Recovery is a measured requirement: RPO and RTO are proven by a restore, not by having backups

practice
established
fragment

Scope. A drill proves one scenario on one store. The replay-timestamp RPO overestimates loss on an idle primary, a scratch-instance restore says nothing about cutover, DNS or dependent services, and recovering several stores to one consistent moment is a separate design problem. The target numbers themselves are a cost decision: AWS guidance allows no DR at all for workloads where recovery costs more than the loss.

Versions. PostgreSQL 18: pg_verifybackup checks tar-format (including compressed) backups, but WAL parsing works only for plain format, hence -n.

Primary sources: Google SRE Book — Chapter 26: Data Integrity: What You Read Is What You Wrote, AWS Well-Architected — REL13-BP03 Test disaster recovery implementation to validate the implementation

References: AWS Whitepaper — Disaster Recovery of Workloads on AWS: Business Continuity Plan (RTO and RPO definitions), AWS Well-Architected — REL11-BP04 Rely on the data plane and not the control plane during recovery, PostgreSQL 18 — Continuous Archiving and Point-in-Time Recovery (PITR), PostgreSQL 18 — pg_verifybackup

Other claims on this card, typed separately:

- heuristic
- established
Recovery paths that are rarely exercised stop working: AWS states from experience that the only error recovery that works is the path tested frequently, and recommends failing over regularly regardless of need.
Primary sources: AWS Well-Architected — REL13-BP03 Test disaster recovery implementation to validate the implementation
- practice
- established
Replication is not a backup: it propagates logical corruption and deletions to every copy.
Primary sources: Google SRE Book — Chapter 26: Data Integrity: What You Read Is What You Wrote

A backup is an artifact; recovery is a capability. RPO (how much data may be lost) and RTO (how long the service may be down) are business numbers set per workload, and neither is a property of a backup job that exits 0. The only evidence for them is a restore that was actually performed, timed end to end and checked for content. Until then “we have backups” is a hypothesis about a procedure nobody has run.

Where the numbers actually come from. Achieved RPO is not the backup schedule but the lag of the last durable, restorable point: for PITR that is the base backup plus an unbroken WAL sequence up to the failure, and a single missing segment ends replay there. RTO is the sum of everything on the path — finding the right backup, fetching it, restoring, replaying, validating, and cutting traffic over — and replay time grows with the distance to the last base backup. Replication does not replace any of this: it carries a DROP TABLE or a corrupting bug to every replica within seconds; the defence against logical damage is a point in time before it. Restoring one store is also not restoring a system: a database and an object store recovered to different moments leave dangling references, the same boundary problem as in 68.

fragment

Bash · PostgreSQL 18

#!/usr/bin/env bash
set -euo pipefail
t0=$(date +%s)                         # the simulated disaster happens now
backup=/backups/latest                 # pg_basebackup -Ft -z --wal-method=stream
pgdata=/drill/pgdata

# 1. Integrity against the manifest (tar format cannot have its WAL parsed, hence -n).
#    Necessary, not sufficient: the PostgreSQL docs still require a real restore.
pg_verifybackup -n "$backup"

# 2. Restore into a scratch instance and replay the archive to its end.
rm -rf "$pgdata" && install -d -m 700 "$pgdata"
tar -xzf "$backup/base.tar.gz" -C "$pgdata"
tar -xzf "$backup/pg_wal.tar.gz" -C "$pgdata/pg_wal"
cat >> "$pgdata/postgresql.auto.conf" <<'CONF'
restore_command = 'cp /archive/%f "%p"'
archive_mode = off   # a promoted drill copy must never write into the production archive
CONF
touch "$pgdata/recovery.signal"      # no recovery target: replay everything, then promote
pg_ctl -D "$pgdata" -o '-p 5499' -l /drill/restore.log -W start
until [ "$(psql -p 5499 -d postgres -Atc 'select pg_is_in_recovery()' 2>/dev/null)" = f ]; do
  sleep 5
done

# 3. Measure the promise instead of asserting it. The RPO figure overestimates
#    the loss when the primary was idle before t0; RTO is wall-clock to a promoted server.
psql -p 5499 -d app -v ON_ERROR_STOP=1 -At <<SQL
select 'rpo_s', extract(epoch from to_timestamp($t0) - pg_last_xact_replay_timestamp());
select 'rto_s', extract(epoch from now()) - $t0;
select 'orders_last_hour', count(*) from orders
 where created_at > to_timestamp($t0) - interval '1 hour';
SQL

Failure modes a drill finds and a dashboard does not. An archive command that returns 0 without having stored the file: PostgreSQL then recycles the segment and the hole surfaces only at restore time. Encryption keys, credentials or the runbook itself living inside the system being recovered. And recovery that depends on the control plane: provisioning instances, changing DNS records, creating resources — APIs that can be impaired while the data plane keeps serving. AWS guidance is explicit here: recover with data-plane actions and pre-provision what recovery needs (101, 105).

Decision rule. A recovery claim is the measured result of the last drill, with its date, for example “on <date>: restored to <N> s before t0, promoted in <M> min, row counts matched” — not a retention setting. Drills run on a schedule, in automation, against the real backup location, and a failed drill pages like an outage, because it is one that has not happened yet. If the measured RTO exceeds the target, the fix is architectural (more frequent base backups, warm standby, smaller restore units via cells, 27) and not a more optimistic runbook.