This is the multi-page printable view of this section. .
Point-in-Time Recovery — A Time Machine for PostgreSQL
If data, a table, or even a database is deleted accidentally, Point-in-Time Recovery (PITR) can return the cluster to an earlier state.
This capability, once treated as specialist DBA work, is enabled by Pigsty’s standard PostgreSQL configuration.
Replication Is Not Backup
High availability can fail over to another instance when hardware fails. It has a natural blind spot, however: replication is not backup.
Streaming replication faithfully sends every primary change to every replica within milliseconds, including a DELETE without a WHERE clause or a DROP TABLE issued against the wrong database. Failover handles a broken machine; when the data itself is wrong, every replica can contain the same error.
Database disasters therefore fall into two broad classes. Redundancy handles physical service failure through multiple copies and automatic failover. Logical errors require history: a base backup plus continuous WAL archives from which PostgreSQL can reconstruct a state before the mistake.
| Threat | High Availability | Delayed Cluster | PITR |
|---|---|---|---|
| Hardware or instance failure | ✔ Automatic failover | ✘ | ✔, with a longer RTO |
| Accidental DML, table drop, or database drop | ✘ The error is replicated | ✔ Within the delay | ✔ At any recoverable point |
| Defective software corrupts data over time | ✘ The error is replicated | ✔ Within the delay | ✔ Try different recovery targets |
| Entire cluster or site is lost | ✘ | ✘ | ✔ Only if the repository survives that failure domain |
These mechanisms complement one another: HA restores service quickly, a delayed cluster provides a short undo window, and PITR is the final historical recovery path.
How the Time Machine Works
A database can be viewed as a state machine. A base backup is a complete physical snapshot at one point, while WAL (Write-Ahead Log) records every subsequent state change. With a snapshot and an unbroken WAL history starting from it, PostgreSQL can replay the database to any target covered by that history. The backup determines how far back recovery can start; the latest archived WAL determines how close to the present it can reach.
Base backup + WAL archive = point-in-time recovery
Pigsty orchestrates both inputs. Cluster initialization attempts an initial full backup by default, and the primary continuously sends completed WAL segments to the selected repository. See How PITR Works for the complete model of backups, archives, targets, and timelines.
Available Out of the Box
PITR is enabled in Pigsty’s standard PostgreSQL configuration. Each cluster is prepared with a backup repository, WAL archiving, and recovery tooling powered by pgBackRest. The policy remains declarative and can be customized with a few parameters:
The default local method stores backups under /pg/backup and retains two full backups. With one successful full backup per day, the resulting window is roughly 24–48 hours. Selecting the remote minio preset places the repository in Silo or compatible S3 storage, enables AES-256-CBC repository encryption, and uses time-based retention. With a 14-day retention setting and weekly full backups, the steady-state recovery window is roughly 14–21 days. Treat both ranges as policy estimates: actual coverage starts at the oldest usable backup and ends at the latest WAL that reached the repository.
Recovery is declarative too: specify a target, then let the playbook stop the cluster, restore files, replay WAL, and rebuild HA. An operator must still verify the recovered business state.
This follows Pigsty’s declarative configuration model: backup policy is part of the cluster definition, and a recovery target is another declared parameter.
Benefits and Costs
PITR materially improves data integrity and availability:
- RPO (maximum data loss) is usually reduced to minutes, bounded by WAL that had not reached a surviving repository.
- RTO (time to restore service) becomes tens of minutes to hours rather than permanent loss, depending on backup size, WAL replay distance, and disk or network throughput.
| Standalone strategy | Event | RTO | RPO |
|---|---|---|---|
| No backup | Host and local data are lost | Permanent loss | All data |
| Base backups only | Host and local data are lost | Backup size and bandwidth, often hours | Changes since the latest backup |
| Base backups + WAL archives | Host and local data are lost | Backup size, replay distance, and bandwidth | WAL not yet present in the surviving repository |
The costs fall mainly into three areas:
- Confidentiality: backups are another copy of business data and need encryption and access control. Pigsty’s remote preset enables repository encryption, but its default password must be changed.
- Resources: backups consume storage and archiving consumes bandwidth. Compression, bundling, and block incremental backup reduce this cost but do not eliminate capacity planning.
- Operations: backup status must be monitored and recovery must be rehearsed. A green backup job alone is not proof that the data can be restored within the required RTO.
PITR by itself does not replace HA. A production design normally combines HA for physical failures with PITR for logical errors and site-level recovery.
Next Steps
- How PITR Works: snapshots, WAL history, recovery windows, targets, and timelines
- PITR Architecture: pgBackRest, repository selection, archive flow, scheduling, and failover behavior
- PITR Tradeoffs: failure domains, capacity, retention, and backup frequency
- Declarative Recovery: the
pg_pitrparameter,pgsql-pitr.yml, andpig pitr - PITR Scenarios: accidental deletion, bad releases, investigation, and site loss
For the operational runbooks, see PGSQL Backup and Recovery.
1 - How PITR Works
If a database is a state machine, WAL (Write-Ahead Log) is its ordered change history. PostgreSQL records each modification in WAL before applying it to data files. Save a physical snapshot at one point, preserve all later WAL, and PostgreSQL can replay that history to a selected consistent state.
PITR is therefore the combination of three simple elements: a snapshot (base backup), history (WAL archive), and a target (where replay should stop).
Snapshot: Base Backup
A base backup is a physical snapshot of the whole PostgreSQL cluster and supplies a starting point for recovery. Pigsty uses pgBackRest to create and manage three backup types:
| Type | Contents | Recovery characteristics |
|---|---|---|
| Full | All database-cluster files | Self-contained, shortest chain, largest backup |
| Differential | Changes since the latest full backup | Restore uses the full plus the differential |
| Incremental | Changes since the latest backup of any type | Smallest backup, restore depends on its complete chain |
The wrapper pg-backup [full|diff|incr] triggers a backup. With no argument it requests incr; pgBackRest creates a full backup instead when no valid full exists. pg_crontab declares recurring jobs and installs them in the postgres user’s crontab.
Backup frequency affects recovery time: the newer the usable backup, the less WAL must be replayed to reach a given target. See PITR Tradeoffs.
WAL History
A snapshot reaches only its own state. WAL archiving preserves every later change needed to advance beyond it. Pigsty’s standard Patroni templates enable archiving and ask PostgreSQL to hand each completed WAL segment to pgBackRest:
Two implementation details matter:
archive_timeout: 300: on a low-write cluster, PostgreSQL can force a segment switch after five minutes so a partially filled segment does not wait indefinitely. This normally keeps the right edge of the recovery window within minutes when WAL is being generated; it is not a promise that every commit is already remote.- Asynchronous archive: pgBackRest uses
/pg/spoolwitharchive-async=yto batch transfers. Pigsty setsarchive-push-queue-max=4GiB; if repository failure lets the queue cross that bound, pgBackRest can drop the queued WAL to protect local disk. That creates an archive gap, so a new full backup is required to establish a fresh recoverable chain.
Expiration is automatic. When old backups expire under the repository policy, pgBackRest also expires archived WAL that no remaining backup needs, unless archive retention is overridden explicitly.
Recovery Window
The backup and its continuous WAL history form a recovery window:
- Left boundary: the start of the oldest usable remaining backup chain. In practical time-based descriptions, this is usually summarized by the oldest retained full backup’s time.
- Right boundary: the latest WAL successfully archived to a repository that survives the incident.
The window moves forward as new backups arrive and old chains expire. Pigsty’s local preset keeps two full backups; with one successful full per day, coverage is roughly one to two days. The minio preset uses retention_full_type: time with retention_full: 14; with weekly full backups, the oldest retained chain normally yields roughly 14–21 days of steady-state coverage. These are estimates, not SLAs: missed backups, archive gaps, explicit archive-retention overrides, or repository loss change the actual window. Verify it with pig pb info and restore drills.
See PITR Tradeoffs and Backup Policy.
Targets: Where Replay Stops
PostgreSQL supports several ways to locate a state inside the recovery window. Pigsty exposes six target types through pg_pitr:
pg_pitr type |
Meaning | Typical use |
|---|---|---|
default |
Replay through all WAL available from the repository | Restore the newest archived state after total loss |
time |
Stop at a timestamp | Recover from accidental DML or DDL |
xid |
Stop at a transaction ID | Exclude a precisely identified bad transaction |
lsn |
Stop at a WAL location | Low-level exact targeting |
name |
Stop at a restore point created with pg_create_restore_point() |
Planned change checkpoint |
immediate |
Stop as soon as the selected backup becomes consistent | Validate or expose the selected backup state quickly |
The set field is different: it chooses which backup set pgBackRest restores as the starting snapshot; it is not itself a replay stop target.
Boundary Semantics
Targets are inclusive by default: the transaction at the target is retained. To stop immediately before a known bad target, set exclusive: true, which maps to recovery_target_inclusive = false.
Transactions remain atomic. Committed transactions before the effective target survive; transactions not committed at that point are rolled back. Recovery produces a consistent database state rather than half of a transaction.
Timelines
Restoring to the past and accepting new writes creates a fork in history. PostgreSQL uses a timeline to distinguish each branch. PITR promotion, replica promotion, and failover can all create a new timeline; new WAL does not overwrite the old timeline’s files.
gitGraph
commit id: "Full backup"
commit id: "Normal writes"
commit id: "Bad change"
commit id: "More writes"
branch Timeline-2
checkout Timeline-2
commit id: "PITR before bad change"
commit id: "New writes"
Keeping the old history allows another attempt if the first target was wrong. The timeline field can select a timeline; Pigsty’s recovery declaration defaults to latest.
Continue with PITR Architecture to see how these concepts map to Pigsty components and configuration.
2 - PITR Architecture
The PITR principle is compact; the engineering is not. WAL archiving must not stall production writes, object-storage backups need encryption, backup jobs must follow the primary after failover, shared repositories must isolate clusters, and large numbers of small objects can limit throughput.
Pigsty uses pgBackRest as its backup engine and ships production-oriented defaults for those concerns. This page describes the engine, repository abstraction, archive path, scheduler, and primary-aware execution model.
Backup Engine: pgBackRest
Pigsty uses pgBackRest for three responsibilities: create base backups with backup, receive WAL with archive-push, and restore data with restore plus archive-get.
Relevant capabilities include:
- Parallelism: backup, archive, and restore operations can use multiple processes.
- Backup chains: full, differential, incremental, and block incremental backups reduce repeated transfer and storage.
- Compression and encryption: zstd compression and AES-256-CBC repository encryption are built in.
- Repository backends: POSIX filesystems, S3-compatible services such as Silo and MinIO, Azure, GCS, and SFTP are supported by pgBackRest.
- Bundling: small files can be packed into larger repository objects, reducing object-storage overhead.
pgBackRest separates cluster histories using a stanza. Pigsty maps the stanza name directly to pg_cluster, allowing multiple clusters to share one storage service without sharing a backup identity:
Repository Abstraction
Two parameters define repository selection. pgbackrest_method chooses one repository name, and pgbackrest_repo is a dictionary of candidate definitions. Pigsty v4.5.0 renders only the selected pgbackrest_repo[pgbackrest_method] entry as pgBackRest repo1; listing both local and minio does not enable two active repositories.
The presets intentionally differ. local favors simplicity and fast local restore; it is unencrypted, unbundled, and retained by full-backup count. minio targets a remote Silo or compatible S3 repository, enabling encryption, bundles, block incremental backup, and time-based retention.
Rendering is mechanical: underscores in the chosen repository’s keys become hyphens and each key gets a repo1- prefix in /etc/pgbackrest/pgbackrest.conf. A custom cloud repository can therefore use pgBackRest options directly:
See Backup Repository for Silo, external S3-compatible storage, versioning, object locking, TLS, and credential details.
Archiving and Scheduling
When pgbackrest_enabled is true, as it is by default, the Patroni templates configure:
Base backups enter the system in two ways:
- Initial backup: after bootstrapping a top-level primary, Pigsty attempts a backup when
pgbackrest_init_backupis true. The task ignores backup failure and writes/etc/pgbackrest/initial.doneonly after success, so the marker means “completed,” not merely “attempted.” - Scheduled backup:
pg_crontabinstalls jobs in the database superuser’s crontab. Its role default is an empty list; standard example configurations usually add a daily 01:00 full backup.
pg-backup [full|diff|incr] is a small wrapper around pgbackrest backup. With no argument it requests an incremental backup, which pgBackRest promotes to a full backup if no usable full exists.
Backups Follow the Primary
pgBackRest and the same scheduled job are installed on every PostgreSQL node, but pg-backup checks /pg/bin/pg-role and only proceeds on the current primary. Replicas fail fast rather than writing a competing backup.
That design decouples the backup schedule from the HA topology:
- all members receive the same repository configuration and crontab;
- after failover, the new primary becomes eligible for subsequent backups and WAL archiving without rewriting the schedule;
- one current primary owns the authoritative write flow to a stanza.
With a non-local repository, Pigsty also adds pgBackRest after basebackup in Patroni’s create_replica_methods. Patroni tries basebackup first; if that method fails, it can restore a replica from the repository with pgbackrest --delta restore, shifting the copy load away from the primary.
Performance Defaults
The shipped pgBackRest template favors light production overhead and aggressive restore throughput:
| Setting | v4.5.0 behavior | Rationale |
|---|---|---|
| Compression | compress-type=zst |
Balance compression ratio and throughput |
| Backup/archive workers | One quarter of CPU, clamped to 2–4 | Limit competition with the database |
| Restore workers | All detected CPU, capped at 8 | Minimize restore time |
| Asynchronous archive | archive-async=y, spool under /pg/spool |
Batch transfer without synchronous object-store latency |
| Archive queue limit | archive-push-queue-max=4GiB |
Bound local spool growth |
| Fast backup start | start-fast=y |
Request an immediate checkpoint |
| Incremental restore | delta=y |
Reuse destination files that already match |
The 4 GiB queue is a safety tradeoff: if the repository remains unavailable and the queue exceeds the limit, pgBackRest can discard queued archive files. PostgreSQL continues running, but the WAL archive becomes incomplete and a new full backup is needed to establish a new recovery chain. See How PITR Works.
Observability
When both backup and exporter settings are enabled, pgbackrest_exporter runs on each PostgreSQL node and exposes metrics on port 9854. The monitoring stack uses those metrics for backup age, type, size, duration, and error visibility.
Useful diagnostic entry points include:
| Entry | Purpose |
|---|---|
pb info |
Shell helper for pgbackrest info using the configured stanza |
/pg/log/pgbackrest/ |
pgBackRest backup, archive, and restore logs |
| `pg-backup full | diff |
See Backup Administration for operational checks, then PITR Tradeoffs for policy design.
3 - PITR Tradeoffs
A backup is an insurance policy. Its premium is storage, network traffic, and operational work; its benefit is how much data can be recovered and how quickly service can return. There is no universal free policy: more history normally needs more capacity, while a shorter RTO normally needs newer backups and tested procedures.
Designing a policy means answering three questions: where is the repository, how long is history retained, and how often are backups taken?
Where: Choose the Failure Domain
Repository location is the most important decision because it defines which disasters the backup survives.
A local repository (pgbackrest_method: local) stores backups on the primary’s local filesystem. It is simple, fast, and has no remote service dependency. But data and backup normally share one host failure domain: loss of the machine, disk, or filesystem can destroy both. Local backup protects well against logical errors, but not total host loss unless /pg/backup is deliberately placed on independent storage.
An object-storage repository (pgbackrest_method: minio or a custom S3 definition) sends backups to Silo or S3. It becomes an independent disaster-recovery copy only when deployed outside the database host or site failure domain. Pigsty’s minio preset also enables AES-256-CBC repository encryption, bundling, and block incremental backup. Recovery throughput then depends on the network and storage service, and that service adds operational responsibility.
| Scenario | Recommended repository | Reason |
|---|---|---|
| Development, test, demo | local |
Minimal dependencies; rebuild is acceptable |
| Production | Dedicated Silo or compatible S3 storage | Independent failure domain and encrypted repository |
| Cloud deployment | Managed S3-compatible or cloud object storage supported by pgBackRest | Independent storage and lower operational burden |
| Ransomware/compliance | Versioned storage plus correctly configured object lock/retention | Prevent privileged database-host access from deleting protected versions |
The backup repository is itself sensitive business data. Change the default access keys and cipher_pass, restrict access, protect credentials separately from the database hosts, and verify any object-lock policy. See Backup Repository.
How Long: Capacity and Recovery Window
Longer retained history generally consumes more storage, but compression, deduplication, block incremental backup, database change rate, and the mix of full/differential/incremental backups determine the actual amount. Measure real backup and WAL growth instead of relying on a fixed multiplier.
For an illustrative 100 GB database changing by 10 GB per day, before compression:
- Daily full, retain two (
localpreset policy): about 200 GB of full backups plus WAL, commonly giving roughly a one-to-two-day window when every job succeeds. - Weekly full, daily incremental, retain full history by 14 days (
miniopreset policy): the oldest surviving weekly chain commonly produces roughly 14–21 days of coverage. Capacity must include multiple full backups, their incrementals, archived WAL, and transient retention-plus-one behavior during expiration.
The precise window is not the configuration number alone. It runs from the oldest usable backup chain to the newest WAL present in the surviving repository. pgBackRest’s time retention removes an old full only when another qualifying full can satisfy the period, and related incrementals and WAL follow the retained full chains. Check pig pb info, monitor archive health, and prove coverage with a restore.
Choose a window long enough to cover the delay between an error occurring and being detected. A dropped table may be noticed in minutes; slow corruption or a month-end reconciliation failure can take weeks to surface.
How Often: Backup Frequency and RTO
Restore time has two main components: restore a backup chain, then replay WAL to the target. Backup size and storage throughput shape the first; the distance between the chosen backup and target shapes the second.
WAL replay is largely serial. On a write-heavy database, restoring from a weekly full immediately before the next full can require nearly a week of replay. Daily incremental backups reduce that replay distance while transferring only changes since the previous backup. They still depend on a valid chain, so monitor and test the entire chain rather than only the newest file.
A useful rule is: within the available backup window and production load budget, take backups often enough that measured restore time meets the RTO.
Pigsty Presets
Pigsty provides two candidate repository definitions, but pgbackrest_method selects one for the generated repo1 configuration.
Standard policy: local repository and daily full backup. It is simple and restores through local I/O, making it suitable for development or environments where host-level disaster recovery is provided separately:
Production policy: remote Silo/S3 repository, weekly full, daily incremental. It separates the repository failure domain and uses the encrypted minio preset:
Do not describe the default pgbackrest_repo dictionary as a “dual-repository” setup: it contains alternative definitions, and the template renders only pgbackrest_repo[pgbackrest_method] as repo1. A real multi-repository pgBackRest design requires explicit advanced configuration and an independently tested backup, expiration, and restore workflow; the two Pigsty presets alone do not create it.
Use Backup Policy for capacity modelling and schedule details.
A Backup Is Proven by Restore
Monitoring a successful backup job is necessary but insufficient. Add clone restore drills to routine operations so you can answer:
- Is the chain usable? Restore it end to end and validate data.
- What is the measured RTO? Database size and WAL volume change over time.
- Can the on-call operator execute the runbook? The first full exercise should not happen during an incident.
A clone recovery leaves the source cluster online but overwrites the designated destination cluster, so verify the exact target and use disposable infrastructure. See Declarative Recovery for the recovery interface.
4 - Declarative Recovery
The value of a backup system is realized at restore time, often during an incident when every minute matters. A traditional PITR procedure requires a long sequence of coupled manual steps: pause HA, stop PostgreSQL, prepare recovery settings, restore the backup, replay WAL, validate the target, rebuild metadata, and start the cluster again.
Pigsty applies the same approach used by declarative configuration to recovery: declare the recovery target, then let the orchestration tools stop the cluster, restore the data, replay WAL, and return control to the operator.
Declare a Recovery Target
Describe the target with the pg_pitr parameter and execute it with pgsql-pitr.yml.
The most common form restores a cluster to a specific time:
The six recovery target types and the rest of the recovery behavior are expressed through fields in this parameter:
See Restore Operations for the complete field reference and examples.
What the Playbook Does
pgsql-pitr.yml turns the manual recovery workflow into six stages and supports Ansible tags for staged execution:
| Stage | Action |
|---|---|
| Print the source cluster, target, and restore command; this stage reports the plan and does not prompt for confirmation | |
| pause | Run patronictl pause so Patroni does not intervene during maintenance |
| stop | Stop Patroni and PostgreSQL on replicas, then on the primary |
| pitr | Render recovery settings, run an incremental pgBackRest restore, start PostgreSQL to replay WAL, wait for consistency, and print control data |
| etcd | Remove stale cluster metadata from etcd so old and new timelines are not mixed |
| start | Start Patroni again, resume HA management, and rebuild replicas |
Several details are important:
- Incremental restore: pgBackRest uses
delta, so it rewrites only files that differ from the backup. For large databases, this can reduce RTO substantially. - Verification, not assumption: the playbook prints checkpoint LSN, timeline, and NextXID data from
pg_controldata; an operator must still verify that the recovered business state is correct. - Rollback copy: with
backup: true, the original data directory is moved to/pg/data-backupbefore recovery. A later run withbackup: trueremoves an existing/pg/data-backup, so this is not a versioned snapshot store. - Staged execution: run
-t down,-t pitr, and-t upseparately when you want an operator checkpoint between phases. Completion of thepitrphase means PostgreSQL reached a consistent recovery state; for a time, XID, LSN, or named target, also confirm WAL replay reached that target.
The action field controls what happens at the target: promote opens a new timeline, pause waits at the target for inspection, and shutdown stops PostgreSQL there.
A targeted recovery defaults to pause when action is omitted. To preserve a manual gate for pause or shutdown, run the stages separately; a one-shot recovery should choose promote explicitly.
The playbook performs the mechanical workflow, but it cannot decide whether the recovered data is correct.
Command-Line Recovery with pig
The pig CLI provides single-instance PITR orchestration directly on a database node, without requiring the management node or an Ansible environment:
pig pitr validates the target, stanza, and available backups; stops Patroni and PostgreSQL; performs the restore; optionally starts PostgreSQL; and prints post-recovery instructions.
For a Patroni-managed data directory, Patroni remains stopped afterward. Validate the data, then use pig pt start to return the instance to HA management.
This single-node workflow does not clear etcd, rebuild replicas, or automatically rejoin the cluster, and it refuses destructive forced shutdown unless --force-stop is supplied explicitly.
The lower-level pig pb commands wrap pgBackRest: pb info lists backups, pb backup creates a backup, and pb restore performs a raw restore.
There is a deliberate safety boundary: pig pb restore refuses to run while Patroni still manages the instance, because Patroni could restart PostgreSQL during the restore.
Use pig pitr or pgsql-pitr.yml for Patroni-managed instances.
In-Place and Clone Recovery
The same mechanism supports two different workflows:
| Dimension | In-place recovery | Clone recovery |
|---|---|---|
| Method | Roll the production cluster back | Restore a source backup into a different cluster |
| Downtime | Required during recovery | The source production cluster remains online |
| Effect | Discards all writes after the target | Does not affect the source; the destination is overwritten and can be retried |
| Best for | Whole-cluster corruption or disaster recovery | Recovering deleted objects, audit work, and recovery drills |
For a clone recovery, the cluster field names the source backup stanza.
This example restores the historical state of pg-meta into pg-test without stopping the source cluster:
Exporting an accidentally deleted table from the clone and importing it into production is generally safer than rolling the entire production cluster back. See Clone a Database Cluster for the complete workflow and cleanup steps.
After Recovery
Recovery completion is not the end of the incident. Include these steps in the closeout checklist:
- New timeline, new backup: after promotion, create a full backup with
pg-backup fullso a recoverable window exists on the new timeline. - Archiving state: if an exploratory restore used
archive: false, restore normal archiving as described in Post-Recovery. - Clone cleanup: a clone’s cluster identity and source backup stanza do not match. Recreate the destination stanza before enabling its own backups; see Clone a Database Cluster.
The tools execute the procedure; operators still decide the target, whether to restore in place or into a clone, and whether the recovered data is correct. Continue with PITR Scenarios for that decision framework.
5 - PITR Scenarios
During an incident, the most expensive resource is often decision time. Pigsty can orchestrate the mechanical recovery steps, but an operator must still answer three questions: what is the target, should recovery be in place or into a clone, and how will the result be validated?
Read and rehearse this framework before an incident.
Decision Framework
| Scenario | Typical problem | Recommended workflow | Target |
|---|---|---|---|
| Accidental DML | DELETE or UPDATE affects the wrong rows |
Clone, validate, then copy back data | time / xid |
| Dropped table, schema, or database | DROP or an incorrect migration |
Clone, validate, then copy back objects | time / name |
| Defective release or batch corruption | Software writes incorrect data for a period | Clone and compare before choosing repair or cutover | time / xid |
| Audit, investigation, or forensics | Inspect historical state | Clone and hold at the target for inspection | time / lsn |
| Whole-cluster or site loss | Hosts or storage are gone or encrypted | Recover in place on replacement infrastructure | default / time |
Two principles apply throughout:
- Stop the damage first. Pause the defective application or remove its write access before choosing a target. The window is moving, but a rushed restore to the wrong cluster can cause a second incident.
- Prefer a clone while production is usable. It leaves the source untouched, supports repeated target selection, and allows validation before export or cutover. It does overwrite the designated destination cluster. In-place recovery is appropriate when the whole cluster is unusable or the business has explicitly accepted rolling every database back.
flowchart TD
A["Data error detected"] --> B["Contain the source of bad writes"]
B --> C{"Can production still serve?"}
C -->|Yes| D["Clone recovery<br/>validate and copy back or cut over"]
C -->|No| E["In-place recovery<br/>or rebuild on new infrastructure"]
D --> F["Validate, take a new backup, review the incident"]
E --> F
Accidental DML
A DELETE without WHERE, an incorrect UPDATE, or a defective batch job is the most common PITR use case.
First locate the error using application logs, PostgreSQL logs, metrics, or audit records. A timestamp is usually sufficient. If the exact transaction ID is known, xid plus exclusive: true can stop immediately before that transaction.
Validate the recovered rows, then copy only the required data back with pg_dump, COPY, or an application-specific reconciliation procedure. If a configured delayed cluster is still inside its delay window, reading from it may be faster than PITR.
Dropped Objects
The same approach applies to DROP TABLE, DROP DATABASE, or a migration executed in the wrong environment, with an even stronger preference for a clone. Rolling the entire production cluster back to recover one object also discards every legitimate write after the target.
Restore a separate destination to before the DDL, validate the object, export it with pg_dump, and import it into production. For planned high-risk changes, create a named restore point with pg_create_restore_point() beforehand; a name target then removes timestamp ambiguity.
Defective Release or Batch Corruption
When a faulty release corrupts data for hours, the challenge is usually identifying the last clean state and the full impact. A clone provides a clean comparison set. Restore repeatedly to candidate times, compare it with production, and decide whether to copy back corrected rows or cut over to a recovered cluster.
This decision needs application-owner validation: a successful PostgreSQL restore proves consistency at a target, not that the target represents correct business state.
Audit and Investigation
Questions such as “what was this balance at month end?” require historical state. Restore into a separate destination, stop at a time, LSN, XID, or named restore point, and inspect without altering the source.
action: pause is the targeted-restore default and holds recovery at the target for inspection; it does not itself configure read-only access or create a separate cluster. The inventory limit and cluster source field determine the destination workflow. Run -t down, -t pitr, and -t up separately when you need an operator gate before promotion, and enforce read-only access explicitly if the investigation requires it. immediate means “stop at the first consistent point,” not “choose a historical timestamp.”
Site Loss
If every database host is destroyed or encrypted, HA cannot help. Recovery requires a repository and the other control-plane assets to have survived outside that failure domain. That survivor can be Silo/S3, another protected host or filesystem, or another tested pgBackRest backend; a remote object store is recommended but the essential property is independent failure-domain survival.
Rebuild hosts, restore the declarative inventory, credentials, and PKI, point the cluster at the surviving repository, then restore through the end of archived WAL:
Inventory and backup data are necessary but not sufficient. Preserve installation media or package repositories, repository credentials and encryption passwords, CA material, custom files, DNS dependencies, and an independently accessible runbook. Keep secrets encrypted and separate from both the database hosts and ordinary source control.
Make Recovery a Routine Drill
The first end-to-end execution of any of these workflows should not occur during a production incident. Use a disposable destination to rehearse clone recovery regularly and after material architecture changes. Measure three outcomes:
- Usability: can the backup and complete WAL chain be restored and validated?
- RTO: how long does the actual restore and replay take now?
- Operator readiness: can the on-call engineer identify source and destination, select a target, and follow the safety gates?
See Restore Operations and Clone a Database Cluster for the task-level runbooks.