Backups, restores, and drills

The backup job has reported success every six hours for nine months. During an incident, the newest dump restores, but the application still fails because the cache contains capabilities and job outputs that nobody knew were authoritative. A green backup job was not a recovery plan.

Recovery begins with a state inventory. The drill is the test.

Recovery map

Postgres            -> compressed logical dump -> off-site remote
shared Redis        -> consistent RDB snapshot -> off-site remote
replication cache   -> discard and rebuild
analytics store     -> expire or start empty by policy
release manifests   -> retain prior image digests
secret provider     -> separate recovery procedure

replacement host
  -> fetch secrets
  -> restore authoritative state
  -> rebuild derived state
  -> deploy recorded images
  -> verify user behavior

Terms

Recovery point objective, or RPO, is the maximum acceptable amount of lost recent data.

Recovery time objective, or RTO, is the target time to restore service.

A logical backup exports database objects and rows through the database engine. It is more portable than copying live data files.

A crash-consistent snapshot reflects storage at one moment but may require database recovery on startup.

A restore drill performs recovery in an isolated environment and records evidence.

A rollback returns an application release or data set to an earlier state. Image rollback and data restore are different operations.

Classify state before scripting

For every database, volume, object bucket, and external service, answer four questions:

1. Does losing it change user-visible history or permissions?
2. Can it be rebuilt from another source?
3. How long would rebuilding take?
4. What privacy or retention promise applies?

Postgres is usually authoritative for conversations, schedules, configuration, and deployment metadata. A replication cache can be deleted and rebuilt. Redis requires closer inspection. Shared mini-app documents, push subscriptions, and scheduled job output may need a snapshot. Aggregate analytics can be intentionally disposable and excluded from backups if the privacy page says so.

That last exclusion is a design decision, not negligence. Restoring expired analytics after a user cleared it or after the stated retention window would violate the product contract. A restart may replay local append-only data, while disaster recovery starts analytics empty and rotates to a new generation.

Produce cell-scoped backups

Use a database-native logical dump such as Postgres custom format. Exclude explicitly disposable high-volume tables only after confirming that their absence does not break foreign keys or startup behavior. Stream a consistent Redis snapshot when Redis holds state worth recovering.

Include the instance ID and UTC timestamp in each filename:

alpha-postgres-20260920T180000Z.dump
alpha-redis-20260920T180000Z.rdb

Upload to a remote path that also contains the instance ID. A local retention window, perhaps 14 days, controls disk use. Remote retention should follow the business RPO, legal requirements, and storage budget.

A generic backup command should accept public instance configuration and a separately materialized runtime secret file:

backup-cell deploy/instances/alpha.env \
  /run/agent/alpha/runtime.env

Use umask 077, write to a partial filename, and rename only after success. Validate that required tools exist before creating a misleading local success. Report both the last successful dump time and whether the off-site copy completed.

Encrypt in transit and at rest. If the remote provider already encrypts storage, still decide who can read objects and delete backups. Keep deletion authority narrower than read or write authority where the provider supports it. Account compromise can otherwise erase the host and its recovery copies together.

Restore in dependency order

Restoration is destructive. Require an explicit confirmation flag and validate that the backup belongs to the target instance. Cross-instance restoration can support disaster migration, but it should require a named override.

Before touching production state, verify file integrity. Postgres restore tools validate format as they read. Redis ships an RDB checker. Checksums on uploaded objects detect incomplete transfer, though they do not prove semantic correctness.

A restore sequence usually does this:

1. Stops public traffic and every writer.
2. Restores Postgres in one transaction where supported.
3. Replaces recoverable Redis state if a snapshot exists.
4. Deletes the stale replication cache.
5. Starts Redis, sync, worker, replication, and edge services in order.
6. Waits for cache rebuild and health checks.
7. Exercises signed-in user behavior.

Do not restore a disposable analytics store merely because its volume exists. Recreate configuration markers from Postgres and set a new availability date so reports do not imply coverage during the missing period.

Drill the whole replacement

Run drills on a disposable host, not in the live Compose project. A useful drill starts from an empty machine with only the deployment bundle, instance file, secret adapter, image digests, and backup objects.

Measure:

• time to provision and secure the host
• time to obtain secrets
• download and restore duration
• replication-cache rebuild time
• public DNS or address-switch time
• total time until a real user flow succeeds

Compare the total with the RTO. If DNS dominates, pre-plan a low-TTL incident procedure. If the dump dominates, test compression, storage region, or physical backup options. If secrets dominate, fix the recovery adapter and provider access before the next drill.

Record the image digests used with each deploy. Restoring old data under the newest application can fail if migrations changed assumptions. The drill should test a supported pairing and then the current upgrade path.

Backup before deploy, restore by decision

A full deployment should take a backup before migration. If new containers fail health checks, restore the previous image digests automatically when schema compatibility permits.

Do not automatically restore the pre-deploy database. Users may have written data after the migration, and automatic restoration would silently discard it. The deployer should print the exact backup paths and recovery command. An operator decides whether downtime, forward repair, or data rollback has the lowest cost.

Prefer additive migrations so old and new application versions overlap safely. Database restore remains the last resort, not the ordinary release mechanism.

Choices and rejected shortcuts

Use logical Postgres dumps for portability across replacement hosts. Provider disk snapshots are useful for fast whole-machine recovery, but they tie restoration to one provider and capture application state without understanding it.

Keep backups off-site. Another directory on the VM is staging, not disaster recovery.

Back up shared Redis only when its content cannot be reconstructed. Dumping every cache wastes time and can reintroduce stale state.

Exclude content-free failure ledgers or revision caches when their loss is acceptable and their volume harms RTO. Document every exclusion.

Do not declare success after pg_dump exits zero. Recovery success requires application-level verification on restored state.

Failure modes

• The remote path is unset and every backup stays local.
• Retention deletes the only known-good dump.
• An instance ID typo writes alpha data under beta's prefix.
• The restore script starts writers before Redis replacement finishes.
• The database restores, but the old replication file points at invalid progress.
• Secrets cannot be recovered without the dead host.
• Backup metrics update before remote upload.
• A drill uses a warm host with cached images and hides registry or bootstrap dependencies.

Drill checklist

• Every state store is classified as authoritative, rebuildable, or disposable.
• RPO and RTO are written in hours or minutes.
• Filenames and remote paths include the cell ID and UTC time.
• Dumps use partial files and atomic rename.
• Off-site completion has its own metric and alert.
• Restore rejects cross-cell data unless explicitly overridden.
• Writers stop before replacement and restart in dependency order.
• Replication caches are rebuilt, not blindly restored.
• Privacy-driven backup exclusions match generated disclosures.
• A blank replacement host has passed a timed restore drill.
• Image rollback and database restoration remain separate decisions.

Previous: Chapter 38, "Observability for agents".

Next chapter

Chapter 40, "Analytics without surveillance", applies the same state and retention discipline to public app measurement.