Backups + disaster recovery

Nightly Postgres dumps, weekly restore tests, RTO/RPO targets, the actual runbook for when something breaks.

The minimum-viable safety net: a nightly Postgres dump to S3-compatible storage, verified weekly with a restore test, with documented recovery time and recovery point objectives. Anything less than this is wishful thinking dressed up as a backup strategy.

Warning — Untested backups are not backups

The most expensive lesson in operations is "we have nightly backups" followed by "they don't restore." Test weekly. Make it boring. Then it'll work when you need it.

What gets backed up

Backup targets

Daily Postgres backup

The repo ships infra/scripts/backup-postgres.sh. It:

1. Runs pg_dump --format=custom against $DATABASE_URL.

2. Compresses with gzip -9.

3. Uploads to s3://${S3_BUCKET}/postgres/<stamp>.dump.gz, with SSE-KMS if AWS_KMS_KEY_ID is set.

4. Lists the object back to confirm the upload landed.

5. Prunes anything older than RETENTION_DAYS (default 30).

It does not write a heartbeat object and does not emit a metric — if it stops running, nothing tells you. Alert on its exit code from whatever runs it, and see the monitoring section below.

# /etc/cron.d/chatly-backup — runs at 03:00 UTC daily
0 3 * * * root cd /srv/livechat && \
  DATABASE_URL="$(grep ^DATABASE_URL .env.prod | cut -d= -f2-)" \
  S3_BUCKET=chatly-backups \
  AWS_KMS_KEY_ID=arn:aws:kms:us-east-1:123456789012:key/abc... \
  RETENTION_DAYS=30 \
  ./infra/scripts/backup-postgres.sh >> /var/log/chatly-backup.log 2>&1

For Kubernetes, the Helm chart provides a CronJob:

backups:
  enabled: true
  schedule: "0 3 * * *"
  s3:
    bucket: chatly-backups
    region: us-east-1
    kmsKeyId: arn:aws:kms:...
  retentionDays: 30

RTO / RPO targets

Reference targets (single-region single-VPS)

Warning — An RTO you have not timed is a wish

Chatly's own runbook carried a 30-minute Postgres RTO for a year against a mechanism nobody had ever run end to end. Time a restore of your data and write down what you got. That number is the only one worth telling anyone.

Continuous WAL archiving + point-in-time recovery

A nightly dump can only put you back on a moment a dump happened to be taken. If the incident is "a bad migration ran at 16:12", that costs you everything since the previous night — for every workspace at once. Continuous archiving fixes exactly that case, and the repo ships it.

Postgres fills a 16MB WAL segment, runs archive_command, and may not recycle the segment until that command returns 0. Point that command at pgBackRest and every segment reaches your object storage before it can be lost. A periodic base backup plus every segment since gives you the cluster as it stood at any instant in between.

# 1. Build a Postgres image with pgBackRest in it. archive_command runs INSIDE
#    the Postgres container, so this step is unavoidable.
docker build -f infra/docker/Dockerfile.postgres-pgbackrest \
  --build-arg BASE_IMAGE=<your postgres image> \
  -t <your postgres image>-pgbackrest .

# 2. Run Postgres on that image. Then, with the PGBACKREST_* vars set:
bash infra/scripts/pgbackrest-configure.sh --container=<c>          # dry run
bash infra/scripts/pgbackrest-configure.sh --container=<c> --apply
#    ... restart Postgres once (archive_mode is not reloadable) ...
bash infra/scripts/backup-postgres-wal.sh --init --apply --container=<c>

Then schedule a weekly full and a nightly differential alongside the logical dump:

0 2 * * 0 root cd /srv/chatly && bash infra/scripts/backup-postgres-wal.sh --type=full --apply --container=<c>
0 2 * * 1-6 root cd /srv/chatly && bash infra/scripts/backup-postgres-wal.sh --type=diff --apply --container=<c>
0 3 * * * root cd /srv/chatly && bash infra/scripts/backup-postgres.sh

Recovering to a moment:

bash infra/scripts/restore-postgres-pitr.sh --target=production --container=<c>
#   ^ no --time: prints the recoverable window and exits

bash infra/scripts/restore-postgres-pitr.sh --target=production --container=<c> \
  --time='2026-08-27 16:11:00+00' --apply

Warning — Keep the nightly dump too

These are not redundant. A physical base backup is a byte-for-byte copy, so it faithfully reproduces corruption the cluster already had and cannot restore into a different major version. A logical dump is version-portable and can restore a single table, but its recovery point is whenever it ran. A corrupt- page bug takes the physical chain and not the dump; a botched pg_dump takes the dump and not the chain. Run both.

Prove it works, on a schedule

bash infra/scripts/pitr-drill.sh

Unattended, about half a minute, throwaway containers. It stands up MinIO and Postgres, writes known-good data after a base backup, applies a corrupting write at a known instant, recovers to a moment before it, and asserts the good data is back and the corruption is not — then recovers again to a moment after the corruption and asserts it comes back, so the drill's assertions are known to be capable of failing. Run it after any change to your Postgres image or your object storage.

Weekly restore test

Build a habit. Every Monday, automated via cron:

#!/bin/bash
set -euo pipefail

# 1. Pull yesterday's snapshot
SNAP="postgres/$(date -ud yesterday +%Y-%m-%d)T030000Z.dump.zst"
aws s3 cp "s3://chatly-backups/$SNAP" /tmp/snap.dump.zst

# 2. Spin up a temp Postgres
docker run --rm -d --name pg-verify -e POSTGRES_PASSWORD=test \
  -p 5433:5432 postgres:16
sleep 5

# 3. Restore into it
zstd -d /tmp/snap.dump.zst -o /tmp/snap.dump
PGPASSWORD=test pg_restore -h localhost -p 5433 -U postgres \
  -d postgres --jobs=4 /tmp/snap.dump

# 4. Run the verifier
node scripts/verify-restore.mjs \
  --db "postgresql://postgres:test@localhost:5433/postgres"

# 5. Cleanup
docker stop pg-verify
rm /tmp/snap.dump*

echo "Restore test PASSED for $SNAP"

scripts/verify-restore.mjs checks:

• Row counts within 5% of yesterday's prod (queried via read-replica).

• Latest message timestamp within 24h of the backup time.

• No orphaned workspace_id foreign keys.

• RLS policies exist on every tenant table.

• Migration history matches current code (no schema drift).

Pipe the script's exit code into your alerting — silent failures defeat the purpose.

Real recovery runbook

If you actually need to restore prod:

Single-VPS (docker-compose)

# 0. Page the team. Update status page. Note the time.

# 1. Take prod read-only
docker compose -f docker-compose.prod.yml stop api realtime worker ai voice

# 2. Provision a FRESH database (don't restore over the live one)
#    On managed: spin a new RDS instance from the snapshot.
#    On self-host: create a new DB on the same Postgres, or a new host.

# 3. Restore the latest dump
aws s3 cp s3://chatly-backups/postgres/$(date -u +%Y-%m-%d)T*.dump.zst /tmp/snap.zst
zstd -d /tmp/snap.zst -o /tmp/snap.dump
pg_restore -d "$NEW_DATABASE_URL" --jobs=8 --no-owner /tmp/snap.dump

# 4. Replay outbox events from the gap window (if WAL archiving is on)
#    Otherwise: accept the RPO loss and document what's missing.

# 5. Re-point env
sed -i "s|DATABASE_URL=.*|DATABASE_URL=$NEW_DATABASE_URL|" .env.prod

# 6. Bring services back up
docker compose -f docker-compose.prod.yml --env-file .env.prod up -d

# 7. Run a sanity check
curl -fsSL https://api.chatly.example/health

# 8. Update status page to "operational"

# 9. Postmortem within 24 hours

Kubernetes (Helm)

# 1. Drain traffic: scale services to 0 (or set ingress to 503)
kubectl scale deployment livechat-api --replicas=0
kubectl scale deployment livechat-realtime --replicas=0
# ... worker, ai, voice

# 2. Restore via managed-DB snapshot (RDS, Cloud SQL, etc.) OR a fresh
#    DB + pg_restore from the S3 dump.

# 3. Update the External Secret / Vault binding to point at new DB.

# 4. Scale back up; HPAs take over.
kubectl scale deployment livechat-api --replicas=2
# ...

Operator secrets

MFA_ENCRYPTION_KEY and the JWT keypair are the secrets that cannot be re-derived from the database. If you lose them, the database is useless.

Recommended backup procedure:

1. After generating, encrypt with gpg to two recipients (you + a coworker).

2. Store one copy in a Bitwarden / 1Password vault.

3. Store the second copy in a sealed envelope in a physical safe.

4. Test restoring quarterly. Yes, really.

The cost of doing this right is one hour of paperwork. The cost of not doing it is every BYOK customer credential becoming permanently inaccessible.

Danger — Without MFA_ENCRYPTION_KEY, you lose every workspace's BYOK

Every Twilio key, every LLM key, every IDV secret, every Meta token, every SAML cert — all sealed with this key. The database rows are encrypted with it; restore the DB without the key and the rows decrypt to garbage. Back. It. Up.

Object storage

Use S3 versioning + 30-day soft delete. With both on:

• Accidental object DELETE is undoable.

• Accidental object OVERWRITE leaves the previous version recoverable.

• KMS-encrypted SSE on every object means a stolen S3 access key without the matching KMS grant gets garbage.

Cross-region replication recommended for production — a second bucket in a different region with aws s3 sync (or S3 CRR). Cost is a few percent of the primary bucket.

Off-site copies

The database backups should live in a different cloud or region than the live database. AWS losing a region is rare; AWS losing your account is more common than you think (compromised IAM credentials).

Recommended: replicate the backup bucket to a completely separate AWS account (or to Backblaze B2, Cloudflare R2, Wasabi).

aws s3 sync s3://chatly-backups-primary s3://chatly-backups-dr \
  --source-region us-east-1 \
  --region eu-west-1

Monitoring

Wire these checks into your alerting:

• Heartbeat file freshness: alert if s3://chatly-backups/heartbeat.json is >36h old (cron skipped).

• Backup size sanity: alert if today's dump is less than 50% of the rolling 7-day average (indicates a partial dump).

• Restore test exit code: alert on any failure.

• PITR lag (if WAL archiving): alert if WAL upload lag >5 min.

Compliance

For SOC 2, ISO 27001, HIPAA, you'll need:

• Documented procedures (this page, signed off by your security lead).

• Evidence of weekly restore tests (CI artifacts, dated log entries).

• Quarterly off-site restore drill with a written report.

• Annual full DR drill that simulates a region loss.

The Helm chart's backups.compliance block emits Prometheus metrics that your auditor's evidence-collection tool can consume.