Backups and restore

Written 2026-09-06, the day the registry moved off Fly.io. Fly's volume snapshots were the implicit backup until then; nothing replaced them. This document is the replacement.

What has to be backed up

Everything else (the image, Caddy route, DNS) is rebuilt by scripts/deploy-fleet.sh and docs/HANDOVER-registry-migration-to-forge-control.md.

Verifying the backups exist

scripts/verify-backups.sh, on its own cron, at a different hour:

43 9 * * * root /opt/packr-registry/verify-backups.sh >> /var/log/packr-backup.log 2>&1

It exits 1 when the newest dump is older than MAX_AGE_HOURS (default 36), when no dump exists, when the newest one is too small to be a backup, or when the bucket itself is missing — with a distinct message for that last case, because "deleted" and "stale" need different reactions.

It is deliberately not part of backup-db.sh. A check that runs inside the thing it checks stops running when that thing stops, which is exactly when it is needed. If the cron is removed, the credential expires, or the bucket is deleted, backup-db.sh produces no output and no alarm — and until this script existed, nothing anywhere noticed. The registry does not look at backups (grep -rn backup internal/ finds only a docs page), and the success line goes to a log file on app-3 that nobody reads. An absent backup was indistinguishable from a healthy one.

Run it at a different hour from the backup so a saturated host does not make both fail together.

Proving a backup restores

verify-backups.sh proves a dump exists. scripts/restore-drill.sh proves it restores, which is the claim anybody actually wants on the day. An untested backup is a hypothesis.

/opt/packr-registry/restore-drill.sh          # exits 1 if the newest dump will not restore

It pulls the newest dump, restores it into a uniquely-stamped throwaway database, counts the tables that would matter (packages, package_versions, users, tokens), and drops the database again on exit.

It never touches production. The dump is written with pg_dump --clean --if-exists, so it issues DROP statements against whatever it is pointed at — restoring it into the wrong database would destroy the registry. The script refuses to run when the target name matches the production name, and refuses any target not matching packr_restore_drill_*.

It uses the container's local superuser (peer auth inside postgres-shared, no password handled) because the app's role is deliberately not permitted to CREATE DATABASE.

First run, 2026-09-09 — the first time these backups had ever been restored:

restore-drill: restoring app_packr_registry-20260909T031701Z.sql.gz into packr_restore_drill_20260909160800
restore-drill: packages=342 versions=826 users=11 tokens=51
restore-drill: ok

Worth running monthly, and after any change to the dump or the schema.

The bucket's name is not the protection

A _DO_NOT_DELETE_README.txt sits at the bucket root explaining that packr-registry-cache is not a cache and holds the only off-host copy of the metadata, so the next person to open it learns that before deciding what to do with it.

That is mitigation, not a fix. The durable protection is B2 Object Lock on backups/, which makes deletion impossible rather than merely regrettable. One constraint if you enable it: any retention window must stay shorter than the pruning horizon in backup-db.sh (KEEP=30 dumps), or pruning will start failing against locked objects and the nightly job will begin exiting non-zero.

The two scripts above are the other half — they make a deletion visible within a day. Neither prevents it.

Nightly database backup

scripts/backup-db.sh runs on app-3 as root via cron. It reads registry.env (so credentials live in exactly one place) and uses the BACKUP_S3_* entries — Backblaze B2, off-host — falling back to S3_* only when the blob store itself is B2, runs pg_dump --clean --if-exists --no-owner inside postgres-shared, gzips, and streams it to s3://packr-registry-cache/backups/db/<db>-<UTC stamp>.sql.gz with the amazon/aws-cli image (no host install). It refuses to count an upload under 1 KB as a backup and prunes to the newest 30 dumps.

The backups/ prefix is never touched by the mirror-cache eviction, which only walks tarballs/.

Install:

scp scripts/backup-db.sh scripts/restore-db.sh root@116.203.97.105:/opt/packr-registry/
ssh root@116.203.97.105 'chmod 750 /opt/packr-registry/backup-db.sh /opt/packr-registry/restore-db.sh
  echo "17 3 * * * root /opt/packr-registry/backup-db.sh >> /var/log/packr-backup.log 2>&1" > /etc/cron.d/packr-backup
  /opt/packr-registry/backup-db.sh'     # first run now, do not wait for 03:17

Verify: ssh root@116.203.97.105 /opt/packr-registry/restore-db.sh lists the dumps with sizes. Expect ~1–3 MB compressed for the 2026-09 dataset (342 packages / 800 versions / 15k audit rows).

Restore

Always restore into a scratch database first, then check it, then swap:

ssh root@116.203.97.105
/opt/packr-registry/restore-db.sh                                   # list
TARGET_DB=app_packr_registry_restore /opt/packr-registry/restore-db.sh app_packr_registry-20260907T031700Z.sql.gz
# prints packages / versions / users / tokens counts for the scratch DB

To make the scratch copy live (registry stopped for the swap, ~10 s):

cd /opt/packr-registry
docker compose -p packr-registry -f registry.docker-compose.yml stop
docker exec postgres-shared psql -U postgres -c 'ALTER DATABASE app_packr_registry RENAME TO app_packr_registry_broken'
docker exec postgres-shared psql -U postgres -c 'ALTER DATABASE app_packr_registry_restore RENAME TO app_packr_registry'
docker compose -p packr-registry -f registry.docker-compose.yml start
curl -s http://127.0.0.1:4873/health          # status "ok", checks.db.ok true

Restoring straight into the live database (restore-db.sh <object> with no TARGET_DB) also works — the script makes you type the database name and you must stop the registry first — but the rename dance above leaves the broken copy for forensics and is what to reach for at 3 a.m.

Blobs are not part of the dump: tarballs referenced by the restored rows are still in B2. If a blob is missing, the admin backfill (POST /api/v1/admin/maintenance/backfill-checksums, body {"apply":true}, header X-Super-Admin-Token) reports it under errors[].

Recovery objectives

Not formally agreed. What the mechanism above gives you:

• RPO ≤ 24 h for metadata (nightly dump); ~0 for blobs once B2 versioning is on.
• RTO ≈ 15 min: restore-into-scratch + rename + restart, all on one host.

If either number is not good enough, the next step is pg_dump every hour (same script, different cron line, KEEP=72), not a different design.

Testing the backup

Do a restore into a scratch database once a month and compare the row counts the script prints against the live registry's /api/v1/ecosystems/stats. An untested backup is a hope, not a backup.