Off-site backup scripts (rig-gitops#897, P4)

deploy/backup-r2/scripts/backup-db.sh and lib.sh run in the rig-backup image. They are inert: P5 runs them from suspended CronJobs, and P9 unsuspends them. How they are deployed, configured and mounted: Off-site backups to R2. lib.sh also serves P5's agent-home job.

Inputs

Input	Where from
LOCATION, R2_ENDPOINT (https://<32-hex account>.eu.r2.cloudflarestorage.com), R2_BUCKET	Config (P5 placeholders, P7 Dell patch). Each is checked against a fixed pattern. The eu is deliberate: the plan puts the bucket in R2's EU jurisdiction, the GCS bucket's residency, and any other endpoint fails closed
AGE_RECIPIENT, AGE_RECIPIENT_PIN	Config. Both must be the same 62-character X25519 recipient. Commit them in different places (for example the pin in rig-conductor deploy/backup-r2/, the value in the rig-gitops Dell patch), so that one edit can't swap the recipient
BACKUP_EXCLUDE_DBS	Config: databases deliberately not dumped, space-separated
PGHOST, PGUSER	Default postgres and conductor (I2)
/secrets/r2-writer/AWS_ACCESS_KEY_ID, …/AWS_SECRET_ACCESS_KEY	Secret rig-backup-r2-writer (P8)
/secrets/rig-conductor/postgres-password, …/discord-webhook-url	rig-conductor-secrets (I2). P5 must mount these two keys with items:, never the whole Secret
/work, /tmp	emptyDirs: ciphertext only, and the pgpass copy (memory-backed)

What a run does

1. Checks every input. Credentials only ever reach curl through --config <(…), a pipe, and the password only reaches libpq through a 0600 pgpass copy in /tmp. Any bad value except the webhook stops the run before anything is dumped.
2. Fails unless pg_dump's major version is at least the server's.
3. Dumps rig_control first, then each tenant in tenants.mt_tenant_databases (its evt DB, and rig_t_<id>_mem if it exists), then every rig_drill_* DB. A failed dump is counted and the run goes on (rc#2321).
4. For each DB, a psql session runs BEGIN ISOLATION LEVEL REPEATABLE READ, READ ONLY, exports a snapshot and counts in it: rows per table, non-null vector values per column, the vector version, every hnsw and ivfflat index definition with indisvalid. pg_dump -Fc --snapshot=<id> then dumps that same snapshot, straight into age -r "$AGE_RECIPIENT" -o ….
5. Fails if any database in pg_database was neither dumped nor excluded by name.
6. PUTs the encrypted manifest.json.age, then SHA256SUMS, if every PUT so far returned 200. Just before SHA256SUMS goes its signature, SHA256SUMS.sig (P6), so a restore can prove the run is the job's: see Restoring from R2.
7. On any failure: posts 🔴 [<location>] rig-backup-db <run> … to Discord and exits 1. Silent on success.

Objects follow the plan's layout: backups/<tenant>/<location>/<run>/{evt,mem}.dump.age, with _control and _drill in place of the tenant, and _runs/<location>/db/<run>/{manifest.json.age,SHA256SUMS}. Objects over 1 GiB go up as <key>.p000, .p001, …. Each line of SHA256SUMS is <sha256> <bytes> <key> for one uploaded object.

Security review conditions

The review is docs/ai-tooling-registry/backup/2026-10-03-security-review-rig-backup-age-jq-curl.md in Stig-Johnny/claude-3, the AI-tooling registry where the plan's P0 puts it, as for the image. claude-3 is private, and Review-E's app can't read it (deny-list). So each condition the scripts rely on is quoted here, under the number their comments cite (age's are in §2, jq's in §3, curl's in §4). A change to a condition is made in claude-3 first, then here.

Review condition	How the scripts meet it
age 1: encryption only, as age -r "$AGE_RECIPIENT" [-o FILE]. Never -R, -i, -j, -p, -d or AGEDEBUG	encrypt_to is the only age call. AGEDEBUG is unset at start
age 2: the recipient matches ^age1[02-9ac-hj-np-z]{58}$ (exactly one 1, so no plugin) and the value committed in Git	load_config checks both: the pattern, then AGE_RECIPIENT_PIN
age 3: no age-plugin-* binary in the image	The image build fails if one exists (P3)
age 4: set -o pipefail, every pipeline's status checked; a failed run deletes its outputs	A failed dump's ciphertext is deleted before upload; the EXIT trap deletes /work and /tmp copies
age 5, §9: the private key never enters the cluster	Only the public recipient is configured
jq 1, 2, 4: fixed literal programs, data only via --arg/--argjson. No -L, import, include, -f, --rawfile, --slurpfile, env or $ENV. Output never evaled	Three jq calls, each a single-quoted literal with --arg
jq 3: HOME=/nonexistent	lib.sh exports it, as does the image
jq 5: jq only processes data the job produced	The JSON of the job's own counts query, its keys, labels and alert text
curl 1: every call begins curl -q --proto =https --proto-redir =https --noproxy '*' --globoff --http1.1 --fail-with-body -sS --connect-timeout 10 --max-time <n>	One CURL array in lib.sh; each call adds --max-time
curl 2: two destinations, R2 with --aws-sigv4 aws:amz:auto:s3, --upload-file and Content-MD5, and the Discord webhook. Host and bucket fixed; keys from a fixed alphabet; no URL from data	R2_ENDPOINT, R2_BUCKET and the webhook are pattern-checked, and every key before its PUT (see "Key alphabet"). x-amz-content-sha256 is set too, as the plan asks
curl 3: credentials only in --config files, user = "<key id>:<secret>" and url = …. Never argv or environment	Both lines are printed by a builtin onto a pipe from the mounted Secret keys. Values must match [A-Za-z0-9] or the webhook pattern, so none can add a curl option
curl 4, 5: never -k, -L, -x, --netrc*, -b, -c, -v, --trace* or the other listed flags; one transfer per process	None is used; one URL per call
§4 and §8: nothing in the environment steers curl or libpq; the password reaches libpq through a 0600 pgpass copy	SSLKEYLOGFILE, proxies, CURL_HOME and PGPASSWORD are unset at start; write_pgpass writes the copy in /tmp
openssl review (P6) 6.1.2, 6.1.3, 6.1.5, 6.1.6: only /usr/bin/openssl, with OPENSSL_CONF=/dev/null; only dgst -sha256 -sign/-verify, pkey, dgst -md5 -binary; the signature checked against the pins before upload; fail closed	ossl is the only openssl call; load_pins and sign_checked in lib.sh. See Restoring from R2

Guard

scripts/check-backup-r2.sh runs the real backup-db.sh against fake psql, pg_dump, age, curl and jq, with no network and no real keys, in CI before the .NET build. It has seven scenarios: a clean run, an unexcluded DB, a failed dump, a failed PUT, a recipient that differs from its pin, a plugin-shaped recipient, and a server newer than pg_dump. After every run, any curl, age or jq call that breaks the review's conditions fails it, and so does a credential in the output, any argv, Discord or R2. --selftest puts back 15 defects and fails unless each is caught: plaintext upload, control not first, coverage skipped, SHA256SUMS early or after a failed PUT, no --snapshot, a wrong tenant prefix, credentials in argv or the log, no HOME=/nonexistent, no -q, no pin, a loose recipient pattern, no version check, no pipefail. P5 adds the agent-home scenarios and defects (Off-site backups to R2).

Where this differs from the plan

• One recipient. The plan puts a primary and a break-glass key in AGE_RECIPIENTS. The review allows only age -r "$AGE_RECIPIENT", so the break-glass copy is a second offline copy of the same identity (review §9). Two recipients need a review update first.
• No plaintext on disk. The plan dumps to the emptyDir and then encrypts. The review's pipe is used instead.
• Key alphabet. Tenant and DB names are limited to [a-z0-9_-]; the fixed parts add T, Z and SHA256SUMS.
• SHA256SUMS after a failed dump. It is written when every PUT succeeded, even if a dump failed. Failures are in the manifest's failed list and the 🔴 post, and the watcher (P9) checks for _control.

Not proven until the proving run

The plan's proving runs on the Dell are the gate: P5's CronJob ships suspended, and P9 unsuspends it only after the proving runs complete with no unexplained 🔴 and drill A has passed.

• That psql flushes each result into the FIFO. The opens can't deadlock, because both sides open in, then out. If psql buffers, the 600 s read times out, the fds close and psql is killed, so the dump fails and alerts; it never hangs.
• That R2 accepts the signed payload hash with Content-MD5, and with a write-only token (P1's smoke test).
• That discord-webhook-url is a https://discord.com/api/webhooks/… URL. Any other value is logged and not used.