Operations Guide — deploys, OAuth login & token setup

Last updated: 2026-09-23 (full rewrite — the previous version was an April-2026 Fly.io/Vercel runbook; Fly was decommissioned 2026-09-09 and the dashboard moved to forge-control. Closes this file's findings from the 2026-09-23 docs audit: pervasive stale Fly/Vercel content, the missing break-glass secrets inventory, and the absent JWT-drift triage entry.)

Where things actually run (verified 2026-09-23):

Related runbooks: registry-outage-runbook.md (incident response), backups.md, deployment-architecture.md, docs/05-cli.md (CLI reference), docs/08-cli-auth.md (login/troubleshooting).

Deploys

Two paths exist. Both end at docker compose on app-3; they differ in who writes the compose/env files and which env file the compose reads.

Normal path (forge-control worker)

Push to main on forge-git → Gitea webhook → build-worker builds the image on app-3, writes /opt/packr-registry/registry.docker-compose.yml + site.docker-compose.yml, and docker compose up -d. Feature-branch pushes are cancelled by design; pushes that touch only non-code paths (docs, changelog.d/) are also cancelled — neither rebuilds production.

# Watch a deploy (forge CLI):
forge app deploys packr-registry          # records + error messages
forge app status packr-registry           # container, image, health, domains
forge deployments show <full-uuid>        # stage-by-stage lifecycle

Known landmines (tracked in forge-control):

• The worker's env render has twice diverged from vault/files — it dropped ?sslmode=disable from DATABASE_URL (registry crash-loops with pq: SSL is not enabled on the server, rolls back) and wrote a site INTERNAL_API_SECRET that didn't match the registry's. After any worker deploy, verify /-/ping, health?strict=1, and a packr-cli doctor before trusting it. (forge-control#1502)
• Rollback messages name the deployment record's commit, not necessarily the image that is actually serving — trust /-/ping's commit, not the message.
• A failed registry deploy rolls back the registry only; the site half may have shipped. Check both containers.

Manual fallback (scripts/deploy-fleet.sh)

Use when the worker path is broken. Builds the image on the host from the exact commit, installs deploy/fleet/registry.docker-compose.yml, and verifies /-/ping on loopback + publicly.

scripts/deploy-fleet.sh            # VERSION from HEAD's tag, else "dev"
scripts/deploy-fleet.sh v0.54.0    # explicit (must match the VERSION file)

• Refuses a dirty tree (the image is built from git archive HEAD).
• Never writes env files — runtime env stays whatever is on the host.
• Ships the deploy-artefact scripts (backup-db.sh, regression-recovery.sh, restore-db.sh, …) to /opt/packr-registry/ so /etc/cron.d/packr-backup keeps working.
• ⚠️ The site container appears as an "orphan" in its output — it is a separate compose file under the same project. Never add --remove-orphans; it would delete the site.
• Rollback: ssh root@app-3 'cd /opt/packr-registry && PACKR_IMAGE_TAG=<old-tag> docker compose -p packr-registry -f registry.docker-compose.yml up -d' (previous image tags are kept; e.g. packr-registry:v0.52.0).

Releases

Tagged v* pushes run release.yml → fold changelog.d/ fragments → tag → forge app-deploy. The step list lives in CONTRIBUTING.md; fragment conventions in changelog.d/README.md.

Secrets & env files

The registry reads env from two files that must stay in sync — which one is live depends on which compose is installed:

Rotating a secret therefore means updating BOTH registry files (and the site's, when it is INTERNAL_API_SECRET), then recreating the container — docker restart reuses the captured env and silently keeps the old value.

Canonical values live in forge vault (forge vault get --project packr-registry --key <KEY>): JWT_SECRET, SESSION_SECRET, INTERNAL_API_SECRET, DATABASE_URL, plus SUPER_ADMIN_TOKEN and OAUTH_GITHUB_* (added 2026-09-23). Hash-compare before and after; never print values.

Current break-glass inventory (as of 2026-09-23, hash-verified):

Vault is the source of truth; the env files are the delivery. If a deploy writes a value that disagrees with vault, the deploy is wrong — that is forge-control#1502's bug class, not a reason to "fix" vault.

OAuth login setup

Prerequisites

• Registry at https://api.packr.blueforge.studio (app-3)
• GitHub OAuth App configured (client ID: Ov23ligJE4NpN7NysXzL)
• OAUTH_GITHUB_CLIENT_ID / OAUTH_GITHUB_CLIENT_SECRET set in both registry env files (device flow reads them from the registry process)
• OAUTH_REDIRECT_BASE=https://packr.blueforge.studio
• GitHub OAuth App callback URL = https://packr.blueforge.studio/api/auth/callback/github
• Dashboard live at https://packr.blueforge.studio (forge-control, app-3)
• INTERNAL_API_SECRET equal on registry and site (see Secrets above)
• packr-cli built at bin/packr-cli

The GitHub OAuth client secret was written out in full in this repo's history, so treat any value that was ever committed as burned: rotate it in the OAuth App and update both env files + vault. The client ID is not a secret and needs no rotation.

First login

./bin/packr-cli login --provider github --registry https://api.packr.blueforge.studio

1. CLI prints a verification URL + code (ABCD-EFGH)
2. Browser opens https://packr.blueforge.studio/cli/auth (live since 2026-08-20)
3. Enter the code → authorize on GitHub's consent screen
4. Browser: "CLI Authorized." → CLI: Logged in as <name> (maintainer) via github — expires <date>

Creates ~/.packr/credentials.json (mode 0600) and updates ~/.npmrc.

Headless/CI: no browser to open? Use the password fallback — packr-cli login --password — see docs/08-cli-auth.md § "Password login escape hatch". On a live registry, whoami reads the local file and can claim a dead token is fine; verify server-side with packr-cli doctor or packr-cli token verify.

Verify install path

pnpm install @blueforge-studio/some-package   # resolves via the token in ~/.npmrc

Creating scoped CI tokens

packr-cli token create uses the admin API — it needs a working login (or the operator route, below), plus --registry or PACKR_REGISTRY.

export PACKR_REGISTRY=https://api.packr.blueforge.studio

# Read-only (installs)
./bin/packr-cli token create ci-read --role ci-readonly

# Publish, scoped to the namespace (releases)
./bin/packr-cli token create ci-publish --role ci-publish --scope @blueforge-studio \
  --output-file /tmp/tok        # never let the JWT hit scrollback/CI logs

The operator route: minting when your login is the broken thing

After a JWT_SECRET change, your stored session may 401 on every admin route while publish still works — and token create goes through the admin API. The break-glass path needs only the super-admin secret, no session:

export PACKR_SUPER_ADMIN_TOKEN=<from ~/.forge-secrets/packr-super-admin.token or vault>
./bin/packr-cli token create replacement --role admin --scope @blueforge-studio \
  --as-user web:<id> --output-file /tmp/tok

--as-user posts to POST /api/v1/admin/tokens/mint with X-Super-Admin-Token and no Authorization header (the audit row names the secret, not a stale session). Requires SUPER_ADMIN_TOKEN to be set server-side — see the Secrets table above. Retiring a credential you can't authenticate as is the mirror operation: packr-cli token revoke <id> --super-admin-token <tok>.

JWT drift — "publish works, every admin call 401s"

The classic post-rotation symptom, and the reason the operator route exists:

Fix: packr-cli login --password (fresh signature) or the operator mint above; then retire the old credential with packr-cli token revoke <id> --super-admin-token …. Full detail: docs/08-cli-auth.md § troubleshooting.

GitHub org secrets — pitfalls

Use stdin with printf, not --body (avoids line-wrap corruption):

unset GITHUB_TOKEN                 # otherwise gh ignores keychain auth
gh auth refresh -h github.com -s admin:org
printf '%s' '<token>' | gh secret set PACKR_TOKEN --org blueforge-studio
printf '%s' '<token>' | gh secret set PACKR_TOKEN_PUBLISH --org blueforge-studio

Repo-level secrets shadow org-level ones. A leftover repo PACKR_TOKEN silently wins:

cd /path/to/repo && gh secret list          # repo-level
gh secret list --org blueforge-studio       # org-level
gh secret delete PACKR_TOKEN               # remove the shadowing duplicate

PACKR_TOKEN vs PACKR_TOKEN_PUBLISH

The original design was: use PACKR_TOKEN (read-only) for pnpm install, PACKR_TOKEN_PUBLISH (publish + scoped) for releases.

It does not survive a single root .npmrc: both install and publish read the same _authToken line, and NODE_AUTH_TOKEN loses to it. Production runs one ci-publish token scoped to @blueforge-studio (the scope is the real authorization boundary), set as both secrets:

printf '%s' '<ci-publish-token>' | gh secret set PACKR_TOKEN --org blueforge-studio
printf '%s' '<ci-publish-token>' | gh secret set PACKR_TOKEN_PUBLISH --org blueforge-studio

Real split-in-two needs per-command temp .npmrcs — documented, not yet the default.

Initializing consumer repos

./bin/packr-cli init --scope @blueforge-studio --registry https://api.packr.blueforge.studio

Writes .packr.json (registry + scope) and an .npmrc scope line with a ${PACKR_TOKEN} reference (safe to commit). Target every repo consuming @blueforge-studio/*: blueforge-org, mailstack-os, and siblings.

Dashboard/site environment

The site is a forge-control app, not a Vercel project. Its env lives in /opt/packr-registry/site.secrets.env on app-3 (same host as the registry); blueforge.config.json#apps.site.secrets declares what belongs there. Key entries:

GITHUB_OAUTH_CLIENT_ID / GITHUB_OAUTH_CLIENT_SECRET   # web sign-in (rotate if burned — see above)
INTERNAL_API_SECRET                                   # MUST equal the registry's value
SESSION_SECRET                                        # site sessions
STRIPE_*                                              # billing

After editing: recreate the site container (docker compose -f site.docker-compose.yml up -d site-1) — restart does not re-read env.

⚠️ The worker has written a site INTERNAL_API_SECRET that didn't match the registry's (forge-control#1502). If dashboard login or device-flow callbacks start failing after a deploy, hash-compare the two files first — this exact drift has now been repaired by hand twice.

Troubleshooting

Device flow / OAuth callback

1. Callback URL in the OAuth App must be https://packr.blueforge.studio/api/auth/callback/github (not /cli/auth/callback)
2. OAUTH_REDIRECT_BASE=https://packr.blueforge.studio in the registry env
3. INTERNAL_API_SECRET equal on registry and site (hash-compare — never print)
4. OAUTH_GITHUB_CLIENT_ID/SECRET present in the registry env (unknown_provider from /-/v1/device/authorize = they are missing)
5. Site route live: curl -I https://packr.blueforge.studio/cli/auth → 200

Admin calls 401 while publish works

JWT drift — see above. Do not chase the token's exp; it lies. packr-cli token verify reports the signature and stored-row paths separately.

Publish fails with 403 in CI

1. Repo-level PACKR_TOKEN shadowing the org-level secret (see pitfalls above)
2. Decode the JWT actually in the secret and check permissions/scopes — echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | python3 -m json.tool
3. Confirm the token's scopes actually cover the namespace you are about to publish into. Publishing a brand-new @scope/* name needs publish plus a matching scope — not admin (README § No admin role is needed to publish a new package)
4. Registry logs: ssh root@app-3 'docker logs packr-registry-registry-1-1 --tail 100 | grep publish:'

Token expired / session keeps dropping

packr-cli login --provider github --registry https://api.packr.blueforge.studio

If sessions drop on every registry boot: SESSION_SECRET (and JWT_SECRET) must be set persistently — unset, the server generates a random one per boot and silently logs everyone out while whoami still reports the local file as valid.

Keychain storage (optional)

./bin/packr-cli config set credential-store keychain

Service packr-cli, account = registry URL (docs/08-cli-auth.md § credential stores). Headless/SSH sessions should stay on file.

Logs & health

ssh root@app-3 'docker logs packr-registry-registry-1-1 --tail 100'
curl -s https://api.packr.blueforge.studio/health?strict=1   # strict: blob+db must be ok
packr-cli doctor                                              # full local→server sweep

/health (bare) answers 200 even when degraded — a liveness probe must not restart a container over a bad bucket credential. Use ?strict=1 or doctor when the question is "is it actually serving".

Token role reference

Per-plan token ceilings (default free = 20 active) gate every mint path; manage with packr-cli token-limit get|set|clear web:<id>.

Lifting a rate limit (super-admin escape hatch)

The login limiter is 10/min per IP. A packr-cli login device flow that polls can spend that, and the operator is then locked out of the one action that would let them fix anything — including whatever made them retry. Observed 2026-09-02.

curl -X POST https://api.packr.blueforge.studio/api/v1/admin/maintenance/rate-limit/reset \
  -H "X-Super-Admin-Token: $SUPER_ADMIN_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"limiter":"login","key":"203.0.113.7"}'
# X-Internal-Secret: <INTERNAL_API_SECRET> is accepted by the same handler

• limiter — login, publish, admin, search, or all. An unknown name is a 400, not a silent no-op: "cleared nothing" and "worked" must not look the same to someone acting in an incident.
• key — the client IP to clear. Omit it to clear every bucket on that limiter.

Gated by X-Super-Admin-Token or X-Internal-Secret, like the other maintenance/* routes. Deliberately not behind a session: an operator locked out by the login limiter has no session to present, and gating it behind one would make it unreachable exactly when it is needed. Both secrets must be set server-side (see the Secrets table) — secret_compare.go fails closed, so a correct local value still 401s against an unprovisioned registry.

Budgeted at 3 per rolling 24h (RATE_LIMIT_RESETS_PER_DAY). A reset switch with no budget is not a safety valve, it is the removal of the limit — anyone holding the token could clear the login limiter on every request and brute-force behind it. The window slides, so a budget spent at 23:00 refills through the following day rather than at midnight.

Every reset is written to the audit log as ratelimit.reset with the limiter, key and what was cleared. The budget keeps those entries rare enough to read.

The response reports what is left:

{"cleared":{"login":1},"resetsRemaining":2,"budgetWindowHours":24}

If sessions are being invalidated rather than rate-limited, the cause is different: SESSION_SECRET unset makes the server generate a random one at boot, so every restart silently logs everyone out while the CLI still reports a valid local session. Set SESSION_SECRET and JWT_SECRET persistently.

Full audit surface (scope=all for super-admin rows, including operator mints): GET /api/v1/admin/audit — docs/API.md § Audit Log.