Janady13/missionmatch / docs/RUNBOOK_OPERATIONS.md

1# Operations runbooks 2 3Day two operations for missionMATCH under the database per tenant model. Each section is a checklist with 4a verification and a rollback. New client onboarding is `docs/RUNBOOK_ONBOARDING.md`; the one time move of 5the existing shared database into per tenant is `docs/MIGRATE_TO_PER_TENANT.md`. 6 7## 1. Provisioning and deploy 8 9Stand up or update the platform. 10 111. Infrastructure: `terraform -chdir=infra/terraform apply -var-file=environments/aws-govcloud.tfvars` 12 (operator, real credentials). See `infra/terraform/README.md`. 132. Schema: run goose against the control plane database (creates the pgvector extension, the 14 missionmatch_app, missionmatch_migrator, and missionmatch_compliance roles, the schema, RLS). For the 15 tenant fleet, `make migrate-all-tenants-check` then `make migrate-all-tenants`. 163. Identity genesis: `provision-org` for the first org and owner. 174. Build and publish the images: `docker build -f infra/docker/Dockerfile.mm-api ...` and the gateway (the 18 in build FIPS gate must pass), push to the registry. 195. Deploy with the pod securityContext in `infra/docker/README.md` (non root 65532, read only rootfs, 20 `/tmp` tmpfs, drop ALL capabilities) and the env: `MM_TENANCY=per_tenant`, `MM_CLOUD`, the three control 21 plane DSNs, `MM_SECRETS_BACKEND` (a government cloud store in production), the pool bounds, and the auth 22 secrets. 23 24Verify: `GET /healthz` is ok, `GET /readyz` is ready (the control plane pool answers), and 25`GET /healthz/fips` reports `fips_enabled: true`. Rollback: redeploy the previous image tag; infrastructure 26changes roll back with the prior Terraform state. 27 28## 2. Scaling 29 30The API is stateless (sessions are opaque and server side in the control plane), so any pod serves any 31request. 32 331. API throughput: increase the replica count. No sticky sessions are needed. 342. Per tenant connection budget (the load bearing limit): `MM_TENANT_MAX_POOLS` times 35 `MM_TENANT_POOL_MAX_CONNS`, plus the control plane and compliance pools, must stay under the database 36 `max_connections`. As the tenant fleet grows, watch the `/metrics` pool gauges and the database 37 connection count; the bounded LRU pool registry evicts idle tenants, so the live pool count is bounded, 38 not the tenant count. Tune the two bounds (lower MaxConns per pool before raising MaxPools). 393. Database: scale the instance class vertically, or add read replicas for read heavy reporting. The 40 control plane is small (identity and the directory); each tenant database scales on its own. 41 42Verify: the pool gauges stay below the connection ceiling under load, and p95 latency holds. Rollback: 43lower the replica count or the pool bounds; a pooler in front of any database MUST be in TRANSACTION mode, 44never statement mode, or the per request org GUC isolation breaks. 45 46## 3. Failover 47 481. Database: managed Postgres is multi AZ, so a node failure fails over automatically to the standby on 49 the same endpoint. The pgx pool reconnects; the per tenant pool manager Pings before serving and 50 returns a typed 503 (never a hang) on a still unreachable tenant, then reopens on recovery. 512. Control plane: it is a single point for identity and routing, so it must be multi AZ like the tenant 52 databases. While it is down, login and routing fail, but each tenant database is independent and its 53 data is intact; recovery is automatic on control plane failover. 543. API: a rolling deploy with the readiness probe gates traffic away from a starting or unhealthy pod. 55 56Verify: after a forced failover, `GET /readyz` returns to ready and a tenant request serves its own data. 57Rollback: failover is automatic; if a node is unhealthy, replace it from the standby. 58 59## 4. Backup and recovery (per database) 60 61Every database (the control plane and each tenant) is covered by the managed automated backups (the 62Terraform `backup_retention_period`) plus on demand logical backups. 63 641. On demand backup: `scripts/backup-tenant.sh -dsn "<migrator DSN to the tenant DB>" -out FILE` (a custom 65 format archive of the whole database, public and eeo and the vectors; the tenant stays online). Pin the 66 client to PostgreSQL 16 (set `PGBIN`). 672. Recovery: take the tenant offline in the directory first 68 (`UPDATE tenant_directory SET status='offline' WHERE organization_id=...`, so the router refuses it with 69 a typed error), then 70 `scripts/restore-tenant.sh -admin-dsn "<superuser>" -migrator-dsn "<migrator>" -source <db> -target <db> -dump FILE`. 71 This is the procedure proven on M4 staging (BUILD_REPORT B4): it mirrors provision-tenant's prep, 72 restores, and verifies row count equality across public and eeo, a vector sample, and the isolation 73 invariants. On a green verify, set the status back to `active`. 74 75Verify: the restore script's own verify must pass (row counts match, vectors present, invariants hold). 76Rollback: if a restore is bad, leave the tenant offline and restore from an earlier backup; never flip a 77tenant active on a failed verify. 78 79## 5. Secret rotation 80 81All secrets rotate without a flag day because the load bearing ones are versioned. 82 831. Per tenant database credential: rotate the value in the secret store under the tenant's reference. The 84 secrets port caches with a short TTL (`MM_SECRET_CACHE_TTL_SECONDS`) and force refreshes on an 85 authentication failure, so the app picks up the new credential within the TTL or on the next failed 86 connect and retries with backoff. Keep a separate higher privilege identity for rotation. Verify: a 87 request to that tenant still serves after the old credential is invalidated. Rollback: restore the prior 88 secret value. 892. The credential pepper (`MM_CREDENTIAL_PEPPER`): set the new pepper, increment 90 `MM_CREDENTIAL_PEPPER_VERSION`, and keep the prior key in `MM_CREDENTIAL_PEPPER_PREVIOUS`. Old hashes 91 verify under the previous version and are rehashed to the new one on the next login. Verify: existing 92 users can still log in; new credentials carry the new version. Rollback: revert the version and the keys. 933. The TOTP KEK (`MM_TOTP_KEK`, 32 byte AES 256): set the new key, increment `MM_TOTP_KEK_VERSION`, keep the 94 prior key in `MM_TOTP_KEK_PREVIOUS` so existing TOTP secrets still decrypt. Verify: enrolled users pass 95 a TOTP check. Rollback: revert. 964. The CSRF key (`MM_CSRF_KEY`) and the recovery lookup key (`MM_RECOVERY_LOOKUP_KEY`): rotate the value. 97 A CSRF key rotation invalidates in flight CSRF tokens (clients fetch a fresh one on the next unsafe 98 request), so rotate at a low traffic window. Verify: a login and an unsafe request succeed after 99 rotation. Rollback: restore the prior key. 100 101Never log a resolved DSN, password, or secret value; the secrets port logs only the reference and the 102tenant id. All keys load from the server side environment file or the cloud secret store, never the repo 103or the browser bundle. 104