Janady13/missionmatch / docs/CREDENTIAL_CONSUMERS.md

1# CREDENTIAL CONSUMERS: THE COMPLETE INVENTORY 2 3Created 29 July 2026 from incident D6. **References only. No values, ever.** 4 5## Why this file exists 6 7The D5 rotation was planned against a list of five consumers, held in memory. Two 8more existed and were found only by breaking: 9 10- the second API node keeps its **own** `MM_DATABASE_URL` in its **own** api.env, 11 so it crashed on restart after the control plane rotation 12- `mm-identity-sync.sh` hardcodes **two** DSNs in **two different clusters**, so a 13 blanket substitution would have written the control plane password into the 14 tenant line 15 16A rotation enumerated from memory is a rotation that will miss something. From now 17on **every rotation enumerates from this file**, and any consumer discovered 18outside it is a defect in this file to be fixed in the same change. 19 20## The clusters, because this is what caused D6 21 22There are **three separate PostgreSQL clusters**, and roles are **cluster scoped** 23while databases are not. Two databases sharing a cluster share its roles. 24 25 CLUSTER 1 M4 100.94.8.94:5432 26 databases: missionmatch (PRODUCTION TENANT), missionmatch_staging 27 THESE TWO SHARE A ROLE NAMESPACE. That is what made a staging 28 rehearsal change a production credential. 29 CLUSTER 2 megamind-agi-wsl 100.77.114.40:5433 30 database: missionmatch (CONTROL PLANE, authoritative identity) 31 CLUSTER 3 none other in service 32 33## The roles 34 35 CLUSTER 1, M4 36 missionmatch_migrator PRODUCTION TENANT only, from 29 Jul 37 missionmatch_staging_migrator STAGING only. Created 29 Jul as the D6 38 structural fix. CONNECT on missionmatch is 39 REVOKED, proven by refusal. 40 missionmatch_app production tenant application role 41 missionmatch_compliance eeo schema, NOBYPASSRLS 42 CLUSTER 2, control plane 43 missionmatch_app 44 missionmatch_migrator 45 missionmatch_compliance 46 missionmatch_aioe 47 48## THE CONSUMERS, every one 49 50### Bubbles, 100.90.97.104 51 52| file | variable | role | cluster | 53|---|---|---|---| 54| `/etc/missionmatch/api.env` | `MM_DATABASE_URL` | app | 2, control plane | 55| `/etc/missionmatch/api.env` | `MM_MIGRATOR_DATABASE_URL` | migrator | 1, tenant | 56| `/etc/missionmatch/api.env` | `MM_TENANT_DSN_TENANT_ACME` | app | 1, tenant | 57| `/etc/missionmatch/api.env` | `MM_TENANT_DSN_TENANT_ACME_COMPLIANCE` | compliance | 1, tenant | 58| `/etc/missionmatch/api.env` | `MM_COMPLIANCE_DATABASE_URL` | compliance | 1, tenant | 59| `/etc/missionmatch/staging.env` | `MM_MIGRATOR_DATABASE_URL` | **staging_migrator** | 1, staging | 60| `/etc/missionmatch/staging.env` | `MM_DATABASE_URL` | app | 1, staging | 61| `/etc/missionmatch/staging.env` | `MM_COMPLIANCE_DATABASE_URL` | compliance | 1, staging | 62| `/etc/missionmatch/backup.dsn` | bare DSN | migrator | 1, tenant | 63| `/etc/missionmatch/sam-sync.env` | `MM_MIGRATOR_DATABASE_URL` | migrator | 1, tenant | 64| `/etc/missionmatch/sam-sync.env` | `MM_COMPLIANCE_DATABASE_URL` | compliance | 1, tenant | 65| `/etc/mm-alert.env` | `SMTP_PASS` | Gmail app password | n/a | 66 67**The two `sam-sync.env` rows were added 29 July 2026 from finding D7**, where they 68were discovered by breaking rather than by reading, which is the same way the last 69two missing consumers were found. That file feeds THREE units, `mm-sam-sync`, 70`mm-awards-sync` and `mm-rates-sync`, so one stale line there stops three jobs at 71once and each one reports the same authentication failure, which reads like three 72problems and is one. It also carries `SAM_API_KEY`, `DATA_GOV_API_KEY`, 73`PDL_API_KEY`, `BLS_API_KEY`, `KAGGLE_KEY`, `HF_TOKEN` and `ONET_API_KEY`, so it is 74a secrets file in its own right and not only a DSN holder. 75 76**`backup.dsn` was already listed here and was still missed by the D5 rotation.** A 77complete inventory is necessary and not sufficient: the rotation has to actually 78enumerate from it. That is why step 1 of the procedure below says read this file, 79and it is now the step that failed rather than a step that was absent. 80 81### megamind-agi WSL, 100.77.114.40 82 83| file | variable | role | cluster | 84|---|---|---|---| 85| `/etc/missionmatch/api.env` | `MM_DATABASE_URL` | app | 2, control plane | 86| `/etc/missionmatch/db.env` | `MM_DATABASE_URL` | app | 2 | 87| `/etc/missionmatch/db.env` | `MM_MIGRATOR_DATABASE_URL` | migrator | 2 | 88| `/etc/missionmatch/db.env` | `MM_COMPLIANCE_DATABASE_URL` | compliance | 2 | 89| `/etc/missionmatch/db.env` | `MM_AIOE_DATABASE_URL` | aioe | 2 | 90| `/etc/missionmatch/backup.dsn` | bare DSN | migrator | 2 | 91| `/etc/missionmatch/db.env` | `MM_TENANT_MIGRATOR_DSN_TENANT_ACME` | migrator | **1, tenant** | 92| `/etc/mm-alert.env` | `SMTP_PASS` | Gmail app password | n/a | 93 94**The trap that used to be here is gone, and it took two months to find.** Until 29 95July 2026 `mm-identity-sync.sh` carried both DSNs INLINE, on two adjacent lines, one 96per cluster. This file described it as a trap and left it standing: a `sed` across the 97file would write the wrong password into one of them and the failure would surface 98hours later as a failed sync rather than immediately. 99 100It was never rewritten because nothing could see it. The out of repo secret sweep found 101both values on its first clean run, and the script now resolves 102`MM_MIGRATOR_DATABASE_URL` and `MM_TENANT_MIGRATOR_DSN_TENANT_ACME` from `db.env`, 103failing by NAME if either key is absent. The two DSN rows above are now env file rows 104like every other consumer, so the rotation procedure reaches them by enumeration 105instead of by memory. 106 107**The same change took both credentials out of `argv`.** The old script passed each DSN 108to `psql` and to `sync-tenant-identity` as a command line flag, and `argv` is world 109readable through `ps`, so any account on that host could read both migrator passwords 110out of the process table for as long as a run lasted. Measured before and after: one 111exposure visible during a run of the old script, zero during the repaired one. A flag is 112a weaker hiding place than an environment variable, not a stronger one. 113 114### Ops scripts, added 29 July 2026 from finding D10 115 116**This section exists because two of the five broken consumers lived here and this file 117did not mention scripts at all.** An inventory that covers env files and not the scripts 118beside them is not an inventory, and the repository secret sweep cannot see any of these 119paths because they are not in the repository. 120 121| file | how it gets its credential | role | cluster | 122|---|---|---|---| 123| `/usr/local/sbin/mm-db-backup.sh` (Bubbles) | reads `/etc/missionmatch/backup.dsn` | migrator | 1, tenant | 124| `/usr/local/sbin/mm-db-backup.sh` (Bubbles) | ssh key auth to VALKYRIE, root's own key | n/a | n/a | 125| `/usr/local/sbin/mm-restore-drill.sh` (Bubbles) | reads `api.env` **since 29 Jul** | migrator | 1, tenant | 126| `/usr/local/sbin/mm-identity-sync.sh` (megamind-agi) | reads `db.env` **since 29 Jul**, two keys, no argv | migrator | **1 and 2** | 127| `/usr/local/sbin/mm-disk-watch.sh` (Bubbles) | none, calls `mm-alert.sh` | n/a | n/a | 128| `/usr/local/sbin/mm-alert.sh` (Bubbles and megamind-agi) | reads `/etc/mm-alert.env` | Gmail app password | n/a | 129| `/usr/local/sbin/mm-watch.sh` (Bubbles) | none, HTTP probes only | n/a | n/a | 130| `/usr/local/sbin/mm-offsite-watch.sh` (megamind-agi) | none, HTTP probes only | n/a | n/a | 131 132**`mm-restore-drill.sh` HARDCODED A PLAINTEXT MIGRATOR PASSWORD until 29 July.** It was 133therefore a consumer no inventory listed, invisible to the repository sweep, and the D5 134rotation left it holding a dead value. The consequence was that the one instrument whose 135entire purpose is to prove a backup can be restored could not run at all. It now reads 136`MM_MIGRATOR_DATABASE_URL` from `api.env` and refuses to start if it is absent, so it 137follows every future rotation without anyone remembering it exists. 138 139**The rule that follows: a script is a consumer.** When adding an ops script that touches a 140database, add a row here in the same change. And prefer reading an env file to embedding a 141value, because a consumer that reads the store follows a rotation for free while a consumer 142that embeds one has to be found. 143 144### Application accounts, not database roles 145 146| where | what | 147|---|---| 148| control plane `users` table | operator, tenant superusers, recruiters | 149| staging `users` table | `patriciacox6@aol.com`, `desk.admin@fg-talent.test` | 150| staging `users` table, Gate C fixtures | `gatec.recruiter@fg-talent.test`, `gatec.manager@fg-talent.test` | 151 152Rotate these with `cmd/rotate-credential`, which goes through the real pepper and 153PBKDF2 path rather than reimplementing the KDF. 154 155### Other secrets in the env files 156 157`MM_CREDENTIAL_PEPPER`, `MM_TOTP_KEK`, `MM_CSRF_KEY`, `MM_RECOVERY_LOOKUP_KEY`, 158`MM_MEGAMIND_TOKEN`, and the OpenAI key in the M4 secrets file. The first two are 159**versioned** and rotate without a flag day; the rest do not, which is recorded in 160the risk register rather than pretended away. 161 162## The rotation procedure 163 1641. Read this file. Do not enumerate from memory. 1652. Confirm the cluster for every consumer. A password is only valid in its own. 1663. Take a fresh backup of the affected cluster. 1674. **Update every consumer FIRST**, while the old credential still works. 1685. `ALTER ROLE` **last**. The moment it commits, consumers already carrying the 169 new value simply start working, and there is no window where nothing connects. 1706. Restart anything that reads its DSN at boot. Both API nodes do. 1717. Prove every consumer **by role name and status code**, never by printing a DSN. 1728. Prove an **authenticated login**, which is the only thing that exercises 173 control plane identity resolution end to end. 1749. If a consumer was found that is not in this file, **add it in the same change**. 17510. **Enumerate the ops scripts too**, not only the env files. Two of the five consumers 176 broken by the D5 rotation were scripts, and one of them held a plaintext password that 177 the repository secret sweep can never see because the file is not in the repository. 178