Janady13/missionmatch / docs/evidence/phase2/deploys/config-fix-deploy.txt

1PRODUCTION DEPLOY, CONFIGURATION ONLY. 28 July 2026, approved by Joseph. 2Closes findings D2 and D3. D1 is PARTLY closed and the remainder is a new 3finding, D1b, recorded below rather than reported as done. 4 5NO APPLICATION CODE WAS DEPLOYED. No binary changed on either node; both still 6carry 8f24414531e80a9a, verified before and after. Scope was environment files, 7systemd units, and one stale credential. 8 9================================================================================ 10D1, THE ADVERSE IMPACT SNAPSHOT. PARTLY CLOSED. 11================================================================================ 12APPLIED. MM_COMPLIANCE_DATABASE_URL added to /etc/missionmatch/api.env on 13Bubbles, copied from the tenant compliance DSN already in that file. Staged to 14/tmp then cp, never sudo tee, which truncates. Line count asserted 27 to 28, 15staged file asserted non empty, perms re asserted 600 root:root, prior file 16backed up. 17 18VERIFIED. The compliance DSN connects as missionmatch_compliance and sees the 19eeo schema, 2 tables. The unit that had failed every daily firing since 20installation now exits 0: 21 Result: success ExecMainStatus: 0 22 23AND IT STILL PRODUCES NOTHING, WHICH IS FINDING D1b. 24 fairness-snapshot: no organizations to snapshot 25This is the exact failure this build keeps relearning. An exit code of 0 is not 26the artifact. Had I stopped at the green result, D1 would have been reported 27closed while the compliance artifact remained nonexistent. 28 29 THE CAUSE, diagnosed rather than guessed. resolveOrgs enumerates organizations 30 with 31 SELECT DISTINCT organization_id FROM audit_logs WHERE organization_id IS NOT NULL 32 while the job deliberately holds the COMPLIANCE role, which is NOBYPASSRLS, and 33 organizations, audit_logs, candidates, eeo.candidate_demographics and 34 eeo.fairness_snapshots ALL have FORCED row level security. With no org GUC set 35 the role sees only rows whose organization_id is NULL. Measured on production: 36 1058 audit rows visible, 0 with an organization. Measured on staging, where 37 2,783 organizations exist: also 0. 38 39 So the job cannot enumerate the orgs it is supposed to iterate. It is not a 40 production data quirk and no amount of configuration fixes it: the design asks 41 a role that can see nothing without an org context to discover which contexts 42 exist. 43 44 NOT FIXED HERE, deliberately. The repair is a SECURITY DEFINER function 45 returning only the org ids that carry eeo data, which is a migration plus a 46 change to resolveOrgs. That is application code and a schema change, outside 47 the configuration only scope Joseph approved, and inventing scope inside an 48 approved deploy is how a config change becomes an incident. Queued. 49 50================================================================================ 51D2, ALERTING COVERAGE. CLOSED. 52================================================================================ 53APPLIED. OnFailure=bubbles-failure-notify@%N.service added to EIGHT missionMATCH 54units that lacked it: mm-alert-sweep, mm-api, mm-api-staging, mm-awards-sync, 55mm-fairness-snapshot, mm-rates-sync, mm-sam-sync, mm-watch. Each unit backed up, 56each edit asserted to add exactly one line, each staged file asserted non empty. 57 58SCOPED BY DESCRIPTION, NOT BY GLOB, and this mattered. The mm- prefix on Bubbles 59is shared with MEGAMIND organs: mm-estate, mm-legacy, mm-live and their reindex 60units belong to a different project. A glob would have wired another system's 61failures into the missionMATCH alert path, and an alert stream carrying someone 62else's noise is precisely how real alerts get ignored. Those units were left 63alone. 64 65POSITIVE CONTROL. A newly wired unit was BROKEN ON PURPOSE, because an OnFailure 66line that has never fired is indistinguishable from one that is silently wrong. 67mm-rates-sync ExecStart was replaced with /bin/false: 68 unit result: exit-code 69 notifier ran: success (ExecMainStatus 0) 70 Starting bubbles-failure-notify@mm-rates-sync.service 71The unit was then restored from its backup and the restore VERIFIED, both the 72ExecStart path and that OnFailure survived the restore. 73 74================================================================================ 75D3, THE CONTROL PLANE MIGRATOR CREDENTIAL. CLOSED. 76================================================================================ 77DIAGNOSED BEFORE ACTING, and the diagnosis changed the fix. The obvious move was 78to rotate the role password. Searching the host first found that 79/etc/missionmatch/backup.dsn and /usr/local/sbin/mm-identity-sync.sh hold a 80WORKING migrator credential for the same database. So the role's password was 81never wrong; /etc/missionmatch/db.env held a STALE COPY of it. 82 83Rotating would have "fixed" the symptom and broken the two consumers that work 84today, including the control plane backup. That is the whole value of looking 85before reaching for the obvious repair. 86 87APPLIED. Only the password field of the migrator line in db.env was replaced, 88leaving host, port, database, and options intact. Line count asserted unchanged 89at 4, staged file asserted non empty, perms re asserted 600 root:root. 90 91VERIFIED, all four roles from that file: 92 MM_DATABASE_URL missionmatch_app 93 MM_MIGRATOR_DATABASE_URL missionmatch_migrator 94 MM_COMPLIANCE_DATABASE_URL missionmatch_compliance 95 MM_AIOE_DATABASE_URL missionmatch_aioe 96And the migrator now reads what only it can read: control plane schema v80. 97 98================================================================================ 99THE EXTERNAL VANTAGE LEG 100================================================================================ 101THE HONEST PROBLEM FIRST. Every machine I can reach is on 192.168.1.0/24: 102Bubbles .132, megamind-agi .190, and this workstation .199. There is no host 103under my control that is off this LAN, so I cannot manufacture a true external 104vantage by running curl somewhere. 105 106WHAT WAS ACTUALLY PROVEN FROM OUTSIDE. Fetched over the public internet from 107infrastructure that is not on this LAN and not under my control: 108 109 https://missionmatch.thatwebhostingguy.com/ 110 served, title "missionMATCH, Mission Workforce Intelligence" 111 PROVES: DNS, WAN routing, TLS, and nginx serving the correct application 112 to the public internet. 113 DOES NOT PROVE the login form renders. The form is client rendered by 114 React and an HTML to text fetch sees only the shell. Saying it proved the 115 login page would be a claim the evidence does not support. 116 117 https://fg-talent.com/ 118 title "Federal Group Talent Partners | Federal Recruiting & Cleared 119 Talent", heading "Federal Recruiting Built for Cleared Talent and Contract 120 Success" 121 https://fg-talent.com/team/ 122 Patricia Cox, Owner. Brian Atwood, Owner. 123 THE CONTENT RULE HOLDS FROM OUTSIDE: Patricia Cox and Brian Atwood only, 124 Tom and Glennette nowhere. 125 https://fg-talent.com/contact/ 126 contact form present, phone (417) 528-8045, 308 Willow St, Clever, MO 127 128 THE LOGIN PAGE RENDER is proven separately by the Gate C public sweep, which 129 drives a real browser at 320, 390, 768, 1024 and 1440 with zero findings. That 130 runs on Bubbles. So one vantage proves public reachability and another proves 131 the render, and NEITHER PROVES BOTH. Stated rather than blurred. 132 133================================================================================ 134THE RECURRING UPTIME PROBE 135================================================================================ 136IT DID NOT RUN FROM ANY VANTAGE BUT BUBBLES. mm-watch.sh probes 137127.0.0.1:8094 and the megamind node, from Bubbles. A host cannot report its own 138death: if Bubbles loses power, loses nginx, or lets a certificate expire, the 139watcher dies with it and the result is SILENCE, which reads identical to health. 140 141ADDED, and running: /usr/local/sbin/mm-offsite-watch.sh on megamind-agi, every 142five minutes via mm-offsite-watch.timer, probing the PUBLIC url plus fg-talent 143plus certificate expiry, alerting on a TRANSITION rather than every cycle. The 144Gmail SMTP alert path was ported to that host to carry it, since none existed. 145 146 counted run: ok app=200 fgt=200 cert_days=66 147 148POSITIVE CONTROL. Pointed at a hostname that cannot serve the app: 149 PROBLEM: public app ... returned 404 150 rc=1 151 SENT: missionMATCH alert 152and the recovery transition also emailed: 153 ok app=200 fgt=200 cert_days=66 154 SENT: missionMATCH alert 155Both directions proven, not just the failure. 156 157WHAT IT IS NOT, and this is the part that must not be rounded up. megamind-agi 158is on the SAME LAN behind the SAME uplink as Bubbles. This is a CROSS NODE 159vantage, not an EXTERNAL one. It catches host death, service death, and 160certificate expiry, which is exactly what Bubbles watching itself cannot. It 161CANNOT catch an ISP outage, a WAN routing failure, or a DNS failure as the 162internet sees it, because it shares all three with the thing it watches. 163 164A genuinely external recurring probe needs a third party monitor on Joseph's 165account. It is recorded as an open item rather than pretended at. 166 167================================================================================ 168POST DEPLOY STATE 169================================================================================ 170 both nodes 8f24414531e80a9a, unchanged, no binary shipped 171 production site 200 172 build routes 404 / 404 / 405, still dark 173 control plane v80, tenant v80, staging v81, unchanged 174