PRODUCTION DEPLOY, CONFIGURATION ONLY. 28 July 2026, approved by Joseph. Closes findings D2 and D3. D1 is PARTLY closed and the remainder is a new finding, D1b, recorded below rather than reported as done. NO APPLICATION CODE WAS DEPLOYED. No binary changed on either node; both still carry 8f24414531e80a9a, verified before and after. Scope was environment files, systemd units, and one stale credential. ================================================================================ D1, THE ADVERSE IMPACT SNAPSHOT. PARTLY CLOSED. ================================================================================ APPLIED. MM_COMPLIANCE_DATABASE_URL added to /etc/missionmatch/api.env on Bubbles, copied from the tenant compliance DSN already in that file. Staged to /tmp then cp, never sudo tee, which truncates. Line count asserted 27 to 28, staged file asserted non empty, perms re asserted 600 root:root, prior file backed up. VERIFIED. The compliance DSN connects as missionmatch_compliance and sees the eeo schema, 2 tables. The unit that had failed every daily firing since installation now exits 0: Result: success ExecMainStatus: 0 AND IT STILL PRODUCES NOTHING, WHICH IS FINDING D1b. fairness-snapshot: no organizations to snapshot This is the exact failure this build keeps relearning. An exit code of 0 is not the artifact. Had I stopped at the green result, D1 would have been reported closed while the compliance artifact remained nonexistent. THE CAUSE, diagnosed rather than guessed. resolveOrgs enumerates organizations with SELECT DISTINCT organization_id FROM audit_logs WHERE organization_id IS NOT NULL while the job deliberately holds the COMPLIANCE role, which is NOBYPASSRLS, and organizations, audit_logs, candidates, eeo.candidate_demographics and eeo.fairness_snapshots ALL have FORCED row level security. With no org GUC set the role sees only rows whose organization_id is NULL. Measured on production: 1058 audit rows visible, 0 with an organization. Measured on staging, where 2,783 organizations exist: also 0. So the job cannot enumerate the orgs it is supposed to iterate. It is not a production data quirk and no amount of configuration fixes it: the design asks a role that can see nothing without an org context to discover which contexts exist. NOT FIXED HERE, deliberately. The repair is a SECURITY DEFINER function returning only the org ids that carry eeo data, which is a migration plus a change to resolveOrgs. That is application code and a schema change, outside the configuration only scope Joseph approved, and inventing scope inside an approved deploy is how a config change becomes an incident. Queued. ================================================================================ D2, ALERTING COVERAGE. CLOSED. ================================================================================ APPLIED. OnFailure=bubbles-failure-notify@%N.service added to EIGHT missionMATCH units that lacked it: mm-alert-sweep, mm-api, mm-api-staging, mm-awards-sync, mm-fairness-snapshot, mm-rates-sync, mm-sam-sync, mm-watch. Each unit backed up, each edit asserted to add exactly one line, each staged file asserted non empty. SCOPED BY DESCRIPTION, NOT BY GLOB, and this mattered. The mm- prefix on Bubbles is shared with MEGAMIND organs: mm-estate, mm-legacy, mm-live and their reindex units belong to a different project. A glob would have wired another system's failures into the missionMATCH alert path, and an alert stream carrying someone else's noise is precisely how real alerts get ignored. Those units were left alone. POSITIVE CONTROL. A newly wired unit was BROKEN ON PURPOSE, because an OnFailure line that has never fired is indistinguishable from one that is silently wrong. mm-rates-sync ExecStart was replaced with /bin/false: unit result: exit-code notifier ran: success (ExecMainStatus 0) Starting bubbles-failure-notify@mm-rates-sync.service The unit was then restored from its backup and the restore VERIFIED, both the ExecStart path and that OnFailure survived the restore. ================================================================================ D3, THE CONTROL PLANE MIGRATOR CREDENTIAL. CLOSED. ================================================================================ DIAGNOSED BEFORE ACTING, and the diagnosis changed the fix. The obvious move was to rotate the role password. Searching the host first found that /etc/missionmatch/backup.dsn and /usr/local/sbin/mm-identity-sync.sh hold a WORKING migrator credential for the same database. So the role's password was never wrong; /etc/missionmatch/db.env held a STALE COPY of it. Rotating would have "fixed" the symptom and broken the two consumers that work today, including the control plane backup. That is the whole value of looking before reaching for the obvious repair. APPLIED. Only the password field of the migrator line in db.env was replaced, leaving host, port, database, and options intact. Line count asserted unchanged at 4, staged file asserted non empty, perms re asserted 600 root:root. VERIFIED, all four roles from that file: MM_DATABASE_URL missionmatch_app MM_MIGRATOR_DATABASE_URL missionmatch_migrator MM_COMPLIANCE_DATABASE_URL missionmatch_compliance MM_AIOE_DATABASE_URL missionmatch_aioe And the migrator now reads what only it can read: control plane schema v80. ================================================================================ THE EXTERNAL VANTAGE LEG ================================================================================ THE HONEST PROBLEM FIRST. Every machine I can reach is on 192.168.1.0/24: Bubbles .132, megamind-agi .190, and this workstation .199. There is no host under my control that is off this LAN, so I cannot manufacture a true external vantage by running curl somewhere. WHAT WAS ACTUALLY PROVEN FROM OUTSIDE. Fetched over the public internet from infrastructure that is not on this LAN and not under my control: https://missionmatch.thatwebhostingguy.com/ served, title "missionMATCH, Mission Workforce Intelligence" PROVES: DNS, WAN routing, TLS, and nginx serving the correct application to the public internet. DOES NOT PROVE the login form renders. The form is client rendered by React and an HTML to text fetch sees only the shell. Saying it proved the login page would be a claim the evidence does not support. https://fg-talent.com/ title "Federal Group Talent Partners | Federal Recruiting & Cleared Talent", heading "Federal Recruiting Built for Cleared Talent and Contract Success" https://fg-talent.com/team/ Patricia Cox, Owner. Brian Atwood, Owner. THE CONTENT RULE HOLDS FROM OUTSIDE: Patricia Cox and Brian Atwood only, Tom and Glennette nowhere. https://fg-talent.com/contact/ contact form present, phone (417) 528-8045, 308 Willow St, Clever, MO THE LOGIN PAGE RENDER is proven separately by the Gate C public sweep, which drives a real browser at 320, 390, 768, 1024 and 1440 with zero findings. That runs on Bubbles. So one vantage proves public reachability and another proves the render, and NEITHER PROVES BOTH. Stated rather than blurred. ================================================================================ THE RECURRING UPTIME PROBE ================================================================================ IT DID NOT RUN FROM ANY VANTAGE BUT BUBBLES. mm-watch.sh probes 127.0.0.1:8094 and the megamind node, from Bubbles. A host cannot report its own death: if Bubbles loses power, loses nginx, or lets a certificate expire, the watcher dies with it and the result is SILENCE, which reads identical to health. ADDED, and running: /usr/local/sbin/mm-offsite-watch.sh on megamind-agi, every five minutes via mm-offsite-watch.timer, probing the PUBLIC url plus fg-talent plus certificate expiry, alerting on a TRANSITION rather than every cycle. The Gmail SMTP alert path was ported to that host to carry it, since none existed. counted run: ok app=200 fgt=200 cert_days=66 POSITIVE CONTROL. Pointed at a hostname that cannot serve the app: PROBLEM: public app ... returned 404 rc=1 SENT: missionMATCH alert and the recovery transition also emailed: ok app=200 fgt=200 cert_days=66 SENT: missionMATCH alert Both directions proven, not just the failure. WHAT IT IS NOT, and this is the part that must not be rounded up. megamind-agi is on the SAME LAN behind the SAME uplink as Bubbles. This is a CROSS NODE vantage, not an EXTERNAL one. It catches host death, service death, and certificate expiry, which is exactly what Bubbles watching itself cannot. It CANNOT catch an ISP outage, a WAN routing failure, or a DNS failure as the internet sees it, because it shares all three with the thing it watches. A genuinely external recurring probe needs a third party monitor on Joseph's account. It is recorded as an open item rather than pretended at. ================================================================================ POST DEPLOY STATE ================================================================================ both nodes 8f24414531e80a9a, unchanged, no binary shipped production site 200 build routes 404 / 404 / 405, still dark control plane v80, tenant v80, staging v81, unchanged