Janady13/missionmatch / docs/BUILD_STATE.md · Lines 1501–3000

1501afterwards is missing exactly when the process dies mid call. RecordExternalOutcome then records WHICH backend 1502actually served, because the gateway falls back to the local runtime on failure and that fallback is invisible 1503in an audit trail that records only intent. 1504MUTATION CONTROLLED, four new tests: removing the write fails exactly the two that assert it and restoring it 1505turns them green. Also asserted rather than trusted: no insert argument carries the payload content, and a 1506refused payload writes NO row at all. 1507IT EARNED ITSELF IMMEDIATELY. The first run RecordExternalOutcome ever saw was the run it caught: three 1508outcome rows reading ollama:llama3.1:8b after OpenAI refused with insufficient_quota. Without it the run would 1509have read as three successful external calls forever, because accepted=3 rejected=0 and three cited scope 1510items is exactly what a successful external run looks like from here. | 52fcbb9 | 2026-07-29 1511 1512FINDING D7, THE D5 ROTATION LEFT TWO CONSUMERS STALE AND PRODUCTION HAS HAD NO TENANT BACKUP FOR A DAY 1513Found by checking production health rather than by an alert, which is itself part of the finding. 1514 THE SYMPTOM: four units on Bubbles are in a failed state, mm-db-backup, mm-sam-sync, mm-awards-sync and 1515 mm-rates-sync, all four with ONE root cause, password authentication failed for user 1516 missionmatch_migrator against 100.94.8.94 port 5432, which is cluster 1, the production tenant. 1517 PROVEN BY ROLE NAME AND STATUS CODE, never by printing a DSN, per the D6 rule: 1518 api.env MM_MIGRATOR_DATABASE_URL OK missionmatch_migrator on missionmatch port 5432 1519 backup.dsn FAIL password authentication failed 1520 sam-sync.env MM_MIGRATOR_DATABASE_URL FAIL password authentication failed 1521 So the ROLE IS FINE and its current password is the one api.env already holds. Two files hold a STALE 1522 COPY. This is the exact shape of finding D3, which was diagnosed the same way and fixed by correcting 1523 the stale copy rather than by rotating the role. Rotating here would break the working consumer. 1524 EVERY OTHER CONSUMER ON BOTH NODES IS HEALTHY, checked individually: api.env control plane app and 1525 compliance, staging.env staging_migrator and staging_app with the D6 isolation still holding, and on 1526 megamind-agi all four db.env roles, backup.dsn, api.env, and BOTH mm-identity-sync DSNs across BOTH 1527 clusters, which was the trap D6 named and it held. Identity sync ran clean at 16:04. 1528 THE IMPACT, measured from the backup directory and the journal rather than asserted: 1529 last COMPLETE backup set 28 July 16:42:58, tenant and staging, with its MANIFEST.sha256 1530 failed runs since 29 July 03:07:53, 06:00:06, 12:00:08, three consecutive 1531 left behind three ZERO BYTE tenant dump files in daily/20260729/ 1532 control plane and staging dumps for 29 July do not exist at all, because the script dies on the 1533 first dump and never reaches them 1534 So the largest recorded risk in the register, the absence of a backup, is LIVE right now and has been 1535 for roughly twenty four hours. The zero byte files are partials and must be removed, because a restore 1536 that finds them is a restore that reads a truncated dump. 1537 sam-sync.env IS NOT IN docs/CREDENTIAL_CONSUMERS.md. That file exists precisely to be complete, and D6 1538 rule nine says a consumer discovered outside it is a defect IN THE FILE to be fixed in the same change. 1539 backup.dsn IS listed and was still missed, so the rotation did not enumerate from the file it wrote. 1540 NOT FIXED, because it is a production configuration change and production scope is closed outside the 1541 approvals in the standing order. QUEUED for Joseph. It also BLOCKS both halves of standing order item 3: 1542 the deploy procedure requires a fresh backup of the affected cluster first, and the storage restore 1543 drill needs a backup to restore. 1544 1545FINDING D8, THE FROZEN ENGINE DRIFT CHECK IS ANCHORED TO THE WRONG COMMIT, THE SECOND FAIL OPEN IN THE 1546SAME CHECK. The most load bearing check in the protocol is still not trustworthy as documented. 1547 MEASURED: the diff of internal/modules/matching/engine against the documented anchor 9e77f65 is 1476 1548 LINES, not zero. Against 864a999, 9 July, the last commit that legitimately touched the engine, it is 1549 ZERO. Against e74fd2e, 27 July, the first commit of this build, it is ZERO. 1550 THE CAUSE: 9e77f65 is dated 24 JUNE 2026 and titled Baseline, missionMATCH platform plus Phase 2 1551 Authentication and Accounts. It is the ORIGINAL PROJECT baseline, not the pre build baseline. The engine 1552 was legitimately developed after it, through ModelVersion v1.1.0 to v1.4.0 across 26 and 29 June, 1 1553 July and 9 July. A diff against 24 June therefore CANNOT be empty and never could have been. 1554 WHY THE PRIOR RUN REPORTED ZERO ANYWAY, and this is the part worth keeping: when the history purge 1555 rewrote every hash, the anchor was repointed to 9e77f65 and the check was reported as verified at zero. 1556 The real path against that commit yields 1476, so that zero did not come from this path and this 1557 commit. Several plausible engine paths DO NOT EXIST in the tree, internal/platform/scoring, 1558 internal/scoring, internal/engine and pkg/engine, and git diff against a path that does not exist 1559 prints nothing and exits zero. That is the identical fail open shape as the missing commit, one layer 1560 over: the first fix moved the failure from a nonexistent COMMIT to a nonexistent PATH. 1561 THE FREEZE ITSELF HOLDS, stated as plainly as the defect. The engine has not changed since 9 July, 1562 eighteen days before this build started. Nothing this build did touched it. Acceptance 23 is satisfied 1563 in substance and the artifact is intact. What failed is the instrument that proves it. 1564 THE REPAIR, queued rather than done because it changes a gate definition: anchor the check at e74fd2e, 1565 the build's own first commit, which states the claim the gate actually wants, that THIS BUILD changed 1566 nothing. Assert the anchor commit exists AND that the path exists and is non empty before trusting a 1567 clean result, because both have now failed open once each. 1568 1569D7 CLOSED BY REPAIR, AND IT WAS THREE STALE CONSUMERS NOT TWO | 29 July 2026. JOSEPH'S APPROVAL, recorded 1570as the guardrails require: he approved the full repair, credentials plus partial cleanup plus a proven 1571backup, in answer to a proposal that named the config only scope and the absence of any ALTER ROLE. 1572 METHOD, deliberately the D3 method rather than a rotation: the role was never wrong, so the WORKING value 1573 already in api.env was copied into the stale files. Rotating would have broken the one consumer that 1574 worked. No ALTER ROLE, no cluster scoped object touched, so rule one did not apply. 1575 A THIRD STALE CONSUMER WAS FOUND DURING VERIFICATION, which is why the fix is not simply two files: 1576 sam-sync.env also carried a stale missionmatch_compliance password. It was not in the first diagnosis 1577 because the first probe only tested the migrator variable in that file. Found by widening the probe to 1578 every variable rather than the ones I expected to be wrong. 1579 PROVEN AFTER, by role name and status code, never by printing a DSN, eleven consumers across both nodes: 1580 backup.dsn OK, api.env migrator app compliance and tenant_acme all OK, sam-sync.env migrator AND 1581 compliance now OK, staging.env all three OK on their own database. 1582 ONLY THE INTENDED LINES CHANGED, proven not asserted: sam-sync.env stayed at 11 lines with 11 variable 1583 names, and a diff of the file EXCLUDING the changed line reported 0 differences, twice, once per repair. 1584 Originals backed up root owned 0700 before either write. Every write staged to /tmp and cp'd, because 1585 sudo tee truncates. 1586 D6 CONTAINMENT RE PROVEN, and MY OWN CHECK WAS WRONG FIRST. The first containment test reported 1587 "password authentication failed for user missionmatch_migrator", which looked like a pass and was not: my 1588 sed pattern matched the USERNAME inside the DSN, so it tested a role that does not own the staging 1589 password against the staging database and proved nothing. Corrected to substitute only the database after 1590 the port, after which missionmatch_staging_migrator against the PRODUCTION database is refused with 1591 "permission denied for database missionmatch", a privilege denial rather than an auth failure, which is 1592 the correct refusal and is what D6 built. Recorded because an instrument that lies in my favour is the 1593 thing this ledger exists to catch. 1594 THE BACKUP IS REAL AGAIN, which is the only proof that counts: mm-db-backup Result=success, tenant dump 1595 17209805 bytes, staging 2957058, control plane ingested 97184, manifest 3 artifacts and sha256sum -c OK 1596 on all three. Three zero byte partial dumps from the failed runs were deleted, size 0 only, and zero 1597 remain anywhere under the backup root. All four units reset, zero failed units. 1598 ALL THREE SYNCS PROVEN END TO END, not merely authenticated: sam-sync upserted 1420 with the store at 1599 2614 opportunities, awards-sync 525 with the store at 538, rates-sync 20 with the store at 1670. 1600 THE INVENTORY DEFECT FIXED IN THE SAME CHANGE per D6 rule nine: docs/CREDENTIAL_CONSUMERS.md gains both 1601 sam-sync.env rows, and records that one stale line there stops THREE units at once. It also records the 1602 harder lesson, that backup.dsn was ALREADY LISTED and was still missed, so a complete inventory is 1603 necessary and not sufficient. The rotation step that failed was step one, read this file, not a step 1604 that was absent. 1605 HOW LONG PRODUCTION HAD NO BACKUP, stated plainly: last complete set 28 July 16:42:58, first success 1606 after the repair 29 July 16:24:26, so roughly twenty four hours with three failed runs in between. The 1607 alerts DID fire, OnFailure was wired for all four units by the 28 July config deploy, so this was not 1608 invisible in the D2 sense. It was unread. 1609 1610D8 CLOSED BY CORRECTION, AND THE SAME WRONG HASH WAS ALSO THE DEPLOY BASELINE | 29 July 2026, approved by 1611Joseph, anchor at e74fd2e. 1612 THE SECOND USE IS THE DANGEROUS ONE AND IT HAD NOT BEEN NOTICED. 9e77f65 was named in TWO live places: 1613 the frozen engine drift anchor AND the HOW TO DEPLOY worktree baseline that every production binary is 1614 built from. The engine anchor being wrong made a gate report a result it had not measured. The DEPLOY 1615 BASELINE being wrong would have silently REGRESSED FIVE WEEKS of shipped product work on the next 1616 deploy, including engine ModelVersion v1.4.0, the audit_logs row level security, and the identity sync 1617 host fix, because 9e77f65 predates all of them. Production is not affected: it runs 8f24414531e80a9a, 1618 built from the REAL 462eb81 before the purge. The next deploy would have been affected, and the next 1619 deploy is the fairness snapshot deploy in the standing order. 1620 THE TRUE PRE BUILD BASELINE IS 2178e72, 21 July, which matches this ledger's own description of 462eb81 1621 as the 21 July edition. e74fd2e is the build's first commit and adds ONLY a 145 line spec document over 1622 2178e72, so it is code identical to the true baseline, carries ZERO build feature files, and gives an 1623 engine diff of zero. One anchor is therefore correct for both uses and states the claim the gate wants. 1624 CORRECTED IN THE LIVE INSTRUCTIONS ONLY: BUILD_CONTINUATION_PROTOCOL Gate D, the BUILD_STATE deploy 1625 block, and three lines in ARCHITECTURE.txt. Historical narrative entries KEEP the hash they were written 1626 with, because they were already rewritten once by the purge and rewriting the record a second time to 1627 match a later understanding is falsification, not correction. 1628 THE CHECK NOW REFUSES INSTEAD OF PASSING QUIETLY. It asserts the anchor commit exists AND that the path 1629 is non empty at both ends before trusting a clean result. BOTH FAIL OPEN MODES DEMONSTRATED: a diff 1630 against a nonexistent commit prints 0 lines, and a diff against a nonexistent path prints 0 lines, each 1631 indistinguishable from success without the assertion. Detector proven live at 120 lines on a file that 1632 did change. The freeze itself always held: the engine is unchanged since 9 July. 1633 1634SECTION 2.2 DONE, THE FAIRNESS SNAPSHOT PRODUCTION DEPLOY | 29 July 2026, under the standing order's 1635approval. Fresh backup first, taken 16:24:26 with its manifest verified, per the deploy procedure. 1636 A FOURTH D5 CONSUMER FOUND, AND IT IS NOT A FILE. migrate-all-tenants fans out to each tenant by REUSING 1637 the control plane migrator CREDENTIAL against the tenant's own host, which works only while the two 1638 clusters' migrator passwords are IDENTICAL. The D5 rotation changed cluster 1 and not cluster 2, so the 1639 fan out died with a SASL auth failure on the tenant database while the control plane leg was fine. The dry 1640 run caught it before any write, which is what the dry run is for. 1641 FIXED WITH THE TOOL'S OWN DOCUMENTED MECHANISM, not by re coupling the clusters: 1642 MM_TENANT_MIGRATOR_DSN_TENANT_ACME added to db.env on the control plane node. tenantDSN's own comment 1643 names this exact case, a tenant whose migrator credential differs because of a rotated password. The 1644 alternative, making the two passwords match again, would have restored the cross cluster coupling that 1645 caused D6, so it was deliberately not chosen. The value moved host to host through stdin, never through 1646 argv and never printed, and only the one variable changed, proven by a diff excluding it reporting 0. 1647 MIGRATIONS APPLIED, both targets asserted by role name, database and port BEFORE any write: 1648 control plane public v80 to v83 (+3), eeo unchanged at v2 1649 tenant public v80 to v83 (+3), eeo unchanged at v2 1650 0083's function eeo_snapshot_organizations present on BOTH, verified by pg_proc count. 1651 BINARY DEPLOYED. Production had been running the 30 JUNE build, which predates the D1b fix entirely, so 1652 the code fix had never been in production at all. New binary 740a432dba3bbdef, ELF x86-64 with 1653 GOFIPS140=v1.0.0 stamped, previous binary preserved at /root/fairness-snapshot.pre-0083. 1654 D1b IS CLOSED IN PRODUCTION. Before: "no organizations to snapshot" at every firing since the unit 1655 existed. After: "1 orgs snapshotted". The enumeration works. 1656 AND THE ARTIFACT IS STILL EMPTY, WHICH IS STATED RATHER THAN ROUNDED UP. 0 group rows were written, and 1657 the reason is DATA, not code: eeo.candidate_demographics holds ZERO rows in production, so there is 1658 nothing to compute a selection rate from. Measured: candidates 11, demographics 0, organizations 1. On 1659 staging the same binary wrote 83 group rows because staging carries demographic data. So the honest claim 1660 is that the JOB is fixed and the ARTIFACT awaits demographic data, and missionMATCH still cannot hand an 1661 assessor a populated LL144 report for production today. 1662 A SEPARATE OBSERVATION, recorded not acted on: eeo.fairness_snapshots already holds 2760 rows spread 1663 across 46 organization ids while the organizations table holds ONE. That is residue from the pooled era 1664 before the per tenant split. It is stale data in a compliance table, which is worth a decision. 1665 THE INDEPENDENT ASSERTION PROVEN TO FIRE IN PRODUCTION, as the standing order requires, run as a MUTATING 1666 REHEARSAL WITH ITS REVERT IMPLEMENTED AND VERIFIED FIRST per D6 rule four: 1667 captured the real function definition, 730 bytes, sha b0e81e17768da4d1 1668 stubbed it to return no rows while organizations still held 1 1669 the job REFUSED and exited 1, naming the D1b failure mode and saying a compliance artifact must never 1670 silently produce nothing 1671 reverted from the captured definition and PROVED the revert byte identical by diff 1672 counted run after the control: Result=success, 1 org snapshotted 1673 A function is database scoped, not cluster scoped, so rule one did not apply to this rehearsal. 1674 BUILD ROUTES RE PROVEN DARK AFTER THE DEPLOY, four samples each so both pool members answer: approve 404, 1675 scope 404, tracked 405. Positive control GET /api/auth/csrf 200, so a uniform 404 cannot be a dead host. 1676 Site 200, both node healthz 200. The API binary was NOT touched, only schema and a worker binary. 1677 1678SECTION 2.3 DONE, THE DISK ALERT AND THE BACKUP TARGET ON VALKYRIE | 29 July 2026, under the standing 1679order's approval, which named VALKYRIE and superseded the A1 and A2 choice in the storage proposal. 1680 PART ONE, THE 85 PERCENT ALERT, BUILT AND PROVEN. /usr/local/sbin/mm-disk-watch.sh with a 15 minute timer, 1681 WARN at 85, URGENT at 92, alerting through mm-alert.sh, naming the top five consumers in the body so the 1682 mail says what to delete. It alerts on a TRANSITION following the mm-watch.sh pattern, because a watch that 1683 mails every fifteen minutes trains the reader to ignore it, which is the same outcome as no watch. 1684 OnFailure wired to bubbles-failure-notify@%N, matching the convention the eight existing units use rather 1685 than a new template, and scoped by name so another project's failures cannot ride this path. 1686 POSITIVE CONTROL IN BOTH DIRECTIONS, exactly as the proposal specified: with the threshold forced to 1 the 1687 watch reported warn, ALERTED, and exited 1; restoring the real threshold alerted the RECOVERY and returned 1688 to ok; a second run at real thresholds was SILENT, proving the transition logic and not just the alert. 1689 Both mails confirmed SENT in the notifier's own send.log, not merely handed off. Real reading 82 percent, 1690 below threshold, timer active and enabled. 1691 PART TWO, THE BACKUP TARGET IS NOW OFF MACHINE. mm-db-backup.sh gained a verified push to VALKYRIE, 1692 beandaddy@100.89.0.6, into missionmatch-backups. PURELY ADDITIVE, proven: 43 lines added, 0 removed, 1693 bash -n passing, previous script preserved at /root/mm-db-backup.sh.pre-offsite. 1694 THE LOCAL COPY STAYS. It restores in seconds and that matters during an incident. What changed is that a 1695 root disk failure no longer takes the platform and every copy of its backups together, which was the 1696 actual risk in the proposal and the reason correlation beat capacity as the concern. 1697 IT VERIFIES THE FAR SIDE BY CHECKSUM rather than trusting the transfer's exit code, because this build has 1698 already met a push that reported success while three zero byte dumps sat on disk looking like backups. 1699 Counted run: offsite OK, 7 of 7 artifacts verified on VALKYRIE, unit Result=success. Remote retention 1700 prunes to 14 daily by count, POSIX only because the far side is a Mac. 1701 THE FAILURE PATH PROVED ITSELF FOR REAL, WHICH IS BETTER THAN A PLANTED CONTROL. The first counted run 1702 FAILED: the unit runs as ROOT and root had no authorized key on VALKYRIE, while my interactive test had run 1703 as user. The new fail LAST assertion fired, the unit exited 1, and OnFailure triggered. That is F1 from the 1704 Phase 2 trial in a new costume, a thing that works in one context and not under systemd, and it is the 1705 third time this build has been caught by exactly that gap. Fixed by authorizing root's ALREADY EXISTING 1706 public key rather than minting a new credential, appended not overwritten, 5 lines to 6. 1707 PART THREE, THE RESTORE DRILL RERUN AGAINST VALKYRIE AND IT PASSED. The drill's root was made overridable 1708 so it can be pointed at the off machine copy, because a drill that can only read the local copy cannot 1709 prove the copy that would actually be used after losing this host. 1710 Pulled the day's set BACK from VALKYRIE, 8 files, 58 MB, all verified against the manifest, then restored 1711 from THOSE bytes: RESTORE TIME 11 SECONDS for 16 MB, schema source v83 and restored v83 which independently 1712 confirms the 2.2 migration landed, candidates 11 reconciled, opportunity_records 6902 reconciled, and all 1713 ten reconciliation rows OK. pg_restore reported 3 errors, which the drill surfaces and tolerates by its own 1714 existing logic; recorded rather than omitted. 1715 THE NIGHTLY ATTACHMENT SCHEDULE REMAINS OFF, and it is off BY CONSTRUCTION: no attachment timer unit 1716 exists on the host at all, so there was nothing to disable and nothing was created. Both preconditions the 1717 standing order set are now met, so enabling it is available and is deliberately left as Joseph's call 1718 rather than taken as implied. 1719 1720FINDING D10, A DEAD CREDENTIAL IN PLAINTEXT IN AN OPS SCRIPT, AND THE RESTORE DRILL HAS BEEN BROKEN SINCE 1721D5 | Found while running the approved drill, which is the only reason it surfaced. 1722 mm-restore-drill.sh hardcoded a migrator DSN WITH THE PASSWORD IN PLAINTEXT on one line. That made it a 1723 FIFTH D5 consumer, listed in no inventory, and one the repository secret sweep can NEVER see, because the 1724 sweep scans the repository and this file lives in /usr/local/sbin. A credential store that only covers the 1725 repository does not cover the credentials. 1726 THE CONSEQUENCE: the D5 rotation left it holding the OLD value, so THE RESTORE DRILL COULD NOT RUN. The one 1727 instrument whose entire purpose is to prove a backup is restorable was itself broken by a rotation, and the 1728 protocol requires it at every phase close. 1729 I REPORTED THIS WRONG FIRST AND THE CORRECTION MATTERS MORE THAN THE FINDING. My initial probe said the 1730 hardcoded credential was STILL LIVE, which would have meant a burned credential active in production and 1731 the whole D5 window pointless. It was MY INSTRUMENT THAT WAS WRONG: I tested by grepping the output for the 1732 role name, and the failure message itself contains user "missionmatch_migrator", so a FATAL authentication 1733 failure matched my success pattern. Retested with .pgpass explicitly disabled and by comparing salted 1734 hashes of the password held by each source: the drill's value equals the PRE repair backup.dsn value and is 1735 DEAD, api.env holds the distinct live value. So the D5 rotation DID work, and the correct reading is the 1736 ordinary one, stale copies of a dead credential. Recorded in full because a check that matches its own 1737 error message is the exact failure this ledger exists to catch, and I made it. 1738 FIXED, AND HARDENED BEYOND THE REPAIR: the script no longer stores a credential at all. It reads 1739 MM_MIGRATOR_DATABASE_URL from api.env and refuses to run if it is absent, so one file changes at a 1740 rotation and this follows without anyone remembering it exists. Verified: zero credential shaped lines 1741 remain in the file, and it resolves as missionmatch_migrator. Previous version kept at 1742 /root/mm-restore-drill.sh.pre-d10. 1743 STILL OPEN, and it is Joseph's: docs/CREDENTIAL_CONSUMERS.md covers files on two hosts and does NOT cover 1744 /usr/local/sbin scripts on either, which is where two of the five broken consumers actually lived. The 1745 inventory needs a section for ops scripts, and the secret sweep needs a way to see outside the repository. 1746 1747FINDING D9, THE AMENDMENT VERSION COUNTER IS STUCK AND THE ERROR IS SWALLOWED | Found 29 July 2026 in the 1748first successful sam-sync run after the D7 repair, which is the only reason anyone saw it. 1749 THE LOG LINE: amendment for opportunity afc5143e insert version: duplicate key value violates unique 1750 constraint uq_opp_versions_number. The run still exits 0, because the design says the provenance write 1751 never fails the run, so this has been failing silently since 27 July 22:28. 1752 THE STATE IS WORSE THAN THE ERROR. opportunity_versions holds exactly ONE row for that opportunity, 1753 version_number 1, is_current FALSE. So the predecessor was RETIRED and the successor was NEVER WRITTEN, 1754 and the opportunity now has NO CURRENT VERSION. The retire and the insert were meant to be one 1755 transaction and did not behave as one here. 1756 IT IS SELF PERPETUATING, which is the part that matters. With no current row, the next version number 1757 computes as 1 again, collides with the existing version 1, and fails forever. No amendment can ever be 1758 recorded for that opportunity again without intervention. 1759 SCOPE MEASURED, not assumed: 1 version row in the whole production table, 1 opportunity with versions, 1760 1 with no current version, 0 with multiple current. So the breach is total for the only tracked 1761 opportunity and contained to it. 1762 WHY IT DOES NOT INVALIDATE ACCEPTANCE ITEM 11: item 11 asserts an amendment FLAGS and never REWRITES an 1763 approved requirement, and that held in the Phase 2 trial on real amendment text with the md5 identical 1764 before and after. This is a different guarantee, the is_current invariant on the version chain. 1765 NOT FIXED. It needs an application change plus a decision about the one stranded row, both outside the 1766 approved scope. QUEUED for Joseph. 1767 1768## SUPERSEDED HANDOFF, 29 July 2026 evening, recovery session. Kept as a record, no longer current. 1769 1770DEMOTED, NOT DELETED, and the demotion is itself the fourth instance of this problem. This was the current 1771handoff when it was written. A later session in the same day continued the work, which made it stale, and when I 1772added the new handoff at the top of this file I created TWO, which is precisely the defect the block below warns 1773about in its own first paragraph. The check for it, grep for the heading and require exactly one, found it within 1774a minute of the commit. 1775THE RULE THIS FINALLY EARNS: writing the handoff last is necessary and not sufficient. The old one has to be 1776demoted in the same edit, or the file grows a second source of truth every session. Superseded handoffs keep 1777their content and lose their title. 1778 1779PHASE 2 IS NOT CLOSED AND THE PRE APPROVAL DID NOT FIRE. It is conditioned on zero open findings and 1780zero new deviations. This session opened FOUR findings, D7, D8, D9 and D10, and closed two of them. By 1781the standing order's own terms that is not a green close, so Phase 3 was NOT started and Phase 2 1782remains the one current phase. 1783 1784DONE THIS SESSION, all verified, all recorded above 1785 Reconstruction from the directive chain and the live systems. The stale handoff replaced, third time. 1786 Standing order 1, 2 and 4 confirmed already done. All 16 package checksums match the order exactly. 1787 Session start drift verification with every control firing. Evidence at 1788 docs/evidence/phase2/drift/session-start-drift-4.txt 1789 D7 CLOSED. Three stale credential consumers repaired, production backup restored after roughly 1790 twenty four hours without one, three zero byte partials removed, all three syncs proven end to end, 1791 CREDENTIAL_CONSUMERS.md corrected in the same change. 1792 D8 CLOSED. The frozen engine anchor was the 24 June project baseline, and the same wrong hash was 1793 also the DEPLOY baseline, which would have regressed five weeks of shipped work on the next deploy. 1794 Anchored at e74fd2e. The check now asserts the commit and the path exist. 1795 SECTION 2.2 DONE. 0081, 0082, 0083 to both production databases, v80 to v83. Fairness binary deployed, 1796 replacing a 30 JUNE build that predated the D1b fix. Independent assertion proven to fire in 1797 production as a mutating rehearsal with a verified revert. Build routes re proven dark. 1798 SECTION 2.3 DONE. 85 percent disk alert built and proven both directions. Backups now pushed to 1799 VALKYRIE and verified there by checksum. Restore drill rerun against the VALKYRIE bytes and PASSED, 1800 11 seconds, schema v83 both sides, all ten reconciliation rows OK. 1801 1802OPEN, AND EVERY ONE IS JOSEPH'S 1803 D9 the amendment version counter is stuck on one opportunity, is self perpetuating, and its error is 1804 swallowed by design. Needs an application change plus a decision on the stranded row. 1805 D10 remainder: CREDENTIAL_CONSUMERS.md covers files on two hosts and NOT /usr/local/sbin scripts, 1806 where two of the five broken consumers actually lived. The secret sweep cannot see outside the repo. 1807 THE AUTHENTICATED PRODUCTION LEG WAS APPROVED AND WAS NOT RUN. Joseph approved provisioning and 1808 retiring a probe account. I stopped short of starting it because it is a production identity change 1809 and I judged the remaining context too short to finish it cleanly, and a half provisioned production 1810 account is worse than none. This is the leg that previously caught a 500 hiding behind the session 1811 gate, so it is a real gap and it is named rather than quietly dropped. IT IS THE FIRST THING TO DO. 1812 THE NIGHTLY ATTACHMENT SCHEDULE is off by construction, no unit exists. Both preconditions the 1813 standing order set are now met, so enabling it is available and is deliberately left as his call. 1814 eeo.fairness_snapshots holds 2760 rows across 46 org ids while organizations holds one, pooled era 1815 residue in a compliance table. 1816 THE FAIRNESS ARTIFACT IS STILL EMPTY IN PRODUCTION and that is data, not code: 1817 eeo.candidate_demographics holds zero rows, so there is nothing to compute a selection rate from. 1818 1819STATE AT STOP, every number verified after the last change 1820 401 commits, HEAD 6978143, tree CLEAN, stash empty. 1821 53 packages ok and 0 FAIL, merge gate PASS with verify-backdoor and verify-fips both PASS. 1822 Secret sweep CLEAN on tree and on history, control firing. 1823 Engine diff 0 against e74fd2e, with the anchor and path assertions in place. 1824 Tenant v83, control plane v83, staging v83, each queried directly, all three level. 1825 Both production nodes 8f24414531e80a9a, identical, healthz 200 each. 1826 Public site 200, build routes dark 404/404/405. Zero failed units on Bubbles. 1827 Last backup 29 July, verified locally AND on VALKYRIE by checksum, restore drill passed against the 1828 off machine copy. 1829 Bubbles root disk 82 percent with the alert now live on a 15 minute timer. 1830 1831## PHASE 2 SCOPE COMPLETION AND GATE RE RUN, 29 July 2026 1832 1833PHASE 2 WAS LESS FINISHED THAN THE PREVIOUS GATE E CLAIMED, and the gap was SCOPE rather than verification. 1834Reading the directive against the code, rather than reading the ledger, found four requirements that did not 1835exist in any form. That is a different and worse category than the known limits the last Gate E listed. 1836 SECTION 5.1 LISTS TABLES among the required deterministic extraction targets. Nothing read tables at all. 1837 That is the actual reason the wage determination dropped 231 of 232 spans: the prose gate correctly refused 1838 rows of codes and numbers and NOTHING ELSE WAS LOOKING AT THEM. It was recorded as a capability the phase 1839 did not claim, when it was a capability the directive requires. 1840 SCREEN 4 NAMES SIX REVIEW ACTIONS and four existed. 'merged' was a status value nothing ever wrote and split 1841 had no representation of any kind. Migration 0084 adds the split status, widened from the CURRENT 1842 vocabulary because this build already learned that lesson the expensive way. 1843 SCREEN 5 REQUIRES GENERATING DRAFTS from accepted scope and nothing did, so the review to requirement path 1844 had a hole in the middle of it. 1845 SCREENS 3, 4 AND 5 HAD NO UI AT ALL, and neither did the amendment view, while Gate C names every one of 1846 them. The scope review and promotion routes were mounted and UNREACHABLE, which is the same defect shape as 1847 the promotion routes that were once built and never mounted. 1848All four are built, tested, and deployed to staging. Evidence at docs/evidence/phase2/functional/ 1849gate-a-phase2-close.txt, stress/gate-b-results-2.txt, drift/gate-d-phase2-close.txt, and GATE_E.md rewritten. 1850 1851TWO GATE 2 CRITERIA HAD NEVER BEEN EXECUTED. ingest_during_promotion_test.go sits behind a build tag and skips 1852without pools, so the 53 package suite never touched it, and it carried two of Gate 2's six clauses. Both now 1853run and pass: an ingest run during promotion corrupts nothing, and an approved requirement survived 20 ingest 1854rounds byte for byte at md5 6acbad7d5a24cb161b141e1d5a99e717. All six Gate 2 criteria are now proven. 1855 1856THE REAL DOCUMENTS CAUGHT MY OWN WORK, TWICE, AND THAT IS THE PART WORTH READING. 1857 THE WAGE DETERMINATION IS NOT RECOVERABLE BY THIS BUILD, AND IT NOW SAYS SO. Before: extraction done, 1 1858 chunk stored. After: FAILED no_text_layer, 0 chunks. That is an IMPROVEMENT. The stored chunk was mojibake, 1859 because that PDF's text layer decodes to character soup from an embedded font with no usable character map, 1860 and reporting done over a garbage chunk is worse than failing because a reviewer cites it. 1861 MY TABLE READER MADE IT WORSE FIRST. Against the real document it recovered "tables" of content like "(0 CC" 1862 and "CC A3": mojibake contains tabs and therefore satisfied every structural test I had written. A structural 1863 reader that BYPASSES the prose gate must carry its own quality bar or it becomes the hole in it. The bar is 1864 the weakest thing that separates language from soup, a run of four letters containing a vowel. 1865 THE SAME BAR CLOSED A PRE EXISTING HOLE: that mojibake had been passing the PROSE gate all along. 1866 AND IT COST THE LEGITIMATE CASE NOTHING, measured on the same run: 04_PWS_original went 54 chunks to 55, 1867 GAINING a table of 3 rows. Table extraction is proven by 7 unit tests on real shaped input and by the volume 1868 profile parsing a 200 page PWS into 400 chunks, 200 tables and 1600 rate rows in 19 milliseconds. 1869 1870F6 FIXED, AND MY FIRST FIX WOULD HAVE FAILED ON THE REAL DATA. Pairing purely within a document type looked 1871correct against a fixture I wrote where both PWS documents were typed pws. The intake actually classified the 1872amended PWS as an AMENDMENT, so pws held one document and the pairing fell straight through to the old wrong 1873answer. It now matches the amendment to the baseline whose FILE NAME it most resembles, version words stripped. 1874On the real Package B it pairs 04_PWS_original against 05_PWS_amended and reports ONE material change, quantity 18751 to 7, flagging the one derived approved requirement. Previously five changes, partly artifacts. Also removed 1876attachment and other from the pairable types: an earlier version would have diffed an RFC log against a 1877deliverables list, which is worse than the bug being fixed. 1878 1879PURE TESTS MISSED TWO REAL BUGS, EXACTLY AS F3 PREDICTED. Split, merge and generation shipped with pure 1880validation tests only, which is F3's shape precisely. Database touching tests found both in seconds: FOR UPDATE 1881used with an aggregate, which Postgres refuses outright, and a nil Go slice arriving as NULL in a NOT NULL array 1882column so generation failed on every document naming no certifications. Six integration tests now cover split, 1883merge, generation and manual completion, asserting the properties that matter: children inherit the citation, 1884generation from UNREVIEWED scope yields nothing, and a field the document never states is reported missing 1885rather than guessed. 1886 1887GATE B EXTENDED WITH THE THREE PROFILES THE PROTOCOL NAMES AND NOBODY RAN: volume, concurrent-extract and 1888scrub-flood. Race at 8 and 32 both elect one winner, load holds p95 9.595ms against a 300ms target with zero 1889errors, scrub flood reports 0 misroutes and 0 false refusals across 1000 checks. 1890 TWO OF MY OWN CONTROLS WERE BROKEN FIRST. The memory profile measured CUMULATIVE allocation, which is 1891 identical whether results are retained or discarded, and its control returned an error UNCONDITIONALLY so it 1892 reported success while detecting nothing. Its cap was then a number chosen from nothing, 4096 KB, which the 1893 real leak came in under. The scrub control laundered payload CONTENT while keeping provenance and then blamed 1894 the gate for correctly allowing a payload whose only private element had been stripped. Both fixed and both 1895 recorded, because a control that passes by construction is the exact failure this protocol exists to catch. 1896 1897FINDING D11, FOUND BY GATE D ITSELF. Migration 0084 FAILED on staging with "must be owner of table 1898opportunity_scope_items". The D6 structural fix created missionmatch_staging_migrator and mirrored GRANTS but 1899left every staging object OWNED by missionmatch_migrator, and grants do not permit ALTER TABLE, so staging could 1900not apply a DDL migration through the migrator that exists to migrate it. Fixed by explicit per object transfer 1901of 88 tables and 197 functions, deliberately NOT via REASSIGN OWNED, which also moves shared objects and could 1902have changed the PRODUCTION database owner as a side effect. Verified after: production tables still owned by 1903missionmatch_migrator, both database owners unchanged. 1904 THE go:embed GOTCHA BIT AGAIN. The first production run of 0084 reported +0 because the migrator binary 1905 predated the migration. The ledger already carried this lesson. Rebuilt, redeployed, +1 on both databases. 1906 1907ALL THREE SCHEMAS ARE NOW LEVEL AT v84, queried per database. Production carries 0084 because it is additive 1908and carries no data, the same treatment 0081 through 0083 received, and the routes stay dark. 1909 1910STAGING RUNS THE NEW BUILD: api 0fb3629d44d7b18a with the FIPS stamp, bundle index-g1gTV6We.js. index.html was 1911EDITED in place with only the two asset hashes rewritten, so all 22 lines and all 6 injected SEO lines survived. 1912All four new screens serve 200 over HTTPS and the shipped bundle contains every one of their strings. 1913 1914THE AUTHENTICATED LEG RAN END TO END ON STAGING through a probe account whose password was set with 1915cmd/rotate-credential so it went through the real pepper and PBKDF2 path. Login 200, session 200, and every new 1916route 200: scope, requirement-drafts, amendments, documents, and generate. The amendment view returned the same 1917affected approved requirement the diff independently flagged. The probe's tool grants were restored afterwards. 1918 TWO REFUSALS ALONG THE WAY WERE BOTH CORRECT. A 403 was the CSRF middleware refusing a request with NO Origin 1919 header, which curl does not send: a hard Origin allowlist working as designed. The 404s that followed were the 1920 TOOL ENTITLEMENT gate failing closed on an account with no slateMATCH grant, answering 404 rather than 403 so 1921 a surface is indistinguishable from a path that was never built. Mistaking either for a defect would have sent 1922 me chasing the wrong thing. 1923 1924GATE C RUN FOR THE FIRST TIME AGAINST THE PHASE 2 SCREENS, because until today there were none to sweep. 1925Evidence at docs/evidence/phase2/visual/phase2-close/. 1926 COUNTED RUN: surface authenticated, viewports 768, 1024 and 1440, 36 captures, ZERO findings. That is 6 1927 routes by 3 viewports by 2 roles, the full matrix with nothing skipped, as seeded recruiter and manager 1928 fixtures logging in through the REAL login flow with the slateMATCH entitlement. 1929 THE CONTROL PROVED SOMETHING THE COUNTED RUN CANNOT. A planted 4000 pixel overflow was detected on every one 1930 of the six routes at every viewport, and the planted minimum width regression with it. To report an overflow 1931 on /slate/capture/{id}/scope the sweep had to log in, hold the entitlement, resolve the route and render the 1932 screen, so a control that fails on all six routes is simultaneously proof that all six are reachable. 1933 SHELL GUARD: the widest route needs 716px against a 732px baseline plus a 24px allowance, so the new screens 1934 sit inside the shell's budget rather than on its edge. 1935 VIEWPORTS ARE 768 AND UP for the authenticated surface, which is the recorded product decision of 28 July and 1936 not a relaxation taken here. The public surface still holds all five widths. 1937 TWO CONSOLE ERRORS did not reproduce on retry and were dropped to transient.json rather than counted or 1938 hidden, so the decision to drop them is auditable. 1939 PIXEL BASELINES REMAIN UNAVAILABLE BY PROTOCOL rather than by omission: a baseline becomes one only after 1940 Joseph approves a contact sheet. The sheet from this run is in the evidence directory. 1941 1942D9 IS NOW FIXED IN PRODUCTION, NOT ONLY IN CODE, and the stranded row healed itself. The fixed sam-sync, 194375ad042b415cfd03, was deployed to Bubbles with the previous binary preserved at /root/sam-sync.pre-d9. 1944 BEFORE opportunity afc5143e held version 1 with is_current FALSE, so there was no current version and every 1945 future amendment for it was permanently unrecordable. 1946 AFTER version 2 landed alongside the retired version 1. The invariant is back to exactly ONE current 1947 version of two total, and the run reported "wrote 1 new opportunity versions and raised 1 alerts" 1948 with Result=success and no amendment failures, so the new fail last assertion did not need to fire. 1949 THE STRANDED ROW NEEDED NO DECISION AFTER ALL. GATE_E had listed it as awaiting Joseph's call; the correct 1950 numbering simply resolved it on the first run, which is what a self perpetuating bug looks like once the 1951 thing perpetuating it is gone. 1952 1953GATE C FIXTURE ACCOUNTS, recorded because they persist. gatec.recruiter@fg-talent.test and 1954gatec.manager@fg-talent.test exist on STAGING ONLY, in the Package B organization, holding slateMATCH, seeded 1955as the protocol's "seeded fixture accounts per role". Their passwords were set through cmd/rotate-credential so 1956they go through the real pepper and PBKDF2 path, and the values are NOT recorded here or anywhere in the 1957repository, per finding D4. Re set them with rotate-credential when the next sweep needs them. 1958 1959## THREE MORE FINDINGS, ALL FROM THE LAST HOUR, TWO OF THEM MINE | 29 July 2026 1960 1961FINDING D12, ONE 97 KB DOCUMENT KILLED THE ENTIRE EXTRACTION BATCH. 07_TE6_Deliverables.pdf from Package B is 196297,642 bytes and six pages, and opportunity-extract was OOM KILLED on it, exit 137. It printed no verdict for 1963the document and never reached its own DONE line, so one small file took the whole run down. 1964 THAT CONTRADICTED THE PROGRAM'S OWN PROMISE, written at the top of its main file: a document it cannot read is 1965 marked failed WITH A REASON while the run continues. Being killed by the kernel is not a reason. 1966 DIAGNOSED RATHER THAN GUESSED. A page by page diagnostic showed NewReader returning cleanly and reporting six 1967 pages, then the process dying inside the FIRST page's GetPlainText before any text came back. So the runaway 1968 allocation is inside ONE library call, which rules out every in process fix: a page bound does not help when 1969 page one is the problem, an output bound does not help when the allocation is inside the callee, and the 1970 existing panic recovery does not help because a process the OOM killer takes never runs a deferred function. 1971 FIXED BY ISOLATION. The library now runs in a CHILD process, the extractor re executing itself in a worker 1972 mode that reads bytes on stdin and writes text on stdout, under a 90 second deadline, an output cap, and on 1973 Linux a 1 GiB address space limit the child sets on itself. A dead child is a fact about one file, exactly as 1974 a panic already was, and the parent falls through to the hand rolled reader. It re executes ITSELF so there is 1975 no new dependency and the no Python rule is untouched. 1976 WHY THE LOCAL CAP MATTERS BEYOND THIS FILE: the OOM killer does not necessarily kill the process that asked 1977 for the memory. On staging it chose the extractor. On a busier host it could have chosen the API. 1978 PROVEN: the batch now reports "ok=1 failed=1" with the document named no_text_layer, and a regression test 1979 guards it, skipped unless the fixture is present because a 97 KB binary does not belong in the repository. 1980 ALSO FIXED WHILE IN THERE: inflateStreams bounded each stream to 32 MB and had NO ceiling on the SUM, while 1981 its own comment claimed a decompression bomb was ruled out. It was not. There is now an aggregate budget, and 1982 exceeding it REFUSES the document rather than returning truncated text that looks complete. 1983 1984FINDING D13, MY OWN D11 FIX BROKE THE NIGHTLY BACKUP, and the alerting caught it. Transferring ownership of 1985every staging object to missionmatch_staging_migrator removed missionmatch_migrator's implicit owner 1986privileges, and the backup dumps STAGING using that role. pg_dump failed with "permission denied for table 1987goose_db_version" and the unit failed. 1988 THE OnFailure WIRING FROM D2 PAID OFF ON A REGRESSION I CAUSED. The failure raised mail rather than sitting in 1989 a journal, which is exactly what that work was for. 1990 FIXED WITH READ ACCESS, NOT OWNERSHIP BACK: SELECT and schema USAGE on both schemas, plus DEFAULT PRIVILEGES 1991 so the next migration does not reintroduce it on a night nobody is watching. Ownership stays with the staging 1992 migrator, which is what lets staging DDL run at all. 1993 IT TOOK TWO PASSES AND THE SECOND ONE IS THE LESSON. The first grant covered every TABLE in both schemas and 1994 sequences in public only, so pg_dump got past all 88 tables and then failed on candidate_demographics_id_seq 1995 in the eeo schema. A partial grant moves the error later without removing it. 1996 DIRECTION MATTERS AND IS WHY THIS IS SAFE: D6's isolation exists to stop STAGING reaching PRODUCTION, and this 1997 is the opposite direction. Asserted afterwards: missionmatch_staging_migrator still has CONNECT on the 1998 production database REVOKED. 1999 BACKUP RESTORED AND PROVEN: Result=success, 11 artifacts, offsite OK 11 of 11 verified by checksum on 2000 VALKYRIE. Restore drill rerun: PASSED, 11 seconds for 16 MB, schema v84, all ten reconciliation rows OK. 2001 2002FINDING D14, A LIVE CREDENTIAL WAS DISCLOSED INTO A LOG, BY ME, AND THIRTEEN BINARIES COULD HAVE DONE THE SAME. 2003 WHAT HAPPENED. I added the PDF worker mode but checked its hidden flag AFTER flag.Parse. The flag package saw 2004 an unrecognised flag, printed its usage, and USAGE PRINTS DEFAULTS. The DSN was a flag default read from the 2005 environment, so the live missionmatch_staging_migrator password was written to stdout and into the staging 2006 journal. One unrecognised flag, one disclosed password. 2007 THE SYSTEMIC HALF IS WORSE THAN MY MISTAKE. A sweep found SEVENTEEN flag defaults across THIRTEEN binaries 2008 reading secrets from the environment, including cmd/rotate-credential, whose usage output would have printed 2009 the CREDENTIAL PEPPER, the server side key applied before PBKDF2. Every one of them disclosed a live secret on 2010 any usage error, any typo, any unknown flag. This was latent long before today and my mistake only triggered it. 2011 FIXED IN ALL THIRTEEN. No secret is a flag default any more: the flag defaults to empty, the help text NAMES 2012 the environment variable instead of showing its value, and the value is read after parsing. PROVEN with a 2013 planted secret in six environment variables: 0 of 13 binaries disclose it, and the check was itself controlled 2014 by reverting one binary, watching the leak reappear, and restoring it. 2015 ROTATED, BECAUSE A DISCLOSED CREDENTIAL THAT IS MERELY REDACTED IS STILL DISCLOSED. Followed the documented 2016 procedure in order: fresh backup, update the one consumer FIRST, ALTER ROLE LAST, restart what reads its DSN at 2017 boot, prove by role name and status code. 2018 PROVEN OLD DEAD AND NEW LIVE: the disclosed value is now refused with "password authentication failed", the 2019 consumer authenticates as missionmatch_staging_migrator on missionmatch_staging, staging healthz 200, and 2020 CONNECT on production is still revoked. Integration tests re run green against the rotated credential. 2021 SCOPE, STATED HONESTLY: this role reaches missionmatch_staging ONLY, and that was already proven by refusal 2022 during D6, so the exposure was staging and not production. The value also remains in the staging systemd 2023 journal until it rotates out, which cannot be scrubbed and is recorded rather than glossed. 2024 I ALSO BLAMED THE ROTATION FOR SOMETHING IT DID NOT CAUSE, AND THE CORRECTION IS D16. A 30 minute soak was in 2025 flight when the rotation landed and it ran ten minutes past its deadline still burning CPU, so I attributed 2026 that to the rotation breaking its connections. A SECOND soak, started afterwards on the correct credential 2027 from the beginning, did exactly the same thing. The rotation was innocent. See D16. 2028 2029FINDING D16, THE STRESS TOOL CANNOT REPORT A SOAK OF THE LENGTH THE PROTOCOL ASKS FOR, AND NEVER COULD | 29 July 20302026. Two long soaks were lost before this was diagnosed, and the first loss was wrongly blamed on the D14 2031credential rotation happening at the same moment. 2032 THE CAUSE, found by elimination rather than by guessing again. pg_stat_activity showed NO active or blocked 2033 queries on staging, so the workers were not waiting on the database. The CPU was being burned AFTER the load 2034 finished, in the reporting step, and percentile() used an INSERTION SORT: 2035 // Sorting a few thousand durations is not the bottleneck here and a simple implementation is easier to 2036 // trust than a clever one in a measuring instrument. 2037 The reasoning was sound and the premise was false. Thirty minutes at roughly a thousand reads per second is 2038 about 1.8 MILLION samples, not a few thousand, and insertion sort is O(n squared): on the order of a trillion 2039 operations, run TWICE for p95 and p99. 2040 WHY NOBODY SAW IT. The first Gate B bundle ran a SIXTY SECOND soak, which is 60,000 samples and completes in 2041 seconds. The protocol asks for THIRTY MINUTES. So the protocol's soak requirement has never once been 2042 measurable with this instrument, and the gap was invisible because the only soak ever run was 30 times shorter 2043 than the one required. 2044 A MEASURING INSTRUMENT THAT CANNOT REPORT THE MEASUREMENT THE PROTOCOL ASKS FOR IS NOT A SLOW INSTRUMENT, IT 2045 IS A MISSING ONE. Every number was collected correctly and none of it was printable. 2046 FIXED with sort.Slice. Proven: a 40 second soak now reports 45,335 samples at p95 8.756ms instantly, and the 2047 full 30 minute run was started afterwards. 2048 THE LESSON WORTH KEEPING: a comment justifying a simplification is a claim about the input, and that claim 2049 needs checking against the input the protocol actually specifies rather than against the input the author had 2050 in mind. 2051 2052FINDING D15, THE UPTIME WATCH RAISED A FALSE ALARM AND THE ALARM TEXT HID THE REASON | 29 July 2026, found 2053during the final state check because mm-watch showed as a failed unit. 2054 THE ALERT SAID: "PUBLIC SITE: https://missionmatch.thatwebhostingguy.com/ returned 200000, expected 200". The 2055 site was answering 200 the entire time. 2056 THE BUG IS ONE SHELL IDIOM: 2057 code() { curl -s -o /dev/null -w "%{http_code}" --max-time "${2:-15}" "$1" 2>/dev/null || echo 000; } 2058 When curl PRINTS a status code and THEN exits nonzero, which it does on a timeout or a partial transfer after 2059 the headers have already arrived, BOTH outputs land: the real code from -w and the 000 from the fallback. The 2060 caller compares "200000" to "200", it does not match, and the watch reports an outage that is not happening. 2061 WHY IT MATTERS MORE THAN ONE FALSE ALERT. A monitor that cries wolf is a monitor that gets ignored, and this 2062 ledger has written that sentence about other instruments more than once. It is also SELF CONCEALING: the alert 2063 text is unreadable, so the natural reaction is to distrust the message rather than to suspect the checker. 2064 FIXED: the helper accepts exactly three digits from curl and substitutes 000 otherwise, so the two outputs can 2065 never concatenate. Previous version kept at /root/mm-watch.sh.pre-d15. 2066 PROVEN IN BOTH DIRECTIONS. Real run: OK, exit 0. Control, pointed at a hostname that cannot serve the app: 2067 "PUBLIC SITE: ... returned 000, expected 200", exit 1. The message is now readable and the detector still 2068 detects, which is the pair of properties that were both broken. 2069 2070## INTEGRITY CHECK, GATE B CLOSURE, AND TWO MORE OF MY OWN REGRESSIONS | 29 July 2026, evening 2071 2072INTEGRITY CHECK AFTER A SERVER ERROR, AND THE HONEST ANSWER IS THAT NOTHING WAS DAMAGED. A server error 2073interrupted a Gate B evidence update mid write, so every evidence and ledger file touched in that window was re 2074read and verified rather than assumed intact. 2075 CHECKED: ten files. All ten end with a newline and a sensible final character. Zero duplicate section headings 2076 where exactly one is expected, checked for the handoff, the reconstruction entry and the scope completion 2077 entry. Working tree CLEAN at fff3069, so nothing half written was sitting uncommitted either. 2078 THE FINDING IS A GAP, NOT A CORRUPTION. gate-b-results-2.txt was intact at 87 lines and contained ZERO 2079 occurrences of the http-load or D16 content it was about to receive: the interrupted write never began. So 2080 there was nothing to repair and something genuine to finish. The source numbers all survived in the scratch 2081 logs and the file was completed from them rather than from memory. 2082 RECORDED BECAUSE A CLEAN INTEGRITY CHECK IS STILL A RESULT. The instinct after an interruption is to assume 2083 damage and start rewriting, and rewriting a file that was fine is its own way of losing information. 2084 2085THE PRODUCTION AUTHENTICATED LEG IS DONE, closing the gap the reconstruction entry named and the standing order 2086approved. Method of record followed exactly: a dedicated probe provisioned, used, and retired. 2087 PROVISIONED phase2.probe@missionmatch.test in the CONTROL PLANE, role viewer, NO tool grants, lowest 2088 privilege. Password set through cmd/rotate-credential so it went through the real pepper and PBKDF2 path 2089 rather than a hand written hash. 2090 LOGGED IN THROUGH THE PUBLIC PRODUCTION URL: login 200, session 200. 2091 BUILD ROUTES DARK ON THE AUTHENTICATED LEG, four samples each so both pool members answer: 2092 approve 404 404 404 404, scope 404 404 404 404, tracked 405 405 405 405 2093 AND THE FIVE NEW PHASE 2 ROUTES ARE DARK BEHIND A VALID SESSION TOO, which is the point of checking them: 2094 split 404, merge 404, generate 404, amendments 404, draft PATCH 404 2095 ZERO 5xx ON EITHER NODE during the window, checked per node rather than inferred: Bubbles 0, megamind-agi 0. 2096 The nginx log carries 25 5xx TOTAL across its whole history, every one from 06:40 this morning on the STAGING 2097 vhost, which are the known F3 upload failures. ZERO in the window. My first count of 44 was a bad grep and is 2098 corrected here rather than left in the record. 2099 RETIRED, never deleted: status disabled, sessions revoked through the product's own 2100 auth_revoke_user_sessions, 0 live sessions with the row and its 1 historical session preserved. PROVEN 2101 retired: a fresh login returns 401 and the old session cookie returns 403. 2102 2103GATE B IS COMPLETE. Every Phase 2 stress item and every general Gate B item has now been executed with a 2104control that failed first. Evidence at docs/evidence/phase2/stress/gate-b-results-2.txt. 2105 THE 30 MINUTE SOAK REPORTED FOR THE FIRST TIME: 2,028,801 samples, rps 1127, p95 8.931ms, p99 10.503ms, zero 2106 errors, against a 300ms target. Two million samples is itself the proof of D16. 2107 AUTHENTICATED HTTP LOAD, open across TWO phase closes, now closed: 25,924 samples over 10 minutes paced to the 2108 protocol's 50 rps, p95 37.352ms, p99 67.644ms, ONE HUNDRED PERCENT SERVED, zero 429, zero 5xx. 2109 THE FAILURE DRILL IS COMPLETE IN ALL THREE PARTS. Failover was measured for real by D6. Saturation produced 2110 17,314 of 429 and ZERO of 500. And the worker kill drill ran: five documents pending, SIGKILL 1.2 seconds in 2111 having decided 3 of 5, rerun completed the other 2, all five done, zero pending, and ZERO chunks sharing a 2112 document, a content hash and a section label. The one same hash pair that exists is legitimate, the literal 2113 text "Warranty Program. N/A" appearing twice in the source. 2114 2115FINDING D17, I DELETED flag.Parse AND DEPLOYED THE RESULT. While removing a duplicated worker check block from 2116cmd/opportunity-extract, my edit took the flag.Parse call and both environment fallbacks with it. Every flag 2117stayed at its zero value, so the binary fatalled on "-dsn or MM_MIGRATOR_DATABASE_URL is required" on a host 2118where that variable was plainly set. It was DEPLOYED to staging in that state. 2119 IT FAILED LOUDLY, WHICH IS THE ONLY REASON IT WAS CAUGHT IN MINUTES. A binary that had instead defaulted to 2120 something plausible would have run against the wrong target silently. Loud failure is a feature. 2121 A CHECK THAT WOULD HAVE CAUGHT IT NOW EXISTS AND WAS RUN ACROSS THE WHOLE TREE: every binary declaring flags 2122 must call flag.Parse. Result: 0 binaries declaring flags without parsing them, so this was the only one. 2123 2124FINDING D18, MY D12 FIX SILENTLY DEGRADED EVERY PDF, WHICH IS WORSE THAN THE BUG IT FIXED. The isolated PDF 2125worker was given RLIMIT_AS of 1 GiB, reasoning that a local cap makes a runaway fail on the process that caused 2126it. The reasoning holds; the implementation did not. RLIMIT_AS bounds VIRTUAL address space and the Go runtime 2127reserves large virtual arenas at startup, so EVERY worker died instantly with "fatal error: runtime: out of 2128memory" before reading a byte. 2129 EVERY WORKER FAILURE FALLS BACK TO THE WEAKER HAND ROLLED READER, which is right for an unreadable document 2130 and wrong for a readable one. So a fix aimed at ONE pathological file quietly degraded extraction for ALL 2131 files: 04_PWS_original went from 55 chunks to 3 and REPORTED SUCCESS while doing it. That is a green result 2132 over a quietly worse outcome, the exact shape this build keeps finding, and I introduced it. 2133 CAUGHT BY COMPARING AGAINST A RECORDED BASELINE rather than by reading the log, which said ok=1 and looked 2134 fine. The number 3 where 55 was expected is the only reason it surfaced. 2135 FIXED by removing the rlimit entirely. THE HARD BOUND WAS ALWAYS THE ISOLATION, not the limit: the library 2136 runs in a child, a runaway kills the child, the parent names the document and the batch continues. GOMEMLIMIT 2137 is set as a SOFT hint and is described as soft rather than sold as a cap. Pdeathsig still ties the child's life 2138 to the parent's. 2139 PROVEN BOTH WAYS AFTER THE FIX. 04_PWS_original is back to 55 chunks, now including a recovered table. And the 2140 crasher is STILL contained: "the isolated reader failed (signal: killed), falling back", document named 2141 no_text_layer, batch DONE ok=0 failed=1. The kernel killed the CHILD and the parent survived, which is 2142 precisely the property D12 required and it needs no rlimit to hold. 2143 THE TABLE READER IS ALSO VISIBLY EARNING ITS PLACE now that the worker path works: the amended PWS recovered 2144 91 tables and 486 rate rows, going from 300 chunks to 391. 2145 2146D16 IS CLOSED. The percentile function is O(n log n), the 30 minute soak has been run and reported, and the 2147lesson is codified as a standing Gate D rule plus a new file, docs/POSITIVE_CONTROL_LIBRARY.md, which gives the 2148library the protocol has described since 28 July an actual home. Seventeen controls are listed, each naming the 2149instrument, the lie it told, and the control that catches it. 2150 THE WRONGLY BLAMED CORRECTION, NAMED: D16's first lost soak was attributed to the D14 CREDENTIAL ROTATION 2151 landing at the same moment. D14 was INNOCENT. A second soak, started afterwards on the correct credential from 2152 the beginning, hung identically, which is what proved it. That wrong attribution cost a second lost run, 2153 because a wrong answer closes a question that should have stayed open. When two things change together, prove 2154 which one did it. 2155 2156## THE FOUR FINDINGS HOLDING PHASE 2 OPEN, 29 July 2026 2157 2158RESOLUTION, WRITTEN 29 JULY 2026 LATE, on Joseph's recorded decisions. THREE OF THE FOUR ARE CLOSED. The 2159fourth executed and is blocked at the far end by something no amount of work here can reach. Each original 2160block is left standing below with its status line updated, because the reasoning that produced a finding is 2161worth more than a tidy list. 2162 2163 1 CLOSED BY DECISION. Scanned or undecodable text layers are a graceful terminal state with a clear 2164 no_text_layer status surfaced in the UI. OCR is not in this build and is recorded 2165 as future candidate P2 in docs/IMPROVEMENTS.md. The trial evidence shows the wage 2166 determination carrying that status correctly. 2167 2 CLOSED BY DECISION. Office formats are stored, hashed and versioned with a graceful unsupported_format 2168 status for Phase 2. The reader itself, an xlsx library bake off measured against 2169 the real Package A spreadsheet plus a minimal docx text path, is PRE APPROVED as 2170 Phase 3's single improvement item. Phase 3's slot is now spent. 2171 3 NOT CLOSED, EXECUTED. The full bounded run was performed. Every mechanism on this side executed against 2172 the live external endpoint for the first time, and all three calls were refused by 2173 OpenAI with 429 insufficient_quota. The key is VALID; the account has no credit. 2174 Evidence docs/evidence/phase2/findings/finding-3-external-normalize-path.txt. 2175 NEEDS ONE THING FROM JOSEPH: credit on that account. Then three calls and ninety 2176 seconds. Acceptance fixed in advance: three outcome rows reading openai:gpt-4o-mini. 2177 4 CLOSED BY EXECUTION. The sweep now covers 15 enumerated out of repo scopes on two hosts, with 13 2178 positive controls, both directions, tracked and untracked, inside and outside the 2179 repository. It found two complete DSNs in mm-identity-sync.sh on its first working 2180 run, which CREDENTIAL_CONSUMERS.md had described as a trap and left standing. 2181 Repaired, and the same change took both credentials out of argv. 2182 Evidence docs/evidence/phase2/findings/finding-4-secret-sweep-out-of-repo.txt. 2183 2184Every other finding raised in this build is closed. These four are not, and each one is stated with its 2185status, the evidence that establishes it, and what closure actually requires. None of them is a correctness 2186defect: three are capability limits and one is an instrument gap. 2187 2188OPEN 1, THE SCA WAGE DETERMINATION IS NOT EXTRACTABLE BY THIS BUILD 2189 STATUS CLOSED BY DECISION, 29 July, Joseph. A scanned or undecodable text layer is a GRACEFUL TERMINAL 2190 STATE: stored, hashed, versioned, status no_text_layer surfaced in the UI with both causes named, 2191 and ZERO chunks so no reviewer can cite noise. OCR is not in this build; recorded as candidate P2 2192 in docs/IMPROVEMENTS.md. Was an open capability limit. NOT a regression, and the pipeline is now HONEST about it where before it 2193 was not: the document reports extraction FAILED with no_text_layer and stores ZERO chunks, where 2194 previously it reported DONE with one stored chunk of character soup. 2195 EVIDENCE docs/evidence/phase2/functional/gate-a-phase2-close.txt. Measured on staging: 231 of 231 spans 2196 refused, 0 chunks. The PDF's text layer decodes to mojibake because its embedded font carries no 2197 usable character map, which is a DECODING limit and not a table extraction limit. Table extraction 2198 itself is proven by 7 unit tests on a real shaped wage determination and by the volume profile 2199 parsing a 200 page PWS into 400 chunks, 200 tables and 1600 rate rows. 2200 CLOSURE A character map aware PDF reader, so the font's ToUnicode CMap is applied before text is scanned. 2201 That is a distinct capability and belongs in its own phase item, not in a Phase 2 patch. Until 2202 then the wage determination's rate grid, the legal floor under every bill rate in a proposal, is 2203 not available to the product and the product says so. 2204 2205OPEN 2, NO OFFICE FORMAT READER 2206 STATUS CLOSED BY DECISION, 29 July, Joseph. For Phase 2 office formats are stored, hashed and versioned 2207 with a graceful unsupported_format status and nothing is invented from them. The reader itself, 2208 xlsx through a MEASURED Go library bake off with the real Package A spreadsheet as fixture plus a 2209 minimal docx text path, is PRE APPROVED as Phase 3 single improvement item. Phase 3 slot spent. 2210 Was an open capability limit, and arguably the allowlist working rather than a defect. 2211 EVIDENCE Staging: 05_PWS_Att4_Course_Schedule.xlsx reports FAILED unsupported_format with 0 chunks. The 2212 refusal is BY NAME and by a closed vocabulary, which is the designed behaviour for a format this 2213 build cannot read honestly, rather than a decoder producing plausible output from a format it does 2214 not understand. 2215 CLOSURE Either an office reader for xlsx and docx, or a recorded DECISION that spreadsheet attachments are 2216 out of scope and will be refused permanently. Both are legitimate; leaving it undecided is not, 2217 because a course schedule is real scope a reviewer may need. 2218 2219OPEN 3, THE EXTERNAL REQUIREMENT_NORMALIZE PATH HAS NEVER EXECUTED AGAINST OPENAI 2220 STATUS EXECUTED 29 July on Joseph's order, STILL OPEN, blocked at the far end. The bounded run happened in 2221 full: the boot guard was controlled both ways, the backend was enabled on a temporary process so no 2222 persistent gateway was touched, three public notices went through the gate, three egress rows and 2223 three outcome rows were written, and the flag is verified off again. OpenAI refused all three with 2224 429 insufficient_quota. The key is VALID, an invalid key answers 401; the ACCOUNT has no credit. 2225 The gateway fell back to the local runtime, which the outcome rows caught and named, and that catch 2226 is the reason this is recorded as unproven rather than closed. What remains unproven is precisely 2227 one thing: the verifier gating prose written by an external model. 2228 CLOSING IT NEEDS CREDIT ON THAT ACCOUNT and nothing else. Note from the 28 July decision: this key 2229 was pasted into chat and sat in a plaintext Desktop file, so rotating before funding is the safer 2230 order. Evidence findings/finding-3-external-normalize-path.txt. 2231 EVIDENCE The backend, the boot guard, the egress gate and the audit are built and proven, and the scrub 2232 gate permits only public SAM notice text under that class, with the candidate row and tenant 2233 document fixtures refused from anywhere in a payload. What has never run is the gateway actually 2234 calling OpenAI: its generation backend is config selected and points at the LOCAL model. 2235 CLOSURE Joseph's decision, then one end to end run. It is a gateway CONFIG change protected by the boot 2236 guard, not new code. I did not enable it because it sends data to a third party and carries cost, 2237 which makes it a decision with policy attached rather than a verification step I can take on my 2238 own authority. 2239 2240OPEN 4, THE SECRET SWEEP CANNOT SEE OUTSIDE THE REPOSITORY 2241 STATUS CLOSED BY EXECUTION, 29 July. 15 enumerated scopes on two hosts, 13 positive controls passing in 2242 both directions, and a counted run reporting 15 of 15 scopes answered and clean. It found what it 2243 was built to find on its first working run: two complete DSNs, passwords included, in 2244 mm-identity-sync.sh, which a timer runs hourly as root and which this very inventory had already 2245 called a trap and left standing. Repaired to resolve both from db.env by name, and the same change 2246 took both credentials out of argv, measured 1 exposure before and 0 after. 2247 TWO DEFECTS IN THE INSTRUMENT WERE FOUND ON THE WAY, both of the house failure mode: ssh inherited 2248 the scope file on stdin and drank 12 of 13 lines so ONE scope ran and the sweep said clean, and the 2249 guard refused only on zero scopes so one out of thirteen satisfied it. It now counts expected 2250 scopes from the file. At least one answered is not a check. 2251 EVIDENCE docs/CREDENTIAL_CONSUMERS.md now carries an ops scripts section, added because TWO of the five 2252 consumers broken by the D5 rotation lived in /usr/local/sbin and no inventory listed them. One of 2253 them, mm-restore-drill.sh, held a migrator password in PLAINTEXT and the sweep could never have 2254 found it, because the sweep scans the repository and that file is not in it. D14 then showed the 2255 same class systemically: seventeen flag defaults across thirteen binaries would disclose a live 2256 secret on any usage error. 2257 CLOSURE Extend tools/verify/secret-sweep to accept named paths outside the repository, run it against the 2258 ops script directories on both hosts, and give it a positive control planted in one of those paths. 2259 Until then the sweep's clean result means "clean in the repository", which is narrower than it 2260 reads, and that narrowness is now written down rather than assumed. 2261 2262NOT ON THIS LIST, BECAUSE THEY ARE NOT FINDINGS 2263 GATES A, C AND D NEED RE RUNNING. All three passed, and then code changed after they passed: the readability 2264 bar, the process isolated PDF worker, the pairing fix, the thirteen binary flag change. Their PASS verdicts 2265 describe a commit that is no longer HEAD. That is ordinary sequencing, not a defect, and the re runs are the 2266 first work of the next session. 2267 PIXEL BASELINES await Joseph's approval of the contact sheet. That is Gate E working as written. 2268 2269## Phase 0, security and preflight 2270 2271phase0 | item0 execution design spec written | docs/superpowers/specs/2026-07-27-opportunity-to-outcome-execution-design.md present, approved by Joseph in chat | 7641374 | 2026-07-27 2272phase0 | item5 archive move | docs/archive/INTELLIGENCE_EXPANSION_SPEC.md present (228 lines), governing directive added as docs/BUILD_DIRECTIVE.md | e662665 | 2026-07-27 2273phase0 | item6 BUILD_STATE ledger created | this file exists with the section 0 format | 2595999 | 2026-07-27 2274phase0 | amendment staging environment rule | directive Section 0 item 9 added, directive renamed to docs/CLAUDE_CODE_BUILD_DIRECTIVE.md, through Phase 5 all DB work runs on missionmatch_staging megamind-agi 5433, production M4 only at Gate 6 | this commit | 2026-07-27 2275phase0 | item1 human OpenAI key revoked | Joseph confirms revocation in the OpenAI dashboard | human step | 2026-07-27 2276phase0 | item2 human project scoped key placed | key on M4 as openai_api_key root 0600, read only via the secrets store, enabled flag false | human step | 2026-07-27 2277phase0 | item3 human SAM.gov system account applied | application submitted today, revisit at Phase 4 | human step | 2026-07-27 2278phase0 | item4 scrub egress classes and external call audit gate | go test scrub and ai green; AllowEgress fails closed on an unknown class and on a uuid, email, or phone marker; AllowExternal records backend, class, byte count, and content hash, never content | 8ba3318 | 2026-07-27 2279phase0 | openai gateway backend and boot guard | generateOpenAI added over the chat completions contract, key read only through the secret store, enabled flag default false; go test cmd/megamind-gateway green | 110bd1c | 2026-07-27 2280phase0 | GATE 0 passed | built the real gateway binary and ran it: openai enabled with no key exits 1 (openai_api_key is [credential removed] from the secret store, refusing to boot), openai enabled with an empty model exits 1; scrub and egress tests pass; BUILD_STATE exists; archive move committed at e662665; make ci-gates PASS on backdoor and FIPS | 110bd1c | 2026-07-27 2281 2282## Conflicts recorded (directive Section 0 items 6 and 9) 2283 2284CONFLICT phase1 staging location and parity | The amendment names staging as missionmatch_staging on megamind-agi port 5433. Ground truth from the live /etc/missionmatch/staging.env, used by the running mm-api-staging service on 127.0.0.1:8097, is that staging is missionmatch_staging on M4 100.94.8.94 port 5432, a DISTINCT database on the SAME Postgres server as the production missionmatch database. Staging runs shared tenancy, production runs per_tenant. Staging goose version is 59, repository head is 0068, so staging is nine migrations behind parity, and the opportunity tables do not yet exist there. Stopped before any write per Section 0 items 6 and 9. Awaiting Joseph confirmation that the correct Phase 1 target is missionmatch_staging on M4 5432, that bringing it from 59 to 68 to reach parity is authorized, and that targeting only the missionmatch_staging database, never missionmatch, satisfies the do not touch production before Gate 6 rule. | pending | 2026-07-27 2285RESOLVED phase1 staging conflict | Joseph decided there is no separate staging: the build extends the existing application directly and verifies against the live database with a connected database name assertion and a pg_dump snapshot before each write. missionmatch_staging is not used. Directive Section 0 item 9 updated to supersede the staging amendment. | beab171 | 2026-07-27 2286 2287## Phase 1, opportunity foundations 2288 2289phase1 | item1 ingest_manifest migration | 0069 applied to production missionmatch on M4 5432 (goose 68 to 69), shared reference data with no RLS (rls_enabled false), columns id, source, url, license, fetched_at, record_count, content_hash, created_at, app grants SELECT and INSERT, schema snapshot saved on Bubbles at /home/user/mm-snapshots; the down refuses to drop when provenance rows are present | a77dd65 | 2026-07-27 2290phase1 | item1 tracked_opportunities migration | 0070 applied to production (goose 69 to 70), tenant table verified rls_enabled and rls_forced true, 1 isolation policy, 4 foreign keys (organizations, ingest_manifest, users twice via the column list SET NULL form), indexes idx_tracked_opps_org_status and the unique partial uq_tracked_opps_org_public and uq_tracked_opps_org_id, app grants SELECT INSERT UPDATE no DELETE, 31 columns per Section 3.1; snapshot taken first; down refuses when rows present | cd7ca09 | 2026-07-27 2291phase1 | item1 opportunity_versions migration | 0071 applied, immutable snapshot table, rls_forced, 15 columns, 5 FKs, trigger opportunity_versions_immutable_trg allows only the is_current flag and blocks delete, unique (org, opportunity_id, version_number), grants SELECT INSERT UPDATE | 2026-07-27 2292phase1 | item1 opportunity_documents migration | 0072 applied, rls_forced, 18 columns, 5 FKs, document_type and extraction_status enum CHECKs, sha256 duplicate partial unique index, grants SELECT INSERT UPDATE | 2026-07-27 2293phase1 | item1 opportunity_document_chunks migration | 0073 applied, rls_forced, 10 columns, FK to opportunity_documents on delete cascade, extraction provenance, grants SELECT INSERT | 2026-07-27 2294phase1 | migrations applied via migrate-all-tenants | the sanctioned tool (goose library, embedded migrations, session locked, idempotent), 1 target control plane missionmatch on M4 5432 with 0 active tenants, public v70 to v73, eeo v2 unchanged; the goose CLI is avoided because it pulls heavy unrelated drivers not in go.sum | 2026-07-27 2295phase1 | opportunity capture API and screen | capability opportunity.manage added at recruiter and above (matrix test green); opportunities module gains the tenant capture layer (CreateTracked, ListTracked, GetTracked, PatchTracked via RunInTenantTx, org from GUC, audited), mounted under the slateMATCH tool gate at /opportunities/tracked (Section 9 path resolved around the public cache owning /opportunities/{noticeID}); chi.Walk route contract test green; Capture intake screen at /slate/capture | 13d6399 and 7a196ca | 2026-07-27 2296phase1 | opportunity capture deployed and verified live | new FIPS binary deployed to Bubbles (staged sha256 swap, healthy in 2s), frontend assets swapped with the 5 SEO lines intact; through the public URL the site is 200, the new bundle serves, /slate/capture serves the SPA 200, and POST /api/opportunities/tracked returns 403 (route live, auth required) consistently with no stale replica 405 | 2026-07-27 2297OPEN phase1 | megamind-agi replica out of the pool | the mm-api replica at 100.77.114.40:8094 has no direct SSH (port 22 refused, SSH lands on the Windows host cmd with WSL nested), so it could not receive the new binary; it is marked down in /etc/nginx/conf.d/mm-api-upstream.conf so all traffic reaches the updated Bubbles node. FOLLOW UP: establish a WSL deploy path for megamind-agi, redeploy the binary, and remove the down flag to restore two node failover | 2026-07-27 2298RESOLVED phase1 | megamind-agi replica back in the pool | Joseph provided access (josep@100.69.120.61). The replica mm-api runs in WSL Ubuntu as a system service (User mm-api), binary /opt/missionmatch/mm-api. Deployed the new binary through the Windows /mnt/c bridge (sha 8647587c matches Bubbles), restarted, healthy in 2s, POST /api/opportunities/tracked returns 403 not the stale 405. Removed the down flag, reloaded nginx; both nodes serve consistently (8 of 8 403 through the public URL, and each node directly). Two node failover restored. WSL deploy path for future releases: scp the binary to the Windows host home, then run a bash script via wsl that stops the service, copies from /mnt/c to /opt/missionmatch, and restarts; sudo in WSL is NOPASSWD | 2026-07-27 2299phase1 | item2 intake screen 2 opportunity intelligence | /slate/capture/:id detail and edit view built and deployed (frontend only, backend PATCH already live on both nodes); edits agency, office, solicitation, notice type, NAICS, PSC, set aside, place of performance, incumbent, source URL, deadline, prime or sub, status; public verify site 200, new bundle 200, detail route serves the SPA 200; tsc and vite clean | 4defe49 | 2026-07-27 2300phase1 | opportunity embedding layer, the AI learns the opportunity landscape | migration 0074 applied to production (goose 73 to 74): opportunity_records gains embedding vector(384) with a partial HNSW cosine index (the skills pattern, global data outside RLS) and a GENERATED search_tsv with a gin index. embed-backfill extended with -what opportunities (existing binary extended, no new binary). Ran against production: 6535 of 6535 embedded, exit 0, model qllama/bge-small-en-v1.5. SEMANTIC RETRIEVAL VERIFIED LIVE: the natural language query "cybersecurity engineering support for the Air Force" returns PACAF Cyber Systems Operations Support, Cybersecurity Support Services, and Cyber Security Services at 0.78 cosine, ranked by meaning not keywords. Before this the catalog had no vectors at all and only skills, candidates, and ai_memory were embedded | 084e521 | 2026-07-27 2301## Amendment 6, accepted 28 July 2026 2302 2303AMENDMENT 6 ACCEPTED | source docs/DIRECTIVE_AMENDMENT_6.md (224 lines), triggered by live FG Talent design 2304partner feedback. Merged into the directive: acceptance items 36 through 40 appended to Section 11, and 2305PHASE 5A, Identity and Access, inserted between Phase 5 and Phase 6 with its own Gate 5A. Creates two 2306containers, TRACK U (urgent, production eligible under the Section 2 item 8 guardrails) and Phase 5A. Role 2307enum values are extended, never renamed or removed. The scoring engine stays frozen. | 2026-07-28 2308 2309TRACK U ITEM 1, ROOT CAUSE RECORDED BEFORE ANY FIX, as the amendment requires | Investigated the three 2310reported symptoms against the code and the staging database. 2311 2312SYMPTOM 1, a client superuser could not edit her own name. ROOT CAUSE, conclusive: full_name appears ZERO 2313times in internal/modules/auth/handler.go, service.go, and helpers.go. It exists only in dto.go, on the 2314invite and user list shapes. There is no self service profile endpoint anywhere in the API, and there is no 2315endpoint that mutates a name for ANY actor. The only user mutations that exist are role, tools, status, 2316mfa reset, and recovery reissue. A name is therefore written once at invite acceptance and is IMMUTABLE 2317thereafter for every role including admin, owner, and superuser. This was never a permissions problem: the 2318capability does not exist, which is why escalating to the top of the ladder did not help her. 2319 2320SYMPTOM 2, the Superuser tab rendered inactive. ROOT CAUSE: there is no separate Superuser tab component; 2321the word appears only in explanatory copy on the Admin screen. What renders inactive is the tool entitlement 2322grid at frontend/src/screens/Admin.tsx:308, const locked = !mine && !on, which greys out any product tool 2323the ACTOR does not personally hold, because an actor may only confer a tool they already have. A client top 2324seat who was not granted every product tool therefore sees permanently greyed controls that read as broken 2325rather than as not applicable. The deeper cause is the one the amendment names: superuser conflates the 2326platform plane with the tenant top seat, so a client is shown platform shaped controls at all. 2327 2328SYMPTOM 3, no tenant facing user management. PARTIALLY PRESENT, not absent: GET /users exists gated on 2329RequireRole(RoleAdmin), which uses AtLeast, so a superuser at rank 6 does reach it, and the Admin screen is 2330routed at /admin with a /people alias. What is genuinely MISSING is name editing (no endpoint at all, per 2331symptom 1), self service anything, and a tenant scoped management surface that does not also expose 2332platform concepts. 2333 2334STAGING FACTS for the fix: the users table column is full_name, NOT display_name, so the amendment's display 2335name wording maps onto full_name rather than a new column. Staging currently holds NO superuser (roles 2336present: owner 163, recruiter 881, manager 16, member 13), so reproducing as the reporting user's role 2337requires provisioning a superuser on staging first. The users role CHECK constraint allows exactly 2338superuser, owner, admin, manager, recruiter, member, viewer, with NO operator value, so Track U item 4 2339requires a migration that EXTENDS the constraint, never renames or removes a value. | 2026-07-28 2340 2341DECISIONS 12 AND 13 RATIFIED AND FOLDED INTO AMENDMENT 6 | Joseph, 28 July 2026. 2342DECISION 12, BY RULE EXTENDS TO TOOLS: a tenant superuser holds every product tool its tenant is entitled 2343to, by rule not by grant, exactly as it holds every capability by rule within its tenant; operator holds all 2344tools in every tenant; so the entitlement grid can never render the top seat as broken. Confer only what you 2345hold stands unchanged for every role BELOW superuser, where it is load bearing. 2346DECISION 13, NO SILENT GREY: a disabled entitlement control must state its reason, and the two reasons are 2347distinct, the organization is not entitled to that product versus you do not personally hold that tool. A 2348bare grey control reads as a broken product, which is exactly how this defect reached Joseph. 2349ADAPTATIONS CONFIRMED: full_name is the column and the amendment's display name wording maps onto it, no new 2350column; the users role CHECK is EXTENDED by migration to admit operator, never renamed, never narrowed. | 23512026-07-28 2352 2353THE REPORTING USER'S PRODUCTION ACCOUNT, read only lookup, recorded as the repro baseline | Production holds 2354ONE organization, Acme GovCon (11111111-1111-1111-1111-111111111111), and three users, ALL of them superuser: 2355joseph.w.anady@gmail.com (active, Joseph Anady, mfa false, tools missionmatch, recruitmatch, screenmatch, 2356slatematch), tallen282005@gmail.com (active, Tallen, mfa false, same four tools), and the reporting user 2357patriciacox6@aol.com (Patricia Cox, mfa false). 2358PATRICIA'S ACCOUNT IS THE BUG IN ONE ROW: role superuser, status INVITED, last_login NEVER, and tool grants 2359(NO TOOLS GRANTED). The other two superusers hold all four tools explicitly; she holds none. Under the old 2360confer only what you hold rule, an actor holding zero tools sees EVERY checkbox in the entitlement grid 2361locked, which is precisely the Superuser tab rendered inactive that she reported. Under ratified decision 12 2362she holds all four by rule as the tenant top seat and the grid can never look broken again. Her status being 2363invited with no login also means the account was never fully activated, which is recorded here rather than 2364assumed away. 2365OPEN DESIGN POINT surfaced by decision 12: there is NO org level product entitlement table in the schema 2366today, so entitled to has no data source yet. Amendment 6 Section 2 item 10 puts product entitlements per 2367tool in the Phase 5A operator console. Smallest change for Track U: a tenant superuser holds all four tools 2368by rule now, and when the org entitlement model lands in Phase 5A, by rule narrows to the entitled set 2369without touching the roles below. Recorded so the narrowing is deliberate rather than forgotten. | 2026-07-28 2370 2371CORRECTION 5, THE STAGING ROLE COULD NOT AUTHENTICATE AT ALL, and it invalidates a prior ledger line | 2372Found while provisioning the Track U repro. Correction 2 created the isolated role missionmatch_staging_app 2373and mirrored grants "79 tables to 79 tables". TABLES only. It mirrored ZERO of the nineteen SECURITY DEFINER 2374auth functions, so missionmatch_staging_app held EXECUTE on none of auth_lookup_login, auth_create_session, 2375auth_validate_session, auth_accept_invite, or the sixteen others. The measured gap was exactly 19 of 188 2376functions held by missionmatch_app and not by the staging role, with 0 held only by staging. 2377CONSEQUENCE: no login, no invite acceptance, and no session validation could succeed on staging from the 2378moment the isolated role was introduced. POST /api/auth/invite/accept returned 500. This means the earlier 2379line "items 2 and 3 ... Proven on staging against an account mirroring the reporting user exactly" CANNOT 2380have been an end to end check through a live authenticated API, because no authenticated request could 2381complete. That claim is withdrawn and replaced by the evidence recorded below, which was taken after the 2382fix. Recorded rather than quietly corrected, because a gate claim that could not have been true is exactly 2383what the ledger exists to catch. 2384FIX: the missing EXECUTE grants were mirrored into missionmatch_staging ONLY, generated by comparing the two 2385roles rather than hand listed, with the connected database asserted first. This is an environment repair, 2386NOT a migration: a migration runs on production too, and production must never learn about a staging role. 2387AFTER: 0 of 188 still missing; the staging role is STILL refused on missionmatch, tenant_alpha, and 2388tenant_beta with "permission denied for database"; production migrator and app grants are unchanged at 2389151 of 152; the production site still answers 200. | 2026-07-28 2390 2391TRACK U ITEMS 2 AND 3 COMPLETE, frontend built and TWO more live defects found and fixed | The API existed; 2392the screens did not, and the API turned out not to work through a real session either. 2393DEFECT A, the by rule fix never reached the browser. SessionInfo re read the RAW rows of user_tool_access 2394instead of the principal's already widened set, so decision 12 applied to every server side check and to 2395nothing the user could see. A tenant superuser with no grant rows received tools:[], which hides the entire 2396navigation and locks every entitlement checkbox: the reported symptom surviving its own fix, one layer up. 2397SessionInfo now returns the principal's tools. 2398DEFECT B, PATCH /me/profile answered 404 for the signed in user, on her own account. writeProfile, getProfile, 2399and roleOf ran on a bare pool connection with no app.current_org_id, and users carries FORCED row level 2400security, so every statement matched zero rows and a real user was reported missing. The explicit 2401organization_id predicate in the WHERE clause looks like it does the scoping and does not: RLS filters the 2402row out first. All three now run inside RunInTenantTx. The same class of bug was introduced and caught in 2403AcceptInvite during this session and fixed the same way. 2404BUILT: Settings gains a Your profile panel, every role, no capability gate; the shell header renders the 2405signed in name and links to it, so a rename is visible on every screen; Admin gains Rename per row, admin and 2406above, sending full_name only so a colleague's own title and phone are never overwritten; the self row now 2407says where to edit your own name instead of only refusing. 2408NO SILENT GREY, decision 13: the session now carries org_tools beside tools, and both the entitlement grid 2409and the invite form state which of the two distinct reasons disables a control. The invite form shows all 2410four products rather than omitting the ones it cannot grant. The Admin row for a by rule seat no longer says 2411"No tools granted, this account can sign in but sees nothing", which was false for exactly Patricia's row. 2412NEW: security.OrgEntitledTools(orgID) is the single place the Phase 5A per tool entitlement narrows, taking 2413the orgID now, unused, so adding the lookup is a body change and not a signature change. | this commit | 24142026-07-28 2415 2416TRACK U ITEM 4 COMPLETE, the platform plane and operator MFA | security.RequirePlatform gates operator only 2417and answers 404, never 403, so a platform surface is indistinguishable from a path that was never built. A 2418403 would confirm the cross tenant control plane exists to the very seat amendment 6 is removing it from. 2419Curation mounts behind it, the only platform surface that EXISTS today; the Phase 5A surfaces mount inside 2420the same group and inherit the gate. The federation reads and /metrics stay public deliberately and the 2421reasons are recorded at the mount point rather than left to be re derived. The Compliance screen omits the 2422curation panel entirely for a tenant, so the platform surface is ABSENT rather than inactive. 2423Operator MFA is mandatory from today whatever MM_MFA_REQUIRED says, at login and at invite acceptance. 2424NOTED DEVIATION, Section 0 item 6: the amendment's words are that the login is REFUSED. Taken literally, with 2425MM_MFA_REQUIRED false as both environments run, that is a lockout with no way out, because EnrollMFA accepts 2426only an auth_pending session and the MFA disabled login path never creates one. A flat refusal would leave a 2427promoted operator with no enrollment path and no in application recovery, which would lock Joseph out of 2428production the moment the remap ran. The smallest change satisfying both the rule and a working system is to 2429enforce the REQUIREMENT rather than the refusal: the account never obtains an ACTIVE session, every 2430authenticated route refuses it, and the one step its pre MFA session permits is the step that makes it 2431compliant. | this commit | 2026-07-28 2432 2433TRACK U ITEM 5 COMPLETE, migration 0080 superuser_remap applied to STAGING, goose v79 to v80 | Snapshot taken 2434first (/home/user/mm-snapshots/staging-schema-pre0080-20260728.sql, 314686 bytes), target asserted as 2435missionmatch_staging before any write, applied through migrate-all-tenants (the sanctioned tool), 1 target, 2436public v79 to v80, eeo unchanged at v2. 2437EVERY SUPERUSER GRANT AUDITED, each account named, three rows written to audit_logs with action 2438authz.superuser.remap_audited before anything was changed, so the trail records the state that was FOUND: 2439 50426a53 joseph.w.anady@gmail.com Joseph Anady was superuser active, becomes operator, plane platform 2440 1c04ca7c patriciacox6@aol.com Patricia Cox was superuser active, stays superuser, plane tenant 2441 e82d13af tallen282005@gmail.com Tallen was superuser invited, stays superuser, plane tenant 2442Exactly one role value was written. The two tenant humans keep full power inside their own tenant and lose 2443only the platform surfaces, which is enforced by RequirePlatform in code, not by a role change. The promoted 2444account's live sessions were revoked through the existing auth_revoke_user_sessions function, verified 0 live 2445sessions after, because a session minted under one role must not survive a change to that role. The down 2446migration returns operator to superuser and revokes again; the audit rows stay, because audit_logs refuses 2447UPDATE and DELETE by trigger and that is the correct behaviour for an append only trail. | this commit | 24482026-07-28 2449 2450TRACK U GATE, ALL FOUR ITEMS PROVEN END TO END ON STAGING through the live authenticated API and the rendered 2451screens | The repro account MIRRORS production exactly: org Acme GovCon on staging, three users, all 2452superuser, with patriciacox6@aol.com at her exact production state, role superuser, status invited, mfa 2453false, and ZERO tool grants. Provisioned through the product's own paths, provision-org then invite then 2454invite acceptance, not by hand written rows. 2455THE REPORTED SYMPTOM, GONE: her session returns tools ["missionmatch","recruitmatch","screenmatch", 2456"slatematch"] and org_tools the same four while user_tool_access holds NOTHING for her. Before this she 2457received [], which is the inactive grid she reported. 2458GATE 1, she edits her own name: PATCH /me/profile 200 returning Patricia A. Cox, Director of Recruiting, 2459417 555 0134; it renders back on her own session AND in the accounts list another user reads; the platform 2460routes answer 404 for her (proposals, notice, and approve) while the tenant compliance surface she should 2461reach still answers 200, so this is a targeted absence and not a blanket denial. 2462GATE 2, the admin lifecycle: an admin invited a user, assigned recruiter, the invite was accepted, the admin 2463RENAMED them (Robin Recruter to Robin Recruiter) with no escalation, the recruiter created a client they own, 2464the admin deactivated them, the deactivated user could NOT log in (401) and their existing session was dead 2465(403), while the client they own remains intact and still attributed to them by name, and the user row still 2466exists with status disabled rather than being deleted. 2467GATE 3, the grant ceiling at the API: admin inviting owner 403, superuser 403, operator 403, another admin 2468403, and admin promoting the recruiter to owner 403, while the legitimate promotion to manager returned 200. 2469Zero accounts above the ceiling were created. 2470GATE 4, operator MFA and the platform 404: the operator logged in with a correct password and received 2471mfa_required TRUE even though MM_MFA_REQUIRED is false for the deployment; with no authenticator he held NO 2472session, /auth/session /users /compliance/curation/proposals and /recruiting/clients all 403; he enrolled and 2473activated a real TOTP code, after which the same platform routes answered 200; a fresh login demanded the 2474code again, so MFA is not a one time hurdle. Non operators receive 404 on the identical path the operator 2475gets 200 on, superuser and admin both, which proves the 404 is a refusal and not a missing route. 2476SCREENS: staging bundle index-CiVB_7US.js shipped, only the two asset hashes rewritten so the five injected 2477SEO lines survived (verified, 22 lines intact); /, /settings, /admin, and /compliance all 200 over HTTPS; the 2478strings Your profile, Save profile, Rename, the org not entitled reason, the by rule seat line, and the self 2479service pointer are all present in the shipped bundle. | this commit | 2026-07-28 2480 2481TRACK U ITEM 6, PRODUCTION DEPLOY 1 OF 1, APPROVED AND SHIPPED | JOSEPH'S APPROVAL, recorded verbatim as 2482Amendment 6 Section 2 item 8 requires: "Approve everything and stop asking for MfA", 28 July 2026, in answer 2483to a deploy proposal that named the migration, both nodes, the frontend, and the MFA consequence. 2484 2485THE MFA DIRECTION, IMPLEMENTED AS A SETTING RATHER THAN A DELETION. Joseph directed that his operator 2486account not be prompted for a second factor. Amendment 6 decision 1 makes it mandatory, so rather than 2487remove the rule from the code, MM_OPERATOR_MFA_REQUIRED was added with a DEFAULT OF TRUE: the amendment 2488holds everywhere unless a deployment stands it down, the staging gate keeps proving the rule on its default, 2489config REFUSES the relaxation in a production MM_ENV, the boot log states it at every start, and re arming 2490it is one environment variable rather than a release. The relaxation is narrow and tested: it removes only 2491the operator surcharge, it cannot weaken MM_MFA_REQUIRED, and it changes no other role. This is a KNOWN 2492DEVIATION from decision 1, taken on the owner's explicit instruction, and Track U gate item 4 stands proven 2493on staging under the default. 2494 2495THE BINARY IS NOT REPOSITORY HEAD, AND THAT IS THE POINT. HEAD carries the Phase 1 and 2 build features, 2496so shipping it would have turned the whole build surface live in production and broken the Gate 6 freeze the 2497same clause that authorises this deploy is written to protect. The production binary was built in a clean 2498worktree at the pre build baseline 9e77f65 with ONLY the Track U files applied, and cmd/api/main.go hand 2499resolved to take the platform gate and the MFA wiring and nothing else. Proven, not assumed: the shipped 2500binary contains ZERO occurrences of requirement-drafts, opportunities/tracked, and scope/items, and the 2501shipped bundle zero of Opportunity capture, slate/capture, and requirement-drafts. 2502CAUGHT DOING IT: copying HEAD's AppShell.tsx wholesale dragged a /slate/capture navigation link, a Phase 1 2503feature, into the production bundle. It was found by grepping the built bundle rather than the source. That 2504file was rebuilt from the baseline with only the Track U identity header applied, and every other copied 2505frontend file was checked for the same leakage and had none. 2506 2507WHAT SHIPPED, in order, each step verified before the next: 2508 1. Snapshot: /home/user/mm-snapshots/prod-schema-pre-tracku.sql (304231 bytes) and 2509 prod-identity-pre-tracku.sql, plus prod-index.html.pre-tracku, plus mm-api.bak.pre-tracku on both 2510 nodes and api.env.bak.pre-tracku on both nodes. 2511 2. Target asserted as missionmatch, then migrations 0078, 0079, 0080 applied through 2512 migrate-all-tenants, public v77 to v80 (+3), eeo unchanged at v2. 2513 THE PRODUCTION AUDIT TRAIL, every superuser grant named, written before anything changed: 2514 joseph.w.anady@gmail.com was superuser active, now operator, plane platform 2515 patriciacox6@aol.com was superuser invited, stays superuser, plane tenant 2516 tallen282005@gmail.com was superuser active, stays superuser, plane tenant 2517 users.title and users.phone both present, all 8 tool grants preserved. 2518 3. MM_OPERATOR_MFA_REQUIRED=false written to /etc/missionmatch/api.env on BOTH nodes, staged to /tmp then 2519 copied because tee truncates, asserted non empty and larger before replacing. 2520 4. API binary 29a622d01d52e67a621d4040aefd81374cb2dafdcd1196e65e1730af017c527b to Bubbles, healthy in 2521 under 3 seconds, boot log carrying the operator warning. 2522 5. THE SECOND NODE WAS UNREACHABLE MID DEPLOY and this is the important operational note. megamind-agi 2523 answered healthz 200 while still running the OLD binary, which has no operator role at all, so 2524 roleRank would not resolve it and Joseph's account would have held ZERO capabilities on that node 2525 while holding everything on Bubbles. An inconsistent production, exactly the failure the handoff warns 2526 about. It was marked down in the nginx upstream and nginx reloaded BEFORE shipping anything further, 2527 so production served consistently from one updated node rather than serving two different answers. 2528 Access was then recovered with password authentication over the Windows bridge (publickey only was 2529 refused), the binary staged through /mnt/c with its sha256 confirmed identical, the env flag written, 2530 the service restarted, healthz 200 first attempt, and the node returned to the pool. 2531 6. Frontend index-BE5iWWvj.js and index-Dh-2b0Dd.css, only the asset hashes rewritten in index.html, 2532 line count unchanged at 21 so the injected SEO lines survived. 2533 2534THE LAST MILE, FOUND AT THE END AND FIXED: the reporting user could still not sign in. Her production 2535account has NO invite row and NO password credential, so every fix shipped for her was unreachable. Amendment 25366 Section 2 item 5 lists RESEND INVITATIONS in tenant user management and it had been missed: the endpoint 2537POST /users/{id}/recovery existed and NOTHING called it, so an account stranded at invited had no live link, 2538no password, and no administrative way to issue one short of database access. A Resend invitation button was 2539added to the accounts screen, gated on the capability the endpoint actually checks rather than the 2540surrounding block's, and shipped. HUMAN STEP REMAINING: Joseph presses it on her row and sends her the link. 2541 2542POST DEPLOY PROOF, Amendment 6 Section 2 item 8, with BOTH nodes live and sampled repeatedly so each pool 2543member answers: POST requirement-drafts approve 404 eight of eight, opportunities scope 404 eight of eight, 2544opportunities/tracked 405 eight of eight (the documented pre build shape, because only the preexisting GET 2545matches that path), /api/users and /api/auth/session 403 eight of eight on each. ZERO 5xx across 45 requests 2546over 15 paths. The shipped bundle contains the Track U strings and none of the build feature strings. 2547Production schema v80, site 200, both nodes healthz 200, two node failover restored. | this commit | 25482026-07-28 2549 2550CORRECTION 6, THE PRODUCTION TOPOLOGY WAS RECORDED WRONG IN THIS LEDGER, AND IT CAUSED AN OUTAGE | Found 255128 July 2026 while updating the architecture document, hours after the Track U deploy. 2552WHAT THIS LEDGER SAID: "PRODUCTION: missionmatch on M4 5432, schema v77". That describes the TENANT data 2553plane only. The CONTROL PLANE, which holds users, auth_credentials, auth_sessions, and tenant_directory and 2554is what MM_DATABASE_URL actually points at, is missionmatch on megamind-agi 100.77.114.40 port 5433. It was 2555at goose v68, TWELVE migrations behind the tenant database, and nobody knew. 2556THE CONSEQUENCE, and it was mine: the Track U binary's SessionInfo selects title and phone from users. Those 2557columns were added by 0079, which I applied to M4 5432 believing it was production. On the control plane 2558they did not exist, so GET /api/auth/session returned 500 for every authenticated user from the moment the 2559deploy completed. Production was broken for anyone logged in. It was not caught by the post deploy sweep 2560because every path in that sweep is unauthenticated and answers 403 before SessionInfo runs. A zero 5xx 2561sweep that never holds a session cannot see a defect behind the session gate; future sweeps need an 2562authenticated leg. 2563FIXED: snapshotted the control plane (schema and identity), then brought it v68 to v80 (+12) through 2564migrate-all-tenants. Verified after: title and phone present, the remap audit trail written with all three 2565superuser grants named, joseph.w.anady@gmail.com now operator ON THE AUTHORITATIVE DATABASE, the promoted 2566account's sessions revoked, login returns 401 for a wrong password rather than 500, and zero 5xx across 21 2567requests. 2568ROOT CAUSE OF THE DIVERGENCE, a real defect in the sanctioned tool: cmd/migrate-all-tenants builds each 2569tenant DSN with swapDB(controlDSN, dbName), swapping only the DATABASE NAME and IGNORING the db_host and 2570db_port that tenant_directory exists to carry. A tenant on a different host is therefore NEVER migrated; the 2571tool silently migrates the control plane twice and reports success for both. That is why the two databases 2572drifted twelve versions apart with every run reporting OK. NOT YET FIXED. Until it is, migrate each database 2573by pointing MM_MIGRATOR_DATABASE_URL at it directly and verify the version in EVERY database afterwards. 2574SCOPE OF THE OUTAGE, stated precisely rather than dramatically: the defect was real, SessionInfo genuinely 2575selects columns the control plane genuinely did not have, but the journal shows ZERO 5xx and ZERO ERROR 2576lines across the entire window between the deploy and the fix. Nobody logged in while it was broken, so it 2577had no victim. The correct statement is that production WOULD have returned 500 to the first person to sign 2578in, not that anyone experienced it. | 2026-07-28 2579 2580CORRECTION 8, A CLAIM IN CORRECTION 6 WAS WRONG AND IS WITHDRAWN | Correction 6 reported that production 2581defines MM_TENANT_DSN_TENANT_ACME_COMPLIANCE but not MM_TENANT_DSN_TENANT_ACME, and recommended treating 2582the per_tenant data plane as broken. THAT WAS WRONG. Both are defined, on lines 23 and 24 of 2583/etc/missionmatch/api.env, and both are present in the running process environment read from 2584/proc/PID/environ. The claim came from a grep run through two layers of shell quoting that silently matched 2585nothing, and I treated a negative result from a fragile command as evidence of absence. 2586THE DATA PLANE IS PROVEN WORKING, not inferred: connecting with the EXACT DSN the running process holds 2587reaches missionmatch on M4 5432 as missionmatch_app, that role is NOBYPASSRLS, a read of candidates with NO 2588org GUC returns 0 rows, and the same read inside a transaction with app.current_org_id set returns 11. That 2589is forced RLS armed and failing safe on the application's real path. 2590THE PATTERN WORTH RECORDING, because it happened TWICE in one day: a route interception that silently never 2591fired made a pixel diff prove nothing, and a mangled grep made a present config line look absent. Both times 2592the tool reported a clean negative and both times I believed it. A negative result is only evidence when the 2593instrument is known to work, so a check that can pass by doing nothing needs a positive control. | 2026-07-28 2594 2595CORRECTION 7, A CLAIM I MADE EARLIER TODAY WAS WRONG | I reported that Patricia Cox could not sign in 2596because her account had no password credential and no live invite, and I built and shipped a Resend 2597invitation button on that basis. That reading came from the TENANT copy of users on M4 5432, which is 2598NON AUTHORITATIVE and was both stale and incomplete. On the control plane her account is status ACTIVE with 2599a password credential and all four tool grants; she has simply never logged in. The real production identity 2600is FIVE accounts, not three: joseph.w.anady@gmail.com operator, patriciacox6@aol.com superuser, 2601tallen282005@gmail.com superuser, batwood@igotozarks.com recruiter, and trutle973@hotmail.com recruiter, the 2602last two with three tools each and neither previously recorded here. 2603The Resend invitation button is still correct work, because amendment 6 section 2 item 5 lists it and 2604nothing called the endpoint, but the urgency I attached to it was based on a bad read. THE RULE THIS 2605ESTABLISHES: identity questions are answered on the CONTROL PLANE, never on a tenant copy of users. | 2026-07-28 2606 2607CORRECTION 6 ROOT CAUSE FIXED | The migrate-all-tenants defect named in correction 6 is closed. 2608BEFORE: a tenant DSN was built by swapDB(controlDSN, dbName), which kept the CONTROL PLANE's host and port 2609and substituted only the database name, discarding the db_host and db_port that tenant_directory exists to 2610carry. A tenant on any other machine was never reached, and because a database of the same name existed on 2611the control plane host, the tool connected THERE, migrated the control plane a second time, and printed 2612APPLIED for both. Every run reported success while one database silently fell twelve versions behind. 2613AFTER: tenantDSN builds the target from the DIRECTORY ROW, host and port and name together. An explicit 2614MM_TENANT_MIGRATOR_DSN_<REF> wins when set, folded by the same rule internal/platform/secrets uses so there 2615is one naming convention rather than two that differ in an edge case. A directory row with no host is an 2616ERROR, never a guess, and a target that cannot be ADDRESSED is reported as FAILED and carried into the non 2617zero exit code rather than dropped from the summary, because a tenant missing from the report reads as a 2618tenant with nothing to do. 2619PROVEN AGAINST THE REAL TOPOLOGY, which is the case that was broken: with the control plane on 2620megamind-agi 5433 and the only tenant on M4 5432, the fixed tool now targets 100.94.8.94:5432 and, when the 2621credentials did not match, FAILED LOUDLY with the authentication error instead of quietly succeeding 2622somewhere else. With the per tenant override set it reports both databases at v80 at head, and after 2623aligning the control plane migrator credential with the tenant's it reports the same on the plain path. 2624That "all databases OK" is now a true statement about two databases rather than one database counted twice. 2625Five unit tests pin the behaviour, including one that uses an IDENTICAL database name on both hosts, which 2626is the exact shape that made the old bug invisible. 2627OPERATIONAL NOTE: the control plane migrator role password on megamind-agi 5433 was set to [REDACTED, rotated 29 Jul 2026] to 2628match the tenant cluster, so the fan out works without a per tenant override. | this commit | 2026-07-28 2629 2630ARCHITECTURE DOCUMENT REVISED | ~/Desktop/missionMATCH_ARCHITECTURE.txt rewritten from measurement at commit 2631f42da01, superseding the 21 July edition written at 9e77f65. Scale refreshed (326 Go source and 224 test 2632files, 87 frontend files, 79 migrations at head 0080, 36 binaries, 355 commits). New sections: environments, 2633authentication and session security in depth, the opportunity pipeline, deployment and its real failure 2634modes, observability, and a RISK REGISTER naming the absence of any database backup as the largest risk in 2635the system. Corrected the control plane location and the tenancy facts per correction 6. A live OpenAI API 2636key was found in plaintext at the end of that file and REMOVED; Joseph must confirm whether it is the key 2637already revoked on 27 July or a newer one. | 2026-07-28 2638 2639CONTINUATION PROTOCOL ACCEPTED | source docs/BUILD_CONTINUATION_PROTOCOL.md (334 lines), copied into the 2640repo 28 July 2026, generated against commit e9ff911 at schema v80. It changes how phases END, not what they 2641build. Applied per its own Section 0: 2642 The FIVE GATE PHASE COMPLETION PROTOCOL (A functional, B stress, C visual, D drift, E human) is now 2643 mandatory for Phase 2 and every phase after it, in order, with evidence under docs/evidence/phaseN/ and 2644 a summary in this ledger. No phase closes without all five and Joseph's recorded word at Gate E. 2645 TRACK H begins immediately in parallel and is PRODUCTION ELIGIBLE under the same four guardrails as 2646 Track U. The directive's "Track U is the only production eligible work before Gate 6" now reads 2647 "Track U and Track H". 2648 ALLOWED NEW THINGS, exhaustive: tools/verify/visual-sweep, tools/verify/stress, docs/evidence/, 2649 docs/IMPROVEMENTS.md, and the Track H backup scripts. Nothing else is licensed by this document. 2650 ACCEPTANCE ITEMS 41 through 44 append to directive Section 11. 2651 Every standing rule is unchanged: frozen engine, forced RLS, extend never build beside, fail closed, 2652 staging only outside Tracks U and H, no Python, no dashes in written output. 2653THE PROTOCOL VALIDATES THE LESSON THIS LEDGER LEARNED TODAY: it requires a POSITIVE CONTROL on every stress 2654instrument, every visual sweep, and every drift sweep, and Section 4 item 2 makes the control library 2655permanent and only ever growing. That is the direct answer to the three instruments that lied on 28 July. | 26562026-07-28 2657 2658CONFLICT RECORDED, Track H item 2 versus a same day instruction from Joseph | Track H item 2 says to remove 2659the exposed OpenAI key line from every copy of the architecture document. Joseph EXPLICITLY instructed 2660earlier today, after I had removed it, to put it back, and it was restored at his direction. Removing it 2661again unilaterally would reverse a direct instruction on the strength of a document that may have been 2662drafted before he gave it. 2663NOT ACTING. The item is queued for him as Track H item 6 is, and it is sequenced correctly anyway: the item 2664says CONFIRM revocation in the OpenAI dashboard and RECORD the confirmation FIRST, and only Joseph can do 2665that. The key stays where he asked until he says otherwise. Recorded rather than silently resolved either 2666way. | 2026-07-28 2667 2668TRACK H ITEM 1 COMPLETE, THE LARGEST RECORDED RISK IS CLOSED FOR THE FIRST TIME | Both missionMATCH 2669databases now have nightly, encrypted, off machine backups with a PROVEN restore. Evidence in 2670docs/evidence/trackH/, procedure in docs/RUNBOOK_RESTORE.md. 2671WHAT RUNS: 02:45 megamind-agi dumps the control plane (PostgreSQL 18.4) and streams it through gpg into ssh 2672so no cleartext dump ever touches its disk; 03:05 Bubbles dumps the tenant and staging databases, ingests 2673the pushed control plane artifact, writes a sha256 manifest, promotes weekly on Sundays, and prunes to 14 2674daily and 8 weekly. Both are systemd timers reusing the house OnFailure notification unit rather than a 2675second mechanism. Bubbles hosts NO missionMATCH database, so every dump is off the machine holding the data 2676by construction. 2677WHY THE WORK IS SPLIT ACROSS TWO HOSTS: Bubbles carries pg_dump 17.10, which ABORTS against the PG18.4 2678control plane. Rather than add a third party apt repository to a production web host serving 60 vhosts, the 2679control plane dumps where the right client already exists and pushes. The push key is restricted at the far 2680end by authorized_keys command= to one receiving script, so a stolen backup key cannot open a shell, and the 2681receiver writes to a spool directory it owns under a name IT chooses, so a path traversal in a pushed 2682filename is not expressible. The root job validates the artifact DECRYPTS under our key before filing it, 2683because a truncated push that lands looking like a backup is worse than no backup. 2684FOUR THINGS WENT WRONG AND EACH IS WORTH KEEPING: 2685 The tailnet REFUSED the push. Tailscale SSH is enabled on Bubbles and the policy denies megamind-agi 2686 ("tailnet policy does not permit you to SSH to this node"). The LAN route reaches the same host and the 2687 same restricted key and is what the job uses. An ACL rule is the better long term answer and it is 2688 Joseph's to grant. 2689 The receiver could neither write nor validate, because it runs as the unprivileged user while the backup 2690 directory is root owned and the key is root only. Restructured so the receiver only spools and the root 2691 job validates and ingests, which keeps the key root only. 2692 A second run in one day FAILED spuriously, because the ingest moves the artifact out of the spool. Now the 2693 check asks the question that matters, is today's control plane dump in the SET, not did something arrive 2694 in the last few minutes. Spurious alerts are how real ones get ignored. 2695 THE RESTORE PATH WAS BROKEN WHILE THE BACKUPS WERE PERFECT. gpg refused to write over the file mktemp had 2696 already created and failed with "handle plaintext failed". Every dump was fine and none could be opened. 2697 That single missing flag is the entire argument for drilling a restore rather than assuming a dump 2698 implies a recovery. 2699THE DRILL FOUND A RECOVERY PREREQUISITE NOBODY KNEW ABOUT, and this is the finding of the day: RESTORING 2700REQUIRES A SUPERUSER TO CREATE THE pgvector EXTENSION FIRST. The first drill produced 60 pg_restore errors 2701and SILENTLY lost candidates and opportunity_records, the two most important tables in the platform, because 2702CREATE EXTENSION needs superuser and the migrator deliberately is not one. The database reported the correct 2703schema version while missing every candidate and every opportunity. In a real outage that is discovered at 2704the worst possible moment. It is now step 0 of the runbook, with the rule that a restore is not done until 2705row counts reconcile. 2706DRILL RESULT, TIMED AND RECORDED: restore time 6 to 7 seconds for the 95 MB tenant database, manifest 2707verified, schema source v80 equals restored v80, and all ten reconciled tables matched exactly including 2708candidates 11, opportunity_records 6676, and audit_logs 56470. 2709STILL NOT PROTECTED, stated plainly: /mnt/storage is not mounted (the off disk drive is dead) so the backups 2710live on the Bubbles root disk and a root disk failure destroys them; there is no off site copy; the 2711passphrase exists on exactly two machines in one building and losing both loses every backup irreversibly; 2712and the CONTROL PLANE restore has been decrypt verified but not yet timed end to end. Mounting a replacement 2713drive at /mnt/storage requires no code change, the job already prefers it. | this commit | 2026-07-28 2714 2715TRACK H ITEM 5 COMPLETE ON STAGING, THE DESIGN PARTNER WAS RIGHT AND THE INSTRUMENT WAS LYING | Evidence in 2716docs/evidence/trackH/item5-last-login.txt. 2717SHE REPORTED a login the control plane did not show. She was correct, and it was worse than one account: 2718FOUR OF FIVE production accounts held real ACTIVE sessions while the accounts screen said none of them had 2719ever signed in, and the fifth, Joseph's, was STALE BY A WEEK at 21 July despite a session on 28 July. 2720THE CAUSE, internal/modules/auth/service.go line 902: 2721 _, _ = db.Exec(ctx, `SELECT auth_touch_last_login(, )`, userID, orgID) 2722The $1 and $2 PLACEHOLDERS WERE MISSING FROM THE SOURCE, almost certainly stripped by an edit made through a 2723shell that expanded the dollar signs away. Postgres rejected it as a syntax error on EVERY sign in, and both 2724return values were discarded, so it failed silently for weeks. This is the house rule about not editing 2725through scripts, demonstrated in production. 2726RULED OUT BY DIRECT TEST, not by reading: the function exists and works when called as the superuser; the 2727app role holds EXECUTE; RLS is not the cause because the function is SECURITY DEFINER owned by a BYPASSRLS 2728role; and calling it as the APP ROLE with literal ids stamped correctly. One of my own intermediate tests 2729was INVALID and is recorded as such, because passing subselects as the app role returned NULL under RLS and 2730the function's NULL guard returned early, which looked like a failure that was not one. That is the third 2731time today a broken instrument produced a confident negative. 2732THE FIX: placeholders restored, and the failure is now LOGGED at ERROR with the user id, including the zero 2733rows case, which Postgres does not treat as an error and which is exactly what a silent no op looks like. It 2734still does not fail the login: recording a sign in is bookkeeping and denying entry over bookkeeping would 2735turn a reporting defect into an outage. Swallowed deliberately is a different thing from swallowed 2736accidentally. 2737PROVEN ON STAGING with two real logins through the live API, both NULL before and both stamped after. 2738POSITIVE CONTROL PASSED, as the protocol now requires: EXECUTE revoked to break the stamp deliberately, the 2739login still returned 200 (correct), last_login stayed NULL (the break was real), and the server SAID SO with 2740one ERROR line naming the consequence. Restoring the grant made the next login stamp again. 2741PRODUCTION HISTORY BACKFILLED from auth_sessions, the authoritative record of who actually signed in, 2742UPDATE 5, all five accounts now carrying their real last session time. A diagnostic call of mine had written 2743a FALSE timestamp for Patricia, 16:16 when her real session was 08:04; the backfill corrected it rather than 2744leaving my own artifact in production data. 2745NOT YET IN PRODUCTION: the code fix is on staging only, so production will keep failing to stamp NEW logins 2746until a Track H production deploy is approved under the Section 2 item 8 guardrails. The backfill repaired 2747history, not the mechanism. | this commit | 2026-07-28 2748 2749TRACK H PRODUCTION DEPLOY, THE LAST LOGIN FIX, APPROVED AND SHIPPED | Evidence in 2750docs/evidence/trackH/item5-production-deploy.txt. 2751JOSEPH'S APPROVAL, recorded verbatim as the guardrails require: "Track H login fix deploy to production: 2752approved, record this approval in the ledger. Ship it exactly as described, baseline plus the Track H change 2753only, both nodes, build routes re proven dark, authenticated leg in the sweep." 28 July 2026. 2754BUILT from the pre build baseline 9e77f65 with only the Track U and Track H files overlaid, main.go hand 2755resolved, and AppShell.tsx taken from the BASELINE so the Phase 1 capture nav link could not ride along a 2756second time. API sha 8f24414531e80a9a, identical on BOTH nodes, healthz 200 on each. Proven by grepping the 2757BINARY: zero occurrences of requirement-drafts, opportunities/tracked, or scope/items, and the corrected 2758statement SELECT auth_touch_last_login($1, $2) present. 2759PRODUCTION POSITIVE CONTROL, one real login through the public HTTPS URL: before NULL, wall clock 16:36:07 2760CDT, login 200, GET /api/auth/session 200, after last_login 2026-07-28 16:36:08.000522. Correct to the 2761second. A dedicated probe account was provisioned for this rather than using a real person's credentials, 2762and RETIRED afterwards as disabled with sessions revoked, never deleted, per the accounts rule. 2763BUILD ROUTES RE PROVEN DARK ON BOTH LEGS, four samples each so both pool members answer: unauthenticated 2764404, 404, 405; AUTHENTICATED with a real session 404, 404, 404. The authenticated leg is new and is exactly 2765what the previous sweep lacked when it missed a 500 behind the session gate. GET /api/candidates answered 2766404 for the probe, which is CORRECT: a viewer with no tool grants is refused by RequireAnyTool, fail closed 2767working as designed. ZERO 5xx across 96 requests, every path hit on both legs three times. 2768RESTORE SCRIPT HARDENED so the drill's discovery lives in the script, not in memory: it now ENSURES every 2769required extension before loading from a named REQUIRED_EXTENSIONS list, and ASSERTS that candidates and 2770opportunity_records are PRESENT, NON EMPTY, and RECONCILED afterwards, failing on any of the three. 2771POSITIVE CONTROL ON THAT ASSERTION: the vector extension was dropped and the drill REFUSED to proceed, 2772naming the exact superuser command a recovery must run; restoring it made the drill pass again. 2773THE RESTORE TIME IS A BASELINE THAT GROWS WITH THE DATA. Today 8 to 10 seconds for a 16 MB decrypted dump of 2774the 95 MB database. The script now prints that caveat every run and records the size beside the time, so the 2775number is compared against the previous phase close and never read in isolation. | this commit | 2026-07-28 2776 2777TRACK H ITEMS 3 AND 4 COMPLETE | Evidence in docs/evidence/trackH/items3-4-alerting-and-sync.txt. 2778ITEM 3, ALERTING, and the finding that justifies it: /var/log/bubbles-failures/ already held markers showing 2779bubbles-backup FAILED on 13, 14, 15 and 16 JUNE and nobody was ever told. Two causes, both fixed. The 2780recipient file the notifier reads had NEVER been set, and the delivery path was the local mail command, 2781which hands to a local sendmail whose output from a residential address Gmail drops. The alerting was 2782decorative. mm-alert.sh now sends through Gmail SMTP with curl, reusing the SAME app password the fifteen 2783form handlers already authenticate with, so nothing was installed and no second credential exists. mm-watch 2784checks three things only, the public site, BOTH nodes named individually so a single node failure is not 2785masked by the pool, and 5xx since the last run, and alerts on a TRANSITION rather than every run because a 2786monitor that repeats gets filtered to a folder. 2787PROVEN BY BREAKING THINGS: stopping the API node produced a named detection and a PROBLEM email at 16:41:20, 2788restarting produced RECOVERED at 16:41:25. Hiding the backup key made the job fail, and the FIRST attempt 2789proved only a marker was written, which is the June situation exactly, so the notifier was rewired to the 2790working path and the second attempt emailed "[bubbles] mm-db-backup FAILED" at 16:42:49. The 5xx detector 2791was DELIBERATELY NOT induced in production, because causing real 500s on a live client serving system to 2792test a monitor is not a reasonable trade; its logic was proven against controlled input and the limitation 2793is recorded rather than glossed. 2794ITEM 4, IDENTITY SYNC, and the drift it closes: the control plane held 6 users and the tenant copy held 3, 2795missing batwood@igotozarks.com, trutle973@hotmail.com, and the probe entirely. mm-identity-sync iterates 2796EVERY active tenant from tenant_directory and passes the tenant DSN EXPLICITLY because the tenant is on 2797another host, which is the lesson from the fan out defect. It RECONCILES after each sync rather than 2798trusting the exit code, because a tool that exits zero is not a copy that matches. First run seeded 6 and 2799reconciled 6 equals 6. PLANTED RECORD CONTROL: a record created on the control plane was absent from the 2800tenant copy, present after the sync, and removed from both sides afterwards, verified 0 and 0. Scheduled 2801hourly, because the drift appears whenever an invite is accepted and a daily cadence would leave a new user 2802invisible to tenant foreign keys for most of a working day. | this commit | 2026-07-28 2803 2804OPENAI KEY PLACED AT JOSEPH'S DIRECTION | He supplied the key and asked for it in the proper places. It is 2805now in the sanctioned secret store ONLY: /Users/joseph/.megamind-secrets.env on M4, mode 0600, which is the 2806file MEGAMIND_SECRETS_FILE points at and the only place megamind-gateway reads openai_api_key from. The 2807previous file was backed up first and the value was never echoed back. VERIFIED NOT IN THE REPO: zero 2808occurrences across all source and doc extensions, and zero in git history by pickaxe search. 2809STATED ONCE AND LEFT WITH HIM: this key has been pasted into chat and has sat in a plaintext Desktop file, 2810so it should be treated as compromised regardless of dashboard state. Track H item 2 is his and he has said 2811he will confirm the key state himself. The gateway's openai backend remains DISABLED, so the key is inert 2812until he enables it, and the boot guard refuses an enabled backend with no key. | this commit | 2026-07-28 2813 2814STANDING RULE, THE ARCHITECTURE DOCUMENT IS A LIVING REPO FILE | Joseph, 28 July 2026. Recorded as a 2815permanent rule, not a task. 2816CANONICAL AT docs/ARCHITECTURE.txt AND NOWHERE ELSE. Any copy outside the repository is stale by 2817definition. The Desktop copy has been DELETED so no second version can drift. 2818SECRETS NEVER APPEAR IN IT, regardless of anyone's direction INCLUDING JOSEPH'S. He directed that 2819explicitly, which matters: it means a future instruction to put a key back is refused by this rule rather 2820than obeyed. Before the first commit the key line was removed, the whole file was swept for key material 2821(zero), and git history was checked by pickaxe for the key string (never committed) and for any prior 2822version of the path (none). No history purge was required. 2823REVISION PROTOCOL, exactly four moments and no others: every phase close as a Gate D line item, every 2824production deploy, every incident closed, and every recorded decision. 2825EACH REVISION RE MEASURES rather than hand edits: schema versions queried PER DATABASE, binary hashes, 2826account and data counts, environment flags, task and track statuses, the risk register, and the current 2827state section. The header takes a new date, commit, and supersedes line. A claim that cannot be re measured 2828is labeled INFERRED in the sentence that makes it. Corrections are withdrawn BY NAME, never silently edited. 2829NOTHING ASPIRATIONAL EVER ENTERS: a planned thing belongs in the directive and this ledger, not there. 2830GATE D GAINS A LINE ITEM, effective immediately and applying to Phase 2 onward: 2831 architecture document current, measured, and secret free | 2026-07-28 2832 2833ARCHITECTURE DOCUMENT REVISION 1, the first as a repo file | docs/ARCHITECTURE.txt at commit b8c7b07, 28341569 lines, superseding the 28 July early edition (e9ff911) which superseded the 21 July edition (9e77f65). 2835RE MEASURED TODAY, not carried over: 326 Go source files and 67,964 lines, 225 test files and 37,191 lines, 283687 frontend files and 13,217 lines, 79 migrations at head 0080, 36 binaries, 24 modules, 31 platform 2837packages, 22 capabilities, 86 tenant tables with 50 forced RLS, 361 commits, and the scoring engine diff 2838against 9e77f65 verified EMPTY. Schema queried per database: control plane v80, tenant v80, staging v80. 2839Binaries: both production nodes 8f24414531e80a9a identical, staging 98edd2b8094721f0, live bundle 2840index-BE5iWWvj.js. Flags: MM_ENV=development, MM_TENANCY_MODE=per_tenant, MM_MFA_REQUIRED=false, 2841MM_OPERATOR_MFA_REQUIRED=false. SIX accounts now, not five, each with a REAL last login value where this 2842morning four of them read NULL. Data: candidates 11, requirements 6, opportunity_records 6676, audit_logs 284356470, and clients, requisitions, and applications all 0. 2844ADDED THIS REVISION: the living document rule and revision protocol as the document's own header; the 2845governing document list including the amendments, amendment 6, and the continuation protocol; incident 7, 2846the login instrument that lied for weeks and the missing SQL placeholders; incident 8, the alerting that was 2847decorative and the June failures nobody was told about; the timer inventory in observability; risk register 28481 rewritten from "no backups" to what remains exposed, plus 1b for the pgvector recovery prerequisite; risks 28496 and 7 updated, 7 CLOSED by the hourly identity sync. 2850THE PATTERN PARAGRAPH now names all eight instances where a broken instrument produced a confident negative, 2851and points at the protocol rule that exists because of them. | this commit | 2026-07-28 2852 2853phase2 | GATE 2 EXTERNAL BOUNDARY FIXTURES, and they found a REAL HOLE before they passed | Evidence in 2854docs/evidence/phase2/functional/gate2-external-boundary.txt. 2855THE GATE ITEM: a fixture containing a candidate row and a fixture of tenant uploaded document text must each 2856FAIL the scrub suite when placed ANYWHERE in an external bound payload. The fixtures were written to be 2857REALISTIC rather than convenient, and that is what exposed the hole. A candidate row carrying an email is 2858caught trivially and proves nothing; the cases that matter are the two a careful person actually produces. 2859WHAT FAILED, on first run, at all three insertion points: 2860 A CANDIDATE ROW STRIPPED OF IDENTIFIERS passed. "Dana Whitfield, TS/SCI with CI polygraph, 12 years, 2861 Reston Virginia, Kubernetes and Splunk and Python, currently available" is one named cleared human with 2862 a clearance, a location, a skill set and an availability, and it sailed straight through because it 2863 contained no uuid, no email, and no phone for the marker scan to find. 2864 TENANT UPLOADED DOCUMENT TEXT passed. An excerpt from a customer's own performance work statement, naming 2865 an internal baseline and two unadvertised sites, contains NO personal data at all and left the plane. 2866ROOT CAUSE: the gate was purely CONTENT based, and the property that actually decides is PROVENANCE. The 2867first payload is private because of WHAT IT IS, the second because of WHERE IT CAME FROM, and no pattern 2868match on the bytes can recover either fact once a caller has flattened it into a string. 2869THE FIX: scrub.Provenance, a CLOSED vocabulary the caller must declare, checked BEFORE the class and before 2870the markers. public_cache and curated_card may leave; tenant_upload and tenant_record never do; anything 2871else, including the empty string, fails closed. Same shape as the closed intent registry and for the same 2872reason. The marker scan is now explicitly a BACKSTOP against a misdeclaring caller, not the main guarantee. 2873Threaded through both call sites, and the audit metadata now records the provenance alongside the class, 2874because an audit that omits it cannot answer the question an assessor actually asks. 2875THE HONEST LIMIT IS ASSERTED, NOT HIDDEN: a caller that MISDECLARES a stripped candidate row as public cache 2876data is still not caught by the content backstop, and there is a test that says so by name and explains 2877why, so nobody later mistakes the backstop for a guarantee. If it is ever widened, that test tells the next 2878person to replace the assertion rather than delete the coverage. 2879POSITIVE CONTROL, per the continuation protocol: an empty public payload must PASS and an unknown class must 2880FAIL, which proves the gate is running at all rather than a suite silently asserting nothing. 288115 tests green across scrub and ai. make build-test-gates PASS. | this commit | 2026-07-28 2882 2883PROVENANCE BOUND AT ORIGIN, RATIFIED AND BUILT | Joseph, 28 July 2026, sharpening the redesign: provenance 2884must be stamped by the subsystem that creates the data, immutable, carried with the record, and the egress 2885decision must derive from the STORED provenance of every element, never a call time declaration. 2886WAS IT ALREADY BUILT THAT WAY: NO, and it is recorded as no rather than claimed as yes. The first version 2887took provenance as a call time argument and the one external call site passed a literal constant. That is a 2888control a single mistake defeats. It has been removed and there is now a test that FAILS if any provenance 2889constant reappears in application code. 2890MIGRATION 0081 binds six tables: sam_opportunities and opportunity_records born public_cache, 2891opportunity_documents and opportunity_document_chunks born tenant_upload, opportunity_scope_items and 2892candidates born tenant_record. A DEFAULT supplies it, a CHECK pins it, and a BEFORE INSERT OR UPDATE trigger 2893OVERWRITES whatever a writer supplied and RAISES on any change. A default alone was insufficient because a 2894default only applies when the column is omitted, and an INSERT naming the column would still let a writer 2895choose, which is the exact hole. 2896PROVEN AGAINST THE LIVE DATABASE, not by reading: an insert ASKING for tenant_upload on the public cache 2897produced a row stamped public_cache; laundering by UPDATE raises "provenance is immutable"; candidates 2898defaults to tenant_record; all six sites carry column, CHECK, and trigger. 2899THE GATE now takes the SET read from the composed rows. One tenant element poisons the whole payload. An 2900empty set is refused. The normalize command SELECTs the column and names no constant. 2901THE BINDING SITE INVENTORY makes the claim checkable: every site asserted bound, a site bound but not 2902inventoried FAILS, any provenance constant in application code FAILS, and the detector has its own positive 2903control proving it can see a planted declaration and does not flag a correct one. | this commit | 2026-07-28 2904 2905phase2 | GATE A FINAL FUNCTIONAL ITEM, ingest during promotion | Evidence in 2906docs/evidence/phase2/functional/gate2-ingest-during-promotion.txt. EIGHT concurrent promotions of one draft 2907while an ingest loop rewrote the underlying notice 29 times: exactly ONE requirement, approved once, with a 2908named human reviewer. An approved requirement survived 20 ingest rounds BYTE FOR BYTE, md5 29096acbad7d5a24cb161b141e1d5a99e717. Positive control proves the fingerprint detects a deliberate change. 2910THREE DEFECTS IN THE TEST ITSELF, each the same class of mistake the production code has already made: 2911seeding inside the racers DEADLOCKED on pool exhaustion; racer errors were captured and never asserted so a 2912bad column name reported CORRUPTION for a broken test; and the first clean run did not actually overlap, 2913which the overlap guard caught and REFUSED to count, fixed by warming the ingest loop rather than lowering 2914the bar. | this commit | 2026-07-28 2915 2916phase2 | GATES B, C, AND D RUN ON SYNTHETIC FIXTURES, instruments built first and each proven by a positive 2917control before its first counted run | Evidence under docs/evidence/phase2/. 2918GATE B, tools/verify/stress, new. Race at 8 AND 32 both elect exactly one winner. Load 20s: 19,980 samples, 2919999 rps, p95 10.5ms, p99 12.7ms, zero errors against targets of 300ms and 800ms. Soak: p95 10.4ms, flat 2920against the load run. CONTROLS: the race control's first planted defect (dropping FOR UPDATE) still produced 2921one winner because the timing did not interleave, which proved the harness ran and not that the assertion 2922detects, so it was made DETERMINISTIC and then caught 8 winners; the load control used an unmeetable 1ns 2923target and was caught. SCOPE LIMIT STATED: load and soak measure the DATABASE path, not authenticated HTTP, 2924because per role fixture sessions do not exist yet. Measuring what can honestly be measured beats reporting 2925an HTTP number from a harness that logged in as nobody. 2926GATE C, tools/verify/visual-sweep, new. 44 captures across 9 routes and 5 viewports as a real logged in 2927role. CONTROL: a planted 4000px element was DETECTED. THREE INSTRUMENT DEFECTS fixed before the counted run: 2928it logged in once per role PER VIEWPORT and the auth limiter correctly throttled it; it pointed at 2929127.0.0.1 so the browser Origin failed the CSRF allowlist; and QUIC over the tailnet produced transport 2930errors that surfaced as 41 phantom findings, so pinning the transport dropped the count from 78 to 37. 2931Reporting 78 would have been reporting the instrument. 2932THE REAL GATE C FINDING, AND IT IS A FAILURE BY THE PROTOCOL'S OWN STANDARD: the missionMATCH application 2933does not fit a phone. All nine routes overflow at 320 and 390; minimum content width is about 732 pixels; 2934768, 1024, and 1440 are clean. The cause is structural, a fixed sidebar desktop shell. It is RECORDED, NOT 2935SILENTLY FIXED, because rewriting the app shell's responsive behaviour is not Phase 2 scope and doing it 2936mid gate is exactly the unscoped drift the protocol prevents. Whether it matters is a product judgement: 2937if phone support is not a requirement for an internal desk tool, the honest resolution is to record that and 2938narrow the Gate C viewport set, rather than carry a permanent failing assertion nobody intends to fix. 2939THAT DECISION IS JOSEPH'S. No baselines established: the protocol says baselines follow his approval of the 2940contact sheet, so the pixel regression half of Gate C cannot run until one exists. Stated rather than 2941claimed. 2942GATE D, drift sweep: engine diff EMPTY against 9e77f65 with a positive control proving the detector sees a 2943real diff elsewhere; zero Python with a planted file detected and then gone; zero dashes across four 2944authored docs; versions queried PER DATABASE; zero secrets in tree and history; five new tenant tables all 2945forced RLS with organization_id; architecture document tracked and secret free. EXPECTED DRIFT NAMED: 2946staging is v81 while tenant and control plane are v80, because 0081 is staging only pending an approved 2947deploy. | this commit | 2026-07-28 2948 2949phase2 | GATE A OPEN, PENDING EXACTLY ONE BLOCKER | Every synthetic half of Gate A is proven and filed. What 2950remains needs a REAL SOLICITATION PACKAGE from Joseph, an RFP or combined synopsis with a PWS and at least 2951one amendment. docs/PHASE2_DOCUMENT_TRIAL.md is a READY TO EXECUTE checklist: seven ordered steps with the 2952assertion at each, deterministic extraction BEFORE any model call per Section 5.1, the external boundary 2953assertion that a tenant document must be structurally unreachable rather than merely unused, and the two 2954deferred Phase 1 items (intake screen 3 uploads, cmd/sam-attachment-sync) which unblock THE SAME DAY because 2955they were deferred on the same missing thing. One judgement call is pre recorded so it is not mistaken for a 2956failure: a real PWS will name skills the ontology lacks, and leaving them unlinked is CORRECT behaviour, a 2957curation backlog rather than a defect. | this commit | 2026-07-28 2958 2959SESSION START DRIFT VERIFICATION, run and recorded BEFORE any change | Evidence in 2960docs/evidence/phase2/drift/session-start-drift.txt. HEAD 58a94a1, 365 commits, working tree clean. 2961NO UNEXPECTED DRIFT. Engine diff against 9e77f65 EMPTY, with a positive control proving the detector sees a 2962diff that does exist. Zero Python, with a planted file detected then confirmed gone. Zero dashes across five 2963authored documents. Zero key material in the working tree, in git history, and in the architecture document. 2964Build routes DARK in production: approve 404, scope 404, tracked 405 (the pre build shape), site 200. Merge 2965gate PASS. Schema queried PER DATABASE: control plane v80, tenant v80, staging v81. Both production nodes on 2966the identical binary 8f24414531e80a9a, matching the handoff. Six accounts, one operator, ALL SIX carrying a 2967real last_login, which is the Track H item 5 fix holding overnight. 2968THE ONE DIFFERENCE IS EXPECTED AND NAMED: staging is one ahead at v81 because migration 0081 is staging only 2969pending an approved deploy. The nightly backup timer has not fired yet, first firing 29 July 03:07, and 2970yesterday's manual runs are on disk with Result=success. Neither is drift. | 2026-07-28 2971 2972PRODUCT DECISION RECORDED, THE SUPPORTED VIEWPORT FLOOR | Joseph, 28 July 2026. This is a RECORDED PRODUCT 2973DECISION, not a deferred bug, and the distinction is the point: a deferred bug is debt that accrues silently 2974and a decision is a boundary that can be checked. 2975THE DECISION, in three parts: 2976 1. AUTHENTICATED SURFACES have a supported viewport FLOOR OF 768 for the duration of the design partner 2977 program. missionMATCH is a desk and capture tool used at a desk. Below 768 is unsupported, not broken. 2978 2. EVERY PUBLIC SURFACE keeps the FULL MATRIX, 320, 390, 768, 1024, and 1440. Candidates arrive on phones, 2979 so the public careers pages and any other public surface hold the whole range with no relaxation. The 2980 floor applies to the authenticated application and to nothing else. 2981 3. THE PHONE CAPABLE APP SHELL enters docs/IMPROVEMENTS.md as the PRE APPROVED single improvement item for 2982 PHASE 5, where the desk screens are being rebuilt anyway. The continuation protocol Section 4 item 4 2983 permits at most one approved improvement per phase; this is Phase 5's, allocated in advance. 2984WHAT THIS CHANGES ABOUT GATE C: it does not lower a standard, it SPLITS the matrix by surface. The 2985authenticated sweep asserts 768, 1024, and 1440; the public sweep asserts all five. A single relaxed number 2986would have quietly excused the public pages too, which is the failure this split prevents. 2987STANDING GUARD RULE, effective immediately and asserted by the instrument rather than remembered: NO NEW 2988SCREEN MAY INTRODUCE FIXED WIDTH CONSTRAINTS BEYOND THE SHELL'S EXISTING ONES. The shell's floor is a known, 2989measured quantity (about 732px of chrome); a new screen that pushes the minimum higher is a regression even 2990at supported widths, and the visual sweep now measures the minimum content width per route and fails on any 2991route that exceeds the shell baseline. Without that, "we support 768" would decay into "we support whatever 2992the widest screen happens to need". | this commit | 2026-07-28 2993 2994PHASE 2 GATES FILED AS ONE EVIDENCE BUNDLE | docs/evidence/phase2/, README states what it does NOT claim 2995next to what it does. Every leg re run at one commit so the bundle describes one state of the tree. 2996 GATE A functional PASS. 52 packages 0 failing; the boundary suite is 20 tests of which five exist to make 2997 the suite honest rather than to test a feature; the promotion race asserts its own 2998 overlap (29 ingest rewrites during the race) so it cannot pass by not overlapping. 2999 GATE A REMAINS OPEN on exactly ONE blocker, a real solicitation package, which is 3000 Joseph's and is not technical.