Janady13/missionmatch / docs/ARCHITECTURE.txt

1================================================================================ 2missionMATCH PLATFORM 3Architecture, Data Flow, Product Breakdown, AI Layer, Operations, Incidents 4================================================================================ 5 6THIS IS A LIVING DOCUMENT. Canonical at docs/ARCHITECTURE.txt in the repository 7and NOWHERE ELSE. Any copy outside the repo is stale by definition. 8 9Revision 29 July 2026, revision 4, at commit 870d5ad. 10Supersedes revision 3 (9a73b61), revision 2 (b1a298d), revision 1 (b8c7b07), the 1128 July early edition (e9ff911), and the 21 July edition (9e77f65). 12Trigger for this revision: a phase close and two production deploys. Phase 2 scope 13completion added table extraction, split and merge, draft generation, and the three 14missing intake screens; migrations 0081 through 0084 shipped to production; and 15Gates A, B, C and D were all re run, Gate C for the first time against these screens. 16 17SECRETS NEVER APPEAR IN THIS DOCUMENT, regardless of anyone's direction, 18including the owner's. A previous copy of this file carried an API key at its 19tail; it was removed before this file was first committed, and git history was 20checked and confirmed never to have contained it. 21 22WHEN THIS DOCUMENT IS REVISED, exactly four moments and no others: 23 every phase close, as a Gate D line item 24 every production deploy 25 every incident closed 26 every recorded decision 27 28HOW IT IS REVISED. Volatile facts are RE MEASURED, never edited by hand: schema 29versions queried per database, binary hashes, account and data counts, 30environment flags, task and track statuses, the risk register, and the current 31state section. The header takes a new date, commit, and supersedes line. A claim 32that cannot be re measured is labeled INFERRED in the sentence that makes it. 33Corrections are withdrawn BY NAME, never silently edited. Nothing aspirational 34ever enters: a planned thing belongs in the directive and the ledger, not here. 35 36GOVERNING DOCUMENTS, read in this order: 37 CLAUDE.md 38 docs/CLAUDE_CODE_BUILD_DIRECTIVE.md the work order 39 docs/DIRECTIVE_AMENDMENTS_CONSOLIDATED.md amendments 1 through 5 40 docs/DIRECTIVE_AMENDMENT_6.md identity, roles, the operator plane 41 docs/BUILD_CONTINUATION_PROTOCOL.md the five gate phase close 42 docs/BUILD_STATE.md the append only ledger 43 docs/RUNBOOK_RESTORE.md how to actually restore 44 this file what the system IS, measured 45 46Everything here was measured. Where a number came from a command, that is said. 47Where something is inferred rather than observed, the sentence says so. Where an 48earlier claim was wrong, it is withdrawn by name rather than quietly edited, 49because a document that hides its corrections cannot be trusted about anything. 50 51CONTENTS 52 0 What this is 53 1 Physical topology 54 2 Control plane and data plane 55 3 Environments 56 4 Request data flow, the map 57 5 Authorization model 58 6 Authentication and session security 59 7 THE FOUR PRODUCTS, IN DEPTH 60 8 The matching engine 61 9 The opportunity pipeline 62 10 THE AI LAYER, UNDER SCRUTINY 63 10A THE LEARNING BOUNDARY, three doors and no fourth 64 11 Federation relationship to PANTHEON 65 12 File architecture 66 13 How it is built 67 14 Deployment and its real failure modes 68 15 Observability and what runs on a timer 69 16 THE 28 JULY 2026 INCIDENT RECORD 70 17 Current state, honestly 71 18 Risk register 72 19 The design decisions that explain the rest 73 20 Glossary 74 75SCALE, re measured 28 July 2026 at commit 9a73b61: 76 327 Go source files 68,304 lines 77 228 Go test files 37,903 lines (36 percent of the Go is tests) 78 87 TypeScript and TSX files 13,217 lines 79 80 SQL migrations, head 0081, plus a segregated EEO set of 2 80 36 command line binaries 81 24 domain modules, 31 cross cutting platform packages 82 22 capabilities, 8 roles, 4 products 83 86 tables in the tenant database, 50 with FORCED row level security 84 372 commits on feature/opportunity-to-outcome 85 scoring engine diff against the e74fd2e baseline: EMPTY, verified 86 (anchor corrected 29 July 2026, finding D8; 9e77f65 was the 24 June 87 project baseline and diffs 1476 lines, so it could never be empty) 88 89MEASURED STATE AT THIS REVISION, every number re measured rather than carried forward: 90 scale 339 Go source files, 236 test files, 94 frontend files, 83 migrations at 91 head 0084, 408 commits. The frozen scoring engine diff against e74fd2e is 92 EMPTY, and the anchor and path are both asserted to exist before that 93 result is trusted, because this check has failed open twice. 94 schema, queried per database, never summarized 95 control plane 100.77.114.40:5433/missionmatch v84 96 tenant 100.94.8.94:5432/missionmatch v84 97 staging 100.94.8.94:5432/missionmatch_staging v84 98 ALL THREE LEVEL for the first time since the build began. 0081 through 99 0084 are additive and carry no data, so they ship to production while the 100 routes stay dark, which keeps the Gate 6 freeze intact. 101 CORRECTION TO THE CONTROL PLANE ADDRESS, and it cost time to rediscover. 102 Earlier revisions wrote "megamind-agi:5433". megamind-agi has TWO tailnet 103 identities: the Windows host at 100.69.120.61, and the WSL2 instance at 104 100.77.114.40 running its own tailscaled. Postgres binds INSIDE WSL2, so 105 it answers on .40 and a connection to .61:5433 times out with no useful 106 error. There is no sshd inside WSL2 either, so the only route in is ssh 107 to the Windows host and then wsl with a STAGED script, never inline 108 quoting. 109 STAGING IS ONE AHEAD deliberately: migration 0081, provenance at origin, 110 is staging only pending an approved production deploy under the Track 111 guardrails. Named here so it reads as intended rather than as drift. 112 binaries 113 both production API nodes 8f24414531e80a9a (identical) 114 staging API 98edd2b8094721f0 115 live bundle /assets/index-BE5iWWvj.js 116 environment flags, production 117 MM_ENV=development MM_TENANCY_MODE=per_tenant 118 MM_MFA_REQUIRED=false MM_OPERATOR_MFA_REQUIRED=false 119 accounts, six, from the authoritative control plane 120 joseph.w.anady@gmail.com operator active last login 28 Jul 11:51 121 patriciacox6@aol.com superuser active last login 28 Jul 08:04 122 tallen282005@gmail.com superuser active last login 27 Jul 14:42 123 batwood@igotozarks.com recruiter active last login 27 Jul 15:17 124 trutle973@hotmail.com recruiter active last login 27 Jul 16:51 125 trackh.probe@missionmatch.test viewer DISABLED last login 28 Jul 16:36 126 The probe was created for one production positive control and retired. 127 Every last login value above is real and current, which was NOT true this 128 morning: see incident 7. 129 data, tenant database 130 candidates 11, requirements 6, opportunity_records 6676, 131 audit_logs 56470, clients 0, requisitions 0, applications 0 132 track and phase status 133 PHASE 2 current. Phases 3, 4, 5, 5A, 6 not started. 134 TRACK U complete, all six items, all four gate items proven. 135 TRACK H items 1, 3, 4, 5 complete. Items 2 and 6 are the owner's. 136 PHASE 2 Gate A is OPEN pending exactly ONE blocker, a real solicitation 137 package. Gate E is pending the real document trial plus the owner's 138 approval. Gates B, C, and D PASS. The complete bundle is filed at 139 docs/evidence/phase2/ with a README that states what it does NOT claim 140 next to what it does. See docs/PHASE2_DOCUMENT_TRIAL.md. 141 Tasks T1 and T2 complete. 142-------------------------------------------------------------------------------- 1430. WHAT THIS IS 144-------------------------------------------------------------------------------- 145 146A Cleared Workforce Intelligence Platform for United States government 147contractors, cleared staffing firms, proposal teams, and capture teams. It is 148NOT a generic applicant tracking system. Everything serves one loop: 149 150 Requirement -> Candidate Intelligence -> Match Scoring -> 151 Ranked Candidate Slate -> Proposal Readiness 152 153The current build program extends the front of that loop: 154 155 Opportunity -> Source Documents -> Scope Extraction -> Human Review -> 156 Approved Requirements -> Staffing Slates -> Proposal Workspace -> 157 Submission -> Outcome 158 159Legally the product is an Automated Employment Decision Tool. That single fact 160drives more architecture than any technical consideration: NYC Local Law 144, 161Title VII, Colorado and California fair employment law, the EU AI Act, and the 162vendor exposure recognised in Mobley v. Workday. Every design choice that looks 163paranoid below traces back to it. 164 165WHO IS ON IT TODAY. One production organization, Acme GovCon, five accounts: 166one operator (Joseph Anady), two tenant superusers (Patricia Cox, Tallen), two 167recruiters (Brian Atwood, one other). FG Talent Partners is the live design 168partner and the first client site. 169 170-------------------------------------------------------------------------------- 1711. PHYSICAL TOPOLOGY 172-------------------------------------------------------------------------------- 173 174 internet 175 | 176 Cloudflare 177 | 178 +------------------------v--------------------------+ 179 | BUBBLES Debian 13, i5-8350U, 15GB | 180 | 100.90.97.104 | 181 | | 182 | nginx TLS, fail2ban, ~60 other vhosts | 183 | | | 184 | +-- upstream mm_api (least_conn, passive | 185 | | health checks, keepalive 32) | 186 | | +-- 127.0.0.1:8094 Bubbles | 187 | | +-- 100.77.114.40:8094 megamind-agi | 188 | | | 189 | +-- limit_req zone=mm_auth 1r/s, answers 429 | 190 | +-- missionmatch.thatwebhostingguy.com | 191 | | root /var/www/sites/missionmatch | 192 | +-- missionmatch-staging.thatwebhostingguy.com | 193 | | root /var/www/sites/missionmatch-staging | 194 | | /api/ proxies to 127.0.0.1:8097 | 195 | +-- fg-talent.com | 196 | root /var/www/sites/fgtp | 197 | | 198 | mm-api /home/user/missionmatch/mm-api | 199 | mm-api-staging /opt/missionmatch-staging/mm-api | 200 | mmorch :9505 mmdb :9506 mmk :9500 mmreason | 201 +---------------------------------------------------+ 202 | | 203 | tailnet | tailnet 204 v v 205 +------------------------+ +---------------------------------+ 206 | M4 Mac mini | | MEGAMIND-AGI | 207 | 100.94.8.94 | | Windows 11 + WSL2 Ubuntu 26.04 | 208 | 10 core, 16GB | | 100.77.114.40 | 209 | | | i7-8700K 12t, 26GB allotted | 210 | PostgreSQL 16.14 :5432 | | | 211 | missionmatch | | PostgreSQL 18 | 212 | TENANT DATA PLANE | | :5432 PANTHEON cluster | 213 | org 11111111... | | :5433 missionmatch cluster | 214 | 86 tables, v84 | | missionmatch | 215 | missionmatch_staging | | CONTROL PLANE, v84 | 216 | full stack, v84 | | missionmatch_staging | 217 | | | EMPTY SHELL, unused | 218 | megamind-gateway :8090 | | | 219 | bearer authenticated | | mm-api :8094 second node | 220 | Ollama :11434 | | /opt/missionmatch/mm-api | 221 | bge-small-en-v1.5 | | | 222 | llama3.1:8b | | PANTHEON organs: mm-oracle 9402,| 223 +------------------------+ | mm-loom 9410, mm-serve, mm-eye, | 224 | mm-sem, mm-world, mm-game, | 225 +------------------------+ | mm-narrate, mm-shard, | 226 | VALKYRIE M1, 8GB | | mm-genserve | 227 | 100.89.0.6 codex 8790 | +---------------------------------+ 228 +------------------------+ 229 230WHY THIS SHAPE 231 232The API tier is genuinely liquid. mm-api holds no per process state: sessions 233live in the control plane database, CSRF is a signed double submit. Any request 234may be served by either node, and a session minted on one is valid on the other. 235Passive health checks take a dead node out and put it back with no operator 236action. Verified by stopping a node mid traffic: 14 of 14 requests succeeded. 237 238The DATABASE tier is deliberately NOT liquid, and this is the single most 239important architectural decision in the system. Recruiting correctness rests on 240FOR UPDATE row locks that serialise seat caps, stage transitions, and ownership 241claims. Multi master would break every one of them silently. So state is one 242primary per tenant. 243 244What IS distributed is TENANTS. tenant_directory records a host and port PER 245TENANT, so different tenants live on different machines while each still has 246exactly one primary and every lock keeps working unchanged. 247 248THE LIABILITY OF THIS SHAPE, stated plainly. Three of the four machines are 249consumer hardware in one physical location, two of them are a laptop and a 250desktop that can be closed or unplugged, and there is no off machine backup of 251either database. The design is cloud portable by construction and is running on 252a home network by circumstance. Those are different facts and only the first is 253an architectural claim. 254 255TOPOLOGY RULE, owner decision 29 July 2026, and it is enforced rather than 256aspirational: 257 258 BUBBLES CARRIES EXACTLY TWO THINGS 259 hosted website content, the 138 nginx vhosts 260 deployed production artifacts, the API node binary and the job binaries 261 systemd actually runs, plus two rollback points 262 263 EVERYTHING ELSE LIVES ON MEGAMIND-AGI 264 build workspace /home/user/missionmatch/workspace 265 document intake /home/user/missionmatch/inbox 266 archived build artifacts /home/user/missionmatch/archive 267 268Bubbles is the revenue web host. Build residue there competes with customer 269vhosts on a disk already at 82 percent, and a scratch file left beside a deployed 270binary is something a later session can mistake for deployed state. Audited and 271enforced on 29 July: 463 MB of stale rollback binaries, 59 MB of gate captures, 272and twelve staged scripts were archived to megamind-agi by checksum and only then 273removed, with every serving surface verified untouched afterwards. 274 275-------------------------------------------------------------------------------- 2762. CONTROL PLANE AND DATA PLANE 277-------------------------------------------------------------------------------- 278 279 CONTROL PLANE 280 megamind-agi 100.77.114.40 PORT 5433, database missionmatch 281 organizations, users, auth_credentials, mfa_totp, mfa_recovery_codes, 282 auth_sessions, user_invites, login_attempts, user_tool_access, 283 tenant_directory. Schema v84. 284 This is what MM_DATABASE_URL points at. 285 ANSWER EVERY IDENTITY QUESTION HERE. 286 287 TENANT DATA PLANE 288 M4 100.94.8.94 PORT 5432, database missionmatch. Schema v84. 289 Everything else. 86 tables, 50 with forced RLS. 290 Reached ONLY by routing through tenant_directory. Named in no env file 291 except as the value of MM_TENANT_DSN_TENANT_ACME. 292 ITS COPY OF users IS NON AUTHORITATIVE, and is currently stale and 293 incomplete: it was missing two of the five real accounts and had a 294 stale status on a third. 295 296 STAGING 297 M4 100.94.8.94 PORT 5432, database missionmatch_staging. Schema v84. 298 2766 organizations, 8872 candidates, its own API, vhost, and isolated 299 role. Runs shared tenancy, not per_tenant. 300 301 NOT USED 302 megamind-agi 5433 database missionmatch_staging is an EMPTY SHELL. 303 Named in an early amendment. Must never be used. 304 305WHY SPLIT. Login resolves a user by GLOBAL email BEFORE any organization is 306known. Handing the router to auth would make authentication depend on the very 307answer it exists to produce. So auth is wired to a direct pool and never to the 308router (cmd/api/main.go line 257), while business queries go through the router. 309 310Global email uniqueness does NOT break under per tenant databases, because 311identity is authoritative in ONE database. The unique index there is genuinely 312global. Copies of users in tenant databases exist only so business tables can 313foreign key to a user, and are synced, not authoritative. 314 315TWO MODES, selected by MM_TENANCY_MODE. shared: one database, isolation by 316forced RLS, the default, what staging runs. per_tenant: each organization's 317business data in its own database, resolved per request, RLS still forced 318INSIDE each tenant database so the mechanisms stack. Production runs per_tenant, 319confirmed from the boot log: secrets_backend=env max_pools=32 320max_conns_per_pool=4 idle_evict=15m0s. 321 322PER ENVIRONMENT ROLE NAMES, from incident D6, 29 July 2026. 323 324ROLES ARE CLUSTER SCOPED. DATABASES ARE NOT. The production tenant database and 325missionmatch_staging live in ONE M4 cluster and therefore share ONE role 326namespace, so for a long time they shared missionmatch_migrator. A rehearsal 327aimed at staging changed the password of a role production depended on, and one 328of the two API nodes was down for one minute fifty nine seconds as a result. 329nginx failed over and no client saw a 5xx, which is the redundancy working rather 330than an argument that the mistake was small. 331 332 missionmatch_migrator PRODUCTION TENANT only 333 missionmatch_staging_migrator STAGING only, CONNECT on the production 334 database REVOKED, proven by refusal 335 336The containment is proven with a positive control rather than asserted: the 337staging role reads 2,784 staging organizations, and the same credential against 338the production database is refused with permission denied. 339 340THE RULE THAT GOES WITH IT: on a shared cluster, roles, role passwords, 341extensions, tablespaces, and cluster settings are PRODUCTION CHANGES by 342definition, whichever database the statement names. Rehearsals of cluster scoped 343mutations do not exist. 344 345Every credential and every consuming location is inventoried in 346docs/CREDENTIAL_CONSUMERS.md, references only. Rotations enumerate from it, 347because the rotation that caused D6 was enumerated from memory and missed two. 348 349-------------------------------------------------------------------------------- 3503. ENVIRONMENTS 351-------------------------------------------------------------------------------- 352 353 DEVELOPMENT STAGING PRODUCTION 354 API local go run Bubbles :8097 Bubbles :8094 and 355 megamind-agi :8094 356 control DB local or staging M4 missionmatch_ megamind-agi:5433 357 staging missionmatch 358 data plane same as control same (shared) M4:5432 missionmatch 359 tenancy shared shared per_tenant 360 DB role missionmatch_app missionmatch_ missionmatch_app 361 staging_app 362 MM_ENV development development development 363 MFA off off off 364 operator MFA on (the default) off off 365 binary HEAD HEAD baseline + track only 366 367THE STAGING ISOLATION STORY, because it failed twice before it passed. 368 369Round one: staging and production ran as the SAME role on the SAME cluster, so 370staging credentials could connect to production and successfully write a row. 371The zero row reads people took for isolation were RLS with no org GUC set, not a 372privilege denial. PostgreSQL grants CONNECT to PUBLIC by default, which is what 373made every role in the cluster able to reach production. Fixed by creating 374missionmatch_staging_app with grants mirrored inside missionmatch_staging only, 375granting CONNECT on production explicitly to the four legitimate roles, and 376revoking the PUBLIC blanket. 377 378Round two: that fix mirrored TABLE grants and not FUNCTION grants, so the new 379role held EXECUTE on ZERO of the nineteen SECURITY DEFINER auth functions. 380Nothing could log in on staging at all and invite acceptance returned 500. Found 38128 July, nineteen grants mirrored in. IF A ROLE IS EVER CLONED AGAIN, CLONE 382FUNCTIONS AS WELL AS TABLES. 383 384-------------------------------------------------------------------------------- 3854. REQUEST DATA FLOW, THE MAP 386-------------------------------------------------------------------------------- 387 388 browser 389 | 390 | https, __Host- session cookie, X-CSRF-Token header 391 v 392 nginx on Bubbles 393 | TLS terminate, limit_req on /api/auth/, proxy_next_upstream 394 | (never non_idempotent: a GET may retry on the other node, a POST never 395 | does, because a retried write could apply a placement twice) 396 v 397 chi router (cmd/api/main.go) 398 | 399 +-- obs middleware structured slog, request id, metrics 400 +-- ratelimit token bucket, PER PROCESS 401 +-- CSRF Protect signed double submit, bound to the session hash 402 +-- SessionMiddleware __Host- cookie -> auth.Validate 403 | | 404 | +-> CONTROL PLANE: session row, state=active, mfa_satisfied 405 | +-> loads role AND tool entitlement onto the principal 406 | +-> EffectiveTools widens the top seats BY RULE, once, here 407 | +-> tenant.WithOrg(ctx, org) + tenant.WithUser(ctx, principal) 408 | 409 +-- RequireCapability what you may DO (22 capabilities) 403 410 +-- RequireTool what you may SEE (4 products) 404 411 +-- RequirePlatform the platform plane (operator only) 404 412 | 413 v 414 module handler (internal/modules/<name>/handler.go) 415 | thin: decode, call the service, map the error 416 v 417 service (service.go) 418 | 419 +-- db.RunInTenantTx(ctx, fn) 420 | | 421 | +-- per_tenant: router resolves org -> tenant_directory -> 422 | | secrets resolver (MM_TENANT_DSN_<REF>) -> tenantpool 423 | | (bounded LRU, singleflight, ping before cache, refcount pin, 424 | | typed errors that NEVER fall back to another tenant) 425 | | 426 | +-- shared: the one pool 427 | | 428 | +-- BEGIN 429 | +-- SELECT set_config('app.current_org_id', $1, true) 430 | | transaction local, so it cannot leak across the pool 431 | +-- fn(ctx, tx) <-- ALL repository work happens here 432 | +-- COMMIT (or ROLLBACK on any error) 433 | 434 v 435 PostgreSQL 436 forced row level security, policy = org matches the GUC, 437 no GUC set yields ZERO rows (default deny, fail safe) 438 439TWO STACKED SAFEGUARDS: 440 Go side no validated org in context, RunInTenantTx refuses before the pool 441 is touched at all. 442 DB side an unset GUC makes current_setting(...,true) NULL, so the policy 443 matches nothing. 444 445PROVEN, not asserted, on 28 July using the production DSN the running process 446holds: connect as missionmatch_app, confirm rolbypassrls=false, read candidates 447with NO org GUC and get 0 rows, read the same table inside a transaction with 448the GUC set and get 11. That is the application's real path. 449 450THE TRAP THAT CATCHES EVERYONE. A query with an explicit 451"WHERE organization_id = $1" predicate LOOKS scoped and is not. Forced RLS 452filters the row out before the predicate is evaluated, so a statement on a bare 453pool connection with no GUC matches ZERO rows and returns "not found" for a row 454that plainly exists. PATCH /me/profile answered 404 to the signed in user on her 455own account for exactly this reason. Any read or write of a tenant table must be 456inside RunInTenantTx, without exception. 457 458THE RATE LIMITER IS PER PROCESS, so a client's effective allowance is the 459configured rate times the node count. That is why the login throttle lives in 460nginx, where it is globally correct regardless of how many nodes exist, and the 461in process limiter is a second layer only. 462 463-------------------------------------------------------------------------------- 4645. AUTHORIZATION MODEL 465-------------------------------------------------------------------------------- 466 467THREE INDEPENDENT AXES. Collapsing any two would be wrong. 468 469 ROLE = what you may DO 470 viewer(0) member(1) recruiter(2) manager(3) admin(4) owner(5) 471 superuser(6) operator(7) 472 473 OPERATOR is the PLATFORM PLANE, above and outside the tenant ladder. It 474 holds every platform capability and every tenant capability in every 475 tenant. It exists because superuser previously meant two things at once, 476 the platform seat and the client's top seat, which is why a client was 477 once shown platform shaped controls that refused her. 478 479 SUPERUSER is now the TENANT top seat: everything BY RULE WITHIN ITS 480 TENANT, explicitly not the platform capabilities. 481 482 Grant rule: strictly BELOW your own, with one exception. Superuser may 483 grant superuser, because a tier that cannot reproduce can only shrink and 484 an organization losing its last holder would be locked out of its own 485 access control. Nobody may change their OWN role, ever. 486 487 PLATFORM CAPABILITIES are the ones that act across every tenant at once. 488 Today exactly {curation.curate}, which writes the GLOBAL skill ontology 489 every tenant is scored against. A client's top seat approving an ontology 490 edge would change what a competitor's candidates score. 491 492 TOOL = what you may SEE 493 missionmatch, screenmatch, slatematch, recruitmatch. 494 Per user, explicit, FAILS CLOSED for every role below superuser. Two 495 bounds on granting: outrank the target, AND confer only tools you hold. 496 Revocation needs only the first, because taking access away is safe. 497 BY RULE for the top seats, because relying on provisioning to keep an 498 invariant true is not the same as the invariant being true: the design 499 partner's top seat held superuser with ZERO grants and saw nothing. 500 Enforced on each tool's route GROUP. Answers 404, never 403. 501 502 PLANE = whether the surface is tenant scoped at all 503 RequirePlatform, operator only, 404. A platform surface must be 504 indistinguishable from a path that was never built. 505 506NO SILENT GREY. A disabled entitlement control must STATE ITS REASON, and the 507two reasons are distinct because they have different fixes: the organization is 508not entitled to that product, or you do not personally hold that tool. 509 510OPEN DESIGN POINT: no organization level product entitlement table exists yet, 511so every tenant is treated as entitled to all four. 512security.OrgEntitledTools(orgID) is the single place that narrows when the 513Phase 5A operator console lands. It takes orgID today, unused, so adding the 514lookup is a body change and not a signature change through every caller. 515 516-------------------------------------------------------------------------------- 5176. AUTHENTICATION AND SESSION SECURITY 518-------------------------------------------------------------------------------- 519 520 PASSWORD KDF PBKDF2 HMAC SHA 256 (SP 800-132). NEVER argon2 or bcrypt, which 521 sit outside the validated boundary. 600,000 iterations. Input 522 capped at 64 characters to stay inside the CAVP tested length 523 bound. Exceeding the tested iteration envelope is a STRONGER 524 choice, not a deviation; state it that way to an assessor. 525 PEPPER Versioned, HMAC SHA 256 under a server held key, applied BEFORE 526 PBKDF2 so it stays in boundary. Per credential pepper_version 527 so it can rotate without a flag day. 528 MFA TOTP RFC 6238, HMAC SHA 256 end to end, 32 byte secret, 6 529 digits, 30 seconds, strictly monotonic replay guard so a used 530 code and every earlier in window step cannot be replayed. 531 Implemented on crypto/hmac rather than a library, so every 532 primitive is provably inside the validated module. Secrets 533 AES 256 GCM at rest with a key_version. 10 recovery codes. 534 SESSIONS Opaque, server side, 32 random bytes, stored ONLY as a SHA 256 535 hash. __Host- cookie, HttpOnly, Secure, SameSite Strict. 536 Instantly revocable. Two stage: auth_pending then active. 537 15 minutes idle, 12 hours absolute, computed by the DATABASE 538 clock, not the application's. 539 CSRF Signed double submit BOUND to the session hash. That binding is 540 what stops an attacker being handed a token that works against 541 a victim's session. 542 LOCKOUT Keyed on EMAIL alone, never (email, ip). Scoped to the pair it 543 was free to bypass by rotating source addresses, which defeated 544 the per ip rate limiter with the same move, so two controls 545 shared one exit. The audit records the source spread. 546 IDENTITY One human, one account, one org. Global unique index on 547 lower(email). Login resolves by global email BEFORE the org GUC 548 is set, through a SECURITY DEFINER cross org lookup. 549 PROVISIONING No public signup. First org and owner by the migrator role 550 through a migrator only SECURITY DEFINER function. Everyone 551 else by authority bounded invite. Delivery is MANUAL at pilot: 552 the endpoint returns the one time link and a human delivers it. 553 DEFERRED OIDC Login.gov, Entra Gov, SAML, WebAuthn, PIV CAC, and public 554 forgot password are sub phase 2.9. The five federal auth 555 libraries are deliberately not in go.mod yet. 556 557CURRENT DEVIATIONS, both deliberate and both recorded: 558 MM_MFA_REQUIRED is FALSE everywhere, so login is password only. Config 559 REFUSES this when MM_ENV is production. 560 MM_OPERATOR_MFA_REQUIRED is FALSE in production only, set 28 July 2026 at 561 the owner's explicit direction. Default is true, staging proves the rule, 562 config refuses it in a production MM_ENV, and the boot log states it at 563 every start. Re arming is one environment variable per node. 564 565-------------------------------------------------------------------------------- 5667. THE FOUR PRODUCTS, IN DEPTH 567-------------------------------------------------------------------------------- 568 569All four are ONE binary, ONE database, ONE candidate spine. A product is a 570TOOL ENTITLEMENT plus a route group plus a set of screens. There is no 571duplicated candidate model, no second scoring engine, and no per product 572database. That is the whole reason a person can sit on a proposal slate and in 573a recruiting pipeline at once and resolve to one candidate id. 574 575................................................................................ 5767.1 missionMATCH "who can support this mission today" 577................................................................................ 578 579WHO IT IS FOR: capture leads and program managers who need to know whether the 580bench can cover a mission right now. 581 582SCREENS (frontend/src/tools/missionmatch/) 583 CommandCenter.tsx mission readiness aggregation, the executive view 584 Candidates.tsx the candidate roster and profile 585 Ask.tsx the natural language interface to the AI layer 586 WhoCanSupport.tsx the named question the product is built around 587 588API SURFACE 589 GET /command-center readiness tiles 590 GET /command-center/federal-signal external signal overlay 591 POST /command-center/upcoming/{id}/adopt adopt a forecast record 592 GET /candidates/intelligence enriched candidate view 593 GET /requirements/{id}/support who can support this 594 POST /ai/query the Ask path 595 596WHAT IT ACTUALLY COMPUTES. The command center derives readiness from live 597counts, not from a stored score: available candidates by clearance tier against 598open requirement demand, producing a readiness band and a list of named gaps 599("NEED 10 Intelligence Analysts, 3 Data Scientists, 2 Cyber Architects"). The 600demo seed is deliberately shaped so that Win Impact reads HIGH on the Candidates 601page while the Command Center still names the open gaps, so the two screens tell 602one connected story rather than contradicting each other. 603 604STATE: works. Renders with zero 5xx. Data is thin: 11 candidates, 6 605requirements in production. 606 607................................................................................ 6087.2 screenMATCH "is this candidate qualified for this requirement" 609................................................................................ 610 611WHO IT IS FOR: recruiters and proposal staff qualifying ONE person against ONE 612requirement, and needing to defend the answer. 613 614SCREENS (frontend/src/tools/screenmatch/) 615 ScreeningQueue.tsx the work queue of pending evaluations 616 CandidateEvaluation.tsx the full explainable result for one pairing 617 ScreenCompare.tsx two candidates side by side 618 AliasSettings.tsx skill alias and term normalization controls 619 620API SURFACE 621 POST /match the flat legacy alias 622 POST /screen the canonical evaluation 623 GET /requirements/{id}/candidates ranked for one requirement 624 GET /candidates/{id}/opportunities the reverse direction 625 GET /candidates/{id}/clearance-provenance where a clearance claim came 626 POST /mission-check/screen the browser parse path 627 GET /mission-check/opportunities 628 629WHAT MAKES IT DEFENSIBLE. Every evaluation returns the per component breakdown, 630the band, which caps were applied and why, the engine version, the timestamp, 631and which candidate skill matched which required skill with the similarity that 632matched them. That last item is the one an auditor asks for and the one most 633products cannot produce. 634 635CLEARANCE PROVENANCE is its own endpoint on purpose: a clearance value is a self 636reported indicator, and the product must be able to show WHERE the claim came 637from rather than presenting it as fact. 638 639STATE: works. The most complete product in the platform after the engine. 640 641................................................................................ 6427.3 slateMATCH "who belongs on the proposal" 643................................................................................ 644 645WHO IT IS FOR: proposal and capture teams building a named staffing slate for a 646bid. THIS IS THE COMMERCIAL DIFFERENTIATOR: requirement to slate. 647 648SCREENS (frontend/src/tools/slatematch/) 649 Slate.tsx the ranked slate with per member decisions 650 Capture.tsx opportunity intake (Phase 1, DARK in production) 651 OpportunityDetail.tsx opportunity intelligence and edit (DARK) 652 653API SURFACE 654 POST /slates/generate build a ranked slate 655 GET /requirements/{id}/slates slates for a requirement 656 GET /slates/{id} one slate 657 POST /slates/{id}/members/{candidateID}/decision approve or reject 658 GET /slates/{id}/members/{candidateID}/trace the decision trace 659 GET /slates/{id}/proposal-readiness readiness for the bid 660 GET /slates/{id}/export csvsafe export 661 662 DARK IN PRODUCTION until Gate 6, mounted only at repository HEAD: 663 POST /opportunities/tracked, PATCH .../{id}, GET .../scope, 664 POST .../scope/items, POST /requirement-drafts/{id}/approve and /reject 665 666DECISION VOCABULARY: slate_members.decision is CHECK constrained to 667pending, approved, rejected. Human in the loop is a database constraint, not a 668convention. Nothing auto rejects. 669 670EXPORT goes through internal/platform/csvsafe, which escapes formula injection, 671because a slate export is opened in Excel by someone who did not create it. 672 673STATE: the slate path works. The opportunity capture front end is BUILT and 674verified on staging but deliberately DARK in production, because the Gate 6 675freeze says no build feature route is reachable in production until the whole 676program passes. Proven dark after every deploy. 677 678................................................................................ 6797.4 recruitMATCH "work the desk" 680................................................................................ 681 682WHO IT IS FOR: the recruiting desk. Clients, job orders, candidate pipeline, 683interviews, offers, placements. This is the operational heart and it is the 684LEAST exercised product in the platform. 685 686SCREENS (frontend/src/tools/recruitmatch/), 11 of them 687 RecruiterDashboard.tsx the desk overview 688 Clients.tsx / ClientDetail.tsx 689 Requisitions.tsx / RequisitionDetail.tsx 690 Pipeline.tsx the stage board for one requisition 691 Assignment.tsx one application in depth 692 Tasks.tsx ActivityFeed.tsx Analytics.tsx Reports.tsx 693 694API SURFACE, the largest in the platform, 40 routes 695 clients GET/POST /clients, GET/PATCH /clients/{id}, 696 POST /clients/{id}/contacts, PATCH /contacts/{id}, 697 POST /clients/{id}/claim | /release | /reassign 698 requisitions GET/POST /requisitions, GET/PATCH /requisitions/{id}, 699 POST /requisitions/{id}/status | /claim | /release, 700 GET /requisitions/{id}/pipeline, 701 POST /requisitions/{id}/applications 702 applications GET /applications, GET /applications/{id}, 703 POST /applications/{id}/stage | /interviews | /offers | 704 /placement, GET /applications/{id}/assignment 705 offers POST /offers/{id}/status 706 placements GET /placements, POST /placements/{id}/end 707 work GET/POST /activities, PATCH /activities/{id}/note, 708 GET/POST /tasks, PATCH /tasks/{id}, GET /tasks/queue 709 reporting GET /analytics, GET /reports, GET /dispositions 710 711THE STATE MACHINES, read from the live database constraints, not from docs: 712 713 applications.stage 714 sourced -> screened -> submitted -> interviewing -> offered -> placed 715 with rejected and withdrawn reachable from any stage 716 requisitions.status 717 draft, open, on_hold, filled, closed, cancelled 718 offers.status 719 extended, accepted, declined, rescinded, expired 720 placements.status 721 pending_start, active, completed, terminated, fell_off 722 723THE LOCKS THAT MAKE IT CORRECT. recruiting/ownership.go holds three FOR UPDATE 724locks (claim, release, reassign), requisition_service.go two, and 725placement_service.go one on the seat cap. These are the reason the database tier 726is not allowed to be multi master. Proven by racing them: 16 concurrent stage 727transitions elected one winner, 8 racing claims produced one owner, 8 placements 728into one seat produced one placement. 729 730STATE, AND THIS IS THE HONEST PART: recruitMATCH has ZERO PRODUCTION DATA. 7310 clients, 0 requisitions, 0 applications, 0 placements. The schema, the 732services, the endpoints, the locks, the screens, and the analytics are all 733built. Nothing has ever been created, because the New Client form is too thin to 734record a real client. Expanding it is Phase 5 item 1. 735Consequences today: 736 The dashboard "NOT TRACKED YET" panel is STALE HARDCODED COPY at 737 internal/modules/recruitmatch/derive.go:73 and is now wrong about 738 interviews, offers, and placements. 739 Revenue, bill rates, and candidate source are genuinely not tracked. 740 Time to fill has a migration but no service. 741 Dashboard stat cards are not clickable through to their filtered lists. 742 743................................................................................ 7447.5 THE SHARED SPINE 745................................................................................ 746 747ONE candidates table. ONE application record linking candidate to requisition. 748The frozen engine is reached from BOTH directions, a job order ranking 749candidates and a candidate ranking open job orders, through the same requisition 750to requirement bridge. screenMATCH is the deep evaluation of one pairing. 751slateMATCH is the proposal branch off the same spine. 752 753Shared read routes are gated on holding ANY tool rather than a specific one, 754because gating the candidate and requirement pickers to one product would break 755the other three, while leaving them open would expose the candidate list to an 756account entitled to nothing. 757 758THE FLAT ALIAS SURFACE. Pre split paths (POST /match, /slates/..., /candidates) 759still exist so older clients and the screenmatch redirect keep working. They 760carry the SAME tool entitlement as their canonical namespaces. They previously 761did not, which made the entitlement layer cosmetic: a user refused at 762/slatematch/slates/{id} could fetch the same slate, its decision trace, and its 763CSV export from /slates/{id} with no gate at all. The bookmark the gate exists 764to defeat was the flat one. 765 766-------------------------------------------------------------------------------- 7678. THE MATCHING ENGINE (the regulated core) 768-------------------------------------------------------------------------------- 769 770 Match Score = Clearance x0.25 + Skills x0.25 + Certifications x0.15 771 + CustomerExperience x0.15 + Experience x0.10 772 + LocationAvailability x0.10 773 774 Bands: Green (Proposal Ready) 90+, Yellow (Potential) 75 to 89.99, 775 Red (Recruiting Required) below 75. 776 777 Clearance: exact OR higher scores 100 percent; lower, expired, or unknown 778 scores 0. The hierarchy is encoded, so TS/SCI satisfies Secret and Secret does 779 NOT satisfy TS/SCI. 780 781 Skills: 100 percent of required met scores full, 80 to 99 partial, below 80 782 reduced. Exact match first, then semantic similarity through embeddings, and 783 the system always records WHICH candidate skill matched WHICH required skill 784 and the similarity, because an AEDT decision must be explainable at that grain. 785 786 TWO DETERMINISTIC BAND CAPS, applied after the band is derived. Neither ever 787 changes the Total; each can only LOWER a band: 788 1. A clearance gap caps at Red. 789 2. An unmet mandatory skill caps a Green at Yellow. 790 791 ALL scoring uses shopspring/decimal, never float64. The engine is a pure Go 792 package with no DB and no I/O, skill_matcher injected. That purity is not 793 aesthetic: it is what makes the score bit stable and reproducible, which is 794 what an AEDT compliance artifact requires. 795 796 THE ENGINE IS FROZEN. internal/modules/matching/engine is untouchable under 797 the current directive and its diff against the starting commit is checked and 798 must stay empty. Verified empty at this revision. 799 800 EEO data lives in a physically and privilege segregated schema the application 801 and scoring roles cannot read at all, with its own migration set and version 802 table so sqlc never models it. Demographics flow ONE WAY, into adverse impact 803 reporting run by a separate compliance role, never into a score. The matching 804 and search code must never reference the EEO tables. 805 806 THE COMPLIANCE ARTIFACT CHAIN, which is the actual product for a regulated 807 buyer: per component breakdown, human readable rationale, gap summary, engine 808 version, weights, caps applied, and a recordable decision trace. Humans 809 approve or reject every slate member and nothing is ever auto rejected. 810 Adverse impact reporting produces selection rates and four fifths ratios so a 811 customer can run their own LL144 or OFCCP audit. The platform SUPPLIES the 812 data for that audit; it does not constitute one, and vendor tooling does not 813 transfer the employer's liability. 814 815-------------------------------------------------------------------------------- 8169. THE OPPORTUNITY PIPELINE 817-------------------------------------------------------------------------------- 818 819An opportunity may enter through the SAM.gov API, manual entry, uploaded 820solicitation documents, a customer or partner scope, a spreadsheet import, or a 821future agency forecast feed. Every path converges into one workflow, and the 822SAFETY BOUNDARY is that nothing a machine derived becomes visible to the scoring 823engine without a named human approving it. 824 825 TABLES, all with organization_id, forced RLS, composite indexes: 826 ingest_manifest shared reference data, no RLS. EVERY synced row 827 and every knowledge card traces to a row here. 828 tracked_opportunities 31 columns. Unique partial index on 829 (org, public_opportunity_id) so a tenant cannot 830 track one notice twice. 831 opportunity_versions IMMUTABLE. A trigger allows only the is_current 832 flag to change and blocks DELETE outright. 833 opportunity_documents sha256 duplicate detection, extraction_status 834 state machine, malware scan hook between upload 835 and pending. 836 opportunity_document_chunks page, section, content hash, method, confidence 837 opportunity_scope_items ck_scope_items_cited REFUSES uncited machine 838 produced scope. A manual item may be uncited. 839 requirement_drafts ck_drafts_approved_has_requirement, 840 ck_drafts_decided_has_reviewer (so no automated 841 path can approve), ck_drafts_skill_flags_align 842 843 THE GATE RULES LIVE IN THE SCHEMA, not only in code. A future service bug 844 cannot write uncited scope or an unattributed approval, because the database 845 refuses it. The constraint names are listed so that claim is checkable. 846 847 PROMOTION is one audited transaction taking FOR UPDATE on the draft row, 848 recording the named approver, writing the requirement, and flipping the draft 849 together. Raced: 8 concurrent promotions produced exactly 1 requirement. 850 851 SEMANTIC RETRIEVAL IS LIVE. opportunity_records carries vector(384) with a 852 partial HNSW cosine index and a generated search_tsv with a gin index. 6,676 853 records embedded with qllama/bge-small-en-v1.5. "cybersecurity engineering 854 support for the Air Force" returns PACAF Cyber Systems Operations Support and 855 Cybersecurity Support Services at 0.78 cosine, ranked by meaning. 856 857 AMENDMENT HANDLING. sam-sync compares the source payload hash per tracking 858 tenant and on a change writes an immutable version, retires its predecessor in 859 the same transaction, cites the manifest, and raises an alert. Tenant authored 860 fields survive a source refresh BYTE FOR BYTE, proven by md5 fingerprint. 861 862 THE EXTERNAL BOUNDARY, redesigned 28 July 2026 after its fixtures found a hole. 863 864 PROVENANCE IS BOUND AT ORIGIN, migration 0081. Six tables carry a provenance 865 column that a DEFAULT supplies, a CHECK pins to one value, and a BEFORE INSERT 866 OR UPDATE trigger overwrites on insert and refuses to change on update. A row 867 in the public SAM cache is BORN public_cache; a tenant upload is BORN 868 tenant_upload; a candidate is BORN tenant_record. The application cannot choose 869 it, so fabricating provenance requires shipping a migration, which is a named 870 reviewable location rather than a constant at a call site. 871 872 WHY, and it is worth keeping. The gate was originally content based, and the 873 Gate 2 fixtures proved that cannot hold: a candidate row with the uuid, email 874 and phone stripped is still one named cleared human with a clearance, a 875 location and an availability, and it crossed; a customer's own performance work 876 statement contains no personal data at all and crossed too. The first is 877 private because of WHAT IT IS, the second because of WHERE IT CAME FROM, and no 878 pattern match on the bytes recovers either fact once a caller has flattened it 879 into a string. A second version took provenance as a CALL TIME argument, which 880 is a control one mistake defeats. Only the stored value survives contact. 881 882 The gate now takes the SET of provenance values read from every row a payload 883 was composed from. ONE TENANT ELEMENT POISONS THE WHOLE PAYLOAD, because a 884 public notice with one paragraph of a customer's PWS folded in is a customer's 885 PWS. AN EMPTY SET IS REFUSED, because silence is not permission. 886 887 A BACKSTOP REMAINS, and its limit is asserted by name rather than assumed away: 888 a caller that MISDECLARES a stripped candidate row is not caught by content 889 scanning, and a test says so explicitly so nobody mistakes the backstop for a 890 guarantee. 891 892 Only four egress classes may leave the machine: 893 PUBLIC_KNOWLEDGE, PUBLIC_DATA_SUMMARY, ONTOLOGY_SUGGEST, REQUIREMENT_NORMALIZE. 894 NEVER external, enforced by the gate and by tests: candidate rows, resume 895 text, names, emails, phone numbers, tenant identifiers, anything from the eeo 896 schema, RouteRAG and RouteGraph over tenant data, scoring inputs, scoring 897 outputs, slate contents. The gate fails closed to the local plane. Every 898 external call audits backend, class, byte count, and content hash, NEVER 899 content. 900 901 CURRENT DATA: 6,676 opportunity_records, 1 tracked_opportunity, 5 manifest 902 rows. 903 904-------------------------------------------------------------------------------- 90510. THE AI LAYER, UNDER SCRUTINY 906-------------------------------------------------------------------------------- 907 908This section is deliberately harder on the AI layer than a sales document would 909be, because the failure mode of an AI feature in a regulated product is not a 910bad answer, it is a confident bad answer that someone acts on. 911 912THE ARCHITECTURE. Routed, not monolithic. One router picks the cheapest 913sufficient strategy, defined in orchestrator.go: 914 915 RouteRefuse unsafe input (EEO, injection) 916 RouteTAG the database must compute (count, sum, aggregate) 917 RouteRAG retrieve the tenant's candidates and narrate 918 RouteDirect answer from stable rules in context (CAG), no retrieval 919 RouteGraph walk a cited graph evidence path 920 RoutePlan bounded plan and execute over closed intents 921 922THE REAL SHAPE OF THE CODE, by size, which tells you where the weight is: 923 graph_route.go 835 planner.go 841 tag.go 798 924 mag.go 642 curation.go 655 orchestrator 524 925 graph_market.go 488 teaching.go 446 graph_teaming 403 926 plus graph_lifecycle, graph_agency, graph_loop, graph_skill, 927 graph_clearance, graph_bench, capture_readiness, knowledge, memory, 928 safety, suppression, vectorguard, egress, feed 929 930WHAT IS GENUINELY STRONG 931 932 RAG, the hybrid retrieval. pgvector cosine kNN fused with tsvector rank by 933 Reciprocal Rank Fusion (k=60), IN SQL. This is the best piece in the layer. 934 It now spans TWO corpora, the tenant's candidates and the public opportunity 935 catalog, and the detector yields to any bench noun so a question touching both 936 planes keeps its candidate corpus rather than being hijacked by the market 937 one. Fusion in SQL rather than in Go means one query plan, one round trip, and 938 no reranking layer to drift. 939 940 THE CLOSED INTENT REGISTRY. 80 curated, parameterised, read only intents in 941 tag.go. The model selects an intent NAME. It never authors SQL, never authors 942 parameters, and never sees a query planner. THIS IS A SECURITY PROPERTY, NOT 943 A MISSING FEATURE, and it is the single most important decision in the layer. 944 Every NL to SQL system is one prompt injection away from an arbitrary read; 945 this one is not, because there is no code path from model output to SQL text. 946 947 THE VERIFIER. A faithfulness gate: model prose ships ONLY if the verifier 948 confirms it against the retrieved evidence, otherwise the user gets the 949 deterministic rendering of the same evidence. It FAILS CLOSED on a parse 950 error. This is what makes the layer safe to put in front of a recruiter: the 951 worst case is a plainer answer, not a wrong one. 952 953 THE POLICY STACK. EEO prohibition on the question itself, read only SQL 954 guard, table allowlist, stacked statement rejection, k anonymity suppression. 955 A federation audit widened the protected noun matching beyond bare terms to 956 close proxies, which is the part most implementations miss. 957 958WHAT IS REAL BUT SHALLOWER THAN IT SOUNDS 959 960 GraphRAG. Recursive CTE traversal, depth capped, cycle guarded, across 20 961 graph files covering agency, bench, clearance, lifecycle, loop, market, route, 962 skill, and teaming. It is real graph traversal over real relational data with 963 real citations. It is NOT a graph database, there is no learned embedding of 964 the graph, and the traversals are hand written per domain. That is a 965 reasonable engineering choice and it does mean "GraphRAG" here means 966 "parameterised recursive SQL with provenance", which is less than the term 967 implies elsewhere. 968 969 PAG, the planner. K=3 self consistency, DAG execution, real data dependency 970 chaining between intents. 841 lines. The plan space is bounded by the same 971 closed intent registry, so it cannot plan its way to something the registry 972 does not already permit. That bound is what makes it safe and also what caps 973 how clever it can be. 974 975 KAG and OAG. 30 cited GovCon knowledge cards plus O*NET, NICE, PSC, and CALC 976 ontologies. Cited is the operative word. But 30 cards is a small corpus and 977 curation is a manual, operator only, two step approve then verify process. It 978 is trustworthy and it is not large. 979 980WHAT IS SCAFFOLDED, OFF, OR HONESTLY ABSENT 981 982 The teaching signal channel. missionMATCH emits contents free craft signals. 983 PANTHEON does not learn from them: the gateway VALIDATES a signal and then 984 DISCARDS it. The working federation path is the knowledge feed harvest, not 985 the teaching channel. Anyone reading "teaching loop" should read "built, 986 validated, and currently a no op at the consumer". 987 988 Federated differential privacy. Scaffolded and OFF. The Gaussian sampler 989 returns ErrNotImplemented. It is a placeholder with a good shape, not a 990 capability. 991 992 The external OpenAI backend. Built, boot guarded (enabled with no secret 993 exits 1, proven against the live gateway), and currently pointed at the LOCAL 994 model. REQUIREMENT_NORMALIZE has therefore never executed against OpenAI end 995 to end. Enabling it is a config change protected by the boot guard, not new 996 code, but the end to end path is UNPROVEN. 997 998 DELIBERATELY ABSENT, and these are decisions, not gaps: 999 Self-RAG needs a fine tuned critic; incompatible with a stateless 1000 commodity plane and the no Python rule. 1001 Web retrieval must not happen with CUI. 1002 NL to SQL replaced by the closed registry, on purpose. 1003 Multimodal, vision no use case in a text domain. 1004 1005THE SCRUTINY THAT MATTERS MOST 1006 1007 1. The layer's safety rests on the verifier and the closed registry, and both 1008 are code paths, not model behaviour. That is the right place to put trust. 1009 2. The EEO prohibition operates on the QUESTION. A question that never names a 1010 protected attribute but correlates with one is not caught by it. The real 1011 defence there is that demographics are in a schema the query role cannot 1012 read at all, which is architectural rather than heuristic. That is the 1013 stronger guarantee and it should be the one cited. 1014 3. Nothing in this layer can write. Every intent is read only and the guard 1015 enforces it. The AI cannot change a score, a stage, or an entitlement. 1016 4. The inference plane is a SEPARATE stateless Go binary on another machine, 1017 bearer authenticated. The FIPS API binary never hosts a model; it speaks 1018 HTTP. So a model process crash cannot take the API down and a model cannot 1019 be reached without the token. 1020 5. UNPROVEN AT SCALE. Every claim above is proven by tests and by small live 1021 checks. No part of this layer has served production traffic at volume, 1022 because production has five users and 11 candidates. 1023 1024-------------------------------------------------------------------------------- 102510A. THE LEARNING BOUNDARY, THREE DOORS AND NO FOURTH 1026-------------------------------------------------------------------------------- 1027 1028A standing rule about what INGESTION is allowed to change, recorded 28 July 2026 1029as the corpus began to grow at scale. 1030 1031INGESTION MAY OPEN EXACTLY THREE DOORS: 1032 1. THE CORPUS new documents, chunks, notices, attachments 1033 2. THE EMBEDDINGS new vectors over that corpus 1034 3. ONTOLOGY SUGGESTIONS candidate skills and adjacencies PROPOSED for curation 1035 1036THERE IS NO FOURTH DOOR. Ingestion never alters scoring weights, never alters the 1037band thresholds, never alters the clearance hierarchy, and never promotes an 1038ontology suggestion into the live ontology by itself. 1039 1040WHY IT IS WRITTEN DOWN. The scoring engine is an AEDT under NYC LL144 and OFCCP. 1041Its weights are frozen, diffed, and exhaustively tested precisely so a customer 1042can point at one number and say what produced it. A pipeline that learned from the 1043market and adjusted those weights would produce a scoring function nobody could 1044reproduce, explain, or audit, and it would do it silently, one federal document at 1045a time. The failure would not look like a bug. It would look like the numbers 1046slowly changing. 1047 1048DOOR 3 IS THE ONE THAT LOOKS SAFEST AND IS NOT. An ontology that grows itself 1049changes which candidate skills match which required skills, which changes the 1050Skills component, which changes the score. So a SUGGESTION is data, a PROMOTION is 1051an operator action by a named human, and the gap between them is load bearing. 1052cmd/promote-ontology exists for exactly that gap. 1053 1054HOW IT IS ENFORCED TODAY, honestly: by the frozen engine diff against e74fd2e 1055asserted EMPTY at every drift check, by ontology promotion being a separate 1056operator binary, and by nothing else. It is a rule with a DETECTOR, not a rule 1057with a LOCK, and it is named that way so nobody mistakes one for the other. 1058 1059-------------------------------------------------------------------------------- 106011. FEDERATION RELATIONSHIP TO PANTHEON 1061-------------------------------------------------------------------------------- 1062 1063ONE WAY, by design. missionMATCH --publishes--> PANTHEON 1064 1065missionMATCH is a standalone product, NOT an organ of the federation. It is not 1066in ORACLE's registry, nothing routes tasks into it, and it has no knowledge that 1067any consumer exists. 1068 1069 GET /knowledge/manifest declares federation_member:false, routable:false, 1070 as DATA not prose 1071 GET /knowledge/cards the whole curated public set in one pull 1072 1073Both sit OUTSIDE /api, because the public vhost proxies only /api/ and /healthz. 1074They are reachable across the tailnet, not from the internet, and a bulk pull 1075does not spend the API rate limit real traffic depends on. 1076 1077PANTHEON harvests hourly through curator-db.sh, content addressed and 1078idempotent, alongside codex-lore and seferim-events. 1079 1080WHY PUBLISH RATHER THAN JOIN: a registered organ is DELEGATED to, and the 1081orchestrator needs it reachable every time. A harvested feed is ABSORBED, and 1082the consumer knows the material afterwards with missionMATCH offline or gone. 1083Learning is what survives the source going away. 1084 1085-------------------------------------------------------------------------------- 108612. FILE ARCHITECTURE 1087-------------------------------------------------------------------------------- 1088 1089missionmatch/ 1090| 1091+-- cmd/ 36 binaries 1092| +-- api/ THE SERVER. Config, FIPS guard, pools, 1093| | modules, routes. ~720 lines, all wiring. 1094| +-- megamind-gateway/ stateless inference proxy (runs on M4) 1095| +-- provision-org/ create an org and its first owner 1096| +-- provision-tenant/ create a NEW empty tenant database 1097| +-- register-tenant/ adopt an EXISTING database into the directory 1098| +-- migrate-all-tenants/ fan out migrations across control plane and 1099| | every active tenant (see Section 14) 1100| +-- sync-tenant-identity/ copy org and users into a tenant DB 1101| +-- sam-sync, opp-sync, awards-sync, rates-sync, oews-national-sync, 1102| | ontology-sync, missionclass-sync, icims-sync external ingestion 1103| +-- opportunity-scope-normalize model normalization into cited drafts 1104| +-- aioe-run, aioe-sim, aioe-backfill adaptive weight optimization 1105| +-- train-model, eval-model, eval learned model lifecycle 1106| +-- teach-aggregate contents free craft signals 1107| +-- fairness-snapshot adverse impact monitoring 1108| +-- embed-backfill, verify-vector-model, verify-resolver 1109| +-- alert-sweep, staffing-proactive-sync, demo-seed, brain-probe, 1110| import-candidates, import-bench, promote-ontology, tag-missions 1111| 1112+-- internal/platform/ 31 cross cutting packages 1113| +-- config/ ALL configuration and validation. Cloud choice 1114| | is CONFIG, never code. 1115| +-- database/ pool construction, pgvector codecs per 1116| | connection, role assertion failing BOOT closed 1117| +-- tenant/ THE CORRECTNESS BOUNDARY. RunInTenantTx, the 1118| | org GUC, the Router interface, Principal 1119| +-- tenantpool/ bounded per tenant registry: LRU evict, 1120| | singleflight, ping before cache, refcount pin, 1121| | typed errors that never fall back 1122| +-- directory/ tenant_directory access 1123| +-- identity/ seed org and users into a tenant DB 1124| +-- secrets/ Store (named) and Resolver (per tenant refs, 1125| | TTL cached, env backend MM_TENANT_DSN_<REF>) 1126| +-- security/ rbac, capabilities, tools, CSRF, password 1127| | hashing, TOTP, session tokens, RequirePlatform 1128| +-- audit/ append only trail, System() and OnTx() 1129| +-- inference/ typed client, circuit breaker, bounded retries 1130| +-- scrub/ egress payload classes, enforced twice 1131| +-- csvsafe/ formula injection escape 1132| +-- federated/ differential privacy scaffolding (OFF) 1133| +-- embeddings/, magstore/, learn/, eval/, aioe/, docextract/, extraction/ 1134| +-- samgov/, samawards/, bls/, marketintel/, forecast/, connector/ 1135| +-- ratelimit/, obs/, storage/, ontology/ 1136| 1137+-- internal/modules/ 24 domain modules 1138| | each: handler.go, service.go, dto.go, repo.go 1139| +-- auth/ login, MFA, sessions, invites, users, roles, 1140| | tool entitlement, self service profile 1141| +-- matching/ THE ENGINE plus screen and slate cores 1142| | +-- engine/ pure deterministic scoring, decimal, FROZEN 1143| +-- screenmatch/ slatematch/ missionmatch/ recruitmatch/ 1144| +-- recruiting/ THE DESK, 40 routes, the state machines 1145| +-- opportunities/ public cache PLUS the tenant capture layer 1146| +-- commandcenter/ mission readiness aggregation 1147| +-- ai/ orchestrator, retrieval, planner, 20 graph 1148| | files, memory, knowledge, safety, suppression, 1149| | vectorguard, curation, teaching, egress, feed 1150| +-- eeo/ adverse impact, SEGREGATED 1151| +-- governance/ retention, AEDT inventory, model registry 1152| +-- aioe/ adaptive weights 1153| +-- opportunity_intelligence/, awards/, wages/, sourcing/, staffing/, 1154| alerts/, bidoutcomes/, ontology/, termalias/, org/, federalsignal/ 1155| 1156+-- db/ 1157| +-- migrations/ 79 files, head 0080. The ONLY way schema 1158| | changes. 0020 is absent from the sequence; 1159| | numbers are never reused. 1160| +-- migrations_eeo/ SEPARATE set, separate version table, so sqlc 1161| | never models the eeo schema 1162| +-- bootstrap/ cluster roles 1163| +-- queries/ sqlc source SQL 1164| 1165+-- frontend/src/ 87 files, 13,217 lines 1166| +-- App.tsx routes + ToolGuard entitlement redirect 1167| +-- main.tsx ErrorBoundary > QueryClient > Router 1168| +-- components/ AppShell (tool switcher, nav, signed in 1169| | identity), ui, ErrorBoundary, ExecutiveCards 1170| +-- lib/ api, auth, AuthContext, can (capabilities, 1171| | tools, the two disabled reasons), homeTool, 1172| | queries, types 1173| +-- screens/ Login, AcceptInvite, Admin, Settings, 1174| | Compliance, Requirements, TimeZonePanel, 1175| | NotFound 1176| +-- tools/ 22 views across the four products (Section 7) 1177| 1178+-- docs/ CLAUDE_CODE_BUILD_DIRECTIVE (the work order), 1179| DIRECTIVE_AMENDMENTS_CONSOLIDATED, 1180| DIRECTIVE_AMENDMENT_6, BUILD_STATE (the append 1181| only ledger), AI_RESEARCH, STACK_DECISIONS, 1182| MODEL_CARD, runbooks 1183+-- ops/m4/ gateway lifecycle 1184+-- scripts/deploy.sh encodes three SILENT failure modes 1185 1186-------------------------------------------------------------------------------- 118713. HOW IT IS BUILT 1188-------------------------------------------------------------------------------- 1189 1190NON NEGOTIABLE RULES (CLAUDE.md): 1191 1. No Python anywhere. Go backend, TypeScript frontend, bash scripts. HTML 1192 written by hand, never generated by a program. 1193 2. No dashes in written output. Hyphens only inside URLs, HTTP header names, 1194 CLI flags, and code identifiers. 1195 3. Every API key stays on the server. Root owned 0600, never in the repo, 1196 never in the browser bundle. 1197 4. Clearance is "clearance intelligence", never "verified". 1198 1199THE BUILD PROGRAM. Work runs under docs/CLAUDE_CODE_BUILD_DIRECTIVE.md, phase 1200locked: work only on the lowest incomplete phase, never start phase N+1 until 1201every item in the phase N gate is verified and recorded in docs/BUILD_STATE.md. 1202That ledger takes only VERIFIED items; aspirations never enter. Phase 0 and 1203Phase 1 complete with gates passed, Phase 2 current, Phases 3 through 6 not 1204started. TRACK U (identity and access) ran in parallel and is complete. Tasks T1 1205and T2 are standalone client site work outside the phase gates. 1206 1207FROZEN AND UNTOUCHABLE: internal/modules/matching/engine, db/migrations_eeo, 1208every existing migration file, generated sqlc output, append only audit 1209semantics. 1210 1211ENGINEERING INVARIANTS: 1212 organization_id, forced RLS, and composite indexes leading with 1213 organization_id, on every tenant table. 1214 ON DELETE SET NULL on a composite FK MUST use the column list form. The bare 1215 form tries to null organization_id, which is NOT NULL, making the 1216 referenced row undeletable. Migration 0053 repaired 24 of these. 1217 audit_logs is append only. UPDATE and DELETE revoked, triggers enforce it. 1218 A down migration may drop what its up added but may NOT delete rows the 1219 application authored. Several downs REFUSE with a count. 1220 Widen a CHECK from the CURRENT constraint, never from an old migration's 1221 vocabulary, or a later migration's additions are silently dropped. Caught 1222 twice. 1223 Role enum values are EXTENDED, never renamed, never removed. 1224 1225VERIFICATION METHOD: 1226 Reproduce the defect against a live database FIRST, fix, verify against the 1227 same database, verify end to end through the live AUTHENTICATED API, then 1228 look at the rendered screen. 1229 1230 Concurrency proven by RACING it: 16 concurrent stage transitions elected one 1231 winner; 8 racing claims one owner; 8 placements into one seat one placement; 1232 8 concurrent promotions of one draft one requirement. 1233 1234 Artifacts are GREPPED, not trusted. A bundle is checked for the strings it 1235 must and must not contain, because a source tree that looks right can still 1236 build the wrong thing. This caught a Phase 1 navigation link riding into a 1237 production bundle on 28 July. 1238 1239 A NEGATIVE RESULT IS ONLY EVIDENCE WHEN THE INSTRUMENT IS KNOWN TO WORK. 1240 Added 28 July after being burned twice in one day (Section 16). A check that 1241 can pass by doing nothing needs a positive control. 1242 1243THE MERGE GATE: make build-test-gates runs gofmt, go vet, the full unit suite, 1244the security gates, and a clean frontend production build. Green at this 1245revision. 1246 1247-------------------------------------------------------------------------------- 124814. DEPLOYMENT AND ITS REAL FAILURE MODES 1249-------------------------------------------------------------------------------- 1250 1251Every item below is something that has actually gone wrong. 1252 1253 BUILD CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 1254 go build -trimpath 1255 1256 API DEPLOY Stage to /tmp, COMPARE SHA256, stop, swap, start, poll healthz. 1257 BOTH nodes must get the same binary or the failover serves two 1258 different answers. On 28 July the second node briefly ran a 1259 binary with no operator role while the first had it, which 1260 would have given the operator zero capabilities on one node and 1261 everything on the other. The correct response is to mark the 1262 stale node DOWN in the nginx upstream and reload BEFORE 1263 shipping anything further. 1264 1265 SECOND NODE megamind-agi has no direct SSH and refuses publickey. Use 1266 PASSWORD authentication to the WINDOWS host 1267 josep@100.69.120.61, scp there, then run a STAGED BASH SCRIPT 1268 through wsl. Multi line quoting through wsl breaks; stage a 1269 file rather than inlining. WSL sudo is NOPASSWD. 1270 1271 MIGRATIONS Use cmd/migrate-all-tenants, NOT the goose CLI (which pulls 1272 heavy unrelated drivers absent from go.sum). It EMBEDS 1273 migrations with go:embed, so an edited .sql does NOTHING until 1274 the migrator binary is REBUILT. 1275 1276 CRITICAL DEFECT, found AND FIXED 28 July 2026, recorded because 1277 the failure mode is worth knowing. The tool built each tenant 1278 DSN with swapDB(controlDSN, dbName), swapping only the DATABASE 1279 NAME and IGNORING the db_host and db_port that tenant_directory 1280 exists to carry. A tenant on another host was never reached: it 1281 connected to a database of the same name on the CONTROL PLANE 1282 host, migrated it a second time, and printed APPLIED for both. 1283 Every run reported success while the production control plane 1284 fell TWELVE migrations behind the tenant database. 1285 1286 The fix builds the target from the directory row, host and port 1287 and name together. MM_TENANT_MIGRATOR_DSN_<REF> overrides for a 1288 tenant whose credential differs, folded by the same rule 1289 internal/platform/secrets uses. A row with no host is an ERROR, 1290 never a guess, and a target that cannot be addressed is 1291 reported FAILED and carried into the exit code rather than 1292 dropped from the summary. Five tests pin it, including one 1293 using an identical database name on both hosts, which is the 1294 shape that made the original bug invisible. 1295 1296 THE GENERAL LESSON: a fan out tool that cannot reach a target 1297 must FAIL, never silently succeed against something else. 1298 Verify the version in EVERY database regardless of the summary. 1299 1300 FRONTEND npm run build, ship dist/assets, then rewrite ONLY the two 1301 asset hashes in the live index.html. Overwriting it destroys 1302 injected SEO lines. Verify the line count is unchanged. 1303 1304 ENV FILES sudo tee TRUNCATES. Stage to /tmp then cp, and assert the 1305 result is non empty and not smaller before replacing. 1306 1307 POST DEPLOY Re prove the build feature routes DARK and run a zero 5xx 1308 sweep, sampled enough times that both pool members answer. 1309 AND INCLUDE AN AUTHENTICATED LEG: a sweep of unauthenticated 1310 paths cannot see a defect behind the session gate, which is 1311 exactly what hid a 500 on 28 July. 1312 1313 PRODUCTION BINARIES ARE NOT REPOSITORY HEAD. HEAD carries Phase 1 and 2 build 1314 features that must stay dark until Gate 6. The production binary is built in a 1315 worktree at the pre build baseline e74fd2e with only the approved track's 1316 files overlaid and cmd/api/main.go hand resolved. Any deploy before Gate 6 1317 must be built the same way and the artifacts grepped to prove it. 1318 1319-------------------------------------------------------------------------------- 132015. OBSERVABILITY 1321-------------------------------------------------------------------------------- 1322 1323 LOGGING Structured slog, JSON, request id, method, route, status, latency, 1324 bytes on every request. Debug in development, info in production. 1325 PII is never logged. 1326 METRICS /metrics outside /api so a scrape never spends the API rate limit. 1327 Feeds database pool gauges so connection saturation is visible. 1328 Unauthenticated by design, firewalled to the scraping network. 1329 HEALTH /healthz per node. nginx passive health checks handle failover. 1330 AUDIT audit_logs is the operational record as well as the compliance one: 1331 logins, failures, lockouts with source spread, permission changes, 1332 identity changes, exports, every material write through OnTx. 1333 ALERTS The alerts module plus alert-sweep. 1334 SCHEDULED mm-sam-sync.timer runs daily SAM.gov ingestion on Bubbles. 1335 1336 WHAT RUNS ON A TIMER, all installed 28 July 2026 and each proven by an 1337 induced failure where one was possible: 1338 02:45 megamind-agi mm-cp-backup control plane dump, streamed through 1339 gpg into ssh, pushed to Bubbles 1340 03:05 Bubbles mm-db-backup tenant and staging dumps, ingest of 1341 the pushed artifact only after it is 1342 proven to decrypt, manifest, prune to 1343 14 daily and 8 weekly 1344 every 5 min Bubbles mm-watch public site, BOTH API nodes named 1345 individually so a single node failure 1346 is not masked by the pool, and 5xx 1347 since the last run. Alerts on a 1348 TRANSITION, once when a problem starts 1349 and once when it clears. 1350 hourly megamind-agi mm-identity-sync control plane to every active tenant, 1351 reconciled after each run rather than 1352 trusting the exit code 1353 Delivery is mm-alert.sh through Gmail SMTP, reusing the app password the 1354 form handlers already use. The systemd OnFailure notifier routes through it. 1355 1356 A TIMER THAT IS NOT ON THAT LIST AND FAILS EVERY DAY, found 28 July 2026 at 1357 the Phase 2 drift check and recorded as finding D1: 1358 05:16 Bubbles mm-fairness-snapshot FAILING SINCE INSTALLATION 1359 It exits 1 with "set MM_COMPLIANCE_DATABASE_URL (the compliance role DSN, the 1360 only path to the eeo schema)". The unit reads /etc/missionmatch/api.env and 1361 that file does not carry the variable. Its sibling on the control plane node, 1362 /etc/missionmatch/db.env, does, and that DSN authenticates, so the repair is 1363 configuration and not code. 1364 1365 WHY THIS ONE MATTERS MORE THAN A FAILED TIMER USUALLY DOES. The adverse impact 1366 snapshot is the AEDT compliance artifact this product exists to provide: the 1367 selection rates and four fifths ratios a customer needs for an LL144 or OFCCP 1368 audit. The claim that missionMATCH supplies it is currently true of the code 1369 and FALSE of the running system. 1370 1371 ALERTING COVERAGE IS PARTIAL, which is why the above survived, recorded as 1372 finding D2. Exactly TWO units on Bubbles carry OnFailure, bubbles-cert-check 1373 and mm-db-backup, both wired during Track H item 3. That work fixed the 1374 alerting PATH and proved it by breaking things; it did not sweep every 1375 existing unit onto that path. A failing unit outside those two is silent. This 1376 is the same shape as the decorative alerting already found once: the mechanism 1377 works, the coverage does not, and partial coverage is harder to notice than a 1378 mechanism that is plainly broken. 1379 1380 Both are production changes and both are QUEUED rather than fixed, because the 1381 Phase 2 lock holds. One deploy closes both. 1382 1383 REMAINING GAPS: no metrics scraper and no dashboard. That is a deliberate 1384 ordering, not an oversight: a phone that buzzes was worth more than a graph 1385 nobody is watching at three in the morning. 1386 1387-------------------------------------------------------------------------------- 138816. THE 28 JULY 2026 INCIDENT RECORD 1389-------------------------------------------------------------------------------- 1390 1391A full account, because the corrections are more instructive than the features. 1392 1393INCIDENT 1: THE STAGING ROLE COULD NOT AUTHENTICATE AT ALL 1394 FOUND Provisioning a Track U repro account. Invite acceptance returned 500. 1395 CAUSE The isolated role missionmatch_staging_app was created with TABLE 1396 grants mirrored and FUNCTION grants not. It held EXECUTE on ZERO of 1397 the nineteen SECURITY DEFINER auth functions. 1398 IMPACT No login, no invite acceptance, no session validation on staging 1399 from the moment the role was introduced. It also means an earlier 1400 ledger claim that Track U items were "proven on staging" could not 1401 have been an end to end check, because no authenticated request 1402 could complete. That claim was withdrawn. 1403 FIX The nineteen missing grants mirrored into missionmatch_staging only, 1404 generated by comparing the two roles rather than hand listed. 1405 VERIFIED 0 of 188 functions still missing. The staging role is STILL refused 1406 on missionmatch, tenant_alpha, and tenant_beta. Production grants 1407 unchanged at 151 of 152. Production site 200. 1408 1409INCIDENT 2: THE BY RULE TOOL FIX NEVER REACHED THE BROWSER 1410 CAUSE SessionInfo re read the RAW rows of user_tool_access instead of the 1411 principal's already widened set. 1412 IMPACT A tenant superuser with no grant rows was widened for every server 1413 side check and still received tools:[], which hides the entire 1414 navigation and locks every entitlement checkbox. The reported 1415 symptom surviving its own fix, one layer up. 1416 FIX SessionInfo returns the principal's tools. 1417 1418INCIDENT 3: PATCH /me/profile ANSWERED 404 ON THE CALLER'S OWN ACCOUNT 1419 CAUSE writeProfile, getProfile, and roleOf ran on a bare pool connection 1420 with no app.current_org_id. users carries forced RLS, so every 1421 statement matched zero rows. 1422 LESSON The explicit "WHERE organization_id = $1" predicate LOOKS like it 1423 does the scoping and does not. RLS filters first. 1424 FIX All three run inside RunInTenantTx. The same class of bug was 1425 introduced in AcceptInvite the same day and fixed the same way. 1426 1427INCIDENT 4: A PHASE 1 NAVIGATION LINK NEARLY SHIPPED TO PRODUCTION 1428 CAUSE Copying HEAD's AppShell.tsx wholesale into the production build 1429 dragged a /slate/capture link with it. 1430 CAUGHT BY Grepping the BUILT BUNDLE, not the source tree. 1431 FIX That file rebuilt from the baseline with only the Track U change. 1432 1433INCIDENT 5: THE PRODUCTION TOPOLOGY WAS RECORDED WRONG, AND IT BROKE PRODUCTION 1434 CAUSE The ledger described production as missionmatch on M4 5432. That is 1435 the TENANT data plane. The CONTROL PLANE is missionmatch on 1436 megamind-agi 5433, and it was at v68, twelve migrations behind. 1437 IMPACT The Track U binary selects title and phone from users; those columns 1438 were added by 0079, applied to the tenant database. On the control 1439 plane they did not exist, so GET /api/auth/session WOULD have 1440 returned 500 to the first person to sign in. 1441 SCOPE Stated precisely: the journal shows ZERO 5xx and ZERO ERROR lines 1442 across the entire window between deploy and fix. Nobody logged in 1443 while it was broken. The defect was real and had no victim. 1444 WHY IT WAS MISSED The post deploy zero 5xx sweep used only unauthenticated 1445 paths, which answer 403 before the failing handler runs. A sweep 1446 with no session cannot see a defect behind the session gate. 1447 FIX Control plane snapshotted and migrated v68 to v80. 1448 VERIFIED title and phone present, remap audit trail written on the 1449 authoritative database with all three superuser grants named, the 1450 promoted account's sessions revoked, login returns 401 for a wrong 1451 password rather than 500, zero 5xx across 21 requests. 1452 ROOT CAUSE The migrate-all-tenants swapDB defect (Section 14). FIXED, with 1453 five tests, and proven against the real two host topology: the fixed 1454 tool now targets the tenant's own host and, when credentials did not 1455 match, FAILED LOUDLY instead of quietly succeeding elsewhere. 1456 DRIFT NOW CLOSED. Control plane v80, tenant v80, staging v80, each verified 1457 by querying that database directly rather than trusting the summary. 1458 1459INCIDENT 6: TWO OF MY OWN FINDINGS WERE WRONG 1460 6a I reported Patricia Cox could not sign in, with no password credential and 1461 no live invite, and built a Resend invitation button on that basis. I had 1462 read the TENANT copy of users, which is non authoritative and was stale. 1463 On the control plane she is ACTIVE with a password and four tool grants; 1464 she has simply never logged in. Production has FIVE accounts, not three. 1465 The button is still correct work (amendment 6 lists resend invitations and 1466 nothing called the endpoint), but the urgency was based on a bad read. 1467 6b I reported MM_TENANT_DSN_TENANT_ACME missing from production and put it in 1468 the risk register as a probable breakage. It is defined on line 23 of 1469 api.env and present in the running process. The claim came from a grep run 1470 through two layers of shell quoting that silently matched nothing. 1471 WITHDRAWN, and the data plane is now PROVEN working instead: connecting 1472 with the exact DSN the running process holds reaches the tenant database 1473 as missionmatch_app, that role is NOBYPASSRLS, a read of candidates with 1474 no org GUC returns 0 rows, and the same read with the GUC set returns 11. 1475 1476INCIDENT 7: THE LOGIN INSTRUMENT LIED FOR WEEKS, and a design partner caught it 1477 REPORTED She said she had logged in and the control plane showed "never 1478 signed in". 1479 MEASURED She was right, and it was worse than one account. FOUR of five 1480 accounts held real ACTIVE sessions while the screen said none had 1481 ever signed in, and the fifth was STALE BY A WEEK. 1482 CAUSE internal/modules/auth/service.go read 1483 _, _ = db.Exec(ctx, `SELECT auth_touch_last_login(, )`, ...) 1484 The $1 and $2 PLACEHOLDERS WERE MISSING FROM THE SOURCE, almost 1485 certainly stripped by an edit made through a shell that expanded the 1486 dollar signs away. Postgres rejected it as a syntax error on EVERY 1487 sign in, and both return values were discarded, so it failed 1488 silently. This is the no scripts rule demonstrated in production. 1489 RULED OUT by direct test, not by reading: the function works when called as a 1490 superuser, the app role holds EXECUTE, and RLS is not the cause 1491 because the function is SECURITY DEFINER owned by a BYPASSRLS role. 1492 ONE OF MY OWN TESTS WAS INVALID and is recorded as such: subselects 1493 run as the app role returned NULL under RLS, so the function's NULL 1494 guard returned early and it looked like a failure that was not one. 1495 FIX Placeholders restored, and the failure is now LOGGED at ERROR 1496 including the zero rows case, which Postgres does not treat as an 1497 error and which is exactly what a silent no op looks like. It still 1498 does not fail the login, because recording a sign in is bookkeeping 1499 and denying entry over bookkeeping would turn a reporting defect into 1500 an outage. Swallowed deliberately is different from swallowed