Janady13/missionmatch / BUILD_REPORT.md

1# BUILD_REPORT.md — AI brain activation 2 3**Date:** 2026-06-29. **Branch:** `ai-brain-live` (off `w3-ai-staffing-recommendations`, which sits on 4the unmerged AI intelligence layer). **Build host:** M2. **Validated on:** M4 staging (`brain_staging`, 5goose v18, bge-small embeddings, the live MEGAMIND plane). 6 7This mission made the AI brain LIVE: the brain library in `internal/modules/ai` (orchestrator, TAG, RAG 8retrieval, memory, KAG SkillAdjacency, CAG rules) was built and tested but unreachable from the running 9app, and the vector layer was dead in the live path. It is now wired: the real embedder, backfilled 10vectors, the HNSW indexed skill matcher, the brain mounted as an authenticated tenant scoped route behind 11the faithfulness gate, the two scoring paths reconciled, proven end to end on staging. 12 13**Done means, and is proven:** a person can ask the brain a question through an authenticated route and get 14an explainable, faithfulness gated, tenant isolated answer. `TestBrainAnswersEndToEnd` exercises exactly 15that (RAG, TAG, CAG, and an EEO refusal, with the AEDT audits), and passes on M4 staging. 16 17## Pre cutover hardening (four fixes, 2026-06-29) 18 19A close review before cutover found four things to fix. All four are now closed and re validated on M4 20staging (`brain_staging`, goose v19). No feature was added; these are correctness and deploy safety. 21 22**FIX 1, the semantic threshold (the live correctness defect).** The engine's `DefaultSemanticThreshold` 23was 0.80, tuned for a different embedder. On the production bge-small vectors a clear synonym scores ~0.74, 24so true semantic matches fell under the live cutoff and were silently discarded: the brain shipped a 25semantic matcher the engine then ignored. A calibration harness on staging 26(`TestSemanticThresholdCalibration`) scored known synonym and adjacent skill pairs against known unrelated 27pairs on the stored bge vectors: 28 29- Genuine synonyms (11 of 12 pairs): cosine 0.678 to 0.944 (for example "Penetration Testing" to "Ethical 30 Hacking" 0.678, "Threat Hunting" to "Threat Detection" 0.875, "Network Engineering" to "Network 31 Infrastructure Engineering" 0.944). 32- Unrelated pairs (all 12): cosine 0.410 to 0.571 (mean 0.498), the highest being a noise spike "Amazon Web 33 Services" to "Underwater Basket Weaving" 0.571. 34- The genuine groups separate in the gap [0.571, 0.678]. One acronym pair, "Signals Intelligence" to 35 "SIGINT" (0.534), sits BELOW the unrelated ceiling, so no safe single threshold can capture it; acronym to 36 expansion belongs to the skills ontology alias layer, not to a cosine cutoff. It is documented, not forced. 37 38Chosen threshold: **0.62**, in the separating gap with margin: 0.0486 above the unrelated ceiling (no 39invented match) and below every genuine synonym (no discarded match). At 0.62 the harness confirms 11 of 12 40synonyms matched and 12 of 12 unrelated rejected. The threshold is a frozen engine contract, so one commit 41changed the constant, bumped the engine **ModelVersion to v1.1.0**, and updated the frozen test together; 42**migration 0019** registers v1.1.0 in `aedt_model_versions` (migrator only, since the app role has SELECT) 43so every decision stamped v1.1.0 has a registry row and a model card entry, with weights and band cuts 44unchanged from v1.0.0. `TestBrainSemanticSkillMatchEnginePath` proves the new threshold FIRES in the engine 45scoring path (not a read only compare): a candidate holding a synonym of a required skill scores Skills 100 46through `engine.ScoreCandidate` over stored bge vectors, while an unrelated skill scores 0 with the required 47skill reported as a gap. The golden `TestSemanticBelowThresholdDoesNotMatch` moved its below threshold 48fixture from 0.79 to 0.55 (a deliberate shift, since 0.79 now matches). 49 50**FIX 2, the enforced re embed gate.** Production holds all-minilm vectors; bge and all-minilm are not 51comparable, so serving over a wrong model store silently mis ranks every retrieval with no error. A new boot 52self check (`ai.VerifyVectorModel`, behind `ai.VectorGate`) asserts the skills and candidates vector columns 53are vector(384) through the catalog (no RLS bypass, no PII), samples stored skill vectors, and re embeds 54them with the configured model to confirm the stored vectors were produced by that model (cosine within 550.98). It re embeds the SAME text the backfill wrote (skill name with category, via a shared 56`embeddings.SkillEmbedText`), so a categorized skill is not a false mismatch. On a proven mismatch, 57`POST /ai/query` returns a typed 503 `vector_model_mismatch` and the mismatch is logged, while the rest of 58the API serves normally; a boot time plane outage leaves the gate to self heal on the first query after the 59plane recovers. The comparison core and the gate state machine are unit tested for match, model swap, wrong 60width, permanent mismatch, transient retry, and the handler 503; a live staging test proves the gate OPENS 61on the bge store. DEPLOY_PLAN.md B2 is now a hard psql assertion (it RAISES unless every live candidate and 62every skill is embedded and a sampled vector is 384 wide) and states the gate in words: the brain stays dark 63until both the re embed is verified complete and the startup check passes. 64 65**FIX 3, the merge sequence.** DEPLOY_PLAN.md now enumerates the three branch merges as discrete, 66individually rollback able steps: M1 merge the AI intelligence layer (migration 0017, ai_memory, the 67foundation), M2 then merge W3 (0018, the two staffing tables), M3 then merge the brain (0019, the v1.1.0 68registry row). Each has its own rollback (revert the merge commit, and the goose down already in the 69migration). The hard rule is stated: do not collapse the three, and do not run the cutover until all three 70are merged in order and the migrations through 0019 are applied to production. 71 72**FIX 4, the forecast test is pre existing (confirmed, not a regression).** 73`TestCommandCenterUpcomingIncumbentAdopt` requires a populated global forecast and contract award catalog 74(it does not seed them; it expects them as global reference data). On a fresh database with an empty 75catalog it fails by construction. Confirmed by a controlled experiment: 76 77- On the **ai-intelligence-layer** branch (neither W3 nor the brain present, head migration 0017), against a 78 CLEAN M4 database (`fix4_ail_baseline`, goose v17, empty catalog), it FAILS at 79 `command_center_d_integration_test.go:43` ("expected forecast records in upcoming opportunities"). 80- On the **ai-brain-live** branch (full stack, migrations to 0019), against a CLEAN M4 database 81 (`fix4_brain_baseline`, goose v19, empty catalog), it FAILS at the SAME assertion, line 43. 82 83Identical failure on the foundation branch alone proves the cause is the empty reference catalog 84(environmental), NOT a regression introduced by W3 or the brain. The "passes on the AI layer, fails only 85with W3 or the brain" regression condition is NOT met. Both scratch databases were dropped after the run. 86 87**Re validation (all on M4 staging, nothing against production).** FIPS production build clean with the 88stamp present (`GOFIPS140=v1.0.0`), `sqlc generate` no drift, goose to v19. Full unit suite green under race 89and fips140=on; `go vet` and `gofmt` clean; `make ci-gates` green (backdoor and FIPS). Full integration 90suite on `brain_staging`: **49 pass, 1 skip, 1 fail**. The skip is `TestCrossTenantDatabaseIsolation` (per 91tenant pools, out of scope). The one fail is the same environmental `TestCommandCenterUpcomingIncumbentAdopt` 92(on `brain_staging` it now clears line 43 because two forecasts exist there, and fails one assertion later at 93line 62 on the empty contract award feed, the same missing reference data cause). Every mission critical 94brain test passes: the threshold fires in the engine path and the frozen test matches, the startup vector 95model check serves on a match, the brain answers end to end with the EEO refusal and audits, the 96reconciliation proves engine equals tag equals dashboard, and cross tenant memory isolation and erasure 97hold. 98 99## Scope held 100- Mounted LIVE on the SHARED RLS database, exactly like matching and commandcenter. Per tenant database 101 routing is a sequenced follow on and was NOT done; the live API was not switched to per tenant pools. 102- No branch merged, no production database or Bubbles host touched. Validated on M4 staging only. 103 104## Non negotiable constraints, all held 1051. **No Python, FIPS intact.** All ML runs out of process on the MEGAMIND plane over HTTP. The API binary 106 stays `CGO_ENABLED=0 GOFIPS140=v1.0.0`; the stamp was verified after every build. No cgo, no new 107 dependency. 1082. **EEO isolation absolute.** No brain path names the eeo schema; the TAG layer runs ONLY curated, 109 parameterized, read only intents over an allowlist that excludes eeo (never free form SQL); 110 `safety.ForbidEEO` guards every string sent to the plane; and a new natural language guard 111 (`MentionsProtectedAttribute`) refuses a question that asks to group people by a protected attribute 112 even in plain words ("list candidates by race"). The EEO unreachability tripwire walks the ai package 113 (21 ai and staffing files scanned, none reach eeo). 1143. **Tenant isolation.** Every brain database read (HybridSearch, ExecuteIntent, memory Recall and Store, 115 the audit) runs inside `RunInTenantTx`, RLS scoped, on the validated org. The skill matcher reads only 116 the GLOBAL skills table (no org, no RLS). MEGAMIND stays stateless. `TestBrainMemoryCrossTenantIsolation` 117 proves one tenant cannot recall another's memory. 1184. **The deterministic engine is the floor.** The brain never recomputes or overrides `engine.ScoreCandidate`; 119 the TAG intents read the frozen `candidate_evaluations` and `slate_members`, not the Command Center 120 composite. No model is in any scoring or eligibility decision; the model only embeds a query, generates 121 prose, and checks faithfulness. `TestBrainTAGReconciliation` proves the brain's answer equals the engine 122 equals the dashboard read. 1235. **Faithfulness gate plus deterministic fallback on every answer.** The model prose ships only when the 124 gate confirms it; otherwise the brain returns the deterministic rendering of the same evidence (a 200, 125 rationale_source deterministic), recording why. A plane outage degrades through the circuit breaker to a 126 typed 503, never a hang. 1276. **Every brain action audits** (`aedt.ai.answered`, `aedt.ai.refused`), carrying no PII, no names, no 128 prose, no question text, no fallback reason. 129 130## Research (primary sources, cited in code) 131A 3 agent research workflow verified each piece before wiring: 132- **pgvector 0.8 HNSW and RRF**: the repo is textbook correct (the cosine operator with vector_cosine_ops, 133 RRF at k=60, SET LOCAL ef_search and iterative_scan in the tenant tx). One tightening applied: the 134 vector arm of `SearchCandidatesHybrid` is now `AS MATERIALIZED` so the relaxed_order HNSW output is 135 re-sorted by true distance before the LIMIT, preventing nearest neighbor mis ranking under the RLS post 136 index filter at scale (pgvector 0.8 README; AWS pgvector 0.8 Aurora guidance). The critical rule, the 137 SAME model and normalization for stored and query vectors, drove the embedder decision below. 138- **bge-small-en-v1.5 on Ollama**: it is NOT in the official Ollama library; `qllama/bge-small-en-v1.5` is 139 the simplest stable 384 dim handle over the standard /api/embed (pulled and probed on M4). The optional 140 query instruction prefix is query only and genuinely optional (BAAI: omitting it is a slight degradation 141 only), so the adopted policy is NO prefix on either path, which keeps stored and query text consistent. 142- **NL2SQL safety and faithfulness**: a curated intent registry beats sanitizing generated SQL; the gate 143 withholds unconfirmed prose and falls back to a deterministic answer while still returning a useful 200 144 (CRAG, Self RAG family). 145 146## What was activated, by phase 1471. **The real embedder.** bge-small-en-v1.5 (the `qllama` handle, 384 dim) is the configured default for the 148 MEGAMIND plane; the gateway forwards the request model, so the API and the backfill name it. The fake 149 deterministic embedder is GONE from the API binary (the embedder was removed from main.go and from the 150 staffing cron) and remains only on the deterministic test path (`EmbeddingSkillMatcher`). 1512. **The backfill and the live skill matcher.** `cmd/embed-backfill` (idempotent, resumable, batched) ran 152 against `brain_staging`: **skills.embedding 32/32, candidates.resume_embedding 10/10**, all bge. The new 153 `matching.NewStoredSkillMatcher` reads the backfilled, HNSW indexed `skills.embedding` column (a global, 154 no RLS table) into a bounded cache and resolves a required skill against a candidate's skills by cosine 155 over the STORED vectors, instead of embedding name strings at request time. `TestBrainSemanticSkillMatchLive` 156 proves the engine's Skills component now resolves semantically (a synonym scores 0.74 versus an unrelated 157 skill 0.57 on stored bge vectors). 1583. **Reconciliation.** The brain consumes the one engine path: the TAG org level intents read 159 `candidate_evaluations`, and `WhoCanSupport` reads the same frozen table, so the brain, the engine, and 160 the dashboard agree by construction. `TestBrainTAGReconciliation` proves equality (engine = tag = support). 1614. **The brain mounted live.** `POST /ai/query` is mounted under the tenant scoped group behind session, 162 CSRF, the new `brain.query` RBAC capability (the search.run tier), rate limit, and audit. The orchestrator 163 gained a top level dispatcher (`Answer`) and the CAG direct path (`AnswerDirect`); the faithfulness gate 164 was upgraded to the deterministic fallback. The registry is validated at boot (an unsafe intent fails the 165 boot). 1665. **Per tenant memory live.** Memory writes are scoped by organization_id and recall is RLS scoped; RAG is 167 memory augmented (the tenant's own recalled notes join the evidence). Memory is wiped on erasure in BOTH 168 anonymize paths (the on demand `sourcing.Delete` and the governance retention sweep). Cross tenant non 169 retrieval and erasure are tested. 1706. **The contract** is in `docs/openapi.yaml`: the query request, the answer object (answer, path, rationale 171 source, citations, faithful, fallback reason), and one typed error envelope with the 200/403/422/503 172 failure surface (a faithfulness fallback is a 200 deterministic; a demographic or unanswerable or unknown 173 intent question is a 422; a plane outage is a 503; never a raw SQL error and never a hang). 174 175## Live embedding counts on staging 176`brain_staging` after the pre cutover hardening was fully re embedded with bge (skills 38/38, live 177candidates 131/131), and the startup vector model check passes against it (the gate opens). (The shared 178production database holds all-minilm vectors today; the cutover re embeds everything with bge, now an 179enforced gate, see DEPLOY_PLAN.md and FIX 2, because mixing models silently corrupts ranking.) 180 181## Test results 182- **Unit (race, no DB):** all pass, including the EEO tripwire over the ai package and the engine frozen 183 and golden tests. 184- **Live integration on M4 staging:** the brain tests pass (semantic skill match via stored embeddings AND 185 in the engine scoring path at the new threshold, the threshold calibration harness, the startup vector 186 model check, the brain answering across RAG/TAG/CAG with the EEO refusal and the audits, the scoring 187 reconciliation engine=tag=dashboard, memory cross tenant isolation with erasure). After the pre cutover 188 hardening the full suite on `brain_staging` is **49 pass, 1 skip, 1 fail** (see the hardening section: the 189 skip is per tenant database isolation, out of scope; the fail is the environmental 190 `TestCommandCenterUpcomingIncumbentAdopt`, the empty global forecast and award catalog, confirmed pre 191 existing on the AI layer branch alone in FIX 4). 192- **Build gates:** FIPS stamp present after every build, `make ci-gates` green (backdoor and FIPS), vet and 193 gofmt clean, no Python, no cgo. 194 195## What the adversarial review found and what was fixed 196A 5 dimension review (EEO safety, the engine floor, the live vector layer, tenancy, the failure surface), 197with every finding independently verified, confirmed six, all fixed: 1981. **HIGH (erasure parity).** The governance retention sweep anonymized a candidate but skipped ai_memory, 199 so a candidate de identified by the automatic sweep left recallable memory behind, unlike the on demand 200 erasure. Fixed: the ai_memory delete was added to the sweep's purge loop, so both anonymize paths leave 201 the same residue. 2022. **MEDIUM (embed model drift).** The API defaulted to bge while the backfill, the gateway, and the env 203 defaulted to all-minilm, the silent corruption mode (query bge versus stored all-minilm mis ranks every 204 neighbor). Fixed: every default aligned to bge so write time and query time vectors share one model. 2053. **MEDIUM (failure surface).** A transient plane failure that had not yet tripped the breaker mapped to a 206 4xx. Fixed: a typed `inference.ErrPlaneUnavailable` is returned on exhausted transient retries, mapped to 207 503. 2084. **LOW.** The API now calls `ValidateRegistry` at boot so an unsafe TAG intent fails the boot. The 209 `ForgetCandidate` comment was corrected (the transactional erasure paths inline the delete; ForgetCandidate 210 is the standalone hook). 211 212## What remains (genuine decisions or sequenced work) 2131. **The production cutover** is the operator's manual step in DEPLOY_PLAN.md (merge the AI layer and W3, 214 apply the migrations through 0018, pull bge on the production Ollama, switch the embedder, re embed all 215 vectors with bge, deploy the brain binary, verify, roll back). Validated on M4 staging, NOT run against 216 production. 2172. **Per tenant database routing** for the brain is the sequenced follow on (out of scope here); the brain 218 is born on the shared RLS database and isolation is enforced by RLS and the org GUC. 2193. **The semantic threshold.** DONE in the pre cutover hardening above (FIX 1). The engine's 0.80 default 220 was recalibrated to 0.62 on the bge-small vectors (engine ModelVersion v1.1.0, migration 0019), proven to 221 fire in the engine scoring path; the old 0.80 silently discarded real synonym matches (cosine ~0.74). 2224. **GraphRAG teaming and Foresight** stay scaffolded and degrade gracefully; the rule based router stays 223 rule based, as directed. 2245. **The Command Center SQL composite** still co exists for the org level dashboard (a different metric than 225 the per candidate engine score); the brain consumes the engine path and agrees with both, so they cannot 226 contradict. Merging the composite into the engine is a larger refactor, recommended but out of scope. 227