Janady13/missionmatch / BUILD_REPORT.md

1# BUILD_REPORT.md — missionMATCH AI Intelligence Layer 2 3**Date:** 2026-06-26. **Branch:** `ai-intelligence-layer` (off `master`). **Build host:** M2. 4**Inference host:** M4 (Apple M4, Metal). **Database:** PostgreSQL 16.14 on M4 (goose v17). 5 6This documents what was built, what was researched and relied on, the test results, what was fixed in 7the final scan, and what remains. Status of each capability is one of **LIVE** (built, configured, 8tested on real data), **WIRED** (built and tested, not yet exposed as an HTTP route), **SCAFFOLD** 9(interface present, fuller build sequenced), or **NEEDS DATA** (waits on labels/history). 10 11## Non-negotiable constraints — all held 121. **No Python in the repo.** Verified: `find . -name '*.py'` outside node_modules is empty. The ML runs 13 out of process (Ollama on M4). The gateway, client, and tools are pure Go. 142. **FIPS intact.** The API binary builds `CGO_ENABLED=0 GOFIPS140=v1.0.0`; the stamp was verified after 15 every build (`GOFIPS140=v1.0.0` present). No cgo in any of the four binaries. The circuit breaker is 16 in house specifically to avoid a third party dependency disturbing the FIPS build. 173. **EEO isolation absolute.** No AI path may name the eeo schema. `internal/modules/ai/safety.go` 18 (`ForbidEEO`) refuses it at runtime; `TestNoAIFileReferencesEEO` walks every AI source file and fails 19 if any executable line references it. The TAG allowlist contains no eeo table; sqlc still models no 20 eeo table. Memory and adjacency are eeo guarded. 214. **Tenant isolation is database per tenant.** `cmd/provision-tenant` creates a complete isolated 22 database; Postgres itself refuses cross database reads (`TestCrossTenantDatabaseIsolation`). MEGAMIND 23 is stateless and the teach boundary refuses any client fact (`TestMegamindTeachBoundaryLive`). 245. **The deterministic engine is the floor.** The AI layer reads engine outputs and tenant data and 25 narrates; it never recomputes a score. Learned ranking (LTR with SHAP and calibration) is scoped to 26 the reranker on top of the engine and is sequenced (see Remaining). 276. **Every AI action audits.** The orchestrator writes an append only `aedt.ai.answered` / 28 `aedt.ai.refused` event (route and verdict only, no PII); the backfill writes `aedt.embedding.backfill`. 29 30## Research (primary sources) 31An 8 agent research workflow ran against primary sources before building; the full brief is 32`docs/AI_RESEARCH.md`. Key decisions taken from it and cited in code: 33- **Hybrid RAG in Postgres** (pgvector cosine + tsvector full text fused with Reciprocal Rank Fusion), 34 with pgvector 0.8 `hnsw.ef_search` and `hnsw.iterative_scan = relaxed_order` set per statement. 35- **Adaptive routing**: aggregation questions go to **TAG** (the database computes), not vector RAG. 36- **CRAG style self correction over Self-RAG**: a faithfulness gate that needs no fine tuned model, which 37 fits a stateless commodity inference plane. 38- **Structured tool use over free form NL2SQL** for cleared data: the TAG layer selects from a curated 39 intent registry; the model never authors SQL. 40- **384 dim embeddings** (all-minilm now, bge-small-en-v1.5 the documented production target, both 384) 41 to match the existing `vector(384)` columns with no migration. 42 43## What was built, by phase 44 45### Phase 0 — MEGAMIND inference plane, client, scrub boundary — LIVE 46- `cmd/megamind-gateway` (pure Go, stateless): `/v1/embed`, `/v1/rerank`, `/v1/generate`, 47 `/v1/faithfulness`, `/v1/teach`, `/healthz`. Proxies Ollama (embeddings all-minilm 384, generation 48 llama3.1:8b), wraps generation with a prompt injection guard, persists nothing. **Deployed and running 49 on M4** (Tailscale only bind `100.94.8.94:8090`). Smoke tested: embed 384, rerank cosine, faithfulness 50 via llama3.1, teach refuses a client fact (422). 51- `internal/platform/inference`: typed client (Embed, Rerank, Generate, CheckFaithfulness, Teach, 52 Health) with per request timeout, bounded retries, and an in house circuit breaker. Embed hard fails 53 on a width other than 384. 54- `internal/platform/scrub`: the teaching boundary. `TeachingSignal` is a closed abstract struct; 55 `Validate` refuses any UUID, email, out of vocabulary value, free text run, or stray field, enforced on 56 both the tenant side and the gateway. 57- Tests: scrub boundary (unit), breaker and client (unit), live reachability and 384 dim and teach 58 boundary (integration on M4). 59 60### Phase 1 — per tenant vector and isolation, vectors made live — LIVE 61- `cmd/provision-tenant`: creates a schema identical, isolated tenant database (embedded migrations + 62 goose). Proven on M4: 17 public + 1 eeo migration, RLS forced, app role has zero grant on 63 `eeo.candidate_demographics`, pgvector ready. 64- `cmd/embed-backfill`: embeds candidates and skills through the plane into the `vector(384)` columns. 65 **Run on M4**: `skills.embedding` 0 to 61, demo `candidates.resume_embedding` to 1511. The HNSW indexes 66 and `SearchSkillsByEmbedding` are now live (AWS nearest neighbors are Cybersecurity, Splunk, Systems 67 Engineering). The skill matcher runs on real cosine (the `megamind` embedding provider). 68- Tests: cross tenant database isolation, erasure wipes the resume vector, the embedder uses real 69 semantics (a synonym matches with the plane, not with the fake hash). 70 71### Phases 2-7 — augmented generation module (`internal/modules/ai`) 72- **Safety (`safety.go`) — LIVE**: `ForbidEEO`, `GuardReadOnlyQuery` (single SELECT, no stacked 73 statements, no hidden CTE writes, allowlisted tables only), `ReadAllowlist`. The required EEO tripwire 74 test passes. 75- **TAG (`tag.go`) — WIRED**: a curated registry of named, parameterized, read only intents (no eeo), 76 each validated at startup and execution, run RLS scoped. Tested live (`count_candidates_by_clearance`). 77 The model selects and parameterizes; it never authors SQL. 78- **RAG retrieval (`retrieval.go`) — WIRED**: hybrid pgvector + tsvector + RRF over the live vectors, 79 RLS scoped, pgvector 0.8 tuned, eeo guarded. 80- **Orchestrator (`orchestrator.go`) — WIRED**: the adaptive router (aggregation to TAG, retrieval to 81 RAG, else direct), the faithfulness gate (an unconfirmed answer is withheld and routed for human 82 review), and the AEDT audit on every action. 83- **Extraction (Phase 2)**: the deterministic server side extractor already exists (resume to fields). 84 A plane backed, layout aware extractor (LayoutLMv3/Donut or a local VLM) and a Section L and M parser 85 are **SCAFFOLD/NEEDS DATA**: they require a model served out of process and labeled solicitations; the 86 extraction port is config switchable for that backend. 87- **Reranker (Phase 3)**: the `/v1/rerank` endpoint is live as a cosine baseline; the bge-reranker-v2-m3 88 cross encoder is the documented upgrade. Boolean query generation for sourcing is **SCAFFOLD**. 89 90### Phase 4 — scoring reconciliation 91Reconciliation by routing, the safe choice that does not change the working Command Center: **the AI/TAG 92layer reads the deterministic engine's persisted outputs (`candidate_evaluations`, `slate_members`), 93never the separate Command Center SQL composite.** So the brain's org level answers are the engine's own 94numbers and cannot contradict it. The two implementations still co exist for their different purposes 95(per candidate engine vs org level dashboard aggregate); merging the Command Center composite into the 96engine is a larger refactor recommended in Remaining. 97 98### Phase 5 — KAG and GraphRAG 99- **KAG skill adjacency (`scaffolds.go`) — LIVE**: `SkillAdjacency` over the now populated 100 `skills.embedding` HNSW (the substitution reasoner the audit found missing). Clearance reciprocity 101 already lives in the engine. 102- **GraphRAG teaming graph — SCAFFOLD**: the node/edge model over awards and opportunity_records is 103 designed; the raw relationships exist; the extraction and community summarization pass is sequenced. 104 105### Phase 6 — CAG — LIVE 106`PublicRules()` preloads the fixed weights, bands, clearance hierarchy and reciprocity, and the standing 107disclaimers into model context (contents free, fits easily), so the model never retrieves or invents them. 108 109### Phase 7 — agentic and self correction — WIRED 110The router + faithfulness gate + guardrails + AEDT audit (above). Multi step planning (slate assembly as 111a plan) is **SCAFFOLD** on top of the router. 112 113### Phase 8 — per tenant memory and learning — WIRED / SCAFFOLD 114- **Memory (`memory.go`, migration 0017) — WIRED**: per tenant episodic and semantic memory inside the 115 tenant database (RLS, vector(384), HNSW), candidate scoped. `ForgetCandidate` wipes it, and the right 116 to erasure path (`sourcing.Delete`) now deletes a candidate's memory. Tested live (store, recall, 117 erase). 118- **Active learning + LTR reranker with SHAP and calibration — SCAFFOLD/NEEDS DATA**: the labels exist 119 (`slate_members.decision`, `candidate_evaluations`); the learned reranker that sits on top of the 120 engine (the only place SHAP and calibration belong) is sequenced. 121- **Federated learning + differential privacy — DESIGNED, DISABLED**: `FederatedLearningEnabled = false`; 122 `DefaultFederatedConfig` fixes the privacy budget and the minimum tenant count. Transmission is off and 123 the scrub boundary withholds it. 124 125### Phase 9 — capture and foresight — SCAFFOLD / NEEDS DATA 126Win probability, bid or no bid, price to win (from BLS and USASpending), clearance survival analysis, and 127demand forecasting are interface fixed and marked needs data (they need a labeled win/loss and clearance 128timing history the platform does not have enough of yet). 129 130## Test results 131- Unit (race + fips): **all pass**, including the engine frozen and golden tests, the scrub boundary, the 132 inference breaker and client, and the AI safety and EEO tripwire. 133- Live integration on M4: **39 pass, 0 fail** (the original 31 plus 8 new: megamind reachability and 134 teach boundary, cross tenant database isolation, erasure wipes vector, real semantics, AI memory 135 erasure, KAG adjacency, TAG intent). 136- `go vet`, `staticcheck`, `oxlint`: clean. `make ci-gates`: backdoor and FIPS gates pass. Frontend 137 build: clean. 138 139## What was fixed in the final scan 140- Removed the four dead env keys the audit flagged (`DATA_GOV_API_KEY`, `KAGGLE_USERNAME`, `KAGGLE_KEY`, 141 `HF_TOKEN`) from `.env.example`; documented the AI plane variables. The host env cleanup is in 142 DEPLOY_PLAN.md. 143- Added `compliance.write` and re gated the governance writes (register an AEDT tool, run retention) on 144 it instead of the read capability (the audit's authz granularity flag). 145- Regenerated sqlc so `ai_memory` is modeled. 146 147## What remains (genuine human decisions or sequenced work) 1481. **Activate the AI HTTP surface.** The orchestrator, TAG, RAG, and memory are WIRED and tested as a 149 library but not yet exposed as authenticated chi routes; that is a small wiring step deferred to the 150 cutover so the live demo is not disturbed. 1512. **Database per tenant routing in the API.** The provisioning primitive and isolation are built and 152 proven; switching the running API from the shared single database to per tenant pools is the migration 153 the cutover performs (the current shared+RLS model still works in the interim). 1543. **Per tenant role hardening.** Today tenant databases share the cluster roles; physical isolation 155 holds (Postgres refuses cross database reads), but per tenant credentials would prevent a compromised 156 credential from reaching another tenant database by switching the database name. Recommended. 1574. **bge-small-en-v1.5 and bge-reranker-v2-m3.** all-minilm (384) is live; swapping to bge-small (also 158 384, no migration) and serving the cross encoder reranker improves quality. A config change plus a 159 one time re backfill. 1605. **Scoring reconciliation as a code merge.** The AI layer is reconciled by routing to engine outputs; 161 merging the Command Center SQL composite into the engine would remove the second gating path entirely. 1626. **Phase 2 layout aware extraction, Phase 9 foresight, GraphRAG, LTR reranker, federated loop.** 163 Sequenced as above; they need a served extraction/rerank model, labeled history, or multiple live 164 clients. 165