Janady13/missionmatch / BUILD_REPORT.md

1# Deployability and operability (Mission B, validated on staging and locally) 2 3Branch `deploy-and-ops` (off `per-tenant-routing`). Container images, automated CI with the security and 4FIPS gates, Terraform for the primary cloud, proven tenant backup and recovery, and the operations 5runbooks. Validated on M4 staging and locally. NOT merged, no production change, no Bubbles, no brain 6cutover. Terraform is authored and validated but NOT applied (no cloud accounts yet). 7 8## B0 research (primary sources relied on) 9- **Container:** `distroless/static-debian12:nonroot` (not scratch, the CA bundle is load bearing for 10 verify full TLS), numeric `USER 65532`, a multi stage build with an in build FIPS stamp gate, base images 11 pinned by digest, and a read only rootfs plus `/tmp` tmpfs plus drop ALL capabilities via the pod 12 securityContext. Aligns with STACK_DECISIONS section 1. Sources: GoogleContainerTools distroless, Go FIPS 13 140 (go.dev), Docker multi stage builds. 14- **CI:** GitHub Actions, actions pinned by SHA, `permissions: contents read`, build plus the race suite 15 under `fips140=on` plus `make ci-gates` with no `continue-on-error` (fail closed); the integration suite 16 is excluded (a runner has no live Postgres or plane). Sources: GitHub Actions Go docs, fail closed status 17 checks. 18- **Terraform:** the HashiCorp standard module split mirroring the app's cloud ports, AWS GovCloud primary 19 (partition from `data.aws_partition`), Azure Government as a tfvars and module swap, with pgvector and the 20 roles staying in goose. Sources: HashiCorp module structure, AWS provider RDS / S3 / Secrets Manager / VPC 21 / IAM, GovCloud partition. 22- **Backup and restore:** one `pg_dump -Fc` of the whole tenant database (public plus eeo plus the 23 vectors), restored into a fresh database from template0 with the extension and grants in place, restored 24 as the superuser so ownership and the extension comment both survive, and a verify of row counts plus a 25 vector sample plus the invariants. Sources: PostgreSQL pg_dump and pg_restore docs. 26 27## B1 container images 28`infra/docker/Dockerfile.mm-api` and `Dockerfile.megamind-gateway`: two stage, `golang:1.26.4` builder 29pinned by digest with the exact FIPS flags and an in build stamp gate (`go version -m | grep 30GOFIPS140=v1.0.0`, the same assertion as `make verify-fips`), final `distroless/static-debian12:nonroot` 31pinned by digest, `USER 65532:65532`. The README documents the build, the in image stamp verification, the 32digest re resolution, and the pod securityContext. A `.dockerignore` keeps the context lean. No container 33runtime on the build host, so the docker build is a CI or operator step; the in image gate's exact assertion 34is proven locally (both binaries carry GOFIPS140=v1.0.0). 35 36## B2 CI fail closed 37`.github/workflows/ci.yml`: on push and pull request, least privilege (`contents: read`), three jobs. 38build-test-gates runs `make build`, the race unit suite under `CGO_ENABLED=1 GODEBUG=fips140=on`, and 39`make ci-gates` with no `continue-on-error`. image builds both Dockerfiles (the in build FIPS gate is the 40proof the stamp is present). terraform runs fmt check, init `-backend=false`, and validate with no 41credentials. Actions are pinned by SHA. `scripts/ci-local.sh` reproduces the build, race suite, and gates 42without the platform and runs green locally. Proven fail closed: a negative test (a backdoor added in a temp 43file) makes `verify-backdoor` exit non zero, and it passes again on removal. 44 45## B3 Terraform (AWS GovCloud, fmt and validate, no apply) 46`infra/terraform`: a root module plus five child modules (network, database managed Postgres 16, 47object_storage, secrets, identity), parameterized so Azure Government is a tfvars and module swap. pgvector, 48the app and migrator and compliance roles, RLS, the per tenant databases, and every secret value stay OUT 49of Terraform (goose, provision-tenant, and the operator). `tofu fmt` is clean and `tofu validate` reports 50the configuration valid (`init -backend=false`, no credentials). plan and apply are the operator's step 51(real GovCloud credentials), never in CI. 52 53## B4 tenant backup and restore (proven on staging) 54`scripts/backup-tenant.sh` and `restore-tenant.sh`. Proven end to end on M4 staging: backed up tenant_alpha 55(184K custom format), restored into a fresh database, verified row counts match across 36 tables (public and 56eeo), 8 candidates with their vectors restored, ownership preserved (migrator owned), and the isolation 57invariants hold. The restore runs as the superuser so the extension comment and the table ownership both 58survive (a real failure caught and fixed). 59 60## B5 operations runbooks 61`docs/RUNBOOK_OPERATIONS.md`: provisioning and deploy, scaling (the per tenant connection budget), failover 62(multi AZ, the control plane as the single point for identity and routing), backup and recovery per 63database (linking B4), and secret rotation (the versioned pepper and TOTP KEK, the per tenant DSN through 64the secrets port, the CSRF and recovery keys). Each is a checklist with a verification and a rollback. 65 66## What remains for the operator (Mission B) 67- Build and push the container images: needs a container runtime (the build host had none); CI builds them 68 on a docker equipped runner, where the in build FIPS gate is the proof. 69- Apply the Terraform: needs real AWS GovCloud credentials; plan and apply are the operator's step. 70- Wire the build-test-gates job as a required status check on the protected branch so the gates actually 71 block merges. 72- Deploy with the documented pod securityContext (non root 65532, read only rootfs, `/tmp` tmpfs, drop ALL 73 capabilities) and the per_tenant env. 74 75--- 76 77# Per tenant database routing (Mission A, validated on staging) 78 79Branch `per-tenant-routing` (off `ai-brain-live`). Validated on M4 staging only, NOT merged, no production 80change, no brain cutover. Moves the platform from one shared RLS database to a shared control plane plus one 81isolated database per tenant. Built and proven across phases A0 to A6; the operator cutover is authored in 82`docs/MIGRATE_TO_PER_TENANT.md` and the new client onboarding in `docs/RUNBOOK_ONBOARDING.md`. 83 84## A0 research (primary sources relied on) 85 86A four facet research workflow verified current best practice before building. 87- **Control plane vs tenant data plane.** Keep the database per tenant silo (the resource boundary is the 88 isolation boundary) and operate the fleet through ONE shared control plane that owns identity, the auth 89 secret tables, and a tenant directory catalog. Sources: AWS SaaS Lens silo/pool/bridge models 90 (https://docs.aws.amazon.com/wellarchitected/latest/saas-lens/silo-pool-and-bridge-models.html) and 91 Tenant Isolation Strategies; Azure multitenant SaaS patterns and control plane considerations 92 (https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/control-planes). 93 CRITICAL FINDING: the repo today replicates identity and auth (0001 organizations and users, 0009 auth_*) 94 into EVERY tenant database via provision-tenant, which makes the repo's own globally unique email and 95 email first login invariants physically unenforceable. Auth must be OPERATED only against the control 96 plane (the schema can stay identical; the per tenant copies of the auth tables go unused). 97- **pgxpool per tenant lifecycle.** Build each tenant pool through the existing `database.NewPool` (so the 98 pgvector AfterConnect codec and the NOBYPASSRLS `AssertRole` run on every pool), lazy open and Ping before 99 caching, a bounded LRU of live pools with a pin or refcount so an in flight transaction is never closed 100 under it, singleflight to avoid duplicate opens, a nonzero `MaxConnLifetimeJitter`, and typed 101 `ErrUnknownTenant` (no network) and `ErrTenantUnreachable`, never a fallback to another tenant. Source: 102 pgx v5 pgxpool godoc and the jackc/pgx discussions on pool per request and RLS vs pool per tenant. Repo 103 reconciliation: org tenancy stays RLS in one pool; the registry is only for the physically separate tenant 104 databases, layered ABOVE `database.NewPool`, never replacing RLS. 105- **Cross tenant migrations.** Use the per instance `goose.NewProvider` (not the process global API), two 106 providers (public plus eeo via `WithTableName`), a Postgres session locker for cross process mutual 107 exclusion, a dry run via `Status` and `HasPending`, and an aggregate then fail design that names the 108 offending database and exits non zero if any database errored or is behind. Source: pressly/goose v3 109 Provider API. Flagged: STACK_DECISIONS pins goose v3.27.1 but go.mod has v3.24.1 indirect; reconcile in A4. 110- **Per tenant secrets.** A secrets port mirroring the storage and embeddings ports; the directory stores 111 only a reference (never a credential); an env adapter for dev and staging; cloud adapters stubbed and 112 wired at deploy (no cloud SDK in the core build); a short TTL cache with force refresh on an auth failure. 113 Sources: AWS Secrets Manager and Azure Key Vault multitenancy guidance, the 12 factor config principle. 114 115## A1 control plane, tenant directory, secrets port 116 117- **config:** `MM_TENANCY` (shared default, or per_tenant), the bounded pool knobs (`MM_TENANT_MAX_POOLS`, 118 `MM_TENANT_POOL_MAX_CONNS`, `MM_TENANT_POOL_IDLE_EVICT_SECONDS`), and `MM_SECRETS_BACKEND` plus 119 `MM_SECRET_CACHE_TTL_SECONDS`. Production validation: per_tenant requires the control plane app DSN and a 120 government cloud secret store, never the env backend. 121- **Migration 0020 tenant_directory:** the control plane catalog (organization_id, db_name, db_host, 122 db_port, connection_secret_ref, status, cloud, region, last_migrated_version/at). It is cross tenant by 123 design so it is NOT under the org GUC RLS; the migrator writes it, the app role reads it to route. Created 124 in the shared migration set so the schema stays identical (harmless and empty in a tenant database). 125- **internal/platform/secrets:** the `Resolver` port (`ResolveDSN` by reference, `Forget` for rotation), an 126 env adapter (`MM_TENANT_DSN_<KEY>`), stubbed cloud adapters, and a TTL cache. Unit tested (resolve, key 127 folding, unknown reference typed error, cloud stub error, cache hit and expiry and force refresh). 128- **internal/platform/directory:** the `Directory` accessor over a control plane pool, with `Lookup`, 129 `List`, `Register`, `SetStatus`, `SetMigrated`, and a typed `ErrUnknownTenant`. Integration tested on 130 staging (the app role reads the catalog for routing, the migrator writes it during provisioning). 131- **Validation:** FIPS build clean, full unit suite green under race and fips140=on, `TestTenantDirectoryRoundTrip` 132 passes on M4 staging at goose v20. 133 134## A2 tenant pool manager 135 136`internal/platform/tenantpool` is the bounded, lazy registry of per tenant pools. Each pool is built 137through the new `database.NewBoundedPool` (so the pgvector codec and the NOBYPASSRLS role assertion run on 138every tenant pool), Pinged before caching, held in a bounded LRU with a pin or refcount so an in flight 139transaction is never closed under it, opened once under singleflight, and evicted whole when idle. Typed 140errors never fall back to another tenant (`ErrUnknownTenant`, `ErrTenantUnavailable`, `ErrTenantUnreachable`), 141and an authentication failure forgets the cached secret and retries once. Unit tested under the race 142detector for lazy open, cache hit, singleflight, LRU eviction, pin protection, idle sweep, close, and every 143typed error. `Config` carries a role and a secret reference suffix so a second registry serves the 144compliance role for the eeo path with a distinct per tenant credential. 145 146## A3 request routing 147 148`tenant.DB` carries an optional `Router`. `RunInTenantTx` (all 23 call sites unchanged) and a new 149`Acquire(ctx)` resolve the request's own tenant database by the validated org in context and run there; 150shared mode is a transparent no op over the single pool. The direct pool readers (the stored skill matcher, 151whose cache is now keyed by org because each tenant has its own skills, plus the reference catalog reads and 152sourcing) route through `Acquire`. The brain, the staffing layer, and embed-backfill all run against the 153tenant database. `cmd/api/main.go` builds a static control plane DB for auth and the System audit, and in 154per_tenant mode makes the app and compliance DBs routing DBs over the two pool managers. Routing unit 155tested (dispatch by org, fail closed without org, shared mode no op); the shared mode integration suite on 156M4 staging is unchanged. 157 158## A4 cross tenant migrations 159 160`cmd/migrate-all-tenants` applies the migration set to the control plane and every active tenant database 161from the directory, both the public and the segregated eeo schema (two goose version tables), using the per 162instance goose Provider (not the process global API) and a Postgres session locker. It is idempotent and 163resumable, supports `-check` (dry run) and `-concurrency`, records the applied version back to the 164directory, and exits non zero if any database failed or is behind. Proven on staging: the control plane 165reports public v20 and eeo v1 at head, and the tool correctly fails closed when a directory row points at a 166database that does not exist. 167 168## A5 auth and identity against the control plane 169 170Auth, sessions, MFA, and user lookups operate against the control plane (the auth service and the System 171audit bind to the static control plane pool); the crypto is unchanged. The five auth integration tests 172pass. For foreign key integrity, each tenant database's business tables reference its own organizations and 173users rows; those are migrator owned, so identity is seeded into each tenant database at onboarding by 174`internal/platform/identity.SeedTenant` (and `cmd/sync-tenant-identity`), re run when the org's users 175change. The running API stays the NOBYPASSRLS app role and never writes identity. 176 177## A6 staging validation (two tenant databases, physical isolation) 178 179Provisioned a control plane plus two tenant databases on M4 (cp_staging, tenant_alpha, tenant_beta), all 180schema identical at goose v20, RLS forced, eeo segregated. `TestPerTenantPhysicalIsolation` drives the real 181routing stack (directory, secrets resolver, pool manager) and proves a request as tenant A reads only its 182own row from tenant_alpha, a request as tenant B reads only its own row from tenant_beta, neither crosses, 183an unknown org is a typed denial, and no org fails closed. Inside a real tenant database the app works in 184full: RLS forced as defense in depth, the brain end to end (RAG, TAG, CAG, the EEO refusal, the audits, the 185reconciliation, memory isolation and erasure, the semantic threshold), the staffing layer end to end (gap, 186portfolio, proactive, decision, erasure), the eeo adverse impact via the compliance role with the app role 187denied on the eeo schema, and embed-backfill populating the tenant's own vectors. Gates green: full unit 188suite under race and fips140=on, FIPS stamp present, ci-gates green (the dev X-Org-ID resolver still cannot 189compile into a release build), vet and gofmt clean, no Python, no cgo. 190 191Fixed a real provisioning bug surfaced here: `provision-tenant` ran the migrations as the admin superuser, 192so the tenant tables were owned by postgres and the migrator could neither write identity nor run future 193migrations. It now runs the migrations as the migrator (a new `-migrator-dsn`, with CREATE granted on the 194schema and the database to the migrator by the admin first), so the migrator owns every table. 195 196## What remains for the operator (Mission A) 197 198- The production cutover is the operator's manual step in `docs/MIGRATE_TO_PER_TENANT.md` (stand up the 199 control plane, provision each tenant database, copy each org's business data, register and activate, 200 flip `MM_TENANCY=per_tenant`, verify, roll back). Authored, NOT executed. The onboarding flow for a new 201 client is `docs/RUNBOOK_ONBOARDING.md`. 202- Production secrets must use a government cloud secret store (the cloud adapter is wired at deploy); the 203 env backend is dev and staging only (config rejects env in production per_tenant). 204- A new user invited after onboarding must be re synced into the tenant database 205 (`sync-tenant-identity`); automating that on invite accept is a sequenced follow on. 206- Per tenant connection budget: confirm `MaxPools` times `MaxConnsPerPool` plus the control plane and 207 compliance pools stays under the server `max_connections` for the fleet size. 208 209--- 210 211# BUILD_REPORT.md — AI brain activation 212 213**Date:** 2026-06-29. **Branch:** `ai-brain-live` (off `w3-ai-staffing-recommendations`, which sits on 214the unmerged AI intelligence layer). **Build host:** M2. **Validated on:** M4 staging (`brain_staging`, 215goose v18, bge-small embeddings, the live MEGAMIND plane). 216 217This mission made the AI brain LIVE: the brain library in `internal/modules/ai` (orchestrator, TAG, RAG 218retrieval, memory, KAG SkillAdjacency, CAG rules) was built and tested but unreachable from the running 219app, and the vector layer was dead in the live path. It is now wired: the real embedder, backfilled 220vectors, the HNSW indexed skill matcher, the brain mounted as an authenticated tenant scoped route behind 221the faithfulness gate, the two scoring paths reconciled, proven end to end on staging. 222 223**Done means, and is proven:** a person can ask the brain a question through an authenticated route and get 224an explainable, faithfulness gated, tenant isolated answer. `TestBrainAnswersEndToEnd` exercises exactly 225that (RAG, TAG, CAG, and an EEO refusal, with the AEDT audits), and passes on M4 staging. 226 227## Pre cutover hardening (four fixes, 2026-06-29) 228 229A close review before cutover found four things to fix. All four are now closed and re validated on M4 230staging (`brain_staging`, goose v19). No feature was added; these are correctness and deploy safety. 231 232**FIX 1, the semantic threshold (the live correctness defect).** The engine's `DefaultSemanticThreshold` 233was 0.80, tuned for a different embedder. On the production bge-small vectors a clear synonym scores ~0.74, 234so true semantic matches fell under the live cutoff and were silently discarded: the brain shipped a 235semantic matcher the engine then ignored. A calibration harness on staging 236(`TestSemanticThresholdCalibration`) scored known synonym and adjacent skill pairs against known unrelated 237pairs on the stored bge vectors: 238 239- Genuine synonyms (11 of 12 pairs): cosine 0.678 to 0.944 (for example "Penetration Testing" to "Ethical 240 Hacking" 0.678, "Threat Hunting" to "Threat Detection" 0.875, "Network Engineering" to "Network 241 Infrastructure Engineering" 0.944). 242- Unrelated pairs (all 12): cosine 0.410 to 0.571 (mean 0.498), the highest being a noise spike "Amazon Web 243 Services" to "Underwater Basket Weaving" 0.571. 244- The genuine groups separate in the gap [0.571, 0.678]. One acronym pair, "Signals Intelligence" to 245 "SIGINT" (0.534), sits BELOW the unrelated ceiling, so no safe single threshold can capture it; acronym to 246 expansion belongs to the skills ontology alias layer, not to a cosine cutoff. It is documented, not forced. 247 248Chosen threshold: **0.62**, in the separating gap with margin: 0.0486 above the unrelated ceiling (no 249invented match) and below every genuine synonym (no discarded match). At 0.62 the harness confirms 11 of 12 250synonyms matched and 12 of 12 unrelated rejected. The threshold is a frozen engine contract, so one commit 251changed the constant, bumped the engine **ModelVersion to v1.1.0**, and updated the frozen test together; 252**migration 0019** registers v1.1.0 in `aedt_model_versions` (migrator only, since the app role has SELECT) 253so every decision stamped v1.1.0 has a registry row and a model card entry, with weights and band cuts 254unchanged from v1.0.0. `TestBrainSemanticSkillMatchEnginePath` proves the new threshold FIRES in the engine 255scoring path (not a read only compare): a candidate holding a synonym of a required skill scores Skills 100 256through `engine.ScoreCandidate` over stored bge vectors, while an unrelated skill scores 0 with the required 257skill reported as a gap. The golden `TestSemanticBelowThresholdDoesNotMatch` moved its below threshold 258fixture from 0.79 to 0.55 (a deliberate shift, since 0.79 now matches). 259 260**FIX 2, the enforced re embed gate.** Production holds all-minilm vectors; bge and all-minilm are not 261comparable, so serving over a wrong model store silently mis ranks every retrieval with no error. A new boot 262self check (`ai.VerifyVectorModel`, behind `ai.VectorGate`) asserts the skills and candidates vector columns 263are vector(384) through the catalog (no RLS bypass, no PII), samples stored skill vectors, and re embeds 264them with the configured model to confirm the stored vectors were produced by that model (cosine within 2650.98). It re embeds the SAME text the backfill wrote (skill name with category, via a shared 266`embeddings.SkillEmbedText`), so a categorized skill is not a false mismatch. On a proven mismatch, 267`POST /ai/query` returns a typed 503 `vector_model_mismatch` and the mismatch is logged, while the rest of 268the API serves normally; a boot time plane outage leaves the gate to self heal on the first query after the 269plane recovers. The comparison core and the gate state machine are unit tested for match, model swap, wrong 270width, permanent mismatch, transient retry, and the handler 503; a live staging test proves the gate OPENS 271on the bge store. DEPLOY_PLAN.md B2 is now a hard psql assertion (it RAISES unless every live candidate and 272every skill is embedded and a sampled vector is 384 wide) and states the gate in words: the brain stays dark 273until both the re embed is verified complete and the startup check passes. 274 275**FIX 3, the merge sequence.** DEPLOY_PLAN.md now enumerates the three branch merges as discrete, 276individually rollback able steps: M1 merge the AI intelligence layer (migration 0017, ai_memory, the 277foundation), M2 then merge W3 (0018, the two staffing tables), M3 then merge the brain (0019, the v1.1.0 278registry row). Each has its own rollback (revert the merge commit, and the goose down already in the 279migration). The hard rule is stated: do not collapse the three, and do not run the cutover until all three 280are merged in order and the migrations through 0019 are applied to production. 281 282**FIX 4, the forecast test is pre existing (confirmed, not a regression).** 283`TestCommandCenterUpcomingIncumbentAdopt` requires a populated global forecast and contract award catalog 284(it does not seed them; it expects them as global reference data). On a fresh database with an empty 285catalog it fails by construction. Confirmed by a controlled experiment: 286 287- On the **ai-intelligence-layer** branch (neither W3 nor the brain present, head migration 0017), against a 288 CLEAN M4 database (`fix4_ail_baseline`, goose v17, empty catalog), it FAILS at 289 `command_center_d_integration_test.go:43` ("expected forecast records in upcoming opportunities"). 290- On the **ai-brain-live** branch (full stack, migrations to 0019), against a CLEAN M4 database 291 (`fix4_brain_baseline`, goose v19, empty catalog), it FAILS at the SAME assertion, line 43. 292 293Identical failure on the foundation branch alone proves the cause is the empty reference catalog 294(environmental), NOT a regression introduced by W3 or the brain. The "passes on the AI layer, fails only 295with W3 or the brain" regression condition is NOT met. Both scratch databases were dropped after the run. 296 297**Re validation (all on M4 staging, nothing against production).** FIPS production build clean with the 298stamp present (`GOFIPS140=v1.0.0`), `sqlc generate` no drift, goose to v19. Full unit suite green under race 299and fips140=on; `go vet` and `gofmt` clean; `make ci-gates` green (backdoor and FIPS). Full integration 300suite on `brain_staging`: **49 pass, 1 skip, 1 fail**. The skip is `TestCrossTenantDatabaseIsolation` (per 301tenant pools, out of scope). The one fail is the same environmental `TestCommandCenterUpcomingIncumbentAdopt` 302(on `brain_staging` it now clears line 43 because two forecasts exist there, and fails one assertion later at 303line 62 on the empty contract award feed, the same missing reference data cause). Every mission critical 304brain test passes: the threshold fires in the engine path and the frozen test matches, the startup vector 305model check serves on a match, the brain answers end to end with the EEO refusal and audits, the 306reconciliation proves engine equals tag equals dashboard, and cross tenant memory isolation and erasure 307hold. 308 309## Scope held 310- Mounted LIVE on the SHARED RLS database, exactly like matching and commandcenter. Per tenant database 311 routing is a sequenced follow on and was NOT done; the live API was not switched to per tenant pools. 312- No branch merged, no production database or Bubbles host touched. Validated on M4 staging only. 313 314## Non negotiable constraints, all held 3151. **No Python, FIPS intact.** All ML runs out of process on the MEGAMIND plane over HTTP. The API binary 316 stays `CGO_ENABLED=0 GOFIPS140=v1.0.0`; the stamp was verified after every build. No cgo, no new 317 dependency. 3182. **EEO isolation absolute.** No brain path names the eeo schema; the TAG layer runs ONLY curated, 319 parameterized, read only intents over an allowlist that excludes eeo (never free form SQL); 320 `safety.ForbidEEO` guards every string sent to the plane; and a new natural language guard 321 (`MentionsProtectedAttribute`) refuses a question that asks to group people by a protected attribute 322 even in plain words ("list candidates by race"). The EEO unreachability tripwire walks the ai package 323 (21 ai and staffing files scanned, none reach eeo). 3243. **Tenant isolation.** Every brain database read (HybridSearch, ExecuteIntent, memory Recall and Store, 325 the audit) runs inside `RunInTenantTx`, RLS scoped, on the validated org. The skill matcher reads only 326 the GLOBAL skills table (no org, no RLS). MEGAMIND stays stateless. `TestBrainMemoryCrossTenantIsolation` 327 proves one tenant cannot recall another's memory. 3284. **The deterministic engine is the floor.** The brain never recomputes or overrides `engine.ScoreCandidate`; 329 the TAG intents read the frozen `candidate_evaluations` and `slate_members`, not the Command Center 330 composite. No model is in any scoring or eligibility decision; the model only embeds a query, generates 331 prose, and checks faithfulness. `TestBrainTAGReconciliation` proves the brain's answer equals the engine 332 equals the dashboard read. 3335. **Faithfulness gate plus deterministic fallback on every answer.** The model prose ships only when the 334 gate confirms it; otherwise the brain returns the deterministic rendering of the same evidence (a 200, 335 rationale_source deterministic), recording why. A plane outage degrades through the circuit breaker to a 336 typed 503, never a hang. 3376. **Every brain action audits** (`aedt.ai.answered`, `aedt.ai.refused`), carrying no PII, no names, no 338 prose, no question text, no fallback reason. 339 340## Research (primary sources, cited in code) 341A 3 agent research workflow verified each piece before wiring: 342- **pgvector 0.8 HNSW and RRF**: the repo is textbook correct (the cosine operator with vector_cosine_ops, 343 RRF at k=60, SET LOCAL ef_search and iterative_scan in the tenant tx). One tightening applied: the 344 vector arm of `SearchCandidatesHybrid` is now `AS MATERIALIZED` so the relaxed_order HNSW output is 345 re-sorted by true distance before the LIMIT, preventing nearest neighbor mis ranking under the RLS post 346 index filter at scale (pgvector 0.8 README; AWS pgvector 0.8 Aurora guidance). The critical rule, the 347 SAME model and normalization for stored and query vectors, drove the embedder decision below. 348- **bge-small-en-v1.5 on Ollama**: it is NOT in the official Ollama library; `qllama/bge-small-en-v1.5` is 349 the simplest stable 384 dim handle over the standard /api/embed (pulled and probed on M4). The optional 350 query instruction prefix is query only and genuinely optional (BAAI: omitting it is a slight degradation 351 only), so the adopted policy is NO prefix on either path, which keeps stored and query text consistent. 352- **NL2SQL safety and faithfulness**: a curated intent registry beats sanitizing generated SQL; the gate 353 withholds unconfirmed prose and falls back to a deterministic answer while still returning a useful 200 354 (CRAG, Self RAG family). 355 356## What was activated, by phase 3571. **The real embedder.** bge-small-en-v1.5 (the `qllama` handle, 384 dim) is the configured default for the 358 MEGAMIND plane; the gateway forwards the request model, so the API and the backfill name it. The fake 359 deterministic embedder is GONE from the API binary (the embedder was removed from main.go and from the 360 staffing cron) and remains only on the deterministic test path (`EmbeddingSkillMatcher`). 3612. **The backfill and the live skill matcher.** `cmd/embed-backfill` (idempotent, resumable, batched) ran 362 against `brain_staging`: **skills.embedding 32/32, candidates.resume_embedding 10/10**, all bge. The new 363 `matching.NewStoredSkillMatcher` reads the backfilled, HNSW indexed `skills.embedding` column (a global, 364 no RLS table) into a bounded cache and resolves a required skill against a candidate's skills by cosine 365 over the STORED vectors, instead of embedding name strings at request time. `TestBrainSemanticSkillMatchLive` 366 proves the engine's Skills component now resolves semantically (a synonym scores 0.74 versus an unrelated 367 skill 0.57 on stored bge vectors). 3683. **Reconciliation.** The brain consumes the one engine path: the TAG org level intents read 369 `candidate_evaluations`, and `WhoCanSupport` reads the same frozen table, so the brain, the engine, and 370 the dashboard agree by construction. `TestBrainTAGReconciliation` proves equality (engine = tag = support). 3714. **The brain mounted live.** `POST /ai/query` is mounted under the tenant scoped group behind session, 372 CSRF, the new `brain.query` RBAC capability (the search.run tier), rate limit, and audit. The orchestrator 373 gained a top level dispatcher (`Answer`) and the CAG direct path (`AnswerDirect`); the faithfulness gate 374 was upgraded to the deterministic fallback. The registry is validated at boot (an unsafe intent fails the 375 boot). 3765. **Per tenant memory live.** Memory writes are scoped by organization_id and recall is RLS scoped; RAG is 377 memory augmented (the tenant's own recalled notes join the evidence). Memory is wiped on erasure in BOTH 378 anonymize paths (the on demand `sourcing.Delete` and the governance retention sweep). Cross tenant non 379 retrieval and erasure are tested. 3806. **The contract** is in `docs/openapi.yaml`: the query request, the answer object (answer, path, rationale 381 source, citations, faithful, fallback reason), and one typed error envelope with the 200/403/422/503 382 failure surface (a faithfulness fallback is a 200 deterministic; a demographic or unanswerable or unknown 383 intent question is a 422; a plane outage is a 503; never a raw SQL error and never a hang). 384 385## Live embedding counts on staging 386`brain_staging` after the pre cutover hardening was fully re embedded with bge (skills 38/38, live 387candidates 131/131), and the startup vector model check passes against it (the gate opens). (The shared 388production database holds all-minilm vectors today; the cutover re embeds everything with bge, now an 389enforced gate, see DEPLOY_PLAN.md and FIX 2, because mixing models silently corrupts ranking.) 390 391## Test results 392- **Unit (race, no DB):** all pass, including the EEO tripwire over the ai package and the engine frozen 393 and golden tests. 394- **Live integration on M4 staging:** the brain tests pass (semantic skill match via stored embeddings AND 395 in the engine scoring path at the new threshold, the threshold calibration harness, the startup vector 396 model check, the brain answering across RAG/TAG/CAG with the EEO refusal and the audits, the scoring 397 reconciliation engine=tag=dashboard, memory cross tenant isolation with erasure). After the pre cutover 398 hardening the full suite on `brain_staging` is **49 pass, 1 skip, 1 fail** (see the hardening section: the 399 skip is per tenant database isolation, out of scope; the fail is the environmental 400 `TestCommandCenterUpcomingIncumbentAdopt`, the empty global forecast and award catalog, confirmed pre 401 existing on the AI layer branch alone in FIX 4). 402- **Build gates:** FIPS stamp present after every build, `make ci-gates` green (backdoor and FIPS), vet and 403 gofmt clean, no Python, no cgo. 404 405## What the adversarial review found and what was fixed 406A 5 dimension review (EEO safety, the engine floor, the live vector layer, tenancy, the failure surface), 407with every finding independently verified, confirmed six, all fixed: 4081. **HIGH (erasure parity).** The governance retention sweep anonymized a candidate but skipped ai_memory, 409 so a candidate de identified by the automatic sweep left recallable memory behind, unlike the on demand 410 erasure. Fixed: the ai_memory delete was added to the sweep's purge loop, so both anonymize paths leave 411 the same residue. 4122. **MEDIUM (embed model drift).** The API defaulted to bge while the backfill, the gateway, and the env 413 defaulted to all-minilm, the silent corruption mode (query bge versus stored all-minilm mis ranks every 414 neighbor). Fixed: every default aligned to bge so write time and query time vectors share one model. 4153. **MEDIUM (failure surface).** A transient plane failure that had not yet tripped the breaker mapped to a 416 4xx. Fixed: a typed `inference.ErrPlaneUnavailable` is returned on exhausted transient retries, mapped to 417 503. 4184. **LOW.** The API now calls `ValidateRegistry` at boot so an unsafe TAG intent fails the boot. The 419 `ForgetCandidate` comment was corrected (the transactional erasure paths inline the delete; ForgetCandidate 420 is the standalone hook). 421 422## What remains (genuine decisions or sequenced work) 4231. **The production cutover** is the operator's manual step in DEPLOY_PLAN.md (merge the AI layer and W3, 424 apply the migrations through 0018, pull bge on the production Ollama, switch the embedder, re embed all 425 vectors with bge, deploy the brain binary, verify, roll back). Validated on M4 staging, NOT run against 426 production. 4272. **Per tenant database routing** for the brain is the sequenced follow on (out of scope here); the brain 428 is born on the shared RLS database and isolation is enforced by RLS and the org GUC. 4293. **The semantic threshold.** DONE in the pre cutover hardening above (FIX 1). The engine's 0.80 default 430 was recalibrated to 0.62 on the bge-small vectors (engine ModelVersion v1.1.0, migration 0019), proven to 431 fire in the engine scoring path; the old 0.80 silently discarded real synonym matches (cosine ~0.74). 4324. **GraphRAG teaming and Foresight** stay scaffolded and degrade gracefully; the rule based router stays 433 rule based, as directed. 4345. **The Command Center SQL composite** still co exists for the org level dashboard (a different metric than 435 the per candidate engine score); the brain consumes the engine path and agrees with both, so they cannot 436 contradict. Merging the composite into the engine is a larger refactor, recommended but out of scope. 437