Janady13/missionmatch / DEPLOY_PLAN.md

1# DEPLOY_PLAN.md — AI Intelligence Layer cutover 2 3The build is validated in a staging posture. This is the exact, ordered, rollback ready plan for the 4**human** to run the irreversible live cutover. It is the one manual step: Bubbles serves other live 5paying client sites and this build provisions cleared data databases. 6 7**Do not run the Bubbles steps automatically.** Everything below was dry run and validated; the operator 8executes the Bubbles and live database steps. 9 10## State at handoff (already done, staging) 11- The MEGAMIND gateway is **running on M4** (`100.94.8.94:8090`, Tailscale only, via `nohup`) and smoke 12 tested. Ollama on M4 has `all-minilm` (384) and `llama3.1:8b`. 13- The shared production database on M4 is at **goose v17**; `0017_ai_memory.sql` is applied (additive, a 14 new table the old API binary does not use, so it is safe alongside the running old binary). 15- `skills.embedding` (61/61) and the demo org `candidates.resume_embedding` (1511/1511) are backfilled. 16- A staging tenant database `tenant_staging_demo` was provisioned on M4 to validate `provision-tenant`. 17- The Bubbles `mm-api` service is **unchanged** (still the pre AI binary). 18 19## Merge sequence (do this FIRST, before any cutover step below) 20The stack is three branches: the brain sits on W3, which sits on the AI intelligence layer. Merge them in 21this order, as three DISCRETE steps, each merged and validated and individually rollback able, because each 22carries its own migration and is a separate revertable unit. Do NOT collapse them into one merge. 23 24**Step M1 - merge the AI intelligence layer branch first (the foundation; carries migration 0017, 25ai_memory).** It is the base of the stack and the only branch the others depend on. Validate (build FIPS, 26unit suite, integration on a staging database) before proceeding. 27Rollback: revert the merge commit. If 0017 was already applied to a database, `goose down` one step (the 280017 down drops ai_memory), per the rollback in `db/migrations/0017_ai_memory.sql`. 29 30**Step M2 - then merge W3 (carries migration 0018, recommendation_runs and staffing_recommendations).** 31Only after M1 is merged and green. Validate again. 32Rollback: revert the W3 merge commit. If 0018 was applied, `goose down` to v17 (the 0018 down DROPs the two 33W3 tables), per the rollback referenced in Step W3.1. 34 35**Step M3 - then merge the brain (carries migration 0019, the v1.1.0 model version registry row).** Only 36after M2 is merged and green. The brain adds NO new schema table; 0019 inserts one row into 37aedt_model_versions registering the recalibrated semantic threshold (v1.1.0), which the engine now stamps 38onto every decision. Validate again. 39Rollback: revert the brain merge commit. If 0019 was applied, `goose down` to v18 (the 0019 down DELETEs the 40v1.1.0 registry row), per the rollback in `db/migrations/0019_model_version_v1_1_0.sql`. Note that reverting 41the brain code without reverting 0019 is safe (a registry row with no stamping engine is inert), and 42reverting to a binary that stamps v1.0.0 with the v1.1.0 row present is also safe. 43 44**Hard rule.** Do not collapse these into one merge, and do NOT run the production cutover sequence below 45until all three branches are merged in this M1, M2, M3 order AND the migrations through 0019 are applied to 46the production shared database. The cutover steps assume a single merged tree at migration head v19. 47 48## Build the artifacts (on the build host, no live mutation) 49``` 50cd ~/missionmatch && git checkout ai-intelligence-layer 51# API (Bubbles target), FIPS, verify the stamp: 52CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/mm-api ./cmd/api 53go version -m /tmp/mm-api | grep GOFIPS140 # must show GOFIPS140=v1.0.0 54# Gateway + tools (M4 target): 55CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/megamind-gateway ./cmd/megamind-gateway 56CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/provision-tenant ./cmd/provision-tenant 57CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/embed-backfill ./cmd/embed-backfill 58# Frontend (unchanged this cutover; rebuild only if you changed it): 59cd frontend && npm run build 60``` 61 62## Step 1 — durable MEGAMIND gateway on M4 (launchd) 63Replaces the staging `nohup` with a managed service. 64``` 65scp /tmp/megamind-gateway joseph@100.94.8.94:/tmp/megamind-gateway 66scp infra/launchd/com.missionmatch.megamind-gateway.plist joseph@100.94.8.94:/tmp/ 67ssh joseph@100.94.8.94 ' 68 pkill -f /tmp/megamind-gateway || true # stop the staging nohup instance 69 sudo cp /tmp/megamind-gateway /usr/local/bin/megamind-gateway && chmod +x /usr/local/bin/megamind-gateway 70 cp /tmp/com.missionmatch.megamind-gateway.plist ~/Library/LaunchAgents/ 71 launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist 2>/dev/null || true 72 launchctl load ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist 73' 74# Verify: 75curl -s http://100.94.8.94:8090/healthz # -> ok 76curl -s http://100.94.8.94:8090/v1/embed -d '{"inputs":["x"]}' | jq .dimension # -> 384 77``` 78Rollback: `ssh joseph@100.94.8.94 'launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist'`. 79 80Optional quality upgrade (no migration, both 384): pull bge-small and set `MEGAMIND_EMBED_MODEL`: 81`ssh joseph@100.94.8.94 'ollama pull <bge-small-en-v1.5-gguf>'` then re run the backfill (Step 4). 82 83## Step 2 — env on Bubbles (operator; root owned 0600 files) 84Add the AI plane variables to `/etc/missionmatch/api.env` (back it up first): 85``` 86sudo cp /etc/missionmatch/api.env /etc/missionmatch/api.env.bak.$(date +%s) 87# append (do not echo secrets to a log): 88MM_EMBED_PROVIDER=megamind 89MM_MEGAMIND_URL=http://100.94.8.94:8090 90MM_EMBED_DIMENSION=384 91MM_MEGAMIND_EMBED_MODEL=all-minilm 92MM_MEGAMIND_GENERATE_MODEL=llama3.1:8b 93MM_MEGAMIND_TEACHING_ENABLED=false 94``` 95Remove the dead keys from `/etc/missionmatch/sam-sync.env` (back it up first): `DATA_GOV_API_KEY`, 96`KAGGLE_USERNAME`, `KAGGLE_KEY`, `HF_TOKEN`. 97Rollback: restore the `.bak` files. 98 99## Step 3 — database (already at v17; only new tenants need provisioning) 100The shared database is at v17. If you maintain it, confirm: 101``` 102ssh joseph@100.94.8.94 'PGURL=... goose -dir db/migrations postgres "$PGURL" status' # 0017 applied 103``` 104For a NEW client (database per tenant), provision a fresh database (this is the irreversible "create a 105live client database" step; run only for a real onboarding): 106``` 107ssh joseph@100.94.8.94 '/usr/local/bin/provision-tenant \ 108 -admin-dsn "host=/tmp user=postgres dbname=postgres sslmode=disable" \ 109 -name tenant_<client_slug>' 110``` 111It applies the full public + eeo migration set, RLS forced, eeo segregated, vector(384) ready, and 112verifies the app role has no grant on `eeo.candidate_demographics`. Rollback (only if the database is 113brand new and unused): `DROP DATABASE tenant_<client_slug>`. 114 115## Step 4 — embed backfill (idempotent) 116For the shared database (the demo org is already done) or a freshly provisioned tenant database, once it 117has candidates: 118``` 119ssh joseph@100.94.8.94 '/usr/local/bin/embed-backfill \ 120 -dsn "postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/<db>?sslmode=disable" \ 121 -megamind http://100.94.8.94:8090 -what all -only-missing -batch 64' 122``` 123`-only-missing` makes it safe to re run. Rollback: `UPDATE skills SET embedding=NULL; UPDATE candidates 124SET resume_embedding=NULL;` then re run (or leave; embeddings are additive). 125 126## Step 5 — THE CUTOVER on Bubbles (operator, irreversible; brief downtime) 127Swap the mm-api binary to the AI build and restart. Back up the current binary first. 128``` 129scp /tmp/mm-api user@bubbles:/tmp/mm-api.new 130ssh user@bubbles ' 131 sudo cp /home/user/missionmatch/mm-api /home/user/mm-api.bak.$(date +%s) # backup 132 sudo systemctl stop mm-api 133 sudo cp /tmp/mm-api.new /home/user/missionmatch/mm-api && sudo chmod +x /home/user/missionmatch/mm-api 134 sudo systemctl start mm-api 135 sleep 2 && systemctl is-active mm-api 136' 137# Verify the live API still answers and FIPS is on: 138curl -s https://missionmatch.thatwebhostingguy.com/healthz # -> ok 139curl -s https://missionmatch.thatwebhostingguy.com/healthz/fips | jq .fips_enabled # -> true 140``` 141**Rollback (fast):** 142``` 143ssh user@bubbles ' 144 sudo systemctl stop mm-api 145 sudo cp /home/user/mm-api.bak.<ts> /home/user/missionmatch/mm-api 146 sudo systemctl start mm-api 147' 148# and restore /etc/missionmatch/api.env.bak.<ts> if the env changes caused the issue. 149``` 150Nginx is **unchanged** by this cutover (the gateway is reached over Tailscale, not through nginx). No 151`nginx -t` or reload is required. The systemd unit `mm-api.service` is unchanged; only the binary and 152the env file change. 153 154## Step 6 — host service hygiene (operator) 155The audit found the `mm-sam-sync.timer` showed no next elapse. The unit is valid (`OnCalendar=*-*-* 15604:00:00`); ensure it is started: 157``` 158ssh user@bubbles 'sudo systemctl daemon-reload && sudo systemctl enable --now mm-sam-sync.timer && systemctl list-timers | grep mm-sam-sync' 159``` 160 161## Activating the AI HTTP surface (a later, separate change) 162The orchestrator, TAG, RAG, and memory are built and tested as a library but are not yet mounted as 163authenticated routes (deferred so this cutover does not change the user facing app). Mounting them is a 164small code change behind the session resolver, RBAC, and the same audit, then a repeat of Step 5. 165 166## Post cutover checks 167- `curl https://missionmatch.thatwebhostingguy.com/healthz/fips` shows `fips_enabled: true`. 168- The MEGAMIND gateway `launchctl list | grep megamind` is running on M4. 169- `make ci-gates` on the build host stays green (backdoor + FIPS). 170- No EEO reach: `go test ./internal/modules/ai/...` (the tripwire) stays green in CI. 171 172## W3 AI Staffing Recommendations cutover (additive, on top of the AI layer) 173 174W3 is a backend addition: migration 0018, four new authenticated routes already mounted in the mm-api 175binary, and one new scheduled job. It is additive and reversible. As with the AI layer, the operator runs 176the Bubbles and production database steps; everything below was built and validated on M4 staging 177(migration 0018 applies cleanly through v18 plus eeo, and 5 of 5 W3 integration tests pass). 178 179### State at handoff (already done, staging) 180- Migration 0018 (recommendation_runs, staffing_recommendations) is written and proven on a fresh M4 181 staging database `tenant_w3_staging` (goose v18). It is NOT applied to the production shared database. 182- The mm-api binary on the build host now carries the four W3 routes (staffing.recommend gated), but the 183 Bubbles mm-api is unchanged (still the pre AI binary), so nothing is live yet. 184- The new `staffing-proactive-sync` binary and `mm-staffing-proactive.{service,timer}` are written but NOT 185 installed on Bubbles. 186 187### Build the artifacts (build host, no live mutation) 188``` 189cd ~/missionmatch && git checkout w3-ai-staffing-recommendations 190CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/mm-api ./cmd/api 191go version -m /tmp/mm-api | grep GOFIPS140 # must show GOFIPS140=v1.0.0 192CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/staffing-proactive-sync ./cmd/staffing-proactive-sync 193``` 194 195### Step W3.1 - apply migration 0018 to the production shared database (operator, irreversible) 196This is the one irreversible step. 0018 only ADDS two tables and is safe alongside the running old binary 197(the old binary does not use them). Run as the migrator role. 198``` 199ssh joseph@100.94.8.94 'cd ~/missionmatch && \ 200 goose -dir db/migrations postgres "postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/missionmatch?sslmode=disable" up' 201# verify: goose ... status shows 0018 applied, head v18 202``` 203Rollback (only before any recommendation rows matter): `goose ... down` once, or 204`DROP TABLE staffing_recommendations; DROP TABLE recommendation_runs;`. 205 206### Step W3.2 - deploy the mm-api binary that carries the W3 routes (operator, brief downtime) 207Same binary swap as the AI layer Step 5; this build also mounts the four W3 routes behind 208staffing.recommend / slate.write / candidate.read. Back up the current binary first. 209``` 210scp /tmp/mm-api user@bubbles:/tmp/mm-api.new 211ssh user@bubbles ' 212 sudo cp /home/user/missionmatch/mm-api /home/user/mm-api.bak.$(date +%s) 213 sudo systemctl stop mm-api 214 sudo cp /tmp/mm-api.new /home/user/missionmatch/mm-api && sudo chmod +x /home/user/missionmatch/mm-api 215 sudo systemctl start mm-api 216 sleep 2 && systemctl is-active mm-api 217' 218curl -s https://missionmatch.thatwebhostingguy.com/healthz/fips | jq .fips_enabled # -> true 219``` 220Rollback: stop mm-api, restore `/home/user/mm-api.bak.<ts>`, start mm-api. 221 222### Step W3.3 - install the proactive recompute timer (operator) 223``` 224scp /tmp/staffing-proactive-sync user@bubbles:/home/user/missionmatch/staffing-proactive-sync 225ssh user@bubbles ' 226 sudo cp infra/systemd/mm-staffing-proactive.* /etc/systemd/system/ 227 sudo systemctl daemon-reload 228 sudo systemctl enable --now mm-staffing-proactive.timer 229 systemctl start mm-staffing-proactive.service # one off, to seed the first bench watch 230 systemctl list-timers | grep mm-staffing-proactive 231' 232``` 233Rollback: `sudo systemctl disable --now mm-staffing-proactive.timer`. The watch freshness depends on the 234ingestion timers (mm-sam-sync, mm-opp-sync); the proactive timer is scheduled after them at 05:00. 235 236### Post cutover checks (W3) 237- `curl -sf -X POST https://missionmatch.thatwebhostingguy.com/api/missionmatch/recommendations/portfolio` with a 238 valid session returns a 200 allocation (or 403 without staffing.recommend); an unauthenticated call is 403. 239- A staffing run writes one `aedt.staffing.recommended` audit and a decision writes one `aedt.staffing.decided`. 240- `go test -tags=integration -run TestStaffing ./internal/integration/` stays green against the database. 241- The EEO tripwire `go test ./internal/modules/ai/...` stays green (it now walks the staffing package too). 242 243### Note: feed freshness 244The proactive bench watch only surfaces an opportunity once the ingestion timers (mm-sam-sync, mm-opp-sync) 245have ingested it. The audit previously flagged the SAM sync timer as possibly not firing; confirm 246`systemctl list-timers | grep mm-sam-sync` shows a next elapse before relying on the watch for new postings. 247 248## AI brain activation cutover (mount the brain live, additive) 249 250This activates the brain on the SHARED RLS database that runs today (per tenant pools are a sequenced 251follow on, out of scope). It mounts one new route (POST /ai/query), switches the live embedder to 252bge-small-en-v1.5, and re embeds the vectors. Validated end to end on M4 staging (the brain_staging 253database, 46 of 48 integration pass, 1 environmental fail unrelated to the brain, 4 of 4 brain tests pass). 254Nothing below was run against production. 255 256### Dependency (must be true first) 257- All three branches are merged in the M1, M2, M3 order above, and the migrations through 0019 are applied 258 to the production shared database (ai_memory is 0017 from the AI layer, the W3 tables are 0018, and the 259 v1.1.0 model version registry row is 0019 from the brain branch, added by the threshold recalibration). 260 The brain branch sits on top of both and carries only that one registry row migration, no new table. 261- Apply 0019 to production if it is not yet at v19 (additive, a single registry row, safe alongside the 262 running binary): 263``` 264ssh joseph@100.94.8.94 'PGURL="postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/missionmatch?sslmode=disable"; goose -dir db/migrations postgres "$PGURL" up' 265ssh joseph@100.94.8.94 'PGURL=...; goose -dir db/migrations postgres "$PGURL" status' # head v19, 0019 applied 266``` 267Rollback: `goose -dir db/migrations postgres "$PGURL" down` to v18 (DELETEs the v1.1.0 registry row). 268- bge-small-en-v1.5 is available on the production Ollama. It is NOT in the official Ollama library; pull 269 the community handle (384 dim, standard /api/embed): 270``` 271ssh joseph@100.94.8.94 'curl -s http://127.0.0.1:11434/api/pull -d "{\"name\":\"qllama/bge-small-en-v1.5\",\"stream\":false}"' 272# probe: the embedding must be 384 wide 273ssh joseph@100.94.8.94 'curl -s http://127.0.0.1:11434/api/embed -d "{\"model\":\"qllama/bge-small-en-v1.5\",\"input\":\"x\"}" | head -c 60' 274``` 275 276### Step B1 - env on Bubbles (operator, root owned 0600 files; back up first) 277Append to /etc/missionmatch/api.env: 278``` 279MM_EMBED_PROVIDER=megamind 280MM_MEGAMIND_URL=http://100.94.8.94:8090 281MM_EMBED_DIMENSION=384 282MM_MEGAMIND_EMBED_MODEL=qllama/bge-small-en-v1.5 283MM_MEGAMIND_GENERATE_MODEL=llama3.1:8b 284``` 285Set the same MM_MEGAMIND_EMBED_MODEL in the sync env used by embed-backfill, and update the M4 gateway 286launchd env MEGAMIND_EMBED_MODEL to qllama/bge-small-en-v1.5 (the rerank path uses the gateway default). 287Rollback: restore the .bak files. 288 289### Step B2 - re backfill the production vectors with bge (operator, irreversible to the prior vectors) 290CRITICAL: production currently holds all-minilm vectors (from the AI layer). bge and all-minilm vectors are 291NOT comparable, so the query (bge) versus stored (all-minilm) mismatch would silently corrupt retrieval. 292Re embed EVERYTHING with bge BEFORE serving the brain. NULL the columns first so a row that errors mid run 293is left unembedded (a safe no semantic match) rather than retaining a stale all-minilm vector, then run a 294FULL re embed (no -only-missing, so every row is rewritten with bge): 295``` 296ssh joseph@100.94.8.94 'psql -U postgres -d missionmatch -c "UPDATE candidates SET resume_embedding=NULL; UPDATE skills SET embedding=NULL"' 297ssh joseph@100.94.8.94 '/usr/local/bin/embed-backfill \ 298 -dsn "postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/missionmatch?sslmode=disable" \ 299 -megamind http://100.94.8.94:8090 -model qllama/bge-small-en-v1.5 -what all -batch 64' 300``` 301 302THE RE EMBED GATE (enforced, not a glance). The verify below is a hard ASSERTION: it RAISES (non zero exit 303under ON_ERROR_STOP) unless every embedding is present and 384 wide. Do NOT proceed to B3 unless this 304assertion passes. The brain routes stay dark until the bge backfill is verified complete. Deleted and 305anonymized candidates legitimately keep a NULL resume_embedding (the right to erasure), so the candidate 306assertion scopes to live rows, exactly the set the backfill targets; skills must be embedded in full. 307``` 308ssh joseph@100.94.8.94 'psql -U postgres -d missionmatch -v ON_ERROR_STOP=1 <<SQL 309DO \$\$ 310DECLARE s_total int; s_emb int; c_total int; c_emb int; w int; 311BEGIN 312 SELECT count(*), count(embedding) INTO s_total, s_emb FROM skills; 313 SELECT count(*), count(resume_embedding) INTO c_total, c_emb 314 FROM candidates WHERE deleted_at IS NULL AND anonymized_at IS NULL; 315 SELECT vector_dims(embedding) INTO w FROM skills WHERE embedding IS NOT NULL LIMIT 1; 316 IF s_emb <> s_total THEN RAISE EXCEPTION ''skills not fully embedded: %% of %%'', s_emb, s_total; END IF; 317 IF c_emb <> c_total THEN RAISE EXCEPTION ''live candidates not fully embedded: %% of %%'', c_emb, c_total; END IF; 318 IF w IS DISTINCT FROM 384 THEN RAISE EXCEPTION ''sampled vector width %% not 384'', w; END IF; 319 RAISE NOTICE ''BACKFILL VERIFIED: skills %%/%%, live candidates %%/%%, width %%'', s_emb, s_total, c_emb, c_total, w; 320END \$\$; 321SQL' 322``` 323Rollback: the prior all-minilm vectors are not retained; if rolling back, leave bge vectors in place (the 324prior binary re embeds skill names at request time and ignores the stored column, so it is unaffected). 325 326### Step B3 - deploy the binary that mounts the brain (operator, brief downtime) 327The new mm-api mounts POST /ai/query (capability brain.query) and uses the stored embedding skill matcher. 328Back up the current binary first; same swap as the prior cutovers. 329``` 330CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/mm-api ./cmd/api 331go version -m /tmp/mm-api | grep GOFIPS140 # must show GOFIPS140=v1.0.0 332scp /tmp/mm-api user@bubbles:/tmp/mm-api.new 333ssh user@bubbles ' 334 sudo cp /home/user/missionmatch/mm-api /home/user/mm-api.bak.$(date +%s) 335 sudo systemctl stop mm-api 336 sudo cp /tmp/mm-api.new /home/user/missionmatch/mm-api && sudo chmod +x /home/user/missionmatch/mm-api 337 sudo systemctl start mm-api && sleep 2 && systemctl is-active mm-api 338' 339``` 340The skill matcher caches the global skill vectors on first use; the restart in this step picks up the bge 341backfill. Rollback: stop mm-api, restore /home/user/mm-api.bak.<ts>, start mm-api (the /ai/query route 342disappears; the older binary is unaffected by the bge vectors). 343 344STARTUP VECTOR MODEL CHECK (the binary enforces the gate, belt and suspenders to the B2 assertion). On 345boot, with a configured plane, the API runs a self check: it asserts the skills.embedding and 346candidates.resume_embedding columns are vector(384), samples stored skill vectors, and re embeds them with 347the configured MM_MEGAMIND_EMBED_MODEL to confirm the stored vectors were produced by that model (cosine 348within tolerance). If the stored vectors do NOT match the configured model (the all-minilm versus bge 349corruption mode, for example if an operator skipped or half ran B2), POST /ai/query returns a typed 503 350vector_model_mismatch and the mismatch is logged, while the rest of the API serves normally; the brain 351stays dark until the store is re embedded and the binary restarted. A boot time plane outage leaves the 352check to self heal on the first query after the plane recovers. So /ai/query stays dark until BOTH the B2 353re embed is verified complete AND the startup vector model check passes; the brain never serves a query 354over a mismatched vector store. 355 356### Step B4 - verify the brain answers end to end (operator) 357``` 358# with a valid session + CSRF (capability brain.query): 359curl -s -X POST https://missionmatch.thatwebhostingguy.com/api/missionmatch/ai/query \ 360 -H 'Content-Type: application/json' -b "$COOKIES" -H "X-CSRF-Token: $CSRF" \ 361 -d '{"question":"who are the top cleared cloud security candidates"}' | jq '{path, rationale_source, citations}' 362# a demographic question must be refused (422), an aggregation intent must compute from the database: 363curl -s ... -d '{"intent":"count_candidates_by_clearance","args":[""]}' | jq .answer 364curl -s -o /dev/null -w "%{http_code}" ... -d '{"question":"list candidates by race"}' # -> 422 365curl -s https://missionmatch.thatwebhostingguy.com/healthz/fips | jq .fips_enabled # -> true 366``` 367 368### Post cutover checks (brain) 369- A brain query writes one aedt.ai.answered (or aedt.ai.refused) audit, no PII, no prose. 370- A plane outage returns a typed 503 (never a hang); a faithfulness fallback returns a 200 with 371 rationale_source deterministic. 372- `go test -tags=integration -run TestBrain ./internal/integration/` stays green against the database, and 373 the EEO tripwire `go test ./internal/modules/ai/...` stays green (it walks the brain files). 374- Staging cleanup when done: `DROP DATABASE brain_staging` (and the W3 tenant_w3_staging) on M4. 375