Janady13/missionmatch / DEPLOY_PLAN.md

1# DEPLOY_PLAN.md — AI Intelligence Layer cutover 2 3The build is validated in a staging posture. This is the exact, ordered, rollback ready plan for the 4**human** to run the irreversible live cutover. It is the one manual step: Bubbles serves other live 5paying client sites and this build provisions cleared data databases. 6 7**Do not run the Bubbles steps automatically.** Everything below was dry run and validated; the operator 8executes the Bubbles and live database steps. 9 10## State at handoff (already done, staging) 11- The MEGAMIND gateway is **running on M4** (`100.94.8.94:8090`, Tailscale only, via `nohup`) and smoke 12 tested. Ollama on M4 has `all-minilm` (384) and `llama3.1:8b`. 13- The shared production database on M4 is at **goose v17**; `0017_ai_memory.sql` is applied (additive, a 14 new table the old API binary does not use, so it is safe alongside the running old binary). 15- `skills.embedding` (61/61) and the demo org `candidates.resume_embedding` (1511/1511) are backfilled. 16- A staging tenant database `tenant_staging_demo` was provisioned on M4 to validate `provision-tenant`. 17- The Bubbles `mm-api` service is **unchanged** (still the pre AI binary). 18 19## Build the artifacts (on the build host, no live mutation) 20``` 21cd ~/missionmatch && git checkout ai-intelligence-layer 22# API (Bubbles target), FIPS, verify the stamp: 23CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/mm-api ./cmd/api 24go version -m /tmp/mm-api | grep GOFIPS140 # must show GOFIPS140=v1.0.0 25# Gateway + tools (M4 target): 26CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/megamind-gateway ./cmd/megamind-gateway 27CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/provision-tenant ./cmd/provision-tenant 28CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/embed-backfill ./cmd/embed-backfill 29# Frontend (unchanged this cutover; rebuild only if you changed it): 30cd frontend && npm run build 31``` 32 33## Step 1 — durable MEGAMIND gateway on M4 (launchd) 34Replaces the staging `nohup` with a managed service. 35``` 36scp /tmp/megamind-gateway joseph@100.94.8.94:/tmp/megamind-gateway 37scp infra/launchd/com.missionmatch.megamind-gateway.plist joseph@100.94.8.94:/tmp/ 38ssh joseph@100.94.8.94 ' 39 pkill -f /tmp/megamind-gateway || true # stop the staging nohup instance 40 sudo cp /tmp/megamind-gateway /usr/local/bin/megamind-gateway && chmod +x /usr/local/bin/megamind-gateway 41 cp /tmp/com.missionmatch.megamind-gateway.plist ~/Library/LaunchAgents/ 42 launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist 2>/dev/null || true 43 launchctl load ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist 44' 45# Verify: 46curl -s http://100.94.8.94:8090/healthz # -> ok 47curl -s http://100.94.8.94:8090/v1/embed -d '{"inputs":["x"]}' | jq .dimension # -> 384 48``` 49Rollback: `ssh joseph@100.94.8.94 'launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist'`. 50 51Optional quality upgrade (no migration, both 384): pull bge-small and set `MEGAMIND_EMBED_MODEL`: 52`ssh joseph@100.94.8.94 'ollama pull <bge-small-en-v1.5-gguf>'` then re run the backfill (Step 4). 53 54## Step 2 — env on Bubbles (operator; root owned 0600 files) 55Add the AI plane variables to `/etc/missionmatch/api.env` (back it up first): 56``` 57sudo cp /etc/missionmatch/api.env /etc/missionmatch/api.env.bak.$(date +%s) 58# append (do not echo secrets to a log): 59MM_EMBED_PROVIDER=megamind 60MM_MEGAMIND_URL=http://100.94.8.94:8090 61MM_EMBED_DIMENSION=384 62MM_MEGAMIND_EMBED_MODEL=all-minilm 63MM_MEGAMIND_GENERATE_MODEL=llama3.1:8b 64MM_MEGAMIND_TEACHING_ENABLED=false 65``` 66Remove the dead keys from `/etc/missionmatch/sam-sync.env` (back it up first): `DATA_GOV_API_KEY`, 67`KAGGLE_USERNAME`, `KAGGLE_KEY`, `HF_TOKEN`. 68Rollback: restore the `.bak` files. 69 70## Step 3 — database (already at v17; only new tenants need provisioning) 71The shared database is at v17. If you maintain it, confirm: 72``` 73ssh joseph@100.94.8.94 'PGURL=... goose -dir db/migrations postgres "$PGURL" status' # 0017 applied 74``` 75For a NEW client (database per tenant), provision a fresh database (this is the irreversible "create a 76live client database" step; run only for a real onboarding): 77``` 78ssh joseph@100.94.8.94 '/usr/local/bin/provision-tenant \ 79 -admin-dsn "host=/tmp user=postgres dbname=postgres sslmode=disable" \ 80 -name tenant_<client_slug>' 81``` 82It applies the full public + eeo migration set, RLS forced, eeo segregated, vector(384) ready, and 83verifies the app role has no grant on `eeo.candidate_demographics`. Rollback (only if the database is 84brand new and unused): `DROP DATABASE tenant_<client_slug>`. 85 86## Step 4 — embed backfill (idempotent) 87For the shared database (the demo org is already done) or a freshly provisioned tenant database, once it 88has candidates: 89``` 90ssh joseph@100.94.8.94 '/usr/local/bin/embed-backfill \ 91 -dsn "postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/<db>?sslmode=disable" \ 92 -megamind http://100.94.8.94:8090 -what all -only-missing -batch 64' 93``` 94`-only-missing` makes it safe to re run. Rollback: `UPDATE skills SET embedding=NULL; UPDATE candidates 95SET resume_embedding=NULL;` then re run (or leave; embeddings are additive). 96 97## Step 5 — THE CUTOVER on Bubbles (operator, irreversible; brief downtime) 98Swap the mm-api binary to the AI build and restart. Back up the current binary first. 99``` 100scp /tmp/mm-api user@bubbles:/tmp/mm-api.new 101ssh user@bubbles ' 102 sudo cp /home/user/missionmatch/mm-api /home/user/mm-api.bak.$(date +%s) # backup 103 sudo systemctl stop mm-api 104 sudo cp /tmp/mm-api.new /home/user/missionmatch/mm-api && sudo chmod +x /home/user/missionmatch/mm-api 105 sudo systemctl start mm-api 106 sleep 2 && systemctl is-active mm-api 107' 108# Verify the live API still answers and FIPS is on: 109curl -s https://missionmatch.thatwebhostingguy.com/healthz # -> ok 110curl -s https://missionmatch.thatwebhostingguy.com/healthz/fips | jq .fips_enabled # -> true 111``` 112**Rollback (fast):** 113``` 114ssh user@bubbles ' 115 sudo systemctl stop mm-api 116 sudo cp /home/user/mm-api.bak.<ts> /home/user/missionmatch/mm-api 117 sudo systemctl start mm-api 118' 119# and restore /etc/missionmatch/api.env.bak.<ts> if the env changes caused the issue. 120``` 121Nginx is **unchanged** by this cutover (the gateway is reached over Tailscale, not through nginx). No 122`nginx -t` or reload is required. The systemd unit `mm-api.service` is unchanged; only the binary and 123the env file change. 124 125## Step 6 — host service hygiene (operator) 126The audit found the `mm-sam-sync.timer` showed no next elapse. The unit is valid (`OnCalendar=*-*-* 12704:00:00`); ensure it is started: 128``` 129ssh user@bubbles 'sudo systemctl daemon-reload && sudo systemctl enable --now mm-sam-sync.timer && systemctl list-timers | grep mm-sam-sync' 130``` 131 132## Activating the AI HTTP surface (a later, separate change) 133The orchestrator, TAG, RAG, and memory are built and tested as a library but are not yet mounted as 134authenticated routes (deferred so this cutover does not change the user facing app). Mounting them is a 135small code change behind the session resolver, RBAC, and the same audit, then a repeat of Step 5. 136 137## Post cutover checks 138- `curl https://missionmatch.thatwebhostingguy.com/healthz/fips` shows `fips_enabled: true`. 139- The MEGAMIND gateway `launchctl list | grep megamind` is running on M4. 140- `make ci-gates` on the build host stays green (backdoor + FIPS). 141- No EEO reach: `go test ./internal/modules/ai/...` (the tripwire) stays green in CI. 142 143## W3 AI Staffing Recommendations cutover (additive, on top of the AI layer) 144 145W3 is a backend addition: migration 0018, four new authenticated routes already mounted in the mm-api 146binary, and one new scheduled job. It is additive and reversible. As with the AI layer, the operator runs 147the Bubbles and production database steps; everything below was built and validated on M4 staging 148(migration 0018 applies cleanly through v18 plus eeo, and 5 of 5 W3 integration tests pass). 149 150### State at handoff (already done, staging) 151- Migration 0018 (recommendation_runs, staffing_recommendations) is written and proven on a fresh M4 152 staging database `tenant_w3_staging` (goose v18). It is NOT applied to the production shared database. 153- The mm-api binary on the build host now carries the four W3 routes (staffing.recommend gated), but the 154 Bubbles mm-api is unchanged (still the pre AI binary), so nothing is live yet. 155- The new `staffing-proactive-sync` binary and `mm-staffing-proactive.{service,timer}` are written but NOT 156 installed on Bubbles. 157 158### Build the artifacts (build host, no live mutation) 159``` 160cd ~/missionmatch && git checkout w3-ai-staffing-recommendations 161CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/mm-api ./cmd/api 162go version -m /tmp/mm-api | grep GOFIPS140 # must show GOFIPS140=v1.0.0 163CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/staffing-proactive-sync ./cmd/staffing-proactive-sync 164``` 165 166### Step W3.1 - apply migration 0018 to the production shared database (operator, irreversible) 167This is the one irreversible step. 0018 only ADDS two tables and is safe alongside the running old binary 168(the old binary does not use them). Run as the migrator role. 169``` 170ssh joseph@100.94.8.94 'cd ~/missionmatch && \ 171 goose -dir db/migrations postgres "postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/missionmatch?sslmode=disable" up' 172# verify: goose ... status shows 0018 applied, head v18 173``` 174Rollback (only before any recommendation rows matter): `goose ... down` once, or 175`DROP TABLE staffing_recommendations; DROP TABLE recommendation_runs;`. 176 177### Step W3.2 - deploy the mm-api binary that carries the W3 routes (operator, brief downtime) 178Same binary swap as the AI layer Step 5; this build also mounts the four W3 routes behind 179staffing.recommend / slate.write / candidate.read. Back up the current binary first. 180``` 181scp /tmp/mm-api user@bubbles:/tmp/mm-api.new 182ssh user@bubbles ' 183 sudo cp /home/user/missionmatch/mm-api /home/user/mm-api.bak.$(date +%s) 184 sudo systemctl stop mm-api 185 sudo cp /tmp/mm-api.new /home/user/missionmatch/mm-api && sudo chmod +x /home/user/missionmatch/mm-api 186 sudo systemctl start mm-api 187 sleep 2 && systemctl is-active mm-api 188' 189curl -s https://missionmatch.thatwebhostingguy.com/healthz/fips | jq .fips_enabled # -> true 190``` 191Rollback: stop mm-api, restore `/home/user/mm-api.bak.<ts>`, start mm-api. 192 193### Step W3.3 - install the proactive recompute timer (operator) 194``` 195scp /tmp/staffing-proactive-sync user@bubbles:/home/user/missionmatch/staffing-proactive-sync 196ssh user@bubbles ' 197 sudo cp infra/systemd/mm-staffing-proactive.* /etc/systemd/system/ 198 sudo systemctl daemon-reload 199 sudo systemctl enable --now mm-staffing-proactive.timer 200 systemctl start mm-staffing-proactive.service # one off, to seed the first bench watch 201 systemctl list-timers | grep mm-staffing-proactive 202' 203``` 204Rollback: `sudo systemctl disable --now mm-staffing-proactive.timer`. The watch freshness depends on the 205ingestion timers (mm-sam-sync, mm-opp-sync); the proactive timer is scheduled after them at 05:00. 206 207### Post cutover checks (W3) 208- `curl -sf -X POST https://missionmatch.thatwebhostingguy.com/api/missionmatch/recommendations/portfolio` with a 209 valid session returns a 200 allocation (or 403 without staffing.recommend); an unauthenticated call is 403. 210- A staffing run writes one `aedt.staffing.recommended` audit and a decision writes one `aedt.staffing.decided`. 211- `go test -tags=integration -run TestStaffing ./internal/integration/` stays green against the database. 212- The EEO tripwire `go test ./internal/modules/ai/...` stays green (it now walks the staffing package too). 213 214### Note: feed freshness 215The proactive bench watch only surfaces an opportunity once the ingestion timers (mm-sam-sync, mm-opp-sync) 216have ingested it. The audit previously flagged the SAM sync timer as possibly not firing; confirm 217`systemctl list-timers | grep mm-sam-sync` shows a next elapse before relying on the watch for new postings. 218