Janady13/missionmatch / DEPLOY_PLAN.md

1# DEPLOY_PLAN.md — AI Intelligence Layer cutover 2 3The build is validated in a staging posture. This is the exact, ordered, rollback ready plan for the 4**human** to run the irreversible live cutover. It is the one manual step: Bubbles serves other live 5paying client sites and this build provisions cleared data databases. 6 7**Do not run the Bubbles steps automatically.** Everything below was dry run and validated; the operator 8executes the Bubbles and live database steps. 9 10## State at handoff (already done, staging) 11- The MEGAMIND gateway is **running on M4** (`100.94.8.94:8090`, Tailscale only, via `nohup`) and smoke 12 tested. Ollama on M4 has `all-minilm` (384) and `llama3.1:8b`. 13- The shared production database on M4 is at **goose v17**; `0017_ai_memory.sql` is applied (additive, a 14 new table the old API binary does not use, so it is safe alongside the running old binary). 15- `skills.embedding` (61/61) and the demo org `candidates.resume_embedding` (1511/1511) are backfilled. 16- A staging tenant database `tenant_staging_demo` was provisioned on M4 to validate `provision-tenant`. 17- The Bubbles `mm-api` service is **unchanged** (still the pre AI binary). 18 19## Build the artifacts (on the build host, no live mutation) 20``` 21cd ~/missionmatch && git checkout ai-intelligence-layer 22# API (Bubbles target), FIPS, verify the stamp: 23CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/mm-api ./cmd/api 24go version -m /tmp/mm-api | grep GOFIPS140 # must show GOFIPS140=v1.0.0 25# Gateway + tools (M4 target): 26CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/megamind-gateway ./cmd/megamind-gateway 27CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/provision-tenant ./cmd/provision-tenant 28CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/embed-backfill ./cmd/embed-backfill 29# Frontend (unchanged this cutover; rebuild only if you changed it): 30cd frontend && npm run build 31``` 32 33## Step 1 — durable MEGAMIND gateway on M4 (launchd) 34Replaces the staging `nohup` with a managed service. 35``` 36scp /tmp/megamind-gateway joseph@100.94.8.94:/tmp/megamind-gateway 37scp infra/launchd/com.missionmatch.megamind-gateway.plist joseph@100.94.8.94:/tmp/ 38ssh joseph@100.94.8.94 ' 39 pkill -f /tmp/megamind-gateway || true # stop the staging nohup instance 40 sudo cp /tmp/megamind-gateway /usr/local/bin/megamind-gateway && chmod +x /usr/local/bin/megamind-gateway 41 cp /tmp/com.missionmatch.megamind-gateway.plist ~/Library/LaunchAgents/ 42 launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist 2>/dev/null || true 43 launchctl load ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist 44' 45# Verify: 46curl -s http://100.94.8.94:8090/healthz # -> ok 47curl -s http://100.94.8.94:8090/v1/embed -d '{"inputs":["x"]}' | jq .dimension # -> 384 48``` 49Rollback: `ssh joseph@100.94.8.94 'launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist'`. 50 51Optional quality upgrade (no migration, both 384): pull bge-small and set `MEGAMIND_EMBED_MODEL`: 52`ssh joseph@100.94.8.94 'ollama pull <bge-small-en-v1.5-gguf>'` then re run the backfill (Step 4). 53 54## Step 2 — env on Bubbles (operator; root owned 0600 files) 55Add the AI plane variables to `/etc/missionmatch/api.env` (back it up first): 56``` 57sudo cp /etc/missionmatch/api.env /etc/missionmatch/api.env.bak.$(date +%s) 58# append (do not echo secrets to a log): 59MM_EMBED_PROVIDER=megamind 60MM_MEGAMIND_URL=http://100.94.8.94:8090 61MM_EMBED_DIMENSION=384 62MM_MEGAMIND_EMBED_MODEL=all-minilm 63MM_MEGAMIND_GENERATE_MODEL=llama3.1:8b 64MM_MEGAMIND_TEACHING_ENABLED=false 65``` 66Remove the dead keys from `/etc/missionmatch/sam-sync.env` (back it up first): `DATA_GOV_API_KEY`, 67`KAGGLE_USERNAME`, `KAGGLE_KEY`, `HF_TOKEN`. 68Rollback: restore the `.bak` files. 69 70## Step 3 — database (already at v17; only new tenants need provisioning) 71The shared database is at v17. If you maintain it, confirm: 72``` 73ssh joseph@100.94.8.94 'PGURL=... goose -dir db/migrations postgres "$PGURL" status' # 0017 applied 74``` 75For a NEW client (database per tenant), provision a fresh database (this is the irreversible "create a 76live client database" step; run only for a real onboarding): 77``` 78ssh joseph@100.94.8.94 '/usr/local/bin/provision-tenant \ 79 -admin-dsn "host=/tmp user=postgres dbname=postgres sslmode=disable" \ 80 -name tenant_<client_slug>' 81``` 82It applies the full public + eeo migration set, RLS forced, eeo segregated, vector(384) ready, and 83verifies the app role has no grant on `eeo.candidate_demographics`. Rollback (only if the database is 84brand new and unused): `DROP DATABASE tenant_<client_slug>`. 85 86## Step 4 — embed backfill (idempotent) 87For the shared database (the demo org is already done) or a freshly provisioned tenant database, once it 88has candidates: 89``` 90ssh joseph@100.94.8.94 '/usr/local/bin/embed-backfill \ 91 -dsn "postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/<db>?sslmode=disable" \ 92 -megamind http://100.94.8.94:8090 -what all -only-missing -batch 64' 93``` 94`-only-missing` makes it safe to re run. Rollback: `UPDATE skills SET embedding=NULL; UPDATE candidates 95SET resume_embedding=NULL;` then re run (or leave; embeddings are additive). 96 97## Step 5 — THE CUTOVER on Bubbles (operator, irreversible; brief downtime) 98Swap the mm-api binary to the AI build and restart. Back up the current binary first. 99``` 100scp /tmp/mm-api user@bubbles:/tmp/mm-api.new 101ssh user@bubbles ' 102 sudo cp /home/user/missionmatch/mm-api /home/user/mm-api.bak.$(date +%s) # backup 103 sudo systemctl stop mm-api 104 sudo cp /tmp/mm-api.new /home/user/missionmatch/mm-api && sudo chmod +x /home/user/missionmatch/mm-api 105 sudo systemctl start mm-api 106 sleep 2 && systemctl is-active mm-api 107' 108# Verify the live API still answers and FIPS is on: 109curl -s https://missionmatch.thatwebhostingguy.com/healthz # -> ok 110curl -s https://missionmatch.thatwebhostingguy.com/healthz/fips | jq .fips_enabled # -> true 111``` 112**Rollback (fast):** 113``` 114ssh user@bubbles ' 115 sudo systemctl stop mm-api 116 sudo cp /home/user/mm-api.bak.<ts> /home/user/missionmatch/mm-api 117 sudo systemctl start mm-api 118' 119# and restore /etc/missionmatch/api.env.bak.<ts> if the env changes caused the issue. 120``` 121Nginx is **unchanged** by this cutover (the gateway is reached over Tailscale, not through nginx). No 122`nginx -t` or reload is required. The systemd unit `mm-api.service` is unchanged; only the binary and 123the env file change. 124 125## Step 6 — host service hygiene (operator) 126The audit found the `mm-sam-sync.timer` showed no next elapse. The unit is valid (`OnCalendar=*-*-* 12704:00:00`); ensure it is started: 128``` 129ssh user@bubbles 'sudo systemctl daemon-reload && sudo systemctl enable --now mm-sam-sync.timer && systemctl list-timers | grep mm-sam-sync' 130``` 131 132## Activating the AI HTTP surface (a later, separate change) 133The orchestrator, TAG, RAG, and memory are built and tested as a library but are not yet mounted as 134authenticated routes (deferred so this cutover does not change the user facing app). Mounting them is a 135small code change behind the session resolver, RBAC, and the same audit, then a repeat of Step 5. 136 137## Post cutover checks 138- `curl https://missionmatch.thatwebhostingguy.com/healthz/fips` shows `fips_enabled: true`. 139- The MEGAMIND gateway `launchctl list | grep megamind` is running on M4. 140- `make ci-gates` on the build host stays green (backdoor + FIPS). 141- No EEO reach: `go test ./internal/modules/ai/...` (the tripwire) stays green in CI. 142