# DEPLOY_PLAN.md — AI Intelligence Layer cutover

The build is validated in a staging posture. This is the exact, ordered, rollback ready plan for the
**human** to run the irreversible live cutover. It is the one manual step: Bubbles serves other live
paying client sites and this build provisions cleared data databases.

**Do not run the Bubbles steps automatically.** Everything below was dry run and validated; the operator
executes the Bubbles and live database steps.

## State at handoff (already done, staging)
- The MEGAMIND gateway is **running on M4** (`100.94.8.94:8090`, Tailscale only, via `nohup`) and smoke
  tested. Ollama on M4 has `all-minilm` (384) and `llama3.1:8b`.
- The shared production database on M4 is at **goose v17**; `0017_ai_memory.sql` is applied (additive, a
  new table the old API binary does not use, so it is safe alongside the running old binary).
- `skills.embedding` (61/61) and the demo org `candidates.resume_embedding` (1511/1511) are backfilled.
- A staging tenant database `tenant_staging_demo` was provisioned on M4 to validate `provision-tenant`.
- The Bubbles `mm-api` service is **unchanged** (still the pre AI binary).

## Build the artifacts (on the build host, no live mutation)
```
cd ~/missionmatch && git checkout ai-intelligence-layer
# API (Bubbles target), FIPS, verify the stamp:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/mm-api ./cmd/api
go version -m /tmp/mm-api | grep GOFIPS140        # must show GOFIPS140=v1.0.0
# Gateway + tools (M4 target):
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/megamind-gateway ./cmd/megamind-gateway
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/provision-tenant ./cmd/provision-tenant
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/embed-backfill ./cmd/embed-backfill
# Frontend (unchanged this cutover; rebuild only if you changed it):
cd frontend && npm run build
```

## Step 1 — durable MEGAMIND gateway on M4 (launchd)
Replaces the staging `nohup` with a managed service.
```
scp /tmp/megamind-gateway joseph@100.94.8.94:/tmp/megamind-gateway
scp infra/launchd/com.missionmatch.megamind-gateway.plist joseph@100.94.8.94:/tmp/
ssh joseph@100.94.8.94 '
  pkill -f /tmp/megamind-gateway || true                 # stop the staging nohup instance
  sudo cp /tmp/megamind-gateway /usr/local/bin/megamind-gateway && chmod +x /usr/local/bin/megamind-gateway
  cp /tmp/com.missionmatch.megamind-gateway.plist ~/Library/LaunchAgents/
  launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist 2>/dev/null || true
  launchctl load ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist
'
# Verify:
curl -s http://100.94.8.94:8090/healthz                  # -> ok
curl -s http://100.94.8.94:8090/v1/embed -d '{"inputs":["x"]}' | jq .dimension   # -> 384
```
Rollback: `ssh joseph@100.94.8.94 'launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist'`.

Optional quality upgrade (no migration, both 384): pull bge-small and set `MEGAMIND_EMBED_MODEL`:
`ssh joseph@100.94.8.94 'ollama pull <bge-small-en-v1.5-gguf>'` then re run the backfill (Step 4).

## Step 2 — env on Bubbles (operator; root owned 0600 files)
Add the AI plane variables to `/etc/missionmatch/api.env` (back it up first):
```
sudo cp /etc/missionmatch/api.env /etc/missionmatch/api.env.bak.$(date +%s)
# append (do not echo secrets to a log):
MM_EMBED_PROVIDER=megamind
MM_MEGAMIND_URL=http://100.94.8.94:8090
MM_EMBED_DIMENSION=384
MM_MEGAMIND_EMBED_MODEL=all-minilm
MM_MEGAMIND_GENERATE_MODEL=llama3.1:8b
MM_MEGAMIND_TEACHING_ENABLED=false
```
Remove the dead keys from `/etc/missionmatch/sam-sync.env` (back it up first): `DATA_GOV_API_KEY`,
`KAGGLE_USERNAME`, `KAGGLE_KEY`, `HF_TOKEN`.
Rollback: restore the `.bak` files.

## Step 3 — database (already at v17; only new tenants need provisioning)
The shared database is at v17. If you maintain it, confirm:
```
ssh joseph@100.94.8.94 'PGURL=... goose -dir db/migrations postgres "$PGURL" status'   # 0017 applied
```
For a NEW client (database per tenant), provision a fresh database (this is the irreversible "create a
live client database" step; run only for a real onboarding):
```
ssh joseph@100.94.8.94 '/usr/local/bin/provision-tenant \
  -admin-dsn "host=/tmp user=postgres dbname=postgres sslmode=disable" \
  -name tenant_<client_slug>'
```
It applies the full public + eeo migration set, RLS forced, eeo segregated, vector(384) ready, and
verifies the app role has no grant on `eeo.candidate_demographics`. Rollback (only if the database is
brand new and unused): `DROP DATABASE tenant_<client_slug>`.

## Step 4 — embed backfill (idempotent)
For the shared database (the demo org is already done) or a freshly provisioned tenant database, once it
has candidates:
```
ssh joseph@100.94.8.94 '/usr/local/bin/embed-backfill \
  -dsn "postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/<db>?sslmode=disable" \
  -megamind http://100.94.8.94:8090 -what all -only-missing -batch 64'
```
`-only-missing` makes it safe to re run. Rollback: `UPDATE skills SET embedding=NULL; UPDATE candidates
SET resume_embedding=NULL;` then re run (or leave; embeddings are additive).

## Step 5 — THE CUTOVER on Bubbles (operator, irreversible; brief downtime)
Swap the mm-api binary to the AI build and restart. Back up the current binary first.
```
scp /tmp/mm-api user@bubbles:/tmp/mm-api.new
ssh user@bubbles '
  sudo cp /home/user/missionmatch/mm-api /home/user/mm-api.bak.$(date +%s)   # backup
  sudo systemctl stop mm-api
  sudo cp /tmp/mm-api.new /home/user/missionmatch/mm-api && sudo chmod +x /home/user/missionmatch/mm-api
  sudo systemctl start mm-api
  sleep 2 && systemctl is-active mm-api
'
# Verify the live API still answers and FIPS is on:
curl -s https://missionmatch.thatwebhostingguy.com/healthz        # -> ok
curl -s https://missionmatch.thatwebhostingguy.com/healthz/fips | jq .fips_enabled   # -> true
```
**Rollback (fast):**
```
ssh user@bubbles '
  sudo systemctl stop mm-api
  sudo cp /home/user/mm-api.bak.<ts> /home/user/missionmatch/mm-api
  sudo systemctl start mm-api
'
# and restore /etc/missionmatch/api.env.bak.<ts> if the env changes caused the issue.
```
Nginx is **unchanged** by this cutover (the gateway is reached over Tailscale, not through nginx). No
`nginx -t` or reload is required. The systemd unit `mm-api.service` is unchanged; only the binary and
the env file change.

## Step 6 — host service hygiene (operator)
The audit found the `mm-sam-sync.timer` showed no next elapse. The unit is valid (`OnCalendar=*-*-*
04:00:00`); ensure it is started:
```
ssh user@bubbles 'sudo systemctl daemon-reload && sudo systemctl enable --now mm-sam-sync.timer && systemctl list-timers | grep mm-sam-sync'
```

## Activating the AI HTTP surface (a later, separate change)
The orchestrator, TAG, RAG, and memory are built and tested as a library but are not yet mounted as
authenticated routes (deferred so this cutover does not change the user facing app). Mounting them is a
small code change behind the session resolver, RBAC, and the same audit, then a repeat of Step 5.

## Post cutover checks
- `curl https://missionmatch.thatwebhostingguy.com/healthz/fips` shows `fips_enabled: true`.
- The MEGAMIND gateway `launchctl list | grep megamind` is running on M4.
- `make ci-gates` on the build host stays green (backdoor + FIPS).
- No EEO reach: `go test ./internal/modules/ai/...` (the tripwire) stays green in CI.