# DEPLOY_PLAN.md — AI Intelligence Layer cutover

The build is validated in a staging posture. This is the exact, ordered, rollback ready plan for the
**human** to run the irreversible live cutover. It is the one manual step: Bubbles serves other live
paying client sites and this build provisions cleared data databases.

**Do not run the Bubbles steps automatically.** Everything below was dry run and validated; the operator
executes the Bubbles and live database steps.

## State at handoff (already done, staging)
- The MEGAMIND gateway is **running on M4** (`100.94.8.94:8090`, Tailscale only, via `nohup`) and smoke
  tested. Ollama on M4 has `all-minilm` (384) and `llama3.1:8b`.
- The shared production database on M4 is at **goose v17**; `0017_ai_memory.sql` is applied (additive, a
  new table the old API binary does not use, so it is safe alongside the running old binary).
- `skills.embedding` (61/61) and the demo org `candidates.resume_embedding` (1511/1511) are backfilled.
- A staging tenant database `tenant_staging_demo` was provisioned on M4 to validate `provision-tenant`.
- The Bubbles `mm-api` service is **unchanged** (still the pre AI binary).

## Build the artifacts (on the build host, no live mutation)
```
cd ~/missionmatch && git checkout ai-intelligence-layer
# API (Bubbles target), FIPS, verify the stamp:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/mm-api ./cmd/api
go version -m /tmp/mm-api | grep GOFIPS140        # must show GOFIPS140=v1.0.0
# Gateway + tools (M4 target):
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/megamind-gateway ./cmd/megamind-gateway
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/provision-tenant ./cmd/provision-tenant
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/embed-backfill ./cmd/embed-backfill
# Frontend (unchanged this cutover; rebuild only if you changed it):
cd frontend && npm run build
```

## Step 1 — durable MEGAMIND gateway on M4 (launchd)
Replaces the staging `nohup` with a managed service.
```
scp /tmp/megamind-gateway joseph@100.94.8.94:/tmp/megamind-gateway
scp infra/launchd/com.missionmatch.megamind-gateway.plist joseph@100.94.8.94:/tmp/
ssh joseph@100.94.8.94 '
  pkill -f /tmp/megamind-gateway || true                 # stop the staging nohup instance
  sudo cp /tmp/megamind-gateway /usr/local/bin/megamind-gateway && chmod +x /usr/local/bin/megamind-gateway
  cp /tmp/com.missionmatch.megamind-gateway.plist ~/Library/LaunchAgents/
  launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist 2>/dev/null || true
  launchctl load ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist
'
# Verify:
curl -s http://100.94.8.94:8090/healthz                  # -> ok
curl -s http://100.94.8.94:8090/v1/embed -d '{"inputs":["x"]}' | jq .dimension   # -> 384
```
Rollback: `ssh joseph@100.94.8.94 'launchctl unload ~/Library/LaunchAgents/com.missionmatch.megamind-gateway.plist'`.

Optional quality upgrade (no migration, both 384): pull bge-small and set `MEGAMIND_EMBED_MODEL`:
`ssh joseph@100.94.8.94 'ollama pull <bge-small-en-v1.5-gguf>'` then re run the backfill (Step 4).

## Step 2 — env on Bubbles (operator; root owned 0600 files)
Add the AI plane variables to `/etc/missionmatch/api.env` (back it up first):
```
sudo cp /etc/missionmatch/api.env /etc/missionmatch/api.env.bak.$(date +%s)
# append (do not echo secrets to a log):
MM_EMBED_PROVIDER=megamind
MM_MEGAMIND_URL=http://100.94.8.94:8090
MM_EMBED_DIMENSION=384
MM_MEGAMIND_EMBED_MODEL=all-minilm
MM_MEGAMIND_GENERATE_MODEL=llama3.1:8b
MM_MEGAMIND_TEACHING_ENABLED=false
```
Remove the dead keys from `/etc/missionmatch/sam-sync.env` (back it up first): `DATA_GOV_API_KEY`,
`KAGGLE_USERNAME`, `KAGGLE_KEY`, `HF_TOKEN`.
Rollback: restore the `.bak` files.

## Step 3 — database (already at v17; only new tenants need provisioning)
The shared database is at v17. If you maintain it, confirm:
```
ssh joseph@100.94.8.94 'PGURL=... goose -dir db/migrations postgres "$PGURL" status'   # 0017 applied
```
For a NEW client (database per tenant), provision a fresh database (this is the irreversible "create a
live client database" step; run only for a real onboarding):
```
ssh joseph@100.94.8.94 '/usr/local/bin/provision-tenant \
  -admin-dsn "host=/tmp user=postgres dbname=postgres sslmode=disable" \
  -name tenant_<client_slug>'
```
It applies the full public + eeo migration set, RLS forced, eeo segregated, vector(384) ready, and
verifies the app role has no grant on `eeo.candidate_demographics`. Rollback (only if the database is
brand new and unused): `DROP DATABASE tenant_<client_slug>`.

## Step 4 — embed backfill (idempotent)
For the shared database (the demo org is already done) or a freshly provisioned tenant database, once it
has candidates:
```
ssh joseph@100.94.8.94 '/usr/local/bin/embed-backfill \
  -dsn "postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/<db>?sslmode=disable" \
  -megamind http://100.94.8.94:8090 -what all -only-missing -batch 64'
```
`-only-missing` makes it safe to re run. Rollback: `UPDATE skills SET embedding=NULL; UPDATE candidates
SET resume_embedding=NULL;` then re run (or leave; embeddings are additive).

## Step 5 — THE CUTOVER on Bubbles (operator, irreversible; brief downtime)
Swap the mm-api binary to the AI build and restart. Back up the current binary first.
```
scp /tmp/mm-api user@bubbles:/tmp/mm-api.new
ssh user@bubbles '
  sudo cp /home/user/missionmatch/mm-api /home/user/mm-api.bak.$(date +%s)   # backup
  sudo systemctl stop mm-api
  sudo cp /tmp/mm-api.new /home/user/missionmatch/mm-api && sudo chmod +x /home/user/missionmatch/mm-api
  sudo systemctl start mm-api
  sleep 2 && systemctl is-active mm-api
'
# Verify the live API still answers and FIPS is on:
curl -s https://missionmatch.thatwebhostingguy.com/healthz        # -> ok
curl -s https://missionmatch.thatwebhostingguy.com/healthz/fips | jq .fips_enabled   # -> true
```
**Rollback (fast):**
```
ssh user@bubbles '
  sudo systemctl stop mm-api
  sudo cp /home/user/mm-api.bak.<ts> /home/user/missionmatch/mm-api
  sudo systemctl start mm-api
'
# and restore /etc/missionmatch/api.env.bak.<ts> if the env changes caused the issue.
```
Nginx is **unchanged** by this cutover (the gateway is reached over Tailscale, not through nginx). No
`nginx -t` or reload is required. The systemd unit `mm-api.service` is unchanged; only the binary and
the env file change.

## Step 6 — host service hygiene (operator)
The audit found the `mm-sam-sync.timer` showed no next elapse. The unit is valid (`OnCalendar=*-*-*
04:00:00`); ensure it is started:
```
ssh user@bubbles 'sudo systemctl daemon-reload && sudo systemctl enable --now mm-sam-sync.timer && systemctl list-timers | grep mm-sam-sync'
```

## Activating the AI HTTP surface (a later, separate change)
The orchestrator, TAG, RAG, and memory are built and tested as a library but are not yet mounted as
authenticated routes (deferred so this cutover does not change the user facing app). Mounting them is a
small code change behind the session resolver, RBAC, and the same audit, then a repeat of Step 5.

## Post cutover checks
- `curl https://missionmatch.thatwebhostingguy.com/healthz/fips` shows `fips_enabled: true`.
- The MEGAMIND gateway `launchctl list | grep megamind` is running on M4.
- `make ci-gates` on the build host stays green (backdoor + FIPS).
- No EEO reach: `go test ./internal/modules/ai/...` (the tripwire) stays green in CI.

## W3 AI Staffing Recommendations cutover (additive, on top of the AI layer)

W3 is a backend addition: migration 0018, four new authenticated routes already mounted in the mm-api
binary, and one new scheduled job. It is additive and reversible. As with the AI layer, the operator runs
the Bubbles and production database steps; everything below was built and validated on M4 staging
(migration 0018 applies cleanly through v18 plus eeo, and 5 of 5 W3 integration tests pass).

### State at handoff (already done, staging)
- Migration 0018 (recommendation_runs, staffing_recommendations) is written and proven on a fresh M4
  staging database `tenant_w3_staging` (goose v18). It is NOT applied to the production shared database.
- The mm-api binary on the build host now carries the four W3 routes (staffing.recommend gated), but the
  Bubbles mm-api is unchanged (still the pre AI binary), so nothing is live yet.
- The new `staffing-proactive-sync` binary and `mm-staffing-proactive.{service,timer}` are written but NOT
  installed on Bubbles.

### Build the artifacts (build host, no live mutation)
```
cd ~/missionmatch && git checkout w3-ai-staffing-recommendations
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/mm-api ./cmd/api
go version -m /tmp/mm-api | grep GOFIPS140                          # must show GOFIPS140=v1.0.0
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o /tmp/staffing-proactive-sync ./cmd/staffing-proactive-sync
```

### Step W3.1 - apply migration 0018 to the production shared database (operator, irreversible)
This is the one irreversible step. 0018 only ADDS two tables and is safe alongside the running old binary
(the old binary does not use them). Run as the migrator role.
```
ssh joseph@100.94.8.94 'cd ~/missionmatch && \
  goose -dir db/migrations postgres "postgres://missionmatch_migrator:<pw>@127.0.0.1:5432/missionmatch?sslmode=disable" up'
# verify: goose ... status  shows 0018 applied, head v18
```
Rollback (only before any recommendation rows matter): `goose ... down` once, or
`DROP TABLE staffing_recommendations; DROP TABLE recommendation_runs;`.

### Step W3.2 - deploy the mm-api binary that carries the W3 routes (operator, brief downtime)
Same binary swap as the AI layer Step 5; this build also mounts the four W3 routes behind
staffing.recommend / slate.write / candidate.read. Back up the current binary first.
```
scp /tmp/mm-api user@bubbles:/tmp/mm-api.new
ssh user@bubbles '
  sudo cp /home/user/missionmatch/mm-api /home/user/mm-api.bak.$(date +%s)
  sudo systemctl stop mm-api
  sudo cp /tmp/mm-api.new /home/user/missionmatch/mm-api && sudo chmod +x /home/user/missionmatch/mm-api
  sudo systemctl start mm-api
  sleep 2 && systemctl is-active mm-api
'
curl -s https://missionmatch.thatwebhostingguy.com/healthz/fips | jq .fips_enabled   # -> true
```
Rollback: stop mm-api, restore `/home/user/mm-api.bak.<ts>`, start mm-api.

### Step W3.3 - install the proactive recompute timer (operator)
```
scp /tmp/staffing-proactive-sync user@bubbles:/home/user/missionmatch/staffing-proactive-sync
ssh user@bubbles '
  sudo cp infra/systemd/mm-staffing-proactive.* /etc/systemd/system/
  sudo systemctl daemon-reload
  sudo systemctl enable --now mm-staffing-proactive.timer
  systemctl start mm-staffing-proactive.service      # one off, to seed the first bench watch
  systemctl list-timers | grep mm-staffing-proactive
'
```
Rollback: `sudo systemctl disable --now mm-staffing-proactive.timer`. The watch freshness depends on the
ingestion timers (mm-sam-sync, mm-opp-sync); the proactive timer is scheduled after them at 05:00.

### Post cutover checks (W3)
- `curl -sf -X POST https://missionmatch.thatwebhostingguy.com/api/missionmatch/recommendations/portfolio` with a
  valid session returns a 200 allocation (or 403 without staffing.recommend); an unauthenticated call is 403.
- A staffing run writes one `aedt.staffing.recommended` audit and a decision writes one `aedt.staffing.decided`.
- `go test -tags=integration -run TestStaffing ./internal/integration/` stays green against the database.
- The EEO tripwire `go test ./internal/modules/ai/...` stays green (it now walks the staffing package too).

### Note: feed freshness
The proactive bench watch only surfaces an opportunity once the ingestion timers (mm-sam-sync, mm-opp-sync)
have ingested it. The audit previously flagged the SAM sync timer as possibly not firing; confirm
`systemctl list-timers | grep mm-sam-sync` shows a next elapse before relying on the watch for new postings.