# AIOE Operations Runbook (Adaptive Intelligence Optimization Engine)

The closed loop beneath every module: outcome_events in, bounded weight profile updates out,
per tenant, on a deterministic batch. Customers see improved recommendations, never the mechanism.

## Roles and isolation

- missionmatch_aioe is provisioned OUT OF BAND (the migrator has no CREATEROLE, on purpose):
  CREATE ROLE missionmatch_aioe LOGIN NOBYPASSRLS PASSWORD '...';
  Migration 0043 grants it outcome_events (SELECT), aioe_weight_profiles (SELECT, INSERT, UPDATE),
  aioe_weight_adjustments (SELECT, INSERT), aioe_runs (SELECT, INSERT), and NOTHING else.
- The integration proof: TestAioeRoleIsDeniedEverywhereButItsOwnTables (42501 on eeo, candidates,
  evaluations, users, auth_sessions, slate members, audit_logs, organizations) and
  TestAioeClosedLoop (capture, update, audit rows, scorer pickup) with MM_AIOE_DSN set.

## The nightly batch

- Binary: cmd/aioe-run (FIPS build like every release binary). Env: MM_AIOE_DATABASE_URL.
- Host timer (the alert-sweep pattern), for example:
  15 5 * * *  MM_AIOE_DATABASE_URL=... /opt/missionmatch/aioe-run >> /var/log/mm/aioe-run.log 2>&1
- On demand: POST /api/aioe/run (admin role) on a deployment with MM_AIOE_DATABASE_URL set.
- Bounds (internal/aioe DefaultParams): eta 0.10, beta 0.30, max step 0.02 per dimension per
  cycle, floor 0.05, ceiling 0.40, minimum 25 outcomes in the 90 day window. Same inputs always
  produce the same outputs; a rerun inside one UTC day consumes the identical window.
- Nothing changes silently: every dimension change is one aioe_weight_adjustments row naming the
  old value, the new value, the window, the outcome count, and the rationale.

## API surface

- POST /api/outcomes (candidate.write): the one click feedback capture. The service denormalizes
  the served dimension scores into event metadata so the updater consumes outcome_events only.
- GET /api/aioe/status, GET /api/aioe/history, POST /api/aioe/run: admin role only.

## Staging (built and verified 2026-07-09)

- Database: missionmatch_staging on the M4 Postgres, goose v44 plus the eeo dir at v2 under the
  separate table goose_db_version_eeo. Migrations proven up, down, up.
- API: mm-api-staging.service on Bubbles 127.0.0.1:8097 (8095 belongs to Atlas, 8094 to prod),
  env /etc/missionmatch/staging.env, secret file /opt/missionmatch-staging/secrets.env (0400,
  owned by the service user; /etc/missionmatch is not traversable by www-data).
- Frontend: /var/www/sites/missionmatch-staging behind
  missionmatch-staging.thatwebhostingguy.com (wildcard cert). Every nginx change ends with
  nginx -t && systemctl reload nginx.
- Inference plane: a SECOND authenticated gateway on the M4 at 100.94.8.94:8091 (Tailscale bound)
  serves staging. The prod gateway on 8090 is untouched and still open; flipping prod to the
  authenticated build is a cutover step (below).

## Pre production training (the knowledge build)

The engine gets smarter before the first client along four axes that carry NO bias liability,
because they make the SYSTEM smarter, not a tenant's weights pre biased. Every client still starts
on global.default; their own outcomes adapt it after the 25 event threshold.

1. Vocabulary (migration 0045): 204 global acronym and term expansions the acronym resolver reads,
   deterministic, no statistical weights. Extend with more curated unambiguous pairs any time.
2. Ontology (db/seed/cleared_skills_growth.sql): 174 curated cleared skills, 37 aliases, 26
   relationships. Apply with the base seeds via psql, then promote-ontology and embed-backfill
   (the latter now needs -token or MM_MEGAMIND_TOKEN for the authenticated gateway).
3. Backfill (cmd/aioe-backfill): convert existing slate decisions, vendor submissions, and bid
   outcomes into outcome_events so the ledger starts with real attributed evidence. Idempotent.
4. Calibration (cmd/aioe-sim): characterize the updater under synthetic regimes for the model card
   (docs/AIOE_CALIBRATION.txt). Pure, no client data; validation only, never shipped weights.

## Shadow mode (the pilot posture)

Set MM_AIOE_SHADOW=true on the API and cmd/aioe-run. A run then proposes a CANDIDATE weight profile
per org WITHOUT flipping the active scorer (which stays global.default until promotion). The
Continuous Learning tile shows the pending candidate and, for admins, a Promote button; the API is
POST /api/aioe/promote (admin, scoped to the caller's org through the aioe pool). This is how a
design partner accrues weeks of real signal before launch and a human, not the engine, decides when
a tenant's weights change. Turn shadow OFF only when a tenant is ready for continuous adaptation.

## Production cutover (HUMAN OPERATOR, not automated)

1. Provision missionmatch_aioe LOGIN on the prod cluster; add MM_AIOE_DATABASE_URL to prod env.
2. goose up (0043, 0044) on the prod database. WARNING: the mm-growth working copy on Bubbles
   holds an uncommitted migration numbered 0043 (solicitation_extractions, applied to prod as raw
   DDL); renumber it to 0045 before it ever enters the repo.
3. Mint MEGAMIND_GATEWAY_TOKEN into the M4 secret file (~/.megamind-staging-secrets.env pattern,
   0600); add MM_MEGAMIND_TOKEN + MM_SECRETS_FILE to prod env; restart the prod gateway through
   ops/m4/gateway-backend.sh (now fails closed without the secret file and binds Tailscale).
4. Deploy the new mm-api build; schedule aioe-run on the host timer.
5. Make the build-test-gates workflow required on the default branch (branch protection).