Source identity
| Field | Recorded value |
|---|---|
| Source type | GitHub source |
| Original record | Janady13/missionmatch / docs/RUNBOOK_AIOE.md |
| Date | Not specified in source |
| Repository | Janady13/missionmatch |
| Revision | 720d47fae476b0d38477ae4401697d372b1ea2bc |
| Branch | feature/opportunity-to-outcome |
| Path or archive member | docs/RUNBOOK_AIOE.md |
| Original SHA-256 | 2378c43bd527aef8e4535a571751ff007f5f39143dc92472f446678594e955c4 |
| Git blob SHA | 987cbe13cf5f48cdbf70576cd8d17358df65b4f5 |
| Reading-copy SHA-256 | 2378c43bd527aef8e4535a571751ff007f5f39143dc92472f446678594e955c4 |
Every recorded occurrence
These 2 records contain identical original source bytes. Their paths and revisions remain part of the archive.
| Original record | Branch or archive | Revision / identifier | Original location |
|---|---|---|---|
| Janady13/missionmatch / docs/RUNBOOK_AIOE.md | feature/opportunity-to-outcome | 720d47fae476b0d38477ae4401697d372b1ea2bc | Open original |
| Janady13/missionmatch / docs/RUNBOOK_AIOE.md | feature/aioe-upgrade | 2178e7298d0e5d9af83afa12402fd587f0ba6ed8 | Open original |
Recovered source transcript
Download complete source text ↓84 lines · 5.4 KB
# AIOE Operations Runbook (Adaptive Intelligence Optimization Engine)
The closed loop beneath every module: outcome_events in, bounded weight profile updates out,
per tenant, on a deterministic batch. Customers see improved recommendations, never the mechanism.
## Roles and isolation
- missionmatch_aioe is provisioned OUT OF BAND (the migrator has no CREATEROLE, on purpose):
CREATE ROLE missionmatch_aioe LOGIN NOBYPASSRLS PASSWORD '...';
Migration 0043 grants it outcome_events (SELECT), aioe_weight_profiles (SELECT, INSERT, UPDATE),
aioe_weight_adjustments (SELECT, INSERT), aioe_runs (SELECT, INSERT), and NOTHING else.
- The integration proof: TestAioeRoleIsDeniedEverywhereButItsOwnTables (42501 on eeo, candidates,
evaluations, users, auth_sessions, slate members, audit_logs, organizations) and
TestAioeClosedLoop (capture, update, audit rows, scorer pickup) with MM_AIOE_DSN set.
## The nightly batch
- Binary: cmd/aioe-run (FIPS build like every release binary). Env: MM_AIOE_DATABASE_URL.
- Host timer (the alert-sweep pattern), for example:
15 5 * * * MM_AIOE_DATABASE_URL=... /opt/missionmatch/aioe-run >> /var/log/mm/aioe-run.log 2>&1
- On demand: POST /api/aioe/run (admin role) on a deployment with MM_AIOE_DATABASE_URL set.
- Bounds (internal/aioe DefaultParams): eta 0.10, beta 0.30, max step 0.02 per dimension per
cycle, floor 0.05, ceiling 0.40, minimum 25 outcomes in the 90 day window. Same inputs always
produce the same outputs; a rerun inside one UTC day consumes the identical window.
- Nothing changes silently: every dimension change is one aioe_weight_adjustments row naming the
old value, the new value, the window, the outcome count, and the rationale.
## API surface
- POST /api/outcomes (candidate.write): the one click feedback capture. The service denormalizes
the served dimension scores into event metadata so the updater consumes outcome_events only.
- GET /api/aioe/status, GET /api/aioe/history, POST /api/aioe/run: admin role only.
## Staging (built and verified 2026-07-09)
- Database: missionmatch_staging on the M4 Postgres, goose v44 plus the eeo dir at v2 under the
separate table goose_db_version_eeo. Migrations proven up, down, up.
- API: mm-api-staging.service on Bubbles 127.0.0.1:8097 (8095 belongs to Atlas, 8094 to prod),
env /etc/missionmatch/staging.env, secret file /opt/missionmatch-staging/secrets.env (0400,
owned by the service user; /etc/missionmatch is not traversable by www-data).
- Frontend: /var/www/sites/missionmatch-staging behind
missionmatch-staging.thatwebhostingguy.com (wildcard cert). Every nginx change ends with
nginx -t && systemctl reload nginx.
- Inference plane: a SECOND authenticated gateway on the M4 at 100.94.8.94:8091 (Tailscale bound)
serves staging. The prod gateway on 8090 is untouched and still open; flipping prod to the
authenticated build is a cutover step (below).
## Pre production training (the knowledge build)
The engine gets smarter before the first client along four axes that carry NO bias liability,
because they make the SYSTEM smarter, not a tenant's weights pre biased. Every client still starts
on global.default; their own outcomes adapt it after the 25 event threshold.
1. Vocabulary (migration 0045): 204 global acronym and term expansions the acronym resolver reads,
deterministic, no statistical weights. Extend with more curated unambiguous pairs any time.
2. Ontology (db/seed/cleared_skills_growth.sql): 174 curated cleared skills, 37 aliases, 26
relationships. Apply with the base seeds via psql, then promote-ontology and embed-backfill
(the latter now needs -token or MM_MEGAMIND_TOKEN for the authenticated gateway).
3. Backfill (cmd/aioe-backfill): convert existing slate decisions, vendor submissions, and bid
outcomes into outcome_events so the ledger starts with real attributed evidence. Idempotent.
4. Calibration (cmd/aioe-sim): characterize the updater under synthetic regimes for the model card
(docs/AIOE_CALIBRATION.txt). Pure, no client data; validation only, never shipped weights.
## Shadow mode (the pilot posture)
Set MM_AIOE_SHADOW=true on the API and cmd/aioe-run. A run then proposes a CANDIDATE weight profile
per org WITHOUT flipping the active scorer (which stays global.default until promotion). The
Continuous Learning tile shows the pending candidate and, for admins, a Promote button; the API is
POST /api/aioe/promote (admin, scoped to the caller's org through the aioe pool). This is how a
design partner accrues weeks of real signal before launch and a human, not the engine, decides when
a tenant's weights change. Turn shadow OFF only when a tenant is ready for continuous adaptation.
## Production cutover (HUMAN OPERATOR, not automated)
1. Provision missionmatch_aioe LOGIN on the prod cluster; add MM_AIOE_DATABASE_URL to prod env.
2. goose up (0043, 0044) on the prod database. WARNING: the mm-growth working copy on Bubbles
holds an uncommitted migration numbered 0043 (solicitation_extractions, applied to prod as raw
DDL); renumber it to 0045 before it ever enters the repo.
3. Mint MEGAMIND_GATEWAY_TOKEN into the M4 secret file (~/.megamind-staging-secrets.env pattern,
0600); add MM_MEGAMIND_TOKEN + MM_SECRETS_FILE to prod env; restart the prod gateway through
ops/m4/gateway-backend.sh (now fails closed without the secret file and binds Tailscale).
4. Deploy the new mm-api build; schedule aioe-run on the host timer.
5. Make the build-test-gates workflow required on the default branch (branch protection).