# MAG learning loop: cmd/train-model and cmd/eval-model > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax. **Goal:** Close the self improving loop on the MAG substrate: a pure Go offline TRAINER that fits interpretable win probability coefficients from the accruing M7 bid_outcomes labels and writes a CHALLENGER version, and a pure Go EVALUATOR that replays a challenger against held out labels and a no adverse impact four fifths gate so a human can promote it to champion. The frozen engine stays the only candidate scorer; this only fits an OPPORTUNITY win probability. **Architecture:** Two `cmd/` tools that read the labels under a migrator scoped read and write ONLY through the migrator role into the MAG coefficient store (aedt_learned_models, aedt_model_coefficients, aedt_calibration_bins). Training is IRLS / Newton Raphson logistic regression (exact for this feature count, converges in under 10 iterations, pure Go), calibration is Platt scaling at cold start. Evaluation replays offline and runs the four fifths regression gate before a HUMAN flips champion. No Python, both cloud, FIPS. **Depends on:** the MAG substrate (Phase 0, branch mag-substrate): the three coefficient tables, IntentOpportunityWinProbability, and the ExecuteModelIntent serving path must exist. This plan is the next task on that branch. **Branch:** `mag-substrate` (continues Phase 0 into the learning loop). Migration: none new (the substrate's 0036 tables are written to, not altered). Validate on M4 `brain_staging` with seeded bid_outcomes labels. --- ## Hard invariants 1. **The trainer NEVER flips champion.** cmd/train-model writes a new (model_key, version) row with champion=false plus its coefficient and calibration rows. Only cmd/eval-model, on a passing eval, and only as an explicit human governance action, flips champion. On any doubt the served model stays the current champion or the prior only v0. 2. **The fitted model scores an OPPORTUNITY, never a candidate.** The label is a bid win or loss (bid_outcomes.result), the unit is a segment or an opportunity, the features are market structure (NONE a protected attribute or proxy). The frozen engine internal/modules/matching is untouched. 3. **Migrator writes, app never.** The coefficient store is app+compliance SELECT only; the cmds write through the migrator role. The trainer reads labels through a migrator scoped read (it aggregates across the tenant's own bid_outcomes; a pooled cross tenant training posture is a founder decision, default per tenant or a documented pooled read with the fairness gate). 4. **The fairness gate is mandatory before promotion.** cmd/eval-model recomputes the eeo.fairness_snapshots four fifths ratios under the challenger and refuses promotion if any group impact ratio drops below 0.80 or regresses from the champion. Reuse the eeo Service and the compliance pool; the eeo schema stays reachable only through the compliance pool. 5. **Events per predictor floor.** The trainer refuses to fit (and logs why) until a segment has enough decided bids for the feature count (roughly 10 to 20 events per predictor, so about 60 to 120 decided win+loss for a 6 feature model). Below the floor the prior only v0 stays champion and the served probability is the shrunk market base rate. 6. Standard: no Python; no dashes in written output except URLs, CLI flags, import paths, identifiers; shopspring/decimal for the surfaced and stored coefficients; FIPS; the FIPS self test gate at cmd boot like cmd/fairness-snapshot. --- ## File Structure - Create `cmd/train-model/main.go`: the FIPS gated trainer. - Create `internal/platform/learn/logistic.go` (+ test): the pure Go IRLS logistic fit and the Platt calibration fit, decimal in the stored output, exhaustively unit tested (this is the numerical core, no DB). - Create `internal/platform/learn/dataset.go` (+ test): assembling the labeled feature matrix from bid_outcomes joined to the segment feature reads, pure over an in memory rowset. - Create `cmd/eval-model/main.go`: the FIPS gated evaluator + the four fifths gate + the human promotion action. - Create `internal/platform/learn/metrics.go` (+ test): held out calibration error (ECE), AUC, and the reliability table, pure functions unit tested. - Create `internal/integration/mag_train_eval_integration_test.go`: the live loop proof. --- ## Phase L.A: the numerical core (pure, no DB) ### Task A1: IRLS logistic regression and Platt calibration **Files:** - Create: `internal/platform/learn/logistic.go`, `internal/platform/learn/logistic_test.go` - [ ] **Step 1: Write the failing tests.** A separable and a noisy fixture with a KNOWN closed form or reference coefficient set: assert FitLogistic(X, y, l2) returns coefficients that classify the fixture, that a higher l2 shrinks the coefficients toward zero, that it converges in under 25 iterations, and that it is deterministic (same input same output). A Platt fit test: FitPlatt(scores, labels) returns a, b that map the raw scores to calibrated probabilities monotonically. - [ ] **Step 2: Implement.** `FitLogistic(x [][]float64, y []int, l2 float64) (coeffs []float64, err error)` by iteratively reweighted least squares (Newton Raphson on the log likelihood with an L2 ridge penalty, a zero mean Gaussian prior, so a thin dataset shrinks toward zero not diverges); solve the weighted normal equations each iteration by Cholesky or Gaussian elimination (a small dense system, feature count under 12). Standardize features (store the means and scales). `FitPlatt(scores []float64, y []int) (a, b float64)` by a 1 D logistic fit of the labels on the raw scores. Internally float64 is fine for the FIT (an offline numerical optimization, not a surfaced score); the OUTPUT coefficients are quantized to the stored numeric(20,10) as decimal. Document that float lives only inside the offline solver, never in the served path. - [ ] **Step 3: Run the tests. Commit.** ### Task A2: the dataset assembly and the metrics **Files:** - Create: `internal/platform/learn/dataset.go` + test, `internal/platform/learn/metrics.go` + test - [ ] **Step 1: dataset.** `BuildDataset(rows []LabeledRow) (x [][]float64, y []int, featureNames []string)` where a LabeledRow is a decided bid (result win=1 loss=0) plus its segment feature values (the same six features IntentOpportunityWinProbability declares). Drop pending and no_bid. Unit test the shape, the win=1/loss=0 mapping, and the feature ordering stability. - [ ] **Step 2: metrics.** `ECE(probs []float64, y []int, bins int) float64` the expected calibration error, `AUC(probs []float64, y []int) float64` the rank AUC, `Reliability(probs, y, bins) []Bin` the reliability table. Unit test each against a hand computed fixture (perfect calibration ECE 0, a perfectly ranked set AUC 1). - [ ] **Step 3: Commit.** --- ## Phase L.B: the trainer ### Task B1: cmd/train-model **Files:** - Create: `cmd/train-model/main.go` - [ ] **Step 1:** FIPS self test gate at boot (mirror cmd/fairness-snapshot). Read MM_MIGRATOR_DATABASE_URL. Flags: `-model opportunity_win_probability` (which MAG model to fit), `-min-events N` (the events per predictor floor, default computed from the feature count). For the model: read the decided bid_outcomes joined to each segment's feature reads (reuse the feature source intents SQL or a direct migrator read), BuildDataset, check the events floor (log and exit 0 without writing if below), FitLogistic with a default l2, FitPlatt on the in sample scores (an out of sample split is the evaluator's job), then WRITE a new challenger: the next version number for the model_key, aedt_learned_models(champion=false, family logistic, output_kind opportunity_score, feature_spec the feature list, notes the training summary), aedt_model_coefficients (intercept + one row per standardized feature, decimal), aedt_calibration_bins from the Platt map. Log the fitted coefficients, the sample size, and the new version. NEVER set champion. - [ ] **Step 2:** Idempotent enough: re running produces a new challenger version (versions are append only, an audit trail of every fit); the champion is unaffected. Commit. --- ## Phase L.C: the evaluator and the fairness gate ### Task C1: cmd/eval-model with the no adverse impact regression gate **Files:** - Create: `cmd/eval-model/main.go` - [ ] **Step 1:** FIPS gate. Read MM_MIGRATOR_DATABASE_URL and MM_COMPLIANCE_DATABASE_URL. Flags: `-model`, `-challenger V`, `-promote` (default false, a dry run report unless set). Replay: hold out a labeled split (the most recent K decided bids, or a fixed fold), score them under BOTH the champion and the challenger coefficients, compute ECE, AUC, and the reliability table for each, and print a comparison. THE GATE: recompute the eeo four fifths ratios under the challenger (reuse the eeo Service through the compliance pool; for a win probability model the fairness surface is the SELECTION the model would drive, so recompute the adverse impact on the cohort the challenger would rank or recommend, conservatively refusing if any group impact ratio drops below 0.80 OR regresses from the champion). Print PASS or FAIL with the numbers. - [ ] **Step 2: promotion.** Only when `-promote` is set AND the eval PASSES (better or equal calibration/AUC AND the fairness gate holds): flip champion in one migrator transaction (set the old champion false, the challenger true, the partial unique index enforces one champion). Log the promotion as an append only audit event. Without `-promote` it is a pure report. Commit. --- ## Phase L.D: the live loop proof on staging ### Task D1: train, eval, promote, serve, end to end **Files:** - Create: `internal/integration/mag_train_eval_integration_test.go` - [ ] **Step 1:** Seed a per run segment with enough decided bid_outcomes (above the events floor) with a learnable signal (say wins correlate with a high incumbent share feature). Run the trainer path (call the trainer's fit + write functions directly, or shell the cmd with the staging DSN), assert a challenger version with non zero learned coefficients appears. Run the evaluator path, assert the challenger's ECE and AUC are computed and the fairness gate reports. Promote, assert champion flips. Then run IntentOpportunityWinProbability through the orchestrator and assert the served probability now reflects the LEARNED coefficients (moved off the prior only base rate toward the signal), with the driver contributions naming the learned features. Assert the frozen engine is untouched. - [ ] **Step 2: Run on staging**, then the full graph suite for no regression, then commit. ```bash GOFIPS140=v1.0.0 CGO_ENABLED=0 go test ./internal/platform/learn/... -v GOFIPS140=v1.0.0 CGO_ENABLED=0 go test -tags=integration -count=1 -run "TestMAGTrain|TestWinProbability|TestLearn" ./internal/integration/... -v ``` --- ## Self review checklist 1. The trainer never sets champion; only eval-model on a passing eval + explicit promotion. 2. The fitted model scores an opportunity; no candidate score; frozen engine byte identical. 3. Migrator writes the store; app never; the fairness gate runs through the compliance pool. 4. The four fifths gate blocks a promotion that regresses fairness. 5. The events per predictor floor keeps the prior only v0 as champion until labels suffice. 6. Float lives only inside the offline solver; every stored and surfaced number is decimal. 7. No Python, no dashes, FIPS, deterministic.