Source identity
| Field | Recorded value |
|---|---|
| Source type | GitHub source |
| Original record | Janady13/missionmatch / docs/plans/2026-06-29-phase2-eval-fairness.md |
| Date | Not specified in source |
| Repository | Janady13/missionmatch |
| Revision | 720d47fae476b0d38477ae4401697d372b1ea2bc |
| Branch | feature/opportunity-to-outcome |
| Path or archive member | docs/plans/2026-06-29-phase2-eval-fairness.md |
| Original SHA-256 | c9dac8af69b73b67bb64ca540938a80eed446ddfad51eb2dbce339e3767520e0 |
| Git blob SHA | 4047b77bb61caa34a5c40add42783f37eb91b133 |
| Reading-copy SHA-256 | c9dac8af69b73b67bb64ca540938a80eed446ddfad51eb2dbce339e3767520e0 |
Every recorded occurrence
These 3 records contain identical original source bytes. Their paths and revisions remain part of the archive.
| Original record | Branch or archive | Revision / identifier | Original location |
|---|---|---|---|
| Janady13/missionmatch / docs/plans/2026-06-29-phase2-eval-fairness.md | feature/opportunity-to-outcome | 720d47fae476b0d38477ae4401697d372b1ea2bc | Open original |
| Janady13/missionmatch / docs/plans/2026-06-29-phase2-eval-fairness.md | master | 1e9249840a4302ec82e123b9c9cb6f75badea3e4 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-29-phase2-eval-fairness.md | feature/aioe-upgrade | 2178e7298d0e5d9af83afa12402fd587f0ba6ed8 | Open original |
Recovered source transcript
Download complete source text ↓76 lines · 11.6 KB
# Phase 2 Implementation Plan: WS4 Evaluation Benchmark and WS5 Adverse Impact Dry Run
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax.
**Goal:** Move accuracy from internally consistent to measured (WS4: a labeled GovCon benchmark with a confusion matrix, plus a threshold recalibration against the grown vocabulary), and turn the empty AEDT adverse impact deliverable into actual numbers (WS5: a 500 plus decision sample with segregated synthetic demographics, run through the existing four fifths report).
**Architecture:** WS4 reuses the exact scoring path: hand authored fixtures (requirement, candidate, expected band, rationale) under versioned testdata, scored through `engine.ScoreCandidate`, with a `cmd/eval` that emits a 3x3 band confusion matrix plus per band precision and recall, and a golden snapshot for drift. The recalibration extends the synonym and unrelated pair corpus to the promoted vocabulary and re proves the 0.62 gap, registering a new model version ONLY if the evidence moves it. WS5 generates a sample on a non production database and runs `eeo.Service.AdverseImpact` on the compliance pool, producing real selection and screening ratios, all while the EEO isolation tripwires stay green.
**Tech stack:** Go 1.26, the frozen engine package (read only), the eeo module (read only), pgx/v5, goose. CGO_ENABLED=0, GOFIPS140=v1.0.0, no Python.
**Two hard rules:** (1) the frozen engine is untouched: `go test ./internal/modules/matching/engine/...` stays green, the three golden scores and the weights and bands do not change, and any threshold change is a registered model version bump (never a silent edit). (2) the EEO isolation tripwires stay green: `TestEEOPrivilegeBoundary` and `TestNoProtectedAttributeInCoreTypes` must pass; WS5 never imports eeo from matching or scoring or AI code, never adds a demographic field to a core type, writes demographics only through the compliance pool with consent_given true, and never puts a demographic value into an audit payload.
## Decisions locked (flag at review)
1. Benchmark labeling is HAND AUTHORED domain truth: each fixture carries the band a cleared workforce expert would assign, plus a rationale. The eval's confusion matrix compares the engine's band to that domain truth, so it can catch a genuine mis banding (for example the alias layer over crediting a Red into Yellow), not merely drift. A separate golden total snapshot guards against drift. Honest caveat carried in the model card and the harness README: these labels are constructed by us from domain rules, NOT real cleared hire outcomes, so the benchmark proves band sanity and regression safety, not real world correctness; customer labeled cases swap in later.
2. Recalibration re proves 0.62. The alias layer now absorbs acronyms, so the cosine cutoff only separates genuine semantic synonyms; a move is not expected. If the evidence moves it, register v1.3.1 (weights and bands frozen), do not edit silently.
3. WS5 sample: 500 screening decisions (5 requirements x 100 synthetic candidates) plus a handful of slates for the selection stage, with a realistic balanced synthetic gender and race spread and a consent coverage gap to exercise the unknown and no demographics disclosures. Report the ACTUAL ratios honestly (the existing integration test already exercises an engineered flagged cut). Staging only dry run, captured as a committed reproducible integration test, clearly labeled demo data.
---
## WS4 Task 1: GovCon benchmark fixtures
**Files:** Create `internal/modules/matching/testdata/v1.3.0/<lane>/fixtures.json` for lanes `intelligence_analyst`, `data_scientist`, `cyber_architect`, `general_labor`; a `manifest.json`; a `README.md` carrying the honest caveat.
- [ ] Author roughly 12 to 15 fixtures per lane (about 50 total) spanning the bands (clear Green, clear Yellow, clear Red, and a few near boundary cases). Each fixture is JSON: a `candidate` block (clearance level/poly/status, skills by name, certs, customer agencies, years, location, availability) and a `requirement` block (required level/poly, required skills with mandatory flags, required certs, customer agency, years, location), plus an `expected` block: `band` (the domain truth Green/Yellow/Red), `rationale` (one line), and optionally `mandatory_gaps` (named). Do NOT hand compute the numeric total; the band is the truth label.
- [ ] The README states the labels are constructed from domain rules, not real hire outcomes, and that the harness proves band sanity and regression safety only.
- [ ] Commit.
## WS4 Task 2: cmd/eval, the confusion matrix harness
**Files:** Create `cmd/eval/main.go`, `internal/modules/matching/eval/eval.go` (the pure scoring and metrics logic, unit testable), `internal/modules/matching/eval/eval_test.go`.
- [ ] **Step 1 (TDD):** Write `eval_test.go` first: a small in package test that feeds two or three hand built fixtures with known expected bands through the eval logic and asserts the confusion matrix counts and per band precision and recall. Run, confirm fail.
- [ ] **Step 2:** Write `eval.go`: load fixtures from a lane directory, convert each fixture JSON to `engine.Candidate` and `engine.Requirement`, call `engine.ScoreCandidate(ctx, c, r, nil)` (exact only; a flag enables an injected semantic matcher later), compare the scored band to the expected band, accumulate a 3x3 confusion matrix (rows scored Red/Yellow/Green, columns expected Red/Yellow/Green) and per band precision and recall, and surface per fixture pass or fail with the total and the gap list. All numeric handling stays decimal. Reuse the engine, never re implement scoring.
- [ ] **Step 3:** Write `cmd/eval/main.go`: flags `-lanes` (comma list, default all), `-testdata` (default the v1.3.0 dir), `-verbose`, `-json`. Print the confusion matrix, the per band precision and recall, the per lane pass rate, and (verbose) the per fixture trace. Exit non zero if any band mismatch (so it can gate CI).
- [ ] **Step 4:** Run the tests, `go build ./...`, and run `eval` over the Task 1 fixtures; record the confusion matrix. Add an `eval` FIPS line to the Makefile build target. Commit.
## WS4 Task 3: Threshold recalibration against the grown vocabulary
**Files:** Modify `internal/integration/calibration_test.go` (extend `synonymPairs` and `unrelatedPairs`).
- [ ] Add new synonym pairs (genuine semantic synonyms among the promoted cleared workforce skills, NOT acronyms, which the alias layer owns) and new unrelated pairs drawn from the 267 skill vocabulary. Keep the existing hard guards (`TestSemanticThresholdCalibration`: zero false positives, threshold at or above unrelated_max plus 0.03, at most one acronym class miss).
- [ ] The controller runs this against staging with the embedded `skills` table and the M4 plane (it needs live embeddings). If the expanded gap still cleanly contains 0.62 with margin, 0.62 is re proven and NO engine or migration change is made. If and only if the evidence moves the separating gap, the controller registers v1.3.1: a new migration `0024_model_version_v1_3_1.sql` (aedt_model_versions insert, weights and bands unchanged), bump `DefaultSemanticThreshold` and `ModelVersion` in engine.go, update frozen_test.go, and re run the golden suite (golden fixtures avoid semantic so they stay green).
- [ ] Commit the pair additions; the version bump, if any, is a separate commit.
## WS5 Task 4: Sample generator (non production)
**Files:** Create `cmd/eeo-sample/main.go` (a non production seeding command; clearly labeled demo data) OR an integration helper. It seeds, on a target non production DB: one org, 500 plus synthetic candidates (NO demographic fields, varied clearance and 5 to 15 random skills), 5 requirements, screens each requirement against 100 candidates via `matching.Service.ScreenCandidate` (persisting `candidate_evaluations` and the `aedt.score.served` audit rows), generates a couple of slates and records synthetic human decisions, and inserts synthetic demographics for the candidates THROUGH THE COMPLIANCE POOL into `eeo.candidate_demographics` with `consent_given = true`, distributing gender and race over a labeled demo pattern with a deliberate consent coverage gap.
- [ ] Build it; it must import the matching service and the eeo write path only through the compliance pool. It must NOT add any demographic field to a core type or write a demographic into an audit payload. Run `go build ./...` and confirm `go test ./internal/modules/staffing/ -run TestNoProtectedAttributeInCoreTypes` and the privilege boundary test still pass.
- [ ] Commit.
## WS5 Task 5: The four fifths worked example
**Files:** Create `internal/integration/eeo_dry_run_integration_test.go` (build tag integration).
- [ ] A test that runs the WS5 sample generation (or assumes it ran) against the integration DB, calls `eeo.NewService(tenant.New(compliance)).AdverseImpact(ctxOrg(org), now)`, and asserts: the report has selection and screening sections, each cut (ByGender, ByRace, Intersectional) produces `ImpactRow`s with a selection rate and an impact ratio, at least two known groups per cut, the unknown and no demographics disclosures are populated, and the `Flagged` field is computed (a ratio below 0.80 sets it). It asserts the numbers are present and internally consistent (rate = selected/considered, ratio = rate/max_known_rate), NOT a specific pass or flag outcome (that depends on the random sample). The controller runs it and captures the actual report.
- [ ] Confirm in the same run that the app pool still cannot read eeo (the privilege boundary tripwire) and the core type guard passes. Commit.
## Task 6: Validate on staging and capture the deliverables
- [ ] **Step 1:** Run `cmd/eval` over the committed fixtures, capture the confusion matrix and any mis banded fixtures. If a fixture reveals a genuine mis banding, investigate: it is either a fixture labeling error (fix the label) or a real engine behavior to document (do NOT change the engine to make the benchmark pass).
- [ ] **Step 2:** Run the extended calibration test against `brain_staging` with the M4 plane; record whether 0.62 is re proven or moved.
- [ ] **Step 3:** Run the WS5 sample generator against a non production DB (a fresh `eeo_dry_run` database or `brain_staging`), run the four fifths worked example test, and capture the actual selection and screening ratios as the worked AEDT example.
- [ ] **Step 4:** Run the full tripwire and frozen suite: `go test ./internal/modules/matching/engine/...`, `TestEEOPrivilegeBoundary`, `TestNoProtectedAttributeInCoreTypes`, all green. No production binary change unless the recalibration moved the threshold (then deploy the v1.3.1 binary gated, as in WS3).
- [ ] **Step 5:** Final whole branch code review, then the finishing handoff.
---
## Self-Review
1. Spec coverage: Phase 2 in `docs/AI_GROUNDING_PROGRAM.md` calls for the benchmark with a confusion matrix and threshold recalibration (WS4) and the four fifths dry run on segregated synthetic demographics (WS5). Covered: fixtures (Task 1), the eval harness (Task 2), the recalibration (Task 3), the sample (Task 4), the worked four fifths (Task 5), validation (Task 6).
2. Frozen engine: WS4 only READS the engine; the only path to a scoring change is the registered v1.3.1 bump in Task 3, gated on evidence.
3. EEO isolation: WS5 code lives in cmd/eeo-sample and internal/integration only, writes demographics solely through the compliance pool with consent, and is verified against both tripwire tests in Tasks 4 and 5.
## Open decisions for owner review
The three locked decisions above (hand authored domain truth labels, re prove 0.62, realistic balanced WS5 sample reported honestly, staging only). Confirm, or adjust (for example a deliberately skewed WS5 cut to force a flag for demonstration).