Janady13/missionmatch / docs/plans/2026-06-29-ws3-clearance.md

1# WS3 Clearance and Compartment Layer Implementation Plan 2 3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task by task. Steps use checkbox (`- [ ]`) syntax. 4 5**Goal:** Make clearance first class reference data (levels, polys, compartments, reciprocity) that is FK enforced and governable, and add additive compartment aware scoring to the engine, without changing the frozen weights, bands, threshold, or the existing golden clearance values. 6 7**Architecture:** New global reference tables (`clearance_levels`, `polygraph_types`, `clearance_compartments`, `clearance_reciprocity`) FK enforced on the candidate and requirement clearance columns. Two new array columns (`candidates.clearance_compartments`, `requirements.required_compartments`) flow into the engine through the existing `SELECT *` reads (sqlc auto picks them up). The engine gains `Compartments`/`RequiredCompartments` fields and one additive check in `scoreClearance` after the poly check: a requirement that lists compartments is only met by a held clearance that carries them all. Existing inputs carry no compartments, so the check never fires and the three frozen golden scores are byte identical. ModelVersion bumps to v1.3.0. A consistency test pins the engine rank enum to the seeded `clearance_levels.rank` so the documented hierarchy equals the scored one. 8 9**Tech stack:** Go 1.26, pgx/v5, sqlc v1.31.1 (now installed), goose, shopspring/decimal. CGO_ENABLED=0, GOFIPS140=v1.0.0, no Python. 10 11**The hard rule:** `go test ./internal/modules/matching/engine/...` stays green; the three frozen golden scores (100.00, 87.50, 26.00) and the WS2 alias and relationship golden cases are unchanged. The only allowed frozen change is the ModelVersion assertion to v1.3.0. 12 13--- 14 15## Task 1: Migration 0023, reference tables, compartment columns, v1.3.0 16 17**Files:** Create `db/migrations/0023_clearance_reference.sql` 18 19```sql 20-- +goose Up 21-- WS3 clearance reference layer. Four GLOBAL reference tables (no organization_id, outside RLS, app 22-- SELECT only, migrator writes), FK enforced on the candidate and requirement clearance columns. Adds 23-- compartment array columns to candidates and requirements for additive compartment aware scoring. The 24-- six component weights, the bands, and the 0.62 threshold are unchanged; this registers ModelVersion 25-- v1.3.0 for the new compartment match dimension. 26 27CREATE TABLE clearance_levels ( 28 code text PRIMARY KEY, 29 rank int NOT NULL UNIQUE, 30 requires_polygraph boolean NOT NULL DEFAULT false, 31 label text NOT NULL 32); 33INSERT INTO clearance_levels (code, rank, label) VALUES 34 ('unknown',0,'Unknown'),('none',1,'None or Public Trust'),('confidential',2,'Confidential'), 35 ('secret',3,'Secret'),('top_secret',4,'Top Secret'),('ts_sci',5,'Top Secret SCI'); 36 37CREATE TABLE polygraph_types ( 38 code text PRIMARY KEY, 39 rank int NOT NULL UNIQUE, 40 label text NOT NULL 41); 42INSERT INTO polygraph_types (code, rank, label) VALUES 43 ('none',0,'None'),('ci',1,'Counterintelligence'),('full_scope',2,'Full Scope'); 44 45CREATE TABLE clearance_compartments ( 46 code text PRIMARY KEY, 47 kind text NOT NULL CHECK (kind IN ('sci','sap')), 48 label text NOT NULL 49); 50-- Publicly documented SCI control systems plus generic markers. A SAP is named per program; SAP here is 51-- the generic marker, specific programs are added by the customer. 52INSERT INTO clearance_compartments (code, kind, label) VALUES 53 ('SCI','sci','Sensitive Compartmented Information (generic)'), 54 ('SI','sci','Special Intelligence (SIGINT)'), 55 ('TK','sci','Talent Keyhole (overhead collection)'), 56 ('HCS','sci','HUMINT Control System'), 57 ('SAP','sap','Special Access Program (generic)'); 58 59-- Stub for future non rank reciprocity exceptions (agency specific). The engine uses the rank ladder 60-- today; this table documents and will later carry exceptions. Seeded empty. 61CREATE TABLE clearance_reciprocity ( 62 held_code text NOT NULL REFERENCES clearance_levels(code), 63 required_code text NOT NULL REFERENCES clearance_levels(code), 64 satisfies boolean NOT NULL, 65 source text NOT NULL DEFAULT '', 66 PRIMARY KEY (held_code, required_code) 67); 68 69ALTER TABLE candidates ADD COLUMN clearance_compartments text[] NOT NULL DEFAULT '{}'; 70ALTER TABLE requirements ADD COLUMN required_compartments text[] NOT NULL DEFAULT '{}'; 71 72-- FK enforce the clearance codes against the reference tables (the existing CHECK constraints already 73-- bound the values to exactly these codes, so the FKs are satisfiable on existing rows). 74ALTER TABLE candidates ADD CONSTRAINT candidates_clearance_level_fk FOREIGN KEY (clearance_level) REFERENCES clearance_levels(code); 75ALTER TABLE candidates ADD CONSTRAINT candidates_clearance_poly_fk FOREIGN KEY (clearance_poly) REFERENCES polygraph_types(code); 76ALTER TABLE requirements ADD CONSTRAINT requirements_required_level_fk FOREIGN KEY (required_level) REFERENCES clearance_levels(code); 77ALTER TABLE requirements ADD CONSTRAINT requirements_required_poly_fk FOREIGN KEY (required_poly) REFERENCES polygraph_types(code); 78 79GRANT SELECT ON clearance_levels, polygraph_types, clearance_compartments, clearance_reciprocity TO missionmatch_app; 80 81INSERT INTO aedt_model_versions (version, weights, band_cuts, notes) VALUES ( 82 'v1.3.0', 83 '{"clearance":0.25,"skills":0.25,"certifications":0.15,"customer_experience":0.15,"experience":0.10,"location_availability":0.10}', 84 '{"green":90,"yellow":75}', 85 'Adds an additive compartment dimension to the clearance component, and makes clearance levels, polygraph types, and compartments FK enforced reference data. A requirement that lists compartments is met only by a held clearance carrying them all. The level and poly logic, the binary 0 or 100 clearance score, the six weights, the bands, and the 0.62 threshold are unchanged from v1.2.0. Clearance remains a self reported indicator, never an input proxy for a protected attribute.' 86) ON CONFLICT (version) DO NOTHING; 87 88-- +goose Down 89DELETE FROM aedt_model_versions WHERE version = 'v1.3.0'; 90ALTER TABLE requirements DROP CONSTRAINT requirements_required_poly_fk; 91ALTER TABLE requirements DROP CONSTRAINT requirements_required_level_fk; 92ALTER TABLE candidates DROP CONSTRAINT candidates_clearance_poly_fk; 93ALTER TABLE candidates DROP CONSTRAINT candidates_clearance_level_fk; 94ALTER TABLE requirements DROP COLUMN required_compartments; 95ALTER TABLE candidates DROP COLUMN clearance_compartments; 96DROP TABLE clearance_reciprocity; 97DROP TABLE clearance_compartments; 98DROP TABLE polygraph_types; 99DROP TABLE clearance_levels; 100``` 101 102- [ ] Write the file. Verify against 0022 (global grant pattern) and 0019 (the aedt_model_versions insert columns). Do NOT apply live (Task 6 validates on staging). Commit. 103 104--- 105 106## Task 2: sqlc regen for the compartment columns 107 108- [ ] Run `sqlc generate` (sqlc is installed; `$(go env GOPATH)/bin` is on PATH). Because `GetCandidate` and `GetRequirement` are `SELECT *`, the new `clearance_compartments` and `required_compartments` columns appear on the `dbgen.Candidate` and `dbgen.Requirement` structs as `[]string` automatically. Confirm `go build ./...` is clean and that `git diff` on dbgen shows only the two new compartment fields (the ontology and alias models were already regenerated on the WS2 branch). 109- [ ] Commit the dbgen diff. 110 111--- 112 113## Task 3: Engine compartment check, convert population, ModelVersion v1.3.0 114 115**Files:** Modify `internal/modules/matching/engine/types.go`, `engine.go`, `internal/modules/matching/convert.go`, `internal/modules/matching/engine/frozen_test.go` 116 117- [ ] **Step 1:** In `types.go`, add `Compartments []string` to `Clearance` and `RequiredCompartments []string` to `Requirement` (both default nil/empty, so every existing construction is unchanged). Bump the `ModelVersion` constant in `engine.go` to `"v1.3.0"`. Update the single ModelVersion assertion in `frozen_test.go`. 118- [ ] **Step 2:** In `engine.go` `scoreClearance`, add a new `case` AFTER the poly check (`case held.Poly < r.RequiredPoly:`) and before `default:`: 119```go 120 case !hasAll(held.Compartments, r.RequiredCompartments): 121 missing := missingCompartments(held.Compartments, r.RequiredCompartments) 122 g := fmt.Sprintf("Requires compartments %v; candidate lacks %v.", r.RequiredCompartments, missing) 123 return zero, g + " Scores 0.", []string{g} 124``` 125Add the two helpers (case insensitive, empty required means satisfied): 126```go 127// hasAll reports whether held carries every required compartment. An empty requirement is always met, 128// so existing inputs (no required compartments) are unaffected and the frozen scores do not change. 129func hasAll(held, required []string) bool { 130 if len(required) == 0 { 131 return true 132 } 133 set := map[string]bool{} 134 for _, h := range held { 135 set[strings.ToUpper(strings.TrimSpace(h))] = true 136 } 137 for _, r := range required { 138 if !set[strings.ToUpper(strings.TrimSpace(r))] { 139 return false 140 } 141 } 142 return true 143} 144func missingCompartments(held, required []string) []string { 145 set := map[string]bool{} 146 for _, h := range held { 147 set[strings.ToUpper(strings.TrimSpace(h))] = true 148 } 149 var out []string 150 for _, r := range required { 151 if !set[strings.ToUpper(strings.TrimSpace(r))] { 152 out = append(out, r) 153 } 154 } 155 return out 156} 157``` 158(Add `strings` to the engine.go imports if not present.) 159- [ ] **Step 3:** In `convert.go`, set `Compartments: c.ClearanceCompartments` on the built `engine.Clearance` (line 89 area) and `RequiredCompartments: r.RequiredCompartments` on the built `engine.Requirement` (line 132 area), reading the new dbgen fields. 160- [ ] **Step 4:** Run `go test ./internal/modules/matching/...` and `go build ./...`. The three frozen golden scores and the WS2 golden cases stay green. Commit. 161 162--- 163 164## Task 4: Golden case for compartment scoring 165 166**Files:** Modify `internal/modules/matching/golden_test.go` (the WS2 matching-package golden file) 167 168- [ ] Add two cases scored through `engine.ScoreCandidate`: (a) a requirement with `RequiredLevel ts_sci` and `RequiredCompartments ["SI"]`, a candidate with active TS/SCI carrying `Compartments ["SI","TK"]`, Clearance scores 100; (b) the same requirement, a candidate with active TS/SCI but `Compartments ["TK"]` (no SI), Clearance scores 0 with a gap naming the missing SI. Lock the totals. Confirm the three frozen and the WS2 cases are unchanged. Commit. 169 170--- 171 172## Task 5: Consistency test, the documented hierarchy equals the scored one 173 174**Files:** Create `internal/integration/clearance_reference_integration_test.go` 175 176- [ ] An integration test (DB tagged) that reads `clearance_levels(code, rank)` and `polygraph_types(code, rank)` and asserts that for every code, the seeded rank equals the engine enum value (`engine.ParseClearanceLevel(code)` cast to int, and the polygraph equivalent). This pins the reference data to the scored hierarchy so they can never silently diverge. Commit. 177 178--- 179 180## Task 6: Validate on M4 staging, then load production 181 182- [ ] **Step 1:** On `brain_staging` apply 0023 (goose from M2 as migrator), confirm up and down both work, the FK constraints are added, the four reference tables are seeded, and the compartment columns exist. 183- [ ] **Step 2:** Run the consistency integration test against staging, confirm the reference ranks match the engine enum. 184- [ ] **Step 3:** Prove compartment scoring end to end on staging: seed one candidate with TS/SCI + compartment SI and a requirement for TS/SCI + SI, screen them, confirm Clearance 100; remove SI from the candidate, confirm Clearance 0 with the gap. 185- [ ] **Step 4:** Production (gated, backup first): apply 0023 to production via goose, confirm the FKs hold against the live candidate and requirement rows (the existing CHECK guarantees this), verify the reference seeds. No binary redeploy is required for the reference layer; the compartment scoring activates when the already deployed engine reads compartment data, so deploy the WS3 binary (same gated swap as WS2) to ship the scoreClearance compartment check. 186- [ ] **Step 5:** Final whole branch code review, then the finishing handoff. 187 188--- 189 190## Self-Review 1911. Spec coverage: WS3 in `docs/AI_GROUNDING_PROGRAM.md` calls for clearance and compartment modeling with reference tables and reciprocity, the engine staying the floor. Covered: reference tables and FK and seed (Task 1), the columns flowing through sqlc (Task 2), the additive engine check and convert and version bump (Task 3), the golden proof (Task 4), the consistency pin (Task 5), staging then production (Task 6). Reciprocity is a documented stub table for future exceptions, the engine keeps the rank ladder. 1922. Frozen contract: the only frozen change is the ModelVersion assertion to v1.3.0; the compartment check returns satisfied for an empty requirement, so all existing inputs score identically. 1933. Type consistency: `Clearance.Compartments`, `Requirement.RequiredCompartments`, `hasAll`, and the dbgen `[]string` fields are used identically across Tasks 2, 3, 4. 194