Janady13/missionmatch / docs/AI_GROUNDING_PROGRAM.md

1# AI Grounding Program, design spec (missionMATCH) 2 3Date: 2026-06-29. Status: draft for owner review. Scope approved: full program, Phases 1 to 3. 4 5## 1. Goal 6 7Make the missionMATCH AI perform well on the cleared GovCon job BEFORE a live customer, so the 8platform demonstrably works on day one for Galapagos Federal (8a, Air Force and Space Force, Hawaii 9and New Mexico, mission lanes SIGINT, GEOINT, Satellite C2, Cross Domain Solutions, cyber, intel). 10 11This is a grounding, evaluation, and feedback program built entirely in Go, SQL, and bash over the 12free federal data sources we already pull. It is NOT model weight training. 13 14## 2. The reframe (why this is grounding, not training) 15 16Two things in this product are deliberately frozen and are NOT trained: 17 181. The deterministic scoring engine. Fixed weights (Clearance 0.25, Skills 0.25, Certifications 0.15, 19 Customer Experience 0.15, Experience 0.10, Location 0.10), enforced byte for byte by the golden and 20 frozen tests. Its immutability is the AEDT compliance asset. We do not touch it. 212. The embedding model. Domain fine tuning of bge-small-en-v1.5 is ruled out by our own constraints: 22 no Python anywhere, cleared resumes are CUI and must not leave the self hosted plane, FIPS 23 (GOFIPS140=v1.0.0, CGO_ENABLED=0), and both cloud portability. The map confirmed every fine tune 24 path (LoRA, continued pretraining, managed Bedrock or Azure OpenAI tuning) violates at least one 25 hard rule, and an in place checkpoint change would silently corrupt every stored vector. 26 27So "train the AI on this job" means: give the AI a rich, domain grounded knowledge layer it can reason 28over deterministically, prove and calibrate it against labeled data, and let curated human feedback 29improve that layer over time. The engine stays the floor; the AI narrates and retrieves on top of it. 30 31## 3. Guardrails (every workstream inherits these) 32 33- AEDT: every match stays explainable with a per component trace. A human approves or rejects every 34 slate decision. Nothing auto rejects. The deterministic engine stays the floor. 35- EEO: protected attributes and close proxies are never a matching input. The matching and ontology 36 code never references the eeo schema. New skill, alias, and mission fields are screened so none is a 37 demographic proxy. 38- Tenancy: tenant scoped tables keep RLS forced with organization_id leading every composite index. 39 The skills, alias, relationship, and mission reference tables are GLOBAL (outside RLS), read only to 40 the app role, written only by the migrator role, exactly like the existing skills table. 41- FIPS and tooling: all new binaries build CGO_ENABLED=0 GOFIPS140=v1.0.0, verify the stamp after every 42 build. No Python. Go commands under cmd/, bash for orchestration, SQL via migrations only. 43- Both cloud: stays 384 dimension so the vector columns and HNSW indexes survive a provider swap. No 44 cloud specific primitive in the core path. External data fetchers sit behind a source interface. 45- Vector integrity: any new embedded column obeys the DEPLOY_PLAN B2 rule, query vectors and stored 46 vectors share one model, enforced by the existing VectorGuard gate. 47 48## 4. Current state (from the 2026-06-29 capability map) 49 50- Ontology is a stub: 13 real skills in production, 32 in the seed, 8 flat free text categories, no 51 hierarchy, no mission vocabulary. A SIGINT or Intel Analyst requirement cannot be expressed today. 52- No alias or synonym table exists. SIGINT to Signals Intelligence is provably unreachable by any 53 single cosine threshold and is routed to an alias layer that was designed but never built. 54- Adjacency runs live as raw k nearest neighbor over skills.embedding but nothing is curated or stored, 55 no strength, no requires, no stronger than, no deprecation edges. 56- Clearance granularity is a static text rule. The generic embedding treats Secret and TS/SCI as 57 similar. Compartments (SCI, SAP), poly type (CI versus full scope), and reciprocity are not modeled. 58- The teaching and federated learning path is built and privacy scrubbed but DISABLED and non 59 persistent. The gateway returns 202 and discards every signal. Nothing improves from a reviewer 60 action. 61- There is no GovCon benchmark. Accuracy is proven internally consistent (formula, determinism, 62 threshold calibration on 12 plus 12 pairs) but never proven correct against real cleared hire 63 judgment. The four fifths adverse impact report has architecture but no sample data. 64- Rich federal signal sits unparsed: incumbent staffing mix and mission keywords live in award and 65 opportunity descriptions and raw JSON in object storage, never extracted. BLS OEWS covers only 5 66 hardcoded occupations. O*NET, GSA CALC, and FPDS are entirely unintegrated. 67- The generated RRF hybrid retrieval SQL is not mounted on any live path, and the production reranker 68 is a cosine baseline rather than a cross encoder. 69 70## 5. Architecture of the grounded knowledge layer 71 72``` 73 FREE FEDERAL SOURCES GLOBAL KNOWLEDGE LAYER (new, outside RLS) 74 O*NET SOC skills ─┐ ┌─► skills (enriched: + mission_area, source, soc_code) 75 GSA CALC labor rates ├─ ingest ┼─► skill_aliases (acronym and synonym to canonical skill) 76 FPDS / USASpending ─┤ (Go ├─► skill_relationships (requires, substitutes_for, broader_than) 77 SAM raw JSON / CLINs │ cmds) ├─► clearance_levels (compartments, poly, reciprocity) 78 curated IC overlay ─┘ └─► mission_areas (SIGINT, GEOINT, Satellite C2, CDS, ...) 79 │ 80 ▼ 81 DETERMINISTIC ENGINE (frozen weights) 82 exact match → alias match → relationship credit → semantic ≥ 0.62 83 │ 84 ▼ 85 BRAIN (retrieval + memory + faithfulness, narrates only) 86 RRF hybrid + reranker, mission filtered, incumbent intelligence 87 │ 88 ▼ 89 EVALUATION HARNESS ──► calibrates threshold, proves bands, four fifths 90 ▲ 91 BOUNDED FEEDBACK LOOP ───┘ scrubbed reviewer outcomes become CURATION 92 proposals for humans, never auto mutate scoring 93``` 94 95The new tables are global reference data, the same class as skills today. The engine consults the 96alias and relationship layer deterministically before falling back to semantic similarity, so domain 97knowledge is encoded as auditable rows, not as opaque model weights. 98 99## 6. Phase 1 — grounding (the load bearing core) 100 101### WS1. GovCon skill ontology 102 103Build a real cleared workforce ontology, hundreds of skills with meaningful categories and a mission 104area dimension. 105 106- Schema: add to the global skills table the columns source (text, for provenance), soc_code (text, 107 nullable), mission_area (text, nullable, FK to mission_areas.code), and last_verified_at. Keep the 108 existing id, name, category, embedding. A new mission_areas table holds the controlled mission 109 vocabulary (code, name, description) for SIGINT, GEOINT, HUMINT, MASINT, Satellite C2, Cross Domain 110 Solutions, COMSEC, EW, cyber operations, intel analysis, space systems, and the rest of the Galapagos 111 lanes. 112- Sourcing strategy (verified 2026-06-29 against live sources): auto pull the breadth, curate only the 113 small IC spine. A new Go command cmd/ontology-sync (CGO_ENABLED=0, FIPS) with an OntologySource 114 interface (Name, Fetch) and six named adapters, mirroring the existing cmd/opp-sync adapter pattern, 115 using net/http with encoding/json, encoding/xml, and an xlsx reader (excelize). No Python, no new 116 crypto. Adapters, in run order: 117 1. psc: the Product Service Code manual XLSX from acquisition.gov (no auth, fully enumerable), the 118 procurement classification floor including AC National Defense R&D and the D series IT codes. 119 2. nice: the NICE Framework components JSON (NIST CPRT, no auth, public domain), pinned to a version, 120 the cyber and IT skill backbone (work roles, competency areas, TKS statements with stable ids). 121 3. dcwf: the open DoD Cyber Workforce Framework factsheet and coding guide PDFs plus the NICE to DCWF 122 crosswalk, projecting the cleared and IC cyber roles (Intelligence Cyberspace, Cyberspace Effects) 123 onto the NICE backbone. KSAT detail is CAC gated, so we ingest the open publications plus crosswalk. 124 4. onet: O*NET Web Services v2.0 (JSON, X-API-Key header). Built but DORMANT behind a missing key, it 125 logs a clear skip until ONET_API_KEY is [credential removed], so WS1 ships without it. 126 5. calc: GSA CALC ceiling rates (no auth), enumerating the federal labor category vocabulary by a 127 bounded suggest sweep and dedupe (there is no catalog endpoint). 128 6. corpus: the differentiator, no external call. Mine our OWN retained text (sam_opportunities.raw, 129 usaspending_awards.description and raw, opportunity_records via the storage.Storer) with the 130 existing deterministic extractor gazetteers and the matching/intel.go labor classifier, emitting 131 the actual skills, certs, clearances, and mission terms that Air Force and Space Force buyers in 132 Hawaii and New Mexico use, then reconciling them against the reference vocabularies above and the 133 curated mission spine. 134- Staging versus matching tables: external adapters write a reference vocabulary (ontology_terms, 135 ontology_crosswalk) with provenance and a version; a curation step promotes selected, screened terms 136 into the matching facing global skills table (and the WS2 alias and relationship tables). Promotion is 137 human reviewed so the live ontology stays controlled. Each promoted skill gets a bge vector via the 138 existing embed-backfill path so the HNSW index stays live and the VectorGuard gate still passes. 139- The IC mission spine is a small curated table, see decision 1 below, not an adapter. 140- Acceptance: the ontology expresses every Galapagos mission lane, an Intel Analyst and a SIGINT and a 141 Satellite C2 requirement can each be authored and matched, and the embed gate (VectorGuard) still 142 passes on the enriched table. 143 144### WS2. Deterministic alias and relationship layer 145 146Encode the domain knowledge that embeddings provably cannot reach. 147 148- skill_aliases (global): alias_text, canonical skill_id, alias_type (acronym, synonym, vendor_term, 149 legacy), source, confidence. Seeded with the load bearing GovCon acronyms (SIGINT to Signals 150 Intelligence, COMSEC to Communications Security, ATO to Authority to Operate, CDS to Cross Domain 151 Solution, RMF to Risk Management Framework, TS/SCI compartment expansions, and the rest). 152- skill_relationships (global): from_skill_id, to_skill_id, relation (requires, substitutes_for, 153 broader_than, related_to), strength (decimal), source. Captures prerequisites (COMSEC requires 154 cryptography), substitutions (one tool credits for an adjacent tool at a strength below 1.0), and 155 hierarchy (a broader skill covers a narrower one). 156- Engine integration: extend the matchOne path so the order is exact match, then alias match (an alias 157 hit is treated as an exact match, similarity 1.0, recorded as alias matched with the alias text in the 158 trace), then relationship credit (a substitutes_for or broader_than edge contributes its strength as a 159 partial match below 1.0, recorded), then the existing semantic similarity at or above 0.62. Every step 160 records which candidate skill matched which required skill and by what mechanism, so the trace stays 161 fully explainable. The fixed weights and band thresholds do not change. New behavior is additive and 162 covered by new golden cases so the existing golden contract still holds for the existing inputs. 163- Curation: the alias and relationship rows are human authored or human approved. The Phase 3 feedback 164 loop proposes new edges, it never writes them directly. 165- Acceptance: a candidate whose resume says Signals Intelligence matches a requirement that asks for 166 SIGINT at similarity 1.0 with alias matched in the trace, and the calibration suite still shows zero 167 false positives. 168 169### WS3. Clearance and compartment modeling 170 171Make clearance granularity first class, because for an intel customer it is the whole game. 172 173- clearance_levels (global reference): level code, rank, requires_polygraph (none, CI, full_scope), 174 compartments (SCI, SAP markers), and an explicit reciprocity matrix (which level satisfies which). 175 This replaces the implicit hierarchy encoded in code with auditable reference rows. 176- Engine integration: the Clearance component (weight 0.25, unchanged) reads the reciprocity matrix 177 instead of a hardcoded ladder. The rule stays deterministic and explainable, exact or higher satisfies, 178 lower or expired or unknown scores zero, and a compartment requirement (TS/SCI with CI poly) is only 179 satisfied by a clearance that carries it. The clearance value stays a self reported indicator labeled 180 clearance intelligence, never verified, per the existing product rule. 181- Acceptance: a Secret clearance does not satisfy a TS/SCI requirement, a TS/SCI with CI poly satisfies a 182 TS/SCI requirement, and every verdict carries a one line reason in the trace. 183 184## 7. Phase 2 — prove and calibrate 185 186### WS4. GovCon evaluation and benchmark harness 187 188Move accuracy from internally consistent to measured. 189 190- A labeled benchmark of realistic Air Force and Space Force requirements crossed with candidate 191 profiles, each with an expected band (Green, Yellow, Red) and a short expected rationale. Authored as 192 versioned fixtures under a new testdata tree, organized by mission lane. 193- Honest sourcing caveat: until Galapagos can label real outcomes, the expected bands are constructed by 194 us from defensible domain rules, not from real hire decisions. This benchmark proves regression safety, 195 threshold calibration, and band sanity. It is explicitly NOT a substitute for validation against real 196 cleared hire outcomes, and the model card will say so. A later workstream swaps in customer labeled 197 cases when they exist. 198- A new Go test target and a runnable cmd/eval that scores the benchmark and reports precision, recall, 199 a confusion matrix over bands, and the per pair traces. The semantic threshold is recalibrated against 200 the expanded labeled skill pair corpus (the current 12 plus 12 grows into the mission vocabulary), with 201 the same hard guards (zero false positives, margin above the unrelated ceiling), and any change is 202 registered as a new model version row, the same mechanism as the 0.62 calibration. 203- Acceptance: the harness runs green, produces a confusion matrix, and any threshold move is evidence 204 backed and version registered. 205 206### WS5. Adverse impact dry run 207 208Turn the empty AEDT deliverable into actual numbers. 209 210- Generate a sample of 500 or more scored decisions in a non production database, with segregated 211 synthetic demographics in the eeo schema (synthetic, never real people, labeled demo data), then run 212 the existing four fifths report to produce real selection rates and ratios. 213- This exercises the compliance reporting path end to end and gives a worked example for a FedRAMP or LL144 214 assessor. The matching code never sees the demographics, the report reads them only through the 215 compliance role, exactly as the architecture already enforces. 216- Acceptance: the four fifths report produces ratios over the sample, and the EEO isolation tripwire 217 tests stay green (matching still cannot reach the eeo schema). 218 219## 8. Phase 3 — retrieval, federal signal, and the feedback loop 220 221### WS6. Mount RRF hybrid retrieval and a real reranker 222 223- Wire the already generated RRF SQL (tsvector BM25 fused with pgvector HNSW by reciprocal rank fusion) 224 into the live brain retrieval path, replacing or augmenting the current call. Add mission_area as a 225 retrieval filter so the brain can answer mission scoped questions. 226- Upgrade the gateway reranker from the cosine baseline toward a cross encoder served by the self hosted 227 plane, behind the existing rerank interface so the API contract does not change. If a FIPS clean, both 228 cloud portable cross encoder is not readily self hostable, this stays the cosine baseline and the 229 workstream documents that limit rather than reaching for a managed service. 230- Acceptance: a mission scoped brain query returns mission relevant evidence, and retrieval quality on the 231 WS4 benchmark improves measurably against the pre RRF baseline. 232 233### WS7. Federal labor and mission signal extraction 234 235- A Go extraction pass over the retained award and opportunity descriptions and the raw JSON and CLIN 236 payloads already in object storage, pulling incumbent staffing mix (for example counts of ISSO, Cyber 237 Engineer, SIGINT Analyst roles) and mission keywords into queryable columns. New fields hang off the 238 existing usaspending_awards and opportunity_records tables, or a child table, with provenance. 239- A deterministic mission classifier (keyword and alias driven, reusing the WS2 alias layer) tags 240 opportunities and awards with mission_area, so the Command Center can answer open SIGINT requirements 241 and align bench to mission, the customer's core organizing dimension. 242- Extends the existing 8 keyword headline classifier in matching/intel.go into a shared, tested, 243 ontology backed labor inference used by both candidates and opportunities. 244- Acceptance: the platform can list open requirements by mission area and surface incumbent staffing mix 245 on a recompete, with each inference traceable to its source text. 246 247### WS8. Bounded teaching and feedback loop 248 249Make the AI improve from reviewer judgment, within the privacy and AEDT boundary already designed. 250 251- Implement the gateway side aggregation and storage that is currently a discard. The enumerated, scrubbed 252 TeachingSignal vocabulary already exists (adjacent_skill_substitution_outcome, 253 clearance_gap_override_outcome, experience_shortfall_outcome, reranker_label_agreement) carrying only 254 coarse band, outcome, counts, and model version, no identifiers, enforced by the two sided scrub 255 boundary. The gateway aggregates these into counts per pattern. 256- The learning target is deterministic and human in the loop, never an auto model update. Aggregated 257 signals produce CURATION PROPOSALS, for example a repeated adjacent_skill_substitution_outcome proposes 258 a new skill_relationships substitutes_for edge for a human to approve, and a repeated 259 clearance_gap_override_outcome flags a reciprocity rule for review. They also feed the WS4 260 recalibration as evidence. No signal ever mutates a score, a weight, or a threshold automatically, and 261 the engine stays the floor. 262- Differential privacy aggregation, currently a comment, is implemented as bounded noise on the published 263 counts so a single tenant's behavior cannot be inferred, keeping the cross tenant learning privacy 264 preserving. Federated learning stays behind its flag and defaults off until reviewed. 265- Acceptance: a reviewer override produces a scrubbed signal, the gateway aggregates it, and a curation 266 proposal appears for human approval, with an audit entry and zero PII. 267 268### WS9. Bulk candidate skill ingestion 269 270- Use the existing extraction port to auto populate candidate_skills from cleared resumes at scale (CUI 271 safe, server side extraction, no browser key, the path the W1 Mission Check already established), so a 272 real bench exists to match against. Skills resolve through the WS1 ontology and the WS2 alias layer, so 273 an extracted Signals Intelligence resolves to the SIGINT canonical skill. 274- Acceptance: a batch of resumes produces grounded candidate_skills rows resolved to canonical ontology 275 skills, and the Command Center bench coverage reflects them. 276 277## 9. Free data sources used 278 279Verified 2026-06-29 against live sources. 280 281| Source | Access now | What it grounds | Limit | Workstream | 282| --- | --- | --- | --- | --- | 283| PSC manual (acquisition.gov) | yes, public XLSX | product and service code taxonomy incl. defense codes | fully enumerable, annual refresh | WS1, WS7 | 284| NICE Framework (NIST CPRT) | yes, public JSON | cyber and IT skill backbone (roles, competencies, TKS) | pin a version | WS1 | 285| DoD DCWF | partial, open PDFs | cleared and IC cyber roles plus NICE crosswalk | KSAT detail is CAC gated, use crosswalk | WS1 | 286| O*NET Web Services v2.0 | KEY NEEDED (free) | civilian SOC skills, knowledge, abilities, tasks | manual approval, 5/sec, 50k/day, attribution | WS1 | 287| GSA CALC ceiling rates | yes, no auth | federal labor category vocabulary | no catalog, sweep and dedupe | WS1, WS7 | 288| FPDS-NG ATOM | yes, public | award transactions tagged by PSC and UEI | 10 records/request, paginate | WS7 | 289| USASpending | yes, keyless | incumbents by UEI, PSC and NAICS hierarchy | approx 10k/query, blurb text only | WS7 | 290| SAM.gov opportunities | yes, SAM_API_KEY held | the high value mine: PWS and SOW attachments carry real labor categories, clearances, certs, mission terms | 1000/day entity tier, 429 Retry-After | WS1, WS7 | 291| Our retained corpus | yes, in hand | already holds SAM and award raw text, mined locally | none external, deterministic, auditable | WS1, WS7 | 292| Curated IC mission spine | author in repo | SIGINT, GEOINT, Satellite C2, CDS, COMSEC vocabulary | not scrapeable, see decision 1 | WS1, WS2 | 293| BLS OEWS | yes, BLS_API_KEY held | wage baseline by SOC and area | widen beyond the 5 hardcoded rows | WS7 | 294 295## 10. Schema and migration plan 296 297New global reference tables: mission_areas, skill_aliases, skill_relationships, clearance_levels. 298Enriched columns on skills (source, soc_code, mission_area, last_verified_at) and on the award and 299opportunity tables (incumbent mix, mission_area). All as additive goose migrations continuing from the 300current production head (v19 on the brain branch), with exact numbers assigned at implementation to avoid 301collision with the unmerged per tenant branch (which already claims 0020). Every new global table grants 302SELECT to the app role and INSERT and UPDATE to the migrator only, RLS not applicable (global), the same 303pattern as skills. Each migration ships a tested down. 304 305## 11. Sequencing and dependencies 306 3071. WS1 ontology is the root, everything matches against it, build first. 3082. WS2 alias and relationship layer and WS3 clearance modeling depend on WS1, build next, in parallel. 3093. WS4 benchmark and WS5 adverse impact depend on WS1 to WS3 (they measure them), build third. WS4 is the 310 gate that proves Phase 1 worked. 3114. WS6 retrieval, WS7 federal signal, WS9 bench ingestion depend on the grounded ontology and can run in 312 parallel after Phase 1 is proven. 3135. WS8 feedback loop is last in build order because its curation proposals target the WS2 and WS3 tables 314 and its evidence feeds the WS4 recalibration, so those must exist first. 315 316Each workstream is its own branch, its own migration where needed, its own tests, reviewed and validated 317on M4 staging before any production step, mirroring the brain cutover discipline. Nothing in this program 318requires a production cutover except, optionally, deploying the enriched ontology and binary the same way 319the brain was deployed, gated and rollback ready. 320 321## 12. Acceptance for the program 322 323The AI is considered grounded for go live when: every Galapagos mission lane is expressible and matchable 324in the ontology, the load bearing acronyms resolve through the alias layer at similarity 1.0, clearance 325and compartment rules are correct and explainable, the WS4 benchmark runs green with a published confusion 326matrix and an evidence backed threshold, a four fifths dry run exists, the brain answers mission scoped 327questions over RRF retrieval with incumbent intelligence, and a reviewer action produces a privacy scrubbed 328curation proposal for human approval. The deterministic engine is unchanged throughout and the EEO 329isolation tripwires stay green. 330 331## 13. Out of scope (deferred) 332 333GraphRAG teaming graphs and the capture and foresight features (win probability, price to win, clearance 334timing) stay deferred. They return ErrScaffold today and need labeled historical outcome data this program 335does not generate. They become viable after the customer accrues real decisions, and get their own spec. 336 337## 14. Open decisions for owner review 338 3391. Ontology sourcing: RESOLVED 2026-06-29 by live source verification. We auto pull the breadth from 340 five working no friction adapters (psc, nice, dcwf, calc, corpus) plus O*NET once its free key lands, 341 and we CURATE the IC mission spine. The spine is not scrapeable: the terms live only as prose across 342 three competing authorities (DoD Dictionary and Joint Pubs, ODNI, CNSSI 4009), the one machine readable 343 IC artifact (the CVE/XSD enumerations) carries classification markings not mission terms, the target is 344 tiny (roughly 150 to 400 terms, about a half day of curation), it barely changes (multi year doctrine 345 cycles), and choosing the governing authority for a contested term needs a human. So scraping it would 346 be more work, less reliable, and would still need hand reconciliation. Confirm this resolution. 3472. Benchmark ground truth: pre customer labels are constructed by us from domain rules, not real hire 348 outcomes, and the model card will state that limit. Confirm this is acceptable as the interim proof. 3493. Teaching loop target: aggregated reviewer signals become human approved curation proposals and 350 calibration evidence, never an automatic score or threshold change. Confirm this stays human in the loop 351 by design even once federated learning is enabled. 352