Janady13/missionmatch / docs/plans/2026-06-29-ws2-alias-relationship.md

1# WS2 Alias and Relationship Layer Implementation Plan 2 3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task by task. Steps use checkbox (`- [ ]`) syntax for tracking. 4 5**Goal:** Make the deterministic engine actually use the WS1 ontology, by resolving curated skill aliases (SIGINT to Signals Intelligence, as an exact match) and curated skill relationships (substitution and broader than, as partial credit) BEFORE the semantic matcher, without changing the frozen weights, bands, threshold, or the three golden scores. 6 7**Architecture:** Add two global reference tables (`skill_aliases`, `skill_relationships`) that FK to the global `skills` table. Add a `DeterministicAliasResolver` (loads the two tables into a lazy cache, the same shape as `StoredSkillMatcher`) and a `ChainedResolver` that runs the resolver first and the existing semantic `StoredSkillMatcher` second. `ChainedResolver` satisfies the engine's existing `SkillMatcher` interface, so the engine package changes zero lines; the swap happens at service init. Promote a curated set of WS1 ontology terms into the matcher facing `skills` table so there are real skills to alias and relate. Bump the engine `ModelVersion` to v1.2.0 and register it. 8 9**Tech stack:** Go 1.26, pgx/v5, goose, shopspring/decimal (all match math stays decimal), bge embeddings via the existing embed-backfill, the existing engine `SkillMatcher` seam. CGO_ENABLED=0, GOFIPS140=v1.0.0, no Python. 10 11**The hard rule:** `go test ./internal/modules/matching/engine/...` (golden_test.go and frozen_test.go) MUST stay green at every step. The three frozen cases score byte for byte identically because they reference no alias or relationship edge and run with no matcher. 12 13--- 14 15## Decisions locked for this plan (flag at review) 16 171. **Promotion scope (DECIDED, curated cleared set plus key tools):** promote into `skills` a curated GovCon relevant set, not the full 2797 skill kind rows. The batch is: the 15 Corpus Mining skills, the 10 mission_spine IC terms, an authored cleared workforce seed (RMF, ATO, Cross Domain Solutions, SIGINT, GEOINT, and the like, roughly 60 to 100 rows), AND the genuinely useful O*NET technology tools filtered to cleared tech relevance by tech category (development environment, operating system, network and security, database, programming language, configuration management, virtualization, cloud, analytical, so Splunk, Python, Kubernetes, Linux, Wireshark land but veterinary and sewing tools do not). Net `skills` table goes from 13 to roughly 150 to 300 clean rows. The bulk O*NET tools, NICE KSAs, and PSC codes stay in `ontology_terms` as the reference pool. 182. Alias hit scores Similarity 1.0, Exact true, and the trace flags it alias sourced for audit. Relationship `substitutes_for` and `broader_than` score the curated `strength` as a partial (Exact false), gated `strength >= 0.62`. `requires` and `related_to` never score. Only `verified_at IS NOT NULL` edges score. 193. Engine integration is the `ChainedResolver` (zero engine edits), not new branches inside matchOne. 204. ModelVersion bumps to v1.2.0, registered in `aedt_model_versions` by the migration, stamped by the engine. 21 22--- 23 24## File structure 25 26| File | Responsibility | New or Modify | 27| --- | --- | --- | 28| `db/migrations/0022_skill_alias_relationship.sql` | the two global tables, skills column adds, v1.2.0 registry row | Create | 29| `db/seed/cleared_skills.sql` | the authored cleared workforce skill seed | Create | 30| `db/seed/skill_aliases.sql` | curated alias and relationship seed | Create | 31| `cmd/promote-ontology/main.go` | promote curated ontology_terms into skills (idempotent) | Create | 32| `internal/modules/matching/aliasresolver.go` | `DeterministicAliasResolver` (lazy cache, ResolveAlias) | Create | 33| `internal/modules/matching/aliasresolver_test.go` | resolver unit tests (alias and relationship) | Create | 34| `internal/modules/matching/chainedresolver.go` | `ChainedResolver` satisfies `SkillMatcher` | Create | 35| `internal/modules/matching/chainedresolver_test.go` | chained order tests | Create | 36| `internal/modules/matching/engine/types.go` | add `AliasMatched bool` to `SkillMatch`; bump `ModelVersion` to v1.2.0 | Modify | 37| `internal/modules/matching/engine/golden_test.go` | add the alias and relationship golden cases (3 frozen unchanged) | Modify | 38| `internal/modules/matching/service.go` / `cmd/api/main.go` | inject `ChainedResolver` in place of bare `StoredSkillMatcher` | Modify | 39 40--- 41 42## Task 1: Migration 0022, the alias and relationship tables and v1.2.0 43 44**Files:** Create `db/migrations/0022_skill_alias_relationship.sql` 45 46- [ ] **Step 1: Write the migration** (global tables mirroring the skills grant pattern; additive skills columns; register the model version like 0019) 47 48```sql 49-- +goose Up 50-- WS2 deterministic alias and relationship layer. Two GLOBAL reference tables, like skills they carry no 51-- organization_id and sit outside RLS, app role SELECT only, migrator writes. They FK to the global skills 52-- table. The engine consults these (curated, ontology backed) before the semantic matcher: an alias is an 53-- exact equivalence, a relationship is a curated partial credit. Weights, bands, and the 0.62 threshold are 54-- unchanged; this registers ModelVersion v1.2.0 for the new curated match capability. 55 56ALTER TABLE skills ADD COLUMN IF NOT EXISTS source text NOT NULL DEFAULT ''; 57ALTER TABLE skills ADD COLUMN IF NOT EXISTS soc_code text NOT NULL DEFAULT ''; 58 59CREATE TABLE skill_aliases ( 60 id uuid PRIMARY KEY DEFAULT gen_random_uuid(), 61 skill_id uuid NOT NULL REFERENCES skills(id) ON DELETE CASCADE, 62 alias_text text NOT NULL, 63 alias_type text NOT NULL CHECK (alias_type IN ('acronym','synonym','vendor_term','legacy')), 64 source text NOT NULL DEFAULT '', 65 confidence numeric(3,2) NOT NULL DEFAULT 1.0 CHECK (confidence >= 0 AND confidence <= 1), 66 verified_at timestamptz, 67 created_at timestamptz NOT NULL DEFAULT now(), 68 UNIQUE (lower(alias_text)) 69); 70CREATE INDEX idx_skill_aliases_skill ON skill_aliases (skill_id); 71 72CREATE TABLE skill_relationships ( 73 id uuid PRIMARY KEY DEFAULT gen_random_uuid(), 74 from_skill_id uuid NOT NULL REFERENCES skills(id) ON DELETE CASCADE, 75 to_skill_id uuid NOT NULL REFERENCES skills(id) ON DELETE CASCADE, 76 relation text NOT NULL CHECK (relation IN ('requires','substitutes_for','broader_than','related_to')), 77 strength numeric(3,2) NOT NULL DEFAULT 1.0 CHECK (strength >= 0 AND strength <= 1), 78 source text NOT NULL DEFAULT '', 79 verified_at timestamptz, 80 created_at timestamptz NOT NULL DEFAULT now(), 81 UNIQUE (from_skill_id, to_skill_id, relation) 82); 83CREATE INDEX idx_skill_rel_from ON skill_relationships (from_skill_id); 84CREATE INDEX idx_skill_rel_to ON skill_relationships (to_skill_id); 85 86GRANT SELECT ON skill_aliases TO missionmatch_app; 87GRANT SELECT ON skill_relationships TO missionmatch_app; 88 89-- Register the recalibrated model capability. The aedt_model_versions table (added by 0019, v1.1.0) has 90-- columns version, weights, band_cuts, notes; the weights and band cuts are unchanged from v1.1.0. 91INSERT INTO aedt_model_versions (version, weights, band_cuts, notes) VALUES ( 92 'v1.2.0', 93 '{"clearance":0.25,"skills":0.25,"certifications":0.15,"customer_experience":0.15,"experience":0.10,"location_availability":0.10}', 94 '{"green":90,"yellow":75}', 95 'Adds the curated deterministic alias and relationship match layer (skill_aliases and skill_relationships). An alias resolves a curated ontology backed equivalence as an exact match (similarity 1.0); a substitutes_for or broader_than relationship contributes its curated strength as a partial match, gated at or above the 0.62 semantic threshold. The six component weights, the green and yellow band cuts, and the 0.62 threshold are unchanged from v1.1.0. Clearance remains a self reported indicator, never an input proxy for a protected attribute.' 96) ON CONFLICT (version) DO NOTHING; 97 98-- +goose Down 99DELETE FROM aedt_model_versions WHERE version = 'v1.2.0'; 100DROP TABLE IF EXISTS skill_relationships; 101DROP TABLE IF EXISTS skill_aliases; 102ALTER TABLE skills DROP COLUMN IF EXISTS soc_code; 103ALTER TABLE skills DROP COLUMN IF EXISTS source; 104``` 105 106- [ ] **Step 2:** Confirm the `aedt_model_versions` columns by reading `db/migrations/0019_model_version_v1_1_0.sql`; if its columns differ (for example a different notes column name), match them exactly. Do NOT apply live (no local Postgres); the apply and rollback are validated in Task 9 on M4 staging. 107- [ ] **Step 3: Commit** `git add db/migrations/0022_skill_alias_relationship.sql` with the no dash rules and the Co-Authored-By trailer. 108 109--- 110 111## Task 2: Promote curated ontology terms into skills 112 113**Files:** Create `cmd/promote-ontology/main.go`, `db/seed/cleared_skills.sql` 114 115The matcher needs real skills to alias and relate. Promote, deduped by lowercased name, into `skills`: every Corpus Mining skill, every mission_spine term, and an authored cleared workforce seed. 116 117- [ ] **Step 1:** Author `db/seed/cleared_skills.sql`: INSERT roughly 60 to 100 cleared workforce skills and tools into `skills (name, category, source)` with `ON CONFLICT (name) DO NOTHING`, covering the Galapagos lanes (SIGINT, GEOINT, HUMINT, Cross Domain Solutions, COMSEC, RMF, ATO, Splunk, Python, AWS, Azure, Kubernetes, Linux, Wireshark, SIEM, Intelligence Analysis, Cybersecurity, and the like). Source `'curated'`. 118- [ ] **Step 2:** Write `cmd/promote-ontology/main.go` (Go, FIPS, migrator DSN), which runs the curated promotion SQL idempotently: 119```sql 120INSERT INTO skills (name, category, source) 121SELECT DISTINCT ot.term, COALESCE(NULLIF(ot.category,''),'mined'), ot.source_name 122FROM ontology_terms ot 123WHERE ot.term_kind = 'skill' AND ot.source_name = 'Corpus Mining' 124 AND NOT EXISTS (SELECT 1 FROM skills s WHERE lower(s.name) = lower(ot.term)) 125ON CONFLICT (name) DO NOTHING; 126 127INSERT INTO skills (name, category, source) 128SELECT ms.term, 'mission_area', 'mission_spine' 129FROM mission_spine ms WHERE ms.version='v1' 130 AND NOT EXISTS (SELECT 1 FROM skills s WHERE lower(s.name) = lower(ms.term)) 131ON CONFLICT (name) DO NOTHING; 132 133-- O*NET technology tools filtered to cleared tech relevance by category, so Splunk, Python, Kubernetes, 134-- Linux, Wireshark land but veterinary and sewing tools do not. 135INSERT INTO skills (name, category, source, soc_code) 136SELECT DISTINCT ot.term, COALESCE(NULLIF(ot.category,''),'tool'), 'onet', ot.code 137FROM ontology_terms ot 138WHERE ot.source_name = 'O*NET Web Services' AND ot.source_record_id LIKE 'onet-tech:%' 139 AND lower(ot.category) ~ 'development|operating system|network|security|database|program|configuration|virtualiz|cloud|analytical|data base|object or component' 140 AND NOT EXISTS (SELECT 1 FROM skills s WHERE lower(s.name) = lower(ot.term)) 141ON CONFLICT (name) DO NOTHING; 142``` 143It also writes one `aedt.ontology.promoted` audit event with the inserted count. It runs as the migrator role. The implementer should first query `SELECT DISTINCT category FROM ontology_terms WHERE source_record_id LIKE 'onet-tech:%'` on a populated database to confirm the real category titles and tune the keyword set so the cleared tech tools land and the noise does not. 144- [ ] **Step 3:** After promotion the new skill rows have NULL embeddings. The plan runs `embed-backfill -what skills` (existing command) to embed them via bge, then VectorGuard re verifies. This is an operator step in Task 9, not code here. 145- [ ] **Step 4: Commit.** 146 147--- 148 149## Task 3: The DeterministicAliasResolver 150 151**Files:** Create `internal/modules/matching/aliasresolver.go`, `aliasresolver_test.go` 152 153It lazily loads `skills` (name to id), `skill_aliases` (lower(alias_text) to canonical skill_id, verified only), and `skill_relationships` (from_id to list of (to_id, relation, strength), verified only) into a global cache, the same shape and graceful degradation as `StoredSkillMatcher`. It exposes `ResolveAlias(ctx, required, candidateSkills) (engine.SkillMatch, bool)`. 154 155Resolution logic (deterministic, decimal): 1561. Resolve the required skill to a canonical skill id: `required.SkillID` if set, else the id for `lower(required.Name)` via the skills map, else the canonical id for `lower(required.Name)` via the alias map. If none, return no match. 1572. For each candidate skill, resolve its canonical id the same way (id, then name, then alias). 1583. If a candidate's canonical id equals the required canonical id (and it was NOT already an exact id or name match, which the engine handled), return `SkillMatch{CandidateSkillID, CandidateName, Similarity: 1.0, Exact: true, AliasMatched: true, Matched: true}`. 1594. Else, for each candidate canonical id, look up relationships from that id to the required id with relation in (`substitutes_for`, `broader_than`); if the best `strength >= 0.62`, return `SkillMatch{..., Similarity: strength, Exact: false, AliasMatched: false, Matched: true}` (record the relation in a trace friendly field). 1605. Else return no match (the engine falls through to semantic). 161 162- [ ] **Step 1:** Write `aliasresolver_test.go` first with an in memory resolver constructor (inject the three maps directly, no DB) and tests: (a) a candidate whose name is an alias of the required skill returns Similarity 1.0 Exact true AliasMatched true; (b) a candidate with a `substitutes_for` edge at strength 0.80 returns Similarity 0.80 Exact false; (c) a `substitutes_for` at strength 0.50 returns no match (below 0.62); (d) a `requires` or `related_to` edge returns no match; (e) an unverified alias (verified_at nil) is not loaded so returns no match. Run, confirm fail. 163- [ ] **Step 2:** Write `aliasresolver.go`: the cache struct, a DB backed loader (`SELECT ... WHERE verified_at IS NOT NULL`), the in memory constructor for tests, and `ResolveAlias`. All comparisons in `shopspring/decimal`. Run, confirm pass. 164- [ ] **Step 3:** `go test ./internal/modules/matching/...` and `go build ./...`, confirm green. Commit. 165 166--- 167 168## Task 4: The ChainedResolver (satisfies SkillMatcher, zero engine edits) 169 170**Files:** Create `internal/modules/matching/chainedresolver.go`, `chainedresolver_test.go` 171 172```go 173// ChainedResolver runs the deterministic alias and relationship resolver first, then the semantic 174// matcher, and satisfies the engine SkillMatcher interface so the engine code is unchanged. 175type ChainedResolver struct { 176 resolver *DeterministicAliasResolver 177 semantic engine.SkillMatcher 178} 179func NewChainedResolver(resolver *DeterministicAliasResolver, semantic engine.SkillMatcher) *ChainedResolver { ... } 180func (c *ChainedResolver) Match(ctx context.Context, required engine.RequiredSkill, cands []engine.CandidateSkill, threshold decimal.Decimal) (engine.SkillMatch, bool) { 181 if m, ok := c.resolver.ResolveAlias(ctx, required, cands); ok { 182 return m, true 183 } 184 if c.semantic == nil { 185 return engine.SkillMatch{}, false 186 } 187 return c.semantic.Match(ctx, required, cands, threshold) 188} 189``` 190 191- [ ] **Step 1:** Write `chainedresolver_test.go`: a fake resolver and a fake semantic matcher prove the order (resolver wins when it matches; semantic runs only on resolver miss; nil semantic degrades to resolver only). Run, confirm fail. 192- [ ] **Step 2:** Write `chainedresolver.go`. Run, confirm pass. `go build ./...`. Commit. 193 194--- 195 196## Task 5: SkillMatch.AliasMatched and ModelVersion v1.2.0 197 198**Files:** Modify `internal/modules/matching/engine/types.go` 199 200- [ ] **Step 1:** Add `AliasMatched bool` to the `SkillMatch` struct (default false, so every existing construction is unchanged and the frozen tests still pass). Bump `ModelVersion` constant to `"v1.2.0"`. 201- [ ] **Step 2:** Run `go test ./internal/modules/matching/engine/...`. The frozen test asserts `ModelVersion`; update that single assertion in `frozen_test.go` to `v1.2.0` (this is the deliberate, audited version bump, the only frozen change allowed). The three golden SCORES stay byte identical. Confirm green. 202- [ ] **Step 3:** Commit. 203 204--- 205 206## Task 6: New golden cases proving the new behavior (frozen three unchanged) 207 208**Files:** Modify `internal/modules/matching/engine/golden_test.go` 209 210- [ ] **Step 1:** Add two new golden cases to the snapshot, run through the engine WITH a ChainedResolver wired over an in memory resolver: 211 - Alias case: a requirement for skill SIGINT, a candidate holding Signals Intelligence (aliased to SIGINT). Skills component must be 100 and the total must equal the same requirement met by the exact skill. Assert the matched skill has `AliasMatched true`, `Similarity 1.0`. 212 - Relationship case: a requirement for skill B, a candidate holding skill A where A substitutes_for B at strength 0.80. Skills component reflects the partial; assert `Similarity 0.80`, `Exact false`. 213- [ ] **Step 2:** Run `go test ./internal/modules/matching/engine/ -run TestGolden`. Confirm the three frozen cases are byte identical and the two new cases pass. Commit. 214 215--- 216 217## Task 7: Seed the curated aliases and relationships 218 219**Files:** Create `db/seed/skill_aliases.sql` 220 221- [ ] **Step 1:** Author the seed: INSERT the load bearing GovCon aliases keyed to the promoted skills (resolve skill_id by name subselect), all `verified_at = now()`, for example: 222```sql 223INSERT INTO skill_aliases (skill_id, alias_text, alias_type, source, verified_at) 224SELECT id, 'Signals Intelligence', 'acronym', 'curated', now() FROM skills WHERE name='SIGINT' 225ON CONFLICT (lower(alias_text)) DO NOTHING; 226``` 227Cover SIGINT, GEOINT, HUMINT, COMSEC (Communications Security), CDS (Cross Domain Solution), ATO (Authority to Operate), RMF (Risk Management Framework), and the other high frequency acronyms. Add a handful of substitutes_for and broader_than relationships (for example Kubernetes broader_than Docker if both are skills) with explicit strengths and `verified_at = now()`. 228- [ ] **Step 2: Commit.** (This seed is applied in Task 9 on staging then production.) 229 230--- 231 232## Task 8: Wire the ChainedResolver into the service 233 234**Files:** Modify `internal/modules/matching/service.go` and/or `cmd/api/main.go` 235 236- [ ] **Step 1:** Find where `matching.NewStoredSkillMatcher(db)` is constructed and passed to the engine path. Replace it with `matching.NewChainedResolver(matching.NewDeterministicAliasResolver(db), matching.NewStoredSkillMatcher(db))`. The engine still sees a `SkillMatcher`, so nothing else changes. 237- [ ] **Step 2:** `go build ./...`, `go test ./...`, confirm green. Commit. 238 239--- 240 241## Task 9: Validate on M4 staging, then load production 242 243- [ ] **Step 1:** On a fresh `brain_staging` apply 0021 then 0022 (goose from M2 as migrator, the WS1 pattern), confirm both up and the 0022 down both work. 244- [ ] **Step 2:** Provision a real skills set: apply `db/seed/cleared_skills.sql`, run `cmd/promote-ontology` (needs ontology_terms; if staging is empty, run ontology-sync first or accept corpus 0), then `db/seed/skill_aliases.sql`, then `embed-backfill -what skills`, then verify VectorGuard passes. 245- [ ] **Step 3:** Prove the resolver end to end with a tiny screen: a candidate holding Signals Intelligence scores Skills 100 against a SIGINT requirement (alias), and a substitute skill scores the partial. Verify the served score audit records the AliasMatched flag. 246- [ ] **Step 4:** Production load (gated, backup first, like the WS1 cutover): apply 0022 to production via goose, apply the two seeds, run promote-ontology, run embed-backfill -what skills, verify. Restart mm-api so the resolver cache loads (the matcher caches on first use; a restart picks up the new skills and edges). 247- [ ] **Step 5:** Final commit and a final whole branch code review, then the finishing handoff. 248 249--- 250 251## Self-Review 252 2531. **Spec coverage:** WS2 in `docs/AI_GROUNDING_PROGRAM.md` calls for the alias and relationship layer with exact match before semantic, prerequisites and substitutions, and the engine staying the floor. Covered: tables (Task 1), promotion so there are skills to relate (Task 2), the resolver (Task 3), the zero edit engine integration (Tasks 4, 5, 8), the frozen preservation and new golden proof (Tasks 5, 6), the curated seed (Task 7), staging then production (Task 9). WS3 clearance is a separate plan. 2542. **Frozen contract:** the only allowed frozen change is the ModelVersion assertion bump to v1.2.0 (Task 5); the three golden SCORES stay byte identical because they use no matcher and no alias or relationship edge. 2553. **Type consistency:** `SkillMatch` (with the new `AliasMatched bool`), the `SkillMatcher` interface, `ChainedResolver`, and `DeterministicAliasResolver.ResolveAlias` are used identically across Tasks 3, 4, 6, 8. 256 257## Open decision for owner review 258The promotion scope (decision 1 above): curated roughly 100 to 150 skills, not the full 2797 skill kind rows. Confirm, or widen to include the broad O*NET tool set. 259