Source identity
| Field | Recorded value |
|---|---|
| Source type | GitHub source |
| Original record | Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md |
| Date | Not specified in source |
| Repository | Janady13/missionmatch |
| Revision | 720d47fae476b0d38477ae4401697d372b1ea2bc |
| Branch | feature/opportunity-to-outcome |
| Path or archive member | docs/plans/2026-06-30-graphrag-phase0-foundation.md |
| Original SHA-256 | f2e3fa398a0188d7687e5de96210838f5467216a44d5537269c179daa6084276 |
| Git blob SHA | 81f18f22ff5fdd5d80e59bdce64173aac000820f |
| Reading-copy SHA-256 | f2e3fa398a0188d7687e5de96210838f5467216a44d5537269c179daa6084276 |
Every recorded occurrence
These 12 records contain identical original source bytes. Their paths and revisions remain part of the archive.
| Original record | Branch or archive | Revision / identifier | Original location |
|---|---|---|---|
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | feature/opportunity-to-outcome | 720d47fae476b0d38477ae4401697d372b1ea2bc | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | collision-deconfliction | 4ee5a09972608ed89f8377c6665542d5f450f0a6 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | m5-price-to-win | 41000ad30fff4129fdaf5ec84e15ba14c27d4890 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | master | 1e9249840a4302ec82e123b9c9cb6f75badea3e4 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | mag-substrate | b82c7100a492dec67dfd1eda6b3f53def859be35 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | time-to-fill-survival | b66e1e505863d4f17741a4b9ede73509e45bd6c6 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | graphrag-foundation | b0ddeb9f633fd138d09309d1a17b7d13bc3a67f7 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | presentation-readiness | eb574acaa29d71ce4d0f852572e87c275c79fdb7 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | vendor-intelligence | fe9796a19c261951d3f2565206c0c1b5ac294465 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | phase2-connectors | e038d4c687aec16f8bd9bc4d907eea14d91f8704 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | feature/aioe-upgrade | 2178e7298d0e5d9af83afa12402fd587f0ba6ed8 | Open original |
| Janady13/missionmatch / docs/plans/2026-06-30-graphrag-phase0-foundation.md | capture-readiness | bae6af7cb2997e819174a9e9cfff5e16067a1975 | Open original |
Recovered source transcript
Download complete source text ↓108 lines · 12.4 KB
# GraphRAG Phase 0: Foundation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax.
**Goal:** Stand up the deterministic graph reasoning substrate and its first two public data layers, so the brain crosses from single table lookup to walking a cited evidence path: G3 multi hop skill resolution and G1 the market and relationship backbone, reached through a new RouteGraph surface, with the shared TAG safety guard fixed to allow recursive traversal.
**Architecture:** One graph, walked as recursive and aggregate reads over the relational tables already in Postgres (no second datastore, no Neptune, no Cosmos, no Python graph library, both cloud identical on RDS and Azure Flexible Server). Every traversal is a named, parameterized QueryIntent in the existing CLOSED Registry, never model authored SQL, run through ExecuteIntent and GuardReadOnlyQuery. A new deterministic RouteGraph maps a natural language question to one graph intent, runs ForbidEEO and MentionsProtectedAttribute FIRST, executes the traversal, and narrates the result behind the existing faithfulness gate. The frozen deterministic engine stays the ONLY scorer; graphs reason about structure and route to it, never compute a score.
**Tech Stack:** Go 1.26, chi, pgx/v5, sqlc v1.31.1, shopspring/decimal, FIPS 140-3 (`GOFIPS140=v1.0.0`, `CGO_ENABLED=0`), no Python.
**Branch:** `graphrag-foundation` off `ws9-bench-ingestion` (the linear grounding stack; migration head 0032). Phase 0 adds NO migration: G3 reuses the WS1 ontology + WS2 skill_relationships/skill_aliases + ontology_crosswalk + mission_areas; G1 reuses the WS7 opportunity_records, usaspending_awards, sam_opportunities, federal_mission_tags, award_incumbent_roles. Validate on M4 `brain_staging` only (goose v32), never production.
---
## Hard invariants (every task)
1. **The frozen engine is the ONLY scorer.** No graph intent computes a match score, a band, or any employment decision number. Graphs select, traverse, and READ engine outputs (`candidate_evaluations` as stored) but never call a scoring path and never feed an LLM derived artifact into a feature. Market aggregates (dollar share, competitor rank) are computed in `shopspring/decimal` and are explicitly NOT match scores. `internal/modules/matching/engine` is not modified; golden/frozen tests stay byte identical.
2. **EEO is structurally unreachable.** No graph intent references the eeo schema or any demographic column or proxy. RouteGraph runs `ForbidEEO` and `MentionsProtectedAttribute` on the question FIRST (the orchestrator precedent). Every intent passes `GuardReadOnlyQuery` and `ValidateRegistry`. The `safety_test.go` EEO walk stays green.
3. **Closed Registry, never model authored SQL.** Every traversal is a FIXED, named QueryIntent with bound params, added to `Registry` and asserted by `ValidateRegistry` at boot. RouteGraph selects a registered intent and binds params; it NEVER concatenates table names or predicates. The recursive CTE text is static and reviewed.
4. **Deterministic, byte stable traversal.** Recursive CTEs carry a hard depth cap, a cycle guard (a visited set or `UNION` not `UNION ALL`), and a unique id tiebreaker on every `ORDER BY`, so the same inputs yield byte identical rows (the AEDT reproducibility mandate, the same discipline the live RRF path uses). Prefer a `full_parent_path` string split over recursion where the data already supports it (the agency hierarchy).
5. **ReadAllowlist changes are reviewed security decisions.** Add ONLY the tables Phase 0 intents reference, one reviewed change, all GLOBAL public reference or public market tables (no PII, safe cross tenant): for G3 add `skill_relationships`, `skill_aliases`, `ontology_crosswalk`, `mission_areas`; for G1 add `federal_mission_tags`, `award_incumbent_roles` (opportunity_records, usaspending_awards, sam_opportunities already present). Keep eeo, auth secret tables, and audit_logs OUT. No wildcard.
6. **Faithfulness gated narration.** The LLM touches only the final narration step, behind the existing per claim faithfulness gate (`finalize`); a low confidence answer falls back to the deterministic rendering of the evidence, never a public web fetch (CUI). The `aedt.ai.answered` audit carries route, rationale source, evidence count only, never a UEI, a candidate id, a score, or answer text.
7. Standard rules: no Python; no dashes in written output except URLs, CLI flags, import paths, identifiers; both cloud (core ANSI SQL plus the existing pgvector, no cloud specific graph primitive); FIPS; decimal not float for any surfaced number.
---
## File Structure
- Modify `internal/modules/ai/safety.go`: fix `cteNameRe` for `WITH RECURSIVE`; expand `ReadAllowlist`.
- Modify `internal/modules/ai/safety_test.go`: negative and positive guard tests for recursive CTEs.
- Create `internal/modules/ai/graph_skill.go` (+ `_test.go`): the G3 skill adjacency QueryIntents and the transitive walk.
- Create `internal/modules/ai/graph_market.go` (+ `_test.go`): the G1 market QueryIntents.
- Modify `internal/modules/ai/tag.go`: register the new graph intents in `Registry`.
- Create `internal/modules/ai/graph_route.go` (+ `_test.go`): RouteGraph + AnswerGraph.
- Modify `internal/modules/ai/orchestrator.go`: add `RouteGraph` and dispatch.
- Integration tests under `internal/integration/`.
---
## Phase 0.A: the recursive CTE guard fix (security critical, do first)
### Task A1: fix cteNameRe and expand ReadAllowlist
**Files:** `internal/modules/ai/safety.go`, `internal/modules/ai/safety_test.go`
- [ ] Fix `cteNameRe` (safety.go:94) to tolerate the optional `RECURSIVE` keyword after `WITH`:
`var cteNameRe = regexp.MustCompile(`(?i)(?:with(?:\s+recursive)?|,)\s+([a-z_][a-z0-9_]*)\s+as\s*\(`)`
- [ ] Add to `ReadAllowlist` (one reviewed change, with a comment splitting the two classes): the GLOBAL public reference tables `skill_relationships`, `skill_aliases`, `ontology_crosswalk`, `mission_areas`, and the public market tables `federal_mission_tags`, `award_incumbent_roles`. Keep eeo, auth, audit OUT.
- [ ] **Negative and positive guard tests** (the load bearing part, treat as a security change): a `WITH RECURSIVE closure AS (...) SELECT ... FROM closure` whose base tables are all allowlisted PASSES; a recursive CTE referencing a non allowlisted base table still FAILS; a hidden write inside a recursive CTE (`WITH RECURSIVE x AS (... DELETE ...)`) still FAILS; the existing `TestGuardReadOnlyQuery` negatives (stacked statements, hidden DELETE, non allowlisted table, eeo marker) still REJECT; multi CTE `WITH RECURSIVE a AS (...), b AS (...)` registers BOTH a and b.
- [ ] Run `go test ./internal/modules/ai/` and prove green. Commit `fix(graphrag): allow WITH RECURSIVE in the read only guard and allowlist the graph reference tables`.
---
## Phase 0.B: G3 Skill and Role Adjacency graph
### Task B1: the transitive skill walk intents
**Files:** Create `internal/modules/ai/graph_skill.go`, `_test.go`; modify `tag.go` Registry.
- [ ] Author FIXED QueryIntents (recursive CTE over `skill_relationships` WHERE `verified_at IS NOT NULL` and `relation IN (substitutes_for, broader_than)`, plus `ontology_crosswalk` for taxonomy hops), each with a hard depth cap (for example 3 hops), a visited set cycle guard, a unique id tiebreaker, and bound params (the start skill, the depth):
- `skill_substitution_closure`: from a skill, the transitive substitutes_for and broader_than neighborhood with the curated strength per hop (carried as numeric to decimal in Go, never float), and the evidence path (the chain of edges).
- `skill_gap_bridge`: given a required skill not exactly held, the nearest curated skills one or two hops away and the cert (from `ontology_crosswalk`) that would close it.
- [ ] Register them in `Registry`; confirm `ValidateRegistry` passes at boot.
- [ ] Tests: the closure terminates at the depth cap, a cycle does not loop, the strength accumulates in decimal, the evidence path is the real edge chain, and the result is byte stable (same input same rows). Live integration test on brain_staging proving the SIGINT to Signals Intelligence curated hop resolves. Commit.
---
## Phase 0.C: G1 Market and Relationship graph
### Task C1: the capture traversal intents
**Files:** Create `internal/modules/ai/graph_market.go`, `_test.go`; modify `tag.go` Registry.
- [ ] Author FIXED QueryIntents over `usaspending_awards` + `opportunity_records` + `sam_opportunities` + `federal_mission_tags` + `award_incumbent_roles` (mostly aggregates and a `full_parent_path` string split for the agency hierarchy, NOT recursion where avoidable; recursion only for genuine multi hop with the depth cap and guards). Honestly key awards self joins on `(agency, naics)` only (awards carry no psc/set_aside/pop_state, per the schema), and label every incumbent edge `prime-of-record` (the sub award staffing reality is a later acquisition):
- `segment_competitor_field`: agency + NAICS to the recency weighted dollar share of recipients (wired vs open), evidence the specific awards.
- `incumbent_of_record`: an opportunity to its predecessor award via `(agency, naics)` and `dedupe_key`, returning the prime UEI, explicitly flagged prime-of-record.
- `recompete_pipeline`: agency to incumbent_on rows whose `pop_end_date` falls in a 12 to 24 month window, the takeaway pipeline.
- `agency_buying_pattern`: agency and sub_agency (via `full_parent_path` split) to NAICS/PSC distribution, set_aside mix, estimated_value band, the where to position view.
- [ ] All dollar and share math in `shopspring/decimal`. Register in `Registry`; `ValidateRegistry` passes.
- [ ] Tests: each intent returns byte stable rows with the cited evidence path; the agency rollup uses the string split not recursion; the incumbent edge is labeled prime-of-record. Live integration test on brain_staging (it has ~1899 opportunity_records + 522 awards) proving a real competitor field and recompete pipeline come back. Commit.
---
## Phase 0.D: RouteGraph and AnswerGraph
### Task D1: the routing surface
**Files:** Create `internal/modules/ai/graph_route.go`, `_test.go`; modify `orchestrator.go`, `handler.go`.
- [ ] Add `RouteGraph Route = "graph"` to the Route consts. In `Classify` (or a dedicated graph classifier), deterministically map a question that names a capture or skill graph traversal (incumbent, competitor, recompete, who teams, what substitutes for X, agency buying) to `RouteGraph` plus the chosen graph intent and bound params, from a curated keyword to intent table (never an LLM, mirroring how `Classify` routes count/sum to TAG). EEO and protected attribute checks run FIRST.
- [ ] `AnswerGraph(ctx, question, intent, args)`: runs `ForbidEEO` and `MentionsProtectedAttribute`, executes the registered graph intent via `ExecuteIntent`, builds the deterministic evidence rendering (the cited path), then narrates through `o.generate` and `o.finalize` (the faithfulness gate), exactly as `AnswerTAG` does. A plane outage degrades to the deterministic rendering, never a hang. The audit is `aedt.ai.answered` route graph, counts only.
- [ ] Wire `RouteGraph` into `Answer` dispatch and the handler. Keep `RouteRAG`, `RouteTAG`, `RouteDirect` unchanged (default path invariance).
- [ ] Tests: a graph question routes to the right intent and is gated; an EEO laced question refuses; a plane outage falls back deterministically; the existing TAG and RAG routing is unchanged. Live integration test: a natural language capture question returns a cited graph answer on brain_staging. Commit.
---
## Phase 0.E: review and validation
- [ ] Dispatch a final adversarial review over the whole Phase 0 diff (the guard change especially; the engine is untouched; EEO unreachable; recursive determinism; decimal; closed registry; both cloud).
- [ ] Re run golden/frozen/safety; prove byte identical engine. Run the full integration suite on brain_staging. Write `docs/RUNBOOK_GRAPHRAG.md` if operational notes are warranted. NOT merged, NOT deployed.
---
## Self review
- The cross cutting CRITICAL blocker (the `WITH RECURSIVE` guard bug that five layer verifiers independently caught) is Phase 0.A, fixed once for every future layer with its own negative test.
- The frozen engine is never in a modify list. Graphs route to it and read its stored outputs; they never score. No migration (Phase 0 reuses existing tables). The person touching layers (G4 onward) are NOT in Phase 0; this phase is public and global data only, so there is no tenant privacy surface beyond the uniform RunInTenantTx posture.