Janady13/missionmatch / docs/ARCHITECTURE.md

1# missionMATCH Architecture Blueprint (live) 2 3A Cleared Workforce Intelligence Platform for US government contractors. It serves one loop: 4**Requirement → Candidate Intelligence → Match Scoring → Ranked Slate → Proposal Readiness**, exposed as 5three tools over one deterministic engine, with an AI brain that narrates but never scores. 6 7Generated from the live codebase: 233 Go files, 22 modules, 18 platform packages, 19 migrations, 12 binaries, 8a React frontend. The deterministic engine is at v1.1.0 (semantic skill threshold 0.62, registered in the 9AEDT model registry). A companion rendered diagram is at `docs/architecture.svg`. 10 11Non negotiables baked into the structure: no Python, no cgo, the API binary stays `CGO_ENABLED=0 12GOFIPS140=v1.0.0` with `GODEBUG=fips140=on`; the deterministic engine is the floor (no model in any scoring 13or eligibility decision); EEO data is physically and privilege isolated and is never a matching input; 14multi tenancy is Postgres Row Level Security with a per request org GUC; every consequential action writes an 15append only audit row; cloud specific I/O (storage, embeddings, secrets, identity) sits behind ports so AWS 16GovCloud and Azure Government are configuration, not code. 17 18## 1. System context 19 20```mermaid 21flowchart TB 22 subgraph CLIENT["Browser SPA (Vite + React 19 + TS, Tailwind, TanStack Query)"] 23 SW["Tool switcher: screenMATCH /screen · slateMATCH /slate · missionMATCH /command · Compliance · Auth"] 24 end 25 EDGE["nginx (Bubbles, Debian) · TLS 1.2+ · reverse proxy"] 26 subgraph API["mm-api · single static Go binary · FIPS 140-3"] 27 MW["Pipeline: RequestID, Recoverer, metrics, Timeout, RateLimit, Session+MFA, CSRF, RBAC"] 28 CORE["Deterministic core: tool routers to matching.Service to engine.ScoreCandidate (the floor)"] 29 BRAIN["AI brain: orchestrator CAG/RAG/TAG/KAG + memory + faithfulness gate + vector gate"] 30 OTHER["Command Center · Staffing W3 · Governance/AEDT · EEO · Auth/MFA · Market catalogs"] 31 end 32 DB[("PostgreSQL 16 + pgvector (HNSW) · RLS forced · 3 roles")] 33 PLANE["MEGAMIND inference plane (Ollama: bge-small, llama3.1, bge-reranker) · stateless"] 34 FED["Free federal data: SAM.gov · USASpending · BLS OEWS · GSA FCO · PeopleDataLabs"] 35 36 CLIENT --> EDGE --> API 37 MW --> CORE 38 MW --> BRAIN 39 MW --> OTHER 40 CORE <--> DB 41 BRAIN <--> DB 42 BRAIN --> PLANE 43 OTHER <--> DB 44 FED -->|daily systemd sync| DB 45``` 46 47## 2. The one engine, three tools principle 48 49`engine.ScoreCandidate(ctx, Candidate, Requirement, SkillMatcher) MatchResult` in 50`internal/modules/matching/engine` is the single source of every match score. It is pure: no DB, no I/O, no 51clock, no randomness, all arithmetic in `shopspring/decimal` so the same inputs always yield byte identical 52scores and explanations (the AEDT reproducibility asset). It reads no demographic field, by type. 53 54``` 55Match = Clearance*0.25 + Skills*0.25 + Certifications*0.15 + CustomerExperience*0.15 56 + Experience*0.10 + LocationAvailability*0.10 (weights frozen, sum to 1.00) 57Bands: Green >= 90 (Proposal Ready) · Yellow 75 to 89 (Potential) · Red < 75 (Recruiting) 58Skills: exact match first, then SEMANTIC over stored bge HNSW vectors at cosine >= 0.62 (v1.1.0) 59Output: per component trace + human readable rationale + gap summary + ModelVersion stamp 60``` 61 62`screenMATCH`, `slateMATCH`, and `missionMATCH` are thin HTTP namespaces that mount the same 63`matching.Handler` (`ScreenRoutes`, `SlateRoutes`, `ReadRoutes`, `IntelRoutes`, `MissionRoutes`), so the same 64candidate against the same requirement shows the same number in all three. `commandcenter` and `staffing` 65read the frozen `candidate_evaluations` the engine persists; they never recompute. A frozen contract test 66pins the weights, bands, threshold, and ModelVersion so a refactor can never silently drift a number. 67 68## 3. Request lifecycle 69 70```mermaid 71flowchart LR 72 R["HTTPS request"] --> N["nginx TLS"] 73 N --> P["RequestID, Recoverer, metrics, Timeout 60s, RateLimit/IP"] 74 P --> S["Session middleware: validate __Host- cookie, MFA satisfied, resolve org+principal"] 75 S --> C["CSRF double submit"] 76 C --> B["RBAC RequireCapability (CapSearchRun, CapBrainQuery, ...)"] 77 B --> T["RunInTenantTx: SET LOCAL app.current_org_id from the VALIDATED session"] 78 T --> H["handler to service to engine / brain"] 79 H --> A["append only audit row"] 80 A --> O["typed JSON response"] 81``` 82 83No org in context yields zero rows (fail safe). The org GUC is set transaction locally and auto resets at 84COMMIT or ROLLBACK, which is the pgx pool correctness rule. A production build cannot compile the dev 85`X-Org-ID` header resolver (a build tag split, hard rejected again by config validation), and CI gates it. 86 87## 4. The AI brain (augmented generation, narrates never scores) 88 89`POST /ai/query` (capability `brain.query`) routes a natural language question. The brain may embed, narrate 90behind a faithfulness gate, and check faithfulness; it is never in a scoring or eligibility decision. 91 92```mermaid 93flowchart TB 94 Q["question"] --> CL{"Classify"} 95 CL -->|"demographic or injection"| REF["Refuse (422): ForbidEEO + MentionsProtectedAttribute"] 96 CL -->|"stable rules"| CAG["CAG: preloaded public rules, no retrieval"] 97 CL -->|"retrieval"| RAG["RAG: embed query, hybrid pgvector + tsvector, RRF fuse, narrate"] 98 CL -->|"explicit intent"| TAG["TAG: curated parameterized read only SQL intent (allowlisted, never eeo)"] 99 RAG --> KAG["KAG: skill adjacency over HNSW (live, no model)"] 100 RAG --> FG{"Faithfulness gate"} 101 FG -->|"grounded"| ANS["model prose (rationale_source ai_faithful)"] 102 FG -->|"not grounded"| DET["deterministic rendering (rationale_source deterministic, still 200)"] 103 ANS --> AUD["audit aedt.ai.answered (no PII, no prose)"] 104 DET --> AUD 105 REF --> AUD2["audit aedt.ai.refused"] 106``` 107 108Supporting guarantees: 109- **Vector gate (enforced re embed).** At boot the API asserts the `skills.embedding` and 110 `candidates.resume_embedding` columns are vector(384) and re embeds a sample of stored skills with the 111 configured model; if the stored vectors were made by a different model (the all-minilm versus bge 112 corruption mode), `/ai/query` returns 503 `vector_model_mismatch` while the rest of the API serves, and it 113 self heals on the first query after a boot time plane outage. 114- **Per tenant memory.** `ai_memory` (episodic and semantic, vector embedded) is RLS scoped, eeo guarded on 115 write, and wiped on erasure in both the on demand and retention paths. 116- **EEO unreachability.** No brain path names the eeo schema; a compile time tripwire walks the ai package 117 and fails the build if it does. 118 119## 5. Multi tenancy and security model 120 121```mermaid 122flowchart TB 123 subgraph ROLES["Postgres roles"] 124 APP["missionmatch_app · NOBYPASSRLS"] 125 MIG["missionmatch_migrator · BYPASSRLS"] 126 CMP["missionmatch_compliance"] 127 end 128 subgraph DATA["one shared database (per tenant DB routing is a sequenced follow on)"] 129 T["RLS tenant tables: candidates, requirements, slates, candidate_evaluations, staffing_recommendations, ai_memory"] 130 G["global catalogs (no org_id): skills HNSW, opportunity_records, sam_opportunities, usaspending_awards, bls_wages"] 131 SEC["secrets OUTSIDE RLS: auth_credentials, auth_sessions, mfa_totp, recovery, invites"] 132 EEO["eeo schema: candidate_demographics"] 133 AUD["audit_logs (append only)"] 134 end 135 APP -->|org GUC, FORCE RLS| T 136 APP -->|read| G 137 APP -.->|no table grant; SECURITY DEFINER fns only| SEC 138 APP -.->|REVOKE ALL, permission denied| EEO 139 MIG -->|migrations, sync writes, provisioning| DATA 140 CMP -->|SELECT public outcomes + eeo, one way to report| EEO 141``` 142 143- **RLS.** Every tenant table has `organization_id` leading its keys and `FORCE ROW LEVEL SECURITY`. The app 144 sets `app.current_org_id` per request transaction from the validated session, never a client value. 145- **Secrets.** The five secret tables carry no app grant and are reached only through about twenty migrator 146 owned `SECURITY DEFINER` functions with a locked `search_path`. 147- **EEO isolation (absolute).** The `eeo` schema lives in a separate goose track and is not in the sqlc path, 148 so no Go model is ever generated for it; only `missionmatch_compliance` has grants (SELECT and append only 149 INSERT); the matching role gets a permission denied. Demographics flow one way into the four fifths adverse 150 impact report and never into a score. 151- **Auth and accounts.** Opaque server side sessions (32 random bytes, stored only as a SHA 256 hash, in a 152 `__Host-` cookie, two stage pre MFA then active), PBKDF2 HMAC SHA 256 at 600000 iterations with a versioned 153 pepper, mandatory in house TOTP pinned to SHA 256 with a monotonic replay guard, TOTP secrets AES 256 GCM 154 encrypted with a key version. Org creation is migrator only provisioning plus invites, no public signup in 155 this phase. 156- **Audit.** `audit_logs` is append only outside RLS, protected by row, statement, and privilege guards; 157 org scoped events ride the tenant transaction, pre auth security events insert directly and fail closed. 158 159## 6. Schema (migrations 0001 to 0019) 160 161| # | Migration | What it adds | 162|------|-----------|--------------| 163| 0001 | foundation | organizations, users (composite FKs), audit_logs (append only) | 164| 0002 | matching | skills (global HNSW), candidates, requirements, slates, slate_members, the candidate/requirement detail tables | 165| 0003 | command_center | opportunity value, requirement labor category and quantity | 166| 0004 | command_center_fns | immutable clearance and poly ordinal functions (engine synced) | 167| 0005 | sam_opportunities | SAM.gov contract cache (global, outside RLS) | 168| 0006 | usaspending_awards | USASpending awards cache (global) | 169| 0007 | bls_wages | BLS OEWS regional wage baseline (global) | 170| 0008 | external_sourcing | candidate source fields, sourcing_log | 171| 0009 | auth | credentials, sessions, MFA, recovery, invites, login attempts + SECURITY DEFINER functions | 172| 0010 | aedt_governance | model_version on decisions, anonymized_at, model version registry, AEDT inventory, candidate notices | 173| 0011 | opportunity_value_source | value provenance (representative/operator/sam/usaspending) | 174| 0012 | clearance_provenance | clearance provenance and verified_at on candidates | 175| 0013 | opportunity_records | unified global opportunity feed (SAM, awards, forecasts) | 176| 0014 | notice_disclosed_by_fk | composite FK for the notice discloser | 177| 0015 | candidate_evaluations | frozen engine result per candidate and requirement (the reconciliation table) | 178| 0016 | slate_member_decision_note | human decision note on a slate member | 179| 0017 | ai_memory | per tenant episodic and semantic memory, vector(384), erasable | 180| 0018 | staffing_recommendations | recommendation_runs and staffing_recommendations (W3) | 181| 0019 | model_version_v1_1_0 | registers v1.1.0 (semantic threshold recalibrated 0.80 to 0.62) | 182 183## 7. Deployment topology and the inference plane 184 185```mermaid 186flowchart LR 187 subgraph BUB["Bubbles (Debian, public edge)"] 188 NGX["nginx TLS"] --> SVC["mm-api (systemd, GODEBUG=fips140=on, /etc/missionmatch/api.env 0600)"] 189 CRON["systemd timers: sam-sync, awards-sync, opp-sync, rates-sync, staffing-proactive"] 190 end 191 subgraph M4["M4 / Thunderport (Apple Silicon, Tailscale 100.94.8.94)"] 192 PG[("Postgres 16 + pgvector")] 193 GW["megamind-gateway (launchd) :8090"] 194 OLL["Ollama (Metal): bge-small-en-v1.5, llama3.1:8b, bge-reranker-v2-m3"] 195 end 196 SVC -->|pgx over Tailscale| PG 197 SVC -->|typed inference client: timeouts, retries, circuit breaker| GW 198 GW --> OLL 199 CRON --> PG 200``` 201 202The gateway is stateless and sees no client identifier on the inference paths; it exposes `/v1/embed`, 203`/v1/rerank`, `/v1/generate`, `/v1/faithfulness`, and a scrub guarded `/v1/teach` that persists nothing. The 204embeddings provider is a port (fake for dev and tests, plus openai, azure_openai, bedrock, and megamind), so 205a cloud swap is configuration. The same `embeddings.SkillEmbedText` defines the skill embed text for both the 206backfill and the boot vector gate, so they cannot drift. 207 208## 8. Binaries 209 210`api` (the server), `megamind-gateway` (the inference plane), `embed-backfill` (skills and resume vectors), 211`provision-org` and `provision-owner` and `provision-tenant` (account and tenant genesis), and the data sync 212jobs `sam-sync`, `awards-sync`, `opp-sync`, `rates-sync`, `import-candidates`, and `staffing-proactive-sync`. 213 214## 9. Responsible AI (AEDT) guarantees 215 216This is an Automated Employment Decision Tool. The platform supplies the data and controls for the deploying 217employer's bias audit; it does not transfer their liability. Every score is explainable (per component trace, 218rationale, gaps, a recordable decision trace) and reproducible (decimal, frozen weights, stamped ModelVersion 219registered in `aedt_model_versions`). Humans approve or reject every slate member and every staffing 220recommendation; nothing is auto rejected. Demographic data is never a feature or proxy. The four fifths 221adverse impact report runs only on the isolated compliance role. A model card lives at `docs/MODEL_CARD.md`. 222