# WS9 Bench Ingestion Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax. **Goal:** A CUI safe bulk loader that ingests a real cleared workforce bench (resumes and structured profiles) into a tenant at scale, server side extracted, deduped, embedded, and mission tagged, so the brain matches requirements against a real bench instead of demo gap fillers. **Architecture:** A new `internal/platform/benchimport` package with a `FileSource` port (resume directory + structured CSV/JSONL) yielding a normalized `BenchRecord` mapped onto the existing `sourcing.SourcedPerson`. Resume text comes from the pure Go `docextract` reader and the deterministic `extraction.Extractor` (no browser, no provider key). Ingestion reuses the existing `sourcing.IngestPerson` dedup, adds an embed step (the vector composed and embedded BEFORE the write tx so the network call is out of the lock window), and writes `candidate_mission_tags` through the app role under RLS. A migrator role CLI `cmd/import-bench` drives it. The frozen engine is never touched. **Tech Stack:** Go 1.26, pgx/v5, sqlc v1.31.1, goose v3.24.1, shopspring/decimal, pgvector, FIPS 140-3 (`GOFIPS140=v1.0.0`, `CGO_ENABLED=0`), no Python. **Branch:** `ws9-bench-ingestion` off `ws8-teaching-loop` (the linear grounding stack; master is the merged ws1 to ws7). Migrations resume at **0032** (WS8 took 0028 to 0031; brain_staging is already at goose v31). Validate on M4 `brain_staging` only, never production. --- ## Hard invariants (every task) — the seven WS9 review corrections, mandatory 1. **Embed gate is the REAL probe, not Health.** `inference.Client.Health` (client.go:72) is a bare liveness ping with no model identity; it cannot detect a model mismatch. Before any inline embed, run the real vector probe: reuse `cmd/verify-vector-model` / the `ai.VectorGate.Check` logic (vectorguard.go:55) to embed a fixed probe string and compare cosine against the known `qllama/bge-small-en-v1.5` reference, and REFUSE to embed (exit non zero) on mismatch. Keep the `Embed` width check (client.go:90) as a secondary guard. `-embed` is gated on the probe passing, never on Health. 2. **Mission tags are written through the APP role under RLS, never the migrator tx.** The WS7 invariant (tenant_tag.go:9) is that EVERY `candidate_mission_tags` write runs inside `RunInTenantTx` as the NOBYPASSRLS app role so the RLS WITH CHECK validates the org. WS9 must NOT collapse the tag write into the migrator `importOne` tx. Use the `cmd/tag-missions` two pool pattern: migrator for the candidate plus skills plus embed plus `bench_import_log` writes, and a SEPARATE app pool to write `candidate_mission_tags` inside `RunInTenantTx` (org from `WithOrg`, never a caller arg) after the candidate commits. 3. **Clearance stays `unknown` on import; a resume regex never scores as held.** Reuse `sourcing.IngestPerson` behavior verbatim: `UpsertExternalCandidate` (sourcing.sql:11) hardcodes `clearance_level='unknown'`, `clearance_poly='none'`, `clearance_status='unknown'`. The resume extracted `RawExtraction.ClearanceLevel` (extraction.go:30) MUST NEVER reach `candidates.clearance_level`. Clearance becomes known only later via an explicit org FSO action (`SetClearanceProvenance`, missioncheck.go). Add a test asserting `clearance_level` stays `'unknown'` on import so it can never feed `scoreClearance` (engine weight 0.25). If the founder wants the extracted code captured for triage, store it ONLY in `bench_import_log` (a non scoring field), clamped, provenance forced `self_reported`. 4. **The EEO tripwire must walk the new code.** Extend `safety_test.go:73` `dirs` (today `[".", "../staffing"]`) to include `internal/platform/benchimport`, `internal/platform/sourcing`, and `cmd/import-bench` (use a relative path that the walk reaches, or add an equivalent walk test in the benchimport package). Prove it green. Confirm `BenchRecord`, the embed profile composer, and the CSV/JSONL column mapping carry NO demographic field. 5. **Resume PII never reaches the embed string or the inference plane.** The embed profile (embed-backfill/main.go:113-118) is name free (headline = JobTitle/Role + first 6 skills, plus skills/certs/agencies/clearance/years). The resume FileSource must map ONLY structured tokens (skills, certs, agencies, clearance code, years, a sanitized JobTitle from the title gazetteer) into `SourcedPerson`, and must NEVER place `FullName`, an email, a phone, or a raw resume span into `JobTitle`, `Role`, or `headline`. `FullName` goes ONLY to `candidates.full_name` (never embedded). Add a unit test that feeds a resume containing a name and email and asserts the composed embed profile string contains neither. 6. **Embed is composed and run BEFORE the write tx.** Do not hold a write transaction open across the embed network round trip (a slow plane holds the row lock for the embed timeout). Compose the embed profile and call `inference.Client.Embed` BEFORE opening the migrator write tx, then pass the finished `vector(384)` into `importOne` so the tx is pure DB writes and the candidate plus vector commit atomically. On embed failure, commit the row WITHOUT the vector and rely on the documented `embed-backfill -only-missing` fallback. 7. **Legal basis is explicit, owner provided, never the PDL string.** Do NOT default `-legal-basis` to the PDL legitimate interest string. Require the operator to pass an explicit `-legal-basis` and refuse to run if empty; the default wording (when documented) is an owner provided controller basis, for example "client provided cleared bench, client is the data controller". Record `source` as `bench_file` or `bench_csv`, never `PDL`. Confirm the exact wording with the founder before the first real Galapagos run. Plus the standard rules: no Python; no dashes in written output (comments, logs, UI copy, docs, migration COMMENT) except URLs, CLI flags, import paths, identifiers; the frozen engine under `internal/modules/matching/engine` is never modified and the golden contract (100.00, 87.50, 26.00) stays green; FIPS stamp gate on the CLI; both cloud (384 dim through the provider interface); decimal not float where it matters (WS9 does no scoring math); RLS multi tenancy; one `bench_import_log` row per committed record; append only audit counts and model only. --- ## File Structure - Create `internal/platform/benchimport/` (`benchimport.go` the `BenchRecord` + `FileSource` port + `BatchResult`, `resume_source.go`, `structured_source.go`, `_test.go`). - Modify `internal/platform/sourcing/` to add `IngestBenchRecord` (or an options struct on `IngestPerson`) for the embed and tag hooks. - Create `db/migrations/0032_bench_import_log.sql` + `db/queries/bench_import.sql` (`InsertBenchImportLog`). - Create `cmd/import-bench/main.go`. - Modify `internal/modules/ai/safety_test.go` (extend the EEO walk). - Modify `Makefile` (import-bench build target + FIPS stamp gate). - Create `docs/RUNBOOK_BENCH_IMPORT.md`. --- ## Phase WS9.0: the benchimport package and FileSource port (PII safe mapping) - [ ] Create `internal/platform/benchimport`: `BenchRecord` (structured tokens only, NO demographic field), a `FileSource` interface (`Resumes(dir)` and `Profiles(path)` iterators), and a `BatchResult` counter. Pure types, no DB, no clock. - [ ] Resume FileSource: walk a directory, read each file, call `docextract.ExtractText`, run `extraction.Extractor.ExtractResume`, map `RawExtraction` onto `sourcing.SourcedPerson` honoring invariant 5 (FullName only to the name field, never to JobTitle/Role/headline; a sanitized JobTitle from the title gazetteer only). Unit test with the docextract testdata fixtures plus a sample resume. - [ ] Structured FileSource: decode CSV (header mapped columns) and JSONL into `sourcing.SourcedPerson` with the standard library. Unit test the column mapping and a malformed row skip. - [ ] **Invariant 5 test:** feed a resume containing a name and an email; assert the composed embed profile string contains NEITHER. Commit. --- ## Phase WS9.1: ingest extension (embed before tx, tag via app pool, clearance unknown) - [ ] Add `IngestBenchRecord` (or an options struct on `IngestPerson`) in `internal/platform/sourcing` that reuses the existing dedup and skills/certs/agencies writes verbatim. Clearance stays `unknown` (invariant 3): do not pass the extracted `ClearanceLevel` to any clearance column. - [ ] Embed BEFORE the tx (invariant 6): the caller composes the name free embed profile and calls `inference.Client.Embed` outside the write tx, then passes the finished vector into the ingest function which UPDATEs `resume_embedding` in the same migrator tx as the candidate. On a nil vector (embed skipped or failed) the row still commits without the embedding. - [ ] Tag via the app pool (invariant 2): the tag write is NOT in this migrator function. Expose a separate `TagCandidateMissions(ctx, appDB, candidateID, idx)` that the CLI calls inside `RunInTenantTx` on the app pool after the candidate commits, reusing `UpsertCandidateMissionTag` and `missionclass.Classify`. - [ ] Unit test: the embed skip path (nil client commits the row without a vector), the clearance stays unknown assertion, and the tag classification with an injected fake embedder and a fixed AliasIndex, no DB. Commit. --- ## Phase WS9.2: migration 0032 and the compliance log - [ ] `db/migrations/0032_bench_import_log.sql`: tenant scoped `bench_import_log` (organization_id uuid, candidate_id uuid, source text, file_or_batch text, legal_basis text, collected_at timestamptz default now()), ENABLE plus FORCE ROW LEVEL SECURITY, the `NULLIF(current_setting('app.current_org_id', true),'')::uuid` USING and WITH CHECK isolation policy, an organization_id leading index, GRANT SELECT and INSERT to missionmatch_app, tested down. Mirror `0008_external_sourcing.sql` exactly. - [ ] Add `InsertBenchImportLog` to `db/queries/bench_import.sql`, run `sqlc generate`. Commit. (Controller applies 0032 on brain_staging.) --- ## Phase WS9.3: the CLI - [ ] `cmd/import-bench/main.go`: flags `-resumes`, `-profiles`, `-org`, `-legal-basis` (required, invariant 7), `-embed`, `-dry-run`, `-size`. FIPS self test at boot (mirror cmd/tag-missions). Assert the migrator role on the migrator pool AND the app role on the app pool (two pools, invariant 2). Build the AliasIndex once from the global alias pairs through the migrator. When `-embed` is set, run the REAL vector probe gate (invariant 1) at startup and exit non zero on mismatch. - [ ] Per record: compose + embed BEFORE the write (invariant 6); `importOne` writes the candidate plus skills plus embed plus `bench_import_log` in one migrator tx; then write `candidate_mission_tags` via the app pool inside `RunInTenantTx` (invariant 2). Record `source` = `bench_file` or `bench_csv` (invariant 7). A single `bench.import.batch` audit summary at the end (counts and model only). A `-dry-run` mode extracts and resolves but rolls back so an operator previews dedup hits without writing. Commit. --- ## Phase WS9.4: EEO tripwire, staging validation, runbook - [ ] Extend `safety_test.go` (invariant 4) to walk `internal/platform/benchimport`, `internal/platform/sourcing`, and `cmd/import-bench`; prove green. Commit. - [ ] Build FIPS validated, verify the GOFIPS140 stamp, run the unit suite and the golden plus frozen engine tests to prove the engine is byte identical and untouched. - [ ] On M4 brain_staging (controller): apply 0032, run import-bench against sample cleared resumes plus a small CSV into a test org, and assert: candidate_skills resolved to canonical ontology skills (Signals Intelligence to SIGINT), `resume_embedding` populated and HNSW searchable, `candidate_mission_tags` present, one `bench_import_log` row per record, **clearance stored `unknown`**, and Command Center bench coverage reflecting the new bench. Run the dedup test (re run the same import, assert enrich not duplicate). - [ ] Write `docs/RUNBOOK_BENCH_IMPORT.md` (no dashes): the dry run, the small batch posture, the clearance self reported invariant, the explicit legal basis requirement, and how to re run `embed-backfill -only-missing` if the inline embed was skipped. Commit. --- ## Self review - The seven review corrections are each an enumerated hard invariant and each maps to a phase: embed probe gate (1, WS9.3), app role tag write (2, WS9.1 + WS9.3), clearance unknown (3, WS9.1), EEO walk (4, WS9.4), resume PII (5, WS9.0), embed before tx (6, WS9.1), explicit legal basis (7, WS9.3). - The frozen engine is never in a modify list. WS9 writes data and vectors only; it never imports the engine for scoring. Migration 0032 is the only schema change, tenant scoped, mirroring 0008.