# STACK_DECISIONS.md — missionMATCH grounding doc (single source of truth) > This document is **the law** for version-sensitive Go/Postgres/cloud choices, per `CLAUDE.md` > ("Don't blindly build" rule). It is research-backed, source-cited, and every Go pattern below was > **compile-verified** under the locally-installed toolchain `go1.26.4 darwin/arm64` > (`/opt/homebrew/bin/go`). Where something could only be confirmed against a live Postgres 16 / AWS > GovCloud / Azure Government — and therefore could NOT be proven on a dev Mac — it is called out > explicitly in **OPEN QUESTIONS / verify during build**. Build the platform to these pins; do not > "upgrade to latest" without re-verifying against this doc. > > Last verified: 2026-06-23. Toolchain: Go 1.26.4. Target runtime: linux/amd64 + linux/arm64 containers. > > **LIVE VALIDATION (2026-06-23):** the data layer was proven against a real **PostgreSQL 16.14 + > pgvector 0.8.0** instance (M4, `/Volumes/MEGAMIND`). `goose up` applies all migrations cleanly; > the 7-test suite in `internal/integration` (run with `-tags=integration`) proves at runtime: > cross-tenant read isolation, clean default-deny on an unset GUC, the composite-FK cross-tenant > injection block (23503), the `numeric(7,4)` perfect-score round-trip via the full service, audit > append-only (app `42501` / owner-trigger `P0001`), the EEO privilege boundary, and the RLS-scoped > pgvector+HNSW+RRF hybrid search. The FIPS binary (`GOFIPS140=v1.0.0`) serves `/api/match` (100/green) > with scram-sha-256 auth fully inside the validated module. **One hardening landed from this run:** > RLS policies now use `NULLIF(current_setting('app.current_org_id', true), '')::uuid` so a reset GUC > (empty string `""` on a reused pooled connection) yields a clean 0-row default-deny instead of a > `22P02` cast error — see §2. --- ## 0. Pinned dependencies (add these to `go.mod`) The repo already ships `go.mod` with `module github.com/missionworkforce/missionmatch`, `go 1.26`, and `github.com/shopspring/decimal v1.4.0`. Add the rest below. **Do not run an unpinned `go get @latest`** — use these exact versions, then `go mod tidy`. | Concern | Module path | Version | Notes | |---|---|---|---| | HTTP router | `github.com/go-chi/chi/v5` | `v5.3.0` | std-lib `net/http` compatible middleware | | Postgres driver/pool | `github.com/jackc/pgx/v5` | `v5.10.0` | `pgxpool` is `.../pgx/v5/pgxpool`; published 2026-06-03 | | pgxpool internal pool | `github.com/jackc/puddle/v2` | `v2.2.2` | transitive (pulled by pgxpool) | | pgvector types | `github.com/pgvector/pgvector-go` | `v0.4.0` | pgx codecs in subpkg `.../pgvector-go/pgx` | | Decimal (AEDT determinism) | `github.com/shopspring/decimal` | `v1.4.0` | already pinned; **all scoring math** | | UUID | `github.com/google/uuid` | `v1.6.0` | RLS `app.current_org_id` casts to `uuid` | | WebAuthn (FIDO2/passkey/PIV) | `github.com/go-webauthn/webauthn` | `v0.17.4` | phishing-resistant authenticator path (compile-verified `/tmp/gostack-auth`) | | OIDC (Login.gov / Entra Gov) | `github.com/coreos/go-oidc/v3` | `v3.18.0` | discovery + ID-token verify (compile-verified) | | OAuth2 | `golang.org/x/oauth2` | `v0.36.0` | token/code exchange; private_key_jwt assertions (compile-verified) | | JWT | `github.com/golang-jwt/jwt/v5` | `v5.3.1` | RS256 client_assertion (Login.gov private_key_jwt) (compile-verified) | | SAML SP (PIV-CAC IdPs) | `github.com/crewjam/saml` | `v0.5.1` | SP middleware for SAML-only federal IdPs | | AWS Bedrock runtime (GovCloud adapter) | `github.com/aws/aws-sdk-go-v2/service/bedrockruntime` | `v1.54.0` | embeddings + LLM, us-gov-west-1 | | AWS config loader | `github.com/aws/aws-sdk-go-v2/config` | `v1.32.25` | `WithUseFIPSEndpoint` | | AWS core | `github.com/aws/aws-sdk-go-v2` | (transitive, matched by config/bedrockruntime) | — | | Azure identity (Entra Gov) | `github.com/Azure/azure-sdk-for-go/sdk/azidentity` | `v1.14.0` | `cloud.AzureGovernment` | | Azure core | `github.com/Azure/azure-sdk-for-go/sdk/azcore` | `v1.22.0` | `azcore.ClientOptions{Cloud:...}` | | OpenAI SDK (Azure Gov adapter) | `github.com/openai/openai-go/v3` | `v3.41.0` | use subpkg `.../v3/azure`; supersedes `azopenai` | **Migration tool (CLI + embeddable library):** | Tool | Module / install | Version | |---|---|---| | **goose** (CHOSEN) | `github.com/pressly/goose/v3` | `v3.27.1` | **Code generators (developer tooling, not runtime imports — `go install`, do not `require`):** | Tool | Install | Version | |---|---|---| | sqlc | `go install github.com/sqlc-dev/sqlc/cmd/sqlc@v1.31.1` | `v1.31.1` | | goose CLI (optional; library is enough) | `go install github.com/pressly/goose/v3/cmd/goose@v3.27.1` | `v3.27.1` | > Honesty note on versions: public docs caches (pkg.go.dev search index) lagged behind for `pgx/v5` > (showed v5.8.0) and `openai-go` — the **live Go module proxy** returns the versions pinned above and > they compiled clean. Re-confirm against `https://proxy.golang.org//@latest` if a fresh > `go mod tidy` resolves something different, and keep this table as the gate. --- ## 1. FIPS 140-3 — standardized build/run incantation ### DECISION Use the **Go-native in-tree FIPS 140-3 module** (introduced in Go 1.24). **No cgo, no BoringCrypto, no openssl-fips.** Pin to the **CMVP-VALIDATED Go Cryptographic Module `v1.0.0` (CMVP Cert #5247, CAVP Cert A6650, validating vendor Geomys LLC)** — **NOT** the `v1.26.0` module bundled with the locally-installed Go 1.26.4, which is only **"Pending Review"** on the CMVP Modules-In-Process list (CAVP A8028) and is therefore **not a validated module** — it yields FIPS *mode* but cannot satisfy a control that requires a CMVP *certificate*. **Standard build incantation (this exact artifact runs on BOTH GovClouds — zero cloud-specific crypto code):** ```bash CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 \ go build -trimpath -o missionmatch ./cmd/api ``` - `GOFIPS140=v1.0.0` selects **which** validated module is compiled in (works like `GOOS`/`GOARCH`) and **auto-stamps `DefaultGODEBUG=fips140=on`** into the binary, so FIPS mode is the runtime default with no env var required. - Alternative pin that always tracks the latest CMVP-**validated** snapshot: `GOFIPS140=certified` (resolves to `v1.0.0-c2097c7c` on this toolchain). Acceptable, but `v1.0.0` is the explicit, audit-legible pin. - **Do NOT** use bare `GOFIPS140=latest` or `=inprocess` → those select **v1.26.0 (unvalidated)**. This is the #1 trap. - **Belt-and-suspenders runtime env** (set at deploy): `GODEBUG=fips140=on`. - **Do NOT** ship `GODEBUG=fips140=only` in production — Go docs state it is best-effort for testing/assessment and "introduces crashes by design." Use `only` ONLY in a dedicated CI stage to surface non-approved algorithm usage. **CI proof:** build-time `go version -m ./missionmatch | grep GOFIPS140` (expect substring `GOFIPS140=v1.0.0`); runtime `crypto/fips140.Enabled()` healthcheck. ### Dockerfile crypto lines ```dockerfile # --- build stage --- FROM golang:1.26.4 AS build ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 WORKDIR /src COPY . . RUN go build -trimpath -o /missionmatch ./cmd/api # verify the FIPS stamp is present in the artifact (fail the build otherwise) RUN go version -m /missionmatch | grep -q 'GOFIPS140=v1.0.0' # --- runtime stage (distroless/static; static binary, no host OpenSSL) --- FROM gcr.io/distroless/static-debian12:nonroot ENV GODEBUG=fips140=on COPY --from=build /missionmatch /missionmatch USER nonroot:nonroot ENTRYPOINT ["/missionmatch"] ``` > Pin the toolchain (`go.mod` `toolchain go1.26.4` + a pinned builder image digest) so the `v1.0.0` > snapshot zip in `GOROOT/lib/fips140` and its checksum are bit-reproducible for SBOM/attestation. ### VERIFIED PATTERN — fail-closed startup guard (`cmd/api`) ```go package main import ( "crypto/fips140" "log" ) // Call FIRST in main(). In a FIPS build this MUST be true; abort otherwise. func mustFIPS() { if !fips140.Enabled() { log.Fatalf("refusing to start: FIPS 140-3 mode is OFF (module=%q). "+ "Build with GOFIPS140=v1.0.0 and run with GODEBUG=fips140=on", fips140.Version()) } log.Printf("FIPS 140-3 mode active; Go Cryptographic Module %s", fips140.Version()) } // Optional /healthz/fips for load balancers + compliance evidence. type fipsStatus struct { Enabled bool `json:"enabled"` Enforced bool `json:"enforced"` // true only under GODEBUG=fips140=only Version string `json:"module_version"` } func fipsSnapshot() fipsStatus { return fipsStatus{ Enabled: fips140.Enabled(), // Go 1.24 Enforced: fips140.Enforced(), // Go 1.26 Version: fips140.Version(), // e.g. "v1.0.0" } } ``` **Compile status:** `go build ./...` + `go vet ./...` clean; `GOFIPS140=v1.0.0` static linux/amd64 cross-build succeeds; `fips140.Enabled()` toggles true/false with the GODEBUG as expected. ### SBOM note — excelize legacy crypto imports (ontology-sync only) The `cmd/ontology-sync` batch binary imports `github.com/xuri/excelize/v2` (pure Go, no cgo) to read the public PSC manual XLSX. excelize links `golang.org/x/crypto/md4` and `golang.org/x/crypto/ripemd160` for ENCRYPTED workbook support, so an SBOM scan of that one binary will list them. They are dead path: the PSC manual is an unencrypted public file, so no md4 or ripemd160 code is ever reached, and the validated crypto boundary (PBKDF2, HMAC, SHA-256, AES-GCM under the in-tree module) is untouched. The serving binary `cmd/api` does NOT import excelize, so the API binary's SBOM stays clean. Pre answer for an assessor: these are unreachable transitive imports in a batch tool, not invoked algorithms. ### SOURCES - https://go.dev/doc/security/fips140 (GOFIPS140 build var, GODEBUG fips140 runtime, supported platforms, algorithm scope) - https://go.dev/blog/fips140 ("no cgo … cross-compile like any other Go program"; Cert #5247; Geomys) - https://go.dev/doc/go1.24 (native module introduced) - https://go.dev/doc/go1.25 (FIPS signing ~4x faster) - https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/5247 (CMVP Cert #5247, Module v1.0.0) - https://pkg.go.dev/crypto/fips140 (`Enabled`, `Version`, `Enforced`, `WithoutEnforcement`) ### GOTCHAS - Local Go 1.26.4 defaults `GOFIPS140=off`; its `latest`/`inprocess` module is the **unvalidated v1.26.0**. You MUST pin `=v1.0.0` (or `=certified`) for Cert #5247. - FIPS mode alone ≠ compliant. Go docs disclaim that the validated module "may not on its own satisfy all relevant regulatory requirements." Treat it as one control (SC-13/SC-8), not the whole SSP. - Under `fips140=on`, `crypto/tls` **silently drops** non-approved TLS versions/cipher suites/sigalgs/KEMs. Re-verify every TLS path handshakes: pgx `sslmode=verify-full`, S3/Blob clients, OIDC/SAML/PIV-CAC IdPs. - `argon2`/`bcrypt`/`scrypt` (`golang.org/x/crypto`) are **NOT FIPS-approved KDFs** — password hashing sits outside the validated boundary. Document it as a non-crypto-boundary storage function, or move to PBKDF2 (SP 800-132, in-module) if an auditor objects. (See auth section.) - FIPS mode unsupported on OpenBSD / WebAssembly / AIX / 32-bit Windows — irrelevant for GovCloud Linux, but a dev binary built on an unsupported target silently won't engage. - The snapshot resolves to `v1.0.0-c2097c7c` in `go version -m`; CI assertions should grep the substring `GOFIPS140=v1.0.0`, not equality on the hash. - Pairwise consistency tests on generated keys can cost up to ~2x for some key types (matters for ephemeral keys). ### OPEN QUESTIONS / verify during build 1. **CMVP currency** — confirm at audit time that Cert #5247 (Module v1.0.0) has not moved to historical/sunset; decide whether to migrate to a future validated v1.26.0 once it clears review. 2. **3PAO acceptance** — confirm your auditor accepts the Go module's CMVP cert for SC-13/SC-8. Go gives no regulatory guarantee. 3. **Both-GovCloud TLS interop under `fips140=on`** — actually handshake the FIPS binary against AWS RDS PostgreSQL and Azure DB for PostgreSQL Flexible Server (`sslmode=verify-full`), S3 GovCloud + Azure Blob, and the OIDC/SAML/PIV-CAC IdP. Confirm none rely on a suite `fips140=on` drops. 4. **Password-hash posture** — get auditor sign-off on bcrypt/argon2, or swap to in-module PBKDF2. --- ## 2. pgx/v5 + pgxpool + Postgres 16 RLS multi-tenancy (the pool gotcha) ### DECISION `github.com/jackc/pgx/v5 v5.10.0` + `pgxpool`. **Enforce tenant isolation in Postgres RLS, never in Go `WHERE` clauses.** The single correct pattern, per request: 1. `tx, _ := pool.Begin(ctx)` — acquire ONE pooled connection in a transaction. 2. On that same tx: `SELECT set_config('app.current_org_id', $1, true)` — **`is_local=true`** makes the GUC **transaction-scoped**, discarded at COMMIT/ROLLBACK, so it **cannot leak** to the next request that reuses the connection. 3. Run all repo work against that tx. 4. `Commit` (with `defer Rollback`). **Roles:** app connects as a non-owner **`NOBYPASSRLS`** role; a **separate `BYPASSRLS` migrator role** owns tables and runs DDL. **Every tenant table:** `ENABLE` **and** `FORCE ROW LEVEL SECURITY` (`FORCE` is mandatory — owners bypass RLS otherwise) and a `FOR ALL` policy with both `USING` and `WITH CHECK`. Unset GUC ⇒ `current_setting(...,true)` returns `NULL` ⇒ predicate `NULL` ⇒ **zero rows (default-deny)**. Go also fails closed: no validated org in context ⇒ refuse to open the tx. **Do NOT** use bare `SET app.current_org_id` (session-scoped → leaks across pooled requests). **Do NOT** use pgxpool `BeforeAcquire`/`AfterRelease` hooks to set/scrub tenant context (`BeforeAcquire` is deprecated in v5.10 — use `PrepareConn`; `AfterRelease` runs after `Release()` and can't run queries safely; a hook-based `RESET` is a race-prone substitute for what `is_local` gives for free). Cloud-portable: pure GUC + standard RLS works **identically** on AWS RDS for PostgreSQL 16 and Azure DB for PostgreSQL Flexible Server 16 — no vendor primitive. ### VERIFIED PATTERN ```go // internal/platform/tenant/rls.go — verified against pgx/v5 v5.10.0 (go build + go vet clean). type ctxKey struct{} var orgKey = ctxKey{} var ErrNoTenant = errors.New("tenant: no organization_id in context") func WithOrg(ctx context.Context, orgID string) context.Context { return context.WithValue(ctx, orgKey, orgID) } func OrgFromContext(ctx context.Context) (string, bool) { v, ok := ctx.Value(orgKey).(string); return v, ok && v != "" } type DB struct{ Pool *pgxpool.Pool } // Pool connects as the NOBYPASSRLS app role // RunInTenantTx is the correctness boundary: fail-closed, one conn, tx-local GUC. func (db *DB) RunInTenantTx(ctx context.Context, fn func(ctx context.Context, tx pgx.Tx) error) (err error) { orgID, ok := OrgFromContext(ctx) if !ok { return ErrNoTenant } // no org => no query (defense in depth atop RLS default-deny) tx, err := db.Pool.Begin(ctx) if err != nil { return fmt.Errorf("begin tenant tx: %w", err) } defer func() { _ = tx.Rollback(ctx) }() // no-op after Commit; always releases conn // is_local=true => GUC lives ONLY for this tx; auto-reset on COMMIT/ROLLBACK; cannot leak across pooled requests. if _, err = tx.Exec(ctx, "SELECT set_config('app.current_org_id', $1, true)", orgID); err != nil { return fmt.Errorf("set tenant guc: %w", err) } if err = fn(ctx, tx); err != nil { return err } if err = tx.Commit(ctx); err != nil { return fmt.Errorf("commit tenant tx: %w", err) } return nil } // chi middleware: org id comes from the validated session/JWT, NEVER a raw client value. func TenantMiddleware(resolveOrg func(*http.Request) (string, bool)) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { orgID, ok := resolveOrg(r) if !ok { http.Error(w, "forbidden", http.StatusForbidden); return } next.ServeHTTP(w, r.WithContext(WithOrg(r.Context(), orgID))) }) } } ``` ```sql -- migrations (run as the BYPASSRLS migrator/owner role) CREATE ROLE missionmatch_app LOGIN NOBYPASSRLS; -- app connects as this CREATE ROLE missionmatch_migrator LOGIN BYPASSRLS; -- owns tables, runs DDL ALTER TABLE documents ENABLE ROW LEVEL SECURITY; ALTER TABLE documents FORCE ROW LEVEL SECURITY; -- MANDATORY: owners bypass RLS without it CREATE POLICY tenant_isolation ON documents FOR ALL -- NULLIF(...,'') is load-bearing: a reset GUC on a reused pooled connection is the empty string -- "", and ""::uuid raises 22P02. NULLIF turns "" into NULL so the predicate is NULL => 0 rows -- (clean default-deny) whether the GUC was never set OR was set-and-reset. Proven live 2026-06-23. USING (organization_id = NULLIF(current_setting('app.current_org_id', true), '')::uuid) WITH CHECK (organization_id = NULLIF(current_setting('app.current_org_id', true), '')::uuid); -- unset OR reset GUC => NULL => predicate NULL => 0 rows (fail-safe default-deny) GRANT SELECT, INSERT, UPDATE, DELETE ON documents TO missionmatch_app; ``` **Compile status:** `/tmp/gostack-rls`: `go build ./...` OK, `go vet ./...` OK. Exercises real pgx types (`pgxpool.Pool`, `pgx.Tx`, `tx.Exec/Query`). Runtime tenant isolation must be proven against live Postgres 16. ### SOURCES - https://proxy.golang.org/github.com/jackc/pgx/v5/@latest (v5.10.0, 2026-06-03) - https://pkg.go.dev/github.com/jackc/pgx/v5/pgxpool (pool API; `PrepareConn` vs deprecated `BeforeAcquire`; `AfterRelease` semantics) - https://www.postgresql.org/docs/16/functions-admin.html (`set_config(...,is_local)` "applies only during the current transaction"; `current_setting(name, missing_ok)` returns NULL) - https://www.postgresql.org/docs/16/ddl-rowsecurity.html (default-deny; FORCE RLS; `USING` vs `WITH CHECK`) - https://dev.to/__8fa66572/postgresql-rls-in-go-architecting-secure-multi-tenancy-4ifm (canonical tx-wrapper + fail-fast) ### GOTCHAS - **NEVER** bare `SET app.current_org_id = ...` on a pooled conn — it persists to the next (possibly different-tenant) request. Always `set_config(...,true)` / `SET LOCAL` inside a tx. - `SET LOCAL` / `set_config(is_local=true)` is a **no-op outside a transaction** (warns, does nothing). The GUC must be set on the **same tx** the queries run on — not via `pool.Exec` (different/auto-released conn). - Forgetting **`FORCE ROW LEVEL SECURITY`** is the silent killer: `ENABLE` alone still lets the owner see all rows. App role must be non-owner + `NOBYPASSRLS`. Verify with a cross-tenant integration test. - App DB role as superuser or `BYPASSRLS` ⇒ RLS entirely bypassed, isolation silently broken. - `is_local` GUC removes any need for `AfterRelease`/`RESET` scrubbing — don't add a redundant hook; `AfterRelease` can't run queries safely anyway. - Cast the GUC explicitly to `uuid` in the policy. A malformed value (incl. **empty string**) errors at query time; `NULL` does not. Standardize on writing a valid uuid string or never writing the GUC at all. - Server-side poolers (PgBouncer / RDS Proxy / Azure built-in pooling): the tx pattern holds in **transaction** pooling mode (GUC is tx-local). **NEVER STATEMENT pooling** — splits the tx context and breaks isolation. Pin transaction mode if a pooler sits in front of pgxpool. - Background jobs/migrations that legitimately span tenants must use the `BYPASSRLS` migrator role or set the GUC per tenant — else they hit default-deny and see zero rows (looks like "data disappeared"). ### OPEN QUESTIONS / verify during build 1. ✅ **VERIFIED 2026-06-23** against live PostgreSQL 16.14: tenant A cannot read B; unset/reset GUC ⇒ 0 rows; the composite FK blocks writing a foreign-tenant child row (`internal/integration`). 2. Audit `pg_class.relrowsecurity` **AND** `relforcerowsecurity` for every tenant table. 3. Confirm AWS RDS Proxy / Azure built-in pooling is in **transaction** (not statement) mode if used in front of pgxpool. 4. Confirm the app role is genuinely `NOBYPASSRLS` and owns nothing under managed-PG constraints. --- ## 3. pgvector + pgvector-go + HNSW (vector type registration, cosine NN, RLS-scoped) ### DECISION `github.com/pgvector/pgvector-go v0.4.0` (subpkg `.../pgvector-go/pgx v0.4.0`) on `pgx/v5 v5.10.0` + pgxpool. **Register vector codecs once per connection** via `pgxpool.Config.AfterConnect` calling `pgxvec.RegisterTypes(ctx, conn)` — mandatory so `pgvector.Vector` works as bind param and Scan target. - Columns declared `vector(384)` (the matching engine's skill embeddings). **Author the type UNQUALIFIED** (`vector(384)`, not `public.vector`) so sqlc emits `pgvector.Vector` not `interface{}` (sqlc#3548). - **HNSW with `vector_cosine_ops`** — cosine is correct for normalized LLM embeddings (skill adjacency is directional). `WITH (m = 16, ef_construction = 64)` baseline; raise `ef_construction` to 200 in production. - **The index op class MUST match the query operator** or Postgres silently seq-scans: cosine `<=>` ↔ `vector_cosine_ops`; L2 `<->` ↔ `vector_l2_ops`; inner-product `<#>` ↔ `vector_ip_ops`; L1 `<+>` ↔ `vector_l1_ops`. - RLS-scoped NN query: `set_config('app.current_org_id',$1,true)` (SET LOCAL) inside the **same tx**, add `WHERE organization_id = current_setting('app.current_org_id')::uuid` (lets the composite btree pre-filter), tune recall with `SET LOCAL hnsw.ef_search = 100`, then `ORDER BY embedding <=> $vec LIMIT k`. - Stay on plain `vector` at 384 dims; `halfvec` is a later storage-halving optimization, not needed now. > Dimension is **config, not code**: at 384 the column is `vector(384)`. If the embedding provider changes > per cloud (Titan v2 = 1024, OpenAI 3-large = 3072), the column width and HNSW index differ — see §6. > Pin embedding dimension per deployment. ### VERIFIED PATTERN ```go import ( "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" "github.com/pgvector/pgvector-go" pgxvec "github.com/pgvector/pgvector-go/pgx" ) // 1) Register codecs per-connection on the pool: cfg, _ := pgxpool.ParseConfig(dsn) cfg.AfterConnect = func(ctx context.Context, conn *pgx.Conn) error { return pgxvec.RegisterTypes(ctx, conn) // sig: RegisterTypes(ctx context.Context, conn *pgx.Conn) error } pool, _ := pgxpool.NewWithConfig(ctx, cfg) // 2) RLS-scoped nearest-neighbor (cosine) in one tx: tx, _ := pool.Begin(ctx); defer tx.Rollback(ctx) tx.Exec(ctx, "SELECT set_config('app.current_org_id',$1,true)", orgID) // SET LOCAL tx.Exec(ctx, "SET LOCAL hnsw.ef_search = 100") rows, _ := tx.Query(ctx, ` SELECT id FROM candidate_embeddings WHERE organization_id = current_setting('app.current_org_id')::uuid ORDER BY embedding <=> $1 LIMIT $2`, pgvector.NewVector(query), k) // []float32 -> pgvector.Vector; Scan(&v); v.Slice() -> []float32 tx.Commit(ctx) ``` ```sql CREATE EXTENSION IF NOT EXISTS vector; -- Azure: extension binary name is "vector", NOT "pgvector" CREATE TABLE candidate_embeddings ( id bigserial PRIMARY KEY, organization_id uuid NOT NULL, embedding vector(384) NOT NULL -- UNQUALIFIED type for sqlc ); ALTER TABLE candidate_embeddings ENABLE ROW LEVEL SECURITY; ALTER TABLE candidate_embeddings FORCE ROW LEVEL SECURITY; CREATE POLICY org_isolation ON candidate_embeddings USING (organization_id = current_setting('app.current_org_id', true)::uuid) WITH CHECK (organization_id = current_setting('app.current_org_id', true)::uuid); CREATE INDEX idx_ce_org ON candidate_embeddings (organization_id); -- composite leads with org (tenant rule) -- HNSW cosine index (op class MUST match the <=> query operator): SET maintenance_work_mem = '2GB'; SET max_parallel_maintenance_workers = 4; CREATE INDEX CONCURRENTLY idx_ce_embedding_hnsw ON candidate_embeddings USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 200); ``` **Compile status:** `/tmp/gostack-pgvector` (Go 1.26.4): `go mod tidy` resolved pgvector-go v0.4.0 + `.../pgx` v0.4.0 + pgx/v5 v5.10.0; `go build ./...` and `go vet ./...` rc=0. `go doc` confirms `func RegisterTypes(ctx context.Context, conn *pgx.Conn) error`. SQL/DDL not runnable here (needs live PG16 + pgvector). ### SOURCES - https://pkg.go.dev/github.com/pgvector/pgvector-go (v0.4.0; `Vector`, `NewVector`, `.Slice()`, `HalfVector`, `SparseVector`) - https://github.com/pgvector/pgvector-go (`RegisterTypes` via `AfterConnect`) - https://github.com/pgvector/pgvector (operator/op-class table; HNSW DDL `m`/`ef_construction`; `hnsw.ef_search`; build tuning; `halfvec` sizing) - https://aws.amazon.com/blogs/database/optimize-generative-ai-applications-with-pgvector-indexing-a-deep-dive-into-ivfflat-and-hnsw-techniques/ (cosine for normalized embeddings) - https://dev.to/philip_mcclarence_2ef9475/pgvector-distance-functions-cosine-vs-l2-vs-inner-product-57pd (op class must match query operator) - https://docs.sqlc.dev/en/latest/howto/ddl.html (sqlc auto-emits pgvector types under pgx) - https://github.com/sqlc-dev/sqlc/issues/3548 (schema-qualified `public.vector` → `interface{}`) ### GOTCHAS - `RegisterTypes` is **per-connection**: register via `AfterConnect`, not once on a single conn, or you get intermittent codec errors under pool churn. - **Op class must match the query operator** — an HNSW index on `vector_l2_ops` will NOT accelerate a `<=>` (cosine) query; Postgres silently seq-scans. Standardize on `vector_cosine_ops` + `<=>` everywhere. - sqlc#3548: never schema-qualify (`public.vector`) in the schema sqlc reads. If your pg_dump/migration tool qualifies it, add the explicit override `db_type: "vector"` (see §4). - The `<=>` operator result type is inferred `interface{}` by sqlc — alias and cast in SQL (`(embedding <=> $2)::float8 AS distance`) to get a Go `float64`. - `hnsw.ef_search` default 40 is low for high-recall slate generation; set it `LOCAL` per query (100–200). Too high inflates latency. - HNSW has no incremental rebalance; heavy re-embedding churn bloats the graph → schedule `REINDEX INDEX CONCURRENTLY` + `VACUUM`. - Builds are slow/RAM-hungry: set `maintenance_work_mem` + `max_parallel_maintenance_workers`; prefer `CREATE INDEX CONCURRENTLY` in production. These GUCs are per-session on RDS/Flexible Server but capped by instance memory. - `<#>` returns the **negative** inner product (smaller = more similar). Sticking with cosine avoids this footgun. ### OPEN QUESTIONS / verify during build 1. Confirm the exact pgvector **extension version** on AWS GovCloud RDS PG16 **and** Azure Gov PG16 Flexible Server (both should expose ≥ 0.8.x for current HNSW/halfvec). Pin identically in migrations; gate via `CREATE EXTENSION ... ` — do not assume preinstalled. 2. Confirm your embedding model actually outputs **L2-normalized** 384-dim vectors; if not, normalize at write time or stay strictly on cosine. 3. Prove end-to-end RLS NN behavior on a live PG16: unset GUC ⇒ 0 rows; tenants isolated under FORCE RLS + `NOBYPASSRLS` app role. --- ## 4. sqlc (typed SQL) + goose (migrations) — the type-override truth ### DECISION **sqlc `v1.31.1`** (config `version: "2"`, `sql_package: "pgx/v5"`) + **goose `v3.27.1`** (`github.com/pressly/goose/v3`) as the single migration tool. Runtime deps as pinned in §0. **Type overrides (load-bearing for AEDT determinism):** `uuid`, `timestamptz`, and `vector` override cleanly by **bare** `db_type`. **NUMERIC does NOT** — sqlc silently ignores `db_type: pg_catalog.numeric` for plain table columns in pgx/v5 mode (issues #1067 / #1796 / #3460). You **MUST** add a **`column:`-scoped override** to force `shopspring/decimal.Decimal` on every numeric table column (this satisfies CLAUDE.md's "all scoring math uses `shopspring/decimal`, never `float64`" rule). Keep BOTH the `pg_catalog.numeric` db_type override (covers `AVG()`/expression results) AND a per-column override for each numeric table column. > Naming caveat verified locally: the bare `timestamptz` db_type matched in pgx/v5; the docs' example > `pg_catalog.timestamptz` did **not** match. Use bare `timestamptz`. **Why goose (not golang-migrate, not atlas):** 1. Single static Go binary **and** an embeddable library (`embed.FS` + run programmatically against a specific `*sql.DB` — exactly how you connect as the `BYPASSRLS` migrator role). Zero cloud lock-in; identical on AWS GovCloud RDS and Azure Flexible Server. 2. `-- +goose NO TRANSACTION` is required for `CREATE EXTENSION vector` / `CREATE INDEX CONCURRENTLY`; `-- +goose StatementBegin/StatementEnd` handles internal semicolons in `CREATE POLICY` / PL/pgSQL. 3. **Decisive:** sqlc natively parses goose-annotated `.sql` as its `schema:` input (ignores `-- +goose Down`), so ONE set of migration files is both the migration source and sqlc's schema — no schema duplication. 4. **Atlas rejected:** declarative RLS-policy support is **Pro/paid only** — unacceptable lock-in for an RLS-centric federal product. golang-migrate works but lacks per-file no-transaction annotation and has weaker library ergonomics for role-scoped programmatic runs. `goose` uses `database/sql` (`*sql.DB`) for migrations as the migrator role; the app uses `pgxpool` as the app role — distinct connection paths by design (migrations are infrequent). ### VERIFIED PATTERN — `sqlc.yaml` ```yaml version: "2" sql: - engine: "postgresql" schema: "db/migrations" # goose-annotated files; sqlc ignores -- +goose Down queries: "db/queries" gen: go: package: "dbgen" out: "internal/gen" sql_package: "pgx/v5" emit_json_tags: true emit_pointers_for_null_types: true emit_interface: true emit_empty_slices: true overrides: - db_type: "uuid" go_type: { import: "github.com/google/uuid", type: "UUID" } - db_type: "uuid" nullable: true go_type: { import: "github.com/google/uuid", type: "UUID", pointer: true } - db_type: "timestamptz" # BARE name (pg_catalog.timestamptz did NOT match in pgx/v5) go_type: "time.Time" - db_type: "timestamptz" nullable: true go_type: { type: "time.Time", pointer: true } - db_type: "pg_catalog.numeric" # covers AVG()/expression numeric results go_type: { import: "github.com/shopspring/decimal", type: "Decimal" } - column: "candidates.match_score" # REQUIRED: db_type override is IGNORED for plain columns go_type: { import: "github.com/shopspring/decimal", type: "Decimal" } - db_type: "vector" go_type: { import: "github.com/pgvector/pgvector-go", package: "pgvector", type: "Vector" } - db_type: "vector" nullable: true go_type: { import: "github.com/pgvector/pgvector-go", package: "pgvector", type: "Vector", pointer: true } ``` ### VERIFIED PATTERN — goose migration (also sqlc's schema) ```sql -- +goose Up -- +goose NO TRANSACTION CREATE EXTENSION IF NOT EXISTS vector; -- +goose StatementBegin CREATE TABLE candidates ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), organization_id uuid NOT NULL, full_name text NOT NULL, match_score numeric(6,4), skill_embedding vector(384), created_at timestamptz NOT NULL DEFAULT now(), deleted_at timestamptz ); -- +goose StatementEnd CREATE INDEX idx_candidates_org ON candidates (organization_id, created_at DESC); ALTER TABLE candidates ENABLE ROW LEVEL SECURITY; ALTER TABLE candidates FORCE ROW LEVEL SECURITY; -- +goose StatementBegin CREATE POLICY candidates_tenant_isolation ON candidates USING (organization_id = current_setting('app.current_org_id', true)::uuid) WITH CHECK (organization_id = current_setting('app.current_org_id', true)::uuid); -- +goose StatementEnd -- +goose Down DROP TABLE IF EXISTS candidates; ``` ### VERIFIED PATTERN — pgx pool + pgvector registration + per-tx RLS ```go cfg, _ := pgxpool.ParseConfig(dsn) cfg.AfterConnect = func(ctx context.Context, c *pgx.Conn) error { return pgxvec.RegisterTypes(ctx, c) } pool, _ := pgxpool.NewWithConfig(ctx, cfg) // per request: tx, _ := pool.Begin(ctx); defer tx.Rollback(ctx) tx.Exec(ctx, "SELECT set_config('app.current_org_id', $1, true)", orgID.String()) q := dbgen.New(tx); /* ...q.CreateCandidate(...)... */; tx.Commit(ctx) ``` **Compile status:** Installed sqlc v1.31.1. In `/tmp/gostack-sqlc`: `sqlc generate` exit 0 — parsed the goose-annotated migration (incl. `CREATE EXTENSION`/RLS/`CREATE POLICY`) as schema; generated models had the **correct** types: `ID uuid.UUID`, `MatchScore decimal.Decimal`, `SkillEmbedding *pgvector.Vector`, `CreatedAt time.Time`, `DeletedAt *time.Time`. Then `go build ./...` and `go vet ./...` exit 0. Live DB execution (actual `goose Up` / RLS enforcement) NOT run — needs a running Postgres 16. ### SOURCES - https://github.com/sqlc-dev/sqlc/releases + https://docs.sqlc.dev/en/stable/reference/config.html (v1.31.1; version "2"; pgx/v5) - https://docs.sqlc.dev/en/latest/howto/ddl.html (parses goose/atlas/dbmate/golang-migrate; ignores down) - https://docs.sqlc.dev/en/stable/howto/overrides.html (override semantics; one db_type per nullable-ness) - https://github.com/sqlc-dev/sqlc/issues/1067 , /1796 , /3460 (NUMERIC db_type override ignored for plain columns → column override) - https://github.com/pgvector/pgvector-go (`pgxvec.RegisterTypes`) - https://github.com/pressly/goose (v3.27.1; `NO TRANSACTION`; `StatementBegin/End`; `SetBaseFS`/embed.FS) - https://pressly.github.io/goose/blog/2024/goose-sqlc/ (official sqlc+goose workflow) - https://atlasgo.io/guides/orms/gorm + https://entgo.io/docs/migration/row-level-security/ (Atlas RLS is Pro-only) - https://github.com/golang-migrate/migrate/releases (v4.19.1; viable but weaker for this use) ### GOTCHAS - **NUMERIC override silently dropped** for plain table columns in pgx/v5 mode — without a `column:` override your `match_score` becomes `pgtype.Numeric`, breaking the AEDT decimal-determinism rule. Add a per-column override for **every** numeric column. - Use **bare** `timestamptz` db_type (not `pg_catalog.timestamptz`) or columns stay `pgtype.Timestamptz`. - Non-null `vector` columns generate as `*pgvector.Vector` (sqlc treats vector as nullable); pass `&vec`. Add the nullable override with `pointer: true` so both cases resolve to a pointer consistently. - The `<=>` distance result is inferred `interface{}`; alias + `::float8` in SQL to get `float64`. - Register the vector type on **every** pooled conn via `AfterConnect`, not once. - `SET LOCAL`/`set_config(...,true)` is tx-scoped — all tenant queries must run inside that same tx via `dbgen.New(tx)`. A bare `pool.Query` after a `SET LOCAL` on a different checkout sees no org ⇒ zero rows. - `CREATE EXTENSION vector` needs `CREATE` on the DB for the migrator role and a `-- +goose NO TRANSACTION` migration. - Enforce migrator/app role separation at connect time: goose `*sql.DB` = `BYPASSRLS` migrator; app `pgxpool` = `NOBYPASSRLS`. Do not reuse one DSN. ### OPEN QUESTIONS / verify during build 1. Confirm `goose Up` actually applies `CREATE EXTENSION vector` / `ENABLE`+`FORCE` RLS / `CREATE POLICY` and that the policy enforces zero-rows-without-context against a live PG16. 2. Confirm pgvector is in the **allowlist at the version you target** on AWS GovCloud RDS PG16 and Azure Gov PG16 Flexible Server before pinning `vector(384)`/HNSW DDL. 3. Confirm the `BYPASSRLS` migrator role is creatable under managed-PG constraints (providers restrict superuser — verify `rolbypassrls` is grantable, or use `rds_superuser` / `azure_pg_admin`). 4. Decide whether to register pgx's shopspring/decimal integration (`github.com/jackc/pgx-shopspring-decimal`) so `decimal.Decimal` round-trips numeric without manual conversion — recommended, not yet exercised. --- ## 5. Phishing-resistant federal auth stack (OIDC / SAML / WebAuthn / PIV-CAC) ### DECISION Pin (versions are the compile-verified resolutions from `/tmp/gostack-auth`, which `go mod tidy` selected — they supersede the older numbers an earlier draft of this doc carried): `github.com/go-webauthn/webauthn v0.17.4` (FIDO2/passkey + PIV authenticators), `github.com/coreos/go-oidc/v3 v3.18.0` (OIDC discovery + ID-token verify), `golang.org/x/oauth2 v0.36.0`, `github.com/golang-jwt/jwt/v5 v5.3.1` (RS256 `client_assertion`), `github.com/crewjam/saml v0.5.1` (SP for SAML-only federal IdPs; pinned at first use). - **Login.gov** = OIDC with **`private_key_jwt`** client authentication (RS256 client assertion signed with golang-jwt v5; no client secret). Use go-oidc for discovery + ID-token verification. - **Microsoft Entra Government** = OIDC against the **`login.microsoftonline.us`** sovereign authority (NOT `.com`). Pair with `cloud.AzureGovernment` for any Graph/Azure calls. - **PIV-CAC**: front via an IdP that does the cert-bound auth, or via WebAuthn authenticators; for SAML-only federal IdPs use crewjam/saml SP middleware. - **Password hashing under FIPS**: bcrypt/argon2 are **not** FIPS-approved (see §1). Use **PBKDF2 (SP 800-132, in-module)** if the auditor requires hashing inside the validated boundary; otherwise document password storage as a non-crypto-boundary function. ### VERIFIED PATTERN (shape) ```go // OIDC: discovery + verify provider, _ := oidc.NewProvider(ctx, issuerURL) // go-oidc v3 verifier := provider.Verifier(&oidc.Config{ClientID: clientID}) idToken, _ := verifier.Verify(ctx, rawIDToken) // Login.gov private_key_jwt: sign an RS256 client assertion (golang-jwt v5) tok := jwt.NewWithClaims(jwt.SigningMethodRS256, jwt.MapClaims{ "iss": clientID, "sub": clientID, "aud": tokenEndpoint, "jti": jti, "exp": exp.Unix(), }) assertion, _ := tok.SignedString(rsaPrivateKey) // WebAuthn: BeginRegistration/FinishRegistration (and Login) with a persisted SessionData w, _ := webauthn.New(&webauthn.Config{RPDisplayName: "missionMATCH", RPID: rpID, RPOrigins: origins}) options, session, _ := w.BeginRegistration(user) // ... persist `session`, return `options` to the browser, then on callback: cred, _ := w.FinishRegistration(user, session, httpRequest) ``` **Compile status:** `go build` and `go vet` clean on Go 1.26. (The auth area was verified at the compile/wiring level; exact module-resolution and live IdP handshakes are build-time tasks.) ### SOURCES - https://pkg.go.dev/github.com/go-webauthn/webauthn (WebAuthn RP API) - https://pkg.go.dev/github.com/coreos/go-oidc/v3/oidc (provider/verifier) - https://pkg.go.dev/golang.org/x/oauth2 (code/token exchange) - https://pkg.go.dev/github.com/golang-jwt/jwt/v5 (RS256 client assertion) - https://pkg.go.dev/github.com/crewjam/saml (SAML SP) - https://developers.login.gov/oidc/ (Login.gov OIDC + private_key_jwt) - https://learn.microsoft.com/en-us/entra/identity-platform/authentication-national-cloud (`login.microsoftonline.us` US Gov authority) ### GOTCHAS - Entra **Gov** authority is `login.microsoftonline.us`, audience/endpoints differ from commercial — do not reuse `.com` config. - Login.gov requires **`private_key_jwt`** (no client secret); the assertion `aud` must be the token endpoint and `jti` must be unique/replay-guarded. - Under `fips140=on`, RS256/ES256 are fine but verify your IdP TLS suites aren't dropped (§1 gotcha). - bcrypt/argon2 outside FIPS boundary — pick PBKDF2 or document the carve-out before ATO. - crewjam/saml: pin metadata/clock-skew and validate signatures strictly; SAML is the fallback only for IdPs without OIDC. ### OPEN QUESTIONS / verify during build 1. Re-resolve exact versions for go-webauthn / go-oidc / oauth2 / crewjam-saml / golang-jwt with `go mod tidy` and re-run `go build`/`go vet`; this area's pins are the most likely to move. 2. Live handshakes: Login.gov sandbox (`private_key_jwt`), Entra Gov (`login.microsoftonline.us`), and the PIV-CAC IdP — confirm none rely on a TLS suite `fips140=on` drops. 3. Auditor decision on password-hash KDF (PBKDF2 vs documented carve-out). --- ## 6. Both-cloud support (Postgres+pgvector AND AI providers on AWS GovCloud + Azure Government) ### DECISION — the "both clouds, no lock-in" claim **HOLDS for the core path** (one residual gap, below). **DB:** Postgres 16 + pgvector + HNSW is available on **AWS RDS for PostgreSQL in GovCloud** (pgvector 0.8.0 on PG 16.5+/17.1+) **and** on **Azure Database for PostgreSQL Flexible Server in Azure Government** (extension allowlisted via `azure.extensions`, binary name **`vector`**). Same `pgx/v5` + sqlc + pgvector-go on both — **no vendor primitive**. **AI provider:** keep the `Embedder`/`LLM` behind a **port with TWO adapters**: - **AWS GovCloud** = `aws-sdk-go-v2/service/bedrockruntime v1.54.0` + `config v1.32.25`, region `us-gov-west-1`, `WithUseFIPSEndpoint(aws.FIPSEndpointStateEnabled)` → resolves `bedrock-runtime-fips.us-gov-west-1.amazonaws.com`. Embeddings: `amazon.titan-embed-text-v2:0` (**1024-dim**) or Cohere Embed. LLM: Claude Sonnet 4.5 / 3.7 / 3.5 v1 / 3 Haiku, or Llama 3 (FedRAMP High + DoD IL4/5). - **Azure Government** = `openai/openai-go/v3 v3.41.0` + `.../v3/azure` (`azure.WithEndpoint("https://.openai.azure.us", apiVersion)` + `azure.WithTokenCredential`), authenticated with `azidentity v1.14.0` using `azcore.ClientOptions{Cloud: cloud.AzureGovernment}`. Embeddings: `text-embedding-3-large` (**3072-dim**) / `-3-small` (1536). LLM: gpt-4o / gpt-4.1 / gpt-5.1. (`azopenai` is now a thin extensions module over `openai-go/v3` — target `openai-go/v3` directly.) **Pin embedding DIMENSION in config, not code.** Titan v2 = 1024 vs OpenAI 3-large = 3072 ⇒ the `vector(N)` column width and HNSW index differ per cloud. Abstract it; treat switching providers as a **data migration** (re-embed), not a config flip. (OpenAI 3-large's `dimensions` param can reduce to 1024 for cross-cloud parity at a quality cost — note for `slateMATCH`'s 384-dim skill space, both providers exceed 384, so you either use a 384-native model or reduce/pad deliberately.) **FIPS posture differs by cloud, by design:** AWS Bedrock has a dedicated FIPS endpoint hostname; **Azure Gov has NO separate FIPS hostname** — FIPS there comes from the Go FIPS 140-3 binary (§1) plus TLS to `*.openai.azure.us`. Don't expect a magic FIPS URL on Azure. ### VERIFIED PATTERN ```go // AWS GovCloud — Bedrock Runtime + FIPS + Titan Text Embeddings V2 (1024-dim): cfg, _ := awsconfig.LoadDefaultConfig(ctx, awsconfig.WithRegion("us-gov-west-1"), awsconfig.WithUseFIPSEndpoint(aws.FIPSEndpointStateEnabled), // -> bedrock-runtime-fips.us-gov-west-1.amazonaws.com ) br := bedrockruntime.NewFromConfig(cfg) body, _ := json.Marshal(map[string]any{"inputText": text}) out, _ := br.InvokeModel(ctx, &bedrockruntime.InvokeModelInput{ ModelId: aws.String("amazon.titan-embed-text-v2:0"), ContentType: aws.String("application/json"), Accept: aws.String("application/json"), Body: body, }) // Azure Government — Azure OpenAI via official openai-go/v3 + Entra ID (sovereign cloud): cred, _ := azidentity.NewDefaultAzureCredential(&azidentity.DefaultAzureCredentialOptions{ ClientOptions: azcore.ClientOptions{Cloud: cloud.AzureGovernment}, // .us authority + audience }) client := openai.NewClient( azure.WithEndpoint("https://my-res.openai.azure.us", "2024-10-21"), azure.WithTokenCredential(cred), ) resp, _ := client.Embeddings.New(ctx, openai.EmbeddingNewParams{ Model: openai.EmbeddingModel("text-embedding-3-large"), // 3072-dim Input: openai.EmbeddingNewParamsInputUnion{OfString: openai.String(text)}, }) ``` ### Support matrix (feature × AWS-GovCloud × Azure-Gov) | Feature | AWS GovCloud | Azure Government | |---|---|---| | pgvector extension | YES, PG 15.2+ all RDS incl GovCloud; 0.8.0 on PG 16.5+/17.1+ | YES, `vector` via `azure.extensions` allowlist + `CREATE EXTENSION vector` | | HNSW index | YES (pgvector ≥ 0.5.0; 0.8.0 build/search perf) | YES (pgvector on Flexible Server supports `hnsw`) | | Managed Postgres | AWS RDS for PostgreSQL (Aurora PG also) | Azure DB for PostgreSQL Flexible Server (GA in USGov regions) | | AI embeddings | Titan Text Embeddings V2 (1024) / Cohere Embed | text-embedding-3-large (3072) / -small (1536); ada-002 limited | | AI LLM | Claude Sonnet 4.5/3.7/3.5v1/3 Haiku, Llama 3 8B/70B | gpt-4o, gpt-4.1(+mini), gpt-5.1, o3-mini | | Compliance | FedRAMP High + DoD IL4/5 (us-gov-west-1 / us-gov-east-1) | FedRAMP High (usgovarizona / usgovvirginia) | | Go SDK | `aws-sdk-go-v2/service/bedrockruntime v1.54.0` | `openai-go/v3 v3.41.0` + `.../azure` + `azidentity v1.14.0` | | FIPS crypto path | `bedrock-runtime-fips.us-gov-{west,east}-1.amazonaws.com` | TLS to `*.openai.azure.us` (FIPS = app-side Go module, no FIPS hostname) | **Compile status:** `/tmp/gostack-portability` (Go 1.26.4): `go build ./...` BUILD OK, `go vet ./...` VET OK. Resolved bedrockruntime v1.54.0, aws config v1.32.25, azidentity v1.14.0, azcore v1.22.0, openai-go/v3 v3.41.0, pgvector-go v0.4.0. Network calls NOT executed (need real GovCloud/Azure-Gov accounts) — wiring is compile-verified; runtime correctness cited below. ### SOURCES - https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-rds-postgresql-pgvector-ml-model-integration/ (pgvector on RDS PG 15.2+, **all regions incl GovCloud**) - https://aws.amazon.com/about-aws/whats-new/2023/10/amazon-rds-postgresql-pgvector-hnsw-indexing/ + https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-rds-for-postgresql-pgvector-080/ (HNSW; pgvector 0.8.0) - https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/govcloud-bedrock.html (Bedrock in GovCloud; FedRAMP High + IL4/5 model list) - https://aws.amazon.com/blogs/publicsector/accelerating-government-innovation-amazon-bedrock-models-get-fedramp-high-and-dod-il-4-5-approval-in-aws-govcloud-us/ (Titan V2 embeddings live in GovCloud) - https://docs.aws.amazon.com/general/latest/gr/bedrock.html (`bedrock-runtime-fips.us-gov-{west,east}-1` endpoints) - https://learn.microsoft.com/en-us/azure/postgresql/extensions/how-to-use-pgvector + https://learn.microsoft.com/en-us/azure/postgresql/extensions/how-to-allow-extensions (`azure.extensions`=`vector`, `CREATE EXTENSION vector`, hnsw) - https://azure.microsoft.com/en-us/updates?id=general-availability-azure-database-for-postgresql-flexible-server-new-us-government-regions (Flexible Server GA in USGov) - https://devblogs.microsoft.com/azuregov/azure-openai-fedramp-high-for-government/ + https://devblogs.microsoft.com/azuregov/azure-openai-authorization/ (Azure OpenAI FedRAMP High in Gov) - https://learn.microsoft.com/en-us/azure/ai-foundry/openai/azure-government?view=foundry-classic (Gov model/region matrix; `*.openai.azure.us`) - https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/ai/azopenai + https://pkg.go.dev/github.com/openai/openai-go/v3/azure (`azopenai` is an extensions module; target openai-go/v3) ### GOTCHAS - **EMBEDDING DIMENSION MISMATCH ACROSS CLOUDS** (the real lock-in trap): Titan V2 = 1024, OpenAI 3-large = 3072, 3-small/ada-002 = 1536. The `vector(N)` column and HNSW index are dimension-specific — you **cannot share one embeddings table across both clouds** without re-embedding. Pin dimension per deployment; treat a provider switch as a data migration. - Azure Gov region pinning: under Standard deployment, `text-embedding-3-large/-small` show available in **usgovarizona** but **not usgovvirginia** per the model matrix; ada-002 is narrower. Verify your subscription/region before committing. - AWS **FIPS + custom endpoint are mutually exclusive** in the SDK: with `WithUseFIPSEndpoint(Enabled)` let the SDK resolve the FIPS hostname; do NOT also pass a custom `BaseEndpoint` ("FIPS and custom endpoint are not supported"). Rely on region+FIPS resolution for GovCloud. - Azure Gov has **no separate FIPS endpoint hostname** — FIPS comes from the Go FIPS binary (§1) + TLS to `*.openai.azure.us`. Don't assume a magic FIPS URL. - The pgvector **0.8.0** GovCloud guarantee is implied by "all applicable regions"; the firm guarantee is the original 15.2+ all-regions-incl-GovCloud statement. Confirm the exact point-version on your launched RDS engine before relying on 0.8.0-specific perf. - Azure extension name is **`vector`** (binary) — NOT `pgvector` — in `CREATE EXTENSION` and the allowlist. Easy to get wrong in migrations/Terraform. - **pgvector 0.8.0 + HNSW illegal-instruction crash on some Azure Flexible Server SKUs** (open: pgvector/pgvector#889) — certain Gov hardware hits a CPU-instruction incompatibility when building an HNSW index on pgvector 0.8.0. Field mitigation is to **pin pgvector 0.7.0 for HNSW** until resolved. **Build-time gate:** actually build an HNSW index on your real US-Gov Flexible Server SKU before committing to 0.8.x; fall back to 0.7.0 if hit. (AWS GovCloud RDS is unaffected.) - **Avoid Azure `pg_diskann`** (Microsoft-specific ANN index) in portable code — it does not exist on AWS. Standardize on the open `vector` + HNSW path, which is identical on both clouds. Using DiskANN would silently re-introduce the lock-in this architecture exists to prevent. - Don't pull the old standalone `azopenai` client as the primary path — build on `openai-go/v3` + `/azure`. - Aurora PG pgvector in GovCloud trails RDS historically; re-verify the point-version if you choose Aurora. ### Both-cloud verdict & residual gap **Verdict: the no-lock-in claim HOLDS for the core path.** DB (Postgres+pgvector+HNSW), tenancy (RLS), and crypto (Go FIPS module) are cloud-agnostic and identical on both clouds. The AI provider is cleanly port/adapter-isolated with a working adapter for each cloud. **Residual gap (the only one):** the embedding **dimension differs by provider** (1024 vs 3072), so the **embeddings table + HNSW index are NOT portable as data** — switching clouds/providers is a re-embed migration, not a config flip. This is a data-layer consequence, not a code-portability failure, and it is mitigated by: (a) pinning dimension in config, (b) optionally using OpenAI's `dimensions` param for parity, or (c) self-hosting one embedding model (vLLM, per CLAUDE.md's CUI/production posture) so a single dimension serves both clouds — **the recommended path for production CUI**, which also sidesteps both managed AI providers entirely. ### OPEN QUESTIONS / verify during build (cannot verify without GovCloud/Azure-Gov accounts) 1. Exact pgvector point-version on the specific RDS PG16 engine launched in `us-gov-west-1` (`aws rds describe-db-engine-versions --engine postgres` / `SHOW rds.allowed_extensions`) — needed to guarantee 0.8.0 HNSW perf vs older 0.5.x. 2. Whether your Azure Gov subscription can actually **deploy** `text-embedding-3-large` in your chosen region (usgovarizona vs usgovvirginia differ; gated by quota). 3. End-to-end FIPS posture across the ATO boundary (Bedrock FIPS endpoint + Azure Gov TLS + Go FIPS binary). 4. Cohere Embed availability in GovCloud if you want a non-Titan fallback (only Titan was explicitly named). 5. Decide production embedding strategy: self-hosted vLLM (single dimension, CUI-safe, both clouds) vs managed per-cloud (dimension split). CLAUDE.md leans self-hosted for production CUI. --- ## 7. Phase 2 Authentication and Accounts (build decisions, locked) ### DECISION - **Password and recovery code KDF: PBKDF2 with HMAC SHA 256 (SP 800-132), `crypto/pbkdf2.Key`** (Go 1.26 stdlib, confirmed `func Key[Hash hash.Hash](h func() Hash, password string, salt []byte, iter, keyLength int) ([]byte, error)` — it RETURNS an error, handle it). 600000 iterations, 16 byte salt, 32 byte derived key, self describing encoding `pbkdf2-sha256$$$`. argon2, bcrypt, scrypt are NOT used: outside the FIPS boundary. The high iteration count is a deliberate SP 800-132 maximize choice above the CAVP tested ceiling of 10000, documented as a strength not a deviation. Password input capped at 64 chars (the tested length envelope upper bound). - **Versioned credential pepper**: HMAC SHA 256 of the password under a server held key from the secrets file, applied before PBKDF2 (in boundary), stored as a per credential `pepper_version`. - **MFA: mandatory TOTP, rolled in house on `crypto/hmac` + `crypto/sha256`**, NOT a third party library. RFC 6238, pinned to HMAC SHA 256 end to end, 32 byte secret, `algorithm=SHA256` in the otpauth URI, strictly monotonic replay guard (a step is accepted only if greater than the stored last used step). This is a deliberate deviation from the design doc's `github.com/pquerna/otp` pin: rolling it in house keeps every primitive inside the validated module (no external crypto to re verify under fips140=on) and matches the design's own in house session and CSRF choice. TOTP secrets are AES 256 GCM encrypted at rest (`crypto/aes` + `crypto/cipher`) with a `key_version`. - **Sessions: opaque, server side**, 32 bytes `crypto/rand` base64url, stored only as SHA 256 hash, in a `__Host-` cookie (HttpOnly, Secure, SameSite Strict). Two stage (auth_pending then active), 15 min idle and 12 h absolute, all expiry math on the database clock. NOT JWT (instant revocation, no signing key rotation under FIPS). - **No new go.mod dependencies in Phase 2 core.** Everything is stdlib (`crypto/pbkdf2`, `crypto/hmac`, `crypto/sha256`, `crypto/rand`, `crypto/subtle`, `crypto/aes`, `crypto/cipher`, `crypto/fips140`) plus the already pinned chi, pgx, and uuid. The five federal auth libraries in section 0 (webauthn, go-oidc, oauth2, jwt, saml) are deferred to sub phase 2.9. ### GRANT MODEL (the hardened, post red team posture) - The secret bearing tables `auth_credentials`, `auth_sessions`, `mfa_totp`, `mfa_recovery_codes`, `user_invites` are OUTSIDE RLS and the `missionmatch_app` role gets NO direct table grants on them. All access is through a tight set of `SECURITY DEFINER` functions owned by `missionmatch_migrator`, `search_path` locked, EXECUTE granted narrowly to `missionmatch_app` (and `provision_org` granted migrator only, with a `pg_proc`/`aclexplode` CI assertion). This restores DB side default deny for the most sensitive data: the app role literally cannot SELECT a credential or session row directly, so a forgotten predicate cannot leak another org's secrets. This consolidates the design's mixed "mfa under RLS, sessions behind definers" into one uniform definer guarded model because MFA enroll and verify run on a pre org auth_pending session where no org GUC exists. - `login_attempts` is OUTSIDE RLS with a narrow app SELECT/INSERT grant (a counter, no secrets), so the brute force throttle reads it through a scoped query. `users`, `organizations`, `audit_logs` are unchanged (users under RLS; audit_logs append only, app INSERT/SELECT, read bound to the GUC). ### OPEN QUESTIONS / verify during build 1. The live M4 dev Postgres is NOT reboot durable (PGDATA on an external volume macOS launchd cannot own). It is a RELEASE BLOCKER for any pilot: credentials, MFA enrollments, and session state now live there, so a reboot would lock every user out with no recovery. Make it durable with scheduled backups and one tested restore before this auth ships. 2. Confirm pgx TLS to Postgres still negotiates under the FIPS build (fips140=on drops non approved suites); exercise password hashing and the DB handshake under GOFIPS140=v1.0.0, not a dev build. 3. The global unique email ALTER can fail if seed data holds cross org duplicate emails; pre flight a duplicate check on the live users table before applying 0009. --- ## Appendix — quick command reference ```bash # FIPS-validated production build (linux/amd64) CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOFIPS140=v1.0.0 go build -trimpath -o missionmatch ./cmd/api go version -m ./missionmatch | grep GOFIPS140 # expect: GOFIPS140=v1.0.0 # dev codegen (developer tooling, not a runtime require) go install github.com/sqlc-dev/sqlc/cmd/sqlc@v1.31.1 sqlc generate # CI-only: surface non-FIPS algorithm usage (NEVER ship this) GODEBUG=fips140=only go test ./... ```