Janady13/missionmatch / docs/AI_RESEARCH.md

1# AI Layer Research Brief (primary sources) 2 3Generated by an 8 agent research workflow against primary sources before building. Tailored to the 4missionMATCH stack (Go 1.26, pgx/sqlc/pgvector 0.8, 384 dim HNSW, FIPS CGO_ENABLED=0, no Python in 5repo, stateless Apple Silicon llama.cpp/Ollama inference plane over Tailscale). 6 7## Augmented Generation patterns end to end (RAG, CAG, KAG, TAG, GraphRAG, CRAG/Self-RAG, agentic + memory-augmented), how to route and compose them 8 9**Recommendation:** For the missionMATCH stack (Go 1.26 / chi / pgx / sqlc / pgvector-go, Postgres 16 + pgvector 0.8.0, 384-dim HNSW, FIPS CGO_ENABLED=0, no Python, stateless Apple-Silicon llama.cpp/Ollama inference over HTTP): 10 111. Default primitive = hybrid RAG in Postgres, not pure vector RAG. Combine pgvector cosine/L2 ANN with Postgres full-text (tsvector) and fuse with Reciprocal Rank Fusion in Go. This is the highest-ROI baseline and stays entirely in your existing pgx/sqlc data plane. 12 132. Lean hard on pgvector 0.8.0's two new capabilities — they were built for exactly your filtered-tenant case: 14 - Improved filter cost-estimation: the planner now correctly chooses HNSW vs B-tree for queries with WHERE filters (org_id/RLS predicates), restoring recall on filtered ANN. This matters because every query here is tenant-scoped under RLS. 15 - Iterative index scans to avoid overfiltering: set `hnsw.iterative_scan = relaxed_order` as the production default (approximate order, faster, "good enough"); use `strict_order` only on ranking-precision-critical paths. Tune `hnsw.max_scan_tuples` (and `hnsw.ef_search`) per workload. Set these as GUCs per-session/per-statement from Go, not globally. 16 173. Storage/index: use `halfvec(384)` with HNSW (m=16, ef_construction=200). Halves storage/index size with <1% recall loss on normalized embeddings, and is the cheapest decision to make on day one. Skip binary quantization at 384-dim (recall drop not worth it without a rerank stage you don't need at this scale). 18 194. Add a lightweight router in Go (Adaptive-RAG pattern) before retrieval, classifying each query into: (a) direct-answer / CAG-style preloaded small static KB; (b) single-step RAG (pgvector hybrid); (c) DB-compute => TAG/Text-to-SQL against your relational tables for any count/sum/filter/aggregate question (e.g., staffing gaps, $ exposure, slate stats) — do NOT answer aggregation questions via vector RAG; (d) multi-step/agentic only when needed. Implement the router as a small classifier or rules over embeddings; keep it cheap since your inference plane is stateless commodity Metal. 20 215. For correctness, adopt CRAG (plug-and-play) over Self-RAG. Self-RAG needs a custom fine-tuned critic+generator model — incompatible with a stateless commodity llama.cpp/Ollama plane and your no-Python/FIPS constraints. CRAG can be implemented as: score retrieved chunks (a small grader prompt or a cosine-margin heuristic), and on low confidence either re-query Postgres differently or fall back to an approved secondary source. Web-search fallback is likely out of scope for CUI/federal data — replace it with a broader-corpus or human-in-the-loop fallback consistent with your AEDT audit rules. 22 236. CAG: reserve for genuinely small, static knowledge (e.g., a fixed compliance ruleset, a model card, AEDT policy text) that fits the model's context. Precompute/reuse the KV cache on the inference node so the policy text is loaded once. Do not use CAG for the opportunity/award corpus (too large and changes daily via your sync crons). 24 257. KAG/GraphRAG: justified only for the structured-reasoning slices — incumbent/award/UEI entity graphs and multi-hop "who held this contract, who are competitors, what past performance connects" questions. You already have relational entity data; prefer querying it directly (TAG) before standing up a GraphRAG indexing pass, which is LLM-expensive and adds a heavy ingest pipeline. If you do build a graph, keep it in Postgres (recursive CTEs / a graph table) rather than introducing a new datastore, to preserve "no lock-in" and RLS tenancy. 26 278. Memory-augmented generation: keep per-user/session memory in Postgres tables (not an external memory service), scoped by the same composite-FK RLS so memory inherits tenant isolation. Treat memory as separate from corpus RAG. Apply the same served-score/PII audit and anonymize-in-place/retention rules to memory rows; be aware of memory-poisoning risk (validate before persisting agent-written memories). 28 299. Everything stays Go-native: routing, RRF fusion, CRAG grading, and KV-cache control live in your chi handlers and the inference HTTP calls; pgvector does ANN+filter+compute. No Python, CGO_ENABLED=0 preserved (pgvector runs in Postgres, not in the Go binary), FIPS Go binary untouched. 30 31**Gotchas:** - Vector RAG cannot do aggregation: count/sum/filter/math questions answered by stuffing retrieved chunks will be wrong. Route these to TAG/SQL so Postgres computes the answer. This is the single most common production failure mode. 32- CAG context-window trap: it ONLY works when the whole KB fits the context window AND is static. Your opportunity/award corpus (1400+ solicitations, daily sync) disqualifies it; using CAG there silently drops knowledge. CAG also needs server-side KV-cache reuse support on the inference node — verify llama.cpp/Ollama actually persist/reuse the prompt cache for your preload, or the "no retrieval" win evaporates. 33- Self-RAG is NOT plug-and-play: it requires training a critic model and a reflection-token generator (GPT-4-distilled labels). Wrong fit for a stateless commodity inference plane and your no-Python/FIPS repo. Use CRAG instead, which is model-agnostic and plug-and-play. 34- CRAG's reference web-search fallback is inappropriate for CUI/federal data — do not call out to public web search. Swap in an approved-corpus or human-in-the-loop fallback, and audit it under your AEDT/served-score rules. 35- pgvector 0.8 iterative scan modes: `relaxed_order` returns APPROXIMATE distance ordering — there are open reports (pgvector issue #862) of the planner assuming strict ordering and producing surprising orderings; for any path where exact rank matters (adverse-impact/fairness ranking, slate ordering) use `strict_order` and verify. Don't blindly global-set relaxed_order. 36- Filtered ANN recall: even with 0.8's better planner, tight WHERE filters + small ef_search can overfilter and return too few rows. Iterative scan mitigates it but you must tune `hnsw.max_scan_tuples`; unbounded iterative scans can degrade latency, so cap it. 37- GraphRAG indexing cost: the entity-extraction + community-summarization pass is LLM-heavy and slow/expensive to rebuild; it is not a free upgrade. Only adopt for genuinely global/sensemaking questions, and beware re-indexing cost on a daily-changing corpus. 38- halfvec at 384-dim is safe (<1% recall loss); binary quantization at this dimensionality drops recall hard and needs a rerank stage — not worth it here. 39- Memory poisoning / security: agent-written long-term memory is an attack surface (MemoryGraft / memory-poisoning literature). Validate and tenant-scope memory writes under RLS; never let cross-tenant memory leak. Apply existing anonymize/retention/audit policy to memory rows. 40- "Augmented generation" zoo is mostly marketing: CAG/KAG/TAG/GraphRAG are a handful of real primitives plus a router. Don't build separate stacks per acronym — build one router + Postgres-resident retrieval/compute and select per query. 41- Embedding/normalization consistency: cosine HNSW assumes normalized vectors; if the inference plane emits unnormalized 384-dim embeddings, recall and halfvec accuracy degrade. Normalize at write time and keep the same model for index and query. 42 43**Primary sources:** 44- [Don't Do RAG: When Cache-Augmented Generation (CAG) is All You Need for Knowledge Tasks (arXiv 2412.15605)](https://arxiv.org/abs/2412.15605) 45- [KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation (arXiv 2409.13731)](https://arxiv.org/abs/2409.13731) 46- [Text2SQL is Not Enough: Unifying AI and Databases with TAG (arXiv 2408.14717)](https://arxiv.org/html/2408.14717v1) 47- [Project GraphRAG - Microsoft Research](https://www.microsoft.com/en-us/research/project/graphrag/) 48- [microsoft/graphrag GitHub (modular graph-based RAG)](https://github.com/microsoft/graphrag) 49- [Corrective Retrieval Augmented Generation / CRAG (arXiv 2401.15884)](https://arxiv.org/abs/2401.15884) 50- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (arXiv 2310.11511)](https://arxiv.org/abs/2310.11511) 51- [Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic RAG for Industry (arXiv 2506.10408)](https://arxiv.org/pdf/2506.10408) 52- [RAGRouter-Bench: A Dataset and Benchmark for Adaptive RAG Routing (arXiv 2602.00296)](https://arxiv.org/pdf/2602.00296) 53- [PostgreSQL.org: pgvector 0.8.0 Released (iterative scans, filter planning)](https://www.postgresql.org/about/news/pgvector-080-released-2952/) 54- [pgvector Iterative Index Scans docs (relaxed_order/strict_order, max_scan_tuples)](https://docs.pgedge.com/pgvector/v0-8-0/iterative-index-scans/) 55- [Supercharging vector search with pgvector 0.8.0 (AWS, filtered HNSW + iterative scans)](https://aws.amazon.com/blogs/database/supercharging-vector-search-performance-and-relevance-with-pgvector-0-8-0-on-amazon-aurora-postgresql/) 56- [Jonathan Katz: Scalar and binary quantization for pgvector (halfvec recall tradeoffs)](https://jkatz05.com/post/postgres/pgvector-scalar-binary-quantization/) 57- [pgvector issue #862: relaxed_order planner strict-ordering inconsistency](https://github.com/pgvector/pgvector/issues/862) 58- [MemGPT / Mem0 / Zep memory-augmented agent surveys (Agent-Memory-Paper-List)](https://github.com/Shichun-Liu/Agent-Memory-Paper-List) 59 60## pgvector 0.8.0 HNSW tuning at small-to-medium scale (thousands of rows) + hybrid BM25/tsvector + dense-vector search via Reciprocal Rank Fusion in SQL 61 62**Recommendation:** For this Go 1.26 / chi / pgx / sqlc / pgvector-go stack on PG16 + pgvector 0.8.0, 384-dim HNSW, at thousands of rows: 63 641) FIPS/CGO is a non-issue here: pgvector is a server-side Postgres C extension, not a Go module. It never links into the CGO_ENABLED=0 FIPS Go binary. pgvector-go is pure Go (registers the vector type with pgx); keep using it. Nothing in this research touches the Go binary's FIPS posture. 65 662) Index: CREATE INDEX ... USING hnsw (embedding vector_cosine_ops) WITH (m=16, ef_construction=64) is fine; at this scale bumping ef_construction to 128-200 costs almost nothing on build and slightly improves recall. Match the op-class to your query operator: cosine -> vector_cosine_ops + <=>. At 384 dims, full float vectors are cheap (1544 bytes/row); skip halfvec unless you later scale to millions. 67 683) At thousands of rows expect the planner to often pick a seqscan (exact brute-force KNN), which is sub-ms and 100% recall — that is correct and desirable, not a bug. Do NOT force the index in production. Use SET LOCAL enable_seqscan=off only in tests to prove the index works. Run ANALYZE after bulk loads so cost estimates are sane. 69 704) Turn ON iterative scans for any query that combines a tenant/metadata WHERE with ORDER BY embedding <=> $q LIMIT n, since RLS tenancy filters can drop the candidate pool below the LIMIT. Per-request, inside the pgx txn: SET LOCAL hnsw.iterative_scan='relaxed_order'; SET LOCAL hnsw.ef_search=100; keep hnsw.max_scan_tuples=20000 default. With relaxed_order, re-sort at the outer query: SELECT * FROM (...) q ORDER BY q.distance. Use strict_order only if exact ordering is contractually required. Set these via SET LOCAL in the same transaction as the query (sqlc: a small exec before the SELECT, or a custom query method) — not globally. 71 725) Hybrid search: do NOT add BM25 and cosine scores. Use rank-based RRF in one SQL statement. Generated tsvector column (GENERATED ALWAYS AS to_tsvector('english',content) STORED) + GIN index; HNSW on embedding. Pattern: CTE per arm with ROW_NUMBER() OVER (ORDER BY ...) and LIMIT 50 each (over-fetch); UNION ALL 1.0/(60+rank); GROUP BY id; SUM; ORDER BY score DESC; LIMIT n. Pass the query embedding as a $param (pgvector-go encodes it) and the text query via websearch_to_tsquery('english',$q). k=60 default; expose it as a tunable. Accept that ts_rank_cd is not true BM25 (no IDF, no length norm) — adequate at thousands of rows; only reach for a Rust BM25 extension (ParadeDB pg_search) if lexical quality becomes a measured problem, and note that adds a non-FIPS C extension to the DB server (not your Go binary). 73 746) Upgrade path: you are on 0.8.0; 0.8.3 (2026-06-17) fixed possible HNSW index corruption during VACUUM and a PG18 perf regression. If autovacuum touches the vector table, plan the bump to 0.8.3 — it is a server extension upgrade, transparent to the Go code. 75 76**Gotchas:** 1) Iterative scans default OFF — filtered ORDER BY vector LIMIT queries silently under-return without it; this directly bites RLS multi-tenant filters that shrink the candidate pool. 2) relaxed_order returns slightly mis-ordered rows — you MUST re-sort by the distance expression in an outer query or results look wrong. 3) Iterative scans only activate when the HNSW index is the chosen plan; at thousands of rows the planner usually (correctly) seqscans, so iterative-scan settings simply no-op then. 4) Op-class/operator mismatch (e.g. vector_l2_ops index queried with <=> cosine) silently falls back to seqscan with no error. 5) ts_rank/ts_rank_cd is NOT BM25 — no IDF, no document-length normalization; do not advertise it as BM25. 6) NEVER linearly combine BM25 and cosine scores (incompatible scales) — RRF fuses by rank, not score. 7) Each RRF arm needs its own LIMIT (over-fetch ~50) before fusion; feeding full result sets defeats RRF. 8) hnsw.scan_mem_multiplier scales off work_mem, not maintenance_work_mem; large iterative scans can exhaust memory if work_mem is tiny. 9) SET (not SET LOCAL) of hnsw.* leaks the setting onto the pooled pgx connection for later queries — always SET LOCAL inside the txn. 10) 0.8.0 specifically lacks the 0.8.3 fix for HNSW-vacuum index corruption. 11) maintenance_work_mem 8GB advice is for million-row builds; irrelevant and wasteful at thousands of rows where builds are instant. 77 78**Primary sources:** 79- [pgvector README (HNSW options, ef_search, Iterative Index Scans, Filtering, build tuning)](https://github.com/pgvector/pgvector/blob/master/README.md) 80- [pgvector CHANGELOG (0.8.0 features incl. iterative scans + cost estimation; 0.8.3 HNSW-vacuum corruption fix)](https://github.com/pgvector/pgvector/blob/master/CHANGELOG.md) 81- [AWS: Supercharging vector search performance and relevance with pgvector 0.8.0 on Aurora PostgreSQL (iterative scan modes, ef_search/recall benchmarks, planner cost changes)](https://aws.amazon.com/blogs/database/supercharging-vector-search-performance-and-relevance-with-pgvector-0-8-0-on-amazon-aurora-postgresql/) 82- [pgvector issue #835 — HNSW index not used for KNN, planner prefers seq scan (PG17.4, pgvector 0.8.0)](https://github.com/pgvector/pgvector/issues/835) 83- [pgvector issue #584 — Can't get the query planner to use HNSW index](https://github.com/pgvector/pgvector/issues/584) 84- [ParadeDB — Hybrid Search in PostgreSQL: The Missing Manual (RRF SQL, why rank-fusion not score-fusion, k=60, per-arm LIMIT)](https://www.paradedb.com/blog/hybrid-search-in-postgresql-the-missing-manual) 85- [ParadeDB — Implementing BM25 in PostgreSQL (ts_rank lacks IDF/length-normalization vs true BM25)](https://www.paradedb.com/learn/search-in-postgresql/bm25) 86- [DEV: Building Hybrid Search for RAG — pgvector + full-text + RRF (full CTE/RRF SQL example)](https://dev.to/lpossamai/building-hybrid-search-for-rag-combining-pgvector-and-full-text-search-with-reciprocal-rank-fusion-6nk) 87 88## Serving bge-small-en-v1.5 (384-dim embeddings) and bge-reranker-v2-m3 (cross-encoder rerank) without Python on an Apple-Silicon/Metal inference plane, over HTTP, matching an existing Postgres 16 + pgvector 0.8.0 vector(384) HNSW column. 89 90**Recommendation:** Run TWO `llama-server` processes on the Apple-Silicon inference node (Metal-accelerated, native arm64 binaries, zero Python), each on its own port, reached from the Go backend over HTTP across Tailscale: 91 921) Embedder: `llama-server --hf-repo ggml-org/bge-small-en-v1.5-Q8_0-GGUF --hf-file bge-small-en-v1.5-q8_0.gguf -c 512 --embedding --pooling cls -ngl 99 --port 8011` (bge max context is 512 tokens; no need for 2048). Call the OpenAI-shaped `POST /v1/embeddings` from Go with stdlib net/http + encoding/json — nothing FIPS-sensitive here since this is a plain HTTPS/JSON client; your `CGO_ENABLED=0` FIPS Go binary is unaffected because the model never runs in-process. Read `data[0].embedding` (384 floats, already L2-normalized) and write straight into your `vector(384)` column; query the HNSW index with `<=>` (cosine) — consistent because both stored and query vectors are CLS-pooled + normalized. 93 942) Reranker: `llama-server -hf gpustack/bge-reranker-v2-m3-GGUF:Q8_0 --reranking --pooling rank -ngl 99 --port 8012`. After pgvector returns your top-K (e.g. K=50) candidates, POST `{query, documents:[...K...]}` to `/v1/rerank`, sort by `relevance_score` desc, take top-N. 95 96REQUIRED Go-side logic: prepend `Represent this sentence for searching relevant passages: ` to QUERY text only before embedding (passages stay raw) — and verify your already-stored vectors were generated the same way (un-prefixed passages, CLS pooling, L2 norm). If the existing pgvector(384) column was built by a different pooling/normalization, regenerate it with this exact server config so query and corpus live in the same space. 97 98Operationally on Apple Silicon: launch both via a `launchd` plist (mirror the durable pattern already used for the M4 services), keep them STATELESS (no per-request state) as intended, and pin a specific llama.cpp build — see gotcha on the /v1/embeddings regression. Prefer the native `/embedding` endpoint if you ever hit the `/v1/embeddings` 501 regression; it returns the same 384 vector and accepts `--pooling cls`. 99 100Do NOT use Ollama for this stack: it cannot serve the cross-encoder reranker correctly, so you'd end up splitting across two runtimes. One llama.cpp toolchain serves both halves. 101 102**Gotchas:** 1) POOLING MUST BE EXPLICIT: `--embd-bge-small-en-default` convenience flag confusingly sets pooling to `none`, which makes `/v1/embeddings` fail ("Pooling type 'none' is not OAI compatible") AND would give per-token output even on the native endpoint. Always pass `--pooling cls` explicitly (GitHub issue #16451). 103 1042) /v1/embeddings 501 REGRESSION: builds after release b5630 had an open bug where `/v1/embeddings` returns HTTP 501 "This server does not support embeddings" even with `--embeddings` set (issue #15406). Pin/test your specific llama.cpp build; if you hit it, fall back to the native `/embedding` endpoint (same 384 vector) or use a known-good build. 105 1063) QUERY PREFIX IS YOUR JOB: llama.cpp embeds raw text and does NOT auto-add the BGE query instruction. Forgetting `Represent this sentence for searching relevant passages:` on queries (or accidentally adding it to passages) silently degrades retrieval — vectors still 384-dim and "valid", just worse recall. Hardest bug to notice. 107 1084) NORMALIZATION ALIGNMENT: with a pooling mode != none the server L2-normalizes output. Make sure pgvector queries use cosine/IP consistently and that your STORED vectors were also normalized; mixing normalized and unnormalized vectors corrupts ranking but throws no error. 109 1105) RERANK SCORE IS A LOGIT, NOT A PROBABILITY: `relevance_score` is unbounded (e.g. 5.97, can be negative). Use for ordering only; sigmoid it if you need a 0-1 threshold. Map the returned `index` back to your candidate array — the endpoint returns indices into the `documents` array you sent. 111 1126) `--pooling rank` belongs to the RERANKER process only; do not set it on the embedder. Reranking needs all three: `--reranking --embedding --pooling rank`. 113 1147) OLLAMA CANNOT RERANK: no native `/api/rerank`; running bge-reranker through `/api/embed` and using embedding magnitude as a relevance proxy is undocumented and unreliable. This is the single biggest reason to choose llama.cpp over Ollama for THIS system. 115 1168) CONTEXT LENGTH: bge-small caps at 512 tokens; chunk longer passages before embedding. The reranker (XLM-RoBERTa) also has a finite max length — keep query+document pairs within model limits or scores degrade. 117 1189) METAL/-ngl: pass `-ngl 99` to offload all layers to the Apple GPU; these models are tiny so they fit easily and run sub-100ms. Keep each server `--host 0.0.0.0` only on the trusted Tailscale interface, not public. 119 120**Primary sources:** 121- [llama.cpp server README (embedding + rerank endpoints, flags, pooling modes, request/response shapes)](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) 122- [PR #9510: llama : add reranking support (exact --reranking/--pooling rank flags + /v1/rerank request and relevance_score response shape)](https://github.com/ggml-org/llama.cpp/pull/9510) 123- [ggml-org/bge-small-en-v1.5-Q8_0-GGUF (canonical 384-dim embedding GGUF + llama-server launch example)](https://huggingface.co/ggml-org/bge-small-en-v1.5-Q8_0-GGUF) 124- [BAAI/bge-small-en-v1.5 model card (384 dims, CLS pooling, L2 normalize, query instruction prefix queries-only)](https://huggingface.co/BAAI/bge-small-en-v1.5) 125- [gpustack/bge-reranker-v2-m3-GGUF (cross-encoder reranker GGUF files and quant sizes)](https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF) 126- [llama.cpp issue #16451: --embd-bge-small-en-default sets pooling=none; use --pooling cls](https://github.com/ggml-org/llama.cpp/issues/16451) 127- [llama.cpp issue #15406: /v1/embeddings 501 regression after build b5630](https://github.com/ggml-org/llama.cpp/issues/15406) 128- [Ollama reranking-endpoint limitation (no native cross-encoder rerank; embed-magnitude hack)](https://github.com/overcuriousity/ollama-utils/blob/main/plugins/reranking-endpoint/README.md) 129 130## Layout-aware document extraction for resumes and government solicitations (Sections L and M): LayoutLMv3 vs Donut vs a local vision-language model, run out-of-process on Apple Silicon. 131 132**Recommendation:** Use a TWO-TIER extraction pipeline, with the VLM as a fallback, not the default. This matches the repo's existing `internal/platform/extraction` deterministic port (per W1 Mission Check memory). 133 134TIER 1 (default, in Go, no inference plane): For digital-born PDFs and DOCX (the overwhelming majority of resumes and SAM-posted Section L/M docs), extract text WITH layout/bounding-box geometry in-process. Avoid Donut/LayoutLMv3/VLM entirely. pdfium exposes FPDFText_CountRects/FPDFText_GetRect/FPDFText_GetBoundedText for text+bbox, but it is C (CGO) — for CGO_ENABLED=0/FIPS, keep PDF parsing in a tiny SIDE process (a pdfium- or pdfplumber-class tool) behind the same HTTP boundary as the inference plane, OR use a pure-Go PDF text extractor and reconstruct reading order from word coordinates. Serialize columns/sections in reading order into text, then send to a SMALL TEXT LLM on the existing inference plane to emit JSON. Do NOT add Python to the repo; if pdfium is needed, isolate it as an out-of-process HTTP extractor exactly like the VLM, so the FIPS Go binary stays CGO-free. 135 136TIER 2 (fallback, VLM on Apple Silicon over Tailscale): When a page is scanned/image-only or text extraction yields garbage, render the page to PNG and call the VLM. Self-host Qwen3-VL-8B-Instruct GGUF (Q4_K_M LLM ~5.0 GB + Q8_0 or F16 mmproj) on the Apple-Silicon node via llama.cpp's `llama-server` (or Ollama `qwen3-vl:8b`). 8B is the sweet spot for OCR/document on M2/M4; drop to 4B if memory-constrained. This is licensed (Apache-2.0), Metal-accelerated, and reached over plain HTTP — no Python, nothing linked into the Go binary, no FIPS impact since the Go service is only an HTTP client. 137 138EXTRACTION API SHAPE (out-of-process, stateless, behind auth — consistent with the W1 browser->server->one-engine path): Expose your own internal Go endpoint `POST /extract` that takes {document bytes or page images, doc_type: resume|solicitation, target_schema}. Internally it (a) tries Tier 1, (b) falls back to Tier 2. The inference call to the Apple node uses the OpenAI-compatible contract: 139 POST http://<apple-node>:8080/v1/chat/completions 140 body: {"model":"qwen3-vl","messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}},{"type":"text","text":"<instruction>"}]}],"response_format":{"type":"json_schema","schema":{...}},"temperature":0} 141Force structured output: llama-server supports `response_format` json_schema and GBNF grammar (`--grammar`) plus `--jinja`; pin temperature=0 for determinism (important for your AEDT auditability — the same input must score the same way). Control cost with `-c/--ctx-size`, and for the dynamic-resolution Qwen-VL use `--image-min-tokens`/`--image-max-tokens` and `--mtmd-batch-max-tokens` to cap image token blow-up on large solicitation pages. Run llama-server with `--mmproj <vision.gguf>`. Keep the plane STATELESS: one request = full document context, no server-side session, retries idempotent. 142 143For Section L/M specifically (long, multi-page, instruction-dense, digital-born): this is a TEXT problem, not a vision problem. Use Tier 1 + a text LLM with the 256K-context Qwen3 class; do NOT default to feeding 30 page-images to a VLM (slow, token-expensive, lossy). Reserve the VLM for embedded tables/figures or scanned amendments. 144 145**Gotchas:** - LayoutLMv3 is NON-COMMERCIAL licensed — do not ship it in a paid federal SaaS. Donut is MIT and Qwen-VL is Apache-2.0, both fine. 146- Donut and LayoutLMv3 have NO GGUF / NO Metal / NO llama.cpp path. Using either forces PyTorch+Python into your inference plane, directly violating the "no Python in the repo" rule. Only GGUF VLMs (Qwen-VL, Gemma 3/4, etc.) run on your existing llama.cpp/Ollama Metal plane. 147- The mmproj (vision projector) GGUF is a SEPARATE file from the LLM GGUF; the server only does vision if you pass `--mmproj`. Mismatched mmproj/LLM versions silently break image input. Pin both from the same Qwen repo. 148- VLM determinism: VLMs are not bit-deterministic across builds/quantizations. For AEDT auditability you must pin model+quant+mmproj versions in your model registry/inventory (you already have one from W0), set temperature=0, and record the served output. Treat a model/quant change as a registry event. 149- Image-token blowup: Qwen-VL uses DYNAMIC resolution; a full-page solicitation scan can consume thousands of vision tokens per page and tank latency/throughput. Cap with --image-max-tokens / --mtmd-batch-max-tokens and downscale renders. This is the #1 reason not to make the VLM the default for multi-page Section L/M. 150- OCR-error propagation makes the LayoutLMv3 + external-OCR combo brittle on uncontrolled scans (~12% token error, 4.2x misclassification). If you go vision at all, prefer the OCR-free VLM over OCR->LayoutLMv3. 151- pdfium gives text+bbox but is CGO/C — keep it OUT of the FIPS CGO_ENABLED=0 binary by running it as a side process, same as the VLM. Don't be tempted to link it in. 152- Don't conflate "layout-aware" with "needs a VLM." For digital-born docs the layout is already in the PDF's coordinate stream; serialize it to text and use a cheap text LLM (per arXiv 2510.09722). Vision is only for pixels-only pages. 153- Apple-Silicon memory: an 8B VLM + mmproj + 256K-ctx KV cache can pressure an 8GB Mac. The cluster nodes in scope (M2/VALKYRIE M1 are 8GB; M4/Thunderport has more headroom) — schedule VLM fallback on the M4-class node, not the 8GB M1/M2, and consider the 4B variant if it must run on 8GB. 154- Keep the inference plane stateless and idempotent; never store CUI/PII server-side on the Apple node (consistent with W1's CUI-safe, browser-extracts-text-only design). The node should hold no document after the response returns. 155 156**Primary sources:** 157- [Qwen3-VL (Ollama library — sizes, 256K ctx, OCR/32-lang)](https://ollama.com/library/qwen3-vl) 158- [Qwen/Qwen3-VL-8B-Instruct-GGUF (HuggingFace — GGUF quants, mmproj, Metal, llama-server usage)](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-GGUF) 159- [Qwen2.5-VL (Ollama library — sizes, 125K ctx, structured doc/invoice/table extraction)](https://ollama.com/library/qwen2.5vl) 160- [llama.cpp multimodal docs (mmproj, --no-mmproj-offload, supported VLM families incl. Qwen2.5/3-VL)](https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md) 161- [llama.cpp server README (OpenAI-compatible image_url base64 API, --mmproj, json_schema/GBNF, image-min/max-tokens)](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) 162- [Donut: OCR-free Document Understanding Transformer (clovaai/donut — MIT, PyTorch-only, 2560x1920 input)](https://github.com/clovaai/donut) 163- [OCR-free Document Understanding Transformer (Donut paper, arXiv 2111.15664)](https://arxiv.org/abs/2111.15664) 164- [Layout-Aware Parsing Meets Efficient LLMs: resume IE without VLMs (arXiv 2510.09722, Oct 2025)](https://arxiv.org/pdf/2510.09722) 165- [Engineering Explained: LayoutLMv3 (KUNGFU.AI — OCR+bbox+image, non-commercial license, perf)](https://www.kungfu.ai/blog-post/engineering-explained-layoutlmv3-and-the-future-of-document-ai) 166- [PDFium text extraction with bounding boxes (chromium/pdfium DeepWiki — FPDFText_GetRect etc.)](https://deepwiki.com/chromium/pdfium/5.1-text-extraction-and-search) 167- [Benchmarking Local LLMs incl. Qwen2.5-VL-7B on Apple Silicon (~70 tok/s, 4.8GB Q4_K_M)](https://jmlab.net/projects/m5-macbook-benchmark-pipeline/) 168 169## Faithfulness / hallucination detection to gate generated answers (AlignScore, MiniCheck, Patronus Lynx, RAGAS faithfulness, + Vectara HHEM-2.1-Open) 170 171**Recommendation:** For this stack (commercial federal SaaS, Go 1.26/chi/pgx, CGO_ENABLED=0 FIPS binary, NO Python in repo, stateless Apple-Silicon llama.cpp/Ollama plane over HTTP), the license filter is decisive: Lynx and Bespoke-MiniCheck-7B are both CC-BY-NC-4.0 and must NOT be used to gate a paid product. Use a commercial-safe checker reached over HTTP — never link a Python lib into the Go binary. 172 173PRIMARY CHOICE — MiniCheck-Flan-T5-Large (MIT, ~770M) as a thin HTTP microservice on the inference plane: 174- It is the strongest commercially-licensed lightweight checker (repo's recommended near-GPT-4 sub-1B model), MIT licensed, runs comfortably on Apple Silicon. Wrap it in a tiny stateless HTTP server (the one piece of Python allowed to live OUTSIDE the Go repo, deployed on the M4/inference node exactly like Ollama already is) exposing POST /check {document, claim} -> {label, prob}. Your Go service stays pure (CGO_ENABLED=0, FIPS) and just makes an HTTP call over Tailscale, identical to how it already calls the LLM plane. 175- If you want ZERO extra Python anywhere: use Vectara HHEM-2.1-Open (Apache-2.0, 0.1B, <600MB, CPU, unlimited ctx) the same way, or — cleanest of all — quantize HHEM/MiniCheck-encoder is harder, so for a no-new-service path put Bespoke-MiniCheck behind Ollama ONLY if you obtain the commercial license; otherwise HHEM-2.1-Open is the no-strings, Apache-2.0 default. 176 177WIRING THE SELF-CORRECTION GATE (before any answer ships): 1781. Generate the answer from your existing llama.cpp/Ollama HTTP plane. 1792. Decompose the answer into atomic claims (RAGAS-style), one sentence/claim each, in Go. Pair each claim with the retrieved pgvector context (the 384-dim HNSW top-k chunks already in your pipeline). 1803. For each (context, claim) call the checker over HTTP and collect prob/label. 1814. Gate: faithfulness = supported_claims/total_claims. Require every claim to clear the threshold (start prob >= 0.5 with HHEM/AlignScore, label==1 for MiniCheck; raise to ~0.65 for a precision-biased ship gate and calibrate on a labeled hold-out). 1825. On FAIL, run ONE self-correction pass: re-prompt the generator with the specific unsupported claims plus instruction to restate using only the context (or to say it cannot answer), then re-check. If it still fails, fall back to an abstention / "not enough grounded evidence" response rather than shipping. 1836. Audit every gate decision into your existing append-only audit trail (served-score pattern from W2): store the per-claim probs, the threshold, the pass/fail, and whether a correction pass fired — this is the same auditability posture as the AEDT/served-score work and keeps the hallucination gate inspectable. Keep PII out of the audit row exactly like the aedt.ephemeral.evaluated event. 184 185Net: MiniCheck-Flan-T5-Large (MIT) as an HTTP sidecar on the inference plane, or HHEM-2.1-Open (Apache-2.0) for a truly Python-free, Apache-only deployment; gate per-claim at a tuned ~0.5-0.65 prob with a single self-correction retry then abstain; never adopt Lynx or Bespoke-7B in production without a commercial license. 186 187**Gotchas:** - LICENSE TRAP (most important): Patronus Lynx (v1.0 AND v1.1) and Bespoke-MiniCheck-7B are ALL CC-BY-NC-4.0 — non-commercial. They are the two highest-accuracy options (Lynx ~82.9% HaluBench, Bespoke-7B 77.4% LLM-AggreFact) and the most tempting, but using either to gate a paid federal SaaS is a license violation. Commercial-safe set = MiniCheck encoder checkpoints (lytang/*, MIT), AlignScore (MIT), HHEM-2.1-Open (Apache-2.0). 188- NO Python in the Go repo: AlignScore, MiniCheck, HHEM, and RAGAS are all Python libraries with heavy deps (torch/transformers). Do NOT try to embed them — they cannot be linked into a CGO_ENABLED=0 FIPS Go binary. The only correct pattern is a separate HTTP service on the inference plane (where Python/torch already effectively live for the model server), called over HTTP exactly like Ollama. 189- RAGAS is orchestration, not a checker: its default faithfulness needs an LLM judge (GPT-4o-mini in the docs) which means an external API call and non-determinism; only the FaithfulnesswithHHEM local variant avoids that. Borrow RAGAS's statement-decomposition + per-claim NLI ALGORITHM (reimplement in Go) rather than importing the library. 190- Thresholds are not portable constants: balanced-accuracy operating points differ per model and per domain. AlignScore's ~72% requires threshold tuning; HHEM ships no formal threshold. You MUST calibrate on a held-out labeled set from your own RAG traffic; do not hardcode 0.5 and assume it generalizes. 191- Per-claim, not per-answer: a whole-answer faithfulness score hides one bad claim among several good ones. Decompose to atomic claims and require ALL to pass; this is also what makes the gate auditable and the self-correction targeted. 192- Context length: HHEM-1.0 was capped at 512 tokens (HHEM-2.1-Open removes this); Lynx v1.0 is 8K, v1.1 is 128K; Bespoke-MiniCheck is 32K. If you feed multi-chunk pgvector context, watch truncation — a claim scored against a truncated context can false-FAIL. 193- AlignScore default mode is nli_sp (chunk+sentence split); using raw 'nli'/'bin' modes changes scores. Pin the evaluation_mode if you adopt it. 194- Self-correction must terminate: cap at one retry then abstain. Unbounded re-generation loops on genuinely-ungrounded questions will burn the inference plane and still ship nothing trustworthy. 195 196**Primary sources:** 197- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (GitHub, EMNLP 2024)](https://github.com/Liyan06/MiniCheck) 198- [MiniCheck-Flan-T5-Large model card (MIT license, ~770M)](https://huggingface.co/lytang/MiniCheck-Flan-T5-Large) 199- [Bespoke-MiniCheck-7B model card (77.4% LLM-AggreFact, CC-BY-NC-4.0)](https://huggingface.co/bespokelabs/Bespoke-MiniCheck-7B) 200- [Bespoke-MiniCheck on Ollama (Document/Claim -> Yes/No, 4.7GB, 32K ctx, HTTP API)](https://ollama.com/library/bespoke-minicheck) 201- [Patronus Lynx 8B v1.1 model card (prompt template, JSON PASS/FAIL, 128K ctx, CC-BY-NC-4.0)](https://huggingface.co/PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct-v1.1) 202- [Patronus Lynx 8B v1.0 model card (HaluBench 82.9%, 8K ctx, CC-BY-NC-4.0)](https://huggingface.co/PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct) 203- [AlignScore (GitHub, ACL 2023, RoBERTa, evaluation_mode nli_sp)](https://github.com/yuh-zha/AlignScore) 204- [AlignScore checkpoints (MIT license)](https://huggingface.co/yzha/AlignScore) 205- [Vectara HHEM-2.1-Open model card (Flan-T5-base, Apache-2.0, <600MB CPU, unlimited ctx)](https://huggingface.co/vectara/hallucination_evaluation_model) 206- [RAGAS Faithfulness metric docs (statement decomposition + NLI; LLM judge vs FaithfulnesswithHHEM local mode)](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/) 207 208## Prompt-injection and output guardrails for an LLM that triggers read-only parameterized SQL (TAG / text-to-SQL) over multi-tenant data 209 210**Recommendation:** For the missionMATCH Go/chi/pgx/sqlc stack with a read-only TAG agent, deliberately break the lethal trifecta at the DATABASE leg and treat the LLM as an untrusted query-intent generator, never as the security boundary: 211 2121. NEVER let the LLM emit raw SQL strings. Use the Context-Minimization + Plan-Then-Execute patterns: have the model emit a constrained query-INTENT object (which named report/template, which allowlisted columns, which filter operators, which literal values) and have deterministic Go code (sqlc-generated, parameterized queries) compile that intent into the actual SQL. This is the natural-language analogue of parameterized statements — values from the model can only ever land in bind parameters ($1,$2), never in identifiers or query structure. This collapses "prompt injection -> SQL injection" to impossible by construction. 213 2142. Make structured output a parsing aid, not a guard. On the llama.cpp/Ollama Metal plane, drive the model with a json_schema/GBNF grammar (llama-server `json_schema` field or `response_format`) so the intent object is always parseable. Then validate the parsed object in Go against an explicit ALLOWLIST: allowed table/view names, allowed columns per role, allowed operators, value type/length caps. Reject (don't sanitize) anything off-list. Re-validate every column/table even though the grammar "looked" valid — grammar guarantees syntax only. 215 2163. Enforce at the DB, leveraging the RLS architecture already in place. Run the agent's queries through the existing least-privilege `app` role under RLS, but additionally: (a) put each LLM query in an explicit READ ONLY transaction (BEGIN; SET TRANSACTION READ ONLY) AND grant the agent a role with no INSERT/UPDATE/DELETE/DDL privileges — do NOT rely on default_transaction_read_only, which is session-overridable and is "not a security setting"; the GRANT is the real boundary. (b) Set statement_timeout (e.g. 5-10s) on that role to kill runaway/DoS queries. (c) Keep tenant scoping in set_config/RLS as today so the model literally cannot reach another tenant's rows even if it tries. This is the same composite-FK / NULLIF default-deny RLS posture already proven in the slate loop. 217 2184. Close the exfiltration leg in the Go API output path. Because returned rows are untrusted (could contain attacker-planted text from a tenant's own data fields), when the API renders model-summarized output: strip/neutralize markdown links and images (the canonical Willison exfiltration vector), set a strict egress/CSP posture so no auto-fetched URL can carry data out, and never let the model trigger an outbound HTTP tool. With no write capability AND no external-communication capability, two of the three trifecta legs are gone — injection becomes a nuisance, not a breach. 219 2205. Human-in-the-loop only where state changes (Meta Rule of Two): for the read-only TAG path this is unneeded, but any "adopt-as-requirement" / write action surfaced from model output must keep the existing audited human decision step (consistent with the never-auto-reject W0 posture). 221 2226. Test it: stand up an AgentDojo-style adversarial suite (injected text inside returned tenant rows attempting cross-tenant reads, write attempts, and link-based exfiltration) in Go integration tests, and gate CI on attack-success-rate = 0 for the deterministic-compile path. No Python is required; the harness is just table-driven Go tests hitting the chi endpoints plus SQL assertions. 223 224Net: do not buy/build a "prompt-injection detector." Spend the effort on the deterministic intent->parameterized-SQL compiler, the read-only GRANT + statement_timeout role, RLS tenant scoping, and the markdown/egress stripping. That is a provable boundary; a classifier is a 95%-is-failing heuristic. 225 226**Gotchas:** - default_transaction_read_only is NOT a security control: any session can SET it back to off. Enforce read-only with a role that simply lacks write GRANTs (and an explicit READ ONLY transaction as belt-and-suspenders). 227- Grammar/JSON-schema constrained decoding guarantees SYNTAX, not SEMANTICS. A schema-valid object can still name a forbidden table or column. Always re-validate the parsed intent against your allowlist in Go. 228- Guardrail/classifier products advertising ~95% block rates are a failing grade for a security boundary (Willison); treat any LLM-as-judge or input filter as one defense-in-depth layer, never as the boundary. Bypasses appear within weeks. 229- The exfiltration leg is easy to leave open by accident: markdown image/link auto-rendering in the frontend is the classic data-theft channel even with a read-only DB, because the LLM can embed tenant data in a URL. Strip links/images and lock egress/CSP. 230- "Read-only" still allows DoS and data-scope abuse: heavy cross-join/seq-scan queries (set statement_timeout) and reading more rows than intended within a tenant (constrain via the intent allowlist + RLS, not via prose instructions). 231- Returned ROWS are untrusted input too, not just the user message. A tenant's own free-text field can carry an injection that fires when the model summarizes results — Context-Minimization (drop the user prompt before results return) and never giving the model a tool after it reads rows are what neutralize this. 232- llama.cpp json_schema->GBNF does not support uniqueItems/contains/conditional/negation and defaults additionalProperties=false; keep the intent schema simple and enforce richer constraints in Go. 233- Multi-statement / stacked queries and comment tricks must be blocked at the compiler layer (single statement, no semicolons in literals, parameterized values) — pgx with parameterized queries already refuses to interpolate, so the risk is only if you ever build SQL via string concatenation from model output. Don't. 234- FIPS/CGO_ENABLED=0 and no-Python constraints are fully satisfiable: the whole pattern is deterministic Go (sqlc/pgx) plus an HTTP call to the stateless Metal inference plane; none of the recommended controls require a Python guardrail library. 235 236**Primary sources:** 237- [The lethal trifecta for AI agents (Simon Willison, Jun 2025)](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/) 238- [OWASP LLM01:2025 Prompt Injection (OWASP GenAI Security Project)](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) 239- [OWASP LLM Prompt Injection Prevention Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html) 240- [Design Patterns for Securing LLM Agents against Prompt Injections (Willison summary of DeepMind arXiv:2506.08837)](https://simonwillison.net/2025/Jun/13/prompt-injection-design-patterns/) 241- [Defeating Prompt Injections by Design (CaMeL, Debenedetti et al., arXiv:2503.18813)](https://arxiv.org/abs/2503.18813) 242- [Agents Rule of Two: A Practical Approach to AI Agent Security (Meta AI, 2026)](https://ai.meta.com/blog/practical-ai-agent-security/) 243- [Prompt injection is the new SQL injection, and guardrails aren't enough (Cisco)](https://blogs.cisco.com/ai/prompt-injection-is-the-new-sql-injection-and-guardrails-arent-enough) 244- [llama.cpp GBNF grammars / JSON-schema constrained decoding (grammars/README.md)](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md) 245- [AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (arXiv:2406.13352)](https://arxiv.org/pdf/2406.13352) 246- [Read-only mode and default_transaction_read_only caveats (PostgreSQL docs: SET TRANSACTION + client connection defaults)](https://www.postgresql.org/docs/current/sql-set-transaction.html) 247 248## Running local instruct models on Apple Silicon (16 GB) via llama.cpp / Ollama, Metal-accelerated, alongside other processes: model sizing/quantization, memory budgeting, concurrency, and a stateless HTTP generation API. 249 250**Recommendation:** Use llama.cpp's llama-server (not Ollama) as the inference plane, fronted by the Go API over HTTP/Tailscale. Rationale tied to this stack: (1) Your Go backend is CGO_ENABLED=0 / FIPS — do NOT use any llama.cpp/Ollama Go cgo bindings. Talk to the inference box ONLY over HTTP from a pure-Go client, so the FIPS Go binary stays cgo-free; the Metal/cgo lives entirely in the separate llama-server process on the Apple Silicon node, exactly as your 'stateless inference plane reached over HTTP' architecture intends. (2) llama-server keeps the model resident for the process lifetime (no 5-min unload roulette like Ollama's default), exposes a clean /health for your readiness probe, supports --api-key for a bearer token over Tailscale, and is a single static binary you can run under launchd/caffeinate the same way as your VALKYRIE Node backend. 251 252Concrete config for one shared 16 GB Apple Silicon node: model = Meta-Llama-3.1-8B-Instruct Q4_K_M GGUF (bartowski quant) for maximum JSON/tool-calling reliability, or Qwen3-8B Q4_K_M if you want longer context / reasoning. Launch flags: -ngl all (full Metal offload), -fa on, -ctk q8_0 -ctv q8_0 (halve KV; safe because -fa is on), --parallel 2, -c 16384 (= 2 slots x 8192 each — note the per-slot division), --host 0.0.0.0 --port <p> --api-key <token>. Budget: ~4.9 GB weights + ~1.1 GB KV (halved by q8 cache, split across 2 slots) + overhead ≈ 6-7 GB resident, leaving 9+ GB for OS + any co-located processes. If pgvector/Postgres also live on this M4 (your dev DB does), drop to --parallel 1 and -c 8192, or move inference to a node that is not also the DB. Keep your 384-dim embeddings in pgvector/HNSW as-is — do NOT use llama-server /v1/embeddings for the 384-dim index (that's a separate embedding model concern), generation and embedding are different model loads and you don't want both resident on 16 GB. 253 254For structured extraction from the Go side, drive it via the OpenAI-compatible response_format with a json_schema and ALSO describe the shape in the prompt (the grammar does not inject the schema), set temperature low, and validate the returned JSON in Go (treat a fail-open unconstrained response as a retryable error). Use a plain net/http client with a generous timeout and a /health-gated readiness check; add OLLAMA_/llama-server queue backpressure expectations into the Go client (bounded concurrency from the API so you never exceed the slot count). 255 256**Gotchas:** 1) CONTEXT IS DIVIDED ACROSS SLOTS: with llama-server -np N, -c is split N ways (and Ollama scales required RAM by NUM_PARALLEL * CONTEXT_LENGTH). Setting -c 8192 -np 4 silently gives each request only 2048 tokens — a frequent cause of truncated prompts. Size -c = slots x per-request-ctx. 2572) KV CACHE QUANT NEEDS FLASH ATTENTION: -ctk/-ctv q8_0 without -fa on makes llama.cpp dequantize per attention op and can be SLOWER than f16. Always pair cache quant with -fa. 2583) FIPS/CGO TRAP: do not pull a cgo llama binding into the Go repo — it breaks CGO_ENABLED=0 and FIPS (GOFIPS140). Keep all Metal/cgo in the out-of-process llama-server and talk HTTP only. 2594) SHARED 16 GB CONTENTION: if Postgres+pgvector and the model co-reside (your M4 dev DB does), unified memory pressure can swap or OOM under load; macOS will compress/swap silently and tank tok/s. Prefer a dedicated inference node, or cap --parallel 1 and ctx when co-located. 2605) OLLAMA SILENT UNLOAD: default keep_alive is 5 min — first request after idle pays a multi-second reload. Pin with keep_alive:-1 or OLLAMA_KEEP_ALIVE, or just use llama-server (resident for process life). 2616) STRUCTURED-OUTPUT FAIL-OPEN: llama-server can return UNCONSTRAINED text when a json_schema grammar fails to parse (GitHub issue #19051); additionalProperties defaults to false; the schema is NOT added to the prompt. Always validate JSON in Go and describe the shape in the prompt; treat schema-violating output as retryable. 2627) Q4 QUALITY FLOOR: don't go below Q4_K_M for instruct/tool-calling reliability on 8B; sub-4-bit saves little memory here and degrades JSON adherence. 2638) /health RETURNS 503 WHILE LOADING: gate your Go readiness/liveness probe on 200, and expect a cold-load window at startup before the first request. 264 265**Primary sources:** 266- [llama.cpp server README (tools/server) — flags, slots, OpenAI endpoints, response_format, --api-key, /health](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) 267- [Ollama FAQ — OLLAMA_NUM_PARALLEL / MAX_LOADED_MODELS / MAX_QUEUE, keep_alive, RAM scaling](https://docs.ollama.com/faq) 268- [Ollama OpenAI compatibility docs — /v1/chat/completions, format/json schema](https://docs.ollama.com/api/openai-compatibility) 269- [llama.cpp GBNF grammar README — grammar-constrained / structured output](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md) 270- [ggml-org/llama.cpp Discussion #9784 — why llama.cpp uses RAM/VRAM (weights + KV + overhead, 8B Q4 ~4.9GB, KV ~1.1GB@8K)](https://github.com/ggml-org/llama.cpp/discussions/9784) 271- [ggml-org/llama.cpp Discussion #4130 — parallelization / batching / slots explanation](https://github.com/ggml-org/llama.cpp/discussions/4130) 272- [llama.cpp issue #19051 — llama-server fails open when JSON schema grammar parsing fails](https://github.com/ggml-org/llama.cpp/issues/19051) 273- [QwenLM/Qwen3 — current small instruct model (8B), tool calling, 128K context](https://github.com/QwenLM/Qwen3) 274 275## Federated learning with differential privacy for cross-tenant model improvement without sharing client contents — privacy accounting, a contents-scrubbed signal channel, and disabled-flag scaffolding (design-only) 276 277**Recommendation:** FOR THIS SYSTEM (Go 1.26 / chi / pgx / sqlc / pgvector 0.8, 384-dim HNSW, FIPS CGO_ENABLED=0, stateless Apple-Silicon inference plane over HTTP, no Python in repo) — DESIGN-ONLY, BEHIND A DISABLED FLAG: 278 279UNIT OF PRIVACY: org/tenant-level DP (each tenant is one "user"). This matches the cross-tenant ask and is the strongest defensible claim for a GovCon compliance story. Pair it cleanly with the existing RLS composite-FK tenancy: the federated signal is computed strictly inside a tenant's RLS scope, then leaves only as a clipped+noised aggregate. 280 281SIGNAL CHANNEL (what crosses the tenant boundary): NOT documents, NOT candidate text, NOT raw 384-dim embeddings. Ship a clipped, Gaussian-noised AGGREGATE in embedding space — e.g. a per-tenant centroid/mean-shift or a low-rank update to a shared projection, L2-clipped to a fixed sensitivity C then perturbed with sigma calibrated to the target (epsilon, delta). The aggregate is order-invariant and carries no row identity. This is the DP-FedEmb pattern, not the DP-SGD-on-a-big-net pattern, which is correct because your shared asset is an embedding/retrieval model, not a deep classifier. Audit every emission to the append-only audit log as a distinct DP-release event (count it against the budget), mirroring how W1/W2 already audit served scores with no PII. 282 283ACCOUNTING (Go-native, since no Python allowed): implement an RDP/zCDP accountant in pure Go — this is the pragmatic choice because (a) the Gaussian-mechanism RDP curve is a closed form (alpha/(2 sigma^2) per release; subsampled-RDP bound from arxiv 1808.00087 if you sample tenants), (b) composition is just summation over a fixed grid of Renyi orders, (c) RDP->(epsilon, delta) conversion is a closed-form min over orders. This is a few hundred lines of decimal/float Go, needs NO cgo, and stays inside the FIPS boundary. Use shopspring/decimal only where determinism matters (the audit-recorded epsilon ledger); the RDP grid math can be float64. Store the running budget per shared-model-version in Postgres (a `dp_budget_ledger` table: model_version, tenant_id_nullable, rdp_orders jsonb, accumulated_rdp, epsilon_at_delta, delta, released_at) and refuse emission once spent — a hard server-side gate, never client-trusted. Do NOT attempt to port Google's PLD/Connect-the-Dots into Go for v1; RDP is tighter than advanced composition, correct, and verifiable. Pin delta <= 1e-7 (well under 1/num_tenants) and target epsilon < 10 (Gboard band 0.994–13.69 is your reference); make epsilon/delta config, not hardcoded. 284 285SECURE AGGREGATION: for a cross-SILO (tenant) deployment with a small number of tenants, full MPC SecAgg is overkill. Two stack-fit options, both design-only: (A) central-DP-with-trusted-aggregator — your own server clips+noises per tenant and only stores the noised aggregate; simplest, but the server is trusted (acceptable inside a single GovCloud boundary, documented as the threat model). (B) TEE/confidential-computation upgrade path — tenants encrypt their pre-clip vector to an attested enclave; matches the arxiv 2410.08892 goal-state and the GovCon "verifiable" story. Scaffold the interface so (A) is the v1 and (B) is a drop-in later. The inference plane stays stateless: the Apple-Silicon llama.cpp/Ollama box only ever produces embeddings of in-tenant text over HTTP; it must NEVER see cross-tenant aggregates — keep the DP/aggregation logic entirely in the Go API tier, behind RLS, not on the inference node. 286 287DISABLED-FLAG SCAFFOLD (design, not activated): gate the whole thing behind a single server-side env flag (pattern already in the repo: MM_MFA_REQUIRED -> use MM_FEDERATED_DP_ENABLED, default false/unset). When false: zero emissions, zero ledger writes, the `/internal/platform/federated` package compiles and is unit-tested but its HTTP/cron hooks are no-ops. Define the package boundary now: an `Aggregator` interface (Clip, Noise, Emit) + an `Accountant` interface (Charge(release) (epsilonNow, error), Remaining()) + the ledger schema as a goose migration that is written but whose feature path is dead until the flag flips. Add a CI gate (mirroring the X-Org-ID backdoor CI gate) asserting that no federated emission code path is reachable when the flag is unset. Ship a model-card / DP-notice entry (you already have a model registry + /compliance page) describing the unit of privacy, the (epsilon, delta) target, and the accountant — design artifacts only. 288 289NET: tenant-level RDP/zCDP accounting in pure Go (FIPS-safe, no Python), DP-FedEmb-style clipped+noised embedding-aggregate as the only cross-tenant signal, central-DP v1 with a TEE/SecAgg upgrade seam, all dead behind MM_FEDERATED_DP_ENABLED=false with a CI reachability gate. 290 291**Gotchas:** 1. NO FIRST-PARTY GO ACCOUNTANT. google/differential-privacy has Go noise mechanisms (Laplace/Gaussian) but the RDP/PLD/Connect-the-Dots accountant is Python-only. You must write the accountant in Go yourself; do not assume a library exists. Good news: RDP composition is closed-form and small. 292 2932. SCRUBBING != DP. Stripping PII/text from the signal does NOT give a formal guarantee; embeddings leak membership and can be partially inverted. DP (clip + calibrated noise) is what bounds it (arxiv 2302.00539). Sell DP as the guarantee, scrubbing as defense-in-depth, never the reverse. 294 2953. RECORD-LEVEL DP IS THE WRONG UNIT and will look strong while being weak for multi-tenant. Use tenant/user-level: one tenant = one privacy unit, clip per-tenant. Otherwise a tenant contributing many rows blows past the claimed guarantee. 296 2974. EPSILON FOLKLORE. epsilon=1 is not magically "safe" and epsilon=8 is not automatically "bad." The defensible band from real deployments is epsilon up to ~13.69 with delta=1e-10 (Gboard). What matters: delta << 1/N, epsilon disclosed, and the accountant tight (RDP/PLD), not advanced composition. Always report the (epsilon, delta) pair AND the unit of privacy together; an epsilon with no unit is meaningless. 298 2995. SUBSAMPLING AMPLIFICATION IS A TRAP IF MISUSED. Privacy amplification by sampling only holds if tenant participation is genuinely random and unknown to the adversary each round. In a small fixed set of GovCon tenants you likely cannot claim it — account WITHOUT amplification (worst case) unless participation is provably random. Banded-MF/DP-FTRL gets correlated-noise benefits without relying on per-round amplification. 300 3016. THE SERVER IS TRUSTED IN CENTRAL DP. If your Go API clips+noises, it sees raw per-tenant vectors before noising — that is central DP and the server is in the TCB. State this explicitly in the threat model. SecAgg or a TEE is required to remove that trust; do not claim "the server never sees tenant data" under plain central DP. 302 3037. FLOATING-POINT DP ATTACKS. Naive Gaussian sampling has known leakage via float representation (Mironov 2012). Use a vetted secure-noise sampling routine (the discrete/snapped-Gaussian approach in google/differential-privacy Go) for any ACTIVATED path; for design-only this is a documented TODO, not a v1 blocker. 304 3058. DETERMINISM vs NOISE. Your AEDT engine is pure-decimal and deterministic by law (STACK_DECISIONS). DP noise is INHERENTLY random and must be quarantined OUT of the scoring/matching path. The federated signal can only ever influence a SHARED model artifact offline, never a per-candidate served score, or you break AEDT auditability/reproducibility. Keep the DP module strictly off the /match and /slates hot paths. 306 3079. STATELESS INFERENCE PLANE BOUNDARY. The Apple-Silicon node must remain stateless and tenant-blind: it embeds in-tenant text on request and forgets. Never route cross-tenant aggregates or the shared federated model through it as state — that would silently merge tenant data outside RLS. 308 30910. DELTA AND N COUPLING. Pin delta below 1/(number of contributing tenants) and re-check it whenever the tenant count grows; a delta that was fine at 30 tenants can become a disclosure risk as N shrinks per cohort. 310 311**Primary sources:** 312- [Federated Learning in Practice: Reflections and Projections (Google, arXiv 2410.08892) — privacy hierarchy, SecAgg+DP+TEE confidential computations, Gboard epsilon [0.994,13.69] delta=1e-10](https://arxiv.org/html/2410.08892) 313- [Learning to Generate Image Embeddings with User-level Differential Privacy / DP-FedEmb (Google Research, arXiv 2211.10844)](https://arxiv.org/pdf/2211.10844) 314- [google/differential-privacy — repo (Go library: Laplace/Gaussian mechanisms, bounded aggregations; accountant in /python/dp_accounting only)](https://github.com/google/differential-privacy) 315- [google/differential-privacy dp_accounting (Python) — RDP + PLD privacy accountant](https://github.com/google/differential-privacy/tree/main/python/dp_accounting) 316- [Differential privacy accounting by connecting the dots — Connect-the-Dots PLD accountant (Google Research blog, 2022)](https://blog.research.google/2022/12/differential-privacy-accounting-by.html) 317- [Subsampled Renyi Differential Privacy and Analytical Moments Accountant (Wang, Balle, Kasiviswanathan, arXiv 1808.00087)](https://arxiv.org/abs/1808.00087) 318- [(Amplified) Banded Matrix Factorization: A unified approach to private training (DP-FTRL, arXiv 2306.08153)](https://arxiv.org/abs/2306.08153) 319- [Scaling up the Banded Matrix Factorization Mechanism for Differentially Private ML (ICLR 2025, arXiv 2405.15913)](https://arxiv.org/html/2405.15913v2) 320- [Secure Stateful Aggregation: A Practical Protocol with Applications in Differentially-Private Federated Learning (arXiv 2410.11368)](https://arxiv.org/pdf/2410.11368) 321- [Analyzing Leakage of Personally Identifiable Information in Language Models — scrubbing is not sufficient, DP complementary (arXiv 2302.00539)](https://arxiv.org/pdf/2302.00539) 322