PHASE 2 DOCUMENT PIPELINE, BUILT AND PROVEN 28 July 2026. Target: missionmatch_staging, now v82. Everything below was measured. WHAT WAS BUILT, closing the gap the trial rehearsal found. opportunity_documents and opportunity_document_chunks were SCHEMA WITHOUT CODE; they now have code. internal/modules/opportunities/document_service.go intake, storage, dedupe internal/modules/opportunities/document_handler.go POST/GET documents cmd/opportunity-extract chunks with citations cmd/opportunity-amendment-diff diff and impact flags cmd/sam-attachment-sync the deferred Phase 1 item db/migrations/0082_provenance_by_source.sql provenance from source ================================================================================ PROVEN GREEN ================================================================================ PROVENANCE IS BORN, NEVER DECLARED, and 0082 made it derive from the SOURCE rather than the table. Proven in one transaction on staging: A: caller declared tenant_upload on a sam_attachment, born public_cache B: caller declared public_cache on an upload, born tenant_upload UPDATE source_type -> ERROR: source_type is immutable on opportunity_documents, because provenance derives from it The third is the one that matters. Without it a row could be inserted as an upload and relabelled an attachment later, laundering a customer's document into the public class after the fact. Live totals: sam_attachment -> public_cache 4 rows, upload -> tenant_upload 7. Chunks inherit from their parent document and fail closed if the parent cannot be found. EXTRACTION SURVIVES BAD INPUT, every case named rather than collapsed to failed: malformed.pdf empty_document locked.pdf encrypted_or_locked empty.txt empty_document scanned.pdf no_text_layer ghost.pdf bytes_missing (positive control: a row promising bytes storage does not have MUST fail, and it did, rather than passing quietly) The batch continued through all five. A job that dies on the first bad file reports one problem instead of all of them. CITATION WORKS, per clause. Six chunks per synthetic PWS, each labelled: C.3.1 The contractor shall provide continuous monitoring of the enclave. C.3.2 The contractor shall staff four Cyber Systems Operations specialists C.3.3 Personnel shall hold Security Plus certification within 30 days. C.4 Period of performance is twelve months base with four option years. C.5 Travel is estimated at 15 percent. Minimum of 5 years experience. The first version produced ONE chunk per document, because the PDF reader joined every line with a space and the clause splitter is anchored to a line start. Chunking exists so a scope item can cite C.3.2 rather than "the document", so losing line breaks lost the entire point. THE AMENDMENT DIFF DETECTS ALL FIVE MATERIAL CHANGES: polygraph "ci" -> "full_scope" quantity "4" -> "6" minimum_years "5" -> "8" travel_percent "15" -> "40" certifications "Security Plus" -> "CISSP" It first found only three. QUANTITY and TRAVEL were missed because the patterns expected symbols and fixed qualifiers, while the document says "four Cyber Systems Operations specialists" and "15 percent". For a cleared staffing product the headcount is the single most important number on the page, and a diff that silently misses it is worse than no diff. ACCEPTANCE ITEM 11, THE SAFETY PROMISE, PROVEN ON REAL AMENDMENT TEXT: requirement md5 before: 7977de0d1d07653b84542a9e6bd57af0 flagged 1 requirement(s) for a named human. NONE was rewritten. requirement md5 after: 7977de0d1d07653b84542a9e6bd57af0 alerts raised: 1, kind=opportunity_amendment, status=unread Byte for byte identical. The guarantee is not that the rewrite is careful, it is that there is no rewrite. THE ATTACHMENT SYNC WORKS AGAINST THE LIVE PUBLIC ENDPOINT, no API key spent: a9d240544bd0400590d5120ae4fde52d: 3/3 files stored 863fb87be21a4cd7873d3d3f299ff9a0: 1/1 files stored 8469e11a72a5431abc33990cb3a63e56: 0/4, all four skipped not_a_file two synthetic notices: SKIPPED, status 400, gracefully DEDUPE POSITIVE CONTROL, second run: stored=0 considered=4 ingest_manifest: SAM.gov attachments, records=4, US Government work THREE THINGS ONLY THE LIVE CALL COULD FIND. The response is nested (_embedded.opportunityAttachmentList[].attachments[]), mimeType carries an EXTENSION not a media type so every real attachment was being refused, and the payload carries accessLevel, exportControlled, and explicitAccess flags that the first version ignored entirely. For a product whose users hold clearances, pulling an export controlled attachment into a tenant corpus is not a bug to fix later, so mayPull now fails closed on any unexpected value. A fourth: four resources were reported as media_type_not_allowed when they were actually external LINKS with no file at all. A misleading skip reason is worse than a blunt one, because it would have prompted someone to widen the allowlist for no reason. ================================================================================ RED. THE PIPELINE CANNOT READ A PRODUCTION GRADE PDF. ================================================================================ This is the finding of the day and it blocks the real document half of Gate A. WHAT HAPPENS. Of four genuine federal attachments: A74 Pre-Sol.docx unsupported_format (no office reader) Salient Characteristics.pdf no_text_layer RFQ_Supplemental Document.pdf no_text_layer Solicitation, FA441725Q0323.pdf no_text_layer Zero real PDFs extract. The synthetic fixtures still extract perfectly, which is precisely why this had to be tested on real documents. HOW IT GOT HERE, in three honest steps. 1. The reader handled only UNCOMPRESSED content streams, so real PDFs produced nothing. Fixed by inflating FlateDecode with the standard library. 2. With inflation on, it produced 16 chunks of BINARY NOISE that stored cleanly, carried correct provenance, and were garbage, because inflating every stream picks up font programs and image data and a scanner looking for parenthesised strings finds plenty inside a compressed font. 3. A prose quality gate now refuses those chunks. It is CORRECT, and its consequence is that real PDFs extract nothing at all. THE GATE WAS NOT WEAKENED TO PRODUCE A GREEN RESULT. Garbage chunks are worse than no chunks: a reviewer cites them, and a citation is the one thing this pipeline promises. A green Gate A built on noise would be the most expensive kind of lie this build could tell. WHY IT IS NOT A TUNING PROBLEM, measured rather than asserted: Solicitation, FA441725Q0323.pdf 149 FlateDecode streams, 20 font programs Salient Characteristics.pdf 24 FlateDecode streams, 0 font programs Telling the few content streams from the ~145 others requires parsing the PDF object graph: the xref table, object streams, /Contents references, and font encoding with ToUnicode maps for anything not WinAnsi. That is a real parser, not a regular expression. THE THREE OPTIONS, and choosing among them is Joseph's, not mine: 1. ADD A GO PDF DEPENDENCY. rsc.io/pdf is small and pure Go; pdfcpu is larger and more capable. Cheapest by far. It is a stack decision under the "don't blindly build" rule and adds a dependency to a FedRAMP bound artifact, so it needs the owner and a note in STACK_DECISIONS. 2. BUILD THE PARSER IN HOUSE. No new dependency, full control, and a substantial piece of work that is genuinely its own phase item. 3. SHELL OUT to pdftotext. Fastest to working, and it breaks the cloud portability mandate by adding a system binary to both target clouds. Listed for completeness; I would not choose it. UNTIL ONE IS CHOSEN: the pipeline is proven end to end on documents it can read, the guarantees around it (provenance, citation, no rewrite, graceful failure) are proven on both synthetic and real inputs, and the reader is the one component that cannot yet do its job on the real thing. Said plainly rather than averaged into a passing summary. ================================================================================ CLEANUP ================================================================================ Synthetic organisation, notices, documents, chunks, drafts, and alerts created for this proof are removed. The four real federal attachments and their rows are retained on STAGING only, because they are published public documents and they are the only real corpus this pipeline has; they are named here so their presence is deliberate rather than residue.