Janady13/missionmatch / docs/evidence/phase2/functional/document-pipeline-proof.txt

1PHASE 2 DOCUMENT PIPELINE, BUILT AND PROVEN 228 July 2026. Target: missionmatch_staging, now v82. Everything below was measured. 3 4WHAT WAS BUILT, closing the gap the trial rehearsal found. opportunity_documents 5and opportunity_document_chunks were SCHEMA WITHOUT CODE; they now have code. 6 7 internal/modules/opportunities/document_service.go intake, storage, dedupe 8 internal/modules/opportunities/document_handler.go POST/GET documents 9 cmd/opportunity-extract chunks with citations 10 cmd/opportunity-amendment-diff diff and impact flags 11 cmd/sam-attachment-sync the deferred Phase 1 item 12 db/migrations/0082_provenance_by_source.sql provenance from source 13 14================================================================================ 15PROVEN GREEN 16================================================================================ 17 18PROVENANCE IS BORN, NEVER DECLARED, and 0082 made it derive from the SOURCE 19rather than the table. Proven in one transaction on staging: 20 A: caller declared tenant_upload on a sam_attachment, born public_cache 21 B: caller declared public_cache on an upload, born tenant_upload 22 UPDATE source_type -> ERROR: source_type is immutable on 23 opportunity_documents, because provenance derives from it 24 The third is the one that matters. Without it a row could be inserted as an 25 upload and relabelled an attachment later, laundering a customer's document 26 into the public class after the fact. 27 28 Live totals: sam_attachment -> public_cache 4 rows, upload -> tenant_upload 7. 29 Chunks inherit from their parent document and fail closed if the parent cannot 30 be found. 31 32EXTRACTION SURVIVES BAD INPUT, every case named rather than collapsed to failed: 33 malformed.pdf empty_document 34 locked.pdf encrypted_or_locked 35 empty.txt empty_document 36 scanned.pdf no_text_layer 37 ghost.pdf bytes_missing (positive control: a row promising bytes 38 storage does not have MUST fail, and it 39 did, rather than passing quietly) 40 The batch continued through all five. A job that dies on the first bad file 41 reports one problem instead of all of them. 42 43CITATION WORKS, per clause. Six chunks per synthetic PWS, each labelled: 44 C.3.1 The contractor shall provide continuous monitoring of the enclave. 45 C.3.2 The contractor shall staff four Cyber Systems Operations specialists 46 C.3.3 Personnel shall hold Security Plus certification within 30 days. 47 C.4 Period of performance is twelve months base with four option years. 48 C.5 Travel is estimated at 15 percent. Minimum of 5 years experience. 49 The first version produced ONE chunk per document, because the PDF reader 50 joined every line with a space and the clause splitter is anchored to a line 51 start. Chunking exists so a scope item can cite C.3.2 rather than "the 52 document", so losing line breaks lost the entire point. 53 54THE AMENDMENT DIFF DETECTS ALL FIVE MATERIAL CHANGES: 55 polygraph "ci" -> "full_scope" 56 quantity "4" -> "6" 57 minimum_years "5" -> "8" 58 travel_percent "15" -> "40" 59 certifications "Security Plus" -> "CISSP" 60 It first found only three. QUANTITY and TRAVEL were missed because the patterns 61 expected symbols and fixed qualifiers, while the document says "four Cyber 62 Systems Operations specialists" and "15 percent". For a cleared staffing 63 product the headcount is the single most important number on the page, and a 64 diff that silently misses it is worse than no diff. 65 66ACCEPTANCE ITEM 11, THE SAFETY PROMISE, PROVEN ON REAL AMENDMENT TEXT: 67 requirement md5 before: 7977de0d1d07653b84542a9e6bd57af0 68 flagged 1 requirement(s) for a named human. NONE was rewritten. 69 requirement md5 after: 7977de0d1d07653b84542a9e6bd57af0 70 alerts raised: 1, kind=opportunity_amendment, status=unread 71 Byte for byte identical. The guarantee is not that the rewrite is careful, it 72 is that there is no rewrite. 73 74THE ATTACHMENT SYNC WORKS AGAINST THE LIVE PUBLIC ENDPOINT, no API key spent: 75 a9d240544bd0400590d5120ae4fde52d: 3/3 files stored 76 863fb87be21a4cd7873d3d3f299ff9a0: 1/1 files stored 77 8469e11a72a5431abc33990cb3a63e56: 0/4, all four skipped not_a_file 78 two synthetic notices: SKIPPED, status 400, gracefully 79 DEDUPE POSITIVE CONTROL, second run: stored=0 considered=4 80 ingest_manifest: SAM.gov attachments, records=4, US Government work 81 THREE THINGS ONLY THE LIVE CALL COULD FIND. The response is nested 82 (_embedded.opportunityAttachmentList[].attachments[]), mimeType carries an 83 EXTENSION not a media type so every real attachment was being refused, and the 84 payload carries accessLevel, exportControlled, and explicitAccess flags that 85 the first version ignored entirely. For a product whose users hold clearances, 86 pulling an export controlled attachment into a tenant corpus is not a bug to 87 fix later, so mayPull now fails closed on any unexpected value. 88 A fourth: four resources were reported as media_type_not_allowed when they were 89 actually external LINKS with no file at all. A misleading skip reason is worse 90 than a blunt one, because it would have prompted someone to widen the allowlist 91 for no reason. 92 93================================================================================ 94RED. THE PIPELINE CANNOT READ A PRODUCTION GRADE PDF. 95================================================================================ 96This is the finding of the day and it blocks the real document half of Gate A. 97 98WHAT HAPPENS. Of four genuine federal attachments: 99 A74 Pre-Sol.docx unsupported_format (no office reader) 100 Salient Characteristics.pdf no_text_layer 101 RFQ_Supplemental Document.pdf no_text_layer 102 Solicitation, FA441725Q0323.pdf no_text_layer 103 Zero real PDFs extract. The synthetic fixtures still extract perfectly, which 104 is precisely why this had to be tested on real documents. 105 106HOW IT GOT HERE, in three honest steps. 107 1. The reader handled only UNCOMPRESSED content streams, so real PDFs produced 108 nothing. Fixed by inflating FlateDecode with the standard library. 109 2. With inflation on, it produced 16 chunks of BINARY NOISE that stored 110 cleanly, carried correct provenance, and were garbage, because inflating 111 every stream picks up font programs and image data and a scanner looking for 112 parenthesised strings finds plenty inside a compressed font. 113 3. A prose quality gate now refuses those chunks. It is CORRECT, and its 114 consequence is that real PDFs extract nothing at all. 115 116 THE GATE WAS NOT WEAKENED TO PRODUCE A GREEN RESULT. Garbage chunks are worse 117 than no chunks: a reviewer cites them, and a citation is the one thing this 118 pipeline promises. A green Gate A built on noise would be the most expensive 119 kind of lie this build could tell. 120 121WHY IT IS NOT A TUNING PROBLEM, measured rather than asserted: 122 Solicitation, FA441725Q0323.pdf 149 FlateDecode streams, 20 font programs 123 Salient Characteristics.pdf 24 FlateDecode streams, 0 font programs 124 Telling the few content streams from the ~145 others requires parsing the PDF 125 object graph: the xref table, object streams, /Contents references, and font 126 encoding with ToUnicode maps for anything not WinAnsi. That is a real parser, 127 not a regular expression. 128 129THE THREE OPTIONS, and choosing among them is Joseph's, not mine: 130 1. ADD A GO PDF DEPENDENCY. rsc.io/pdf is small and pure Go; pdfcpu is larger 131 and more capable. Cheapest by far. It is a stack decision under the "don't 132 blindly build" rule and adds a dependency to a FedRAMP bound artifact, so it 133 needs the owner and a note in STACK_DECISIONS. 134 2. BUILD THE PARSER IN HOUSE. No new dependency, full control, and a 135 substantial piece of work that is genuinely its own phase item. 136 3. SHELL OUT to pdftotext. Fastest to working, and it breaks the cloud 137 portability mandate by adding a system binary to both target clouds. Listed 138 for completeness; I would not choose it. 139 140UNTIL ONE IS CHOSEN: the pipeline is proven end to end on documents it can read, 141the guarantees around it (provenance, citation, no rewrite, graceful failure) are 142proven on both synthetic and real inputs, and the reader is the one component 143that cannot yet do its job on the real thing. Said plainly rather than averaged 144into a passing summary. 145 146================================================================================ 147CLEANUP 148================================================================================ 149Synthetic organisation, notices, documents, chunks, drafts, and alerts created 150for this proof are removed. The four real federal attachments and their rows are 151retained on STAGING only, because they are published public documents and they 152are the only real corpus this pipeline has; they are named here so their presence 153is deliberate rather than residue. 154