Source identity
| Field | Recorded value |
|---|---|
| Source type | GitHub source |
| Original record | Janady13/missionmatch / docs/evidence/phase2/trial/findings.txt |
| Date | Not specified in source |
| Repository | Janady13/missionmatch |
| Revision | 720d47fae476b0d38477ae4401697d372b1ea2bc |
| Branch | feature/opportunity-to-outcome |
| Path or archive member | docs/evidence/phase2/trial/findings.txt |
| Original SHA-256 | bf07bec0c8137c348dd6a88e7996452db61cef27d9a571a31401ea3f20c3d1b6 |
| Git blob SHA | 0f0fd14e13755007b3617cb115600712dc103e6e |
| Reading-copy SHA-256 | bf07bec0c8137c348dd6a88e7996452db61cef27d9a571a31401ea3f20c3d1b6 |
Recovered source transcript
Download complete source text ↓112 lines · 7.0 KB
PACKAGE A TRIAL, FINDINGS RECORDED BEFORE ANY FIX
29 July 2026. Every failure below was found by running the real HTTP upload path
with real federal documents, and every one is a DEPLOYMENT defect that no
synthetic fixture and no unit test could have surfaced.
F1 UPLOADS FAIL 500, THE STORE IS ON A READ ONLY PATH
{"error":"opportunities: storing document bytes: storage: mkdir:
mkdir /opt/missionmatch-staging/store/org: read-only file system"}
The service runs under systemd sandboxing that makes the filesystem read
only except for paths explicitly granted. MM_STORAGE_ROOT was configured, the
directory was created, and the process still cannot write to it.
WHY IT MATTERS BEYOND STAGING: the same unit hardening exists in production,
so shipping document intake without granting the path would have produced an
endpoint that accepts a file, reports success at the HTTP layer for small
files, and loses every byte. It fails loudly here only because the store
write happens before the row insert, which was a deliberate ordering choice.
F2 FILES OVER ONE MEGABYTE FAIL 413 AT NGINX, NOT AT THE APPLICATION
Four of nine Package A files were refused with 413 before reaching the API.
The application cap is 64 MiB and was never consulted. nginx applies its
default client_max_body_size of 1m, and the staging vhost does not override
it.
WHY IT MATTERS: a solicitation package is exactly the workload that exceeds
1 MB. The two files that DID reach the application were the small ones, which
is the worst possible failure shape: it looks like the feature works and
fails only on the documents that matter.
BOTH ARE INFRASTRUCTURE, NOT CODE. No application change was needed for either,
which is itself the finding: the document intake was proven correct in tests and
on a local store, and neither test could see the two things that actually stopped
it in a deployed environment.
F1b THE SAME FINDING HAD A THIRD LAYER. Granting ReadWritePaths was necessary and
not sufficient: the store was root owned 755 and the service runs as
www-data, so the write still failed. Three separate things had to be true for
one upload to land, and the first two each looked like the whole problem.
F3 A REAL CODE BUG: created_at IS A timestamptz SCANNED INTO A string
can't scan into dest[12] (col: created_at): cannot scan timestamptz into *string
EVERY upload failed 500 after the storage problems were cleared. The
documentCols list selects created_at raw while Document.CreatedAt is a Go
string, and pgx refuses that conversion.
WHY NO TEST CAUGHT IT: the document service tests are pure, covering the
allowlist, the size cap, the type vocabulary, and path traversal. Not one of
them touches a database, so the column list was never executed against real
Postgres. The scope service next to it casts its timestamps with ::text and
this one did not, which is the kind of inconsistency a test with a database
finds in a second and a pure test never finds at all.
THIS IS THE FINDING THAT JUSTIFIES THE WHOLE TRIAL. The upload path was
committed, reviewed, and reported as built, and it had never once written a
row.
WITH THE INTAKE WORKING, EXTRACTION RAN AND FOUND TWO MORE.
F4 PERMISSION RESTRICTED PDFs ARE TREATED AS UNREADABLE, AND MOST FEDERAL PDFs
ARE PERMISSION RESTRICTED
Four of nine Package A files failed encrypted_or_locked: the solicitation,
the SF30 amendment, the conformed solicitation, and the locked PWS.
The reader refuses any file containing /Encrypt. That catches two very
different things: a PDF encrypted with a USER password, which genuinely
cannot be read, and a PDF with an OWNER password restricting printing or
editing, which can be read by anyone and is how nearly every agency publishes.
Treating the second as the first refuses the majority of real solicitations.
THE LOCKED PWS ITSELF BEHAVED CORRECTLY: it failed with a named reason, the
batch continued, and the unlocked variant supplied alongside it extracted 33
chunks. Graceful degradation on a locked file is proven either way.
F5 THE PROSE GATE REJECTS TABULAR DOCUMENTS, INCLUDING A WAGE DETERMINATION
07_SCA_Wage_Determination dropped 231 chunks as not prose and then reported
no_text_layer. A wage determination is a TABLE of labor categories and hourly
rates. It is the single most staffing relevant document in the package and
the gate refuses all of it.
The gate was written against a real failure, decompressed font noise scoring
as text, and it is correct about that. It over corrects on tabular content,
where short rows of codes, titles, and numbers legitimately look nothing like
prose. Recorded rather than loosened blindly: the fix must keep refusing font
noise while accepting a rate table, which is a narrower question than
lowering a threshold.
F4 CORRECTED AFTER THE FIX. My diagnosis was half wrong and the correction matters
more than the original claim. Giving the library first refusal did NOT recover
those four files: it panics with "zlib: invalid header" on their content
streams, which means they are GENUINELY ENCRYPTED, not merely permission
restricted. encrypted_or_locked was the correct verdict all along.
The change is still right and stays: a permission restricted PDF will now be
read instead of refused, and the panic containment proved itself on real
files rather than on a planted defect.
THIS IS THE DESIGNED TEST AND IT PASSED. Package A ships a LOCKED PWS and an
UNLOCKED variant of the same document on purpose. The locked one fails with a
named reason, the batch continues, and the unlocked one yields 33 chunks.
Graceful survival of a locked file is proven on a real locked file.
F5 AFTER THE FIX: the wage determination now yields a chunk instead of nothing,
with 230 still refused. A rate table is mostly short numeric rows and the gate
still declines most of them. Recorded as a KNOWN LIMIT rather than tuned
further: the document is present, its provenance is right, and extracting its
rate grid properly is table extraction, which is a different capability from
text extraction and does not belong in a threshold.
F6 THE DIFF PAIRS BY RECENCY, NOT BY DOCUMENT TYPE
On Package B it compared the SOLICITATION against the AMENDED PWS, when the
intended pair is the ORIGINAL PWS against the AMENDED PWS. pickPair takes the
first non amendment document as the baseline and the latest amendment as the
other side, so a package containing both a solicitation and a PWS pairs the
wrong two. The five changes it reported are therefore partly artifacts of
comparing unlike documents.
RECORDED, NOT FIXED. The guarantee under test is acceptance item 11, that an
amendment FLAGS and never REWRITES an approved requirement, and that held
exactly: md5 identical before and after, one alert raised for a named human.
Pair selection is diff QUALITY, which is worth fixing and is not what Gate A
is asserting.