Source identity
| Field | Recorded value |
|---|---|
| Source type | GitHub source |
| Original record | Janady13/missionmatch / docs/evidence/phase2/deploys/production-deploy.txt |
| Date | Not specified in source |
| Repository | Janady13/missionmatch |
| Revision | 720d47fae476b0d38477ae4401697d372b1ea2bc |
| Branch | feature/opportunity-to-outcome |
| Path or archive member | docs/evidence/phase2/deploys/production-deploy.txt |
| Original SHA-256 | 40c82f420bce2dc4b7316bb30b0ac724010a856f0786e3b36d8039ed8933b518 |
| Git blob SHA | 92ec7b4f397a8b34ed3767aae75998a887f4c803 |
| Reading-copy SHA-256 | 40c82f420bce2dc4b7316bb30b0ac724010a856f0786e3b36d8039ed8933b518 |
Recovered source transcript
Download complete source text ↓75 lines · 5.4 KB
PHASE 2 PRODUCTION DEPLOY, 30 July 2026, at Joseph's direction. VERDICT: SHIPPED AND VERIFIED.
WHAT SHIPPED
API node 1, Bubbles /home/user/missionmatch/mm-api 8f24414531e80a9a -> 8ce5a2b8141a2ab1
API node 2, megamind-agi WSL /opt/missionmatch/mm-api 8f24414531e80a9a -> 8ce5a2b8141a2ab1
Frontend, Bubbles index-BE5iWWvj.js -> index-DmaEX_GX.js, plus self hosted fonts
Gateway, M4 production -> 9e1ff5da245989cd, the teaching allow list fix
Migrations NONE. All three databases were already at 85, so there was no schema risk.
ORDER, AND WHY. API FIRST, THEN FRONTEND. A new bundle in front of routes that do not exist yet is a broken
UI; a new API under an old bundle is simply unused capability. The reverse order would have put a customer in
front of a broken screen for the length of the deploy.
ROLLBACK PATH, established BEFORE the first change and still in place
node 1 /home/user/missionmatch/mm-api.rollback.8f24414531e80a9a
node 2 /opt/missionmatch/mm-api.rollback.8f24414531e80a9a
frontend /var/www/sites/missionmatch/index.html.rollback
gateway /Users/joseph/megamind-gateway.rollback
database nightly dump from 20260729 on disk, 186M, and no schema change was made.
THREE THINGS WENT WRONG AND ALL THREE WERE CAUGHT BY THE DEPLOY'S OWN CHECKS.
1. A BAD HEALTH CHECK TRIGGERED A FALSE ROLLBACK. Node 2 binds ":8094", a bind spec with no host, and my
check built http://:8094/healthz which is not a URL. curl failed, the rollback fired, and the binary was
never at fault. THE GUARD BEHAVED CORRECTLY AND THE CHECK DID NOT. Fixed by normalising a bare port to
127.0.0.1.
2. I CORRUPTED THE BINARY IN TRANSIT AND IT SEGFAULTED. The node 2 script ran the staged file through
tr -d '\r', which is correct for a text script and DESTROYS an executable: it strips every 0x0D byte out
of 21 megabytes. The service died with code=killed status=11/SEGV and no Go output at all, which is
exactly what a corrupted executable looks like. Bubbles was unaffected because the binary was scp'd
straight there.
THE FIX IS A HASH ASSERTION, not care. The deploy now compares the staged file's sha256 against the
source before the service is stopped, and REFUSES if they differ. A corrupted artifact can no longer
reach a service start, which is a stronger guarantee than remembering not to use tr.
Cost: two rollbacks and a period where production ran two different versions across its nodes.
3. MY VERIFICATION REPORTED SIX 5xx THAT DID NOT EXIST. The grep pattern " 5[0-9][0-9] " matched the BYTE
COUNT 548 in the access log rather than a status code. Actual 5xx: zero. That is the same class of error
this build has catalogued seventeen times, and this time it was in the check I wrote to confirm the
deploy was safe. Named rather than quietly corrected.
POST DEPLOY VERIFICATION, with controls
CONTROLS FIRST, routes that must still answer: / is 200, /api/auth/csrf is 200. So a non 200 below means
something about the route rather than about the host.
THE PHASE 2 ROUTES ARE LIVE, and they were 404 before this deploy:
GET /api/opportunities/tracked/{id}/scope 403, auth required
GET /api/opportunities/tracked/{id}/scope/items 405, exists, GET not the verb
GET /api/opportunities/tracked/{id}/amendments 403, auth required
POST /api/opportunities/tracked/{id}/scope/items/{i}/split 403, auth required
A 403 or a 405 proves the route EXISTS and is gated. A 404 is what all of them returned yesterday.
ONE ROUTE IS STILL DARK AND THAT IS BY DESIGN:
GET/POST /api/opportunities/{id}/documents 404, with a real uuid too
Document intake ships dark unless MM_STORAGE_ROOT names a place for the bytes, and production does not set
it. The API logs "document intake DISABLED, MM_STORAGE_ROOT is not set" at boot. The reasoning is in the
code: an endpoint with nowhere to put a document would accept an upload and lose it, which is worse than a
404. Staging sets it, which is why the pipeline was exercisable there.
ENABLING IT IS A DECISION, NOT A DEFAULT. The root disk is 82 percent full, 39G free of 221G, and the
database backups live on that same disk. Choosing where solicitation documents land is the same decision
IMPROVEMENTS item 2 already flags as needing a storage ceiling and a retention rule first.
THE TEACHING BOUNDARY IS OPEN FOR THE FIRST TIME, proven both ways against the production gateway:
a clean craft signal on the current contract v1.4.0 HTTP 202 ACCEPTED
a planted client fact HTTP 422, "payload contains a client fact and
cannot cross the teaching boundary: uuid"
Before this deploy every signal was refused whatever it named, so teaching_signal_aggregates and
curation_proposals were unreachable rather than unused.
FONTS: zero requests leave for Google. /fonts/fonts.css answers 200 and the page carries no third party font
tag, so no visitor IP reaches a third party on the render path.
ERRORS: 0 ERROR level entries in the API journal since the deploy. 0 actual 5xx.
BOTH NODES agree on 8ce5a2b8141a2ab1, so production no longer runs two versions.
WHAT THIS DEPLOY DID NOT DO
It did not change any schema. It did not enable document intake. It did not change MFA posture, which
remains as recorded. It did not touch the staging environment.