Janady13/missionmatch / docs/BUILD_CONTINUATION_PROTOCOL.md

1# BUILD CONTINUATION PROTOCOL: PHASES, STRESS, VISUAL PROOF, AND DRIFT 2 3Companion to docs/CLAUDE_CODE_BUILD_DIRECTIVE.md and the amendments. 4Generated 28 July 2026 against commit e9ff911, schema v80, Phases 0 and 1 5complete, Phase 2 current, Track U complete, Tasks T1 and T2 done. This 6protocol governs every remaining phase. It changes how phases END, not 7what they build. 8 9-------------------------------------------------------------------------------- 100. HOW TO APPLY 11-------------------------------------------------------------------------------- 12 131. Record "continuation protocol accepted" in BUILD_STATE with the date. 142. The Phase Completion Protocol in Section 2 becomes mandatory for 15 Phase 2 and every phase after it. No phase closes without all five 16 gates, in order, with evidence. 173. Track H (Section 1) begins immediately, in parallel, production 18 eligible under the same four guardrails as Track U: staging verified 19 first, Joseph approves each deploy explicitly and it is recorded, 20 the deploy is recorded with commit hash and evidence, and build 21 routes are re proven dark after every deploy. The directive's "Track 22 U is the only production eligible work before Gate 6" now reads 23 "Track U and Track H." 244. Allowed new things, exhaustive additions to the directive's list: 25 tools/verify/visual-sweep (Node, Puppeteer, pixelmatch), 26 tools/verify/stress (Go load profiles and drill scripts), 27 docs/evidence/ (per phase evidence bundles), docs/IMPROVEMENTS.md 28 (the betterment backlog), and the Track H backup scripts. Nothing 29 else new is licensed by this document. 305. Every rule stands: frozen engine, forced RLS, extend never build 31 beside, fail closed, staging only outside Tracks U and H, no 32 Python, no dashes in written output. 33 34-------------------------------------------------------------------------------- 351. TRACK H: HARDENING NOW, NOT AT GATE 6 36-------------------------------------------------------------------------------- 37 38Scope is exactly these six items: 39 401. BACKUPS, the largest recorded risk. Nightly pg_dump of the control 41 plane and every tenant database, compressed, encrypted, shipped off 42 machine to a second host, 14 daily and 8 weekly retained, failure of 43 the job itself alerting. Then the drill: restore last night's dumps 44 onto staging, run the migration version check and a row count 45 reconciliation, and record the restore TIME. An untested backup is a 46 hope, not a backup. The drill repeats at every phase close from now 47 on. 482. KEY HYGIENE. Confirm in the OpenAI dashboard that the exposed key is 49 revoked, record the confirmation, remove the key line from every 50 copy of the architecture document, and if any document carrying it 51 was ever committed, purge it from history. The replacement lives 52 only in the secrets store. 533. MINIMAL ALERTING. An uptime check on the public site and /healthz on 54 both nodes, plus a journal watch that alerts on any 5xx and on 55 backup job failure, delivered to Joseph by email. Dashboards can 56 wait; a phone that buzzes cannot. 574. IDENTITY SYNC. sync-tenant-identity on a schedule with a positive 58 control proving it moves a record, closing the stale tenant copy 59 risk. 605. LAST LOGIN INSTRUMENT. One real login on staging, confirm 61 last_login records, because a design partner reported a login the 62 control plane does not show, and an identity instrument that lies 63 is not an instrument. 646. OPERATOR MFA DECISION, Joseph's call queued for him: enroll TOTP 65 with printed backup codes, then lift the production stand down. The 66 protocol only keeps the decision visible; it does not decide. 67 68TRACK H GATE: a restore drill has succeeded with its time recorded, an 69induced failure proves each alert fires (positive control), the sync 70moves a planted record, and the ledger holds the key confirmation. 71 72-------------------------------------------------------------------------------- 732. THE PHASE COMPLETION PROTOCOL, FIVE GATES IN ORDER 74-------------------------------------------------------------------------------- 75 76A phase is CLOSED only when all five pass, in this order, with evidence 77filed under docs/evidence/phaseN/ and summarized in BUILD_STATE. 78 79GATE A, FUNCTIONAL. The phase's own gate from the directive and 80amendments, unchanged. Every item verified against staging, reproduced 81before fixed, recorded with the command and the observation. 82 83GATE B, STRESS. Strict means numbers. Run on staging, which carries 842,766 organizations and is the right place to hurt things. 85 86 Concurrency Every lock guarded transition in the phase raced at 8 87 and at 32 concurrent attempts. Exactly one winner, 88 counts reconcile, nothing corrupts. The existing 89 racing style, widened. 90 Load Go load tooling (vegeta or hey) drives the phase's 91 hot routes with authenticated sessions at a mixed 92 profile. Targets, measured on staging hardware: 93 sustained 50 requests per second for 10 minutes, p95 94 under 300 milliseconds, p99 under 800, ZERO 5xx. 95 Soak 30 minutes at half load. RSS stable within 10 percent 96 after warmup on both nodes, database pool gauges 97 never saturate, no goroutine growth trend. 98 Failure drill Stop one API node mid load: nginx failover carries it 99 with zero client visible 5xx. Kill the phase's worker 100 binary mid batch and rerun: completes without 101 duplicates. Saturate the rate limited routes and 102 observe 429, never 500. 103 Volume The phase's data heavy path at its documented cap, 104 memory bounded by streaming, completion time 105 recorded. 106 107 Every stress instrument gets a positive control first: a deliberate 108 overload or planted defect must FAIL the check before the passing 109 run counts. Numbers land in the evidence bundle. Regressions 110 against the prior phase's numbers are called out, not smoothed. 111 112GATE C, VISUAL. How Claude verifies with its eyes, made mechanical. 113 114 The sweep tools/verify/visual-sweep logs in through the real 115 login flow as EACH role the phase touches (seeded 116 fixture accounts per role), walks the phase's route 117 manifest, and renders at five viewports: 320, 390, 118 768, 1024, 1440. 119 Assertions Zero console errors, zero failed network requests, 120 zero horizontal overflow (document scrollWidth must 121 not exceed clientWidth), every stat tile navigates 122 when clicked, and no screen renders a raw error or an 123 empty state where fixture data should be. 124 Capture Full page screenshots into docs/evidence/phaseN/ 125 visual/ plus a generated contact sheet HTML so Joseph 126 reviews a whole phase in one scroll. 127 Regression Screens the phase did NOT touch are pixel diffed 128 against their approved baselines with pixelmatch, the 129 same technique that proved T2 left desktop unchanged. 130 A diff above threshold on an untouched screen is a 131 failure, not a curiosity. 132 Baselines New or changed screens become new baselines only after 133 Joseph approves the contact sheet, and the approval is 134 recorded. Baselines are the visual ledger. 135 Eyes The sweep does not replace looking. Claude reviews the 136 captures, lists anything odd even if assertions 137 passed, and the sweep itself has a positive control: a 138 deliberately broken fixture page must fail the run 139 before the real run counts. 140 Public Where a phase touches fg-talent.com, add the live 141 checks over HTTPS: widths 320 through 1440, curl 142 proving real HTML in the initial response, and the 143 Rich Results test on one live posting. 144 145GATE D, DRIFT. The freeze proven, every phase, no exceptions. 146 147 The scoring engine diff against e74fd2e is empty, path 148 internal/modules/matching/engine. THE ANCHOR AND THE PATH ARE BOTH 149 ASSERTED TO EXIST FIRST, because this check has now failed open twice, 150 once on a commit that ceased to exist after a history rewrite and once 151 on a path that never existed. git diff against either prints nothing 152 and exits zero, which is indistinguishable from no drift. Corrected 153 29 July 2026, finding D8: the previous anchor 9e77f65 is the 24 June 154 PROJECT baseline, not the pre build baseline, and the engine was 155 legitimately developed after it through ModelVersion v1.4.0, so a diff 156 against it is 1476 lines and can never be empty. e74fd2e is this 157 build's own first commit, is code identical to the true pre build 158 baseline 2178e72 of 21 July, and states the claim the gate actually 159 wants, that THIS BUILD changed nothing. 160 rg proves zero Python and zero dashes in written docs, each with a 161 planted positive control. 162 Migration versions match across control plane, tenant, and staging, 163 each queried directly, never summarized. 164 Build feature routes beyond the released set are re proven dark in 165 production, unauthenticated AND authenticated legs. 166 A secrets sweep of repo and docs finds nothing, positive control 167 planted and removed, values never echoed. 168 New tenant tables spot checked for organization_id, forced RLS, and 169 the leading composite index. 170 Endpoint catalogue, table index, runbooks, and the Owner's Manual 171 reflect what shipped. BUILD_STATE, the task list, and reality 172 agree, exactly one current phase named. 173 EVERY MEASURING INSTRUMENT IS CALIBRATED ONCE AT THE LARGEST INPUT THIS 174 PROTOCOL CAN DEMAND OF IT. Added as a standing rule 29 July 2026 from 175 finding D16. Before an instrument's number is quoted anywhere, it must 176 have been run once at the FULL scale the protocol specifies, not at a 177 convenient fraction of it, and that calibration run must be recorded. 178 CONVENIENCE SCALE VALIDATION PROVES NOTHING. D16: the stress tool's 179 percentile function used an insertion sort behind a comment reasoning 180 that "sorting a few thousand durations is not the bottleneck here". The 181 reasoning was sound and the premise was false. This protocol asks for a 182 THIRTY MINUTE soak, which is roughly 1.8 million samples, and O(n 183 squared) on 1.8 million is a trillion operations run twice. The only 184 soak ever executed was SIXTY SECONDS, 60,000 samples, which completes in 185 seconds. So the soak this protocol requires had never once been 186 reportable, every number was collected correctly, none of it was 187 printable, and the gap was invisible because the input was always thirty 188 times smaller than the requirement. 189 A MEASURING INSTRUMENT THAT CANNOT REPORT THE MEASUREMENT ASKED OF IT IS 190 NOT A SLOW INSTRUMENT, IT IS A MISSING ONE. Two long soaks were lost 191 before this was diagnosed, and the first loss was WRONGLY BLAMED on the 192 D14 credential rotation that happened to land at the same moment. A 193 wrong attribution is worse than no attribution, because it closes the 194 question. 195 THE CHECK: for every instrument, name the largest input the protocol can 196 demand, and state where it was run at that size. An instrument with no 197 such run is uncalibrated and its numbers are provisional. 198 A COMMENT JUSTIFYING A SIMPLIFICATION IS A CLAIM ABOUT THE INPUT, and it 199 must be checked against the input the protocol specifies rather than the 200 input the author had in mind. 201 202 NO CHANGE TO A RESOURCE LIMIT OR A RUNTIME CONSTRAINT COUNTS UNTIL A 203 RECORDED KNOWN GOOD FIXTURE HAS BEEN RERUN AND MATCHED ITS WRITTEN 204 NUMBERS. Added as a standing rule 29 July 2026 from finding D18. The rule 205 covers rlimits, memory caps, worker and concurrency counts, pool sizes, 206 and timeouts. 207 D18: the PDF library was moved into a child process, correctly, and that 208 child was given an RLIMIT_AS of 1 GiB. RLIMIT_AS bounds VIRTUAL address 209 space and the Go runtime reserves large virtual arenas at startup, so 210 every worker died instantly with "fatal error: runtime: out of memory" 211 before reading a byte. Every worker failure falls back to the weaker 212 reader, which is right for an unreadable document and WRONG for a 213 readable one, so a fix aimed at ONE pathological file silently degraded 214 extraction for ALL of them. The original PWS went from 55 chunks to 3 and 215 the run reported ok=1. 216 THIS CLASS OF DEFECT DOES NOT FAIL, IT DEGRADES, and a degraded pipeline 217 reports success. No test failed. Nothing in the output was red. The only 218 reason it did not ship is that 55 had been written down in an evidence 219 file and 3 did not match it. 220 THE COROLLARY IS THE HABIT: record the number before changing the thing 221 that produces it. A limit change with no fixture to compare against is 222 unproven no matter how green the suite looks. 223 THE CHECK: name the fixture, name its written numbers, and state where 224 the matching rerun is recorded. A limit change with none of those three 225 is provisional. 226 227 THE POSITIVE CONTROL LIBRARY IS A FILE, NOT AN INTENTION. Section 4 item 2 228 said the library only grows; it had no home, so it did not exist. It is 229 now docs/POSITIVE_CONTROL_LIBRARY.md and every entry names the 230 instrument, the lie it told, and the control that catches it. 231 232 ARCHITECTURE DOCUMENT CURRENT, MEASURED, AND SECRET FREE. Added as a 233 standing rule 28 July 2026. docs/ARCHITECTURE.txt is canonical in 234 the repository and nowhere else; any copy outside it is stale by 235 definition. Its volatile facts are RE MEASURED at every revision, 236 never hand edited: schema versions queried per database, binary 237 hashes, account and data counts, environment flags, task and track 238 statuses, the risk register, and the current state section. The 239 header takes a new date, commit, and supersedes line. A claim that 240 cannot be re measured is labeled inferred in the sentence. 241 Corrections are withdrawn by name, never silently edited. Nothing 242 aspirational enters it. SECRETS NEVER APPEAR IN IT regardless of 243 anyone's direction, INCLUDING THE OWNER'S, which is the point of 244 writing the rule down rather than relying on judgement each time. 245 It revises at exactly four moments: every phase close as this line 246 item, every production deploy, every incident closed, and every 247 recorded decision. 248 249GATE E, HUMAN. The evidence bundle goes to Joseph: contact sheet, 250stress numbers against targets, drift checklist, restore drill time, 251and the ledger extract. Joseph says go. The phase closes on his word 252and not before, recorded. 253 254-------------------------------------------------------------------------------- 2553. THE REMAINING PHASES UNDER THIS PROTOCOL 256-------------------------------------------------------------------------------- 257 258PHASE 2, CURRENT: SCOPE AND REQUIREMENTS 259 Objective Close the extraction loop on REAL documents: upload, 260 deterministic extraction, cited scope, review, drafts, 261 promotion, amendment impact. Includes the two deferred 262 Phase 1 items, intake uploads and sam-attachment-sync, 263 which unblock the moment Joseph's solicitation package 264 lands. 265 Stress 32 way promotion race on one draft. Concurrent extraction 266 of 25 documents memory bounded. A 200 page PWS parses 267 within cap. Amendment ingest during promotion corrupts 268 nothing. Scrub gate flooded with mixed payloads misroutes 269 zero private items, positive control planted. 270 Visual Roles recruiter and manager. Screens: capture list and 271 detail, upload with progress and failure states, scope 272 review with accept, edit, reject, split, merge, draft 273 builder, promotion confirmation, amendment diff view. 274 Drift External boundary: a tenant document fixture anywhere in 275 an external bound payload fails the suite. Engine diff 276 empty. Blocked on Joseph: the real RFP or combined 277 synopsis with PWS and at least one amendment. 278 279PHASE 3: PROPOSAL OPERATIONS 280 Objective Proposal workspace, compliance matrix, sections, 281 milestones, risks, staffing links, the deterministic 282 readiness engine, submission with hard blockers, outcomes 283 into bidoutcomes, plus the slate approval lifecycle with 284 withdrawal. 285 Stress 32 way approve and withdraw races reconcile counts. A 286 readiness recompute storm across 100 proposals is bit 287 stable run to run. Submission raced against a blocker 288 appearing loses to the blocker. Snapshot job killed mid 289 run reruns clean. 290 Visual Roles recruiter, manager, admin. Every workspace tab with 291 fixture data, blocker states rendered, readiness drivers 292 legible, approve and withdraw with the reason modal, audit 293 trail visible in Activity. 294 Drift Readiness engine is pure, versioned, and shares zero code 295 with the candidate engine, proven by import graph. 296 297PHASE 4: INTELLIGENCE TIER A 298 Objective oews-metro-sync, sca-wd-sync, usajobs-sync, awards task 299 order depth, O*NET bump, the eleven cited knowledge cards, 300 five closed TAG intents, evidence pack stub. 301 Stress Every sync run twice at full volume: identical counts, no 302 duplicates, manifests reconcile. Rate limit budgets 303 respected under a deliberate over ask, backoff observed. 304 Intent storm: 100 concurrent TAG calls, all parameterized, 305 zero model authored SQL by construction. 306 Visual Cards render with citations in the knowledge feed, Ask 307 answers cite them, each new intent demonstrated in the 308 console with its provenance. 309 Drift ingest_manifest rows for every source, licenses recorded, 310 CC BY 4.0 attribution present, external flag state 311 re proven. 312 313PHASE 5: THE DESK, HEAVIEST PHASE IN THE PLAN 314 Objective Everything amendments 2 through 5 assign here: recruit 315 desk completion and metrics, requisition to requirement 316 bridge, resume pipeline, Import Center, screenMATCH 317 uploads, Candidates page with fusion search, reverse 318 match, add to job order, job order self updates, and 319 publishing to fg-talent.com with same origin apply. 320 Stress A 5,000 row import commits within cap, killed and rerun 321 without twins. 32 way add to order races produce one 322 application. Seat fill raced at the last opening flips 323 filled exactly once and moves the rest to 324 filled_by_another. Publish and unpublish churned 50 times 325 leaves site, sitemap, and cache consistent. The apply 326 endpoint flooded answers 429 and the honeypot swallows 327 the bots. Fusion search under load holds the p95 target. 328 Visual Roles viewer through superuser, because this phase is 329 where role differences become visible. Candidates page, 330 pipeline board with aging, Import Center all four flows 331 including the protected class rejection message, job 332 order detail with live counts, and the public side: the 333 careers index and a detail page on fg-talent.com at all 334 five widths, apply form success and duplicate paths, Rich 335 Results pass, Patricia and Brian only, the excluded names 336 asserted absent from render and schema. 337 Drift One candidate spine: the shared components serve both 338 surfaces, proven by import graph, and the public surface 339 hygiene checks rerun in full. 340 341PHASE 5A: IDENTITY AND ACCESS 342 Objective The matrix rebuild with ownership scopes, rate field 343 sensitivity, impersonation, organizations provisioning, 344 regenerated exhaustive tests. 345 Stress The full matrix suite at the new shape. A grant ceiling 346 fuzz attempts every illegal mint at every rung. 100 347 concurrent sessions across mixed roles hold targets. 348 Impersonation expiry raced against writes. Provisioning 349 run end to end twice is idempotent on the second attempt. 350 Visual One contact sheet PER ROLE across the same route set, so 351 the differences between viewer, member, recruiter, 352 manager, admin, owner, and superuser are visible side by 353 side, plus the impersonation banner on screen, plus proof 354 by payload inspection AND by eye that rates above pay are 355 absent below manager. 356 Drift The nineteen legacy capability aliases still resolve, 357 404 not 403 holds everywhere, EEO grant remains outside 358 the ladder. 359 360PHASE 6: HARDENING AND CUTOVER 361 Objective MFA on everywhere, MM_ENV production with every gate 362 live, monitoring beyond minimal, the Gate 6 checklist, 363 and the production release of the whole build. 364 Stress The full system run: 100 concurrent authenticated users 365 mixed across all four products for 30 minutes on staging 366 at production data shape, targets held, then the failover 367 drill under that load, then the timed restore drill one 368 more time. These numbers become the baseline the live 369 system is judged against. 370 Visual The complete route manifest, every role, every viewport, 371 fresh baselines approved as the production visual ledger. 372 Drift The full Section 11 acceptance list, items 1 through 40 373 plus this protocol's additions, every line with evidence. 374 375-------------------------------------------------------------------------------- 3764. THE IMPROVEMENT LOOP, MAKING IT BETTER WITHOUT DRIFTING 377-------------------------------------------------------------------------------- 378 3791. RETRO. Closing any phase writes a retro block into BUILD_STATE: 380 what broke, what instrument lied, what took longest, what surprised. 3812. POSITIVE CONTROL LIBRARY. Every time an instrument is caught lying, 382 the control that would have caught it joins the permanent library 383 and runs at every subsequent Gate D. The library only grows. 3843. BACKLOG. Improvement ideas go to docs/IMPROVEMENTS.md with a one 385 line rationale, never straight into code. Joseph triages at each 386 Gate E. 3874. ONE PER PHASE. Each phase may carry at most ONE approved 388 improvement item alongside its scoped work. That is the entire 389 licensed channel for unscoped betterment; anything more is drift by 390 definition. 3915. TARGETS RATCHET. Stress targets may tighten between phases and may 392 never quietly loosen. Loosening requires a recorded decision. 393 394-------------------------------------------------------------------------------- 3955. EVIDENCE CONVENTION 396-------------------------------------------------------------------------------- 397 398 docs/evidence/phaseN/ 399 functional/ gate item transcripts and observations 400 stress/ load outputs, race results, drill timings 401 visual/ screenshots, contact sheet, pixel diffs 402 drift/ sweep outputs and their positive controls 403 GATE_E.md the bundle summary Joseph approves 404 405BUILD_STATE references the bundle path at every phase close. Evidence 406is append only like everything else: superseded bundles are kept, never 407deleted. 408 409-------------------------------------------------------------------------------- 4106. ACCEPTANCE ADDITIONS TO SECTION 11 411-------------------------------------------------------------------------------- 412 41341. Every phase from 2 onward closed through all five gates in order, 414 with an evidence bundle on disk and Joseph's recorded approval. 41542. Stress targets met at every phase and never loosened without a 416 recorded decision; Gate 6 numbers stand as the production baseline. 41743. Visual baselines exist for every screen, updated only through 418 approved contact sheets, and untouched screens pixel diff clean at 419 every phase close. 42044. A timed, successful restore drill exists no older than the most 421 recent phase close. 422