A continuation of Janady13/missionmatch / docs/ARCHITECTURE.txt. The complete source identity, every recorded occurrence, and linked equations are on its first page.
Source transcript · continued
Download complete source text ↓1,876 lines · 107.7 KB
accidentally.
PROVEN Staging: two real logins, NULL before, stamped after. POSITIVE
CONTROL: EXECUTE revoked, login still 200 (correct), stamp stayed
NULL, and the server SAID SO with one ERROR line naming the
consequence. Production after deploy: wall clock 16:36:07, stamp
written 16:36:08, correct to the second.
HISTORY Backfilled from auth_sessions, the authoritative record of who
actually signed in. All six accounts now carry a real value.
INCIDENT 8: THE ALERTING WAS DECORATIVE, discovered while building more of it
MEASURED /var/log/bubbles-failures/ already held markers showing the house
backup FAILED on 13, 14, 15 and 16 JUNE. Nobody was ever told.
CAUSE Two, both real. The recipient file the notifier reads had NEVER been
set, and the delivery path was the local mail command handing to a
local sendmail, whose output from a residential address Gmail drops
or files as spam. An alert that is silently discarded is worse than
no alert, because the silence is indistinguishable from health.
FIX Alerts now send through Gmail SMTP with curl, reusing the SAME app
password the fifteen form handlers on this host already
authenticate with. Nothing installed, no second credential.
PROVEN By breaking things. Stopping an API node produced a named detection
and both a PROBLEM and a RECOVERED email. Hiding the backup key made
the job fail, and the FIRST attempt proved only a marker was written,
which is the June situation exactly, so the notifier was rewired and
the second attempt emailed.
NOT DONE The 5xx detector was DELIBERATELY NOT induced in production, because
causing real 500s on a live client serving system to test a monitor
is not a reasonable trade. Its logic is proven against controlled
input and this limitation is recorded rather than glossed.
THE PATTERN ACROSS 4, 5, 6, AND 7, and the most portable lesson in this document:
A route interception that silently never fired made a pixel diff prove
nothing. A grep mangled by nested quoting made a present config line look
absent. A migration fan out that could not reach a host reported success. A
SQL statement whose placeholders had been stripped failed on every login for
weeks with its error discarded. A backup notifier wrote markers to a
directory nobody reads and called that alerting. An unauthenticated only 5xx
sweep could not see a defect behind the session gate. And one of my own
diagnostic queries returned NULL under row level security and looked like a
failure that was not one.
In every case the tool returned a clean negative and was believed.
A NEGATIVE RESULT IS ONLY EVIDENCE WHEN THE INSTRUMENT IS KNOWN TO WORK.
Every check that can pass by doing nothing needs a positive control. The
continuation protocol now REQUIRES one at every stress, visual, and drift
gate, and Section 4 item 2 makes the control library permanent and only ever
growing. That rule exists because of the eight incidents above.
WHAT ELSE SHIPPED THE SAME DAY
Track U items 1 through 6 complete: operator role, platform plane answering
404, self service profile, tenant name editing, resend invitations, the
audited superuser remap (migration 0080), and one approved production
deploy with build routes re proven dark.
Operator MFA became MM_OPERATOR_MFA_REQUIRED, default true, refused in a
production MM_ENV, stood down in production only by explicit direction.
Task T2: fg-talent.com made to fit every screen. The header needed 1365px
while the drawer only took over at 980, so EVERY viewport from 981 to 1365
was broken, including every 1280 laptop. Fixed with rules bounded to
1439.98 and narrower so the desktop cannot move, proven by pixel diff
against the previous stylesheet on 10 pages at 1440 and 1600. Zero
horizontal overflow across 340 measurements. One file changed, 0 of 35
HTML files touched.
--------------------------------------------------------------------------------
INCIDENT 9, THE GATE C INSTRUMENT WAS MEASURING ITSELF
This is filed as an incident rather than a footnote because of what it says
about every clean result this build has ever reported.
FOUND Re running Gate C on the split viewport matrix.
CAUSE Six separate defects in the sweep, each of which produced findings
about the instrument and attributed them to the application:
the shell guard called flex-shrink:0 with width:100% a fixed
minimum, so it reported "route requires 1440px" on a 1440 viewport
and would have false positived on every route forever
that guard was a brand new assertion that had never been seen to
fail, which is indistinguishable from one silently skipped
a network blip inside an unguarded page.evaluate threw and killed
the whole sweep on its first route
failed requests were logged as a bare URL, so a transport error, the
sweep's own retry aborting a navigation, and a genuine refusal all
looked identical
when the DOCUMENT itself never arrived, the sweep asserted on the
empty page and manufactured an EMPTY_SCREEN finding, which is not
transport class and so was reported as an application defect
an inconclusive first attempt returned before the screenshot, so
routes vanished from the contact sheet a human reviews
ROOT Bubbles resolves its own public hostname to its public address and
hairpins through the router, across a host with TWO active interfaces
on the same LAN. Chromium saw the routes flap and reported
ERR_NETWORK_CHANGED. The hostname is now mapped to the loopback
UNDERNEATH AN UNCHANGED URL, so Origin, SNI, and certificate are
identical and the server receives byte for byte the request it would
receive from outside.
IMPACT Finding counts across one day, same application, better instrument:
78, then 37 with QUIC pinned, then 24, then 3, then 0. NOTHING ABOUT
missionMATCH CHANGED BETWEEN THE 37 AND THE 0. That number is the
measure of how much of a gate result can be instrument rather than
system, and it is the reason every assertion here carries a control.
FIX All six, each with the reason written at the line that fixes it. A
route whose findings are ALL transport class is now loaded a second
time and a finding is kept only if it happens twice; anything that is
NOT transport class is a verdict on first sighting and nothing is
retried or dropped. Dropped findings are written to transient.json,
because a suppression nobody can see is indistinguishable from an
instrument that measures nothing.
VERIFIED Both surfaces, zero findings, with both planted defects caught first.
--------------------------------------------------------------------------------
17. CURRENT STATE, HONESTLY
--------------------------------------------------------------------------------
THE PHASE 2 GATE RESULTS, filed as one bundle at docs/evidence/phase2/ and all
measured at commit ff2df39 so they describe one state of the tree:
Gate A functional PASS 52 packages, 0 failing. The boundary suite is 20
tests, five of which exist to make the suite
honest rather than to test a feature.
GATE A STAYS OPEN on one non technical blocker.
Gate B stress PASS race elects exactly one winner at 8 and at 32,
load and soak p95 10.4ms against a 300ms target,
zero errors, every control failing first.
Gate C visual PASS both surfaces, zero findings, 64 captures.
Gate D drift PASS run at BOTH ends of the session. No drift in
anything the phase touched; three findings in the
environment around it, all production, all queued.
Gate E human PENDING the real document trial plus the owner's approval.
THE SUPPORTED VIEWPORT FLOOR, a recorded product decision of 28 July 2026 and
not a deferred bug:
authenticated surfaces 768 and up for the duration of the design partner
program. missionMATCH is a desk and capture tool
used at a desk. Below the floor is UNSUPPORTED,
which is a different statement from broken.
public surfaces 320, 390, 768, 1024, 1440, no relaxation whatsoever,
because candidates arrive on phones. The login page
fits a 320 pixel phone with zero overflow, measured.
A single relaxed number would have quietly excused the public pages too.
The phone capable app shell is docs/IMPROVEMENTS.md item 1, pre approved as
Phase 5's single improvement item, where the desk screens are being rebuilt
anyway. Until then a STANDING GUARD is asserted in the sweep: no new screen
may raise the shell's minimum width past its 732px baseline plus 24px. It
measures 320 to 706 across every authenticated route today, so a regression
is caught while it is one screen rather than a pattern.
WORKING AND VERIFIED IN PRODUCTION, re measured at this revision:
Two API nodes on identical binary 8f24414531e80a9a, failover proven.
Control plane 100.77.114.40:5433 v84, tenant 100.94.8.94:5432 v84, staging
100.94.8.94:5432 v84, each version queried in its own database rather than
taken from a summary. ALL THREE ARE LEVEL as of 29 July, the first time in
this build. Migrations 0081 through 0084 shipped to production because they
are additive and carry no data; the routes they support stay dark.
The earlier correction about staging v80 versus v81 stands as a record of
what was wrong then and is left in place rather than edited away.
Data plane routing proven with the app's own DSN: connects as
missionmatch_app, NOBYPASSRLS, zero rows with no org GUC and 11 with it.
Zero 5xx across every sweep, including an AUTHENTICATED leg, which the
earlier sweeps lacked and which is how a 500 once hid behind the session
gate. 96 requests on both legs at the last deploy.
Tool entitlement enforced at the API including the flat alias paths.
The platform plane answers 404 to every non operator, proven for superuser
and admin against a route the operator gets 200 on.
Self service profile, tenant name editing, resend invitation.
Last login now records, proven in production to the second: wall clock
16:36:07, stamp 16:36:08. It did NOT this morning; see incident 7.
CSRF bound to session. Denials survive the refusal that caused them.
6 accounts: 1 operator, 2 tenant superusers, 2 recruiters, 1 disabled probe.
6,676 opportunity records, all embedded, semantic retrieval proven live.
fg-talent.com fits 320 to 1440 with zero horizontal overflow.
Nightly encrypted backups of all three databases, off the machine that holds
the data, with a restore drilled and timed at 8 to 10 seconds.
Alerting that has been proven by breaking things: a stopped node emails, a
failing backup emails, and both recoveries email.
Identity sync hourly, proven by a planted record.
IN PROGRESS, PHASE 2 DOCUMENT PIPELINE, and one part of it is RED:
document intake, extraction, amendment diff, and attachment sync are BUILT
and proven on staging at v82.
provenance is bound FROM THE SOURCE by migration 0082 for documents and
chunks, because one table holds both a customer's private package and
published federal attachments. A caller still cannot declare it, and
source_type is immutable so it cannot be relabelled afterwards.
sam-attachment-sync pulls from the PUBLIC endpoint and spends no key budget.
THE EXTRACTOR CANNOT READ A PRODUCTION GRADE PDF. Zero of four real federal
attachments extract. A prose quality gate refuses the binary noise that
naive stream inflation produces, correctly, and the consequence is no
output at all on real files. One real solicitation carries 149 FlateDecode
streams and 20 embedded font programs, so this needs PDF object parsing,
which is a dependency decision or a build and is the owner's call. Stated
here rather than averaged into a passing summary.
NOT DONE:
The adverse impact fairness snapshot has NEVER produced output in
production. It fails every daily firing for want of one environment
variable, and nothing alerted because the unit has no OnFailure hook.
Findings D1 and D2, section 15, queued.
The control plane cannot migrate itself with its own recorded credential.
Finding D3, risk 9.
MFA is OFF everywhere; operator MFA additionally stood down in production.
MM_ENV is development, so production gates are inert.
recruitMATCH has ZERO data. Built, never used. Section 7.4.
derive.go:73 stale hardcoded dashboard copy.
Revenue, bill rates, candidate source not tracked. Time to fill has a
migration and no service. Stat cards not clickable.
Federated differential privacy scaffolded and OFF.
Metrics have no scraper and no dashboard, a deliberate ordering: a phone that
buzzes was worth more first.
PANTHEON does not learn from the teaching channel; the gateway discards it.
Phase 1 has two items deferred until client go live: intake screen 3 uploads
and cmd/sam-attachment-sync, both blocked on real documents existing.
The external OpenAI path has never executed end to end.
--------------------------------------------------------------------------------
18. RISK REGISTER
--------------------------------------------------------------------------------
1. LARGELY CLOSED 28 July 2026. What remains is named precisely rather than
left as a solved heading. All three databases now have nightly encrypted
dumps written off the machine that holds the data, with a sha256 manifest,
14 daily and 8 weekly retention, and a restore that has been DRILLED AND
TIMED rather than assumed: 8 to 10 seconds for a 16 MB decrypted dump,
schema version matched, all ten reconciled tables equal, and the critical
tables asserted present and non empty.
STILL EXPOSED, and this is the honest remainder: /mnt/storage is NOT
mounted, because the off disk drive is dead, so the backups live on the
BUBBLES ROOT DISK. That survives accidental deletion and a bad deploy, NOT
a root disk failure. Mounting a replacement drive requires no code change,
the job already prefers it. There is no off site copy, and the encryption
passphrase exists on exactly two machines in one building; losing both
loses every backup irreversibly.
THE RESTORE TIME IS A BASELINE THAT GROWS WITH THE DATA, recorded beside
the dump size at every drill so the trend is visible rather than a number
read once and trusted forever.
1b. THE DISCOVERY THAT MAKES THE ABOVE REAL: restoring requires a SUPERUSER to
create the pgvector extension FIRST. Without it the restore reports the
correct schema version while silently losing candidates and
opportunity_records, the two most important tables in the platform. It is
now enforced by the drill script and documented as step 0 of the runbook,
so it lives in code rather than in anyone's memory.
2. CLOSED 28 July 2026. The migrate-all-tenants fan out defect. Fixed, tested,
and the divergence it caused is closed, each database verified directly.
3. WITHDRAWN 28 July 2026. An earlier revision claimed
MM_TENANT_DSN_TENANT_ACME was missing. It is not. The data plane is proven
working (Section 16, incident 6b).
4. Consumer hardware, single location, no database redundancy. A power cut
takes the platform down. A sleeping laptop takes a node out.
5. Secrets live in root owned env files per host with no rotation mechanism
beyond the versioned pepper and TOTP KEK.
CORRECTED 28 July 2026. Revision 2 said an OpenAI API key sits in plaintext
at the end of this document. IT DOES NOT AND MUST NOT. The owner's standing
rule is that secrets never appear in this document regardless of anyone's
direction, including the owner's; the key was removed before this file was
first committed and git history was checked and confirmed never to have
carried it. The key itself lives in the root owned env files where every
other secret lives. Its revocation status remains the owner's to confirm
and is tracked in the ledger, not here.
6. MFA is off, and the operator seat is additionally stood down in production
at the owner's explicit direction (MM_OPERATOR_MFA_REQUIRED=false, default
true, refused by config in a production MM_ENV, announced at every boot).
For an AEDT handling CUI adjacent data with a federal buyer in view, this
is the compliance gap a first assessor will find. The owner holds this
decision and it is tracked as Track H item 6.
7. CLOSED 28 July 2026. The tenant copy of users WAS stale and incomplete,
holding 3 of 6 accounts. sync-tenant-identity now runs HOURLY from the
control plane across every active tenant in the directory, passing the
tenant DSN explicitly because the tenant is on another host, and it
RECONCILES after each run rather than trusting an exit code. Proven by a
planted record: absent from the tenant copy before, present after, removed
from both sides afterwards.
8. One design partner, one production organization, 11 candidates. Multi
tenant behaviour is proven by tests and by staging's 2,766 organizations,
not by production traffic.
CORRECTED 28 July 2026. Revision 2 said nothing in this platform has been
load tested. That is no longer true and the correction is narrow: the
DATABASE path has been load and soak tested at about a thousand reads per
second for a minute with a p95 of 10.4ms and zero errors, and the FOR
UPDATE discipline elects exactly one winner at 8 and at 32 way concurrency.
AUTHENTICATED HTTP has still never been load tested, because that needs
seeded fixture sessions per role which do not exist yet. The distinction is
kept sharp on purpose, so the database number is never quoted as an end to
end request latency.
9. NEW 28 July 2026, finding D3. The control plane's own recorded migrator
credential is refused: FATAL, password authentication failed for user
missionmatch_migrator. Three of the four DSNs in that file authenticate
cleanly, so it is the credential and not the parsing, and the role exists
with rolcanlogin true, so the stored password and the role's password
disagree.
CONSEQUENCE: a migration run from that node with the recorded DSN fails and
the operator has to find another path under time pressure. That is the
exact condition that produced the twelve version divergence in risk 2.
Resetting a production role's password is a production change, so it is
queued rather than done under the phase lock.
10. NEW 28 July 2026, findings D1 and D2. The adverse impact fairness snapshot
has never run in production, and nothing alerted. Full detail in section 15.
For an AEDT, a compliance artifact the product claims to produce and does
not is a different class of gap from a feature that is merely unbuilt: the
code is right and the running system is wrong, which is the harder failure
to notice from the outside.
--------------------------------------------------------------------------------
19. THE DESIGN DECISIONS THAT EXPLAIN THE REST
--------------------------------------------------------------------------------
1. Postgres RLS, forced, org GUC set transaction locally. Isolation belongs in
the database, not in remembered WHERE clauses. A hand written predicate is
not a substitute and is actively misleading.
2. Decimal, never float, for anything scored. Bit stability is a compliance
requirement, not a preference.
3. The scoring engine is pure and frozen, with its diff checked.
4. The model never authors SQL. A closed intent registry instead. This is why
there is no NL to SQL feature and why that is correct.
5. Fail closed everywhere: no org yields zero rows; no tool grant yields 404;
an unrecordable AEDT decision is not served; a gateway with a backend
enabled and no secret refuses to boot.
6. Tenants distribute, rows do not. One primary per tenant preserves every row
lock while still spreading load across machines.
7. Identity is centralised in the control plane, because login must resolve a
person before any tenant is known. Copies elsewhere are non authoritative.
8. Role, tool, and plane are orthogonal: authority, surface, and whether the
surface is tenant scoped at all are three different questions.
9. A refusal's STATUS CODE is a design decision. 403 confirms a thing exists.
Tool and platform surfaces answer 404.
10. Extract, draft, review, approve, promote. Nothing a machine derived reaches
the scoring engine without a named human, and the DATABASE enforces it.
11. missionMATCH publishes to the federation and never joins it.
12. Every non obvious decision is explained in a comment at the point of the
decision. Several comments are longer than the code they explain,
deliberately, because the reason is the part that gets lost.
--------------------------------------------------------------------------------
20. GLOSSARY
--------------------------------------------------------------------------------
AEDT Automated Employment Decision Tool. The regulatory category the
product falls into, driving explainability and audit obligations.
Band Green, Yellow, Red. The human facing summary of a match score.
Band cap A deterministic rule that can only LOWER a band, never raise it or
change the numeric total.
CAG Cache Augmented Generation. RouteDirect: answer from stable rules
already in context, with no retrieval.
Control plane The database holding identity and the tenant routing catalog.
CUI Controlled Unclassified Information. Cleared resumes are CUI,
which is why production self hosts inference.
Data plane A tenant's own business database.
DISS The government system where an FSO verifies a clearance. The
platform never does; it reports self reported indicators.
Egress class One of four labels that permit data to leave the machine.
Forced RLS ALTER TABLE ... FORCE ROW LEVEL SECURITY, applying the policy even
to the table owner.
GUC app.current_org_id, set transaction locally, the value every RLS
policy compares against.
KAG / OAG Knowledge and Ontology Augmented Generation: cited knowledge cards
and the O*NET, NICE, PSC, CALC ontologies.
Operator The platform plane role, above and outside the tenant ladder.
PAG Plan Augmented Generation. The bounded planner over closed intents.
RRF Reciprocal Rank Fusion, k=60, fusing vector and keyword rank in SQL.
Superuser The tenant top seat. Everything by rule WITHIN its tenant.
TAG Table Augmented Generation. The model picks a curated intent NAME;
it never writes SQL.
Track U The urgent identity and access track from amendment 6, the only
production eligible work before Gate 6.
================================================================================
END
Written from measurement. Where this document and the system disagree, the
system is right and this document is a bug.
================================================================================