Source identity
| Field | Recorded value |
|---|---|
| Source type | GitHub source |
| Original record | Janady13/missionmatch / docs/evidence/trackH/items3-4-alerting-and-sync.txt |
| Date | Not specified in source |
| Repository | Janady13/missionmatch |
| Revision | 720d47fae476b0d38477ae4401697d372b1ea2bc |
| Branch | feature/opportunity-to-outcome |
| Path or archive member | docs/evidence/trackH/items3-4-alerting-and-sync.txt |
| Original SHA-256 | 0ee6877e8c7046ca3b1ca6df5a3d706d57387f5cca935583070db819110fc138 |
| Git blob SHA | 80e102e0c2cca049363909754a77998dca13e6b4 |
| Reading-copy SHA-256 | 0ee6877e8c7046ca3b1ca6df5a3d706d57387f5cca935583070db819110fc138 |
Recovered source transcript
Download complete source text ↓79 lines · 4.3 KB
TRACK H ITEMS 3 AND 4. Evidence, 28 July 2026.
ITEM 3, MINIMAL ALERTING
------------------------
THE FINDING THAT JUSTIFIES THE WHOLE ITEM: /var/log/bubbles-failures/ already
held markers showing bubbles-backup FAILED on 13, 14, 15 and 16 June. Nobody was
ever told. Two reasons, both now fixed: /etc/bubbles-notify.recipient had NEVER
been set, and the delivery path was the local `mail` command, which hands off to
a local sendmail whose output from a residential address Gmail drops or files as
spam. The alerting was decorative.
BUILT, reusing what already works rather than adding a stack:
/usr/local/sbin/mm-alert.sh sends through Gmail SMTP with curl, using the SAME
app password the fifteen form handlers on this host already authenticate
with. Nothing installed, no second credential: if that password rotates, one
file changes and every consumer follows. Credentials in /etc/mm-alert.env,
root owned 0600.
/usr/local/sbin/mm-watch.sh checks three things and only three, because a
monitor that cries wolf trains people to ignore the one that matters: the
public site answers, BOTH API nodes answer healthz individually so a single
node failure is not masked by the pool, and no 5xx has appeared in the API
journal since the last run. It alerts on a TRANSITION, once when a problem
starts and once when it clears, rather than every five minutes.
mm-watch.timer every 5 minutes. The house failure notifier now routes through
mm-alert.sh first and keeps the old local mail attempt as a fallback.
POSITIVE CONTROLS, each by ACTUALLY BREAKING THE THING:
1. API NODE DOWN. systemctl stop mm-api.
detected and NAMED: "API NODE 127.0.0.1:8094: healthz returned 000"
email SENT 16:41:20 "[missionMATCH] PROBLEM detected"
node restarted, email SENT 16:41:25 "[missionMATCH] RECOVERED"
2. BACKUP JOB FAILURE. The encryption key was hidden so the job would fail.
first attempt proved only a marker file was written, which is the June
situation exactly, so the notifier was rewired to the working path
second attempt after the rewire: email SENT 16:42:49
"[bubbles] mm-db-backup FAILED"
key restored, job healthy, 9 artifacts in the set
3. 5xx DETECTION. DELIBERATELY NOT INDUCED IN PRODUCTION, and this limitation
is recorded rather than glossed: causing real 500s on a live client serving
system to test a monitor is not a reasonable trade. The detection logic was
proven against controlled input instead:
a 500 line matches (1), a 200 line does not (0)
a 503 line matches (1), a 404 line does not (0)
The real journal path runs on every execution and currently reports 0.
ITEM 4, IDENTITY SYNC ON A SCHEDULE
-----------------------------------
THE DRIFT IT CLOSES, measured before the first run: the control plane held 6
users and the tenant copy held 3. Missing from the tenant copy entirely:
batwood@igotozarks.com
trutle973@hotmail.com
trackh.probe@missionmatch.test
The tenant copy is non authoritative and exists so business tables can foreign
key to a user, so nothing is known to depend on it being right, and nothing
enforced that either. That is the worst of both.
BUILT: /usr/local/sbin/mm-identity-sync.sh on megamind-agi, where the control
plane lives. It iterates EVERY active tenant from tenant_directory, so adding a
client needs no change, and passes the tenant DSN EXPLICITLY because the tenant
is on another host. That last point is the lesson from the migration fan out
defect: assuming a tenant shares the control plane's host is what let two
databases drift twelve migrations apart while every run reported success.
It RECONCILES after each sync rather than trusting the exit code, because a tool
that exits zero is not the same as a copy that matches.
FIRST RUN: seeded 6 users, reconciled 6 = 6.
POSITIVE CONTROL, a planted record:
planted sync.control@missionmatch.test on the CONTROL PLANE
present in the tenant copy BEFORE the sync: 0
ran the sync: seeded 7 users, reconciled 7 = 7
present in the tenant copy AFTER the sync: 1
planted record then deleted from BOTH sides, verified 0 and 0
SCHEDULED hourly with a randomized delay, because the drift appears whenever an
invite is accepted or a role changes, and a daily cadence would leave a new user
invisible to tenant foreign keys for most of a working day.