Source identity
| Field | Recorded value |
|---|---|
| Source type | GitHub source |
| Original record | Janady13/missionmatch / ops/m4/README.md |
| Date | Not specified in source |
| Repository | Janady13/missionmatch |
| Revision | 720d47fae476b0d38477ae4401697d372b1ea2bc |
| Branch | feature/opportunity-to-outcome |
| Path or archive member | ops/m4/README.md |
| Original SHA-256 | a8ad0874593243fca161c4cd732b868e8a2bb2f9a7113909e6b4fabb810d997e |
| Git blob SHA | 976e8cb7cdb2ab2d52a39b576f7acf34e6a4001f |
| Reading-copy SHA-256 | a8ad0874593243fca161c4cd732b868e8a2bb2f9a7113909e6b4fabb810d997e |
Every recorded occurrence
These 3 records contain identical original source bytes. Their paths and revisions remain part of the archive.
| Original record | Branch or archive | Revision / identifier | Original location |
|---|---|---|---|
| Janady13/missionmatch / ops/m4/README.md | feature/opportunity-to-outcome | 720d47fae476b0d38477ae4401697d372b1ea2bc | Open original |
| Janady13/missionmatch / ops/m4/README.md | master | 1e9249840a4302ec82e123b9c9cb6f75badea3e4 | Open original |
| Janady13/missionmatch / ops/m4/README.md | feature/aioe-upgrade | 2178e7298d0e5d9af83afa12402fd587f0ba6ed8 | Open original |
Recovered source transcript
Download complete source text ↓91 lines · 5.3 KB
# M4 service durability and recovery
M4 (the Mac at `joseph@100.94.8.94`) singly hosts the services the live platform depends on:
1. **PostgreSQL 16** (the missionMATCH backing store), PGDATA on the external MEGAMIND volume at
`/Volumes/MEGAMIND/missionmatch-pg/data`.
2. **Ollama** (`~/ollama/ollama serve` on `127.0.0.1:11434`), the local model runtime for embeddings and
generation. A reboot can leave the server up but its model runner stuck with zero weights loaded, returning
an empty generation, so the recovery script probes a REAL generation and restarts Ollama when it is stuck,
not just when the port is down.
3. **The MEGAMIND inference gateway** on `0.0.0.0:8090` (`~/megamind-gateway`), which proxies Ollama for
embeddings and generation. The live `mm-api` on Bubbles reaches both over Tailscale.
Neither survives a reboot on its own: macOS TCC (privacy) blocks a launchd job from reading the external volume,
and there is no root LaunchDaemon. This directory makes recovery a one command fix and adds a best effort
auto start.
## The one command manual fix (works today, over SSH)
```
ssh joseph@100.94.8.94 'bash ~/missionmatch-recover.sh'
```
`missionmatch-recover.sh` is idempotent: it starts Postgres and the gateway only when they are down, waits for
each to accept connections, then verifies both (a `SELECT 1` and the gateway `/healthz`). Run it any time; it is
a fast no op when everything is healthy. Run over SSH it has the filesystem access it needs, so it always works.
## The auto start (LaunchAgent)
`com.missionmatch.recover.plist` installs to `~/Library/LaunchAgents/`. It runs the recovery script when the
user session loads (login or boot into the session) and again every five minutes as a lightweight watchdog, so a
service that dies is brought back within five minutes. Install or reload it with:
```
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.missionmatch.recover.plist 2>/dev/null
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.missionmatch.recover.plist
launchctl kickstart -k gui/$(id -u)/com.missionmatch.recover
tail -f ~/missionmatch-recover.launchd.log
```
## Two one time GUI steps for FULL reboot durability
The LaunchAgent runs, but two things must be granted once on the M4 desktop for it to start Postgres from the
external volume after an unattended reboot:
1. **Full Disk Access.** System Settings, Privacy and Security, Full Disk Access. Add
`/opt/homebrew/opt/postgresql@16/bin/postgres` (use Cmd Shift G to type the path), and if Postgres still will
not start from the agent, also add `/bin/bash`. This lets the launchd job read the external PGDATA volume that
TCC otherwise blocks. Until this is granted, use the SSH one command fix above (the SSH context is not
blocked).
2. **Automatic login.** System Settings, Users and Groups, Automatically log in as joseph. A LaunchAgent only
runs once the user session starts, so on a headless reboot the session must come up on its own for the agent
to fire.
For true root level durability instead of the two steps above, a LaunchDaemon (runs at boot before login) would
work, but it needs administrator rights on M4.
## Generation backend selector
The gateway can proxy generation to three runtimes, flipped with one command (embeddings always stay on
Ollama's bge model so the stored vectors and the boot gate keep matching; only generation is selectable):
```
gateway-backend.sh ollama the local Ollama binary (default, no Python, Metal accelerated, most durable)
gateway-backend.sh mlx an external MLX server on 8081 (Apple Silicon, fastest generation on an M chip)
gateway-backend.sh megamind the MEGAMIND reasoning brain on 9999 (answers from its own learned corpus)
```
The running `mm-api` reads the gateway per request, so a flip needs no `mm-api` redeploy, and if the selected
backend fails at request time the gateway falls back to Ollama. Set the backend explicitly with
`MEGAMIND_GENERATE_BACKEND`; the recovery script starts the gateway on Ollama by default after a reboot.
Trade-offs on this 16GB M4:
- **Ollama** serves generation, faithfulness, and (via bge) embeddings from one model family, so it is the
safest default and is what the recovery script heals. Warm generation is about 1.3 seconds.
- **MLX** (a 4 bit 3B model such as Qwen2.5-3B) is the fastest generator, about 0.5 seconds warm, and the flip
helper brings its server up on 8081. But faithfulness still runs on the Ollama model, so on 16GB two model
families are resident at once. On a machine with more memory, routing faithfulness through MLX as well makes
it the clean choice. Note: a stuck MLX server (a bad model on 8081) is slow, so keep it on a known good 4 bit
model.
- **MEGAMIND** answers from its own learned corpus rather than the injected evidence, so it declines domain
questions it was not taught. It is wired for when its corpus covers the domain; the faithfulness gate and the
deterministic fallback protect against an ungrounded answer shipping.
## Second device (planned)
When the second device arrives the single point of failure goes away. The intended topology: a Postgres
streaming replica (hot standby) on the second device fed from M4, and a second gateway there, so a read path and
the model plane survive M4 going down. The recovery script is written to extend cleanly to a second node. Design
this when the device is in hand.