Janady13/missionmatch / docs/evidence/phase2/findings/finding-3-external-normalize-path.txt

1FINDING THREE, EXECUTED AND BLOCKED AT THE FAR END 2The external normalize path had never executed. 3Run 29 July 2026 at commit 52fcbb9, staging cluster missionmatch_staging on M4. 4 5STATUS: NOT CLOSED. Everything on this side of the boundary is proven. The 6external service refused all three calls with insufficient_quota, so the one 7thing this finding is ultimately about, prose returned by an external model 8passing through the verifier, has no evidence and cannot be manufactured. It 9needs one action from Joseph. It is stated at the bottom. 10 11WHAT WAS ORDERED AND WHAT HAPPENED, ITEM BY ITEM 12 13 1. RE PROVE THE SCRUB GATE POSITIVE CONTROLS. DONE, 22 of 22 green. 14 Including TestSuitePositiveControl and TestBindingSiteDetectorPositiveControl, 15 which are the two that prove the suite is measuring at all. Full list in the 16 run log below. 17 18 2. ENABLE THE OPENAI BACKEND FOR ONE BOUNDED RUN. DONE. 19 The key resolved from /Users/joseph/.megamind-secrets.env and nowhere else, 20 per the recorded owner decision of 28 July. No copy was made, no second file 21 was created, and the value was never printed. 22 23 The backend was enabled on a SEPARATE, TEMPORARY gateway process rather than 24 by flipping a flag on either running gateway. Reason: the staging gateway 25 reads .megamind-staging-secrets.env, which per the recorded decision does not 26 hold the key and must not, and flipping production would have widened the 27 enabled window to every consumer for the length of the run. A temporary 28 process on port 8097 bound the change to this run and to nothing else. 29 30 BOOT GUARD CONTROLLED BOTH WAYS BEFORE THE RUN, because a guard nobody has 31 seen fire is a guard nobody has tested: 32 enabled with an EMPTY MODEL refused to boot, by name 33 enabled with the KEY ABSENT refused to boot, by name 34 enabled with both present booted, logging openai_enabled=true and 35 generation backend=openai 36 37 3. EXACTLY THREE PUBLIC SAM NOTICE TEXTS THROUGH REQUIREMENT_NORMALIZE. DONE. 38 Three, not four, not five. The eligible pool held six and holds three now. 39 40 All three notices carry provenance public_cache, SELECTED from the database 41 rather than asserted by the program, and the gate permitted all three on that 42 basis. 43 44 4. CAPTURE THE AUDIT ROWS WITH BACKEND, PAYLOAD CLASS, BYTE COUNT, AND CONTENT 45 HASH. DONE, and it required building the thing that writes them. See 46 FINDING D19 below, which is the real discovery of this work. 47 48 entity_type content_hash backend payload_class bytes served_model 49 external_inference a179923bf11d megamind-gateway REQUIREMENT_NORMALIZE 237 50 external_inference_result a179923bf11d ollama:llama3.1:8b 51 external_inference c51e24dfc65a megamind-gateway REQUIREMENT_NORMALIZE 117 52 external_inference_result c51e24dfc65a ollama:llama3.1:8b 53 external_inference f3c46e811ea5 megamind-gateway REQUIREMENT_NORMALIZE 114 54 external_inference_result f3c46e811ea5 ollama:llama3.1:8b 55 56 3 egress rows, 3 outcome rows, all org scoped, 0 egress rows without a 57 matching outcome row, and 0 rows carrying payload content, checked by 58 joining every audit row against every notice title in the cache. 59 60 5. PROVE THE VERIFIER GATES THE RETURNED PROSE AND THE TYPED JSON CARRIES 61 CITATIONS. HALF DONE, and the half that is missing is the point of the 62 finding. 63 64 PROVEN, and backend independent: 65 TestMalformedPayloadFailsClosed prose that is not JSON is refused 66 TestUncitedModelOutputIsRejected an item with no quote is refused 67 TestFabricatedCitationIsRejected a quote NOT PRESENT in the source is 68 refused, which is the one that 69 matters, because a fabricated 70 citation reads exactly like a real one 71 TestInvalidScopeTypeIsRejected an out of vocabulary type is refused 72 TestEmptySourceRejectsEverything no source means nothing survives 73 TestGroundedItemIsAcceptedAndHashed a grounded item is accepted and hashed 74 75 PROVEN AGAINST LIVE OUTPUT: all three written items are extraction_method 76 model_normalize, review_status extracted, and every one carries a citation 77 hash. 0 uncited extracted items exist in the database, which the schema CHECK 78 independently forbids. 79 80 NOT PROVEN: that the verifier gates prose written by an EXTERNAL model. The 81 prose it gated came from the local runtime. The mechanism is identical and the 82 validator cannot tell where bytes came from, so the risk this leaves is small, 83 and it is not zero and should not be written up as though it were. 84 85 6. SET THE FLAG BACK TO FALSE. DONE, and verified rather than assumed. 86 the temporary process stopped, and its binary deleted 87 port 8097 closed, 0 listeners 88 gateway-backend.sh 0 mentions of MEGAMIND_OPENAI, unchanged 89 the secrets file 0 mentions of MEGAMIND_OPENAI_ENABLED 90 both persistent gateways openai NOT enabled, verified per process 91 8092 and the 8090 proxy healthz 200, still serving 92 93 7. RECORD THE SPEND. ZERO. Not "small", zero. 94 All three requests were refused with HTTP 429 insufficient_quota, which is a 95 refusal before service, so nothing was billable. The 129 completion tokens in 96 the outcome rows were produced by the LOCAL runtime and cost nothing. 97 98WHY IT IS BLOCKED 99 100 The gateway logged, three times: 101 generate backend "openai" failed (backend status 429 Too Many Requests ... 102 "You exceeded your current quota, please check your plan and billing details" 103 "type": "insufficient_quota"), falling back to ollama 104 105 Probed directly at the boundary, key never printed: 106 key present yes, length 164, prefix sk-proj 107 HTTP 429, code insufficient_quota 108 109 THE KEY IS VALID. An invalid or revoked key answers 401. This answered 429 with 110 insufficient_quota, which is an authenticated request against an account with no 111 available credit. So the credential works and the account cannot serve. 112 113 WHAT THIS MEANS FOR THE BOUNDARY, stated carefully. The request left this 114 machine, carrying the composed public payload, and was refused at the far end. 115 The gate permitted it, the audit row was written BEFORE it left, and the outcome 116 row recorded what actually served it. Every mechanism on our side executed for 117 real against a live external endpoint for the first time. What did not happen is 118 a completion coming back. 119 120 I did not retry, switch models, or try a second account. A quota refusal is an 121 account state, not a transient error, and the recorded decision on this key is 122 that its state is Joseph's to confirm. 123 124THE SILENT FALLBACK, AND WHY IT WAS NOT SILENT 125 126 The gateway falls back to the local runtime when the selected backend fails. 127 That is correct for availability and it is invisible in an audit trail that only 128 records intent. Had this run recorded only the egress row it would read, forever, 129 as three successful external calls. The pipeline would have reported accepted=3 130 rejected=0, three cited scope items would have been written, and every number 131 would have looked exactly like a successful external run. 132 133 RecordExternalOutcome is why it did not. It was written hours earlier in this 134 same session for exactly this reason, and the first run it ever saw was the run 135 it caught. The served_model column says ollama:llama3.1:8b three times. 136 137 This is the same shape as D18, where 55 chunks became 3 while the status still 138 read ok, and it was caught only because 55 had been written down. An instrument 139 that reports success while measuring something else is the recurring failure of 140 this build, and it is now caught twice by the same discipline: write down what 141 actually served, not what you asked for. 142 143FINDING D19, FOUND WHILE DOING THIS: THE AUDIT ACTION WAS DECLARED, NEVER WRITTEN 144 145 ai.ActionExternalInference has existed since 27 July. AllowExternal carried a 146 comment stating that binding the gate and the audit in one call means a caller 147 cannot send without both. THE BINDING WAS NEVER REAL. AllowExternal returned a 148 map and trusted its caller to record it, and the only caller wrote it to a log 149 line and persisted nothing. 150 151 So the single class of call that physically leaves this machine was the only 152 class with no audit trail, for two days, while five unit tests passed. They 153 passed because they assert the SHAPE OF A MAP. Not one of them asked whether a 154 row existed. 155 156 Fixed at commit 52fcbb9. AllowExternalAudited gates and writes as one call and 157 FAILS CLOSED: an external call whose audit row cannot be written does not happen, 158 the same posture the auth module takes for security events. The row is written 159 BEFORE the payload leaves, which is the only ordering an auditor can trust, since 160 a row written afterwards is missing exactly when the process dies mid call. 161 162 Four new tests, MUTATION CONTROLLED: removing the write fails exactly the two 163 that assert it, and restoring it turns them green. Also asserted, because a 164 promise is worth checking: no insert argument carries the payload content, and a 165 refused payload writes no row at all. 166 167ONE OBSERVATION, RECORDED NOT ACTIONED 168 169 Two of the three MM_MEGAMIND_URL settings point at port 8090, which is 170 /Users/joseph/pantheon-mlx/gateway-auth.mjs, a fifteen line node reverse proxy 171 belonging to PANTHEON. It injects the gateway bearer and forwards to 127.0.0.1:8092, 172 which IS megamind-gateway, so the boundary is intact and no payload reaches a 173 foreign service. Worth knowing anyway: a component outside this repository sits 174 in missionMATCH's inference path and makes a loopback only gateway reachable 175 across the tailnet with an injected credential. missionMATCH's own guarantees, 176 the scrub gate and the provenance check, run before the call and are unaffected. 177 178WHAT CLOSING THIS NEEDS, and it is one thing 179 180 Credit on the OpenAI account behind the key in /Users/joseph/.megamind-secrets.env. 181 Nothing else. When it has credit, this rerun is three calls, a few cents, and 182 about ninety seconds: 183 184 on M4: 185 nohup env MEGAMIND_SECRETS_FILE=/Users/joseph/.megamind-secrets.env \ 186 MEGAMIND_OPENAI_ENABLED=true MEGAMIND_OPENAI_MODEL=gpt-4o-mini \ 187 MEGAMIND_GENERATE_BACKEND=openai MEGAMIND_ADDR=100.94.8.94:8097 \ 188 ./megamind-gateway-oaiprobe > /tmp/oaiprobe.log 2>&1 & 189 on Bubbles: 190 set -a; . /etc/missionmatch/staging.env; set +a 191 MM_SECRETS_FILE=/etc/missionmatch/api.env MM_MEGAMIND_URL=http://100.94.8.94:8097 \ 192 /tmp/opportunity-scope-normalize -limit 3 193 194 Three notices are still eligible, so the rerun needs no seeding. 195 196 ACCEPTANCE, decided in advance so it cannot be softened afterwards: three 197 outcome rows reading openai:gpt-4o-mini. If served_model says ollama again, the 198 external lane did not execute and this finding stays open no matter what else 199 the run reports. 200 201 ALSO NOTED, from the recorded decision of 28 July: this key was pasted into chat 202 and sat in a plaintext Desktop file, so it should be treated as compromised 203 regardless of dashboard state. Adding credit to it is a decision about a key 204 already recorded as suspect. Rotating first would be the safer order. 205