Janady13/missionmatch / docs/evidence/phase2/findings/finding-4-secret-sweep-out-of-repo.txt

1FINDING FOUR, CLOSED BY EXECUTION 2The secret sweep was blind outside the repository. 3Run 29 July 2026 on megamind-agi WSL and Bubbles from the M4 working copy. 4 5THE FINDING, RESTATED HONESTLY 6 7 tools/verify/secret-sweep/sweep.sh scanned the repository and reported "clean". 8 That sentence was true and it read as something much broader than it was. 9 10 Two of the five credential consumers broken by the D5 rotation lived in 11 /usr/local/sbin, which is not in the repository. One of them, 12 mm-restore-drill.sh, held a migrator password in PLAINTEXT for weeks. The 13 sweep could not have found it at any point, and it never said so. A clean 14 result meant clean in the repository and was read as clean. 15 16WHAT WAS BUILT 17 18 outofrepo.scope 15 enumerated scopes, HOST tab KIND tab GLOB. Every line 19 traces to a section of docs/CREDENTIAL_CONSUMERS.md, to a 20 systemd unit carrying an EnvironmentFile, or to an ops 21 script. The scope is ENUMERATED, not discovered by walking 22 the disk, so the inventory and the sweep can be diffed 23 against each other and a consumer in one but not the other 24 is a defect in whichever is behind. 25 26 remote-scan.sh The remote half. Staged onto a host and run there. 27 28 sweep.sh --out-of-repo runs the enumerated scopes. 29 --control --out-of-repo controls every scope it claims. 30 31THE ASSERTION IS REFERENCES, NOT ABSENCE 32 33 ENV FILES ARE THE STORE. /etc/missionmatch/*.env and backup.dsn are SUPPOSED 34 to hold values. They are checked for MODE ONLY, never content. Flagging them 35 would train the reader to ignore the sweep, and a sweep people ignore is worse 36 than no sweep because it also provides cover. 37 38 SCRIPTS AND UNITS MUST REFERENCE. A script names a variable or reads an env 39 file. It never embeds a value. That is exactly what mm-restore-drill.sh 40 violated. It is the assertion here. 41 42NO VALUE IS EVER PRINTED 43 44 The remote half reports KIND, PATH, LINE, PATTERN NAME. Never the matched 45 text. grep runs with -n only, never in a mode that echoes the line. The 46 control mode counts findings and does not echo them even though its planted 47 value is fake. 48 49 This is not a hypothetical concern. This build disclosed a live credential 50 into a log this week through flag usage output, finding D14. A sweep that 51 echoes what it finds is a disclosure with good intentions. 52 53WHAT IT FOUND ON ITS FIRST WORKING RUN 54 55 megamind:/usr/local/sbin/mm-identity-sync.sh line 16 dsn_with_password 56 megamind:/usr/local/sbin/mm-identity-sync.sh line 17 dsn_with_password 57 58 Two complete DSNs, passwords included, one per cluster, in a script a systemd 59 timer runs every hour as root. 60 61 docs/CREDENTIAL_CONSUMERS.md ALREADY DESCRIBED THESE TWO LINES AS A TRAP and 62 left them standing. The inventory knew. Nothing could see it, so nothing acted 63 on it, and that gap between knowing and seeing is the whole finding. 64 65 Verified before touching anything, values never printed: 66 the embedded tenant DSN and MM_TENANT_MIGRATOR_DSN_TENANT_ACME IDENTICAL 67 by salted hash 68 the embedded control DSN and MM_MIGRATOR_DATABASE_URL same user, SAME 69 PASSWORD by salted hash, differing only 127.0.0.1 against 100.77.114.40, 70 which is the same cluster reached two ways 71 both env DSNs connect from inside WSL 72 the hourly timer was healthy, Result=success, last run 2 minutes prior 73 74 So the repair was a substitution of equals, not a credential change. 75 76THE REPAIR 77 78 mm-identity-sync.sh now sources /etc/missionmatch/db.env and resolves 79 MM_MIGRATOR_DATABASE_URL and MM_TENANT_MIGRATOR_DSN_TENANT_ACME, failing by 80 NAME if either is absent. A wrong credential and an absent one look identical 81 from the far side of a socket, and that ambiguity is what made the D5 rotation 82 take five separate discoveries instead of one. 83 84 IT ALSO LEFT argv. The old script passed both DSNs as command line flags to 85 psql and to sync-tenant-identity. argv is world readable through ps, so any 86 account on that host could read both migrator passwords out of the process 87 table for the length of a run. The Go binary already falls back to the same 88 two environment variables when its flags are empty, and psql reads the libpq 89 variables, so this cost nothing. 90 91 MEASURED BOTH WAYS, counts only, no values: 92 peak argv DSN exposures during a run of the REPAIRED script 0 93 peak argv DSN exposures during a run of the OLD script 1 94 95 The second number is the positive control. Without it the first proves only 96 that the sampler ran. 97 98 Verified after install: syntax clean, 750 root:root, and a real run 99 reconciled 7 control plane users against 7 in the tenant copy. 100 101THE POSITIVE CONTROLS, ONE PER SCOPE, BOTH DIRECTIONS 102 103 A planted credential must be FOUND, and then must stop being reported once 104 removed. A check that always fires is as useless as one that never does, and 105 only the second direction distinguishes them. 106 107 control OK repo untracked planted credential detected, 1 108 control OK repo untracked, cleared clean once the plant was removed 109 control OK repo tracked planted credential detected, 1 110 control OK repo tracked, restored hash matches 111 control OK repo tracked, cleared clean once the plant was removed 112 control OK bubbles script planted DSN detected, 1 113 control OK bubbles unit inline unit secret detected, 2 114 control OK bubbles mode, loose world readable file flagged, 1 115 control OK bubbles mode, correct 0600 root owned correctly NOT flagged 116 control OK megamind script planted DSN detected, 1 117 control OK megamind unit inline unit secret detected, 2 118 control OK megamind mode, loose world readable file flagged, 1 119 control OK megamind mode, correct 0600 root owned correctly NOT flagged 120 121 positive control: 13 passed, 0 failed 122 123 TRACKED AND UNTRACKED, INSIDE AND OUTSIDE, as the order required. The tracked 124 control MODIFIES an already tracked file and never touches the index, because 125 staging a credential shaped string even briefly risks a crash leaving it 126 staged for the next commit. Its content is saved and restored and the restore 127 is VERIFIED BY HASH rather than assumed. 128 129 The mode control uses a file of its own at /etc/missionmatch/zz-sweep-control.env 130 so no real credential file's permissions are ever loosened to test the 131 instrument. Remote plants are removed by a TRAP rather than a trailing line, 132 because a control that plants a credential shaped file on a server and then 133 dies has created the problem it was testing for. 134 135 Cleanup verified after the control run: 3 of 3 plant paths absent on each 136 host, 0 dirty control paths in git. 137 138TWO DEFECTS IN THE INSTRUMENT ITSELF, FOUND AND FIXED DURING THIS WORK 139 140 1. ssh CONSUMED THE SCOPE FILE. The scope loop read from the file on stdin and 141 ssh inherited it, so ssh drank the remaining 12 lines. Exactly ONE scope 142 ran and the sweep reported clean. Fixed with ssh -n. 143 144 The guard was ALSO wrong: it refused only when ZERO scopes answered, so one 145 scope out of thirteen satisfied it. It now counts the expected scopes out of 146 the file and refuses unless every one returns a summary. "At least one 147 answered" is not a check. 148 149 2. NESTED printf %q DID NOT SURVIVE THE WSL BRIDGE, so all five megamind 150 scopes failed to reach the host. The sweep correctly refused to report 151 clean for them, which is the guard working rather than a second defect. A 152 single unnested invocation survives. 153 154THE COUNTED RUN 155 156 secret sweep: working tree 157 secret sweep: out of repo scope 158 out of repo: 15 of 15 enumerated scopes scanned, all answered 159 secret sweep: clean 160 161WHAT THIS STILL DOES NOT CLAIM 162 163 The scope is 15 enumerated targets on two hosts. It does not cover VALKYRIE, 164 which is a backup target holding encrypted dumps and no scripts, and it does 165 not cover M4, which runs the databases and holds no ops scripts of its own. 166 Adding a host means adding scope lines, and the enumeration is deliberately 167 the thing you have to edit, so a new host cannot arrive silently unswept. 168 169 It reads files. It cannot see a credential that exists only in a running 170 process's memory, in a shell history, or in a systemd drop in nobody listed. 171 The argv measurement above was a separate deliberate check, not something the 172 file sweep does. 173 174REPRODUCE 175 176 MM_MEGAMIND_PW=... bash tools/verify/secret-sweep/sweep.sh --control --out-of-repo 177 MM_MEGAMIND_PW=... bash tools/verify/secret-sweep/sweep.sh --out-of-repo 178 179 The control comes FIRST. A clean sweep whose control has not run this session 180 is an unverified instrument reporting a negative. 181