Source identity
| Field | Recorded value |
|---|---|
| Source type | GitHub source |
| Original record | Janady13/missionmatch / cmd/opportunity-extract/limits_linux.go |
| Date | Not specified in source |
| Repository | Janady13/missionmatch |
| Revision | 720d47fae476b0d38477ae4401697d372b1ea2bc |
| Branch | feature/opportunity-to-outcome |
| Path or archive member | cmd/opportunity-extract/limits_linux.go |
| Original SHA-256 | 17256e0a0d62c2abc0ced2608b20bf5ebf87d88812d8de88001f617ef6f4095e |
| Git blob SHA | 0f7d721dfe33da059530d7a576d9ec0188c56153 |
| Reading-copy SHA-256 | 17256e0a0d62c2abc0ced2608b20bf5ebf87d88812d8de88001f617ef6f4095e |
Recovered source transcript
Download complete source text ↓43 lines · 2.3 KB
//go:build linux
package main
import (
"os/exec"
"syscall"
)
// THE HARD BOUND IS THE ISOLATION, NOT AN RLIMIT, and that correction cost a silent regression to learn.
//
// The first version set RLIMIT_AS to 1 GiB on the worker, reasoning that a local cap makes a runaway fail on
// the process that caused it rather than inviting the kernel to choose a victim. The reasoning still holds. The
// implementation did not: RLIMIT_AS bounds VIRTUAL address space, and the Go runtime reserves large virtual
// arenas at startup, so every worker died immediately with "fatal error: runtime: out of memory" before reading
// a single byte.
//
// THE CONSEQUENCE WAS WORSE THAN THE BUG I WAS FIXING. Every worker failure falls back to the hand rolled
// reader, which is the correct behaviour for an unreadable document and the WRONG outcome for a readable one.
// So a fix aimed at one pathological file silently degraded extraction for ALL files: the original PWS went
// from 55 chunks to 3 and reported success while doing it. That is the exact failure shape this build keeps
// finding, a green result over a quietly worse outcome, and I introduced it.
//
// WHAT ACTUALLY DELIVERS THE PROPERTY D12 NEEDED: the library runs in a CHILD. A runaway kills the child, the
// parent learns the library could not read the document, names it, and the batch continues. No rlimit is
// required for that and none is set.
//
// GOMEMLIMIT IS SET AS A SOFT BOUND, and it is described as soft rather than sold as a cap. It makes the
// collector work harder as the heap approaches the value and it does NOT refuse a large allocation, so it
// reduces the chance of a balloon without pretending to prevent one.
const workerSoftMemLimit = "GOMEMLIMIT=512MiB"
func applyWorkerLimits(cmd *exec.Cmd) {
// Pdeathsig ties the child's life to the parent's, so an orphaned worker cannot outlive a killed batch and
// keep allocating. This is the one kernel level guarantee worth taking here.
cmd.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL}
cmd.Env = append(cmd.Environ(), workerSoftMemLimit)
}
// selfLimit deliberately does NOTHING. It exists so the call site in main reads the same on every platform and
// so the reason is recorded here rather than as an absence somebody has to reconstruct.
func selfLimit() {}