Persistence Integrity for a Continuously-Learning Agent State: Heartbeat Locks with Token-Verified Release, Concurrent-Writer Detection, and Wipe/Bloat Sanity Gates

Abstract

Agent frameworks increasingly persist mutable state that several independent loops write concurrently — a per-action hook, a background consolidation or reflection daemon, and an operator-facing UI process are a common shape. This engineering note reports on a single JSON state file ("the brain") in a persistent bio-inspired agent substrate, written by exactly three such processes, and on three defensive layers built and falsified against it, in the order they were needed. First, a cross-process advisory lock was hardened with heartbeat renewal (staleness measures the heartbeat gap, not total hold duration, so a legitimately slow holder is never reclaimed while alive) and token-verified, rename-based atomic release (an ownership token is checked before every delete; a stale-lock reclaim is a single atomic rename so exactly one of several racing contenders wins). This was proven with a real cross-process test harness — a Node process running one implementation against a tsx process running a parallel implementation, both against the same lock path — covering mutual exclusion, heartbeat survival in both process directions, exactly-once reclaim of a crashed holder, and refusal of a foreign-token release, plus three 500-round live passes (1,500 rounds) over a copy of the live state with zero lost updates, zero stale reclaims, zero cross-releases, and zero acquire timeouts across all rounds; wall-clock blackout rounds (>5 s) were zero only on a quiet host — 5 and 3 under heavy external machine load, attributed by an instrumented probe to the scheduler rather than the lock. Second, a concurrent-writer detector was added: a state signature mismatch at write time is logged and stamped into the state as a bounded ring of recent detection timestamps, feeding two alert rules. This layer exists because full write-path serialization was deliberately deferred pending observation of whether the underlying race is real. Third, precipitated by a real incident (a 441 MB state bloat that could not be parsed, silently replaced by a fresh "newborn" state, and persisted over a month-plus of accumulated learning before being caught and restored from a daily archive), a write-time sanity gate now refuses three classes of catastrophic write before they reach disk: bloat past a fixed ceiling, a newborn state landing over a substantial one, and an implausible backward jump in the state's monotonic counter. Two of these three checks are mirrored across the three writer processes' primary write paths; the third currently lives only in the highest-frequency writer, and four further dashboard request handlers that rewrite the full state carried no gate at all at the time of writing (a gap found in pre-submission review and closed on 2026-08-25) — asymmetries this note reports rather than papers over. As of the draft freeze (2026-07-07) the detector shows the underlying race is real under sustained load: a 60-second burst rule has fired live and the freeze day itself recorded at least 547 distinct detections (an earlier mid-morning reading of 41 undercounted the day by roughly 13×, in the conservative direction). A fourth layer — full in-process serialization of the highest-risk write path — has been designed and preregistered but is explicitly not implemented at the time of writing. The reusable contribution is methodological: a minimal, falsified recipe for persistence integrity under multi-writer agent state — heartbeat-backed locking with verified release, cheap detection before expensive serialization, and independent write-time sanity gates as a last line — validated with real cross-process harnesses rather than in-process mocks.