Pull: Lazy Materialization of Working Memory for Stateful LLM Conversations

Abstract

As LLM conversations grow to hundreds of turns, full-context injection incurs O(N²) cumulative token costs, while lossy summarization or hard truncation irreversibly discards historical state. We propose Pull, a session router that maintains an addressable metadata directory via a local, deterministic Purifier (zero LLM calls, millisecond-level latency). At query time, the LLM lazily materializes only the turns it needs; unmaterialized turns remain accessible but collapsed. Unlike irreversible compression, Pull's materialization is reversible—subsequent queries can expand any collapsed turn. On LoCoEval (128 conversations, 12,780 turns), Pull reduces per-query context tokens (Phase 2) by 75.1% on single-hop tasks with equivalent quality (Δ = −0.002, n.s.) and by 72.0% on multi-hop tasks with no quality loss (Δ = +0.017). A controlled routing benchmark (7,831 queries × 10 methods) shows that entity lifecycle tracking is empirically a prerequisite for distance-independent routing. On BEAM 1M (14 conversations, 263 questions), Pull improves F1 by +55.2% over a truncation baseline.