Faithful, Query-Aware Compression of LLM Agent Tool Output

Abstract

Tool outputs—web pages, JSON API responses, logs, CSV exports—are the largest andmost-ignored token sink in LLM agent loops. A single verbose tool result can dwarf the agent’sinstructions, and the standard mitigation, blind truncation, silently discards the fact the agentneeds. We present tooltrim, a drop-in, provider-agnostic library that compresses tool out-puts before they re-enter context using a content-type router and query-aware extraction, whilekeeping the full output retrievable on demand via a content-addressed store. We introduce theTool-Output Faithfulness Benchmark (tofb), 66 cases that plant a checkable fact inside real-istic, bloated tool output across five content types, with single-fact, multi-fact, distractor, andaggregation variants. On tofb, query-aware compression retains 100% of full-context accu-racy while removing 94–99% of tokens, whereas truncation collapses to 2–14% (p<0.001 atevery budget by a paired McNemar test). We report three findings: (i) compression can im-prove accuracy even on frontier models—full context depresses Claude Sonnet 5 and Opus 4.8to 73%, and compressing to 256 tokens raises them to 92% and 100% (p≤0.001)—by remov-ing distracting tool-output noise; (ii) compression acts as rate-limit admission control—undera hosted provider’s per-request token cap, only 42% of raw tool outputs are admissible versus100% of compressed ones; and (iii) against strong query-aware RAG top-k, tooltrim ties onanswer accuracy (p≥0.25 on four frontier models) but its output stays machine-parseable—thegold fact is recoverable from a real parse 57% of the time versus 39% for RAG, which shredsstructured output into fragments an agent cannot re-parse. We then close the loop this parityleaves open: on an end-to-end agent-task suite where success is decided by real parsing andarithmetic over the compressed output (no LLM judge), tooltrim reaches 100% task successversus 20% for RAG top-kat a 256-token budget (∆=+80pp, paired McNemar p=0.013)—thestructural advantage that is invisible to answer-recall converts into a significant task-successgap once a broken json.loads halts the agent loop. Because those wins are scored by realparsing rather than a live model, we complement them with a live, pre-registered study onthe τ-bench retail benchmark (gpt-4o-mini as agent and user simulator, 40 tasks ×5 tri-als). Its methodological core is a placebo-controlled difference-in-differences: the 19 tasks onwhich tooltrim never fires form a byte-identical control that measures the benchmark’s ownrun-to-run stochasticity, which the estimator nets out. We find no detectable task-successharm from compression in a real multi-step loop (DiD = −4.0pp, 95% cluster-bootstrap CI[−23.6,+14.6]—a CI-bounded no-harm result, not a formal equivalence certificate), and theplacebo control reproducibly exposes a large intrinsic noise floor—two trials of the unchangedsystem disagree on 25.7% of task pairs—which we argue makes uncontrolled single-trial agenticA/B comparisons uninterpretable and is a reusable instrument for evaluating any such interven-tion. We release tooltrim (PyPI, MIT), the tofb dataset (HuggingFace), and a reproduciblebaseline harness comparing tooltrim against truncation, RAG top-k, and LLMLingua-2.