Synchronous Online Verification Gating in Semantic Caches: An Empirical Study — Part I: Core Findings and the Go/No-Go Verdict

Abstract

Semantic caches replace exact matching with vector similarity to reuse an LLM's past answers, but similarity and answer correctness are not the same quantity: static similarity thresholds have been reported in production to produce false-positive rates in the double digits. A recent line of work, vCache, learns per-embedding adaptive thresholds online in exchange for a formal error-rate bound; another, Krites, introduces an LLM judge in the similarity gray zone, but that verification is designed to run asynchronously, off the serving path, and its own evaluation always substitutes a ground-truth oracle for the real judge. This paper fills that specific gap: under a single-tier cache architecture, using a real (non-oracle) lightweight verifier, gating cache hits synchronously and online, and running a formal Pareto-frontier comparison against static-threshold and adaptive-threshold baselines on two public benchmarks (roughly 210,000 real requests combined). The headline finding is conditional, not unqualified: with an oracle verifier, synchronous gating raises hit rate by 20-28 percentage points at matched error rate, proving the mechanism has substantial theoretical headroom. Swapping in an off-the-shelf real verifier and re-measuring with honest calibration gives a best reproducible net gain of about +1.9 percentage points of hit rate on LmArena. Accordingly, this paper's Go/No-Go verdict is a weak Go: the mechanism is not falsified, but how well the verifier matches the data domain is the variable that actually determines whether it has practical value. Fine-tuning the same verifier on each dataset's own gray-zone labels confirms this diagnosis and closes most of the gap on all three datasets tested. This is Part I of a four-part series distilled from a single complete technical report (Zenodo DOI: 10.5281/zenodo.21703364) originally covering all experiments together. This paper focuses on the core empirical Go/No-Go verdict for synchronous verification gating; companion papers cover formal risk control, adversarial robustness, and integration under non-stationarity.