Measuring What Retrieval Misses: Probe-Calibrated Recall and Absence Estimation Without Assessing the Unretrieved Segment

Abstract

Deployed retrieval systems routinely need two numbers they cannot currently obtain: an estimate of the relevant material they failed to retrieve, and a calibrated statement of what a run of failed searches does — and does not — imply about absence. The measurement tools that exist each require a precondition that deployment does not meet: sample-based recall estimation requires human assessment of documents drawn from the unretrieved segment; pooling-based evaluation requires a fixed judged pool; learned truncation requires labeled training data; conformal risk control requires a labeled calibration set. This note describes a measurement protocol that replaces these preconditions with one the target field can actually satisfy: ground truth by construction. Queries generated from documents yield known-relevant probes; planted decoy documents yield a measured false-positive rate; an occupancy-class estimator with a document-level covariate turns repeated, varied searches into detection probabilities and posterior estimates of missed relevant mass with prediction intervals; a mandatory adequacy statistic, computable only because probes exist, indicates when the covariate model is misspecified — the protocol's own validity conditions are part of its output. The construction composes four measurement traditions (spike-in standards, target–decoy, site-occupancy estimation, empirical null calibration). In a pre-registered simulation the estimator attains small bias and stated interval coverage where its assumptions hold, degrades by approximately the hidden mass where they are violated, and flags the violation. A pre-registered transport test on a public collection (v2, §4b) then measures the protocol's remaining exposed assumption — that probe-calibrated detection probabilities transfer to real queries — and reports it failed for document-side covariates, loudly rather than silently, with a finding that reframes the assumption: two independent judgment sources for "real" relevance disagreed with each other more than the probes disagreed with either. Absence calibration is therefore stated relative to a named query population and a named judgment provenance.