Measuring What Retrieval Misses: Probe-Calibrated Recall and Absence Estimation Without Assessing the Unretrieved Segment

Abstract

Deployed retrieval systems routinely need two numbers they cannot currently obtain: an estimate of the relevant material they failed to retrieve, and a calibrated statement of what a run of failed searches does - and does not - imply about absence. The measurement tools that exist each require a precondition that deployment does not meet: sample-based recall estimation requires human assessment of documents drawn from the unretrieved segment; pooling-based evaluation requires a fixed judged pool; learned truncation requires labeled training data; conformal risk control requires a labeled calibration set. This note describes a measurement protocol that replaces these preconditions with one the target field can actually satisfy: ground truth by construction. Queries generated from documents yield known-relevant probes; planted decoy documents yield a measured false-positive rate; an occupancy-class estimator with a document-level covariate turns repeated, varied searches into detection probabilities and posterior estimates of missed relevant mass with prediction intervals; a mandatory adequacy statistic, computable only because probes exist, indicates when the covariate model is misspecified - the protocol's own validity conditions are part of its output. The construction composes four measurement traditions (spike-in standards, target-decoy, site-occupancy estimation, empirical null calibration). In a pre-registered simulation the estimator attains small bias and stated interval coverage where its assumptions hold, degrades by approximately the hidden mass where they are violated, and flags the violation.