Look Before You Train: Train-Test Contamination in GlobalQA and a Training-Free Method that Reverse-Engineers Its Gold Rules

Abstract

GlobalQA is a recent benchmark for corpus-level question answering (counting, sorting, and extremum queries over a resume corpus), on which RL-based retrieval agents such as MARAG-R1 report a state-of-the-art answer-F1 of 28.60-31.22. While calibrating a simple training-free system against the training split, we discovered three facts that change how results on this benchmark should be interpreted. (1) Contamination: 1,073 of the 1,160 test questions (92.5%) appear verbatim in the training set with identical answers and identical gold document sets; a trivial lookup baseline that copies the duplicated training answer scores 92.5 answer-F1, roughly three times the published state of the art. (2) Reverse-engineerable gold rules: the gold document sets are not a pure function of document content. With a scored LLM judge calibrated on 928 labeled pairs, 30-46% of non-gold documents in counting and minimum-ID questions receive perfect relevance scores, evidencing an annotation process of the form retrieval exposure ∩ lenient LLM judgment. Because condition phrases are heavily reused across training questions (median support 86), the gold rule can be reconstructed as an explicit phrase-to-document-pool dictionary estimated from training labels. (3) A strong training-free baseline: query compilation into disjunctive normal form, the dictionary, and a lightweight LLM judge fallback yield 48.8 answer-F1 on the full test set under a twin-exclusion protocol that removes the verbatim duplicate before answering each question -- exceeding the published RL state of the art, trained with the duplicates available, by 17.6 points, with no model training and about $40 of API cost. On the 87 uncontaminated test questions our method scores 22.0. We propose reporting both twin-excluded and clean-subset numbers, and we have notified the benchmark authors to help strengthen the next revision.