Valid Inference with Synthetic Data via Task Exchangeability: replication deposit

Abstract

Replication deposit for the paper Valid Inference with Synthetic Data via Task Exchangeability (Lezhi Tan, Tijana Zrnic). Synthetic data is cheap to produce but can be biased, and treating it as real makes inference confidently wrong. Task exchangeability corrects for this: find historical tasks where both real and synthetic data exist, measure how far the synthetic answer fell from the real one there, and correct the current synthetic-only interval by that learned amount. The deposit runs the method and its baselines end to end on all five experiments in the paper: simulated exchangeable Bernoulli tasks, ANES feeling thermometers, Pew American Trends Panel presidential approval, Arena model win rates, and Arena Bradley-Terry scores. experiments.ipynb explains each experiment and regenerates every figure and table from the stored results in seconds; reproduce.sh --check rebuilds everything from the raw data and fails on any numerical difference from the deposited copy. Licensing. Code (Alg/, experiments.ipynb, reproduce.sh) is MIT, see LICENSE-CODE.txt. Data is CC BY 4.0 except where an upstream source imposes its own terms, see LICENSE-DATA.txt. The single licence field on this record refers to the data; the code is MIT. Arena data. Models under pre-release testing at collection time are pseudonymized as model-NN; the mapping is not distributed.