Assessing residual bias in real-world vaccine effectiveness studies: validating negative controls to avoid misleading results

Abstract

Observational studies and randomized controlled trials demonstrate that vaccines protect against infections and severe outcomes, but real-world vaccine effectiveness estimates remain vulnerable to residual biases. Negative control outcomes can help detect and mitigate such biases, but their validity is often assumed rather than systematically evaluated. In COVID-19 research, short post-vaccination infections have been proposed as negative controls because biological protection may not yet be established. We assessed residual bias in vaccine effectiveness studies and examined when negative controls provide appropriate benchmarks for bias detection and correction. We conducted a target trial emulation using electronic health records from PEDSnet, comprising eight US children’s hospitals and health institutions, from January 1 to November 16, 2022. Adolescents aged 12 to 20 years without prior documented SARS-CoV-2 infection or COVID-19 vaccination were included. Vaccinated and unvaccinated patients were matched 1:1 on propensity scores. Outcome included documented SARS-CoV-2 infection. To further evaluate residual bias, we considered other respiratory infections (influenza/RSV), which share clinical presentation and healthcare-seeking patterns with COVID-19 but are not causally affected by COVID-19 vaccination. We estimated hazard ratios for days 1–14 and day 15 onward. Empirical calibration was performed using prespecified vaccination-unrelated health events as default negative control outcomes. Each comparison group included 24,008 participants. During days 1–14, HR was 0.33 (95% CI, 0.24 to 0.47) for documented SARS-CoV-2 infections and 0.30 (95% CI, 0.14 to 0.63) for other respiratory infections. Both differed significantly from the empirical null distribution (both p < 0.001). From day 15 onward, the HR was 0.32 (95% CI, 0.27 to 0.36) for documented SARS-CoV-2 infections and 0.68 (95% CI, 0.60 to 0.77) for other respiratory infections; the latter no longer differed significantly from the empirical null distribution ( p = 0.142). Immediate post-vaccination outcomes are affected by behavioral and temporal biases that can produce spurious protective associations. These biases diminish with longer follow-up, indicating that negative control validity depends on both outcome selection and time-at-risk. Short-term documented infections may therefore be inappropriate negative controls for later follow-up. Careful evaluation of negative control validity across time-at-risk windows is essential in real-world vaccine effectiveness studies.