Automated surveillance of primary outcome reporting in clinical trials: Development and validation of a scalable screening pipeline

Abstract

Background: Detecting discrepancies between registered and published trial outcomes currently requires manual expert review, limiting surveillance to small samples. No scalable automated system exists for continuous monitoring of outcome reporting integrity across the clinical trial literature. We aimed to develop and validate an automated screening pipeline for primary outcome discrepancies and to characterise its performance at scale. Methods: We developed an automated pipeline that retrieves trial registrations from ClinicalTrials.gov, matches them to publications via PubMed and screens for primary outcome discrepancies using a term-matching algorithm. We applied the pipeline to all 46,365 completed interventional trials with posted results (2015–2025) and validated performance against manual expert review of 123 trial-publication pairs, with independent large language model re-review as a second rater. Results: The pipeline processed 46,365 trials in under 48 h, matching 27,555 (59.4%) to publications and confirming 13,595 pairs via trial identifier verification. Automated screening classified 10,231 (75.3%) as concordant and flagged 3353 (24.7%) for review. Against manual review of 123 pairs, the screen achieved 100% specificity and 16.7% sensitivity (1 of 6 discrepancies); the flag’s positive predictive value could not be estimated because the validation sample contained only one flagged pair. Independent large language model re-review showed 99.2% agreement (Cohen’s kappa = 0.91). Manual review estimated a true discrepancy rate of 4.9% (95% confidence interval = 1.6%–8.9%), predominantly minor modifications (timeframe extensions, operationalisation changes) rather than substantive outcome switching. Automated flag rates were not associated with primary outcome statistical significance (21.0% vs 20.6%; p = 0.745). Discussion: Automated screening can efficiently triage tens of thousands of trials for outcome reporting surveillance, deprioritising 75% of pairs as concordant and concentrating manual review on the flagged subset. The pipeline provides a foundation for continuous, near-real-time monitoring of outcome reporting integrity that could be integrated into journal peer review or regulatory workflows.