Across five leading conferences, papers sharing both code and data rose from 11% in 2014 to 64% in 2024.
Researchers examined seven reproducibility-related features in all papers published at five leading AI conferences over a decade. They found broad improvements in documentation practices, including a rise in papers that shared both their code and data from 11% in 2014 to 64% in 2024.
Using reproducibility rates from an earlier study, the researchers estimated that reproducibility rose from 28% to 64% over the same period. The timing suggests the changes began before formal reproducibility checklists were introduced and may reflect a wider move toward open science rather than a direct response to those checklists.
More papers share code and data
The researchers quality-assured seven variables related to reproducibility and analysed 56,800 publications from five leading AI conferences. Between 2014 and 2024, documentation practices improved, and the share of papers providing both code and data increased nearly sixfold, from 11% to 64%.
Based on empirical reproducibility rates reported in a previous study, the researchers estimated that reproducibility increased from 28% in 2014 to 64% in 2024. This estimate was inferred from documentation practices; the researchers did not directly test whether each paper's results could be reproduced. The improvements in documentation began before the introduction of reproducibility checklists.
Evidence and caveats
This journal article reports an analysis of 56,800 papers from five leading AI conferences, using seven quality-assured measures of reproducibility-related documentation. The study covers published papers at those conferences, so its findings may not describe all AI research. Most importantly, the reported rise in reproducibility is an estimate based on documentation practices and results from a prior study, not the outcome of direct reproducibility tests on the papers analysed.