Artificial Intelligence-Assisted Systematic Literature Reviews in Metabolic Phenotyping: A Methodological Framework and Validation Study

Abstract

Abstract Systematic literature reviews are widely acknowledged to be at the top of most evidence hierarchies but are very time-consuming and labour-intensive. The advent of artificial intelligence (AI) offers possibilities to accelerate the review process. We propose a systematic literature review protocol that incorporates AI and evaluate the accuracy of AI against a human-constructed gold standard. Human researchers performed a systematic literature review on the PubMed metabolomic literature investigating Long COVID. OpenAI model o3-mini was then used via the application programming interface to carry out the three key stages of a systematic literature review: 1) screening articles for inclusion criteria, 2) evaluating articles' risk of bias (RoB) using a customised metabolomics tool based on well-established quality criteria, and 3) extracting data from articles. The performance of AI was evaluated against a manually created reference by the assessment of accuracy, negative predictive value, precision, sensitivity, specificity, F1 score, and Cohen's kappa. Out of the 320 identified articles, 33 met all inclusion criteria. AI performed screening, RoB assessment, and data extraction with mean accuracies of 92.7% ± 4.9%, 91.9% ± 5.9%, and 92.0% ± 9.2%, respectively, in ~ 10 h and at a cost of ~ 15.35 USD, while human reviewers took ~ 116 h, equivalent to ~ 2000 USD at minimum wages, to complete these steps in duplicate. OpenAI model o3-mini can independently complete the three main stages of systematic literature reviews in the metabolomics domain, thus considerably reducing the time to complete such reviews, at minimal cost. Nonetheless, human reviewers should always verify the AI-generated output to ensure robustness.