An Independent Evaluation of AI Scientist v2 on Biomedical Research Tasks

Abstract

We conduct a pre-registered independent evaluation of Sakana AI’s AI Scientist v2 on 11 biomedical research prompts spanning epidemiology, pharmacovigilance, medical imaging, respiratory acoustics, drug repurposing, cancer omics, chem-informatics, variant pathogenicity, and adolescent health. Our scoring corpus is 24 v2-generated manuscript PDFs, 20 v2-generated research ideas, 3 student-authored human baseline papers (JEI, high-school first authors), and 3 v1 template outputs, scored on a common anchored rubric covering ideation quality, experimentation, manuscript quality, qualitative reading-level, trust calibration, and a catalog of v2 failure modes. Every reported value is drawn from a human-reviewed scoring form. We report three principal results. First, on trust calibration, v2 passes 79% of the “high-school AP Bio homework” bar but 0 of 24 of the “peer-reviewed publication” bar, while the 3 JEI baselines pass all four tiers. Second, the v2-vs-baseline gap is not in domain understanding, methods soundness, or writing fluency, these are near-ties, but in citation honesty (v2 hallucinates 1.12 citations per manuscript on average; baselines hallucinate zero), figure quality (mean 2.38/5 vs 4.00/5), and methods reproducibility (2.67/5 vs 3.67/5). Third, v2 self-flags all 20 generated ideas as novel; independent human review verifies zero as fully novel and all 20 as partially novel. Two catalogued v2 code bugs fire in every one of 24 runs at the pinned SHA. The 3 v1 template outputs exhibit a silent write-up failure: all three PDFs are template scaffolds containing only placeholder text, a stronger v1-vs-v2 finding than a quality comparison. Sample sizes for the baseline (n=3) and v1 (n=3) arms are small; point differences are directional.