AI scores for grant proposals were less reliable than expert reviews

A comparison with Swedish fellowship applications suggests these systems may help sort proposals, but they do not match reviewers as closely.

Editorial illustration — not from the study.

The study tested medium-sized, openly available large language models on a dataset of 142 post-doctoral fellowship applications submitted to the Swedish Medical Council in 1994. The researchers compared the models’ scores with the average scores given by expert reviewers and with the agreement among reviewers.

The models’ scores were moderately related to one another but only weakly related to the experts’ average scores. The authors say the systems might assist with application triage or help break ties, rather than replace expert review.

How the scores compared

The language models’ scores had a mean Spearman correlation of 0.34 with one another. Their mean correlation with the average expert scores was 0.22, and the relationship was weak but positive and mostly statistically significant. The strongest correlation with expert scores was 0.33, produced by Gemma 3 27B using proposal titles and summaries without the main texts. This was about 56% of the correlation between reviewers.

A possible review aid

Grant review can be slow and difficult to staff, especially when academics are busy or lack expertise in a proposal’s subject. The findings suggest that language models may help with initial application sorting or tie-breaking, but their scores were quantitatively weaker than those of expert reviewers in this analysis. The results therefore do not support treating model scores as a substitute for expert assessment.

Evidence and caveats

This article is a comparison of model scores with peer-review scores from a previously studied dataset of 142 Swedish Medical Council post-doctoral fellowship applications from 1994. The analysis involved a range of medium-sized open-weight language models. Its conclusions are limited by the small sample, the age of the funding call and the use of heterogeneous evaluation criteria, which the authors say undermine the robustness of the analysis. The results come from this particular dataset and do not establish how the models would perform across other funding programs or review systems.

High-mounted cameras spot pedestrians on the device at busy intersections

Researchers introduced a 13,314-image dataset for testing AI systems that view urban intersections from above. The results show that detection can run on edge devices such as the Jetson Orin Nano, although pedestrians remain harder to detect than vehicles and performance changes with scene conditions.

22 hours ago·6 views

AI & Computing

Where smaller AI models can work alongside—and compete with—larger ones

A survey examines the roles smaller AI models can play alongside large language models. It focuses on both collaboration and competition, including how smaller systems may help researchers and businesses use computing resources more efficiently.

1 day ago·48 views

AI & Computing

AI watermarks in Europe do not yet meet all four legal tests

The European Union’s AI Act calls for providers to mark and detect outputs from general-purpose AI models. A new analysis finds that no current watermarking approach for large language models meets all four of the law’s stated standards: reliability, interoperability, effectiveness and robustness.

3 days ago·19 views