AI-generated tests struggle with real-world Python functions
A new benchmark found that language models performed much worse on complex, carefully screened functions than on simpler test-generation tasks.
Editorial illustration — not from the study.
ULT was built from permissively licensed Python code using a multi-stage process intended to select self-contained functions with high cyclomatic complexity, a measure of how many independent paths code can take. The researchers also created PLT, a version with leaked tests, to compare the effects of memorization and reasoning.
Across 12 language models, tests generated for ULT averaged 41.32% accuracy, 45.10% statement coverage, 30.22% branch coverage and a 40.21% mutation score. These results were lower than the corresponding averages on TestEval and PLT, suggesting that ULT presents a more difficult evaluation of test generation.
How the tests performed
The researchers selected 3,909 function-level tasks for ULT and designed the benchmark to address two problems in earlier evaluations: possible contamination from tests already available to models and overly simple code. ULT functions had cyclomatic complexity ranging from 10.0 to 82.0, with a mean of 14.87; TestEval functions had a mean of 12.35 and were concentrated in a narrower range.
For the 12 language models tested, ULT results averaged 41.32% for accuracy, 45.10% for statement coverage, 30.22% for branch coverage and 40.21% for mutation score. On TestEval, the corresponding results were 91.79%, 92.18%, 82.04% and 49.69%; on PLT, they were 47.07%, 55.13%, 40.07% and 50.80%.
ULT also showed a stronger relationship between a model's code-generation ability and its test-generation performance. The correlation coefficient for Pass@1 was 0.79 on ULT, compared with 0.56 on TestEval and 0.52 on PLT. The researchers report that this indicates ULT more effectively measures how well models generalize to new tasks.
Why harder benchmarks matter
Results from simple or contaminated benchmarks can make automated test generation look more capable than it is on unfamiliar, complex code. By using real-world Python functions and withholding the ground-truth tests, ULT is intended to provide a harder evaluation while reducing the chance that those tests enter future model-training data.
The benchmark gives researchers a common way to compare new language models with the 12 models studied here without releasing the hidden tests. That makes it possible to assess progress while preserving the benchmark's resistance to test leakage.
Evidence and caveats
The evidence comes from a benchmark study of 12 language models across ULT, PLT and existing benchmarks. ULT contains function-level tasks drawn from The Stack v2 and selected through filtering for complexity, self-containment and, for ULT, the absence of pre-existing test cases. The reported results are averages across the models and use measures including accuracy, statement coverage, branch coverage and mutation score.
The benchmark focuses on real-world Python functions, so the findings do not by themselves establish how language models perform on other programming languages, larger software components or complete applications. The ground-truth tests are not publicly released, specifically to limit future leakage; researchers instead provide evaluation results for the tested models. The study therefore enables comparisons with its results but does not provide the hidden tests for independent inspection.
// Source
ACM Transactions on Software Engineering and Methodology · 2026 · DOI: 10.1145/3805043
A study of 26 people using ChatGPT for a complex web development task identified nine types of failure, including wrong or incomplete answers, too much mental effort and lost context. Participants who received unhelpful responses were 11 times more likely to abandon ChatGPT, while additional prompts were linked to a lower chance of giving up.
A study of five language models found that they can improve several measures of code quality and pass many unit tests after refactoring software. Developers still performed better on complex changes involving code structure, dependencies and encapsulation.
A study of three language models and six datasets found that retrieval-augmented systems do not have one best design. Question answering generally worked best with five to 10 retrieved documents, while code generation required more case-by-case choices.