Tests on 30 open-source Java projects found language models effective at repetitive cleanup, while developers were better at changes involving architecture and dependencies.
Researchers evaluated StarCoder2, GPT-4o Mini, GPT-4o, LLaMA 3 and DeepSeek-v3 on refactoring tasks across 30 open-source Java projects. They assessed whether the revised code passed unit tests, reduced code smells—patterns associated with maintainability problems—and improved other static code-quality measures.
The models were generally strongest at systematic and repetitive changes, such as renaming or extracting methods. Developers were better at context-sensitive work, including changes to modularization, encapsulation and relationships between classes. Prompt design also affected results, with one-shot and chain-of-thought prompts improving some outcomes.
Where AI helps most
The production-grade models GPT-4o and DeepSeek-v3 achieved unit-test pass rates above 90% at Pass@5 for multi-file refactorings. LLaMA 3 produced the largest overall reduction in code smells, with a median reduction of 15.1%, while DeepSeek-v3 and GPT-4o made the greatest improvements in cohesion, coupling and complexity. StarCoder2 was particularly effective at systematic changes and modularity improvements.
Across the study’s comparison of StarCoder2 with developers, the model reduced more implementation-level code smells and performed well on issues such as long statements, magic numbers, empty catch clauses and long identifiers. Developers did better on complex, context-dependent tasks such as broken modularization, deficient encapsulation and changes requiring dependency analysis.
Prompting changed performance. For StarCoder2, one-shot prompting raised the unit-test pass rate by 6.15% over zero-shot prompting and increased code-smell reduction by 3.52%. Generating five candidate refactorings per input raised the unit-test pass rate by 28.8% compared with generating one.
Why human review remains important
The findings suggest that language models can assist with a substantial range of routine code-cleanup tasks, particularly when prompts are tailored and several candidate changes are generated. They do not remove the need for human review: changes that depend on a broad understanding of a program’s architecture, data dependencies or intended design remained areas where developers performed better.
That points to a complementary role for the technology. Models may help with repetitive implementation-level improvements, while developers check behavior and take responsibility for larger structural changes.
How the study was tested
This was an empirical evaluation using 30 open-source Java projects. The researchers compared model-generated refactorings with developer refactorings using unit-test results, static code-quality measures and reductions in code smells, and also tested different prompting approaches. The study reports results for several named models, while some detailed comparisons in the paper focus specifically on StarCoder2.
The evidence is limited to the projects, language and evaluation measures included in the study. Passing unit tests and reducing code smells do not by themselves establish that a change is appropriate for every software system, especially when architectural context is important. The authors identify evaluation across more diverse projects and programming languages as future work.
// Source
ACM Transactions on Software Engineering and Methodology · 2026 · DOI: 10.1145/3801158
Authors: Jonathan Cordeiro, Shayan Noei, Ying Zou
Institutions: Queen's University