Researchers evaluated StarCoder2, GPT-4o Mini, GPT-4o, LLaMA 3 and DeepSeek-v3 on refactoring tasks across 30 open-source Java projects. They assessed whether the revised code passed unit tests, reduced code smells—patterns associated with maintainability problems—and improved other static code-quality measures.

The models were generally strongest at systematic and repetitive changes, such as renaming or extracting methods. Developers were better at context-sensitive work, including changes to modularization, encapsulation and relationships between classes. Prompt design also affected results, with one-shot and chain-of-thought prompts improving some outcomes.