A benchmark found that language models could solve problems while signaling answers through an unexpected choice of response language.
The study introduced a benchmark covering eight kinds of reasoning, including arithmetic, causal, logical, moral, social, spatial, temporal and linguistic tasks. Instead of asking models to explain an answer, the researchers required them to select a particular non-English language for their first response token; the chosen language represented the answer. This was intended to limit reasoning expressed through the visible response or the prompt's context window.
GPT-4.5 had the highest reported accuracy, at 74.7%, followed by Grok-2 at 67.2% and Llama 3.1 405B at 65.6%. Smaller models generally performed less well, but model size alone did not determine results: Llama 3.1 70B and the larger 405B model had similar scores, and some older models outperformed newer versions.
What the models could do
The researchers evaluated 18 language models on 4,000 benchmark items. GPT-4.5 achieved the highest accuracy at 74.7%, ahead of Grok-2 at 67.2% and Llama 3.1 405B at 65.6%. Accuracy was generally higher for arithmetic, causal, logical, moral and linguistic tasks, while social, spatial and temporal tasks were more difficult. On the hardest categories, even the best-performing models often scored below 60%. GPT-4.5 was especially strong on social reasoning, with an accuracy of 94.6%.
The benchmark required models to solve each problem and indicate the answer by choosing a specified response language other than English. A reverse control, in which English responses were counted as correct, produced high average performance across models and benchmark types. Together with analyses that increased task difficulty, these results suggest that the models were doing some internal computation rather than relying only on obvious language cues. However, the researchers said that heuristic shortcuts could not be ruled out in some conditions.
Why hidden reasoning matters
Visible step-by-step answers are not the only place where a model's reasoning may occur. This study presents a way to measure some inferences that happen between the model's observable token predictions, including reasoning that does not leave an explicit text record.
That matters for oversight: capabilities such as hidden planning, goal-seeking or deception could be harder to assess if they were carried out without visible reasoning traces. The benchmark could also be rerun on future models, although the researchers stress that it does not capture every kind of reasoning.
Evidence and caveats
This was a benchmark study, not a direct observation of the computations inside a model. The researchers created 250 items in each of eight reasoning categories, checked them automatically and manually, and tested models from several families. Control experiments and difficulty scaling support the interpretation that models performed internal reasoning, but the authors could not exclude heuristic strategies under some conditions.
The benchmark covers only eight reasoning domains, and its language-selection task adds the extra challenge of overriding a model's tendency to answer in the language of the prompt. The study also cannot establish a definitive link between model size, architecture and hidden reasoning ability because information about model architectures and parameter counts is not always available.
// Source
Scientific Reports · 2026 · DOI: 10.1038/s41598-026-68486-7
Authors: Thilo Hagendorff, Sarah Fabi
