Models designed to understand spatial relationships predicted structure-driven material properties more reliably than language models.
Researchers tested how well language models learn from data that combine coordinates with categories, a format used to describe crystal structures and predict properties such as formation energy and elastic modulus. Using synthetic datasets and the MatText benchmarking framework, they varied how much the coordinates and categories contributed to the answer.
The language models performed well on category-based patterns but consistently failed to capture the coordinate information. This limitation remained in tests of models with up to 70 billion parameters, datasets containing up to 2 million structures and different text representations. Models designed specifically to use geometric information outperformed the language models on material-property tasks dominated by structure.
How the models used structure
Language models showed a consistent split in performance: they learned category patterns but did not reliably learn from coordinates, which describe the spatial arrangement of the data. The researchers observed this limitation across model sizes up to 70 billion parameters, dataset sizes up to 2 × 10^6 structures and different ways of representing the data in text.
For material-property prediction tasks in which structure was the main influence, specialized geometric architectures performed better than language models. The researchers describe a persistent performance gap between graph-based geometric models and language models as a “Graph Neural Network Language Model wall.”
Why model choice matters
The findings suggest that making a language model larger or providing more structures may not solve tasks in which spatial arrangement is central. For those material-property problems, the study supports choosing models built to represent geometry rather than relying on a general language model alone.
The work offers practical guidance for selecting machine-learning architectures for scientific data, especially when predictions depend more on how components are arranged than on their categories.
Evidence and caveats
The conclusions come from systematic experiments on synthetic coordinate–category datasets and from the MatText benchmarking framework. The tests covered a range of coordinate and category contributions, model sizes, dataset scales and text representations, but the abstract does not specify the full set of real-world materials or properties included in the benchmark.
The results directly support the reported limitations under these experimental conditions. They do not establish that every language model will fail on every materials task, or that specialized geometric models will outperform them when category information is more important than structure.
// Source
Nature Computational Science · 2026 · DOI: 10.1038/s43588-026-01045-2
Authors: Nawaf Alampara, Santiago Miret, Kevin Maik Jablonka
