A study of more than 72,000 documents finds multilingual models can identify 25 kinds of web writing, especially in languages with less training data.
The study introduces the Multilingual CORE corpora, a collection of more than 72,000 web documents labeled with a hierarchy of 25 text varieties. Because a document can contain more than one variety, the researchers used multi-label classification rather than assigning every document just one category.
Multilingual models performed better than models trained separately for each language, particularly when training data were limited. But performance fell by an average of 7% when the models classified languages they had not seen during training, showing that web writing shares some features across languages while also retaining language-specific differences.
How the labels performed
The best model achieved 79% F1 averaged across the 16 languages. The researchers observed a similar performance ceiling across the models they tested. When they removed documents with uncertain labels, performance rose above 90% F1, suggesting that the main barrier was ambiguity in how web texts fit the categories rather than a simple lack of model capability.
The hardest issue involving hybrid documents—texts that combine several varieties—was distinguishing them from non-hybrid documents, rather than labeling the hybrids themselves. Multilingual models consistently outperformed monolingual models, especially for languages with limited training data. In tests on languages absent from training, performance dropped by an average of 7%, with declines ranging from 3% to 8%.
Why web labels matter
Large web collections contain many kinds of writing but often lack information about what kind of text each document represents. Automatic labels could help researchers describe and organize these collections, and could support more deliberate control over the mixture of text used in language-model training.
The results also show why such labeling is difficult: web documents can combine several text varieties or sit near the boundary between categories. Multilingual training may help extend register identification to languages that have less labeled data, but it does not remove language-specific differences.
Evidence and caveats
This was an experimental evaluation of monolingual, multilingual and zero-shot classification models using the Multilingual CORE corpora. The reported scores measure how closely the models' labels matched the annotated data; the study also tested how performance changed after removing uncertain examples.
The language collections were sampled in different ways, which may affect comparisons across languages. The models used only the beginnings of documents—512 or 2,048 tokens—because of token limits, so they may miss information later in a text, particularly in hybrid documents. The interpretability analysis covered only two registers in three languages, rather than all 25 categories. The study also did not test generative language models for this task.