Nazarbayev Intellectual Schools

Tokenization Disparity in Kazakh: Cross-Tokenizer Measurement and Low-Cost Vocabulary Adaptation for Large Language Models

Large language models are billed by the token, and tokens are not distributed evenly across languages. We measure how many tokens ten widely deployed tokenizers require to encode identical content in Kazakh and English, using 900 sentence-aligned pairs from OPUS-100 with bootstra...

Tokenization Disparity in Kazakh: Cross-Tokenizer Measurement and Low-Cost Vocabulary Adaptation for Large Language Models

Large language models are billed by the token, and tokens are not distributed evenly across languages. We measure how many tokens ten widely deployed tokenizers require to encode identical content in Kazakh and English, using 900 sentence-aligned pairs from OPUS-100 with bootstra...