How Far Can Multilingual Text Embeddings Be Trained From Scratch? A Compute-Efficient Study of Arabic, English, and Urdu

Abstract

We investigate how much useful multilingual embedding capability can be obtained when the student encoder starts from random initialization, under a strict single-GPU compute budget. We present the mentee-embed series — three controlled experiments (v1–v3) using a 41M-parameter Transformer encoder with a custom 50,000-entry BPE tokenizer, trained entirely from scratch on Arabic, English, and Urdu. The student is randomly initialized; a frozen pretrained teacher (multilingual-e5-base) supplies relational distillation signal only during Stage B but transfers no weights in to the student. Our three-version controlled progression isolates the dominant training factors: batch size and retrieval training data domain matter more than model size in this regime. The v1→v2→v3 ablation demonstrates this strikingly: scaling from 41M to 125M parameters while reducing batch size (v2) degrades Protocol A avg MRR@10 from 0.585 to 0.429, while returning to 41M with batch size 512 and adding MS-MARCO retrieval data (v3) recovers to 0.655 — a 3× Protocol C MS-MARCO improvement (0.215→0.645) and STS-B Spearman ρ = 0.683. Model weights: https://huggingface.co/MenteEAI/mentee-embed-v3 Code: https://github.com/MenteE-s/mentee-embeddings