Yiran Zhao | Chaoqun Liu | Yue Deng | Jiahao Ying | Mahani Aljunied | Zhaodonghui Li | Lidong Bing | Hou Pong Chan | Yu Rong | Deli Zhao | Wenxuan Zhang* (Corresponding Author)

• 📄 Paper
• 💻 GitHub Repository
• 🤗 Hugging Face Models

People built the Tower of Babel to reach heaven and achieve unity,
but God confused their language and scattered them across the earth.
— Story from Genesis, Old Testament

Large language models (LLMs) have transformed natural language processing (NLP), yet open-source multilingual LLMs remain scarce, with existing models often limited in language coverage.

To address this disparity, we introduce Babel, a multilingual LLM that covers the top 25 languages by number of speakers, supports over 90% of the global population, and includes many languages neglected by other open multilingual LLMs.

Unlike traditional continue pretraining approaches, Babel expands its parameter count through a layer extension technique that elevates Babel's performance ceiling. We introduce two variants:

• Babel-9B: Designed for efficient inference and fine-tuning.
• Babel-83B: Sets a new standard for open multilingual LLMs.

Extensive evaluations on multilingual tasks demonstrate its superior performance compared to open LLMs of comparable size. Using existing supervised fine-tuning datasets, Babel achieves remarkable performance, with Babel-9B-Chat leading among 10B-sized LLMs and Babel-83B-Chat setting a new standard for open LLMs, performing comparably to GPT-4o on certain tasks.

The following table lists the languages supported by Babel, sorted by the number of speakers. Highlighted languages are those underexplored by previous multilingual LLMs.

We evaluate Babel using diverse multilingual tasks:

1. World Knowledge: MMMLU, M3Exam (authentic human exam questions).
2. Reasoning: MGSM, XCOPA.
3. Understanding: XNLI.
4. Translation: Flores-200.

We thank Guanzheng Chen for assisting with the training codebase. Special thanks to Tantong Champaiboon, Nguyen Ngoc Yen Nhi, and Tara Devina Putri for their contributions to dataset evaluation and fact-checking. Additional thanks to Fan Wang, Jiasheng Tang, Xin Li, and Hao Zhang for coordinating computing resources.

If you find our project useful, please cite it using:

@misc{zhao2025babelopenmultilinguallarge,
      title={Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakers}, 
      author={Yiran Zhao and Chaoqun Liu and Yue Deng and Jiahao Ying and Mahani Aljunied and Zhaodonghui Li and Lidong Bing and Hou Pong Chan and Yu Rong and Deli Zhao and Wenxuan Zhang},
      year={2025},
      eprint={2503.00865},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.00865}, 
}