Summary of Adapting Multilingual Llms to Low-resource Languages Using Continued Pre-training and Synthetic Corpus, by Raviraj Joshi et al.
Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus
by Raviraj Joshi, Kanishk Singla, Anusha Kamath, Raunak Kalani, Rakesh Paul, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar, Eileen Long
First submitted to arxiv on: 18 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed work focuses on improving the performance of multilingual language models (LLMs) in low-resource languages like Hindi. To achieve this, the authors emphasize the importance of continued pre-training and use translation-based synthetic corpora. They introduce Nemotron-Mini-Hindi 4B, a bilingual model that supports both Hindi and English, trained on a mix of real and synthetic tokens. The models are evaluated on Hindi benchmarks and found to be competitive with state-of-the-art results while also performing well on English tasks. The study demonstrates the effectiveness of continued pre-training in enhancing the factual accuracy of LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper is about making language models better for languages that don’t have a lot of data available. They want to see if they can improve these models by giving them more information to learn from. To do this, they’re using fake texts that are translated versions of real texts. They create a new model that can understand both Hindi and English, and train it on a mix of real and fake texts. They find that the model does really well on Hindi tests and is also good at understanding English. This shows that giving the model more information helps it get better. |
Keywords
» Artificial intelligence » Translation
