Summary of Romansetu: Efficiently Unlocking Multilingual Capabilities Of Large Language Models Via Romanization, by Jaavid Aktar Husain et al.
RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization
by Jaavid Aktar Husain, Raj Dabre, Aswanth Kumar, Jay Gala, Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan
First submitted to arxiv on: 25 Jan 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed approach utilizes romanized text as an interface for Large Language Models (LLMs) to extend them to non-English languages that use non-Roman scripts. The method involves continual pretraining of an English LLM on romanized text, followed by instruction tuning on romanized data. This leads to reduced token fertility and matching or outperforming native script representation across various NLU, NLG, and MT tasks. Additionally, the embeddings computed from romanized text exhibit closer alignment with their English translations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study helps us talk to computers in more languages that don’t use Roman letters. Right now, most computer language models only understand a few languages. To fix this, we used Romanized text (text written using Roman letters) as a way for these models to learn new languages. We trained an English model on Romanized text from other languages and then fine-tuned it to work even better. This made the model more accurate and able to do tasks like understanding and generating text in different languages. |
Keywords
* Artificial intelligence * Alignment * Instruction tuning * Pretraining * Token
