Summary of Few-shot Cross-lingual Transfer For Prompting Large Language Models in Low-resource Languages, by Christopher Toukmaji
Few-Shot Cross-Lingual Transfer for Prompting Large Language Models in Low-Resource Languages
by Christopher Toukmaji
First submitted to arxiv on: 9 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A recent surge in Natural Language Processing (NLP) research has seen large pre-trained language models (PLMs) excel in “prompting” or in-context learning. However, only the largest PLMs can effectively perform this task, and these models are typically trained on English corpora, leaving other languages behind. The data limitations in most languages prevent the training of language-specific PLMs capable of prompting. This paper evaluates three methods to adapt LLaMa, a 7B parameter open-source PLM primarily trained in English, for prompting in low-resource languages such as Kinyarwanda, Hausa, and Luganda. These methods include few-shot prompting (prompt), language-adaptive fine-tuning (LAFT), and neural machine translation (translate). The authors evaluate these methods on abstractive summarization, multi-class topic classification, and named-entity recognition tasks. Surprisingly, the results show that LAFT is not always the optimal choice for adapting PLMs for prompting, and instead, the translate and prompt settings are a compute-efficient and cost-effective method of few-shot prompting. The findings suggest that the prompting method performs better than both translating and LAFT with statistical significance across all tasks and languages. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models can be very good at learning new things when given examples to follow. But most of these models were only trained on English texts, which means they’re not as good at understanding other languages. To help them learn about new languages, researchers are trying different ways to adapt these models for use in languages like Kinyarwanda, Hausa, and Luganda. In this paper, the authors test three methods: using a little bit of training data from each language (few-shot prompting), changing the model’s behavior based on the language it’s learning (language-adaptive fine-tuning), or translating text into English so the model can learn about it there. The results show that one method is better than the others at helping the models understand these new languages and perform tasks like summarizing texts, classifying topics, and identifying important words. |
Keywords
* Artificial intelligence * Classification * Few shot * Fine tuning * Llama * Named entity recognition * Natural language processing * Nlp * Prompt * Prompting * Summarization * Translation
