Summary of Mixtures Of In-context Learners, by Giwon Hong et al.
Mixtures of In-Context Learners
by Giwon Hong, Emile van Krieken, Edoardo Ponti, Nikolay Malkin, Pasquale Minervini
First submitted to arxiv on: 5 Nov 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces Mixtures of In-Context Learners (MoICL), a novel method for adapting large language models (LLMs) using in-context learning. Unlike traditional approaches, MoICL treats subsets of demonstrations as experts and learns a weighting function to merge their output distributions based on a training set. The proposed approach outperforms strong baselines on 5 out of 7 classification datasets, achieving up to +13% performance improvements compared to in-context learning (ICL) and LENS. Moreover, MoICL reduces the inference time needed to achieve the same performance with fewer demonstrations, enhancing the Pareto frontier of ICL. Additionally, MoICL is more robust to out-of-domain, imbalanced, or noisy demonstrations, or can filter these out from datasets. Overall, MoICL presents a more expressive approach to learning from demonstrations without exhausting the context window or memory. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper talks about how to make computers learn better by showing them examples instead of just telling them what to do. The usual way is called “in-context learning,” but it has some problems, like using too much computer memory and not working well with noisy or mixed-up data. To fix this, the researchers came up with a new approach called MoICL (Mixtures of In-Context Learners). They tested it on lots of different datasets and found that it works better than other methods in many cases. It’s also faster and more robust to mistakes in the training data. |
Keywords
» Artificial intelligence » Classification » Context window » Inference
