Summary of Context-scaling Versus Task-scaling in In-context Learning, by Amirhesam Abedsoltan et al.
Context-Scaling versus Task-Scaling in In-Context Learning
by Amirhesam Abedsoltan, Adityanarayanan Radhakrishnan, Jingfeng Wu, Mikhail Belkin
First submitted to arxiv on: 16 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Transformers exhibit In-Context Learning (ICL), where they can solve new tasks by using examples in the prompt without additional training. Our work identifies two key components of ICL: context-scaling, where model performance improves with more in-context examples, and task-scaling, where model performance improves with more pre-training tasks. We find that transformers are capable of both context-scaling and task-scaling, whereas standard Multi-Layer Perceptrons (MLPs) can only perform task-scaling. To understand how transformers achieve context-scaling, we propose a simplified transformer architecture without key, query, value weights, showing comparable ICL performance to GPT-2 in various statistical learning tasks. We also demonstrate that a single block of our simplified transformer acts as a powerful predictor capable of context-scaling but not task-scaling. By concatenating the output of this feature map with vectorized data and inputting it into MLPs, we enable both context-scaling and task-scaling. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about how some special kinds of artificial intelligence (AI) models called transformers can learn new things by looking at examples. They don’t need to be retrained for each new task. The researchers found two important parts that make this happen: using more examples to help the model, and giving the model many tasks to practice on. They tested these ideas with some special computer programs and showed that they work well. They also came up with a simpler version of the transformer that can do this learning too. |
Keywords
» Artificial intelligence » Feature map » Gpt » Prompt » Transformer
