Summary of Why Larger Language Models Do In-context Learning Differently?, by Zhenmei Shi et al.
Why Larger Language Models Do In-context Learning Differently?
by Zhenmei Shi, Junyi Wei, Zhuoyan Xu, Yingyu Liang
First submitted to arxiv on: 30 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the in-context learning (ICL) abilities of large language models (LLMs), which can learn from brief task examples without modifying their parameters. Researchers observed that larger LLMs are more sensitive to noise in test contexts, leading to varying ICL behaviors between models of different scales. To improve understanding of LLM and ICL, the study analyzes two stylized settings: linear regression with one-layer transformers and parity classification with two-layer attention heads transformers. The analysis provides closed-form optimal solutions, revealing that smaller models focus on important hidden features while larger ones cover more features, making them more susceptible to noise. This insight sheds light on transformer attention patterns and their impact on ICL. Preliminary experiments support the findings for large base and chat models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how big language models can learn new tasks just by seeing a few examples. These models are powerful because they don’t need to be changed much after learning from those examples. One curious thing scientists noticed is that bigger models are more affected by noise in the test data. This paper tries to understand why this happens by looking at simple cases of linear regression and parity classification. The results show that smaller models focus on important parts of the data, while bigger models look at everything, making them more likely to be distracted by noise. This helps us understand how big language models work and how they can learn new things. |
Keywords
» Artificial intelligence » Attention » Classification » Linear regression » Transformer
