Summary of Memorization in In-context Learning, by Shahriar Golchin et al.
Memorization in In-Context Learning
by Shahriar Golchin, Mihai Surdeanu, Steven Bethard, Eduardo Blanco, Ellen Riloff
First submitted to arxiv on: 21 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A medium-difficulty summary of the abstract reveals that researchers investigated the relationship between in-context learning (ICL) and large language model performance. They found that ICL surfaces memorized training data, particularly when using demonstrations without labels. The surfaced memorization correlates strongly with performance on downstream tasks across various ICL regimes. Notably, few-shot ICL performs better when memorization reaches a high level (around 40%). These findings highlight memorization as a crucial factor in ICL’s success, raising questions about the extent to which language models generalize from demonstrations and how much is due to memorization. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary In simple terms, this study looked at how large language models learn new skills. They found that giving these models examples of what they should do helps them remember old knowledge and perform better. The more examples they get, the better they become. This discovery shows that large language models don’t just learn from new information, but also rely on remembering things they learned before. |
Keywords
» Artificial intelligence » Few shot » Large language model
