Summary of Learning and Unlearning Of Fabricated Knowledge in Language Models, by Chen Sun et al.
Learning and Unlearning of Fabricated Knowledge in Language Models
by Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov, Max Vladymyrov, Mark Sandler
First submitted to arxiv on: 29 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates how large language models (LMs) retain new knowledge introduced during training and whether it can be erased or mitigated. Specifically, it studies how LMs respond to fact injection from a probing dataset called “Outlandish”, designed to test various fact types. The results show that there’s an optimal range of fact novelty where the injected memory is most enduring. Facts conflicting with common knowledge are remembered for tens of thousands of training steps, while mundane and randomly jumbled prompts are forgotten rapidly. Furthermore, knowledge-conflicting facts can prime LMs’ hallucinations on unrelated prompts, demonstrating non-target generalization. However, this priming effect can be largely erased by novel multi-step sparse updates without affecting the model’s training ability. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how large language models (LMs) learn and remember new things when they’re trained. They created a special dataset called “Outlandish” to test different types of facts. The results show that LMs remember some facts for a really long time, but forget others quickly. The interesting part is that LMs can start to make up new ideas based on the facts they learned, even if those ideas aren’t related to the original fact. However, this effect can be undone by updating the model in a special way without making it worse at learning. |
Keywords
» Artificial intelligence » Generalization
