Summary of Representation Shattering in Transformers: a Synthetic Study with Knowledge Editing, by Kento Nishi et al.
Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing
by Kento Nishi, Maya Okawa, Rahul Ramesh, Mikail Khona, Hidenori Tanaka, Ekdeep Singh Lubana
First submitted to arxiv on: 22 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel knowledge editing (KE) algorithm is designed to update incorrect or outdated factual associations in large language models. Recent studies have shown that applying KE can lead to decreased factual recall accuracy and impaired reasoning abilities, but the reasons behind these destructive failures are not well understood. This paper investigates why KE methods may distort representations of concepts beyond the targeted fact, leading to a degradation of model performance. A synthetic task is defined to test the effects of KE on a Transformer model trained from scratch to internalize a structured knowledge graph. The results show that KE inadvertently affects representations of entities beyond the targeted one, distorting relevant structures that allow a model to infer unseen knowledge about an entity. This phenomenon, referred to as representation shattering, leads to a degradation of factual recall and reasoning performance. Preliminary experiments with pre-trained Llama and Mamba models also reproduce this effect. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A big question in AI is: what happens when we try to fix mistakes in large language models? Some researchers have found that fixing mistakes can actually make the models worse, not better! But why does this happen? This paper tries to answer this question by looking at how a special type of editing algorithm works. The algorithm changes the way the model stores information about facts, and it seems to cause problems for the model’s ability to reason and learn new things. The researchers created a special task to test how well the algorithm works, and they found that it can actually make the model worse at understanding relationships between different pieces of information. This is bad news for AI because it means we need to be careful when trying to fix mistakes in our models. |
Keywords
» Artificial intelligence » Knowledge graph » Llama » Recall » Transformer
