Summary of Swea: Updating Factual Knowledge in Large Language Models Via Subject Word Embedding Altering, by Xiaopeng Li et al.
SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering
by Xiaopeng Li, Shasha Li, Shezheng Song, Huijun Liu, Bin Ji, Xi Wang, Jun Ma, Jie Yu, Xiaodong Liu, Jing Wang, Weimin Zhang
First submitted to arxiv on: 31 Jan 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Subject Word Embedding Altering (SWEA) framework and optimizing then suppressing fusion method efficiently update a small amount of knowledge in large language models, achieving state-of-the-art performance on factual knowledge editing tasks. Local editing methods are suitable for updating small amounts of knowledge, but they require significant resources and can disrupt the original organization of model parameters. The SWEAmethod optimizes learnable embedding vectors for the editing target, suppresses Knowledge Embedding Dimensions to obtain final editing embeddings, and demonstrates superior performance on CounterFact, zsRE, and RippleEdits benchmarks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models can be updated efficiently with a new technique called recent model editing. This method is suitable for updating small amounts of knowledge in these models. However, it requires significant resources and can disrupt the original organization of the model’s parameters. The authors propose a new framework called SWEA (Subject Word Embedding Altering) that finds the editing embeddings through token-level matching and adds them to the subject word embeddings in Transformer input. They also develop an optimizing then suppressing fusion method to obtain these editing embeddings. This method optimizes learnable embedding vectors for the editing target and then suppresses Knowledge Embedding Dimensions to get final editing embeddings. The authors demonstrate the effectiveness of this method on several benchmark datasets. |
Keywords
* Artificial intelligence * Embedding * Token * Transformer
