Summary of Rwku: Benchmarking Real-world Knowledge Unlearning For Large Language Models, by Zhuoran Jin et al.
RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
by Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, Jun Zhao
First submitted to arxiv on: 16 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a Real-World Knowledge Unlearning benchmark (RWKU) to efficiently remove specific knowledge from large language models (LLMs). The RWKU is designed with three key factors: task setting, knowledge source, and evaluation framework. In the task setting, neither the forget corpus nor the retain corpus is accessible, making it a more challenging unlearning scenario. The knowledge source consists of 200 real-world famous people, showing that popular knowledge is widely present in various LLMs. For the evaluation framework, two sets are designed: the forget set and the retain set. Four membership inference attack (MIA) methods and nine kinds of adversarial attack probes are used to rigorously test unlearning efficacy in the forget set. The retain set assesses locality, utility, neighbor perturbation, general ability, reasoning ability, truthfulness, factuality, and fluency. Extensive experiments were conducted across two unlearning scenarios, two models, and six baseline methods, yielding meaningful findings. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about a way to erase unwanted information from large language models that accidentally learn it during training. They propose a benchmark called RWKU (Real-World Knowledge Unlearning) to test how well these models can forget specific things they learned. The benchmark has three main parts: the task, what they want to forget, and how to measure if they’re forgetting it correctly. They tested this benchmark on two different models and found that it’s possible to make them forget certain information without losing their ability to understand language. |
Keywords
* Artificial intelligence * Inference
