Summary of Unlearnable Algorithms For In-context Learning, by Andrei Muresanu et al.
Unlearnable Algorithms for In-context Learning
by Andrei Muresanu, Anvith Thudi, Michael R. Zhang, Nicolas Papernot
First submitted to arxiv on: 1 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Machine unlearning is crucial as models are increasingly deployed on data with unknown provenance. However, achieving exact unlearning – obtaining a model matching the distribution when the data to be forgotten was never used – is challenging or inefficient, often requiring significant retraining. This paper focuses on efficient unlearning methods for task adaptation phase of pre-trained large language models (LLMs). We observe that LLMs’ in-context learning ability allows for efficient exact unlearning of task adaptation training data. An algorithm, ERASE, is provided to select few-shot training examples to prepend to the prompt given to an LLM for task adaptation, with unlearning operation cost independent of model and dataset size, scaling to large models and datasets. We compare our approach to fine-tuning approaches and discuss trade-offs between them. A new holistic measure of unlearning cost is proposed, accounting for varying inference costs, concluding that in-context learning can be more favourable than fine-tuning for deployments involving unlearning requests. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Machine unlearning is important because models are often used with data we don’t know where it came from. Right now, making a model forget something it learned requires a lot of retraining or isn’t very efficient. This paper tries to make this process faster and more efficient by using special methods for pre-trained language models. These models can learn new things quickly just by reading some examples, which makes them good at forgetting too. The researchers came up with an algorithm that selects the most important examples for the model to learn from, making it easy to forget something without needing to retrain the whole model. They also compare this method to another way of making a model forget something and show why their way might be better. |
Keywords
* Artificial intelligence * Few shot * Fine tuning * Inference * Prompt
