Summary of Min-k%++: Improved Baseline For Detecting Pre-training Data From Large Language Models, by Jingyang Zhang et al.
Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models
by Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, Hai Li
First submitted to arxiv on: 3 Apr 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed methodology, Min-K%++, is a novel and theoretically motivated approach for pre-training data detection in large language models (LLMs). Building upon the insight that training samples are local maxima of the modeled distribution along each input dimension through maximum likelihood training, the method translates the problem into identifying local maxima. Specifically, it determines whether an input forms a mode or has relatively high probability under the conditional categorical distribution. The proposed approach achieves state-of-the-art (SOTA) performance across multiple settings, outperforming existing methods by up to 10.5% in detection AUROC on the WikiMIA benchmark and performing on par with reference-based methods on the MIMIR benchmark. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper solves a problem that affects big language models. It’s called pre-training data detection, which means finding out when training samples are used for testing. This is important because it can help prevent things like copyright violation and test data contamination. Right now, there are methods to detect this, but they’re not very good. They mostly use simple rules and don’t have a strong foundation. The authors of this paper propose a new method called Min-K%++. It works by looking at the distribution of the model along each input dimension and finding where it forms a local maximum. This helps identify when training samples are used for testing. In tests, this new method performs much better than existing methods. |
Keywords
* Artificial intelligence * Likelihood * Probability
