Summary of Mosaic Memory: Fuzzy Duplication in Copyright Traps For Large Language Models, by Igor Shilov et al.
Mosaic Memory: Fuzzy Duplication in Copyright Traps for Large Language Models
by Igor Shilov, Matthieu Meeus, Yves-Alexandre de Montjoye
First submitted to arxiv on: 24 May 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed solution generates fuzzy copyright traps to improve content detectability in Large Language Models (LLMs). These traps feature slight modifications across duplication, making them resilient against common data deduplication techniques. The generated fuzzy trap sequences are memorized nearly as well as exact duplicates in a 1.3B LLM during fine-tuning. Membership Inference Attack (MIA) ROC AUC drops from 0.9 to 0.87 when 4 tokens are replaced across fuzzy duplicates. Selecting replacement positions to minimize exact overlap between fuzzy duplicates leads to similar memorization, making them highly unlikely to be removed by deduplication processes. The study highlights the importance of considering fuzzy duplicates in LLM memorization and questions the effectiveness of exact data deduplication as a privacy protection technique. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Language Models (LLMs) use massive datasets to learn from, but these datasets often include copyrighted content without permission. To fix this, we propose using “fuzzy traps” that are slightly modified versions of original text. We tested these fuzzy traps and found that they’re just as well-remembered by LLMs as the exact originals. This means that even if someone tries to remove duplicates from a dataset, the fuzzy traps will still be detectable. We also looked at how well LLMs memorize across different types of duplicates and found that it’s an important consideration for understanding how they work. |
Keywords
» Artificial intelligence » Auc » Fine tuning » Inference
