Summary of Efficient and Private: Memorisation Under Differentially Private Parameter-efficient Fine-tuning in Language Models, by Olivia Ma and Jonathan Passerat-palmbach and Dmitrii Usynin
Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models
by Olivia Ma, Jonathan Passerat-Palmbach, Dmitrii Usynin
First submitted to arxiv on: 24 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Fine-tuning large language models (LLMs) for specific tasks raises concerns about privacy risks, as they may unintentionally memorize and leak sensitive training data. Differential Privacy (DP) is a solution to mitigate these risks but comes with significant computational and performance trade-offs when combined with standard fine-tuning approaches. This paper investigates Parameter-Efficient Fine-Tuning (PEFT) methods under DP constraints, demonstrating that PEFT achieves comparable performance to standard fine-tuning while requiring fewer parameters and reducing privacy leakage. The authors also conduct a data poisoning experiment involving intentional mislabelling to assess model memorisation and directly measure privacy risks, showing that PEFT not only provides an alternative but also serves as a complementary approach for privacy-preserving, resource-efficient fine-tuning of LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are great at doing certain tasks, but they can accidentally remember and share private information from their training data. This is a problem because it could expose sensitive info. To solve this issue, scientists use something called Differential Privacy (DP). However, using DP with these large language models makes them work slower and less well. In this study, researchers looked at ways to make fine-tuning these models more efficient while still keeping their private data safe. They found that a method called PEFT (Parameter-Efficient Fine-Tuning) works just as well as the standard way but uses fewer parts and keeps privacy risks lower. |
Keywords
» Artificial intelligence » Fine tuning » Parameter efficient
