Summary of On the Convergence Of Differentially-private Fine-tuning: to Linearly Probe or to Fully Fine-tune?, by Shuqi Ke et al.
On the Convergence of Differentially-Private Fine-tuning: To Linearly Probe or to Fully Fine-tune?
by Shuqi Ke, Charlie Hou, Giulia Fanti, Sewoong Oh
First submitted to arxiv on: 29 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Optimization and Control (math.OC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the two-phase differentially private (DP) machine learning pipeline, which typically involves non-private pre-training followed by DP optimization techniques. The study analyzes the training dynamics of DP linear probing and full fine-tuning, as well as the phenomenon of sequential fine-tuning, starting with linear probing and transitioning to full fine-tuning. The authors provide theoretical insights into the convergence of DP fine-tuning within an overparameterized neural network and establish a utility curve for allocating privacy budget between linear probing and full fine-tuning. Empirical evaluations on various benchmarks and models support the theoretical results, revealing the complex nature of DP fine-tuning methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how to make machine learning more private. Right now, many people use a two-step process: train a model on public data, then adjust it using secret information. But researchers have found that this doesn’t always work well. They’re trying to figure out why and come up with better ways to do things. The study looks at different methods for adjusting the model, like linear probing and full fine-tuning, and tries to understand how they work together. The results show that it’s not just about using one method or another, but also about deciding when to use each one. |
Keywords
* Artificial intelligence * Fine tuning * Machine learning * Neural network * Optimization
