Summary of Pre-training Differentially Private Models with Limited Public Data, by Zhiqi Bu et al.
Pre-training Differentially Private Models with Limited Public Data
by Zhiqi Bu, Xinwei Zhang, Mingyi Hong, Sheng Zha, George Karypis
First submitted to arxiv on: 28 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Cryptography and Security (cs.CR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the role of differential privacy (DP) in securing large foundation models that rely on massive datasets containing sensitive information. The authors highlight the limitations of applying DP only during the fine-tuning stage, as it can degrade performance when applied during the pre-training phase. They demonstrate that current methods are insufficient in protecting a significant portion of the data used during initial training. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making sure big computer models don’t accidentally share private information. These models need lots of data to work well, but some of this data might be personal or belong to someone else. To keep this information safe, researchers use something called differential privacy (DP). The problem is that DP only works well when it’s used at the very end, during fine-tuning. If you try to apply it earlier in the process, it can make the model perform worse. This means we still don’t have a good way to protect most of the data these models use. |
Keywords
* Artificial intelligence * Fine tuning
