Summary of Differentially Private Stochastic Gradient Descent with Fixed-size Minibatches: Tighter Rdp Guarantees with or Without Replacement, by Jeremiah Birrell et al.
Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement
by Jeremiah Birrell, Reza Ebrahimi, Rouzbeh Behnia, Jason Pacheco
First submitted to arxiv on: 19 Aug 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Cryptography and Security (cs.CR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The researchers present a new accountant for differentially private stochastic gradient descent (DP-SGD) with fixed-size subsampling without replacement (FSwoR) and with replacement (FSwR). The DP-SGD framework is used to privately train deep learning models, providing a mechanism to control and track the privacy loss incurred during training. The team improves upon existing computable bounds for FSwoR by a factor of 4 and shows that Poisson subsampling and FSwoR with replace-one adjacency have the same privacy guarantees to leading order in the sampling probability. The study also compares fixed-size and Poisson subsampling, revealing lower variance in practice for DP-SGD gradients in a fixed-size regime. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary DP-SGD is a way to train deep learning models while keeping people’s information private. It works by adding noise to the training process to make it harder for others to figure out what people are doing. The researchers looked at how to make this process even better by using something called fixed-size subsampling. This means that instead of looking at all the data, they only look at a small part of it. They found that this way is actually better than another method called Poisson subsampling because it uses less memory and gives more accurate results. |
Keywords
» Artificial intelligence » Deep learning » Probability » Stochastic gradient descent
