Summary of Towards Efficient and Scalable Training Of Differentially Private Deep Learning, by Sebastian Rodriguez Beltran et al.
Towards Efficient and Scalable Training of Differentially Private Deep Learning
by Sebastian Rodriguez Beltran, Marlon Tobaben, Joonas Jälkö, Niki Loppi, Antti Honkela
First submitted to arxiv on: 25 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Cryptography and Security (cs.CR); Distributed, Parallel, and Cluster Computing (cs.DC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A comprehensive study on the computational cost of training deep learning models under differential privacy (DP) is presented, focusing on the implementation of Poisson subsampling in DP-stochastic gradient descent (DP-SGD). The authors re-implement efficient methods using Poisson subsampling and benchmark them to quantify the computational cost. The results show that naive implementations have lower throughput than standard stochastic gradient descent (SGD), but efficient gradient clipping can halve this cost. Alternative computationally efficient implementations are proposed, and the scaling behavior of DP-SGD is studied up to 80 GPUs, showing better scalability than SGD. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Training deep learning models with differential privacy is crucial for protecting user data. However, implementing Poisson subsampling in DP-stochastic gradient descent (DP-SGD) can be computationally expensive. This study compares the efficiency of different implementations and proposes alternative methods to reduce the cost. The results show that using efficient gradient clipping can make a big difference. |
Keywords
» Artificial intelligence » Deep learning » Stochastic gradient descent
