Summary of Doppler: Differentially Private Optimizers with Low-pass Filter For Privacy Noise Reduction, by Xinwei Zhang et al.
DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction
by Xinwei Zhang, Zhiqi Bu, Mingyi Hong, Meisam Razaviyayn
First submitted to arxiv on: 24 Aug 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Cryptography and Security (cs.CR); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper addresses a pressing concern in deep-learning systems, ensuring privacy during training by preventing sensitive information leakage from collected data to trained models. To achieve this, differentially private (DP) optimizers like DPSGD and its variants are used, which privatize the training procedure through gradient clipping and DP noise injection. However, these optimizers often result in significant performance degradation, hindering their application in tasks like foundation model pretraining. This study proposes a novel signal processing perspective for designing and analyzing DP optimizers, introducing a new component called DOPPLER that effectively reduces DP noise by amplifying gradients while suppressing DP noise within the frequency domain. Experiments demonstrate that DP optimizers with DOPPLER outperform their counterparts without it, achieving 3%-10% improvements in test accuracy on various models and datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research paper tackles a crucial issue in deep learning: keeping training data private while still getting good results from machine learning models. Right now, there are some methods that can do this, but they often make the model perform worse. The authors of this paper came up with a new idea to fix this problem by looking at it like a signal processing task. They created a special component called DOPPLER that helps reduce noise in the training data while keeping the important information. This improved the performance of the models and made them more accurate. |
Keywords
» Artificial intelligence » Deep learning » Machine learning » Pretraining » Signal processing
