Summary of Diagonalisation Sgd: Fast & Convergent Sgd For Non-differentiable Models Via Reparameterisation and Smoothing, by Dominik Wagner et al.
Diagonalisation SGD: Fast & Convergent SGD for Non-Differentiable Models via Reparameterisation and Smoothing
by Dominik Wagner, Basim Khajwal, C.-H. Luke Ong
First submitted to arxiv on: 19 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper addresses a critical issue in machine learning, where the reparameterisation gradient estimator, despite being low-variance, can be biased for non-differentiable models. This bias can compromise the correctness of gradient-based optimisation methods like stochastic gradient descent (SGD). To tackle this problem, the authors introduce a novel approach to define non-differentiable functions piecewisely and develop a systematic method to obtain smoothings that ensure the reparameterisation gradient estimator is unbiased. The main contribution is a new variant of SGD, Diagonalisation Stochastic Gradient Descent, which progressively improves the accuracy of the smoothed approximation during optimisation, ensuring convergence to stationary points of the unsmoothed objective. Empirical evaluation shows significant benefits over existing methods, including reduced work-normalised variance and improved stability. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper talks about a problem in machine learning where some models can’t be used with certain algorithms. These algorithms are important for training models, but they don’t work well with non-differentiable models. The authors came up with a new way to make these models work better with the algorithms. They also created a new type of algorithm that is more accurate and stable than current methods. This means that people can use these algorithms to train their models faster and more accurately. |
Keywords
* Artificial intelligence * Machine learning * Stochastic gradient descent
