Summary of A New Perspective on Shampoo’s Preconditioner, by Depen Morwani et al.
A New Perspective on Shampoo’s Preconditioner
by Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, Lucas Janson
First submitted to arxiv on: 25 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Optimization and Control (math.OC); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper explores Shampoo, a second-order optimization algorithm that uses a Kronecker product preconditioner. The authors establish an explicit connection between Shampoo’s approximation and the optimal Kronecker product approximation of the Gauss-Newton component of the Hessian or the covariance matrix of gradients maintained by Adagrad. They show that the square of Shampoo’s approximation is equivalent to a single step of the power iteration algorithm for computing the optimal Kronecker product approximation. The authors also investigate the impact of various practical tricks on the quality of Hessian approximation using Shampoo, and demonstrate its effectiveness across various datasets and architectures. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Shampoo is a new way to make machines learn faster. It uses a special trick called a Kronecker product preconditioner. This paper shows that Shampoo’s trick is actually really close to the best possible trick for making machines learn. They also show that making some small changes to how Shampoo works can make it even better at learning. The authors tested Shampoo on many different types of data and machine learning models, and it worked really well. |
Keywords
» Artificial intelligence » Machine learning » Optimization
