Summary of Adafisher: Adaptive Second Order Optimization Via Fisher Information, by Damien Martins Gomes and Yanlei Zhang and Eugene Belilovsky and Guy Wolf and Mahdi S. Hosseini
AdaFisher: Adaptive Second Order Optimization via Fisher Information
by Damien Martins Gomes, Yanlei Zhang, Eugene Belilovsky, Guy Wolf, Mahdi S. Hosseini
First submitted to arxiv on: 26 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Optimization and Control (math.OC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes an adaptive second-order optimizer, AdaFisher, that leverages a diagonal block-Kronecker approximation of the Fisher information matrix for adaptive gradient preconditioning. This approach aims to bridge the gap between enhanced convergence and generalization capabilities and computational efficiency in training deep neural networks (DNNs). The authors compare AdaFisher to state-of-the-art optimizers like Adam and SGD, showcasing its ability to outperform them in terms of both accuracy and convergence speed on image classification, language modeling tasks. The proposed optimizer is demonstrated to be stable and robust in hyper-parameter tuning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper introduces a new way to train deep neural networks using an adaptive second-order optimizer called AdaFisher. This method tries to make training faster and better by using more information about the data. The authors compare their approach to other methods that are already popular, like Adam and SGD, and show that it can do even better on certain tasks. |
Keywords
» Artificial intelligence » Generalization » Image classification
