Summary of Dynamical Loss Functions Shape Landscape Topography and Improve Learning in Artificial Neural Networks, by Eduardo Lavin and Miguel Ruiz-garcia
Dynamical loss functions shape landscape topography and improve learning in artificial neural networks
by Eduardo Lavin, Miguel Ruiz-Garcia
First submitted to arxiv on: 14 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper derives dynamical loss functions from standard loss functions used in supervised classification tasks by introducing oscillations that globally alter the loss landscape without affecting the global minima. It transforms cross-entropy and mean squared error into dynamical loss functions, showing how they improve validation accuracy for networks of varying sizes. The authors also explore the evolution of these dynamical loss functions during training, highlighting the emergence of instabilities that may be linked to edge-of-instability minimization. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper takes standard loss functions used in supervised classification tasks and adds oscillations that make the loss landscape change over time. This changes how neural networks learn and improves their performance on validation sets. The authors show how this works for different-sized networks and explore what happens during training, revealing some interesting instabilities. |
Keywords
» Artificial intelligence » Classification » Cross entropy » Supervised
