Summary of Progressive Feedforward Collapse Of Resnet Training, by Sicong Wang and Kuo Gai and Shihua Zhang
Progressive Feedforward Collapse of ResNet Training
by Sicong Wang, Kuo Gai, Shihua Zhang
First submitted to arxiv on: 2 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Statistics Theory (math.ST)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The research paper explores the phenomenon of neural collapse in deep neural networks (DNNs) at the terminal phase of training. It shows that the last-layer features collapse to their class means and form a simplex equiangular tight frame aligning with classifier vectors. The study characterizes the geometry of intermediate layers in ResNet models, proposing progressive feedforward collapse (PFC), which claims the degree of collapse increases during forward propagation. The paper also derives a transparent model for well-trained ResNets, showing that they approximate geodesic curves in Wasserstein space at the terminal phase. The results demonstrate monotonic decrease in PFC metrics across depth on various datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Neural networks are super smart machines that can learn from data. But did you know that these networks have a secret? When they’re fully trained, their inner workings become super simple and symmetrical! This phenomenon is called neural collapse, or NC for short. Researchers wanted to understand what’s going on inside these networks during training. They found that the inner layers of the network are getting simpler and more aligned with the final answers as it learns. They even came up with a new idea called progressive feedforward collapse (PFC) to describe this process. The study also created a special model that shows how well-trained networks relate to their input data. |
Keywords
» Artificial intelligence » Resnet
