Summary of Geometric Inductive Biases Of Deep Networks: the Role Of Data and Architecture, by Sajad Movahedi et al.
Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture
by Sajad Movahedi, Antonio Orvieto, Seyed-Mohsen Moosavi-Dezfooli
First submitted to arxiv on: 15 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed geometric invariance hypothesis (GIH) suggests that neural network input space curvature remains invariant under architecture-dependent transformations during training. Researchers investigate a high-dimensional binary classification problem on a plane and find that ResNets, unlike MPLs, fail to generalize based on the plane’s orientation. The study defines average geometry and evolution as summaries of the model’s input-output geometry and its evolution during training. By analyzing average geometry at initialization, the team discovers that geometry evolves according to data covariance projected onto average geometry, leading to an architecture-dependent invariance property dubbed GIH. This property causes low-rank average geometry, affecting generalization performance. The paper presents extensive experimental results on GIH’s consequences and its relation to neural network generalization. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study proposes the geometric invariance hypothesis (GIH), which suggests that a neural network’s input space curvature stays the same during training when certain transformations are applied. Researchers looked at a problem where they had to classify things as either 0 or 1, and found that some types of neural networks did better than others depending on how the data was oriented. They defined new ways to measure how neural networks work and found that this affects how well they do their job. |
Keywords
» Artificial intelligence » Classification » Generalization » Neural network
