Summary of Infinite Width Models That Work: Why Feature Learning Doesn’t Matter As Much As You Think, by Luke Sernau
Infinite Width Models That Work: Why Feature Learning Doesn’t Matter as Much as You Think
by Luke Sernau
First submitted to arxiv on: 27 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper investigates the performance of infinite-width Neural Tangent Kernels (NTK) compared to finite models in various tasks. Contrary to previous assumptions, the study reveals that NTKs do not require feature learning to achieve identical behavior, as they can select relevant subfeatures from their frozen feature vector. Experimental results show that even when feature learning is disabled, NTKs underperform traditional finite models. The authors attribute this weak performance to the dependence on weak optimizers like SGD and propose a new infinite-width limit based on ADAM-like learning dynamics, which empirically erases the performance gap. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study looks at how well computers can learn from big data sets using special kinds of neural networks called Neural Tangent Kernels. Normally, these networks don’t do very well compared to smaller ones. Researchers thought this was because they didn’t learn good features from the data. But actually, it’s not that simple. The big networks can still work by picking the most important parts from their frozen feature vector. Even when trying to make them work like the small ones, they still don’t do as well. Instead, the problem is with the way they’re trained, and a new way of training them makes them much better. |
