Summary of Local Loss Optimization in the Infinite Width: Stable Parameterization Of Predictive Coding Networks and Target Propagation, by Satoki Ishikawa et al.
Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation
by Satoki Ishikawa, Rio Yokota, Ryo Karakida
First submitted to arxiv on: 4 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Neural and Evolutionary Computing (cs.NE)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers explore alternative methods for training neural networks without relying on backpropagation. They focus on a technique called “local learning,” where each layer learns through local targets and losses. However, the algorithms used in local learning can be complex and require additional hyperparameters to work effectively. To address these challenges, the authors introduce a new parameterization called (maximal update parameterization) and apply it to two different types of local targets: predictive coding (PC) and target propagation (TP). The study shows that enables the transfer of hyperparameters across models with different widths and reveals unique properties not present in conventional backpropagation. The authors also analyze deep linear networks and find that PC’s gradients interpolate between first-order and Gauss-Newton-like gradients, while TP favors feature learning over kernel learning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about finding new ways to train artificial neural networks without using a technique called backpropagation. Neural networks are like super powerful computers that can learn from data and make predictions or decisions. Backpropagation is one way that we currently use to teach these networks, but it’s not the only way. The researchers in this paper are exploring an alternative method called local learning, where each layer of the network learns through its own targets and losses. This approach can be tricky because it requires extra settings to work well. To help with this, the authors introduce a new way of setting these parameters, which they call (maximal update parameterization). They test this method on two different types of local learning: predictive coding and target propagation. The study shows that makes it easier to transfer settings between networks with different sizes. |
Keywords
» Artificial intelligence » Backpropagation
