Summary of A Granger-causal Perspective on Gradient Descent with Application to Pruning, by Aditya Shah et al.
A Granger-Causal Perspective on Gradient Descent with Application to Pruning
by Aditya Shah, Aditya Challa, Sravan Danda, Archana Mathur, Snehanshu Saha
First submitted to arxiv on: 4 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the causality aspect of Stochastic Gradient Descent (SGD), a widely used approach for optimizing neural networks. The authors show that SGD has an implicit granger-causal relationship between loss reduction and parameter changes, which can be made explicit with suitable modifications. This causal approach to gradient descent has significant applications, such as pruning, where it allows greater control. In the application of Pruning, the causal approach reveals interesting properties, including a phase shift in accuracy as the percentage of pruned parameters increases. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how Stochastic Gradient Descent works and what happens when we make changes to neural networks using this method. It finds that there is a connection between making these changes and reducing the loss, which can be controlled by modifying the way we do SGD. This control is important for applications like Pruning, where it helps us understand when and how much we should prune weights to get better accuracy. |
Keywords
* Artificial intelligence * Gradient descent * Pruning * Stochastic gradient descent
