Summary of Complexity-aware Training Of Deep Neural Networks For Optimal Structure Discovery, by Valentin Frank Ingmar Guenter and Athanasios Sideris
Complexity-Aware Training of Deep Neural Networks for Optimal Structure Discovery
by Valentin Frank Ingmar Guenter, Athanasios Sideris
First submitted to arxiv on: 14 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed algorithm combines unit/filter and layer pruning of deep neural networks during training, without requiring a pre-trained network. The algorithm optimally trades-off learning accuracy and pruning levels using three user-defined parameters. The optimal network structure is found as the solution of a stochastic optimization problem over the network weights and variational Bernoulli distributions. Pruning occurs when a variational parameter converges to 0, rendering the corresponding structure permanently inactive. The cost function combines prediction accuracy and network pruning in a computational/parameter complexity-aware manner. The algorithm evaluates on CIFAR-10/100 and ImageNet datasets using ResNet architectures, demonstrating improved performance compared to layer only or unit only pruning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Our paper proposes a new way to make deep neural networks more efficient by removing parts that aren’t helping with predictions during training. This is done without needing a pre-trained network. The algorithm finds the best balance between keeping the right information and making the network faster to compute. We show how this works mathematically and test it on images from CIFAR-10/100 and ImageNet datasets. |
Keywords
» Artificial intelligence » Optimization » Pruning » Resnet
