Summary of Proact: Progressive Training For Hybrid Clipped Activation Function to Enhance Resilience Of Dnns, by Seyedhamidreza Mousavi et al.
ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs
by Seyedhamidreza Mousavi, Mohammad Hasan Ahmadilivani, Jaan Raik, Maksim Jenihhin, Masoud Daneshtalab
First submitted to arxiv on: 10 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Deep learning models are widely used in safety-critical applications where hardware reliability is crucial. To improve the resilience of these models against hardware faults, activation restriction techniques can be employed at the network structure level, regardless of the underlying accelerator architecture. State-of-the-art methods typically use neuron-wise or layer-wise clipping activation functions, with optimal threshold determination achieved through heuristic and learning-based approaches. However, these methods have limitations, including the inability to preserve DNN resilience at high bit error rates (layer-wise clipped) or introducing significant memory overhead (neuron-wise). Furthermore, heuristic-based optimization requires numerous fault injections during the search process, while learning-based techniques may yield sub-optimal results. This work proposes a hybrid clipped activation function that integrates neuron-wise and layer-wise methods, applying neuron-wise clipping only in the last layer of DNNs. Additionally, ProAct, a progressive training methodology, is introduced to iteratively train thresholds on a layer-by-layer basis for optimal threshold values. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Deep learning models are used in important applications where they must work even if there’s a problem with the hardware. To make these models more reliable, some techniques can be used that change how the model works without changing what it does. Current methods try to find the best way to do this by trying different things and seeing which one works best. However, these methods have some problems, such as not being able to handle very many mistakes or taking a long time to figure out what to do. This new method tries to solve these problems by combining different techniques together and training them in a special way. |
Keywords
» Artificial intelligence » Deep learning » Optimization
