Summary of Layer-adaptive State Pruning For Deep State Space Models, by Minseon Gwak et al.
Layer-Adaptive State Pruning for Deep State Space Models
by Minseon Gwak, Seongrok Moon, Joohwan Ko, PooGyeon Park
First submitted to arxiv on: 5 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Systems and Control (eess.SY)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper presents a novel method for pruning deep state space models (SSMs), called Layer-Adaptive STate pruning (LAST). The proposed approach reduces the state dimension of each layer in SSMs by minimizing model-level output energy loss. LAST scores are evaluated using the H infinity norms of subsystems and layer-wise energy normalization, serving as global pruning criteria for cross-layer comparison of states and adaptive pruning. Experimental results show that pruning 33% of states maintains performance with minimal accuracy loss (0.52%) in multi-input multi-output SSMs without retraining. The paper’s contribution is the development of a structured pruning method for SSMs, which optimizes previous models, revealing redundancy and compressibility in their state spaces. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research paper helps make deep learning models more efficient by removing unnecessary parts called “states” from the model. The authors developed a new way to do this called Layer-Adaptive STate pruning (LAST). LAST makes sure that the remaining states are still working well together, which is important for the model’s performance. By reducing the number of states, the model becomes faster and uses less energy without losing too much accuracy. The researchers tested their method on different types of data and found that it can work with most models to make them more efficient. |
Keywords
* Artificial intelligence * Deep learning * Pruning
