Summary of A Comparative Study Of Pruning Methods in Transformer-based Time Series Forecasting, by Nicholas Kiefer et al.
A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting
by Nicholas Kiefer, Arvid Weyrauch, Muhammed Öz, Achim Streit, Markus Götz, Charlotte Debus
First submitted to arxiv on: 17 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the application of pruning techniques on Transformer-based models for multivariate time series forecasting. Pruning helps reduce neural network parameter count, which is beneficial for real-world deployment on low-power embedded devices. The study compares unstructured and structured pruning methods on state-of-the-art models, evaluating their effects on predictive performance, model size, operations, and inference time. The results show that certain models can be pruned to high sparsity levels without compromising performance, but fine-tuning is necessary. Moreover, the paper finds that even with optimized hardware and software support, structured pruning does not lead to significant time savings. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how to make Transformer-based models for forecasting smaller and faster. These models are used a lot in real-world applications, but they can be too big and hungry for resources to use on devices like smartphones or embedded systems. The researchers test two ways of making the models smaller: unstructured pruning and structured pruning. They look at how these methods affect the performance of different forecasting models and how much faster they are compared to the original models. The results show that some models can be made very small without losing their ability to make good predictions, but the models still need to be fine-tuned after being pruned. Also, the researchers find that even with special hardware or software, structured pruning doesn’t help that much. |
Keywords
* Artificial intelligence * Fine tuning * Inference * Neural network * Pruning * Time series * Transformer
