Summary of Excp: Extreme Llm Checkpoint Compression Via Weight-momentum Joint Shrinking, by Wenshuo Li et al.
ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking
by Wenshuo Li, Xinghao Chen, Han Shu, Yehui Tang, Yunhe Wang
First submitted to arxiv on: 17 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Extreme Checkpoint Compression (ExCP) framework reduces the required storage of training checkpoints for large language models while maintaining nearly lossless performance. The approach calculates residuals between adjacent checkpoints to extract essential information, and then applies a weight-momentum joint shrinking method to discard redundant parameters. Non-uniform quantization is also employed to further compress checkpoint storage. Experiments on various models demonstrate significant storage reduction (up to 70x) with preserved performance on downstream tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are getting attention in AI, but training them takes up a lot of computer power and space. To solve this problem, researchers created a new way to store model “checkpoints” – the points where the model saves its progress during training. This new method, called Extreme Checkpoint Compression (ExCP), can store these checkpoints more efficiently without losing any important information. It works by comparing nearby checkpoint points and throwing away the extra details that aren’t needed. The result is a much smaller storage size while still keeping the model’s performance accurate. |
Keywords
» Artificial intelligence » Attention » Quantization
