Summary of Pushing the Envelope Of Low-bit Llm Via Dynamic Error Compensation, by Yeonhong Park et al.
Pushing the Envelope of Low-Bit LLM via Dynamic Error Compensation
by Yeonhong Park, Jake Hyun, Hojoon Kim, Jae W. Lee
First submitted to arxiv on: 28 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed QDEC inference scheme improves the quality of low-bit Large Language Models (LLMs) while preserving quantization’s benefits: reduced GPU memory usage and inference latency. By storing residual matrices in CPU memory, QDEC dynamically fetches residuals for salient channels marked by activation outliers to correct quantization errors. This adaptation allows effective error compensation. The scheme outperforms state-of-the-art quantization methods, reducing the perplexity of a 3-bit Llama-3-8B-Instruct model from 10.15 to 9.12, while adding less than 0.0003% to GPU memory usage and incurring only a 1.7% inference slowdown. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary QDEC is a new way to make Large Language Models work better on devices with limited space or speed. Right now, we can make these models smaller by “quantizing” them, but this makes them not as good at understanding what they’re reading. QDEC fixes this problem by keeping extra information in the computer’s memory that helps correct mistakes made when the model is shrunk. This makes it better than before and works well even on devices with limited power. |
Keywords
» Artificial intelligence » Inference » Llama » Perplexity » Quantization
