Summary of Continuous Approximations For Improving Quantization Aware Training Of Llms, by He Li et al.
Continuous Approximations for Improving Quantization Aware Training of LLMs
by He Li, Jianhang Hong, Yuanzhuo Wu, Snehal Adbol, Zonglin Li
First submitted to arxiv on: 6 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Model compression methods are used to reduce computation and energy requirements for Large Language Models (LLMs). The paper proposes Quantization Aware Training (QAT) as an effective method to reduce performance degradation after quantization. To further minimize this degradation, the authors introduce two continuous approximations to the QAT process on the rounding function, traditionally approximated by the Straight-Through Estimator (STE), and the clamping function. By applying both methods, the perplexity (PPL) on the WikiText-v2 dataset of the quantized model reaches 9.0815, outperforming 9.9621 by the baseline. Additionally, the authors achieve a 2.76% improvement on BoolQ, and a 5.47% improvement on MMLU, proving that the step sizes and weights can be learned more accurately with this approach. The method achieves better performance with the same precision, model size, and training setup. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Language Models (LLMs) need to be made more energy-efficient for widespread use. One way to do this is by compressing models using methods like Quantization Aware Training (QAT). This paper proposes new ways to improve QAT, making it better at reducing performance loss when models are compressed. The results show that these new approaches can reduce the difference in performance between the original and compressed models. |
Keywords
* Artificial intelligence * Model compression * Perplexity * Precision * Quantization
