Summary of When Quantization Affects Confidence Of Large Language Models?, by Irina Proskurina et al.
When Quantization Affects Confidence of Large Language Models?
by Irina Proskurina, Luc Brun, Guillaume Metzler, Julien Velcin
First submitted to arxiv on: 1 May 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This study explores the effects of compressing Large Language Models (LLMs) through post-training quantization or low-bit weight representation. The authors investigate how quantization impacts the confidence and calibration of LLMs, considering factors like language model type and scale as contributors to quantization loss. They find that quantizing GPTQ models to 4-bit results in a decrease in confidence regarding true labels, with varying impacts observed among different language models. The study also reveals fluctuations in the impact on confidence across different scales. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how compressing big language models affects their predictions and how confident they are about those predictions. It finds that making these models smaller makes them less sure of themselves, especially for certain types of models and certain types of data. The study tries to explain why this happens and shows that it might be because the smaller models were already unsure about some things in the first place. |
Keywords
» Artificial intelligence » Language model » Quantization
