Summary of Oh! We Freeze: Improving Quantized Knowledge Distillation Via Signal Propagation Analysis For Large Language Models, by Kartikeya Bhardwaj et al.
Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models
by Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi, Kyunggeun Lee, Jun Ma, Harris Teague
First submitted to arxiv on: 26 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel technique is proposed to improve the performance of large language models (LLMs) on edge devices by applying knowledge distillation (KD-QAT) for 4-bit weight quantization. The authors study the stability of KD-QAT during training and propose ov-freeze, a simple method to stabilize the process. This approach enables near-floating-point-precision performance at 4-bit quantization level, with only a 0.7% loss in accuracy on Commonsense Reasoning benchmarks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are super powerful! But they can be slow and use too much memory. To fix this, scientists came up with a clever way to make them smaller while still keeping them good at understanding human language. They used something called knowledge distillation to teach the small model to behave like a big one. By doing this, they were able to shrink the model down to just 4 bits (which is really tiny!) and still keep it super accurate. |
Keywords
» Artificial intelligence » Knowledge distillation » Precision » Quantization
