Summary of Quarot: Outlier-free 4-bit Inference in Rotated Llms, by Saleh Ashkboos et al.
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
by Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, James Hensman
First submitted to arxiv on: 30 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary We present QuaRot, a novel Quantization scheme based on Rotations that enables end-to-end quantization of Large Language Models (LLMs) in 4 bits. This approach rotates LLMs to remove outliers from the hidden state without altering the output, simplifying quantization. QuaRot applies this computational invariance to the residual and activations of feed-forward components, attention mechanisms, and KV cache. The resulting 4-bit quantized model performs matrix multiplications in 4 bits without retaining any channels in higher precision. Our 4-bit quantized LLaMa2-70B model achieves losses of up to 0.47 WikiText-2 perplexity and retains 99% of zero-shot performance. We also demonstrate lossless 6 and 8 bit LLaMa2 models using round-to-nearest quantization without calibration data. The code is available at: https://github.com/spcl/QuaRot. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary We created a new way to make big language models smaller, called QuaRot. This helps us understand what’s inside the model and makes it easier to use on devices with limited power. Our method, QuaRot, removes things that don’t matter from the hidden state of the model without changing its output. We applied this idea to different parts of the model, including how it learns new words and how it remembers old ones. The result is a smaller language model that still works well and can be used on devices with limited power. |
Keywords
» Artificial intelligence » Attention » Language model » Perplexity » Precision » Quantization » Zero shot
