Summary of Gptvq: the Blessing Of Dimensionality For Llm Quantization, by Mart Van Baalen et al.
GPTVQ: The Blessing of Dimensionality for LLM Quantization
by Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, Paul Whatmough
First submitted to arxiv on: 23 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed GPTVQ method significantly improves the size-versus-accuracy trade-off of neural network quantization by increasing the quantization dimensionality. This method interleaves the quantization of one or more columns with updates to the remaining unquantized weights, leveraging the Hessian of the per-layer output reconstruction MSE. The codebooks are initialized using an efficient data-aware version of the EM algorithm and then updated for further compression via integer quantization and SVD-based methods. GPTVQ achieves state-of-the-art results on a wide range of Large Language Models (LLMs), including Llama-v2 and Mistral, while also being efficient in processing times. On a single H100 processor, it takes between 3 to 11 hours to process a Llamav2-70B model depending on the quantization setting. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary GPTVQ is a new way to make neural networks smaller and more accurate. Normally, as you try to make a network smaller, its performance gets worse. But GPTVQ shows that if you increase the size of the “quantization dimensionality,” you can actually improve the accuracy while keeping the size small. This method works by updating the weights in the network in a special way, using information about how well the network is doing. The codebooks used to store the quantized weights are initialized and updated efficiently. GPTVQ beats other methods on many different types of language models, like Llama-v2 and Mistral. |
Keywords
* Artificial intelligence * Llama * Mse * Neural network * Quantization
