Summary of I-llm: Efficient Integer-only Inference For Fully-quantized Low-bit Large Language Models, by Xing Hu et al.
I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
by Xing Hu, Yuan Cheng, Dawei Yang, Zhihang Yuan, Jiangyong Yu, Chen Xu, Sifan Zhou
First submitted to arxiv on: 28 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Post-training quantization (PTQ) is a technique used to accelerate inference in large language models (LLMs). However, existing works still require a significant number of floating-point operations during inference, including additional quantization and de-quantization. This limitation hinders the deployment of LLMs on edge and cloud devices. The authors identify that the primary obstacle to integer-only quantization for LLMs lies in the large fluctuation of activations across channels and tokens. To address this issue, they propose I-LLM, a novel integer-only fully-quantized PTQ framework tailored for LLMs. Specifically, they develop Fully-Smooth Block-Reconstruction (FSBR) to smooth inter-channel variations of all activations and weights, Dynamic Integer-only MatMul (DI-MatMul) to alleviate degradation caused by inter-token variations, and DI-ClippedSoftmax, DI-Exp, and DI-Normalization to execute non-linear operators efficiently. The experiment shows that I-LLM achieves comparable accuracy to the FP baseline and outperforms non-integer quantization methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making big language models run faster on devices like smartphones or computers. Right now, it’s hard to make these models work efficiently because they need a lot of special numbers called floating-point operations. The researchers found that the main problem is that the model’s “activations” (like the excitement level) can be very different across different parts of the model. To fix this, they created a new way to make the model work with only whole numbers, which is faster and uses less energy. They also developed special ways to deal with tricky math operations like squaring or normalizing. The results show that their method works just as well as using all those floating-point operations, but it’s much faster! |
Keywords
» Artificial intelligence » Inference » Quantization » Token
