Summary of Rolora: Fine-tuning Rotated Outlier-free Llms For Effective Weight-activation Quantization, by Xijie Huang et al.
RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization
by Xijie Huang, Zechun Liu, Shih-Yang Liu, Kwang-Ting Cheng
First submitted to arxiv on: 10 Jul 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Low-Rank Adaptation (LoRA) is a Parameter-Efficient Fine-Tuning (PEFT) method that updates only a small portion of the weights in Large Language Models (LLMs). Recent weight-only quantization techniques have improved training efficiency, but applying weight-activation quantization to LoRA pipelines is under-explored. This paper proposes RoLoRA, a novel scheme for effective weight-activation quantization using rotation for outlier elimination and rotation-aware fine-tuning. Experimental results show RoLoRA improves low-bit LoRA convergence and post-training quantization robustness across various LLaMA models, achieving up to 29.5% absolute accuracy gain on commonsense reasoning tasks compared to the LoRA baseline. The paper evaluates RoLoRA’s effectiveness on Large Multimodal Models (LLaVA-1.5-7B). Codes are available at this GitHub URL. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making computer models learn faster and more accurately. It proposes a new way to update only the most important parts of large language models, called RoLoRA. This helps improve how well the models work when they’re used in everyday applications like answering questions or generating text. The researchers tested RoLoRA on several different types of models and found that it worked better than other methods in certain situations. This could lead to more accurate and efficient computer models that can be used for many things, from helping people with tasks to creating new ideas. |
Keywords
» Artificial intelligence » Fine tuning » Llama » Lora » Low rank adaptation » Parameter efficient » Quantization
