Summary of Prilora: Pruned and Rank-increasing Low-rank Adaptation, by Nadav Benedek et al.
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation
by Nadav Benedek, Lior Wolf
First submitted to arxiv on: 20 Jan 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed PRILoRA method tackles the issue of large pre-trained language models (PLMs) requiring fine-tuning by introducing trainable rank decomposition matrices into each target module. This approach builds upon LoRA, but adds layer-wise linear allocation of ranks and pruning during training to account for varying importance. Extensive experiments on eight GLUE benchmarks demonstrate PRILoRA’s effectiveness, achieving a new state-of-the-art. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A team of researchers developed a new way to fine-tune large language models without using all the model’s parameters. They created an algorithm called PRILoRA that works by breaking down each part of the model into smaller pieces and adjusting how important those pieces are based on what data they’re seeing. This makes the process more efficient and uses less storage space. The team tested PRILoRA on eight different tasks and found it worked better than other methods, setting a new record. |
Keywords
* Artificial intelligence * Fine tuning * Lora * Pruning
