Summary of Aflora: Adaptive Freezing Of Low Rank Adaptation in Parameter Efficient Fine-tuning Of Large Models, by Zeyu Liu et al.
AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models
by Zeyu Liu, Souvik Kundu, Anni Li, Junrui Wan, Lianghao Jiang, Peter Anthony Beerel
First submitted to arxiv on: 20 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed AFLoRA method is a novel parameter-efficient fine-tuning technique that achieves state-of-the-art performance on the GLUE benchmark while reducing computational requirements by up to 9.5 times. The approach involves adding low-rank matrices with trainable projection paths, which are incrementally frozen during training based on a novel freezing score. This leads to an average improvement of up to 0.85% compared to existing methods. Additionally, AFLoRA can result in up to 1.86 times faster runtime compared to similar PEFT alternatives. The paper provides insights into the trainability requirements of LoRA paths at different modules and the optimal freezing schedule for projection matrices. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary AFLoRA is a new way to fine-tune models that makes them better without using too many resources. This helps reduce over-fitting, which means the model won’t get too good at fitting the training data and then perform poorly on new, unseen data. The method uses special low-rank matrices that can be adjusted during training to help the model learn more efficiently. As a result, AFLoRA achieves better performance than other similar methods while using less computational power. |
Keywords
* Artificial intelligence * Fine tuning * Lora * Parameter efficient
