Summary of Clover: Cross-layer Orthogonal Vectors Pruning and Fine-tuning, by Fanxu Meng et al.
CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning
by Fanxu Meng, Pingzhi Tang, Fan jiang, Muhan Zhang
First submitted to arxiv on: 26 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary CLOVER (Cross-Layer Orthogonal Vectors) is a novel approach to address memory-bound inference in decoder-only models. These models generate tokens autoregressively by caching key/value vectors, which can become memory-intensive as the cache grows. CLOVER treats pairs of attention layers as low-rank decompositions using Singular Value Decomposition (SVD). The resulting singular values guide pruning or serve as trainable parameters for efficient fine-tuning of orthogonal vectors. This approach is applied to various models, including GPT-2 XL, DeepSeek-V2-Lite, Whisper-Large-v3, Stable Diffusion XL, and LLaMA-3.2-11B-Vision. The results show that CLOVER significantly improves pruning efficiency, achieving similar perplexity to vanilla methods when pruning 70% of the Q-K pairs in GPT-2 XL. Fine-tuning the singular values further outperforms state-of-the-art methods on eight commonsense tasks for LLaMA-2 7B. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you have a super powerful computer that can quickly generate text, but it gets slower and uses more memory as it goes along. To solve this problem, scientists created a new way to make the computer work more efficiently called CLOVER. It takes two important parts of the computer and breaks them down into smaller pieces, making it easier for the computer to use less memory while still working well. They tested this method on several different text generators and found that it made them all work much faster and use less memory. |
Keywords
» Artificial intelligence » Attention » Decoder » Diffusion » Fine tuning » Gpt » Inference » Llama » Perplexity » Pruning
