Summary of Reducing Transformer Key-value Cache Size with Cross-layer Attention, by William Brandon et al.
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
by William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan Kelly
First submitted to arxiv on: 21 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper explores strategies to reduce the memory requirements of key-value (KV) caching in transformer-based autoregressive large language models (LLMs), particularly at long sequence lengths and large batch sizes. Specifically, it investigates the effectiveness of Multi-Query Attention (MQA) and its generalization, Grouped-Query Attention (GQA), which modify the attention block design to reduce the number of distinct key/value heads. The authors then propose a new approach called Cross-Layer Attention (CLA), which shares key and value heads between adjacent layers, further reducing the KV cache size while maintaining accuracy. Experimental results demonstrate that CLA offers a Pareto improvement over traditional MQA, enabling inference with longer sequence lengths and larger batch sizes. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at ways to make big language models work faster on bigger computers. Right now, these models need a lot of memory (like how much RAM you have on your computer) to do their job well. The researchers found two ways that help: Multi-Query Attention (MQA) and Grouped-Query Attention (GQA). They both make the model use less memory while still working pretty well. Then, they came up with a new way called Cross-Layer Attention (CLA) that makes it even better! With CLA, we can do more complex tasks like long text analysis or larger language processing jobs without needing as much computer power. |
Keywords
» Artificial intelligence » Attention » Autoregressive » Generalization » Inference » Transformer
