Summary of Hashevict: a Pre-attention Kv Cache Eviction Strategy Using Locality-sensitive Hashing, by Minghui Liu et al.
HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing
by Minghui Liu, Tahseen Rabbani, Tony O’Halloran, Ananth Sankaralingam, Mary-Anne Hartley, Brian Gravelle, Furong Huang, Cornelia Fermüller, Yiannis Aloimonos
First submitted to arxiv on: 13 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Data Structures and Algorithms (cs.DS); Performance (cs.PF)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces HashEvict, an algorithm that uses locality-sensitive hashing (LSH) to compress the key-value (KV) cache of transformer-based large language models (LLMs). This compression is achieved by computing the Hamming distance between binarized Gaussian projections of query and cached token keys. The authors show that HashEvict can reduce the KV cache size by 30-70% while maintaining high performance on reasoning, multiple-choice, long-context retrieval, and summarization tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper shows a new way to make large language models work faster on computers. It uses a special kind of memory called locality-sensitive hashing (LSH) to help the model remember things it has already seen. This makes the model use less computer power and can do more tasks like answering questions, choosing answers, and summarizing text. |
Keywords
» Artificial intelligence » Summarization » Token » Transformer
