Summary of Deft: Decoding with Flash Tree-attention For Efficient Tree-structured Llm Inference, by Jinwei Yao et al.
DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference
by Jinwei Yao, Kaiqi Chen, Kexun Zhang, Jiaxuan You, Binhang Yuan, Zeke Wang, Tao Lin
First submitted to arxiv on: 30 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large language models are increasingly used for complex tasks that process multiple generation calls in a tree structure with shared prefixes of tokens. However, existing inference systems for tree-based applications are inefficient due to improper partitioning of queries and KV cache during attention calculation. To address these challenges, the proposed DeFT algorithm reduces the number of read/write operations of KV cache during attention calculation through KV-Guided Grouping, a method that avoids repeatedly loading KV cache of shared prefixes in attention computation. Additionally, Flattened Tree KV Splitting ensures even distribution of the KV cache across partitions with little computation redundancy. By reducing 73-99% KV cache IO and nearly 100% IO for partial results during attention calculation, DeFT achieves up to 2.23/3.59x speedup in end-to-end/attention latency compared to state-of-the-art attention algorithms. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are used for complex tasks that process many pieces of information at once. The problem is that the current systems aren’t very efficient because they don’t handle the information properly. To solve this, a new algorithm called DeFT was developed. It helps by reducing the amount of information that needs to be loaded and processed during attention calculation. This makes it faster than other algorithms, with some tasks completing up to 2.23 or 3.59 times faster. |
Keywords
» Artificial intelligence » Attention » Inference
