Summary of Taming Throughput-latency Tradeoff in Llm Inference with Sarathi-serve, by Amey Agrawal et al.
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
by Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee
First submitted to arxiv on: 4 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Distributed, Parallel, and Cluster Computing (cs.DC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper examines the internal workings of language models’ (LLMs) serving mechanisms, focusing on two distinct phases: prefill and decode. Prefill processes the entire input prompt, producing the first output token, while decode generates subsequent tokens one-at-a-time. Notably, prefill iterations exhibit high latency due to parallel processing of the input prompt, whereas decode iterations demonstrate low latency but limited compute utilization. This disparity highlights the benefits of batching for decodes, enhancing overall throughput. However, batching raises challenges in balancing high throughput and low latency. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Language models have two main steps: prefill and decode. The prefill part does a lot of work upfront to produce one output token. It takes a little longer because it processes the whole input prompt at once. On the other hand, the decode step is much faster but only does a small amount of work, processing one token at a time. This difference makes batching helpful for decoding, which improves overall speed. However, combining these steps can make it hard to achieve both high speed and low latency. |
Keywords
* Artificial intelligence * Prompt * Token
