Summary of Nomad-attention: Efficient Llm Inference on Cpus Through Multiply-add-free Attention, by Tianyi Zhang et al.
NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention
by Tianyi Zhang, Jonah Wonkyu Yi, Bowen Yao, Zhaozhuo Xu, Anshumali Shrivastava
First submitted to arxiv on: 2 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes an efficient attention algorithm called NoMAD-Attention, which leverages the Single-Instruction-Multiple-Data (SIMD) registers in Central Processing Units (CPU) to accelerate large language model inference. The algorithm replaces Multiply-Add (MAD) matrix operations with ultra-low-latency lookups in batch, achieving significant speedups without sacrificing quality. NoMAD-Attention is designed for pre-trained attention-based large language models like LLaMA-7B and can be used without model finetuning. Empirical evaluations show that the algorithm maintains the quality of the original models while speeding up computation by up to 2x at a context length of 16k. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary NoMAD-Attention is a new way to make big language models run faster on regular computers. The problem is that these models need lots of calculations, which can be slow. The idea is to use special registers in the computer’s processor to do these calculations really fast. This works well with pre-trained models and doesn’t require changing them at all. Tests show that this method keeps the quality of the original model while making it run up to 2 times faster. |
Keywords
* Artificial intelligence * Attention * Context length * Inference * Large language model * Llama
