Summary of Inference Performance Optimization For Large Language Models on Cpus, by Pujiang He and Shan Zhou and Wenhuan Huang and Changqing Li and Duyi Wang and Bin Guo and Chen Meng and Sheng Gui and Weifei Yu and Yi Xie
Inference Performance Optimization for Large Language Models on CPUs
by Pujiang He, Shan Zhou, Wenhuan Huang, Changqing Li, Duyi Wang, Bin Guo, Chen Meng, Sheng Gui, Weifei Yu, Yi Xie
First submitted to arxiv on: 10 Jul 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents an inference performance optimization solution for large language models (LLMs) on CPUs, aiming to alleviate the financial burden and constraints imposed by hardware resources. The proposed approach reduces KV cache size while ensuring precision, and a distributed inference optimization method is implemented using oneAPI Collective Communications Library. Optimization techniques are tailored for commonly used LLMs, with an open-sourced code available at this URL. This work has significant potential in low-resource environments where GPU hardware is limited. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper solves a big problem. Large language models are really good at doing lots of tasks, but they need special computers to run fast. The problem is that not everyone has those computers. So, the researchers found a way to make these language models work faster on regular computers. They did this by making sure the computer uses its memory efficiently and by working together with other computers if needed. This will help people who don’t have super powerful computers use these language models too. |
Keywords
* Artificial intelligence * Inference * Optimization * Precision
