Summary of Cats: Contextually-aware Thresholding For Sparsity in Large Language Models, by Donghyun Lee et al.
CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models
by Donghyun Lee, Je-Yong Lee, Genghan Zhang, Mo Tiwari, Azalia Mirhoseini
First submitted to arxiv on: 12 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces Contextually Aware Thresholding for Sparsity (CATS), a new framework to reduce the computational costs of Large Language Models (LLMs) while maintaining downstream task performance. CATS is designed to be simple, easy to implement, and highly effective, using a novel non-linear activation function at its core. The proposed method can be applied to various base models, including Mistral-7B and Llama2-7B, and outperforms existing sparsification techniques in downstream task performance. Specifically, CATS-based models often achieve performance within 1-2% of their base models without fine-tuning, even at activation sparsity levels of 50%. The paper also presents a custom GPU kernel implementation that translates the activation of sparsity to real wall-clock time speedups, resulting in a ~15% improvement in wall-clock inference latency for token generation on both Llama-7B and Mistral-7B. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper helps make Large Language Models (LLMs) more useful by making them faster and cheaper to use. It does this with a new way of reducing the amount of computing needed, called Contextually Aware Thresholding for Sparsity (CATS). This method is easy to use and works well, even when you’re trying to make LLMs do specific tasks. The paper shows that CATS can be used with different types of base models and that it’s better than other methods at getting the job done. |
Keywords
* Artificial intelligence * Fine tuning * Inference * Llama * Token
