Summary of Densing Law Of Llms, by Chaojun Xiao et al.
Densing Law of LLMs
by Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Biyuan Lin, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, Maosong Sun
First submitted to arxiv on: 5 Dec 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces a novel metric called “capacity density” to evaluate the quality of Large Language Models (LLMs) across different scales. It describes the trend of LLMs in terms of both effectiveness and efficiency, proposing a scaling law to predict downstream performance based on parameter sizes. The concept of effective parameter size is introduced, formalizing capacity density as the ratio of actual parameter size to effective parameter size required for equivalent performance. Capacity density provides a unified framework for assessing model effectiveness and efficiency. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how big language models are getting and how that affects their ability to be useful. It comes up with a new way to measure how good these models are, called “capacity density”. This helps us understand why bigger models aren’t always better. The paper also finds a rule (the densing law) that shows how capacity density changes over time – it doubles every three months! This can help guide the development of future language models. |
