Summary of Physics Of Language Models: Part 3.3, Knowledge Capacity Scaling Laws, by Zeyuan Allen-zhu and Yuanzhi Li
Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws
by Zeyuan Allen-Zhu, Yuanzhi Li
First submitted to arxiv on: 8 Apr 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates how language models scale with their capabilities by estimating the number of knowledge bits they store. Unlike previous studies that evaluate model performance using loss or benchmarks, this research focuses on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. The study uses multiple controlled datasets to establish a scaling law, demonstrating that language models can only store 2 bits of knowledge per parameter, even when quantized to int8. This knowledge can be flexibly extracted for downstream applications. Furthermore, the paper shows that a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Language models are getting bigger, but what do they know? This research tries to answer this question by counting how much factual information language models can store. Instead of looking at how well they perform on tests or benchmarks, the study looks at specific facts like (USA, capital, Washington D.C.) from a Wikipedia page. The results show that language models can only store 2 bits of knowledge per parameter, even when using special storage techniques. This means that a really big language model (7B parameters) can remember an enormous amount of information – actually more than all the text in English Wikipedia and textbooks combined! |
Keywords
» Artificial intelligence » Language model
