Summary of Confidence Regulation Neurons in Language Models, by Alessandro Stolfo et al.
Confidence Regulation Neurons in Language Models
by Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, Neel Nanda
First submitted to arxiv on: 24 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This study delves into the uncertainty mechanisms underlying large language models (LLMs) next-token predictions. Researchers investigate two key components: entropy neurons and token frequency neurons. Entropy neurons, characterized by high weight norms, regulate LayerNorm scales to scale down logits. The study finds that entropy neurons operate by influencing residual stream norms through an unembedding null space, affecting logit outputs minimally. Entropy neurons are present across models up to 7 billion parameters. Token frequency neurons, discovered for the first time, boost or suppress token logits proportionally to their log frequencies, shifting output distributions towards or away from unigram distributions. The study presents a case study where entropy neurons manage confidence in induction settings, detecting and continuing repeated subsequences. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how large language models make predictions about the next word. It examines two special parts that help these models be more confident: entropy neurons and token frequency neurons. Entropy neurons control how sure the model is by adjusting a special scale. The study finds that these neurons work by changing the way the model’s internal calculations are done, without directly affecting what words it predicts. These neurons are found in many large language models. Token frequency neurons are new and help the model by making it more likely to choose certain words based on how often they appear. The paper also shows an example of how entropy neurons can be used to make a model better at detecting repeated patterns. |
Keywords
* Artificial intelligence * Logits * Token
