Summary of Tokens, the Oft-overlooked Appetizer: Large Language Models, the Distributional Hypothesis, and Meaning, by Julia Witte Zimmerman et al.
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
by Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer, Alejandro J. Ruiz, Calla Beauregard, Ashley Fehr, Mikaela Irene Fudolig, Bradford Demarest, Yoshi Meke Bird, Milo Z. Trujillo, Christopher M. Danforth, Peter Sheridan Dodds
First submitted to arxiv on: 14 Dec 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper examines the role of tokenization in transformer-based large language models (LLMs), including those from Generative AI. The authors argue that the Distributional Hypothesis (DH) is sufficient for human-like language performance, and that linguistically-informed interventions are needed to improve existing tokenization techniques. The study explores various tokenizations, including BPE, Hugging Face, tiktoken, and RoBERTa model vocabularies. The results show that suboptimal semantic building blocks can be created, obscuring the model’s access to distributional patterns. Additionally, the authors highlight how tokenization pretraining can introduce bias and unwanted content, which current alignment practices may not address. This research demonstrates the impact of tokenization on LLM cognition and emphasizes the need for linguistically-informed interventions. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how language models work and why they’re important. The authors say that these models can learn human-like language skills, but they need a special way to break down words into smaller units called tokens. Right now, this process is not very good, so it affects how well the model works. The study shows that if we do tokenization better, we can make the model more human-like and reduce problems like bias in the data. |
Keywords
» Artificial intelligence » Alignment » Pretraining » Tokenization » Transformer
