Summary of From Tokens to Words: on the Inner Lexicon Of Llms, by Guy Kaplan et al.
From Tokens to Words: On the Inner Lexicon of LLMs
by Guy Kaplan, Matanel Oren, Yuval Reif, Roy Schwartz
First submitted to arxiv on: 8 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The abstract presents research on natural language processing by large language models (LLMs). It reveals that LLMs have an intrinsic detokenization process where sub-word sequences are combined into coherent whole-word representations. This process occurs primarily within the early and middle layers of the model, allowing it to recognize words even when they’re split or contain typos. The research also shows that LLMs can understand out-of-vocabulary words without additional training. These findings provide a practical application for expanding pre-trained models’ vocabulary without sacrificing accuracy. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models (LLMs) are very smart computers that can understand human language. They break down words into smaller parts called sub-words, but sometimes they need to put these sub-words back together to make sense of the whole word. This research shows how LLMs do this and why it’s important for them to be able to recognize words even when they’re broken or misspelled. The study also reveals that LLMs can understand new words they’ve never seen before, which is useful because it means they don’t need extra training to learn these new words. |
Keywords
» Artificial intelligence » Natural language processing
