Summary of Algorithmic Capabilities Of Random Transformers, by Ziqian Zhong et al.
Algorithmic Capabilities of Random Transformers
by Ziqian Zhong, Jacob Andreas
First submitted to arxiv on: 6 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Transformed transformer models have been found to implement interpretable procedures for tasks like arithmetic and associative recall, but little is understood about how the circuits that implement these procedures originate during training. To what extent do they depend on the supervisory signal provided to models, and to what extent are they attributable to behavior already present in models at the beginning of training? This paper investigates what functions can be learned by randomly initialized transformers, finding that these random transformers can perform a wide range of meaningful algorithmic tasks, including modular arithmetic, in-weights and in-context associative recall, decimal addition, parenthesis balancing, and even some aspects of natural language text generation. The results indicate that some algorithmic capabilities are present in transformers (and accessible via appropriately structured inputs) even before these models are trained. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Transformers can do math! They’re really good at it too. But how do they learn to do this? Do they need someone telling them what’s right or wrong, or is there already something inside them that lets them figure things out? This paper looks at transformers when they start with nothing, just like a blank slate. It finds that these “random” transformers can do all sorts of math problems, like adding numbers together and balancing parentheses. They even try generating some text! The results show that transformers have some built-in math skills from the very beginning. |
Keywords
» Artificial intelligence » Recall » Text generation » Transformer
