Summary of Transformers Are Expressive, but Are They Expressive Enough For Regression?, by Swaroop Nath et al.
Transformers are Expressive, But Are They Expressive Enough for Regression?
by Swaroop Nath, Harshad Khadilkar, Pushpak Bhattacharyya
First submitted to arxiv on: 23 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Transformers have revolutionized Natural Language Processing, excelling in applications like Machine Translation and Summarization. Despite their widespread adoption, several works have attempted to analyze the expressivity of Transformers. Expressivity refers to the class of functions a neural network can approximate. A fully expressive neural network can act as a universal function approximator. Our study reveals that Transformers struggle to reliably approximate smooth functions, relying on piecewise constant approximations with sizable intervals. The central question emerges: “Are Transformers truly Universal Function Approximators?” To address this, we conducted a thorough investigation, providing theoretical insights and supporting evidence through experiments. We theoretically proved that Transformer Encoders cannot approximate smooth functions and experimentally showed that the full Transformer architecture cannot approximate smooth functions. By shedding light on these challenges, we advocate for a refined understanding of Transformers’ capabilities. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Transformers have been very good at helping computers understand language. They’re great at things like translating languages and making summaries. But some people have tried to figure out if they can do even more complicated things. One important question is: “Can Transformers be used for anything?” We wanted to find the answer, so we did a lot of research and experiments. We found that Transformers are not as good at doing some things as we thought. They’re better at making simple approximations than complex ones. This helps us understand what Transformers can really do. |
Keywords
* Artificial intelligence * Natural language processing * Neural network * Summarization * Transformer * Translation
