Summary of Weight-based Decomposition: a Case For Bilinear Mlps, by Michael T. Pearce et al.
Weight-based Decomposition: A Case for Bilinear MLPs
by Michael T. Pearce, Thomas Dooms, Alice Rigg
First submitted to arxiv on: 6 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Gated Linear Units (GLUs) are a fundamental component in modern foundation models. Bilinear layers, which drop the non-linearity in the “gate,” demonstrate comparable performance to other GLUs while offering an attractive feature: they can be fully expressed as a third-order tensor and linear operations. Building upon this concept, researchers develop a method to decompose the bilinear tensor into sparse eigenvectors, showcasing promising interpretability properties in preliminary experiments with shallow image classifiers (MNIST) and small language models (Tiny Stories). This decomposition is fully equivalent to the original computations, making bilinear layers an attractive architecture for interpretability. The application of this method may not be limited to pretrained bilinar models, as researchers find that language models like TinyLlama-1.1B can be fine-tuned into bilinear variants. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Gated Linear Units are a type of building block in modern computer programs. Researchers have found a new way to make these units more understandable by breaking them down into smaller, simpler pieces. This new method works well with small image and language recognition tasks and shows promise for helping us understand how these models work. The good news is that this method can be applied not just to the original models but also to other related models. |
