Summary of Features That Make a Difference: Leveraging Gradients For Improved Dictionary Learning, by Jeffrey Olmo et al.
Features that Make a Difference: Leveraging Gradients for Improved Dictionary Learning
by Jeffrey Olmo, Jared Wilson, Max Forsey, Bryce Hepner, Thomas Vin Howe, David Wingate
First submitted to arxiv on: 15 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers introduce Gradient SAEs (g-SAEs), a modified sparse autoencoder architecture that considers the effect of activations on downstream computations. Traditional SAEs only learn from activation values, which limits feature learning and biases the model towards neglecting features with small activation values but significant influence. g-SAEs augment the TopK activation function to rely on gradients when selecting elements, producing reconstructions more faithful to original network performance. This approach also learns latents that are more effective at steering models in arbitrary contexts. By accounting for both representation and action aspects of neural network features, g-SAEs represent a step towards more accurate feature discovery. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Gradient SAEs is a new way to make computers learn from pictures or words by looking at how those things affect what they do next. Right now, computers just look at the pictures or words themselves and don’t think about how they might be used later on. This makes it harder for them to learn really important features. The Gradient SAEs fix this problem by making the computer also look at how the pictures or words change things as it uses them. This helps the computer learn better features that can be used in lots of different situations. |
Keywords
» Artificial intelligence » Autoencoder » Neural network
