Summary of Localizing Memorization in Ssl Vision Encoders, by Wenhao Wang et al.
Localizing Memorization in SSL Vision Encoders
by Wenhao Wang, Adam Dziedzic, Michael Backes, Franziska Boenisch
First submitted to arxiv on: 27 Sep 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates memorization in self-supervised learning (SSL) encoders, which are trained on large datasets without labels. Despite being trained on millions of images, these encoders still memorize individual data points. The authors propose two metrics to localize memorization within the encoder: layermem and unitmem. These metrics can be computed during a forward pass and do not require any label information. By applying their methods to various encoder architectures and datasets, they find that (1) memorization increases with layer depth, but highly memorizing units are distributed across the entire encoder; (2) many units in SSL encoders experience high memorization of individual data points; (3) atypical data points cause higher layer and unit memorization than standard data points; and (4) vision transformers have most memorization in fully-connected layers. The authors also show that localizing memorization can improve fine-tuning and inform pruning strategies. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks into how self-supervised learning (SSL) models remember certain things even when they’re not labeled. SSL models are trained on huge amounts of data without labels, but they still “remember” individual images. The researchers came up with two ways to figure out where this remembering happens inside the model: layermem and unitmem. These methods can be used while the model is running and don’t need any labeled information. By testing these methods on different models and datasets, they found some interesting things. For example, (1) as you go deeper into the model, it remembers more; (2) many parts of the model remember individual images; (3) weird or outlier images make the model remember even more; and (4) a certain type of model called a vision transformer has most of its remembering happening at the end. The researchers also showed that understanding where the model remembers things can help it work better. |
Keywords
» Artificial intelligence » Encoder » Fine tuning » Pruning » Self supervised » Vision transformer
