Summary of Why Is Sam Robust to Label Noise?, by Christina Baek et al.
Why is SAM Robust to Label Noise?
by Christina Baek, Zico Kolter, Aditi Raghunathan
First submitted to arxiv on: 6 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper presents a study on Sharpness-Aware Minimization (SAM), a technique known for achieving state-of-the-art performances on natural image and language tasks. However, its most significant improvements are seen in the presence of label noise, with tens of percent gains. The authors decompose SAM’s robustness into two effects: changes to the logit term and changes to the network Jacobian. They find that the first effect is observable in linear logistic regression, but surprisingly, removing this effect does not degrade performance in neural networks. Instead, they attribute SAM’s effectiveness to its impact on the network Jacobian, which induces implicit regularization. The authors theoretically derive this effect in two-layer linear networks and propose cheaper alternatives to SAM that recover benefits in deep networks trained on real-world datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about a new way to train AI models called Sharpness-Aware Minimization (SAM). It’s already been shown to work really well on certain types of tasks. But what’s interesting is that it gets even better when there are mistakes in the training data. The researchers looked at how SAM makes this happen and found that it’s not just because it’s good at ignoring errors, but because it changes the way the model is trained overall. They also came up with some simpler versions of SAM that still work well. |
Keywords
» Artificial intelligence » Logistic regression » Regularization » Sam
