Summary of Efficient Local Linearity Regularization to Overcome Catastrophic Overfitting, by Elias Abad Rocamora et al.
Efficient local linearity regularization to overcome catastrophic overfitting
by Elias Abad Rocamora, Fanghui Liu, Grigorios G. Chrysos, Pablo M. Olmos, Volkan Cevher
First submitted to arxiv on: 21 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper aims to address the issue of catastrophic overfitting (CO) in single-step adversarial training (AT), which can result in a significant drop in adversarial test accuracy. The authors introduce a regularization term, called ELLE, that mitigates CO effectively and efficiently in classical AT evaluations as well as more challenging regimes such as large adversarial perturbations and long training schedules. The proposed method is computationally cheaper than previous methods and can be theoretically linked to the curvature of the loss function. The paper provides thorough experimental validation demonstrating that the proposed approach does not suffer from CO, even in challenging settings where previous works experience it. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper tackles a problem called catastrophic overfitting in single-step adversarial training. This means that when you train a model to be good at recognizing things that are slightly different from what you’ve seen before, it can suddenly become really bad at recognizing those same things if they’re just a little bit different. The authors propose a new way to prevent this from happening, called ELLE, which is more efficient and effective than other methods. They also show that their method works well in challenging situations where previous approaches didn’t work so well. |
Keywords
* Artificial intelligence * Loss function * Overfitting * Regularization
