Summary of Imagenet-rib Benchmark: Large Pre-training Datasets Don’t Always Guarantee Robustness After Fine-tuning, by Jaedong Hwang et al.
ImageNet-RIB Benchmark: Large Pre-Training Datasets Don’t Always Guarantee Robustness after Fine-Tuning
by Jaedong Hwang, Brian Cheung, Zhang-Wei Hong, Akhilan Boopathy, Pulkit Agrawal, Ila Fiete
First submitted to arxiv on: 28 Oct 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces a new benchmark, ImageNet-RIB (Robustness Inheritance Benchmark), to assess the robustness of pre-trained models fine-tuned for specialized tasks. The goal is to achieve both specialization and maintain robustness by starting from a good general-purpose model. To measure this, the authors iterate across multiple downstream datasets, fine-tuning on one and assessing on others. The study finds that despite continual learning methods helping maintain robustness, fine-tuning generally reduces generalization performance on related tasks. Interestingly, models are most fragile when fine-tuned on datasets with the richest and most diverse pre-training data. This suggests that starting with the strongest foundation model may not be the best approach for specialist tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research paper is about testing how well big AI models can adapt to new tasks while still being good at their original job. The researchers created a special test to see if these models can handle different kinds of data without getting confused. They found that even with extra training, the models often don’t do as well on similar but slightly different tasks. Surprisingly, they also discovered that when given very diverse and rich training data, the models actually become less robust and more prone to mistakes. This could change how we develop AI models for specific jobs. |
Keywords
» Artificial intelligence » Continual learning » Fine tuning » Generalization
