Summary of On Pretraining Data Diversity For Self-supervised Learning, by Hasan Abed Al Kader Hammoud et al.
On Pretraining Data Diversity for Self-Supervised Learning
by Hasan Abed Al Kader Hammoud, Tuhin Das, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem
First submitted to arxiv on: 20 Mar 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary We investigate the effects of training self-supervised learning (SSL) models with diverse datasets, characterized by unique sample counts, under a fixed computational budget. Our results consistently show that increasing pretraining data diversity improves SSL performance, but only when the distribution distance to downstream data is minimal. Interestingly, even with exceptionally large pretraining data diversities achieved through methods like web crawling or diffusion-generated data, distribution shift remains a challenge. We conducted comprehensive experiments using seven SSL methods and large-scale datasets such as ImageNet and YFCC100M, amounting to over 200 GPU days. Our code and trained models are available at https://github.com/hammoudhasan/DiversitySSL. This paper explores the role of diversity in self-supervised learning and its implications for real-world applications. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study looks at how using more diverse training data affects how well self-supervised learning (SSL) works. We found that having a lot of different training examples helps SSL perform better, but only if the type of data used to train it is similar to what we want it to learn from in the end. Even with really big and diverse training sets, there’s still a problem with adapting to new types of data. To test this, we tried seven different SSL methods using big datasets like images and videos, which took over 200 days to do on our computers. You can find the code and trained models used in this study at https://github.com/hammoudhasan/DiversitySSL. |
Keywords
* Artificial intelligence * Diffusion * Pretraining * Self supervised
