Summary of Improving Ood Generalization Of Pre-trained Encoders Via Aligned Embedding-space Ensembles, by Shuman Peng et al.
Improving OOD Generalization of Pre-trained Encoders via Aligned Embedding-Space Ensembles
by Shuman Peng, Arash Khoeini, Sharan Vaswani, Martin Ester
First submitted to arxiv on: 20 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a solution to improve the generalization of self-supervised pre-trained embeddings on out-of-distribution (OOD) data without fine-tuning. The approach involves using deep ensembles, but finding an effective ensemble in the embedding space with only unlabeled data remains a challenge. The authors first analyze the relationship between individual hyperspherical embedding spaces in an ensemble and then design a principled method to align these spaces unsupervisedly. Experimental results on MNIST show that their method improves pre-trained embedding quality on both in-distribution and OOD data compared to single encoders. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper solves a problem with self-supervised pre-training. When you train an AI model on some data, it usually performs well on the same type of data. But what happens when you use it on different types of data? It often does poorly without more training. The authors suggest using “deep ensembles” to help. They also develop a new way to align these ensemble embeddings unsupervisedly. This leads to better results on both familiar and unfamiliar data. |
Keywords
» Artificial intelligence » Embedding » Embedding space » Fine tuning » Generalization » Self supervised
