Summary of On Discriminative Probabilistic Modeling For Self-supervised Representation Learning, by Bokun Wang and Yunwen Lei and Yiming Ying and Tianbao Yang
On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning
by Bokun Wang, Yunwen Lei, Yiming Ying, Tianbao Yang
First submitted to arxiv on: 11 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper studies discriminative probabilistic modeling on a continuous domain for self-supervised representation learning. The authors leverage the multiple importance sampling (MIS) technique to compute the integral in the partition function, which can recover InfoNCE-based contrastive loss as a special case. They conduct generalization error analysis to reveal limitations of current InfoNCE-based approaches and derive insights for developing better methods by reducing Monte Carlo integration errors. The authors propose a novel non-parametric method for approximating conditional probability densities using convex optimization, yielding a new contrastive objective for self-supervised representation learning. An efficient algorithm is designed to solve this objective, and experimental results on the CC3M and CC12M datasets demonstrate superior overall performance compared to representative baselines. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper uses math to help computers learn from themselves without labeled data. They use a technique called multiple importance sampling to make sure the computer’s training process is accurate. The authors then analyze how well this method works and find some flaws in current methods. They develop a new way to fix these issues by using something called convex optimization, which helps the computer learn better representations of images and text. This is tested on real data and shows that their new method performs better than existing ones. |
Keywords
» Artificial intelligence » Contrastive loss » Generalization » Optimization » Probability » Representation learning » Self supervised
