Summary of The Hidden Pitfalls Of the Cosine Similarity Loss, by Andrew Draganov et al.
The Hidden Pitfalls of the Cosine Similarity Loss
by Andrew Draganov, Sharvaree Vadgama, Erik J. Bekkers
First submitted to arxiv on: 24 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores two under-explored settings where the gradient of the cosine similarity between two points approaches zero, namely when a point has large magnitude or when points are on opposite ends of the latent space. It surprisingly shows that optimizing cosine similarity forces points to grow in magnitude, making the first setting unavoidable in practice. The findings are generalizable across deep learning architectures and standard self-supervised learning loss functions, leading to the proposal of cut-initialization, a simple network initialization change that accelerates convergence for various SSL methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper investigates two unusual situations where the relationship between points becomes zero. If a point is very big or if it’s far from another point on the other side of the space, something interesting happens. The study shows that trying to make these points similar actually makes them grow bigger. This means that the first situation always occurs in real-life applications. The research also finds that this phenomenon applies to many different types of artificial intelligence models and learning techniques. |
Keywords
» Artificial intelligence » Cosine similarity » Deep learning » Latent space » Self supervised
