Summary of Enhancing Diversity in Bayesian Deep Learning Via Hyperspherical Energy Minimization Of Cka, by David Smerkous et al.
Enhancing Diversity in Bayesian Deep Learning via Hyperspherical Energy Minimization of CKA
by David Smerkous, Qinxun Bai, Fuxin Li
First submitted to arxiv on: 31 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the use of Centered Kernel Alignment (CKA) as an optimization objective in Bayesian deep learning. The goal is to generate diverse ensembles and hypernetworks that output a network posterior, which requires a similarity metric that preserves permutation invariance. CKA on feature kernels has been used for comparing deep networks, but not in the context of Bayesian deep learning. The paper proposes adapting CKA with hyperspherical energy (HE) to address diminishing gradients when two networks are very similar and improve training stability. Additionally, it derives feature repulsive terms using CKA-based feature kernels and synthetically generated outlier examples. Experiments show that this approach significantly outperforms baselines in uncertainty quantification for both synthetic and realistic outlier detection tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper uses a special method to compare deep learning networks. It wants to find networks that are similar but not the same, so it creates a way to measure how close they are. This is important because deep learning networks can be very big and complex, so we need ways to understand them better. The new approach combines two ideas: one that makes sure the similarity metric works for all network structures, and another that helps when the networks are very similar. It also adds some extra steps to make the training process more stable. This is useful because it can help us create better models that predict things accurately. |
Keywords
» Artificial intelligence » Alignment » Deep learning » Optimization » Outlier detection
