Summary of Towards Continual Learning Desiderata Via Hsic-bottleneck Orthogonalization and Equiangular Embedding, by Depeng Li et al.
Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding
by Depeng Li, Tianqi Wang, Junwei Chen, Qining Ren, Kenji Kawaguchi, Zhigang Zeng
First submitted to arxiv on: 17 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed method for continual learning addresses the issue of catastrophic forgetting in deep neural networks when trained on sequential tasks without access to previous training data or network expansion. The approach attributes forgetting to layer-wise parameter overwriting and decision boundary distortion, which is mitigated by two key components: HSIC-Bottleneck Orthogonalization (HBO) and EquiAngular Embedding (EAE). HBO implements non-overwritten parameter updates in an orthogonal space mediated by Hilbert-Schmidt independence criterion. EAE enhances decision boundary adaptation between old and new tasks with predefined basis vectors. The method demonstrates competitive accuracy performance, even without using exemplar buffers or network expansion. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper finds a way to help deep neural networks learn new things without forgetting what they already know. This is important because it’s hard for AI models to remember what they learned in the past when they’re trained on lots of different tasks. The researchers came up with a simple yet effective method that works even when you don’t have access to all the old training data or can’t make the model bigger. They use two key ideas: one helps keep track of important information and the other helps adjust how the model makes decisions. |
Keywords
* Artificial intelligence * Continual learning * Embedding
