Summary of Distill the Best, Ignore the Rest: Improving Dataset Distillation with Loss-value-based Pruning, by Brian B. Moser et al.
Distill the Best, Ignore the Rest: Improving Dataset Distillation with Loss-Value-Based Pruning
by Brian B. Moser, Federico Raue, Tobias C. Nauen, Stanislav Frolov, Andreas Dengel
First submitted to arxiv on: 18 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel framework called “Prune First, Distill After” that improves dataset distillation by systematically pruning datasets via loss-based sampling prior to distillation. The framework combines pruning with classical distillation techniques and generative priors to create a representative core-set that enhances generalization for unseen architectures. Experimental results show that the proposed method significantly boosts distilled quality, achieving up to a 5.2 percentage points accuracy increase even with substantial dataset pruning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making it easier to train artificial intelligence models by reducing the amount of data they need to learn from. The problem is that many datasets contain unnecessary information that can actually make it harder for the model to learn. To solve this, the researchers came up with a new way of preparing datasets called “Prune First, Distill After”. This method helps get rid of unimportant parts of the dataset before training the model, which makes the model better at generalizing to new situations. |
Keywords
» Artificial intelligence » Distillation » Generalization » Pruning
