Summary of Sketchy Moment Matching: Toward Fast and Provable Data Selection For Finetuning, by Yijun Dong et al.
Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuning
by Yijun Dong, Hoang Phan, Xiang Pan, Qi Lei
First submitted to arxiv on: 8 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Numerical Analysis (math.NA); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper revisits data selection from a fundamental perspective, focusing on fine-tuning deep neural networks. The authors extend classical wisdom on variance minimization to high-dimensional settings, highlighting the importance of controlling bias induced by low-rank approximation. They introduce Sketchy Moment Matching (SkMM), a scalable scheme with two stages: first, gradient sketching explores the parameter space for an informative subspace; then, moment matching reduces variance over this subspace. The authors prove that gradient sketching is fast and accurate, preserving fast-rate generalization. Empirical results demonstrate the effectiveness of SkMM for fine-tuning in vision tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how to pick the right data to train a deep neural network. It’s all about finding the best way to make sure the network learns from good examples, not bad ones. The authors came up with a new method called Sketchy Moment Matching (SkMM) that works by first looking for patterns in the data and then choosing the best bits to use. This helps the network learn faster and more accurately. They tested their method on some vision tasks and it worked really well. |
Keywords
* Artificial intelligence * Fine tuning * Generalization * Neural network
