Summary of Scaling Laws For Learning with Real and Surrogate Data, by Ayush Jain et al.
Scaling laws for learning with real and surrogate data
by Ayush Jain, Andrea Montanari, Eren Sasoglu
First submitted to arxiv on: 6 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper proposes an innovative approach to overcome the limitations of collecting large quantities of high-quality data in machine learning. The authors introduce the concept of “surrogate data,” which involves augmenting a small set of data points with data from more accessible sources, such as generative models or datasets collected under different circumstances. They develop a weighted empirical risk minimization (ERM) method for integrating surrogate data into training and analyze its effectiveness mathematically and empirically on various datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study shows that using surrogate data can significantly reduce the test error on the original distribution, even when the surrogate data is unrelated to the original ones. The authors attribute this surprising behavior to Stein’s paradox, a classical statistical phenomenon. To reap the benefits of surrogate data, it is crucial to use optimally weighted ERM and predict the optimal weighting scheme using a scaling law that describes the test error of models trained on mixtures of real and surrogate data. |
Keywords
* Artificial intelligence * Machine learning
