Summary of Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws, by Yiding Jiang et al.
Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws
by Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, J. Zico Kolter
First submitted to arxiv on: 15 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces Adaptive Data Optimization (ADO), an algorithm that optimizes the composition of pretraining data in real-time during model training. ADO estimates the learning potential of each domain using per-domain scaling laws and adjusts the data mixture accordingly, making it more scalable and efficient than existing methods. The authors demonstrate that ADO can achieve comparable or better performance while reducing computational complexity, offering a practical solution for dynamic data distribution adjustment. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper is about creating an algorithm to help train models by optimizing the mix of different types of data they use. It’s like finding the right combination of ingredients to make a cake. The algorithm, called ADO, uses special rules to figure out how well each type of data helps the model learn and adjusts the mix as it trains. This makes it more efficient and easier to use than other methods. The results show that ADO can do just as well or better than current methods while using less computer power. |
Keywords
» Artificial intelligence » Optimization » Pretraining » Scaling laws
