Summary of Mini-batch Coresets For Memory-efficient Language Model Training on Data Mixtures, by Dang Nguyen et al.
Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures
by Dang Nguyen, Wenhan Yang, Rathul Anand, Yu Yang, Baharan Mirzasoleiman
First submitted to arxiv on: 28 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper addresses a significant issue in training Large Language Models (LLMs) by developing an effective approach to find small mini-batch co-sets that closely match the gradient of larger mini-batches. This is crucial for achieving superior performance while reducing memory requirements. The authors propose CoLM, which leverages zeroth-order methods to find smooth gradients and sparsify them to keep dimensions with the largest normalized gradient magnitude. CoLM reduces memory requirements by 2x and even outperforms training with larger mini-batches on benchmarks like MathInstruct and SuperGLUE. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps make Large Language Models (LLMs) work better while using less computer memory. The problem is that LLMs need a lot of memory to train, but this makes them slow. To solve this, the authors developed a new way to find small groups of examples in the training data that are similar to the larger group. This helps the model learn faster and more efficiently. They tested their approach on different models and datasets and showed it can reduce memory usage by 2x while still achieving good results. |
