Summary of Balanced Data Sampling For Language Model Training with Clustering, by Yunfan Shao et al.
Balanced Data Sampling for Language Model Training with Clustering
by Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, Xipeng Qiu
First submitted to arxiv on: 22 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper addresses the open question of determining the data sampling strategy in training Large Language Models (LLMs). Most LLMs are trained using a simple random sampling approach, but this ignores the unbalanced nature of training data distribution. The authors propose ClusterClip Sampling to balance text distribution and improve model training. This method utilizes data clustering to reflect the training set’s data distribution and balances common and rare samples based on cluster results. A repetition clip operation is introduced to mitigate overfitting from certain clusters. Experimental results demonstrate the effectiveness of ClusterClip Sampling, outperforming random sampling and other variants under various training datasets and LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Language Models (LLMs) need data to learn. Right now, we use a simple way to pick which pieces of text to train with – it’s called random sampling. But this approach doesn’t take into account how the data is distributed. Some texts are very common, while others are rare. This paper suggests a new way to sample data, called ClusterClip Sampling. It groups similar texts together and balances the number of common and rare texts used for training. This helps prevent the model from getting too good at recognizing patterns in some types of texts and forgetting others. The results show that this method works better than random sampling and other approaches. |
Keywords
» Artificial intelligence » Clustering » Overfitting
