Summary of Datacomp-lm: in Search Of the Next Generation Of Training Sets For Language Models, by Jeffrey Li et al.
DataComp-LM: In search of the next generation of training sets for language models
by Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, Vaishaal Shankar
First submitted to arxiv on: 17 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces DataComp for Language Models (DCLM), a testbed for controlled experiments to improve language models. The authors provide a standardized corpus, pretraining recipes based on OpenLM, and 53 downstream evaluations. Participants can experiment with dataset curation strategies at various model scales. The baseline DCLM-Baseline enables training a 7B parameter language model from scratch, outperforming previous state-of-the-art models MAP-Neo and Mistral-7B-v0.3 on the MMLU benchmark while using less compute. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about creating a special tool for testing language models. It has a big dataset with lots of text that can be used to train these models. The authors also share some tips on how to make the training process better, and they compare their results to other similar projects. The goal is to help improve language models by understanding what makes them work well or poorly. |
Keywords
» Artificial intelligence » Language model » Pretraining
