Summary of Task-adaptive Pretrained Language Models Via Clustered-importance Sampling, by David Grangier et al.
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
by David Grangier, Simin Fan, Skyler Seto, Pierre Ablin
First submitted to arxiv on: 30 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces a novel method for building specialist language models from large generalist training sets, rather than relying on limited domain-specific data. The proposed approach, called ClusteRed Importance SamPling (CRISP), clusters the generalist dataset and samples from these clusters based on their frequencies in the smaller specialist dataset. This allows CRISP to be scalable and suitable for both pretraining and continued pretraining, as well as multi-task settings. Compared to other methods that adjust the training distribution of the generalist data with guidance from limited domain-specific data, CRISP performs favorably in terms of language modeling perplexity and accuracy on multiple-choice question tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates specialist language models by using a big training set instead of small data specific to one task. The method is called ClusteRed Importance SamPling (CRISP). It groups the big dataset into clusters and picks from these clusters based on how often words appear in the smaller task-specific data. This makes CRISP good for pretraining, continued pretraining, and doing many tasks at once. The results show that CRISP does better than other methods when it comes to language modeling and getting answers right on multiple-choice questions. |
Keywords
» Artificial intelligence » Multi task » Perplexity » Pretraining
