Summary of The Unreasonable Effectiveness Of Easy Training Data For Hard Tasks, by Peter Hase et al.
The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
by Peter Hase, Mohit Bansal, Peter Clark, Sarah Wiegreffe
First submitted to arxiv on: 12 Jan 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This study investigates the “scalable oversight problem” in language models, where it’s challenging to train models on hard test data when training data is difficult to label correctly. The researchers found that current pre-trained language models can generalize surprisingly well from easy to hard data, often performing as well as oracle models fine-tuned on hard data. They demonstrated this using simple finetuning methods and seven measures of datapoint hardness. Furthermore, they showed that collecting easy data for finetuning is often better than collecting hard data, as the latter is generally noisier and costlier to collect. The study used open models up to 70b in size and four publicly available question-answering datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper explores how language models can perform well on difficult test data when training data is tricky to label correctly. Surprisingly, the researchers found that current pre-trained language models can generalize from easy to hard data without needing lots of labeled training data. They used simple methods to fine-tune these models and showed that they often work just as well as specialized models trained specifically for difficult tasks. The study also found that collecting easy data for finetuning is usually a better approach than trying to gather more difficult data. |
Keywords
* Artificial intelligence * Question answering
