Summary of Questionable Practices in Machine Learning, by Gavin Leech et al.
Questionable practices in machine learning
by Gavin Leech, Juan J. Vazquez, Niclas Kupper, Misha Yagudin, Laurence Aitchison
First submitted to arxiv on: 17 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL); Computers and Society (cs.CY)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper delves into the challenges of evaluating modern machine learning (ML) models, highlighting the prevalence of questionable research practices (QRPs). The authors identify 44 such practices, providing examples where possible, with a focus on large language models (LLMs) and public benchmarks. Additionally, they discuss “irreproducible research practices” that hinder the ability to reproduce, build upon, or audit previous research. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper talks about how it’s hard to evaluate modern machine learning models because people often try to show that their results are better than others’. This can lead to some not-so-good ways of doing research. The authors make a list of 44 things people might do that aren’t great, and they give examples for some of them. They’re particularly interested in how big language models do on public tests. They also talk about when it’s hard or impossible for other researchers to repeat or build upon previous work. |
Keywords
» Artificial intelligence » Machine learning
