Summary of Why You Don’t Overfit, and Don’t Need Bayes If You Only Train For One Epoch, by Laurence Aitchison
Why you don’t overfit, and don’t need Bayes if you only train for one epoch
by Laurence Aitchison
First submitted to arxiv on: 19 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the relationship between standard maximum likelihood training and Bayesian inference in a data-rich setting where models are only trained on each datapoint once. It shows that both methods optimize the same objective, the true data generating process loss, which is equivalent to the test loss. The findings suggest that Bayesian inference may not offer any advantages over standard maximum likelihood training in terms of overfitting or calibration in these settings. This has implications for the use of Bayesian models in areas such as language models, which are often trained with a single epoch. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper shows how standard machine learning training and Bayesian methods work together when we only train on each piece of data once. It finds that both types of training aim to achieve the same goal: to make accurate predictions. This means that using Bayesian methods might not be as important in some cases, like language models that are trained quickly. This is because the benefits of Bayesian methods are smaller when we’re working with large amounts of data. |
Keywords
» Artificial intelligence » Bayesian inference » Likelihood » Machine learning » Overfitting
