Summary of Tune Without Validation: Searching For Learning Rate and Weight Decay on Training Sets, by Lorenzo Brigato and Stavroula Mougiakakou
Tune without Validation: Searching for Learning Rate and Weight Decay on Training Sets
by Lorenzo Brigato, Stavroula Mougiakakou
First submitted to arxiv on: 8 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces Tune without Validation (Twin), a novel pipeline for tuning learning rate and weight decay without relying on validation sets. Building upon recent theoretical work on learning phases in hypothesis space, the authors develop a heuristic that predicts which hyperparameter combinations lead to better generalization. This is achieved through a grid search of trials, using an early- or non-early-stopping scheduler, followed by segmenting the region providing the best training loss results. The weight norm is found to strongly correlate with predicting generalization. Extensive experiments are conducted on 20 image classification datasets and various deep network families (convolutional, transformer, and feed-forward models) to assess Twin’s effectiveness in both scratch-trained and fine-tuned scenarios, highlighting its benefits for small-sample scenarios. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Twin is a new way to adjust learning rates and weight decay without needing extra data. It uses recent research on how learning works to make predictions about which settings will work best. The authors tested it on many image classification datasets and different types of deep networks, showing that it can help with both training from scratch and fine-tuning. |
Keywords
* Artificial intelligence * Early stopping * Fine tuning * Generalization * Grid search * Hyperparameter * Image classification * Transformer
