Summary of In-context Freeze-thaw Bayesian Optimization For Hyperparameter Optimization, by Herilalaina Rakotoarison and Steven Adriaensen and Neeratyoy Mallik and Samir Garibov and Edward Bergman and Frank Hutter
In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization
by Herilalaina Rakotoarison, Steven Adriaensen, Neeratyoy Mallik, Samir Garibov, Edward Bergman, Frank Hutter
First submitted to arxiv on: 25 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes FT-PFN, a novel surrogate model for freeze-thaw Bayesian optimization. The authors address limitations in current automated hyperparameter optimization methods, which rely heavily on black-box Bayesian optimization. Freeze-thaw BO offers a grey-box alternative, but its frequent surrogate model updates pose challenges for existing methods. The proposed FT-PFN leverages transformers’ in-context learning ability to efficiently and reliably perform Bayesian learning curve extrapolation in a single forward pass. Empirical analysis across three benchmark suites shows that FT-PFN’s predictions are more accurate and 10-100 times faster than those of deep Gaussian process and deep ensemble surrogates used previously. When combined with the novel acquisition mechanism MFPI-random, the resulting in-context freeze-thaw BO method (ifBO) achieves state-of-the-art performance in three families of deep learning HPO benchmarks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper solves a problem with how computers find the best settings for machine learning models. Right now, it takes too long and uses too much computer power to do this automatically. The researchers came up with a new way to do it called freeze-thaw Bayesian optimization. It’s like a middle ground between using all or nothing of computer power. They also made a special kind of model that can quickly make good predictions about what will work best. This helps the computer find the right settings faster and more accurately than before. |
Keywords
» Artificial intelligence » Deep learning » Hyperparameter » Machine learning » Optimization
