Loading Now

Summary of In-context Freeze-thaw Bayesian Optimization For Hyperparameter Optimization, by Herilalaina Rakotoarison and Steven Adriaensen and Neeratyoy Mallik and Samir Garibov and Edward Bergman and Frank Hutter


In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization

by Herilalaina Rakotoarison, Steven Adriaensen, Neeratyoy Mallik, Samir Garibov, Edward Bergman, Frank Hutter

First submitted to arxiv on: 25 Apr 2024

Categories

  • Main: Machine Learning (cs.LG)
  • Secondary: None

     Abstract of paper      PDF of paper


GrooveSquid.com Paper Summaries

GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!

Summary difficulty Written by Summary
High Paper authors High Difficulty Summary
Read the original abstract here
Medium GrooveSquid.com (original content) Medium Difficulty Summary
This paper proposes FT-PFN, a novel surrogate model for freeze-thaw Bayesian optimization. The authors address limitations in current automated hyperparameter optimization methods, which rely heavily on black-box Bayesian optimization. Freeze-thaw BO offers a grey-box alternative, but its frequent surrogate model updates pose challenges for existing methods. The proposed FT-PFN leverages transformers’ in-context learning ability to efficiently and reliably perform Bayesian learning curve extrapolation in a single forward pass. Empirical analysis across three benchmark suites shows that FT-PFN’s predictions are more accurate and 10-100 times faster than those of deep Gaussian process and deep ensemble surrogates used previously. When combined with the novel acquisition mechanism MFPI-random, the resulting in-context freeze-thaw BO method (ifBO) achieves state-of-the-art performance in three families of deep learning HPO benchmarks.
Low GrooveSquid.com (original content) Low Difficulty Summary
This paper solves a problem with how computers find the best settings for machine learning models. Right now, it takes too long and uses too much computer power to do this automatically. The researchers came up with a new way to do it called freeze-thaw Bayesian optimization. It’s like a middle ground between using all or nothing of computer power. They also made a special kind of model that can quickly make good predictions about what will work best. This helps the computer find the right settings faster and more accurately than before.

Keywords

» Artificial intelligence  » Deep learning  » Hyperparameter  » Machine learning  » Optimization