Summary of Bilevel Optimization Under Unbounded Smoothness: a New Algorithm and Convergence Analysis, by Jie Hao et al.
Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis
by Jie Hao, Xiaochuan Gong, Mingrui Liu
First submitted to arxiv on: 17 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Optimization and Control (math.OC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a new bilevel optimization algorithm, BO-REP, to address limitations in current algorithms. Bilevel optimization is crucial for many machine learning problems, but recent studies reveal that certain neural networks like RNNs and LSTMs exhibit unbounded smoothness, making conventional algorithms unsuitable. The proposed algorithm updates the upper-level variable using normalized momentum and incorporates two novel techniques: initialization refinement and periodic updates. For nonconvex, unbounded smooth upper-level problems and strongly convex lower-level problems, BO-REP requires (1/^4) iterations to find an -stationary point in the stochastic setting. The algorithm’s effectiveness is demonstrated through experiments on hyper-representation learning, hyperparameter optimization, and data hyper-cleaning for text classification tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper solves a big problem in machine learning called bilevel optimization. Right now, most algorithms assume that certain things are “smooth” or easy to understand. But some neural networks don’t follow these rules, making it hard for computers to learn from them. The researchers created a new algorithm called BO-REP that can work with these tricky networks. They also showed that their algorithm is really good at solving problems and works well in different situations. |
Keywords
* Artificial intelligence * Hyperparameter * Machine learning * Optimization * Representation learning * Text classification
