Summary of Improving Discrete Optimisation Via Decoupled Straight-through Gumbel-softmax, by Rushi Shah et al.
Improving Discrete Optimisation Via Decoupled Straight-Through Gumbel-Softmax
by Rushi Shah, Mingyuan Yan, Michael Curtis Mozer, Dianbo Liu
First submitted to arxiv on: 17 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel extension is proposed for the Straight-Through Gumbel-Softmax (ST-GS) estimator, which combines the Straight-Through Estimator (STE) and the Gumbel-based reparameterization trick. The new approach, Decoupled ST-GS, employs decoupled temperatures for forward and backward passes to enhance the performance of ST-GS. This extension is shown to significantly improve the original ST-GS through extensive experiments across multiple tasks and datasets. The impact of this method on gradient fidelity is investigated from multiple perspectives, including the gradient gap and the bias-variance trade-off of estimated gradients. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Discrete representations are important in many deep learning models. However, they can be tricky to optimize because they’re not easy to change smoothly. To help with this, researchers have developed ways to estimate gradients. One such method is called Straight-Through Gumbel-Softmax (ST-GS). But ST-GS has a problem: it’s very sensitive to temperature settings. In this paper, scientists propose an easy-to-use extension to ST-GS that solves this problem. They call it Decoupled ST-GS. By testing this method on many different tasks and datasets, they show that it works much better than the original ST-GS. |
Keywords
* Artificial intelligence * Deep learning * Softmax * Temperature
