Summary of Deal, or No Deal (or Who Knows)? Forecasting Uncertainty in Conversations Using Large Language Models, by Anthony Sicilia et al.
Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models
by Anthony Sicilia, Hyunwoo Kim, Khyathi Raghavi Chandu, Malihe Alikhani, Jack Hessel
First submitted to arxiv on: 5 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes FortUne Dial, a new approach to the “conversation forecasting” task that evaluates language models’ ability to represent uncertainty in conversations. Unlike traditional methods, which focus on accuracy, FortUne Dial uses uncertainty-aware metrics to enable abstention on individual instances. The authors study two ways language models can represent outcome uncertainty and propose fine-tuning strategies to improve calibration. Experiments on eight negotiation corpora show that their proposed strategies can calibrate smaller open-source models to compete with larger pre-trained models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Language models are getting better at understanding conversations, but they still struggle to understand when things might go wrong. The paper proposes a new way to measure how well language models can predict the outcome of a conversation. Instead of just trying to get the answer right, this approach also considers how uncertain the model is about its prediction. The authors test two ways that language models can represent this uncertainty and show that their proposed methods can improve the performance of smaller language models. |
Keywords
* Artificial intelligence * Fine tuning
