Summary of Calibrating Large Language Models Using Their Generations Only, by Dennis Ulmer et al.
Calibrating Large Language Models Using Their Generations Only
by Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, Seong Joon Oh
First submitted to arxiv on: 9 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed APRICOT method aims to calibrate large language models (LLMs) by predicting their confidence in predictions based solely on textual input and output. This approach is conceptually simple, doesn’t require access to the target model beyond its output, and has various potential uses such as verbalizing predicted confidence or adjusting answers. The method performs competitively in terms of calibration error for white-box and black-box LLMs on closed-book question-answering tasks, particularly detecting incorrect LLM answers. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary APRICOT is a way to make sure large language models are honest about how confident they are in what they say. Right now, it’s hard to know if these models are really sure or just pretending. APRICOT helps by creating another model that looks at the text input and output of the original model and says “hey, I think this model is 80% sure” or something like that. This can help us figure out when the model is wrong and maybe even adjust its answers to make them more accurate. |
Keywords
* Artificial intelligence * Question answering
