Summary of On the Attribution Of Confidence to Large Language Models, by Geoff Keeling et al.
On the attribution of confidence to large language models
by Geoff Keeling, Winnie Street
First submitted to arxiv on: 11 Jul 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper explores the concept of credences in Large Language Models (LLMs). Credences refer to mental states indicating degrees of confidence in propositions. Researchers often attribute credences to LLMs when evaluating their performance. However, the theoretical foundation for this practice remains unclear. The authors defend three claims: first, that LLM credence attributions are interpreted literally as truth-apt beliefs; second, that the existence of LLM credences is plausible, although current evidence is inconclusive; and third, that LLM credence attributions in the empirical literature on LLM evaluation are subject to skeptical concerns. The authors argue that even if LLMs have credences, their attributions may be generally false due to non-truth-tracking experimental techniques. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you’re trying to understand how well a computer program knows something. You might say the program has a certain amount of confidence in its answer. This idea is called a “credence.” Researchers often give credences to big language models, like those used for chatbots or language translation. But are these credences even real? The authors of this paper think about what it means when we talk about computer models having credences and whether our methods for understanding them are working correctly. They raise some important questions about how we evaluate these models. |
Keywords
* Artificial intelligence * Tracking * Translation
