Summary of Log Probabilities Are a Reliable Estimate Of Semantic Plausibility in Base and Instruction-tuned Language Models, by Carina Kauf et al.
Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models
by Carina Kauf, Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, Anna A. Ivanova
First submitted to arxiv on: 21 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers evaluate the effectiveness of log probabilities (LogProbs) and basic prompting to measure semantic plausibility in language models. They find that LogProbs offers a more reliable measure than direct zero-shot prompting, which yields inconsistent results. Instruction-tuning does not significantly alter the sensitivity of LogProbs to semantic plausibility. The study also explores how context affects LogProbs, using three novel metrics to compare with human judgments. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about how well language models understand what makes sense in the world. It looks at two ways to measure this: log probabilities and prompts. They find that one way is better than the other, and that having more training doesn’t always make it better. The study also shows how different situations affect how well the models understand things. |
Keywords
» Artificial intelligence » Instruction tuning » Prompting » Zero shot
