Summary of Are Llms Good Pragmatic Speakers?, by Mingyue Jian et al.
Are LLMs good pragmatic speakers?
by Mingyue Jian, N. Siddharth
First submitted to arxiv on: 3 Nov 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates whether large language models (LLMs) can behave like pragmatic speakers by using the Rational Speech Act (RSA) framework. It compares scores from a state-of-the-art LLM (Llama3-8B-Instruct) and the RSA model on a reference game constructed from the TUNA corpus. While there is some positive correlation between the scores, the results suggest that the LLM does not fully exhibit pragmatic speaker behavior. The study sets the stage for future research exploring different models and settings to see if LLMs can be made to behave like pragmatic speakers. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at whether big language models are good at understanding human communication rules. It uses a special framework called RSA, which helps explain how humans talk to each other. The researchers compare the model’s scores with those of a really smart computer program (Llama3-8B-Instruct) on a game-like task using real-life conversations. While there is some connection between the scores, it doesn’t look like the computer program is as good at understanding human communication rules as humans are. |
