Summary of Posix: a Prompt Sensitivity Index For Large Language Models, by Anwoy Chatterjee et al.
POSIX: A Prompt Sensitivity Index For Large Language Models
by Anwoy Chatterjee, H S V N S Kowndinya Renduchintala, Sumit Bhatia, Tanmoy Chakraborty
First submitted to arxiv on: 3 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes POSIX, a novel metric to measure prompt sensitivity in Large Language Models (LLMs). Prompt sensitivity refers to the model’s tendency to generate different outputs in response to minor variations in prompts. While LLMs’ performance on downstream tasks is often evaluated, their prompt sensitivity is largely overlooked. The authors demonstrate that increasing the parameter count or instruction tuning does not necessarily reduce prompt sensitivity, but adding a few-shot exemplar can lead to significant decreases. They also find that alterations to prompt templates and paraphrasing have different effects on prompt sensitivity depending on the task type. POSIX provides a reliable measure of prompt sensitivity, allowing for a more comprehensive evaluation of LLM performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Language Models (LLMs) are amazing tools that can understand and generate human-like text. But they’re not perfect – sometimes small changes in what you ask them to do can lead to very different answers. This is called “prompt sensitivity”. The problem is, nobody really knows how well LLMs handle these small changes, or how to make them better at it. To solve this problem, the authors of this paper created a new way to measure prompt sensitivity – called POSIX. They tested POSIX on several LLMs and found that some ways of making LLMs “smarter” don’t actually help with prompt sensitivity. But they also found that adding just one or two examples of what you want the model to do can make a big difference. |
Keywords
» Artificial intelligence » Few shot » Instruction tuning » Prompt
