Summary of Towards Understanding the Robustness Of Llm-based Evaluations Under Perturbations, by Manav Chaudhary et al.
Towards Understanding the Robustness of LLM-based Evaluations under Perturbations
by Manav Chaudhary, Harshit Gupta, Savita Bhat, Vasudeva Varma
First submitted to arxiv on: 12 Dec 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the potential of Large Language Models (LLMs) to serve as automatic evaluators for non-standardized metrics in summarization and dialog-based tasks. Specifically, it examines Google Gemini 1’s performance in evaluating summarization and dialogue quality using multiple prompting strategies on the SummEval and USR datasets. The model is asked to generate both a score and a justification for the score, and its robustness against perturbed inputs is also tested. While the LLM shows promise, its alignment with human evaluators is limited, it lacks robustness against perturbations, and significant improvements are required for standalone use. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how well big language models can be used to evaluate how good summaries or conversations are. It tests a Google model called Gemini 1 on two datasets and sees if the model’s scores match up with human scores. The model has to give both a score and an explanation for why it gave that score, and also handles changed input questions. While the model does some things right, it doesn’t always agree with people, isn’t very good at handling changes, and needs more work before it can be used on its own. |
Keywords
» Artificial intelligence » Alignment » Gemini » Prompting » Summarization
