Summary of Reife: Re-evaluating Instruction-following Evaluation, by Yixin Liu et al.
ReIFE: Re-evaluating Instruction-Following Evaluation
by Yixin Liu, Kejian Shi, Alexander R. Fabbri, Yilun Zhao, Peifeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan
First submitted to arxiv on: 9 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a comprehensive evaluation of large language models (LLMs) used to assess response quality in instruction following. The study compares 25 different LLMs and 15 evaluation protocols on four human-annotated datasets, analyzing their accuracy and robustness. The results show that the best-performing base LLMs and evaluation protocols are consistent across different tasks, but the effectiveness of an evaluation protocol depends on the base LLM used. Additionally, the study highlights the importance of using multiple datasets with distinct features to ensure rigorous evaluation. The authors release a meta-evaluation suite called ReIFE, which provides codebase and evaluation result collection for over 500 LLM- evaluator configurations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how well computers can evaluate people’s responses when following instructions. They compared many different computer programs (called large language models or LLMs) with different ways of testing these programs on lots of examples. The results show that some LLMs are better than others, and some methods work better than others too. But the authors found out that what works best depends on which type of LLM is used. They also say that it’s important to test these computer programs on many different types of examples to make sure they’re accurate. |
