Summary of How Reliable Are Automatic Evaluation Methods For Instruction-tuned Llms?, by Ehsan Doostmohammadi et al.
How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?
by Ehsan Doostmohammadi, Oskar Holmström, Marco Kuhlmann
First submitted to arxiv on: 16 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the reliability of automatic methods used in evaluating Large Language Models (LLMs) for instruction tuning, as an alternative to human evaluation. The authors assess the performance of these methods across various tasks, including short-answer English and free-form generation tasks, using correlation metrics and an alternative approach called Pairwise Accuracy. They find that while automatic methods can approximate human ratings under specific conditions, their validity is highly context-dependent. Specifically, ROUGE-L correlates well with human ratings for short-answer English tasks but is unreliable in free-form generation tasks and cross-lingual scenarios. The study highlights the importance of understanding how to apply and interpret automatic evaluation metrics when developing and evaluating instruction-tuned LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how computers can be used to evaluate language models that are designed for teaching. Right now, people usually do this evaluation by hand, which is time-consuming and expensive. The researchers in this study tested different ways that computers can do this job, like comparing text and asking a computer program (called GPT-4) to judge. They found that these methods work okay in some situations, but not so well in others. For example, they are good at grading short answers on English questions, but not as good for longer free-writing tasks or when the question is asked in another language. This study helps us understand how to use computers to evaluate language models more effectively. |
Keywords
* Artificial intelligence * Gpt * Instruction tuning * Rouge
