Summary of Mirror: a Novel Approach For the Automated Evaluation Of Open-ended Question Generation, by Aniket Deroy et al.
MIRROR: A Novel Approach for the Automated Evaluation of Open-Ended Question Generation
by Aniket Deroy, Subhankar Maity, Sudeshna Sarkar
First submitted to arxiv on: 16 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a novel system called MIRROR (Multi-LLM Iterative Review and Response for Optimized Rating) that leverages large language models (LLMs) to automate the evaluation process for questions generated by automated question generation systems. The proposed approach improves human-like understanding and judgment, addressing limitations in current automated systems. By leveraging state-of-the-art LLMs like GPT-4, Gemini, and Llama2-70b, MIRROR enhances scores on metrics such as relevance, appropriateness, novelty, complexity, and grammaticality, getting closer to human baseline scores. The paper also demonstrates the effectiveness of Pearson’s correlation coefficient between GPT-4 and human experts when using MIRROR. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Automatic question generation is important for evaluating questions’ quality based on factors like engagement, pedagogical value, and critical thinking. Human evaluations are costly and impractical for large-scale samples, so a system called MIRROR was developed to automate this process. MIRROR uses big language models to help evaluate questions generated by other systems. The paper tested several of these models, including GPT-4, Gemini, and Llama2-70b. It found that using MIRROR improved scores on relevance, appropriateness, novelty, complexity, and grammar, making them closer to human evaluations. |
Keywords
» Artificial intelligence » Gemini » Gpt
