Summary of Question Difficulty Ranking For Multiple-choice Reading Comprehension, by Vatsal Raina et al.
Question Difficulty Ranking for Multiple-Choice Reading Comprehension
by Vatsal Raina, Mark Gales
First submitted to arxiv on: 16 Apr 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers explore automated methods for ranking multiple-choice (MC) questions by difficulty, which is useful for creating efficient exams for English learners. Traditional approaches rely on human test takers, but these methods are expensive and not scalable. The authors compare two approaches: task transfer, which adapts level classification and reading comprehension systems, and zero-shot prompting of instruction finetuned language models. They find that level classification transfers better than reading comprehension, and that zero-shot comparative assessment is more effective at difficulty ranking than absolute assessment or task transfer. Combining the systems further boosts the correlation. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper explores ways to make English exams more efficient by automatically ranking multiple-choice questions by difficulty. Traditionally, humans do this job, but it’s time-consuming and expensive. The researchers try two different approaches: one that adapts existing systems and another that uses language models. They find that one approach works better than the other, and that combining them makes things even better. |
Keywords
» Artificial intelligence » Classification » Prompting » Zero shot
