Summary of Answer, Assemble, Ace: Understanding How Lms Answer Multiple Choice Questions, by Sarah Wiegreffe et al.
Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions
by Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, Ashish Sabharwal
First submitted to arxiv on: 21 Jul 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The research paper investigates how successful transformer language models perform multiple-choice question answering (MCQA) tasks with varying formats. The study employs vocabulary projection and activation patching methods to identify key hidden states responsible for predicting correct answers. It finds that specific middle layers, particularly their multi-head self-attention mechanisms, play a crucial role in this process. The paper also explores how different models adapt to alternative answer symbols and develops a synthetic task to disentangle model errors, enabling the identification of when a model has learned formatted MCQA. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The study looks at how language models do well on multiple-choice question tests that ask you to pick one correct answer from several options. The researchers used special techniques to figure out which parts of the model are important for making predictions. They found that certain layers in the model are key to picking the right answer, and that these layers use a specific attention mechanism. The study also shows how different models handle different answer choices and develops a new task to help understand when a model has learned to do well on this type of test. |
Keywords
» Artificial intelligence » Attention » Question answering » Self attention » Transformer
