Summary of Towards Robust Extractive Question Answering Models: Rethinking the Training Methodology, by Son Quoc Tran et al.
Towards Robust Extractive Question Answering Models: Rethinking the Training Methodology
by Son Quoc Tran, Matt Kretchmar
First submitted to arxiv on: 29 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel training method to enhance the robustness of Extractive Question Answering (EQA) models. Despite previous research showing that existing EQA models struggle with distribution shifts and adversarial attacks, including unanswerable questions in EQA datasets is crucial for ensuring real-world reliability. The proposed training method introduces a new loss function for the EQA problem, challenging an implicit assumption prevalent in numerous EQA datasets. Models trained using this approach retain in-domain performance while achieving a notable improvement on out-of-domain datasets, resulting in a 5.7 F1 score increase across all testing sets. Additionally, our models demonstrate significantly enhanced robustness against two types of adversarial attacks, with a performance decrease of approximately one-third compared to default models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps make question-answering models more reliable by improving their ability to handle unexpected questions and attacks. Previous models were not good at dealing with changes in the way questions are asked or when someone tries to trick them. To fix this, the researchers developed a new training method that includes a special way of calculating the model’s performance. This new approach allows the model to perform well even when it’s faced with unfamiliar questions and attacks. The results show that the new model is much better at handling these challenges than previous models, which could have important implications for how we use AI in the future. |
Keywords
» Artificial intelligence » F1 score » Loss function » Question answering
