Summary of Towards Unbiased Evaluation Of Detecting Unanswerable Questions in Ehrsql, by Yongjin Yang et al.
Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL
by Yongjin Yang, Sihyeon Kim, SangMook Kim, Gyubok Lee, Se-Young Yun, Edward Choi
First submitted to arxiv on: 29 Apr 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a simple debiasing method to address a data bias issue in Electronic Health Record (EHR) Question Answering (QA) systems. The authors identify that unanswerable questions in the EHRSQL dataset can often be detected by filtering with specific N-gram patterns, which jeopardizes the authenticity and reliability of QA system evaluations. To mitigate this bias, they propose adjusting the split between validation and test sets to neutralize the undue influence of N-gram filtering. The effectiveness of their method is demonstrated through experiments on the MIMIC-III dataset. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper talks about making sure that a computer program that helps doctors diagnose patients doesn’t give fake answers. Right now, this program can get tricked into giving false information because some questions are too easy to figure out just by looking at certain patterns in the data. The researchers found that this is happening with some of the questions in a special dataset called EHRSQL. They came up with an easy way to fix this problem by adjusting how they test the program’s answers. This makes sure the program doesn’t get tricked into giving false information and can be trusted to give accurate diagnoses. |
Keywords
» Artificial intelligence » N gram » Question answering
