Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset

Abstract

Machine reading comprehension has made great progress in recent years owing to largescale annotated datasets. In the clinical domain, however, creating such datasets is quite difficult due to the domain expertise required for annotation. Recently, Pampari et al. (2018) tackled this issue by using expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering (QA) based on clinical notes. In this paper, we provide an indepth analysis of this dataset and the clinical reading comprehension (CliniRC) task. From our qualitative analysis, we find that (i) emrQA answers are often incomplete, and (ii) emrQA questions are often answerable without using domain knowledge. From our quantitative experiments, surprising results include that (iii) using a small sampled subset (5%-20%), we can obtain roughly equal performance compared to the model trained on the entire dataset, (iv) this performance is close to human expert's performance, and (v) BERT models do not beat the best performing base model. Following our analysis of the emrQA, we further explore two desired aspects of CliniRC systems: the ability to utilize clinical domain knowledge and to generalize to unseen questions and contexts. We argue that both should be considered when creating future datasets. 1

GatorTron: A Large Language Model for Clinical Natural Language Processing

“…and relations-related questions as Yue et al 75 found that the two subsets are more consistent. We utilized both F1-score and exact match score for evaluation.…”

Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text

“…SeaReader is proposed to answer questions in clinical medicine using documents extracted from publications in the medical domain. Yue et al (2020) conduct a thorough analysis of the emrQA dataset (Pampari et al, 2018) and explore the ability of QA systems to utilize clinical domain knowledge and to generalize to unseen questions. introduce PubMedQA where questions are derived based on article titles and can be answered with its respective abstracts.…”

“…Some recent works are trying to construct medical MRC dataset such as PubMedQA , emrQA (Pampari et al, 2018) and HEAD-QA (Vilares and Gómez-Rodríguez, 2019), etc. However, either these data sets are noisy (e.g., due to semi-automatically or heuristic rules generated), or the annotated data scale is too small (Yoon et al, 2019;Yue et al, 2020). Instead, we constructs a large scale medical MRC dataset by collecting 21.7k multiplechoice problems with human-annotated answers for the National Licensed Pharmacist Examination in China.…”

“…Though several pre-trained language models have been introduced for domain-specific MRC, BERT based models are not as consistently dominant as they are in open field MRC tasks (Zhong et al, 2020;Yue et al, 2020). Another challenge is that medical questions are often more difficult; no labeled paragraph contains the answer to a given question.…”

PHICON: Improving Generalization of Clinical Text De-identification Models via Data Augmentation

“…To prevent the model overfitting on specific cases and encourage it to learn general language patterns, one possible way is to enlarge training data (Yang et al, 2019). However, clinical texts are usually difficult to obtain, not to mention the requirement of tremendous expert effort for annotations (Yue et al, 2020). To solve this, we introduce our data augmentation method PHICON, which consists of PHI augmentation and Context augmentation.…”