The 2019 n2c2/UMass Lowell shared task on clinical concept normalization

Abstract

Objective The 2019 National Natural language processing (NLP) Clinical Challenges (n2c2)/Open Health NLP (OHNLP) shared task track 3, focused on medical concept normalization (MCN) in clinical records. This track aimed to assess the state of the art in identifying and matching salient medical concepts to a controlled vocabulary. In this paper, we describe the task, describe the data set used, compare the participating systems, present results, identify the strengths and limitations of the current state of the art, and identify directions for future research. Materials and Methods Participating teams were provided with narrative discharge summaries in which text spans corresponding to medical concepts were identified. This paper refers to these text spans as mentions. Teams were tasked with normalizing these mentions to concepts, represented by concept unique identifiers, within the Unified Medical Language System. Submitted systems represented 4 broad categories of approaches: cascading dictionary matching, cosine distance, deep learning, and retrieve-and-rank systems. Disambiguation modules were common across all approaches. Results A total of 33 teams participated in the MCN task. The best-performing team achieved an accuracy of 0.8526. The median and mean performances among all teams were 0.7733 and 0.7426, respectively. Conclusions Overall performance among the top 10 teams was high. However, several mention types were challenging for all teams. These included mentions requiring disambiguation of misspelled words, acronyms, abbreviations, and mentions with more than 1 possible semantic type. Also challenging were complex mentions of long, multi-word terms that may require new ways of extracting and representing mention meaning, the use of domain knowledge, parse trees, or hand-crafted rules.

Triplet-Trained Vector Space and Sieve-Based Search Improve Biomedical Concept Normalization

“…There are a few similar works to our vector space model, CNN-triplet (Mondal et al, 2019), BIOSYN (Sung et al, 2020), RoBERTa-Node2Vec (Pattisapu et al, 2020), and TTI (Henry et al, 2020). CNN-triplet is a two-step approach, requiring a generator to generate candidates for training the triplet network, and requiring various embedding resources as input to CNN-based encoder.…”

“…Research on concept normalization has grown thanks to shared tasks such as disorder normalization in the 2013 ShARe/CLEF (Suominen et al, 2013), chemical and disease normalization in BioCreative V Chemical Disease Relation (CDR) Task , and medical concept normalization in 2019 n2c2 shared task (Henry et al, 2020), and to the availability of annotated data (Dogan et al, 2014;Luo et al, 2019). Existing approaches can be divided into three categories: rule-based approaches using string-matching or dictionary look-up (Leal et al, 2015;D'Souza and Ng, 2015;Lee et al, 2016), which rely heavily on handcrafted rules and domain knowledge; supervised multi-class classifiers (Limsopatham and Collier, 2016;Lee et al, 2017;Tutubalina et al, 2018;Niu et al, 2019;Li et al, 2019), which cannot generalize to concept types not present in their training data; and two-step frameworks based on a nontrained candidate generator and a supervised candidate ranker (Leaman et al, 2013;Li et al, 2017;Liu and Xu, 2017;Nguyen et al, 2018;Murty et al, 2018;Mondal et al, 2019;Ji et al, 2020;Xu et al, 2020), which require complex pipelines and fail if the candidate generator does not find the gold truth concept.…”

Clinical Term Normalization Using Learned Edit Patterns and Subconcept Matching: System Development and Evaluation

“…Recently, a new corpus, called MCN [ 3 ], was created exclusively for the clinical term normalization task, which also includes clinical terms of other semantic types. This corpus was provided as the data set for 2019 n2c2 Track 3 [ 17 ], a shared task for clinical term normalization. In this paper, we describe our system that we had submitted for this shared task.…”

“…Our system, UWM, achieved an accuracy of 80.79% on the test data set of the MCN corpus, which ranked sixth among the 33 system submissions and was behind by only 1.15% (absolute) to the second ranked system (81.94%) and was well above the mean (74.26%) and the median (77.33%) of all the participating systems [ 17 ]. The top system scored 85.26% and used a massive end-to-end deep learning architecture.…”

Clinical Term Normalization Using Learned Edit Patterns and Subconcept Matching: System Development and Evaluation (Preprint)

“…Recently, a new corpus, called MCN [3], was created exclusively for the clinical term normalization task, which also includes clinical terms of other semantic types. This corpus was provided as the data set for 2019 n2c2 Track 3 [17], a shared task for clinical term normalization. In this paper, we describe our system that we had submitted for this shared task.…”