Negation’s Not Solved: Generalizability Versus Optimizability in Clinical Natural Language Processing

Abstract

A review of published work in clinical natural language processing (NLP) may suggest that the negation detection task has been “solved.” This work proposes that an optimizable solution does not equal a generalizable solution. We introduce a new machine learning-based Polarity Module for detecting negation in clinical text, and extensively compare its performance across domains. Using four manually annotated corpora of clinical text, we show that negation detection performance suffers when there is no in-domain development (for manual methods) or training data (for machine learning-based methods). Various factors (e.g., annotation guidelines, named entity characteristics, the amount of data, and lexical and syntactic context) play a role in making generalizability difficult, but none completely explains the phenomenon. Furthermore, generalizability remains challenging because it is unclear whether to use a single source for accurate data, combine all sources into a single model, or apply domain adaptation methods. The most reliable means to improve negation detection is to manually annotate in-domain training data (or, perhaps, manually modify rules); this is a strategy for optimizing performance, rather than generalizing it. These results suggest a direction for future work in domain-adaptive and task-adaptive methods for clinical NLP.

Unsupervised Domain Adaptation for Clinical Negation Detection

“…We make use of the (Wu et al, 2014) system in these experiments, as it is freely available as part of the Apache cTAKES (Savova et al, 2010) 1 clinical NLP software, and can be easily retrained.…”

“…However generalizability of negation systems is still lacking, as cross-domain experiments suffer dramatic performance losses, even while obtaining F1 scores over 90% in the domain of the training data (Wu et al, 2014).…”

“…The MITRE system (Clark et al, 2011) used conditional random fields to tag cues and their scopes, then incorporated cue information, section features, semantic and syntactic class features, and lexical surface features into a maximum entropy classifier. Finally, Wu et al (2014) incorporated many of the dependency features from rule-based DepNeg system (Sohn et al, 2012) and the best features from the i2b2 Challenge into a machine learning system.…”

Don't Let Notes Be Misunderstood: A Negation Detection Method for Assessing Risk of Suicide in Mental Health Records

“…A comprehensive study on current state-of-the-art negation detection algorithms and their performance on different corpora is presented by Wu et al (2014). As is concluded in this study, none of the existing state-of-the-art systems are guaranteed to work well on a new domain or corpus, and there are still open issues when it comes to creating a generalizable negation detection solution.…”

Proceedings of the Second Workshop on Extra-Propositional Aspects of Meaning in Computational Semantics (ExProM 2015)

“…This should be leveraged to harness the benefits offered by machine learning solutions. Recently, Wu et al (2014) argued that negation detection is not of practical value without in-domain training and/or development, and described an SVM-based approach using hand-crafted features.…”