NLPReViz: an interactive tool for natural language processing on clinical text

Abstract

The gap between domain experts and natural language processing expertise is a barrier to extracting understanding from clinical text. We describe a prototype tool for interactive review and revision of natural language processing models of binary concepts extracted from clinical notes. We evaluated our prototype in a user study involving 9 physicians, who used our tool to build and revise models for 2 colonoscopy quality variables. We report changes in performance relative to the quantity of feedback. Using initial training sets as small as 10 documents, expert review led to final F1scores for the "appendiceal-orifice" variable between 0.78 and 0.91 (with improvements ranging from 13.26% to 29.90%). F1for "biopsy" ranged between 0.88 and 0.94 (-1.52% to 11.74% improvements). The average System Usability Scale score was 70.56. Subjective feedback also suggests possible design improvements.

Local Decision Pitfalls in Interactive Machine Learning

“…Interactive machine learning (IML) systems aim to ease the process of training a model by providing tools that support more rapid, focused, and incremental model updates than seen in a traditional machine learning (ML) process [2]. These properties enable everyday users to interactively explore the model space through trial-and-error and drive the system toward an intended behavior, hopefully reducing the need for supervision by ML experts [14,30,63]. IML enables two-way interaction between human and machines: On one hand, the system explains to users how the learner is making predictions, usually through visual [38,41] or textual [34] feedback on model performance [10,52,63].…”

“…These properties enable everyday users to interactively explore the model space through trial-and-error and drive the system toward an intended behavior, hopefully reducing the need for supervision by ML experts [14,30,63]. IML enables two-way interaction between human and machines: On one hand, the system explains to users how the learner is making predictions, usually through visual [38,41] or textual [34] feedback on model performance [10,52,63]. On the other hand, the user then communicates modifications back to the learning system to enhance the resulting model.…”

“…On the other hand, the user then communicates modifications back to the learning system to enhance the resulting model. For example, the user may label more data points [25,57], adjust learning parameters [27], or add and delete features [33,63].…”

“…In the first case, users may trust in their own domain knowledge too strongly, lowering learner performance by acting on their intuitions [37,60]. In the second case, they may overly rely on IML system feedback specific to the current model and overfit to the training or validation dataset [10,63]. Both over-confidence and over-reliance may hamper learner generalization due to localized decisions.…”

“…Text data is a common modality in IML research [10,30,33,41,55,60,61,63], and sentiment analysis is an interesting and representative task. Feature selection, while just one activity supported by IML systems, is provided by many IML systems [10,15,33,55,60,61,63]. It also focuses on the immediate current iteration of a learner: classic feature selection systems in the wild typically show the performance of the most up-to-date model that users have created with previous manipulations.…”