Ease of adoption of clinical natural language processing software: An evaluation of five systems

Abstract

Objective In recognition of potential barriers that may inhibit the widespread adoption of biomedical software, the 2014 i2b2 Challenge introduced a special track, Track 3—Software Usability Assessment, in order to develop a better understanding of the adoption issues that might be associated with the state-of-the-art clinical NLP systems. This paper reports the ease of adoption assessment methods we developed for this track, and the results of evaluating five clinical NLP system submissions. Materials and Methods A team of human evaluators performed a series of scripted adoptability test tasks with each of the participating systems. The evaluation team consisted of four “expert evaluators” with training in computer science, and eight “end user evaluators” with mixed backgrounds in medicine, nursing, pharmacy, and health informatics. We assessed how easy it is to adopt the submitted systems along the following three dimensions: communication effectiveness (i.e., how effective a system is in communicating its designed objectives to intended audience), effort required to install, and effort required to use. We used a formal software usability testing tool, TURF, to record the evaluators’ interactions with the systems and ‘think-aloud’ data revealing their thought processes when installing and using the systems and when resolving unexpected issues. Results Overall, the ease of adoption ratings that the five systems received are unsatisfactory. Installation of some of the systems proved to be rather difficult, and some systems failed to adequately communicate their designed objectives to intended adopters. Further, the average ratings provided by the end user evaluators on ease of use and ease of interpreting output are −0.35 and −0.53, respectively, indicating that this group of users generally deemed the systems extremely difficult to work with. While the ratings provided by the expert evaluators are higher, 0.6 and 0.45, respectively, these ratings are still low indicating that they also had considerable struggles. Discussion The results of the Track 3 evaluation show that the adoptability of the five participating clinical NLP systems has a great margin for improvement. Remedy strategies suggested by the evaluators included (1) more detailed and operation system specific use instructions; (2) provision of more pertinent onscreen feedback for easier diagnosis of problems; (3) including screen walk-throughs in use instructions so users know what to expect and what might have gone wrong; (4) avoiding jargon and acronyms in materials intended for end users; and (5) packaging prerequisites required within software distributions so that prospective adopters of the software do not have to obtain each of the third-party components on their own.

Replicability of Research in Biomedical Natural Language Processing: a pilot evaluation for a coding task

“…The results obtained were be compared to those submitted by the teams using the same system. During this process, the analysts took notes on the various aspects of working with the systems (ease of installing and using, ease of understanding supplied instructions, success of the replication attempt), using a specific score sheet developed by the analysts, following some of the criteria evaluated by (Zheng et al, 2015). The score sheet comprised 10 questions addressing the experience of analysts at each stage of the experiment: system configuration, system installation, running the system, obtaining results, and overall impressions.…”

Practical applications for natural language processing in clinical research: The 2014 i2b2/UTHealth shared tasks

“…2 Two of these systems were on concept extraction and understanding, two were on medication extraction, and one was on de-identification. Zheng et al [16] describes these systems and their evaluation in detail, with one major take away that affects all NLP systems in the clinical domain: the long pipeline of preprocessing components, from tokenizers to metathesauri, that are essential to most NLP goals reduce the adoptability and portability of systems, especially if the systems are to be used by novices. While these preprocessing components cannot be excluded from NLP systems, they can be standardized in their input and output formats to allow some degree of interchangeability so that each new system does not come with a completely new set of preprocessing components.…”

“…The software usability track aimed to assess the usability of systems developed for any of the past i2b2 shared tasks since 2006 [16]. The novel data use track, on the other hand, built on the observation that past i2b2 corpora have often been successfully put to use for purposes outside of their original goals and opened the 2014 shared-task corpus to any research project that fit the participants’ existing goals.…”

“… 2 The remaining three were dropped because “one was withdrawn before the evaluations started, a second was not an NLP system, and the third was a software library that did not have a user interface.” [16] …”

A Neuro-ontology for the neurological examination

“…Much of the clinical data in electronic health records (EHRs) are represented as free text. Although progress is being made in the conversion of free text into structured data by natural language processing (NLP), these methods are not in general use [6][7][8][9][10]. The entry of data about neurological patients in EHRs into large databases requires a method for converting symptoms (patient complaints) and signs (examination abnormalities) into machine-readable codes.…”