Use of ontology structure and Bayesian models to aid the crowdsourcing of ICD-11 sanctioning rules

Abstract

The International Classification of Diseases (ICD) is the de facto standard international classification for mortality reporting and for many epidemiological, clinical, and financial use cases. The next version of ICD, ICD-11, will be submitted for approval by the World Health Assembly in 2018. Unlike previous versions of ICD, where coders mostly select single codes from pre-enumerated disease and disorder codes, ICD-11 coding will allow extensive use of multiple codes to give more detailed disease descriptions. For example, “severe malignant neoplasms of left breast” may be coded using the combination of a “stem code” (e.g., code for malignant neoplasms of breast) with a variety of “extension codes” (e.g., codes for laterality and severity). The use of multiple codes (a process called post-coordination), while avoiding the pitfall of having to pre-enumerate vast number of possible disease and qualifier combinations, risks the creation of meaningless expressions that combine stem codes with inappropriate qualifiers. To prevent that from happening, “sanctioning rules” that define legal combinations are necessary. In this work, we developed a crowdsourcing method for obtaining sanctioning rules for the post-coordination of concepts in ICD-11. Our method utilized the hierarchical structures in the domain to improve the accuracy of the sanctioning rules and to lower the crowdsourcing cost. We used Bayesian networks to model crowd workers’ skills, the accuracy of their responses, and our confidence in the acquired sanctioning rules. We applied reinforcement learning to develop an agent that constantly adjusted the confidence cutoffs during the crowdsourcing process to maximize the overall quality of sanctioning rules under a fixed budget. Finally, we performed formative evaluations using a skin-disease branch of the draft ICD-11 and demonstrated that the crowd-sourced sanctioning rules replicated those defined by an expert dermatologist with high precision and recall. This work demonstrated that a crowdsourcing approach could offer a reasonably efficient method for generating a first draft of sanctioning rules that subject matter experts could verify and edit, thus relieving them of the tedium and cost of formulating the initial set of rules.

A Bibliometric Analysis of the Development of ICD-11 in Medical Informatics

“…Researchers can also develop a customized version of WebProtégé [42] to support the collaborative development of ICD-11 content. Other ontology-based algorithms may also be useful to enhance ICD-11-based decision-making [43]. erefore, an important use of ICD-11 is the implementation of knowledge discovery in healthcare by transforming big data into healthcare knowledge [44].…”

DrugSemantics: A corpus for Named Entity Recognition in Spanish Summaries of Product Characteristics

“…This Crowdsourcing 18 is a collaborative approach for obtaining larger annotated corpora that allows annotators to work independently no matter the distance. Notwithstanding certain researchers have used the crowd to annotate in the healtcare field, such as [74,75], "a remaining challenge is that the cost to define a single annotation crowdsourcing project can outweigh the benefits" [76].…”

Semantic Web and Human Computation: The status of an emerging field

“…The latter type of workflows dovetails with recent efforts to construct Human-in-the-Loop systems and still raises several open research issues as discussed in [9]. There are also interesting efforts to exploring novel interfaces for HC based knowledge acquisition, such as chatbots [8] and aiming to collect more complex knowledge structures (e.g., rules) [26].…”

“…We found a total of 75 papers in this category, which cover context-aware knowledge acquisition on mobile devices [8], socio-technical systems that support communities, such as the Paleoclimate community, to develop and extend a community ontology in a collaborative effort [16]. Lou et al focus on the crowdsourcedacquisition of more complex knowledge structures, namely sanctioning rules in a use case related to the International Classification of Diseases (ICD-11) medical standard [26].…”