Diagnosis code assignment: models and evaluation metrics

Abstract

Background and objectiveThe volume of healthcare data is growing rapidly with the adoption of health information technology. We focus on automated ICD9 code assignment from discharge summary content and methods for evaluating such assignments.MethodsWe study ICD9 diagnosis codes and discharge summaries from the publicly available Multiparameter Intelligent Monitoring in Intensive Care II (MIMIC II) repository. We experiment with two coding approaches: one that treats each ICD9 code independently of each other (flat classifier), and one that leverages the hierarchical nature of ICD9 codes into its modeling (hierarchy-based classifier). We propose novel evaluation metrics, which reflect the distances among gold-standard and predicted codes and their locations in the ICD9 tree. Experimental setup, code for modeling, and evaluation scripts are made available to the research community.ResultsThe hierarchy-based classifier outperforms the flat classifier with F-measures of 39.5% and 27.6%, respectively, when trained on 20 533 documents and tested on 2282 documents. While recall is improved at the expense of precision, our novel evaluation metrics show a more refined assessment: for instance, the hierarchy-based classifier identifies the correct sub-tree of gold-standard codes more often than the flat classifier. Error analysis reveals that gold-standard codes are not perfect, and as such the recall and precision are likely underestimated.ConclusionsHierarchy-based classification yields better ICD9 coding than flat classification for MIMIC patients. Automated ICD9 coding is an example of a task for which data and tools can be shared and for which the research community can work together to build on shared models and advance the state of the art.

EMR Coding with Semi-Parametric Multi-Head Matching Networks

“…We evaluate using two versions of MIMIC: MIMIC II (Lee et al, 2011) and MIMIC III (Johnson et al, 2016), where the former is a relatively smaller and older dataset and the latter is the most recent version. Following prior work (Perotte et al, 2013; Vani et al, 2017), we use the free text discharge summaries in MIMIC to predict the ICD–9–CM 2 codes. The dataset statistics are shown in Table 1.…”

Leveraging hierarchy in medical codes for predictive modeling

“…Perotte et al [12] use a hierarchy-based SVM that exploits hierarchy in the codes for automated ICD-9 code assignment to discharge summary notes. Their task is to use features derived from the discharge summary notes to generate appropriate ICD-9 codes.…”

A Machine Learning Approach to Clinical Terms Normalization

“…On a similar view there are a number of works on automated clinical coding (Friedman et al, 2004;Pakhomov et al, 2006;Patrick et al, 2006;Suominen et al, 2008;Stanfill et al, 2010;Perotte et al, 2014). This work explores traditional soft string matching methods along with n-gram character and word features in a machine learning approach using MaxEnt and XGBoost classifiers.…”