Classification of Clinical Notes from a Heart Failure Telehealth Network

Abstract

Heart failure is a common chronic disease which is associated with high re-hospitalization and mortality rates. Within the telemedicine-assisted transitional care disease management program HerzMobil, monitoring data such as daily measured vital parameters and various other heart failure related data are collected in a structured way. Additionally, involved healthcare professionals communicate with one another via the system using free-text clinical notes. Since manual annotation of such notes is too time-consuming for routine care applications, an automated analysis process is needed. In the present study, we established a ground truth classification of 636 randomly selected clinical notes from HerzMobil based on annotations of 9 experts with different professional background (2 physicians, 4 nurses, and 3 engineers). We analyzed the influence of the professional background on the inter annotator reliability and compared the results with the accuracy of an automated classification algorithm. We found significant differences depending on the profession and on the category. These results indicate that different professional backgrounds should be considered when selecting annotators in such scenarios.

Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes

“…Figure 3 shows the inter-annotator variability of the annotations. Similar to the previous work [7] the variability varied significantly throughout the categories. While in [7], all annotators were also active users of the HM Tirol system, leading to specific knowledge concerning notes and patients depending on their group, all annotators in the present paper were presented with the same amount of information.…”

“…Similar to the previous work [7] the variability varied significantly throughout the categories. While in [7], all annotators were also active users of the HM Tirol system, leading to specific knowledge concerning notes and patients depending on their group, all annotators in the present paper were presented with the same amount of information. The category Home visitation had the best F1 score concerning the inter-annotator variability, followed by the category Therapeutic regime, whilst the category Absence had the lowest score.…”

“…The notes were divided between the annotators so that every note was annotated by two annotators of each background resulting in approximately 400 notes per annotator and four annotations per note. A previously developed annotation guide was used to ensure a uniform definition of the categories [7].…”

“…For supervised machine learning models to be developed and trained, a set of annotated data with a defined ground truth is necessary. Therefore, an annotation guideline was developed and a subset of HM Tirol notes was manually annotated by nine independent annotators with different professional backgrounds into eight categories in previous work [7]. Annotations differ significantly between annotators of different professions.…”

“…The goal of this scientific paper was to further analyze the inter-annotator variability among different professional groups (healthcare, engineering) involved in the annotation process. While in [7], all annotators were users of HM Tirol, i.e., they had different views on the related data based on their professional role, the present paper analyses the impact of the professional background only. All annotators in the present paper had the same information (annotation guide with definition of categories) available for annotating the notes, without further knowledge of HM Tirol and its patients.…”