Relation Classification for Bleeding Events From Electronic Health Records Using Deep Learning Systems: An Empirical Study (Preprint)

Abstract

BACKGROUND Accurate detection of bleeding events from electronic health records (EHR) is crucial for identifying and characterizing different common and serious medical problems. To extract such information from EHRs, it is essential to identify the relations between bleeding events and related clinical entities (e.g., bleeding anatomic sites, lab tests). With the advent of natural language processing (NLP) and deep learning (DL) based techniques, many studies have focused on their applicability for various clinical applications. However, there has been no prior work that utilized deep learning to extract relations between bleeding events and relevant entities. OBJECTIVE In this study, we aim to evaluate multiple deep learning systems on a novel EHR dataset for bleeding event related relation classification. METHODS We first expert-annotated a new dataset of 1283 de-identified EHR notes for bleeding events and their attributes. On this dataset, we evaluated three state-of-the-art deep learning architectures, namely, convolutional neural network (CNN), graph convolutional network with attention (AGGCN) and BERT-based models (BioBERT, Bio+Clinical BERT and EhrBERT) for bleeding event relation classification task. RESULTS Our experiments show that the BERT-based models significantly outperformed CNN and AGGCN. Specifically, BioBERT achieved a macro F1 score of 0.842, outperforming both AGGCN (macro F1 score, 0.828) and CNN (macro F1 score, 0.763) by 1.4% (P<.001) and 7.9% (P<.001) respectively. CONCLUSIONS In this comprehensive study, we explored and compared different DL systems to classify relations between bleeding events and other medical concepts. On our corpus, BERT-based models outperformed other deep learning models for identifying the relations of bleeding related entities. BERT-based models were benefited from their pre-trained contextualized word representation and the use of target entity representation over traditional sequence representation.

Doctors Identify Hemorrhage Better during Chart Review when Assisted by Artificial Intelligence

“…25,26 Automatic hemorrhage identification has been investigated in previous studies. [9][10][11][12][13][14][15][16] Pedersen et al used an AI model to classify Danish clinical text as positive or negative for hemorrhage and achieved an accuracy of 90% on a balanced test set. 16 For English EHRs, Li et al used an AI model to detect hemorrhage events in EHR sentences with an F1 score of 94%, 11 and Taggart et al detected hemorrhage events at a note level using a rule-based approach with an F1 score of 74%.…”

“…10 Mitra et al used an AI model to extract single words that indicated hemorrhage and achieved an F1 score of 75%. 13 However, most studies used small test sets during model development (<1,000 samples) [10][11][12][13][14]16 and did not investigate model performance between different hemorrhage types or patient characteristics. [9][10][11][12][13][14]16 In contrast to previous studies on hemorrhage identification, we made an evaluation in a test cohort.…”

“…Artificial intelligence (AI) methods can be used for fast, automatic identification of information in EHR text. [9][10][11][12][13][14][15][16] This can be used in an acute clinical setting to find information in the EHR. Clinical evaluation of these methods is currently lacking.…”

“…To compare manual chart review with AI-assisted chart review, we performed a clinical use study with two admissions from the EHR system of the OUH. 13 MDs participated in the study. Seven participants were given admission A and six participants were given admission B.…”

Identification and Impact Analysis of Family History of Psychiatric Disorder in Mood Disorder Patients With Pretrained Language Model

“…Owing to the successful development of various deep learning (DL) techniques, accurate and effective DL-based natural language processing (NLP) systems can be built to alleviate this problem ( 18 – 22 ). On most NLP tasks, bidirectional transformer encoder representation (BERT) can achieve state-of-the-art performance while requiring minimal architectural modification ( 23 ).…”