Using Character-Level and Entity-Level Representations to Enhance Bidirectional Encoder Representation From Transformers-Based Clinical Semantic Textual Similarity Model: ClinicalSTS Modeling Study

Abstract

Background With the popularity of electronic health records (EHRs), the quality of health care has been improved. However, there are also some problems caused by EHRs, such as the growing use of copy-and-paste and templates, resulting in EHRs of low quality in content. In order to minimize data redundancy in different documents, Harvard Medical School and Mayo Clinic organized a national natural language processing (NLP) clinical challenge (n2c2) on clinical semantic textual similarity (ClinicalSTS) in 2019. The task of this challenge is to compute the semantic similarity among clinical text snippets. Objective In this study, we aim to investigate novel methods to model ClinicalSTS and analyze the results. Methods We propose a semantically enhanced text matching model for the 2019 n2c2/Open Health NLP (OHNLP) challenge on ClinicalSTS. The model includes 3 representation modules to encode clinical text snippet pairs at different levels: (1) character-level representation module based on convolutional neural network (CNN) to tackle the out-of-vocabulary problem in NLP; (2) sentence-level representation module that adopts a pretrained language model bidirectional encoder representation from transformers (BERT) to encode clinical text snippet pairs; and (3) entity-level representation module to model clinical entity information in clinical text snippets. In the case of entity-level representation, we compare 2 methods. One encodes entities by the entity-type label sequence corresponding to text snippet (called entity I), whereas the other encodes entities by their representation in MeSH, a knowledge graph in the medical domain (called entity II). Results We conduct experiments on the ClinicalSTS corpus of the 2019 n2c2/OHNLP challenge for model performance evaluation. The model only using BERT for text snippet pair encoding achieved a Pearson correlation coefficient (PCC) of 0.848. When character-level representation and entity-level representation are individually added into our model, the PCC increased to 0.857 and 0.854 (entity I)/0.859 (entity II), respectively. When both character-level representation and entity-level representation are added into our model, the PCC further increased to 0.861 (entity I) and 0.868 (entity II). Conclusions Experimental results show that both character-level information and entity-level information can effectively enhance the BERT-based STS model.

Impact of a Clinical Text–Based Fall Prediction Model on Preventing Extended Hospital Stays for Elderly Inpatients: Model Development and Performance Evaluation

“…It has often been reported that BERT exhibits high performance, even with clinical text [36][37][38][39]. This is also true for this study, in which a model combining BERT and Bi-LSTM using clinical text recorded in daily practice allowed for fall prediction with an accuracy equal to or higher than that of conventional risk assessment tools.…”

EARA: Improving Biomedical Semantic Textual Similarity with Entity-Aligned Attention and Retrieval Augmentation

“…We compare our results to existing methods and conduct ablation studies. Model N2C2STS (Xiong et al, 2020a) 0.868 (Ormerod et al, 2021) 0.870 (Chen et al, 2021) (single) 0.87 (Mulyar et al, 2021) 0.867 (Wang et al, 2022b) 0.875 EARA (BlueBERT) 0.887 across three benchmark datasets: N2C2STS, EBM-SASS, and BIOSSES. Notably, our approach consistently outperforms the baseline models, with average improvements ranging from 1.94% to 4.22%.…”

“…The first used data augmentation strategies (Wang et al, 2020c;Li et al, 2021a) or multi-task learning (Mulyar et al, 2021;Mahajan et al, 2020) to enhance the model's representation. The second introduced external knowledge into the neural network models, which can capture implicit information (Xiong et al, 2020a;Chang et al, 2021). These methods only integrate traditional features and lack interpretations.…”

“…In this section, we give a brief introduction of datasets on which we evaluate all models. (Xiong et al, 2020a), we train our supervised model on the training set using a 5-fold crossvalidation and evaluate our model on the gold test set. All experimental results are evaluated by the PCC, which measures the linear correlation between two sets of data.…”