BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition

Abstract

With the growing number of electronic health record data, clinical NLP tasks have become increasingly relevant to unlock valuable information from unstructured clinical text. Although the performance of downstream NLP tasks, such as named-entity recognition (NER), in English corpus has recently improved by contextualised language models, less research is available for clinical texts in low resource languages. Our goal is to assess a deep contextual embedding model for Portuguese, so called BioBERTpt, to support clinical and biomedical NER. We transfer learned information encoded in a multilingual-BERT model to a corpora of clinical narratives and biomedical-scientific papers in Brazilian Portuguese. To evaluate the performance of BioBERTpt, we ran NER experiments on two annotated corpora containing clinical narratives and compared the results with existing BERT models. Our in-domain model outperformed the baseline model in F1-score by 2.72%, achieving higher performance in 11 out of 13 assessed entities. We demonstrate that enriching contextual embedding models with domain literature can play an important role in improving performance for specific NLP tasks. The transfer learning process enhanced the Portuguese biomedical NER model by reducing the necessity of labeled data and the demand for retraining a whole new model.

Ensemble of Deep Masked Language Models for Effective Named Entity Recognition in Health and Life Science Corpora

“…Alsentzer et al 2019 trained BERT andBioBERT (Lee et al, 2019), on MIMIC notes, and showed that Bio + Clinical BERT performed better than BERT and BioBERT trained on MedNLI dataset and i2b2 2010 datasets. Similarly, Schneider et al (2020) demonstrated that the fine-tuned BERT using Portuguese clinical notes outperformed BERT trained on general corpora.…”

“…Various NER challenges and shared tasks, such as the i2b2 and n2c2 NLP challenges (Uzuner et al, 2010;Suominen et al, 2013;Kelly et al, 2014;Bethard et al, 2015;Névéol et al, 2015;Henry et al, 2020), fostered the development of NER methods (De Bruijn et al, 2011;Jiang et al, 2011;Kim et al, 2015;Van Mulligen et al, 2016;El Boukkouri et al, 2019) for the clinical domain in different languages (Lopes et al, 2019;Sun and Yang, 2019;Andrioli de Souza et al, 2020;Schneider et al, 2020). The DEFT challenge proposed an information extraction task for the French clinical corpus, with entities distributed across four categories: anatomy, clinical practices, treatments, and time (Cardon et al, 2020).…”

Ensemble of deep masked language models for effective named entity recognition in multi-domain corpora

“…Various NER challanges and shared tasks [Uzuner et al, 2010, Kelly et al, 2014, Névéol et al, 2015, Suominen et al, 2013, Bethard et al, 2015] fostered the development of NER methods Van Mulligen et al, 2016 Kim et al, 2015 Jiang et al, 2011, De Bruijn et al, 2011, El Boukkouri et al, 2019] for clinical domain in different languages [Lopes et al, 2019, Schneider et al, 2020, Sun and Yang, 2019]. Performance varies greatly across the different methods and corpora, with more modern methods achieving F 1 -score as high as 95%.…”

Sentiment Analysis in Portuguese Texts from Online Health Community Forums: Data, Model and Evaluation

“…As input features to the classic classifiers, we used the TF-IDF vector representations of the texts to be classified. As state-of-the-art approaches, we fine-tuned BERT-based models such as its multilingual version [Devlin et al 2019]; BERTimbau [Souza et al 2020], its Brazilian Portuguese version; and BioBERTpt [Schneider et al 2020], a Brazilian Portuguese version of BERT focused on the clinical domain.…”