Improving biomedical named entity recognition with syntactic information

Abstract

Background Biomedical named entity recognition (BioNER) is an important task for understanding biomedical texts, which can be challenging due to the lack of large-scale labeled training data and domain knowledge. To address the challenge, in addition to using powerful encoders (e.g., biLSTM and BioBERT), one possible method is to leverage extra knowledge that is easy to obtain. Previous studies have shown that auto-processed syntactic information can be a useful resource to improve model performance, but their approaches are limited to directly concatenating the embeddings of syntactic information to the input word embeddings. Therefore, such syntactic information is leveraged in an inflexible way, where inaccurate one may hurt model performance. Results In this paper, we propose BioKMNER, a BioNER model for biomedical texts with key-value memory networks (KVMN) to incorporate auto-processed syntactic information. We evaluate BioKMNER on six English biomedical datasets, where our method with KVMN outperforms the strong baseline method, namely, BioBERT, from the previous study on all datasets. Specifically, the F1 scores of our best performing model are 85.29% on BC2GM, 77.83% on JNLPBA, 94.22% on BC5CDR-chemical, 90.08% on NCBI-disease, 89.24% on LINNAEUS, and 76.33% on Species-800, where state-of-the-art performance is obtained on four of them (i.e., BC2GM, BC5CDR-chemical, NCBI-disease, and Species-800). Conclusion The experimental results on six English benchmark datasets demonstrate that auto-processed syntactic information can be a useful resource for BioNER and our method with KVMN can appropriately leverage such information to improve model performance.

On the Use of Parsing for Named Entity Recognition

“…As we explained in the previous section, existing approaches have been limited by the difficulty of integrating hierarchical information such as a parse tree into a task that is linear in nature. Thus, either they make limited use of such syntactic information [128,129], or they develop ad hoc architectures that result in more complex, less generic, and less efficient models [95,136,137]. What we know for sure is that the use of information from parsers is beneficial but, since they have been tested on different data sets, it is difficult to determine which of those approaches for incorporating parsing information is more effective in general terms.…”

“…However, this will require a reduction of semantic parsing to sequence labeling. Some NER systems, notably [129], resort to pre-trained language models. End-toend-models based on large pre-trained language models suffer from high computational costs, with the associated environmental costs [141]; reduced inclusivity in multilingual settings (e.g., GPT-3 is currently only available for English, and training it for a new language has been estimated to cost more than USD 4 million with current hardware [142]); as well as lack of explainability, which can be provided with parsing.…”

“…Although these studies demonstrate the potential benefits of incorporating syntactic information, they are limited in either treating noisy syntactic information as gold references for training their taggers, or using direct concatenation to combine that information with context information without weighing it with respect to its contribution to the NER task. Tian et al [129] tried to find a better way to incorporate syntactic information into deep learning models for NER. For this purpose, they built BioKMNER, a NER model for biomedical texts based on Key-Value Memory Networks (KVMN) [130].…”

“…The system by Tian et al [129] is based on bioBERT [131], a pre-trained biomedical language model designed for biomedical text mining tasks. It is worth remarking that there is a recent trend in end-to-end NLP systems that use powerful pre-trained language models with huge parameter spaces based on transformers to solve a variety of tasks, as in the case of the Bidirectional Encoder Representations from Transformers (BERT) [132] or Generative Pre-trained Transformer 3 (GPT-3) models [133].…”

Discovering and Summarizing Relationships Between Chemicals, Genes, Proteins, and Diseases in PubChem

“…While available data includes trusted curated sets, experimental data provided by various depositors, as well as literature and biomedical publications that are annotated manually by indexers ( MEDLINE, 2021 ); an abundance of data can be extracted from unstructured text using named-entity recognition software ( Ratinov, 2009 ). Current named-entity recognition approaches include dictionary matching, use of rules to recognize specialized terminology, and context analysis using statistical and neural language models ( Sayle et al, 2011 ; Vazquez et al, 2011 ; Jessop et al, 2012 ; Rocktäschel et al, 2012 ; Gurulingappa et al, 2013 ; Lowe and Sayle, 2015 ; Pletscher-Frankild et al, 2015 ; Song et al, 2018 ; Devlin et al, 2019 ; Lee et al, 2020 ; Tian et al, 2020 ). To produce data for the PubChem literature knowledge panels, entities are annotated in a PubMed record using a third-party named-entity recognition software, LeadMine ( Lowe and Sayle, 2015 ), and matched to chemical synonyms in the PubChem Compound database and to gene, protein, and disease names, as described in Materials and Methods .…”