Chinese Named Entity Recognition based on BERT-Transformer-BiLSTM-CRF Model

Gan, Yong; Yang, Ruisen; Zhang, Chenfang; Jia, Dongwei

doi:10.1109/isssr53171.2021.00029

Cited by 5 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(6) BERT-Transformer-BiLSTM-CRF [23]: The BERT pre-trained model generates character vectors, the Transformer encoding area constructs contextual long-range semantic features of text, the BiLSTM neural network extracts semantic features, and the CRF inference layer classifies different entities.…”

Section: Results and Analysis A Comparative Experimental Designmentioning

confidence: 99%

Chinese Named Entity Recognition of Epidemiological Investigation of Information on COVID-19 Based on BERT

et al. 2022

View full text Add to dashboard Cite

The named entity recognition based on the epidemiological investigation of information on COVID-19 can help analyze the source and route of transmission of the epidemic to control the spread of the epidemic better. Therefore, this paper proposes a Chinese named entity recognition model BERT-BiLSTM-IDCNN-ELU-CRF (BBIEC) based on the epidemiological investigation of information on COVID-19 of the BERT pre-training model. The model first processes the unlabeled epidemiological investigation of information on COVID-19 into the character-level corpus and annotates it with artificial entities according to the BIOES character-level labeling system and then uses the BERT pre-training model to obtain the word vector with position information; then, through the bidirectional long-short term memory neural network (BiLSTM) and the improved iterated dilated convolutional neural network (IDCNN) extract global context and local features from the generated word vectors and concatenate them serially; output all possible label sequences to the conditional random field (CRF); finally pass the condition random The airport decodes and generates the entity tag sequence. The experimental results show that the model is better than other traditional models in recognizing the entity of the epidemiological investigation of information on COVID-19.INDEX TERMS chinese named entity recognition,the epidemiological investigation of information on COVID-19,bidirectional encoder representations from transformer, bidirectional long-short term memory network, iterated dilated convolutional neural network, conditional random field

show abstract

Section: Results and Analysis A Comparative Experimental Designmentioning

confidence: 99%

Chinese Named Entity Recognition of Epidemiological Investigation of Information on COVID-19 Based on BERT

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Even though the current performance in data constrained settings is not ideal, there can be lot of work to carry out towards better model performance via mode refinement, optimisation, and augmentation, e.g. the more complex structures and system combinations used by Wang and Su (2022); Gan et al (2021);Yan et al (2019).…”

Section: Discussionmentioning

confidence: 99%

“…To facilitate this research, we adopt off-the-shelf open-source Bioformer 5 from HuggingFace (Wolf et al, 2019) which is one of the latest PLMs in biomedical domain and reported comparable and even better scores than BioBERT and PubMed- and Wang, 2019;Gan et al, 2021;Zheng et al, 2021) which applied Transformer and CRF combined models for spoken language understanding, Chinese NER, and power meter NER 6 . However, none of above mentioned work has ever applied TransformerCRF models into clinical domain text mining which topic is the focus of this paper.…”

Section: Related Workmentioning

confidence: 99%

“…Pre-trained language models (PLMs) with fine tuning have been one of the dominant methods in current natural language processing (NLP) tasks, including text mining (Zhang et al, 2021), named entity recognition (Dernoncourt et al, 2017), reading comprehension (Sun et al, 2020), machine translation (Vaswani et al, 2017;Devlin et al, 2019), and summarisation (Gokhan et al, 2021;Wu et al, 2022), etc. They also demonstrated strong performances in comparison to conventional methods.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On Cross-Domain Pre-Trained Language Models for Clinical Text Mining: How Do They Perform on Data-Constrained Fine-Tuning?

Wu¹,

Han²,

Antonini³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have been deployed in many natural language processing (NLP) tasks and in various domains. Language model pre-training from general or mixed domain rich data plus fine-tuning using small amounts of available data in a low resource domain demonstrated beneficial results by researchers. In this work, we question this statement and verify if BERT-based PLMs from the biomedical domain can perform well in clinical text mining tasks via fine-tuning. We test the state-of-the-art models, i.e. Bioformer which is pre-trained on a large amount of biomedical data from PubMed corpus. We use a historical n2c2 clinical NLP challenge dataset for fine-tuning its task-adapted version (BioformerApt), and show that their performances are actually very low. We also present our own end-to-end model, TransformerCRF, which is developed using Transformer and conditional random fields (CRFs) as encoder and decoder. We further create a new variation model by adding a CRF layer on top of PLM Bioformer (BioformerCRF). We investigate the performances of TransformerCRF on clinical text mining tasks by training from scratch using a limited amount of data, as well as the model BioformerCRF. Experimental evaluation shows that, in a constrained setting, all tested models are far from ideal regarding extreme low-frequency special token recognition, even though they can achieve relatively higher accuracy on overall text tagging. Our models including source codes will be hosted at https://github.com/ poethan/TransformerCRF.

show abstract

“…Liu et al [39] proposed the BERT-BiLSTM-CRF recognition method and applied it to research on citrus pests and diseases. Gan et al [40] proposed the BERT-Transformer-BiLSTM-CRF model to handle the challenges posed by pronouns and polysemous words in Chinese NER. Li et al [41] addressed the issue of excessive parameters and long training times in BERT by introducing the BERT-IDCNN-CRF model.…”

Section: Methods Based On Deep Learningmentioning

confidence: 99%

LB-BMBC: MHBiaffine-CNN to Capture Span Scores with BERT Injected with Lexical Information for Chinese NER

Guo,

Zhang

2024

Int J Comput Intell Syst

View full text Add to dashboard Cite

A substantial body of research has shown that introducing lexical information in Chinese Named Entity Recognition (NER) tasks can enhance the semantic and boundary information of Chinese words. However, in most methods, the introduction of lexical information occurs at the model architecture level, which cannot fully leverage the lexicon learning capability of pre-trained models. Therefore, we propose seamless integration of external Lexicon knowledge into the Transformer layer of BERT. Additionally, we have observed that in span-based recognition, adjacent spans have special spatial relationships. To capture this relationship, we extend the work after Biaffine and use Convolutional Neural Networks (CNN) to treat the score matrix as an image, allowing us to interact with the spatial relationships of spans. Our proposed LB-BMBC model was experimented on four publicly available Chinese NER datasets: Resume, Weibo, OntoNotes v4, and MSRA. In particular, during ablation experiments, we found that CNN can significantly improve performance.

show abstract

Chinese Named Entity Recognition based on BERT-Transformer-BiLSTM-CRF Model

Cited by 5 publications

References 13 publications

Chinese Named Entity Recognition of Epidemiological Investigation of Information on COVID-19 Based on BERT

Chinese Named Entity Recognition of Epidemiological Investigation of Information on COVID-19 Based on BERT

On Cross-Domain Pre-Trained Language Models for Clinical Text Mining: How Do They Perform on Data-Constrained Fine-Tuning?

LB-BMBC: MHBiaffine-CNN to Capture Span Scores with BERT Injected with Lexical Information for Chinese NER

Contact Info

Product

Resources

About