Delayed Combination of Feature Embedding in Bidirectional LSTM CRF for NER

Abstract

Named Entity Recognition (NER) plays a vital role in natural language processing (NLP). Currently, deep neural network models have achieved significant success in NER. Recent advances in NER systems have introduced various feature selections to identify appropriate representations and handle Out-Of-the-Vocabulary (OOV) words. After selecting the features, they are all concatenated at the embedding layer before being fed into a model to label the input sequences. However, when concatenating the features, information collisions may occur and this would cause the limitation or degradation of the performance. To overcome the information collisions, some works tried to directly connect some features to latter layers, which we call the delayed combination and show its effectiveness by comparing it to the early combination. As feature encodings for input, we selected the character-level Convolutional Neural Network (CNN) or Long Short-Term Memory (LSTM) word encoding, the pre-trained word embedding, and the contextual word embedding and additionally designed CNN-based sentence encoding using a dictionary. These feature encodings are combined at early or delayed position of the bidirectional LSTM Conditional Random Field (CRF) model according to each feature’s characteristics. We evaluated the performance of this model on the CoNLL 2003 and OntoNotes 5.0 datasets using the F1 score and compared the delayed combination model with our own implementation of the early combination as well as the previous works. This comparison convinces us that our delayed combination is more effective than the early one and also highly competitive.

Information extraction model from Ge’ez texts

“…From recent literature, there has been little research on information extraction systems in Ethiopian and foreign languages [3], [7], [8], [10]. Because of its grammatical nuances, syntactic structure, lexical structure, writing system, morphological structure, and encoding style, existing approaches created for other languages are difficult to apply directly to the Ge'ez language [21].…”

Forecasting PM2.5 Concentration Using a Single-Dense Layer BiLSTM Method

“…Unlike a traditional LSTM, the BiLSTM can run input sequence in two ways: forward layer from past to future and backward layer from future to past. By this approach, two hidden states from two LSTM models are combined with preserving information from both the past and future [38], which improve the long-term learning dependencies and the model's accuracy.…”

Context-Aware Bidirectional Neural Model for Sindhi Named Entity Recognition

“…More recently, NN-based models have attained a state-of-the-art status in NER [6,15,22,23] and other sequence tagging tasks [8,24]. These models primarily rely on the feature representation at character-level [10,11], word-level [25,26], and joint character-level and word-level [9,13] to encode a word boundary and its context.…”

“…The limitation of the pretrained word embedding model is the representation of a word in a single embedding without encoding its context [23].To tackle this drawback, two different word representation techniques have been developed: one is pattern-based wordlevel encoding, and the other is LSTM-based character-level word encoding. The former (word-level) approach makes it possible to learn pattern-based word representations by encoding them with different literals but with the same representations [8,9].…”