End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Abstract

State-of-the-art sequence labeling systems traditionally require large amounts of taskspecific knowledge in the form of handcrafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both word-and character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF. Our system is truly end-to-end, requiring no feature engineering or data preprocessing, thus making it applicable to a wide range of sequence labeling tasks. We evaluate our system on two data sets for two sequence labeling tasks -Penn Treebank WSJ corpus for part-of-speech (POS) tagging and CoNLL 2003 corpus for named entity recognition (NER). We obtain state-of-the-art performance on both datasets -97.55% accuracy for POS tagging and 91.21% F1 for NER.

Deep Reference Mining From Scholarly Literature in the Arts and Humanities

“…More recent work on sequence labeling tasks relies instead on deep learning techniques such as convolutional or recurrent neural network models (CNNs LeCun et al, 1989 andRNNs Rumelhart, 1986, respectively), without the need for any hand-crafted features (Kim, 2014;Huang et al, 2015;Zhang et al, 2015;Chiu and Nichols, 2016;Lample et al, 2016;Ma and Hovy, 2016;Yang et al, 2016;Strubell et al, 2017). RNNs in particular, typically rely on a neural network architecture built using one or more Bidirectional LongShort Term Memory (BiLSTM) layers, as this type of neural cell provides for variable-length memory allowing the model to capture relationships within sequences of proximal words.…”

“…Such architectures have achieved state-of-the-art performance for both POS and NER tasks on popular datasets (Reimers and Gurevych, 2017b). Current state-of-the-art architectures for sequence labeling include the use of a CRF prediction layer (Huang et al, 2015) and the use of character-level word embeddings to complement word embeddings, trained either with CNNs (Ma and Hovy, 2016) or BiLSTM RNNs (Lample et al, 2016). Character-level word embeddings have indeed been shown to perform well on a variety of NLP tasks (Dos Santos and Gatti de Bayser, 2014;Kim et al, 2015;Zhang et al, 2015).…”

“…Attention mechanisms have also been proposed for the same tasks (Rei et al, 2016;Shen and Lee, 2016). In this paper we will apply, tune and compare two architectures (Lample et al, 2016;Ma and Hovy, 2016) to the specific task of reference mining.…”

“…We consider a recurrent architecture organized into three layers: input (word representations), inner and prediction, following the best performing models for sequence labeling tasks (Lample et al, 2016;Ma and Hovy, 2016). The network firstly receives a sequence of (one-hot encoded) words w (1) , w (2) , ..., w (n) as input and transforms it into a sequence of dense vectors x (1) , x (2) , ..., x (n) , using a combination of word and character-level word embeddings.…”

A Chinese Named Entity Recognition System with Neural Networks

“…Lample et al [8] gives a more specific implementation of neural network model for NER tasks, their model introduced pretrained word embeddings from large corpus, and gives F1 scores of 90.97% on CoNLL 2003 dataset using no context features or spelling features. Ma et al [9] combines Chiu's CNN model [10] and Lample's bidirectional LSTM-CRF model [8] for end-to-end SLPs. These models use neural networks strategies and/or pretrained word embeddings to give better generalization performance.…”