Pooled Contextualized Embeddings for Named Entity Recognition

Abstract

Contextual string embeddings are a recent type of contextualized word embedding that were shown to yield state-of-the-art results when utilized in a range of sequence labeling tasks. They are based on character-level language models which treat text as distributions over characters and are capable of generating embeddings for any string of characters within any textual context. However, such purely character-based approaches struggle to produce meaningful embeddings if a rare string is used in a underspecified context. To address this drawback, we propose a method in which we dynamically aggregate contextualized embeddings of each unique string that we encounter. We then use a pooling operation to distill a global word representation from all contextualized instances. We evaluate these pooled contextualized embeddings on common named entity recognition (NER) tasks such as CoNLL-03 and WNUT and show that our approach significantly improves the state-of-the-art for NER. We make all code and pre-trained models available to the research community for use and reproduction.

Leverage Lexical Knowledge for Chinese Named Entity Recognition via Collaborative Graph Network

“…This includes statistic methods, such as SVM (Isozaki and Kazawa, 2002), HMMs (Bikel et al, 1997) and CRF (Lafferty et al, 2001), suffering from feature engineering. There are also a number of recent neural network approaches applied to NER, such as (Collobert et al, 2011;Huang et al, 2015;Lample et al, 2016;Ma and Hovy, 2016;Chiu and Nichols, 2016;Akbik et al, 2018;Jie et al, 2019;Akbik et al, 2019 (Bastings et al, 2017;Yao et al, 2019;Wang et al, 2018;Mishra et al, 2019;Cao et al, 2019;Zhang et al, 2019). Cetoli et al (2017) use GCN to investigate the role of the dependency tree in English named entity recognition.…”

Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition

“…Neural architecture search has been proposed to automatically search for better architectures, showing competitive results on several tasks, e.g., image recognition and language modeling. A s- Model F1 best published BiLSTM-CRF (Lample et al, 2016) 90.94 BiLSTM-CRF+ELMo (Peters et al, 2018) 92.22 BERT Base (Devlin et al, 2018) 92.40 BERT Large (Devlin et al, 2018) 92.80 BiLSTM-CRF+PCE (Akbik et al, 2019) 93 trand of NAS research focuses on reinforcement learning (Zoph and Le, 2016) and evolutionary algorithm-based (Xie and Yuille, 2017) methods. They are powerful but inefficient.…”

When Specialization Helps: Using Pooled Contextualized Embeddings to Detect Chemical and Biomedical Entities in Spanish

“…For our best performing model, we used two different token-level embeddings, a WANG2VECbased embedding (Ling et al, 2015) and a FAST-TEXT-based embedding (Bojanowski et al, 2017), a single byte-pair sub-word embedding (Heinzerling and Strube, 2018) and one context sensitive character-level language model (Akbik et al, 2019b). Figure 1 gives a visual depiction of our best performing model.…”