BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition

Abstract

We study the problem of learning a named entity recognition (NER) tagger using noisy labels from multiple weak supervision sources. Though cheap to obtain, the labels from weak supervision sources are often incomplete, inaccurate, and contradictory, making it difficult to learn an accurate NER model. To address this challenge, we propose a conditional hidden Markov model (CHMM), which can effectively infer true labels from multi-source noisy labels in an unsupervised way. CHMM enhances the classic hidden Markov model with the contextual representation power of pretrained language models. Specifically, CHMM learns token-wise transition and emission probabilities from the BERT embeddings of the input tokens to infer the latent true labels from noisy observations. We further refine CHMM with an alternate-training approach (CHMM-ALT). It fine-tunes a BERT-NER model with the labels inferred by CHMM, and this BERT-NER's output is regarded as an additional weak source to train the CHMM in return. Experiments on four NER benchmarks from various domains show that our method outperforms state-of-the-art weakly supervised NER models by wide margins.

WRENCH: A Comprehensive Benchmark for Weak Supervision

“…Most studies focus on developing label models while leaving the end model flexible to the downstream tasks. Existing label models include Majority Voting (MV), Probabilistic Graphical Models (PGM) [14,77,75,22,53,82,50], etc.. Note that prior crowd-worker modeling work can be included and subsumed by this set of approaches, e.g.…”

“…[53,71] use a standard HMM with multiple observed variables, each from one labeling source. [82] improves HMM by introducing unique linking rules as an additional supervision source; [50] predicts token-wise transition and emission probabilities from BERT embeddings to utilize the context information. Besides, [45] is an one-stage method that models each labeling source by a CRF layer and aggregates their transitions with an attention network.…”

“…(We discuss the detailed procedure on adapting label model for sequence tagging tasks in Appendix D) However, these models neglect the internal dependency between labels within the sequence. In contrast, HMM and CHMM take the whole Following the standard protocols, we use entity-level F1-score as the metric [53,60] and use BIO schema [88,50,53], which labels the beginning token of an entity as B-X and the other tokens inside that entity as I-X, while non-entity tokens are marked as O. For methods that predict token-level labels (e.g.MV), we transform token-level predictions to entity-level predictions when calculating the F1 score.…”

“…This dataset accompanies the BioCreative V CDR challenge and consists of 1,500 PubMed articles and is annotated with chemical and disease mentions. The labeling functions are selected from [50]. (We use DictCore-Chemical, DictCore-Chemical-Exact, DictCore-Disease, DictCore-Disease-Exact, Element, Ion, or Isotope, Organic Chemical, Antibiotic, Disease or Syndrome, PostHyphen, ExtractedPhrase as weak supervision sources.)…”

“…To reduce the efforts of annotation, recent weak supervision (WS) frameworks have been proposed which focus on enabling users to leverage a diversity of weaker, often programmatic supervision sources [76,77,75] to label and manage training data in an efficient way. Recently, WS has been widely applied to various machine learning tasks in a diversity of domains: scene graph prediction [9], video analysis [23,92], image classification [12], image segmentation [35], autonomous driving [96], relation extraction [36,107,57], named entity recognition [82,53,50,45,27], text classification [78,100,85,86], dialogue system [63], biomedical [43,19,64], healthcare [20,17,21,80,93,81], software engineering [74], sensors data [24,39], E-commerce [66,103], and multi-agent systems [102].…”