BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition

Abstract

We study the problem of learning a named entity recognition (NER) tagger using noisy labels from multiple weak supervision sources. Though cheap to obtain, the labels from weak supervision sources are often incomplete, inaccurate, and contradictory, making it difficult to learn an accurate NER model. To address this challenge, we propose a conditional hidden Markov model (CHMM), which can effectively infer true labels from multi-source noisy labels in an unsupervised way. CHMM enhances the classic hidden Markov model with the contextual representation power of pretrained language models. Specifically, CHMM learns token-wise transition and emission probabilities from the BERT embeddings of the input tokens to infer the latent true labels from noisy observations. We further refine CHMM with an alternate-training approach (CHMM-ALT). It fine-tunes a BERT-NER model with the labels inferred by CHMM, and this BERT-NER's output is regarded as an additional weak source to train the CHMM in return. Experiments on four NER benchmarks from various domains show that our method outperforms state-of-the-art weakly supervised NER models by wide margins.

Named entity recognition of chemical experiment operations based on BERT

“…Early named entity recognition tasks were generally based on manually constructed rules and dictionary methods [9]，but entity recognition is heavily reliant on human intervention, which often leads to limitations in terms of cost control and portability. With the development of machine learning techniques, researchers have utilized statistical machine learning algorithms to construct mathematical models, such as Hidden Markov Models (HMM) [10,11] and Support Vector Machines (SVM) [12][13][14]. Although these models have achieved good results in some corpora, they can only extract partial contextual information.…”

Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler

“…(1) In probabilistic graphical model approach (and in addition to the HMM-based models [20,21,26,36,39]), Rodrigues et al [32] in early 2014 used a partially directed graph containing a CRF for modeling to solve the truth inference from crowdsourcing labels; (2) In deep learning model approach (and in addition to the "source-specific perturbation" methods [17,26,46]), other methods [17,[33][34][35] are either based on the end-to-end deep neural architecture [33], or the customized optimization objective along with coordinate ascent optimization technology [34,35], or the iterative solving framework similar to expectation-maximization algorithm [4]. However, all these methods do not have the advantages of the recently proposed neuralized HMM-based graphical models [18,19] and our Neural-Hidden-CRF in principled modeling for variants of interest and in harnessing the context information that provided by advanced deep learning models. Additionally, it is worth mentioning the presence of numerous established WS methods that address the normal independent classification scenario [3,5,[43][44][45].…”

“…This global normalization approach is unlike all HMMs, where (y (𝑖 ) , t (𝑖 ) ) is split into multiple uni-directional dependent random variables (i.e., a set consisting of many (𝑦 (𝑖,𝑗 ) 𝑙 , 𝑡 (𝑖 ) 𝑙 )) based on some strict independence assumptions and each conditional probability distribution between random variables is normalized (e.g., local normalization for per-step in Li et al [18,19]) to further obtain the probability of the joint distribution (y (𝑖 ) , t (𝑖 ) ). Simply put, our approach models/trains holistically (the learned knowledge is global), while the HMMs [18,19] decompose the modeling into multiple uni-directional dependent local regions and model the patterns for the scale of a step (the learned knowledge is local). As a result of the holistic undirected graphical modeling, our method can result in model parameters that are not constrained by probabilistic forms, thus enjoying more flexible scoring.…”

“…(Then, these inferred labels, in turn, can be used to train a final sequence labeler.) Though principled, these models fall short in leveraging token semantics and context information [18], as they either model input tokens as one-hot observations [26,39] or do not model them at all [20,21,36]. • The "source-specific perturbation" deep learning models [17,26,46], train multiple weak source-specific deep models, obtained by inserting the source-specific perturbation parameters to the unique shared deep model parameters, and perform test using the assumed optimal classifier obtained by the shared deep model straightforwardly or a certain combination of the source-specific deep models.…”

“…• The recently proposed neuralized HMM-based graphical models [18,19] construct HMM-based directed graphical models in which the dependencies among variables of word sequence, latent ground truth sequences and weak labels are sophisticatedly modeled, and rich contextual semantic information is introduced using deep learning techniques (e.g., the language model BERT [8]).…”