Embedded-State Latent Conditional Random Fields for Sequence Labeling

Abstract

Complex textual information extraction tasks are often posed as sequence labeling or shallow parsing, where fields are extracted using local labels made consistent through probabilistic inference in a graphical model with constrained transitions. Recently, it has become common to locally parametrize these models using rich features extracted by recurrent neural networks (such as LSTM), while enforcing consistent outputs through a simple linear-chain model, representing Markovian dependencies between successive labels. However, the simple graphical model structure belies the often complex non-local constraints between output labels. For example, many fields, such as a first name, can only occur a fixed number of times, or in the presence of other fields. While RNNs have provided increasingly powerful context-aware local features for sequence tagging, they have yet to be integrated with a global graphical model of similar expressivity in the output distribution. Our model goes beyond the linear chain CRF to incorporate multiple hidden states per output label, but parametrizes their transitions parsimoniously with low-rank logpotential scoring matrices, effectively learning an embedding space for hidden states. This augmented latent space of inference variables complements the rich feature representation of the RNN, and allows exact global inference obeying complex, learned non-local output constraints. We experiment with several datasets and show that the model outperforms baseline CRF+RNN models when global output constraints are necessary at inference-time, and explore the interpretable latent structure.

Using BibTeX to Automatically Generate Labeled Data for Citation Field Extraction

“…We train a variety of citation field extraction models including one based on RoBERTa [Liu et al, 2019]. We show that this model trained only on the UMass CFE dataset matches state-ofthe-art results [Thai et al, 2018]. We then show that training the BERT-based model on our large automatically generated dataset drastically improves the results, outperforming the state of the art approach by 1.2 points of F1, a 24.48% relative reduction in error.…”

Low-Rank Constraints for Fast Inference in Structured Models

“…Low-rank structure has been explored in both HMMs [Siddiqi et al, 2009], a generalization of PCFGs called weighted tree automata [Rabusseau et al, 2015], and conditional random fields [Thai et al, 2018]. The reduced-rank HMM [Siddiqi et al, 2009] has at most 50 states, and relies on spectral methods for training.…”

“…We extend the low-rank assumption to neural parameterizations, which have been shown to be effective for generalization [Kim et al, 2019, Chiu andRush, 2020], and directly optimize the evidence via gradient descent. Finally, Thai et al [2018] do not take advantage of the low-rank parameterization of their CRF potentials for faster inference via low-rank matrix products, a missed opportunity. Instead, the low-rank parameterization is used only as a regularizer, with the full potentials instantiated during inference.…”

Factored Latent-Dynamic Conditional Random Fields for Single and Multi-label Sequence Modeling

“…LDCRF has been shown to outperform HMM, CRF and HCRF on several sequence labeling tasks Sun et al, 2008). Thai et al (2018) recently proposed a very similar model called Embedded-State Latent CRFs. They claim to factorize the log potential as the novelty over a LDCRF, however such factorization is not reflected in their model structure and mathematical descriptions.…”