Learning Constraints for Structured Prediction Using Rectifier Networks

Abstract

Various natural language processing tasks are structured prediction problems where outputs are constructed with multiple interdependent decisions. Past work has shown that domain knowledge, framed as constraints over the output space, can help improve predictive accuracy. However, designing good constraints often relies on domain expertise. In this paper, we study the problem of learning such constraints. We frame the problem as that of training a two-layer rectifier network to identify valid structures or substructures, and show a construction for converting a trained network into a system of linear constraints over the inference variables. Our experiments on several NLP tasks show that the learned constraints can improve the prediction accuracy, especially when the number of training examples is small.

New Frontiers of Information Extraction

“…Besides, we will also discuss about unsuper-vised techniques for out-of-distribution (OOD) detection (Zhou et al, 2021b;Hendrycks et al, 2020), prediction with abstention (Dhamija et al, 2018;Hendrycks et al, 2018) and novelty class detection (Perera and Patel, 2019) that seek to help the IE model identify invalid inputs or inputs with semantic shifts during its inference phase. Specifically, to demonstrate how models can ensure the global consistency of the extraction, we will cover constraint learning methods that automatically capture logical constraints among relations (Wang et al, , 2022cPan et al, 2020), and techniques to enforce the constraints in inference Li et al, 2019a;Han et al, 2019;Lin et al, 2020). To assess if the systems give faithful extracts, we will also talk about the spurious correlation problems of current IE models and how to address them with counterfactual analysis Qian et al, 2021).…”

Learning Constraints and Descriptive Segmentation for Subevent Detection

“…In terms of enforcing declarative constraints in neural models, early efforts (Roth and Yih, 2004; formulate the inference process as Integer Linear Programming (ILP) problems. Pan et al (2020) also employ ILP to enforce constraints learned automatically from Rectifier Networks with strong expressiveness (Pan and Srikumar, 2016). Yet the main drawback of solving an ILP problem is its inefficiency in a large feasible solution space.…”

“…When we construct three-event subgraphs from documents, a binary label t for structure legitimacy is created for each subgraph. Inspired by how constraints are learned for several structured prediction tasks (Pan et al, 2020), we represent constraints for a given subgraph-label pair (X, t) as K linear inequalities. 2 Formally, t = 1 if X satisfies constraints c k for all k = 1, • • • , K. And the k th constraint c k is expressed by a linear inequality…”

“…whose weights w k and bias b k are learned. Since a system of linear inequalities is proved to be equivalent to the Rectifier Network proposed in Pan et al (2020), we adopt a two-layer rectifier network for 2 Here we assume K constraints is the upper bound for all the rules to be learned. Figure 2: A legitimate structure for three-event subgraph obtained from the example shown in Fig.…”

“…To capture the logical dependency between subevent structure and EVENTSEG, our second contribution is an approach to automatically learning and enforcing logical constraints. Motivated by Pan et al (2020), we use Rectifier Networks to learn constraints in the form of linear inequalities, and then convert the constraints to a regularization term that can be incorporated into the loss function of the neural model. This allows any hard-to-articulate constraints to be automatically captured for interrelated tasks, and efficiently guides the model to make globally consistent inference.…”