Lightly-supervised Representation Learning with Global Interpretability

Abstract

We propose a lightly-supervised approach for information extraction, in particular named entity classification, which combines the benefits of traditional bootstrapping, i.e., use of limited annotations and interpretability of extraction patterns, with the robust learning approaches proposed in representation learning. Our algorithm iteratively learns custom embeddings for both the multi-word entities to be extracted and the patterns that match them from a few example entities per category. We demonstrate that this representation-based approach outperforms three other state-of-theart bootstrapping approaches on two datasets: CoNLL-2003 and OntoNotes. Additionally, using these embeddings, our approach outputs a globally-interpretable model consisting of a decision list, by ranking patterns based on their proximity to the average entity embedding in a given class. We show that this interpretable model performs close to our complete bootstrapping model, proving that representation learning can be used to produce interpretable models with small loss in performance. This decision list can be edited by human experts to mitigate some of that loss and in some cases outperform the original model.

End-to-End Bootstrapping Neural Network for Entity Set Expansion

“…Because the seed entities are sparse comparing to the number of unexpanded entities, the learned Boot-strapTeacher tends to be underfitting. Inspired by Zupon et al (2019), and based on the intuition that the pattern and entity embeddings are similar to their neighbors but dissimilar to their unrelated patterns or entities, we leverage the graph structure as a regularizer to the learning procedure, and let the BootstrapTeacher maximize the following unsupervised learning objective:…”

“…Datasets: We use two datasets, CoNLL and OntoNotes, constructed by Zupon et al (2019). CoNLL is constructed from the CoNLL 2003 shared task dataset (Tjong Kim Sang and De Meulder 2003), which contains 4 entity types.…”

“…OntoNotes is constructed from the OntoNotes datasets (Pradhan et al 2013) without numerical categories, which finally contains 11 entity types. Zupon et al (2019) use the n-grams of the size up to 4 tokens on either side of an entity as the patterns and filter out some patterns. Baselines.…”

“…• Gupta (Gupta and Manning 2014): this method is a classical bootstrapping system that iteratively evaluates and selects patterns, and scores new entities by a learned entity classifier 4 . • Emboot (Zupon et al 2019): this method follows Gupta and Manning (2014), but learns custom embeddings for entities and patterns at each iteration, which are used to guide the entity classifier. • LTB (Yan et al 2019): this method uses the MCTS algorithm to perform lookahead search for estimating delayed feedback and jointly learns an entity similarity function.…”

“…Second, it is hard to exploit the high-order entity-pattern relations for entity set expansion. Due to the lack of extra supervision, previous studies mainly used the entity-pattern matching statistics for entity/pattern evaluation (Riloff and Jones 1999;Curran, Murphy, and Scholz 2007;Shi et al 2014;Zupon et al 2019). However, these methods only exploit the first-order entity-pattern relations, while ignoring the useful information from the high-order relations, which have been proven useful in many information extraction tasks (Riedel et al 2013;Chen et al 2006).…”