Modeling Fine-Grained Entity Types with Box Embeddings

Abstract

Neural entity typing models typically represent fine-grained entity types as vectors in a high-dimensional space, but such spaces are not well-suited to modeling these types' complex interdependencies. We study the ability of box embeddings, which embed concepts as d-dimensional hyperrectangles, to capture hierarchies of types even when these relationships are not defined explicitly in the ontology. Our model represents both types and entity mentions as boxes. Each mention and its context are fed into a BERT-based model to embed that mention in our box space; essentially, this model leverages typological clues present in the surface text to hypothesize a type representation for the mention. Box containment can then be used to derive both the posterior probability of a mention exhibiting a given type and the conditional probability relations between types themselves. We compare our approach with a vector-based typing model and observe state-of-the-art performance on several entity typing benchmarks. In addition to competitive typing performance, our box-based model shows better performance in prediction consistency (predicting a supertype and a subtype together) and confidence (i.e., calibration), demonstrating that the box-based model captures the latent type hierarchies better than the vector-based model does. 1

Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference

“…Onoe and Durrett (2019) trained a filtering and relabeling model with the human annotated data to denoise the automatically generated data for training. Onoe et al (2021) introduced box embeddings (Vilnis et al, 2018) to represent the dependency among multiple levels of type labels as topology of axis-aligned hyperrectangles (boxes). To further cope with insufficient training data, Dai et al (2021) used pretrained language model for augmenting (noisy) training data with masked entity generation.…”

“…Moreover, to handle the dependency of type labels in different granularities, we also utilize the inference ability of NLI model to learn that the finer label hypothesis of an entity mention entails its general label hypothesis. Experimental results on the UFET benchmark (Choi et al, 2018) show that LITE drastically outperforms the recent state-of-the-art (SOTA) systems (Dai et al, 2021;Onoe et al, 2021;Liu et al, 2021) without any need of distantly supervised data as they do. In addition, our LITE also yields the best performance on traditional (less) fine-grained entity typing tasks.…”

TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrieval

“…Entity typing A task closely related to our work is entity typing, or predicting the set of types for a mention (e.g., 31,16,37). A key difference is that entity typing methods often learn explicit type embeddings to perform type classification, whereas TABi only learns query and entity embeddings.…”

Box Embeddings: An open-source library for representation learning using geometric structures

“…Representing relations between the nodes of a hierarchy is useful for various NLP and Machine Learning tasks such as natural language inference (Wang et al, 2019;Sharma et al, 2019), entity typing (Onoe et al, 2021), multi-label classification (Chatterjee et al, 2021), and question answering (Jin et al, 2019;Fang et al, 2020). For example, in Figure 2, knowing the hypernym relationship between the pairs (herb, basil), (herb, thyme), and (herb, rosemary) can help paraphrase the sentence "This dish requires basil, thyme and rosemary" into "This dish requires several herbs.".…”

“…Various applications of probabilistic box embeddings (eg. modeling joint-hierarchies (Patel et al, 2020), uncertain knowledge graph representation (Chen et al, 2021), or fine-grained entity typing (Onoe et al, 2021)) have relied on bespoke implementations, adding unnecessary difficulty and differences in implementation when applying box embeddings to new tasks. To mitigate this issue and make applying and extending box embeddings easier, we saw the need to introduce a reusable, unified, stable library that provides the basic functionalities needed in studying box embeddings.…”