Unified Semantic Typing with Meaningful Label Inference

Abstract

Semantic typing aims at classifying tokens or spans of interest in a textual context into semantic categories such as relations, entity types, and event types. The inferred labels of semantic categories meaningfully interpret how machines understand components of text. In this paper, we present UNIST, a unified framework for semantic typing that captures label semantics by projecting both inputs and labels into a joint semantic embedding space. To formulate different lexical and relational semantic typing tasks as a unified task, we incorporate task descriptions to be jointly encoded with the input, allowing UNIST to be adapted to different tasks without introducing task-specific model components. UNIST optimizes a margin ranking loss such that the semantic relatedness of the input and labels is reflected from their embedding similarity. Our experiments demonstrate that UNIST achieves strong performance across three semantic typing tasks: entity typing, relation classification and event typing. Meanwhile, UNIST effectively transfers semantic knowledge of labels and substantially improves generalizability on inferring rarely seen and unseen types. In addition, multiple semantic typing tasks can be jointly trained within the unified framework, leading to a single compact multi-tasking model that performs comparably to dedicated single-task models, while offering even better transferability. 1 * Equal contributions. 1 Our code and pre-trained models are available at https: //github.com/luka-group/UniST.

Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy

“…Following the original paper, we report macro-averaged precision, recall and F1. We compare our model with Box4Types (Onoe et al, 2021), LRN (Liu et al, 2021b), MLMET (Dai et al, 2021), DenoiseFET (Pan et al, 2022), UNIST (Huang et al, 2022), NPCRF (Jiang et al, 2022) and LITE . The baseline results were obtained from the original papers.…”

“…Pan et al (2022) initialise the scoring function for each label based on that label's representation in the decoder of the language model (see Section 3). Rather than using pre-trained embeddings, Huang et al (2022) encode the labels using a language model, which is fine-tuned together with the entity mention encoder. A similar approach was used by Ma et al (2022) in the context of few-shot entity typing.…”

“…State-of-the-art methods, therefore rely on prior knowledge about the labels themselves. For instance, Pan et al (2022) use the representation of the labels in the decoder of BERT to initialise the weights of the label classifiers, while Huang et al (2022) use a fine-tuned BERT (Devlin et al, 2019) model to map type labels onto prototypes. However, the token embeddings from BERT are known to be sub-optimal as pre-trained representations of word meaning (Bommasani et al, 2020), whereas fine-tuning a BERT encoder may lead to overfitting.…”

Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing

“…BiLSTM (Choi et al, 2018) 47.1 24.2 32.0 BERT (Onoe and Durrett, 2019) 51.6 33.0 40.2 Box4Types (Onoe et al, 2021) 52.8 38.8 44.8 MLMET (Dai et al, 2021) 53.6 45.3 49.1 UniST (Huang et al, 2022) 50.2 49.6 49.9 LITE 52.4 48.9 50.6 CASENT (Ours) 53.3 49.5 51.3 the F1 score. Among the fully-supervised models, cross-encoder models demonstrate superior performance over both bi-encoder methods and multilabel classifier-based models.…”

Dense Retrieval as Indirect Supervision for Large-space Decision Making

“…Baselines We consider two categories of competitive entity typing models as baselines: 1) methods capturing the example-label and label-label relations, e.g., BiLSTM (Choi et al, 2018) that concatenates the context representation learned by a bidirectional LSTM and the mention representation learned by a CNN, LabelGCN (Xiong et al, 2019) that learns to encode global label co-occurrence statistics and their word-level similarities, LRN (Liu et al, 2021a) that models the coarse-to-fine label dependency as causal chains, Box4Types (Onoe et al, 2021) that captures hierarchies of types as topological relations of boxes, and UniST (Huang et al, 2022a) that conduct namebased label ranking; 2) methods leveraging inductive bias from pre-trained models for entity typing, e.g., MLMET (Dai et al, 2021) that utilizes the pretrained BERT to predict the most probable words for "[MASK]" earlier incorporated around the mention as type labels, LITE and Context-TE ) that both leverage indirect supervision from pre-trained natural language inference.…”