What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study

Abstract

The degree of semantic relatedness (or, closeness in meaning) of two units of language has long been considered fundamental to understanding meaning. Automatically determining relatedness has many applications such as question answering and summarization. However, prior NLP work has largely focused on semantic similarity (a subset of relatedness), because of a lack of relatedness datasets. Here for the first time, we introduce a dataset of semantic relatedness for sentence pairs. This dataset, STR-2021, has 5,500 English sentence pairs manually annotated for semantic relatedness using a comparative annotation framework. We show that the resulting scores have high reliability (repeat annotation correlation of 0.84). We use the dataset to explore a number of questions on what makes two sentences more semantically related. We also evaluate a suite of sentence representation methods on their ability to place pairs that are more related closer to each other in vector space.

Just Rank: Rethinking Evaluation with Word and Sentence Similarities

“…Interestingly, according to our results, even though STR evaluation does not correlate well with downstream tasks, the positive pairs collected from STR have better quality than STS-B. It also confirms the argument that STR improves the dataset collection process (Abdalla et al, 2021). Benchmarking Results.…”

“…Reimers et al (2016), Eger et al (2019) and Zhelezniak et al (2019) states current evaluation paradigm for Semantic Textual Similarity (STS) tasks are not ideal. One most recent work (Abdalla et al, 2021) questions about the data collection process of STS datasets and creates a new semantic relatedness dataset (STR) by comparative annotations (Louviere and Woodworth, 1991).…”

“…Sentence-Level. Similarly, the pairs with top 25% similarity/relatedness score from STS-Benchmark dataset (Cer et al, 2017) and STR dataset (Abdalla et al, 2021) are collected as positive pairs. All sentences that appear at least once are used as the background sentence samples.…”

“…). Recently,Abdalla et al (2021) questioned the labeling process of STS datasets and released a new semantic textual relatedness (STR) dataset, which is also included in our experiments.…”

What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitions

“…We ran pilots for obtaining importance annotations for each sentence in a contract, as well as a pair of sentences on a scale of 0-5 taking inspiration from prior works (Sakaguchi et al, 2014;Sakaguchi and Van Durme, 2018) but found they had a poor agreement (see details in A.1). Thus, following Abdalla et al (2021), we use Best-Worst Scaling (BWS), a comparative annotation schema, which builds on pairwise comparisons and does not require N 2 labels. Annotators are presented with n=4 sentences from a contract and a party, and are instructed to choose the best (i.e., most important) and worst (i.e., least important) sentence.…”