Annotating Relation Inference in Context via Question Answering

Levy, Omer; Dagan, Ido

doi:10.18653/v1/p16-2041

Cited by 30 publications

(58 citation statements)

References 18 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chen et al (2016) revealed problems with the CNN/DailyMail dataset (Hermann et al, 2015) which resulted from apply-ing automatic tools for annotation. Levy and Dagan (2016) showed that a relation inference benchmark (Zeichner et al, 2012) is severely biased towards distributional methods, since it was created using DIRT (Lin and Pantel, 2001). Schwartz et al (2017) and Cai et al (2017) showed that certain biases are prevalent in the ROC stories cloze task (Mostafazadeh et al, 2016), which allow models trained on the endings alone, and not the story prefix, to yield state-of-the-art results.…”

Section: Discussionmentioning

confidence: 99%

Annotation Artifacts in Natural Language Inference Data

Gururangan¹,

Swayamdipta²,

Levy³

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

Self Cite

770

858

View full text Add to dashboard Cite

Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et al., 2015) and 53% of MultiNLI (Williams et al., 2018). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.

show abstract

Section: Discussionmentioning

confidence: 99%

Annotation Artifacts in Natural Language Inference Data

Gururangan¹,

Swayamdipta²,

Levy³

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

Self Cite

770

858

View full text Add to dashboard Cite

show abstract

“…Our work is more closely related to the dataset by Levy and Dagan (2016), who frame relation entailment as the task of judging the appropriateness of candidate answers. Their hypothesis is that an answer is only appropriate if it entails the predicate of the question.…”

Section: Meta Rules and Implicative Verbsmentioning

confidence: 99%

“…Lexical ontologies, such as WordNet (as used by Levy and Dagan, 2016) likewise lack this connection between relations and types. Moreover, relations between real-world entities are more often events than relations between common nouns.…”

Section: Meta Rules and Implicative Verbsmentioning

confidence: 99%

SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language Inference

Schmitt

Schütze

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

We present SherLIiC, 1 a testbed for lexical inference in context (LIiC), consisting of 3985 manually annotated inference rule candidates (InfCands), accompanied by (i)~960k unlabeled InfCands, and (ii)~190k typed textual relations between Freebase entities extracted from the large entity-linked corpus ClueWeb09. Each InfCand consists of one of these relations, expressed as a lemmatized dependency path, and two argument placeholders, each linked to one or more Freebase types. Due to our candidate selection process based on strong distributional evidence, SherLIiC is much harder than existing testbeds because distributional evidence is of little utility in the classification of InfCands. We also show that, due to its construction, many of SherLIiC's correct InfCands are novel and missing from existing rule bases. We evaluate a number of strong baselines on SherLIiC, ranging from semantic vector space models to state of the art neural models of natural language inference (NLI). We show that SherLIiC poses a tough challenge to existing NLI systems.

show abstract

“…Entailment Detection Evaluation. For the entailment detection task, we evaluate on Levy/Holt's dataset (Levy and Dagan, 2016;. Each example in the dataset contains a pair of triples where the entities are the same (possibly in the reverse order), but the relations are different.…”

Section: Evaluation Datasetsmentioning

confidence: 99%

“…In order to have a fair comparison with Berant's ILP method, we first test a set of rule-based constraints proposed by them (Berant et al, 2011). We also apply the lemma baseline heuristic process of Levy and Dagan (2016) before testing the methods. Figure 3 shows the precision-recall curve of all the methods in both local (A) and global (B) settings.…”

Section: Entailment Scores Based On Link Predictionmentioning

confidence: 99%

Duality of Link Prediction and Entailment Graph Induction

Hosseini¹,

Cohen²,

Johnson³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Link prediction and entailment graph induction are often treated as different problems. In this paper, we show that these two problems are actually complementary. We train a link prediction model on a knowledge graph of assertions extracted from raw text. We propose an entailment score that exploits the new facts discovered by the link prediction model, and then form entailment graphs between relations. We further use the learned entailments to predict improved link prediction scores. Our results show that the two tasks can benefit from each other. The new entailment score outperforms prior state-of-the-art results on a standard entialment dataset and the new link prediction scores show improvements over the raw link prediction scores.

show abstract

Annotating Relation Inference in Context via Question Answering

Cited by 30 publications

References 18 publications

Annotation Artifacts in Natural Language Inference Data

Annotation Artifacts in Natural Language Inference Data

SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language Inference

Duality of Link Prediction and Entailment Graph Induction

Contact Info

Product

Resources

About