SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

Marelli, Marco; Bentivogli, Luisa; Baroni, Marco; Bernardi, Raffaella; Menini, Stefano; Zamparelli, Roberto

doi:10.3115/v1/s14-2001

Cited by 374 publications

(365 citation statements)

References 19 publications

Supporting

Mentioning

299

Contrasting

Unclassified

Order By: Relevance

“…In data collection for NLI, different annotator decisions about the coreference between entities and events across the two sentences in a pair can lead to very different assignments of pairs to labels (de Marneffe et al, 2008;Marelli et al, 2014a;Bowman et al, 2015). Drawing an example from Bowman et al, the pair "a boat sank in the Pacific Ocean" and "a boat sank in the Atlantic Ocean" can be labeled either CONTRADICTION or NEU-TRAL depending on (among other things) whether the two mentions of boats are assumed to refer to the same entity in the world.…”

Section: Data Collectionmentioning

confidence: 99%

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Williams

Nangia

Bowman

2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

2,446

1,993

View full text Add to dashboard Cite

This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. recognizing textual entailment), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by offering data from ten distinct genres of written and spoken English, making it possible to evaluate systems on nearly the full complexity of the language, while supplying an explicit setting for evaluating cross-genre domain adaptation. In addition, an evaluation using existing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement.

show abstract

Section: Data Collectionmentioning

confidence: 99%

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Williams

Nangia

Bowman

2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

2,446

1,993

View full text Add to dashboard Cite

show abstract

“…In this study, we verify the effectiveness of CWE on the benchmark dataset of a textual entailment recognition task which consists of data with contradiction, entailment or neutral relation. The dataset is from task 1 in SemEval 2014 [13]. The distribution of data is shown in Table 3.…”

Section: Data Set For Contradiction Detectionmentioning

confidence: 99%

“…We run experiments on benchmark datasets from SemEval 2014 [13]. The experiment results show that the proposed method with CWE performs comparably with top-performing systems in terms of overall classification accuracy.…”

Section: Introductionmentioning

confidence: 99%

“…Condoravdi et al first argued the importance of handling contradiction in text understanding [2]. As a kind of relation in the entailment recognition problem, contradiction has been studied continually in Recognizing Textual Entailment Challenges [13,[24][25][26][27][28][29]. In previous years, researchers undertook the task by resolving some of the contradiction phenomena [1,30], like negation, antonyms, data/number mismatch and different structure.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Contradiction Detection with Contradiction-Specific Word Embedding

Qin

Liu

2017

Algorithms

View full text Add to dashboard Cite

Contradiction detection is a task to recognize contradiction relations between a pair of sentences. Despite the effectiveness of traditional context-based word embedding learning algorithms in many natural language processing tasks, such algorithms are not powerful enough for contradiction detection. Contrasting words such as "overfull" and "empty" are mostly mapped into close vectors in such embedding space. To solve this problem, we develop a tailored neural network to learn contradiction-specific word embedding (CWE). The method can separate antonyms in the opposite ends of a spectrum. CWE is learned from a training corpus which is automatically generated from the paraphrase database, and is naturally applied as features to carry out contradiction detection in SemEval 2014 benchmark dataset. Experimental results show that CWE outperforms traditional context-based word embedding in contradiction detection. The proposed model for contradiction detection performs comparably with the top-performing system in accuracy of three-category classification and enhances the accuracy from 75.97% to 82.08% in the contradiction category.

show abstract

“…CDSMs assume that the meaning of a word can be interpreted by its context and the meaning of a sentence can be derived from its compositions. [17,24] Central to CDSM is the notion of compositionality, i.e., the meaning of complex expressions is determined by the meanings of their constituent expressions and the rules used to combine them. However, access to the annotated text and rules or corpora of symbolic-logic representation is challenging to evaluate in operational settings such as items generated from an item model.…”

Section: Measure Of Semantic Relatednessmentioning

confidence: 99%

Information-based methods for evaluating the semantics of automatically generated test items

Latifi

Gierl

Ren

et al. 2016

AIR

View full text Add to dashboard Cite

Multiple-choice questions are the popular type of test items that are used for testing the knowledge of health-science students in north America and elsewhere. The motivation of this article is to present the recent advances in the automatic item generation (AIG) and to propose a novel unsupervised approach that extends the information-based Compositional Distributional Semantic Model (CDSM) to measure the semantic relatedness among the pool of automatically generated items. We have used operational item bank from the medical science domain for developing the CDSM and demonstrated our approach using the concepts from AIG research. We illustrated our approach using eleven item models from the medical education domain, and discussed the possible applications to advance the AIG research.

show abstract

SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

Cited by 374 publications

References 19 publications

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Contradiction Detection with Contradiction-Specific Word Embedding

Information-based methods for evaluating the semantics of automatically generated test items

Contact Info

Product

Resources

About