The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity

Abstract

Shared Task 1 of SemEval-2014 comprised two subtasks on the same dataset of sentence pairs: recognizing textual entailment and determining textual similarity. We used an existing system based on formal semantics and logical inference to participate in the first subtask, reaching an accuracy of 82%, ranking in the top 5 of more than twenty participating systems. For determining semantic similarity we took a supervised approach using a variety of features, the majority of which was produced by our system for recognizing textual entailment. In this subtask our system achieved a mean squared error of 0.322, the best of all participating systems.

Adding Semantics to Data-Driven Paraphrasing

“…RTE systems vary considerably in their choice of representation and inference procedure. In the most recent shared task on RTE, some systems used deep logical representations of text, allowing them to invoke theorem provers (Bjerva et al, 2014) or Markov Logic Networks (Beltagy et al, 2014) to perform the inference, while others used shallower representations, relying on machine learning to perform inference (Lai and Hockenmaier, 2014;Zhao et al, 2014). Systems based on natural logic (MacCartney and Manning, 2007) use natural language as a representation, but still perform inference using a structured algebra rather than a statistical model.…”

“…We run our experiments using Nutcracker, a state-of-the-art RTE system based on formal semantics (Bjerva et al, 2014). Baselines are in gray, this work in blue, human references in gold.…”

SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

“…In the table, systems in bold are those for which the authors submitted a paper (Ferrone and Zanzotto, 2014;Bjerva et al, 2014;Beltagy et al, 2014;Lai and Hockenmaier, 2014;Alves et al, 2014;León et al, 2014;Bestgen, 2014;Zhao et al, 2014;Vo et al, 2014;Biçici and Way, 2014;Lien and Kouylekov, 2014;Jimenez et al, 2014;Proisl and Evert, 2014;Gupta et al, 2014). For the others, we used the brief description sent with the system's results, double-checking the information with the authors.…”

On-demand Injection of Lexical Knowledge for Recognising Textual Entailment

“…These semantic interpretations were composed using Boxer (Bos et al, 2004) from derivations of a Combinatory Categorial Grammar (CCG) (Steedman, 2000) automatically obtained by C&C, a wide-coverage CCG parser (Clark and Curran, 2007). This system was later extended into Nutcracker (Bjerva et al, 2014), where WordNet (Miller, 1995) and relations from Paraphrase Database (PPDB) (Ganitkevitch et al, 2013) are used to introduce external linguistic resources to account for lexical divergences (Pavlick et al, 2015). Pavlick et al (2015) study the characteristics of linguistic relations that may signal entailment or contradiction at subsentential level.…”