Winventor: A Machine-driven Approach for the Development of Winograd Schemas

Abstract

The Winograd Schema Challenge-the task of resolving pronouns in certain carefully-constructed sentences-has recently been proposed as a basis for a novel form of CAPTCHAs. Such uses of the task necessitate the availability of a large, and presumably continuously-replenished, collection of available Winograd Schemas, which goes beyond what human experts can reasonably develop by themselves. Towards tackling this issue, we introduce Winventor, the first, to our knowledge, system that attempts to fully automate the development of Winograd Schemas, or at least to considerably help humans in this development task. Beyond describing the system, the paper presents a series of three studies that demonstrate, respectively, Winventor's ability to replicate existing Winograd Schemas from the literature, automatically develop reasonable Winograd Schemas from scratch, and aid humans in developing Winograd Schemas by post-processing the system's suggestions.

WinoReg: A New Faster and More Accurate Metric of Hardness for Winograd Schemas

“…It seems that the Connective-Triple, the Sentence-Type, the Sentence-Length, the TextBlob-Polarity, and the Word-Relations are the most useful features. This is in line with previous works that showed that the sentence length and the sentence type of each schema play an important role on the quality of the schema [10,11]. Additionally, according to Rahman et al, the Word-Relations feature is one of the most important ones in their work [17].…”

“…Recent work has shown that the sentence length of each schema plays an important role in the the resolution of the target pronoun [11]. Given that the schemas that are built on sentences that have a big number of words are harder to resolve, we create a feature that holds the number of words of each Schema Half sentence (SL).…”

Blending NLP and Machine Learning for the Development of Winograd Schemas

“…The current paper extends an earlier version [12] presented at the 12th International Conference on Agents and Artificial Intelligence (ICAART). Compared to the conference paper, which was based on NLP-only techniques, we enhanced the schema development process through Deep Learning.…”

“…Fig. 1: Winventor's high-level architecture: A system that automates the schema development process (adapted from [12]).…”

The Defeat of the Winograd Schema Challenge

“…The Winventor program (Isaak and Michael, 2020) uses automated methods to generate candidates for pronoun disambiguation problems, which could then be polished or adapted by human post-editors. Using hand-coded techniques, the program collected Wikipedia sentences with a pronoun with two potential referents and formulated a question based on the resolution of that pronoun.…”