Using the Winograd Schema Challenge as a CAPTCHA

Isaak, Nicos; Michael, Loizos

doi:10.29007/rnk8

Cited by 7 publications

(22 citation statements)

References 9 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The availability of Winograd schemas seems disproportional to their demand and their potential impact. A recent study [5] showed that the WSC can form the basis of a new type of CAPTCHA, which might encourage more AI researchers to work on the problem of actually trying to tackle the WSC, and perhaps, to help towards the building of machines able to reason with commonsense knowledge. On the other hand, the development of carefully-crafted pronoun resolution tasks towards the development of Winograd schemas is a hard process [4]: it requires creativity and inspiration, and it is too troublesome to be done on a regular basis to support, for instance, competitions on the WSC or the testing of systems that might have been trained on existing collections of Winograd schemas.…”

Section: The Winograd Schema Challengementioning

confidence: 99%

See 1 more Smart Citation

WinoFlexi: A Crowdsourcing Platform for the Development of Winograd Schemas

Isaak

Michael

2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

The Winograd Schema Challenge, the task of resolving pronouns in certain carefully-structured sentences, has received considerable interest in the past few years as an alternative to the Turing Test. Systems developed to tackle this challenge have typically been evaluated on a small set of hand-crafted collections of sentences, since the development of new sentences by individuals is itself a rather challenging task, requiring care and creativity. In this paper we approach the problem of developing Winograd schemas via the introduction of WinoFlexi, a flexible online crowdsourcing system. Our empirical evaluation of the system's performance suggests that WinoFlexi allows crowdworkers to develop Winograd schemas of quality similar to that of most typical existing collections.

show abstract

Section: The Winograd Schema Challengementioning

confidence: 99%

“…To the best of our knowledge, this is the first work that attempts to use crowdsourcing for this task. We envision the use of this platform as a source of Winograd schemas for use in WSC-based CAPTCHAs [5] and in WSC competitions for the evaluation of systems that attempt to pass the challenge [4].…”

Section: Introductionmentioning

confidence: 99%

WinoFlexi: A Crowdsourcing Platform for the Development of Winograd Schemas

Isaak

Michael

2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…The importance of this suggestion becomes even more clear when one considers that the general problem of anaphora resolution remains a significant task for many natural language understanding applications (Deepa and Deisy, 2017). a https://orcid.org/0000- 0003-2353-2192 On the other hand, this disparity presents an opportunity for the use of WSC as the basis for developing CAPTCHAs (automated tests that aim to distinguish humans from computers), as proposed in recent work (Isaak and Michael, 2018).…”

Section: Introductionmentioning

confidence: 99%

Winventor: A Machine-driven Approach for the Development of Winograd Schemas

Isaak

Michael

2020

Proceedings of the 12th International Conference on Agents and Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

The Winograd Schema Challenge-the task of resolving pronouns in certain carefully-constructed sentences-has recently been proposed as a basis for a novel form of CAPTCHAs. Such uses of the task necessitate the availability of a large, and presumably continuously-replenished, collection of available Winograd Schemas, which goes beyond what human experts can reasonably develop by themselves. Towards tackling this issue, we introduce Winventor, the first, to our knowledge, system that attempts to fully automate the development of Winograd Schemas, or at least to considerably help humans in this development task. Beyond describing the system, the paper presents a series of three studies that demonstrate, respectively, Winventor's ability to replicate existing Winograd Schemas from the literature, automatically develop reasonable Winograd Schemas from scratch, and aid humans in developing Winograd Schemas by post-processing the system's suggestions.

show abstract

“…(as pursued, for example, in [7]), and could use our system to ensure that the generated tests are not overly demanding for human users.…”

mentioning

confidence: 99%

A Data-Driven Metric of Hardness for WSC Sentences

Isaak¹,

Michael²

EPiC Series in Computing

Self Cite

View full text Add to dashboard Cite

The Winograd Schema Challenge (WSC) — the task of resolving pronouns in certain sentences where shallow parsing techniques seem not to be directly applicable — has been proposed as an alternative to the Turing Test. According to Levesque, having access to a large corpus of text would likely not help much in the WSC. Among a number of attempts to tackle this challenge, one particular approach has demonstrated the plausibility of using commonsense knowledge automatically acquired from raw text in English Wikipedia.Here, we present the results of a large-scale experiment that shows how the performance of that particular automated approach varies with the availability of training material. We compare the results of this experiment with two studies: one from the literature that investigates how adult native speakers tackle the WSC, and one that we design and undertake to investigate how teenager non-native speakers tackle the WSC. We find that the performance of the automated approach correlates positively with the performance of humans, suggesting that the performance of the particular automated approach could be used as a metric of hardness for WSC instances.

show abstract

Using the Winograd Schema Challenge as a CAPTCHA

Cited by 7 publications

References 9 publications

WinoFlexi: A Crowdsourcing Platform for the Development of Winograd Schemas

WinoFlexi: A Crowdsourcing Platform for the Development of Winograd Schemas

Winventor: A Machine-driven Approach for the Development of Winograd Schemas

A Data-Driven Metric of Hardness for WSC Sentences

Contact Info

Product

Resources

About