The Winograd Schema Challenge: Are You Sure That We Are on the Right Track?

Blending NLP and Machine Learning for the Development of Winograd Schemas

“…To put it succinctly, strictly following the purpose of the challenge, that is, focusing on the science and not the technology of AI, could lead us towards developing machines with commonsense and reasoning abilities similar to those found in humans. This is well-stated and analyzed in another paper of ours, where we shed light on how to build commonsense and reasoning systems that analyze and tackle schemas the way humans do [10].…”

“…The whole idea behind WinoFusion is to provide the crowd with the necessary tools to develop quality and diverse schemas based on various subjects. As previously stated, Winventor [10,16] can motivate and inspire experts to develop rich and diverse schema halves. In this regard, previous experiments showed that human experts were able to develop schemas for a variety set of sentence types.…”

“…However, along a completely different tangent, when approaching the challenge as another pronoun resolution problem, there are subsets of schemas that can be solved with high accuracy by utilizing clever tricks that, sometimes, even humans can hardly determine [18]. In this sense, when completely removing the science of AI factor, which propels us to figure out how humans analyze Winograd schemas to come to conclusions about events and participants, and by focusing on the technology of AI, we can build narrow savants that show but not exhibit real commonsense and reasoning abilities when tackling Winograd schemas [10].…”

“…Especially with the advent of large language models, many researchers have pursued a course of achieving high accuracy results in specialized subsets of schemas, often comparable to those of humans, but with ways that have nothing to tell us about human behavior itself [10]. It is well-known that these systems, although they can achieve high accuracy results in specified subsets of schemas by analyzing the pattern of words, have nothing to do with human results, as they often fail to answer with scores even below chance [18].…”