Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

Abstract

Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce COSMOS QA, a large-scale dataset of 35, 600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people's everyday narratives, asking such questions as "what might be the possible reason of ...?", or "what would have happened if ..." that require reasoning beyond the exact text spans in the context. To establish baseline performances on COSMOS QA, we experiment with several state-of-the-art neural architectures for reading comprehension, and also propose a new architecture that improves over the competitive baselines. Experimental results demonstrate a significant gap between machine (68.4%) and human performance (94%), pointing to avenues for future research on commonsense machine comprehension. Dataset, code and leaderboard is publicly available at https://wilburone.

Uncertainty over Uncertainty: Investigating the Assumptions, Annotations, and Text Measurements of Economic Policy Uncertainty

“…Although there has been a rise of common sense reasoning research in natural language processing (e.g. Bhagavatula et al (2020); Huang et al (2019); Sap et al (2019)), we suspect current state-of-the-art NLP systems would be unable to accurately resolve the inferences stated above. Furthermore, if there is inherent ambiguity in the language that expresses EPU, and, as we argue in Section 2.2, human perception is important, then we may desire to build models that can identify ambiguous documents and account for the uncertainty from ambiguity of language into measurement predictions, e.g.…”

LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

“…Similar to our dataset, recent datasets for commonsense reasoning, including MCScript [Ostermann et al, 2018] and COSMOS [Huang et al, 2019], also contain candidate answers not directly included in the input passage. They test a model's capability of making use of external background knowledge about spatial relations, cause and effect, scientific facts and social conventions.…”

“…In a typical task setting, a system is given a passage and a question, and asked to select a most appropriate answer from a list of candidate answers. With recent advances of deep learning in NLP, reading comprehension research has seen rapid advances, with a development from simple factual question answering [Rajpurkar et al, 2016] to questions that involve the integration of different pieces of evidences via multi-hop reasoning [Welbl et al, 2018; and questions that involve commonsense knowledge outside the given passage [Ostermann et al, 2018;Huang et al, 2019], where more varieties of challenges in human reading comprehension are investigated. One important aspect of human reading comprehension and question answering is logical reasoning, which was also one of the main research topics of early AI [McCarthy, 1989;Colmerauer and Roussel, 1996].…”

On Making Reading Comprehension More Comprehensive

“…There is room for interesting datasets along these lines. Cosmos QA (Huang et al, 2019) is an attempt to make such a dataset, though the fact that it is multiple choice puts it outside of our strict definition of "reading comprehension".…”