A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset

Boratko, Michael; Padigela, Harshit; Mikkilineni, Divyendra; Yuvraj, Pritish; Das, Rajarshi; McCallum, Andrew; Chang, Maria Hsia; Fokoue-Nkoutche, Achille; Kapanipathi, Pavan; Mattei, Nicholas; Musa, Ryan; Talamadupula, Kartik; Witbrock, Michael

doi:10.18653/v1/w18-2607

Cited by 31 publications

(29 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many systems answer science questions by composing multiple facts from semi-structured and unstructured knowl-edge sources (Khashabi et al 2016;Khot, Sabharwal, and Clark 2017;Jansen et al 2017;Khashabi et al 2018b). However, these often require careful manual tuning due to the large variety of reasoning techniques needed for these questions (Boratko et al 2018) and the large number of facts that often must be composed together (Jansen 2018;Jansen et al 2016). By limiting QASC to require exactly 2 hops (thereby avoiding semantic drift issues with longer paths (Fried et al 2015;Khashabi et al 2019)) and explicitly annotating these hops, we hope to constrain the problem enough so as to enable the development of supervised models for identifying and composing relevant knowledge.…”

Section: Comparison With Existing Datasetsmentioning

confidence: 99%

QASC: A Dataset for Question Answering via Sentence Composition

Khot

Clark

Guerquin

et al. 2020

AAAI

114

135

View full text Add to dashboard Cite

Composing knowledge from multiple pieces of texts is a key challenge in multi-hop question answering. We present a multi-hop reasoning dataset, Question Answering via Sentence Composition (QASC), that requires retrieving facts from a large corpus and composing them to answer a multiple-choice question. QASC is the first dataset to offer two desirable properties: (a) the facts to be composed are annotated in a large corpus, and (b) the decomposition into these facts is not evident from the question itself. The latter makes retrieval challenging as the system must introduce new concepts or relations in order to discover potential decompositions. Further, the reasoning model must then learn to identify valid compositions of these retrieved facts using common-sense reasoning. To help address these challenges, we provide annotation for supporting facts as well as their composition. Guided by these annotations, we present a two-step approach to mitigate the retrieval challenges. We use other multiple-choice datasets as additional training data to strengthen the reasoning model. Our proposed approach improves over current state-of-the-art language models by 11% (absolute). The reasoning and retrieval problems, however, remain unsolved as this model still lags by 20% behind human performance.

show abstract

Section: Comparison With Existing Datasetsmentioning

confidence: 99%

QASC: A Dataset for Question Answering via Sentence Composition

Khot

Clark

Guerquin

et al. 2020

AAAI

114

135

View full text Add to dashboard Cite

show abstract

“…LoBue and Yates (2011) and Sammons et al (2010) analyzed entailment phenomena using detailed classifications in RTE. For the ARC dataset, Boratko et al (2018) proposed knowledge and reasoning types.…”

Section: Related Workmentioning

confidence: 99%

What Makes Reading Comprehension Questions Easier?

Sugawara¹,

Inui²,

Sekine³

et al. 2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

107

View full text Add to dashboard Cite

A challenge in creating a dataset for machine reading comprehension (MRC) is to collect questions that require a sophisticated understanding of language to answer beyond using superficial cues. In this work, we investigate what makes questions easier across recent 12 MRC datasets with three question styles (answer extraction, description, and multiple choice). We propose to employ simple heuristics to split each dataset into easy and hard subsets and examine the performance of two baseline models for each of the subsets. We then manually annotate questions sampled from each subset with both validity and requisite reasoning skills to investigate which skills explain the difference between easy and hard questions. From this study, we observed that (i) the baseline performances for the hard subsets remarkably degrade compared to those of entire datasets, (ii) hard questions require knowledge inference and multiple-sentence reasoning in comparison with easy questions, and (iii) multiplechoice questions tend to require a broader range of reasoning skills than answer extraction and description questions. These results suggest that one might overestimate recent advances in MRC.

show abstract

“…Similarly, we leverage this dataset to build an essential term selector using a neural network-based algorithm. More recently, Boratko et al (2018) developed a labeling interface to obtain high quality labels for the ARC dataset. One finding is that human annotators tend to retrieve better evidence after they reformulate the search queries which are originally constructed by a simple concatenation of question and answer choice.…”

Section: Related Workmentioning

confidence: 99%

Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering

Zhu

Chen

et al. 2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

Open-domain question answering remains a challenging task as it requires models that are capable of understanding questions and answers, collecting useful information, and reasoning over evidence. Previous work typically formulates this task as a reading comprehension or entailment problem given evidence retrieved from search engines. However, existing techniques struggle to retrieve indirectly related evidence when no directly related evidence is provided, especially for complex questions where it is hard to parse precisely what the question asks. In this paper we propose a retriever-reader model that learns to attend on essential terms during the question answering process. We build (1) an essential term selector which first identifies the most important words in a question, then reformulates the query and searches for related evidence; and (2) an enhanced reader that distinguishes between essential terms and distracting words to predict the answer. We evaluate our model on multiple open-domain multiplechoice QA datasets, notably performing at the level of the state-of-the-art on the AI2 Reasoning Challenge (ARC) dataset.

show abstract

A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset

Cited by 31 publications

References 34 publications

QASC: A Dataset for Question Answering via Sentence Composition

QASC: A Dataset for Question Answering via Sentence Composition

What Makes Reading Comprehension Questions Easier?

Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering

Contact Info

Product

Resources

About