QASC: A Dataset for Question Answering via Sentence Composition

Abstract

Composing knowledge from multiple pieces of texts is a key challenge in multi-hop question answering. We present a multi-hop reasoning dataset, Question Answering via Sentence Composition (QASC), that requires retrieving facts from a large corpus and composing them to answer a multiple-choice question. QASC is the first dataset to offer two desirable properties: (a) the facts to be composed are annotated in a large corpus, and (b) the decomposition into these facts is not evident from the question itself. The latter makes retrieval challenging as the system must introduce new concepts or relations in order to discover potential decompositions. Further, the reasoning model must then learn to identify valid compositions of these retrieved facts using common-sense reasoning. To help address these challenges, we provide annotation for supporting facts as well as their composition. Guided by these annotations, we present a two-step approach to mitigate the retrieval challenges. We use other multiple-choice datasets as additional training data to strengthen the reasoning model. Our proposed approach improves over current state-of-the-art language models by 11% (absolute). The reasoning and retrieval problems, however, remain unsolved as this model still lags by 20% behind human performance.

UNIFIEDQA: Crossing Format Boundaries with a Single QA System

“…Multiple-choice QA (MC). We use the following MC datasets: MCTest (Richardson et al, 2013), RACE (Lai et al, 2017), OpenBookQA/OBQA (Mihaylov et al, 2018), ARC (Clark et al, , 2016, QASC (Khot et al, 2019), CommonsenseQA/CQA , PIQA (Bisk et al, 2020), SIQA (Sap et al, 2019), and Winogrande (Sakaguchi et al, 2020). Several of the MC datasets do not come with accompanying paragraphs (such as ARC, QASC, OBQA).…”

“…The first few rows of the table show T5 models trained for individual formats, followed by UNI-FIEDQA. For completeness, we include the highest previous scores for each dataset; one must be careful when reading these numbers as the best previous numbers follow the fully supervised protocol (for NewsQA (Zhang et al, 2020), Quoref (Segal et al, 2019), DROP (Lan et al, 2019), ROPES (Lin et al, 2019), QASC (Khot et al, 2019), CommonsenseQA (Zhu et al, 2020) and x-CS datasets (Gardner et al, 2020). )…”

Sentence embeddings in NLI with iterative refinement encoders

“…2017), we extend their architecture with a hierarchy-like structure of bidirectional LSTM (BiLSTM) layers with max pooling. All in all, our model improves the previous state of the art for SciTail (Khot, Sabharwal, and Clark 2018) and achieves strong results for the Stanford Natural Language Inference (SNLI) and Multi-Genre Natural Language Inference corpus (MultiNLI; Williams, Nangia, and Bowman 2018).…”

“…SciTail: SciTail (Khot et al . 2018) is an NLI dataset created from multiple-choice science exams consisting of 27k sentence pairs. Each question and the correct answer choice have been converted into an assertive statement to form the hypothesis.…”

“…It has been argued that the lexical similarity of the sentences in SciTail sentence pairs make it a particularly difficult dataset (Khot et al . 2018). If this is the case, we hypothesize that our model is indeed better at identifying entailment relations beyond focusing on the lexical similarity of the sentences.…”