DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension

Abstract

We propose DuoRC, a novel dataset for Reading Comprehension (RC) that motivates several new challenges for neural approaches in language understanding beyond those offered by existing RC datasets. DuoRC contains 186,089 unique questionanswer pairs created from a collection of 7680 pairs of movie plots where each pair in the collection reflects two versions of the same movie -one from Wikipedia and the other from IMDb -written by two different authors. We asked crowdsourced workers to create questions from one version of the plot and a different set of workers to extract or synthesize answers from the other version. This unique characteristic of DuoRC where questions and answers are created from different versions of a document narrating the same underlying story, ensures by design, that there is very little lexical overlap between the questions created from one version and the segments containing the answer in the other version. Further, since the two versions have different levels of plot detail, narration style, vocabulary, etc., answering questions from the second version requires deeper language understanding and incorporating external background knowledge. Additionally, the narrative style of passages arising from movie plots (as opposed to typical descriptive passages in existing datasets) exhibits the need to perform complex reasoning over events across multiple sentences. Indeed, we observe that state-of-the-art neural RC models which have achieved near human performance on the SQuAD dataset (Rajpurkar et al., 2016b), even when coupled with tra-ditional NLP techniques to address the challenges presented in DuoRC exhibit very poor performance (F1 score of 37.42% on DuoRC v/s 86% on SQuAD dataset). This opens up several interesting research avenues wherein DuoRC could complement other RC datasets to explore novel neural approaches for studying language understanding.

Adversarial Domain Adaptation for Machine Reading Comprehension

“…We further test our model on the newly-released DuoRC dataset (Saha et al, 2018). This dataset contains two subsets: movie descriptions collected from Wikipedia (SelfRC) and from IMDB (ParaphraseRC).…”

“…Note that the answers of the same question could be different in the two subsets (only 40.7% of the questions have the same answers in both domains). We preprocess the dataset and test the answer-span extraction task following Saha et al (2018). Results are reported in Table 3.…”

Neural Machine Reading Comprehension: Methods and Trends

“…As questions are not proposed directly from documents, this task is challenging and some information extraction methods fail to deal with it. This methodology of creating MRC datasets enlightens lots of other researches [77,52,69]. In order to avoid that questions can be answered by knowledge out of the documents, all entities in documents are anonymized by random markers.…”

On Making Reading Comprehension More Comprehensive

“…One way to reduce lexical overlap between the question and passage is to expose the author of the question to a different passage that conveys a similar meaning. Examples include NARRA-TIVEQA (Kočiskỳ et al, 2018), where question authors were shown a summary of a movie script that will be used for answering questions, and DUORC (Saha et al, 2018), where questions are authored given a passage that is comparable to the one that will later be employed.…”