Neural Conversational QA: Learning to Reason vs Exploiting Patterns

Abstract

Neural Conversational QA tasks like ShARC require systems to answer questions based on the contents of a given passage. On studying recent state-of-the-art models on the ShARC QA task, we found indications that the models learn spurious clues/patterns in the dataset. Furthermore, we show that a heuristic-based program designed to exploit these patterns can have performance comparable to that of the neural models. In this paper we share our findings about four types of patterns found in the ShARC corpus and describe how neural models exploit them. Motivated by the aforementioned findings, we create and share a modified dataset that has fewer spurious patterns, consequently allowing models to learn better.

Dialogue Graph Modeling for Conversational Machine Reading

“…The major differences lie in two sides: 1) machines are required to formulate follow-up questions for clarification before confident enough to make the decision, 2) machines have to make a question-related conclusion by interpreting a set of complex decision rules, instead of simply extracting the answer from the text. Existing works (Zhong and Zettlemoyer, 2019;Lawrence et al, 2019;Verma et al, 2020;Gao et al, 2020a,b) have made progress in improving the reasoning ability by modeling the interactions between the rule document and other elements. As a widely-used manner, the existing models commonly extracted the rule documents into individual rule items, and track the rule fulfillment for the dialogue states.…”

“…The corresponding rule document and the question are marked in the same color in the figure . The major challenges for the conversational machine reading include the rule document interpretation, and reasoning with the background knowledge, e.g., the provided rule document, user scenario and the input question. Existing works (Zhong and Zettlemoyer, 2019;Lawrence et al, 2019;Verma et al, 2020;Gao et al, 2020a,b) have made progress in improving the reasoning ability by modeling the interactions among rule document, user scenario and the other elements implicitly. As for rule document interpretation, most existing approaches simply split the rule document into several rule conditions to be satisfied.…”

Smoothing Dialogue States for Open Conversational Machine Reading

“…The left part introduces the retrieval and tagging process for rule documents, which is then fed into the encoder together with other necessary information. Lawrence et al, 2019;Verma et al, 2020;Gao et al, 2020a,b;Ouyang et al, 2021) have made progress in modeling the matching relationships between the rule document and other elements such as user scenarios and questions. These studies are based on the hypothesis that the supporting information for answering the question is provided, which does not meet the real-world applications.…”

“…Existing studies deal with decision making and question generation independently (Zhong and Zettlemoyer, 2019;Lawrence et al, 2019;Verma et al, 2020;Gao et al, 2020a,b), and use hard-label decisions to activate question generation. These methods inevitably suffer from error propagation if the model makes the wrong decisions.…”

“…A variety of methods have been proposed for the CMR task, including 1) sequential models that encode all the elements and model the matching relationships with attention mechanisms (Zhong and Zettlemoyer, 2019;Lawrence et al, 2019;Verma et al, 2020;Gao et al, 2020a,b); 2) graph-based methods that capture the discourse structures of the rule texts and user scenario for better interactions (Ouyang et al, 2021). However, there are two sides of challenges that have been neglected: 1) Open-retrieval of supporting evidence.…”