Interpretation of Natural Language Rules in Conversational Machine Reading

Abstract

Most work in machine reading focuses on question answering problems where the answer is directly expressed in the text to read. However, many real-world question answering problems require the reading of text not because it contains the literal answer, but because it contains a recipe to derive an answer together with the reader's background knowledge. One example is the task of interpreting regulations to answer "Can I...?" or "Do I have to...?" questions such as "I am working in Canada. Do I have to carry on paying UK National Insurance?" after reading a UK government website about this topic. This task requires both the interpretation of rules and the application of background knowledge. It is further complicated due to the fact that, in practice, most questions are underspecified, and a human assistant will regularly have to ask clarification questions such as "How long have you been working abroad?" when the answer cannot be directly derived from the question and text. In this paper, we formalise this task and develop a crowd-sourcing strategy to collect 32k task instances based on real-world rules and crowd-generated questions and scenarios. We analyse the challenges of this task and assess its difficulty by evaluating the performance of rule-based and machine-learning baselines. We observe promising results when no background knowledge is necessary, and substantial room for improvement whenever background knowledge is needed.

Attending to Future Tokens for Bidirectional Sequence Generation

“…We report the results of our approach, the various baselines, as well as the previous state-of-the-art (SOTA) scores where applicable in Table 1 and 2 for SHARC and in Table 3 for DAILY DIALOG. On the SHARC dataset, we observe very poor BLEU-4 performance for the encoder-decoder Transformer (E&D), which is consistent with results from Saeidi et al (2018), who could not get a LSTM-based network to work without an additional classification head. Adding BERT (E&D+B) slightly improves performance.…”

Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion

“…The CSQA dataset [30] takes preliminary steps towards the sequential KG-QA paradigm, but it is extremely artificial: initial and follow-up questions are generated semi-automatically via templates, and sequential utterances are only simulated by stitching questions with shared entities or relations in a thread, without a logical flow. QBLink [9], CoQA [27], ans ShARC [29] are recent resources for sequential QA over text. The SQA resource [16], derived from WikiTableQuestions [25], is aimed at driving conversational QA over (relatively small) Web tables.…”

“…However, such table-cell search methods cannot scale to real-world, large-scale curated KGs. QBLink [9], CoQA [27], and ShARC [29] are recent benchmarks aimed at driving conversational QA over text, and the allied paradigm in text comprehension on interactive QA [18]. Hixon et al [13] try to learn concept knowledge graphs from conversational dialogues over science questions, but such KGs are fundamentally different from curated ones like Wikidata with millions of facts.…”

Reasoning Over Paragraph Effects in Situations

“…The most closely related datasets to ROPES are ShARC (Saeidi et al, 2018), Open-BookQA (Mihaylov et al, 2018), and QuaRel (Tafjord et al, 2019). ShARC shares the same goal of understanding causes and effects (in terms of specified rules), but frames it as a dialogue where the system has to also generate questions to gain complete information.…”