Open-Retrieval Conversational Question Answering

Abstract

Recent studies on Question Answering (QA) and Conversational QA (ConvQA) emphasize the role of retrieval: a system first retrieves evidence from a large collection and then extracts answers. This open-retrieval ConvQA setting typically assumes that each question is answerable by a single span of text within a particular passage (a span answer). The supervision signal is thus derived from whether or not the system can recover an exact match of this ground-truth answer span from the retrieved passages. This method is referred to as spanmatch weak supervision. However, information-seeking conversations are challenging for this span-match method since long answers, especially freeform answers, are not necessarily strict spans of any passage. Therefore, we introduce a learned weak supervision approach that can identify a paraphrased span of the known answer in a passage. Our experiments on QuAC and CoQA datasets show that the span-match weak supervisor can only handle conversations with span answers, and has less satisfactory results for freeform answers generated by people. Our method is more flexible as it can handle both span answers and freeform answers. Moreover, our method can be more powerful when combined with the span-match method which shows it is complementary to the span-match method. We also conduct in-depth analyses to show more insights on open-retrieval ConvQA under a weak supervision setting.

Technical Question Answering across Tasks and Domains

“…It should be noted that different from previous work that only leverages the first term in reading score, i.e., Xiong et al, 2020;Qu et al, 2020), our added second term improved inference performance. This is because during the training time, the span label of a document that does not contain an answer is set to (0, 0), and such negative documents are the majority.…”

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers

“…It should be noted that different from previous work that only leverages the first term in reading score, i.e., Qu et al, 2020), our added second term improved inference performance. This is because during the training time, the span label of a document that does not contain an answer is set to (0, 0), and such negative documents are the majority.…”

“…While there are some prior data engineering solutions to "model patching", including augmentation Wei and Zou, 2019;Kaushik et al, 2019;Goel et al, 2021a), weak labeling Chen et al, 2020), and synthetic data generation (Murty et al, 2020), due to the noise in WIKIPEDIA, we repurpose BOOTLEGSPORT using weak labeling to modify training labels and correct for this noise. Our weak labeling technique works as follows: any existing mention from strong-sport-cues that is labeled as a country is relabeled as a national sports team for Subpop.…”

“…Variational auto encoders (Yang et al (2017); Gururangan et al (2019)) and virtual adversarial training (Miyato et al, 2016) have been extensively used in semi-supervised setting. Recently interpolations in textual hidden space (Chen et al, 2020) have been used for semi-supervised learning as well.…”

“…In addition to PL and KD, Virtual Adversarial Training (VAT) and Cross-View Training (CVT) have achieved state-of-the-art SSL performance on various tasks including text classification, named entity recognition, and dependency parsing (Miyato et al, 2019;Miyato et al, 2017;Chen et al, 2020). In this paper, we conduct comprehensive experiments and analysis related to these commonly used SSL techniques, and discuss their pros and cons in the industry setting.…”