AmbigQA: Answering Ambiguous Open-domain Questions

Min, Sewon; Michael, Julian; Hajishirzi, Hannaneh; Zettlemoyer, Luke

doi:10.18653/v1/2020.emnlp-main.466

Cited by 95 publications

(90 citation statements)

References 23 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We observed there are many ambiguous questions included in NQ data. Consistent with the findings of Min et al (2020), we have found that many of the ambiguous questions or illposed questions can be fixed by small edits, and we suggest asking annotators to edit those questions or asking them a follow-up clarification instead of simply marking and leaving the questions as is in the future information-seeking QA dataset creation.…”

Section: Discussionsupporting

confidence: 86%

See 1 more Smart Citation

Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval

Asai¹,

Choi²

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Recent pretrained language models "solved" many reading comprehension benchmarks, where questions are written with the access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA. Our controlled experiments suggest two headrooms -paragraph selection and answerability prediction, i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, existing models easily outperform a human annotator. However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct percategory answerability prediction, revealing issues in the current dataset collection as well as task formulation. Together, our study points to avenues for future research in informationseeking question answering, both for dataset creation and model development. 11 Our code and annotated data is publicly available at https://github.com/AkariAsai/ unanswerable_qa.

show abstract

Section: Discussionsupporting

confidence: 86%

“…While all the individual pieces might be revealed in independent studies (Min et al, 2020;Oguz et al, 2020), our study quantifies how much each factor accounts for reducing answer coverage.…”

Section: Discussionmentioning

confidence: 89%

Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval

Asai¹,

Choi²

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…Question Rewriting QR has been studied for augmenting training data (Buck et al, 2018;Sun et al, 2018;Zhu et al, 2019; or clarifying ambiguous questions (Min et al, 2020). In CQA, QR can be viewed as a task of simplifying difficult questions that include anaphora and ellipsis in a conversation.…”

Section: Related Workmentioning

confidence: 99%

Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering

Kim¹,

Kim²,

Park³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

One of the main challenges in conversational question answering (CQA) is to resolve the conversational dependency, such as anaphora and ellipsis. However, existing approaches do not explicitly train QA models on how to resolve the dependency, and thus these models are limited in understanding human dialogues. In this paper, we propose a novel framework, EXCORD (Explicit guidance on how to resolve Conversational Dependency) to enhance the abilities of QA models in comprehending conversational context. EXCORD first generates self-contained questions that can be understood without the conversation history, then trains a QA model with the pairs of original and self-contained questions using a consistency-based regularizer. In our experiments, we demonstrate that EXCORD significantly improves the QA models' performance by up to 1.2 F1 on QuAC , and 5.2 F1 on CANARD (Elgohary et al., 2019), while addressing the limitations of the existing approaches. 1 † Corresponding author 1 Our models and code are available at: https://github.com/dmis-lab/excord 2 While the term "context" usually refers to the evidence document from which the answer is extracted, in CQA, it refers to conversational context.

show abstract

“…To ensure that our datasets properly "isolate the property that motivated [the dataset] in the first place" (Zaenen, 2006), we need to explicitly appreciate the unavoidable ambiguity instead of silently glossing over it. 14 This is already an active area of research, with conversational QA being a new setting actively explored by several datasets (Reddy et al, 2018;Choi et al, 2018); and other work explicitly focusing on identifying useful clarification questions (Rao and Daumé III), thematically linked questions (Elgohary et al, 2018) or resolving ambiguities that arise from coreference or pragmatic constraints by rewriting underspecified question strings (Elgohary et al, 2019;Min et al, 2020).…”

Section: A Call To Actionmentioning

confidence: 99%

What Question Answering can Learn from Trivia Nerds

Boyd-Graber¹,

Börschinger²

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Question answering (QA)is not just building systems; this NLP subfield also creates and curates challenging question datasets that reveal the best systems.We argue that QA datasets-and QA leaderboards-closely resemble trivia tournaments: the questions agents-humans or machines-answer reveals a "winner". However, the research community has ignored the lessons from decades of the trivia community creating vibrant, fair, and effective QA competitions. After detailing problems with existing QA datasets, we outline several lessons that transfer to QA research: removing ambiguity, identifying better QA agents, and adjudicating disputes.

show abstract

AmbigQA: Answering Ambiguous Open-domain Questions

Cited by 95 publications

References 23 publications

Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval

Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval

Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering

What Question Answering can Learn from Trivia Nerds

Contact Info

Product

Resources

About