ASQA: Factoid Questions Meet Long-Form Answers

Stelmakh, Ivan; Luan, Yi; Dhingra, Bhuwan; Chang, Ming‐Wei

doi:10.48550/arxiv.2204.06092

Cited by 4 publications

(17 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One response strategy to such questions is a long-form answer, studied here. Both ASQA (Stelmakh et al, 2022) and AQuaMuSe (Kulkarni et al, 2020) require that systems consolidate information from multiple sources to generate multifaceted long-form answers to questions from the Natural Questions. ASQA focuses on the subset of questions labeled in AmbigQA (Min et al, 2020) for which it is possible to enumerate a collection of refinements and factoid answers that should be covered in a long-form answer.…”

Section: Long-form Question Answeringmentioning

confidence: 99%

“…There are multiple reasons why a long-form answer would be more felicitous than a factoid answer to a question. Table 2 shows six common types of multifaceted questions in the ASQA (Stelmakh et al, 2022) and AQuAMuSe datasets that fall into this category. These are highly related to the ambiguity categories in AmbigQA (Min et al, 2020), which we used as a seed set for exploring the data in this work.…”

Section: Types Of Multifaceted Questionsmentioning

confidence: 99%

“…We conducted experiments on two question answering tasks that require long-form answers: Ambiguous Question Answering (ASQA; Stelmakh et al, 2022) and Query-focused Multi-document Summarization (AQuAMuSe; Kulkarni et al, 2020). In this section, we present results on ASQA; the following section presents results on AquaMuSe.…”

Section: Experiments On Ambiguous Question Answeringmentioning

confidence: 99%

“…Our study is based on two long-form question answering benchmarks, ASQA (Stelmakh et al, 2022) and AQuAMuSe (Kulkarni et al, 2020), which focus on queries that may be ambiguous and/or require information from multiple sources. Using these two datasets, our work is the first to show that LLMs are capable of generating long-form answers to complex questions of various types in a closed-book setting.…”

mentioning

confidence: 99%

“…We evaluate our long-form answers using ROUGE (measuring stylistic similarity to gold answers; Lin, 2004) and reading comprehension models (measuring correctness; Stelmakh et al, 2022). Using our query refinement prompts in the few-shot prompting and prompt tuning settings, we are able to achieve significantly better performance to fully finetuned closed-book systems, as well as comparable performance to open-book retrieve-then-generate systems, both finetuned on the full training dataset.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Query Refinement Prompts for Closed-Book Long-Form Question Answering

Amplayo¹,

Webster²,

Collins³

et al. 2022

Preprint

View full text Add to dashboard Cite

Large language models (LLMs) have been shown to perform well in answering questions and in producing long-form texts, both in few-shot closed-book settings. While the former can be validated using well-known evaluation metrics, the latter is difficult to evaluate. We resolve the difficulties to evaluate long-form output by doing both tasks at once -to do question answering that requires long-form answers. Such questions tend to be multifaceted, i.e., they may have ambiguities and/or require information from multiple sources. To this end, we define query refinement prompts that encourage LLMs to explicitly express the multifacetedness in questions and generate long-form answers covering multiple facets of the question. Our experiments on two long-form question answering datasets, ASQA and AQuAMuSe, show that using our prompts allows us to outperform fully finetuned models in the closed book setting, as well as achieve results comparable to retrieve-then-generate open-book models.

show abstract

Section: Long-form Question Answeringmentioning

confidence: 99%

Section: Types Of Multifaceted Questionsmentioning

confidence: 99%

Section: Experiments On Ambiguous Question Answeringmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Query Refinement Prompts for Closed-Book Long-Form Question Answering

Amplayo¹,

Webster²,

Collins³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering

Han,

Park,

Lee

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Information-seeking questions in long-form question answering (LFQA) often prove misleading due to ambiguity or false presupposition in the question. While many existing approaches handle misleading questions, they are tailored to limited questions, which are insufficient in a real-world setting with unpredictable input characteristics. In this work, we propose PreWoMe, a unified approach capable of handling any type of information-seeking question. The key idea of PreWoMe involves extracting presuppositions in the question and exploiting them as working memory to generate feedback and action about the question. Our experiment shows that PreWoMe is effective not only in tackling misleading questions but also in handling normal ones, thereby demonstrating the effectiveness of leveraging presuppositions, feedback, and action for real-world QA settings.

show abstract

Asking Clarification Questions to Handle Ambiguity in Open-Domain QA

Lee,

Kim,

Lee

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Ambiguous questions persist in open-domain question answering, because formulating a precise question with a unique answer is often challenging. Previous works have tackled this issue by generating and answering disambiguated questions for all possible interpretations of the ambiguous question. Instead, we propose to ask a clarification question, where the user's response will help identify the interpretation that best aligns with the user's intention. We first present CAMBIGNQ, a dataset consisting of 5,653 ambiguous questions, each with relevant passages, possible answers, and a clarification question. The clarification questions were efficiently created by generating them using InstructGPT and manually revising them as necessary. We then define a pipeline of three tasks-(1) ambiguity detection, (2) clarification question generation, and (3) clarification-based QA. In the process, we adopt or design appropriate evaluation metrics to facilitate sound research. Lastly, we achieve F1 of 61.3, 25.1, and 40.5 on the three tasks, demonstrating the need for further improvements while providing competitive baselines for future work.

show abstract

ASQA: Factoid Questions Meet Long-Form Answers

Cited by 4 publications

References 24 publications

Query Refinement Prompts for Closed-Book Long-Form Question Answering

Query Refinement Prompts for Closed-Book Long-Form Question Answering

PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering

Asking Clarification Questions to Handle Ambiguity in Open-Domain QA

Contact Info

Product

Resources

About