Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension

Xu, Ying; Wang, Dakuo; Yu, Mei; Ritchie, Daniel; Wu, Tongshuang; Zhang, Zheng; Li, Toby Jia-Jun; Bradford, Nora; Sun, Branda; Hoang, Tran Bao; Sang, Yisi; Hou, Yufang; Ma, Xiao‐Jun; Yang, Diyi; Peng, Nanyun; Zhou, Yu; Warschauer, Mark

doi:10.48550/arxiv.2203.13947

Cited by 3 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite this major difference in learning paradigm, most GPT-3-based models proposed here outperform previous results by significant margins on the SQuAD dataset -even the least performant samples M s (lowerbound) achieve competitive results. For Fairytale QA, however, only the best samples M s (upperbound) outperform previous results (Xu et al, 2022), indicating margins for improvement on question selection strategies for future work.…”

Section: Resultsmentioning

confidence: 77%

“…Datasets and model: We adopt two question generation datasets with distinctive characteristics, namely SQuAD (Rajpurkar et al, 2016) and Fairytale QA (Xu et al, 2022). SQuAD was originally proposed as an extractive QA dataset.…”

Section: Problem Settingmentioning

confidence: 99%

“…Evaluation Metrics : We use two metrics to evaluate the selected question q ′ = M (q S(Q) ): • Reference-based evaluation: Following prior works, we use BLEU-4 for SQuAD (Du and Cardie, 2018;Bao et al, 2020) and ROUGE-L for Fairytale QA (Xu et al, 2022) 3 . These metrics compare q ′ against the reference question q (a.k.a.…”

Section: Problem Settingmentioning

confidence: 99%

See 2 more Smart Citations

Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation

Yuan

Wang

Wang³

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

Large Language Models (LLMs) have in recent years demonstrated impressive prowess in natural language generation. A common practice to improve generation diversity is to sample multiple outputs from the model. However, partly due to the inaccessibility of LLMs, there lacks a simple and robust way of selecting the best output from these stochastic samples. As a case study framed in the context of question generation, we propose two prompt-based approaches, namely round-trip and prompt-based score, to selecting high-quality questions from a set of LLM-generated candidates. Our method works without the need to modify the underlying model, nor does it rely on human-annotated references -both of which are realistic constraints for real-world deployment of LLMs. With automatic as well as human evaluations, we empirically demonstrate that our approach can effectively select questions of higher qualities than greedy generation.1 * Equal contribution. 1 We open-source all code and annotated data on github.

show abstract

Section: Resultsmentioning

confidence: 77%

Section: Problem Settingmentioning

confidence: 99%

See 1 more Smart Citation

Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation

Yuan

Wang

Wang³

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

show abstract

“…There are many forms of reading comprehension tasks such as cloze tests (Bajgar et al, 2016;Ma et al, 2018), question answering (Richardson et al, 2013;Kočiskỳ et al, 2018;Yang and Choi, 2019;Lal et al, 2021;Xu et al, 2022), and text summarization (Ladhak et al, 2020;Kryściński et al, 2021;Chen et al, 2021). Most of these tasks are built on very short stories or can be solved in segments of a story, thus presenting limited challenges to understanding the elements, especially the characters, of the story.…”

Section: Assessment Of Narrative Comprehensionmentioning

confidence: 99%

“…The most popular form of narrative comprehension evaluation is through question answering, starting from the early work of MCTest (Richardson et al, 2013), to the more recent crowd-sourced tasks like Narra-tiveQA (Kočiskỳ et al, 2018), FriendsQA Choi, 2019), TellMeWhy (Lal et al, 2021) and FairytaleQA (Xu et al, 2022). Among them, the MCTest and TellMeWhy conduct multi-choice question answering on short stories.…”

Section: Question Answeringmentioning

confidence: 99%

TVShowGuess: Character Comprehension in Stories as Speaker Guessing

Sang¹,

Mou²,

Yu³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

We propose a new task for assessing machines' ability to understand fictional characters in narrative stories. The task, TVSHOWGUESS, builds on the scripts of TV series and takes the form of guessing the anonymous main characters based on the backgrounds of the scenes and dialogues. Our human study supports that this form of task covers comprehension of multiple types of character persona, including understanding characters' personalities, facts and memories of personal experience, which are well aligned with the psychological and literary theories about the theory of mind (ToM) of human beings on understanding fictional characters during reading. We further propose new model architectures to support the contextualized encoding of long scene texts. Experiments show that our proposed approaches significantly outperform baselines, yet still largely lag behind the (nearly perfect) human performance. Our work serves as a first step toward the goal of narrative character comprehension. 1

show abstract

A Self-Questioning Framework Towards Knowledge Self-Organization in Children's Readings via Hybrid Prompt Learning

He,

Yao,

2024

Preprint

View full text Add to dashboard Cite

The cultivation of children's intellectual and literacy skills will benefit from heuristic questioning in educational readings like fairy tales. However, as not all stories naturally encompass expert-derived questions, machine-generated questions need to serve as indispensable supplements to enrich the learning experience. Unfortunately, current text generation models fail to generate high-cognitive educational questions closely related to diverse knowledge of stories. To this end, we propose a novel framework that employs automatic hybrid prompt learning to enable self-questioning to organize knowledge. Initially, we design an identifier to locate sentences containing knowledge within a given text, and then train a model to generate corresponding knowledge inferences. Each inference, which acts as the hard prompt, is concatenated with the soft prompt (\textit{i.e.}, learnable parameters) to construct the hybrid prompt. Equipped with these prompts, pre-trained language models can be facilitated to generate questions and then their answers. These question and answer pairs are distillations of the reading's knowledge. We evaluate the generation performance of our framework on an educational question-answering benchmark known as FairytaleQA. Experimental results demonstrate that our framework outperforms baselines according to automatic and manual evaluation metrics. Notably, our approach excels at generating diverse heuristic questions. Moreover, our work holds the potential to contribute significantly to the advancement of children's education.

show abstract

Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension

Cited by 3 publications

References 0 publications

Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation

Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation

TVShowGuess: Character Comprehension in Stories as Speaker Guessing

A Self-Questioning Framework Towards Knowledge Self-Organization in Children's Readings via Hybrid Prompt Learning

Contact Info

Product

Resources

About