Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension

Xu, Ying; Wang, Dakuo; Yu, Mei; Ritchie, Daniel; Wu, Tongshuang; Zhang, Zheng; Li, Toby Jia-Jun; Bradford, Nora; Sun, Branda; Hoang, Tran Bao; Sang, Yisi; Hou, Yufang; Ma, Xiao‐Jun; Yang, Diyi; Peng, Nanyun; Zhou, Yu; Warschauer, Mark

doi:10.18653/v1/2022.acl-long.34

Cited by 17 publications

(35 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Question Answering (QA) aims to provide answers a to the corresponding questions q , Based on the availability of context c , QA can be categorized into Open-domain QA (without context) [ 21 , 22 ] and Machine Reading comprehension (with context) [ 23 , 24 ]. Besides, QA can also be categorized into generative QA [ 25 , 26 ] and extractive QA [ 27 , 28 , 29 , 30 ]. Generally, the optimization objective of QA models is to maximize the log likelihood of the ground-truth answer a for the given context c .…”

Section: Background: Question Answering Question Generation and Evalu...mentioning

confidence: 99%

QAScore—An Unsupervised Unreferenced Metric for the Question Generation Evaluation

Lyu

Jones

et al. 2022

Entropy

View full text Add to dashboard Cite

Question Generation (QG) aims to automate the task of composing questions for a passage with a set of chosen answers found within the passage. In recent years, the introduction of neural generation models has resulted in substantial improvements of automatically generated questions in terms of quality, especially compared to traditional approaches that employ manually crafted heuristics. However, current QG evaluation metrics solely rely on the comparison between the generated questions and references, ignoring the passages or answers. Meanwhile, these metrics are generally criticized because of their low agreement with human judgement. We therefore propose a new reference-free evaluation metric called QAScore, which is capable of providing a better mechanism for evaluating QG systems. QAScore evaluates a question by computing the cross entropy according to the probability that the language model can correctly generate the masked words in the answer to that question. Compared to existing metrics such as BLEU and BERTScore, QAScore can obtain a stronger correlation with human judgement according to our human evaluation experiment, meaning that applying QAScore in the QG task benefits to a higher level of evaluation accuracy.

show abstract

Section: Background: Question Answering Question Generation and Evalu...mentioning

confidence: 99%

QAScore—An Unsupervised Unreferenced Metric for the Question Generation Evaluation

Lyu

Jones

et al. 2022

Entropy

View full text Add to dashboard Cite

show abstract

“…Salient events are defined as events that are relevant, or essential, to the main narrative of the story as defined by (Zhang et al, 2021). To find the salient events, we trained an inverse document frequency(idf) dictionary on a fairy-tale corpus (Xu et al, 2022) to calculate a tf-idf based salience score. Inverse document frequency can filter out generic events, and document frequency can be used to identify locally important events in the analyzed text.…”

Section: System Pipelinementioning

confidence: 99%

“…Dataset Experiments are conducted on the Fairy-taleQA dataset, a corpus of 278 open-source fairytales (Xu et al, 2022). First, we evaluate the event ordering model compared to heuristics-based method of ordering events using f-score based metrics.…”

Section: Experiments Setupmentioning

confidence: 99%

“…To measure the quality of the event chain, and for practical reasons, we adopt a precision based method to ask humans to judge whether the output event is correct in each step on the respective four dimensions. We perform in-group annotation on three randomly selected stories from the FairyTale corpus (Xu et al, 2022) consisting of around 200 events and 600 data points. We annotate based on the following procedure: for salient events, we annotate one or more salient events in each sentence to examine if the toolkit's prediction correctly matches the human-identified salient events; for character extraction, we annotate the name of the character that is the subject argument of the event to compare with model's character coding; for gender prediction, we annotate the gender of the event's subject argument; lastly, to annotate temporal ordering, at each step of the event chain, we ask the annotator whether the current event's posi-tion is reasonable given its previous and subsequent events in the chain.…”

Section: Evaluation Of the Four Dimensions Of An Event Chainmentioning

confidence: 99%

“…NECE toolkit is also helpful in enhancing a Question-Answering model's performance on certain narrative events related questions. Xu et al (2022) leverages a SOTA sequence-to-sequence model for the baseline QA-task experiment on the FairytaleQA dataset which contains 10,580 QApairs annotated by education experts. The education experts split each story into multiple sections which contain multiple paragraphs.…”

Section: Question-answeringmentioning

confidence: 99%

See 2 more Smart Citations

NECE: Narrative Event Chain Extraction Toolkit

Xu¹,

Isaza²,

Li³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

NECE is an event-based text analysis toolkit built for narrative documents. NECE aims to provide users open and easy accesses to an event-based summary and abstraction of long narrative documents through both a graphic interface and a python package, which can be readily used in narrative analysis, understanding, or other advanced purposes. Our work addresses the challenge of long passage events extraction and temporal ordering of key events; at the same time, it offers options to select and view events related to narrative entities, such as main characters and gender groups. We conduct human evaluation to demonstrate the quality of the event chain extraction system and character features mining algorithms. Lastly, we shed light on the toolkit's potential downstream applications by demonstrating its usage in gender bias analysis and Question-Answering tasks.

show abstract

Neural Question Generation for the Portuguese Language: A Preliminary Study

Leite

Cardoso

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Question Generation aims to automatically generate questions based on a given input provided as context. A controllable question generation scheme focuses on generating questions with specific attributes, allowing better control. In this study, we propose a few-shot prompting strategy for controlling the generation of question-answer pairs from children's narrative texts. We aim to control two attributes: the question's explicitness and underlying narrative elements. With empirical evaluation, we show the effectiveness of controlling the generation process by employing few-shot prompting side by side with a reference model. Our experiments highlight instances where the few-shot strategy surpasses the reference model, particularly in scenarios such as semantic closeness evaluation and the diversity and coherency of question-answer pairs. However, these improvements are not always statistically significant. The code is publicly available at github.com/bernardoleite/few-shot-prompting-qg-control.

show abstract

Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension

Cited by 17 publications

References 28 publications

QAScore—An Unsupervised Unreferenced Metric for the Question Generation Evaluation

QAScore—An Unsupervised Unreferenced Metric for the Question Generation Evaluation

NECE: Narrative Event Chain Extraction Toolkit

Neural Question Generation for the Portuguese Language: A Preliminary Study

Contact Info

Product

Resources

About