GANDALF: a General Character Name Description Dataset for Long Fiction

Abstract

This paper introduces a long-range multiplechoice Question Answering (QA) dataset, based on full-length fiction book texts. The questions are formulated as 10-way multiplechoice questions, where the task is to select the correct character name given a character description, or vice-versa. Each character description is formulated in natural text and often contains information from several sections throughout the book. We provide 20,000 questions created from 10,000 manually annotated descriptions of characters from 177 books containing 152,917 words on average. We address the current discourse regarding dataset bias and leakage by a simple anonymization procedure, which in turn enables interesting probing possibilities. Finally, we show that suitable baseline algorithms perform very poorly on this task, with the book size itself making it non-trivial to attempt a Transformer-based QA solution. This leaves ample room for future improvement, and hints at the need for a completely different type of solution.

Evaluate AMR Graph Similarity via Self-supervised Learning

“…The use of self-supervised methods is a wellestablished approach in semantic text similarity tasks (STS, Carlsson et al, 2021;Gao et al, 2021). To assess AMR graph similarity efficiently and well-correlated with human judgments, we propose migrating the self-supervised training process in STS to AMR similarity evaluation.…”

“…Thus, self-supervised learning methods are an alternative solution. We adopt an efficient self-supervised approach Contrastive Tension (CT; Carlsson et al, 2021) in AMRSim metrics. The basic assumption is that AMR graphs with adjacent distributions have similar meanings.…”