Using Crowdsourcing to Investigate Perception of Narrative Similarity

Abstract

For many applications measuring the similarity between documents is essential. However, little is known about how users perceive similarity between documents. This paper presents the first large-scale empirical study that investigates perception of narrative similarity using crowdsourcing. As a dataset we use a large collection of Dutch folk narratives. We study the perception of narrative similarity by both experts and non-experts by analyzing their similarity ratings and motivations for these ratings. While experts focus mostly on the plot, characters and themes of narratives, non-experts also pay attention to dimensions such as genre and style. Our results show that a more nuanced view is needed of narrative similarity than captured by story types, a concept used by scholars to group similar folk narratives. We also evaluate to what extent unsupervised and supervised models correspond with how humans perceive narrative similarity.

Adopting Seekers’ Solution Exemplars in Crowdsourcing Ideation Contests: Antecedents and Consequences

“…To do this, we recruited third-party raters from MTurk to compare the ads and the corresponding exemplars. Past studies also relied on raters on microtask platforms to evaluate the similarity between objects (Dow et al 2011, Rahmanian andDavis 2014), and such raters could perform reliable similarity evaluations despite their (possible) lack of expertise with the objects being compared (Nguyen et al 2014). Following Dow et al (2011), we asked MTurk raters to compare the similarity between an ad and an assigned exemplar on a seven-point scale.…”

Exploring aspects of similarity between spoken personal narratives by disentangling them into narrative clause types

“…Additionally, their method does not generalize well to other story types (or even movie plots) since they include specific movie parameters, like characters' name and gender, as the basis of their solution, which does not apply to our case since we do not attempt to match stories based on these surface-level indicators. The closest work to ours was done by Nguyen et al (2014), who proposed a set of crowdsourcing tasks to analyze perception of similarity in folk narratives. They tried various approaches to retrieve these narratives.…”

“…There have been some attempts to match stories (Nguyen et al, 2014;Chaturvedi et al, 2018) and to understand human judgment about matched stories (Nguyen et al, 2014;Reagan et al, 2016). Nevertheless, these efforts have been mostly developed in supervised scenarios that already have a set of matched stories in hand, and they are mostly focused on non-personal narratives (e.g., fictional).…”

Where Have I Heard This Story Before? Identifying Narrative Similarity in Movie Remakes

“…Also noteworthy in this context is the Aarne-Thompson classification system (Aarne and Thompson, 1961), which has been extensively used in the analysis of folk-tales to organize types of stories, based on an index of motifs. Our work is most closely related to that of Nguyen et al (2014) who attempt to understand the various dimensions that experts and non-experts consider while judging narrative similarity.…”