Read-Agree-Predict: A Crowdsourced Approach to Discovering Relevant Primary Sources for Historians

Abstract

Historians spend significant time evaluating the relevance of primary sources that they encounter in digitized archives and through web searches. One reason this task is time-consuming is that historians’ research interests are often highly abstract and specialized. These topics are unlikely to be manually indexed and are difficult to identify with automated text analysis techniques. In this article, we investigate the potential of a new crowdsourcing model in which the historian delegates to a novice crowd the task of evaluating the relevance of primary sources with respect to her unique research interests. The model employs a novel crowd workflow, Read-AgreePredict (RAP), that allows novice crowd workers to perform as well as expert historians. As a useful byproduct, RAP also reveals and prioritizes crowd confusions as targeted learning opportunities. We demonstrate the value of our model with two experiments with paid crowd workers (n=170), with the future goal of extending our work to classroom students and public history interventions. We also discuss broader implications for historical research and education.

Annotator Rationales for Labeling Tasks in Crowdsourcing

“…Whereas several prior crowdsourcing studies have asked workers to provide open-ended textual justifications for their answers (Alonso, 2009;Drapeau et al, 2016;Chang et al, 2017;Wang et al, 2019;Han et al, 2020), the rationales approach (Zaidan et al, 2007) is differentiated by requiring a far more restricted form of justification. For a given instance x to be labeled, annotators identify a subset of x as the rationale for their label; there is no free-form text response.…”