Can the online crowd match real expert judgments? How task complexity and coder location affect the validity of crowd‐coded data

Abstract

Crowd‐coding is a novel technique that allows for fast, affordable and reproducible online categorisation of large numbers of statements. It combines judgements by multiple, paid, non‐expert coders to avoid miscoding(s). It has been argued that crowd‐coding could replace expert judgements, using the coding of political texts as an example in which both strategies produce similar results. Since crowd‐coding yields the potential to extend the replication standard to data production and to ‘scale’ coding schemes based on a modest number of carefully devised test questions and answers, it is important that its possibilities and limitations are better understood. While previous results for low complexity coding tasks are encouraging, this study assesses whether and under what conditions simple and complex coding tasks can be outsourced to the crowd without sacrificing content validity in return for scalability. The simple task is to decide whether a party statement counts as positive reference to a concept – in this case: equality. The complex task is to distinguish between five concepts of equality. To account for the crowd‐coder's contextual knowledge, the IP restrictions are varied. The basis for comparisons is 1,404 party statements, coded by experts and the crowd (resulting in 30,000 online judgements). Comparisons of the expert‐crowd match at the level of statements and party manifestos show that the results are substantively similar even for the complex task, suggesting that complex category schemes can be scaled via crowd‐coding. The match is only slightly higher when IP restrictions are used as an approximation of coder expertise.

COVID-19 Government Response Event Dataset (CoronaNet v.1.0)

“…Each RA is responsible for tracking government policy actions for at least one country. RAs were allocated on the basis of their background, language skills and expressed interest in certain countries 87 . Note that depending on the level of policy coordination at the national level, certain countries were assigned multiple RAs, for example, the United States, Germany and France.…”

Editorial: using smartphones to collect linguistic data

“…Studies of the quality of crowd sourced data indicate that these worries are not entirely justified. Lind et al (2017) find that crowd sourced analyses from paid volunteers is comparable to analyses produced by five research assistants, but that there is variability across different types of tasks, and within groups of volunteers, Especially the complexity of the task that the data comes from is central when determining validity of crowd-sourced data (Horn 2018;Shing et al 2018). Quality of crowd sourced language data concerns primarily its reliability and ecological validity: whether the method provides results that are the same as those found with a different method, and whether the properties of the data can be said to be equal, i.e.…”

Measuring and Visualizing Coders’ Reliability: New Approaches and Guidelines From Experimental Data

“…According to Morris (1977:679), experts have special knowledge about the topic and the material to be coded. Crowdcoders are very lightly trained, have no expertise on the specific topic, and are typically associated with enterprises such as Crowdflower and Mechanical Turk (Cabrera and Reiner 2018;Horn 2018;Lind, Gruber, and Boomgaarden 2017). Trained coders stand somewhere between experts and crowd-coders and are generally prescreened for suitability and trained to complete the task.…”