Scoring from Contests

Schnakenberg, Keith E.; Penn, Elizabeth Maggie

doi:10.1093/pan/mpt018

Cited by 2 publications

(2 citation statements)

References 26 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Once we obtain the crowd-sourced rankings, we will use contest scoring techniques (Schnakenberg & Penn 2014, Coppedge, Glynn, Lindberg, Pemstein & Seim 2015 to estimate latent scales from the participant-provided rankings and compare country-year scores on the crowd-sourced scales to those produced by applying IRT models to expert-produced Likert scores.…”

Section: Likert Scales Vs Paired Comparisonsmentioning

confidence: 99%

See 1 more Smart Citation

Experts, Coders, and Crowds::An Analysis of Substitutability

et al. 2017

View full text Add to dashboard Cite

Varieties of Democracy (V-Dem) is a new approach to conceptualization and measurement of democracy. It is co-hosted by the University of Gothenburg and University of Notre Dame. With a V-Dem Institute at University of Gothenburg with almost ten staff, and a project team across the world with four Principal Investigators, fifteen Project Managers (PMs), 30+ Regional Managers, 170 Country Coordinators, Research Assistants, and 2,500 Country Experts, the V-Dem project is one of the largest ever social science research-oriented data collection programs.Please address comments and/or queries for information to: AbstractRecent work suggests that crowd workers can replace experts and trained coders in common coding tasks. However, while many political science applications require coders to both find relevant information and provide judgment, current studies focus on a limited domain in which experts provide text for crowd workers to code. To address potential over-generalization, we introduce a typology of data producing actors-experts, coders, and crowds-and hypothesize factors which affect crowd-expert substitutability. We use this typology to guide a comparison of data from crowdsourced and expert surveys. Our results provide sharp scope conditions for the substitutability of crowd workers: when coding tasks require contextual and conceptual knowledge, crowds produce substantively different data from coders and experts. We also find that crowd workers can cost more than experts in the context of cross-national panels, and that one purported advantage of crowdsourcingreplicability-is undercut by an insufficient number of crowd workers.Political scientists often rely on experts to code data. While expertise plays an important role in a wide range of coding tasks, an increasing number of "expert surveys" (e.g., the British Election Study Expert Survey, the Chapel Hill Expert Survey, the Electoral Integrity Project, Quality of Government, Transparency International, and Varieties of Democracy) prominently tout the advantages of expert coders. Experts' purported virtues are manifold. They allow researchers to obtain information on complex topics; gather data cross-nationally and over time, even when observable indicators (e.g. roll call votes or election manifestos) are not universally available; and deductively determine the content of their measures (Hooghe, Bakker, Brigevich, de Vries, Edwards, Marks, Rovny & Steenbergen 2010). Surveys also distribute coding efforts across the research community, providing a widely accessible public good, and are inexpensive compared to fieldwork, archival research, and large-scale public and elite surveys.Crowdsourcing-the large-scale recruitment of lay persons to code data-has emerged as a tool for data collection in traditionally expert-reliant domains (Kittur, Chi & Suh 2008, Cooper, Khatib, Treuille, Barbero, Lee, Beenen, Leaver-Fay, Baker, Popovic & Players 2010, Honaker, Berkman, Ojeda & Plutzer 2013, D'Orazio, Kenwick, Lane, Palmer & Reitter 2016. The key distinction between...

show abstract

Section: Likert Scales Vs Paired Comparisonsmentioning

confidence: 99%

“…Then, once for each pay group, we will fit a contest scoring model (Schnakenberg & Penn 2014) to the paired rank data, bootstrapping the procedure 1000 times to produce estimates of uncertainty. For each question this will produce a rank ordering of the cases considered by crowd coders.…”

Section: Comparing Model Based Ranksmentioning

confidence: 99%