Salience Bias in Crowdsourcing Contests

Abstract

Crowdsourcing relies on online platforms to connect a community of users to perform specific tasks. However, without appropriate control, the behavior of the online community might not align with the platform's designed objective, which can lead to an inferior platform performance. This paper investigates how the feedback information on a crowdsourcing platform and systematic bias of crowdsourcing workers can affect crowdsourcing outcomes. Specifically, using archival data from the online crowdsourcing platform Kaggle, combined with survey data from actual Kaggle contest participants, we examine the role of a systematic bias, namely the salience bias, in influencing the performance of the crowdsourcing workers and how the number of crowdsourcing workers moderates the impact of the salience bias as a result of the parallel path effect and competition effect. Our results suggest that the salience bias influences the performance of contestants, including the winners of the contests. Furthermore, the parallel path effect cannot completely eliminate the impact of the salience bias, but it can attenuate it to a certain extent. By contrast, the competition effect is likely to amplify the impact of the salience bias. Our results have critical implications for crowdsourcing firms and platform designers.

Introduction—Platforms and Infrastructures in the Digital Age

“…For instance, the research conducted by Lee, Ba, Li, and Stallaert (2018) in this issue examines the performance of crowdsourcing contests on Kaggle. The authors show that despite the well-reported merits of using open contests for research and development tasks previously performed within internal business units (Boudreau and Lakhani 2013), the performance of these contests is very much dependent on in-progress feedback to the contestants.…”

Community interactions at crowd scale: hybrid crowds on the GitHub platform

“…To identify and examine projects that may exhibit both attributes of both communities and crowds, we used an in-depth analysis of archival data obtained through the GitHub platform. Similar archival methods have been used in other open innovation studies of both communities (Roberts, Hann, & Slaughter, 2006) and crowds (Lee, Ba, Li, & Stallaert, 2018). More broadly, the study of open source has been a popular setting for the study of open innovation (Dahlander & Magnusson, 2008;West & Lakhani, 2008).…”

Gamifying the gig: transitioning the dark side to bright side of online engagement

“…Archival data also helps in understanding the changing needs and requirements posted on the portals across different geographies. However, studies have also reported that data validation is key for archival data to improve the robustness of the results (Lee et al, 2018). We then contacted website owners seeking permission to send a questionnaire to the participants in these tasks.…”