Cognitively inspired task design to improve user performance on crowdsourcing platforms

Sampath, Harini; Rajeshuni, Rajeev; Indurkhya, Bipin

doi:10.1145/2556288.2557155

Cited by 33 publications

(19 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Task Operation Problems (6-8) Several studies indicate that there are often (6) missing responses from requesters on inquiries from workers related to tasks or desired solutions (Silberman, 2010;Silberman et al, 2010a;Silberman et al, 2010b;Bederson and Quinn, 2011;Dow et al, 2012;Chandler et al, 2013;Alagarai Sampath et al, 2014;Silberman, 2015;Brawley and Pury, 2016;Deng and Joshi, 2016;Schwartz, 2018;Berg et al, 2018). Requesters often give only (7) minor feedback to submitted results (Dow et al, 2012;Gaikwad et al, 2017;Schwartz, 2018;Berg et al, 2018).…”

Section: Resultsmentioning

confidence: 99%

Mining Crowdsourcing Problems from Discussion Forums of Workers

Nouri¹,

Wachsmuth²,

Engels³

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Crowdsourcing is used in academia and industry to solve tasks that are easy for humans but hard for computers, in natural language processing mostly to annotate data. The quality of annotations is affected by problems in the task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. To learn about the major problems, we provide a short but comprehensive survey based on two complementary studies: (1) a literature review where we collect and organize problems known from interviews with workers, and (2) an empirical data analysis where we use topic modeling to mine workers' complaints from a new English corpus of workers' forum discussions. While literature covers all process phases, problems in the task evaluation are prevalent, including unfair rejections, late payments, and unjustified blockings of workers. According to the data, however, poor task design in terms of malfunctioning environments, bad workload estimation, and privacy violations seems to bother the workers most. Our findings form the basis for future research on how to improve crowdsourcing processes. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. create task Platform (1) Task design (2) Task operation (3) Task evaluation assess results post task take on task solve task obtain results submit results accept or reject get paid or not questions, inquiries responses, feedback explanations, responses complaints Requester Workers Figure 1: The three phases of a crowdsourcing process. (1) Task design: The requester creates a task.(2) Task operation: Workers accept and solve the task, and submit the results.(3) Task evaluation: The requester accept or reject the results. Communication may happen during task operation and evaluation.approaches in this paper, bringing together knowledge from human computation with ideas from NLP: First, a literature review where we collect crowdsourcing problems discussed in related work, mostly obtained from interviews with and surveys among workers (Section 3). We consolidated the reported challenges from a wide range of studies in order to form a unified view on problems already known in literature. Second, an empirical data analysis where we use topic modeling to mine the workers' problems from what they complain about on a daily basis in online discussion forums (Section 4). Given its dominance, we focus on MTurk here, building a new corpus of 27,041 reviews from Turkopticon, a reputation portal where MTurk workers write reviews on requesters. In our analysis, we focus on the 8,610 negative reviews, since they are likely to contain complaints. 1 The hypothesis underlying our combined approach is that the literature gives a more comprehensive view of problems existing in crowdsourcing, whereas the data better reflects their significance. In light Rof this hypothesis, we study two research questions:RQ1. What problems do workers face in the different phases of crowdsour...

show abstract

Section: Resultsmentioning

confidence: 99%

Mining Crowdsourcing Problems from Discussion Forums of Workers

Nouri¹,

Wachsmuth²,

Engels³

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Multiple studies investigated different techniques to alleviate these limitations, enhance both respondents' performance and data quality acquired from crowdsourcing platforms. Some of the recommendations include increasing participants' attentiveness with engaging features of the data collection instruments (Alagarai Sampath, Rajeshuni, & Indurkhya, 2014), or using filtering algorithms for participants' selection in order to improve data accuracy and avoid unreliable workers (Kan & Drummey, 2018).…”

Section: Literature Reviewmentioning

confidence: 99%

Mechanical Turk or volunteer participant? Comparing the two samples in the study of intelligent personal assistants

Lopatovska

Korshakova

2020

Proc Assoc Inf Sci Technol

View full text Add to dashboard Cite

A challenge in academic and practitioner research is recruiting study participants that match target demographics, possess a desired skillset, and will participate for little to no compensation. An alternative to traditional participant recruitment struggles is crowdsourcing participants through online labor markets, such as Amazon Mechanical Turk (AMT). AMT is a platform that provides the tool for finding and recruiting participants with diverse demographics, skills, and experiences. This paper aims to demystify the use of crowdsourcing, and particularly AMT, by comparing the performance of traditionally recruited volunteers and AMT participants on tasks related to the evaluation of intelligent personal assistants (IPAs such as Amazon Alexa, Google Assistant, Apple Siri, and Microsoft Cortana). The comparison of AMT and non‐AMT samples indicated that while the two samples differed on demographics, their task performance was not significantly different. The paper discusses the costs and benefits of using AMT samples and would be of particular relevance to researchers who employ questionnaires and/or task‐specific data collection methods in their work.

show abstract

“…of UX design and Machine Learning also focuses on improving explicit feedback elicitation. For example, Sampath et al [1] show that different visual designs can alleviate some of the cognitive and perceptual burden of crowd workers in a text extraction task which consequently increases the accuracy of their responses. Another example is the work of Kulesza et al [26], where the authors present an interface that supports users in concept labeling tasks that are typical in Machine Learning.…”

Section: Interface Design For Machine Learningmentioning

confidence: 99%

Shaping Feedback Data in Recommender Systems with Interventions Based on Information Foraging Theory

Schnabel

Bennett

Joachims

2019

Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

Recommender systems rely heavily on the predictive accuracy of the learning algorithm. Most work on improving accuracy has focused on the learning algorithm itself. We argue that this algorithmic focus is myopic. In particular, since learning algorithms generally improve with more and better data, we propose shaping the feedback generation process as an alternate and complementary route to improving accuracy. To this effect, we explore how changes to the user interface can impact the quality and quantity of feedback data-and therefore the learning accuracy. Motivated by information foraging theory, we study how feedback quality and quantity are influenced by interface design choices along two axes: information scent and information access cost. We present a user study of these interface factors for the common task of picking a movie to watch, showing that these factors can effectively shape and improve the implicit feedback data that is generated while maintaining the user experience. CCS CONCEPTS • Information systems → Recommender systems; • Humancentered computing → Interaction design theory, concepts and paradigms; User models; • Computing methodologies → Learning from implicit feedback;

show abstract

Cognitively inspired task design to improve user performance on crowdsourcing platforms

Cited by 33 publications

References 22 publications

Mining Crowdsourcing Problems from Discussion Forums of Workers

Mining Crowdsourcing Problems from Discussion Forums of Workers

Mechanical Turk or volunteer participant? Comparing the two samples in the study of intelligent personal assistants

Shaping Feedback Data in Recommender Systems with Interventions Based on Information Foraging Theory

Contact Info

Product

Resources

About