Are your participants gaming the system?

Downs, Julie S.; Holbrook, Mandy B.; Sheng, Steve; Cranor, Lorrie Faith

doi:10.1145/1753326.1753688

Cited by 311 publications

(148 citation statements)

References 3 publications

Supporting

Mentioning

139

Contrasting

Order By: Relevance

“…Experiments 3, 4, and 5 had similar failure rates (38 %, 45 %, and 40 %, respectively). These rates are above or at the high end of those reported in research investigating Mechanical Turk as a subject pool (10 %-39 %; Downs, Holbrook, Sheng, & Cranor, 2010;Goodman, Cryder, & Cheema, 2013;Kapelner & Chandler, 2010). We suspect that our high failure rates are an artifact of the attention check we used.…”

Section: Resultsmentioning

confidence: 60%

“…We suspect that our high failure rates are an artifact of the attention check we used. The article that subjects read was six paragraphs long and came at the end of the experiment, when subjects would be most fatigued and tempted to skim or skip material (see Downs et al, 2010). Moreover, the effort involved in the reading task was different from that of judging the truth of claims.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Nonprobative photos rapidly lead people to believe claims about their own (and other people’s) pasts

et al. 2016

View full text Add to dashboard Cite

Photos lead people to believe that both true and false events have happened to them, even when those photos provide no evidence that the events occurred. Research has shown that these nonprobative photos increase false beliefs when combined with misleading suggestions and repeated exposure to the photo or target event. We propose that photos exert similar effects without those factors, and test that proposition in five experiments. In Experiment 1, people saw the names of several animals and pretended to give food to or take food from each. Then people saw the animal names again, half with a photo of the animal and half alone, and decided whether they had an experience with each. The photos led people to believe they had experiences with the animals. Moreover, Experiments 2-5 provided evidence that photos exerted these effects by making it easier to bring related thoughts and images to mind-a feeling that people mistook as evidence of genuine experience. In each experiment, photos led people to believe positive claims about the past (but not negative claims), consistent with evidence that feelings of ease selectively increase positive judgments. Experiment 4 also showed that photos (like other manipulations of ease) bias people's judgments broadly, producing false beliefs about other people's pasts. Finally, in Experiment 5, photos exerted more powerful effects when they depicted unfamiliar animals, and thus could most help bring information to mind. These findings suggest that nonprobative photos can distort the past without other factors that encourage false beliefs, and that they operate by helping related thoughts and images come to mind.

show abstract

Section: Resultsmentioning

confidence: 60%

Section: Resultsmentioning

confidence: 99%

Nonprobative photos rapidly lead people to believe claims about their own (and other people’s) pasts

et al. 2016

View full text Add to dashboard Cite

show abstract

“…Another potential benefit of improvements to task routing and assignment, is that it might mitigate the need for error controlling approaches that are currently predominantly used in crowdsourcing, such as the Gold Standard [10]. This can be quite important in certain situations, as authoring gold data can be burdensome, and gold standards are challenging to implement in subjective or generative tasks, like writing an essay [31].…”

Section: Benefits Of Appropriate Task Routing and Assignment In Crowdmentioning

confidence: 99%

“…Currently, quality control is typically implemented as posthoc filtering of substandard answers and "smart" aggregation of the crowd contributions. A common technique is to adopt a Gold Standard [10], which entails the creation and inclusion of tasks that have known answers to the requested crowdsourcing job. Another approach is to analyse the extent to which workers agree with each other in their answers [4,17,28].…”

Section: Introductionmentioning

confidence: 99%

Task Routing and Assignment in Crowdsourcing based on Cognitive Abilities

Gonçalves

Feldman

et al. 2017

Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion

View full text Add to dashboard Cite

Appropriate task routing and assignment is an important, but often overlooked, element in crowdsourcing research and practice. In this paper, we explore and evaluate a mechanism that can enable matching crowdsourcing tasks to suitable crowd-workers based on their cognitive abilities. We measure participants' visual and fluency cognitive abilities with the well-established Kit of FactorReferenced Cognitive Test, and measure crowdsourcing performance with our own set of developed tasks. Our results indicate that participants' cognitive abilities correlate well with their crowdsourcing performance. We also built two predictive models (beta and linear regression) for crowdsourcing task performance based on the performance on cognitive tests as explanatory variables. The model results suggest that it is feasible to predict crowdsourcing performance based on cognitive abilities. Finally, we discuss the benefits and challenges of leveraging workers' cognitive abilities to improve task routing and assignment in crowdsourcing environments.

show abstract

“…In some implementations, training results in a pass/fail that screens out untrustworthy or inaccurate users (Downs et al 2010, Le et al 2010. In others, the score attributed to each user's vote is weighted based upon how well they perform during the training (Sheng et al 2014).…”

Section: Improving the Voting Systemmentioning

confidence: 99%

Determining the Accuracy of Crowdsourced Tweet Verification for Auroral Research

Case¹,

MacDonald²,

McCloat³

et al. 2016

CSTP

View full text Add to dashboard Cite

The Aurorasaurus project harnesses volunteer crowdsourcing to identify sightings of an aurora (the "northern/southern lights") posted by citizen scientists on Twitter. Previous studies have demonstrated that aurora sightings can be mined from Twitter with the caveat that there is a large background level of non-sighting tweets, especially during periods of low auroral activity. Aurorasaurus attempts to mitigate this, and thus increase the quality of its Twitter sighting data, by using volunteers to sift through a pre-filtered list of geolocated tweets to verify real-time aurora sightings. In this study, the current implementation of this crowdsourced verification system, including the process of geolocating tweets, is described and its accuracy (which, overall, is found to be 68.4%) is determined. The findings suggest that citizen science volunteers are able to accurately filter out unrelated, spam-like, Twitter data but struggle when filtering out somewhat related, yet undesired, data. The citizen scientists particularly struggle with determining the real-time nature of the sightings, so care must be taken when relying on crowdsourced identification.

show abstract

Are your participants gaming the system?

Cited by 311 publications

References 3 publications

Nonprobative photos rapidly lead people to believe claims about their own (and other people’s) pasts

Nonprobative photos rapidly lead people to believe claims about their own (and other people’s) pasts

Task Routing and Assignment in Crowdsourcing based on Cognitive Abilities

Determining the Accuracy of Crowdsourced Tweet Verification for Auroral Research

Contact Info

Product

Resources

About