Predicting result quality in Crowdsourcing using application layer monitoring

Hirth, Matthias; Scheuring, Sven; Hoßfeld, Tobias; Schwartz, Christian; Tran-Gia, Phuoc

doi:10.1109/cce.2014.6916756

Cited by 24 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Approaches that predict worker quality from their behavior have been proposed [5], [6], [7], [15]. They do not require the creation of a gold task and do not assume redundancy; hence, they make it possible to reduce costs.…”

Section: Related Workmentioning

confidence: 99%

“…They gathered feature data such as the number of clicks, keyboard operations, and processing time from the workers. On the other hand, Hirth et al [6] used the following features: the duration in which the worker reads the target text, and the answer time in which the answer is considered as determined from scrolling of the page or the click interval of the radio button. Moreover, they predicted the quality of the workers by using a machine learning algorithm that took the workers' behaviors as input.…”

Section: Related Workmentioning

confidence: 99%

“…To solve this problem, researchers have proposed several methods that measure worker behaviors, such as working time and number of clicks, for predicting their quality [4], [5], [6], [7]. The systems embodying these approaches automatically obtain worker behaviors on their work screens and estimate worker qual-ity by using machine learning.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Additional Operations of Simple HITs on Microtask Crowdsourcing for Worker Quality Prediction

Suzuki

Matsuda

Nakamura

2019

Journal of Information Processing

View full text Add to dashboard Cite

In microtask crowdsourcing, low-quality workers damage the quality of work results. Therefore, if a system automatically eliminates the low-quality workers, the requesters will obtain high-quality work results with low wages. When we consider simple Human Intelligent Tasks (HITs), such as yes-no questions of a labeling task, the requesters have difficulty assessing the worker quality only from the work results. Therefore, we need a method to accurately predict the worker quality automatically from the behaviors of workers, such as working time and the number of clicks. When we accurately predict the worker quality, we are able to prepare many features from the worker behaviors. However, when we submit simple HITs, we can capture only a small number of behaviors of workers, then the accuracy of predicted worker quality will be low. To solve this issue, we propose a method to insert into the simple task of obtaining many features of worker behaviors. We prepared a classification task of tweets as simple HITs. We added a button to the work screen. The workers can browse the target tweets on the work screen during the time the workers are pressing the button, but the workers cannot browse the target tweets when the workers have released the button. Using this button, we can obtain six more kinds of features of worker behaviors. Using our method, we can improve the recall ratio 12% of identifying low-quality workers. However, as the load of workers increases, then the processing time becomes longer, and the motivation of workers decreases. From this result, we also discovered that there is a trade-off between the number of obtained behaviors and the load of workers.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Additional Operations of Simple HITs on Microtask Crowdsourcing for Worker Quality Prediction

Suzuki

Matsuda

Nakamura

2019

Journal of Information Processing

View full text Add to dashboard Cite

show abstract

“…Nevertheless, the heterogeneous environment and unsupervised nature of such QoE studies can introduce pitfalls, which have to be avoided by proper study design and the ltering of unreliable results [77]. Therefore, in the eld of crowdsourced QoE studies, research deals with motivation and incentives of participants (e.g., [59,152]), methods for screening the reliability of participants (e.g., [54]), mechanisms for asserting a high quality of results (e.g., [32,36,153]), and the development of crowdsourcing frameworks and platforms (e.g., [154]). A comprehensive report of best practices and lessons learned for crowdsourced QoE studies can be found in [77].…”

Section: Impact Of Adaptation On the Qoe Of Hasmentioning

confidence: 99%

Quality of experience and access network traffic management of HTTP adaptive video streaming

Seufert

Tran-Gia

2018

NOMS 2018 - 2018 IEEE/IFIP Network Operations and Management Symposium

Self Cite

View full text Add to dashboard Cite

“…A defensive task design, i.e., making it easier to complete the task in a meaningful manner than to find a means to cheat, can be applied to measurements in which no user interaction is required. If user interaction is required, e.g., the worker must access certain web pages or videos, such interactions can be monitored [52][53][54] or additional validation questions [48] about the content of the visited pages or videos can be added to verify correct task completion by the worker. …”

Section: Best Practicesmentioning

confidence: 99%