Artificial Intelligence for Artificial Artificial Intelligence

Abstract

Crowdsourcing platforms such as Amazon Mechanical Turk have become popular for a wide variety of human intelligence tasks; however, quality control continues to be a significant challenge. Recently, we propose TurKontrol, a theoretical model based on POMDPs to optimize iterative, crowd-sourced workflows. However, they neither describe how to learn the model parameters, nor show its effectiveness in a real crowd-sourced setting. Learning is challenging due to the scale of the model and noisy data: there are hundreds of thousands of workers with high-variance abilities. This paper presents an end-to-end system that first learns TurKontrol's POMDP parameters from real Mechanical Turk data, and then applies the model to dynamically optimize live tasks. We validate the model and use it to control a successive-improvement process on Mechanical Turk. By modeling worker accuracy and voting patterns, our system produces significantly superior artifacts compared to those generated through nonadaptive workflows using the same amount of money.

Dynamically Switching between Synergistic Workflows for Crowdsourcing

“…Since the true answer values v are unknown, an option is supervised learning, where experts label true answers and difficulties. This approach was used by Dai et al (2011). But, this is not a scalable option, since it requires significant expert time upfront.…”

“…• We evaluate the benefits of our approach first in a simulated environment and then with live experiments on Amazon Mechanical Turk. We show that AGENTHUNT outperforms the state-of-the-art single-workflow task controller, TURKONTROL, (Dai, Mausam, and Weld 2011), achieving up to 50% error reduction and greater net utility for the task of generating NLP training data. Surprisingly, we also show that our adaptive RL method yields almost as high a utility as the approach requiring an explicit training phase.…”

Octopus: A Framework for Cost-Quality-Time Optimization in Crowdsourcing

“…The other branch studies intelligent control, which dynamically decides whether to ask for a new ballot on a task, or stop and submit the answer. These include control of binary or multiple choice tasks (Dai et al 2013;Parameswaran et al 2012;Kamar et al 2013), multi-label tasks (Bragg, Mausam, and Weld 2013), and tasks beyond multiple choice answers (Lin, Mausam, and Weld 2012;Dai, Mausam, and Weld 2011). All these works design agents to control a single task and assume a constant pay per ballot.…”

Adaboost modular tensor locality preservative projection: face detection in video using Adaboost modular‐based tensor locality preservative projections

“…He et al [15] proposed tensor subspace analysis for PCA, linear discriminant analysis (LDA) and LPP, which considers the image as second order tensors. This method was further extended by Dai and Yeung [16] to tensor LPP, which provided the relationship between dimensions of a tensor representation thus allowing efficient characterisation. Adaboost algorithm proposed by Freund and Schapire [17,18], is used to considerably reduce the errors in the learning algorithm that constantly generate classifiers whose performance is slightly better than a random guess.…”