POMDP-based control of workflows for crowdsourcing

Dai, Peng; Lin, Christopher H.; Weld, Daniel S.

doi:10.1016/j.artint.2013.06.002

Cited by 111 publications

(88 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Typical workflow adaptions include: varying the number of workers to recruit for a task [30,47], assigning stronger workers to harder versions of a task [10], and/or fundamentally changing the task an individual worker is given [29,35] These workflow decisions are made automatically by workflow controllers often by analyzing worker performance history, inferring task difficulty, or estimating cost.…”

Section: Dynamic Workflow Allocationmentioning

confidence: 99%

Tohme

Hara

Sun

Moore

et al. 2014

Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

Building on recent prior work that combines Google Street View (GSV) and crowdsourcing to remotely collect information on physical world accessibility, we present the first "smart" system, Tohme, that combines machine learning, computer vision (CV), and custom crowd interfaces to find curb ramps remotely in GSV scenes. Tohme consists of two workflows, a human labeling pipeline and a CV pipeline with human verification, which are scheduled dynamically based on predicted performance. Using 1,086 GSV scenes (street intersections) from four North American cities and data from 403 crowd workers, we show that Tohme performs similarly in detecting curb ramps compared to a manual labeling approach alone (Fmeasure: 84% vs. 86% baseline) but at a 13% reduction in time cost. Our work contributes the first CV-based curb ramp detection system, a custom machine-learning based workflow controller, a validation of GSV as a viable curb ramp data source, and a detailed examination of why curb ramp detection is a hard problem along with steps forward.

show abstract

Section: Dynamic Workflow Allocationmentioning

confidence: 99%

Tohme

Hara

Sun

Moore

et al. 2014

Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

show abstract

“…The AMT tools for managing workers and tasks are often not sufficient to block poor quality workers and spammers. Principled methods for improving the quality of results from AMT are currently being explored [20], but their applicability to the robotics domain remains to be explored. Fig.…”

Section: B Crowdsourcingmentioning

confidence: 99%

Accelerating imitation learning through crowdsourcing

Chung

Forbes

Çakmak

et al. 2014

2014 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Abstract-Although imitation learning is a powerful technique for robot learning and knowledge acquisition from naïve human users, it often suffers from the need for expensive human demonstrations. In some cases the robot has an insufficient number of useful demonstrations, while in others its learning ability is limited by the number of users it directly interacts with. We propose an approach that overcomes these shortcomings by using crowdsourcing to collect a wider variety of examples from a large pool of human demonstrators online. We present a new goal-based imitation learning framework which utilizes crowdsourcing as a major source of human demonstration data. We demonstrate the effectiveness of our approach experimentally on a scenario where the robot learns to build 2D object models on a table from basic building blocks using knowledge gained from locals and online crowd workers. In addition, we show how the robot can use this knowledge to support human-robot collaboration tasks such as goal inference through object-part classification and missing-part prediction. We report results from a user study involving fourteen local demonstrators and hundreds of crowd workers on 16 different model building tasks.

show abstract

“…Clearly, other solving approaches are possible, and, indeed, a significant number of works ( [24], [15], [7], [9], [5], [29], [12]) cast the task assignment problem as an online decision-making problem, thus assuming that each annotator must be assigned to a task in an online fashion. Unlike our work, the accuracy of each annotator is initially unknown, and must be learnt through observation.…”

Section: Related Workmentioning

confidence: 99%

A workload-dependent task assignment policy for crowdsourcing

et al. 2017

View full text Add to dashboard Cite

Crowdsourcing marketplaces have emerged as an effective tool for high-speed, low-cost labeling of massive data sets. Since the labeling accuracy can greatly vary from worker to worker, we are faced with the problem of assigning labeling tasks to workers so as to maximize the accuracy associated with their answers. In this work, we study the problem of assigning workers to tasks under the assumption that workers' reliability could change depending on their workload, as a result of, e.g., fatigue and learning. We offer empirical evidence of the existence of a workload-dependent accuracy variation among workers, and propose solution procedures for our Crowdsourced Labeling Task Assignment Problem, which we validate on both synthetic and real data sets.

show abstract

POMDP-based control of workflows for crowdsourcing

Cited by 111 publications

References 10 publications

Tohme

Tohme

Accelerating imitation learning through crowdsourcing

A workload-dependent task assignment policy for crowdsourcing

Contact Info

Product

Resources

About