Scheduling Human Intelligence Tasks in Multi-Tenant Crowd-Powered Systems

Abstract

Micro-task crowdsourcing has become a popular approach to e↵ectively tackle complex data management problems such as data linkage, missing values, or schema matching. However, the backend crowdsourced operators of crowd-powered systems typically yield higher latencies than the machineprocessable operators, this is mainly due to inherent efficiency di↵erences between humans and machines. This problem can be further exacerbated by the lack of workers on the target crowdsourcing platform, or when the workers are shared unequally among a number of competing requesters; including the concurrent users from the same organization who execute crowdsourced queries with di↵erent types, priorities and prices. Under such conditions, a crowd-powered system acts mostly as a proxy to the crowdsourcing platform, and hence it is very di cult to provide e ency guarantees to its end-users.Scheduling is the traditional way of tackling such problems in computer science, by prioritizing access to shared resources. In this paper, we propose a new crowdsourcing system architecture that leverages scheduling algorithms to optimize task execution in a shared resources environment, in this case a crowdsourcing platform. Our study aims at assessing the e ciency of the crowd in settings where multiple types of tasks are run concurrently. We present extensive experimental results comparing i) di↵erent multi-tenant crowdsourcing jobs, including a workload derived from real traces, and ii) di↵erent scheduling techniques tested with real crowd workers. Our experimental results show that task scheduling can be leveraged to achieve fairness and reduce query latency in multi-tenant crowd-powered systems, although with very di↵erent tradeo↵s compared to traditional settings not including human factors. General Terms

Deadline-Aware Fair Scheduling for Multi-Tenant Crowd-Powered Systems

“…In this line of work, we propose to overhaul the pull-crowdsourcing scheme by taking control of the distribution process of tasks submitted by multiple requesters. In [20], we have proposed a push-crowdsourcing architecture that automatically assigns tasks to workers. Such architecture allows us to order the execution of the tasks and to apply scheduling strategies tailored to the unpredictable nature of the humans.…”

“…Sharing heterogeneous resources across tasks having diferent priorities is a well-known problem, one popular approach is Fair Scheduling (FS) [25]. In this work, we expand on our approach introduced in [20] and propose a universal model of fairness, one that can directly embed several aspects deined in our requirements described in Section 2.3.…”

“…In our previous work [20], we proposed a Fair Scheduler that operates with the following strategy: Ωhenever a worker is available, assign a HIT from the batch with the lowest number of currently assigned HITs R b . Unlike Round Robin, this ensures that all the batches continuously get a non-zero amount of resources.…”

Implementation of cost effective hierarchical Hadoop cluster–a case study for education

“…al. proposed crowd sourcing system architecture to enhance the efficiency of the crowd while running different types of tasks in parallel [12] [14]. Dazhao Cheng et.…”

Crowd intelligence in AI 2.0 era

“…Through experiments, they observed that selecting a subset of participants can indeed increase the task accuracy. They then proposed a scheduling workflow in a subsequent study (Difallah et al, 2016).…”