Crowdsourced POI labelling: Location-aware result inference and Task Assignment

Abstract

Identifying the labels of points of interest (POIs), aka POI labelling, provides significant benefits in location-based services. However, the quality of raw labels manually added by users or generated by artificial algorithms cannot be guaranteed. Such low-quality labels decrease the usability and result in bad user experiences. In this paper, by observing that crowdsourcing is a best-fit for computer-hard tasks, we leverage crowdsourcing to improve the quality of POI labelling. To our best knowledge, this is the first work on crowdsourced POI labelling tasks. In particular, there are two sub-problems: (1) how to infer the correct labels for each POI based on workers' answers, and(2) how to effectively assign proper tasks to workers in order to make more accurate inference for next available workers. To address these two problems, we propose a framework consisting of an inference model and an online task assigner. The inference model measures the quality of a worker on a POI by elaborately exploiting (i) worker's inherent quality, (ii) the spatial distance between the worker and the POI, and (iii) the POI influence, which can provide reliable inference results once a worker submits an answer. As workers are dynamically coming, the online task assigner judiciously assigns proper tasks to them so as to benefit the inference. The inference model and task assigner work alternately to continuously improve the overall quality. We conduct extensive experiments on a real crowdsourcing platform, and the results on two real datasets show that our method significantly outperforms state-of-the-art approaches.

From Social to Individuals: A Parsimonious Path of Multi-Level Models for Crowdsourced Preference Aggregation

“…Beyond the CF approach, there are various techniques to model annotators' abnormal behaviors in general crowdsourcing [18], [21], [23]- [25], [43], [46], [56]- [59], etc. The basic idea of these work is to characterize user quality using some probabilistic behavior models.…”

A Real-Time Framework for Task Assignment in Hyperlocal Spatial Crowdsourcing

“…A recent study distinguishes SC from related fields, including generic crowdsourcing, participatory sensing, volunteered geographic information, and online matching. Research efforts in SC have focused on different aspects, such as task assignment (e.g., [Kazemi and Shahabi 2012;Tong et al 2016;Liu et al 2016b;Hu et al 2016]), task scheduling (e.g., [Deng et al 2016;Li et al 2015;Sales Fonteles et al 2016]), quality control and trust (e.g., [Kazemi et al 2013;Cheng et al 2015]), privacy (e.g., Wang et al 2017]), incentive mechanism (e.g., [Gao et al 2015;Kandappu et al 2016;). The authors in [Kazemi and Shahabi 2012] proposed task assignment problem whose goal is to maximize the number of assigned tasks.…”

SLADE: A Smart Large-Scale Task Decomposer in Crowdsourcing

“…. , bm}, a reliability threshold t Output: An optimal priority queue OP Q 1 Enumerate(1, 0, ∅, B, t); 2 Remove any OP Qi with OP Qi.LCM ≥ OP Qj.LCM and OP Qi.U C ≥ OP Qj.U C for some j; 3 return OP Q; 4 SubFunction:Enumerate(p, q, S, B, t) 5 for k ← p to m do 6 Add b k into S; assigned in this combination, where each atomic task is assigned to six task bins (three 1-cardinality bins, two 2-cardinality bins and one 3-cardinality bins). For example, as shown in the last row in Figure 5, the atomic task a 1 is assigned into the six task bins…”

“…. , S β−1 all ∅; 5 foreach a i ∈ T do 6 Find the lowest j s.t. θ i ≤ 2 j ; 7 Assign a i into S j−α−1 ;…”

“…Crowdsourcing has attracted extensive research attention due to its success in human intrinsic applications. Particularly, a wide spectrum of fundamental data-driven operations have been studied, such as max [2], [3], filtering [4], inference [5], [6], [7] and so on. In addition, researchers and practitioners also pave the way for building crowdpowered databases and data mining systems, and a couple 1. https://www.mturk.com/mturk/ 2. http://www.odesk.com/ of prototypes have been successfully developed, such as CrowdDB [8], Deco [9], [10], Qurk [11], Crowd Miner [12], DOCS [13] and CDB [14].…”