Crowdsourced Data Management: A Survey

Abstract

Some important data management and analytics tasks cannot be completely addressed by automated processes. These "computer-hard" tasks such as entity resolution, sentiment analysis, and image recognition, can be enhanced through the use of human cognitive ability. Human Computation is an effective way to address such tasks by harnessing the capabilities of crowd workers (i.e., the crowd). Thus, crowd sourced data management has become an area of increasing interest in research and industry. There are three important problems in crowd sourced data management. (1) Quality Control: Workers may return noisy results and effective techniques are required to achieve high quality; (2) Cost Control: The crowd is not free, and cost control aims to reduce the monetary cost; (3) Latency Control: The human workers can be slow, particularly in contrast to computing time scales, so latency-control techniques are required. There has been significant work addressing these three factors for designing crowd sourced tasks, developing crowd sourced data manipulation operators, and optimizing plans of multiple operators. In this paper, we survey and synthesize a wide spectrum of existing studies on crowd sourced data management. Based on this analysis we then outline key factors that need to be considered to improve crowd sourced data management.

Computing Crowd Consensus with Partial Agreement

“…Answer aggregation, thus, shall capture worker characteristics, to assess the likelihood of them providing correct answers and to justify their effects in the aggregated result. The existence of different worker types, as illustrated above, has been verified in various studies [26], [27] as well as in our experimental evaluation. (R2) Support for partial answer validity.…”

Citizen Science: An Information Quality Research Frontier

“…Explicit open calls to members of the general public for specific information underlie the booming practice of crowdsourcing (Deng et al 2016;Ghezzi et al 2017;Love and Hirschheim 2017;Lukyanenko and Parsons 2018). To offer crowd services at massive scale, platforms like Amazon's Mechanical Turk and CrowdFlower.com allow companies to tap into thousands of paid crowd workers (Chittilappilly et al 2016;Deng et al 2016;Garcia-Molina et al 2016;Li et al 2016;Stewart et al 2015). Organizations also create their own platforms, such as the widely popular Zooniverse (www.zooniverse.org), where scientists can post data processing tasks (e.g., image annotation, classification, transcription) to over a million engaged volunteers (Simpson et al 2014).…”

“…User-generated content stands to rapidly expand the scope of IQ research as it presents novel challenges and opens many new questions (Li et al 2016;Lukyanenko and Parsons 2015b). Unlike information produced by organizational employees, suppliers, and contractors with close ties to the organization, UGC is created by Bcasualĉ ontent contributors with varying levels of subject matter expertise, reasons for producing content, backgrounds, and worldviews (Daugherty et al 2008;Levina and Arriaga 2014;Lukyanenko et al 2014b;Susarla et al 2012).…”

High-Dimensional Crowdsourced Data Distribution Estimation with Local Privacy

“…With the development of various integrated sensors and crowd sensing systems [19], crowdsourced information from all aspects can be collected and analyzed to better produce rich knowledge about the group, which can benefit everyone in the crowdsourced system [20]. Particularly, with multi-dimensional crowdsourced data (data with multiple attributes), a lot of potential information and patterns behind the data can be mined or extracted to provide accurate dynamics and reliable prediction for both group and individuals.…”