Detecting malicious campaigns in crowdsourcing platforms

Abstract

Crowdsourcing systems enable new opportunities for requesters with limited funds to accomplish various tasks using human computation. However, the power of human computation is abused by malicious requesters who create malicious campaigns to manipulate information in web systems such as social networking sites, online review sites, and search engines. To mitigate the impact and reach of these malicious campaigns to targeted sites, we propose and evaluate a machine learning based classification approach for detecting malicious campaigns in crowdsourcing platforms as a first line of defense, and build a malicious campaign blacklist service for targeted site providers, researchers and users.Specifically, we (i) conduct a comprehensive analysis to understand the characteristics of malicious campaigns and legitimate campaigns in crowdsourcing platforms, (ii) propose various features to distinguish between malicious campaigns and legitimate campaigns, (iii) evaluate a classification approach against baselines, and (iv) build a malicious campaign blacklist service. Our experimental results show that our proposed approaches effectively detect malicious campaigns with low false negative and false positive rates. Web. To solve this problem, we build a predictor that detects whether campaign/task in crowdsourcing sites is malicious or not so that malicious campaigns/tasks can be removed by crowdsourcing site providers as soon as they are created. In particular, we (i) analyze characteristics of malicious and legitimate campaigns; (ii) extract commonly available features from four crowdsourcing sites; and (iii) build predictors and evaluate their performance. Our experimental results show that our predictors are more effective and robust compared with several baselines. In the end, we design and build a malicious campaign blacklist service, which provides users with various information.

Building a task blacklist for online social platforms

“…The small dataset consists of 23,220 crowdsourcing campaigns with manually labeled information (i.e., which campaign is malicious or legitimate), including 3,356,153 tasks 3 . This dataset was collected from MTurk, Microworkers, Rapidworkers, and Shorttask by using a crawler that we developed for the prior work [1]. The crawler collected 23,220 campaigns, including detailed campaign descriptions within a period of three months between November 2014 and January 2015.…”

“…On another note, Fayazi et al [3] showed that malicious requestors targeted popular search engines (e.g., Google, Bing), social media sites (e.g., Facebook, Pinterest), and online e-commerce sites (e.g., Amazon) by creating malicious tasks and hiring workers from crowdsourcing platforms. Choi et al [1] proposed a novel approach to define, classify and detect malicious campaigns that exist on several popular crowdsourcing platforms. They first analyzed the characteristics of malicious campaigns as opposed to legitimate campaigns, and then made use of these characteristics (or features) to build a classification model to identify malicious campaigns with a high accuracy.…”

“…Crowdsourcing platforms such as Amazon Mechanical Turk (MTurk) 1 and Figure Eight 2 have gained considerable favor among companies and users, as they have made the process of appealing to the mass for completion of micro-tasks much easier. The crowdsourcing model provides requesters with a low-cost, constantly available workforce, mitigating the need to hire fixed-time employees to perform simple tasks.…”

“…While crowdsourcing seems to be an ideal model for both requesters and workers, crowdsourcing platforms face a risk of being exploited by malicious requesters, who create malicious tasks. Choi et al [1] found that malicious tasks aimed at information manipulation by manipulating search engines, writing fake reviews or creating manipulated accounts, etc. In the work, they found that malicious campaigns have several notable characteristics: shorter estimated completion time along with higher rewards rate, which makes them more appealing to workers when selecting tasks to complete because some workers may not care about ethical issues or consequences.…”

“…So far, there exists no research or product that compiles the findings of malicious campaigns into a comprehensive list where affected parties can utilize to moderate the crowdsourcing environment. it has given us the motivation to build classification models inspired by the findings from [1]'s research, and detect additional malicious campaigns within over 446K campaigns crawled from several crowdsourcing sites. In the end, we build a task blacklist, a web service, which provides a keyword-based search service so that users can query for malicious campaigns based on the targeted site, task description or requesters' information.…”