Multi-Objective Crowd Worker Selection in Crowdsourced Testing

Abstract

Abstract-Crowdsourced testing is an emerging trend in software testing, which relies on crowd workers to accomplish test tasks. Typically, a crowdsourced testing task aims to detect as many bugs as possible within a limited budget. For a specific test task, not all crowd workers are qualified to perform it, and different test tasks require crowd workers to have different experiences, domain knowledge, etc. Inappropriate workers may miss true bugs, introduce false bugs, or report duplicated bugs, which could not only decrease the quality of test outcomes, but also increase the cost of hiring workers. Thus, how to select the appropriate crowd workers for specific test tasks is a challenge in crowdsourced testing.This paper proposes a Multi-Objective crowd wOrker SElection approach (MOOSE), which includes three objectives: maximizing the coverage of test requirement, minimizing the cost, and maximizing bug-detection experience of the selected crowd workers. Specifically, MOOSE leverages NSGA-II, a widely used multi-objective evolutionary algorithm, to optimize the three objectives when selecting workers. We evaluate MOOSE on 42 test tasks (involve 844 crowd workers and 3,984 test reports) from one of the largest crowdsourced testing platforms in China, and the experimental results show MOOSE could improve the best baseline by 17% on average in bug detection rate.

Iterative Learning for K-Approval Votes in Crowdsourcing Systems

“…The authors of [34] modeled trustworthy worker selection as a multi-objective combinatorial optimization problem and solved the problem using evolutionary algorithms. Similarly, the authors of [35] developed a worker selection method with multiple objects including bug detection and cost minimization. Recently, the authors of [36] suggested a generic algorithm that uses video games as a way to gather novel solutions to optimization problems, and the authors of [37] introduced an expertise estimation as a meta-heuristic optimization harmony search problem.…”

A Study on Testers’ Learning Curve in Crowdsourced Software Testing

“…In crowdsourced software testing, there are many researchers studying the recommendations of testers. Qiang et al [27] proposed a multi-target crowd worker selection method (MOOSE) to select crowd testers by maximizing the coverage of testing requirements, minimizing the cost, and maximizing the bug detection experience of the selected group workers. The experiment found that the selected workers could improve the detection rate by 17%.…”

“…According to the different initiative, relevance, and diversity of mass testing workers, Cui et al [8] proposed a selection method that considers the three aspects of work at the same time to select appropriate mass testing personnel for each test task, thus improving the defect detection rate and critical point coverage of testing requirements, reducing the testing cost, and improving the testing efficiency.…”

A systemic framework for crowdsourced test report quality assessment

“…Some studies attempt to resolve the problems existing in crowdsourced testing (Chen and Luo 2014;Starov 2013;Chen et al 2019). To select appropriate workers for testing, Cui et al presented a multi-objective crowd worker selection approach which aims to maximize the coverage of test requirement and bug-detection experience, and minimize the cost (Cui et al 2017). Taking the poor performance of workers, Zhang et al proposed an approach to help workers to acquire domain knowledge and guide them to complete test tasks (Zhang et al 2016).…”