An Empirical Study on Task Documentation in Software Crowdsourcing on TopCoder

Abstract

In the Software Crowdsourcingcompetitive model, crowd members seek for tasks in a platform and submit their solutions seeking rewards. In this model, the task description is important to support the choice and the execution of a task. Despite its importance, little is known about the role of task description as support for these processes. To fill this gap, this paper presents a study that explores the role of documentation on TopCoder platform, focusing on the task selection and execution. We conducted a two-phased study with professionals that had no prior contact with TopCoder. Based on data collected with questionnaires, diaries, and a retrospective session, we could understand how people choose and perform the tasks, and the role of documentation in the platform. We could find that poorly specified or incomplete tasks lead developers to look for supplementary material or invest more time and effort than initially estimated. To better support the crowd members, we proposed a model on how to structure the documentation that composes the task description in competitive software crowdsourcing. We evaluated the model with another set of professionals, again relying on questionnaires, reports, and a retrospective session. Results showed that although the documentation available covered the elements of the proposed model, the participants had issues to find the necessary information, suggesting the need for a reorganization. Participants agreed that the proposed model would help them understand the task description. Therefore, our study provides a better understanding of the importance of task documentation in software crowdsourcing and points out what information is important to the crowd.

Crowdsourcing in Software Development: Empirical Support for Configuring Contests

“…Crowd participation was found to be influenced by the clarity of the description of the associated tasks [35]. Tasks with unclear objective description, without specifying required technologies or environment setup instructions, discourage developers from selecting them.…”

“…In particular, the context of the case study is crowdsourced software development, whereas as cases and units of analysis we consider crowdsourced competitions hosted by the TopCoder platform. TopCoder platform is a pioneer for practicing Crowdsourced Software Engineering (CSE), systematically used for empirical research the last years [2], [39], [35].…”

Microtasking Activities in Crowdsourced Software Development: A Systematic Literature Review

“…These platforms support variety of tasks and provide different facilities to their users in terms of remuneration, social recognition, bonuses and e-gifts [9], [10]. Few platforms support the specific tasks (Quicktate and iDictate for call auditing and Topcoder for programming), while most of the platforms facilitate their clients with a variety of tasks related to designing, programming and development, testing and quality assurance, interpretation and analysis and content writing [11], [12].…”

“…This model is suitable when high quality and diversified results are required by the client. Different crowdsourcing platforms implement competitions model e.g., Topcoder, 99designs, testbirds and uTests to crowdsource the development, designing, usability and system testing related tasks respectively [9], [12], [34].…”

Can I Solve it? Identifying the APIs required to complete OSS tasks

“…Finding tasks to contribute to in Open Source projects is challenging [1,2,3,4,5]. Open tasks vary in complexity and required skills, which can be difficult to determine solely by reading the task descriptions alone, especially for new contributors [6,7,8]. Adding labels to the issues (a.k.a tasks, bug reports) help new contributors when they are choosing their tasks [9].…”