Process Improvements for Crowdsourced Software Testing

Abstract

Crowdsourced software testing has been a common practice lately. It refers to the use of crowdsourcing in software testing activities. Although the crowd testing is a collaborative process by nature, there is no available research that provides a critical assessment of the key collaboration activities offered by the current crowdsourced testing platforms. In this paper, we review the process used in the crowd testing platforms including identifying the workflow used in managing the crowd testing process starting from submitting testing requirements and ending with reviewing testing report. Understanding the current process is then utilized to identify a set of limitations in the current process and has led to propose three process improvements (improving assigning crowd manager, improving building test team, monitoring testing progress). We have designed and implemented these process improvements and then evaluated them using two techniques: 1) questionnaire and 2) workshop. The questionnaire shows that the process improvements are significantly sound and strong enough to be added to crowd testing platforms. In addition, the evaluation through conducting a workshop was useful to assess the design and implementation of the process improvements. The participants were satisfied with them but asked for further modifications. Nevertheless, because crowd testing requires participation from a large number of people, the automation suggested improving managing the current process which was highly appreciated.

Mobile devices compatibility testing strategy via crowdsourcing

“…Most of these testing platforms and/or companies are not directly targeting compatibility testing in their testing solutions. These testing platforms and companies have a more focus on other types of testing such as functional, security, usability, load, localization and automation (Alyahya and Alrugebh, 2017). The crowdtesting method used in these aforementioned testing platforms and companies mainly concentrates on testing the mobile app as a whole to make sure that the app is free of errors.…”

“…At that time, crowdtesting included only four key elements: developer, crowd manager, crowd leader, and crowd testers. Alyahya and Alrugebh (2017) emphasized that C-CT workflow has several limitations. They addressed these limitations by proposing certain automated processes as illustrated in Figure 1:…”

“…Recently, several testing approaches such as Automated testing (Prathibhan et al , 2014), Cloud testing services (Huang, 2014b; Bhojan et al , 2015; Huang, 2012a; Starov, 2013) and industrial crowdtesting platforms or companies (Huang, 2014b; Alyahya and Alrugebh, 2017; Mao et al , 2017; Starov, 2013; Huang, 2012a), MyCrowd QA[2] and Pay4Bug[3] have been built to address the compatibility issue. However, the issue still remains.…”

Hybrid Crowd-powered Approach for Compatibility Testing of Mobile Devices and Applications

“…Most of these platforms and companies are not directly targeting compatibility testing in their testing solutions. These platforms and companies focus more on other types of testing such as functional, security, usability, load, localization, and automation [13]. The crowdtesting method used in these aforementioned platforms and companies is mainly focused on testing the app as a whole to make sure that the app is free of errors.…”

“…(d) The lack of relevant knowledge and experience [33]. Recently, several testing approaches like Automated testing [28], Cloud testing services [15,19,20,30] and industrial crowdtesting platforms or companies [5,7,11,13,19,20,26,30] have been built to address the compatibility issue. However, the issue remains [16].…”