The Power of the Crowd: Performing Usability Testing Using an On-Demand Workforce

Schneider, Christoph; Cheung, Terence

doi:10.1007/978-1-4614-4951-5_44

Cited by 12 publications

(15 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous studies show various applications of MTurk in research. It has been widely used for different purposes becoming widely accepted in business research [53], [60], [61], transcriptions [56], experiment designs [57], qualitative designs [58] and user evaluation studies [59]. As our theory is based on general memory systems, we conducted the experiments with the general population.…”

Section: Methodsmentioning

confidence: 99%

Data Collection Interfaces in Online Communities: The Impact of Data Structuredness and Nature of Shared Content on Perceived Information Quality

Maddah

Lukyanenko

VanderMeer

et al. 2020

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

The growth of online communities has resulted in an increased availability of user-generated content (UGC). Given the varied sources of UGC, the quality of information it provides is a growing challenge. While many aspects of UGC have been studied, the role of data structures in gathering UGC and nature of to-be-shared content has yet to receive attention. UGC is created in online platforms with varying degrees of data structure, ranging from unstructured to highly-structured formats. These platforms are often designed without regard to how the structure of the input format impacts the quality of outcome. In this study, we investigate the impact of the degree of data structure on the perceived quality of information from the novel perspective of data creators. We also propose and evaluate a novel moderating effect due to the nature of content online users wish to share. The preliminary findings support our claims of the importance of these factors for information quality. We conclude the paper with directions for future research and expected contributions for theory and practice.

show abstract

Section: Methodsmentioning

confidence: 99%

Data Collection Interfaces in Online Communities: The Impact of Data Structuredness and Nature of Shared Content on Perceived Information Quality

Maddah

Lukyanenko

VanderMeer

et al. 2020

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

show abstract

“…Recruiting online ad-hoc crowd labour may be a way to ameliorate these issues, by exploiting a large potential user pool and providing lower labour rates with extended incentives to the end users. Crowdsourced usability testing has demonstrated its capability for detecting usability problems as good as the testing done by 'experts' [154]. However, Liu et al [155] showed that the quality of crowdsourced usability testing was worse than that of the face-to-face usability testing in a laboratory setting.…”

Section: Crowdsourcing For Software Testingmentioning

confidence: 99%

“…Schneider and Cheung [154] first demonstrated the viability of employing on-demand crowd users for usability testing. They also proposed methods to help observe the testers during the process.…”

Section: Crowdsourcing For Software Testingmentioning

confidence: 99%

“…It has been applied to usability testing [30], [154]- [159], performance testing [160], GUI testing [87], [161], test case generation [166], [167], and the oracle problem [88]. We discuss each of these below: 1) Usability Testing: Traditional usability testing is labour-intensive and can be expensive and time consuming [155].…”

Section: Crowdsourcing For Software Testingmentioning

confidence: 99%

See 1 more Smart Citation

A survey of the use of crowdsourcing in software engineering

Mao

Capra

Harman

et al. 2017

Journal of Systems and Software

304

231

View full text Add to dashboard Cite

Crowdsourcing can be used to support software engineering activities and research into these activities. In this paper we provide a comprehensive survey of the use of crowdsourcing to support software engineering activities (Crowdsourced Software Engineering), seeking to cover all literature on this topic. We describe the software engineering domains, tasks and applications for crowdsourcing and the platforms and stakeholders involved in realising Crowdsourced Software Engineering solutions. We also expose trends, issues and opportunities for Crowdsourced Software Engineering.Please cite as: Ke Mao, Licia Capra, Mark Harman and Yue Jia. A Survey of the Use of Crowdsourcing in SoftwareEngineering.

show abstract

“…There are many research papers that explore the use of crowdsourced testing platforms in the various types of testing such as usability testing [4], performance testing [5], GUI testing [6], etc.…”

Section: Introductionmentioning

confidence: 99%

Process Improvements for Crowdsourced Software Testing

Alyahya¹,

Alrugebh²

2017

ijacsa

View full text Add to dashboard Cite

Crowdsourced software testing has been a common practice lately. It refers to the use of crowdsourcing in software testing activities. Although the crowd testing is a collaborative process by nature, there is no available research that provides a critical assessment of the key collaboration activities offered by the current crowdsourced testing platforms. In this paper, we review the process used in the crowd testing platforms including identifying the workflow used in managing the crowd testing process starting from submitting testing requirements and ending with reviewing testing report. Understanding the current process is then utilized to identify a set of limitations in the current process and has led to propose three process improvements (improving assigning crowd manager, improving building test team, monitoring testing progress). We have designed and implemented these process improvements and then evaluated them using two techniques: 1) questionnaire and 2) workshop. The questionnaire shows that the process improvements are significantly sound and strong enough to be added to crowd testing platforms. In addition, the evaluation through conducting a workshop was useful to assess the design and implementation of the process improvements. The participants were satisfied with them but asked for further modifications. Nevertheless, because crowd testing requires participation from a large number of people, the automation suggested improving managing the current process which was highly appreciated.

show abstract

The Power of the Crowd: Performing Usability Testing Using an On-Demand Workforce

Cited by 12 publications

References 13 publications

Data Collection Interfaces in Online Communities: The Impact of Data Structuredness and Nature of Shared Content on Perceived Information Quality

Data Collection Interfaces in Online Communities: The Impact of Data Structuredness and Nature of Shared Content on Perceived Information Quality

A survey of the use of crowdsourcing in software engineering

Process Improvements for Crowdsourced Software Testing

Contact Info

Product

Resources

About