Aligning Crowdworker Perspectives and Feedback Outcomes in Crowd-Feedback System Design

Abstract

Leveraging crowdsourcing in software development has received growing attention in research and practice. Crowd feedback offers a scalable and flexible way to evaluate software design solutions and the potential of crowd-feedback systems has been demonstrated in different contexts by existing research studies. However, previous research lacks a deep understanding of the effects of individual design features of crowd-feedback systems on feedback quality and quantity. Additionally, existing studies primarily focused on understanding the requirements of feedback requesters but have not fully explored the qualitative perspectives of crowd-based feedback providers. In this paper, we address these research gaps with two research studies. In study 1, we conducted a feature analysis (N=10) and concluded that from a user perspective, a crowd-feedback system should have five core features (scenario, speech-to-text, markers, categories, and star rating). In the second study, we analyzed the effects of the design features on crowdworkers' perceptions and feedback outcomes (N=210). We learned that offering feedback providers scenarios as the context of use is perceived as most important. Regarding the resulting feedback quality, we discovered that more features are not always better as overwhelming feedback providers might decrease feedback quality. Offering feedback providers categories as inspiration can increase the feedback quantity. With our work, we contribute to research on crowd-feedback systems by aligning crowdworker perspectives and feedback outcomes and thereby making the software evaluation not only more scalable but also more human-centered.

Scalable Design Evaluation for Everyone! Designing Configuration Systems for Crowd-Feedback Request Generation

“…A big advantage compared to simple surveys as they can be created with tools like LimeSurvey or Google Forms is that crowd-feedback systems usually enable the feedback collection while the feedback provider is actually using the system (e.g., [9,30]. Crowd-feedback systems can also apply dedicated features that are not available in online survey tools like markers to pin comments to a specific element [10]. According to Haug and Maedche [11], the core element of crowd-feedback systems are their design characteristics and the selected crowd configuration.…”

“…Many of the studies focused on achieving a high feedback quality or optimizing the resulting designs. Thereby, the applicability of these systems in practice was often neglected [10]. Consequently, all existing crowdfeedback systems in research are fixed to a specific use case or to the evaluation of a specific software system [11].…”

“…Consequently, all existing crowdfeedback systems in research are fixed to a specific use case or to the evaluation of a specific software system [11]. One system that allows the evaluation of interactive designs and combines multiple design characteristics of crowd-feedback systems to understand how they are perceived by users is Feeasy [10]. Feeasy collects feedback via a free text field, categories, and star ratings that are attached to the categories.…”

“…Additionally, it contains markers, context, and recording, in the form of a speech-to-text feature as interactivity cues. Haug et al [10] used Feeasy to understand the effect of the design characteristics of crowd-feedback systems on the resulting feedback quality and quantity. Their results can serve as a basis to better support designers in creating individual crowd-feedback requests and tailoring the design to the respective requirements and context.…”

“…For example, the anonymous crowd is found to be mostly financially incentivized [11]. Also, the features of crowd-feedback systems need to be adapted to the goal of the feedback study and to the perceptions of feedback providers [10]. Supplementary, one must consider that different end-users have different design needs [16].…”