Scalable Design Evaluation for Everyone! Designing Configuration Systems for Crowd-Feedback Request Generation

Abstract

Design evaluation is an important step during software development to ensure users' requirements are met. Crowd feedback represents an effective approach to tackling scalability issues of traditional design evaluation methods. Crowd-feedback systems are usually developed for a fixed use case and designers lack knowledge on how to build individual crowd-feedback systems by themselves. Consequently, they are rarely applied in practice. To address this challenge, we propose the design of a configuration system to support designers in creating individual crowd-feedback requests. By conducting expert interviews (N=14) and an exploratory literature review, we derive four design rationales for such configuration systems and propose a prototypical configuration system instantiation. We evaluate this instantiation in exploratory focus groups (N=10). The results show that feedback requesters appreciate guidance. However, there seems to be a trade-off between complexity and flexibility. With our research, we contribute with a generalizable concept to support feedback requesters to create individualized crowd-feedback requests to support scalable design evaluation for everyone. CCS CONCEPTS• Human-centered computing → HCI design and evaluation methods.