A case study of post-deployment user feedback triage

Abstract

Many software requirements are identified only after a product is deployed, once users have had a chance to try the software and provide feedback. Unfortunately, addressing such feedback is not always straightforward, even when a team is fully invested in usercentered design. To investigate what constrains a teams evolution decisions, we performed a 6-month field study of a team employing iterative user-centered design methods to the design, deployment and evolution of a web application for a university community. Across interviews with the team, analyses of their bug reports, and further interviews with both users and non-adopters of the application, we found most of the constraints on addressing user feedback emerged from conflicts between users heterogeneous use of information and inflexible assumptions in the team's software architecture derived from earlier user research. These findings highlight the need for new approaches to expressing and validating assumptions from user research as software evolves.

User involvement in software evolution practice: A case study

“…Other studies (e.g. [7], [9], [18]) support our findings, what makes us confident that the identified problems are real. Ensuring construct validity for empirical studies of software developers is always a complex task, specifically as such studies typically require the researcher to abstract from observed behavior or gathered information.…”

“…Ko et al [9] found two main factors which constrain evolution decisions in development teams. First, developers were more likely to address feedback they believed to be shared by the majority of the users.…”

“…The consequence of these changes is an increasing distance between developers and users [6], while more focus on users would actually be necessary to satisfy their increasing demands [11]. As a consequence, postdeployment user feedback such as feature requests and bug reports become increasingly important to developers [9].…”

Continuous deployment of software intensive products and services: A systematic mapping study

“…MacCormack [51] suggests that a valuable avenue for identifying lead (beta) customers is through exploring the company's customer-support database. Ko et al [45] also warn about the importance of selecting a sample of customers that is representative of the user community and discusses challenges when using a vocal minority of existing users.…”

Adaptive software-based Feedback Acquisition: A Persona-based design

“…From a Requirement Engineering perspective, users' involvement via their feedback while they are using the software in practice is more credible to assess how the software is playing its role in meeting their requirements in practice [1,38,3]. In fact, many of the users' requirements are only identified after the software is being deployed and once users get the chance to use it in a real context [4]. This becomes even more evident when we consider the requirements which emerge because of the existence of competitive technology and peer pressure.…”