What’s (Not) Working in Programmer User Studies?

Abstract

A key goal of software engineering research is to improve the environments, tools, languages, and techniques programmers use to efficiently create quality software. Successfully designing these tools and demonstrating their effectiveness involves engaging with tool users — software engineers. Researchers often want to conduct user studies of software engineers to collect direct evidence. However, running user studies can be difficult, and researchers may lack solution strategies to overcome the barriers, so they may avoid user studies. To understand the challenges researchers face when conducting programmer user studies, we interviewed 26 researchers. Based on the analysis of interview data we contribute: (i) a taxonomy of 18 barriers researchers encounter; (ii) 23 solution strategies some researchers use to address 8 of the 18 barriers in their own studies; and (iii) 4 design ideas, which we adapted from the behavioral science community, that may lower 8 additional barriers. To validate the design ideas, we held an in-person all-day focus group with 16 researchers.

A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations

“…Echoing others [e.g., 85,155], Ledo et al [125] reiterate that evaluation methods for a specific toolkit should be determined in light of the claims researchers wish to make about the toolkit. 5 As we explore further in Sections 5.2 and 5.3, we agree with the call to align evaluation methods with research claims, and draw on HCI guidelines for evaluation design that have been developed in response to this call [e.g., 53,146] to inform an RAI tool effectiveness evaluation framework. As RAI tools aim to address ethical issues in AI development, claims related to the consequences of their use or effectiveness are wide in their sociopolitical scope, implicating not only RAI tool developers and users, but also AI system stakeholders and communities who may be affected by AI system deployment.…”

“…The objective of an effectiveness evaluation framework is to support evaluators of RAI tools to design robust evaluations, commensurate to similar efforts in the HCI and software engineering fields to develop frameworks for usability evaluations [e.g., 53,118], evaluations of ML models [103], and the design of empirical studies in software engineering [e.g., 116]. Below we provide initial design desiderata for the development of a framework that meets this objective.…”

“…In HCI, for instance, the "RepliCHI" special interest group was convened in response to the replicability crisis in psychology and related fields [106], as a way to foster replication studies (and to focus on replicability as a desirable aspect of HCI research) [215,216], while the 2023 CHI conference saw the first Special Interest Group on "Human-Centered Responsible AI" [192]. Additionally, HCI researchers have proposed addressing participant recruitment and design replicability challenges through the development of shared evaluation platforms, which might include reusable evaluation design guides and participant recruitment pools [53].…”