Crisis Ahead? Why Human-Robot Interaction User Studies May Have Replicability Problems and Directions for Improvement

Abstract

There is a confidence crisis in many scientific disciplines, in particular disciplines researching human behavior, as many effects of original experiments have not been replicated successfully in large-scale replication studies. While human-robot interaction (HRI) is an interdisciplinary research field, the study of human behavior, cognition and emotion in HRI plays also a vital part. Are HRI user studies facing the same problems as other fields and if so, what can be done to overcome them? In this article, we first give a short overview of the replicability crisis in behavioral sciences and its causes. In a second step, we estimate the replicability of HRI user studies mainly 1) by structural comparison of HRI research processes and practices with those of other disciplines with replicability issues, 2) by systematically reviewing meta-analyses of HRI user studies to identify parameters that are known to affect replicability, and 3) by summarizing first replication studies in HRI as direct evidence. Our findings suggest that HRI user studies often exhibit the same problems that caused the replicability crisis in many behavioral sciences, such as small sample sizes, lack of theory, or missing information in reported data. In order to improve the stability of future HRI research, we propose some statistical, methodological and social reforms. This article aims to provide a basis for further discussion and a potential outline for improvements in the field.

Development and Validation of a Basic Psychological Needs Scale for Technology Use

“…As Cronbach (1971) notes, construct validation is an ever extending and thus never-ending process. This article describes a successful start of such a validation process, of which there are still far too few in the literature on HCI (Leichtmann et al, 2022(Leichtmann et al, , 2023. It is thus a first step toward achieving more reliable measurements and thus more robust results in user studies.…”

Interdisciplinary views of fNIRS: Current advancements, equity challenges, and an agenda for future needs of a diverse fNIRS research community

“…Robots show great promise in helping those varying socioeconomic status, in settings like schools and hospitals. However, many HRI studies study interactions with typically white and young samples, often sourced from universities, which are often not representative of the desired population (Leichtmann et al, 2022). A narrow sample may allow researchers to miss how cultural and racial differences impact HRIs.…”

Explainable Artificial Intelligence improves human decision-making: Results from a mushroom picking experiment at a public art festival

“…While we already tested the effects of XAI methods with a mushroom picking task (Leichtmann, Humer, et al, 2022), the replication of the results is needed. The value of replication studies was recently highlighted in discussions on the so-called replicability crisis in behavioral sciences (e.g., Leichtmann, Nitsch, et al, 2022;Pashler & Wagenmakers, 2012;Romero, 2019;Świątkowski & Dompnier, 2017;Wiggins & XAI AT PUBLIC ART EXHIBITION 9 Christopherson, 2019). In the subsequent reform movement it was emphasized that replications test the robustness of effects, verify ideas and theories, control for sampling errors, control for fraud, and control for artifacts (Schmidt, 2009).…”

“…Historically younger approaches in psychology (e.g., feminist psychology, Eagly and Riger, 2014;critical psychology, Teo, 2015; or performative perspectives, Derksen and Morawski, 2022) therefore emphasize that (i) scientists themselves are part of social systems with certain social values, (ii) psychological research situations are special social situations (e.g., through unequal power relations between investigator and subject), and (iii) psychological research is not a mere observation of existing phenomena, but empirical studies themselves have effects on participants and can thus create new phenomena. This reflection on the transformative power of (psychological) studies can also be understood as a new responsibility of researchers (Leichtmann, Nitsch, et al, 2022).…”