Reproducibility in Human-Robot Interaction: Furthering the Science of HRI

Abstract

Purpose of Review To discuss the current state of reproducibility of research in human-robot interaction (HRI), challenges specific to the field, and recommendations for how the community can support reproducibility. Recent Findings As in related fields such as artificial intelligence, robotics, and psychology, improving research reproducibility is key to the maturation of the body of scientific knowledge in the field of HRI. The ACM/IEEE International Conference on Human-Robot Interaction introduced a theme on Reproducibility of HRI to their technical program in 2020 to solicit papers presenting reproductions of prior research or artifacts supporting research reproducibility. Summary This review provides an introduction to the topic of research reproducibility for HRI and describes the state of the art in relation to the HRI 2020 Reproducibility theme. As a highly interdisciplinary field that involves work with technological artifacts, there are unique challenges to reproducibility in HRI. Biases in research evaluation and practice contribute to challenges in supporting reproducibility, and the training of researchers could be changed to encourage research reproduction. The authors propose a number of solutions for addressing these challenges that can serve as guidelines for the HRI community and related fields.

(Hu)man-Like Robots: The Impact of Anthropomorphism and Language on Perceived Robot Gender

“…In our opinion, the fact that two research groups have dedicated their research independently to the same research question further strengthens its relevance. Moreover, if the results point in the same direction, this would support the validity of the results by [ 27 ] and contribute to more reproducibility of HRI research [ 28 ].…”

Human-Robot Relationship: long-term effects on disclosure, perception and well-being

“…Furthermore, this study can be replicated and tested with various populations, clinical and healthy, in order to understand how social robots could be introduced in different care settings and as interventions using speech-based interactions [c.f., 114]. By introducing this novel paradigm in detail here, and documenting results from a rigorous empiriucal study using this paradigm, we aim to provide a tool that we hope will be of use to the HRI research community more broadly, while also assisting with facilitating research rigour and reproducibility [1,8,115,116], as well as the development of data-centric robotic models [c.f., 117,118]. Moreover, we would argue that the online computer-mediated means of human-robot communication used in this experimental design can overcome some of the challenges and barriers that are related to long-term HRI studies in natural ecologically valid settings (such as the costs associated with sending individual robots home for an extended period of time with participants) and suggest alternative means for conducting HRI research in people's natural settings.…”

RISE: an open-source architecture for interdisciplinary and reproducible human–robot interaction research

“…Extensibility and robot independence : One goal of a reproducible architecture in a system for interdisciplinary research is to enable the reproduction, replication, or re-creation of previous HRI work ( Gunes et al, 2022 ). Furthermore, it should reduce the effort of developing new HRI studies.…”

“…In this context, reproducibility is required at different levels. Replication— a mere re-run of a study with only minor parameter variations—and reproduction— a study is carried out in a different laboratory—are necessary to ascertain reliability, while conceptual reproducibility aims at specifying the conditions under which a finding holds true and ascertains generalizable results ( Gunes et al, 2022 ). To achieve conceptual reproducibility as well as replication, theoretical principles and practical resources are needed that allow us to specify what kind of variability can be neglected and which variations are relevant ( Gunes et al, 2022 ).…”

“…To reproduce an HRI study, all software components and their documentations must be available. Despite the current discourse on reproducibility, this rarely ever happens ( Gunes et al, 2022 ). For reproducing whole-interaction studies, Lier et al (2014) proposed the Cognitive Interaction Toolkit (CITK) for the aggregation and discovery of required research artifacts and an automated software build and deployment.…”