Review of Human Studies Methods in HRI and Recommendations

Abstract

This article provides an overview on planning, designing, and executing human studies for Human-Robot Interaction (HRI) that leads to ten recommendations for experimental design and study execution. Two improvements are described, using insights from the psychology and social science disciplines. First is to use large sample sizes to better represent the populations being investigated to have a higher probability of obtaining statistically significant results. Second is the application of three or more methods of evaluation to have reliable and accurate results, and convergent validity. Five primary methods of evaluation exist: selfassessments, behavioral observations, psychophysiological measures, interviews, and task performance metrics. The article describes specific tools and procedures for operationalizing these improvements, as well as suggestions for recruiting participants. A recent large-scale, complex, controlled human study in HRI using 128 participants and four methods of evaluation is presented to illustrate planning, design, and execution choices.

Improving Speech Recognition with the Robot Interaction Language

“…In our opinion, collecting rich and contextual data via psychological methodologies is the way to go with regard to studying the state-of-art human-robot interaction. Empirical and experimental research is gaining popularity and ascendancy as far as the evaluation of human-robot interaction is concerned 37 ; however, there are not many large-scale evaluations to report when it comes to studying speech-enabled robotic systems. In summary, the focus of HRI evaluations has not been primarily on the modality of speech.…”

Enhancing Interpretation of Ambiguous Voice Instructions based on the Environment and the User’s Intention for Improved Human-Friendly Robot Navigation

“…The evaluation was carried out with five healthy participants (average and standard deviation of the age of the participant are 25.2 years and 1.7 years, respectively) and they were graduate students in the university. The experiments have been carried out based on the guidelines suggested in [25] for designing, planning and executing human studies for human-robot interactions in order to avoid the subjectivity of the experimental results. The scalar constants δ max and δ min are chosen experimentally as 1.5 and 0.75, respectively, in for achieving the desired characteristics.…”

New instrumentation for human robot interaction assessment based on observational methods

“…Whereas the main research focus of HRI is on the development of specific robotic systems and applications, some methods for the evaluation of interactive behaviour between people and robots have been adopted and/or modified from fields as humancomputer interaction, user experience, and Psychology [4,14,49,50]. These complementary approaches face the central question of studying whether the manner in which people interact with a robot is similar either to interactions between humans and computers or to the interaction among humans.…”

“…Categories related to interaction with objects and interpersonal relationships are of interest for the present work. Autonomy/intervention level [4] Non-Autonomous (High intervention) Autonomous (Low intervention)…”

“…Unfortunately, it is also highlighted that these metrics would differ from one field to another. According to [4], there exist five main evaluation methods for HRI: self assessment, behavioural observation, psychophysiology measures, interviews, and task performance metrics. The most employed techniques for exploring the extent to which social robots elicit some desired behaviour are questionnaires and observational methodology.…”