Is the Social Desirability Effect in Human–Robot Interaction overestimated? A Conceptual Replication Study Indicates Less Robust Effects

Abstract

The “Computers are social actors” (CASA) assumption (Nass and Moon in J Soc Issues 56:81–103, 2000. https://doi.org/10.1111/0022-4537.00153) states that humans apply social norms and expectations to technical devices. One such norm is to distort one’s own response in a socially desirable direction during interviews. However, findings for such an effect are mixed in the literature. Therefore, a new study on the effect of social desirability bias in human–robot evaluation was conducted, aiming for a conceptual replication of previous findings. In a between-subject laboratory experiment, $$N = 107$$ N = 107 participants had to evaluate the robot and the interaction quality after a short conversation in three different groups: In one group, the evaluation was conducted using (1) the same robot of the former interaction, (2) a different robot, (3) a tablet computer. According to the CASA assumption, it was expected, that evaluations on likability and quality of interaction, are higher in the condition with the same robot conducting the evaluation, compared to a different robot or a tablet computer because robots are treated as social actors and hence humans distort ratings in a socially desirable direction. Based on previous findings, we expected robots to evoke higher anthropomorphism and feelings of social presence compared to the tablet computer as potential explanation. However, the data did not support the hypotheses. Low sample size, low statistical power, lack of measurement validation and other problems that could lead to an overestimation of effect sizes—in this study and the literature in general—are discussed in light of the replicability crisis.

Measuring the impact and reach of informal educational videos on YouTube: The case of Scientific Animations Without Borders

“…These situational aspects include but are not limited to an assured anonymity of response, decreased or no fear of reprisals for answering accurately, and decreased pressure from social-desirability bias ( Durmaz et al., 2020 ; Short et al., 2009 ), i.e., the tendency of respondents “to stretch the truth in an effort to make a good impression” ( Martin and Nagao, 1989 , p. 72). Moreover, although it is intuitive that computer-mediated solicitation of user-response data could decrease social desirability bias, the results are actually mixed ( Booth-Kewley et al., 1992 ; Lautenschlager and Flaherty, 1990 ; Leichtmann and Nitsch, 2020 ). Lautenschlager and Flaherty (1990) found that social desirability bias increased when participants used computers to self-report compared to pen and paper, but Booth-Kewley et al.…”

Speed Dating with Voice User Interfaces: Understanding How Families Interact and Perceive Voice User Interfaces in a Group Setting

“…The studies presented in this paper represent an iterative study processes that promotes conceptual replication. Lack of replication is a critical area to address in HRI studies ( Irfan et al, 2018 ; Hoffman and Zhao, 2020 ; Leichtmann and Nitsch, 2020b ; Ullman et al, 2021 ). Through conceptual replication, this paper demonstrates how consistent the results are for VUIs with higher social embodiment.…”

“…In recent years, conversations around replicability in HRI have grown as researchers increasingly seek to understand if there is a replication crisis in HRI ( Irfan et al, 2018 ; Belhassein et al, 2019 ; Hoffman and Zhao, 2020 ; Leichtmann and Nitsch, 2020b ; Ullman et al, 2021 ). It is critical for the HRI community to address this area as replication crisis can create fundamental problems of trust that can “cast serious doubt on previous research and undermine the public’s trust in research studies in general” ( Hoffman and Zhao, 2020 ).…”

“…While replication issues have been well documented in fields such as psychology, medicine, and neuroscience, very few works in HRI have systematically examined if the same replication issues exist in HRI ( Irfan et al, 2018 ; Hoffman and Zhao, 2020 ). HRI is rampant with low sample sizes resulting in low statistical power, lack of systematic methodological training, and lack of theory around proxemics studies, suggesting that replicability is a key issue to be addressed by the HRI community ( Hoffman and Zhao, 2020 ; Leichtmann and Nitsch, 2020a ; Leichtmann and Nitsch, 2020b ; Ullman et al, 2021 ). It is rare to see replications in HRI largely due to the fact that a wide variety of robots are used and those robots are often proprietary, limited, and/or expensive ( Baxter et al, 2016 ; Ullman et al, 2021 ).…”

“…These replication studies can reproduce previous findings as well as reveal novel insights; both of which are valuable outcomes from replication studies ( Ullman et al, 2021 ). In our work, the three studies are not direct replications, however, we follow a similar goal as Leichtmann and Nitsch (2020b) replicating the basic idea and format of the study through a conceptual replication ( Ward and Kemp, 2019 ): A speed dating interaction with small groups engage with three VUIs of various social embodiments followed by three activities.…”