Student Evaluations of a (Rude) Spoken Dialogue System Insights from an Experimental Study

Abstract

Communicating with spoken dialogue systems (SDS) such as Apple’s Siri® and Google’s Now is becoming more and more common. We report a study that manipulates an SDS’s word use with regard to politeness. In an experiment, 58 young adults evaluated the spoken messages of our self-developed SDS as it replied to typical questions posed by university freshmen. The answers were either formulated politely or rudely. Dependent measures were both holistic measures of how students perceived the SDS as well as detailed evaluations of each single answer. Results show that participants not only evaluated the content of rude answers as being less appropriate and less pleasant than the polite answers, but also evaluated the rude system as less accurate. Lack of politeness also impacted aspects of the perceived trustworthiness of the SDS. We conclude that users of SDS expect such systems to be polite, and we then discuss some practical implications for designing SDS.

Impact of politeness and performance quality of android robots on future interaction decisions: a conversational design perspective

“…The finding that politeness was less of a concern in expert situations than in general service situations (Experiment 1 in this study, Inbar and Meyer, 2019;Rea et al, 2021) indicates that the robot design of politeness should be more effective in general service situations than in expert situations. As studies proposing personalize agents' politeness levels according to each participant (Firdaus et al, 2022;Smith et al, 2022), participants' demographic differences (e.g., age and gender) may affect politeness impressions (Jucks et al, 2018;Kumar et al, 2022;Miyamoto et al, 2019;Rana et al, 2021;Zhu andKaber, 2012, but not observed in Inbar andMeyer, 2015;Inbar and Meyer, 2019;Lee et al, 2019). As this study did not aim to investigate demographic differences, we did not control the numbers of male and female participants and participants' ages, but conducted a posteriori analysis regarding gender differences.…”

“…As studies proposing personalize agents’ politeness levels according to each participant ( Firdaus et al, 2022 ; Smith et al, 2022 ), participants’ demographic differences (e.g., age and gender) may affect politeness impressions ( Jucks et al, 2018 ; Kumar et al, 2022 ; Miyamoto et al, 2019 ; Rana et al, 2021 ; Zhu and Kaber, 2012 , but not observed in Inbar and Meyer, 2015 ; Inbar and Meyer, 2019 ; Lee et al, 2019 ). As this study did not aim to investigate demographic differences, we did not control the numbers of male and female participants and participants’ ages, but conducted a posteriori analysis regarding gender differences.…”

“…It refers to the extent to which service agents are perceived as considerate and trustworthy ( Brown and Swartz, 1989 ) and, specifically, has a significant impact on initial trust ( Coulter and Coulter, 2002 ). Humans as well as robots and chatbots have pointed out that politeness is comforting, and studies have investigated effective ways to use politeness (e.g., Castro-González et al, 2016 ; Jucks et al, 2018 ; Kaiser et al, 2019 ; Lee et al, 2019 ; Miyamoto et al, 2019 ; Smith et al, 2022 ). Particularly in human robot collaboration, the politest robots receive higher evaluations on enjoyment, satisfaction, and trustworthiness, regardless of participants’ age ( Kumar et al, 2022 ).…”

Um Agente Pedagógico Gentil é Mais Efetivo? Efeito das Atitudes de Agentes Pedagógicos Animados na Aprendizagem, Engajamento, Emoções e Ansiedade dos Estudantes

“…Os resultados indicaram que os alunos do grupo da agente amigável apresentaram um maior engajamento e ganho de aprendizado, confirmando as avaliações de outros trabalhos daárea [Jucks et al 2018, Guo and Goh 2015, McLaren et al 2011. Também houve uma maior intensidade de emoções positivas nos alunos que utilizaram as agentes amigável e direta.…”

“…Existem uma série de trabalhos que buscaram verificar a eficiência de APAs com atitudes empática, polida, rude e com rapport 1 no aumento da aprendizagem e engajamento dos alunos [Jucks et al 2018, Hastie et al 2016, Krämer et al 2016, e alguns estudos também analisaram o impacto desses agentes nos estados afetivos dos alunos [Moridis andEconomides 2012, Wang et al 2010]. Entretanto, esses trabalhos não compararam os efeitos de agentes pedagógicos animados com atitudes opostas (como a educação e a rudeza, por exemplo) e neutras (sem emoções), e não consideraram o estado de ansiedade dos estudantes.…”