Rushing to Judgement: How do Laypeople Rate Caller Engagement in Thin-Slice Videos of Human–Machine Dialog?

Abstract

We analyze the efficacy of a small crowd of naïve human raters in rating engagement during human-machine dialog interactions. Each rater viewed multiple 10 second, thin-slice videos of non-native English speakers interacting with a computerassisted language learning (CALL) system and rated how engaged and disengaged those callers were while interacting with the automated agent. We observe how the crowd's ratings compared to callers' self ratings of engagement, and further study how the distribution of these rating assignments vary as a function of whether the automated system or the caller was speaking. Finally, we discuss the potential applications and pitfalls of such a crowdsourced paradigm in designing, developing and analyzing engagement-aware dialog systems.

Engagement Recognition in Spoken Dialogue via Neural Network by Aggregating Different Annotators' Models

“…However, we observed some bias where the subjects tend to give positive evaluations on themselves. This kind of bias was also observed in other works [33,34]. The second method is to ask the operator to evaluate the subject engagement.…”

Engagement recognition by a latent character model based on multimodal listener behaviors in spoken dialogue

“…Furthermore, we sometimes observe a bias where subjects tend to give positive evaluations of themselves. This kind of bias was observed in other works [50,51]. Another method is to ask the ERICA's operators to evaluate the subject engagement.…”