What are We Measuring Anyway?

Abstract

Research into artificial social agents aims at constructing these agents and at establishing an empirically grounded understanding of them, their interaction with humans, and how they can ultimately deliver certain outcomes in areas such as health, entertainment, and education. Key for establishing such understanding is the community's ability to describe and replicate their observations on how users perceive and interact with their agents. In this paper, we address this ability by examining questionnaires and their constructs used in empirical studies reported in the intelligent virtual agent conference proceedings from 2013 to 2018. The literature survey shows the identification of 189 constructs used in 89 questionnaires that were reported across 81 papers. We found unexpectedly little repeated use of questionnaires as the vast majority of questionnaires (more than 76%) were only reported in a single paper. We expect that this finding will motivate joint effort by the IVA community towards creating a unified measurement instrument.

Empathic pedagogical conversational agents: A systematic literature review

“…Our results are broadly consistent with previous research such as the one presented by Fitrianie et al. (2019) on Intelligent Virtual Agents, where there is no reuse of instruments even in the evaluation of empathic PCA, while the trend towards the study of new dimensions is maintained.…”

Mandarin Chinese translation of the Artificial-Social-Agent questionnaire instrument for evaluating human-agent interaction

“…When developing and studying Artificial Social Agents (ASAs), such as virtual agents, chatbots, and social robots, it is apparent to gain research insights beyond a single individual agent. To this end, the research community has developed the ASA questionnaire (Fitrianie et al, 2022) for evaluating human-ASA interaction, which makes it possible to compare results obtained with different agents on different topics deemed relevant by the research community (Fitrianie et al, 2020). To extend the usability, the community also needs translations of the questionnaire into different languages.…”

Conversational Agents in Software Engineering: Survey, Taxonomy and Challenges

“…The use of pre or post quantitative surveys can be used to the same end, with the ability to provide quantitative layouts before and after the session, which can later be used to compute and measure the level of task or goal achievement, e.g., as in the learning success of an educational chatbot for a specific skill or state of knowledge [50]. Finally, Maroengsit et al [80] and Fitrianie et al [33] highlight the use of expert content evaluation surveys to facilitate the annotation process for expert evaluation of the response or even the ranked set of response candidates of a conversational agent, which is used to report prediction quality metrics like top-N response accuracy or task completion measures [102].…”