The 19 Unifying Questionnaire Constructs of Artificial Social Agents

Abstract

In this paper, we report on the multi-year Intelligent Virtual Agents (IVA) community effort, involving more than 80 researchers worldwide, researching the IVA community interests and practises in evaluating human interaction with an artificial social agent (ASA). The effort is driven by previous IVA workshops and plenary IVA discussions related to the methodological crisis on the evaluation of ASAs. A previous literature review showed a continuous practise of creating new questionnaires instead of reusing validated questionnaires. We address this issue by examining questionnaire measurement constructs used in empirical studies between 2013 to 2018 published in the IVA conference. We identified 189 constructs used in 89 questionnaires that are reported across 81 studies. Although these constructs have different names, they often measure the same thing. In this paper, we, therefore, present a unifying set of 19 constructs that captures more than 80% of the 189 constructs initially identified. We established this set in two steps. First, 49 researchers classified the constructs in broad theoretically based categories. Next, 23 researchers grouped the constructs in each category on their similarity. The resulting 19 groups form a unifying set of constructs, which will be the basis for the future questionnaire instrument of human-ASA interaction. CCS CONCEPTS • Human-centered computing → Empirical studies in HCI; • Computing methodologies → Intelligent agents;

Psychology in Virtual Reality: Toward a Validated Measure of Social Presence

“…the research initiative, similar to Fitrianie et al ( 2020 ) to approach the problem of terminology;…”

“…Another example which could help solving the terminological confusion in the field of social presence is described in the initiative by Fitrianie et al ( 2020 ). In the mentioned initiative, independent researchers in the field of artificial virtual agents, were asked to sort semantically overlapping concepts into a reduced set of groups.…”

“…Roughly counting, there are over 40 different questionnaires mentioned. Intelligent Virtual Agents (IVA) researchers recently noted that there is a continuous practice in the community of creating new questionnaires instead of reusing existing ones (Fitrianie et al, 2020 , p. 1). Moreover, only a minority of the social presence questionnaires which are in use were properly psychometrically validated (rare exceptions, e.g., Harms and Biocca, 2004 ; Poeschl and Doering, 2015 ) and usually the tools are constructed ad-hoc for particular research purposes (Table 2, Oh et al, 2018 ).…”

eXtended Artificial Intelligence: New Prospects of Human-AI Interaction Research

“…Both desiderates hints to the significance of the right sight of our continuum. Further, Fitrianie et al (2020) pointed to a methodological crisis on the evaluation of artificial social agents (Fitrianie et al, 2020). They discuss that most studies use different approaches to investigate human-agent interactions, resulting in a lack of comparability and replicability.…”

Tailoring coaching conversations with virtual health coaches

“…Within the agent community there is an ongoing initiative to come to questionnaire items for the evaluation of artificial social agents by the 'Artificial Social Agent Evaluation Instrument' workgroup. This workgroup showed the wide variance of evaluation metrics used in agent studies in a literature survey (Fitrianie et al, 2019) (189 constructs in 89 questionnaires for 81 studies), from which they created a set of 19 unifying constructs (Fitrianie et al, 2020), and recently presented the first results of creating questionnaire items (Fitrianie et al, 2021) (131 expert-generated and content-validated items).…”