Multimodal Corpus Analysis as a Method for Ensuring Cultural Usability of Embodied Conversational Agents

Abstract

Abstract. In this paper we propose the method of multimodal corpus analysis to collect enough empirical data for modeling the behavior of embodied conversational agents. This is a prerequisite to ensure the usability of such complex interactive systems. So far, the development of embodied agents suffers from a lack of explicit usability methods. In most cases, the consideration of usability aspects is constrained to preliminary user tests at the end of the development process.

Some Pitfalls for Developing Enculturated Conversational Agents

“…The scheduling stage at last is necessary to ensure appropriate timing in turn-taking of the interlocutors, which again is culturespecific. For instance in the above mentioned study on German and Japanese behavior [22] we found that German interlocutors are generally uncomfortable with longer pauses in conversations compared to the Japanese samples.…”

“…One way to deal with this problem is to gather data in a standardized way, tailored to the modeling endeavor. In [22] in this volume, we describe such an approach for the German and Japanese culture.…”

Cross-cultural assessment of automatically generated multimodal referring expressions in a virtual world

“…The work by Jan et al (2006), for instance, models a set of cultural parameters that define an agent's behaviour according to reviewed literature and test the realism of the output on people from different cultural backgrounds. In the Cube-G project (Nakano and Rehm, 2009), a corpus elicited from lab studies in which participants are interacting with professional actors is used to train a system to mimic human communicative behaviour in a number of social settings. Yet another approach models ECAs based on observations of people in their natural environment and tests the result in comparable situations Huang et al, 2009).…”

“…In the Japanese presentations, the size of the nondeictic gestures was adapted to occupy less space (Nakano and Rehm, 2009) and the pointing gestures were made using the whole hand instead of with just the index finger. The virtual furniture shop that was built for the study displayed the furniture in a relatively large space, which made it easy for the agents to move around.…”

Empirical Multimodality Research: The State of Play

“…Nevertheless, as research objects continue to diversify, the interest of neighboring disciplines increases, and the field shows signs of becoming a stand-alone discipline in its own right (see Wildfeuer et al 2019), the need for solid empirical grounding is also becoming ever more apparent. In fact, diverse scholars in multimodality have been pointing for some time, and often quite independently of one another, to the usefulness of 'large n' empirical investigations (see, e.g., Stöckl 1997;Bateman et al 2004;Gu 2006;Carter & Adolphs 2008;Nakano & Rehm 2009;Bednarek 2015;Hiippala 2015;Pederson & Cohn 2016;Bezemer & Cowan 2021). Furthermore, scholars are increasingly stressing the shortcomings of exclusively broad-brushed orientations (e.g., Jewitt 2017;Kohrs 2018).…”