Timing in Multimodal Turn-Taking Interactions: Control and Analysis Using Timed Petri Nets

Abstract

Turn-taking interactions with humans are multimodal and reciprocal in nature. In addition, the timing of actions is of great importance, as it influences both social and task strategies. To enable the precise control and analysis of timed discrete events for a robot, we develop a system for multimodal collaboration based on a timed Petri net (TPN) representation. We also argue for action interruptions in reciprocal interaction and describe its implementation within our system. Using the system, our autonomously operating humanoid robot Simon collaborates with humans through both speech and physical action to solve the Towers of Hanoi, during which the human and the robot take turns manipulating objects in a shared physical workspace. We hypothesize that action interruptions have a positive impact on turn-taking and evaluate this in the Towers of Hanoi domain through two experimental methods. One is a between-groups user study with 16 participants. The other is a simulation experiment using 200 simulated users of varying speed, initiative, compliance, and correctness. In these experiments, action interruptions are either present or absent in the system. Our collective results show that action interruptions lead to increased task efficiency through increased user initiative, improved interaction balance, and higher sense of fluency. In arriving at these results, we demonstrate how these evaluation methods can be highly complementary in the analysis of interaction dynamics.

Four-participant group conversation: A facilitation robot controlling engagement density as the fourth participant

“…In such situations, conversational context, including engagement (Sidner et al, 2004) and turn-taking (Sacks et al, 1995), is commonly grounded between two interlocutors. Many dialogue systems have dealt with turn-taking within two-participant engagement (Raux and Eskenazi, 2009;Chao and Thomaz, 2012). However, in three-participant conversations as shown in Fig.…”

Timing multimodal turn-taking for human-robot cooperation

“…In addition, I presented a novel system evaluation technique using TPN simulations in [6], in which the user is modeled as a TPN graph connected to the robot's TPN controller. The user graph generates inputs according to parameters of the task that are hypothesized to be important.…”

“…An example from my previous research is an interruptible communicative action template (Figure 2) [6]. Interactions can be more fluent when communicative actions are interruptible.…”

“…A timed Petri net (TPN) includes management of timing by placing a stochastic timer in each place; transitions do not fire until the timer runs down to 0. A more detailed specification of my TPN system can be found in [6]. Petri nets without timing have also been used for realizing Behavior Markup Language (BML) on robots [10].…”

Nonstrict Hierarchical Reinforcement Learning for Interactive Systems and Robots

“…For other kinds of joint optimizations, see [Dethlefs et al 2012b;Dethlefs and Cuayáhuitl 2011b;2011a;Lemon 2011]. (5) Optimize dialogue control by adding linguistically-motivated phenomena such as multimodal turn-taking [Chao and Thomaz 2012; and incremental processing [Schlangen and Skantze 2009;Dethlefs et al 2012a;. These features have shown to be important contributors to improve performance and enhance naturalness in the interactions.…”