Multimodal acting in mixed reality interactive storytelling

Abstract

Copyright © 2005 IEEE. This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of Teesside University's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to pubs-permissions@ieee.org.By choosing to view this document, you agree to all provisions of the copyright laws protecting it.This document was downloaded from http://tees.openrepository.com/tees/handle/10149/58295 Please do not use this version for citation purposes.All items in TeesRep are protected by copyright, with all rights reserved, unless otherwise indicated. I nteractive storytelling immerses users in fantasy worlds in which they play parts in evolving narratives that respond to their intervention. Implementing the interactive storytelling concept involves many computing technologies: virtual or mixed reality for creating the artificial world, and artificial intelligence techniques and formalisms for generating the narrative and characters in real time.As a character in the narrative, the user communicates with virtual characters much like an actor communicates with other actors. This requirement introduces a novel context for multimodal communication as well as several technical challenges. Acting involves attitudes and body gestures that are highly significant for both dramatic presentation and communication. At the same time, spoken communication is essential to realistic interactive narratives. This kind of multimodal communication faces several difficulties in terms of real-time performance, coverage, and accuracy.We've developed an experimental system that provides a small-scale but complete integration of multimodal communication in interactive storytelling. It uses a narrative's semantic context to focus multimodal input processing-that is, the system interprets users' acting (the multimodal input) in the mixed reality stage in terms of narrative functions representing users' contributions to the unfolding plot.System overview: The mixed reality installation Figure 1 shows the mixed reality system architecture. The system uses a "magic mirror" paradigm, which we derived from the Transfiction approach.1 In our approach, a video camera captures the user's image in real time, and the Transfiction engine extracts the image from the background and mixes it with a 3D graphic model of a virtual stage, which includes the story's synthetic characters. The system projects the resulting image on a large screen facing the user, who sees his or her image embedded in the virtual stage with the synthetic actors.We based the mixed reality world's graphic component on the Unreal Tournament 2003 game engine (http://www.unrealtournament.com). This engine not only renders graphics and animates characters but, most importantly, contains a sophisticated deve...

Draw your own story: Paper and pencil interactive storytelling

“…Dow et al [5] present an augmented reality version of the desktop-based interactive drama Façade [6]. With a similar approach, Cavazza et al [7] present an interactive storytelling application that captures the user's video image and inserts him/her in a world populated by virtual actors. Users are able to interact with the virtual actors using body gestures and natural language speech.…”

“…Users are able to interact with the virtual actors using body gestures and natural language speech. While Dow et al [5] bring the virtual characters to the real world, Cavazza et al [7] place the user inside the virtual world. Zhou et al [16] explore the use of tangible cubes as interaction interface for mixed reality interactive storytelling.…”

Social Interaction for Interactive Storytelling

“…They cover the subject from traditional GUI interfaces [6][7] to more complex interaction mechanisms, such as speech recognition [8][9] [10], body gestures combined with speech [11][12] [13], hand-drawn sketches [14][15] and physiological inputs [16]. However, few of these works consider multiuser interactions.…”

An implementation of real-time 3D interactive drama

“…In free interfaces, the user interacts with the system via natural language. The free interfaces method uses either speech [Cavazza et al 2003] or typed text [Stern and Mateas 2003] to enable the user to dialog naturally with other characters. But there are two major problems:…”