Intermediated Reality: A Framework for Communication Through Tele-Puppetry

Abstract

We introduce Intermediated Reality (IR), a framework for intermediated communication enabling collaboration through remote possession of entities (e.g., toys) that come to life in mobile Mediated Reality (MR). As part of a two-way conversation, each person communicates through a toy figurine that is remotely located in front of the other participant. Each person's face is tracked through the front camera of their mobile devices and the tracking pose information is transmitted to the remote participant's device along with the synchronized captured voice audio, allowing a turn-based interactive avatar chat session, which we have called ToyMeet. By altering the camera video feed with a reconstructed appearance of the object in a deformed pose, we perform the illusion of movement in real-world objects to realize collaborative tele-present augmented reality (AR). In this turn based interaction, each participant first sees their own captured puppetry message locally with their device's front facing camera. Next, they receive a view of their counterpart's captured response locally (in AR) with seamless visual deformation of their local 3D toy seen through their device's rear facing camera. We detail optimization of the animation transmission and switching between devices with minimized latency for coherent smooth chat interaction. An evaluation of rendering performance and system latency is included. As an additional demonstration of our framework, we generate facial animation frames for 3D printed stop motion in collaborative mixed reality. This allows a reduction in printing costs since the in-between frames of key poses can be generated digitally with shared remote review.

Intermediated Reality with an AI 3D Printed Character

“…Intermediated Reality [Casas and Mitchell 2019] is a term used to describe a Mixed Reality experience that combines elements of both Augmented Reality (AR) and Mediated Reality [Mann 1999] with the outcome of providing telepresence via remote physical objects. This technique creates a seamless integration of virtual content into the real-world by enabling real-time interaction and manipulation of virtual objects within a physical environment (see figure 1).…”

“…The Augmented Reality live AI character interaction showcase is the result of a groundbreaking technology that employs a photorealistic Intermediated Reality [Casas and Mitchell 2019] technique. Morphing optically captured real-time lit appearance allows us to animate live facial performance from generative AI characters (e.g.…”

DPWP - Uma nova Abordagem para o Escalonamento Dinâmico em Computação Paralela Virtual

“…al., 1995), Dynamic PVM (Overeinder et. al., 1996) e UPVM (Casas et. al., 1994) dentre muitos outros.…”