Real-time 3D video communication in 3D virtual worlds: Technical realization of a new communication concept

Abstract

After a failed initial attempt in the 1950s to bring three dimensionality into the entertainment sector, 3D technologies today have reached a point where media production, transportation, as well as consumption are of sufficient speed and quality to finally advance the complete media chain from two dimensional (2D) to three dimensional information. Production is based either on stereo-or multi-scopic video capturing techniques or on synthetic environments that can mimic or adopt real-life behavior. The resulting 3D media can be transported by providers over the existing telecommunication infrastructure to be consumed by users using single-or multi-view 3D devices such as TVs, goggles, or projectors. The Visual Communication Department in Bell Labs' Application Research Domain has taken on the research challenge to replace a user's avatar in a 3D virtual environment with a real time representation of the user derived from his streaming 3D video image. With this, the tedious process of avatar creation becomes obsolete as people are able to gesture naturally and convey facial expressions within these synthetic worlds. This paper highlights the technical challenges around building and implementing an architecture for innovative and immersive communication systems. We also discuss the technical limitations of the current approach, and pinpoint major open issues. © 2011 Alcatel-Lucent.with a mission. Immersive online multi-user social applications such as SimCity* and Second Life* build on this trend by providing people with the opportunity to create and expand a social network in a simulated adaptable environment. Both the standard video communication applications as well as the immersive multi-user social applications have specific advantages and disadvantages [7] that will ultimately lock them into a specific area of use for a specific market segment. IntroductionThanks to applications like Skype*, Yahoo!* and instant messaging (IM), people are becoming used to using webcams for one-to-one video communication with relatives and friends. These video communication applications typically pop up a window in which the local user can see an image captured by the webcam at the remote side while his local webcam will convey his image to the remote user. At the same time, users are also familiar with immersive multiuser games in which they are represented by avatars The research goal of the Bell Labs Visual Communication Department is to develop and implement applications for immersive visual communication between multiple (residential as well as professional) users by prototyping innovative ideas. Within the department's Mixed Reality activity, team members study the use of real-time three-dimensional (3D) video images of users embedded in immersive online multi-user social applications, more specifically virtual environments [16].Ongoing research [6,8] in this area concentrates on improving an avatar's behavior so as to mimic that of the real user as much as possible. The research is conducted using synthetic models that c...