TogetherVR: A Framework for Photorealistic Shared Media Experiences in 360-Degree VR

Abstract

Virtual Reality (VR) and 360-degree video are reshaping the media landscape, creating a fertile business environment. In 2016 many new 360-degree cameras and VR headsets entered the consumer market. Distribution platforms are being established and new production studios are emerging. VR is a hot topic in research and industry, and many new and exciting interactive VR content and experiences are emerging. The biggest gap we see in these experiences is the lack of social and shared aspects of VR usage, as today's VR applications tend to be an isolated endeavour. In this paper, we present TogetherVR, a web-based framework for the creation and evaluation of social and shared VR experiences in which users can communicate with a high degree of presence and in photo-realistic video quality. We further elaborate on three multiuser VR cases: watching TV together in VR, social collaboration in VR, and social VR conferencing in a mixed reality setting.

Generative RGB-D Face Completion for Head-Mounted Display Removal

“…This concept involves the assembly of a group of people in the same VR environment that supports some form of human-to-human communication and collaboration. In subsequent years, technological advances in VR technology have resulted in ongoing research efforts towards the creation of immersive social VR experiences such as collaborative learning [4,5], entertainment [6], treatment of mental disorders [7,8] and teleconferencing [9][10][11][12].…”

“…However, the HMD worn by the user occludes the upper part of their face and prevents the sensor from capturing it. As a result, during social interaction in a virtual environment such as the aforementioned, it is a challenge for users to estimate gaze direction [28], make eye contact [29], interpret non-verbal information [11,30,31] or to recognize the identity of others. For this reason, we want to reconstruct the occluded region of the user's face in a realistic way, which is referred to as the task of HMD removal.…”

“…This forms a major barrier during face-to-face interactions in shared virtual environments, as the obstruction caused by HMDs makes it impossible to fully observe the wearer's face. This forms a key problem for social VR applications such as teleconferencing [9][10][11] and remote collaboration [4,5,65]. HMD removal describes the task of recovering the missing image information caused by the occlusion of an HMD in a coherent and realistic way.…”

Multi-sensor capture and network processing for virtual reality conferencing

“…In [6] we introduced TogetherVR as a modular platform based on web technologies, that allows both to easily create VR experiences that are social and to consume them with off-the-shelf hardware. The platform included browser screen share functionality to provide flexibility in the type of shared application within the VR room.…”

Watching Videos Together in Social Virtual Reality: An Experimental Study on User's QoE

“…The VE can be a computer-generated 3D scene or a 360-degree natural scene. Each user is represented in the VE as a computergenerated avatar or, in recently proposed systems, with a virtual representation based on live captures [6].…”