A Mixed Reality Telepresence System for Collaborative Space Operation

Fairchild, Allen J.; Campion, Simon P.; García, Arturo S.; Wolff, Robin; Fernando, Terrence; Roberts, David

doi:10.1109/tcsvt.2016.2580425

Cited by 57 publications

(28 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The idea here was to reproduce all the communication cues (audio, visual, body expressions, facial expressions and gestures) that we enjoy in face-to-face meetings. The interested reader would be able to find a detailed description of the telepresence aspects of this project (physical setup, algorithms and evaluation) in [11], as the focus of the current paper is on the software architecture supporting the whole system. Figure 5 shows a 3D reconstructed user waving at two collaborators, one local and other remote (represented by a traditional avatar).…”

Section: Communication Viewmentioning

confidence: 99%

Collaborative virtual reality platform for visualizing space data and mission planning

García

Fernando

Roberts

et al. 2019

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

This paper presents the system architecture of a collaborative virtual environment in which distributed multidisciplinary teams involved in space exploration activities come together and explore areas of scientific interest of a planet for future missions. The aim is to reduce the current challenges of distributed scientific and engineering meetings that prevent the exploitation of their collaborative potential, as, at present, expertise, tools and datasets are fragmented. This paper investigates the functional characteristics of a software framework that addresses these challenges following the design science research methodology in the context of the space industry and research. An implementation of the proposed architecture and a validation process with end users, based on the execution of different use cases, are described. These use cases cover relevant aspects of real science analysis and operation, including planetary data visualization, as the system aims at being used in future European missions. This validation suggests that the system has the potential to enhance the way space scientists will conduct space science research in the future.

show abstract

Section: Communication Viewmentioning

confidence: 99%

Collaborative virtual reality platform for visualizing space data and mission planning

García

Fernando

Roberts

et al. 2019

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

show abstract

“…2013). It was developed onward for a mixed reality system with multiple sites collaborating on shared data (Fairchild et al, 2016).…”

Section: Literature Reviewmentioning

confidence: 99%

“…With chairs evenly distributed around the Telethrone, it should in principle also be possible to support mutual awareness between a telepresent seated person and a person walking past. withyou (Roberts et al, 2015, Fairchild 2016) was a telepresence system that recreates a 3D CGI copy of a remote person that can be viewed from any perspective but was derived from multiple discrete 2D video streams. Before integration of withyou, the mona lisa effect meant that a pulled chair would need to exactly align to a remote camera if mutual gaze were to be supported.…”

Section: Introductionmentioning

confidence: 99%

Telethrone Reconstructed; Ongoing Testing Toward a More Natural Situated Display

O'Hare

Fairchild

Wolff

et al. 2017

Augmented Reality and Virtual Reality

Self Cite

View full text Add to dashboard Cite

The concept of supporting ad-hoc or dynamic membership tele-present meetings through pulling up a chair is novel. In real world business situations, people pull up a chair after catching the eye of someone already seated. Telethrone is a situated display on a chair which allows multiple correct views of a remote collaborator. The system has been expanded to support informal meetings where chairs can be moved around. This is facilitated through the novel integration of a 3D reconstructed model of a person, with live viewpoint dependent rendering onto a retro-reflective surface. This removes the need for painstaking alignment of multiple cameras and projectors each time a chair is moved. A between subjects experiment tested accuracy of reconnected mutual gaze mediated by part of the system. Subjectively easier and harder situations are compared. Specifically best and worst cases, both in terms of orientation of eyes in the reconstructed head, and angle of observer gaze onto the display. Discussion compares results to experiments that used other systems to attempt to convey eye gaze by different techniques. This research builds toward a scalable system for adhoc business meetings; a paradigm poorly supported by current video conferencing. It is also applicable to supporting conversations between seated people in any scenario where seats might be moved, for example in interaction between client and therapist in tele-therapy.

show abstract

“…A particular challenge, therefore, is to find a suitable coupling between the acquisition and viewing stages that respects the practical limitations imposed by available network bandwidth and client-side compute hardware while still guaranteeing an immersive exploration experience. For this purpose, teleconferencing systems for transmit-ting dynamic 3D models of their users typically rely on massive wellcalibrated acquisition setups with several statically mounted cameras around the region of interest [9,41,50]. Instead, we direct our attention to the remote exploration of places using portable, consumer-grade acquisition devices, for instance in scenarios of remote inspection or consulting.…”

Section: Introductionmentioning

confidence: 99%

SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence

Stotko

Krumpen

Hullin

et al. 2019

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Fig. 1. Illustration of our novel multi-client live telepresence framework for remote collaboration: RGB-D data captured with consumergrade cameras represent the input to our real-time large-scale reconstruction technique that is based on a novel thread-safe GPU hash map data structure. Efficient data streaming is achieved by transmitting a novel compact representation of the reconstructed model in terms of Marching Cubes indices. Multi-client live telepresence is achieved by the server's independent handling of client requests.Abstract-Real-time 3D scene reconstruction from RGB-D sensor data, as well as the exploration of such data in VR/AR settings, has seen tremendous progress in recent years. The combination of both these components into telepresence systems, however, comes with significant technical challenges. All approaches proposed so far are extremely demanding on input and output devices, compute resources and transmission bandwidth, and they do not reach the level of immediacy required for applications such as remote collaboration. Here, we introduce what we believe is the first practical client-server system for real-time capture and many-user exploration of static 3D scenes. Our system is based on the observation that interactive frame rates are sufficient for capturing and reconstruction, and real-time performance is only required on the client site to achieve lag-free view updates when rendering the 3D model. Starting from this insight, we extend previous voxel block hashing frameworks by introducing a novel thread-safe GPU hash map data structure that is robust under massively concurrent retrieval, insertion and removal of entries on a thread level. We further propose a novel transmission scheme for volume data that is specifically targeted to Marching Cubes geometry reconstruction and enables a 90% reduction in bandwidth between server and exploration clients. The resulting system poses very moderate requirements on network bandwidth, latency and client-side computation, which enables it to rely entirely on consumer-grade hardware, including mobile devices. We demonstrate that our technique achieves state-of-the-art representation accuracy while providing, for any number of clients, an immersive and fluid lag-free viewing experience even during network outages.

show abstract

A Mixed Reality Telepresence System for Collaborative Space Operation

Cited by 57 publications

References 41 publications

Collaborative virtual reality platform for visualizing space data and mission planning

Collaborative virtual reality platform for visualizing space data and mission planning

Telethrone Reconstructed; Ongoing Testing Toward a More Natural Situated Display

SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence

Contact Info

Product

Resources

About