HoloKinect: Holographic 3D Video Conferencing

Abstract

Recent world events have caused a dramatic rise in the use of video conferencing solutions such as Zoom and FaceTime. Although 3D capture and display technologies are becoming common in consumer products (e.g., Apple iPhone TrueDepth sensors, Microsoft Kinect devices, and Meta Quest VR headsets), 3D telecommunication has not yet seen any appreciable adoption. Researchers have made great progress in developing advanced 3D telepresence systems, but often with burdensome hardware and network requirements. In this work, we present HoloKinect, an open-source, user-friendly, and GPU-accelerated platform for enabling live, two-way 3D video conferencing on commodity hardware and a standard broadband internet connection. A Microsoft Azure Kinect serves as the capture device and a Looking Glass Portrait multiscopically displays the final reconstructed 3D mesh for a hologram-like effect. HoloKinect packs color and depth information into a single video stream, leveraging multiwavelength depth (MWD) encoding to store depth maps in standard RGB video frames. The video stream is compressed with highly optimized and hardware-accelerated video codecs such as H.264. A search of the depth and video encoding parameter space was performed to analyze the quantitative and qualitative losses resulting from HoloKinect’s lossy compression scheme. Visual results were acceptable at all tested bitrates (3–30 Mbps), while the best results were achieved with higher video bitrates and full 4:4:4 chroma sampling. RMSE values of the recovered depth measurements were low across all settings permutations.

On the Way to Holographic-Type Communications: Perspectives and Enabling Technologies

“…Holographic teleconferencing and holographic telepresence [13]- [18] are impressive HTC applications. There is a small conceptual difference between these terms.…”

Toward immersive communications in 6G

“…In the current video teleconferencing, remote participants can only be displayed on flat screens, which results in a very different perception in a virtual conference from that in an on-site conference. In order to provide an immersive experience in teleconferences, holographic teleconferences depict realistic 3D presence for people by projecting 3D images of remote participants as holograms (Jiang et al, 2021;Siemonsma and Bell, 2022;Zhang Y. et al, 2022;Zhou F. et al, 2022). Specifically, when…”

“…Gaming Hu et al, 2019;Carroll and Yildirim, 2021;Rega andSaxena, 2022 E-learning Kavanagh et al, 2017;Ahmad et al, 2021;Harvey et al, 2021 Makransky andPetersen, 2021 Teleconference Siemonsma and Bell, 2022;Zhang Y. et al, 2022 Tele-operation Choi et al, 2018;Lee et al, 2021E-heath Jumreornvong et al, 2020Velana et al, 2022E-commerce Speicher et al, 2017Ornati, 2022 Smart home Eckstein et al, 2019;Zhu et al, 2020Manufacturing Lipton et al, 2017Aijaz and Sooriyabandara, 2018 Tourism and travel Han and Jung, 2018;Chun et al, 2022Metaverse Wang et al, 2022bXu et al, 2022; a remote participant joins the holographic teleconference, 3D visual information and the corresponding audio information of the participant can be captured by multiple sensors, transmitted, and then reconstructed as a hologram on the side of other participants to provide 3D audiovisual information for interactions among participants (Strinati et al, 2019). In this case, holographic teleconference can reduce the impact on participants of the separation between the virtual and the physical worlds.…”

Toward Immersive Communications in 6G

“…In the current video teleconferencing, remote participants can only be displayed on flat screens, which results in a very different perception in a virtual conference from that in an on-site conference. In order to provide an immersive experience in teleconferences, holographic teleconferences depict realistic 3D presence for people by projecting 3D images of remote participants as holo-grams [28], [73]- [75]. Specifically, when a remote participant joins the holographic teleconference, 3D visual information and the corresponding audio information of the participant can be captured by multiple sensors, transmitted, and then reconstructed as a hologram on the side of other participants to provide 3D audiovisual information for interactions among participants [76].…”

“…1, i.e., XR, haptic communication, and holographic communication. 1 Via introducing the concept, basic implementation procedure, and the network requirements Teleconference [28], [75] Tele-operation [54], [68] E-heath [64], [87] E-commerce [88], [89] Smart home [90], [91] Manufacturing [92], [93] Tourism and travel [94], [95] Metaverse [9], [45], [77], [96] for each of the three forms, we aim to sketch an overall picture of immersive communications. The requirements of representative immersive communications use cases are illustrated in Fig.…”