InDepth

Abstract

Mobile Augmented Reality (AR) demands realistic rendering of virtual content that seamlessly blends into the physical environment. For this reason, AR headsets and recent smartphones are increasingly equipped with Time-of-Flight (ToF) cameras to acquire depth maps of a scene in real-time. ToF cameras are cheap and fast, however, they suffer from several issues that affect the quality of depth data, ultimately hampering their use for mobile AR. Among them, scale errors of virtual objects - appearing much bigger or smaller than what they should be - are particularly noticeable and unpleasant. This article specifically addresses these challenges by proposing InDepth, a real-time depth inpainting system based on edge computing. InDepth employs a novel deep neural network (DNN) architecture to improve the accuracy of depth maps obtained from ToF cameras. The DNN fills holes and corrects artifacts in the depth maps with high accuracy and eight times lower inference time than the state of the art. An extensive performance evaluation in real settings shows that InDepth reduces the mean absolute error by a factor of four with respect to ARCore DepthLab. Finally, a user study reveals that InDepth is effective in rendering correctly-scaled virtual objects, outperforming DepthLab.

Ambient Intelligence for Next-Generation AR

“…In our evaluation of depth estimates obtained by a Microsoft HoloLens 2 (in the long throw mode) across a range of representative indoor environments, we found that on average 30% of depth pixels in a frame were missing [220]. We also collected an indoor dataset of 18.6K depth maps on a Samsung Galaxy Note 10+ smartphone, of which 58% had greater than 40% missing pixels [220].…”

“…Even some challenging reflections may be avoided from different viewpoints. We envision this collaborative sensing approach being combined with existing techniques for depth map completion such as [126,160,219,220], with the more complete depth data from multiple sensors combined on the edge server for a less challenging depth inpainting task.…”