Design, Implementation, and Evaluation of a Point Cloud Codec for Tele-Immersive Video

Abstract

Abstract-we present a generic and real-time time-varying point cloud codec for 3D immersive video. This codec is suitable for mixed reality applications where 3D point clouds are acquired at a fast rate. In this codec, intra frames are coded progressively in an octree subdivision. To further exploit interframe dependencies, we present an inter-prediction algorithm that partitions the octree voxel space in N times N times N macroblocks (N=8,16,32). The algorithm codes points in these blocks in the predictive frame as a rigid transform applied to the points in the intra coded frame. The rigid transform is computed using the iterative closest point algorithm and compactly represented in a quaternion quantization scheme. To encode the color attributes, we defined a mapping of color per vertex attributes in the traversed octree to an image grid and use legacy image coding method based on JPEG. As a result, a generic compression framework suitable for real-time 3D tele-immersion is developed. This framework has been optimized to run in realtime on commodity hardware for both encoder and decoder. Objective evaluation shows that a higher rate-distortion (R-D) performance is achieved compared to available point cloud codecs. A subjective study in a state of art mixed reality system shows that introduced prediction distortions are negligible compared to the original reconstructed point clouds. In addition, it shows the benefit of reconstructed point cloud video as a representation in the 3D Virtual world. The codec is available as open source for integration in immersive and augmented communication applications and serves as a base reference software platform in JCT1/SC29/WG11 (MPEG) for the further development of standardized point cloud compression solutions.

Benchmarking of Objective Quality Metrics for Colorless Point Clouds

“…In particular, following the notations of Figure 1, for each point b k of the content under evaluation B, its nearest neighbor a i from the reference point cloud A is determined. Then, the Euclidean distance between them, E(a i , b k ), is calculated based on Equation 1.…”

“…In every metric, each point of the content under evaluation is associated with an individual error, calculated through a corresponding equation, as given above. The level of degradation of a test content with respect to its original version is expressed through a total error value that can be estimated either as the Root Mean Squared (RMS) or the Mean Squared Error (MSE) or a simple average of the individual errors, or by taking the Hausdorff distance 1 . Different weights could be potentially assigned to the individual errors.…”

“…Mekuria et al [1] collected subjective scores for dynamic colored point clouds that were captured in real-time by multiple Microsoft Kinect sensors. The acquired models represented the avatars of the subjects that were able to navigate in a virtual environment in a desktop setup.…”

On the performance of metrics to predict quality in point cloud representations

“…However, it is not clearly reported by the authors whether the evaluation was passive or interactive and what was the adopted subjective methodolgy (i.e., ACR, DSIS, etc.). Mekuria et al 18 proposed a 3D tele-immersive system in which users represented by avatars (i.e., 3D point clouds captured by multiple Microsoft Kinect sensors) were able to interact in a virtual (i.e., synthetic) room, enabling a mixed reality scenario. The subjects were able to control the interaction of their avatar with the virtual environment through the use of the mouse in a desktop setup.…”

“…The state-of-the-art objective metrics for geometric distortions can be classified as point-to-point (p2point) and point-to-plane (p2plane). 18,28 In the first case, p2point error is calculated by connecting each point of the content under evaluation to the closest point that belongs to the reference point cloud. In the second case, the normal of each point of the reference point cloud should be given or estimated.…”