Loss-Resilient Coding of Texture and Depth for Free-Viewpoint Video Conferencing

Abstract

Abstract-Free-viewpoint video conferencing allows a participant to observe the remote 3D scene from any freely chosen viewpoint. An intermediate virtual viewpoint image is commonly synthesized using two pairs of transmitted texture and depth maps from two neighboring captured viewpoints via depthimage-based rendering (DIBR). To maintain high quality of synthesized images, it is imperative to contain the adverse effects of network packet losses that may arise during texture and depth video transmission. Towards this end, we develop an integrated approach that exploits the representation redundancy inherent in the multiple streamed videos-a voxel in the 3D scene visible to two captured views is sampled and coded twice in the two views. In particular, at the receiver we first develop an error concealment strategy that adaptively blends corresponding pixels in the two captured views during DIBR, so that pixels from the more reliable transmitted view are weighted more heavily. We then couple it with a sender-side optimization of reference picture selection (RPS) during real-time video coding, so that blocks containing samples of voxels that are visible in both views are more error-resiliently coded in one view only, given adaptive blending will erase errors in the other view. Further, synthesized view distortion sensitivities to texture versus depth errors are analyzed, so that relative importance of texture and depth code blocks can be computed for system-wide RPS optimization. Experimental results show that the proposed scheme can outperform the use of a traditional feedback channel by up to 0.82 dB on average at 8% packet loss rate, and by as much as 3 dB for particular frames.Index Terms-Free viewpoint video conferencing, reference picture selection, error concealment, depth-image-based rendering.

Adaptive Multiview Video Delivery Using Hybrid Networking

“…Free-viewpoint video has also attracted attention in the domain of video conferencing applications. In [25], the authors have proposed an error-resilient streaming scheme for free-viewpoint video conferencing. The presented approach favours the more reliably reconstructed view in the decoder-side view synthesis process and employs an encoder-side optimised reference picture selection technique for real-time operation.…”

View Synthesis Distortion Estimation With a Graphical Model and Recursive Calculation of Probability Distribution

“…In [30], a quadratic model is proposed to relate the disparity errors caused by packet loss in the depth maps to the distortion contribution in the synthesized view. In [31]- [33], the quadratic-model-based distortion estimation is used at the encoder for reference frame selection and quantization parameter optimization when coding the depth and texture images. In [33], the view synthesis at the decoder is designed such that reliability factors are introduced and related to the estimated synthesis distortion to give higher synthesis quality.…”

“…In [31]- [33], the quadratic-model-based distortion estimation is used at the encoder for reference frame selection and quantization parameter optimization when coding the depth and texture images. In [33], the view synthesis at the decoder is designed such that reliability factors are introduced and related to the estimated synthesis distortion to give higher synthesis quality. The overall transmission distortion estimation framework used in them is a block-based recursive approach that is similar to the one used in [10].…”

Joint source and channel coding of view and rate scalable multi-view video

“…In addition, in [15], RLC is proposed for transmission of H.264 SVC video over LTE/LTE-Advanced networks at the MAC layer, as a replacement of traditional ARQ. In [16], loss-resilient source coding of VpD MVV is studied, however, with no channel coding considerations. Similarly, [17] considers multicast of MVV, where the captured video and depth signals are SVC encoded, and each client is served two reference video and depth signals.…”