Transform domain sparsification of depth maps using iterative quadratic programming

Abstract

Compression of depth maps is important for "texture plus depth" format of multiview images, which enables synthesis of novel intermediate views via depth-image-based rendering (DIBR) at decoder. Previous depth map coding schemes exploit unique depth data characteristics to compactly and faithfully reproduce the original signal. In contrast, since depth map is only a means to the end of view synthesis and not itself viewed, in this paper we explicitly manipulate depth values, without causing severe synthesized view distortion, in order to maximize representation sparsity in the transform domain for compression gain-we call this process transform domain sparsification (TDS). Specifically, for each pixel in the depth map, we first define a quadratic penalty function, with minimum at ground truth depth value, based on synthesized view's distortion sensitivity to the pixel's depth value during DIBR. We then define an objective for a depth signal in a block as a weighted sum of: i) signal's sparsity in the transform domain, and ii) per-pixel synthesized view distortion penalties for the chosen signal. Given that sparsity (l0-norm) is non-convex and difficult to optimize, we replace the l0-norm in the objective with a computationally inexpensive weighted l2-norm; the optimization is then an unconstrained quadratic program, solvable via a set of linear equations. For the weighted l2-norm to promote sparsity, we solve the optimization iteratively, where at each iteration weights are readjusted to mimic sparsity-promoting lτ -norm, 0 ≤ τ ≤ 1. Using JPEG as an example transform codec, we show that our TDS approach gained up to 1.7dB in rate-distortion performance for the interpolated view over compression of unaltered depth maps.

Depth map compression using multi-resolution graph-based transform for depth-image-based rendering

“…For example, depth pixels close to an object edge are likely to affect synthesized view distortion more than depth pixels interior to an object. Exploiting this observation, [6] optimized mode selection during H.264 coding of depth video, while [5] manipulated unimportant depth pixels (without causing severe synthesized view distortion) towards a sparse representation of the depth signal in the transform domain. In contrast, our work exploits the unique characteristics of depth maps (sharp edges and smooth interior surfaces) for coding gain.…”

“…While compression of texture maps is well studied, compression of depth maps is relatively new, and has been the focus of many recent research efforts [3,4,5]. Typical depth maps exhibit unique characteristics, such as sharp edges and smooth surfaces interior to the sharp edges, that are quite different from texture maps, and previous depth map compression algorithms have attempted to exploit these characteristics for coding gain.…”

“…The surface-smoothness prior ensures us that high-frequencies lost (if any) during low-pass filtering before down-sampling (to avoid aliasing) would be minimal. On the other hand, to preserve sharpness (which greatly affects DIBR-synthesized view quality for depth maps [5]), we should encode edges in original HR.…”

Reference frame selection for loss-resilient depth map coding in multiview video conferencing

“…We leverage the recent results by Cheung et al 7 to derive weights of importance for individual MBs in a depth frame.…”

“…6, 7 These studies 6,7 observed that depth maps are encoded solely for the purpose of view synthesis and designed specialized coding optimization accordingly. In one work, 6 coding modes in H.264 are selected based on individual MBs' impact on synthesized view distortion.…”