Image interpolation for DIBR viewsynthesis using graph fourier transform

Abstract

Given texture and depth maps of one or more reference viewpoint(s), depth-image-based rendering (DIBR) can synthesize a novel viewpoint image by mapping texture pixels from reference to virtual view using geometric information provided by corresponding depth pixels. If the virtual view camera is located closer to the 3D scene than the reference view camera, objects close to the camera will increase in size in the virtual view, and DIBR's simple pixel-to-pixel mapping will result in expansion holes that require proper filling. Leveraging on recent advances in graph signal processing (GSP), in this paper we propose to select appropriate graph Fourier transforms (GFT)-adaptive to unique signal structures of the local pixel patches-for expansion hole filling. Our algorithm consists of two steps. First, using structure tensor we compute an adaptive kernel centered at a target empty pixel to identify suitable neighboring pixels for construction of a sparse graph. Second, given the constructed graph with carefully tuned edge weights, to complete the target pixel we formulate an iterative quadratic programming problem (with a closed form solution in each iteration) using a smoothness prior in the GFT domain. Experimental results show that our algorithm can outperform inpainting procedure employed in VSRS 3.5 by up to 4.57dB.

Analysis of sports statistics via graph-signal smoothness prior

“…As done in [3][4][5][6][7]10], in this paper we also employ a graph-signal smoothness prior: a signal x is more probable if x T Lx is small, i.e.,…”

“…Observed statistics are also incomplete; an N -team league typically has far fewer than N (N −1) sets of reliable statistics due to scheduling constraints. GSP provides tools [8][9][10] to interpolate missing samples in a desired graph-signal-point differential for every pair of teams in our case. Interpolated samples are thus our predicted game outcomes in future matchups in terms of point differential.…”

In-network view re-sampling for interactive free viewpoint video streaming

“…We first assume that the received N camera views v from server at the proxy are distorted due to compression, which can be simply evaluated as d(v). Further, a synthesized view u is distorted due to pixel rounding and disocclusion hole filling procedures [10,11] during DIBR-based view synthesis [1] at proxies or clients. Specifically, we assume that the rendered image distortion Du(vL, vR, d(vL), d(vR)) of virtual view u ∈ U synthesized using left and right reference views vL and vR, with respective distortion d(vL) and d(vR), depends on two factors: i) reference view distances |vL − u| and |vR − u|, and ii) distortions of the reference views d(vL) and d(vR).…”

Graph-based interpolation for zooming in 3D scenes

“…More recently, graph-based methods have been proposed for regularization [5], [6], [7]. Interpolation of graph signals is handled in [8] from a sampling perspective, while sparsity based interpolation methods using spectral graph theory are presented in [9], [10], [11]. The work in [12] formulates a patch-based maximum aposteriori problem to fill the expansion holes using a smoothness prior on the graph signal.…”