Stereo-video inpainting

Abstract

Abstract. As the production of stereoscopic content increases, so does the need for post-production tools for that content. Video inpainting has become an important tool for rig removal, but there has been little consideration of the problem in stereo. We present an algorithm for stereo-video inpainting that builds on existing exemplar-based video completion and also considers the issues of view consistency. Given user-selected regions in the sequence, which may be in the same location in several frames and in both views, the objective is to fill in this area using all the available picture information. Existing algorithms lack temporal consistency, causing flickering and other artifacts. We explore the use of long-term picture information across many frames in order to achieve temporal consistency at the same time as exploiting inter-view dependencies within the same framework.

Stereo video completion for rig and artefact removal

“…In [8] we only compare 2D patches with the measure d(p, x) = 1 − SSIM(p, x) using a SSIM [9] weighted by G. In this paper we extend the distance to explicitly enforce view consistency by also comparing the respective stereo-counterpart patches at p and x, i.e. patches centred at disparity-compensated locations p and x on the other view:…”

“…As in [8] our algorithm is iterative, starting with a frame in the middle of the video in a given view. A key deviation from [8] to impose more constraints on the reconstruction and speed up the tracking step is to process the stereo-counterpart of a given frame immediately after having reconstructed the frame.…”

“…A key deviation from [8] to impose more constraints on the reconstruction and speed up the tracking step is to process the stereo-counterpart of a given frame immediately after having reconstructed the frame.…”

“…The reconstructed vectors allow to find candidates for replacement for each missing site by tracking each corresponding patch along time and across view in the whole stereo sequence (using bilinear interpolation on the vectors to avoid drifting errors). Similar to [8] the revealed area around the shifted position of a missing pixel should be maximised to increase the likelihood of selecting relevant information for completion. But we deviate from [8] in three ways to encourage the selection of patches that contain new information.…”

“…Contribution: Our proposed method builds on the technique presented in [8] which reconstructs a hole pixel by pixel, without explicit view consistency constraint. It builds a set of candidate pixels by tracking the missing site, compares all candidates and selects the best match for replacement.…”