Spatio-Temporally Consistent Color and Structure Optimization for Multiview Video Color Correction

Abstract

When compared to conventional 2-D video, multiview video can significantly enhance the visual 3-D experience in 3-D applications by offering horizontal parallax. However, when processing images originating from different views, it is common that the colors between the different cameras are not well-calibrated. To solve this problem, a novel energy function-based color correction method for multiview camera setups is proposed to enforce that colors are as close as possible to those in the reference image but also that the overall structural information is well-preserved. The proposed system introduces a spatio-temporal correspondence matching method to ensure that each pixel in the input image gets bijectively mapped to a reference pixel. By combining this mapping with the original structural information, we construct a global optimization algorithm in a Laplacian matrix formulation and solve it using a sparse matrix solver. We further introduce a novel forward-reverse objective evaluation model to overcome the problem of lack of ground truth in this field. The visual comparisons are shown to outperform state-of-the-art multiview color correction methods, while the objective evaluation reports PSNR gains of up to 1.34 dB and SSIM gains of up to 3.2%, respectively.

Detecting and Removing Visual Distractors for Video Aesthetic Enhancement

“…The results suggest that only using video frames in the input layer generates worse results. As mentioned earlier, this is because long-term temporal Ground truth Zhang et al [25] Our method Fig. 8.…”

“…Temporal information [23] and camera stability [24] were also considered to improve video aesthetic quality. More recently, Lu et al proposed a spatio-temporally consistent color and structure optimization method [25]. However, in spite of being important factors in video aesthetic quality, the detection of distractors, and their uses to improve video quality have been minimally explored.…”

Lighting transfer across multiple views through local color transforms

“…Spatialtemporal correspondences are also used in Refs. [8,9] for multi-view color correction. These methods work well for optimizing color consistency for image collections or videos, but they are not intended to transfer spatially-varying lighting.…”

Synthesis of a stroboscopic image from a hand-held camera sequence for a sports analysis

“…About color alignment, the method for reducing undesirable tonal fluctuations in video [12] and a color collection between multi-camera [13] have been proposed. Reference [12] designates one or more frames as anchors and executes a color alignment on basis of these frames.…”