Scalable and coherent video resizing with per-frame optimization

Abstract

original video cube deformed video cube original frames deformed frames Figure 1: We introduce a scalable content-aware video retargeting method. Here, we render pairs of original and deformed motion trajectories in red and blue. Making the relative transformation of such pathlines consistent ensures temporal coherence of the resized video. AbstractThe key to high-quality video resizing is preserving the shape and motion of visually salient objects while remaining temporallycoherent. These spatial and temporal requirements are difficult to reconcile, typically leading existing video retargeting methods to sacrifice one of them and causing distortion or waving artifacts. Recent work enforces temporal coherence of content-aware video warping by solving a global optimization problem over the entire video cube. This significantly improves the results but does not scale well with the resolution and length of the input video and quickly becomes intractable. We propose a new method that solves the scalability problem without compromising the resizing quality. Our method factors the problem into spatial and time/motion components: we first resize each frame independently to preserve the shape of salient regions, and then we optimize their motion using a reduced model for each pathline of the optical flow. This factorization decomposes the optimization of the video cube into sets of subproblems whose size is proportional to a single frame's resolution and which can be solved in parallel. We also show how to incorporate cropping into our optimization, which is useful for scenes with numerous salient objects where warping alone would degenerate to linear scaling. Our results match the quality of state-of-the-art retargeting methods while dramatically reducing the computation time and memory consumption, making content-aware video resizing scalable and practical.

Scalable motion-aware panoramic videos

“…Between all the works in this subject, the ones closest to ours are [Wang et al 2009], [Wang et al 2010] and [Wang et al 2011]. The preservation of moving objects that we impose are related to the ones proposed by Wang et al [2009] and our concern of making the method scalable has connections with [Wang et al 2011].…”

“…For example, moving features could be preserved in the same way Wang et al [2011] did for the video resizing problem. Artistic perspective control could be included in our formulation with energy terms similar to the ones proposed by Carroll et al [2010].…”

GPU video retargeting with parallelized SeamCrop

“…As with previous techniques, warping tries to maintain important content unchanged while distorting homogenuous regions that the viewer will most likely not notice. An interesting approach considers the optical flow as an additional information [17]. The retargeting process is divided into a spatial and a temporal component that can be computed one after the other.…”

A Real-Time Smart Assistant for Video Surveillance Through Handheld Devices

“…The size of the video is thus reduced by removing unstructured areas between the important objects. Similarly, approaches based on Warping (see [9]) subdivide video frames into a rectangular mesh grid and transform each cell non-uniformly. Cells containing high importance remain mostly unchanged while unimportant cells may be warped.…”