Deep Video Stabilization Using Adversarial Networks

Abstract

Video stabilization is necessary for many hand‐held shot videos. In the past decades, although various video stabilization methods were proposed based on the smoothing of 2D, 2.5D or 3D camera paths, hardly have there been any deep learning methods to solve this problem. Instead of explicitly estimating and smoothing the camera path, we present a novel online deep learning framework to learn the stabilization transformation for each unsteady frame, given historical steady frames. Our network is composed of a generative network with spatial transformer networks embedded in different layers, and generates a stable frame for the incoming unstable frame by computing an appropriate affine transformation. We also introduce an adversarial network to determine the stability of apiece of video. The network is trained directly using the pair of steady and unsteady videos. Experiments show that our method can produce similar results as traditional methods, moreover, it is capable of handling challenging unsteady video of low quality, where traditional methods fail, such as video with heavy noise or multiple exposures. Our method runs in real time, which is much faster than traditional methods.

Deep Iterative Frame Interpolation for Full-frame Video Stabilization

“…Among other techniques, spatiotemporal optimization [Wang et al 2013] and user interactions [Bai et al 2014] have been applied to generate smooth camera motion. Online video stabilization methods [Liu et al 2017[Liu et al , 2016Xu et al 2018] use historical frames generated from the model to estimate current stabilization parameters. Although offline methods generally show better results, deep learning-based approaches Xu et al 2018] show promising results.…”

“…This work also provides a labeled dataset consisting of unstable and stable video sets. Xu et al [2018] propose an adversarial network also trained in a supervised manner, where warping parameters are estimated by the adversarial network to generate stabilized frames. Our method inherits the merits of 2D methods and performs video stabilization in the context of deep frame interpolation.…”

“…First, we conduct a category-wise comparison as conducted in Xu et al [2018], with six categories: Regular, Crowd, Parallax, Quick Rotation, Running, and Zooming as shown in Figure 11. The six categories contain challenging scenarios that include many moving people, large difference in parallax, quick camera rotation, severe shaking due to running, and shaking due to zooming.…”

“…In the past years, we have seen exceptional progress in the field of video stabilization. Various approaches have been explored including 3D [Liu et al 2009[Liu et al , 2012Zhou et al 2013], 2.5D [Goldstein and Fattal 2012;Liu et al 2011Liu et al , 2013a, 2D [Liu et al 2016[Liu et al , 2013b[Liu et al , 2014, and deep learning based Xu et al 2018] approaches. These approaches have shown significant success and continue to affect commercial stabilization algorithms.…”

“…Most existing video stabilization methods stabilize videos offline as postprocessing. Although offline methods generally show better stabilization results compared to online methods, recent deep learningbased Xu et al 2018] approaches have shown promising quality. The deep learning-based methods take a supervised approach that requires unstable (shaky) and stable (motion smoothed) video paired data.…”