Self-Supervised Real-time Video Stabilization

Abstract

Videos are a popular media form, where online video streaming has recently gathered much popularity. In this work, we propose a novel method of real-time video stabilization -transforming a shaky video to a stabilized video as if it were stabilized via gimbals in real-time. Our framework is trainable in a self-supervised manner, which does not require data captured with special hardware setups (i.e., two cameras on a stereo rig or additional motion sensors). Our framework consists of a transformation estimator between given frames for global stability adjustments, followed by scene parallax reduction module via spatially smoothed optical flow for further stability. Then, a margin inpainting module fills in the missing margin regions created during stabilization to reduce the amount of post-cropping. These sequential steps reduce distortion and margin cropping to a minimum while enhancing stability. Hence, our approach outperforms state-of-theart real-time video stabilization methods as well as offline methods that require camera trajectory optimization. Our method procedure takes approximately 24.3 ms yielding 41 fps regardless of resolution (e.g., 480p or 1080p).

Adaptive Sampling-based Particle Filter for Visual-inertial Gimbal in the Wild

“…To tackle the problem of video stabilization in dynamic scenes, a dense warping field as scene representation was trained from consecutive video frames by Liu et al [38], then the warped parts are blended to synthesize the stabilized image. The framework by J. Choi et al [37] even made use of a motion prediction model based on optical flow tracking. However, all those models involve a lot of processing overhead and are quite heavy to deploy on an edge device in real time.…”