Real-time high-resolution video stabilization using high-frame-rate jitter sensing

Abstract

In this study, the novel approach of real-time video stabilization system using a high-frame-rate (HFR) jitter sensing device is demonstrated to realize the computationally efficient technique of digital video stabilization for high-resolution image sequences. This system consists of a high-speed camera to extract and track feature points in graylevel 512 × 496 image sequences at 1000 fps and a high-resolution CMOS camera to capture 2048 × 2048 image sequences considering their hybridization to achieve real-time stabilization. The high-speed camera functions as a real-time HFR jitter sensing device to measure an apparent jitter movement of the system by considering two ways of computational acceleration; (1) feature point extraction with a parallel processing circuit module of the Harris corner detection and (2) corresponding hundreds of feature points at the current frame to those in the neighbor ranges at the previous frame on the assumption of small frame-to-frame displacement in high-speed vision. The proposed hybrid-camera system can digitally stabilize the 2048 × 2048 images captured with the high-resolution CMOS camera by compensating the sensed jitter-displacement in real time for displaying to human eyes on a computer display. The experiments were conducted to demonstrate the effectiveness of hybrid-camera-based digital video stabilization such as (a) verification when the hybrid-camera system in the pan direction in front of a checkered pattern, (b) stabilization in video shooting a photographic pattern when the system moved with a mixed-displacement motion of jitter and constant low-velocity in the pan direction, and (c) stabilization in video shooting a real-world outdoor scene when an operator holding hand-held hybrid-camera module while walking on the stairs.

Projection-mapping-based object pointing using a high-frame-rate camera-projector system

“…Temporal dithering of high-speed illumination was reported for fast active vision [51]. HFR vision systems have also been used as sensing devices in many applications such as optical flow [52], color histogram-based cam-shift tracking [53,54], face tracking [55], image mosaicking, and stabilization [56,57]. Hence, HFR vision systems can be used as an environment sensing device in CPS.…”

Projection-Mapping-Based Object Pointing Using a High-Frame-Rate Camera-Projector System

“…An HFR camera-projector-based structured-light range finding was used to eliminate common problems. Because of their high sampling rate, HFR vision systems have been used as sensing devices in many applications such as optical flow [51], color histogram-based cam-shift tracking [52,53], face tracking [54], image mosaicking, and stabilization [55,56]. HFR vision systems have also been used in dynamic sensing applications including the scratching behavior analysis of laboratory mice [57,58], cell deformation analysis [59,60,61], vision-assisted structure analysis [62,63], motion-blur-free video shooting [64,65], robot manipulation [66,67], and multicopter tracking [68,69].…”