System-on-Chip Solution of Video Stabilization for CMOS Image Sensors in Hand-Held Devices

Abstract

Majority of CMOS image sensors in consumer market utilize a rolling shutter to increase sensitivity. However, it causes severe distortions, such as jitter, wobble, or skew. Since most of these kinds of sensors are used in hand-held devices, the approach of undistorting and generating stabilized images is restricted to resource limited systems. It has also been one of the major challenges to have a mathematical representation of CMOS rolling effect depicting the practical scenario, while keeping accuracy and stability. We propose that a CMOS sensor can be modeled by a section-wise charge-coupled devices model which has multiple homographies and exploit the observation that rolling shutter mechanism gives close relationships among them. We present a CMOS seven-parameter model, and show video stabilization algorithm by the iterative parameter estimation technique. We address four issues while accelerating our stabilization algorithm within resource limited environment: accuracy, stability, computation time, and resource utilization. We developed cache based optimization techniques to meet the requirement of the memory bandwidth and computational time for the iterative parameter estimation and final output image interpolation, and also proposed the incremental form of the seven-parameter model to greatly reduce resource consumption while maintaining the same results as the previous. The validity and effectiveness of our approach is demonstrated by experiments for different types of camera motions. The cache based optimization technique can be used to accelerate other types of iterative vision algorithms that require repetitive memory access: feature tracking, motion estimation, motion compensation, various types of image distortion correction, and also image warping and scaling.Index Terms-Cache-based optimization, CMOS sensor model, hardware interpolation technique, rolling shutter mechanism, video stabilization.

Online real-time presentation of virtual experiences forexternal viewers

“…Many researchers have used filtering techniques to stabilize video sequences in real-time [2, 6, 18, 21]. This type of filtering has become a particular focus for researchers dealing with mobile platforms [12, 15]. Others have used real-time stabilization as a means of treatment.…”

TreeBASIS Feature Descriptor and Its Hardware Implementation

“…Relevant computer vision applications involve target tracking [1,2], object identification [3], optical flow [4], stereo vision [5], super resolution [6], image stabilization [7], and image rectification, localization, and pose estimation [1,[8][9][10][11]. These vision-based applications require high quality feature description and matching.…”

Low Power Parallel Rolling Shutter Artifact Removal

“…Kim et al proposed an algorithm designed to provide video stabilization for CMOS image sensors in 2011 [15]. This approach uses a seven degree of freedom camera model to estimate motion from frame to frame.…”