Robust Global Motion Compensation in Presence of Predominant Foreground

Abstract

The objective of global motion compensation (GMC) is to remove intentional (due to camera pan/tilt/zoom) and unwanted (e.g., due to hand shaking) camera motion. GMC is utilized in applications such as video stitching, or as pre-processing for motion-based video analysis. Normally, GMC estimates the homography transformation between two consecutive frames by matching keypoints on the frames, and mapping the frames to a global coordinate. To remedy outliers in keypoint matches, robust techniques are proposed for homography estimation, e.g., RANSAC [1], by assuming the number of outliers to the correct homography is much less than inliers. However, in the presence of predominant foreground, i.e., moving objects and people, a larger proportion of the putative matches are mismatches. Predominant foreground may result from a higher percentage of coverage by foreground pixels, or occlusion, textureless and noninformative background, blurred background, or a combination of these reasons. In presence of predominant foreground, the common variations of RANSAC have little chance of selecting a minimal set of background keypoints by random sub-sampling in a limited number of iterations.In this paper, we propose a robust GMC (RGMC) method for suppressing foreground keypoint matches and mismatches, enabling a reliable homography estimation in presence of predominant foreground and textureless background (Fig. 1). We perform foreground suppression by clustering motion vectors computed from keypoint matches and identifying potential clusters corresponding to the background. We use SURF algorithm for keypoint detection and description and to detect sufficient background keypoints even for textureless backgrounds, the Fast-Hessian keypoint detection threshold is decreased drastically. Since motion vectors on the background result from camera motion and are more consistent than foreground motion vectors, clustering will likely lead to some candidate regions from the background (see Fig. 1 (a)). Each cluster is analyzed separately by random subsampling of matches in that cluster and evaluating the resultant homography against the cost function, discussed later. However, background motion vectors may be assigned to multiple clusters. To merge background clusters, based on the estimated homography and cost function value (CFV) of each cluster, a subset of the best clusters are selected to be merged in a greedy algorithm ( Fig. 1(b)).To evaluate the estimated homography from a quadruplet of keypoints matches, we derive a cost function that unifies the keypoint matching score, edge matching score, and the information from compensating previous frames. Denote the matching frames as I t−1 and I t , their candidate homography as θ t , and the set of keypoint matches under study as D. In Bayesian framework, θ t can be estimated by maximizingwhere θ t−1 is the obtained prior homography of frames I t−1 and I t−2 . The p(θ t |θ t−1 ) is the conditional probability of θ t given the prior homography θ t−1 . The denominator of Eqn. 1 is constant...

Temporally Robust Global Motion Compensation by Keypoint-Based Congealing

“…Baselines and details We choose three sequential GMC algorithms as the baselines for comparison: MLESAC [15] and HEASK [19] both based on our own implementation, and RGMC [10] based on the authors' Matlab code available online. We implement TRGMC in Matlab, and will publish the code.…”

“…Then, the labeler selects the foreground regions which subsequently identify the background region. Similar to [10], we quantify the consistency of two warped frames I (i) (p i ) and I (j) (p j ) (0 to 1 grayscale pixels) via the background region error (BRE),…”

“…Safdarnejad et al [10] incorporate edge matching into a probabilistic framework that scores candidate homographies. Although [10,12] improve the robustness to foreground, error accumulation and failure in a single frame pair still deteriorate the overall performance. Thus, TRGMC targets robustness of the GMC in terms of both the presence of foreground and long-term consistency by joint alignment of frames.…”

“…Schematic diagrams of proposed TRGMC and existing sequential GMC algorithms, and resultant motion panorama for a video shot by panning the camera up and down. Background continuity breaks easily in the case of the sequential GMC [10].…”

Recurrent Flow-Guided Semantic Forecasting

“…Numerous works have been published in motion prediction [1,4,34,39,45,46,47,48,49], which motivate flow prediction and using a recurrent structure for future anticipation. Villegas et al [44] encode motion as the subtraction between two past frames and utilize a ConvLSTM [18,42] to better aggregate temporal features.…”