DeepMeshFlow: Content Adaptive Mesh Deformation for Robust Image Registration

Abstract

Image alignment by mesh warps, such as meshflow, is a fundamental task which has been widely applied in various vision applications(e.g., multi-frame HDR/denoising, video stabilization). Traditional mesh warp methods detect and match image features, where the quality of alignment highly depends on the quality of image features. However, the image features are not robust in occurrence of low-texture and low-light scenes. Deep homography methods, on the other hand, are free from such problem by learning deep features for robust performance. However, a homography is limited to plane motions. In this work, we present a deep meshflow motion model, which takes two images as input and output a sparse motion field with motions located at mesh vertexes. The deep meshflow enjoys the merics of meshflow that can describe nonlinear motions while also shares advantage of deep homography that is robust against challenging textureless scenarios. In particular, a new unsupervised network structure is presented with content-adaptive capability. On one hand, the image content that cannot be aligned under mesh representation are rejected by our learned mask, similar to the RANSAC procedure. On the other hand, we learn multiple mesh resolutions, combining to a non-uniform mesh division. Moreover, a comprehensive dataset is presented, covering various scenes for training and testing. The comparison between both traditional mesh warp methods and deep based methods show the effectiveness of our deep meshflow motion model.

Parametric chamfer alignment based on mesh deformation

“…DeTone et al 25 and Nguyen et al 26 presented supervised and unsupervised learning methods to estimate global homography models, respectively. Ye et al 27 introduced a deep meshflow model, but suffered from high training cost.…”

TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations

“…Issues of parallax due to content at different depths can be better addressed by mesh-based warping [32,28,30] or pixel-wise dense optical flow [17,60,50,51,57,20] posed deep meshflow [66] to make mesh estimation more robust on different scenes. Due to the sparsity of the mesh, image contents can be better retained while warping.…”