BA-Net: Dense Bundle Adjustment Network

Abstract

This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error. The whole pipeline is differentiable, so that the network can learn suitable features that make the BA problem more tractable. Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth. The network first generates several basis depth maps according to the input image, and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA. The basis depth maps generator is also learned via end-to-end training. The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem. Experiments on large scale real data prove the success of the proposed method.

Multi-view Monocular Depth and Uncertainty Prediction with Deep SfM in Dynamic Environments

“…Our primary focus lies on the depth prediction performance and we include both multi-view and single-view comparisons. As common in deep monocular multi-view SfM we report results for the ground truth scale aligned depth [72,69,70]. We compare with current SotA multi-view frameworks and report results for their publicly available models.…”

“…One of the first deep SfM systems was published by [72]. Since then, a series of frameworks combine multi-view image information for inferring camera motion and scene geometry [88,11,69,70,23,75]. While most works rely on generic network architectures, few combine learning with a traditional geometric optimization [70,69,11].…”

“…Since then, a series of frameworks combine multi-view image information for inferring camera motion and scene geometry [88,11,69,70,23,75]. While most works rely on generic network architectures, few combine learning with a traditional geometric optimization [70,69,11]. We base our model on DeepV2D [70], which couples supervised training of depth based on a cost volume architecture with a geometric pose graph optimization.…”

“…Single-view networks may not generalize well from one dataset to another [48,14] or need to be trained on massive datasets [59,58]. Using a true multi-view learning approach [72,69,70,75,23] turned out to be favorable in terms of performance. An Supported by Robert Bosch GmbH.…”

“…Recent developments have shown, that modeling scene geometry explicitly inside the architecture [70,69,23] leads to better reconstruction results than loosely coupling neural networks with a common training loss. Inside this paradigm, learned features are aligned temporally and spatially based on scene geometry and camera motion.…”