Learning to Adapt for Stereo

Abstract

Real world applications of stereo depth estimation require models that are robust to dynamic variations in the environment. Even though deep learning based stereo methods are successful, they often fail to generalize to unseen variations in the environment, making them less suitable for practical applications such as autonomous driving. In this work, we introduce a "learning-to-adapt" framework that enables deep stereo methods to continuously adapt to new target domains in an unsupervised manner. Specifically, our approach incorporates the adaptation procedure into the learning objective to obtain a base set of parameters that are better suited for unsupervised online adaptation. To further improve the quality of the adaptation, we learn a confidence measure that effectively masks the errors introduced during the unsupervised adaptation. We evaluate our method on synthetic and real-world stereo datasets and our experiments evidence that learning-to-adapt is, indeed beneficial for online adaptation on vastly different domains. * Work done while at University of Oxford. † Second two authors contributed equally.arXiv:1904.02957v1 [cs.CV] 5 Apr 2019 Model Agnostic Meta LearningModel Agnostic Meta Learning (MAML) [5] is a popular meta-learning algorithm designed for few-shot learning problems. The objective is to learn a base model θ * , which

Online Depth Learning Against Forgetting in Monocular Videos

“…Recently, Park et al [38] introduced a method based on meta-learning to obtain an initial network for online tracking. A more closely related work to ours is [49], where an approach for learning to better adapt models to stereo videos is proposed. In contrast, our paper focuses on online mono-depth learning.…”

“…In such open-world condition, we process the video frames sequentially and continuously adapt our model at each time step in order to predict more accurate depth maps with t increasing. Similar to [49,62] on open-world stereo, we follow the paradigm for learning and evaluating. At time t, we first predict depth from I t (the current frame).…”

“…Results are reported in table 4. In this comparison, we include the L2A method [49] which can also be implemented in our model. Importantly, this method is not design to prevent catastrophic forgetting.…”

“…source domain), and keeps changing continuously over time. Several recent works are proposed to tackle this practical open-world problem for stereo matching [62,50,49], but few work focus on online monocular depth adaptation. Compared with online stereo, online mono-depth learning is even more challenging from two aspects: (i) scale ambiguity and lack of support from geometrical information inherent to a monocular setting make the model extremely dependent on domain-specific visual features and prone to overfit on the current domain [11,10]; (ii) the environmental variation (e.g., speed or scene changes) introduces additional challenges both for depth or pose estimation, making the whole model fragile.…”

“…θ 0 to find a good initial weight for online adaptation. Our method is different from the approach in [49] from two aspects: i) it meta-learns and updates adapters rather than the whole model, which preserves and adjusts (rather than totally changes) reliable basic knowledge against drastic changes; ii) it performs evaluation and meta-gradient descent after adapting on video sequences rather than single frame, aiming to achieve good long-range adaptation for future frames.…”