RAFT-MSF: Self-Supervised Monocular Scene Flow using Recurrent Optimizer

Abstract

Learning scene flow from a monocular camera still remains a challenging task due to its ill-posedness as well as lack of annotated data. Self-supervised methods demonstrate learning scene flow estimation from unlabeled data, yet their accuracy lags behind (semi-)supervised methods. In this paper, we introduce a self-supervised monocular scene flow method that substantially improves the accuracy over the previous approaches. Based on RAFT, a state-of-the-art optical flow model, we design a new decoder to iteratively update 3D motion fields and disparity maps simultaneously. Furthermore, we propose an enhanced upsampling layer and a disparity initialization technique, which overall further improves accuracy up to 7.2%. Our method achieves the state-of-the-art accuracy among all self-supervised monocular scene flow methods, improving accuracy by 34.2%. Our finetuned model outperforms the best previous semisupervised method with 228 times faster runtime. Code will be publicly available.

Self-Supervised Ego-Motion Estimation Based on Multi-Layer Fusion of RGB and Inferred Depth

“…Hur et al [20] first present a novel selfsupervised model capable of inferring depth and 3D motion field from monocular sequences and surpass the performance of previous multi-task methods. Subsequent studies extend their method into a multi-frame model [21], or employ a recurrent network architecture [2] for better accuracy. Rigidity in Scene Flow.…”

“…Spatial Photometric Loss. To address scale ambiguity in monocular scene flow learning, we utilize stereo samples during training as proposed in previous works [20,21,2]. We use the stereoscopic image synthesis loss utilized in [60] to regularize depth estimation on an absolute scale and denote it as L d in our method.…”

“…Datasets. For the scene flow task, we use the same data setting as previous self-supervised monocular scene flow methods [20,21,2], which use KITTI Scene Flow Training and Testing as two test sets, and spilt the remaining data into 25801 samples for training and 1684 samples for validation. For comparison in the task of monocular depth estimation, we follow the data split used in [60], but remove the samples which are the last images of sequences, which gives us 22568 samples for training and 1774 for validation.…”

“…On the other hand, monocular scene flow estimation methods [5,57,58,62,30,26,20,21,2] which only require a monocular camera for obtaining both 3D structure and 3D motion, have been presented as an economical yet effective solution for dynamic 3D perception. The methods [5,57] combined with supervised learning have yielded promising results, yet the primary challenge facing them has been the limited availability of ground-truth training data.…”

“…To address this limitation, several multi-task methods [58,62,30,53,26] have been proposed to jointly learn the depth, 2D optical flow and camera ego-motion networks from monocular sequences in a self-supervised manner, and the scene flow can be calculated from the outputs. Recently, [20,21,2] have shown it feasible to train a single network to directly estimate both depth and 3D scene flow from two monocular images and outperform the previous multi-task methods. These methods typically build upon a standard optical flow pipeline (e.g., PWC-Net [43] or RAFT [44]) as basis and adapt it for monocular scene flow.…”