MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching

Abstract

Recent methods in stereo matching have continuously improved the accuracy using deep models. This gain, however, is attained with a high increase in computation cost, such that the network may not fit even on a moderate GPU. This issue raises problems when the model needs to be deployed on resource-limited devices. For this, we propose two light models for stereo vision with reduced complexity and without sacrificing accuracy. Depending on the dimension of cost volume, we design a 2D and a 3D model with encoder-decoders built from 2D and 3D convolutions, respectively. To this end, we leverage 2D Mo-bileNet blocks and extend them to 3D for stereo vision application. Besides, a new cost volume is proposed to boost the accuracy of the 2D model, making it performing close to 3D networks. Experiments show that the proposed 2D/3D networks effectively reduce the computational expense (27%/95% and 72%/38% fewer parameters/operations in 2D and 3D models, respectively) while upholding the accuracy. Our code is available at https://github.com/ cogsys-tuebingen/mobilestereonet.

Attention-Aware Feature Aggregation for Real-Time Stereo Matching on Edge Devices

“…Xing et al [8] has proposed adjust multi-branch module which combines depth-wise convolution to reduce the number of channels. The mobilestereonet's [9] main contribution is that they have proposed to reform the cost volume with convolution and use the 2D convolution only for the disparity regression. However, the final FLOPs and latency are still too larger and far from real-time.…”

“…With the goal of achieving real-time processing on edge devices (GPU/NPU), many lightweight methods have been proposed [7]- [12]. Those methods can be roughly divided into two categories: multi-stage method [8], [10], [13] and model compression method [7], [9], [11]. The computational complexity of the network depends on two factors: the size 1 The authors are with the UBTech Robotics Corp, Shenzhen, China {baiyu.pan,jichao.jiao,walton}@ubtrobot.com 2 The author is with the Beijing University of Posts and Telecommunications, Beijing, China jiaojichao@bupt.edu.cn 3 The authors are with the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China jun.cheng@siat.ac.cn of the feature map and the number of convolution kernels.…”

“…However, replacing the 3D convolution is challenging. Shamsafar et al [9] has achieved promising result by compressing the channel. More importantly, only 2D convolution is used in their network.…”

TriStereoNet: A Trinocular Framework for Multi-baseline Disparity Estimation

“…With the rise of deep learning, stereo matching continued to be reformed by these modern techniques. Following the general paradigm for stereo reconstruction, deep models can be divided into two categories: the methods that formulate one or some of the steps with a deep learning framework [2,28,37], and the approaches that transfer the full process in an end-to-end scheme [4,12,19,21,29,39]. Following the recent research, our model is also an end-to-end one, processing a 3-tuple sample.…”

The Implicit Values of A Good Hand Shake: Handheld Multi-Frame Neural Depth Refinement

“…The cell-phone of the 90s was a phone, the modern cellphone is a handheld computational imaging platform [9] that is capable of acquiring high-quality images, pose, and depth. Recent years have witnessed explosive advances in passive depth imaging, from single-image methods that leverage large data priors to predict structure directly from image features [39,40] to efficient multi-view approaches grounded in principles of 3D geometry and epipolar projection [49,46]. Alongside, progress has been made in the miniaturization and cost-reduction [3] of active depth systems such as LiDAR and correlation time-of-flight sensors [29].…”