Veritatem Dies Aperit - Temporally Consistent Depth Prediction Enabled by a Multi-Task Geometric and Semantic Scene Understanding Approach

Abstract

Robust geometric and semantic scene understanding is ever more important in many real-world applications such as autonomous driving and robotic navigation. In this paper, we propose a multi-task learning-based approach capable of jointly performing geometric and semantic scene understanding, namely depth prediction (monocular depth estimation and depth completion) and semantic scene segmentation. Within a single temporally constrained recurrent network, our approach uniquely takes advantage of a complex series of skip connections, adversarial training and the temporal constraint of sequential frame recurrence to produce consistent depth and semantic class labels simultaneously. Extensive experimental evaluation demonstrates the efficacy of our approach compared to other contemporary state-of-the-art techniques.

To Complete or to Estimate, That is the Question: A Multi-Task Approach to Depth Completion and Monocular Depth Estimation

“…Using adversarial training, a deep architecture with skip connections and a blend of synthetic and real-world training data to guarantee the accuracy and density of the depth output, our approach can produce high quality scene depth. Our extensive experimental evaluation demonstrates the efficacy of our approach compared to contemporary state-of-the-art methods across both domains of monocular depth estimation [5,7,14,20,31,36,62,66] and sparse depth completion [10,16,40,50,54].…”

“…Depth completion can refer to a range of related problems with different input modalities [3]. The existing literature contains a variety of techniques capable of completing relatively dense depth images that contain missing values, such as those utilising exemplar-based depth inpainting [4], low-rank matrix completion [60], object-aware interpolation [2], tensor voting [30], Fourier-based depth filling [8], background surface extrapolation [41], learning-based approaches using deep networks [1,7,63], and alike [9,37].…”

“…• A joint multi-task framework for depth prediction encouraging improved geometric and contextual learning to boost performance. • Monocular depth estimation via adversarial training, a deep architecture with skip connections and a robust compound objective function directly supervised using this framework to outperform prior contemporary work [5,7,14,20,31,36,62,66]. • Sparse to dense depth completion via the same multitask model, capable of generating a dense depth output given a sparse depth input captured via a LiDAR sensor with results superior to prior contemporary work [10,16,40,50,54].…”

Multi-Task Regression-Based Learning for Autonomous Unmanned Aerial Vehicle Flight Control Within Unstructured Outdoor Environments

“…Within the existing literature, various end-to-end learningbased approaches have been employed to derive a set of navigational parameters from a given image, allowing for obstacle avoidance [16], [18], [20], [22]. Additionally, the recent advances made in multi-task systems partially focusing on depth estimation [23]- [26] can also be potentially beneficial towards a successful obstacle avoidance and path planning approach. However, most existing approaches offers only three degrees of freedom navigation, which makes them unsuitable for autonomous flight in unstructured environments, where the UAV may require to change altitude to avoid certain obstacles.…”

Monocular Segment-Wise Depth: Monocular Depth Estimation Based on a Semantic Segmentation Prior

“…There are also supervised approaches [9,19] that use synthetic images to produce depth outputs. Here, we also employ synthetic images [13] in a directly supervised training framework to perform the task of monocular depth estimation.…”