OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving

Abstract

Surround View fisheye cameras are commonly deployed in automated driving for 360°near-field sensing around the vehicle. This work presents a multi-task visual perception network on unrectified fisheye images to enable the vehicle to sense its surrounding environment. It consists of six primary tasks necessary for an autonomous driving system: depth estimation, visual odometry, semantic segmentation, motion segmentation, object detection, and lens soiling detection. We demonstrate that the jointly trained model performs better than the respective single task versions. Our multi-task model has a shared encoder providing a significant computational advantage and has synergized decoders where tasks support each other. We propose a novel camera geometry based adaptation mechanism to encode the fisheye distortion model both at training and inference. This was crucial to enable training on the WoodScape dataset, comprised of data from different parts of the world collected by 12 different cameras mounted on three different cars with different intrinsics and viewpoints. Given that bounding boxes is not a good representation for distorted fisheye images, we also extend object detection to use a polygon with non-uniformly sampled vertices. We additionally evaluate our model on standard automotive datasets, namely KITTI and Cityscapes. We obtain the state-of-the-art results on KITTI for depth estimation and pose estimation tasks and competitive performance on the other tasks. We perform extensive ablation studies on various architecture choices and task weighting methodologies. A short video at https://youtu.be/xbSjZ5OfPes provides qualitative results.

Near-field Perception for Low-Speed Vehicle Automation using Surround-view Fisheye Cameras

“…For the next generation, a unified CNN model with high synergies would be the likely path. We have recently published an initial prototype Omnidet [114] showing joint modelling of reconstruction and recognition. Figure 16 illustrates its high level architecture with cross links shown across the different tasks.…”

“…Fig.16: Overview of our next generation unified multi-task visual perception framework. Refer to our OmniDet paper[114] for more details.…”

Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation

“…Segmentation of panoramic data, which is often captured through distortion-pronounced fisheye lenses [38], [39], [40] or multiple surround-view cameras [41], [42], [43], is challenging as it entails a set of hard tasks like distortion elimination, camera synchronization and calibration, as well as data fusion, resulting in higher latency and complexity. Yang et al introduce the PASS [7] and the DS-PASS [44] frameworks which naturally mitigate the effect of distortions by using a single-shot panoramic annular lens system, but come with an expensive memory-and computation cost, as it requires separating the panorama into multiple partitions for predictions, each resembling a narrow-FoV pinhole image.…”

VM-MODNet: Vehicle Motion aware Moving Object Detection for Autonomous Driving

“…We start with the OmniDet [18] motion segmentation network using two-stream RGB only network. The network consists of two ResNet18 streams with shared weights and a motion segmentation decoder with deconv layers for upsampling to the higher resolution output.…”