Generalized Object Detection on Fisheye Cameras for Autonomous Driving: Dataset, Representations and Baseline

Rashed, Hazem; Mohamed, Eslam; Sistu, Ganesh; Kumar, Varun; Eising, Ciaran; El-Sallab, Ahmad; Yogamani, Senthil

doi:10.1109/wacv48630.2021.00232

Cited by 63 publications

(40 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The standard bounding box representation fails in fisheye cameras due to heavy radial distortion, particularly in the periphery. In concurrent work [15], we explored different output representations for fisheye images, including oriented bounding boxes, curved boxes, ellipses, and polygons. We have integrated this model in our MTL framework where we use a 24-sided polygon representation for object detection.…”

Section: B Generalized Object Detectionmentioning

confidence: 99%

OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving

Kumar

Yogamani

Rashed³

et al. 2021

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Surround View fisheye cameras are commonly deployed in automated driving for 360°near-field sensing around the vehicle. This work presents a multi-task visual perception network on unrectified fisheye images to enable the vehicle to sense its surrounding environment. It consists of six primary tasks necessary for an autonomous driving system: depth estimation, visual odometry, semantic segmentation, motion segmentation, object detection, and lens soiling detection. We demonstrate that the jointly trained model performs better than the respective single task versions. Our multi-task model has a shared encoder providing a significant computational advantage and has synergized decoders where tasks support each other. We propose a novel camera geometry based adaptation mechanism to encode the fisheye distortion model both at training and inference. This was crucial to enable training on the WoodScape dataset, comprised of data from different parts of the world collected by 12 different cameras mounted on three different cars with different intrinsics and viewpoints. Given that bounding boxes is not a good representation for distorted fisheye images, we also extend object detection to use a polygon with non-uniformly sampled vertices. We additionally evaluate our model on standard automotive datasets, namely KITTI and Cityscapes. We obtain the state-of-the-art results on KITTI for depth estimation and pose estimation tasks and competitive performance on the other tasks. We perform extensive ablation studies on various architecture choices and task weighting methodologies. A short video at https://youtu.be/xbSjZ5OfPes provides qualitative results.

show abstract

Section: B Generalized Object Detectionmentioning

confidence: 99%

OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving

Kumar

Yogamani

Rashed³

et al. 2021

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Motion segmentation employs two frames and predicts either a binary moving or static mask and employs Lovasz-Softmax [19], and Focal [20] loss for managing class imbalance instead of the cross-entropy loss. For object detection, we make use of YOLOv3 loss and add IoU loss using segmentation mask [21].…”

Section: Supervised Trainingmentioning

confidence: 99%

Adversarial Attacks on Multi-task Visual Perception for Autonomous Driving

Sobh¹,

Hamed²,

Kumar³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks (DNNs) have accomplished impressive success in various applications, including autonomous driving perception tasks, in recent years. On the other hand, current deep neural networks are easily fooled by adversarial attacks. This vulnerability raises significant concerns, particularly in safety-critical applications. As a result, research into attacking and defending DNNs has gained much coverage. In this work, detailed adversarial attacks are applied on a diverse multi-task visual perception deep network across distance estimation, semantic segmentation, motion detection, and object detection. The experiments consider both white and black box attacks for targeted and un-targeted cases, while attacking a task and inspecting the effect on all the others, in addition to inspecting the effect of applying a simple defense method. We conclude this paper by comparing and discussing the experimental results, proposing insights and future work. The visualizations of the attacks are available at https://youtu.be/R3JUV41aiPY.

show abstract

“…There is relatively less work on using convolutional neural networks for fisheye cameras. Recently, it has been explored for various tasks such as object detection [13], soiling detection [14], depth prediction [15], [16], [17], [18], SLAM [19] and in general for multi-task outputs [20]. As we shall see later, of particular interest for chargepad detection is the joint learning of object, depth, and semantic segmentation, which we shall deploy for the refinement of auto-annotation for online learning.…”

Section: Front Rightmentioning

confidence: 99%

“…Motion segmentation employs two frames and predicts either a binary moving or static mask and also employs Lovasz-Softmax and Focal [29] loss for managing class imbalance instead of the cross-entropy loss. For object detection, we make use of YOLOv3 loss [30] and add IoU loss using segmentation mask [13].…”

Section: B Near-field Perception Stack Through Multi-task Learning (Mtl)mentioning

confidence: 99%

An Online Learning System for Wireless Charging Alignment using Surround-view Fisheye Cameras

Dahal¹,

Kumar²,

Yogamani³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Generalized Object Detection on Fisheye Cameras for Autonomous Driving: Dataset, Representations and Baseline

Cited by 63 publications

References 21 publications

OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving

OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving

Adversarial Attacks on Multi-task Visual Perception for Autonomous Driving

An Online Learning System for Wireless Charging Alignment using Surround-view Fisheye Cameras

Contact Info

Product

Resources

About