Real-time 3D Traffic Cone Detection for Autonomous Driving

Abstract

Considerable progress has been made in semantic scene understanding of road scenes with monocular cameras. It is, however, mainly focused on certain specific classes such as cars, bicyclists and pedestrians. This work investigates traffic cones, an object category crucial for traffic control in the context of autonomous vehicles. 3D object detection using images from a monocular camera is intrinsically an ill-posed problem. In this work, we exploit the unique structure of traffic cones and propose a pipelined approach to solve this problem. Specifically, we first detect cones in images by a modified 2D object detector. Following which the keypoints on a traffic cone are recognized with the help of our deep structural regression network, here, the fact that the cross-ratio is projection invariant is leveraged for network regularization. Finally, the 3D position of cones is recovered via the classical Perspective n-Point algorithm using correspondences obtained form the keypoint regression. Extensive experiments show that our approach can accurately detect traffic cones and estimate their position in the 3D world in real time. The proposed method is also deployed on a realtime, autonomous system. It runs efficiently on the low-power Jetson TX2, providing accurate 3D position estimates, allowing a race-car to map and drive autonomously on an unseen track indicated by traffic cones. With the help of robust and accurate perception, our race-car won both Formula Student Competitions held in Italy and Germany in 2018, cruising at a top-speed of 54 km/h on our driverless platform "gotthard driverless" https://youtu.be/HegmIXASKow?t=11694. Visualization of the complete pipeline, mapping and navigation can be found on our project page http://people.ee. ethz.ch/˜tracezuerich/TrafficCone/.

Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding

“…The last few years have also witnessed a leap in object recognition. A great deal of effort is made specifically in semantic road scene understanding [3,12,16]. However, the extension of these techniques to other weather/illumination conditions has not received due attention, despite its importance in outdoor applications.…”

AMZ Driverless: The full autonomous racing system

“…This paper presents a revision of our previous works (de la Iglesia Valls et al, 2018; Dhall, Dai, & Van Gool, 2019; Gosala et al, 2019) with extended experiments and results. In addition, it includes sections that had not been published previously, such a description of the full autonomous system, path planning (Section 4.3), control (Section 5) and the testing framework (Section 6) which is now being used by several other teams in the Formula Student community and in industry.…”

“…The ill‐posed problem can be solved by leveraging additional prior geometric information of the objects along with the 2D information obtained from the image. A “keypoint regression” scheme is used, that exploits this prior information about the object's shape and size to regress and find specific feature points on the image that match their 3D correspondences whose locations can be measured from a frame of reference (Dhall et al, 2019) as shown in Figure 13.…”

“…Schematic illustrating the matching of a 2D–3D correspondence and estimation of a transformation between the camera and world frames. With permission from Dhall et al (2019) [Color figure can be viewed at wileyonlinelibrary.com]…”

“…More recently, end‐to‐end pose estimation has been proposed where a six degree of freeedom (DoF) pose is the output. In our work, instead of directly producing the pose of an object from a CNN model, we use a CNN for locating the keypoints in the image (Dhall et al, 2019) and subsequently use information about the cone's appearance and geometry to estimate its pose via 2D–3D correspondences.…”