Learning to Drive from Simulation without Real World Labels

Bewley, Alex; Rigley, Jessica; Liu, Yuxuan; Hawke, Jeffrey; Shen, Richard; Lam, Vinh-Dieu; Kendall, Alex

doi:10.1109/icra.2019.8793668

Cited by 63 publications

(55 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The raw objects are represented by a list of points with global and local coordinates, normals, colors attributes and semantic labelsOther works synthesize challenges on CAD data by introducing noise simulated by Gaussians [4,12] or created with a parametic model [6]. Recently, the trend of sim2real [3] also aims to bridge the gap between synthetic and real data. As in the experiment with synthetic data, we sample all raw objects to 1024 points as input to the networks and all methods were trained using only the local (x, y, z) coordinates.…”

Section: Data Collectionmentioning

confidence: 99%

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Pham

Hua

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

407

319

View full text Add to dashboard Cite

Deep learning techniques for point cloud data have demonstrated great potentials in solving classical problems in 3D computer vision such as 3D object classification and segmentation. Several recent 3D object classification methods have reported state-of-the-art performance on CAD model datasets such as ModelNet40 with high accuracy (∼92%). Despite such impressive results, in this paper, we argue that object classification is still a challenging task when objects are framed with real-world settings. To prove this, we introduce ScanObjectNN, a new real-world point cloud object dataset based on scanned indoor scene data. From our comprehensive benchmark, we show that our dataset poses great challenges to existing point cloud classification techniques as objects from real-world scans are often cluttered with background and/or are partial due to occlusions. We identify three key open problems for point cloud object classification, and propose new point cloud classification neural networks that achieve state-of-the-art performance on classifying objects with cluttered background. Our dataset and code are publicly available in our project page 1 .

show abstract

Section: Data Collectionmentioning

confidence: 99%

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Pham

Hua

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

407

319

View full text Add to dashboard Cite

show abstract

“…Given visual observations of the environment (i.e., camera images), our system learns a lane-stable control policy This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/ over a wide variety of different road and environment types, as opposed to current end-to-end systems [2], [3], [8], [9] which only imitate human behavior. This is a major advancement as there does not currently exist a scalable method for training autonomous vehicle control policies that go beyond imitation learning and can generalize to and navigate in previously unseen road and complex, near-crash situations.…”

Section: Fig 1 Training and Deployment Of Policies From Data-driven Simulationmentioning

confidence: 99%

“…Training agents in simulation capable of robust generalization when deployed in the real world is a long-standing goal in many areas of robotics [9]- [12]. Several works have demonstrated transferable policy learning using domain randomization [13] or stochastic augmentation techniques [14] on smaller mobile robots.…”

Section: Related Workmentioning

confidence: 99%

“…Performing style transformation, such as adding realistic textures to synthetic images with deep generative models, has been used to deploy learned policies from model-based simulation engines into the real world [9], [24]. While these approaches can successfully transfer low-level details such as textures or sensory noise, these approaches are unable to transfer higher-level semantic complexities (such as vehicle or pedestrian behaviors) present in the real-world that are also required to train robust autonomous controllers.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning Robust Control Policies for End-to-End Autonomous Driving From Data-Driven Simulation

Amini

Gilitschenski

Phillips

et al. 2020

IEEE Robot. Autom. Lett.

152

View full text Add to dashboard Cite

In this work, we present a data-driven simulation and training engine capable of learning end-to-end autonomous vehicle control policies using only sparse rewards. By leveraging real, human-collected trajectories through an environment, we render novel training data that allows virtual agents to drive along a continuum of new local trajectories consistent with the road appearance and semantics, each with a different view of the scene. We demonstrate the ability of policies learned within our simulator to generalize to and navigate in previously unseen real-world roads, without access to any human control labels during training. Our results validate the learned policy onboard a full-scale autonomous vehicle, including in previously un-encountered scenarios, such as new roads and novel, complex, near-crash situations. Our methods are scalable, leverage reinforcement learning, and apply broadly to situations requiring effective perception and robust operation in the physical world.Index Terms-Deep learning in robotics and automation, autonomous agents, real world reinforcement learning, data-driven simulation. I. INTRODUCTIONE ND-TO-END (i.e., perception-to-control) trained neural networks for autonomous vehicles have shown great promise for lane stable driving [1]-[3]. However, they lack methods to learn robust models at scale and require vast amounts of training data that are time consuming and expensive to collect. Learned end-to-end driving policies and modular perception components in a driving pipeline require capturing training data from all necessary edge cases, such as recovery from off-orientation positions or even near collisions. This is not only prohibitively expensive, but also potentially dangerous [4]. Training and evaluating robotic controllers in simulation [5]-[7]

show abstract

“…Accordingly, the driving policy can be easily adapted to real environment. A method for transferring a vision-based lane following driving policy from a simulated to a real environment is presented in [40] where a model for end-to-end driving is constructed by learning to translate between simulated and real images, jointly learning a control policy from this common latent space using labels from an expert driver in the simulated environment. It was shown that the proposed system is capable of leveraging simulation to learn a driving policy to directly transfer to real world scenarios.…”

Section: Advanced Data Augmentationmentioning

confidence: 99%

Yes, we GAN: Applying adversarial techniques for autonomous driving

Uřičář¹,

Křížek²,

Hurych³

et al. 2019

View full text Add to dashboard Cite

Generative Adversarial Networks (GAN) have gained a lot of popularity from their introduction in 2014 till present. Research on GAN is rapidly growing and there are many variants of the original GAN focusing on various aspects of deep learning. GAN are perceived as the most impactful direction of machine learning in the last decade. This paper focuses on the application of GAN in autonomous driving including topics such as advanced data augmentation, loss function learning, semi-supervised learning, etc. We formalize and review key applications of adversarial techniques and discuss challenges and open problems to be addressed.

show abstract

Learning to Drive from Simulation without Real World Labels

Cited by 63 publications

References 31 publications

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Learning Robust Control Policies for End-to-End Autonomous Driving From Data-Driven Simulation

Yes, we GAN: Applying adversarial techniques for autonomous driving

Contact Info

Product

Resources

About