On Offline Evaluation of Vision-Based Driving Models

Codevilla, Felipe; López, Antonio; Koltun, Vladlen; Dosovitskiy, Alexey

doi:10.1007/978-3-030-01267-0_15

Cited by 85 publications

(78 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This proves that online test is the real significant indicator for IL when it is used for active control. Note that this is in line with the findings of [7], which highlights that the correlation between offline metrics and online performance is weak. Table 5: Comparison of MAE on train and validation data (in m), with none, partial and full data augmentation (None, Partial, Full), less is better Table 5 also shows that the error is greater for the neighbors than for the ego.…”

Section: Ablation Studiessupporting

confidence: 88%

Conditional Vehicle Trajectories Prediction in CARLA Urban Environment

Buhet

Wirbel

Perrotton

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Imitation learning is becoming more and more successful for autonomous driving. End-to-end (raw signal to command) performs well on relatively simple tasks (lane keeping and navigation). Mid-to-mid (environment abstraction to mid-level trajectory representation) or direct perception (raw signal to performance) approaches strive to handle more complex, real life environment and tasks (e.g. complex intersection). In this work, we show that complex urban situations can be handled with raw signal input and mid-level representation. We build a hybrid end-to-mid approach predicting trajectories for neighbor vehicles and for the ego vehicle with a conditional navigation goal. We propose an original architecture inspired from social pooling LSTM taking low and mid level data as input and producing trajectories as polynomials of time. We introduce a label augmentation mechanism to get the level of generalization that is required to control a vehicle. The performance is evaluated on CARLA 0.8 benchmark, showing significant improvements over previously published state of the art.

show abstract

Section: Ablation Studiessupporting

confidence: 88%

Conditional Vehicle Trajectories Prediction in CARLA Urban Environment

Buhet

Wirbel

Perrotton

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

show abstract

“…We validate every 20k iterations and if the validation error increases for three iterations we stop the training process and use this checkpoint to test on the benchmarks, both CARLA and NoCrash. We build a validation dataset as described in [9].…”

Section: Training Detailsmentioning

confidence: 99%

Exploring the Limitations of Behavior Cloning for Autonomous Driving

Codevilla

Santana

López³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

369

340

View full text Add to dashboard Cite

Figure 1. Driving scenarios from our new benchmark where the agent needs to react to dynamic changes in the environment, handle clutter (only part of the environment is causally relevant), and predict complex sensorimotor controls (lateral and longitudinal). We show that Behavior Cloning yields state-of-the-art policies in these complex scenarios and investigate its limitations. AbstractDriving requires reacting to a wide variety of complex environment conditions and agent behaviors. Explicitly modeling each possible scenario is unrealistic. In contrast, imitation learning can, in theory, leverage data from large fleets of human-driven cars. Behavior cloning in particular has been successfully used to learn simple visuomotor policies end-to-end, but scaling to the full spectrum of driving behaviors remains an unsolved problem. In this paper, we propose a new benchmark to experimentally investigate the scalability and limitations of behavior cloning. We show that behavior cloning leads to state-of-the-art results, including in unseen environments, executing complex lateral and longitudinal maneuvers without these reactions being explicitly programmed. However, we confirm well-known limitations (due to dataset bias and overfitting), new generalization issues (due to dynamic objects and the lack of a causal model), and training instability requiring further research before behavior cloning can graduate to real-world driving. We will release our benchmark and code.

show abstract

“…While imitation learning based approaches have shown important progress in autonomous driving [ 27 , 28 , 29 , 30 ], they present limitations when deployed in environments beyond the training distribution [ 31 ]. These driving models relying on supervised techniques are often evaluated on performance metrics on pre-collected validation datasets [ 32 ], however low prediction error on offline testing is not necessarily correlated with driving quality [ 33 ]. Even when demonstrating desirable performance during closed-loop testing in naturalistic driving scenarios, imitation learning models often degrade in performance due to distributional shift [ 26 ], unpredictable road users [ 34 ], or causal confusion [ 35 ] when exposed to a variety of driving scenarios.…”

Section: Related Workmentioning

confidence: 99%

Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages

Kuutti

Bowden

Fallah

2021

Sensors

View full text Add to dashboard Cite

The use of neural networks and reinforcement learning has become increasingly popular in autonomous vehicle control. However, the opaqueness of the resulting control policies presents a significant barrier to deploying neural network-based control in autonomous vehicles. In this paper, we present a reinforcement learning based approach to autonomous vehicle longitudinal control, where the rule-based safety cages provide enhanced safety for the vehicle as well as weak supervision to the reinforcement learning agent. By guiding the agent to meaningful states and actions, this weak supervision improves the convergence during training and enhances the safety of the final trained policy. This rule-based supervisory controller has the further advantage of being fully interpretable, thereby enabling traditional validation and verification approaches to ensure the safety of the vehicle. We compare models with and without safety cages, as well as models with optimal and constrained model parameters, and show that the weak supervision consistently improves the safety of exploration, speed of convergence, and model performance. Additionally, we show that when the model parameters are constrained or sub-optimal, the safety cages can enable a model to learn a safe driving policy even when the model could not be trained to drive through reinforcement learning alone.

show abstract

On Offline Evaluation of Vision-Based Driving Models

Cited by 85 publications

References 20 publications

Conditional Vehicle Trajectories Prediction in CARLA Urban Environment

Conditional Vehicle Trajectories Prediction in CARLA Urban Environment

Exploring the Limitations of Behavior Cloning for Autonomous Driving

Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages

Contact Info

Product

Resources

About