Simulation to scaled city

Abstract

Using deep reinforcement learning, we successfully train a set of two autonomous vehicles to lead a fleet of vehicles onto a roundabout and then transfer this policy from simulation to a scaled city without fine-tuning. We use Flow, a library for deep reinforcement learning in microsimulators, to train two policies, (1) a policy with noise injected into the state and action space and (2) a policy without any injected noise. In simulation, the autonomous vehicles learn an emergent metering behavior for both policies which allows smooth merging. We then directly transfer this policy without any tuning to the University of Delaware's Scaled Smart City (UDSSC), a 1:25 scale testbed for connected and automated vehicles. We characterize the performance of the transferred policy based on how thoroughly the ramp metering behavior is captured in UDSSC. We show that the noise-free policy results in severe slowdowns and only, occasionally, it exhibits acceptable metering behavior. On the other hand, the noise-injected policy consistently performs an acceptable metering behavior, implying that the noise eventually aids with the zero-shot policy transfer. Finally, the transferred, noise-injected policy leads to a 5% reduction of average travel time and a reduction of 22% in maximum travel time in the UDSSC. Videos of the proposed self-learning controllers can be found at https://sites.google.com/view/iccps-policy-transfer.

Separation of learning and control for cyber–physical systems

“…We seek to derive the optimal control strategy g * ∈ G of the system represented in (17) which minimizes the following expected cost:…”

“…If the dynamics of the actual system given in (17) were known, then we could use (24) to derive the optimal control strategy g * ∈ G. Since the primitive random variables are Gaussian with zero mean, variance 1, and covariance 0.5, we can use the linear least-squares estimator to compute the unique optimal solution of (24), which is…”

“…In this section, we consider that the dynamics of the actual system (17) are not known, but we have the model (22) of the system available. Using this model, we can obtain the optimal control strategy by applying the framework presented in Section III.…”

“…There have also been research efforts on developing learning approaches using Bayesian analysis to address such problems [4]. Other approaches over the years have focused on direct or indirect RL methods including robust learning-based [5], [6], learning-based model predictive control [7]- [9] on autonomous racing cars [10], real-time learning [11], [12] of powertain systems with respect to the driver's driving style [13], [14], learning for traffic control [15] for transferring optimal policies [16], [17], decentralized learning for stochastic games [18], learning for optimal social routing [19] and congestion games [20], and learning for enhanced security against replay attacks in cyber-physical systems [21], [22].…”

Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies

“…While testing single-agent AVs is challenging, the problem is even more complex in the presence of other agents [6]. AV testing in a multi-agent setting is a complex problem since it is hard to find which states of interactions will most likely lead them to fail.…”