Understanding Failures of Deterministic Actor-Critic with Continuous Action Spaces and Sparse Rewards

Matheron, Guillaume; Perrin, Nicolás; Sigaud, Olivier

doi:10.1007/978-3-030-61616-8_25

Cited by 12 publications

(6 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both DDPG and TD3, despite their overall good behavior on average, still exhibit specific flaws in their optimization process. For instance, Matheron et al (2020) illustrated how sparse rewards in deterministic environments could prevent altogether the convergence of these methods by inducing value function plateaus and zero-gradient updates. A state-of-the-art alternative is SAC, which makes the actor policy stochastic (the behavior policy is then the actor policy), forces this policy to imitate the soft-max of (instead of the hard-max in DDPG and TD3), and introduces an entropy regularization term in the resolution of the Bellman equation to make the optimization landscape smoother (Geist et al, 2019).…”

Section: Reinforcement Learning For Airfoil Trajectory Controlmentioning

confidence: 99%

Reliability assessment of off-policy deep reinforcement learning: A benchmark for aerodynamics

Berger,

Arroyo Ramo,

Guillet

et al. 2024

DCE

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) is promising for solving control problems in fluid mechanics, but it is a new field with many open questions. Possibilities are numerous and guidelines are rare concerning the choice of algorithms or best formulations for a given problem. Besides, DRL algorithms learn a control policy by collecting samples from an environment, which may be very costly when used with Computational Fluid Dynamics (CFD) solvers. Algorithms must therefore minimize the number of samples required for learning (sample efficiency) and generate a usable policy from each training (reliability). This paper aims to (a) evaluate three existing algorithms (DDPG, TD3, and SAC) on a fluid mechanics problem with respect to reliability and sample efficiency across a range of training configurations, (b) establish a fluid mechanics benchmark of increasing data collection cost, and (c) provide practical guidelines and insights for the fluid dynamics practitioner. The benchmark consists in controlling an airfoil to reach a target. The problem is solved with either a low-cost low-order model or with a high-fidelity CFD approach. The study found that DDPG and TD3 have learning stability issues highly dependent on DRL hyperparameters and reward formulation, requiring therefore significant tuning. In contrast, SAC is shown to be both reliable and sample efficient across a wide range of parameter setups, making it well suited to solve fluid mechanics problems and set up new cases without tremendous effort. In particular, SAC is resistant to small replay buffers, which could be critical if full-flow fields were to be stored.

show abstract

Section: Reinforcement Learning For Airfoil Trajectory Controlmentioning

confidence: 99%

Reliability assessment of off-policy deep reinforcement learning: A benchmark for aerodynamics

Berger,

Arroyo Ramo,

Guillet

et al. 2024

DCE

View full text Add to dashboard Cite

show abstract

“…In a (D)RL framework, the representation of a reward function is critical, as it quantifies the value associated with each state and action pair and assists the agent in learning an optimal policy. [26] remarks that (D)RL with sparse rewards can lead to instability and suboptimal policy convergence. Likewise, each reward component should be weighted optimally in a multi-objective DRL agent to achieve the desired outcome and faster convergence.…”

Section: Related Workmentioning

confidence: 99%

An ML-Aided Reinforcement Learning Approach for Challenging Vehicle Maneuvers

Selvaraj

Hegde

Amati

et al. 2023

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

The richness of information generated by today's vehicles fosters the development of data-driven decision-making models, with the additional capability to account for the context in which vehicles operate. In this work, we focus on Adaptive Cruise Control (ACC) in the case of such challenging vehicle maneuvers as cut-in and cut-out, and leverages Deep Reinforcement Learning (DRL) and vehicle connectivity to develop a data-driven cooperative ACC application. Our DRL framework accounts for all the relevant factors, namely, passengers' safety and comfort as well as efficient road capacity usage, and it properly weights them through a two-layer learning approach. We evaluate and compare the performance of the proposed scheme against existing alternatives through the CoMoVe framework, which realistically represents vehicle dynamics, communication and traffic. The results, obtained in different real-world scenarios, show that our solution provides excellent vehicle stability, passengers' comfort, and traffic efficiency, and highlight the crucial role that vehicle connectivity can play in ACC. Notably, our DRL scheme improves the road usage efficiency by being inside the desired range of headway in cut-out and cut-in scenarios for 69% and 78% (resp.) of the time, whereas alternatives respect the desired range only for 15% and 45% (resp.) of the time. We also validate the proposed solution through a hardware-in-the-loop implementation, and demonstrate that it achieves similar performance to that obtained through the CoMoVe framework.

show abstract

“…They show that the episode reward of the EIDM‐guided DDPG converges to a steady value much faster than that of DDPG, which indicates that the EIDM‐guided DDPG can effectively improve the sample efficiency, training stability, and convergence of DDPG. The reason why DDPG performs poorly and cannot converge in a stable manner could be due to being stuck in a local optimum with poor solutions during the training (Matheron et al., 2019), which entails more human efforts to revise hyperparameters and DNN structures to improve the training performance. Figure 8c shows the episode rewards of the EIDM‐guided DDPG for the remaining three cases (i.e., two, three, and four preceding vehicles).…”

Section: Model Trainingmentioning

confidence: 99%

Cooperative control of a platoon of connected autonomous vehicles and unconnected human‐driven vehicles

Zhou

Peeta

Wang

2023

Computer aided Civil Eng

View full text Add to dashboard Cite

By using kinematic state information obtained through vehicle‐to‐vehicle communications, connected autonomous vehicles (CAVs) can drive cooperatively to alleviate shockwave propagation associated with traffic disturbances. However, during the transition to full autonomy, CAVs and human‐driven vehicles (HDVs) will coexist on the road, creating mixed‐flow traffic. The inherent heterogeneity and randomness in human driving behavior can generate additional disturbances in the traffic flow. Further, HDVs without communication functionality (unconnected HDVs) can cause the control performance of CAVs to degrade by negatively impacting platoon formation. To proactively mitigate the negative impacts of HDVs in mixed‐flow traffic, this study proposes a cooperative control strategy with three components for platoons consisting of CAVs and unconnected HDVs: (i) a number estimator for estimating the number of HDVs between two CAVs, (ii) a kinematic state predictor for predicting the kinematic states of HDVs, and (iii) a multi‐anticipative car‐following controller (i.e., control strategy using kinematic state information of multiple preceding vehicles) for CAVs to maintain string stability and desired time headway. To initialize the proposed strategy, the number estimator is developed using a deep neural network (DNN). Then, a DNN‐based kinematic state predictor predicts the kinematic states of HDVs for CAVs to enable multi‐anticipative car‐following control. The multi‐anticipative car‐following controller is implemented using an extended intelligent driver model‐guided deep deterministic policy gradient algorithm, which ensures safety, string stability, and traffic efficiency. The effectiveness of the proposed control strategy is validated through numerical experiments using NGSIM data. Results indicate that the proposed strategy can produce accurate estimations of the number and the kinematic states of HDVs between CAVs. Further, it can achieve string stability while maintaining smaller time headways, compared to car‐following models used for training guidance under different market penetration rates of CAVs, which significantly improves traffic smoothness and mobility.

show abstract

Understanding Failures of Deterministic Actor-Critic with Continuous Action Spaces and Sparse Rewards

Cited by 12 publications

References 2 publications

Reliability assessment of off-policy deep reinforcement learning: A benchmark for aerodynamics

Reliability assessment of off-policy deep reinforcement learning: A benchmark for aerodynamics

An ML-Aided Reinforcement Learning Approach for Challenging Vehicle Maneuvers

Cooperative control of a platoon of connected autonomous vehicles and unconnected human‐driven vehicles

Contact Info

Product

Resources

About