Model-Based Reinforcement Learning for Time-Optimal Velocity Control

Hartmann, Gabriel; Shiller, Zvi; Azaria, Amos

doi:10.1109/lra.2020.3012128

Cited by 11 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the extensive application of RL in the fields of the transportation systems, the research on RL has become more and more in-depth in recent years. Hartmann et al have proposed a distributed predictive cruise control algorithm based on RL to reduce driving time and fuel consumption [17]. Haydari et al have pointed out that data-driven methodology will bring profound changes to its and play a key role in ITS in the future, have discussed in detail the application of deep reinforcement learning (DRL) in problems such as traffic signal control (TSC) and autonomous driving, and have pointed out that TSC can be abstracted into different problem formulas and involve different RL parameters [18].…”

Section: Reinforcement Learning In Path Planningmentioning

confidence: 99%

A Vehicle Path Planning Algorithm Based on Mixed Policy Gradient Actor‐Critic Model with Random Escape Term and Filter Optimization

Nai¹,

Yang²,

Lin³

et al. 2022

Journal of Mathematics

View full text Add to dashboard Cite

The transportation system of those countries has a huge traffic flow is bearing great pressure on transportation planning and management. Vehicle path planning is one of the effective ways to alleviate such pressure. Deep reinforcement learning (DRL), as a state-of-the-art solution method in vehicle path planning, can better balance the ability and complexity of the algorithm to reflect the real situation. However, DRL has its own disadvantages of higher search cost and earlier convergence to the local optimum, as vehicle path planning issues are usually in a complex environment, and their action set can be diverse. In this paper, a mixed policy gradient actor-critic (AC) model with random escape term and filter operation is proposed, in which the policy weight is both data driven and model driven. The empirical data-driven method is used to improve the poor asymptotic performance, and the model-driven method is used to ensure the convergence speed of the whole model. At the same time, in order to avoid the model converging local optimum, a random escape term has been added in the policy weight update process to overcome the problem that it is difficult to optimize the non-convex loss function, and the random escape term can help to explore the policy in more directions. In addition, filter optimization has been innovatively introduced in this paper, and the step size of each iteration of the model is selected through the filter optimization algorithm to achieve the better iterative effect. Numerical experiment results have shown that the model proposed in this paper can not only improve the accuracy of the solution without losing the accuracy but also speed up the convergence speed and improve the utilization of data.

show abstract

Section: Reinforcement Learning In Path Planningmentioning

confidence: 99%

A Vehicle Path Planning Algorithm Based on Mixed Policy Gradient Actor‐Critic Model with Random Escape Term and Filter Optimization

Nai¹,

Yang²,

Lin³

et al. 2022

Journal of Mathematics

View full text Add to dashboard Cite

show abstract

“…Reinforcement learning based methods have recently shown great success in many domains, including Atari games [17], Go [22], and autonomous vehicles [9,10,19,21]. Human-aided reinforcement learning introduces methods that enable the reinforcement learning agent to take advantage of human knowledge in order to learn more efficiently.…”

Section: Related Workmentioning

confidence: 99%

Criticality-Based Varying Step-Number Algorithm for Reinforcement Learning

Spielberg¹,

Azaria²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In the context of reinforcement learning we introduce the concept of criticality of a state, which indicates the extent to which the choice of action in that particular state influences the expected return. That is, a state in which the choice of action is more likely to influence the final outcome is considered as more critical than a state in which it is less likely to influence the final outcome.We formulate a criticality-based varying step number algorithm (CVS) -a flexible step number algorithm that utilizes the criticality function provided by a human, or learned directly from the environment. We test it in three different domains including the Atari Pong environment, Road-Tree environment, and Shooter environment. We demonstrate that CVS is able to outperform popular learning algorithms such as Deep Q-Learning and Monte Carlo.

show abstract

“…In time-optimal problems, we minimize the time to finish a lap, given an energy budget. It has been proven that these problems are solvable with a number of methods, among which are: reinforcement learning [22], yielding nearoptimal solutions; convex programming [23]- [25], in order to obtain the optimal control strategies; and by leveraging PMP, which can provide the control policies offline [26] and online [27], [28]. These methods are directly applicable to the energy-optimal time-constrained problem if time is properly constrained, as we will show in this paper.…”

Section: Introductionmentioning

confidence: 99%

Joint Fuel-optimal Control of the Velocity and Power-split of Hybrid Electric Vehicles

Borsboom,

Salazar,

Keulen

2024

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

This paper explores the eco-driving problem of parallel hybrid electric vehicles, intended to drive a certain distance within a limited amount of time, where the longitudinal vehicle velocity and powertrain controls are optimized to minimize the fuel consumption. In particular, we incorporate Pontryagin's Minimum Principle (PMP) and singular control theory in an optimization framework to find the fuel-optimal velocity and power-split control policy for the prime mover and the electric machine with global optimality guarantees. In addition, we present reformulations and derivations, so that the same problem can be solved jointly using another framework based on convex optimization, with the same global optimality properties, employing methods originally derived for timeoptimal control of race cars. Thereby, we formally show the equivalence between the eco-driving and the racing problem. We showcase both our frameworks with numerical solutions, drawing three comparisons: First, we solve the velocity and power-split problem, both sequentially and jointly, using the PMP framework. We show that the latter can improve the fuel consumption by 2.6 %. Second, we benchmark the PMP and the convex framework by solving the joint problem with both methods and observe a discrepancy of 0.14 % in terms of the resulting fuel energy consumption. Finally, in a numerical study addressing the performance of both methods individually, we observe that the efficiency of the PMP and the convex framework are strongly dependent on the stopping criteria and the discretization step size, respectively.

show abstract

Model-Based Reinforcement Learning for Time-Optimal Velocity Control

Cited by 11 publications

References 19 publications

A Vehicle Path Planning Algorithm Based on Mixed Policy Gradient Actor‐Critic Model with Random Escape Term and Filter Optimization

A Vehicle Path Planning Algorithm Based on Mixed Policy Gradient Actor‐Critic Model with Random Escape Term and Filter Optimization

Criticality-Based Varying Step-Number Algorithm for Reinforcement Learning

Joint Fuel-optimal Control of the Velocity and Power-split of Hybrid Electric Vehicles

Contact Info

Product

Resources

About