Mixed Reinforcement Learning for Efficient Policy Optimization in Stochastic Environments

Mu, Yao; Peng, Baiyu; Gu, Ziqing; Li, Shengbo Eben; Liu, Chang; Nie, Bingbing; Zheng, Jianfeng; Zhang, Bo

doi:10.23919/iccas50221.2020.9268413

Cited by 8 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the minimum return to work in the task, we first choose a tolerable value of each state during an episode, i.e., x = 2, θ = 0.1, ẋ = 0.1, θ = 0.05, to compute a tolerable reward using (18). Then the minimum return equals the reward times the episode length 100.…”

Section: B Inverted Pendulum Taskmentioning

confidence: 99%

“…To address this problem, Parmas et al (2019) developed a total propagation algorithm that introduces an additional PG estimator with the likelihood ratio trick and automatically gives greater weight to estimators with lower variance [17]. The second drawback is that the PG from BPTT is sensitive to model errors, especially for a long prediction-horizon [18]. This problem has also been shown in [16], in which a value function is integrated after certain forward steps driven by the model to increase robustness to model error and extend BPTT to infinite-horizon control.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Mixed Policy Gradient

Guan,

Duan,

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) has great potential in sequential decision-making. At present, the mainstream RL algorithms are data-driven, relying on millions of iterations and a large number of empirical data to learn a policy. Although data-driven RL may have excellent asymptotic performance, it usually yields slow convergence speed. As a comparison, modeldriven RL employs a differentiable transition model to improve convergence speed, in which the policy gradient (PG) is calculated by using the backpropagation through time (BPTT) technique. However, such methods suffer from numerical instability, model error sensitivity and low computing efficiency, which may lead to poor policies. In this paper, a mixed policy gradient (MPG) method is proposed, which uses both empirical data and the transition model to construct the PG, so as to accelerate the convergence speed without losing the optimality guarantee. MPG contains two types of PG: 1) data-driven PG, which is obtained by directly calculating the derivative of the learned Q-value function with respect to actions, and 2) model-driven PG, which is calculated using BPTT based on the model-predictive return. We unify them by revealing the correlation between the upper bound of the unified PG error and the predictive horizon, where the data-driven PG is regraded as 0-step model-predictive return. Relying on that, MPG employs a rule-based method to adaptively adjust the weights of data-driven and model-driven PGs. In particular, to get a more accurate PG, the weight of the data-driven PG is designed to grow along the learning process while the other to decrease. Besides, an asynchronous learning framework is proposed to reduce the wall-clock time needed for each update iteration. Simulation results show that the MPG method achieves the best asymptotic performance and convergence speed compared with other baseline algorithms.

show abstract

Section: B Inverted Pendulum Taskmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Mixed Policy Gradient

Guan,

Duan,

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…As a self-learning method, RL is promising to reduce the massive engineering efforts in autonomous driving. In recent years, there has been a growing interest towards RL in autonomous driving community, such as adaptive cruise control [6], lane-keeping [7], trajectory tracking [8] and multi-vehicle cooperation [9]. However, despite achieving decent performance, these RL methods mostly lack explicit safety constraints, which significantly limits their application in safety-critical autonomous driving.…”

Section: Introductionmentioning

confidence: 99%

Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

Peng¹,

Mu²,

Duan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Safety is essential for reinforcement learning (RL) applied in real-world tasks like autonomous driving. Chance constraints which guarantee the satisfaction of state constraints at a high probability are suitable to represent the requirements in real-world environment with uncertainty. Existing chance constrained RL methods like the penalty method and the Lagrangian method either exhibit periodic oscillations or cannot satisfy the constraints. In this paper, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. Taking a control perspective, we first interpret the penalty method and the Lagrangian method as proportional feedback and integral feedback control, respectively. Then, a proportional-integral Lagrangian method is proposed to steady learning process while improving safety. To prevent integral overshooting and reduce conservatism, we introduce the integral separation technique inspired by PID control. Finally, an analytical gradient of the chance constraint is utilized for model-based policy optimization. The effectiveness of SPIL is demonstrated by a narrow car-following task. Experiments indicate that compared with previous methods, SPIL improves the performance while guaranteeing safety, with a steady learning process.

show abstract

“…With the development of artificial intelligence technologies, autonomous driving has become an important tendency in the automotive industry for its potential to improve road safety, reduce fuel consumption, and improve traffic efficiency [1], [2]. There are two kinds of schemes widely employed by the autonomous driving system: hierarchical scheme [3], [4], [5], [6] and end-to-end scheme [7], [8], [9]. The hierarchical scheme divides the entire autonomous driving system into several modules, including environment perception, decision-making, and motion control [3], [4], [10].…”

Section: Introductionmentioning

confidence: 99%

Steadily Learn to Drive with Virtual Memory

Zhang¹,

Mu²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning has shown great potential in developing high-level autonomous driving. However, for highdimensional tasks, current RL methods suffer from low data efficiency and oscillation in the training process. This paper proposes an algorithm called Learn to drive with Virtual Memory (LVM) to overcome these problems. LVM compresses the high-dimensional information into compact latent states and learns a latent dynamic model to summarize the agent's experience. Various imagined latent trajectories are generated as virtual memory by the latent dynamic model. The policy is learned by propagating gradient through the learned latent model with the imagined latent trajectories and thus leads to high data efficiency. Furthermore, a double critic structure is designed to reduce the oscillation during the training process. The effectiveness of LVM is demonstrated by an image-input autonomous driving task, in which LVM outperforms the existing method in terms of data efficiency, learning stability, and control performance.

show abstract

Mixed Reinforcement Learning for Efficient Policy Optimization in Stochastic Environments

Cited by 8 publications

References 11 publications

Mixed Policy Gradient

Mixed Policy Gradient

Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

Steadily Learn to Drive with Virtual Memory

Contact Info

Product

Resources

About