Discretionary Lane Change Decision Making using Reinforcement Learning with Model-Based Exploration

Zhang, Songan; Peng, Huei; Nageshrao, Subramanya; Tseng, Eric

doi:10.1109/icmla.2019.00147

Cited by 18 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the exploration of the environment is not likely to exhaust all states though the action policy contains random actions or noise [44,45], which leads to the global optimum may being buried in the quagmire. In particular, the proposed communication scenarios and indoor layouts have high complexity, impelling the efficient and sufficient exploration to be a problem.…”

Section: Federated Learning Modelmentioning

confidence: 99%

Mobile Reconfigurable Intelligent Surfaces for NOMA Networks: Federated Learning Approaches

Zhong¹,

Liu²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

A novel framework of reconfigurable intelligent surfaces (RISs)-enhanced indoor wireless networks is proposed, where an RIS mounted on the robot is invoked to enable mobility of the RIS and enhance the service quality for mobile users. Meanwhile, non-orthogonal multiple access (NOMA) techniques are adopted to further increase the spectrum efficiency since RISs are capable to provide NOMA with artificial controlled channel conditions, which can be seen as a beneficial operation condition to obtain NOMA gains. To optimize the sum rate of all users, a deep deterministic policy gradient (DDPG) algorithm is invoked to optimize the deployment and phase shifts of the mobile RIS as well as the power allocation policy. In order to improve the efficiency and effectiveness of agent training for the DDPG agents, a federated learning (FL) concept is adopted to enable multiple agents to simultaneously explore similar environments and exchange experiences. We also proved that with the same random exploring policy, the FL armed deep reinforcement learning (DRL) agents can theoretically obtain a reward gain compare to the independent agents. Our simulation results indicate that the mobile RIS scheme can significantly outperform the fixed RIS paradigm, which provides about three times data rate gain compare to the fixed RIS paradigm. Moreover, the NOMA scheme is capable to achieve a gain of 42% in contrast with the OMA scheme in terms of sum rate. Finally, the multi-cell simulation proved that the FL enhanced DDPG algorithm has a superior convergence rate and optimization performance than the independent training framework.Index terms-Deep reinforcement learning (DRL), federated learning (FL), intelligent reflecting surfaces (IRSs), non-orthogonal multiple access (NOMA), reconfigurable intelligent surfaces (RIS), resource management

show abstract

Section: Federated Learning Modelmentioning

confidence: 99%

Mobile Reconfigurable Intelligent Surfaces for NOMA Networks: Federated Learning Approaches

Zhong¹,

Liu²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For evaluating the performance of trained meta agents, we compare the PEARL trained agent and the MAML trained agent with a fine-tune method based on the Trust Region Policy Optimization (TRPO) [35] method with safety check implemented from [36]. The fine-tune method will just keep updating the initial policy in a new environment given collected data.…”

Section: B Varying Social Behaviors From Egoistic To Altruisticmentioning

confidence: 99%

“…Therefore, we will also report the crash rate of the fine-tune agent, MAML agent, and the PEARL agent in Section VI. We will compare the crash rate with the benchmark policy trained in the original environment using the method designed in [36].…”

Section: B Varying Social Behaviors From Egoistic To Altruisticmentioning

confidence: 99%

Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning

Zhang¹,

Wen²,

Peng³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

It is essential for an automated vehicle in the field to perform discretionary lane changes with appropriate roadmanship -driving safely and efficiently without annoying or endangering other road users -under a wide range of traffic cultures and driving conditions. While deep reinforcement learning methods have excelled in recent years and been applied to automated vehicle driving policy, there are concerns about their capability to quickly adapt to unseen traffic with new environment dynamics. We formulate this challenge as a multi-Markov Decision Processes (MDPs) adaptation problem and developed Meta Reinforcement Learning (MRL) driving policies to showcase their quick learning capability. Two types of distribution variation in environments were designed and simulated to validate the fast adaptation capability of resulting MRL driving policies which significantly outperform a baseline RL.

show abstract

“…Pan et al (2017) built a new framework on the basis of A3C to train a self-driving vehicle by interacting with a synthesized real environment [7]. Recently, Zhang et al (2019) employed RL with modelbased exploration for autonomous lane change decisionmaking on highways [8]. Duan et al (2020) employed RL with a hierarchical architecture for autonomous decisionmaking on highways [9] [10].…”

Section: Introductionmentioning

confidence: 99%

Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization

Wen

Duan

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) is attracting increasing interests in autonomous driving due to its potential to solve complex classification and control problems. However, existing RL algorithms are rarely applied to real vehicles for two predominant problems: behaviors are unexplainable, and they cannot guarantee safety under new scenarios. This paper presents a safe RL algorithm, called Parallel Constrained Policy Optimization (PCPO), for two autonomous driving tasks. PCPO extends today's common actor-critic architecture to a threecomponent learning framework, in which three neural networks are used to approximate the policy function, value function and a newly added risk function, respectively. Meanwhile, a trust region constraint is added to allow large update steps without breaking the monotonic improvement condition. To ensure the feasibility of safety constrained problems, synchronized parallel learners are employed to explore different state spaces, which accelerates learning and policy-update. The simulations of two scenarios for autonomous vehicles confirm we can ensure safety while achieving fast learning.

show abstract

Discretionary Lane Change Decision Making using Reinforcement Learning with Model-Based Exploration

Cited by 18 publications

References 10 publications

Mobile Reconfigurable Intelligent Surfaces for NOMA Networks: Federated Learning Approaches

Mobile Reconfigurable Intelligent Surfaces for NOMA Networks: Federated Learning Approaches

Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning

Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization

Contact Info

Product

Resources

About