Learning Highway Ramp Merging Via Reinforcement Learning with Temporally-Extended Actions

Triest, Samuel; Villaflor, Adam; Dolan, John M.

doi:10.1109/iv47402.2020.9304841

Cited by 34 publications

(12 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lubars et al [10] tackle the on-ramp merging problem through leveraging on MPC and RL and demonstrate promising performance. In [11], a hierarchical approach is proposed to address the decision-making problem in the on-ramp merging scenario, in which a high-level decision is produced by the RL agent and actual control is executed by the low-level controller. A cooperation-aware framework is proposed in [12], where the RL agent learns to interact with the road users and different cooperation levels are considered during training.…”

Section: Human Controlled Vehicle Ego Vehiclementioning

confidence: 99%

See 1 more Smart Citation

Autonomous Highway Merging in Mixed Traffic Using Reinforcement Learning and Motion Predictive Safety Controller

Liu¹,

Dang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) has a great potential for solving complex decision-making problems in autonomous driving, especially in mixed-traffic scenarios where autonomous vehicles and human-driven vehicles (HDVs) drive together. Safety is a key during both the learning and deploying reinforcement learning (RL) algorithms process. In this paper, we formulate the on-ramp merging as a Markov Decision Process (MDP) problem and solve it with an off-policy RL algorithm, i.e., Soft Actor-Critic for Discrete Action Settings (SAC-Discrete). In addition, a motion predictive safety controller including a motion predictor and an action substitution module, is proposed to ensure driving safety during both training and testing. The motion predictor estimates the trajectories of the ego vehicle and surrounding vehicles from kinematic models, and predicts potential collisions. The action substitution module replaces risky actions based on safety distance, before sending them to the low-level controller. We train, evaluate and test our approach on a gym-like highway simulator with three different levels of traffic modes. The simulation results show that even in harder traffic densities, the proposed method still significantly reduces collision rate while maintaining high efficiency, outperforming several state-of-the-art baselines in the considered on-ramp merging scenarios. The video demo of the evaluation process can be found at: https://www.youtub e.com/watch?v=7FvjbAM4oFw

show abstract

Section: Human Controlled Vehicle Ego Vehiclementioning

confidence: 99%

“…A cooperation-aware framework is proposed in [12], where the RL agent learns to interact with the road users and different cooperation levels are considered during training. However, most of them consider the ego vehicle to be spawned only on the through lane [11] or on the merging lane [13].…”

Section: Human Controlled Vehicle Ego Vehiclementioning

confidence: 99%

Autonomous Highway Merging in Mixed Traffic Using Reinforcement Learning and Motion Predictive Safety Controller

Liu¹,

Dang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This method can reduce the congestion caused by the merging vehicle as well as the average fuel consumption. Treist et al [5] formulated a Partially Observable MDP (POMDP) and proposes a high level decision making using Advantage Actor Critic (A2C) to control low level controllers. However, the above methods formulated RL agents using a single modality, which may generate unreliable merging maneuvers under noisy environment observations, either noisy image or noisy BSM data.…”

Section: Related Workmentioning

confidence: 99%

“…On the other hand, the training modality for on-ramp merging is another critical consideration. Most existing merging algorithms use a single-modality [5]- [7] for making decisions, which is unreliable in certain conditions [8], [9]. For example, single-modality systems have a low accuracy when driving in adverse weather conditions, such as snow and fog [9].…”

Section: Introductionmentioning

confidence: 99%

Towards Robust On-Ramp Merging via Augmented Multimodal Reinforcement Learning

Bagwe¹,

Li²,

Yuan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Despite the success of AI-enabled onboard perception, on-ramp merging has been one of the main challenges for autonomous driving. Due to limited sensing range of onboard sensors, a merging vehicle can hardly observe main road conditions and merge properly. By leveraging the wireless communications between connected and automated vehicles (CAVs), a merging CAV has potential to proactively obtain the intentions of nearby vehicles. However, CAVs can be prone to inaccurate observations, such as the noisy basic safety messages (BSM) and poor quality surveillance images. In this paper, we present a novel approach for Robust on-ramp merge of CAVs via Augmented and Multi-modal Reinforcement Learning, named by RAMRL. Specifically, we formulate the on-ramp merging problem as a Markov decision process (MDP) by taking driving safety, comfort driving behavior, and traffic efficiency into account. To provide reliable merging maneuvers, we simultaneously leverage BSM and surveillance images for multi-modal observation, which is used to learn a policy model through proximal policy optimization (PPO). Moreover, to improve data efficiency and provide better generalization performance, we train the policy model with augmented data (e.g., noisy BSM and noisy surveillance images). Extensive experiments are conducted with Simulation of Urban MObility (SUMO) platform under two typical merging scenarios. Experimental results demonstrate the effectiveness and efficiency of our robust on-ramp merging design.

show abstract

“…In an interactive traffic scenario, the driving environment is highly cooperative and dynamic, and the mutual effect between different traffic participants has a great influence on decision-making. [1]. It is critically important for each autonomous vehicle in interactive traffic scenarios to generate appropriate and cooperative behaviors.…”

Section: Introductionmentioning

confidence: 99%

Graph Convolution-Based Deep Reinforcement Learning for Multi-Agent Decision-Making in Mixed Traffic Environments

Liu¹,

Li²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

An efficient and reliable multi-agent decisionmaking system is highly demanded for the safe and efficient operation of connected autonomous vehicles in intelligent transportation systems. Current researches mainly focus on the Deep Reinforcement Learning (DRL) methods. However, utilizing DRL methods in interactive traffic scenarios is hard to represent the mutual effects between different vehicles and model the dynamic traffic environments due to the lack of interactive information in the representation of the environments, which results in low accuracy of cooperative decisions generation. To tackle these difficulties, this research proposes a framework to enable different Graph Reinforcement Learning (GRL) methods for decision-making, and compares their performance in interactive driving scenarios. GRL methods combinate the Graph Neural Network (GNN) and DRL to achieve the better decisions generation in interactive scenarios of autonomous vehicles, where the features of interactive scenarios are extracted by the GNN, and cooperative behaviors are generated by DRL framework. Several GRL approaches are summarized and implemented in the proposed framework. To evaluate the performance of the proposed GRL methods, an interactive driving scenarios on highway with two ramps is constructed, and simulated experiment in the SUMO platform is carried out to evaluate the performance of different GRL approaches. Finally, results are analyzed in multiple perspectives and dimensions to compare the characteristic of different GRL approaches in intelligent transportation scenarios. Results show that the implementation of GNN can well represents the interaction between vehicles, and the combination of GNN and DRL is able to improve the performance of the generation of lanechange behaviors. The source code of our work can be found at https://github.com/Jacklinkk/TorchGRL.

show abstract

Learning Highway Ramp Merging Via Reinforcement Learning with Temporally-Extended Actions

Cited by 34 publications

References 12 publications

Autonomous Highway Merging in Mixed Traffic Using Reinforcement Learning and Motion Predictive Safety Controller

Autonomous Highway Merging in Mixed Traffic Using Reinforcement Learning and Motion Predictive Safety Controller

Towards Robust On-Ramp Merging via Augmented Multimodal Reinforcement Learning

Graph Convolution-Based Deep Reinforcement Learning for Multi-Agent Decision-Making in Mixed Traffic Environments

Contact Info

Product

Resources

About