Reinforcement Learning with Iterative Reasoning for Merging in Dense Traffic

Bouton, Maxime; Nakhaei, Alireza; Isele, David; Fujimura, Kikuo; Kochenderfer, Mykel J.

doi:10.1109/itsc45102.2020.9294338

Cited by 35 publications

(19 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2). In line with previous works [20], [23], [34], we let ql-0 agents be reflective (nonstrategic) agents who do not explicitly take into account their opponents' possible responses but rather maximize their immediate rewards by treating other agents as stationary objects. The ground-truth reward functions of synthetic agents in each environment are manually tuned to achieve reasonable behaviors.…”

Section: Results and Analysismentioning

confidence: 99%

“…One of our approach's limitations is the ability to treat continuous states and actions. Fortunately, the procedure for computing π i,k can be realized in an iterative deep Q-learning fashion [36] and plugged into Algorithm 1 seamlessly. Another limitation is related to the assumption about humans' constant intelligence levels during interactions.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data

Tian¹,

Tomizuka²,

Sun³

2021

Preprint

View full text Add to dashboard Cite

Reward function, as an incentive representation that recognizes humans' agency and rationalizes humans' actions, is particularly appealing for modeling human behavior in human-robot interaction. Inverse Reinforcement Learning is an effective way to retrieve reward functions from demonstrations. However, it has always been challenging when applying it to multi-agent settings since the mutual influence between agents has to be appropriately modeled. To tackle this challenge, previous work either exploits equilibrium solution concepts by assuming humans as perfectly rational optimizers with unbounded intelligence or pre-assigns humans' interaction strategies a priori. In this work, we advocate that humans are bounded rational and have different intelligence levels when reasoning about others' decision-making process, and such an inherent and latent characteristic should be accounted for in reward learning algorithms. Hence, we exploit such insights from Theory-of-Mind and propose a new multi-agent Inverse Reinforcement Learning framework that reasons about humans' latent intelligence levels during learning. We validate our approach in both zero-sum and general-sum games with synthetic agents and illustrate a practical application to learning human drivers' reward functions from real driving data. We compare our approach with two baseline algorithms. The results show that by reasoning about humans' latent intelligence levels, the proposed approach has more flexibility and capability to retrieve reward functions that explain humans' driving behaviors better.

show abstract

Section: Results and Analysismentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data

Tian¹,

Tomizuka²,

Sun³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…V-H. To further increase safety, we believe that it is beneficial to make the vehicle able to deviate the route a bit in such dangerous situations. Second, this work assumes perfect measurements from a perception or V2X module like many previous works [30]- [32]. However, measurement noises are inevitable in the real world.…”

Section: Discussionmentioning

confidence: 99%

“…However, these work only considers static environments without interaction with other vehicles. By contrast, [20], [21], [30] and [31] considered dynamic traffic scenarios, thus they are more applicable for urban driving. However, current methods have not been well designed for scalable AD in a uniform setup.…”

Section: B Deep Reinforcement Learningmentioning

confidence: 99%

“…However, current methods have not been well designed for scalable AD in a uniform setup. Particularly, most works focus on specialized network design (e.g., input representations and reward) tailored for scattered traffic scenarios such as intersections [20], roundabouts [21], merging [30] and highway [32] scenarios, where one model can not generalize across different scenarios due to the varied requirements of observation space. For example, [32] used a list of vehicle state vectors such as position and velocity to depict the environment.…”

Section: B Deep Reinforcement Learningmentioning

confidence: 99%

See 1 more Smart Citation

DQ-GAT: Towards Safe and Efficient Autonomous Driving with Deep Q-Learning and Graph Attention Networks

Cai¹,

Wang²,

Sun³

et al. 2021

Preprint

View full text Add to dashboard Cite

Autonomous driving in multi-agent and dynamic traffic scenarios is challenging, where the behaviors of other road agents are uncertain and hard to model explicitly, and the egovehicle should apply complicated negotiation skills with them to achieve both safe and efficient driving in various settings, such as giving way, merging and taking turns. Traditional planning methods are largely rule-based and scale poorly in these complex dynamic scenarios, often leading to reactive or even overly conservative behaviors. Therefore, they require tedious human efforts to maintain workability. Recently, deep learning-based methods have shown promising results with better generalization capability but less hand engineering effort. However, they are either implemented with supervised imitation learning (IL) that suffers from the dataset bias and distribution mismatch problems, or trained with deep reinforcement learning (DRL) but focus on one specific traffic scenario. In this work, we propose DQ-GAT to achieve scalable and proactive autonomous driving, where graph attention-based networks are used to implicitly model interactions, and asynchronous deep Q-learning is employed to train the network end-to-end in an unsupervised manner. Extensive experiments through a high-fidelity driving simulation show that our method can better trade-off safety and efficiency in both seen and unseen scenarios, achieving higher goal success rates than the baselines (at most 4.7×) with comparable task completion time. Demonstration videos are available at https: //caipeide.github.io/dq-gat/.

show abstract