Graph Neural Network Reinforcement Learning for Autonomous Mobility-on-Demand Systems

Abstract

Optimization problems over dynamic networks have been extensively studied and widely used in the past decades to formulate numerous real-world problems. However, (1) traditional optimization-based approaches do not scale to large networks, and (2) the design of good heuristics or approximation algorithms often requires significant manual trial-and-error. In this work, we argue that data-driven strategies can automate this process and learn efficient algorithms without compromising optimality. To do so, we present network control problems through the lens of reinforcement learning and propose a graph networkbased framework to handle a broad class of problems. Instead of naively computing actions over high-dimensional graph elements, e.g., edges, we propose a bi-level formulation where we (1) specify a desired next state via RL, and (2) solve a convex program to best achieve it, leading to drastically improved scalability and performance. We further highlight a collection of desirable features to system designers, investigate design decisions, and present experiments on real-world control problems showing the utility, scalability, and flexibility of our framework.

Graph Meta-Reinforcement Learning for Transferable Autonomous Mobility-on-Demand

“…A Three-Step Framework. As in [10,15], we formulate the AMoD control problem as a three-step decision-making framework, whereby the control of an AMoD fleet follows three steps: matching, RL for rebalancing, and post-processing. This three-step framework, as shown in the rest of this section, has the advantage of reducing the action space from 𝑁 2 𝑣 to 𝑁 𝑣 , since the learned policy defines an action at each node as opposed to along each OD pair (as in the majority of literature).…”

“…Related work. Within existing literature, AMoD systems can be coordinated with simple rule-based heuristics [6,7], model predictive control (MPC) approaches [8,9], and RL approaches [10][11][12][13][14][15]. Interested readers can refer to [16] for a comprehensive survey of the analysis and control of AMoD systems and [17] for RL approaches for ridesharing systems.…”

“…However, the learned agents can only take reactive decisions based on the current vehicle distribution, hence may not perform well in dynamic scenarios with time-varying travel demand and traffic conditions. Gammelli et al [15] developed a deep RL approach based on graph neural networks to exploit the connectivity embedded in the transportation network to learn a policy that is scalable and near-optimal. Overall, the aforementioned RL approaches can ensure computational tractability without significantly compromising optimality.…”

“…However, to the best of our knowledge, existing RL approaches are limited to the single-city scenario, wherein an RL agent is trained in a single transportation network or geographical region. Although transferability across cities is discussed through experiments in several works (e.g., [14,15]), there lacks a systematic and formalized methodology that can explicitly train controllers capable of adapting to new cities. Several works employ meta-learning to develop prediction models for traffic states [18] and travel demand [19] that are adaptable to multiple cities, however, these works solely develop predictive models without considering the adaptation of the control strategy, and it is not clear how the insights can be applied to coordination algorithms of AMoD fleets.…”

A Robust and Constrained Multi-Agent Reinforcement Learning Framework for Electric Vehicle AMoD Systems

“…Electric vehicles (EVs) are being adopted worldwide for environmental and economical benefits [4], and AMoD systems embrace this trend without exception. However, the trips sporadically appear, and the origins and destinations are asymmetrically distributed and hard to predict in AMoD systems [3], [5]. Such spatial-temporal nature of urban mobility increases the management difficulty of a large-scale vehicle fleet and makes the system sensitive to disturbances [5]- [7].…”