Deep Reinforcement Learning Approach to Solve Dynamic Vehicle Routing Problem with Stochastic Customers

Abstract

In real-world urban logistics operations, changes to the routes and tasks occur in response to dynamic events. To ensure customers' demands are met, planners need to make these changes quickly (sometimes instantaneously). This paper proposes the formulation of a dynamic vehicle routing problem with time windows and both known and stochastic customers as a route-based Markov Decision Process. We propose a solution approach that combines Deep Reinforcement Learning (specifically neural networks-based Temporal-Difference learning with experience replay) to approximate the value function and a routing heuristic based on Simulated Annealing, called DRLSA. Our approach enables optimized re-routing decision to be generated almost instantaneously. Furthermore, to exploit the structure of this problem, we propose a state representation based on the total cost of the remaining routes of the vehicles. We show that the cost of the remaining routes of vehicles can serve as proxy to the sequence of the routes and time window requirements. DRLSA is evaluated against the commonly used Approximate Value Iteration (AVI) and Multiple Scenario Approach (MSA). Our experiment results show that DRLSA can achieve on average, 10% improvement over myopic, outperforming AVI and MSA even with small training episodes on problems with degree of dynamism above 0.5.

Reinforcement Learning Approach to Solve Dynamic Bi-objective Police Patrol Dispatching and Rescheduling Problem

“…The same reward function and rescheduling heuristic are used in the experiment for fairer comparison. Here, we do not include the commonly-used online approaches to solve DVRP such as Multiple Scenario Approach (MSA) (Bent and Van Hentenryck 2004) as baseline because such sampling-based approaches takes longer computation time and may not be operationally suitable (see experiments done in Joe and Lau (2020)).…”

“…Recently, many works have emerged that proposed learning-based approaches to solve combinatorial optimization problems, including routing and scheduling problems. There also have been many recent works that addressed the dynamic variants of those problems Joe and Lau 2020;Li et al 2021;Chen, Ulmer, and Thomas 2022). Most of these works adopt a two-stage approach.…”

“…To learn the parameter θ, we propose the use of on-policy TD learning with experience replay. We refer our readers to Joe and Lau (2020) for the detailed description of the learning algorithm. We propose the use of VFA because the decision variable, x k is multi-dimensional.…”

Talking Trucks: Decentralized Collaborative Multi-Agent Order Scheduling for Self-Organizing Logistics

“…Gombolay et al (2018) highlight the value of incorporating human expertise and heuristics in RL performance on VRPs. Joe and Lau (2020) show that combining RL and a meta-heuristic is effective for centralized solving of dynamic VRP problems. In the SOL domain, Irannezhad, Prato, and Hickman (2020) successfully incorporate a multi-agent RL solution into a full-fledged port decision support system and evaluate agent collaboration strategies, showing that a cooperative strategy, rather than one focused on individual reward maximization, results in the highest overall vehicle utilization and lowest travel distance and costs.…”

An End-to-End Automatic Cache Replacement Policy Using Deep Reinforcement Learning

“…Because RL has the ability to adapt to dynamic changes in the workload (environment) and handle the non-trivial consequences of chosen policies (actions), it is a good fit for the problems encountered in this paper. We consider cache replacement as a decision-making problem for choosing different replacement policies given the corresponding workload distribution (Joe and Lau 2020). At the same time, to describe the differences between different workloads, we use a neural network (NN) to represent the diverse workload distributions.…”