Offline–Online Approximate Dynamic Programming for Dynamic Vehicle Routing with Stochastic Requests

Abstract

Although increasing amounts of transaction data make it possible to characterize uncertainties surrounding customer service requests, few methods integrate predictive tools with prescriptive optimization procedures to meet growing demand for small-volume urban transport services. We incorporate temporal and spatial anticipation of service requests into approximate dynamic programming (ADP) procedures to yield dynamic routing policies for the single-vehicle routing problem with stochastic service requests, an important problem in city-based logistics. We contribute to the routing literature as well as to the field of ADP. We combine offline value function approximation (VFA) with online rollout algorithms resulting in a high-quality, computationally tractable policy. Our offline-online policy enhances the anticipation of the VFA policy, yielding spatial and temporal anticipation of requests and routing developments. Our combination of VFA with rollout algorithms demonstrates the potential benefit of using offline and online methods in tandem as a hybrid ADP procedure, making possible higher-quality policies with reduced computational requirements for real-time decision-making. Finally, we identify a policy improvement guarantee applicable to VFA-based rollout algorithms, showing that base policies composed of deterministic decision rules lead to rollout policies with performance at least as strong as that of their base policy.

Special Issue on Recent Advances in Urban Transport and Logistics Through Optimization and Analytics

“…Also on the topic of same-day delivery is "Offline-Online Approximate Dynamic Programming for Dynamic Vehicle Routing with Stochastic Requests." Ulmer et al (2019) focus on dynamic routing policies for a single-vehicle routing problem. The authors introduce temporal and spatial anticipation of service requests within an approximate dynamic programming approach.…”

Same‐day delivery with heterogeneous fleets of drones and vehicles

“…We use π PFA as base‐policy. As shown in Ulmer et al , given a sufficient number of samples, the RA performs at least as well as the base policy. The necessary number of samples though depends on the complexity of the problem.…”

“…For simple problems with low dimensional MDPs, a few simulations may be sufficient. RAs and related sampling methods in dynamic vehicle routing usually draw on a number of samples between 10 and 30 to achieve significant improvements .…”

Anticipation versus reactive reoptimization for dynamic vehicle routing with stochastic requests

“…The offline value evaluation could, therefore, not be transferred to the reoptimized online routes. The current state‐of‐the‐art heuristic for the VRPSR is presented by Ulmer et al combining online and offline simulations within a rollout algorithm (RA). We select this approach for our computational evaluation and describe the procedure in Section 3.2.…”