An Approximate Dynamic Programming Approach to Urban Freight Distribution with Batch Arrivals

Heeswijk, Wouter van; Mes, Martijn; Schutten, Johannes M.J.

doi:10.1007/978-3-319-24264-4_5

Cited by 7 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Van Heeswijk & La Poutré [22] compare centralized and decentralized transport for networks with fixed line transport services, concluding that decentralized planning yields considerable computational benefits. Van Heeswijk et al [23,24,26] study several variants of the DDP -including a routing component and a multi-agent setting -and use value function approximation to find strategies. Voccia et al [27] solve a variant that includes both pickups and deliveries.…”

Section: Literature Reviewmentioning

confidence: 99%

Strategic bidding in freight transport using deep reinforcement learning

Heeswijk¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents a multi-agent reinforcement learning algorithm to represent strategic bidding behavior in freight transport markets. Using this algorithm, we investigate whether feasible market equilibriums arise without any central control or communication between agents. Studying behavior in such environments may serve as a stepping stone towards self-organizing logistics systems like the Physical Internet. We model an agent-based environment in which a shipper and a carrier actively learn bidding strategies using policy gradient methods, posing bid-and ask prices at the individual container level. Both agents aim to learn the best response given the expected behavior of the opposing agent. A neutral broker allocates jobs based on bid-ask spreads.Our game-theoretical analysis and numerical experiments focus on behavioral insights. To evaluate system performance, we measure adherence to Nash equilibria, fairness of reward division and utilization of transport capacity. We observe good performance both in predictable, deterministic settings (∼95% adherence to Nash equilibria) and highly stochastic environments (∼85% adherence). Risk-seeking behavior may increase an agent's reward share, as long as the strategies are not overly aggressive. The results suggest a potential for full automation and decentralization of freight transport markets.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Strategic bidding in freight transport using deep reinforcement learning

Heeswijk¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We aim to provide a generic formulation, such that it can be applied to a variety of instances. For the notation of this problem, we expand upon our earlier work (Van Heeswijk et al 2015). We assume that the characteristics of arriving orders are stochastic and have a known associated probability distribution.…”

Section: Problem Formulationmentioning

confidence: 99%

The Delivery Dispatching Problem with Time Windows for Urban Consolidation Centers

Heeswijk

Mes

Schutten

2019

Transportation Science

View full text Add to dashboard Cite

This paper addresses the dispatch decision problem faced by an urban consolidation center. The center receives orders according to a stochastic arrival process, and dispatches them for the last-mile distribution in batches. The operator of the center aims to find the cost-minimizing consolidation policy, depending on the orders at hand, pre-announced orders, and stochastic arrivals. We present this problem as a variant of the Delivery Dispatching Problem that includes dispatch windows, and model it as a Markov decision problem. For toy-sized instances, we solve this model to optimality. Through numerical experiments on these instances, we show that we approximate the optimal values with an error of less than 2%. Larger instances suffer from intractably large state-, outcome-, and action spaces. We propose an Approximate Dynamic Programming (ADP) algorithm that can handle such instances, using value function approximation to estimate the downstream costs. To cope with large action spaces -with sizes up to 2 120 in our experiments -we formulate an integer linear program to be used within our ADP algorithm. To evaluate the performance of our ADP policies, we test against various benchmark policies, including a lookahead policy based on scenario sampling. We test the performance of ADP on a variety of networks. When the dispatching problem provides sufficient flexibility in dispatch times, ADP outperforms our myopic benchmark policies by more than 15%, and lookahead policies by over 10%. * Corresponding author. Address:

show abstract

“…In addition, we refer to Van Heeswijk et al (2015) for a similar ADP approach on a completely different transportation problem.…”

Section: Value Function Approximationmentioning

confidence: 99%