Feature Article—Merging AI and OR to Solve High-Dimensional Stochastic Optimization Problems Using Approximate Dynamic Programming

Abstract

W e consider the problem of optimizing over time hundreds or thousands of discrete entities that may be characterized by relatively complex attributes, in the presence of different forms of uncertainty. Such problems arise in a range of operational settings such as transportation and logistics, where the entities may be aircraft, locomotives, containers, or people. These problems can be formulated using dynamic programming but encounter the widely cited "curse of dimensionality." Even deterministic formulations of these problems can produce math programs with millions of rows, far beyond anything being solved today. This paper shows how we can combine concepts from artificial intelligence and operations research to produce practical solution methods that scale to industrial-strength problems. Throughout, we emphasize concepts, techniques, and notation from artificial intelligence and operations research to show how these fields can be brought together for complex stochastic, dynamic problems.

Policy Iteration Based on a Learned Transition Model

“…These two tasks are investigated experimentally in Section 5 below, using the primary and secondary intuitions just described. The second variety of tasks includes resource management scenarios, where stocks and flows must be controlled in order to maximize profits and minimize costs [13]. In these scenarios, aspects of the estimated state can be the levels of various resources, their prices, environmental conditions such as weather, and so on.…”

Rejoinder—The Languages of Stochastic Optimization

“…In operations research (and I believe that this is often true in AI), the state is generally viewed as a description of a snapshot of the "system," which might be the location of a vehicle, the amount in inventory, or the trajectory of a helicopter. In my single-trucker example in the paper in §3 (Powell 2010), it would include not only the position of the truck but also the additional information about loads to be moved. This illustrates that the state variable has to cover all the information you need to make a decision, a concept that appears to be well defined and understood in the systems and controls community.…”

Semiconvex Regression for Metamodeling-Based Optimization

“…In reinforcement learning, most action spaces are finite and small, so little attention has been given to maintaining convexity of Q(S, a) in a when the action space is continuous, vector-valued and convex. However, vector-valued action spaces can be handled by retaining convexity along with the use of the post-decision state variable, as illustrated in [53].…”