Simulation-Based Approximate Policy Iteration with Generalized Logistic Functions

Abstract

W e present an approximate dynamic programming method based on simulation, policy iteration, a postdecision state formulation, and a logistic value function approximation. This method was developed as part of our efforts to determine whether nonlinear value function approximations could provide cost-effective policies for advance patient scheduling problems, and as a way of identifying the main advantages and disadvantages of using simulation versus linear programming to approximately solve dynamic capacity allocation problems. We first apply the proposed method to a queueing problem and then study a more practical application based on an advance multipriority patient scheduling problem. We investigate the quality and practical implications of the resulting appointment scheduling policies using simulation, and compare their performance to that of four other policies. Patient scheduling policies obtained by the new method not only depend on the number of appointments already booked on each day but also on the overall system workload. In particular, these policies provide lower discounted cost values and shorter average wait times for higher priority patients than policies directly obtained using linear programming and an affine value function approximation in the predecision state variables.

An approximate dynamic programming approach for production‐delivery scheduling under non‐stationary demand

“…We set K to be 10 times the dimension of parameter 𝜽 t , due to the fact that a good rule of thumb in regression analysis is to have a minimum of 30 observations plus at least 10 observations for each independent variable (Sauré et al, 2015).…”

Dynamic multi-priority, multi-class patient scheduling with stochastic service times

“…Papers in the area of advance scheduling mostly use dynamic programming or approximate dynamic programming due to the stochastic nature of the appointment request arrivals and the sequential nature of the decision process (Patrick et al, 2008;Sauré et al, 2012;Kolisch, 2012, 2013;Sauré et al, 2015;Truong, 2015). Some of the objectives considered in these studies are: maximizing the number of patients booked within their medically acceptable wait times (Patrick et al, 2008;Sauré et al, 2012;Sauré et al, 2015), maximizing revenue (Gupta and Denton, 2008;Schütz and Kolisch, 2013), improving resource utilization (Santibáñez et al, 2009), satisfying specific appointment date windows (Gocgun and Puterman, 2014), taking patient preferences into account (Gupta and Denton, 2008;Wang and Gupta, 2011;Feldman et al, 2014), and reducing wait times (Green et al, 2006). Application areas of advance scheduling include the scheduling of diagnostic tests such as MRI/CT scans (Green et al, 2006;Patrick et al, 2008;Schütz and Kolisch, 2012), radiation therapy treatment scheduling (Sauré et al, 2012), primary care clinics (Green and Savin, 2008;Dobson et al, 2011;Liu, 2016), and surgical scheduling (Astaraky and Patrick, 2015).…”

“…There are two main options in seeking to solve an MDP model using ADP. Simulation-based ADP iteratively produces simulated runs of the model to approximate the value function at a subset of initial states while using a form of least squares regression or a recursive update function to converge on a good approximation (e.g., see Sauré et al, 2015). The other option is to transform the MDP model into an equivalent linear program and then substitute into it the value function approximation of choice.…”

“…tributable to either the use of an a ne approximation architecture (possibly curtailing the advantage of incorporating stochastic service times) or the fact that we concentrated our e↵orts on a setting where within-day waiting/idle time was of little consequence. Thus, two further avenues of research are the implementation of a non-linear approximation architecture similar to Sauré et al (2015) as well as the incorporation of the sequencing of patients into the MDP model in order to address scenarios where the within-day waiting/idle time is relevant.…”

An aggregation-based approximate dynamic programming approach for the periodic review model with random yield

“…In this way, we approximate the cost difference myopically. Myopic policies that are based on cost estimates of few periods are frequently used in the approximate dynamic programming literature (e.g., Abdulwahab and Wahab 2014, Fang et al 2013, Sauré et al 2015. We use cost estimates not for a myopic policy itself, but for estimating the cost difference.…”