Exploiting structure in adaptive dynamic programming algorithms for a stochastic batch service problem

Papadaki, Katerina; Powell, Warren B.

doi:10.1016/s0377-2217(01)00297-1

Cited by 44 publications

(22 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The optimality of monotone policies has been broadly studied in the literature on uniformizable problems; see, e.g., [13], [14]. However, obtaining structural results for nonuniformizable problems is much more involved.…”

Section: Optimality Of Threshold Policiesmentioning

confidence: 99%

Efficient Content Delivery in the Presence of Impatient Jobs

Larrañaga

Boxma

Núñez-Queija

et al. 2015

2015 27th International Teletraffic Congress

View full text Add to dashboard Cite

Abstract-We consider a content delivery problem in which jobs are processed in batches and may abandon before their service has been initiated. We model the problem as a Markovian single-server queue and analyze two different settings: (1) the system is cleared as soon as the server is activated, i.e., service rate µ = ∞, and (2) the service speed is exponentially distributed with rate µ < ∞. The objective is to determine the optimal clearing strategy that minimizes the average cost incurred by holding jobs in the queue, having jobs renege, and performing setups. This last cost is incurred upon activation of the server in the case µ = ∞, and per unit of time the server is active otherwise. Our first contribution is to prove that policies of threshold type are optimal in both frameworks. In order to do so we have used the Smoothed Rate Truncation method which overcomes the problem arising from unbounded transition rates. For our second contribution, we derive the steady-state job-length distribution under threshold policies. The latter yields a characterization of the optimal threshold strategy, which can be easily implemented. Finally, we present numerical results for our solution across a wide range of parameters. We show that the performance of nonoptimal threshold policies can be very poor, which highlights the importance of computing the optimal threshold.

show abstract

Section: Optimality Of Threshold Policiesmentioning

confidence: 99%

Efficient Content Delivery in the Presence of Impatient Jobs

Larrañaga

Boxma

Núñez-Queija

et al. 2015

2015 27th International Teletraffic Congress

View full text Add to dashboard Cite

show abstract

“…The set of actions is {0, 1} with the interpretation that 0 and 1 respectively correspond to not running and running the service station. This problem has been studied extensively and its Q-factors satisfy the monotonicity property in (8); see, for example, Ignall and Kolesar [1972], Kosten [1973], Deb and Serfozo [1973], Ignall and Kolesar [1974] and Papadaki and Powell [2002].…”

Section: Batch Service Problem With a Single Productmentioning

confidence: 99%

A stochastic approximation method with max-norm projections and its applications to the Q-learning algorithm

Kunnumkal

Topaloğlu

2010

ACM Trans. Model. Comput. Simul.

View full text Add to dashboard Cite

In this paper, we develop a stochastic approximation method to solve a monotone estimation problem and use this method to enhance the empirical performance of the Q-learning algorithm when applied to Markov decision problems with monotone value functions. We begin by considering a monotone estimation problem where we want to estimate the expectation of a random vector η. We assume that the components of E{η} are known to be in increasing order. The stochastic approximation method that we propose is designed to exploit this information by projecting its iterates onto the set of vectors with increasing components. The novel aspect of the method is that it uses projections with respect to the max norm. We show the almost sure convergence of the stochastic approximation method. After this result, we consider the Q-learning algorithm when applied to Markov decision problems with monotone value functions. We study a variant of the Q-learning algorithm that uses projections to ensure that the value function approximation that is obtained at each iteration is also monotone. Computational results indicate that the performance of the Q-learning algorithm can be improved significantly by exploiting the monotonicity property of the value functions.

show abstract

“…Closing this section, we note that Q-factors satisfying (6) appear in numerous queue admission, equipment replacement, batch service and pricing applications (see, for example, Ignall and Kolesar (1974), Papadaki and Powell (2002) and Puterman (1994)). In particular, the following corollary to Lemma 4.7.2 and Theorem 6.11.6 in Puterman (1994) gives sufficient conditions under which the Q-factors satisfy (6).…”

Section: Description Of the Algorithmmentioning

confidence: 99%

“…Similarly, the assumption that n j=k p ij (u) is increasing in i is satisfied by essentially any sensible aging process. We refer the reader to Papadaki and Powell (2002) and Puterman (1994) for similar observations for batch service and pricing applications.…”

Section: ) Increase T By 1 and Go Tomentioning

confidence: 99%

“…Topaloglu and Powell (2006) formulate a dynamic programming-based fleet management model, and motivated by the fact that the value function for the problem is concave, they use concave value function approximations. Papadaki and Powell (2002) and Papadaki and Powell (2003) show that the value function arising from a batch service problem is monotone in the number of products waiting to be served and exploit this property to enhance the performance of their approximate dynamic programming algorithm.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exploiting the Structural Properties of the Underlying Markov Decision Problem in the Q-Learning Algorithm

Kunnumkal

Topaloğlu

2008

INFORMS Journal on Computing

View full text Add to dashboard Cite

This paper shows how to exploit the structural properties of the underlying Markov decision problem to improve the convergence behavior of the Q-learning algorithm. In particular, we consider infinite-horizon discounted-cost Markov decision problems where there is a natural ordering between the states of the system and the value function is known to be monotone in the state. We propose a new variant of the Q-learning algorithm that ensures that the value function approximations obtained during the intermediate iterations are also monotone in the state. We establish the convergence of the proposed algorithm and experimentally show that it significantly improves the convergence behavior of the standard version of the Q-learning algorithm.

show abstract

Exploiting structure in adaptive dynamic programming algorithms for a stochastic batch service problem

Cited by 44 publications

References 16 publications

Efficient Content Delivery in the Presence of Impatient Jobs

Efficient Content Delivery in the Presence of Impatient Jobs

A stochastic approximation method with max-norm projections and its applications to the Q-learning algorithm

Exploiting the Structural Properties of the Underlying Markov Decision Problem in the Q-Learning Algorithm

Contact Info

Product

Resources

About