The Role of Information in System Stability with Partially Observable Servers

Abstract

We consider a simple discrete time controlled queueing system, where the controller has a choice of which server to use at each time slot and server performance varies according to a Markov modulated random environment. We explore the role of information in the system stability region. At the extreme cases of information availability, that is when there is either full information or no information, stability regions and maximally stabilizing policies are trivial. But in the more realistic cases where only the environment state of the selected server is observed, only the service successes are observed or only queue length is observed, finding throughput maximizing control laws is a challenge. To handle these situations, we devise a Partially Observable Markov Decision Process (POMDP) formulation of the problem and illustrate properties of its solution. We further model the system under given decision rules, using Quasi-Birth-and-Death (QBD) structure to find a matrix analytic expression for the stability bound. We use this formulation to illustrate how the stability region grows as the number of controller belief states increases.Our focus in this paper is on the simple case of two servers where the environment of each is modulated according to a two-state Markov chain. As simple as this case seems, there appear to be no closed form descriptions of the stability region under the various regimes considered. Our numerical approximations to the POMDP Bellman equations and the numerical solutions of the QBDs hint at a variety of structural results.

A Survey of Parameter and State Estimation in Queues

“…For each value of ∆, we simulate M = 10 4 Monte-Carlo simulations of the queue, letting it "warm up" for 10 3 time units each time (this sets it close to "steady state"). Each simulation run samples the queue as per (10). We then estimate the coverage probability of the resulting confidence interval via,…”

Asymptotic Optimal Control of Markov-Modulated Restless Bandits

“…The latter involves continuity of the state space and as such there is a lack of tractable solution methodologies [34]. For results regarding speci c models, we mention here for example [5,6,29]. In [5] optimal load balancing is studied when the queue lengths of the servers are unobservable.…”

“…A set of round-robin policies is proved to be optimal in a heavy-tra c many-server limiting regime. Optimal control in a multi-class queueing system where the server performance varies according to a Markov modulated random environment is studied in [6,29]. The environment can be partially observed for the activated server.…”

“…where the rst step follows from the ergodicity theorem, the second step from uniform integrability of X /N (interchange of limit and summations) and (16), the third step from the fact that π * ,a j,k ( x * ) = x * ,a j,k (see (6) and lim t →∞ x π * (t ) = x * ), and the last step follows since x * is an optimal solution of (LP). This concludes the proof of Proposition 4.5.…”

Control and inference of structured Markov models

“…Moreover, we extend the model to the case of having a multi-absorption PH distribution as the sojourn time distribution or intensity distribution of the semi-Markov model. The result of the projects mentioned above have been published or submitted to prestigious journals and peer-reviewed conferences, see [8][9][10][11]146]. Further papers are currently in final stages of preparation.…”