Dynamically Identifying Regenerative Cycles in Simulation-Based Optimization Algorithms for Markov Chains

Abstract

Simulation-based algorithms for maximizing the average reward of a parameterized Markov chain often rely upon the existence of a state which is recurrent for all choices of parameter values. The question of which recurrent state should serve to mark the end of a regenerative cycle is a very important practical consideration in applications. Even when all of the states of the process are recurrent, some states tend to be visited more often than others, and lengthy renewal cycles tend to result in high variance estimates of the gradient. To address this difficulty, we analyze a mechanism for adjusting this special state dynamically ( -adaptation) as applied to the "batch" simulation-based optimization algorithm of a previous paper. We show that the desirable convergence properties of the original "batch" algorithm are retained with -adaptation, namely the almost sure convergence of the parameter vector to a critical point.

On-line tuning of prices for network services

“…In this section, we adapt the discrete-time simulation-based methodology of [29] (as extended in [30], [32]) for on-line use in a continuous-time setting.…”

“…Unfortunately, for a given set of control parameters, transitions to i * can be infrequent, resulting in high-variance gradient estimates and ultimately in unacceptably slow convergence. This issue has been addressed in [30] through the introduction of an i * -adaptation procedure, where the marked state i * can be reset as needed to account for changes in the steadystate distribution of the process as the control parameters are adjusted.…”

“…In Algorithm 1 below, we present a price-setting algorithm, which can be interpreted as a continuous-time implementation of the discrete-time algorithm in [30]. Since the algorithm proceeds in response to arrival and departure events of the actual process, it applies in an on-line setting.…”

“…This paper explores the possibility of adapting the simulation-based procedure of [30] (based on [29]) to allow for efficient and robust on-line pricing of network services. Our first contribution is to spell out a version of the simulationbased procedure from [30] that applies to the continuous-time operation of the system and, in the case where arrival rate functions and expected holding times are known, is guaranteed to converge with probability one to an extremum of the average reward process. Next, to address the case where the parameters of the model are unknown (and possibly slowly time varying), we develop an adaptive price-setting algorithm by allowing for the simultaneous estimation of model parameters.…”

“…Even though this algorithm operates by adjusting prices in response to arrivals and departures in real-time, we refer to it as "model-based" because it requires explicit knowledge of the parameters of the underlying system. (The main difference between Algorithm 1 and the earlier approach from [29] is that Algorithm 1 incorporates a device known as i * -adaptation [30] which resolves an important technical issue associated with the original simulation-based procedure.) In Section IV, we propose an online algorithm (Algorithm 2) which is adapted from the modelbased procedure of Section III and allows for the simultaneous estimation of model parameters.…”