Multiscale Stochastic Approximation for Parametric Optimization of Hidden Markov Models

Abstract

A two–time scale stochastic approximation algorithm is proposed for simulation-based parametric optimization of hidden Markov models, as an alternative to the traditional approaches to “infinitesimal perturbation analysis.” Its convergence is analyzed, and a queueing example is presented.

Optimal structured feedback policies for ABR flow control using two-timescale SPSA

“…In Section III, we describe our two-timescale SPSA scheme for obtaining the optimal structured policy, briefly compare it with two previously proposed two-timescale stochastic approximation algorithms (cf. [5]) and present our main result. The convergence analysis is briefly presented in the Appendix, with detailed proofs provided in [6].…”

“…The advantages of using the SPSA approach are best appreciated after first presenting the original two-timescale algorithm of [5]. Let be a small fixed constant.…”

“…Define positive real-valued sequences and as follows: , , , , and is the unit vector in the th direction. Then the algorithm is as follows: For (8) An alternative (proposed and used in [5]) to using ( ) parallel simulations is to move the algorithm in cycles during each of which only two simulations are used as follows. The first simulation corresponds to and is governed by , defined as earlier, and the second simulation is represented as , which is governed by defined by for , , .…”

“…The first simulation corresponds to and is governed by , defined as earlier, and the second simulation is represented as , which is governed by defined by for , , . The algorithm of [5] is (9) Here, and (used above for notational simplicity). Thus instead of all components being updated every steps, , as in (8), only one component is updated now every steps and the algorithm thus moves in bigger loops or cycles of with all components updated once at the end of the bigger loop.…”

“…We develop a simultaneous perturbation stochastic approximation (SPSA) [27], [14] variant of a two-timescale stochastic approximation algorithm in [5] to obtain the optimal policy with this structure. The two-timescale stochastic approximation algorithm developed in [5] (see also [2]) for simulation-based parametric optimization had the advantage that it updates the parameter at deterministic instants obtained using two different step sizes (time scales), without the need for regeneration, as is typically the case with traditional perturbation analysis schemes [11], [12]. On the other hand, like other finite-difference schemes, it requires simulations for any -vector parameter to obtain the gradient estimate.…”