Adaptive Strategies for Certain Classes of Controlled Markov Processes

Abstract

One usually defines an adaptive strategy to be a strategy of controlling by a random process with incompletely known probability description. Mostly, only the class of random processes to which the controlled process under examination belongs is indicated. The probability characteristics within the class are unknown but can be reconstructed by statistical methods by observation of a realization of the process. Different variants of problems of adaptive control and methods for solving them have been presented in the books [1], [2]. A large number of works have been devoted to adaptive control of Markov chains with finite sets of states and controls. Adaptive strategies for processes whose spaces of states and controls are compact sets in R have been studied in [3], [4] and in certain other works. In this article a class of controlled Markov processes with discrete time defined by the recurrent relation (1)x, (x,_, at, ), t= 1,2,. , is examined. Here the unknown "parameter" is the probability distribution of the random vectors :t. The goal of the control is to maximize the average "revenue" per unit time. For constructing optimal adaptive strategies, controls are used that are close to optimal ones in an ancillary model with a known probability description. This description is given by means of an empirical distribution constructed from realizations of ,..., so,. The results contained herein were published without proof in [5].

Continuous dependence of stochastic control models on the noise distribution

“…This, again, is a well-known result [3], [5], [6], [8], [9], [15], [17], [19]; the measurability of the functionsj*(/~) and J(x, Ix) in the "parameter"/z c M follows from standard measurable selection theorems [6], [13], [15], [17]. Before stating our continuity result in the average-reward case, let us first assume that condition R5 holds.…”

“…Unfortunately, to obtain (20) is not as simple as it may look, and therefore a related, but different, approach has to be taken. For details, as well as adaptive policies other than 6 above, see [8] and [12].…”

“…Other conditions sufficient for R5 are given by Gordienko [8] and Tijms [19]. We next have: attains the maximum in the right-hand side of (14) for all x c X, and…”

“…Suppose we want to maximize a given reward criterion and let N v, (x) denote the optimal reward function given that the initial state is x0 = x and the planning horizon is N-< oo. The purpose of this note, motivated by questions arising in problems on the adaptive control of stochastic systems with unknown noise distribution [8], [ 12], is to investigate the dependence of N v~ (x) on ~z.…”

Empirical approximation in Markov games under unbounded payoff: discounted and average criteria

“…Let ν ∈ (0, 1/2m) be an arbitrary real number where m is the constant introduced in Assumption 5.1(b). By borrowing the ideas in [7,15], we fix an arbitrary nondecreasing sequence of discount factors {ᾱ t } such that…”