Nonstationary value iteration in controlled Markov chains with risk-sensitive average criterion

Abstract

This work concerns Markov decision chains with finite state spaces and compact action sets. The performance index is the long-run risk-sensitive average cost criterion, and it is assumed that, under each stationary policy, the state space is a communicating class and that the cost function and the transition law depend continuously on the action. These latter data are not directly available to the decision-maker, but convergent approximations are known or are more easily computed. In this context, the nonstationary value iteration algorithm is used to approximate the solution of the optimality equation, and to obtain a nearly optimal stationary policy.

Risk-sensitive average optimality in Markov decision processes

“…In contrast to the risk-neutral model, if the resulting transition probability matrix P (f ) is unichain and contains also transient states, solution of equations (26)-(28) can be guaranteed only for the small values of the risk sensitivity coefficient (see e. g. [7,8,9,25]). Conditions guaranteeing existence of solutions to (26)- (27) were studied in many papers, see e. g. [3][4][5][6][7][8][9][10][11][12][25][26][27][28].…”

“…Unfortunately, risk-sensitive analogies to (9), (10) and (12) as well as optimality conditions are more complicated and the unichain property itself (cf. Assumption 1) is not sufficient for the existence of g, w i 's fulfilling (18)- (20).…”