Infinite horizon risk sensitive control of discrete time Markov processes with small risk

Masi, Giovanni B. Di; Stettner, Łukasz

doi:10.1016/s0167-6911(99)00118-8

Cited by 51 publications

(31 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that the iteration in (3.10) already appears in Di Masi & Stettner (1999) p.68. However, there the authors do not consider a finite horizon problem.…”

Section: Finite Horizon Problemsmentioning

confidence: 99%

“…Cavazos-Cadena & Hernández-Hernández (2011);Cavazos-Cadena & Fernández-Gaucherand (2000); Jaśkiewicz (2007); Di Masi & Stettner (1999)). The infinite horizon discounted classical risk-sensitive MDP and its relation to the average cost problem is considered in Di Masi & Stettner (1999). As far as applications are concerned, risk-sensitive problems can e.g.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

More Risk-Sensitive Markov Decision Processes

2014

View full text Add to dashboard Cite

Abstract. We investigate the problem of minimizing a certainty equivalent of the total or discounted cost over a finite and an infinite horizon which is generated by a Markov Decision Process (MDP). The certainty equivalent is defined by U −1 (E U (Y )) where U is an increasing function. In contrast to a risk-neutral decision maker this optimization criterion takes the variability of the cost into account. It contains as a special case the classical risk-sensitive optimization criterion with an exponential utility. We show that this optimization problem can be solved by an ordinary MDP with extended state space and give conditions under which an optimal policy exists. In the case of an infinite time horizon we show that the minimal discounted cost can be obtained by value iteration and can be characterized as the unique solution of a fixed point equation using a 'sandwich' argument. Interestingly, it turns out that in case of a power utility, the problem simplifies and is of similar complexity than the exponential utility case, however has not been treated in the literature so far. We also establish the validity (and convergence) of the policy improvement method. A simple numerical example, namely the classical repeated casino game is considered to illustrate the influence of the certainty equivalent and its parameters. Finally also the average cost problem is investigated. Surprisingly it turns out that under suitable recurrence conditions on the MDP for convex power utility U , the minimal average cost does not depend on U and is equal to the risk neutral average cost. This is in contrast to the classical risk sensitive criterion with exponential utility.

show abstract

“…Note that the iteration in (3.10) already appears in Di Masi & Stettner (1999) p.68. However, there the authors do not consider a finite horizon problem.…”

Section: Finite Horizon Problemsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

More Risk-Sensitive Markov Decision Processes

2014

View full text Add to dashboard Cite

show abstract

“…As mentioned in Section 1, characterizations of the optimal λ-sensitive average cost for MDPs with denumerable or Borel state spaces via the λ-OE have recently been given in Borkar and Meyn (2002) and Di Masi and Stettner (1999), (2000), respectively. Extending Theorem 3.1 to the cases considered in those papers is an interesting problem.…”

Section: Nonstationary Value Iteration In Controlled Markov Chainsmentioning

confidence: 99%

Nonstationary value iteration in controlled Markov chains with risk-sensitive average criterion

Cavazos–Cadena

Montes-de-Oca

2005

J. Appl. Probab.

View full text Add to dashboard Cite

This work concerns Markov decision chains with finite state spaces and compact action sets. The performance index is the long-run risk-sensitive average cost criterion, and it is assumed that, under each stationary policy, the state space is a communicating class and that the cost function and the transition law depend continuously on the action. These latter data are not directly available to the decision-maker, but convergent approximations are known or are more easily computed. In this context, the nonstationary value iteration algorithm is used to approximate the solution of the optimality equation, and to obtain a nearly optimal stationary policy.

show abstract

“…In the past two decades, there has been a renewed interest in this type of cost criteria as, when the 'risk factor' is strictly positive, i.e., in the risk-averse case, the use of the exponential reduces the possibility of rare but devastating large excursions of the state process. Though this criterion has been studied extensively in the literature of Markov decision processes (see, e.g., Borkar and Meyn [13], Cavazos-Cadena and Fernandez-Gaucherand [14], Di Masi and Stettner [15,16,17], Fleming and Hernández-Hernández [21], Fleming and McEneaney [22], Hernández-Hernández and 2 Article submitted to Mathematics of Operations Research; manuscript no. MOR-2015-241 Marcus [24], Whittle [35,36]), the corresponding results on stochastic games seem to be limited (see e.g., Basar [3], El-Karoui and Hamadene [18], Jacobson [27], James et.…”

Section: Introductionmentioning

confidence: 99%

“…This change makes our analysis totally novel and substantially different from those in the existing literature. Also, in most of the existing literature in this domain see, e.g., Balaji and Meyn [2], Borkar and Meyn [13], Cavazos-Cadena and Fernandez-Gaucherand [14], Di 3 Masi and Stettner [16], and, Hernández-Hernández and Marcus [24], the 'risk factor' is assumed to be sufficiently small. We make an assumption, for the ergodic game only, on the smallness of the cost function as in Basu and Ghosh [4] which essentially implies that the 'risk factor' cannot be too large.…”

Section: Introductionmentioning

confidence: 99%

Nonzero-Sum Risk-Sensitive Stochastic Games on a Countable State Space

Basu

Ghosh

2018

Mathematics of OR

View full text Add to dashboard Cite

The infinite horizon risk-sensitive discounted-cost and ergodic-cost nonzero-sum stochastic games for controlled Markov chains with countably many states are analyzed. For the discounted-cost game, we prove the existence of Nash equilibrium strategies in the class of Markov strategies under fairly general conditions. Under an additional geometric ergodicity condition and a small cost criterion, the existence of Nash equilibrium strategies in the class of stationary Markov strategies is proved for the ergodic-cost game.

show abstract

Infinite horizon risk sensitive control of discrete time Markov processes with small risk

Cited by 51 publications

References 5 publications

More Risk-Sensitive Markov Decision Processes

More Risk-Sensitive Markov Decision Processes

Nonstationary value iteration in controlled Markov chains with risk-sensitive average criterion

Nonzero-Sum Risk-Sensitive Stochastic Games on a Countable State Space

Contact Info

Product

Resources

About