State-Augmentation Transformations for Risk-Sensitive Reinforcement Learning

Ma, Shuai; Yu, Jia Yuan

doi:10.48550/arxiv.1804.05950

Cited by 3 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…How to apply the method to practical problems with stochastic rewards is the problem. In next section, we review the SAT [22] as an MDP homomorphism, which enables the variance formula in a Markov process with a stochastic reward.…”

Section: Variance Formula For Markov Processesmentioning

confidence: 99%

“…We restate the SAT as an MDP homomorphism. Comparing with the original SAT theorem [22], the homomorphism version of SAT is on a more abstract level. An MDP homomorphism is a formalism that captures an intuitive notion of specific equivalence between MDPs [30].…”

Section: Sat Homomorphismmentioning

confidence: 99%

“…In this study, for any given q e , we construct an MDP model, and by virtue of the SAT [22], we can preserve the variance for a Markov process with a stochastic reward. As claimed in Section 2.2, most inherent risk measures can be evaluated or estimated by return variance with or without some assumption.…”

Section: Approximatormentioning

confidence: 99%

See 2 more Smart Citations

A Scheme for Dynamic Risk-Sensitive Sequential Decision Making

Ma¹,

Yu²,

Şatır³

2019

Preprint

Self Cite

View full text Add to dashboard Cite

We present a scheme for sequential decision making with a risk-sensitive objective and constraints in a dynamic environment. A neural network is trained as an approximator of the mapping from parameter space to space of risk and policy with risk-sensitive constraints. For a given risksensitive problem, in which the objective and constraints are, or can be estimated by, functions of the mean and variance of return, we generate a synthetic dataset as training data. Parameters defining a targeted process might be dynamic, i.e., they might vary over time, so we sample them within specified intervals to deal with these dynamics. We show that: i). Most risk measures can be estimated using return variance; ii). By virtue of the state-augmentation transformation, practical problems modeled by Markov decision processes with stochastic rewards can be solved in a risk-sensitive scenario; and iii). The proposed scheme is validated by a numerical experiment.

show abstract

Section: Variance Formula For Markov Processesmentioning

confidence: 99%

Section: Sat Homomorphismmentioning

confidence: 99%

See 1 more Smart Citation

A Scheme for Dynamic Risk-Sensitive Sequential Decision Making

Ma¹,

Yu²,

Şatır³

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…A commonly used quantile-based risk measure is value at risk (VaR). For VaR estimation with the SAT, see [7].…”

Section: Arxiv:190705231v1 [Cslg] 9 Jul 2019mentioning

confidence: 99%

“…However, the reward functions are usually stochastic in many practical problems. It has been shown that, when the objective is risk-sensitive, and the reward needs to be converted to a simple form, the SAT should be implemented instead of a reward simplification [7]. In this paper, we present the SAT in a homomorphism version, and thoroughly discuss its pros and cons.…”

Section: Introductionmentioning

confidence: 99%

Variance-Based Risk Estimations in Markov Processes via Transformation with State Lumping

2019

2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)

Self Cite

View full text Add to dashboard Cite

Variance plays a crucial role in risk-sensitive reinforcement learning, and most risk measures can be analyzed via variance. In this paper, we consider two law-invariant risks as examples: mean-variance risk and exponential utility risk. With the aid of the state-augmentation transformation (SAT), we show that, the two risks can be estimated in Markov decision processes (MDPs) with a stochastic transition-based reward and a randomized policy. To relieve the enlarged state space, a novel definition of isotopic states is proposed for state lumping, considering the special structure of the transformed transition probability. In the numerical experiment, we illustrate state lumping in the SAT, errors from a naive reward simplification, and the validity of the SAT for the two risk estimations.

show abstract

Variance-Based Risk Estimations in Markov Processes via Transformation with State Lumping

Ma¹,

Yu²

2019

Preprint

View full text Add to dashboard Cite

State-Augmentation Transformations for Risk-Sensitive Reinforcement Learning

Cited by 3 publications

References 0 publications

A Scheme for Dynamic Risk-Sensitive Sequential Decision Making

A Scheme for Dynamic Risk-Sensitive Sequential Decision Making

Variance-Based Risk Estimations in Markov Processes via Transformation with State Lumping

Variance-Based Risk Estimations in Markov Processes via Transformation with State Lumping

Contact Info

Product

Resources

About