State-Augmentation Transformations for Risk-Sensitive Reinforcement Learning

Abstract

In the framework of MDP, although the general reward function takes three arguments-current state, action, and successor state; it is often simplified to a function of two arguments-current state and action. The former is called a transition-based reward function, whereas the latter is called a state-based reward function. When the objective involves the expected total reward only, this simplification works perfectly. However, when the objective is risk-sensitive, this simplification leads to an incorrect value. We propose three successively more general state-augmentation transformations (SATs), which preserve the reward sequences as well as the reward distributions and the optimal policy in risk-sensitive reinforcement learning. In risk-sensitive scenarios, firstly we prove that, for every MDP with a stochastic transition-based reward function, there exists an MDP with a deterministic state-based reward function, such that for any given (randomized) policy for the first MDP, there exists a corresponding policy for the second MDP, such that both Markov reward processes share the same reward sequence. Secondly we illustrate that two situations require the proposed SATs in an inventory control problem. One could be using Q-learning (or other learning methods) on MDPs with transition-based reward functions, and the other could be using methods, which are for the Markov processes with a deterministic state-based reward functions, on the Markov processes with general reward functions. We show the advantage of the SATs by considering Value-at-Risk as an example, which is a risk measure on the reward distribution instead of the measures (such as mean and variance) of the distribution. We illustrate the error in the reward distribution estimation from the reward simplification, and show how the SATs enable a variance formula to work on Markov processes with general reward functions. ∞ t=0 γ t−1 R t -in an infinite-horizon MDP with finite state and action spaces, and consider the Value-at-Risk (VaR) objective as a risk-sensitive example. We generalize the transformation in (Ma and Yu 2017) to three successively more general SATs (Cases 1, 2, and 3), give a proof for the most general one, and illustrate the error from the reward simplification on the return distribution.

Variance-Based Risk Estimations in Markov Processes via Transformation with State Lumping

“…A commonly used quantile-based risk measure is value at risk (VaR). For VaR estimation with the SAT, see [7]. In short, we claim that most, if not all, inherent risk measures depend on the reward sequence (R t : t ∈ {1, · · · , N }), which can be preserved by the SAT in a risksensitive scenario.…”

“…In short, we claim that most, if not all, inherent risk measures depend on the reward sequence (R t : t ∈ {1, · · · , N }), which can be preserved by the SAT in a risksensitive scenario. For law-invariant risk estimations, since VaR estimation in a Markov process with a stochastic reward has been throughly studied in [7], in this paper, we focus on the first two risk estimations with Eq. ???…”

“…in which S is a finite state space, and X t ∈ S represents the state at (decision) epoch t ∈ N; A x is the allowable action set for x ∈ S, A = x∈S A x is a finite action space, and K t ∈ A represents the action at epoch t; r is a bounded reward function, and R t denotes the immediate reward at epoch t; p(y | x, a) = P(X t+1 = y | X t = x, K t = a) denotes the homogeneous transition probability; µ is the initial state distribution; γ ∈ (0, 1) is the discount factor. Since a number of risk-sensitive RL studies require the reward function to be deterministic and state-based [7], we only consider r : S → R in a transformed MDP (or Markov process).…”

“…In this study, we consider an MDP with a randomized policy. The function of SAT is to transform M † to M and preserve (R t ), and this problem is concerned in four cases [7]. To illustrate how the SAT with state lumping works, we consider the Case 3 as an example, which refers to an MDP with a randomized policy.…”

“…However, the reward functions are usually stochastic in many practical problems. It has been shown that, when the objective is risk-sensitive, and the reward needs to be converted to a simple form, the SAT should be implemented instead of a reward simplification [7]. In this paper, we present the SAT in a homomorphism version, and thoroughly discuss its pros and cons.…”