Transition-based versus state-based reward functions for MDPs with Value-at-Risk

Abstract

In reinforcement learning, the reward function on current state and action is widely used. When the objective is about the expectation of the (discounted) total reward only, it works perfectly. However, if the objective involves the total reward distribution, the result will be wrong. This paper studies Value-at-Risk (VaR) problems in short-and longhorizon Markov decision processes (MDPs) with two reward functions, which share the same expectations. Firstly we show that with VaR objective, when the real reward function is transition-based (with respect to action and both current and next states), the simplified (state-based, with respect to action and current state only) reward function will change the VaR. Secondly, for long-horizon MDPs, we estimate the VaR function with the aid of spectral theory and the central limit theorem. Thirdly, since the estimation method is for a Markov reward process with the reward function on current state only, we present a transformation algorithm for the Markov reward process with the reward function on current and next states, in order to estimate the VaR function with an intact total reward distribution.

State-Augmentation Transformations for Risk-Sensitive Reinforcement Learning

“…Since many RL methods require the reward function to be deterministic and statebased, the transformation is needed for the MDPs with other types of reward functions in the risk-sensitive problems. We generalize the transformation (Ma and Yu 2017) in different settings, and consider VaR as an example to show the effect of reward simplification on distribution.…”

“…In RL, when the expected return is considered, and the Q-function or the value function is accessed, which implies such a reward simplification. The effect of the reward simplification on return distribution in a finite-horizon Markov reward process has been studied in (Ma and Yu 2017). Here we estimate the distribution with assuming it is approximately normal, illustrate the similar effect on return distribution, and generalize the transformation for more practical cases.…”

“…Generate the state space S † = S × S; for all x † = (x, y) where x, y ∈ S do Construct the reward function r † π (x † ) = r π (x, y); Construct the transition kernel p † π (x † | y † ) = p π (y | x) for all y † = (·, x) ∈ S † , and p † π (x † | y † ) = 0 otherwise; Set the initial state distribution µ † (x † ) = µ(x)p π (y | x); end for have a Markov reward process illustrated in Figure 2(a). In order to attach each possible reward value to a state to construct a r DS , we consider each transition as a state, and apply the State-Transition Transformation (Algorithm 1 (Ma and Yu 2017)). The transformed Markov reward process is presented in Figure 2(b), where only some of the transitions are shown.…”

“…We study the return-the discounted total reward ∞ t=0 γ t−1 R t -in an infinite-horizon MDP with finite state and action spaces, and consider the Value-at-Risk (VaR) objective as a risk-sensitive example. We generalize the transformation in (Ma and Yu 2017) to three successively more general SATs (Cases 1, 2, and 3), give a proof for the most general one, and illustrate the error from the reward simplification on the return distribution.…”

A Scheme for Dynamic Risk-Sensitive Sequential Decision Making

“…In many cases, conditional VaR (also known as expected shortfall) is preferred over VaR since it is coherent [19], i.e., it has some intuitively reasonable properties (convexity, for example). However, when the return can be assumed to be approximately normally distributed, VaR can be simply estimated with E(Φ) and V(Φ) [20].…”