Risk-Averse Trust Region Optimization for Reward-Volatility Reduction

Abstract

<jats:p>The use of reinforcement learning in algorithmic trading is of growing interest, since it offers the opportunity of making profit through the development of autonomous artificial traders, that do not depend on hard-coded rules. In such a framework, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in reinforcement learning through measures related to the distribution of returns. However, in trading it is essential to keep under control the risk of portfolio positions in the intermediate steps. In this paper, we define a novel measure of risk, which we call reward volatility, consisting of the variance of the rewards under the state-occupancy measure. This new risk measure is shown to bound the return variance so that reducing the former also constrains the latter. We derive a policy gradient theorem with a new objective function that exploits the mean-volatility relationship. Furthermore, we adapt TRPO, the well-known policy gradient algorithm with monotonic improvement guarantees, in a risk-averse manner. Finally, we test the proposed approach in two financial environments using real market data.</jats:p>

Hedging of Financial Derivative Contracts via Monte Carlo Tree Search

“…This advancement was also noticed by researchers of the financial derivatives area, where several publications reported on promising hedging performance of trained RL agents, see e.g. [25,32,23,15,13,14,8,51]. In terms of the choice of training algorithm, the research focus has lain on variations of Deep Q-Learning (DQN) [37], which combines Q-learning [52] with a deep neural network for policy representation.…”

“…The article [32] assumes that derivative prices are known and studies a DQN approach for hedging in the presence of transaction costs, focusing on mean-variance equivalent loss distributions. Article [51] studies risk-averse policy search for hedging when price information is available following the mean-volatility approach of [8].…”

A unified algorithm framework for mean-variance optimization in discounted Markov decision processes

“…A variety of algorithms for variance-related optimization in previous works can be unified and analyzed in the proposed framework. When the equivalence in ( 18) is concerned, we have the policy gradient for the mean-variance optimizations in discounted MDPs (Bisi et al 2020), and the policy iteration for the mean-variance optimization in discounted and average MDPs (Zhang et al 2021). However, no convergence analysis is given in either of the works, such as the analyses for the policy iterations in (Xia 2016(Xia , 2020.…”

“…This work is later extended to the mean-variance optimization in average MDPs (Xia 2020). Bisi et al (2020) study the discounted mean-variance in RL, where the steady-state variance is evaluated to bound the limiting average variance. They develop a gradient-based trust region policy optimization (originally proposed by Schulman et al (2015)) algorithm with a monotonic policy improvement.…”

“…Variance is a quadratic function of mean, and a variance-related problem is equivalent to an MDP with a special reward function, whose value for each state depends on policy instead of action, i.e., the variance value function at a current state will be affected by actions chosen at not only the current epoch but also future epochs. This dependency deprives the time-consistency property and revokes traditional dynamic programming (DP) methods (Puterman 2005, Eckstein et al 2016, Bisi et al 2020. In other words, the Bellman optimality equation does not optimize over the admissible action set for a state x (max a∈A(x) ), but over the policy space for the whole state space S (max d∈D ), and we can not divide and conquer this problem in a traditional manner.…”