Recent advances in reinforcement learning in finance

Abstract

The rapid changes in the finance industry due to the increasing amount of data have revolutionized the techniques on data processing and data analysis and brought new theoretical and computational challenges. In contrast to classical stochastic control theory and other analytical approaches for solving financial decision‐making problems that heavily reply on model assumptions, new developments from reinforcement learning (RL) are able to make full use of the large amount of financial data with fewer model assumptions and to improve decisions in complex financial environments. This survey paper aims to review the recent developments and use of RL approaches in finance. We give an introduction to Markov decision processes, which is the setting for many of the commonly used RL approaches. Various algorithms are then introduced with a focus on value‐ and policy‐based methods that do not require any model assumptions. Connections are made with neural networks to extend the framework to encompass deep RL algorithms. We then discuss in detail the application of these RL algorithms in a variety of decision‐making problems in finance, including optimal execution, portfolio optimization, option pricing and hedging, market making, smart order routing, and robo‐advising. Our survey concludes by pointing out a few possible future directions for research.

Deep Reinforcement Learning for Dynamic Stock Option Hedging: A Review

“…SARSA is an example of a value-based RL method, as the aim of SARSA is to find the optimal Q-function by directly evaluating which state-action pairs are more promising in terms of expected cumulative future rewards [9]. Value methods such as SARSA do not explicitly consider the optimal policy [12]. Conversely, policy-based methods keep the current estimate of the optimal policy in memory during the learning process, and updates are performed on this optimal policy estimate rather than the value function [12].…”

“…Value methods such as SARSA do not explicitly consider the optimal policy [12]. Conversely, policy-based methods keep the current estimate of the optimal policy in memory during the learning process, and updates are performed on this optimal policy estimate rather than the value function [12]. Using a value method rather than the policy method may impose the constraint of discrete action spaces.…”

“…In addition to these reviews of the entire field of RL, there are works that consider the use of RL in finance. A recent paper [12] provides a comprehensive overview of RL usage for optimal execution and market making in electronic markets, portfolio optimization, automated trading, order routing and option hedging. Ref.…”

Dynamics of market making algorithms in dealer markets: Learning and tacit collusion

“…In practice, market makers interact with client order flow and learn to adjust their quotes in order to maximize their profit. We model this learning process using a decentralized multi-agent deep reinforcement learning algorithm (Hambly et al, 2023) using a policy gradient method (Fazel et al, 2018) to update market makers strategies, parameterized via neural networks. Our simulation results show that the interaction of market making algorithms through market prices, without any sharing of information, may give rise to tacit collusion, as evidenced by quoted spread levels significantly higher than in competitive (Nash) equilibrium.…”

Markov decision processes under model uncertainty

“…The Markov decision problem leads to an infinite horizon stochastic optimal control problem in discrete‐time, which finds many applications in finance and economics, compare, for example, Bäuerle and Rieder (2011), Hambly et al. (2021), or White (1993) for an overview. It can, among a multitude of other applications, be used to learn the optimal structure of portfolios and the optimal trading behavior, see, for example, Bertoluzzo and Corazza (2012), Chang and Lee (2017), Gold (2003), Hu and Lin (2019), Xiong et al.…”