Recent Advances in Reinforcement Learning in Finance

Abstract

The rapid changes in the finance industry due to the increasing amount of data have revolutionized the techniques on data processing and data analysis and brought new theoretical and computational challenges. In contrast to classical stochastic control theory and other analytical approaches for solving financial decision-making problems that heavily reply on model assumptions, new developments from reinforcement learning (RL) are able to make full use of the large amount of financial data with fewer model assumptions and to improve decisions in complex financial environments. This survey paper aims to review the recent developments and use of RL approaches in finance. We give an introduction to Markov decision processes, which is the setting for many of the commonly used RL approaches. Various algorithms are then introduced with a focus on value and policy based methods that do not require any model assumptions. Connections are made with neural networks to extend the framework to encompass deep RL algorithms. Our survey concludes by discussing the application of these RL algorithms in a variety of decision-making problems in finance, including optimal execution, portfolio optimization, option pricing and hedging, market making, smart order routing, and robo-advising.

Recent Developments in Machine Learning Methods for Stochastic Control and Games

“…Numerous algorithms have been developed under the topic of RL; see, e.g., the surveys and books [116,30,134,166,93]. Most of them focus on RL itself, with state-of-the-art methods in single-agent or multi-agent problems and/or theoretical guarantees of numerical performances.…”

Applications of Reinforcement Learning in Finance -- Trading with a Double Deep Q-Network

“…There are policy-based and value-based methods [9] that can be applied to a model-free setting such as the financial market. [10]. On the one hand, for instance, the Policy Gradient Method (policy-based), approximates the policy directly (action probabilities).…”

“…They illustrate actor-critic methods based on the Policy-Gradient Theorem and their approximation. In addition, Hambly et al [10] provide an overview of recent advances in RL, focusing on deep reinforcement learning and applications in finance. This involves using a large amount of financial data with fewer model assumptions, which can improve decisions in complex financial environments.…”

“…The comparison with the buy-and-hold benchmark shows that the agent can achieve better returns after a certain number of training episodes. Hambly et al [10] presents the use of Reinforcement Learning in various aspects of finance. The author provides initial insights, followed by familiar concepts such as RL in portfolio optimization, and concludes the usage with Smart Order Routing.…”

On Parametric Optimal Execution and Machine Learning Surrogates

“…Perhaps the closest is Leal et al [42] who also study execution problems, but in a model with only temporary and permanent price impact à la Bertsimas and Lo [19], Almgren and Chriss [10], Cartea and Jaimungal [23]. For other approaches in optimal execution in the presence of temporary price impact more in the flavor of reinforcement learning see, e.g., the recent survey articles by Hambly et al [35] and Jaimungal [41] and the references therein.…”