Applications of Reinforcement Learning in Finance -- Trading with a Double Deep Q-Network

Abstract

This paper presents a Double Deep Q-Network algorithm for trading single assets, namely the E-mini S&P 500 continuous futures contract. We use a proven setup as the foundation for our environment with multiple extensions. The features of our trading agent are constantly being expanded to include additional assets such as commodities, resulting in four models. We also respond to environmental conditions, including costs and crises. Our trading agent is first trained for a specific time period and tested on new data and compared with the long-and-hold strategy as a benchmark (market). We analyze the differences between the various models and the in-sample/out-of-sample performance with respect to the environment. The experimental results show that the trading agent follows an appropriate behavior. It can adjust its policy to different circumstances, such as more extensive use of the neutral position when trading costs are present. Furthermore, the net asset value exceeded that of the benchmark, and the agent outperformed the market in the test set. We provide initial insights into the behavior of an agent in a financial domain using a DDQN algorithm. The results of this study can be used for further development.

Enhancing Stock Market Forecasts with Double Deep Q-Network in Volatile Stock Market Environments

“…In this reinforcement learning experiment, the action space [76] is critically designed to enable the agent's decision making with two fundamental actions: "BUY" (0) and "SELL" (1). This binary structure serves the experiment's goal of evaluating the agent's…”

“…In this reinforcement learning experiment, the action space [76] is critically designed to enable the agent's decision making with two fundamental actions: "BUY" (0) and "SELL" (1). This binary structure serves the experiment's goal of evaluating the agent's ability to predict daily stock price movements, either upward or downward, thereby assessing its capability for making profitable trading decisions.…”

Using Deep Reinforcement Learning with Hierarchical Risk Parity for Portfolio Optimization

“…In a different line of work, DRL has been applied with some degree of success in many markets Betancourt and Chen (2021); Jiang and Liang (2017); Zejnullahu et al (2022); Zhang et al (2020). One of the major differences between the two approaches, is the period at which they are evaluated on, with HRP and related approaches tackling much larger out-of-sample sets.…”

A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation

“…One successful use of RL is in the game of Go [ 9 ], where DeepMind’s AlphaGo program defeated the world champion Lee Sedol in 2016 to attain unparalleled success. In addition to its application in finance to improve trading tactics and portfolio management, RL has been used to create recommendation systems that can learn to give individualized suggestions depending on user behavior [ 10 ]. Moreover, RL has been used for complicated issues in which conventional rule-based systems or supervised learning approaches fall short, including those in the domains of natural language processing [ 11 ], drug discovery [ 12 ], and autonomous driving [ 13 ], among others.…”