An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization

Abstract

<jats:p>In this article, we propose an end-to-end adaptive framework for optimal trade execution based on Proximal Policy Optimization (PPO). We use two methods to account for the time dependencies in the market data based on two different neural network architecture: 1) Long short-term memory (LSTM) networks, 2) Fully-connected networks (FCN) by stacking the most recent limit orderbook (LOB) information as model inputs. The proposed framework can make trade execution decisions based on level-2 limit order book (LOB) information such as bid/ask prices and volumes directly without manually designed attributes as in previous research. Furthermore, we use a sparse reward function, which gives the agent reward signals at the end of each episode as an indicator of its relative performances against the baseline model, rather than implementation shortfall (IS) or a shaped reward function. The experimental results have demonstrated advantages over IS and the shaped reward function in terms of performance and simplicity. The proposed framework has outperformed the industry commonly used baseline models such as TWAP, VWAP, and AC as well as several Deep Reinforcement Learning (DRL) models on most of the 14 US equities in our experiments.</jats:p>

Recent Advances in Reinforcement Learning in Finance

“…The most popular types of RL methods that have been used in optimal execution problems are Q-learning algorithms and (double) DQN [102,159,241,108,51,181,158]. Policy-based algorithms are also popular in this field, including (deep) policy gradient methods [100,241], A2C [241], PPO [51,140], and DDPG [235]. The benchmark strategies studied in these papers include the Almgren-Chriss solution [102,100], the TWAP strategy [159,51,140], the VWAP strategy [140], and the SnL policy [158,235].…”

“…Policy-based algorithms are also popular in this field, including (deep) policy gradient methods [100,241], A2C [241], PPO [51,140], and DDPG [235]. The benchmark strategies studied in these papers include the Almgren-Chriss solution [102,100], the TWAP strategy [159,51,140], the VWAP strategy [140], and the SnL policy [158,235]. In some models the trader is allowed to buy or sell the asset at each time point [108,241,217,56], whereas there are also many models where only one trading direction is allowed [158,102,100,159,51,181,235,140].…”

“…The benchmark strategies studied in these papers include the Almgren-Chriss solution [102,100], the TWAP strategy [159,51,140], the VWAP strategy [140], and the SnL policy [158,235]. In some models the trader is allowed to buy or sell the asset at each time point [108,241,217,56], whereas there are also many models where only one trading direction is allowed [158,102,100,159,51,181,235,140]. The state variables are often composed of time stamp, the market attributes including (mid-)price of the asset and/or the spread, the inventory process and past returns.…”

“…Experiments on real LOB data with 10 price levels show that the proposed approach significantly outperforms the existing methods, including the SnL policy (as baseline), the Q-learning algorithm, and the method in [102]. [140] proposed an adaptive framework based on PPO with neural networks including LSTM and fully-connected networks, and showed that the framework outperforms the baseline models including TWAP and VWAP, as well as several deep RL models on most of 14 US equities. [100] applied the (vanilla) policy gradient method to the LOB data of five stocks in different sectors and showed that they improve the Implementation Shortfall of the Almgren-Chriss solution by around 20%.…”

Reinforcement Learning for Quantitative Trading

“…PPO is another widely used RL method for OE. Lin and Beling [78] proposed an end-to-end PPO-based framework.…”