Quantitative Trading through Random Perturbation Q-Network with Nonlinear Transaction Costs

Abstract

In recent years, reinforcement learning (RL) has seen increasing applications in the financial industry, especially in quantitative trading and portfolio optimization when the focus is on the long-term reward rather than short-term profit. Sequential decision making and Markov decision processes are rather suited for this type of application. Through trial and error based on historical data, an agent can learn the characteristics of the market and evolve an algorithm to maximize the cumulative returns. In this work, we propose a novel RL trading algorithm utilizing random perturbation of the Q-network and account for the more realistic nonlinear transaction costs. In summary, we first design a new near-quadratic transaction cost function considering the slippage. Next, we develop a convolutional deep Q-learning network (CDQN) with multiple price input based on this cost functions. We further propose a random perturbation (rp) method to modify the learning network to solve the instability issue intrinsic to the deep Q-learning network. Finally, we use this newly developed CDQN-rp algorithm to make trading decisions based on the daily stock prices of Apple (AAPL), Meta (FB), and Bitcoin (BTC) and demonstrate its strengths over other quantitative trading methods.

EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading

“…DRA (Briola et al 2021) uses LSTM and PPO. CDQNRP (Zhu and Zhu 2022) uses a random perturbation to increase the stability of training a DQN. However, these algorithms focus mainly on designing only one RL agent to conduct profitable trading in short-term scenarios, neglecting its failure to maintain performance over long periods.…”

“…PPO and DQN. 4 DRA and CDQNRP• CDQNRP(Zhu and Zhu 2022) uses a random perturbed target frequency to enhance the stability during training. • MACD(Krug, Dobaj, and Macher 2022) is an upgraded method based on the traditional moving average method.…”

Reinforcement Learning: Theory and Applications in HEMS

“…In Equation (42), 𝜋𝜋(𝑠𝑠 𝑡𝑡 (𝑛𝑛)|𝛉𝛉 𝑎𝑎 ) is the output of DDPG's actor. DDPG uses a replay buffer ℬ that includes samples from older policies.…”

“…Other domains where RL has been used include hospital decision making [37], precision agriculture [38], and fluid mechanics [39]. The financial industry is another important sector where RL has been adopted for several scenarios [40][41][42]. It is of little surprise that RL has been extensively used to solve various problems in energy systems [43][44][45][46][47].…”

Optimal strategy of the simultaneous dice game Pig for multiplayers: when reinforcement learning meets game theory

“…Reinforcement learning (RL) serves as an important branch of machine learning. As a powerful approach in decision and control theory, RL has attracted extensive focus, with wide applications in the fields of robotics 1 , quantitative finance 2 , computer vision 3 , healthcare 4 , career planning 5 , gaming 6 etc. The most common object of a games is to beat the opponents, whether they are computers or other human players.…”