Reinforcement-Learning Based Portfolio Management with Augmented Asset Movement Prediction States

Abstract

Portfolio management (PM) is a fundamental financial planning task that aims to achieve investment goals such as maximal profits or minimal risks. Its decision process involves continuous derivation of valuable information from various data sources and sequential decision optimization, which is a prospective research direction for reinforcement learning (RL). In this paper, we propose SARL, a novel State-Augmented RL framework for PM. Our framework aims to address two unique challenges in financial PM: (1) data heterogeneity – the collected information for each asset is usually diverse, noisy and imbalanced (e.g., news articles); and (2) environment uncertainty – the financial market is versatile and non-stationary. To incorporate heterogeneous data and enhance robustness against environment uncertainty, our SARL augments the asset information with their price movement prediction as additional states, where the prediction can be solely based on financial data (e.g., asset prices) or derived from alternative sources such as news. Experiments on two real-world datasets, (i) Bitcoin market and (ii) HighTech stock market with 7-year Reuters news articles, validate the effectiveness of SARL over existing PM approaches, both in terms of accumulated profits and risk-adjusted profits. Moreover, extensive simulations are conducted to demonstrate the importance of our proposed state augmentation, providing new insights and boosting performance significantly over standard RL-based PM method and other baselines.

Time Your Hedge With Deep Reinforcement Learning

“…Secondly, open high low prices tend to be highly correlated creating some noise in the inputs. Third, the concept of volatility is crucial to detect regime change and is surprisingly absent from these works as well as from other works like (Yu et al 2019;Wang and Zhou 2019;Liu et al 2020;Ye et al 2020;Li et al 2019;Xiong et al 2019).…”

“…Third, there is no consideration of online learning to adapt to changing environment as well as the incorporation of transaction costs. A second stream of research around deep reinforcement learning has emerged to address these points (Jiang and Liang 2016;Jiang, Xu, and Liang 2017;Liang et al 2018;Yu et al 2019;Wang and Zhou 2019;Liu et al 2020;Ye et al 2020;Li et al 2019;Xiong et al 2019;Benhamou et al 2020a;2020b). The dynamic nature of reinforcement learning makes it an obvious candidate for changing environment (Jiang and Liang 2016;Jiang, Xu, and Liang 2017;Liang et al 2018).…”

“…The dynamic nature of reinforcement learning makes it an obvious candidate for changing environment (Jiang and Liang 2016;Jiang, Xu, and Liang 2017;Liang et al 2018). Transaction costs can be easily included in rules (Liang et al 2018;Yu et al 2019;Wang and Zhou 2019;Liu et al 2020;Ye et al 2020;Yu et al 2019). However, these works, except (Ye et al 2020) and (Benhamou et al 2020a) rely only on time series of open high low close prices, which are known to be very noisy.…”

“…Transaction costs can be easily included in rules (Liang et al 2018;Yu et al 2019;Wang and Zhou 2019;Liu et al 2020;Ye et al 2020;Yu et al 2019). However, these works, except (Ye et al 2020) and (Benhamou et al 2020a) rely only on time series of open high low close prices, which are known to be very noisy. Secondly, they all assume an immediate action after observing prices which is quite different from reality.…”

Equity Research Report-Driven Investment Strategy in Korea Using Binary Classification on Stock Price Direction

“…Such studies either design a neural network with an ensemble of evolving clustering and LSTM [36], or propose a new follow-the-loser portfolio strategy from the post of stock micro-blogs using semi-supervised learning method [37], or establish a trading strategy from new sentiment data using learning-to-rank algorithms [38]. Also, recently, a portfolio investment strategy that considers shareholders' confidence index by combining the existing random forest and sentimental analysis [39] and an investment strategy that encodes external information from financial news using reinforcement learning have been proposed [40].…”