Commission Fee is not Enough: A Hierarchical Reinforced Framework for Portfolio Management

Wang, Rundong; Wei, Hongxin; An, Bo; Feng, Zhouyan; Yao, Jun

doi:10.1609/aaai.v35i1.16142

Cited by 20 publications

(9 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are some hierarchical RL frameworks for quantitative trading. HRPM (Wang et al 2021) utilizes a hierarchical framework to simulate portfolio management and order execution. MetaTrader (Niu, Li, and Li 2022) proposes a router to pick the most suitable strategy for the current market situation.…”

Section: Rl For Quantitative Tradingmentioning

confidence: 99%

EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading

Qin,

Sun,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

High-frequency trading (HFT) is using computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market, (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorithms fail to maintain satisfactory performances. To tackle these challenges, we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action value based on dynamic programming, for enhancing the performance and training efficiency of second level RL agents. In stage II, we construct a pool of diverse RL agents for different market trends, distinguished by return rates, where hundreds of RL agents are trained with different preferences of return rates and only a tiny fraction of them will be selected into the pool based on their profitability. In stage III, we train a minute-level router which dynamically picks a second-level agent from the pool to achieve stable performance across different markets. Through extensive experiments in various market trends on Crypto markets in a high-fidelity simulation trading environment, we demonstrate that EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.

show abstract

Section: Rl For Quantitative Tradingmentioning

confidence: 99%

EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading

Qin,

Sun,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…However, all these methods target on individual order execution and do not consider practical constraints under multi-order execution as shown in Figure (1b), leading to sub-optimal or impractical trading behaviors. Although MARL has been widely adopted in financial area for market simulation [28][29][30][31] and portfolio management [7,32,33], these is no existing method utilizing MARL directly for order execution. To our best knowledge, this is the first work using MARL for multi-order execution task with practical constraint.…”

Section: Related Work 21 Rl For Order Executionmentioning

confidence: 99%

Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance

Tang

Ren

et al. 2023

Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Order execution is a fundamental task in quantitative finance, aiming at finishing acquisition or liquidation for a number of trading orders of the specific assets. Recent advance in model-free reinforcement learning (RL) provides a data-driven solution to the order execution problem. However, the existing works always optimize execution for an individual order, overlooking the practice that multiple orders are specified to execute simultaneously, resulting in suboptimality and bias. In this paper, we first present a multi-agent RL (MARL) method for multi-order execution considering practical constraints. Specifically, we treat every agent as an individual operator to trade one specific order, while keeping communicating with each other and collaborating for maximizing the overall profits. Nevertheless, the existing MARL algorithms often incorporate communication among agents by exchanging only the information of their partial observations, which is inefficient in complicated financial market. To improve collaboration, we then propose a learnable multi-round communication protocol, for the agents communicating the intended actions with each other and refining accordingly. It is optimized through a novel action value attribution method which is provably consistent with the original learning objective yet more efficient. The experiments on the data from two real-world markets have illustrated superior performance with significantly better collaboration effectiveness achieved by our method.

show abstract

“…Compared to the supervised learning methods, RL provides a seamless and flexible framework for portfolio management [43]. With different risk appetite, previous RL-based portfolio management algorithms adopt various reward functions including the Sharpe ratio [49], the maximum drawdown [1,51], and the total profits [21,50,54]. The performance of the methods utilizing a pure RL objective is constrained by not fully exploring the fluctuate markets.…”

Section: Related Workmentioning

confidence: 99%

MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization

Niu,

Li,

2022

Preprint

View full text Add to dashboard Cite

Portfolio management is a fundamental problem in finance. It involves periodic reallocations of assets to maximize the expected returns within an appropriate level of risk exposure. Deep reinforcement learning (RL) has been considered a promising approach to solving this problem owing to its strong capability in sequential decision making. However, due to the non-stationary nature of financial markets, applying RL techniques to portfolio optimization remains a challenging problem. Extracting trading knowledge from various expert strategies could be helpful for agents to accommodate the changing markets. In this paper, we propose MetaTrader, a novel two-stage RL-based approach for portfolio management, which learns to integrate diverse trading policies to adapt to various market conditions. In the first stage, MetaTrader incorporates an imitation learning objective into the reinforcement learning framework. Through imitating different expert demonstrations, MetaTrader acquires a set of trading policies with great diversity. In the second stage, MetaTrader learns a meta-policy to recognize the market conditions and decide on the most proper learned policy to follow. We evaluate the proposed approach on three real-world index datasets and compare it to state-of-the-art baselines. The empirical results demonstrate that MetaTrader significantly outperforms those baselines in balancing profits and risks. Furthermore, thorough ablation studies validate the effectiveness of the components in the proposed approach. CCS CONCEPTS• Computing methodologies → Reinforcement learning; • Applied computing → Economics.

show abstract

Commission Fee is not Enough: A Hierarchical Reinforced Framework for Portfolio Management

Cited by 20 publications

References 15 publications

EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading

EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading

Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance

MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization

Contact Info

Product

Resources

About