Universal Trading for Order Execution with Oracle Policy Distillation

Abstract

As a fundamental problem in algorithmic trading, order execution aims at fulfilling a specific trading order, either liquidation or acquirement, for a given instrument. Towards effective execution strategy, recent years have witnessed the shift from the analytical view with model-based market assumptions to model-free perspective, i.e., reinforcement learning, due to its nature of sequential decision optimization. However, the noisy and yet imperfect market information that can be leveraged by the policy has made it quite challenging to build up sample efficient reinforcement learning methods to achieve effective order execution. In this paper, we propose a novel universal trading policy optimization framework to bridge the gap between the noisy yet imperfect market states and the optimal action sequences for order execution. Particularly, this framework leverages a policy distillation method that can better guide the learning of the common policy towards practically optimal execution by an oracle teacher with perfect information to approximate the optimal trading strategy. The extensive experiments have shown significant improvements of our method over various strong baselines, with reasonable trading actions.

Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance

“…In quantitative finance, the primary goal of the investor is to maximize the long-term value through continuously trading of multiple assets in the market [1,2]. The process consists of two parts, portfolio management, which dynamically allocate the portfolio across the assets, and order execution whose goal is to fulfill a number of acquisition or liquidation orders specified by the portfolio management strategy, within a time horizon, and close the loop of investment [3,4]. Figure (1a) presents the trading process within one trading day.…”

“…Although there exists many works for order execution, few of them manage to address the above three challenges. Traditional financial model based methods [5][6][7] and some recently developed model-free reinforcement learning (RL) methods [4,8,9] only optimize the strategy for single-order execution without considering practice of multi-order execution, which would result in low trading efficacy. Moreover, it is not applicable to directly transfer the existing methods to multi-order execution since utilizing only one agent to conduct the execution of multiple orders would lead to scalability issue as the action space of one individual agent grows exponentially with the number of orders.…”

Reinforcement Learning with Automated Auxiliary Loss Search

“…Reinforcement learning (RL) has achieved remarkable progress in games [31,47,50], financial trading [8] and robotics [13]. However, in its core part, without designs tailored to specific tasks, general RL paradigms are still learning implicit representations from critic loss (value predictions) and actor loss (maximizing cumulative reward).…”

LSTMcon: A Novel System of Portfolio Management Based on Feedback LSTM with Confidence

“…Market traders buy and sell volatile assets frequently, with a goal to maximize their total return. Nowadays, it is not difficult for us to find trading strategies suitable for our preferences in many academic articles or forums [1]. However, it is still a problem of how to distinguish the good and bad of these strategies and avoid making some common mistakes, such as survivorship bias, look-ahead bias, and trading cost [2].…”

“…Let RP be the score given by RP, Seq be the score of Consecutive Days and His of Historical Data. The mapping relationship is: Conf = (His + Seq + RP − 1000)/50000 + 0.94(1)…”