Robo-Advising: Enhancing Investment with Inverse Optimization and Deep Reinforcement Learning

Abstract

Machine Learning (ML) has been embraced as a powerful tool by the financial industry, with notable applications spreading in various domains including investment management. In this work, we propose a full-cycle data-driven investment roboadvising framework, consisting of two ML agents. The first agent, an inverse portfolio optimization agent, infers an investor's risk preference and expected return directly from historical allocation data using online inverse optimization. The second agent, a deep reinforcement learning (RL) agent, aggregates the inferred sequence of expected returns to formulate a new multi-period mean-variance portfolio optimization problem that can be solved using deep RL approaches. The proposed investment pipeline is applied on real market data from April 1, 2016 to February 1, 2021 and has shown to consistently outperform the S&P 500 benchmark portfolio that represents the aggregate market optimal allocation. The outperformance may be attributed to the the multi-period planning (versus single-period planning) and the data-driven RL approach (versus classical estimation approach).

Recent advances in reinforcement learning in finance

“…The authors proved that, with high probability, the proposed exploration-exploitation algorithm performs near optimally with the number of time steps depending polynomially on various model parameters. Wang and Yu (2021) proposed an investment robo-advising framework consisting of two agents. The first agent, an inverse portfolio optimization agent, infers an investor's risk preference and expected return directly from historical allocation data using online inverse optimization.…”

“…As introduced in Section 4.6, Alsabah et al (2021) considered learning within a set of 𝑚 prespecified investment portfolios, and Wang and Yu (2021) and developed learning algorithms and procedures to infer risk preferences, respectively, under the framework of Markowitz mean-variance portfolio optimization. It would be interesting to consider a modelfree RL approach where the robo-advisor has the freedom to learn and improve decisions beyond a prespecified set of strategies or the Markowitz framework.…”

Intelligent Systematic Investment Agent: an ensemble of deep learning and evolutionary strategies

“…We found very limited literature that focuses on wealth building over time by making marginally better daily buying decisions over a lifetime of investing (Wang & Yu, 2021;Philps et al, 2018). However, these models still rely on either a 12 month evaluation window to train agents or are more focused on risk profile assessment and portfolio balancing over time.…”

Recent Advances in Reinforcement Learning in Finance

“…Robo-advising in a Model-free Setting. As introduced in Section 4.6, [9] considered learning within a set of m pre-specified investment portfolios, and [209] and [237] developed learning algorithms and procedures to infer risk preferences, respectively, under the framework of Markowitz mean-variance portfolio optimization. It would be interesting to consider a model-free RL approach where the roboadvisor has the freedom to learn and improve decisions beyond a pre-specified set of strategies or the Markowitz framework.…”