Adversarial Attacks Against Reinforcement Learning-Based Portfolio Management Strategy

Abstract

Many researchers have incorporated deep neural networks (DNNs) with reinforcement learning (RL) in automatic trading systems. However, such methods result in complicated algorithmic trading models with several defects, especially when a DNN model is vulnerable to malicious adversarial samples. Researches have rarely focused on planning for long-term attacks against RL-based trading systems. To neutralize these attacks, researchers must consider generating imperceptible perturbations while simultaneously reducing the number of modified steps. In this research, an adversary is used to attack an RLbased trading agent. First, we propose an extension of the ensemble of the identical independent evaluators (EIIE) method, called enhanced EIIE, in which information on the best bids and asks is incorporated. Enhanced EIIE was demonstrated to produce an authoritative trading agent that yields better portfolio performance relative to that of an EIIE agent. Enhanced EIIE was then applied to the adversarial agent for the agent to learn when and how much to attack (in the form of introducing perturbations).In our experiments, our proposed adversarial attack mechanisms were > 30% more effective at reducing accumulated portfolio value relative to the conventional attack mechanisms of the fast gradient sign method (FSGM) and iterative FSGM, which are currently more commonly researched and adapted to compare and improve.

Automated Cyber Defence: A Review

“…This focuses on the area includes the robustification of Deep RL algorithms against poisoning and evasion attacks that aim to attack the algorithmic functionalities of the automated agent. While very low number of publications have focused on such attacks for Deep RL algorithms, it is evident that the future cyber attackers will implement such attacks in the future through Deep RL and neural network based research within other domains [14,26,108,130]. If this challenge is not addressed, future networked systems could be vulnerable to algorithmic attacks that could potentially take control of the automated blue agent, and eventually the entire network.…”

Reinforcement Learning for Feedback-Enabled Cyber Resilience

“…Hence, it is imperative to study RL under an adversarial environment. In the last five years, there is a surge in terms of the number of papers that studies the security issues of RL [104,105,106,107,108,109,110,111,112,113,114,115,116,117]. The vulnerabilities of RL comes from the information exchange between the agent and the environment.…”

Reinforcement Learning for Linear Quadratic Control is Vulnerable Under Cost Manipulation

“…Without consistent or accurate feedback from the environment, the agent can either fail to learn an implementable control policy or be tricked into a 'nefarious' control policy favored by the adversaries. In the last five years, there is a surge in the number of research studies that focus on the security threats faced by RL with discrete state space [13], [16]- [23]. Very few if not none of these studies have investigated the security threats to RL-enabled control systems, which often comes with continuous state and action spaces.…”

“…Related works: There is a recent trend in studying security threats faced by RL algorithms [13], [16]- [23], [28]. These studies approach the security threats faced by Rl algorithms through two different vulnerable targets: attacks on the reward or cost signals [13], [18], [19], [21], [23], attacks on the state sensing [15], [20], [22].…”