Reinforcement Learning for Autonomous Defence in Software-Defined Networking

The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning

“…The literature on test-time attacks on DRL that are not based on adversarial examples is still very scarce. For instance, Han et al [55] investigate the case of a DRL agent in a Software Defined Network (SDN), tasked with preventing the propagation of a malware in the network by identifying the compromised nodes and deciding on taking one of the following actions at each time step: isolating and patching a node, reconnecting a node and its links, migrating the critical server, and taking no action. The reward value for this agent depends on whether the critical servers are compromised and the number of reachable nodes from such servers, as well as the number of compromised nodes, and the cost of migration.…”

“…It is also assumed that the detection mechanism of the agent can be manipulated by the adversary (i.e., the adversary can induce False Positive (FP) or False Negative (FN) results in the detector), but is constrained by a threshold on how many such manipulations can be implemented at each time step. The test-time attacks proposed in [55] are two-fold: indiscriminate attacks aim to prevent the DRL agent from taking the optimal action a t at time t, and targeted attacks aim to force the agent into taking a specific action a t at time t. Considering DDQN and A3C as DRL algorithms for the target agent, the objective for targeting DDQN agents is to maximize Q(s t + δ t , a t ) for action a t at state s t using perturbation δ t . Similarly, the objective for targeting A3C is to maximize π(a t |s t + δ t ) for the stochastic policy π.…”

“…Similarly, the objective for targeting A3C is to maximize π(a t |s t + δ t ) for the stochastic policy π. For these attacks, [55] develops a whitebox attack methodology, where the attacker can access the target's model. This attack requires the computation of those nodes whose FP or FN detection would facilitate the adversarial objective.…”

“…This attack requires the computation of those nodes whose FP or FN detection would facilitate the adversarial objective. Accordingly, [55] proposes an integer programming approach to deriving the set of such nodes at each time step. The authors also propose a blackbox attack technique, which is based on training surrogate models of the target with either the same or different hyperparameters and then following the procedure of the whitebox attacks.…”

“…Similar to the case of test-time attacks, the body of work on training-time attacks that are not based on adversarial examples is very thin. In the previously discussed paper by Han et al [55], the target model is considered to be an online learner, and hence the authors investigate attacks that aim to manipulate the training phase of the target DRL. To this end, [55] presents a poisoning attack based on flipping the reward signs, with the goal of maximizing the loss function in the target DDQN agent.…”