Cyberspace attack and defense game based on reward randomization reinforcement learning

Learning Near-Optimal Intrusion Responses Against Dynamic Attackers

“…7: A mixed threshold strategy where σ( θ(1) l ) is the threshold (0.5 in this example); the x-axis indicates the defender's belief state b(1) ∈ [0, 1] and the y-axis indicates the probability prescribed by π1, θ (1) to the stop action S. (32). Similarly, the attacker's best response strategy π2 is parameterized with the vector θ(2) ∈ R 2L (33).…”

“…Although a growing body of work uses reinforcement learning and game theory to find intrusion response strategies (see Section VII), a direct comparison between the defender strategies learned in our framework and those found in previous work is not feasible for two reasons. First, nearly all of the prior works have developed defender strategies for custom simulations [8], [10], [10], [11], [18]- [27], [33]- [35], [38], [39], [60], [62], [63], [67]- [72], [107]- [119] and there is no obvious way to map their solutions to an emulated environment like ours (see Fig. 1 and Appendix C).…”

“…A large number of studies have focused on applying reinforcement learning to use cases similar to the intrusion response use case we discuss in this paper [9]- [11], [17]- [52], [64], [72]. These works use a variety of models, including MDPs [20], [23], [25], [26], [31], [34], [36], [42], [51], [52], [64], Stochastic games [10], [18], [28], [33], [45], [72], attack graphs [35], Petri nets [43], and POMDPs [9], [11], [21], [27], as well as various reinforcement learning algorithms, including Q-learning [18], [20], [23], [40], [43], [48], [64], [69], SARSA [21], PPO [10], [11], [34], [35], [37], hierarchical reinforcement learning [25], DQN [26], [36]-…”

“…Third, our method to find effective defender strategies includes using an emulation system in addition to a simulation system. The advantage of our method compared to the simulation-only approaches [10], [11], [18]- [27], [33]- [35], [38], [39], [44]- [46], [50], [52], [52], [64], [69], [70], [72] is that the parameters of our simulation system are determined by measurements from an emulation system instead of being chosen by a human expert. Further, the learned strategies are evaluated in the emulation system, not in the simulation system.…”

“…Third, we provide evaluation results from an emulated infrastructure. This addresses a drawback in related research that relies solely on simulations to learn and evaluate strategies [8], [10], [11], [18]- [27], [33]- [35], [38], [39], [43]- [46], [50], [52], [54], [60], [62]- [72].…”