Deep Reinforcement Learning for Mitigating Cyber-Physical DER Voltage Unbalance Attacks

Roberts, Ciaran; Ngo, Sy-Toan; Milesi, Alexandre; Scaglione, Anna; Peisert, Sean; Arnold, Dan

doi:10.23919/acc50511.2021.9482815

Cited by 11 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They use PPO and are explicitly agnostic to the concrete attack, but their agent also trains without prior knowledge. Roberts et al (2021) work from a similar premise (compromised DERs), paying special attention to non-DRL controllers, but otherwise achieving similar results.…”

Section: Deep Reinforcement Learning and Its Application In Smart Gri...mentioning

confidence: 89%

Learning new attack vectors from misuse cases with deep reinforcement learning

2023

View full text Add to dashboard Cite

Modern smart grids already consist of various components that interleave classical Operational Technology (OT) with Information and Communication Technology (ICT), which, in turn, have opened the power grid to advanced approaches using distributed software systems and even Artificial Intelligence (AI) applications. This IT/OT integration increases complexity, however, without argument, this advance is necessary to accommodate the rising numbers of prosumers, Distributed Energy Resources (DERs), to enable new market concepts, and to tackle world-wide CO2 emission goals. But the increasing complexity of the Critical National Infrastructure (CNI) power grid gives way to numerous new attack vectors such that a priori robustness cannot be guaranteed anymore and run-time resilience, especially against the “unknown unknowns”, is the focus of current research. In this article, we present a novel combination of so called misuse-case modelling and an approach based on Deep Reinforcement Learning (DRL) to analyze a power grid for new attack vectors. Our approach enables learning from domain knowledge (offline learning), while expanding on that knowledge through learning agents that eventually uncover new attack vectors.

show abstract

Section: Deep Reinforcement Learning and Its Application In Smart Gri...mentioning

confidence: 89%

Learning new attack vectors from misuse cases with deep reinforcement learning

2023

View full text Add to dashboard Cite

show abstract

“…As shown in Fig. 1, in this paper, instead of controlling arbitrarily the parameters of the VV/VW curves, we only shift the value from the default value defining the curves [6,24]. The vector of actions a, output of the NN that approximates the optimum policy, is a function of the three-phase voltages at all, or part, of buses in the distribution system, which is the observation/input of the STGCN-DRL.…”

Section: B Drl Design For Vvcmentioning

confidence: 99%

Deep Reinforcement Learning with Graph ConvNets for Distribution Network Voltage Control

Wu¹,

Carreño²,

Scaglione³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper proposes a model-free Volt-VAR control (VVC) algorithm via the spatio-temporal graph ConvNet-based deep reinforcement learning (STGCN-DRL) framework, whose goal is to control smart inverters in an unbalanced distribution system. We first identify the graph shift operator (GSO) based on the power flow equations. Then, we develop a spatiotemporal graph ConvNet (STGCN), testing both recurrent graph ConvNets (RGCN) and convolutional graph ConvNets (CGCN) architectures, aimed at capturing the spatiotemporal correlation of voltage phasors. The STGCN layer performs the feature extraction task for the policy function and the value function of the reinforcement learning architecture, and then we utilize the proximal policy optimization (PPO) to search the action spaces for an optimum policy function and to approximate an optimum value function. We further utilize the low-pass property of voltage graph signal to introduce an GCN architecture for the the policy whose input is a decimated state vector, i.e. a partial observation. Case studies on the unbalanced 123-bus systems validate the excellent performance of the proposed method in mitigating instabilities and maintaining nodal voltage profiles within a desirable range.

show abstract

Reinforcement Learning using Physics Inspired Graph Convolutional Neural Networks

Scaglione

Arnold

2022

2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

Deep Reinforcement Learning for Mitigating Cyber-Physical DER Voltage Unbalance Attacks

Cited by 11 publications

References 13 publications

Learning new attack vectors from misuse cases with deep reinforcement learning

Learning new attack vectors from misuse cases with deep reinforcement learning

Deep Reinforcement Learning with Graph ConvNets for Distribution Network Voltage Control

Reinforcement Learning using Physics Inspired Graph Convolutional Neural Networks

Contact Info

Product

Resources

About