A New Model-Free Space Vector Modulation Technique for Multilevel Inverters Based On Deep Reinforcement Learning

Qashqai, Pouria; Al‐Haddad, Kamal; Zgheib, Rawad

doi:10.1109/iecon43393.2020.9254989

Cited by 8 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [29], a DRL approach was introduced for the operation and control of a three-level NPC (Neutral Point Clamped) converter, utilizing a DQN (Deep Q-Networks) agent. This method was compared to traditional MPC controllers and demonstrated enhanced robustness.…”

Section: A Related Workmentioning

confidence: 99%

Empowering Dynamic Active and Reactive Power Control: A Deep Reinforcement Learning Controller for Three-Phase Grid-Connected Electric Vehicles

Ahmadian,

Sedghisigarchi,

Gadh

2024

IEEE Access

View full text Add to dashboard Cite

Advancing bidirectional voltage source inverters in the on-board chargers of electric vehicles is vital for smart grid electric power management. These electric vehicle inverters play a crucial role in actively contributing to active/reactive power flow control in vehicle-to-grid and grid-to-vehicle applications. In this research, a proof-of-concept, model-free reinforcement learning-based agent/controller is introduced that eliminates the need for system identification or linearization techniques for control design. The goal is to dynamically control user-specified active and reactive electric vehicle on-board charger power outputs in all four quadrants for three-phase grid-connected electric vehicle voltage source inverters. A Twin-Delayed Deep Deterministic Policy Gradient deep-reinforcement learning-based controller was designed (1) to smoothly learn continuous control variables and to generate deterministic actions from the learned control policy, (2) to avoid exhaustive system identification techniques associated with nonlinearities involved in the on-board charger of electric vehicles, and (3) to ultimately track continuous active and reactive powers across all four-quadrant operation modes, ensuring safe and minimal steady-state errors. In this paper, a Markov Decision Process framework was established, encompassing the environment, actions, states, and a unique reward function. The minimal steady state error achieved in active and reactive power reference tracking across all four quadrant operation modes substantiates the trained agent's capacity to sustain tracking accuracy and stability across a variety of operational settings. This research could potentially pave the way for any type of electric vehicle to actively play a role in smart grid management, especially through vehicle-to-grid and grid-to-vehicle capabilities.INDEX TERMS EV (Electric Vehicle), OBC (On-Board Charger), VSI (Voltage Source Inverter), RL (Reinforcement Learning).

show abstract

Section: A Related Workmentioning

confidence: 99%

Empowering Dynamic Active and Reactive Power Control: A Deep Reinforcement Learning Controller for Three-Phase Grid-Connected Electric Vehicles

Ahmadian,

Sedghisigarchi,

Gadh

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…For instance, in [40], a DRL agent is used for the efficiency optimization design of a three-level NPC. In [41], a model-free DRL method for controlling the three-level NPC is proposed. This method utilizes an actor-critic method to apply all the possible switching states under different conditions to learn the optimal switching algorithm that satisfies the control objectives.…”

Section: Big Data Dnn Figure 1 Diagram Of Overlap Between Data Scienc...mentioning

confidence: 99%

A Model-Free Switching and Control Method for Three-Level Neutral Point Clamped Converter Using Deep Reinforcement Learning

Qashqai,

Babaie,

Zgheib

et al. 2023

IEEE Access

Self Cite

View full text Add to dashboard Cite

This paper presents a novel model-free switching and control method for three-level neutral point clamped (NPC) converter using deep reinforcement learning (DRL). In this method, voltage balancing, and selection of optimal switches are achieved using a reward function which is calculated based on various signals measured as observations of the DRL agent. Since the action space is discrete, a deep Q-network (DQN) agent is utilized. DQN is used due to its capability of handling high-dimensional state spaces. In order to highlight its pros and cons, the proposed method is compared with model predictive control (MPC), which is another popular non-linear control method for power electronic converters. The proposed method is evaluated and compared with the MPC method in grid-connected mode using simulations in Matlab/Simulink. To evaluate the practical performance of the DRL method, various experimental results are obtained. The simulation and experimental results demonstrate that the proposed method effectively achieves accurate voltage balancing and ensures steady operation even in the presence of various dynamic changes, including variations in the reference currents and grid voltage. Additionally, the method successfully handles uncertainties, such as sensor measurement noise, and accommodates parameter variations, such as changes in the capacity of the DC-link capacitors and line impedance. The results demonstrate that this method exhibits superior adaptability to real-time changes and uncertainties, delivering more robust performance compared to similar conventional methods like MPC. Thus, this method can be considered a promising approach for intelligent control of power electronic converters, especially when conventional methods such as MPC face challenges in performance and accuracy under severe parameter variations and uncertainties.

show abstract

“…As described in above Section, the DQN agent in RL is based on the Q-function defined in (13) and acts to maximize the expected reward under the given state. In other words, the design of an appropriate reward function is one of the most important factors in reinforcement learning because the evaluation of state and action is performed with a reward function.…”

Section: B Reward Functionmentioning

confidence: 99%

“…In order to train the agent, various reward functions have been applied in many studies. [10][11][12][13][14] Fig. 5 depicts commonly used four reward functions.…”

Section: B Reward Functionmentioning

confidence: 99%

“…The RL based controller can smooth the voltage tracking while operating under large power loads variations. The paper [13] proposes a space vector modulation based on RL using the DQN agent for d-q current and neutralpoint voltage controls in a three-level NPC converter. In spite of effective results, the use of RL for simple tasks that can be solved with existing control algorithms should be reconsidered.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning Based Modulation for Balancing Capacitor Voltage and Thermal Stress to Enhance Current Capability of MMCs

Jung

Hosseini

et al. 2022

2022 IEEE 13th International Symposium on Power Electronics for Distributed Generation Systems (PEDG)

View full text Add to dashboard Cite

Balancing DC capacitor voltage of many submodules (SMs) is one of the important issues in modular multilevel converter (MMC) systems. In addition, the balance of thermal stress between SMs should be considered to equalize the lifetime expectation of semiconductors and to enhance the current capability of MMC systems. However, it is complicated to balance all the various factors satisfactorily at the same time. Recent machine learning (ML) techniques can achieve optimal results through learning using numerous data acquired in complex environments. Therefore, this paper proposes a new modulation based on reinforcement learning (RL), which is a subclass of ML methods, to optimally balance the capacitor voltage and thermal stress of SMs. A deep Q-network (DQN) agent, which is one of the RL algorithms, is applied in accordance with a nearest-level modulation (NLM), and main features of the DQN agent are described in this paper. The effectiveness of the proposed modulation based on RL is verified by simulations results.

show abstract

A New Model-Free Space Vector Modulation Technique for Multilevel Inverters Based On Deep Reinforcement Learning

Cited by 8 publications

References 20 publications

Empowering Dynamic Active and Reactive Power Control: A Deep Reinforcement Learning Controller for Three-Phase Grid-Connected Electric Vehicles

Empowering Dynamic Active and Reactive Power Control: A Deep Reinforcement Learning Controller for Three-Phase Grid-Connected Electric Vehicles

A Model-Free Switching and Control Method for Three-Level Neutral Point Clamped Converter Using Deep Reinforcement Learning

Reinforcement Learning Based Modulation for Balancing Capacitor Voltage and Thermal Stress to Enhance Current Capability of MMCs

Contact Info

Product

Resources

About