Artificial Neural Network Based Reinforcement Learning for Wind Turbine Yaw Control

Abstract

The yaw angle control of a wind turbine allows maximization of the power absorbed from the wind and, thus, the increment of the system efficiency. Conventionally, classical control algorithms have been used for the yaw angle control of wind turbines. Nevertheless, in recent years, advanced control strategies have been designed and implemented for this purpose. These advanced control strategies are considered to offer improved features in comparison to classical algorithms. In this paper, an advanced yaw control strategy based on reinforcement learning (RL) is designed and verified in simulation environment. The proposed RL algorithm considers multivariable states and actions, as well as the mechanical loads due to the yaw rotation of the wind turbine nacelle and rotor. Furthermore, a particle swarm optimization (PSO) and Pareto optimal front (PoF)‐based algorithm have been developed in order to find the optimal actions that satisfy the compromise between the power gain and the mechanical loads due to the yaw rotation. Maximizing the power generation and minimizing the mechanical loads in the yaw bearings in an automatic way are the objectives of the proposed RL algorithm. The data of the matrices Q (s,a) of the RL algorithm are stored as continuous functions in an artificial neural network (ANN) avoiding any quantification problem. The NREL 5‐MW reference wind turbine has been considered for the analysis, and real wind data from Salt Lake, Utah, have been used for the validation of the designed yaw control strategy via simulations with the aeroelastic code FAST.

Performance enhancement of the artificial neural network–based reinforcement learning for wind turbine yaw control

“…A multivariate RL algorithm (two states, two actions, and two immediate reward variables are considered) is proposed in this document. The objective of considering an extended RL algorithm, in comparison to the simple RL algorithm considered in the work Saenz‐Aguirre et al, is to provide an improved characterization of the states, actions, and rewards of the RL algorithm associated with the yaw control system of the wind turbine. The following state, action, and reward variables have been considered in the RL algorithm proposed in this paper.…”

“…where, as it is described in the work of Saenz‐Aguirre et al, P_control [W] refers to the power generation of the wind turbine when the yaw control system is activated, P_no_control [W] refers to the power generation of the wind turbine when the yaw control is not activated, and P_no_deviation [W] refers to the power generation of the wind turbine if the yaw angle was zero. YawMoment [N·m] indicates the value of the sum of the mechanical moment with respect to the z axis induced in the yaw system bearing by performing a concrete action (YawRateK [‐] and YawToMove [deg]) in a defined state (Yaw angle [deg] and Wind speed [m/s]). …”

“…The design process of the ANN‐based RL yaw control strategy of the wind turbine has been carried out with a simplified model of the yaw control system, see Figure . This simplified model is very similar to the model introduced in the work of Saenz‐Aguirre et al However, a limiter to include the actuation of the RL action YawToMove [deg] has been introduced. The successful completion of the design process with the simplified model would prove its validity and enables its use for the online training of the system during operation of the wind turbine.…”

“…The use of estimators to anticipate the wind direction is shown to improve the performance of the classical yaw control methods. Saenz‐Aguirre et al present an artificial neural network (ANN)‐based reinforcement learning (RL) control strategy for the yaw control of a wind turbine. According to the work of Saenz‐Aguirre et al, an RL‐based yaw control algorithm shows important advantages in comparison to the conventional PID‐based yaw control methods, especially in form of lack of control parameters tuning necessity and a fully automatic performance, due to the self‐learning process.…”

“…In this paper, an improved version of the ANN‐based RL yaw control strategy introduced by Saenz‐Aguirre et al is developed. The performance enhancement of the ANN‐based RL yaw control strategy is aimed to improve the operation of the yaw control system for wind speed values over the rated value and to reduce the mechanical moments in the yaw system bearings.…”