Data-based reinforcement learning approximate optimal control for an uncertain nonlinear system with control effectiveness faults

Safe Exploration in Model-based Reinforcement Learning using Control Barrier Functions

“…In contrast to (20), the version of the BE in ( 22) is a function of ∇B and therefore selecting the weight estimates to minimize (22) may not correspond to the minimization of the original BE in (20). That is, even if ( Ŵc , Ŵa ) → W , the BE in ( 22) may be large at certain points in the statespace because of the influence of the safeguarding component of ( 21), making ( 22) a non-ideal performance metric for learning.…”

“…However, since a model of the system is known (or, as discussed in Sec. VI, an approximation of the model is known), the BE can be evaluated at any point in the statespace [28] using a different policy to generate data more representative of (20). To facilitate this approach, define the family of mappings {x i : χ × R ≥t0 → χ} N i=1 such that each x i (x(t), t) ∈ B l (x(t)) maps the current state x(t) to some unexplored point in B l (x(t)).…”

“…(23) Note that the exploratory policy u i is not augmented with the safeguarding controller, thereby allowing for maximum exploration of the state-space by the extrapolated system trajectories without risking safety violation of the original system. Consequently, the BE in ( 23) is representative of the original BE from (20) and is used to update the weight estimates using a recursive least squares update law as 5 [20]…”

“…The proof follows the same steps as that of Theorem 2 and is omitted. See [28], [20] for proofs that establish stability of a similar ADP scheme in the presence of model uncertainty.…”

“…One learning-based control method that has garnered significant attention over the past decade is ADP, which relies on ideas from RL and tools from adaptive control [14] to solve optimal control problems online for potentially uncertain nonlinear systems. Although a variety of such methods have been proposed over the past decade [15], [16], [17], the majority of traditional approaches [18] are limited to unconstrained optimal control problems or those with input constraints [19], [20]. More recently, BFs and ADP have been merged in [21], [22], [23], [24] as a means of bridging the gap between learning-based control and safety-critical control.…”