Data‐enabled extremum seeking: A cooperative concurrent learning‐based approach

Abstract

Summary This paper introduces a new class of feedback‐based data‐driven extremum seeking algorithms for the solution of model‐free optimization problems in smooth continuous‐time dynamical systems. The novelty of the algorithms lies on the incorporation of memory to store recorded data that enables the use of information‐rich datasets during the optimization process, and allows to dispense with the time‐varying dither excitation signal needed by standard extremum seeking algorithms that rely on a persistence of excitation (PE) condition. The model‐free optimization dynamics are developed for single‐agent systems, as well as for multi‐agent systems with communication graphs that allow agents to share their state information while preserving the privacy of their individual data. In both cases, sufficient richness conditions on the recorded data, as well as suitable optimization dynamics modeled by ordinary differential equations are characterized in order to guarantee convergence to a neighborhood of the solution of the extremum seeking problems. The performance of the algorithms is illustrated via different numerical examples in the context of source‐seeking problems in multivehicle systems.

Assured learning‐enabled autonomy: A metacognitive reinforcement learning framework

“…It is immediate from (17) that 𝜉 𝜃 (x, t, 𝜎) < 0 results in 𝜉 a 𝜃 (x, t, 𝜎) < 0. On the other hand, if 𝜉 𝜃 (x, t, 𝜎) > 0, then based on (16), 𝜌 𝜎 i (x, t) > 0 for all i ∈ [1, … , N + 1] and thus − log(e −𝜌 𝜎 i (x,t) ) > 0, , i = [1, … , N + 1], which completes the proof. ▪ Now, a fitness function for a hyperparameter vector 𝜃 as a metric for measuring the system's safety and liveness in terms of overall robustness is defined as…”

“…Lemma 3. Consider conjunction of N + 1 predicate functions given in (13) and their overall robustness 𝜉(x, t, 𝜎) defined in (16). Then,…”

“…13 That is, current RL practice has mainly been used to achieve pre-specified goals in structured environments by handcrafting a cost or reward function for which its minimization guarantees reaching the goal. [14][15][16][17][18][19] Strong AI, on the other hand, holds the promise of designing agents that can learn to achieve goals across multiple circumstances by generalizing to unforeseen and novel situations. As the designer cannot foresee all the circumstances that the agent might encounter, pre-specifying and handcrafting the reward function cannot guarantee reaching goals in an uncertain and non-stationary environment.…”

Data-Enabled Gradient Flow as Feedback Controller: Regulation of Linear Dynamical Systems to Minimizers of Unknown Functions

“…Assumption 8 is satisfied if the basis functions {b i } N i=1 and {d i } M i=1 are strongly smooth. Assumption 9 is a technical condition that is required in our proof; nevertheless, this condition is satisfied for parametric convex models and several non-parametric models; see Hastie et al (2009); Bazerque and Giannakis (2013), and the recent results of Poveda et al (2021).…”

“…Related Works. We note that a key differentiating aspect relative to extremum seeking methods (see, e.g., Krstic and Wang (2000); Ariyur and Krstić (2003); Teel and Popovic (2001) and many others), the Q-learning of Devraj and Meyn (2017), and methods based on concurrent learning Chowdhary and Johnson (2010); Chowdhary et al (2013); Poveda et al (2021) is that we consider a setting where only sporadic functional evaluations are available (i.e., we do not have continuous access to functional evaluations). Regarding the problem of regulating LTI systems towards solutions of optimization problems, existing approaches leveraged gradient flows in Menta et al (2018); Bianchin et al (2020), proximal-methods in Colombino et al (2020), saddle-flows in Brunner et al (2012), prediction-correction methods in Zheng et al (2020), and the hybrid accelerated methods proposed in Bianchin et al (2020).…”