Spatio-Temporal Stochastic Optimization: Theory and Applications to Optimal Control and Co-Design

Abstract

There is a rising interest in Spatio-temporal systems described by Partial Differential Equations (PDEs) among the control community. Not only are these systems challenging to control, but the sizing and placement of their actuation is an NP-hard problem on its own. Recent methods either discretize the space before optimziation, or apply tools from linear systems theory under restrictive linearity assumptions. In this work we consider control and actuator placement as a coupled optimization problem, and derive an optimization algorithm on Hilbert spaces for nonlinear PDEs with an additive spatio-temporal description of white noise. We study first and second order systems and in doing so, extend several results to the case of second order PDEs. The described approach is based on variational optimization, and performs joint RL-type optimization of the feedback control law and the actuator design over episodes. We demonstrate the efficacy of the proposed approach with several simulated experiments on a variety of SPDEs.

Stochastic Spatio-Temporal Optimization for Control and Co-Design of Systems in Robotics and Applied Physics

“…Our approach is founded on a general principle coming from thermodynamics that also has had success in stochastic optimal control literature [11] Free Energy ≤ Work − Temperature × Entropy (1) We leverage this principle in order to derive a measure-theoretic loss function that utilizes exponential averaging over importance sampled system trajectories in order to choose network and actuator design parameters that simultaneously minimize state cost and control effort. This work unifies our prior work in [12] and [13], and provides several extensions including scaling policy and actuator co-design optimization to large 2D systems, and extending the approach to complex nonlinear 2D second-order systems that more closely resemble a soft-robotic limb. Specifically we contribute the following:…”

“…Consider the class of SPDEs that are of semi-linear form. Similar mathematical preliminaries are treated in our prior work [13]. Let H denote a separable Hilbert space with inner product •, • , σ-field B(H), and probability space (Ω, F, P) with filtration F t , t ∈ [0, T ].…”

Leveraging Stochasticity for Open Loop and Model Predictive Control of Spatio-Temporal Systems

“…In our experiments we apply model predictive control through re-optimization and turn Equation ( 12) into an implicit feedback-type control. Optimization using Equation (9) with policies that explicitly depend on the stochastic field is also possible and is considered, using gradient-based optimization in [19][20][21].…”

“…In SDE control, probabilistic representations of the HJB PDE can solve scalability via sampling techniques [15,16], including iterative sampling and/or parallelizable implementations [17,18]. These methods have been explored in a reinforcement learning context for SPDEs [19][20][21].…”

Stochastic optimization for learning quantum state feedback control

“…This is discussed with greater detail in the supplemental material, including some common instances of degeneracy. These degeneracies prove prohibitive for a variety of methods introduced in the stochastic optimal control literature, including path integral control [56][57][58][59], forwardbackward stochastic differential equations using importance sampling [60,61], and recently spatio-temporal stochastic optimization [62,63]. In each case, such degeneracies must be carefully addressed.…”