Stochastic Optimal Control via Hilbert Space Embeddings of Distributions

Data-Driven Chance Constrained Control using Kernel Distribution Embeddings

“…Kernel distribution embeddings have been thoroughly studied in the recent years (Song et al, 2009;Smola et al, 2007;Grünewälder et al, 2012;Park and Muandet, 2020), but are not yet popular within the control community. Controller synthesis applications have been explored in (Thorpe and Oishi, 2021), but the authors do not consider constraints and only consider one-step transition kernels. In contrast, we tackle a joint chance constrained problem formulation that explicitly accounts for constraints and represent the distribution of the entire state trajectory.…”

SOCKS: A Stochastic Optimal Control and Reachability Toolbox Using Kernel Methods

“…Because 𝑄 is unknown, the integral with respect to 𝑄 in (3) is intractable. Thus, we form an approximation of Problem 1 by computing an empirical approximation of the integral operator with respect to 𝑄 using the sample S. Following [51], we can view this as a learning problem by embedding the integral operator as an element in a high-dimensional space of functions known as a reproducing kernel Hilbert space. Details regarding the kernelbased stochastic optimal control method are provided in [51] and in Appendix A.…”

“…Kernel distribution embeddings have been applied to modeling of Markov processes [23,44], robust optimization [57], and statistical inference [45]. In addition, these techniques have also been applied to solve stochastic reachability problems [50,54], forward reachability analysis [52], and to solving stochastic optimal control problems [29,51]. Because these techniques are inherently data-driven, SOCKS can accommodate systems with nonlinear dynamics, black-box elements, and arbitrary stochastic disturbances.…”

“…Data-driven stochastic optimal control is an active area of research [17,18], and provides a promising avenue for controls problems which suffer from high model complexity or system uncertainty, such as robotic motion planning [25,30] and model predictive control [39,40]. Recently, approaches using Gaussian processes [16,36] and kernel methods [23,29,51] have also been explored. In SOCKS, we implement the algorithms in [51], which uses data consisting of observations of the system evolution to compute an implicit approximation of the dynamics in a reproducing kernel Hilbert space (RKHS).…”

“…Recently, approaches using Gaussian processes [16,36] and kernel methods [23,29,51] have also been explored. In SOCKS, we implement the algorithms in [51], which uses data consisting of observations of the system evolution to compute an implicit approximation of the dynamics in a reproducing kernel Hilbert space (RKHS). The novelty of the approach in [51] is that it exploits the structure of the RKHS to approximate the stochastic optimal control problem as a linear program that converges in probability to the original problem, and computes an approximately optimal controller without invoking a model-based approach.…”