Model-Free Stochastic Reachability Using Kernel Distribution Embeddings

Abstract

We present a data-driven solution to the terminalhitting stochastic reachability problem for a Markov control process. We employ a nonparametric representation of the stochastic kernel as a conditional distribution embedding within a reproducing kernel Hilbert space (RKHS). This representation avoids intractable integrals in the dynamic recursion of the stochastic reachability problem since the expectations can be calculated as an inner product within the RKHS. We demonstrate this approach on a high-dimensional chain of integrators and on Clohessy-Wiltshire-Hill dynamics.

SReachTools Kernel Module: Data-Driven Stochastic Reachability Using Hilbert Space Embeddings of Distributions

“…By the reproducing property, for any function ∈ H X , we can compute the expectation of with respect to the distribution (• | , ), where ( , ) ∈ X ×U, as an inner product in Hilbert space with the embedding | , [20]. Thus, we can compute the safety probabilities for the first-hitting time problem and the terminal-hitting time problem by computing the expectations in ( 4) and (2) as Hilbert space inner products.…”

“…Thus, we can approximate the expectation of a function ∈ H X using an inner product ˆ | , , H X . As shown in [20,22], we can substitute the inner product into the backward recursion in ( 4) and (2) to approximate the safety probabilities. We have implemented this in SReachTools as KernelEmbeddings.…”

“…As a nonparametric technique, kernel distribution embeddings use principles from functional analysis to embed probability distributions as elements in a high-dimensional Hilbert space. These techniques have applications to Markov models [7], partially-observable models [13,16], and have recently been used to solve stochastic reachability problems [20][21][22]. We incorporate an implementation of the algorithms presented in [20][21][22] into SReachTools, enabling data-driven solutions for stochastic reachability problems that are model-free and distribution agnostic.…”

“…Second, the algorithms largely avoid the curse of dimensionality [1], which can be a significant computational roadblock when solving dynamic programs. Without modification, [20] computes the safety probabilities for a stochastic chain of integrators up to ten thousand dimensions, which is beyond the scope of many existing toolsets. However, the techniques are also amenable to several approximative speedup techniques [6,14,28], which have been shown to reduce the computational complexity down to loglinear time.…”

ARCH-COMP21 Category Report: Stochastic Models

“…The SReachTools Kernel Module [89] is the most recent addition to the toolbox, which implements a collection of nonparametric data-driven algorithms for stochastic reachability based on kernel methods. These algorithms leverage techniques from functional analysis and statistical learning theory and can handle discrete-time linear and nonlinear dynamical systems with arbitrary stochastic disturbances [86,90].…”