No abstract

One of the most fundamental problems in Markov decision processes is analysis and control synthesis for safety and reachability specifications. We consider the stochastic reach-avoid problem, in which the objective is to synthesize a control policy to maximize the probability of reaching a target set at a given time, while staying in a safe set at all prior times. We characterize the solution to this problem through an infinite dimensional linear program. We then develop a tractable approximation to the infinite dimensional linear program through finite dimensional approximations of the decision space and constraints. For a large class of Markov decision processes modeled by Gaussian mixtures kernels we show that through a proper selection of the finite dimensional space, one can further reduce the computational complexity of the resulting linear program. We validate the proposed method and analyze its potential with numerical case studies.

We describe an approximate dynamic programming method for stochastic control problems on infinite state and input spaces. The optimal value function is approximated by a linear combination of basis functions with coefficients as decision variables. By relaxing the Bellman equation to an inequality, one obtains a linear program in the basis coefficients with an infinite set of constraints. We show that a recently introduced method, which obtains convex quadratic value function approximations, can be extended to higher order polynomial approximations via sum of squares programming techniques. An approximate value function can then be computed offline by solving a semidefinite program, without having to sample the infinite constraint. The policy is evaluated online by solving a polynomial optimization problem, which also turns out to be convex in some cases. We experimentally validate the method on an autonomous helicopter testbed using a 10-dimensional helicopter model.

We discuss the computational complexity and feasibility properties of scenario based techniques for uncertain optimization programs. We consider different solution alternatives ranging from the standard scenario approach to recursive variants, and compare feasibility as a function of the total computation burden. We identify trade-offs between the different methods depending on the problem structure and the desired probability of constraint satisfaction. Our motivation for this work stems from the applicability and complexity reduction when making decisions by means of recursive algorithms. We illustrate our results on an example from the area of approximate dynamic programming. Index TermsScenario approach, randomized optimization, uncertain systems, approximate dynamic programming.

No abstract

No abstract

We propose a computational approach to approximate the value function and control policies for a finite horizon stochastic reach-avoid problem as follows. First, we formulate an infinite dimensional linear program whose solution characterizes the optimal value function of the stochastic reach-avoid. Next, we introduce sum-of-squares polynomials to approximate the solution of this linear program through a semidefinite program. We compare our proposed tool to alternative numerical approaches via several case studies.

We develop and implement a framework to address autonomous surveillance problems with a collection of pan-tilt (PT) cameras. Using tools from stochastic reachability with random sets, we formulate the problems of target acquisition, target tracking, and acquisition while tracking as reach-avoid dynamic programs for Markov decision processes (MDPs). It is well known that solution methods for MDP problems based on dynamic programming (DP), implemented by state space gridding, suffer from the curse of dimensionality. This becomes a major limitation when one considers a network of PT cameras. To deal with larger problems we propose a hierarchical task allocation mechanism that allows cameras to calculate reach-avoid objectives independently while achieving tasks collectively. We evaluate the proposed algorithms experimentally on a setup involving industrial PT cameras and mobile robots as targets.