ARES: Adaptive Receding-Horizon Synthesis of Optimal Plans

Abstract

Abstract. We introduce ARES, an efficient approximation algorithm for generating optimal plans (action sequences) that take an initial state of a Markov Decision Process (MDP) to a state whose cost is below a specified (convergence) threshold. ARES uses Particle Swarm Optimization, with adaptive sizing for both the receding horizon and the particle swarm. Inspired by Importance Splitting, the length of the horizon and the number of particles are chosen such that at least one particle reaches a next-level state, that is, a state where the cost decreases by a required delta from the previous-level state. The level relation on states and the plans constructed by ARES implicitly define a Lyapunov function and an optimal policy, respectively, both of which could be explicitly generated by applying ARES to all states of the MDP, up to some topological equivalence relation. We also assess the effectiveness of ARES by statistically evaluating its rate of success in generating optimal plans. The ARES algorithm resulted from our desire to clarify if flying in V-formation is a flocking policy that optimizes energy conservation, clear view, and velocity alignment. That is, we were interested to see if one could find optimal plans that bring a flock from an arbitrary initial state to a state exhibiting a single connected V-formation. For flocks with 7 birds, ARES is able to generate a plan that leads to a V-formation in 95% of the 8,000 random initial configurations within 63 seconds, on average. ARES can also be easily customized into a model-predictive controller (MPC) with an adaptive receding horizon and statistical guarantees of convergence. To the best of our knowledge, our adaptive-sizing approach is the first to provide convergence guarantees in receding-horizon techniques.

Attacking the V: On the Resiliency of Adaptive-Horizon MPC

“…Recently, Lukina et al [9] have modeled this problem as a deterministic Markov Decision Process (MDP) M, where the goal was to generate actions that caused M to reach a desired state.…”

“…Once the current acceleration and displacement are sampled, the next state is uniquely determined by (1) from the current state in M [9]. The problem of whether we can go from a random flock to a V-formation is a reachability question.…”

“…The reachability goal is the set of states representing a V-formation. A key assumption in [9] was that the reachability goal can be specified using a fitness function J, which assigns a non-negative real (fitness) value to each state in M.…”

“…Thus, AMPC can adapt to the severity of the action played by its adversary by choosing its own horizon accordingly. The AMPC procedure is inspired by an adaptive optimization procedure recently presented in [9], which dynamically changes the amount of the effort it uses to search for a better solution in each step. The motivation for adaptation in [9] however was different, namely to take the optimizer out of a local minimum, and thus, ensure convergence to a global optimum.…”

“…If no new level is reached within h max horizons, the first action of the best a h using horizon h max is applied. The dynamic threshold ∆ i is defined as in [9]. Its initial value ∆ 0 is obtained by dividing the fitness range to be covered into m equal parts, that is, This approach allows us to force PSO to escape from a local minimum, even if this implies passing over a bump, by gradually increasing the exploration horizon h. We assume that the MDP is controllable and that the set G of good states is not empty, which means, that from any state, it is possible to reach a state whose fitness decreased by at least ∆ i .…”