Distributed Algorithms for Linearly-Solvable Optimal Control in Networked Multi-Agent Systems

Abstract

Distributed algorithms for both discrete-time and continuous-time linearly solvable optimal control (LSOC) problems of networked multi-agent systems (MASs) are investigated in this paper. A distributed framework is proposed to partition the optimal control problem of a networked MAS into several local optimal control problems in factorial subsystems, such that each (central) agent behaves optimally to minimize the joint cost function of a subsystem that comprises a central agent and its neighboring agents, and the local control actions (policies) only rely on the knowledge of local observations. Under this framework, we not only preserve the correlations between neighboring agents, but moderate the communication and computational complexities by decentralizing the sampling and computational processes over the network. For discrete-time systems modeled by Markov decision processes, the joint Bellman equation of each subsystem is transformed into a system of linear equations and solved using parallel programming. For continuous-time systems modeled by Itô diffusion processes, the joint optimality equation of each subsystem is converted into a linear partial differential equation, whose solution is approximated by a path integral formulation and a sample-efficient relative entropy policy search algorithm, respectively. The learned control policies are generalized to solve the unlearned tasks by resorting to the compositionality principle, and illustrative examples of cooperative UAV teams are provided to verify the effectiveness and advantages of these algorithms.

Generalization of Safe Optimal Control Actions on Networked Multi-Agent Systems

“…The factorial subsystem for agent i includes all the agents directly communicating with agent i and the agent itself, and is denoted by Ni := N i ∪{i}, and the cardinality of set Ni is denoted by | Ni |. Consider the joint continuous-time dynamics for factorial subsystem Ni as in [29]:…”

“…under the condition that σi σ i = λ i R−1 i is satisfied to cancel the nonlinear terms. The linear-form joint optimal control action for continuous-time stochastic networked MAS under aforementioned decentralization topology is derived in [29] and the result is in the form:…”

“…Path integral control approach has also been extended to MASs, and an approximate inference method was applied to approximate the exact solution in [28]. A distributed algorithm proposed in [29] partitions the networked MAS into multiple subsystems, from which local control actions are computed with limited resources.…”

Markovian Decentralized Ensemble Control for Demand Response

“…Comparing with existing decentralized control method [10 -12], our proposed model provides an elegant analytical solution which is easy to implement and can be easily extended to other demand response scenarios such as heating systems. The decentralized consensusbased approach proposed in this paper is closely related to the decentralized LS-MDP [13], which studied a network of Bayesian agents in continuous Markov Processes (MP) without consensus. The introduced algorithm resembles distributed gradient algorithms over (random) networks [14,15].…”