Reinforcement Learning in Deep Structured Teams: Initial Results with Finite and Infinite Valued Features

Abstract

In this paper, we consider Markov chain and linear quadratic models for deep structured teams with discounted and time-average cost functions under two non-classical information structures, namely, deep state sharing and no sharing. In deep structured teams, agents are coupled in dynamics and cost functions through deep state, where deep state refers to a set of orthogonal linear regressions of the states. In this article, we consider a homogeneous linear regression for Markov chain models (i.e., empirical distribution of states) and a few orthonormal linear regressions for linear quadratic models (i.e., weighted average of states). Some planning algorithms are developed for the case when the model is known, and some reinforcement learning algorithms are proposed for the case when the model is not known completely. The convergence of two model-free (reinforcement learning) algorithms, one for Markov chain models and one for linear quadratic models, is established. The results are then applied to a smart grid.

Deep Structured Teams in Arbitrary-Size Linear Networks: Decentralized Estimation, Optimal Control and Separation Principle

“…Following the terminology of deep structured models [8], [11], aggregate variables (linear regressions) defined in ( 1) and ( 3) are called deep variables due to the fact that their evolutions across time horizon are similar to feed-forward deep neural networks. Hence, for ease of reference, we refer to xt as deep state at time t in the sequel.…”

“…In the perfect sharing and deep state sharing, the certainty equivalence principle simplifies the analysis and results in two standard decoupled Riccati equations [8], [11]. In the imperfect observation case, however, the certainty equivalence principle does not hold.…”

“…Note that the separation principle is weaker than the certainty equivalence principle [34]. The remaining problem is an optimal LQ deep structured team with (perfect) deep state sharing whose solution is obtained from the Riccati equations in (17); see [8], [9], [11] for more details.…”

“…In this paper, we study a newly emergent class of largescale decentralized control systems called deep structured teams [8]- [15], which may be viewed as a generalization of the notion of weighted mean-field teams introduced in [16]. Since the closest model to such systems is feed-forward deep neural networks, we refer to them as deep structured teams/games.…”

“…Furthermore, we demonstrate in [9] that today's most common feed-forward deep neural networks (i.e., those with rectified linear unit activation function) may be viewed as a special case of deep structured teams, where layers are time steps and neurons are simple integrator agents whose goal is to collaborate in order to minimize a common loss (cost) function. For more applications of deep structured models, the reader is referred to reinforcement learning [8], [13], [14], nonzerosum game [12], [15], minmax optimization [17], leaderfollowers [18], [19], epidemics [20], smart grids [21], meanfield teams [22]- [25], and networked estimation [26], [27].…”