Decentralized control of finite state Markov processes

A Concise Introduction to Decentralized POMDPs

“…We start investigating communication that arrives with a one-step delay (1-SD), which is referred to as the 'one-step delay observation sharing pattern' in the control literature [Witsenhausen, 1971, Varaiya and Walrand, 1978, Hsu and Marcus, 1982. The consequence is that during execution at stage t the agents knowθ t−1 , the joint action-observation history up to time step t − 1, and the joint action a t−1 that was taken at the previous time step.…”

“…For the motivation to indicate this value using the letter 'V' and for the relation to other value functions, please see the discussion by Oliehoek [2010, chap. 3] Hsu andMarcus [1982] already showed that the value function for 1-step delayed communication is piecewise-linear and convex (PWLC); i.e., representable using sets of vectors. Not surprisingly, more recent approximation methods for POMDPs, e.g., Perseus [Spaan and Vlassis, 2005], can be transferred to its computation [Oliehoek et al, 2007].…”

On submodular value functions and complex dynamic programming

“…This situation arises in team problems with special information structure (see [16,22]), and in control problems with a random environment [4, 14, 151. (iii) Coupling of the dp equation with other linear equations. This occurred in equations describing the time evolution of free choice Petri-nets (which are useful for the modeling of parallel computing), see [7].…”

Stochastic games with one step delay sharing information pattern with application to power control

“…The team problem with decentralized information can be transformed into an equivalent Partially Observable Markov Decision Process (PO-MOP), that can be solved using dynamic programming once we transform it to an equivalent Completely Observable Markov Decision Process (CO-MOP) see [5], [2], [3], [4]. The problem is that this transformation comes at a cost of enlarging the state space.…”