A stochastic gradient type algorithm for closed-loop problems

Abstract

We focus on solving closed-loop stochastic problems, and propose a perturbed gradient algorithm to achieve this goal. The main hurdle in such problems is the fact that the control variables are infinite dimensional, and have hence to be represented in a finite way in order to numerically solve the problem. In the same way, the gradient of the criterion is itself an infinite dimensional object. Our algorithm replaces this exact (and unknown) gradient by a perturbed one, which consists in the product of the true gradient evaluated at a random point and a kernel function which extends this gradient to the neighbourhood of the random point. Proceeding this way, we explore the whole space iteration after iteration through random points. Since each kernel function is perfectly known by a finite (and small) number of parameters, say N , the control at iteration k is perfectly known as an infinite dimensional object by at most N × k parameters. The main strength of this method is that it avoids any discretization of the underlying space, provided that we can draw as many points as needed in this space. Hence, we can take into account in a new way the possible measurability constraints of the problem. Moreover, the randomization of this algorithm implies that the most probable parts of the space are the most explored ones, what is a priori an interesting feature.In this paper, we first show a convergence result of this algorithm in the general case, and then give a few numerical examples showing the interest of this method for solving practical stochastic optimization problems.

Stochastic Relaxed Inertial Forward-Backward-Forward splitting for Monotone Inclusions in Hilbert spaces

“…In fact, this setting has been investigated in detail in the context of stochastic Nash games [28]. Further examples for stochastic approximation schemes in a Hilbert-space setting obeying Assumption 6 are [7,8] and [38]. We now discuss an example that further clarifies the requirements on the estimator.…”

Stochastic decomposition applied to large-scale hydro valleys management

“…The natural extension of these techniques in the closed-loop stochastic case (see Barty et al (2009)) fails to provide decomposed state dependent strategies. Indeed, the optimal strategy of a subproblem depends on the state of the whole system, and not only on the local state.…”

Decomposition of large-scale stochastic optimal control problems

“…when controls do not rely on any observation, Cohen and Culioli (1990) proposed to take advantage of both decomposition techniques and stochastic gradient algorithms. These techniques have been extended in the closed-loop stochastic case by Barty, Roy, and Strugarek (2009), but so far 1 In the case of power management, the state dimension is usually the number of power units. they fail to provide decomposed state dependent strategies in the Markovian case.…”

“…when controls do not rely on any observation, [CC90] proposed to take advantage of both decomposition techniques and stochastic gradient algorithms. These techniques have been extended in the closed-loop stochastic case by [BRS07], but so far they fail to provide decomposed state dependent strategies in the Markovian case. This is because a subproblem's optimal strategy depends on the state of the whole system, not only on the local state.…”