Dynamic consistency for stochastic optimal control problems

Abstract

Abstract. For a sequence of dynamic optimization problems, we aim at discussing a notion of consistency over time. This notion can be informally introduced as follows. At the very first time step t 0 , the decision maker formulates an optimization problem that yields optimal decision rules for all the forthcoming time step t 0 , t 1 , . . . , T ; at the next time step t 1 , he is able to formulate a new optimization problem starting at time t 1 that yields a new sequence of optimal decision rules. This process can be continued until final time T is reached. A family of optimization problems formulated in this way is said to be time consistent if the optimal strategies obtained when solving the original problem remain optimal for all subsequent problems. The notion of time consistency, well-known in the field of Economics, has been recently introduced in the context of risk measures, notably by Artzner et al. (2007) and studied in the Stochastic Programming framework by Shapiro (2009) and for Markov Decision Processes (MDP) by Ruszczynski (2009). We here link this notion with the concept of "state variable" in MDP, and show that a significant class of dynamic optimization problems are dynamically consistent, provided that an adequate state variable is chosen.

Minimax and risk averse multistage stochastic programming

“…For a discussion and survey of time consistency concepts we may refer to [5], [17]; we will discuss this further in the next section. For the nested formulation (4.8) it is possible to write dynamic programming equations in a way similar to (2.5)-(2.6) (cf., [16]).…”

“…It turns out that some suggested approaches to dynamic risk averse optimization are not time consistent (cf., [19]). For a discussion of time consistency concepts we may refer to [5], [17] and references therein. As far as we know time consistency was not discussed in the context of minimax multistage stochastic programming.…”

A survey of time consistency of dynamic risk measures and dynamic performance measures in discrete time: LM-measure perspective

“…From a practical point of view, this property essentially means that assessment of risks propagates in a consistent way over time: assessing at time t future risk, represented by random variable X, is the same as assessing at time t a risky assessment of X done at time t + 1 and represented by −ρ t+1 (X ). Additionally, the property (4) is closely related to the Bellman principle of optimality or to the dynamic programming principle (see, for instance, (Bellman and Dreyfus 1962;Carpentier et al 2012). Delbaen (2006) studies the recursivity property in terms of m-stable sets of probability measures, and also describes the time consistency of dynamic coherent risk measures in the context of martingale theory.…”

Risk aversion in multistage stochastic programming: A modeling and algorithmic perspective

“…The definitions in the literature differ mainly by their focus: the works of Ruszczyński (2010) and Kovacevic and Pflug (2014) deal with sequences of random variables, while Detlefsen and Scandolo (2005), Cheridito et al (2006), and Bion-Nadal (2008, define time consistency for continuous-time dynamic models. The definitions in Shapiro (2009), Carpentier et al (2012, Rudloff et al (2014) and De Lara and Leclère (2014) are centered on optimization and on the stability of decision variables at every stage. Xin et al (2013) propose definitions of time-consistency of policies in the context of distributionally robust MSSPs, whereas Pflug and Pichler (2014a) propose a related notion of time-consistent decisions.…”

“…Remark: As discussed in Carpentier et al (2012), it is possible that by enlarging the state space consistency may be recovered. However, the resulting dynamic programming equations would be intractable since one might end up with an infinite dimensional problem.…”