Reduction of a Controlled Markov Model with Incomplete Data to a Problem with Complete Information in the Case of Borel State and Control Space

Yushkevich, A. A.

doi:10.1137/1121014

Cited by 57 publications

(59 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This section starts with the description of known results on the general reduction of a POMDP to the COMDP; Bertsekas [24], and Yushkevich [34]. To simplify notations, we sometimes drop the time parameter.…”

Section: Reduction Of Pomdps To Comdps and Main Resultsmentioning

confidence: 99%

“…The reduction of MDMIIs with Borel state and action sets to MDPs was described by Rhenius [24] and Yushkevich [34]; see also Dynkin and Yushkevich [12,Chapter 8]. MDMIIs with transition probabilities having densities were studied by Rieder [25]; see also Bäuerle (ii) the graph of the mapping A : Y → 2 A , defined as Gr(A) = {(y, a) : y ∈ Y, a ∈ A(y)} is measurable, that is, Gr(A) ∈ B(Y × A), and this graph allows a measurable selection, that is, there exists a measurable mapping φ : Y → A such that φ(y) ∈ A(y) for all y ∈ Y;…”

Section: Markov Decision Model With Incomplete Information (Mdmii)mentioning

confidence: 99%

“…Section 7 introduces a sufficient condition for the weak continuity of transition probabilities for the COMDP that combines Assumption (H) and the weak continuity of H. Combining these properties together is important because Assumption (H) may hold for some observations and continuity of H may hold for others. Section 8 contains three illustrative examples: (i) a model defined by stochastic equations including Kalman's filter, (ii) a model for inventory control with incomplete records (for particular inventory control problems of such type see Bensoussan et al [4]- [7] and references therein), and (iii) the classic Markov decision model with incomplete information studied by Aoki [1], Dynkin [11], Shiryaev [30], Hinderer [21,Section 7.1], Sawarigi and Yoshikawa [27], Rhenius [24], Yushkevich [34], Dynkin and Yushkevich [12,Chapter 8], for which we provide a sufficient condition for the existence of optimal policies, convergence of value iterations to optimal values, and other optimality properties formulated in Theorems 3.2.…”

Section: This Example Motivated Us To Introduce Assumption (H)mentioning

confidence: 99%

“…The first author of this paper learned about this question from Alexander A. Yushkevich around the time when Yushkevich was working on [34]. The following theorem provides such a condition.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

Feinberg

Kasyanov²,

Zgurovsky³

2016

Mathematics of OR

102

View full text Add to dashboard Cite

This paper describes sufficient conditions for the existence of optimal policies for Partially Observable Markov Decision Processes (POMDPs) with Borel state, observation, and action sets and with the expected total costs. Action sets may not be compact and one-step cost functions may be unbounded. The introduced conditions are also sufficient for the validity of optimality equations, semi-continuity of value functions, and convergence of value iterations to optimal values. Since POMDPs can be reduced to Completely Observable Markov Decision Processes (COMDPs), whose states are posterior state distributions, this paper focuses on the validity of the above mentioned optimality properties for COMDPs. The central question is whether transition probabilities for a COMDP are weakly continuous. We introduce sufficient conditions for this and show that the transition probabilities for a COMDP are weakly continuous, if transition probabilities of the underlying Markov Decision Process are weakly continuous and observation probabilities for the POMDP are continuous in the total variation. Moreover, the continuity in the total variation of the observation probabilities cannot be weakened to setwise continuity. The results are illustrated with counterexamples and examples.

show abstract

Section: Reduction Of Pomdps To Comdps and Main Resultsmentioning

confidence: 99%

Section: Markov Decision Model With Incomplete Information (Mdmii)mentioning

confidence: 99%

Section: This Example Motivated Us To Introduce Assumption (H)mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

Feinberg

Kasyanov²,

Zgurovsky³

2016

Mathematics of OR

102

View full text Add to dashboard Cite

show abstract

“…This reduction was introduced by Aoki [1],Åström [3], Dynkin [8], and Shiryaev [22]. For problems with Borel state and action spaces, this reduction was independently justified by Rhenius [17] and Yushkevich [25]. It reduces finding an optimal policy for a POMDP to finding an optimal policy for the corresponding COMDP, but it says nothing about the existence of optimal policies for the COMDP.…”

Section: Introductionmentioning

confidence: 99%

Uniform Fatou's lemma

Feinberg

Kasyanov

Zgurovsky

2016

Journal of Mathematical Analysis and Applications

View full text Add to dashboard Cite

Fatou's lemma is a classic fact in real analysis stating that the limit inferior of integrals of functions is greater than or equal to the integral of the inferior limit. This paper introduces a stronger inequality that holds uniformly for integrals on measurable subsets of a measurable space. The necessary and sufficient condition, under which this inequality holds for a sequence of finite measures converging in total variation, is provided. This statement is called the uniform Fatou lemma, and it holds under the minor assumption that all the integrals in the inequality are well-defined. The uniform Fatou lemma improves the classic Fatou lemma in the following directions: the uniform Fatou lemma states a more precise inequality, it provides the necessary and sufficient condition, and it deals with variable measures. Various corollaries of the uniform Fatou lemma are formulated. The examples in this paper demonstrate that: (a) the uniform Fatou lemma may indeed provide a more accurate inequality than the classic Fatou lemma; (b) the uniform Fatou lemma does not hold if convergence of measures in total variation is relaxed to setwise convergence.

show abstract

Stochastic Control with Complete Observations on a Finite Horizon

Schuppen¹

2021

Communications and Control Engineering

View full text Add to dashboard Cite

Reduction of a Controlled Markov Model with Incomplete Data to a Problem with Complete Information in the Case of Borel State and Control Space

Cited by 57 publications

References 6 publications

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

Uniform Fatou's lemma

Stochastic Control with Complete Observations on a Finite Horizon

Contact Info

Product

Resources

About