Partially Observable<scp>MDPs</scp>(<scp>POMDPS</scp>): Introduction and Examples

Yaylalı, Emine; Ivy, Julie S.

doi:10.1002/9780470400531.eorms0646

Cited by 5 publications

(5 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The overall goal of reinforcement learning is to achieve the maximum reward over time by learning a policy through which the agent can choose proper actions for given states 1 . Typical reinforcement learning algorithms include Markov decision processes and partially observed Markov decision processes 47,48 …”

Section: Machine Learningmentioning

confidence: 99%

Artificial intelligence in multiparametric magnetic resonance imaging: A review

Liu

et al. 2022

Medical Physics

View full text Add to dashboard Cite

Multiparametric magnetic resonance imaging (mpMRI) is an indispensable tool in the clinical workflow for the diagnosis and treatment planning of various diseases. Machine learning-based artificial intelligence (AI) methods, especially those adopting the deep learning technique, have been extensively employed to perform mpMRI image classification, segmentation, registration, detection, reconstruction, and super-resolution. The current availabilities of increasing computational power and fast-improving AI algorithms have empowered numerous computer-based systems for applying mpMRI to disease diagnosis, imaging-guided radiotherapy, patient risk and overall survival time prediction, and the development of advanced quantitative imaging technology for magnetic resonance fingerprinting.However,the wide application of these developed systems in the clinic is still limited by a number of factors, including robustness, reliability, and interpretability. This survey aims to provide an overview for new researchers in the field as well as radiologists with the hope that they can understand the general concepts, main application scenarios, and remaining challenges of AI in mpMRI.

show abstract

Section: Machine Learningmentioning

confidence: 99%

Artificial intelligence in multiparametric magnetic resonance imaging: A review

Liu

et al. 2022

Medical Physics

View full text Add to dashboard Cite

show abstract

“…If Monte Carlo simulations are carried out enough times, then the distribution of the cumulative number of cycles N will be obtained. This distribution is finally parameterized as the GMD pattern, defined in Equation (31). In this case, the accumulative number of cycles in a year can be represented using the Gaussian distribution N ;½75, 823, 1149 2 .…”

Section: Bdlm Establishingmentioning

confidence: 99%

“…The observation parameters in continuous POMDPs are evolved from the probability matrix to the probability density functions. After the BDLMs are established, a stochastic observation value o t + 1 is obtained based on GMM in Equation (31). The observed belief state is updated by Equation (32), where parameter P DC in Equation ( 32) is given via Equation ( 5) and STRs in Table 2.…”

Section: Management Model Establishingmentioning

confidence: 99%

“…27 It is also employed to consider the value of information for the SHM system. [28][29][30] However, the aforementioned studies are restricted to numerical studies and existing research lacks sufficient attention to how to verify and update some assumptions, such as discrete observation value and stationary state transition matrix in traditional POMDPs, 31 by incorporating the long-term health monitoring data. To accommodate the dynamic deterioration process and actual monitoring data, the solver for non-stationary 32 and continuous observation POMDP 33 should be utilized.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SHM-informed life-cycle intelligent maintenance of fatigue-sensitive detail using Bayesian forecasting and Markov decision process

Lai

Dong

Smyl

2023

Structural Health Monitoring

View full text Add to dashboard Cite

Civil and maritime engineering systems must be efficiently managed to control the failure risk at an acceptable level as their performance is gradually degraded throughout the operational life, caused by fatigue and corrosion. Structural health monitoring develops a timely capability to assess the structural condition and performance metrics. However, using actual long-term monitoring data to guide the life-cycle management under stochastic environments has not been sufficiently studied. To realize an optimal maintenance strategy within the service life, an integrated monitoring-based optimal management framework is developed on the basis of the partially observable Markov decision processes (POMDPs) and Bayesian forecasting. In the proposed framework, the stochastic fatigue processes are quantified by the state transition matrix. The Bayesian dynamic linear model is embedded in POMDPs as a continuous observation part to forecast the cycling impacts and estimate the deterioration rate using long-term dynamic strain responses. In addition, making use of the special features of the problem considered in this paper, an adaptive discretization strategy is proposed to alleviate the complexity of large discrete observed spaces in the POMDP. The applicability and feasibility of the framework are evaluated by intelligent maintenance of fatigue-sensitive components with real-world monitoring data. After solving the POMDP by an efficient offline solver, the results obtained in this paper demonstrate that structural interventions are uneconomical to extend the life when a welded detail is approaching its end of life due to the normal service. Furthermore, if multiple interventions are available, the framework can find optimal maintenance actions based on the trade-off between long-term utility and the corresponding cost. This framework as the prototype could also be adjusted to aid life-cycle intelligent maintenance of other types of components under different deterioration scenarios.

show abstract

“…General POMDP models have found wide applicability in multistate maintenance optimization problems (cf. and references therein). Maintenance models pertaining to wind turbines in particular are fairly sparse with the exception of Byon et al who formulated a POMDP model to optimally maintain a wind turbine component whose degradation state evolves as a finite, discrete‐time Markov chain (DTMC).…”

Section: Introductionmentioning

confidence: 99%

Optimal replacement of continuously degrading systems in partially observed environments

Flory

Kharoufeh

Abdul-Malak

2015

Naval Research Logistics

View full text Add to dashboard Cite

Motivated by wind energy applications, we consider the problem of optimally replacing a stochastically degrading component that resides and operates in a partially observable environment. The component's rate of degradation is modulated by the stochastic environment process, and the component fails when it is accumulated degradation first reaches a fixed threshold. Assuming periodic inspection of the component, the objective is to minimize the long‐run average cost per unit time of performing preventive and reactive replacements for two distinct cases. The first case examines instantaneous replacements and fixed costs, while the second considers time‐consuming replacements and revenue losses accrued during periods of unavailability. Formulated and solved are mixed state space, partially observable Markov decision process models, both of which reveal the optimality of environment‐dependent threshold policies with respect to the component's cumulative degradation level. Additionally, it is shown that for each degradation value, a threshold policy with respect to the environment belief state is optimal if the environment alternates between two states. The threshold policies are illustrated by way of numerical examples using both synthetic and real wind turbine data. © 2015 Wiley Periodicals, Inc. Naval Research Logistics 62: 395–415, 2015

show abstract

Partially ObservableMDPs(POMDPS): Introduction and Examples

Cited by 5 publications

References 46 publications

Artificial intelligence in multiparametric magnetic resonance imaging: A review

Artificial intelligence in multiparametric magnetic resonance imaging: A review

SHM-informed life-cycle intelligent maintenance of fatigue-sensitive detail using Bayesian forecasting and Markov decision process

Optimal replacement of continuously degrading systems in partially observed environments

Contact Info

Product

Resources

About