Continuous-Observation Partially Observable Semi-Markov Decision Processes for Machine Maintenance

Abstract

Partially observable semi-Markov decision processes (POS-MDPs) provide a rich framework for planning under both state transition uncertainty and observation uncertainty. In this paper, we widen the literature on POSMDP by studying discretestate, discrete-action yet continuous-observation POSMDPs. We prove that the resultant α-vector set is continuous and therefore propose a point-based value iteration algorithm. This paper also bridges the gap between POSMDP and machine maintenance by incorporating various types of maintenance actions, such as actions changing machine state, actions changing degradation rate, and the temporally extended action "do nothing". Both finite and infinite planning horizons are reviewed, and the solution methodology for each type of planning horizon is given. We illustrate the maintenance decision process via a real industrial problem and demonstrate that the developed framework can be readily applied to solve relevant maintenance problems.

A Dynamic Prescriptive Maintenance Model Considering System Aging and Degradation

“…A detailed derivation of the equation is provided in the appendix. Numerical approaches like Monte Carlo integration can be employed to evaluated Equation (7). The optimal inspection time t * i can be obtained as…”

“…Degradation models can be either continuous or discrete. In a discrete degradation model, the system condition is divided into a finite number of states, which is usually characterized by a Markov or semi-Markov chain [3]- [7]. The disadvantage of Markov or semi-Markov models lies in the arbitrary classification of the system states and fails to fully characterize its degradation evolution.…”

ChronosPerseus: Randomized Point-based Value Iteration with Importance Sampling for POSMDPs

“…While Zhang and Revie (2017) developed a POSMDP solver based on Perseus, we introduce importance sampling to use the collected samples more efficiently. This leads us to the significant contribution of this paper, which is our importance sampling point-based POSMDP solver called ChronosPerseus that we will present in Section 3.…”

“…Finally, we describe in Section 3 the main contribution of this paper-the ChronosPerseus algorithm. In Section 4, we conclude this paper with some solved POSMDP examples: the bus problem of an agent deciding to stay on the bus or ride its bike, and maintenance of water filters in the real world based on parameters from Zhang and Revie (2017). The first one includes a mixture of random continuous and fixed sojourn times as well as mixed-observability on an episodic task, while the second includes continuous sojourn times and observation space on a non-episodic problem.…”

Stochastic Filtering Approach for Condition-Based Maintenance Considering Sensor Degradation

“…POMDP model was formulated to achieve the optimal maintenance decision. [22] investigated the CBM issue for a machine subject to imperfect continuous monitoring. A continuous-observation partially observable semi-Markov decision process was presented to select various maintenance actions.…”