Denumerable continuous-time Markov decision processes with multiconstraints on average costs

Liu, Qiuli; Tan, Hangsheng; Guo, Xianping

doi:10.1080/00207721.2010.517868

Cited by 1 publication

(1 citation statement)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For some large-scale MCMC problems related to internet network, social computing, or bioinformatics (Bremaud 1999;Liu, Tan, and Guo 2012), the associated Markov chain might be too large to be stored in a single computer. Besides parallel or distributed computing methods, the lowrank approximation could also be used to store the Markov chain inexpensively in the computer in its SVD form.…”

Section: Introductionmentioning

confidence: 99%

Optimal Kullback–Leibler approximation of Markov chains via nuclear norm regularisation

Deng

Huang

2013

International Journal of Systems Science

View full text Add to dashboard Cite

This paper is concerned with model reduction for Markov chain models. The goal is to obtain a low-rank approximation to the original Markov chain. The Kullback-Leibler divergence rate is used to measure the similarity between two Markov chains; the nuclear norm is used to approximate the rank function. A nuclear-norm regularised optimisation problem is formulated to approximately find the optimal low-rank approximation. The proposed regularised problem is analysed and performance bounds are obtained through the convex analysis. An iterative fixed point algorithm is developed based on the proximal splitting technique to compute the optimal solutions. The effectiveness of this approach is illustrated via numerical examples. IntroductionMarkov chain model is an important modelling tool to various applications in electrical engineering, system biology, computer science, and economics (Bremaud 1999;Meyn and Tweedie 2009;Kowalczuk and Domzalski 2012;Brownlee, Regnier-Coudert, McCall, Massie, and Stulajter 2013). A fundamental problem arising from practice is the large dimension of the state space, which causes immense computational difficulties in decision making (e.g. Markov Decision Process), performance analysis (e.g. queuing networks or multi-agent systems), or simulation (e.g. complex dynamical systems), etc. The model reduction of Markov chains is an important problem relevant to many applications.Model reduction of Markov chains has a long history, with many different approaches to the construction of simplified models and the analysis of resulting models (Bremaud 1999;Meyn and Tweedie 2009). The most straightforward approach to construct reduced Markov models is based on aggregation of states: A reduced model is obtained by constructing groups of states with strong interactions. The strongly interacting states within each group are treated as an aggregated super-state, and a reduced Markov model is then obtained to describe the transitions from one super-state to another. Elegant results have been obtained for nearly completely decomposable Markov chains (NCDMC), using singular perturbation method and spectral theory (Yin and Zhang 1998;Huisinga, Meyn, and Schütte 2004). It is widely known that an optimal aggregation of an NCDMC can be obtained through the sign structure of the second eigenvector (Deuflhard, Huisinga,

show abstract

Section: Introductionmentioning

confidence: 99%