Computational Methods for DTMCs

Abstract

We consider methods for computing transient and stationary distributions of ergodic discrete‐time Markov chains. We also briefly consider the computation of moments of first passage times. The computation of transient distributions seldom presents difficulties and hence the major part of this article is devoted to the computation of stationary distributions. We discuss the theory, application, and suitability of direct methods based on Gaussian elimination and two of its variants, LU factorization and the GTH algorithm. Iterative methods are the most commonly used methods for computing stationary distributions, and in this context we examine three different approaches: (i) Methods based on splittings, including the power method, (block) Jacobi, and (block) successive overrelaxation. (ii) Decompositional methods, primarily the iterative aggregation‐disaggregation method. (iii) Krylov subspace projection methods. In this context, we discuss both popular methods and the concept of preconditioning which is vital to the success of such methods.