The system being modelled is assumed to occupy one and only one state at any moment in time and its evolution is represented by transitions from state to state. Also, the physical or mathematical behaviour of this system may be represented by describing all the different states it may occupy and by indicating how it moves among these states. In this work, the concept of the classification of groups of states, between states that are recurrent, meaning that the Markov chain is guaranteed to return to these states infinitely often, and states that are transient, meaning that there is a nonzero probability that the Markov chain will never return to such a state are investigated, in order to provide some insight into the performance measure analysis such as the mean first passage time, 𝑅𝑖𝑗, the mean recurrence time of state 𝑅𝑗𝑗 as well as recurrence iterative matrix 𝑅(𝑘+1). Our quest is to demonstrate with illustrative examples on Markov chains with different classes of states, and the following results are obtained, the mean recurrence time of state 1 is infinite, as well as the mean first passage times from states 2 and 3 to state 1. The mean first passage time from state 2 to state 3 or vice versa is given as 1, while the mean recurrence time of both state 2 and state 3 is given as 2.

The evolution of this model is represented by transitions from one state to the next. Also, the physical or mathematical behavior of this system can also be illustrated by identifying all of the possible states and explaining how it transitions between them. The iterative solution approaches for the stationary distribution of Markov chains, which begin with an initial estimate of the solution vector  and it becomes closer and closer to the true solution with each iteration are investigated. Our goal is to compute solutions of stationary distribution of Markov chain by utilizing the power iterative method which leaves the transition matrices unchanged and saves time by considering the discretization effect, and the convergency. Matrices operations such as multiplication with one or more vectors, lower, diagonal and upper concepts of matrix, with the help of several existing Markov chain laws, theorems, formulas, and the normalization principle are applied. For the illustrative examples, the stationary distribution vectors  and table of convergence are obtained.

The Physical or Mathematical behaviour of this model may be represented by describing all the different states it may occupy and by indicating how it moves among these states. In this study, the stationary distribution of Markov chains was solved using iterative methods that begin with an initial estimate of the solution vector and then modified it in a way that brings it closer and closer to the real solution with each step or iteration. These methods also involved matrix operations like multiplication with one or more vectors, which preserves the transition matrices while speeding up the process. We computed the solutions using Jacobi iterative method and Gauss-Seidel iterative method in order to shed more light on the solutions of stationary distribution in Markov chain. This was done with the aid of several already-existing laws, theorems, and formulas of Markov chain and the application of normalization principle and matrix operations such as lower, upper, and diagonal matrices. The stationary distribution vector’s 𝜋𝑖,𝑖=1,2,…,4 are obtained for the illustrative example one as 𝜋(3) = (0.078125,0.109375,0.21875,0.59375) as well as the four eigenvalues of the matrix as 𝜆1=1.0, 𝜆2=−0.7718, 𝜆3,4=−0.1141±0.5576𝑖 using Jacobi iterative technique, and for illustrative example two using Gauss-Siedel method as 𝝅(𝟑) = (0.090909, 0.181818, 0.363636, 0.363636). The research shown that Gauss Siedel method converged faster than Jacobi method.