Red Light Green Light Method for Solving Large Markov Chains

Abstract

Discrete-time discrete-state finite Markov chains are versatile mathematical models for a wide range of real-life stochastic processes. One of most common tasks in studies of Markov chains is computation of the stationary distribution. Without loss of generality, and drawing our motivation from applications to large networks, we interpret this problem as one of computing the stationary distribution of a random walk on a graph. We propose a new controlled, easily distributed algorithm for this task, briefly summarized as follows: at the beginning, each node receives a fixed amount of cash (positive or negative), and at each iteration, some nodes receive 'green light' to distribute their wealth or debt proportionally to the transition probabilities of the Markov chain; the stationary probability of a node is computed as a ratio of the cash distributed by this a node to the total cash distributed by all nodes together. Our method includes as special cases a wide range of known, very different, and previously disconnected methods including power iterations, Gauss-Southwell, and online distributed algorithms. We prove exponential convergence of our method, demonstrate its high efficiency, and derive scheduling strategies for the green-light, that achieve convergence rate faster than state-of-the-art algorithms.

New ways of solving large Markov chains

“…Most common approach for computing a stationary distribution of a Markov chain, is power iterations (PI), when the initial probability vector is iteratively multiplied by the transition matrix till convergence. Here I will explain a new Red-Light-Green-Light (RLGL) algorithm that we developed with Konstantin Avrachenkov and Patrick Brown [2]. RLGL is fast, and generalizes many methods, including PI, and the state-of-the-art Gauss-Southwell method for PageRank.…”

“…This results in || πt − π * || 1 reducing as O(1/t) rather than exponentially. Figure 2b (from [2]) shows an example of the excellent performance of RLGL in a larger real-life Markov chain. The results are promising, and there are many open problems, which we will discuss in the next section.…”

“…GS stands for the Gauss-Seidel algorithm. Different versions of the RLGL algorithm correspond to the different choices of green light (see [2]). RLGL-Theta gives green light to a fraction of states with maximal absolute cash gramming.…”

“…Another problem is, how to derive convergence rates. In [2] we could express ||C t || 1 through the total variation distance between two Markov chains and used coupling. However, these two Markov chains are inhomogeneous and dependent through the sequence of green lights, so we could prove exponential convergence only in special cases, e.g., when green lights are cyclic.…”

“…Notice that in Fig.2a, if Anna transferred cash first, convergence would take longer. In[2] we computed the optimal policy for a three-state Markov chain, using dynamic pro-…”