Revisiting Asynchronous Linear Solvers: Provable Convergence Rate through Randomization

Abstract-This paper presents a unifying convergence result for asynchronous iterations involving pseudo-contractions in the block-maximum norm. Contrary to previous results which only established asymptotic convergence or studied simplified models of asynchronism, our result allows to bound the convergence rates for both partially and totally asynchronous implementations. Several examples are worked out to demonstrate that our theorem recovers and improves on existing results, and that it allows to characterize the solution times for several classes of asynchronous iterations that have not been addressed before. I. INTRODUCTIONAsynchronous algorithms appear naturally in parallel and distributed systems and are heavily exploited applications ranging from large-scale linear algebra and optimization to distributed coordination of small embedded devices. Allowing nodes to operate in an asynchronous manner simplifies the implementation of distributed algorithms and eliminates the overhead associated with synchronization. However, care has to be taken, since asynchrony runs the risk of rendering an otherwise stable iteration unstable.The dynamics of asynchronous iterations are much richer than their synchronous counterparts, and quantifying the impact of asynchrony on the convergence properties of iterative algorithms remains challenging. Some of the first results on the convergence of asynchronous iterations were derived by Chazan and Miranker [1], who studied chaotic relaxations for solving linear systems of equations. Several authors have proposed extensions of this pioneering work to nonlinear iterations involving maximum norm contractions (e.g., [2], [3]) and for monotone iterations (e.g., [4], [5]). Powerful convergence results for broad classes of asynchronous algorithms, including maximum norm contractions and monotone mappings, under different assumptions on communication delays and update rates were presented by Bertsekas [6] and Bertsekas and Tsitsiklis [7]. Most of the results in the literature only guarantee asymptotic convergence. This paper complements the existing work by developing convergence theorems that characterize the rate of convergence of asynchronous iterations and quantify how these rates depend on the update intervals and information delays in the system.We focus on iterations involving block-maximum norm pseudo-contractions under the general asynchronous model introduced in [6], [7], which allows for heterogeneous and time-varying update rates and communication delays. Such iterations arise in a variety of algorithms, such as certain classes of linear fixed-point iterations and gradient

show abstract

“…The last equality uses (14). It follows that condition (iii) of Theorem 1 holds for allρ satisfying (15).…”

Section: Resultsmentioning

confidence: 87%

“…As also stressed in [14], very few results on convergence rates of asynchronous algorithms have appeared in the literature (see e.g., [2], [7] for exceptions). In particular, [7,Section 6.3] showed that if delays are bounded and…”

Section: Resultsmentioning

confidence: 99%

On the convergence rates of asynchronous iterations

Feyzmahdavian

Johansson

2014

53rd IEEE Conference on Decision and Control

View full text Add to dashboard Cite

show abstract

“…18) was a negative term (seeLemma 13). This term was critical in the proof of the norm convergence and FPR strong convergence criterion in Section 2.4.…”

mentioning

confidence: 99%

On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithms

Hannah

Yin

2017

J Sci Comput

View full text Add to dashboard Cite

The need for scalable solvers for massive optimization problems has motivated the development of asynchronous-parallel algorithms, where a set of nodes runs in parallel with little or no synchronization, thus computing with delayed information. This paper develops powerful Lyapunov-functions techniques, and uses them to study the convergence of the asynchronous-parallel algorithm ARock under potentially unbounded delays.ARock is a very general asynchronous algorithm, that takes many popular algorithms as special cases: For instance, asynchronous block gradient descent, forward backward, ADMM, etc. Therefore our results have broad implications, and a range of applications. ARock parallelizes a fixed-point iterations by letting a set of nodes randomly choose solution coordinates to update in an asynchronous parallel fashion. The existing analysis of ARock assumes the delays to be bounded, and uses this bound to set a step size that is important to both convergence and efficiency. Other works, though allowing unbounded delays, impose strict conditions on the underlying fixed-point operator, resulting in limited applications.In this paper, convergence is established under unbounded delays, which can be either stochastic or deterministic. The proposed step sizes are more practical and larger than those in the existing work. The step size adapts to the delay distribution or the current delay being experienced in the system, instead of being limited by worst-case scenario delays. New Lyapunov functions, which are the key to analyzing asynchronous algorithms, are generated to obtain our results. A general strategy for generating Lyapunov functions is presented, which may find application in convergence analyses of other algorithms. A set of applicable optimization algorithms with large-scale applications are given, including machine learning and scientific computing algorithms.1.1 Motivation and importance of asynchronous algorithms since 2005. Before this point, running the same serial algorithm on the same problem would become faster year-over-year because of the increasing power of individual cores, however this is no longer the case. Moving forward, CPUs will only become faster through the addition of more cores rather than more powerful cores (see [1], [2]). Therefore the only way to utilize more powerful processors is to exploit parallelism. The trade off is that parallel algorithms are difficult to analyze and implement. However, the rewards for success are great: breakthroughs in parallel computing have implications for many other scientific fields. Motivation and importance of asynchronous algorithmsThe vast majority of parallel algorithms are synchronous algorithms. For instance the synchronous-parallel Gauss-Jacobi algorithm divides the problem space R N into p coordinate blocks. At every iteration, these blocks are updated by a corresponding set of p processors, and each processor's update is communicated to every other processor. Synchronous algorithms are simpler to analyze and implement; however, they have...

show abstract

“…However, the iteration in (6) is sequential which might represent an issue for its implementation. Nevertheless, there are randomized and asynchronous (i.e., where any order of the updates can be followed during each iteration) versions of the Gauss-Seidel iteration, such as the one in [19]. The main conclusion is that each processor can employ the GaussSeidel iteration to compute the PageRank using updated values sent by the other processors if the web data does not belong to that processor and use the past iteration values for all the remaining ones.…”

Section: Iterationsmentioning

confidence: 99%

A PageRank Algorithm based on Asynchronous Gauss-Seidel Iterations

Silvestre

Hespanha

Silvestre

2018

2018 Annual American Control Conference (ACC)

View full text Add to dashboard Cite

Abstract-We address the PageRank problem of associating a relative importance value to all web pages in the Internet so that a search engine can use them to sort which pages to show to the user. This precludes finding the eigenvector associated with a particular eigenvalue of the link matrix constructed from the topology graph of the web. In this paper, we investigate the potential benefits of addressing the problem as a solution of a set of linear equations. Initial results suggest that using an asynchronous version of the Gauss-Seidel method can yield a faster convergence than using the traditional power method while maintaining the communications according to the sparse link matrix of the web and avoiding the strict sequential update of the Gauss-Seidel method. Such an alternative poses an interesting path for future research given the benefits of using other more advanced methods to solve systems of linear equations. Additionally, it is investigated the benefits of having a projection after all page ranks have been updated as to maintain all its entries summing to one and positive. In simulations, it is provided evidence to support future research on approximation rules that can be used to avoid the need for the projection to the n-simplex (the projection represents in some cases a threefold increase in the convergence rate over the power method) and on the loss in performance by using an asynchronous algorithm.

show abstract

Revisiting Asynchronous Linear Solvers: Provable Convergence Rate through Randomization

Cited by 36 publications

References 13 publications

On the convergence rates of asynchronous iterations

On the convergence rates of asynchronous iterations

On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithms

A PageRank Algorithm based on Asynchronous Gauss-Seidel Iterations

Contact Info

Product

Resources

About