On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithms

Abstract

The need for scalable solvers for massive optimization problems has motivated the development of asynchronous-parallel algorithms, where a set of nodes runs in parallel with little or no synchronization, thus computing with delayed information. This paper develops powerful Lyapunov-functions techniques, and uses them to study the convergence of the asynchronous-parallel algorithm ARock under potentially unbounded delays.ARock is a very general asynchronous algorithm, that takes many popular algorithms as special cases: For instance, asynchronous block gradient descent, forward backward, ADMM, etc. Therefore our results have broad implications, and a range of applications. ARock parallelizes a fixed-point iterations by letting a set of nodes randomly choose solution coordinates to update in an asynchronous parallel fashion. The existing analysis of ARock assumes the delays to be bounded, and uses this bound to set a step size that is important to both convergence and efficiency. Other works, though allowing unbounded delays, impose strict conditions on the underlying fixed-point operator, resulting in limited applications.In this paper, convergence is established under unbounded delays, which can be either stochastic or deterministic. The proposed step sizes are more practical and larger than those in the existing work. The step size adapts to the delay distribution or the current delay being experienced in the system, instead of being limited by worst-case scenario delays. New Lyapunov functions, which are the key to analyzing asynchronous algorithms, are generated to obtain our results. A general strategy for generating Lyapunov functions is presented, which may find application in convergence analyses of other algorithms. A set of applicable optimization algorithms with large-scale applications are given, including machine learning and scientific computing algorithms.1.1 Motivation and importance of asynchronous algorithms since 2005. Before this point, running the same serial algorithm on the same problem would become faster year-over-year because of the increasing power of individual cores, however this is no longer the case. Moving forward, CPUs will only become faster through the addition of more cores rather than more powerful cores (see [1], [2]). Therefore the only way to utilize more powerful processors is to exploit parallelism. The trade off is that parallel algorithms are difficult to analyze and implement. However, the rewards for success are great: breakthroughs in parallel computing have implications for many other scientific fields. Motivation and importance of asynchronous algorithmsThe vast majority of parallel algorithms are synchronous algorithms. For instance the synchronous-parallel Gauss-Jacobi algorithm divides the problem space R N into p coordinate blocks. At every iteration, these blocks are updated by a corresponding set of p processors, and each processor's update is communicated to every other processor. Synchronous algorithms are simpler to analyze and implement; however, they have...

A Partition-Based Implementation of the Relaxed ADMM for Distributed Convex Optimization over Lossy Networks

“…Therefore Equations (11) and (12) can be derived from (26) and (27) using the particular structure of the problem, proving Proposition 1.…”

“…Notice now that the trajectory k → x(k) generated by (26) is equivalent to that generated by (19) if the initial condition for x is the same and if z(0) = w(0)+ρy(0) since Equation (21) has to hold at time k = 0. Therefore Propositon 1 is proved if we can show that (26) and (27) can be rewritten as (11) and (12).…”

“…Well-known examples are the Peaceman-Rachford Splitting (PRS) [22] with its generalized versions as well as the Douglas-Rachford Splitting (DRS) [23], [24]. Moreover we refer to [25], [26] for more details and possible applications to asynchronous setups. We start our analysis from the fact that ADMM can be shown to be equivalent to the DRS applied to the Lagrange dual of the original problem [27].…”

Distributed Optimization over Lossy Networks via Relaxed Peaceman-Rachford Splitting: a Robust ADMM Approach

“…The previous proposition naturally suggests an alternative distributed implementation of the R-ADMM Algorithm 1, in which each node i stores in its local memory the variables x i and z ij , j ∈ N i . Then, at each iteration of the algorithm, each node i first collects the variables z ji , j ∈ N i ; second, updates x i and z ij according to (25) and the first of (26), respectively; finally, it sends z ij to j ∈ N i . Differently to the natural implementation just briefly described, we present a slightly different implementation building upon the observation that each node i, to update x i as in (25) requires the variables z ji rather than z ij for j ∈ N i .…”

“…Indeed, the classical formulation of the ADMM naturally arises as application of the DRS to the Lagrange dual problem of the original optimization problem [23]. For further details on a variety of splitting operators and their application in asynchronous setups we refer to [24] and [25], respectively. In this paper we present and analyze different formulation for the ADMM algorithm.…”