Superlinearly Convergent Asynchronous Distributed Network Newton Method

Mansoori, Fatemeh; Wei, Ermin

doi:10.48550/arxiv.1705.03952

Cited by 4 publications

(10 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One example of local but not global strong convexity is the objective function of logistic regression [1], i.e., [29] is quasi-Newton and therefore allows each f i to be a general convex function, yet it requires, like other second-order algorithms in [25], [27], [28], [30]- [33], each ∇f i to be (globally) Lipschitz continuous, which is unnecessary for problem (1) under Assumption 1. Moreover, the local Lipschitz continuity of ∇ 2 f i in Assumption 1 is weaker than the three times continuous differentiability of f i in [33] and the (global) Lipschitz continuity of ∇ 2 f i in [25], [26], [30], [31].…”

Section: B In-network Optimizationmentioning

confidence: 99%

“…Comparison in convergence rates: We compare the convergence rate results of DEAN and the existing decentralized second-order methods [25]- [33]. Like the inexact secondorder methods D-BFGS [29], NN [30], ANN [31], and DQN [32], the particular DEAN in Theorem 3 linearly converges to a suboptimal solution. We also provide an explicit error bound for this suboptimal solution, which is near zero when the step-sizes are very small, whereas [29]- [32] do not offer such bounds.…”

Section: Convergence Analysismentioning

confidence: 99%

“…The second category is the Newton-type methods, such as the Distributed Broyden-Fletcher-Goldfarb-Shanno (D-BFGS) method [29], the Network Newton (NN) method [30], the Asynchronous Network Newton method (ANN) [31], the Distributed Quasi-Newton (DQN) method [32], and the Newton-Raphson Consensus (NRC) method [33]. Among these methods, D-BFGS, NN, ANN, and DQN employ a penalized objective function in order to relax the consensus constraint and approximate the Newton direction of the penalized objective in a decentralized manner.…”

Section: Introductionmentioning

confidence: 99%

“…NRC [33] requires every node to transmit 3|N i | vectors and |N i | n × n matrices. ANN [31] is the only asynchronous second-order algorithm, in which each active node transmits 2|N i | vectors and each of its neighbor j transmits |N j | vectors. Among the above secondorder methods, DEAN and DQM are iteration-wise the most communication-efficient synchronous algorithms and might require fewer communications than the asynchronous method ANN on densely-connected networks.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Decentralized Approximate Newton Methods for Convex Optimization on Networked Systems

Wei

et al. 2019

Preprint

View full text Add to dashboard Cite

This paper proposes a class of Decentralized Approximate Newton (DEAN) methods for addressing innetwork convex optimization, where nodes in a network seek for a consensus that minimizes the sum of their individual objective functions through local interactions only. The proposed DEAN algorithms allow each node to repeatedly perform a local approximate Newton update, so that the nodes not only jointly track the global Newton direction but also drive each other closer. Under a less restrictive assumption (i.e., local strong convexity) in comparison with the existing second-order methods, the DEAN algorithms enable the nodes to reach a consensus that can be arbitrarily close to the optimum. Moreover, for a particular DEAN algorithm, the nodes linearly converge to a common suboptimal solution with an explicit error bound and we also provide the iteration complexity for the suboptimal solution to achieve any given accuracy. Furthermore, we show that when the problem reduces to a quadratic program, the DEAN algorithms are guaranteed to converge to the exact optimum at a linear rate. Finally, simulations demonstrate the competitive performance of DEAN in convergence speed, accuracy, and efficiency.

show abstract

Section: B In-network Optimizationmentioning

confidence: 99%

Section: Convergence Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Decentralized Approximate Newton Methods for Convex Optimization on Networked Systems

Wei

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…While this technique cannot be used directly in distributed optimization due to the non-sparsity of the Hessian inverse, there exist ways of using second order information to approximate the Newton step for distributed settings. This has been done for consensus optimization problems reformulated as both the penalty-based methods [17], [24] and dual-based methods [20], as well as the more recent primal-dual methods [22], [25]. These approximate Newton methods exhibit faster convergence relative to their corresponding first order methods.…”

Section: Introductionmentioning

confidence: 99%

A Primal-Dual Quasi-Newton Method for Exact Consensus Optimization

Eisen

Mokhtari

Ribeiro

2019

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

We introduce the primal-dual quasi-Newton (PD-QN) method as an approximated second order method for solving decentralized optimization problems. The PD-QN method performs quasi-Newton updates on both the primal and dual variables of the consensus optimization problem to find the optimal point of the augmented Lagrangian. By optimizing the augmented Lagrangian, the PD-QN method is able to find the exact solution to the consensus problem with a linear rate of convergence. We derive fully decentralized quasi-Newton updates that approximate second order information to reduce the computational burden relative to dual methods and to make the method more robust in ill-conditioned problems relative to first order methods. The linear convergence rate of PD-QN is established formally and strong performance advantages relative to existing dual and primal-dual methods is shown numerically.

show abstract