A framework for parallel second order incremental optimization algorithms for solving partially separable problems

Abstract

We propose HAMSI (Hessian Approximated Multiple Subsets Iteration), which is a provably convergent, second order incremental algorithm for solving large-scale partially separable optimization problems. The algorithm is based on a local quadratic approximation, and hence, allows incorporating curvature information to speed-up the convergence. HAMSI is inherently parallel and it scales nicely with the number of processors. Combined with techniques for effectively utilizing modern parallel computer architectures, we illustrate that the proposed method converges more rapidly than a parallel stochastic gradient descent when both methods are used to solve large-scale matrix factorization problems. This performance gain comes only at the expense of using memory that scales linearly with the total size of the optimization variables. We conclude that HAMSI may be considered as a viable alternative in many large scale problems, where first order methods based on variants of stochastic gradient descent are applicable.

Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks

“…Figures 7(c) and 7(d) show the results. We observe that, the convergence behavior improves when we increase N from 4 to 5; however, further increasing N results in a degraded performance, since the overall computation time is dominated by the communication cost, a typical situation observed in synchronized distributed optimization (Kaya et al, 2019;S ¸imşekli et al, 2018).…”