A Distributed Frank-Wolfe Algorithm for Communication-Efficient Sparse Learning

Abstract

Learning sparse combinations is a frequent theme in machine learning. In this paper, we study its associated optimization problem in the distributed setting where the elements to be combined are not centrally located but spread over a network. We address the key challenges of balancing communication costs and optimization errors. To this end, we propose a distributed Frank-Wolfe (dFW) algorithm. We obtain theoretical guarantees on the optimization error and communication cost that do not depend on the total number of combining elements. We further show that the communication cost of dFW is optimal by deriving a lower-bound on the communication cost required to construct an -approximate solution. We validate our theoretical analysis with empirical studies on synthetic and real-world data, which demonstrate that dFW outperforms both baselines and competing methods. We also study the performance of dFW when the conditions of our analysis are relaxed, and show that dFW is fairly robust.

Distributing Frank-Wolfe via map-reduce

“…An effort to distribute FW is made by Bellet et al in [22]. They propose a distributed version of FW for objectives of the form F (θ) = g(Aθ), for some A ∈ R d×N , where d N .…”

“…Indeed, as we discuss in Sec. 2.3.4, FW assumes a very simple, elegant form under this simplex constraint, and has important computational advantages [20,21,22]. Our main contribution is to identify and formalize a set of conditions under which solving Problem (1.1) through FW admits a massively parallel implementation via map-reduce.…”

“…Several important problems, including experimental design, training SVMs, Adaboost, and projection to a convex hull indeed take this form [20,22,23] (see Section 4.1).…”

“…More recently, and more relevant to our work, Bellet et al [22] propose a distributed version of FW for objectives of the form F (θ) = g(Aθ), for some A ∈ R d×N , where d N . Several examples fall in this class, including two we study here (convex approximation and Adaboost);…”

“…The Frank-Wolfe algoithm has attracted interest recently due to its numerous computational advantages [16,17,18,19,20,21]. It maintains feasibility throughout execution while being projection-free, and minimizes a linear objective in each step; the latter yields sparse solutions for several interesting constraint sets, which often accelerates computation [20,22,21]. Therefore, distributing the Frank-Wolfe algorithm is very useful as it can solve this wide set of problems.…”