Greedy Sparsity-Promoting Algorithms for Distributed Learning

Abstract

Abstract-This paper focuses on the development of novel greedy techniques for distributed learning under sparsity constraints. Greedy techniques have widely been used in centralized systems due to their low computational requirements and at the same time their relatively good performance in estimating sparse parameter vectors/signals. The paper reports two new algorithms in the context of sparsity-aware learning. In both cases, the goal is first to identify the support set of the unknown signal and then to estimate the non-zero values restricted to the active support set. First, an iterative greedy multi-step procedure is developed, based on a neighborhood cooperation strategy, using batch processing on the observed data. Next, an extension of the algorithm to the online setting, based on the diffusion LMS rationale for adaptivity, is derived. Theoretical analysis of the algorithms is provided, where it is shown that the batch algorithm converges to the unknown vector if a Restricted Isometry Property (RIP) holds. Moreover, the online version converges in the mean to the solution vector under some general assumptions. Finally, the proposed schemes are tested against recently developed sparsity-promoting algorithms and their enhanced performance is verified via simulation examples.

Distributed greedy sparse learning over doubly stochastic networks

“…Further improvement on the work in [10] was proposed in [11] to provide a reduced communication cost. Another distributed iterative hard thresholding (DiHaT) algorithm is developed in [4] where observations, measurement matrices and local estimates are exchanged over network to achieve consensus. The DiHaT algorithm provides fast convergence compared to D-LASSO, and provides competitive performance, but at the expense of a high communication cost.…”

“…The DiHaT algorithm provides fast convergence compared to D-LASSO, and provides competitive performance, but at the expense of a high communication cost. In [4], an alternate algorithm was also proposed that only uses estimate exchange, but without any theoretical analysis. Our algorithm can be implemented in more general networks with different measurement sizes at different nodes.…”

“…This condition is necessary for comparing with DiHaT [4]. Define signal-tomeasurement-noise ratio (SMNR) for the l'th node as, We further assume that ∀l, SMNR l are of the same value.…”

“…In this context, low complexity distributed algorithms for large-scale sparse learning has a high potential [2], [3]. To realize large-scale distributed sparse learning, recent activities are reported in [4], [5] where greedy algorithms are designed and analyzed. A major advantage of greedy algorithms is their low computational complexity and hence their suitability for large-scale scenarios.…”

“…The proposed algorithm exchanges estimates of underlying sparse signal over a doubly stochastic network, unlike the case of [4] where other data is also exchanged. The main contributions of this article are as follows:…”