FedNL: Making Newton-Type Methods Applicable to Federated Learning

Abstract

Inspired by recent work of Islamov et al (2021), we propose a family of Federated Newton Learn (FedNL) methods, which we believe is a marked step in the direction of making second-order methods applicable to FL. In contrast to the aforementioned work, FedNL employs a different Hessian learning technique which i) enhances privacy as it does not rely on the training data to be revealed to the coordinating server, ii) makes it applicable beyond generalized linear models, and iii) provably works with general contractive compression operators for compressing the local Hessians, such as Top-K or Rank-R, which are vastly superior in practice. Notably, we do not need to rely on error feedback for our methods to work with contractive compressors.Moreover, we develop FedNL-PP, FedNL-CR and FedNL-LS, which are variants of FedNL that support partial participation, and globalization via cubic regularization and line search, respectively, and FedNL-BC, which is a variant that can further benefit from bidirectional compression of gradients and models, i.e., smart uplink gradient and smart downlink model compression.We prove local convergence rates that are independent of the condition number, the number of training data points, and compression variance. Our communication efficient Hessian learning technique provably learns the Hessian at the optimum.Finally, we perform a variety of numerical experiments that show that our FedNL methods have state-of-the-art communication complexity when compared to key baselines.

Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning

“…We allow both unbiased compressors, such as random sparsification (Rand-K) or random dithering, and contractive compressors, such as greedy sparsification (Top-K) or low-rank approximations (Rank-R). In the special case of choosing the standard basis, our method recovers FedNL [Safaryan et al, 2021]. Thus, basis learn can be viewed as a generalization of FedNL.…”

“…Motivated by these recent developments on distributed second-order methods with communication compression, we extend the results of FedNL [Safaryan et al, 2021] allowing even more aggressive compression for some applications.…”

“…Again, this idea was originated from the first-order methods employing communication compression [Wangni et al, 2018, Alistarh et al, 2018, Wen et al, 2017. Recently developed second-order methods DAN-LA [Zhang et al, 2020], Quantized Newton [Alimisis et al, 2021], NewtonLearn [Islamov et al, 2021] and FedNL [Safaryan et al, 2021] are based on this idea of properly incorporating compression strategies for second-order information. In contrast to the previous approach, this strategy relies on the computation of full Hessian matrices, which might be computationally intensive for some applications.…”

EF21 with Bells & Whistles: Practical Algorithmic Extensions of Modern Error Feedback

“…The operator is allowed to be randomized, and typically operates on models Khaled & Richtárik (2019) or on gradients Alistarh et al (2017); Beznosikov et al (2020), both of which can be described as vectors in R 𝑑 . Besides sparsification (Alistarh et al, 2018), typical examples of useful compression mechanisms include quantization (Alistarh et al, 2017;Horváth et al, 2019a) and low-rank approximation (Vogels et al, 2019;Safaryan et al, 2021).…”

“…We emphasize that we do not assume 𝒞 to be unbiased. Hence, our theory works with the Top-𝑘 (Alistarh et al, 2018) and the Rank-𝑟 (Safaryan et al, 2021) compressors, for example.…”