FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learning

Abstract

Federated Averaging (FedAvg, also known as Local-SGD) (McMahan et al., 2017) is a classical federated learning algorithm in which clients run multiple local SGD steps before communicating their update to an orchestrating server. We propose a new federated learning algorithm, FedPAGE, able to further reduce the communication complexity by utilizing the recent optimal PAGE method (Li et al., 2021) instead of plain SGD in FedAvg. We show that FedPAGE uses much fewer communication rounds than previous local methods for both federated convex and nonconvex optimization. Concretely, 1) in the convex setting, the number of communication rounds of FedPAGE is O( N 3/4 S ), improving the best-known result O( N S ) of SCAFFOLD (Karimireddy et al., 2020) by a factor of N 1/4 , where N is the total number of clients (usually is very large in federated learning), S is the sampled subset of clients in each communication round, and is the target error; 2) in the nonconvex setting, the number of communication rounds of FedPAGE is O(S 2/3 2 ) of SCAFFOLD (Karimireddy et al., 2020) by a factor of N 1/6 S 1/3 , if the sampled clients S ≤ √ N . Note that in both settings, the communication cost for each round is the same for both FedPAGE and SCAFFOLD. As a result, FedPAGE achieves new state-of-the-art results in terms of communication complexity for both federated convex and nonconvex optimization.

EF21 with Bells & Whistles: Practical Algorithmic Extensions of Modern Error Feedback

“…Therefore, for this setup we modify EF21 and combine it with variance reduction. In particular, this time we replace ∇𝑓 𝑖 (𝑥 𝑡+1 ) in the formula for 𝑐 𝑡 𝑖 with the PAGE estimator (Li et al, 2021) 𝑣 𝑡+1…”

“…In the finite-sum setting (3), we enhance EF21 with a variance reduction technique to reduce the computational complexity. In particular, we adopt the simple and efficient variance-reduced method PAGE (Li et al, 2021;Li, 2021b) (which is optimal for solving problems (3)) into EF21, and call the resulting method EF21-PAGE (Algorithm 3). See Appendix E for more details.…”

“…To achieve this, variance reduction techniques are instrumental. One approach is to apply a PAGE-estimator (Li et al, 2021) instead of a random minibatch applied in SGD. Note that PAGE has optimal complexity for nonconvex problems of the form (3).…”

Faster Rates for Compressed Federated Learning with Client-Variance Reduction

“…In the nonconvex setting, the functions {f i } i∈ [N ] are arbitrary functions that satisfy the following standard smoothness assumption (Johnson and Zhang, 2013;Defazio et al, 2014;Nguyen et al, 2017;Zhou et al, 2018;Fang et al, 2018;Li, 2019;, and we assume that the unbiased local stochastic gradient oracle ∇f i (x) has bounded local variance (McMahan et al, 2017;Zhao et al, 2021), which is also standard in the federated learning literature. For simplicity, we use ∇b f i (x) to denote the stochastic gradient oracle that uses a minibatch size b, which is the average of b independent unbiased stochastic gradients oracle ∇f i (x).…”

“…Among these algorithms, most of them need to periodically compute full gradients to reduce the variance of the gradient estimator, e.g. SARAH (Nguyen et al, 2017), PAGE ,PP-MARINA , FedPAGE (Zhao et al, 2021), except ZeroSARAH (Li and Richtárik, 2021b) which never needs to compute the full gradient. The idea of ZeroSARAH is to maintain another unbiased approximate gradient estimator for the full gradient, and update both the biased recursive and unbiased approximate estimators in each round.…”