Federated Adversarial Debiasing for Fair and Transferable Representations

Hong, Junyuan; Zhu, Zhuangdi; Yu, Shuyang; Wang, Zhangyang; Dodge, Hiroko H.; Zhou, Jiayu

doi:10.1145/3447548.3467281

Cited by 24 publications

(15 citation statements)

References 23 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Real-world FL settings present many challenges due to non-iid data distributions, client heterogeneity, and limited compute, memory, and bandwidth at the edge (Kairouz et al 2019;Hong et al 2021). Heterogeneity in the compute capabilities of clients causes the so-called straggler problem, in which certain clients take "too long" to form model updates and the server must proceed without them.…”

Section: Related Work Federated Learningmentioning

confidence: 99%

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Bibikar

Vikalo

Wang

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. Unfortunately, current deep networks remain not only too compute-heavy for inference and training on edge devices, but also too large for communicating updates over bandwidth-constrained networks. In this paper, we develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) by which complex neural networks can be deployed and trained with substantially improved efficiency in both on-device computation and in-network communication. At the core of FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network. With this scheme, "two birds are killed with one stone:'' instead of full models, each client performs efficient training of its own sparse networks, and only sparse networks are transmitted between devices and the cloud. Furthermore, our results reveal that the dynamic sparsity during FL training more flexibly accommodates local heterogeneity in FL agents than the fixed, shared sparse masks. Moreover, dynamic sparsity naturally introduces an "in-time self-ensembling effect'' into the training dynamics, and improves the FL performance even over dense training. In a realistic and challenging non i.i.d. FL setting, FedDST consistently outperforms competing algorithms in our experiments: for instance, at any fixed upload data cap on non-iid CIFAR-10, it gains an impressive accuracy advantage of 10% over FedAvgM when given the same upload data cap; the accuracy gap remains 3% even when FedAvgM is given 2 times the upload data cap, further demonstrating efficacy of FedDST. Code is available at: https://github.com/bibikar/feddst.

show abstract

Section: Related Work Federated Learningmentioning

confidence: 99%

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Bibikar

Vikalo

Wang

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…2) Heterogeneous data distributions, e.g., non-i.i.d. features, D k ∼ D i for ordered domain i, induces additional challenges with a skewed budget distribution, since training a single model on multiple domains (Li et al, 2020b) or models in a single domain (due to budget constraints) (Hong et al, 2021c;Dong et al, 2021) are known to be suboptimal on transfer performance.…”

Section: Problem Settingmentioning

confidence: 99%

“…The degradation could be worsened as facing data heterogeneity: The training datasets from participants are not independent and identically distributed (non-i.i.d.) (Li et al, 2020b;Fallah et al, 2020;Hong et al, 2021c;Zhu et al, 2021). When one device with a unique data distribution cannot afford training a large model, the global large model may not transfer to the unseen distribution (Pan & Yang, 2010).…”

Section: Introductionmentioning

confidence: 99%

Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization

Hong¹,

Wang²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) provides a distributed learning framework for multiple participants to collaborate learning without sharing raw data. In many practical FL scenarios, participants have heterogeneous resources due to disparities in hardware and inference dynamics that require quickly loading models of different sizes and levels of robustness. The heterogeneity and dynamics together impose significant challenges to existing FL approaches and thus greatly limit FL's applicability. In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is done, provides in-situ customization of model sizes and robustness. Specifically, we achieve customization by learning a set of base sub-networks of different sizes and robustness levels, which are later aggregated on-demand according to inference requirements. This split-mix strategy achieves customization with high efficiency in communication, storage, and inference. Extensive experiments demonstrate that our method provides better in-situ customization than the existing heterogeneous-architecture FL methods. Codes and pre-trained models are available: https://github.com/illidanlab/SplitMix.

show abstract

Section: Related Work Federated Learningmentioning

confidence: 99%

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Bibikar¹,

Vikalo²,

Wang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. Unfortunately, current deep networks remain not only too compute-heavy for inference and training on edge devices, but also too large for communicating updates over bandwidth-constrained networks. In this paper, we develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) by which complex neural networks can be deployed and trained with substantially improved efficiency in both ondevice computation and in-network communication. At the core of FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network. With this scheme, "two birds are killed with one stone:" instead of full models, each client performs efficient training of its own sparse networks, and only sparse networks are transmitted between devices and the cloud. Furthermore, our results reveal that the dynamic sparsity during FL training more flexibly accommodates local heterogeneity in FL agents than the fixed, shared sparse masks. Moreover, dynamic sparsity naturally introduces an "in-time self-ensembling effect" into the training dynamics, and improves the FL performance even over dense training. In a realistic and challenging non i.i.d. FL setting, FedDST consistently outperforms competing algorithms in our experiments: for instance, at any fixed upload data cap on non-iid CIFAR-10, it gains an impressive accuracy advantage of 10% over FedAvgM when given the same upload data cap; the accuracy gap remains 3% even when FedAvgM is given 2× the upload data cap, further demonstrating efficacy of FedDST. Code is available at: https://github.com/bibikar/feddst.

show abstract

Federated Adversarial Debiasing for Fair and Transferable Representations

Cited by 24 publications

References 23 publications

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Contact Info

Product

Resources

About