FedMask

Li, Ang; Sun, Jingwei; Zeng, Xiao; Zhang, Mi; Li, Hai; Chen, Yiran

doi:10.1145/3485730.3485929

Cited by 47 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Federated Mask (FedMask) is an efficient framework in terms of processing and communication. When the Fed-Mask algorithm is implemented, each node can learn a sparse binary mask that is heterogeneous and structured ( Li et al, 2021 ). IoT devices can create a sparse model using this approach, resulting in lower computational costs and a smaller memory footprint.…”

Section: Resultsmentioning

confidence: 99%

Integration of federated learning with IoT for smart cities applications, challenges, and solutions

Ghadi,

Mazhar,

Shah

et al. 2023

PeerJ Computer Science

View full text Add to dashboard Cite

In the past few years, privacy concerns have grown, making the financial models of businesses more vulnerable to attack. In many cases, it is hard to emphasize the importance of monitoring things in real-time with data from Internet of Things (IoT) devices. The people who make the IoT devices and those who use them face big problems when they try to use Artificial Intelligence (AI) techniques in real-world applications, where data must be collected and processed at a central location. Federated learning (FL) has made a decentralized, cooperative AI system that can be used by many IoT apps that use AI. It is possible because it can train AI on IoT devices that are spread out and do not need to share data. FL allows local models to be trained on local data and share their knowledge to improve a global model. Also, shared learning allows models from all over the world to be trained using data from all over the world. This article looks at the IoT in all of its forms, including “smart” businesses, “smart” cities, “smart” transportation, and “smart” healthcare. This study looks at the safety problems that the federated learning with IoT (FL-IoT) area has brought to market. This research is needed to explore because federated learning is a new technique, and a small amount of work is done on challenges faced during integration with IoT. This research also helps in the real world in such applications where encrypted data must be sent from one place to another. Researchers and graduate students are the audience of our article.

show abstract

Section: Resultsmentioning

confidence: 99%

Integration of federated learning with IoT for smart cities applications, challenges, and solutions

Ghadi,

Mazhar,

Shah

et al. 2023

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Thus, the communication cost is reduced from s a M p to s a K, communication trips are reduces from M p to K, and server only need to conduction sum operations for K − 1 times. However, some algorithms (Li et al, 2021a;Collins et al, 2021;Lin et al, 2020) need to communicate parameters that should be collected at the server but not averaged (Special Params.). For these parameters of size s e , devices wrap them into a message and send to the server.…”

Section: Hierarchical Aggregationmentioning

confidence: 99%

“…Many FL variant algorithms (Li et al, 2020;Karimireddy et al, 2020b;Wang et al, 2020;Acar et al, 2021;Luo et al, 2021;Li et al, 2021c;Chen & Chao, 2021;Collins et al, 2021) are developed to tackle the data heterogeneity problem where clients typically have different data distributions and/or various data sizes, making simple FL algorithms, like FedAvg, difficult to converge and leads to bad generalization performance (Woodworth et al, 2020;Acar et al, 2021). These algorithms may not be limited to exchanging model parameters during training, but possibly include other parameters like intermediate features (Collins et al, 2021), masks of model (Li et al, 2021a), auxiliary gradient corrections (Karimireddy et al, 2020b), third-party datasets (Lin et al, 2020;Tang et al, 2022), etc. Moreover, many FL algorithms require stateful clients to store some client state, like the control variates (Karimireddy et al, 2020b), old gradients (Acar et al, 2021), personalized models or layers (Liang et al, 2020;Chen & Chao, 2021), model masks (Li et al, 2021a) etc.…”

Section: Introductionmentioning

confidence: 99%

“…These algorithms may not be limited to exchanging model parameters during training, but possibly include other parameters like intermediate features (Collins et al, 2021), masks of model (Li et al, 2021a), auxiliary gradient corrections (Karimireddy et al, 2020b), third-party datasets (Lin et al, 2020;Tang et al, 2022), etc. Moreover, many FL algorithms require stateful clients to store some client state, like the control variates (Karimireddy et al, 2020b), old gradients (Acar et al, 2021), personalized models or layers (Liang et al, 2020;Chen & Chao, 2021), model masks (Li et al, 2021a) etc.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

HARMONY: Heterogeneity-Aware Hierarchical Management for Federated Learning System

Tian

Shi

et al. 2022

2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO)

View full text Add to dashboard Cite

Federated Learning (FL) enables collaborations among clients for train machine learning models while protecting their data privacy. Existing FL simulation platforms that are designed from the perspectives of traditional distributed training, suffer from laborious code migration between simulation and production, low efficiency, low GPU utility, low scalability with high hardware requirements and difficulty of simulating stateful clients. In this work, we firstly demystify the challenges and bottlenecks of simulating FL, and design a new FL system named as FedML Parrot. It improves the training efficiency, remarkably relaxes the requirements on the hardware, and supports efficient large-scale FL experiments with stateful clients by: (1) sequential training clients on devices; (2) decomposing original aggregation into local and global aggregation on devices and server respectively; (3) scheduling tasks to mitigate straggler problems and enhance computing utility; (4) distributed client state manager to support various FL algorithms. Besides, built upon our generic APIs and communication interfaces, users can seamlessly transform the simulation into the real-world deployment without modifying codes. We evaluate Parrot through extensive experiments for training diverse models on various FL datasets to demonstrate that Parrot can achieve simulating over 1000 clients (stateful or stateless) with flexible GPU devices setting (4 ∼ 32) and high GPU utility, 1.2 ∼ 4 times faster than FedScale, and 10 ∼ 100 times memory saving than FedML. And we verify that Parrot works well with homogeneous and heterogeneous devices in three different clusters. Two FL algorithms with stateful clients and four algorithms with stateless clients are simulated to verify the wide adaptability of Parrot to different algorithms. Code will be merged into https://github.com/FedML-AI/FedML.

show abstract

“…This discovery is particularly interesting for FL training, due to the lower communication overhead associated with exchanging binary masks in the UL and DL instead of float-bit representations of the weight updates. In [8], the authors introduce FedMask, a personalized Federated Learning (FL) algorithm based on pruning overparameterized random networks. FedMask is a deterministic algorithm that involves pruning a random network by optimizing personalized binary masks using Stochastic Gradient Descent (SGD), aiming to approximate the personalized target networks that fit the heterogeneous datasets found at the devices.…”

Section: Introductionmentioning

confidence: 99%

UAV-Aided Multi-Community Federated Learning

Mestoukirdi

Esrafilian

Gesbert

et al. 2022

GLOBECOM 2022 - 2022 IEEE Global Communications Conference

View full text Add to dashboard Cite

This work presents a new method for enhancing communication efficiency in stochastic Federated Learning that trains over-parameterized random networks. In this setting, a binary mask is optimized instead of the model weights, which are kept fixed. The mask characterizes a sparse sub-network that is able to generalize as good as a smaller target network. Importantly, sparse binary masks are exchanged rather than the floating point weights in traditional federated learning, reducing communication cost to at most 1 bit per parameter. We show that previous state of the art stochastic methods fail to find the sparse networks that can reduce the communication and storage overhead using consistent loss objectives. To address this, we propose adding a regularization term to local objectives that encourages sparser solutions by eliminating redundant features across sub-networks. Extensive experiments demonstrate significant improvements in communication and memory efficiency of up to five magnitudes compared to the literature, with minimal performance degradation in validation accuracy in some instances.

show abstract

FedMask

Cited by 47 publications

References 16 publications

Integration of federated learning with IoT for smart cities applications, challenges, and solutions

Integration of federated learning with IoT for smart cities applications, challenges, and solutions

HARMONY: Heterogeneity-Aware Hierarchical Management for Federated Learning System

UAV-Aided Multi-Community Federated Learning

Contact Info

Product

Resources

About