Communication Compression for Decentralized Training

Tang, Hanlin; Gan, Shaoduo; Zhang, Ce; Zhang, Tong; Ji, Liu

doi:10.48550/arxiv.1803.06443

Cited by 36 publications

(10 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Overall, the end-to-end training time is an acceptable training time (less than 1 hour) for practical applications. We can also find that the ratio of communication almost costs half of the end-to-end training time, indicating that the communication compression technique [7,18] is essential to improve the system performance in the IoT setting.…”

Section: Analysis Of System Efficiencymentioning

confidence: 96%

Federated Learning for Internet of Things

Zhang¹,

He²,

Ma³

et al. 2021

Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems

View full text Add to dashboard Cite

Federated learning can be a promising solution for enabling IoT cybersecurity (i.e., anomaly detection in the IoT environment) while preserving data privacy and mitigating the high communication/storage overhead (e.g., high-frequency data from time-series sensors) of centralized over-the-cloud approaches. In this paper, to further push forward this direction with a comprehensive study in both algorithm and system design, we build FedIoT platform that contains FedDetect algorithm for on-device anomaly data detection and a system design for realistic evaluation of federated learning on IoT devices. Furthermore, the proposed FedDetect learning framework improves the performance by utilizing a local adaptive optimizer (e.g., Adam) and a cross-round learning rate scheduler. In a network of realistic IoT devices (Raspberry PI), we evaluate FedIoT platform and FedDetect algorithm in both model and system performance. Our results demonstrate the efficacy of federated learning in detecting a wider range of attack types occurred at multiple devices. The system efficiency analysis indicates that both end-to-end training time and memory cost are affordable and promising for resource-constrained IoT devices. The source code is publicly available at https://github.com/FedML-AI/FedIoT. CCS CONCEPTS• Security and privacy → Distributed systems security.

show abstract

Section: Analysis Of System Efficiencymentioning

confidence: 96%

Federated Learning for Internet of Things

Zhang¹,

He²,

Ma³

et al. 2021

Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems

View full text Add to dashboard Cite

show abstract

“…However, FL does allow a more peer-to-peer learning approach wherein each node that is trained can benefit from the other node that is trained on the FL network too. Even in decentralized training similar challenges of communication exist and to tackle it different methods like compression [60] can be used. In this subsection how decentralized or peer-to-peer learning is utilized or integrated into an FL environment.…”

Section: ) Decentralizedmentioning

confidence: 99%

Communication Efficiency in Federated Learning: Achievements and Challenges

Shahid,

Pouriyeh,

Parizi

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) is known to perform Machine Learning tasks in a distributed manner. Over the years, this has become an emerging technology especially with various data protection and privacy policies being imposed FL allows performing machine learning tasks whilst adhering to these challenges. As with the emerging of any new technology, there are going to be challenges and benefits. A challenge that exists in FL is the communication costs, as FL takes place in a distributed environment where devices connected over the network have to constantly share their updates this can create a communication bottleneck. In this paper, we present a survey of the research that is performed to overcome the communication constraints in an FL setting.

show abstract

“…In addition, there are also multiple works investigating the communication reduction in decentralized optimization. [44] Fig. 1.…”

Section: Decentralized Optimizationmentioning

confidence: 99%

Communication Efficient Tensor Factorization for Decentralized Healthcare Networks

Ma¹,

Zhang²,

Lou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Tensor factorization has been proved as an efficient unsupervised learning approach for health data analysis, especially for computational phenotyping, where the high-dimensional Electronic Health Records (EHRs) with patients history of medical procedures, medications, diagnosis, lab tests, etc., are converted to meaningful and interpretable medical concepts. Federated tensor factorization distributes the tensor computation to multiple workers under the coordination of a central server, which enables jointly learning the phenotypes across multiple hospitals while preserving the privacy of the patient information. However, existing federated tensor factorization algorithms encounter the single-point-failure issue with the involvement of the central server, which is not only easily exposed to external attacks, but also limits the number of clients sharing information with the server under restricted uplink bandwidth. In this paper, we propose CiderTF, a communication-efficient decentralized generalized tensor factorization, which reduces the uplink communication cost by leveraging a four-level communication reduction strategy designed for a generalized tensor factorization, which has the flexibility of modeling different tensor distribution with multiple kinds of loss functions. Experiments on two real-world EHR datasets demonstrate that CiderTF achieves comparable convergence with the communication reduction up to 99.99%.

show abstract

Communication Compression for Decentralized Training

Cited by 36 publications

References 17 publications

Federated Learning for Internet of Things

Federated Learning for Internet of Things

Communication Efficiency in Federated Learning: Achievements and Challenges

Communication Efficient Tensor Factorization for Decentralized Healthcare Networks

Contact Info

Product

Resources

About