Detailed comparison of communication efficiency of split learning and federated learning

Singh, Rajesh; Vepakomma, Praneeth; Gupta, Otkrist; Raskar, Ramesh

doi:10.48550/arxiv.1909.09145

Cited by 35 publications

(43 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It simply discards the gradients it received from the second part of the client model, and computes a malicious loss function using the intermediate output it received from the first client model, propagating the malicious loss back to the first client model. The primary advantage of SplitNN compared to federated learning is its lower communication load [18]. While federated learning clients have to share their entire parameter updates with the server, SplitNN clients only share the output of a single layer.…”

Section: B Split Learningmentioning

confidence: 99%

SplitGuard: Detecting and Mitigating Training-Hijacking Attacks in Split Learning

Erdogan¹,

Küpçü²,

Çiçek³

2021

Preprint

View full text Add to dashboard Cite

Distributed deep learning frameworks, such as split learning, have recently been proposed to enable a group of participants to collaboratively train a deep neural network without sharing their raw data. Split learning in particular achieves this goal by dividing a neural network between a client and a server so that the client computes the initial set of layers, and the server computes the rest. However, this method introduces a unique attack vector for a malicious server attempting to steal the client's private data: the server can direct the client model towards learning a task of its choice. With a concrete example already proposed, such training-hijacking attacks present a significant risk for the data privacy of split learning clients.In this paper, we propose SplitGuard, a method by which a split learning client can detect whether it is being targeted by a training-hijacking attack or not. We experimentally evaluate its effectiveness, and discuss in detail various points related to its use. We conclude that SplitGuard can effectively detect traininghijacking attacks while minimizing the amount of information recovered by the adversaries.

show abstract

Section: B Split Learningmentioning

confidence: 99%

SplitGuard: Detecting and Mitigating Training-Hijacking Attacks in Split Learning

Erdogan¹,

Küpçü²,

Çiçek³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The communication overhead of split learning linearly scales with the amount of training data at the client [10]. While split learning has less communication overhead than federated learning [5] when the data size is small, it is a bottleneck if the data size is large.…”

Section: Motivationmentioning

confidence: 99%

“…However, the communication overhead linearly increases with the number of training samples. In the extreme case, where the number of edge devices is small and each edge device has to process a large amount of data, communication overhead can be way higher than federated learning [10], [11].…”

Section: Introductionmentioning

confidence: 99%

Communication and Computation Reduction for Split Learning using Asynchronous Training

Chen,

Li,

Chakrabarti

2021

Preprint

View full text Add to dashboard Cite

Split learning is a promising privacy-preserving distributed learning scheme that has low computation requirement at the edge device but has the disadvantage of high communication overhead between edge device and server. To reduce the communication overhead, this paper proposes a loss-based asynchronous training scheme that updates the client-side model less frequently and only sends/receives activations/gradients in selected epochs. To further reduce the communication overhead, the activations/gradients are quantized using 8-bit floating point prior to transmission. An added benefit of the proposed communication reduction method is that the computations at the client side are reduced due to reduction in the number of client model updates. Furthermore, the privacy of the proposed communication reduction based split learning method is almost the same as traditional split learning. Simulation results on VGG11, VGG13 and ResNet18 models on CIFAR-10 show that the communication cost is reduced by 1.64x-106.7x and the computations in the client are reduced by 2.86x-32.1x when the accuracy degradation is less than 0.5% for the single-client case. For 5 and 10-client cases, the communication cost reduction is 11.9x and 11.3x on VGG11 for 0.5% loss in accuracy.

show abstract

“…However, the federated learning (data sharing) constraint means that the GNN cannot be trained in a centralized manner, since each node can only access the data stored on itself. To address this, CNFGNN employs Split Learning [25] to train the spatial and temporal modules. Further, to alleviate the associated high communication cost incurred by Split Learning, we propose an alternating optimization-based training procedure of these modules, which incurs only half the communication overhead as compared to a comparable Split Learning architecture.…”

Section: Introductionmentioning

confidence: 99%

Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modeling

Meng

Rambhatla

Liu

2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Vast amount of data generated from networks of sensors, wearables, and the Internet of Things (IoT) devices underscores the need for advanced modeling techniques that leverage the spatio-temporal structure of decentralized data due to the need for edge computation and licensing (data access) issues. While federated learning (FL) has emerged as a framework for model training without requiring direct data sharing and exchange, effectively modeling the complex spatio-temporal dependencies to improve forecasting capabilities still remains an open problem. On the other hand, state-of-the-art spatio-temporal forecasting models assume unfettered access to the data, neglecting constraints on data sharing. To bridge this gap, we propose a federated spatio-temporal model -Cross-Node Federated Graph Neural Network (CNFGNN) -which explicitly encodes the underlying graph structure using graph neural network (GNN)-based architecture under the constraint of cross-node federated learning, which requires that data in a network of nodes is generated locally on each node and remains decentralized. CN-FGNN operates by disentangling the temporal dynamics modeling on devices and spatial dynamics on the server, utilizing alternating optimization to reduce the communication cost, facilitating computations on the edge devices. Experiments on the traffic flow forecasting task show that CNFGNN achieves the best forecasting performance in both transductive and inductive learning settings with no extra computation cost on edge devices, while incurring modest communication cost. CCS CONCEPTS• Information systems → Sensor networks; Data mining; • Computing methodologies → Neural networks.

show abstract

Detailed comparison of communication efficiency of split learning and federated learning

Cited by 35 publications

References 1 publication

SplitGuard: Detecting and Mitigating Training-Hijacking Attacks in Split Learning

SplitGuard: Detecting and Mitigating Training-Hijacking Attacks in Split Learning

Communication and Computation Reduction for Split Learning using Asynchronous Training

Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modeling

Contact Info

Product

Resources

About