Performance and Information Leakage in Splitfed Learning and Multi-Head Split Learning in Healthcare Data and Beyond

Abstract

Machine learning (ML) in healthcare data analytics is attracting much attention because of the unprecedented power of ML to extract knowledge that improves the decision-making process. At the same time, laws and ethics codes drafted by countries to govern healthcare data are becoming stringent. Although healthcare practitioners are struggling with an enforced governance framework, we see the emergence of distributed learning-based frameworks disrupting traditional-ML-model development. Splitfed learning (SFL) is one of the recent developments in distributed machine learning that empowers healthcare practitioners to preserve the privacy of input data and enables them to train ML models. However, SFL has some extra communication and computation overheads at the client side due to the requirement of client-side model synchronization. For a resource-constrained client side (hospitals with limited computational powers), removing such conditions is required to gain efficiency in the learning. In this regard, this paper studies SFL without client-side model synchronization. The resulting architecture is known as multi-head split learning (MHSL). At the same time, it is important to investigate information leakage, which indicates how much information is gained by the server related to the raw data directly out of the smashed data—the output of the client-side model portion—passed to it by the client. Our empirical studies examine the Resnet-18 and Conv1-D architecture model on the ECG and HAM-10000 datasets under IID data distribution. The results find that SFL provides 1.81% and 2.36% better accuracy than MHSL on the ECG and HAM-10000 datasets, respectively (for cut-layer value set to 1). Analysis of experimentation with various client-side model portions demonstrates that it has an impact on the overall performance. With an increase in layers in the client-side model portion, SFL performance improves while MHSL performance degrades. Experiment results also demonstrate that information leakage provided by mutual information score values in SFL is more than MHSL for ECG and HAM-10000 datasets by 2×10−5 and 4×10−3, respectively.

Enabling All In-Edge Deep Learning: A Literature Review

“…Therefore, each client-side model sends its activations to a single common server-side sub-network, thereby reducing the required aggregation step and the need to keep multiple copies of the server-side networks as compared to the first variant as shown in Figure 10 (b). Moreover, as the server keeps only one copy of the server-side sub-network, it makes the server-side do forward and backward pass sequentially with each of the client's data (activations of the cut layer) [121,122].…”

“…Splitfed overcomes the drawback of federated learning of training a large ML model in resourceconstrained ESs [123]. At the same time, it eliminates the weakness of split learning to deal with one client at a time while training [122].…”

“…During the training phase, we can use the mutual information score (MIS) to measure the level of direct leakage (activation or gradients being sent from one edge server to another) or indirect leakage (access to DL model parameters) [122].…”

“…Nonetheless, the complexity of the data, whether it's grayscale images or multi-channel image datasets like RGB, can influence the degree of information leakage from activations sent to the client-side model. This leakage can be reduced in instances involving multi-channel datasets but could be greater with grayscale images [2].…”

“…Federated split learning is a distributed collaborative machine learning (DCML) approach that combines federated learning (FL) [1] and split learning (SL) [2]. This way, it jointly leverages the benefits of FL and SL, such as parallel computations across devices for faster DCML and splitting the model into multiple portions to support resource-constrained IoT devices.…”