Budget Distributed Support Vector Machine for Non-ID Federated Learning Scenarios

Abstract

In recent years there has been remarkable growth in Federated Learning (FL) approaches because they have proven to be very effective in training large Machine Learning (ML) models and also serve to preserve data confidentiality, as recommended by the GDPR or other business confidentiality restrictions that may apply. Despite the success of FL, performance is greatly reduced when data is not distributed identically (non-ID) across participants, as local model updates tend to diverge from the optimal global solution and thus the model averaging procedure in the aggregator is less effective. Kernel Methods such as Support Vector Machines (SVMs) have not seen an equivalent evolution in the area of privacy preserving edge computing because they suffer from inherent computational, privacy and scalability issues. Furthermore, non-linear SVMs do not naturally lead to federated schemes, since locally trained models cannot be passed to the aggregator because they reveal training data (they are built on Support Vectors), and the global model cannot be updated at every worker using gradient descent. In this paper, we explore the use of a particular controlled complexity (“Budget”) Distributed SVM (BDSVM) in the Federated Learning scenario with non-ID data, which is the least favourable situation, but very common in practice. The proposed BDSVM algorithm is as follows: model weights are broadcasted to workers, which locally update some kernel Gram matrices computed according to a common architectural base and send them back to the aggregator, which finally combines them, updates the global model and repeats the procedure until a convergence criterion is met. Experimental results using synthetic 2D datasets show that the proposed method can obtain maximal margin decision boundaries even when the data is non-ID distributed. Further experiments using real world datasets with non-ID data distribution show that the proposed algorithm provides better performance with less communication requirements than a comparable Multilayer Perceptron (MLP) trained using FedAvg. The advantage is more remarkable for a larger number of edge devices. We have also demonstrated the robustness of the proposed method against information leakage, membership inference attacks and situations with dropout or straggler participants. Finally, in experiments run on separate processes/machines interconnected via the cloud messaging service developed in the context of the EU-H2020 MUSKETEER project, BDSVM is able to train better models than FedAvg in about half the time.

Quantitative risk analysis of treatment plans for patients with tumor by mining historical similar patients from electronic health records using federated learning

“…In recent years, some studies have been conducted for extending machine learning algorithms into the FL framework, such as support vector machine (SVM) (Brisimi et al., 2018; Navia‐Vázquez et al., 2022; S. Wang et al., 2021), decision tree (Cheng et al., 2021), k‐means (Huang et al., 2019), DNN (Bai et al., 2021; Dayan et al., 2021). For example, S. Wang et al.…”

“…Navia‐Vázquez et al. (2022) extended a particular controlled complexity (“Budget”) distributed SVM into the FL framework. Cheng et al.…”

“…The process is iterated until the global model converges. In recent years, some studies have been conducted for extending machine learning algorithms into the FL framework, such as support vector machine (SVM) (Brisimi et al, 2018;Navia-Vázquez et al, 2022;, decision tree (Cheng et al, 2021), k-means (Huang et al, 2019), DNN (Bai et al, 2021;Dayan et al, 2021). For example, S. Wang et al (2021) proposed an SVM-based FL algorithm to dynamically adjust the user association, service sequence, and task allocation of mobile users in flying wireless base stations.…”

“…Figure 2 that FL is an iterative procedure, and each iteration can be divided into three steps: (1) the central server distributes the global model parameters to all clients; (2) based on the received global model parameters, each client trains its local model with its private dataset and uploads the local model parameters to the central server; and (3) the central server updates the global model by aggregating the uploaded local model parameters of all clients using the aggregation algorithm, for example, Federated Averaging (McMahanet al, 2017), Federated Personalization(Manoj et al, 2019), and Federated Distillation(Jeong et al, 2018). The process is iterated until the global model converges.In recent years, some studies have been conducted for extending machine learning algorithms into the FL framework, such as support vector machine (SVM)(Brisimi et al, 2018;Navia-Vázquez et al, 2022;, decision tree(Cheng et al, 2021), k-means(Huang et al, 2019), DNN(Bai et al, 2021;Dayan et al, 2021). For example, S Wang et al (2021).…”

Improving the Efficiency of Budgeting in Industrial Enterprise: The case of Russia, Italy, and the Middle East

“…By reducing the amount of borrowing and receivables that industrial companies require, modernization and the digitalization transition techniques help increase profitability and revenue as well as the efficiency of equipment loading and production resources. The digitization of budgeting at industrial firms is carried out by IC software packages [4], and two models exist for running an autonomous budgeting system through the programs shown in Figure 1.…”