SecureGBM: Secure Multi-Party Gradient Boosting

Abstract

Federated machine learning systems have been widely used to facilitate the joint data analytics across the distributed datasets owned by the different parties that do not trust each others. In this paper, we proposed a novel Gradient Boosting Machines (GBM) framework SecureGBM built-up with a multi-party computation model based on semi-homomorphic encryption, where every involved party can jointly obtain a shared Gradient Boosting machines model while protecting their own data from the potential privacy leakage and inferential identification. More specific, our work focused on a specific "dualparty"secure learning scenario based on two partiesboth party own an unique view (i.e., attributes or features) to the sample group of samples while only one party owns the labels. In such scenario, feature and label data are not allowed to share with others.To achieve the above goal, we firstly extent -LightGBM -a well known implementation of tree-based GBM through covering its key operations for training and inference with SEAL homomorphic encryption schemes. However, the performance of such re-implementation is significantly bottle-necked by the explosive inflation of the communication payloads, based on ciphertexts subject to the increasing length of plaintexts. In this way, we then proposed to use stochastic approximation techniques to reduced the communication payloads while accelerating the overall training procedure in a statistical manner. Our experiments using the real-world data showed that SecureGBM can well secure the communication and computation of LightGBM training and inference procedures for the both parties while only losing less than 3% AUC, using the same number of iterations for gradient boosting, on a wide range of benchmark datasets. More specific, compared to LightGBM, the proposed SecureGBM would slowdown with 3x ∼ 64x time consumption per iteration in the training procedure, while SecureGBM becomes more and more efficient when the scale of the training dataset increases (i.e., the larger training set, the lower slowdown ratio). 1

SoK: Privacy-Preserving Collaborative Tree-based Model Learning

“…In this case, each party communicates with all the others. Although initially considered for the setting of two parties by Lindell and Pinkas [111] and subsequent works [41,55,56,76,160,181], it was later extended to arbitrary number of parties [38,48,70,100,102,104,106,149,168,179,182]. In this setting, some parties might be assigned specific tasks.…”

“…We identify five categories of PETs: (a) input randomization ( §6.1), (b) differential privacy-based solutions ( §6.2), (c) cryptographic approaches ( §6.3), (d) hardware-based solutions ( §6.4), and (e) hybrid solutions that combine the above ( §6.5). [41,55,56,76,96,160] FD (2 parties) [111,121,181] FD (2 parties) [80] Leader [32,37,153,166,186] Leader [72,104,156,187] Leader [190] Sequential [110,162] Aggregator [52,117,119,152,154,172] Aggregator…”

“…Similarly, Fang et al [55] propose a solution combining additive secret-sharing and HE. Contrary to similar works relying on pure HE [32] or combined with hypervisorenforced domain isolation and OT [56], their solution maintains every value encrypted or secret-shared, thus does not leak intermediate information. Wu et al [179] port XGBoost to the fully encrypted setting for more than two parties by using a threshold version of Paillier HE and the SPDZ framework [36].…”

“…For completely random approaches, the intermediate values are the class counts at the leaves. Other statistics, such as the number of local samples reaching a specific node, residual accuracy errors [56], the "owner" of a node in the vertically distributed scenario (i.e., which party holds the selected attribute for the node), might also be leaked. For gradient boosting ensemble models, the intermediate values are the weight updates or residuals.…”

Data Placement for Multi-Tenant Data Federation on the Cloud

“…In order to protect data security, data accessibility is controlled by attributing different levels of permission to avoid unauthorized or malicious access to data on the cloud [25]. In addition, encryption techniques [26], [27] and distributed data storage plan based on data partitioning [28], [29], [30] can be exploited. Federated learning is proposed to train a model while ensuring data privacy [31], yet it is not applicable to the general data processing among different organizations on the cloud.…”