DP-XGBoost: Private Machine Learning at Scale

Abstract

The big-data revolution announced ten years ago [MCB + 11] does not seem to have fully happened at the expected scale [Ana16]. One of the main obstacle to this, has been the lack of data circulation. And one of the many reasons people and organizations did not share as much as expected is the privacy risk associated with data sharing operations.There has been many works on practical systems to compute statistical queries with Differential Privacy (DP). There have also been practical implementations of systems to train Neural Networks with DP [MAE + 18, YSS + 21], but relatively little efforts have been dedicated to designing scalable classical Machine Learning (ML) models providing DP guarantees.In this work we describe and implement a DP fork of a battle tested ML model: XGBoost. Our approach beats by a large margin previous attempts at the task, in terms of accuracy achieved for a given privacy budget. It is also the only DP implementation of boosted trees that scales to big data and can run in distributed environments such as: Kubernetes, Dask or Apache Spark.

Federated Boosted Decision Trees with Differential Privacy

“…For the rest of this paper, we present algorithms as if the data were held centrally, with the understanding that all the operations we use can be performed in the federated model (with rounding to fixed precision) 2 . This means that we avoid techniques designed for central evaluation such as the exponential mechanism [33,45,65]. Threat Model: In this work, in common with many other works in the federated setting, we assume an honest-but-curious model, where the clients do not trust others with their raw data.…”

“…While this is suitable in non-private settings, it is difficult to calculate such quantiles (or quantile sketches) accurately without incurring an appreciable privacy cost. Existing work on DP-GBDTs has computed split candidates either with LDP quantiles in the local setting [43], DP quantiles in the central setting [33] or with MPC methods (without DP guarantees) in distributed settings [61]. As we assume bounds on features are public knowledge, we do not need to query participants' data, and hence 𝜅 𝑐 = 0.…”

“…In parallel, many works have studied decision tree models under the central model of DP [25,52,65]. Most studies focus on training random forest (RF) models and there has been little research to explore trade-offs between gradient boosting and DP; those that do often use central DP mechanisms that cannot easily be extended to federated settings [33]. It therefore remains an open problem to implement GBDTs in the federated DP setting, and show how to obtain utility comparable to their centralized non-private counterparts.…”