Secure training of decision trees with continuous attributes

Abstract

We apply multiparty computation (MPC) techniques to show, given a database that is secret-shared among multiple mutually distrustful parties, how the parties may obliviously construct a decision tree based on the secret data. We consider data with continuous attributes (i.e., coming from a large domain), and develop a secure version of a learning algorithm similar to the C4.5 or CART algorithms. Previous MPC-based work only focused on decision tree learning with discrete attributes (De Hoogh et al. 2014). Our starting point is to apply an existing generic MPC protocol to a standard decision tree learning algorithm, which we then optimize in several ways. We exploit the fact that even if we allow the data to have continuous values, which a priori might require fixed or floating point representations, the output of the tree learning algorithm only depends on the relative ordering of the data. By obliviously sorting the data we reduce the number of comparisons needed per node to O(N log2N) from the naive O(N2), where N is the number of training records in the dataset, thus making the algorithm feasible for larger datasets. This does however introduce a problem when duplicate values occur in the dataset, but we manage to overcome this problem with a relatively cheap subprotocol. We show a procedure to convert a sorting network into a permutation network of smaller complexity, resulting in a round complexity of O(log N) per layer in the tree. We implement our algorithm in the MP-SPDZ framework and benchmark our implementation for both passive and active three-party computation using arithmetic modulo 264. We apply our implementation to a large scale medical dataset of ≈ 290 000 rows using random forests, and thus demonstrate practical feasibility of using MPC for privacy-preserving machine learning based on decision trees for large datasets.

SoK: Privacy-Preserving Collaborative Tree-based Model Learning

“…De Hoogh et al [38] opt for a generic framework [159] for SMC by using Shamir secret sharing: They train trees using the ID3 algorithm with the Gini index metric, but their solution supports only categorical data. This limitation was recently addressed by Abspoel et al [5] by using SPDZ [94].…”

“…An aggregator is an entity that combines, during the learning process, information from multiple parties. We also employ these definitions to account for the non-colluding servers model employed in several works [5,53,54,107,109,113,123]. A collective is a group of parties interested in training a tree-based machine-learning model on their joint global dataset.…”

“…We classify as "ensemble" any work that creates a forest rather than single trees. For optimization reasons, several works modify the quality metric used by the learning algorithm [5,17,34,38,53,54,112,114,122,149,178,183] (see §6). Overall in the literature, we observe a wide range of combinations on tree-types, tasks, data, and algorithms, as illustrated CART [34] CART-like [5] C4.5-like [168] CRT [116] ExRT [119] ExRT [62] CART [15,17,71,85,96,110,152,154,158,181] C4.5 [41] CART [172,192] CART-like [156] CRT [68,171] ID3/C4.5 [112] C45 [7,8,27] CART [9] C4.5 [16,26,79,183] CART [95,108,177] CRT [84] ID3/CRT…”

“…Brickell and Shmatikov propose a model where a miner creates a private model on a single remote secret database [26]. For privacy, Abspoel et al assume parties offload their data by sharing it across three non-colluding servers [5]. The offloading to only two servers is also possible [123].…”

“…Recent schemes merge directly HE and SMC for efficiency [29,97,132]. Additionally, HE and SS can be used to keep the final model secret and support oblivious predictions [5,12,75,119,123,168]. We now present how SMC and HE techniques are used to protect distributed tree-based model induction by recognizing the constraints imposed by the learning and collaborative model chosen (e.g., required information, communication topology, or data partitioning).…”