Evaluating privacy-preserving record linkage using cryptographic long-term keys and multibit trees on large medical datasets

Abstract

BackgroundIntegrating medical data using databases from different sources by record linkage is a powerful technique increasingly used in medical research. Under many jurisdictions, unique personal identifiers needed for linking the records are unavailable. Since sensitive attributes, such as names, have to be used instead, privacy regulations usually demand encrypting these identifiers. The corresponding set of techniques for privacy-preserving record linkage (PPRL) has received widespread attention. One recent method is based on Bloom filters. Due to superior resilience against cryptographic attacks, composite Bloom filters (cryptographic long-term keys, CLKs) are considered best practice for privacy in PPRL. Real-world performance of these techniques using large-scale data is unknown up to now.MethodsUsing a large subset of Australian hospital admission data, we tested the performance of an innovative PPRL technique (CLKs using multibit trees) against a gold-standard derived from clear-text probabilistic record linkage. Linkage time and linkage quality (recall, precision and F-measure) were evaluated.ResultsClear text probabilistic linkage resulted in marginally higher precision and recall than CLKs. PPRL required more computing time but 5 million records could still be de-duplicated within one day. However, the PPRL approach required fine tuning of parameters.ConclusionsWe argue that increased privacy of PPRL comes with the price of small losses in precision and recall and a large increase in computational burden and setup time. These costs seem to be acceptable in most applied settings, but they have to be considered in the decision to apply PPRL. Further research on the optimal automatic choice of parameters is needed.

Privacy preserving linkage using multiple dynamic match keys

“…The number of hashes computed for each bigram in each field depended on the weight of the field, as well as the average length of the field. Address information was not added to record-level Bloom filters as preliminary testing indicated reduced linkage quality when these fields were included; previous research has also noted this issue [17]. The middle name field was also excluded due to its high proportion of missing values.…”

Population Data Centre Profiles: Centre for Data Linkage

“…PPRL techniques may be particularly useful given the reticence of many custodians to provide person identifiers for linkage given privacy concerns. The CDL, along with a number of other groups, have played a key role in developing, implementing and popularising techniques allowing record linkage to occur on encoded personal identifiers [13][14][15][16].…”

“…The method provides strong protection as the encoding process is irreversible and the encoded output is distorted to the extent that accidental recognition of an individual is impossible. The Bloom Filter encoding means that the matching can be carried out within the context of a traditional Fellegi-Sunter probabilistic linkage [13][14][15][16]. These techniques have now been deployed for project-based linkage, with the CDL using privacy-preserving record linkage methods for a number of real-world projects, including an NHMRCfunded project investigating the continuity of care provided by primary and secondary health services and a more recent project linking health and non-health records to create a Social Investment Data Resource (described below).…”

The Past, Present and Future of the German Record Linkage Center (GRLC)

“…GRLC applied for research grants with all three groups. Due to this collaborative efforts, two joint papers have been published so far (Brown et al, 2017;Christen et al, 2017).…”

“…Combing external blocks such as year of birth with multi-bit trees allows for privacy preserving linkage of two census scale data sets within a few hours (Schnell, 2014a). For most applications, this solution is sufficient with regard to speed, accuracy and privacy (Brown et al, 2017). Therefore, this combination is provided with the record-linkage software of the GRLC (see Section 2.2).…”