The effect of data cleaning on record linkage quality

Abstract

Background: Within the field of record linkage, numerous data cleaning and standardisation techniques are employed to ensure the highest quality of links. While these facilities are common in record linkage software packages and are regularly deployed across record linkage units, little work has been published demonstrating the impact of data cleaning on linkage quality. Methods: A range of cleaning techniques was applied to both a synthetically generated dataset and a large administrative dataset previously linked to a high standard. The effect of these changes on linkage quality was investigated using pairwise F-measure to determine quality. Results: Data cleaning made little difference to the overall linkage quality, with heavy cleaning leading to a decrease in quality. Further examination showed that decreases in linkage quality were due to cleaning techniques typically reducing the variability -although correct records were now more likely to match, incorrect records were also more likely to match, and these incorrect matches outweighed the correct matches, reducing quality overall. Conclusions: Data cleaning techniques have minimal effect on linkage quality. Care should be taken during the data cleaning process.

Secure and scalable deduplication of horizontally partitioned health data for privacy-preserving distributed statistical computation

“…We also assume appropriate data cleaning occurs before the protocol is run. Various data-cleaning techniques, such phonetic encoding algorithms, have been proposed in the literature [58]. …”

Privacy‐preserving record linkage

“…These are always prone to errors, for example by typing or optical character recognition errors or by changes in values such as name changes after marriage or changes of addresses. Slightly different ways of spelling names, for example the inclusion or exclusion of academic or generational titles or other name suffixes, usually do not strongly affect record linkage procedures with unencrypted identifiers (for an example, see Randall et al 2013). However, if the identifiers are encrypted with HMACs, even small variations will result in completely different encodings after encryption.…”

Comparative validity of methods to select appropriate cutoff weight for probabilistic linkage without unique personal identifiers

“…A recent study by Randall et al . tested a range of data‐cleaning techniques in real and simulated data and showed that the overall linkage quality was mostly unaffected …”