Automated Linking of Historical Data

Abstract

The recent digitization of complete count census data is an extraordinary opportunity for social scientists to create large longitudinal datasets by linking individuals from one census to another or from other sources to the census. We evaluate different automated methods for record linkage, performing a series of comparisons across methods and against hand linking. We have three main findings that lead us to conclude that automated methods perform well. First, a number of automated methods generate very low (less than 5%) false positive rates. The automated methods trace out a frontier illustrating the tradeoff between the false positive rate and the (true) match rate. Relative to more conservative automated algorithms, humans tend to link more observations but at a cost of higher rates of false positives. Second, when human linkers and algorithms use the same linking variables, there is relatively little disagreement between them. Third, across a number of plausible analyses, coefficient estimates and parameters of interest are very similar when using linked samples based on each of the different automated methods. We provide code and Stata commands to implement the various automated methods.

The Intergenerational Effects of a Large Wealth Shock: White Southerners After the Civil War

“…Our match rates of around 20 percent from the 1860 Census to either 1870 or 1900 is standard for Census-based linking in the nineteenth century, due to factors like the widespread use of first initials, rather than complete names, on Census manuscripts, and the old-fashioned handwriting that can lead to transcription errors in the digitization process (see, for example, Bleakley and Ferrie, 2016;Salisbury, 2017;Eli, Salisbury and Shertzer, 2018). Abramitzky, et al (2019) document that, even in more recent Census files (=1940), the maximum match rate is around 50 percent, particularly due to the prevalence of common names that cannot be distinguished within year of birth/ state of birth cells.…”

“…We present results using a more conservative matching strategy that requires individuals to be unique by name and state of birth within a five-year age band. This conservative procedure is roughly as successful at reducing the "false positive" rate as are a series of more computationally-intensive matching approaches (Bailey et al, 2017;Abramitzky et al, 2019).…”

Long-run Impacts of Agricultural Shocks on Educational Attainment: Evidence from the Boll Weevil

“…However, we only retain in our linked sample individuals between the ages of 3 and 18, inclusive, in childhood census years.8 Our results are not sensitive to this restriction. 9 SeeAbramitzky et al (2018) for a description of the small differences between the matching method used in this paper and the original method described byAbramitzky, Boustan, and Eriksson (2012). Specifically, we use the abematch command provided at: https://ranabr.people.stanford.edu/matching-codes.…”

Administrative Data Linking and Statistical Power Problems in Randomized Experiments

“…Not limited to arrest, this issue is applicable to any context in which the goal of the linking process is to determine the presence of an outcome and there is no prior prediction for how many records should match, or that changes relative to the prior prediction are the quantity of interest. These points distinguish this linking case from other kinds of longitudinal record linking cases (Abramitzky et al, 2019;Bailey et al, 2017;Feigenbaum, 2016) but, to our knowledge, this distinction has not been discussed in the prior literature on administrative data linking.…”