d-blink: Distributed End-to-End Bayesian Entity Resolution

Marchant, Neil G.; Kaplan, Andee; Elazar, Daniel N.; Rubinstein, Benjamin I. P.; Steorts, Rebecca C.

doi:10.1080/10618600.2020.1825451

Cited by 14 publications

(24 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Handing out multiple posterior sets of matched entities may be impractical, together with the associated variables needed for analysis, especially if the analysis requires a large number of posterior draws. Although there are improvements in the direction of scalability (Marchant et al, 2019), there still does not exist any reported Bayesian linkage application to files of the size of a population census. Goldstein et al (2012) and Gutman et al (2015) apply multiple imputation techniques to analysis of linkage data, which do not handle the problem of linkage data structure like the other Bayesian methods above.…”

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Linkage-Data Linear Regression

Zhang

Tuoto

2020

Journal of the Royal Statistical Society Series A: Statistics in Society

View full text Add to dashboard Cite

Data linkage is increasingly being used to combine data from different sources with the aim of identifying and bringing together records from separate files, which correspond to the same entities. Usually, data linkage is not a trivial procedure and linkage errors, false and missed links, are unavoidable. In these cases, standard statistical techniques may produce misleading inference. In this paper, we propose a method for secondary linear regression analysis, where the linked data have to be prepared by someone else, and neither the match‐key variables nor the unlinked records are available to the analyst. We develop also a diagnostic test for the assumption of non‐informative linkage errors, which is required for all existing secondary analysis adjustment methods. Our approach provides important advantages: it relies on the realistic assumption that the probabilities of correct linkage vary across the records but it does not assume that one is able to estimate the probability of correct linkage for each individual record. Moreover, it accommodates in a simple manner the general situation where the files are of different sizes and none of them is a subset of another. The proposed methodology of adjustment and testing is studied by simulation and applied to real data.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Linkage-Data Linear Regression

Zhang

Tuoto

2020

Journal of the Royal Statistical Society Series A: Statistics in Society

View full text Add to dashboard Cite

show abstract

“…We thank Professor Murray for this observation as we believe that the uniform prior on the linkage structure (or label space) may not be adequate in some record linkage problems where N is fixed and given, since prior uncertainty may be too low. Nevertheless, it has been shown to work quite well in a number of situations, where N is not known, namely, in the work of Marchant et al (2019) on an application to the United States Census Bureau.…”

Section: Prior Modeling and Identifiabilitymentioning

confidence: 99%

“…Moreover, the prior probability that two records are co-referent can be shown to be B n−1 /B n . In fact, if we turn to the recent literature on Bayesian record linkage that take a clustering approach, most assume a uniform prior on the linkage structure or co-reference matrix Marchant et al, 2019;McVeigh et al, 2020). Therefore, we ask why these particular assumptions should be considered less dangerous than ours?…”

Section: Prior Modeling and Identifiabilitymentioning

confidence: 99%

See 1 more Smart Citation

A Unified Framework for De-Duplication and Population Size Estimation (with Discussion)

Tancredi¹,

Steorts²,

Liseo³

2020

Bayesian Anal.

View full text Add to dashboard Cite

Data de-duplication is the process of detecting records in one or more datasets which refer to the same entity. In this paper we tackle the de-duplication process via a latent entity model, where the observed data are perturbed versions of a set of key variables drawn from a finite population of N different entities. The main novelty of our approach is to consider the population size N as an unknown model parameter. As a result, a salient feature of the proposed method is the capability of the model to account for the de-duplication uncertainty in the population size estimation. As by-products of our approach we illustrate the relationships between de-duplication problems and capture-recapture models and we obtain a more adequate prior distribution on the linkage structure. Moreover we propose a novel simulation algorithm for the posterior distribution of the matching configuration based on the marginalization of the key variables at population level. We apply our method to two synthetic data sets comprising German names. In addition we illustrate a real data application, where we match records from two lists which report information about people killed in the recent Syrian conflict.

show abstract

A Bayesian record linkage model incorporating relational data

Sosa

Rodríguez

2023

Appl Stoch Models Bus & Ind

View full text Add to dashboard Cite

In this article, we introduce a novel Bayesian approach for linking multiple social networks in order to discover the same real world person having different accounts across networks. In particular, we develop a latent model that allows us to jointly characterize the network and linkage structures relying on both relational and profile data. In contrast to other existing approaches in the machine learning literature, our Bayesian implementation naturally provides uncertainty quantification via posterior probabilities for the linkage structure itself or any function of it. Our findings clearly suggest that our methodology can produce accurate point estimates of the linkage structure even in the absence of profile information, and also, in an identity resolution setting, our results confirm that including relational data into the matching process improves the linkage accuracy. We illustrate our methodology using real data from popular social networks such as Twitter, Facebook, and YouTube.

show abstract

d-blink: Distributed End-to-End Bayesian Entity Resolution

Cited by 14 publications

References 44 publications

Linkage-Data Linear Regression

Linkage-Data Linear Regression

A Unified Framework for De-Duplication and Population Size Estimation (with Discussion)

A Bayesian record linkage model incorporating relational data

Contact Info

Product

Resources

About