Probabilistic Integration of Large Brazilian Socioeconomic and Clinical Databases

Abstract

The integration of disparate large and heterogeneous socioeconomic and clinical databases is now considered essential to capture and model longitudinal and social aspects of diseases. However, such integration has significant challenges associated with it. Databases are often stored in disparate locations, make use of different identifiers, have variable data quality, record information in bespoke purpose-specific formats and have different levels of associated metadata. Novel computational methods are required to integrate such databases and enable their statistical analyses for clinical research purposes. In this paper, we describe a probabilistic approach for constructing a very large population-based cohort comprised of 114 million individuals using linkages between clinical databases from the National Health System and other administrative databases from various government entities in order to facilitate epidemiological research. We discuss and evaluate the design and validation of our data integration model and probabilistic data linkage methods for creating research data marts that can be statistically analyzed.

The effect of primary health care on tuberculosis in a nationwide cohort of 7·3 million Brazilian people: a quasi-experimental study

“…Demographic and socioeconomic adjusting variables showed an overall protective effect of being a woman, having a higher level of education, and having a higher income per capita, and an increased risk among Black people, older people, and those living in households built with precarious materials (table 3). The associations for all adjusting variables for the models in table 4 are shown in the appendix 2 (pp [15][16][17][18].…”

“…Algorithms and codes were developed to make efficient, highly sensitive, and specific linkages using the name of each individual, date of birth, sex, and municipality of residence present in each of these data systems. 11,14,15 The 100 Million Brazilians Cohort baseline and SINAN datasets were linked by five individual-level identifiers in two steps using the CIDACS-record linkage tool. In the first step, entries were deterministically linked.…”

Road Data Enrichment Framework Based on Heterogeneous Data Fusion for ITS

“…There are some techniques to mitigate data incompleteness in LBSM. For instance, Pinto et al [23] proposed a record linkage approach to enrich incomplete data. Dubois and Prade [24] and Yagger [25] used possibility theory and the probability of fuzzy events to handle imperfect data.…”

Evaluation of approximate comparison methods on Bloom filters for probabilistic linkage

“…Field level Bloom filters encode each identifier into a separate Bloom filter [14]. Record linkage techniques (deterministic and probabilistic) can then be used to link records in much the same way as with unencrypted identifiers [15][16][17]. Record level (or composite) Bloom filters encode two or more identifiers into a single Bloom filter [5,18].…”

“…Probabilistic record linkage is preferred by many data linkage centres due to its proven track record of producing high quality linkage results from unencrypted identifiers [21][22][23]. It has been shown to produce equally good results when applied to Bloom filters [1,15,16]. An extension to the basic probabilistic model of record linkage allows for approximate matches between fields.…”