A Scalable Parallel Deduplication Algorithm

Teixeira, Thiago; Machado, Carlos Alberto Tosta; Meira, Wagner; Silva, A.S. Da; Ferreira, Diogo R.; Guedes, Dorgival

doi:10.1109/sbac-pad.2007.32

Cited by 14 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The group has also developed a parallel deduplication algorithm, called FER-APARDA, using probabilistic record linkage, as well as PAREIA (Santos et al, 2007; dos Santos Filho, 2008). PAREIA’s crucial contributions are two-fold: 1) the proposed blocking scheme uses predicates from fields or portions of them, making a junction of disjunctions to prevent input errors to separate true matches from the right blocks.…”

Section: Data Linkage Initiatives In Brazilmentioning

confidence: 99%

Administrative Data Linkage in Brazil: Potentials for Health Technology Assessment

et al. 2019

View full text Add to dashboard Cite

Health technology assessment (HTA) is the systematic evaluation of the properties and impacts of health technologies and interventions. In this article, we presented a discussion of HTA and its evolution in Brazil, as well as a description of secondary data sources available in Brazil with potential applications to generate evidence for HTA and policy decisions. Furthermore, we highlighted record linkage, ongoing record linkage initiatives in Brazil, and the main linkage tools developed and/or used in Brazilian data. Finally, we discussed the challenges and opportunities of using secondary data for research in the Brazilian context. In conclusion, we emphasized the availability of high quality data and an open, modern attitude toward the use of data for research and policy. This is supported by a rigorous but enabling legal framework that will allow the conduct of large-scale observational studies to evaluate clinical, economical, and social impacts of health technologies and social policies.

show abstract

Section: Data Linkage Initiatives In Brazilmentioning

confidence: 99%

Administrative Data Linkage in Brazil: Potentials for Health Technology Assessment

et al. 2019

View full text Add to dashboard Cite

show abstract

“…This reduces the latency of the hash lookup process during duplicate detection. Scalable dedupe [10] partitions the incoming data stream based on the k-least significant bits of their corresponding fingerprints. These partitioned blocks and the fingerprints are mapped to the respective nodes using a DHT.…”

Section: Related Workmentioning

confidence: 99%

A distributed load balancing algorithm for deduplicated storage

Balasundaram¹,

Babu²,

Rengaswamy³

2019

Turk J Elec Eng & Comp Sci

View full text Add to dashboard Cite

While deduplication brings the advantage of significant space savings in storage, it nevertheless incurs the overhead of maintaining huge metadata. Updating such huge metadata during the data migration that arises due to load balancing activity results in significant overhead. In order to reduce this metadata update overhead, this paper proposes a suitable alternate index that tracks the data blocks even when they migrate across the nodes without explicitly storing the location information. In addition, a virtual server-based load balancing (VSLB) algorithm has been proposed in order to reduce the migration overhead. The experimental results indicate that the proposed index reduces the metadata update overhead by 74% when compared to the existing index. Furthermore, VSLB reduces the migration overhead by 33% when compared to the existing ID reassignment-based load balancing approach.

show abstract

“…There are many tools/software available for record linkage (16)(17)(18)(19). We tested some of these alternatives, but due to the large volume of data, we had trouble with computer performance.…”

Section: Software and Hardwarementioning

confidence: 99%

Building the National Database of Health Centred on the Individual: Administrative and Epidemiological Record Linkage - Brazil, 2000-2015

Guerra

Pereira

Gurgel

et al. 2018

IJPDS

View full text Add to dashboard Cite

IntroducationIn Brazil, the National Health System (SUS) provides healthcare to the public. The system has multiple administrative databases; the major databases record hospital (SIH) and outpatient (SIA) procedures. Epidemiological information is collected for all populations in subsystems, such as mortality (SIM), live births (SINASC) and diseases of compulsory declaration (SINAN). Each subsystem has its own information system, which is able to provide information about consultations, clinical information and medicines dispensed. However, these systems are not linked, thereby preventing individual-centred analysis. ObjectiveTo describe the methods and results of parameter setting that are needed to execute the probabilistic deduplication of large administrative and epidemiological databases in Brazil and to create a National Health Database Centred on the individual. MethodsThis paper shows the results of a record linkage model to integrate data from SIH, SIA, SIM, and SINAN, which have different formats and attributes between them and over time. These data consist of 1.3 billion records from 2000-2015. Probabilistic and deterministic record linkages were used to deduplicate these data. The Kappa statistic and clerical review were used to ensure the quality of the linkage. The graph algorithm and depth-first search were used to generate the identifiers. ResultsThe deterministic deduplication process resulted in a database with 403,113,527 possible unique individuals. After the probabilistic deduplication process of the former database was performed, 159,703,805 unique individuals were identified. This result had an estimated a false positive error rate of 3.3%, and the false negative error was estimated at 12.3%. ConclusionsThe National Health Database centred on the individual was generated and will allow researchers to use real-world evidence to conduct clinical, epidemiological, economic and other studies. This database represents a significant cohort, spanning 15 years of historical data and preserving patient privacy. The success of the process described will allow repeating and appending the data for future years and enable important studies to promote SUS efficiency and provide better treatments for patients.

show abstract

A Scalable Parallel Deduplication Algorithm

Cited by 14 publications

References 7 publications

Administrative Data Linkage in Brazil: Potentials for Health Technology Assessment

Administrative Data Linkage in Brazil: Potentials for Health Technology Assessment

A distributed load balancing algorithm for deduplicated storage

Building the National Database of Health Centred on the Individual: Administrative and Epidemiological Record Linkage - Brazil, 2000-2015

Contact Info

Product

Resources

About