Towards an End-User Layer for Data Integrity

Abu-Rayyan, Lu'ay; Hacid, Hakim; Leonce, Andrew

doi:10.1145/3350546.3352538

Cited by 2 publications

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ensuring the correctness of the transferred data is one of the primary concerns of data transfer frameworks. Many studies have been performed on the design and implementation of data integrity verification and their optimization in data storage [34,36,37], cloud storage [38][39][40][41][42], file systems [32][33][34][35]43], databases [32,44,45], and data transfer systems [18,19,46,47].…”

Section: Related Workmentioning

confidence: 99%

TPBF: Two-Phase Bloom-Filter-Based End-to-End Data Integrity Verification Framework for Object-Based Big Data Transfer Systems

2022

View full text Add to dashboard Cite

Computational science simulations produce huge volumes of data for scientific research organizations. Often, this data is shared by data centers distributed geographically for storage and analysis. Data corruption in the end-to-end route of data transmission is one of the major challenges in distributing the data geographically. End-to-end integrity verification is therefore critical for transmitting such data across data centers effectively. Although several data integrity techniques currently exist, most have a significant negative influence on the data transmission rate as well as the storage overhead. Therefore, existing data integrity techniques are not viable solutions in high performance computing environments where it is very common to transfer huge volumes of data across data centers. In this study, we propose a two-phase Bloom-filter-based end-to-end data integrity verification framework for object-based big data transfer systems. The proposed solution effectively handles data integrity errors by reducing the memory and storage overhead and minimizing the impact on the overall data transmission rate. We investigated the memory, storage, and data transfer rate overheads of the proposed data integrity verification framework on the overall data transfer performance. The experimental findings showed that the suggested framework had 5% and 10% overhead on the total data transmission rate and on the total memory usage, respectively. However, we observed significant savings in terms of storage requirements, when compared with state-of-the-art solutions.

show abstract