Concurrent and Robust End-to-End Data Integrity Verification Scheme for Flash-Based Storage Devices

Abstract

The amount of data generated by scientific applications on high-performance computing systems is growing at an ever-increasing pace. Most of the generated data are transferred to storage in remote systems for various purposes such as backup, replication, or analysis. To detect data corruption caused by network or storage failures during data transfer, the receiver system verifies data integrity by comparing the checksum of the data. However, the internal operation of the storage device is not sufficiently investigated in the existing end-to-end integrity verification techniques. In this paper, we propose a concurrent and reliable end-to-end data integrity verification scheme considering the internal operation of the storage devices for data transfer between high-performance computing systems with flash-based storage devices. To perform data integrity verification including data corruptions that occurred inside the storage devices, we control the order of I/O operations considering the internal operations of the storage devices. Also, to prove the effectiveness of the proposed scheme, we devise a prototype that injects faults on the specific layer of the storage stack and examines detection of faults. We parallelize checksum computation and overlap it with I/O operations to mitigate the overhead caused by I/O reordering. The experimental results show that the proposed scheme reduces the entire data transfer time by up to 62% compared with the existing schemes while ensuring robust data integrity. With the prototype implementation, our scheme detects failures on NAND flash memory inside storage devices that cannot be detected with the existing schemes.

A Survey on Anomalies and Faults that may Impact the Reliability of Renewable-based Power Systems

“…fiber breakage, excessive bending, connectors or splice breakage [169,170]), receiver failure involving a malfunction of the receiver, such as a high data packet reception time [171,172] and data integrity that affect the integrity of the transferred data, degrading the accuracy and reliability of the transmission, and caused by alteration or loss of part of the transmitted data packet [173]. These last are, generally, recognized by the receiver using checksum [174]. The survey has highlighted that failures due to the support medium are more frequent than the other two types.…”