Enabling petascale science: data management, troubleshooting, and scalable science services

Baranovski, A.; Beattie, K.; Bharathi, S.; Boverhof, Joshua; Bresnahan, John; Chervenak, Ann; Foster, Ian; Freeman, Tim; Gunter, Dan; Keahey, Kate; Kesselman, Carl; Kettimuthu, Rajkumar; LeRoy, N.; Link, Michael; Livny, Miron; Madduri, Ravi; Oleynik, G.; Pearlman, Laurie Anne; Schuler, Robert; Tierney, Brian

doi:10.1088/1742-6596/125/1/012068

Cited by 3 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, approximately 1 in every 300,000,000 (65 K × 5 K) packets are accepted with corruption. It has been reported that an average of 40 errors per 1,000,000 transfers is detected on data transferred by the D0 experiment [16]. Projects such as DES require verification of checksums as part of their regular data movement process in order to detect file corruption due to software bugs or human error.…”

Section: Introductionmentioning

confidence: 99%

High-Performance End-to-End Integrity Verification on Big Data Transfer

Jung

Liu

Kettimuthu

et al. 2019

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

The scale of scientific data generated by experimental facilities and simulations in high-performance computing facilities has been proliferating with the emergence of IoT-based big data. In many cases, this data must be transmitted rapidly and reliably to remote facilities for storage, analysis, or sharing, for the Internet of Things (IoT) applications. Simultaneously, IoT data can be verified using a checksum after the data has been written to the disk at the destination to ensure its integrity. However, this end-to-end integrity verification inevitably creates overheads (extra disk I/O and more computation). Thus, the overall data transfer time increases. In this article, we evaluate strategies to maximize the overlap between data transfer and checksum computation for astronomical observation data. Specifically, we examine file-level and block-level (with various block sizes) pipelining to overlap data transfer and checksum computation. We analyze these pipelining approaches in the context of GridFTP, a widely used protocol for scientific data transfers. Theoretical analysis and experiments are conducted to evaluate our methods. The results show that block-level pipelining is effective in maximizing the overlap mentioned above, and can improve the overall data transfer time with end-to-end integrity verification by up to 70% compared to the sequential execution of transfer and checksum, and by up to 60% compared to file-level pipelining.

show abstract

Section: Introductionmentioning

confidence: 99%

High-Performance End-to-End Integrity Verification on Big Data Transfer

Jung

Liu

Kettimuthu

et al. 2019

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

“…However, with an exception of integration of GridFTP with OPeNDAP [7] (which only provides limited flexibility and efficiency), the unit of data transfer for GridFTP is a single file. While enhancing and optimizing data transfer frameworks [2,14,16,17,18,13,4] has continued to be an active area of research, the ability to reduce data volume that needs to be transferred over the wide-area, by providing support for user-defined data subsetting at the server-side, is clearly needed.…”

Section: Introductionmentioning

confidence: 99%

SDQuery DSI

Wang

Agrawal

et al. 2013

Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

In many science areas where datasets need to be transferred or shared, rapid growth in dataset size, coupled with much slower increases in wide area data transfer bandwidths, is making it extremely hard for scientists to analyze the data. This paper addresses the current limitations by developing SDQuery DSI, a GridFTP plug-in that supports flexible server-side data subsetting. An existing GridFTP server is able to dynamically load this tool to support new functionality. Different queries types (query over dimensions, coordinates and values) are supported by our tool. A number of optimizations, like parallel indexing, performance model for data subsetting, and parallel streaming are also applied. We compare our SDQuery DSI with GridFTP default File DSI in different network environments, and show that our method can achieve better efficiency in almost all cases.

show abstract

Taming massive distributed datasets

Agrawal

Woodring

et al. 2013

Proceedings of the 22nd International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

Enabling petascale science: data management, troubleshooting, and scalable science services

Cited by 3 publications

References 5 publications

High-Performance End-to-End Integrity Verification on Big Data Transfer

High-Performance End-to-End Integrity Verification on Big Data Transfer

SDQuery DSI

Taming massive distributed datasets

Contact Info

Product

Resources

About