Leveraging High Performance Computing for Managing Large and Evolving Data Collections

Abstract

The process of developing a digital collection in the context of a research project often involves a pipeline pattern during which data growth, data types, and data authenticity need to be assessed iteratively in relation to the different research steps and in the interest of archiving. Throughout a project’s lifecycle curators organize newly generated data while cleaning and integrating legacy data when it exists, and deciding what data will be preserved for the long term. Although these actions should be part of a well-oiled data management workflow, there are practical challenges in doing so if the collection is very large and heterogeneous, or is accessed by several researchers contemporaneously. There is a need for data management solutions that can help curators with efficient and on-demand analyses of their collection so that they remain well-informed about its evolving characteristics. In this paper, we describe our efforts towards developing a workflow to leverage open science High Performance Computing (HPC) resources for routinely and efficiently conducting data management tasks on large collections. We demonstrate that HPC resources and techniques can significantly reduce the time for accomplishing critical data management tasks, and enable a dynamic archiving throughout the research process. We use a large archaeological data collection with a long and complex formation history as our test case. We share our experiences in adopting open science HPC resources for large-scale data management, which entails understanding usage of the open source HPC environment and training users. These experiences can be generalized to meet the needs of other data curators working with large collections.

Towards Use And Reuse Driven Big Data Management

“…As of this writing, within XSEDE only the Texas Advanced Computing Center (TACC) is capable of allocating sufficient long-term storage space for the Goodwin Hall sensor data from its RANCH long-term storage system [46]. Even with more advanced data transfer tools [11], moving data across XSEDE still presents a major challenge [3]. If we are to avoid moving data far from the storage, TACC's Lonestar and Stampede become the only viable resources for the associated data processing and reuse.…”

“…If we are to avoid moving data far from the storage, TACC's Lonestar and Stampede become the only viable resources for the associated data processing and reuse. Furthermore, HPC services may experience frequent interruptions and overall do not match the quality of service promised by their commercial counterparts [3].…”

“…It reports the experience running file format identification tool DROID on TACC's Stampede to extract metadata from a 4.3 TB dataset. A single Stampede node can reduce the processing time from 2 days to 6 hours, but moving data from where they are stored to Stampede, both of which are on the University of Texas Austin campus, took 28 hours [3]. In contrast, if using our 16-worker test results as a reference, we may be able to copy the same amount of data from S3 to EC2 in 7 minutes or even shorter if we use more workers.…”

Library cultures of data curation: Adventures in astronomy

“…Actions taken to preserve the data set in a stable format are essential. Cleaning data sets can improve their quality and make it easier for prospective reusers to understand its contents (Arora, Esteva, & Trelogan, 2014). Providing accompanying metadata and other information about provenance can help prospective reusers to find and interpret the data set, and to make judgments about its quality (Fear & Donaldson, 2012; Greenberg, 2017).…”