Towards Use And Reuse Driven Big Data Management

Abstract

We propose a use and reuse driven big data management approach that fuses the data repository and data processing capabilities in a co-located, public cloud. It answers to the urgent data management needs from the growing number of researchers who don't fit in the big science/small science dichotomy. This approach will allow researchers to more easily use, manage, and collaborate around big data sets, as well as give librarians the opportunity to work alongside the researchers to preserve and curate data while it is still fresh and being actively used. This also provides the technological foundation to foster a sharing culture more aligned with the open source software development paradigm than the lone-wolf, gift-exchanging small science sharing or the top-down, highly structured big science sharing. To materialize this vision, we provide a system architecture consisting of a scalable digital repository system coupled with the co-located cloud storage and cloud computing, as well as a job scheduler and a deployment management system. Motivated by Virginia Tech's Goodwin Hall instrumentation project, we implemented and evaluated a prototype. The results show not only sufficient capacities for this particular case, but also near perfect linear storage and data processing scalabilities under moderately high workload.

Advancing library cyberinfrastructure for big data sharing and reuse

“…It is therefore imperative to manage data with the use and reuse-driven approach [17]. Using the DCC Curation Lifecycle Model terms [8] developed by the Digital Curation Centre (DCC) in the UK, the library big data service should focus more on facilitating data use and reuse instead of spreading the library resources evenly among storage, preservation, resource description, and various transformations, each for its own purpose.…”

Evaluating Cost of Cloud Execution in a Data Repository

“…The cost of this instance was not counted towards the execution costs. The data used for experiments were vibration signals collected from 214 accelerometers mounted in Virginia Tech's Goodwin Hall [1][2][3][4], an engineering building and a highly instrumented smart infrastructure laboratory facility. The data were written into one-minute interval zlib-compressed chunked HDF5 files.…”

On-Demand Big Data Analysis in Digital Repositories: A Lightweight Approach

“…A use and reuse driven approach to manage big data [9] differs from the traditional library repository in that the emphasis is geared more towards serving the researcher's needs to answer domain-specific research questions, instead of building "preservation-ready" systems to satisfy the librarian's urge to document and arrange materials in certain ways to facilitate unspecified future access. The argument is that unless we make fresh data immediately usable and reusable to researchers in their research process, the data will quickly turn cold, become less valuable for long-term preservation, and crowd out limited IT resources for big data management.…”

“…We need to add an important component missing from the traditional library repository, namely a co-located data analysis infrastructure, to accomplish the goals laid out. Our prior research [9] compared a number of IT infrastructure options with which the use and reuse driven approach may be implemented. Given the IT environment and conditions currently prevalent in most academic libraries, we proposed the public cloud as a viable candidate.…”

“…Our prior approach [9] utilized the computing cloud in its most conventional sense by provisioning virtual machines in lieu of physical machines. We then installed usersupplied analysis code on each of them, and crunched data as if we had a traditional computing cluster on hand.…”