BOSS-LDG: A Novel Computational Framework That Brings Together Blue Waters, Open Science Grid, Shifter and the LIGO Data Grid to Accelerate Gravitational Wave Discovery

Abstract

Abstract-We present a novel computational framework that connects Blue Waters, the NSF-supported, leadership-class supercomputer operated by NCSA, to the Laser Interferometer Gravitational-Wave Observatory (LIGO) Data Grid via Open Science Grid technology. To enable this computational infrastructure, we configured, for the first time, a LIGO Data Grid Tier-1 Center that can submit heterogeneous LIGO workflows using Open Science Grid facilities. In order to enable a seamless connection between the LIGO Data Grid and Blue Waters via Open Science Grid, we utilize Shifter to containerize LIGO's workflow software. This work represents the first time Open Science Grid, Shifter, and Blue Waters are unified to tackle a scientific problem and, in particular, it is the first time a framework of this nature is used in the context of large scale gravitational wave data analysis. This new framework has been used in the last several weeks of LIGO's second discovery campaign to run the most computationally demanding gravitational wave search workflows on Blue Waters, and accelerate discovery in the emergent field of gravitational wave astrophysics. We discuss the implications of this novel framework for a wider ecosystem of Higher Performance Computing users.

Optimal Search for an Astrophysical Gravitational-Wave Background

“…In order to reduce the cost, it is worthwhile to consider the development of new computational methods and implementation on new hardware architecture in order to realize the ultimate goals of the search. These advances might come, for instance, by exploring greater parallelization of parameter estimation methods through the use of, e.g., graphical processor unit (GPU) clusters or larger supercomputer clusters; see, e.g., [31]. We note in passing that it might be possible to detect some individually resolvable sources using parameter estimation if they were just below the threshold for detection with matched filter pipelines [32].…”

Supporting High-Performance and High-Throughput Computing for Experimental Science

“…In order to connect the LDG to Blue Waters, the NSFsupported, leadership-class supercomputer operated by NCSA, authors of this article spearheaded the unification of OSG, Shifter, and Blue Waters [68,69].…”

“…Since Shifter is supported natively by Blue Waters, LIGO used it to encapsulate a full analysis software stack and to use Blue Waters as a computing resource during O2 [68], adding gravitational wave science to the portfolio of science enabled by Blue Waters. Figure 3 shows the setup used to start Shifter jobs.…”

“…The open source, PyCBC pipeline used for this work can be obtained at [76]. Documentation to use this pipeline on OSG resources is available at [77], whereas instructions to run PyCBC on Blue Waters using Shifter and OSG is presented in [68,69]. Recent developments to provide access to XSEDE resources through OSG infrastructure is described in [78].…”

Container solutions for HPC Systems

“…In this paper, we showcase a number of efforts that have successfully harnessed the unique computing capabilities of the Blue Waters supercomputer, the NSF-supported, leadership-class supercomputer operated by National Center for Supercomputing Applications (NCSA), to enable scientific discovery. We focus on efforts that have been spearheaded by researchers at NCSA and the University of Illinois at Urbana-Champaign [7,17]. These efforts provide just a glimpse of the wide spectrum of applications in which containers help advance fundamental science: from the discovery of gravitational waves from the collision of two neutron stars with the LIGO and Virgo detectors [1], to the study of the fundamental building blocks of nature and their interactions with CERN's Large Hadron Collider (LHC).…”