A Science Driven Production Cyberinfrastructure—the Open Science Grid

Abstract

This article describes the Open Science Grid, a large distributed computational infrastructure in the United States which supports many different high-throughput scientific applications, and partners (federates) with other infrastructures nationally and internationally to form multi-domain integrated distributed systems for science. The Open Science Grid consortium not only provides services and software to an increasingly diverse set of scientific communities, but also fosters a collaborative team of practitioners and researchers who use, support and advance the state of the art M. Altunay · T. Levshina · R. Pordes (B) · C. Sehgal in large-scale distributed computing. The scale of the infrastructure can be expressed by the daily throughput of around seven hundred thousand jobs, just under a million hours of computing, a million file transfers, and half a petabyte of data movement. In this paper we introduce and reflect on some of the OSG capabilities, usage and activities.

GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs

“…Currently, this framework is being used to provision resources to high-level systems, such as workflow managers, e.g., Pegasus [36] (through Corral [37]), or it is used directly through the Condor CLI by users allowed to access core services in OSG [38]. The main differences from a pure glidein system are an enhancement of the grid job wrapper for condor startd and the inclusion of a specialised PiF compound of two principal modules: the VO (or Corral) Frontend and the glidein Factory.…”

“…A similar two-level scheduling concept [38] is proposed by the PanDA pilot framework [21], but based on Condor-G and datasets. Several PiFs (AutoPilot [44]) are installed with Condor-G when it is deployed at central services to adjust the number of pilots and tasks in such a way that computing elements (CEs) will not be overloaded.…”

“…In this way, an approach based on site-allocated PiFs was also proposed in DIRAC [46], but the utilisation was limited to standalone clusters [47]. Similarly, some PanDA developers [48] considered the creation of a new site-allocated PiF that supplied glideins [49], but this solution did not seem to be feasible because it forced an unnecessary re-implementation of glideinWMS functionalities and the infrastructure was overloaded with two pilot systems [38].…”

A Comparison of a Campus Cluster and Open Science Grid Platforms for Protein-Guided Assembly Using Pegasus Workflow Management System

“…The OSG is organized into Virtual Organizations (VO's) which include not only the people from an academic community, but also their services, software and policies [25]. OSG does not own any computing or storage resources, but allows users to use the resources contributed by the other members of the OSG and VO's.…”

Rethinking data management for big data scientific workflows

“…Such an environment is readily available on traditional supercomputers [1], [2] usually in the form of a shared parallel file system. Scientists today have multiple alternative computational platforms available to them, including commercial clouds such as Amazon AWS [3], academic science clouds such as FutureGrid [4], [5] and distributed environments such as Open Science Grid [6]. A common feature in cloud and grid infrastructures is some kind of object storage close to the compute nodes.…”