The Dark Energy Survey data management system

Abstract

The Dark Energy Survey (DES) collaboration will study cosmic acceleration with a 5000 deg 2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are developing the DESDM system for operation in the high performance computing (HPC) environments at the National Center for Supercomputing Applications (NCSA) and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products will be available to the Collaboration and later to the public through a virtual-observatory compatible web portal. Our approach leverages investments in publicly available HPC systems, greatly reducing hardware and maintenance costs to the project, which must deploy and maintain only the storage, database platforms and orchestration and web portal nodes that are specific to DESDM. In Fall 2007, we tested the current DESDM system on both simulated and real survey data. We used Teragrid to process 10 simulated DES nights (3TB of raw data), ingesting and calibrating approximately 250 million objects into the DES Archive database. We also used DESDM to process and calibrate over 50 nights of survey data acquired with the Mosaic2 camera. Comparison to truth tables in the case of the simulated data and internal crosschecks in the case of the real data indicate that astrometric and photometric data quality is excellent.

Dark Energy Survey year 1 results: Cosmological constraints from galaxy clustering and weak lensing

“…In this paper, we analyze DECam images taken from August 31, 2013 to February 9, 2014 ("DES Year 1" or Y1), covering 1786 square degrees in griz after coaddition and before masking [86]. The data were processed through the DES Data Management (DESDM) system [96][97][98][99], which detrends and calibrates the raw DES images, combines individual exposures to create coadded images, and detects and catalogs astrophysical objects. Further vetting and subselection of the DESDM data products was performed by [86] to produce a high-quality object catalog (Y1 Gold) augmented by several ancillary data products including a star/galaxy separator.…”

Galaxy Populations in Massive Galaxy Clusters to $z$ = 1.1: Color Distribution, Concentration, Halo Occupation Number and Red Sequence Fraction

“…The DES-SV observations were acquired between November 1, 2012 and February 22, 2013 using the Dark Energy Camera (DECam; Flaugher et al 2015). The data were processed with an improved version of the pipeline used to process the Blanco Cosmology Survey Data (Desai et al 2012), which has its heritage in the early DES data management system (Ngeow et al 2006;Mohr et al 2008Mohr et al , 2012. Following a data flow similar to that adopted for the BCS processing, we process data from every night using the single epoch pipeline.…”

Optical–SZE scaling relations for DES optically selected clusters within the SPT-SZ Survey

“…Data have been processed through the DES Data Management system (DESDM) that is an advanced version of development versions described in several earlier publications (Ngeow et al 2006;Mohr et al 2008Mohr et al , 2012Desai et al 2012). The coadd catalog produced by DESDM of the SV dataset was analyzed and tested, resulting in the generation of the SVA1 Gold catalog (Rykoff et al 2016).…”