The Sloan Digital Sky Survey Data Transfer Infrastructure

Weaver, B. A.; Blanton, Michael R.; Brinkmann, J.; Brownstein, Joel R.; Stauffer, Fritz

doi:10.1086/680999

Cited by 9 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data and metadata from the plate drilling quality assurance process are backed up to the SAS. At APO, the plate-plugging metadata, observing logs, telescope telemetry, and the raw data are transferred each day from the previous night's observing to the SAS (Weaver et al 2015) and backed up on the SAM and HPSS. A similar system is installed at LCO.…”

Section: Data Managementmentioning

confidence: 99%

Sloan Digital Sky Survey IV: Mapping the Milky Way, Nearby Galaxies, and the Distant Universe

Blanton

Bershady

Abolfathi

et al. 2017

Self Cite

1,453

918

View full text Add to dashboard Cite

show abstract

Section: Data Managementmentioning

confidence: 99%

Sloan Digital Sky Survey IV: Mapping the Milky Way, Nearby Galaxies, and the Distant Universe

Blanton

Bershady

Abolfathi

et al. 2017

Self Cite

1,453

918

View full text Add to dashboard Cite

show abstract

“…This flexibility allows Marvin to know precisely where to look for a given file locally and also quickly switch to a remote host when needed. sdss-access has the ability to download files from an SDSS server using multi-stream rsync, a technology derived from the SDSS Transfer Product (Weaver et al 2015). This enables fast and robust file transfers, which are particularly helpful for speeding up downloads of many files.…”

Section: Abstract Path Generationmentioning

confidence: 99%

Marvin: A Tool Kit for Streamlined Access and Visualization of the SDSS-IV MaNGA Data Set

et al. 2019

View full text Add to dashboard Cite

The Mapping Nearby Galaxies at Apache Point Observatory (MaNGA) survey, one of three core programs of the fourth-generation Sloan Digital Sky Survey (SDSS-IV), is producing a massive, highdimensional integral field spectroscopic data set. However, leveraging the MaNGA data set to address key questions about galaxy formation presents serious data-related challenges due to the combination of its spatially inter-connected measurements and sheer volume. For each galaxy, the MaNGA pipelines produce relatively large data files to preserve the spatial correlations of the spectra and measurements, but this comes at the expense of storing the data set in a coarsely-chunked manner. The coarse chunking and total volume of the data make it time-consuming to download and curate locally-stored data. Thus, accessing, querying, visually exploring, and performing statistical analyses across the whole data set at a fine-grained scale is extremely challenging using just FITS files. To overcome these challenges, we have developed Marvin: a toolkit consisting of a Python package, Application Programming Interface (API), and web application utilizing a remote database. Marvin's robust and sustainable design minimizes maintenance, while facilitating user-contributed extensions such as high level analysis code. Finally, we are in the process of abstracting out Marvin's core functionality into a separate product so that it can serve as a foundation for others to develop Marvin-like systems for new science applications.

show abstract

“…The road to success was not without lessons about how to conduct big, collaborative, data-based scientific projects. SDSS data is accessed through the Catalog Archive Server (CAS), the Science Archive Server (SAS), or direct data file access via globus online, http by wget or curl (Cherinka et al 2019;Weaver et al 2015). CAS offers catalog level data and interactive tools to browse through SDSS images with links to associated spectra and catalog data about the objects on the images.…”

Section: Sdss Digital Infrastructure -Joel Brownsteinmentioning

confidence: 99%