Practices and Standards for Data and Processing at the APS

Toby, Brian H.; Gürsoy, Doğa; Carlo, Francesco De; Schwarz, Nina; Sharma, Hemant; Jacobsen, Chris

doi:10.1080/08940886.2015.1013415

Cited by 5 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… 99 At the Large Hadron Collider, ∼1 PB/s data streams are reduced by custom electronics and then processed on a distributed computing grid with hundreds of thousands of cores. 40 More routine linking of instruments with computers 100 , 101 , 102 , 103 predates the Internet. 104 Automation has involved both experiment-specific code 105 , 106 and orchestration and analysis solutions targeted at specific communities.…”

Section: Discussionmentioning

confidence: 99%

Linking scientific instruments and computation: Patterns, technologies, and experiences

Vescovi¹,

Chard²,

Saint³

et al. 2022

Patterns

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

Linking scientific instruments and computation: Patterns, technologies, and experiences

Vescovi¹,

Chard²,

Saint³

et al. 2022

Patterns

View full text Add to dashboard Cite

“…XSD computing is performed as part of our mission to enable research using X-rays and develop scientific methodologies utilizing X-rays [247]. As part of that, computation may be employed for any or all of the following processes, depending on the beamline in question:…”

Section: Facility Overview: Computing In Near Termmentioning

confidence: 99%

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

Bethel¹,

eds²

2016

View full text Add to dashboard Cite

The Department of Energy (DOE) Office of Science (SC) operates dozens of national science user facilities that span many different disciplines. These facilities include accelerators, colliders, supercomputers, light sources and neutron sources, as well as facilities for studying the nanoworld, the environment, and the atmosphere. In Fiscal Year 2014 over 33,000 researchers from academia, industry, and government laboratories, spanning all fifty states and the District of Columbia, utilized these unique facilities to perform new scientific research. Each of these facilities generates vast amounts of scientific data, and the rate, size, and complexity of this data is rapidly increasing, thanks to advances in technology. A growing concern, which motivates this workshop, is the likely significant adverse impact on science programs that will result without significant advances in the capabilities needed to manage and gain knowledge from these collections of data. The purpose of this workshop, held 29 September 2015 through 1 October 2015 in Bethesda, MD, is to help the Advanced Scientific Computing Research (ASCR) and research community better understand needs related to the management, analysis, and visualization of experimental and observational data (EOD) collected and generated by experimental and observational science projects (EOS) at Office of Science user facilities. The science needs articulated in this report, along with the findings, recommendations, and detailed discussion of issues, collectively are consistent with and show opportunity for cultivating a research, development, and deployment path that takes steps towards realizing the vision articulated in the National Strategic Computing Initiative (NCSI) [1, 2] and the Big Data Research and Development Initiative [3, 4]. Specifically, the science use cases reveal a trend towards the convergence of data and computing: data-and compute-centric needs and opportunities are increasingly intertwined, interrelated, and symbiotic. Advances in our ability to collect data in turn require advances in computational capabilities to understand, preserve, share and make optimal use of data, and can even favorably impact the quality and value of science we perform by improving the quality of data we collect. This workshop report consists of input from a set of representatives from DOE EOS facilities and researchers in mathematics and computer science. The findings, drawn from use cases that articulate science drivers, indicate acute and urgent needs. This report articulates a path forward for meeting those needs, a path that includes advances in mathematics, computer and data science, as well as advances in and the use of HPC computational and networking infrastructures. One major theme recurring in the use cases is that individual EOS projects are presently pursuing their own independent paths towards meeting data-centric challenges, which results in the duplication of effort and increased costs across the entire program. EOS projects, and the EO community as a whole, ...

show abstract