Cyberinfrastructure Center of Excellence Pilot: Connecting Large Facilities Cyberinfrastructure

Leveraging user access patterns and advanced cyberinfrastructure to accelerate data delivery from shared-use scientific observatories

“…For example, the LIGO project generates terabytes of data every day during its "observing" mode [28], and the inconstruction Square Kilometer Array (SKA) project is estimated to generate an exabyte of raw data every day though it will then be compressed to around 10 petabytes [29]. As noted at the 2019 NSF Workshops on Connecting Large Facilities and Cyberinfrastructure [4], [30], many large facilities still struggle to provide broad data transfer and access services, especially in remote environments. Furthermore, using the wide-area network (WAN) is not the most effective approach for transferring large data volumes.…”

“…Users can manually download data from the observatory website, run scripts that automatically query data, or subscribe to real-time data streams. For example, automated script-based downloads are typically a part of application workflows [4], [9], and while they often required stricter performance guarantees, their query and access patterns are more predictable.…”