BigData Express: Toward Schedulable, Predictable, and High-Performance Data Transfer

Lu, Qiming; Zhang, Liang; Sasidharan, Sajith; Wu, Wenji; DeMar, Phil; Guok, Chin; Macauley, John B.; Monga, Inder; Yu, Se-young; Chen, Jim Hao; Mambretti, Joe; Kim, Jin; Noh, S. Y.; Yang, Xi; Lehman, Tom; Liu, Gary

doi:10.1109/indis.2018.00011

Cited by 11 publications

(8 citation statements)

References 22 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…multi-ONE) and network-aware data transfer systems (e.g. BigDataExpress [13]. • Research & Education Networks Programmable Services are planned by both ESNet [14] and GEANT [15] and include projects such as ESNet6 [16], FABRIC [17], GEANT OAV [18] and GTS [19].…”

Section: Programmable Networkmentioning

confidence: 99%

Network Capabilities for the HL-LHC Era

Babik

Kee

2020

EPJ Web Conf.

View full text Add to dashboard Cite

High Energy Physics (HEP) experiments rely on the networks as one of the critical parts of their infrastructure both within the participating laboratories and sites as well as globally to interconnect the sites, data centres and experiments instrumentation. Network virtualisation and programmable networks are two key enablers that facilitate agile, fast and more economical network infrastructures as well as service development, deployment and provisioning. Adoption of these technologies by HEP sites and experiments will allow them to design more scalable and robust networks while decreasing the overall cost and improving the effectiveness of the resource utilization. The primary challenge we currently face is ensuring that WLCG and its constituent collaborations will have the networking capabilities required to most effectively exploit LHC data for the lifetime of the LHC. In this paper we provide a high level summary of the HEPiX NFV Working Group report that explored some of the novel network capabilities that could potentially be deployment in time for HL-LHC.

show abstract

Section: Programmable Networkmentioning

confidence: 99%

Network Capabilities for the HL-LHC Era

Babik

Kee

2020

EPJ Web Conf.

View full text Add to dashboard Cite

show abstract

“…One challenge is a need for the highest possible performance for the transfers because of the volume of data. Another challenge is the need to address explicit or implicit time constraints determined by scientific applications [3]. Timeconstraint categories include real-time data transfer (i.e., data transfer is on the critical path of a workflow with a specific deadline), deadline bound data transfer (data transfer is not on the critical path but does have an explicit deadline, and background data transfer, which has no explicit deadline.…”

Section: Primary Challenges and Our Solutionmentioning

confidence: 99%

“…An end-to-end network path typically consists of LAN and WAN segments. In the BigData Express end-to-end data transfer model, LAN segments are provisioned and guaranteed by AmoebaNet [3][9], while WAN segments are provisioned via the SENSE service to provide the path between the data source and destination sites.…”

Section: On-demand Provisioning Of End-to-end Network Paths With Guaranteed Qosmentioning

confidence: 99%

“…The comprehensive approach used here is key. The services include Rucio [2], a data management service widely used by particle physics research communities; BigData Express [3], a schedulable, predictable, and high-performance data transfer service; and SENSE [4], a distributed network resource orchestrator. Each service is comprised of multiple subservices.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ROBIN (RuciO/BIgData Express/SENSE) A Next-Generation High-Performance Data Service Platform

Zhang

et al. 2020

2020 IEEE/ACM Innovating the Network for Data-Intensive Science (INDIS)

Self Cite

View full text Add to dashboard Cite

Increasingly, scientific research discovery depends on the analysis of extremely large amounts of data. This data is gathered through very large scientific instruments, very large numbers of small devices, and mid-scale instruments. These devices generate exponentially increasing volumes of data. Such data is often obtained, analyzed, stored, visualized, and transferred at and among sites world-wide by large collaborations of science communities. Managing and moving extremely large volumes of data across today's networks is a special multidimensional challenge. Also, existing data management and movement services and tools often are inadequate to address all specific requirements. This paper describes the various components of this challenge and how those issues can be addressed by ROBIN, a unique comprehensive set of integrated services designed specifically for managing and moving extremely large amounts of data over long distances, e.g., thousands of miles around the globe. This paper also describes the results of initial experiments using that set of integrated services.

show abstract

“…Compared to Internet WANs, which have periodic patterns [3], R-WANs have random traffic spikes that are difficult to understand and anticipate. The traffic on R-WANs depends on which science experiments and devices are running and which groups are involved, and it is characterized by high-variability data transfers lasting minutes or even hours [4].…”

Section: Introductionmentioning

confidence: 99%

Dynamic Graph Neural Network for Traffic Forecasting in Wide Area Networks

Mallick

Kiran

Mohammed

et al. 2020

2020 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Wide area networking infrastructures (WANs), particularly science and research WANs, are the backbone for moving large volumes of scientific data between experimental facilities and data centers. With demands growing at exponential rates, these networks are struggling to cope with large data volumes, real-time responses, and overall network performance. Network operators are increasingly looking for innovative ways to manage the limited underlying network resources. Forecasting network traffic is a critical capability for proactive resource management, congestion mitigation, and dedicated transfer provisioning. To this end, we propose a nonautoregressive graph-based neural network for multistep network traffic forecasting. Specifically, we develop a dynamic variant of diffusion convolutional recurrent neural networks to forecast traffic in research WANs. We evaluate the efficacy of our approach on real traffic from ESnet, the U.S. Department of Energy's dedicated science network. Our results show that compared to classical forecasting methods, our approach explicitly learns the dynamic nature of spatiotemporal traffic patterns, showing significant improvements in forecasting accuracy. Our technique can surpass existing statistical and deep learning approaches by achieving ≈20% mean absolute percentage error for multiple hours of forecasts despite dynamic network traffic settings.

show abstract

BigData Express: Toward Schedulable, Predictable, and High-Performance Data Transfer

Cited by 11 publications

References 22 publications

Network Capabilities for the HL-LHC Era

Network Capabilities for the HL-LHC Era

ROBIN (RuciO/BIgData Express/SENSE) A Next-Generation High-Performance Data Service Platform

Dynamic Graph Neural Network for Traffic Forecasting in Wide Area Networks

Contact Info

Product

Resources

About