The NOTED software tool-set improves efficient network utilization for Rucio data transfers via FTS

Busse-Grawitz, Coralie; Martelli, Edoardo; Lassnig, M.; Manzi, Andrea; Keeble, Oliver; Cass, Tony

doi:10.1051/epjconf/202024507022

Cited by 3 publications

(4 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are multiple ongoing projects focused on these types of network resource automation and integrated orchestration technologies. The NOTED [85] project is building workflow specific network use optimization tools. The SENSE [97] project has developed a multi-resource multi-domain orchestration system.…”

Section: Resource Orchestration and Automationmentioning

confidence: 99%

Snowmass 2021 Computational Frontier CompF4 Topical Group Report Storage and Processing Resource Access

Bhimji

Carder

Dart

et al. 2023

Comput Softw Big Sci

View full text Add to dashboard Cite

show abstract

Section: Resource Orchestration and Automationmentioning

confidence: 99%

Snowmass 2021 Computational Frontier CompF4 Topical Group Report Storage and Processing Resource Access

Bhimji

Carder

Dart

et al. 2023

Comput Softw Big Sci

View full text Add to dashboard Cite

show abstract

“…The first implementation of NOTED estimated the impact of future data transfers by checking files in the FTS 'SUBMITTED' state [1]. Multiple tests have showed that this approach to the problem wasn't good enough to correctly detect large transfers.…”

Section: The Transfer Brokermentioning

confidence: 99%

“…The goal of the NOTED (Network Optimized Transfer of Experimental Data) project is to predict the arrival and duration of large data transfers and, if and where relevant, to suggest appropriate changes to optimise the network configuration and so reduce transfer duration. One possible network optimisation is to load-balance traffic between the main path and an alternative such as a backup path or an unused connection [1]. Alternatively, a larger capacity link provided by a dynamic provisioning service could be activated.…”

Section: Introductionmentioning

confidence: 99%

NOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services

et al. 2021

Self Cite

View full text Add to dashboard Cite

Network traffic optimisation is difficult as the load is by nature dynamic and seemingly unpredictable. However, the increased usage of file transfer services may help the detection of future loads and the prediction of their expected duration. The NOTED project seeks to do exactly this and to dynamically adapt network topology to deliver improved bandwidth for users of such services. This article introduces, and explains the features of, the two main components of NOTED, the Transfer Broker and the Network Intelligence component. The Transfer Broker analyses all queued and on-going FTS transfers, producing a traffic report which can be used by network controllers. Based on this report and its knowledge of the network topology and routing, the Network Intelligence (NI) component makes decisions as to when a network reconfiguration could be beneficial. Any Software Defined Network controller can then apply these decision to the network, so optimising transfer execution time and reducing operating costs.

show abstract

“…The project was introduced in 2020 and presented at several international conferences. First achievements and outcomes are documented in articles [2,3]. Furthermore, a comprehensive study on traffic forecasting has been conducted using a machine learning approach with Long Short Term Memory (LSTM) neural networks, as detailed in [4].…”

Section: Introductionmentioning

confidence: 99%

NOTED: An intelligent network controller to improve the throughput of large data transfers in File Transfer Services by handling dynamic circuits

Misa Moreira,

Martelli,

Cass

2024

EPJ Web of Conf.

Self Cite

View full text Add to dashboard Cite

The NOTED (Network Optimised Transfer of Experimental Data) project has successfully demonstrated the ability to dynamically provision network links to increase the effective bandwidth available for FTS-driven transfers between endpoints, such as WLCG sites, by inspecting on-going data transfers and so identifying those that are bandwidth-limited for a long period of time. Recently, the architecture of NOTED has been improved and the software has been packaged for easy distribution. These improved capabilities and features of NOTED have been tested and demonstrated at various international conferences. For example, during demonstrations at Supercomputing 2022, independent instances of NOTED at CHCERN (Switzerland) and DE-KIT (Germany) monitored large data transfers generated by the ATLAS experiment between these sites and CA-TRIUMF (Canada). We report here on this and other events, highlighting how NOTED can predict link congestion or a notable increase in the network utilisation over an extended period of time and, where appropriate, automatically reconfigure network topology to introduce an additional or an alternative and betterperforming path by using dynamic circuit provisioning systems such as SENSE and AutoGOLE.

show abstract

The NOTED software tool-set improves efficient network utilization for Rucio data transfers via FTS

Cited by 3 publications

References 1 publication

Snowmass 2021 Computational Frontier CompF4 Topical Group Report Storage and Processing Resource Access

Snowmass 2021 Computational Frontier CompF4 Topical Group Report Storage and Processing Resource Access

NOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services

NOTED: An intelligent network controller to improve the throughput of large data transfers in File Transfer Services by handling dynamic circuits

Contact Info

Product

Resources

About