Deployment of a WLCG network monitoring infrastructure based on the perfSONAR-PS technology

Brown, Alexander L.; Bonacorsi, D.; Capone, Valentina; Girolamo, D De; Casaní, Álvaro Fernández; Flix, J.; Forti, A.; Gable, Ian; Gutsche, O.; Hesnaux, A; Liu, S; Muñoz, Fernando L. García-Gil de; Magini, N.; Kee, S. P. Mc; Mohammed, Kashif; Rand, D; Reale, Mario; Roiser, S.; Zieliński, M.; Zurawski, Jason

doi:10.1088/1742-6596/513/6/062008

Cited by 6 publications

(2 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This tool then produces a file that the FileRouter agent can use to supplement its internal statistics, giving it a more complete overview of the network performance. The next step is to integrate it with perfSONAR [10], effectively gaining a more comprehensive picture of what's happening on the network. At the time of writing this paper, perfSONAR is yet to be deployed on all WLCG sites.…”

Section: Integrating External Sources Of Information 31 Sources Of Imentioning

confidence: 99%

Integrating network-awareness and network-management into PhEDEx: first results from the ANSE project

Lapadatescu¹,

Wildish²,

Barczyk³

et al. 2014

Proceedings of International Symposium on Grids and Clouds (ISGC) 2014 — PoS(ISGC2014)

View full text Add to dashboard Cite

ANSE Collaboration †PhEDEx is the data placement and management tool for the CMS experiment at the LHC. It controls the large-scale data-flows on the WAN across the experiment, typically handling 1 PB of data per week. While robust, its architecture is now ten years old and has yet to fully adapt to today's production environment, an environment in which the network is the fastest and most reliable component. The ANSE (Advanced Network Services for Experiments) project, in the context of CMS, aims to greatly improve PhEDEx' network awareness for smart source selection, as well as to integrate bandwidth provisioning capabilities in the data transfer management. Both parts require a good knowledge of the network status, topology and of course, access to useful and up-to-date performance metrics. One of the first steps towards this goal involved the identification of a mechanism for informing PhEDEx about independent network performance metrics. Methods for providing these metrics have been prototyped and verified in a LAN testbed using fake data transfer requests. This mechanism is already directly usable by CMS in their production environment. Currently, the ANSE-PhEDEx testbed is spread over many servers at a number of sites. It is composed of several machines dedicated to PhEDEx site agents, one server holding the PhEDEx central agents, a central database and one server which contains the PhEDEx website and dataservice. Some of the site nodes have additional attached storage nodes. In this paper, we present the work that has been done in ANSE for PhEDEx. This includes performance measurements using the Fast Data Transfer (FDT) tool and the extension of the PhEDEx agent that downloads files to a site to allow it to control the network via creation and use of dynamic circuits. We present the results of our tests using these new features, on highspeed WAN circuits ranging from a few Gbps to 40Gbps and detail the development done within PhEDEx itself. Finally, the paper will also describe the future plans for the project.

show abstract

Section: Integrating External Sources Of Information 31 Sources Of Imentioning

confidence: 99%

Integrating network-awareness and network-management into PhEDEx: first results from the ANSE project

Lapadatescu¹,

Wildish²,

Barczyk³

et al. 2014

Proceedings of International Symposium on Grids and Clouds (ISGC) 2014 — PoS(ISGC2014)

View full text Add to dashboard Cite

show abstract

“…This accountability is an essential conceptual element of our vision as it allows for data analytics to find problems as promises on some routes, involving some network segments, or some sites are systematically not quite met. Independent monitoring traffic via systems like perfSonar [4] are thus augmented by the accounting of the promises to understand the relevant high level performance characteristics.…”

Section: Introductionmentioning

confidence: 99%

Managed Network Services for Exascale Data Movement Across Large Global Scientific Collaborations

Würthwein¹,

Guiang²,

Arora³

et al. 2022

Preprint

View full text Add to dashboard Cite

Unique scientific instruments designed and operated by large global collaborations are expected to produce Exabytescale data volumes per year by 2030. These collaborations depend on globally distributed storage and compute to turn raw data into science. While all of these infrastructures have batch scheduling capabilities to share compute, Research and Education networks lack those capabilities. There is thus uncontrolled competition for bandwidth between and within collaborations. As a result, data "hogs" disk space at processing facilities for much longer than it takes to process, leading to vastly over-provisioned storage infrastructures. Integrated co-scheduling of networks as part of high-level managed workflows might reduce these storage needs by more than an order of magnitude. This paper describes such a solution, demonstrates its functionality in the context of the Large Hadron Collider (LHC) at CERN, and presents the nextsteps towards its use in production.

show abstract

Optimising costs in WLCG operations

Pradillo

Dimou

Flix

et al. 2015

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Deployment of a WLCG network monitoring infrastructure based on the perfSONAR-PS technology

Cited by 6 publications

References 1 publication

Integrating network-awareness and network-management into PhEDEx: first results from the ANSE project

Integrating network-awareness and network-management into PhEDEx: first results from the ANSE project

Managed Network Services for Exascale Data Movement Across Large Global Scientific Collaborations

Optimising costs in WLCG operations

Contact Info

Product

Resources

About