Achieving High Performance With TCP Over 40 GbE on NUMA Architectures for CMS Data Acquisition

Bawej, Tomasz; Behrens, U.; Branson, J. G.; Chaze, Olivier; Cittolin, S.; Darlea, Georgiana-Lavinia; Deldicque, Christian; Dobson, M.; Dupont, Aymeric; Erhan, S.; Forrest, Andrew; Gigi, D.; Glege, F.; Gómez-Ceballos, G.; Gómez-Reino, R.; Hegeman, J.; Holzner, A.; Masetti, L.; Meijers, F.; Meschi, E.; Mommsen, R. K.; Morovic, S.; Nunez-Barranco-Fernandez, Carlos; O’Dell, V.; Orsini, L.; Paus, C.; Petrucci, Andrea; Pieri, M.; Rácz, A.; Sakulin, H.; Schwick, C.; Stieger, B.; Sumorok, K.; Veverka, J.; Wakefield, Christopher Colin; Zejdl, Petr

doi:10.1109/tns.2015.2409898

Cited by 4 publications

(2 citation statements)

References 7 publications

(8 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This was likely due to an increase in message rate that the system needed to sustain, resulting in an additional overhead to handle the control messages. This feature is well known and documented for the CMS use case [6,7].…”

Section: Tests and Resultsmentioning

confidence: 95%

The CMS Orbit Builder for the HL-LHC at CERN

Amoiridis,

Behrens,

Bocci

et al. 2024

EPJ Web of Conf.

Self Cite

View full text Add to dashboard Cite

The Compact Muon Solenoid (CMS) experiment at CERN incorporates one of the highest throughput data acquisition systems in the world and is expected to increase its throughput by more than a factor of ten for High-Luminosity phase of Large Hadron Collider (HL-LHC). To achieve this goal, the system will be upgraded in most of its components. Among them, the event builder software, in charge of assembling all the data read out from the different sub-detectors, is planned to be modified from a single event builder to an orbit builder that assembles multiple events at the same time. The throughput of the event builder will be increased from the current 1.6 Tb/s to 51 Tb/s for the HL-LHC orbit builder. This paper presents preliminary network transfer studies in preparation for the upgrade. The key conceptual characteristics are discussed, concerning differences between the CMS event builder in Run 3 and the CMS Orbit Builder for the HL-LHC. For the feasibility studies, a pipestream benchmark, mimicking event-builder-like traffic has been developed. Preliminary performance tests and results are discussed.

show abstract

Section: Tests and Resultsmentioning

confidence: 95%

The CMS Orbit Builder for the HL-LHC at CERN

Amoiridis,

Behrens,

Bocci

et al. 2024

EPJ Web of Conf.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The XDAQ system builds upon industry standards, open protocols, and libraries, e.g., TCP, HTTP, XML, and Apache Xerces. Notable enhancements to XDAQ include the development of new "peer transport" plug-ins to support new network technologies, such as RDMA using Infiniband Verbs [278,279], and a new service-based approach to the configuration of the built-in monitoring and alarming infrastructure [280]. New XDAQ-based applications have been added to control new types of custom hardware and existing applications, such as the event builder enhanced with features such as load balancing and fault tolerance.…”

Section: Online Cloudmentioning

confidence: 99%

Development of the CMS detector for the CERN LHC Run 3

Hayrapetyan,

Tumasyan,

Adam

et al. 2024

J. Inst.

View full text Add to dashboard Cite

Since the initial data taking of the CERN LHC, the CMS experiment has undergone substantial upgrades and improvements. This paper discusses the CMS detector as it is configured for the third data-taking period of the CERN LHC, Run 3, which started in 2022. The entire silicon pixel tracking detector was replaced. A new powering system for the superconducting solenoid was installed. The electronics of the hadron calorimeter was upgraded. All the muon electronic systems were upgraded, and new muon detector stations were added, including a gas electron multiplier detector. The precision proton spectrometer was upgraded. The dedicated luminosity detectors and the beam loss monitor were refurbished. Substantial improvements to the trigger, data acquisition, software, and computing systems were also implemented, including a new hybrid CPU/GPU farm for the high-level trigger.

show abstract

The new CMS DAQ system for run-2 of the LHC

Bawej

Behrens

Branson

et al. 2014

2014 19th IEEE-NPSS Real Time Conference

Self Cite

View full text Add to dashboard Cite

The data acquisition system (DAQ) of the CMS experiment at the CERN Large Hadron Collider assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GB/s to the high level trigger (HLT) farm. The HLT farm selects interesting events for storage and offline analysis at a rate of around 1 kHz. The DAQ system has been redesigned during the accelerator shutdown in 2013/14. The motivation is twofold Firstly, the current compute nodes, networking, and storage infrastructure will have reached the end of their lifetime by the time the LHC restarts. Secondly, in order to handle higher LHC luminosities and event pileup, a number of sub-detectors will be upgraded, increasing the number of readout channels and replacing the off-detector readout electronics with a micro-TCA implementation. The new DAQ architecture will take advantage of the latest developments in the computing industry. For data concentration, 10/40 Gb/s Ethernet technologies will be used, as well as an implementation of a reduced TCP/IP in FPGA for a reliable transport between custom electronics and commercial computing hardware. A 56 Gb/s Infiniband FDR CLOS network has been chosen for the event builder with a throughput of 4 Tbps. The HLT processing is entirely file based. This allows the DAQ and HLT systems to be independent, and to use the same framework for the HLT as for the offline processing. The fully built events are sent to the HLT with 1/10/40 Gb/s Ethernet via network file systems. Hierarchical collection of HLT accepted events and monitoring meta-data are stored into a global file system. This paper presents the requirements, technical choices, and performance of the new system. The New CMS DAQ System for Run-2 of the LHC Abstract-The data acquisition system (DAQ) of the CMS experiment at the CERN Large Hadron Collider assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GB/s to the high level trigger (HLT) farm. The HLT farm selects interesting events for storage and offline analysis at a rate of around 1 kHz. The DAQ system has been redesigned during the accelerator shutdown in 2013/14. The motivation is twofold: Firstly, the current compute nodes, networking, and storage infrastructure will have reached the end of their lifetime by the time the LHC restarts. Secondly, in order to handle higher LHC luminosities and event pileup, a number of sub-detectors will be upgraded, increasing the number of readout channels and replacing the off-detector readout electronics with a µTCA implementation. The new DAQ architecture will take advantage of the latest developments in the computing industry. For data concentration, 10/40 Gb/s Ethernet technologies will be used, as well as an implementation of a reduced TCP/IP in FPGA for a reliable transport between custom electronics and commercial computing hardware. A 56 Gb/s Infiniband FDR Clos network has been chosen for the event builder with a throughput of ~4 Tb/s. The HLT processing is entirely file based. This allo...

show abstract

Achieving High Performance With TCP Over 40 GbE on NUMA Architectures for CMS Data Acquisition

Cited by 4 publications

References 7 publications

The CMS Orbit Builder for the HL-LHC at CERN

The CMS Orbit Builder for the HL-LHC at CERN

Development of the CMS detector for the CERN LHC Run 3

The new CMS DAQ system for run-2 of the LHC

Contact Info

Product

Resources

About