Achieving high performance with TCP over 40GbE on NUMA architectures for CMS data acquisition

Bawej, Tomasz; Behrens, U.; Branson, J. G.; Chaze, Olivier; Cittolin, S.; Darlea, Georgiana-Lavinia; Deldicque, Christian; Dobson, M.; Dupont, Aymeric; Erhan, S.; Forrest, A. P. M.; Gigi, D.; Glege, F.; Gómez-Ceballos, G.; Gómez-Reino, R.; Hegeman, J.; Holzner, A.; Masetti, L.; Meijers, F.; Meschi, E.; Mommsen, R. K.; Morovic, S.; Nunez-Barranco-Fernandez, Carlos; O’Dell, V.; Orsini, L.; Paus, C.; Petrucci, Andrea; Pieri, M.; Rácz, A.; Sakulin, H.; Schwick, C.; Stieger, B.; Sumorok, K.; Veverka, J.; Wakefield, Christopher Colin; Zejdl, Petr

doi:10.1109/rtc.2014.7097439

Cited by 2 publications

(2 citation statements)

References 5 publications

(6 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to achieve full throughput at 40 Gb/s, two threads are used for receiving data. CPU affinities of threads, interrupts and memory are fine-tuned manually [12]. Fig.…”

Section: The New Front-end-readout Optical Linkmentioning

confidence: 99%

The New CMS DAQ System for Run-2 of the LHC

Bawej

Behrens

Branson

et al. 2015

IEEE Trans. Nucl. Sci.

Self Cite

View full text Add to dashboard Cite

The data acquisition (DAQ) system of the CMS experiment at the CERN Large Hadron Collider assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GB/s to the high level trigger (HLT) farm. The HLT farm selects interesting events for storage and offline analysis at a rate of around 1 kHz. The DAQ system has been redesigned during the accelerator shutdown in 2013/14. The motivation is twofold: Firstly, the current compute nodes, networking, and storage infrastructure will have reached the end of their lifetime by the time the LHC restarts. Secondly, in order to handle higher LHC luminosities and event pileup, a number of sub-detectors will be upgraded, increasing the number of readout channels and replacing the off-detector readout electronics with a TCA implementation. The new DAQ architecture will take advantage of the latest developments in the computing industry. For data concentration, 10/40 Gb/s Ethernet technologies will be used, as well as an implementation of a reduced TCP/IP in FPGA for a reliable transport between custom electronics and commercial computing hardware. A Clos network based on 56 Gb/s FDR Infiniband has been chosen for the event builder with a throughput of Tb/s. The HLT processing is entirely file based. This allows the DAQ and HLT systems to be independent, and to use the HLT software in the same way as for the offline processing. The fully built events are sent to the HLT with 1/10/40 Gb/s Ethernet via network file systems. Hierarchical collection of HLT accepted events and monitoring meta-data are stored into a global file system. This paper presents the requirements, technical choices, and performance of the new system.

show abstract

“…In order to achieve full throughput at 40 Gb/s, two threads are used for receiving data. CPU affinities of threads, interrupts and memory are fine-tuned manually [12]. Fig.…”

Section: The New Front-end-readout Optical Linkmentioning

confidence: 99%

The New CMS DAQ System for Run-2 of the LHC

Bawej

Behrens

Branson

et al. 2015

IEEE Trans. Nucl. Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In order to achieve full throughput at 40 Gb/s, two threads are used for receiving data. CPU affinities of threads, interrupts and memory are finetuned manually [11]. Fig.…”

Section: Data Concentrationmentioning

confidence: 99%

The new CMS DAQ system for run-2 of the LHC

Bawej

Behrens

Branson

et al. 2014

2014 19th IEEE-NPSS Real Time Conference

Self Cite

View full text Add to dashboard Cite

The data acquisition system (DAQ) of the CMS experiment at the CERN Large Hadron Collider assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GB/s to the high level trigger (HLT) farm. The HLT farm selects interesting events for storage and offline analysis at a rate of around 1 kHz. The DAQ system has been redesigned during the accelerator shutdown in 2013/14. The motivation is twofold Firstly, the current compute nodes, networking, and storage infrastructure will have reached the end of their lifetime by the time the LHC restarts. Secondly, in order to handle higher LHC luminosities and event pileup, a number of sub-detectors will be upgraded, increasing the number of readout channels and replacing the off-detector readout electronics with a micro-TCA implementation. The new DAQ architecture will take advantage of the latest developments in the computing industry. For data concentration, 10/40 Gb/s Ethernet technologies will be used, as well as an implementation of a reduced TCP/IP in FPGA for a reliable transport between custom electronics and commercial computing hardware. A 56 Gb/s Infiniband FDR CLOS network has been chosen for the event builder with a throughput of 4 Tbps. The HLT processing is entirely file based. This allows the DAQ and HLT systems to be independent, and to use the same framework for the HLT as for the offline processing. The fully built events are sent to the HLT with 1/10/40 Gb/s Ethernet via network file systems. Hierarchical collection of HLT accepted events and monitoring meta-data are stored into a global file system. This paper presents the requirements, technical choices, and performance of the new system. The New CMS DAQ System for Run-2 of the LHC Abstract-The data acquisition system (DAQ) of the CMS experiment at the CERN Large Hadron Collider assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GB/s to the high level trigger (HLT) farm. The HLT farm selects interesting events for storage and offline analysis at a rate of around 1 kHz. The DAQ system has been redesigned during the accelerator shutdown in 2013/14. The motivation is twofold: Firstly, the current compute nodes, networking, and storage infrastructure will have reached the end of their lifetime by the time the LHC restarts. Secondly, in order to handle higher LHC luminosities and event pileup, a number of sub-detectors will be upgraded, increasing the number of readout channels and replacing the off-detector readout electronics with a µTCA implementation. The new DAQ architecture will take advantage of the latest developments in the computing industry. For data concentration, 10/40 Gb/s Ethernet technologies will be used, as well as an implementation of a reduced TCP/IP in FPGA for a reliable transport between custom electronics and commercial computing hardware. A 56 Gb/s Infiniband FDR Clos network has been chosen for the event builder with a throughput of ~4 Tb/s. The HLT processing is entirely file based. This allo...

show abstract

Achieving high performance with TCP over 40GbE on NUMA architectures for CMS data acquisition

Cited by 2 publications

References 5 publications

The New CMS DAQ System for Run-2 of the LHC

The New CMS DAQ System for Run-2 of the LHC

The new CMS DAQ system for run-2 of the LHC

Contact Info

Product

Resources

About