The New CMS DAQ System for Run-2 of the LHC

Abstract

The data acquisition (DAQ) system of the CMS experiment at the CERN Large Hadron Collider assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GB/s to the high level trigger (HLT) farm. The HLT farm selects interesting events for storage and offline analysis at a rate of around 1 kHz. The DAQ system has been redesigned during the accelerator shutdown in 2013/14. The motivation is twofold: Firstly, the current compute nodes, networking, and storage infrastructure will have reached the end of their lifetime by the time the LHC restarts. Secondly, in order to handle higher LHC luminosities and event pileup, a number of sub-detectors will be upgraded, increasing the number of readout channels and replacing the off-detector readout electronics with a TCA implementation. The new DAQ architecture will take advantage of the latest developments in the computing industry. For data concentration, 10/40 Gb/s Ethernet technologies will be used, as well as an implementation of a reduced TCP/IP in FPGA for a reliable transport between custom electronics and commercial computing hardware. A Clos network based on 56 Gb/s FDR Infiniband has been chosen for the event builder with a throughput of Tb/s. The HLT processing is entirely file based. This allows the DAQ and HLT systems to be independent, and to use the HLT software in the same way as for the offline processing. The fully built events are sent to the HLT with 1/10/40 Gb/s Ethernet via network file systems. Hierarchical collection of HLT accepted events and monitoring meta-data are stored into a global file system. This paper presents the requirements, technical choices, and performance of the new system.

Development of the CMS detector for the CERN LHC Run 3

“…Prior to Run 2, Infiniband and 40 Gb/s Ethernet became established mainstream standards in high-performance computing (HPC). They were evaluated [244,250] and ultimately adopted [245,251] as technologies of choice for the Run 2 DAQ upgrade. This allowed the system to be scaled down by an order of magnitude and implemented in a single slice, while increasing the overall EVB throughput [252].…”

Towards a container-based architecture for CMS data acquisition

“…Due to the CMS DAQ's utilization of high bandwidth RDMA devices [18][19][20][21], an investigation and prototyping effort was made on RDMA-based device access (RoCE) in Kubernetes, demonstrating successful communication through RDMA devices in study modules 2 and 5.…”

Overview of the CMS detector performance in LHC run II

“…During the long shutdown, lasted from March 2013 up to February 2015, the CMS experiment has undergone major changes involving the detector, the data acquisition and the trigger system. The data acquisition (DAQ) [3] system has been completely redesigned to take the advantage of the latest developments in the computing industry and to cope with the upgraded electronics used for some sub-detectors. In addition, the High Level Trigger (HLT) processing is now file based to allow the DAQ and the HLT systems to be independent.…”