10 Gbps TCP/IP streams from the FPGA for High Energy Physics

Bauer, G.; Bawej, Tomasz; Behrens, U.; Branson, J. G.; Chaze, Olivier; Cittolin, S.; Coarasa, J. A.; Darlea, Georgiana-Lavinia; Deldicque, Christian; Dobson, M.; Dupont, Aymeric; Erhan, S.; Gigi, D.; Glege, F.; Gómez-Ceballos, G.; Gómez-Reino, R.; Hartl, C.; Hegeman, J.; Holzner, A.; Masetti, L.; Meijers, F.; Meschi, E.; Mommsen, R. K.; Morovic, S.; Nunez-Barranco-Fernandez, Carlos; O’Dell, V.; Orsini, L.; Ozga, Wojciech; Paus, C.; Petrucci, Andrea; Pieri, M.; Rácz, A.; Raginel, Olivier; Sakulin, H.; Sani, M.; Schwick, C.; Spataru, Andrei Cristian; Stieger, B.; Sumorok, K.; Veverka, J.; Wakefield, Christopher Colin; Zejdl, Petr

doi:10.1088/1742-6596/513/1/012042

Cited by 14 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The choice of the exact protocol and network technology is still under evaluation. The CMS DAQ group has experience with TCP/IP implementations in various FPGA families, and at multiple speeds [8,9]. The baseline D2S technology at the time of writing is 100 Gbit/s Ethernet (four 25 Gbit/s lanes) using 100GBASE-CWDM4 interfaces to handle the approximately 250 m cable length to the computing center on the surface.…”

Section: Randd On Input and Output Technologiesmentioning

confidence: 99%

Design and development of the DAQ and Timing Hub for CMS Phase-2

Hegeman¹,

André²,

Behrens³

et al. 2019

Proceedings of Topical Workshop on Electronics for Particle Physics — PoS(TWEPP2018)

Self Cite

View full text Add to dashboard Cite

The CMS detector will undergo a major upgrade for Phase-2 of the LHC program, starting around 2026. The upgraded Level-1 hardware trigger will select events at a rate of 750 kHz. At an expected event size of 7.4 MB this corresponds to a data rate of up to 50 Tbit/s. Optical links will carry the signals from on-detector front-end electronics to back-end electronics in ATCA crates in the service cavern. A DAQ and Timing Hub board aggregates data streams from back-end boards over point-to-point links, provides buffering and transmits the data to the commercial data-to-surface network for processing and storage. This hub board is also responsible for the distribution of timing, control and trigger signals to the back-ends. This paper presents the current development towards the DAQ and Timing Hub and the design of the first prototype, to be used as for validation and integration with the first back-end prototypes in

show abstract

Section: Randd On Input and Output Technologiesmentioning

confidence: 99%

Design and development of the DAQ and Timing Hub for CMS Phase-2

Hegeman¹,

André²,

Behrens³

et al. 2019

Proceedings of Topical Workshop on Electronics for Particle Physics — PoS(TWEPP2018)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The full implementation of the standard TCP/ IP protocol consumes a lot of resources in the FPGA. Therefore other non-standard or simplified protocols are used [31,32]. The network interface may also be used to control the FPGA part, for example via the IPbus [33] protocol.…”

Section: Network Interfaces For Fpga-es Communicationmentioning

confidence: 99%

FPGA and Embedded Systems Based Fast Data Acquisition and Processing for GEM Detectors

et al. 2018

View full text Add to dashboard Cite

Gas electron multiplier (GEM) detectors (Sauli in Nucl Instrum Methods Phys Res A 805:2-24, 2016. https://doi.org/10. 1016/j.nima.2015.07.060 (special issue in memory of Glenn F. Knoll); Buzulutskov in Instrum Exp Tech 50(3):287-310, 2007. https://doi.org/10.1134/S0020441207030013) are widely used for detection of ionizing radiation. When used in the proportional mode, they provide information about time, location, and energy of a detected particle (Chernyshova et al. in Fusion Eng Design, 2017. https://doi.org/10.1016/j.fusengdes.2017.03.107; Altunbas et al. in Nucl Instrum Methods Phys Res A 490(1-2):177-203, 2002. https://doi.org/10.1016/S0168-9002(02)00910-5. http://linkinghub.elsevier.com/retrieve/ pii/S0168900202009105). Modern technologies allow full utilization of detector properties, by acquiring the waveform of output current pulses and processing them using sophisticated digital signal processing (DSP) algorithms. The current pulses must be digitized at high speed (up to 125 MHz) with high resolution (up to 12-bits). Due to the high volume of the produced data, it is necessary to provide the high-performance data acquisition system (DAQ) to transmit the data to processing units. Efficient processing of the GEM data requires distributed parallel processing system to perform multiple tasks (Czarski et al. in Rev Sci Instrum 87(11), 11E336, 2016. https://doi.org/10.1063/1.4961559): (1) Filter out the background and transmit only hit related data. (2) Extract the parameters of a hit, describing the time and charge (related to energy). (3) Estimate the hit position by combining information from multiple anode pads. (4) In case of 2D GEM detectors, correlate pulses received from X and Y pads (pixels) or W, U and V pads (pixels). (5) Separate the hits overlapping in space or in time (if possible) to support detector operation at higher rates. The above functionalities may be achieved in different hardware architectures. The typical hardware platforms include FPGA chips, standard or embedded computer systems with different computation accelerators (Wojenski et al. in J Instrum 11(11):C11035, 2016. http://stacks. iop.org/1748-0221/11/i=11/a=C11035; Nowak et al. in J Phys Conf Ser 513(5):052-024, 2014. https://doi.org/10.1088/ 1742-6596/513/5/052024. http://stacks.iop.org/1742-6596/513/i=5/a=052024?key=crossref.c5912cfa72c30b309821e14c43 84948f. The paper shows possible solutions with their feasibility for particular applications.

show abstract

“…This has been achieved though a simplification of the TCP/IP protocol for unidirectional use, which reduces the number of states from 11 to 3 [10]. TCP/IP streams (one per input) are sent via a commercial 10/40 Gb/s Ethernet switch to a commercial 40 Gb/s Ethernet NIC in the Readout Unit (RU) PC where they can be received with the standard Linux TCP/IP stack.…”

Section: The New Front-end-readout Optical Linkmentioning

confidence: 99%

The New CMS DAQ System for Run-2 of the LHC

Bawej

Behrens

Branson

et al. 2015

IEEE Trans. Nucl. Sci.

Self Cite

View full text Add to dashboard Cite

The data acquisition (DAQ) system of the CMS experiment at the CERN Large Hadron Collider assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GB/s to the high level trigger (HLT) farm. The HLT farm selects interesting events for storage and offline analysis at a rate of around 1 kHz. The DAQ system has been redesigned during the accelerator shutdown in 2013/14. The motivation is twofold: Firstly, the current compute nodes, networking, and storage infrastructure will have reached the end of their lifetime by the time the LHC restarts. Secondly, in order to handle higher LHC luminosities and event pileup, a number of sub-detectors will be upgraded, increasing the number of readout channels and replacing the off-detector readout electronics with a TCA implementation. The new DAQ architecture will take advantage of the latest developments in the computing industry. For data concentration, 10/40 Gb/s Ethernet technologies will be used, as well as an implementation of a reduced TCP/IP in FPGA for a reliable transport between custom electronics and commercial computing hardware. A Clos network based on 56 Gb/s FDR Infiniband has been chosen for the event builder with a throughput of Tb/s. The HLT processing is entirely file based. This allows the DAQ and HLT systems to be independent, and to use the HLT software in the same way as for the offline processing. The fully built events are sent to the HLT with 1/10/40 Gb/s Ethernet via network file systems. Hierarchical collection of HLT accepted events and monitoring meta-data are stored into a global file system. This paper presents the requirements, technical choices, and performance of the new system.

show abstract

10 Gbps TCP/IP streams from the FPGA for High Energy Physics

Cited by 14 publications

References 0 publications

Design and development of the DAQ and Timing Hub for CMS Phase-2

Design and development of the DAQ and Timing Hub for CMS Phase-2

FPGA and Embedded Systems Based Fast Data Acquisition and Processing for GEM Detectors

The New CMS DAQ System for Run-2 of the LHC

Contact Info

Product

Resources

About