CERN Accelerating science

Article
Title	Feasibility tests of RoCE v2 for LHCb event building
Author(s)	Krawczyk, Rafał Dominik (CERN) ; Colombo, Tommaso (CERN) ; Neufeld, Niko (CERN) ; Pisani, Flavio (CERN ; U. Bologna (main)) ; Valat, Sébastien (Unlisted, FR)
Publication	2020
Number of pages	7
In:	EPJ Web Conf. 245 (2020) 01011
In:	24th International Conference on Computing in High Energy and Nuclear Physics, Adelaide, Australia, 4 - 8 Nov 2019, pp.01011
DOI	10.1051/epjconf/202024501011
Subject category	Computing and Computers
Accelerator/Facility, Experiment	CERN LHC ; LHCb
Abstract	This paper evaluates the utilization of Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) for the Run 3 LHCb event building at CERN. The acquisition system of the detector will collect partial data from approximately 1000 separate detector streams. The total estimated throughput equals 32 Terabits per second. Full events will be assembled for subsequent processing and data selection in the filtering farm of the online trigger. High-throughput transmissions with up to 90% links utilization will be an essential feature of the system. The data exchange mechanism must support zero-copy transmissions. In this work, the RoCE high-throughput kernel bypass Ethernet protocol is benchmarked as a potential alternative to InfiniBand. A RoCE-based event building network is presented and two implementations are considered. The former variant combined shallow-buffered and deep-buffered switches with enabled flow control. In the latter setup, only deep-buffered devices are used, where operation relied on their memory throughput and capacity. Feasibility tests were conducted with selected Ethernet switches. Memory bandwidth utilization was investigated, in comparison with InfiniBand. Relevant utilization and interoperability issues of RoCE flow control are detailed with lessons learned along the road.
Copyright/License	© 2020-2024 The Authors (License: CC-BY-4.0)

Corresponding record in: Inspire