CERN Accelerating science

ATLAS Note
Report number	ATL-SOFT-PROC-2020-032
Title	Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment
Author(s)	Nilsson, Paul (Brookhaven National Laboratory (BNL)) ; Anisenkov, Alexey (Budker Institute of Nuclear Physics and Novosibirsk State University, Siberian Branch of Russian Academy of Sciences) ; Benjamin, Douglas (Argonne National Laboratory) ; Guan, Wen (Department of Physics, University of Wisconsin) ; Javurek, Tomas (European Laboratory for Particle Physics, CERN) ; Oleynik, Danila (Joint Institute for Nuclear Research)
Corporate Author(s)	The ATLAS collaboration
Collaboration	ATLAS Collaboration
Publication	2020
Imprint	23 Mar 2020
Number of pages	5
In:	EPJ Web Conf. 245 (2020) 03025
In:	24th International Conference on Computing in High Energy and Nuclear Physics, Adelaide, Australia, 4 - 8 Nov 2019, pp.03025
DOI	10.1051/epjconf/202024503025
Subject category	Particle Physics - Experiment
Accelerator/Facility, Experiment	CERN LHC ; ATLAS
Free keywords	PanDA ; Pilot ; Grid ; HPC ; Supercomputers ; ATLAS ; LHC
Abstract	The unprecedented computing resource needs of the ATLAS experiment have motivated the Collaboration to become a leader in exploiting High Performance Computers (HPCs). To meet the requirements of HPCs, the PanDA system has been equipped with two new components; Pilot 2 and Harvester, that were designed with HPCs in mind. While Harvester is a resource-facing service which provides resource provisioning and workload shaping, Pilot 2 is responsible for payload execution on the resource. The presentation focuses on Pilot 2, which is a complete rewrite of the original PanDA Pilot used by ATLAS and other experiments for well over a decade. Pilot 2 has a flexible and adaptive design that allows for plugins to be defined with streamlined workflows. In particular, it has plugins for specific hardware infrastructures (HPC/GPU clusters) as well as for dedicated workflows defined by the needs of an experiment. Examples of dedicated HPC workflows are discussed in which the Pilot either uses an MPI application for processing fine-grained event level service under the control of the Harvester service or acts like an MPI application itself and runs a set of job in an assemble. In addition to describing the technical details of these workflows, results are shown from its deployment on Titan (OLCF) and other HPCs in ATLAS.
Copyright/License	© 2020-2025 The Authors (License: CC-BY-4.0)

Corresponding record in: Inspire