CERN Accelerating science

ATLAS Slides
Report number	ATL-SOFT-SLIDE-2024-538
Title	Advancements in the in-file metadata system for the ATLAS experiment
Author(s)	Szymanski, Maciej Pawel (Argonne National Laboratory (US)) ; Van Gemmeren, Peter (Argonne National Laboratory (US)) ; Mete, Alaettin Serhan (Argonne National Laboratory (US)) ; Nowak, Marcin (Brookhaven National Laboratory (US)) ; Krasznahorkay, Attila (University of Massachusetts (US))
Corporate author(s)	The ATLAS collaboration
Collaboration	ATLAS Collaboration
Submitted to	27th International Conference on Computing in High Energy & Nuclear Physics, Kraków, Pl, 19 - 25 Oct 2024
Submitted by	[email protected] on 06 Nov 2024
Subject category	Particle Physics - Experiment
Accelerator/Facility, Experiment	CERN LHC ; ATLAS
Abstract	The High-Luminosity upgrade of the Large Hadron Collider (HL-LHC) will increase luminosity and the number of events by an order of magnitude, demanding more concurrent processing. Event processing is trivially parallel, but metadata handling is more complex and breaks that parallelism. However, correct and reliable in-file metadata is crucial for all workflows of the experiment, enabling tasks such as job configuration, decoding trigger information, and keeping track of event selection. Therefore, ATLAS is enhancing its current in-file metadata system to support metadata creation and propagation in more robust ways. This talk presents developments in the areas of the evolution of storage technology for metadata and the redesign of metadata tools. Firstly, we delve into the investigation of storage technologies tailored for in-file metadata payload, exploring advancements in the ROOT framework, used for storing data collected by the ATLAS experiment. Not only will this work allow ATLAS to utilize modern storage containers (such as RNTuple) for event and metadata, but the goal is to enhance performance and enable seamless handling of metadata. We also discuss whether a challenging process of merging metadata objects could be performed generically, e.g. using RNTuple features. Furthermore, we introduce a novel approach to metadata tools by developing dual-mode functionality. Such tools, offering both creation and propagation capabilities, improve maintainability and facilitate handling in workflows making use of shared I/O functionality. This work also enables investigation of metadata propagation outside of event processing.