CERN Accélérateur de science

Published Articles
Report number	FERMILAB-CONF-15-605-CD
Title	AsyncStageOut: Distributed User Data Management for CMS Analysis
Author(s)	Riahi, H (CERN) ; Wildish, T (Princeton U.) ; Ciangottini, D (INFN, Perugia ; Perugia U.) ; Hernández, J M (Madrid, CIEMAT) ; Andreeva, J (CERN) ; Balcas, J (Vilnius U.) ; Karavakis, E (CERN) ; Mascheroni, M (INFN, Milan Bicocca) ; Tanasijczuk, A J (UC, San Diego) ; Vaandering, E W (Fermilab)
Publication	2015
Number of pages	9
In:	J. Phys.: Conf. Ser. 664 (2015) 062052
In:	21st International Conference on Computing in High Energy and Nuclear Physics, Okinawa, Japan, 13 - 17 Apr 2015, pp.062052
DOI	10.1088/1742-6596/664/6/062052
Subject category	Computing and Computers
Accelerator/Facility, Experiment	CERN LHC ; CMS
Abstract	AsyncStageOut (ASO) is a new component of the distributed data analysis system of CMS, CRAB, designed for managing users' data. It addresses a major weakness of the previous model, namely that mass storage of output data was part of the job execution resulting in inefficient use of job slots and an unacceptable failure rate at the end of the jobs. ASO foresees the management of up to 400k files per day of various sizes, spread worldwide across more than 60 sites. It must handle up to 1000 individual users per month, and work with minimal delay. This creates challenging requirements for system scalability, performance and monitoring. ASO uses FTS to schedule and execute the transfers between the storage elements of the source and destination sites. It has evolved from a limited prototype to a highly adaptable service, which manages and monitors the user file placement and bookkeeping. To ensure system scalability and data monitoring, it employs new technologies such as a NoSQL database and re-uses existing components of PhEDEx and the FTS Dashboard. We present the asynchronous stage-out strategy and the architecture of the solution we implemented to deal with those issues and challenges. The deployment model for the high availability and scalability of the service is discussed. The performance of the system during the commissioning and the first phase of production are also shown, along with results from simulations designed to explore the limits of scalability.
Copyright/License	publication: © 2015-2025 The Author(s) (License: CC-BY-3.0)

Corresponding record in: Inspire