Report number
| ATL-SOFT-SLIDE-2022-001 |
Title
| intelligent Data Delivery Service (iDDS) |
Author(s)
| Guan, Wen (University of Wisconsin Madison (US)) ; Maeno, Tadashi (Brookhaven National Laboratory (US)) ; Padolski, Siarhei (Brookhaven National Laboratory (US)) ; Bockelman, Brian Paul (University of Wisconsin Madison (US)) ; Wenaus, Torre (Brookhaven National Laboratory (US)) ; Barreiro Megino, Fernando Harald (University of Texas at Arlington (US)) ; Lin, Fa-Hui (University of Texas at Arlington (US)) ; Zhang, Rui (University of Wisconsin Madison (US)) |
Corporate author(s)
| The ATLAS collaboration |
Collaboration
| ATLAS Collaboration |
Submitted to
| 30th International Symposium on Lepton Photon Interactions at High Energies, Manchester, Gb, 10 - 14 Jan 2022 |
Submitted by
| wen.guan@cern.ch on 07 Jan 2022 |
Subject category
| Particle Physics - Experiment |
Accelerator/Facility, Experiment
| CERN LHC ; ATLAS |
Abstract
| The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service that has been deployed to serve data carousel, hyperparameter optimization, multiple-steps DAG (Directed Acyclic Graphs) workflows and so on. Here we will present the motivation for iDDS, the architecture, use cases and current status for ATLAS and Rubin Observatory exercise, and plans for the future. |