-
INDIGO-DataCloud:A data and computing platform to facilitate seamless access to e-infrastructures
Authors:
INDIGO-DataCloud Collaboration,
:,
Davide Salomoni,
Isabel Campos,
Luciano Gaido,
Jesus Marco de Lucas,
Peter Solagna,
Jorge Gomes,
Ludek Matyska,
Patrick Fuhrman,
Marcus Hardt,
Giacinto Donvito,
Lukasz Dutka,
Marcin Plociennik,
Roberto Barbera,
Ignacio Blanquer,
Andrea Ceccanti,
Mario David,
Cristina Duma,
Alvaro López-García,
Germán Moltó,
Pablo Orviz,
Zdenek Sustr,
Matthew Viljoen,
Fernando Aguilar
, et al. (40 additional authors not shown)
Abstract:
This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applicat…
▽ More
This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications.
△ Less
Submitted 5 February, 2019; v1 submitted 6 November, 2017;
originally announced November 2017.
-
Improved Cloud resource allocation: how INDIGO-DataCloud is overcoming the current limitations in Cloud schedulers
Authors:
Alvaro Lopez Garcia,
Lisa Zangrando,
Massimo Sgaravatto,
Vincent Llorens,
Sara Vallero,
Valentina Zaccolo,
Stefano Bagnasco,
Sonia Taneja,
Stefano Dal Pra,
Davide Salomoni,
Giacinto Donvito
Abstract:
Performing efficient resource provisioning is a fundamental aspect for any resource provider. Local Resource Management Systems (LRMS) have been used in data centers for decades in order to obtain the best usage of the resources, providing their fair usage and partitioning for the users. In contrast, current cloud schedulers are normally based on the immediate allocation of resources on a first-co…
▽ More
Performing efficient resource provisioning is a fundamental aspect for any resource provider. Local Resource Management Systems (LRMS) have been used in data centers for decades in order to obtain the best usage of the resources, providing their fair usage and partitioning for the users. In contrast, current cloud schedulers are normally based on the immediate allocation of resources on a first-come, first-served basis, meaning that a request will fail if there are no resources (e.g. OpenStack) or it will be trivially queued ordered by entry time (e.g. OpenNebula). Moreover, these scheduling strategies are based on a static partitioning of the resources, meaning that existing quotas cannot be exceeded, even if there are idle resources allocated to other projects. This is a consequence of the fact that cloud instances are not associated with a maximum execution time and leads to a situation where the resources are under-utilized. These facts have been identified by the INDIGO-DataCloud project as being too simplistic for accommodating scientific workloads in an efficient way, leading to an underutilization of the resources, a non desirable situation in scientific data centers. In this work, we will present the work done in the scheduling area during the first year of the INDIGO project and the foreseen evolutions.
△ Less
Submitted 20 July, 2017;
originally announced July 2017.
-
Run Control and Monitor System for the CMS Experiment
Authors:
M. Bellato,
L. Berti,
V. Brigljevic,
G. Bruno,
E. Cano,
S. Cittolin,
A. Csilling,
S. Erhan,
D. Gigi,
F. Glege,
R. Gomez-Reino,
M. Gulmini,
J. Gutleber,
C. Jacobs,
M. Kozlovszky,
H. Larsen,
I. Magrans,
G. Maron,
F. Meijers,
E. Meschi,
S. Murray,
A. Oh,
L. Orsini,
L. Pollet,
A. Racz
, et al. (8 additional authors not shown)
Abstract:
The Run Control and Monitor System (RCMS) of the CMS experiment is the set of hardware and software components responsible for controlling and monitoring the experiment during data-taking. It provides users with a "virtual counting room", enabling them to operate the experiment and to monitor detector status and data quality from any point in the world. This paper describes the architecture of t…
▽ More
The Run Control and Monitor System (RCMS) of the CMS experiment is the set of hardware and software components responsible for controlling and monitoring the experiment during data-taking. It provides users with a "virtual counting room", enabling them to operate the experiment and to monitor detector status and data quality from any point in the world. This paper describes the architecture of the RCMS with particular emphasis on its scalability through a distributed collection of nodes arranged in a tree-based hierarchy. The current implementation of the architecture in a prototype RCMS used in test beam setups, detector validations and DAQ demonstrators is documented. A discussion of the key technologies used, including Web Services, and the results of tests performed with a 128-node system are presented.
△ Less
Submitted 18 June, 2003;
originally announced June 2003.
