CERN Accelerating science

Article
Title Data analysis with the DIANA meta-scheduling approach
Author(s) Anjum, A (CCS, Bristol) ; McClatchey, R (CCS, Bristol) ; Willers, I (CERN)
Publication 2008
In: J. Phys.: Conf. Ser. 119 (2008) 072004
In: International Conference on Computing in High Energy and Nuclear Physics, Victoria, Canada, 2 - 7 Sep 2007, pp.072004
DOI 10.1088/1742-6596/119/7/072004
Subject category Computing and Computers
Abstract The concepts, design and evaluation of the Data Intensive and Network Aware (DIANA) meta-scheduling approach for solving the challenges of data analysis being faced by CERN experiments are discussed in this paper. Our results suggest that data analysis can be made robust by employing fault tolerant and decentralized meta-scheduling algorithms supported in our DIANA meta-scheduler. The DIANA meta-scheduler supports data intensive bulk scheduling, is network aware and follows a policy centric meta-scheduling. In this paper, we demonstrate that a decentralized and dynamic meta-scheduling approach is an effective strategy to cope with increasing numbers of users, jobs and datasets. We present 'quality of service' related statistics for physics analysis through the application of a policy centric fair-share scheduling model. The DIANA meta-schedulers create a peer-to-peer hierarchy of schedulers to accomplish resource management that changes with evolving loads and is dynamic and adapts to the volatile nature of the resources.
Copyright/License publication: (License: CC-BY)

Corresponding record in: Inspire


 Element opprettet 2009-05-15, sist endret 2022-08-17



  • Send to ScienceWise.info