CERN Accelerating science

ATLAS Slides
Report number	ATL-SOFT-SLIDE-2019-778
Title	An Information Aggregation and Analytics System for ATLAS Frontier.
Author(s)	Formica, Andrea (IRFU, CEA, Université Paris-Saclay) ; Ozturk, Nurcan (The University of Texas at Arlington) ; Gallas, Elizabeth (University of Oxford, Particle Physics) ; Vukotic, Ilija (University of Chicago, Enrico Fermi Institute) ; Lozano Bahilo, Jose Julio (Instituto de Fisica Corpuscular (IFIC), Centro Mixto Universidad de Valencia - CSIC) ; Si Amer, Millissa (Ecole nationale Superieure d'Informatique, Algiers)
Corporate author(s)	The ATLAS collaboration
Collaboration	ATLAS Collaboration
Submitted to	24th International Conference on Computing in High Energy and Nuclear Physics, Adelaide, Australia, 4 - 8 Nov 2019
Submitted by	[email protected] on 18 Oct 2019
Subject category	Particle Physics - Experiment
Accelerator/Facility, Experiment	CERN LHC ; ATLAS
Free keywords	ATLAS ; Frontier ; Docker ; Kubernetes ; Analytics ; Elasticsearch ; COOLR
Abstract	ATLAS event processing requires access to centralized database systems where information about calibrations, detector status and data-taking conditions are stored. This processing is done on more than 150 computing sites on a world-wide computing grid which are able to access the database using the squid-Frontier system. Some processing workflows have been found which overload the Frontier system due to the Conditions data model currently in use, specifically because some of the Conditions data requests have been found to have a low caching efficiency. The underlying cause is that non-identical requests as far as the caching are actually retrieving a much smaller number of unique payloads. While ATLAS is undertaking an adiabatic transition during LS2 and Run-3 from the current COOL Conditions data model to a new data model called CREST for Run 4, it is important to identify the problematic Conditions queries with low caching efficiency and work with the detector subsystems to improve the storage of such data within the current data model. For this purpose ATLAS put together an information aggregation and analytics system. The system is based on aggregated data from the squid-Frontier logs using the Elastic Search technology. This talk describes the components of this analytics system from the server based on Flask/Celery application to the user interface and how we use Spark SQL functionalities to filter data for making plots, storing the caching efficiency results into a PostgreSQL database and finally deploying the package via a Docker container.