Scribd
Upload
7 views9 pages

Hpcsa Block Logging Slides

The document discusses scalable logging and log-file analysis in high-performance computing, emphasizing the need to collect and analyze log-files rather than metrics. It outlines various types of logging services, including provider-managed, SaaS, self-hosted, and custom solutions, with a focus on the Elastic Stack for implementation. Challenges such as requirements engineering, scalability, and integration with existing systems are also highlighted.

Uploaded by

vmkkolli
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
7 views9 pages

Hpcsa Block Logging Slides

The document discusses scalable logging and log-file analysis in high-performance computing, emphasizing the need to collect and analyze log-files rather than metrics. It outlines various types of logging services, including provider-managed, SaaS, self-hosted, and custom solutions, with a focus on the Elastic Stack for implementation. Challenges such as requirements engineering, scalability, and integration with existing systems are also highlighted.

Uploaded by

vmkkolli
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 9

Scalable logging and log-file analysis

High-Performance Computing System Administration

Linus Weber

Institute of Computer Science Göttingen

February 2023

Linus Weber (Institute of Computer Science Göttingen)


Scalable logging and log-file analysis February 2023 1/9
Log-file analysis

Similar to monitoring (TIG stack), but instead of collecting metrics


(time-series data) it is about collecting and analyzing log-files (or
log-entries).
Store collected files on centralized server.
Variety of possible input sources.
Agents - Collector - Storage - Visualization & Analysis

Linus Weber (Institute of Computer Science Göttingen)


Scalable logging and log-file analysis February 2023 2/9
Scalable logging

Here: scale the level of detail, not the capacity of the infrastructure.
File selection
Log level (debug, info, warning, error)
Filters
Aggregation
Metrics polling interval
Retention period

Linus Weber (Institute of Computer Science Göttingen)


Scalable logging and log-file analysis February 2023 3/9
Why do scalable logging?

Different environments (dev, test, staging, production)


New instance
System update
Traffic peaks
Unexpected failure
Use-cases require custom-tailored scaling solutions.

Linus Weber (Institute of Computer Science Göttingen)


Scalable logging and log-file analysis February 2023 4/9
Types of services

1 Provider-managed
2 Software-as-a-Service: Google Cloud logging, Sumo Logic, Loggly
3 Self-hosted: Elastic Stack, Icinga, Nagios Log Server, Graylog, Splunk
(available as SaaS)
4 Custom solution using applications developed in-house and
components like Redis, Kafka, Elasticsearch, InfluxDB, SaaS, etc.

Linus Weber (Institute of Computer Science Göttingen)


Scalable logging and log-file analysis February 2023 5/9
Elastic Stack

Agent - Beats
Collector - Logstash
Storage - Elasticsearch
Visualization & Analysis - Kibana

Linus Weber (Institute of Computer Science Göttingen)


Scalable logging and log-file analysis February 2023 6/9
Kibana Logs

Figure: View logs in Kibana. Source: https://www.elastic.co/de/kibana/

Linus Weber (Institute of Computer Science Göttingen)


Scalable logging and log-file analysis February 2023 7/9
Kibana Charts

Figure: Various chart types in Kibana. Source:


https://www.elastic.co/de/kibana/

Linus Weber (Institute of Computer Science Göttingen)


Scalable logging and log-file analysis February 2023 8/9
Challenges

Requirements engineering
Make it scale
Integrate with existing systems

Linus Weber (Institute of Computer Science Göttingen)


Scalable logging and log-file analysis February 2023 9/9

You might also like