Scribd
Upload
590 views11 pages

Elasticsearch, Logstash and Kibana

This document provides an overview of the Elastic Stack, including Elasticsearch, Logstash, and Kibana. Elasticsearch is described as a schema-less, distributed, RESTful search and analytics engine. Logstash is a data processing pipeline that collects, transforms, and forwards data to outputs like Elasticsearch. Kibana provides visualization and exploration of data stored in Elasticsearch. The Elastic Stack works together to collect, parse, store, and visualize log and machine data.

Uploaded by

Sachin Jain
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
590 views11 pages

Elasticsearch, Logstash and Kibana

This document provides an overview of the Elastic Stack, including Elasticsearch, Logstash, and Kibana. Elasticsearch is described as a schema-less, distributed, RESTful search and analytics engine. Logstash is a data processing pipeline that collects, transforms, and forwards data to outputs like Elasticsearch. Kibana provides visualization and exploration of data stored in Elasticsearch. The Elastic Stack works together to collect, parse, store, and visualize log and machine data.

Uploaded by

Sachin Jain
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

The Elastic Stack

Existing ELK Model


Log Analysis
Collective AIM:
▪ Collect, transform, parse, analyze Various Unix System and N/W Logs.

▪ Provide centralized access and search abilities to search Filter and aggregate the log
data.

▪ Related use case: Point in time Critical log searching based on desired Keywords or
string patterns.
Elasticsearch
The Search Engine
▪ Schema-less

▪ Distributed

▪ REST-ful, Document-oriented, and Speaks JSON For searching and analytics

▪ Highly scalable search index server

▪ Based on Apache Lucene

▪ Near real-time search and analysis capabilities

▪ Document-oriented (structured) index entries which can (but must not) be associated with a
schema

▪ and more...
Scalability of Elasticsearch
▪ Elasticsearch can run as one integrated
application on multiple nodes of a cluster.

▪ Indexes are stored in Lucene instances


called “Shards” which can be distributed
over several nodes.

▪ There a two types of “Shards” Primary


Shards & Replica.

▪ Replicas of “Primary Shards” provide Failure


tolerance and therefore protect data Make
queries (search faster) faster
Logstash
The Dataflow Engine
▪ Open source software to collect, transform, filter and forward data (e.g. log data) from input
sources to output sources (e.g. Elasticsearch)

▪ Implemented in JRuby and runs on a JVM (Java Virtual Machine)

▪ Simple message based architecture

▪ Extendable by plugins (e.g. input, output, filter plugins)


Logstash Internal architecture
Logstash has a three-stage pipeline implemented in Jruby as Follows:

Input: The input stage plugins extract data, This can be from logfiles, a TCP or UDP listener,
one of several protocol-specific plugins such as syslog or IRC, or even queuing systems such
as Redis etc.

Filter: A filter plugin performs intermediary processing on an event which transforms and
enrich the data by using various other plugins like grok etc.

Output: The output stage plugins loads the processed events into something else, such
as Elasticsearch or another document-database.
Configuration File

Multiple inputs of different types

Conditionally filter and


transform data; some common
formats are already known

Forward to multiple outputs


Kibana
▪ Web-based application for exploring and visualizing data

▪ Modern Browser-based interface (HTML5 + JavaScript)

▪ Ships with its own web server for easy setup

▪ Seamless integration with Elasticsearch.

▪ Highly configurable/customizable, build panels with user defined charts, tables, etc.

▪ Built on AngularJS
The Main ELK Configuration Files

▪ The Kibana default configuration is stored kibana/config/kibana.yml.


✓ /etc/kibana/kibana.yml

▪ The Logstash configuration is stored in logstash/config/logstash.yml.


✓ /etc/logstash/logstash.yml

▪ The Logstash pipeline configuration is stored in logstash/pipeline/logstash.conf


✓ /etc/logstash/conf.d/syslog.conf

▪ The Elasticsearch configuration is stored in elasticsearch/config/elasticsearch.yml.


✓ /etc/elasticsearch/elasticsearch.yml
Thanks!
Author: Karn Kumar

You might also like