Scribd
Upload
425 views4 pages

Introduction To Logstash

Introduction to Logstash use. Logstash is a tool for extracting data and import to Elasticsearch index.

Uploaded by

Pablo Ernesto Vigneaux Wilton
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
425 views4 pages

Introduction To Logstash

Introduction to Logstash use. Logstash is a tool for extracting data and import to Elasticsearch index.

Uploaded by

Pablo Ernesto Vigneaux Wilton
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

01/06/2016

IntroductiontoLogstash

IntroductiontoLogstash
TableofContents
WhatisLogstash?
Downloading/InstallingLogstash
RunningLogstash
Agent
Web
Logstasharchitecture
Simplestconfiguration
Changingthewaydataisrepresented
Readinginputfromfilesondisk
OutputingtoanembeddedElasticsearch
OutputingtoaseparateElasticsearch
Addingafilterintothemix
The"grok"filter
Combiningeverythingtogether
TheUIforLogstash

WhatisLogstash?
Agenericconceptforreceivingdata,transformingit,andoutputtingit.

Downloading/InstallingLogstash
Downloadingisaseasyasgettingitfromhttp://logstash.net,butI'llbeusingversion1.4(whichiscurrentlybetaas
ofthiswriting)
Toinstall,untarthepackagesomewhere,orusethe.deb/.rpmrepositoriesforyourrespectiveoperatingsystem.

RunningLogstash
Thereare2mainmodesofrunningLogstash.Notethatbothcanberunatonce.

Agent
Runningasanagentcollectsinformation,forwardingittothebackend(inourcase,Elasticsearch)

Web
RunsthewebUI(knownasKibana)bundledinLogstash

Logstasharchitecture
Logstashisacollectionof:
Inputs
Codecs
Filters
Outputs

Simplestconfiguration
Startingwiththesimplestinput,standardin:
input{
stdin{}
}

Andthesimplestoutput,standardout(nofiltersfornow):
output{
stdout{}
}

Torunthis,youcando:
bin/logstashagentflogstashsimple.conf

Changingthewaydataisrepresented
Let'schangethecodec(datarepresentation)toprintmoreinformation:
input{
stdin{}
}

http://writequit.org/articles/logstashintro.html

1/4

01/06/2016

IntroductiontoLogstash
output{
stdout{
codec=>rubydebug
}
}

Readinginputfromfilesondisk
Thistime,insteadofreadinginfromstdin,readfromafile:
input{
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
}

OutputingtoanembeddedElasticsearch
Logstashcanoutputtomanymoreplacesthanjuststdout,itcomeswithelasticsearchasanoutputoptionthatcanrun
embedded:
input{
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
elasticsearch{
embedded=>true
}
}

Addafewlogstothefile:
echo"thisisalogmessageaboutfoo">>example.log
echo"thisisalogmessageaboutbar">>example.log
echo"thisisalogmessageaboutbaz">>example.log

Logstashcreatesanindex,noticethatitcreateditforthedaythiswasrun.Logstashwillcreatedailyindicesby
default:
curl'localhost:9200/_cat/health?v'
echo""
curl'localhost:9200/_cat/shards?v'

epochtimestampclusterstatusnode.totalnode.datashardsprireloinitunassign
139504637202:52:52elasticsearchyellow2155005

indexshardprirepstatedocsstoreipnode
logstash2014.03.172pSTARTED099b172.22.255.231MultipleMan
logstash2014.03.172rUNASSIGNED
logstash2014.03.170pSTARTED099b172.22.255.231MultipleMan
logstash2014.03.170rUNASSIGNED
logstash2014.03.173pSTARTED24.2kb172.22.255.231MultipleMan
logstash2014.03.173rUNASSIGNED
logstash2014.03.171pSTARTED13.9kb172.22.255.231MultipleMan
logstash2014.03.171rUNASSIGNED
logstash2014.03.174pSTARTED099b172.22.255.231MultipleMan
logstash2014.03.174rUNASSIGNED

Andyoucansearchforlogmessages(here'sanexamplequery)
{
"query":{
"simple_query_string":{
"query":"foo|bar",
"fields":["message"]
}
},
"size":3
}

Andgetbacktheresults:
HTTP/1.1200OK
ContentType:application/json;charset=UTF8
ContentLength:1248

http://writequit.org/articles/logstashintro.html

2/4

01/06/2016

IntroductiontoLogstash

{
"took":76,
"timed_out":false,
"_shards":{
"total":20,
"successful":20,
"failed":0
},
"hits":{
"total":4,
"max_score":0.35355338,
"hits":[{
"_index":"logstash2014.03.17",
"_type":"apache",
"_id":"q8EqCk2RjWwB70rxz7bw",
"_score":0.35355338,"_source":{"message":"thisisalogmessageaboutfoo","@version":"1","@timestamp":"20140317T08:52:
},{
"_index":"logstash2014.03.17",
"_type":"apache",
"_id":"e0KXf2eCQjmm302UB6n60g",
"_score":0.35355338,"_source":{"message":"thisisalogmessageaboutbar","@version":"1","@timestamp":"20140317T08:52:
},{
"_index":"logstash2014.03.13",
"_type":"apache",
"_id":"DXwFHMvTTsauxjr9lJ5Xcg",
"_score":0.25427115,"_source":{"message":"thisisalogmessageaboutbar","@version":"1","@timestamp":"20140313T08:45:
}]
}
}

OutputingtoaseparateElasticsearch
Embeddedisgreatfordevelopment,butoutputtingtoadifferentElasticsearchserverisbetterforproduction:
input{
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
elasticsearch{
host=>"localhost"
port=>9300
node_name=>"logstashagent007"
workers=>2
}
}

Addingafilterintothemix
Filtersallowyoutomodifyoutput
Themostusefulisgrok,butlet'sstartwithmutate.Sothestandardinput/outputconfigurationfirst:
input{
stdin{}
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
}

Addingthemutatefiltertoaddafieldaswellaslowercasethe"message"field
filter{
mutate{
add_field=>["myhost","Hellofrom%{host}!"]
lowercase=>["message"]
}
}

The"grok"filter
Logstash'sarguablymostusefulfilter.~120differentpatternsthatcanbecomibned.
https://github.com/elasticsearch/logstash/tree/1.4.x/patterns
https://grokdebug.herokuapp.com/
Again,standardboilerplate:
input{

http://writequit.org/articles/logstashintro.html

3/4

01/06/2016

IntroductiontoLogstash
stdin{}
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
}

Andthenagrokfiltermeanttomatchthetext"name:John"intheinputs:
filter{
grok{
match=>["message","name:%{WORD:custom_name}"]
}
mutate{
lowercase=>["custom_name"]
}
}

Combiningeverythingtogether
Readfromafile(thistimeanElasticsearchlogfile),usetheeslogtypewhenputtingthelogmessageinto
Elasticsearch.OutputwillbewrittentoaseparateElasticsearchclusteratlocalhostonport9300:
input{
file{
type=>"eslog"
path=>"/Users/hinmanm/introtologstash/es/logs/elasticsearch.log"
}
}
output{
stdout{
codec=>rubydebug
}
elasticsearch{
host=>"localhost"
port=>9300
}
}

ThisexamplefilterwillmatchElasticsearch'slogformat,extracttheusefulpiecesofthelog(time,level,package,
node_name,andlogmessage).
Themutatefilterwillthen:
lowercasetheloglevel(INFO=>info)
stripthewhitespaceforthepackage("indices.recovery"=>"indices.recovery")
Additionally,themultilinefilterwillmatchlinesthatlooklikeaJavaException,andcollapsethemintoasingle
messagefromthepreviousline.
filter{
grok{
match=>["message",
"^\[%{TIMESTAMP_ISO8601:time}\]\[%{LOGLEVEL:level}.*\]\[%{DATA:package}\]\[%{DATA:node_name}\]%{DATA:logmsg}$"
}

mutate{
lowercase=>["level"]
strip=>["package"]
}

multiline{
pattern=>"(org\.elasticsearch\.Exception.+|(at.+))"
what=>"previous"
}
}

TheUIforLogstash
LogstashbundlesKibana,whichcanbeusedforvisualizingdata,andisaseasyasrunning:
bin/logstashweb

orbothatoncewith:
bin/logstashagentflogstash.confweb

Author:LeeHinman
Created:20140319Wed01:07
Emacs24.3.50.1(Orgmode8.2.5h)
Validate

http://writequit.org/articles/logstashintro.html

4/4

You might also like