forked from elastic/built-docs
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathadvanced-pipeline.html
899 lines (879 loc) · 40.8 KB
/
advanced-pipeline.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
<!DOCTYPE html>
<html lang="en-us">
<head>
<meta charset="UTF-8">
<title>Parsing Logs with Logstash | Logstash Reference [6.7] | Elastic</title>
<meta class="elastic" name="content" content="Parsing Logs with Logstash | Logstash Reference [6.7]">
<link rel="home" href="index.html" title="Logstash Reference [6.7]"/>
<link rel="up" href="getting-started-with-logstash.html" title="Getting Started with Logstash"/>
<link rel="prev" href="first-event.html" title="Stashing Your First Event"/>
<link rel="next" href="multiple-input-output-plugins.html" title="Stitching Together Multiple Input and Output Plugins"/>
<meta class="elastic" name="product_version" content="6.7"/>
<meta class="elastic" name="product_name" content="Logstash"/>
<meta class="elastic" name="website_area" content="documentation"/>
<meta name="DC.type" content="Learn/Docs/Logstash/Reference/6.7"/>
<meta name="DC.subject" content="Logstash"/>
<meta name="DC.identifier" content="6.7"/>
<meta name="robots" content="noindex,nofollow"/>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<script src="https://cdn.optimizely.com/js/18132920325.js"></script>
<link rel="apple-touch-icon" sizes="57x57" href="/apple-icon-57x57.png">
<link rel="apple-touch-icon" sizes="60x60" href="/apple-icon-60x60.png">
<link rel="apple-touch-icon" sizes="72x72" href="/apple-icon-72x72.png">
<link rel="apple-touch-icon" sizes="76x76" href="/apple-icon-76x76.png">
<link rel="apple-touch-icon" sizes="114x114" href="/apple-icon-114x114.png">
<link rel="apple-touch-icon" sizes="120x120" href="/apple-icon-120x120.png">
<link rel="apple-touch-icon" sizes="144x144" href="/apple-icon-144x144.png">
<link rel="apple-touch-icon" sizes="152x152" href="/apple-icon-152x152.png">
<link rel="apple-touch-icon" sizes="180x180" href="/apple-icon-180x180.png">
<link rel="icon" type="image/png" href="/favicon-32x32.png" sizes="32x32">
<link rel="icon" type="image/png" href="/android-chrome-192x192.png" sizes="192x192">
<link rel="icon" type="image/png" href="/favicon-96x96.png" sizes="96x96">
<link rel="icon" type="image/png" href="/favicon-16x16.png" sizes="16x16">
<link rel="manifest" href="/manifest.json">
<meta name="apple-mobile-web-app-title" content="Elastic">
<meta name="application-name" content="Elastic">
<meta name="msapplication-TileColor" content="#ffffff">
<meta name="msapplication-TileImage" content="/mstile-144x144.png">
<meta name="theme-color" content="#ffffff">
<meta name="naver-site-verification" content="936882c1853b701b3cef3721758d80535413dbfd" />
<meta name="yandex-verification" content="d8a47e95d0972434" />
<meta name="localized" content="true" />
<meta name="st:robots" content="follow,index" />
<meta property="og:image" content="https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt280217a63b82a734/6202d3378b1f312528798412/elastic-logo.svg" />
<meta property="og:image:width" content="500" />
<meta property="og:image:height" content="172" />
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
<link rel="icon" href="/favicon.ico" type="image/x-icon">
<link rel="apple-touch-icon-precomposed" sizes="64x64" href="/favicon_64x64_16bit.png">
<link rel="apple-touch-icon-precomposed" sizes="32x32" href="/favicon_32x32.png">
<link rel="apple-touch-icon-precomposed" sizes="16x16" href="/favicon_16x16.png">
<!-- Give IE8 a fighting chance -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
<link rel="stylesheet" type="text/css" href="/guide/static/styles.css" />
</head>
<!--© 2015-2022 Elasticsearch B.V. -->
<!-- All Elastic documentation is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. -->
<!-- http://creativecommons.org/licenses/by-nc-nd/4.0/ -->
<body>
<!-- Google Tag Manager -->
<script>dataLayer = [];</script><noscript><iframe src="//www.googletagmanager.com/ns.html?id=GTM-58RLH5" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= '//www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-58RLH5');</script>
<!-- End Google Tag Manager -->
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-12395217-16"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-12395217-16');
</script>
<!-- Google Tag Manager for GA4 -->
<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-KNJMG2M');</script>
<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-KNJMG2M" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
<!-- End Google Tag Manager for GA4-->
<div id='elastic-nav' style="display:none;"></div>
<script src='https://www.elastic.co/elastic-nav.js'></script>
<div class="main-container">
<section id="content" >
<div class="content-wrapper">
<section id="guide" lang="en">
<div class="container-fluid">
<div class="row pb-3">
<div class="col-12 order-2 col-md-4 order-md-1 col-lg-3 h-almost-full-md sticky-top-md" id="left_col">
<!-- The TOC is appended here -->
</div>
<div class="col-12 order-1 col-md-8 order-md-2 col-lg-7 order-lg-2 guide-section" id="middle_col">
<!-- start body -->
<div class="page_header">
<strong>IMPORTANT</strong>: No additional bug fixes or documentation updates
will be released for this version. For the latest information, see the
<a href="../current/index.html">current release documentation</a>.
</div>
<div id="content">
<div class="breadcrumbs">
<span class="breadcrumb-link"><a href="/guide/">Elastic Docs</a></span>
<span class="chevron-right">›</span><span class="breadcrumb-link"><a href="index.html">Logstash Reference [6.7]</a></span>
<span class="chevron-right">›</span><span class="breadcrumb-link"><a href="getting-started-with-logstash.html">Getting Started with Logstash</a></span>
</div>
<div class="navheader">
<span class="prev">
<a href="first-event.html">« Stashing Your First Event</a>
</span>
<span class="next">
<a href="multiple-input-output-plugins.html">Stitching Together Multiple Input and Output Plugins »</a>
</span>
</div>
<div class="section">
<div class="titlepage"><div><div>
<h2 class="title"><a id="advanced-pipeline"></a>Parsing Logs with Logstash<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/logstash/edit/6.7/docs/static/advanced-pipeline.asciidoc">edit</a></h2>
</div></div></div>
<p>In <a class="xref" href="first-event.html" title="Stashing Your First Event">Stashing Your First Event</a>, you created a basic Logstash pipeline to test your Logstash setup. In the real world, a Logstash
pipeline is a bit more complex: it typically has one or more input, filter, and output plugins.</p>
<p>In this section, you create a Logstash pipeline that uses Filebeat to take Apache web logs as input, parses those
logs to create specific, named fields from the logs, and writes the parsed data to an Elasticsearch cluster. Rather than
defining the pipeline configuration at the command line, you’ll define the pipeline in a config file.</p>
<p>To get started, go <a href="https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz" class="ulink" target="_top">here</a> to
download the sample data set used in this example. Unpack the file.</p>
<div class="section">
<div class="titlepage"><div><div>
<h3 class="title"><a id="configuring-filebeat"></a>Configuring Filebeat to Send Log Lines to Logstash<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/logstash/edit/6.7/docs/static/advanced-pipeline.asciidoc">edit</a></h3>
</div></div></div>
<p>Before you create the Logstash pipeline, you’ll configure Filebeat to send log lines to Logstash.
The <a href="https://github.com/elastic/beats/tree/master/filebeat" class="ulink" target="_top">Filebeat</a> client is a lightweight, resource-friendly tool
that collects logs from files on the server and forwards these logs to your Logstash instance for processing.
Filebeat is designed for reliability and low latency. Filebeat has a light resource footprint on the host machine,
and the <a href="/guide/en/logstash/6.7/plugins-inputs-beats.html" class="ulink" target="_top"><code class="literal">Beats input</code></a> plugin minimizes the resource demands on the Logstash
instance.</p>
<div class="note admon">
<div class="icon"></div>
<div class="admon_content">
<p>In a typical use case, Filebeat runs on a separate machine from the machine running your
Logstash instance. For the purposes of this tutorial, Logstash and Filebeat are running on the
same machine.</p>
</div>
</div>
<p>The default Logstash installation includes the <a href="/guide/en/logstash/6.7/plugins-inputs-beats.html" class="ulink" target="_top"><code class="literal">Beats input</code></a> plugin. The Beats
input plugin enables Logstash to receive events from the Elastic Beats framework, which means that any Beat written
to work with the Beats framework, such as Packetbeat and Metricbeat, can also send event data to Logstash.</p>
<p>To install Filebeat on your data source machine, download the appropriate package from the Filebeat <a href="/downloads/beats/filebeat" class="ulink" target="_top">product page</a>. You can also refer to
<a href="/guide/en/beats/filebeat/6.7/filebeat-getting-started.html" class="ulink" target="_top">Getting Started with Filebeat</a> in the Beats documentation for additional
installation instructions.</p>
<p>After installing Filebeat, you need to configure it. Open the <code class="literal">filebeat.yml</code> file located in your Filebeat installation
directory, and replace the contents with the following lines. Make sure <code class="literal">paths</code> points to the example Apache log file,
<code class="literal">logstash-tutorial.log</code>, that you downloaded earlier:</p>
<div class="pre_wrapper lang-yaml">
<pre class="programlisting prettyprint lang-yaml">filebeat.prospectors:
- type: log
paths:
- /path/to/file/logstash-tutorial.log <a id="CO1-1"></a><i class="conum" data-value="1"></i>
output.logstash:
hosts: ["localhost:5044"]</pre>
</div>
<div class="calloutlist">
<table border="0" summary="Callout list">
<tr>
<td align="left" valign="top" width="5%">
<p><a href="#CO1-1"><i class="conum" data-value="1"></i></a></p>
</td>
<td align="left" valign="top">
<p>Absolute path to the file or files that Filebeat processes.</p>
</td>
</tr>
</table>
</div>
<p>Save your changes.</p>
<p>To keep the configuration simple, you won’t specify TLS/SSL settings as you would in a real world
scenario.</p>
<p>At the data source machine, run Filebeat with the following command:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">sudo ./filebeat -e -c filebeat.yml -d "publish"</pre>
</div>
<div class="note admon">
<div class="icon"></div>
<div class="admon_content">
<p>If you run Filebeat as root, you need to change ownership of the configuration file (see
<a href="/guide/en/beats/libbeat/6.7/config-file-permissions.html" class="ulink" target="_top">Config File Ownership and Permissions</a>
in the <em>Beats Platform Reference</em>).</p>
</div>
</div>
<p>Filebeat will attempt to connect on port 5044. Until Logstash starts with an active Beats plugin, there
won’t be any answer on that port, so any messages you see regarding failure to connect on that port are normal for now.</p>
</div>
<div class="section">
<div class="titlepage"><div><div>
<h3 class="title"><a id="_configuring_logstash_for_filebeat_input"></a>Configuring Logstash for Filebeat Input<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/logstash/edit/6.7/docs/static/advanced-pipeline.asciidoc">edit</a></h3>
</div></div></div>
<p>Next, you create a Logstash configuration pipeline that uses the Beats input plugin to receive
events from Beats.</p>
<p>The following text represents the skeleton of a configuration pipeline:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json"># The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
}</pre>
</div>
<p>This skeleton is non-functional, because the input and output sections don’t have any valid options defined.</p>
<p>To get started, copy and paste the skeleton configuration pipeline into a file named <code class="literal">first-pipeline.conf</code> in your home
Logstash directory.</p>
<p>Next, configure your Logstash instance to use the Beats input plugin by adding the following lines to the <code class="literal">input</code> section
of the <code class="literal">first-pipeline.conf</code> file:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json"> beats {
port => "5044"
}</pre>
</div>
<p>You’ll configure Logstash to write to Elasticsearch later. For now, you can add the following line
to the <code class="literal">output</code> section so that the output is printed to stdout when you run Logstash:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json"> stdout { codec => rubydebug }</pre>
</div>
<p>When you’re done, the contents of <code class="literal">first-pipeline.conf</code> should look like this:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">input {
beats {
port => "5044"
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
stdout { codec => rubydebug }
}</pre>
</div>
<p>To verify your configuration, run the following command:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">bin/logstash -f first-pipeline.conf --config.test_and_exit</pre>
</div>
<p>The <code class="literal">--config.test_and_exit</code> option parses your configuration file and reports any errors.</p>
<p>If the configuration file passes the configuration test, start Logstash with the following command:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">bin/logstash -f first-pipeline.conf --config.reload.automatic</pre>
</div>
<p>The <code class="literal">--config.reload.automatic</code> option enables automatic config reloading so that you don’t have to stop and restart Logstash
every time you modify the configuration file.</p>
<p>As Logstash starts up, you might see one or more warning messages about Logstash ignoring the <code class="literal">pipelines.yml</code> file. You
can safely ignore this warning. The <code class="literal">pipelines.yml</code> file is used for running <a class="xref" href="multiple-pipelines.html" title="Multiple Pipelines">multiple pipelines</a>
in a single Logstash instance. For the examples shown here, you are running a single pipeline.</p>
<p>If your pipeline is working correctly, you should see a series of events like the following written to the console:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">{
"@timestamp" => 2017-11-09T01:44:20.071Z,
"offset" => 325,
"@version" => "1",
"beat" => {
"name" => "My-MacBook-Pro.local",
"hostname" => "My-MacBook-Pro.local",
"version" => "6.0.0"
},
"host" => "My-MacBook-Pro.local",
"prospector" => {
"type" => "log"
},
"source" => "/path/to/file/logstash-tutorial.log",
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"tags" => [
[0] "beats_input_codec_plain_applied"
]
}
...</pre>
</div>
<h4><a id="configuring-grok-filter"></a>Parsing Web Logs with the Grok Filter Plugin<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/logstash/edit/6.7/docs/static/advanced-pipeline.asciidoc">edit</a></h4>
<p>Now you have a working pipeline that reads log lines from Filebeat. However you’ll notice that the format of the log messages
is not ideal. You want to parse the log messages to create specific, named fields from the logs.
To do this, you’ll use the <code class="literal">grok</code> filter plugin.</p>
<p>The <a href="/guide/en/logstash/6.7/plugins-filters-grok.html" class="ulink" target="_top"><code class="literal">grok</code></a> filter plugin is one of several plugins that are available by default in
Logstash. For details on how to manage Logstash plugins, see the <a class="xref" href="working-with-plugins.html" title="Working with plugins">reference documentation</a> for
the plugin manager.</p>
<p>The <code class="literal">grok</code> filter plugin enables you to parse the unstructured log data into something structured and queryable.</p>
<p>Because the <code class="literal">grok</code> filter plugin looks for patterns in the incoming log data, configuring the plugin requires you to
make decisions about how to identify the patterns that are of interest to your use case. A representative line from the
web server log sample looks like this:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png
HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"</pre>
</div>
<p>The IP address at the beginning of the line is easy to identify, as is the timestamp in brackets. To parse the data, you can use the <code class="literal">%{COMBINEDAPACHELOG}</code> grok pattern, which structures lines from the Apache log using the following schema:</p>
<div class="informaltable">
<table border="0" cellpadding="4px">
<colgroup>
<col/>
<col/>
</colgroup>
<tbody valign="top">
<tr>
<td valign="top">
<p>
<span class="strong strong"><strong>Information</strong></span>
</p>
</td>
<td valign="top">
<p>
<span class="strong strong"><strong>Field Name</strong></span>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
IP Address
</p>
</td>
<td valign="top">
<p>
<code class="literal">clientip</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
User ID
</p>
</td>
<td valign="top">
<p>
<code class="literal">ident</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
User Authentication
</p>
</td>
<td valign="top">
<p>
<code class="literal">auth</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
timestamp
</p>
</td>
<td valign="top">
<p>
<code class="literal">timestamp</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
HTTP Verb
</p>
</td>
<td valign="top">
<p>
<code class="literal">verb</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
Request body
</p>
</td>
<td valign="top">
<p>
<code class="literal">request</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
HTTP Version
</p>
</td>
<td valign="top">
<p>
<code class="literal">httpversion</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
HTTP Status Code
</p>
</td>
<td valign="top">
<p>
<code class="literal">response</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
Bytes served
</p>
</td>
<td valign="top">
<p>
<code class="literal">bytes</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
Referrer URL
</p>
</td>
<td valign="top">
<p>
<code class="literal">referrer</code>
</p>
</td>
</tr>
<tr>
<td valign="top">
<p>
User agent
</p>
</td>
<td valign="top">
<p>
<code class="literal">agent</code>
</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="tip admon">
<div class="icon"></div>
<div class="admon_content">
<p>If you need help building grok patterns, try out the
<a href="/guide/en/kibana/6.7/xpack-grokdebugger.html" class="ulink" target="_top">Grok Debugger</a>. The Grok Debugger is an
X-Pack feature under the Basic License and is therefore <span class="strong strong"><strong>free to use</strong></span>.</p>
</div>
</div>
<p>Edit the <code class="literal">first-pipeline.conf</code> file and replace the entire <code class="literal">filter</code> section with the following text:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}</pre>
</div>
<p>When you’re done, the contents of <code class="literal">first-pipeline.conf</code> should look like this:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
stdout { codec => rubydebug }
}</pre>
</div>
<p>Save your changes. Because you’ve enabled automatic config reloading, you don’t have to restart Logstash to
pick up your changes. However, you do need to force Filebeat to read the log file from scratch. To do this,
go to the terminal window where Filebeat is running and press Ctrl+C to shut down Filebeat. Then delete the
Filebeat registry file. For example, run:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">sudo rm data/registry</pre>
</div>
<p>Since Filebeat stores the state of each file it harvests in the registry, deleting the registry file forces
Filebeat to read all the files it’s harvesting from scratch.</p>
<p>Next, restart Filebeat with the following command:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">sudo ./filebeat -e -c filebeat.yml -d "publish"</pre>
</div>
<p>There might be a slight delay before Filebeat begins processing events if it needs to wait for Logstash to reload the
config file.</p>
<p>After Logstash applies the grok pattern, the events will have the following JSON representation:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">{
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"offset" => 325,
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"prospector" => {
"type" => "log"
},
"source" => "/path/to/file/logstash-tutorial.log",
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
"@timestamp" => 2017-11-09T02:51:12.416Z,
"response" => "200",
"bytes" => "203023",
"clientip" => "83.149.9.216",
"@version" => "1",
"beat" => {
"name" => "My-MacBook-Pro.local",
"hostname" => "My-MacBook-Pro.local",
"version" => "6.0.0"
},
"host" => "My-MacBook-Pro.local",
"httpversion" => "1.1",
"timestamp" => "04/Jan/2015:05:13:42 +0000"
}</pre>
</div>
<p>Notice that the event includes the original message, but the log message is also broken down into specific fields.</p>
<h4><a id="configuring-geoip-plugin"></a>Enhancing Your Data with the Geoip Filter Plugin<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/logstash/edit/6.7/docs/static/advanced-pipeline.asciidoc">edit</a></h4>
<p>In addition to parsing log data for better searches, filter plugins can derive supplementary information from existing
data. As an example, the <a href="/guide/en/logstash/6.7/plugins-filters-geoip.html" class="ulink" target="_top"><code class="literal">geoip</code></a> plugin looks up IP addresses, derives geographic
location information from the addresses, and adds that location information to the logs.</p>
<p>Configure your Logstash instance to use the <code class="literal">geoip</code> filter plugin by adding the following lines to the <code class="literal">filter</code> section
of the <code class="literal">first-pipeline.conf</code> file:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json"> geoip {
source => "clientip"
}</pre>
</div>
<p>The <code class="literal">geoip</code> plugin configuration requires you to specify the name of the source field that contains the IP address to look up. In this example, the <code class="literal">clientip</code> field contains the IP address.</p>
<p>Since filters are evaluated in sequence, make sure that the <code class="literal">geoip</code> section is after the <code class="literal">grok</code> section of
the configuration file and that both the <code class="literal">grok</code> and <code class="literal">geoip</code> sections are nested within the <code class="literal">filter</code> section.</p>
<p>When you’re done, the contents of <code class="literal">first-pipeline.conf</code> should look like this:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
}</pre>
</div>
<p>Save your changes. To force Filebeat to read the log file from scratch, as you did earlier, shut down Filebeat (press Ctrl+C),
delete the registry file, and then restart Filebeat with the following command:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">sudo ./filebeat -e -c filebeat.yml -d "publish"</pre>
</div>
<p>Notice that the event now contains geographic location information:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">{
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"geoip" => {
"timezone" => "Europe/Moscow",
"ip" => "83.149.9.216",
"latitude" => 55.7485,
"continent_code" => "EU",
"city_name" => "Moscow",
"country_name" => "Russia",
"country_code2" => "RU",
"country_code3" => "RU",
"region_name" => "Moscow",
"location" => {
"lon" => 37.6184,
"lat" => 55.7485
},
"postal_code" => "101194",
"region_code" => "MOW",
"longitude" => 37.6184
},
...</pre>
</div>
<h4><a id="indexing-parsed-data-into-elasticsearch"></a>Indexing Your Data into Elasticsearch<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/logstash/edit/6.7/docs/static/advanced-pipeline.asciidoc">edit</a></h4>
<p>Now that the web logs are broken down into specific fields, you’re ready to get
your data into Elasticsearch.</p>
<div class="tip admon">
<div class="icon"></div>
<div class="admon_content">
<p>You can run Elasticsearch on your own hardware, or use our
<a href="/cloud/elasticsearch-service" class="ulink" target="_top">hosted Elasticsearch Service</a> on
Elastic Cloud. The Elasticsearch Service is available on both AWS and GCP.
<a href="/cloud/elasticsearch-service/signup" class="ulink" target="_top">Try the Elasticsearch Service
for free</a>.</p>
</div>
</div>
<p>The Logstash pipeline can index the data into an
Elasticsearch cluster. Edit the <code class="literal">first-pipeline.conf</code> file and replace the entire <code class="literal">output</code> section with the following
text:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}</pre>
</div>
<p>With this configuration, Logstash uses http protocol to connect to Elasticsearch. The above example assumes that
Logstash and Elasticsearch are running on the same instance. You can specify a remote Elasticsearch instance by using
the <code class="literal">hosts</code> configuration to specify something like <code class="literal">hosts => [ "es-machine:9092" ]</code>.</p>
<p>At this point, your <code class="literal">first-pipeline.conf</code> file has input, filter, and output sections properly configured, and looks
something like this:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}</pre>
</div>
<p>Save your changes. To force Filebeat to read the log file from scratch, as you did earlier, shut down Filebeat (press Ctrl+C),
delete the registry file, and then restart Filebeat with the following command:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">sudo ./filebeat -e -c filebeat.yml -d "publish"</pre>
</div>
<h5><a id="testing-initial-pipeline"></a>Testing Your Pipeline<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/logstash/edit/6.7/docs/static/advanced-pipeline.asciidoc">edit</a></h5>
<p>Now that the Logstash pipeline is configured to index the data into an
Elasticsearch cluster, you can query Elasticsearch.</p>
<p>Try a test query to Elasticsearch based on the fields created by the <code class="literal">grok</code> filter plugin.
Replace $DATE with the current date, in YYYY.MM.DD format:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=response=200'</pre>
</div>
<div class="note admon">
<div class="icon"></div>
<div class="admon_content">
<p>The date used in the index name is based on UTC, not the timezone where Logstash is running.
If the query returns <code class="literal">index_not_found_exception</code>, make sure that <code class="literal">logstash-$DATE</code> reflects the actual
name of the index. To see a list of available indexes, use this query: <code class="literal">curl 'localhost:9200/_cat/indices?v'</code>.</p>
</div>
</div>
<p>You should get multiple hits back. For example:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">{
"took": 50,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 98,
"max_score": 2.793642,
"hits": [
{
"_index": "logstash-2017.11.09",
"_type": "doc",
"_id": "3IzDnl8BW52sR0fx5wdV",
"_score": 2.793642,
"_source": {
"request": "/presentations/logstash-monitorama-2013/images/frontend-response-codes.png",
"agent": """"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"""",
"geoip": {
"timezone": "Europe/Moscow",
"ip": "83.149.9.216",
"latitude": 55.7485,
"continent_code": "EU",
"city_name": "Moscow",
"country_name": "Russia",
"country_code2": "RU",
"country_code3": "RU",
"region_name": "Moscow",
"location": {
"lon": 37.6184,
"lat": 55.7485
},
"postal_code": "101194",
"region_code": "MOW",
"longitude": 37.6184
},
"offset": 2932,
"auth": "-",
"ident": "-",
"verb": "GET",
"prospector": {
"type": "log"
},
"source": "/path/to/file/logstash-tutorial.log",
"message": """83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] "GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1" 200 52878 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"""",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": """"http://semicomplete.com/presentations/logstash-monitorama-2013/"""",
"@timestamp": "2017-11-09T03:11:35.304Z",
"response": "200",
"bytes": "52878",
"clientip": "83.149.9.216",
"@version": "1",
"beat": {
"name": "My-MacBook-Pro.local",
"hostname": "My-MacBook-Pro.local",
"version": "6.0.0"
},
"host": "My-MacBook-Pro.local",
"httpversion": "1.1",
"timestamp": "04/Jan/2015:05:13:45 +0000"
}
},
...</pre>
</div>
<p>Try another search for the geographic information derived from the IP address.
Replace $DATE with the current date, in YYYY.MM.DD format:</p>
<div class="pre_wrapper lang-shell">
<pre class="programlisting prettyprint lang-shell">curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=geoip.city_name=Buffalo'</pre>
</div>
<p>A few log entries come from Buffalo, so the query produces the following response:</p>
<div class="pre_wrapper lang-json">
<pre class="programlisting prettyprint lang-json">{
"took": 9,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 2.6390574,
"hits": [
{
"_index": "logstash-2017.11.09",
"_type": "doc",
"_id": "L4zDnl8BW52sR0fx5whY",
"_score": 2.6390574,
"_source": {
"request": "/blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29",
"agent": """"Tiny Tiny RSS/1.11 (http://tt-rss.org/)"""",
"geoip": {
"timezone": "America/New_York",
"ip": "198.46.149.143",
"latitude": 42.8864,
"continent_code": "NA",
"city_name": "Buffalo",
"country_name": "United States",
"country_code2": "US",
"dma_code": 514,
"country_code3": "US",
"region_name": "New York",
"location": {
"lon": -78.8781,
"lat": 42.8864
},
"postal_code": "14202",
"region_code": "NY",
"longitude": -78.8781
},
"offset": 22795,
"auth": "-",
"ident": "-",
"verb": "GET",
"prospector": {
"type": "log"
},
"source": "/path/to/file/logstash-tutorial.log",
"message": """198.46.149.143 - - [04/Jan/2015:05:29:13 +0000] "GET /blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 9316 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)"""",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": """"-"""",
"@timestamp": "2017-11-09T03:11:35.321Z",
"response": "200",
"bytes": "9316",
"clientip": "198.46.149.143",
"@version": "1",
"beat": {
"name": "My-MacBook-Pro.local",
"hostname": "My-MacBook-Pro.local",
"version": "6.0.0"
},
"host": "My-MacBook-Pro.local",
"httpversion": "1.1",
"timestamp": "04/Jan/2015:05:29:13 +0000"
}
},
...</pre>
</div>
<p>If you are using Kibana to visualize your data, you can also explore the Filebeat data in Kibana:</p>
<div class="imageblock">
<div class="content">
<img src="static/images/kibana-filebeat-data.png" alt="Discovering Filebeat data in Kibana">
</div>
</div>
<p>See the <a href="/guide/en/beats/filebeat/6.7/filebeat-getting-started.html" class="ulink" target="_top">Filebeat getting started docs</a> for info about loading the Kibana
index pattern for Filebeat.</p>
<p>You’ve successfully created a pipeline that uses Filebeat to take Apache web logs as input, parses those logs to
create specific, named fields from the logs, and writes the parsed data to an Elasticsearch cluster. Next, you
learn how to create a pipeline that uses multiple input and output plugins.</p>
</div>
</div>
<div class="navfooter">
<span class="prev">
<a href="first-event.html">« Stashing Your First Event</a>
</span>
<span class="next">
<a href="multiple-input-output-plugins.html">Stitching Together Multiple Input and Output Plugins »</a>
</span>
</div>
</div>
<!-- end body -->
</div>
<div class="col-12 order-3 col-lg-2 order-lg-3 h-almost-full-lg sticky-top-lg" id="right_col">
<div id="sticky_content">
<!-- The OTP is appended here -->
<div class="row">
<div class="col-0 col-md-4 col-lg-0" id="bottom_left_col"></div>
<div class="col-12 col-md-8 col-lg-12">
<div id="rtpcontainer">
<div class="mktg-promo" id="most-popular">
<p class="aside-heading">Most Popular</p>
<div class="pb-2">
<p class="media-type">Video</p>
<a href="https://www.elastic.co/webinars/getting-started-elasticsearch?baymax=default&elektra=docs&storm=top-video">
<p class="mb-0">Get Started with Elasticsearch</p>
</a>
</div>
<div class="pb-2">
<p class="media-type">Video</p>
<a href="https://www.elastic.co/webinars/getting-started-kibana?baymax=default&elektra=docs&storm=top-video">
<p class="mb-0">Intro to Kibana</p>
</a>
</div>
<div class="pb-2">
<p class="media-type">Video</p>
<a href="https://www.elastic.co/webinars/introduction-elk-stack?baymax=default&elektra=docs&storm=top-video">
<p class="mb-0">ELK for Logs & Metrics</p>
</a>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
<div id='elastic-footer'></div>
<script src='https://www.elastic.co/elastic-footer.js'></script>
<!-- Footer Section end-->
</section>
</div>
<script src="/guide/static/jquery.js"></script>
<script type="text/javascript" src="/guide/static/docs.js"></script>
<script type="text/javascript">
window.initial_state = {}</script>
</body>
</html>