Log Management

with Graylog
Graylog - architecture
• Elasticsearch - main "database" for logs

• MongoDB - stores configurations and meta

information

• Graylog - processes events and GUI

www.cybexer.com 2
Graylog - architecture

www.cybexer.com 3
Graylog - prerequisites
Login to Ubuntu system (graylog-snort.envXX.csirt.crp
- 10.X.32.4).
Verify that you logged in as a 'root' user

• id

www.cybexer.com 4
Graylog - prerequisites
Make sure your system is fully updated
• apt-get update && apt-get upgrade

If system updated kernel packages, then reboot the

machine
• reboot

www.cybexer.com 5
Graylog - prerequisites
Graylog 5.0 requires the following to maintain
compatibility with its software dependencies:
• OpenJDK 17 (This is embedded in Graylog 5.0 and
does not need to be separately installed.)
• OpenSearch 1.x, 2.x or Elasticsearch 7.10.2
• MongoDB 5.x or 6.x

www.cybexer.com 6
Graylog - prerequisites
Graylog requires, that some packages are installed on
the system. Let's install additional packages
• apt -y install apt-transport-https uuid-runtime
pwgen dirmngr gnupg wget curl jq net-tools

www.cybexer.com 7
Graylog - MongoDB installation
To install MongoDB from repository, first we have to
fetch and save the authentication key
• curl -fsSL
https://www.mongodb.org/static/pgp/server-
6.0.asc | gpg --dearmor -o
/usr/share/keyrings/mongodb.gpg

www.cybexer.com 8
Graylog - MongoDB installation
Next, we have to add MongoDB repository location
• echo "deb [ arch=amd64,arm64 signed-
by=/usr/share/keyrings/mongodb.gpg]
https://repo.mongodb.org/apt/ubuntu
jammy/mongodb-org/6.0 multiverse" | tee -a
/etc/apt/sources.list.d/mongodb-org-6.0.list

www.cybexer.com 9
Graylog - MongoDB installation
After adding new repository, we have to update the
package list
• apt-get update

www.cybexer.com 10
Graylog - MongoDB installation
Now we can proceed with MongoDB installation
• apt-get -y install mongodb-org

www.cybexer.com 11
Graylog - MongoDB installation
After MongoDB installed, we must reload system start-
up scripts and enable MongoDB automatic start-up
after boot

• systemctl daemon-reload
• systemctl enable mongod.service
• systemctl restart mongod.service

www.cybexer.com 12
Graylog - MongoDB installation
verification
We must verify MongoDB is running. There're several
ways how to do it.
By default, MongoDB listens on port 27017, let's check
if this port is listening

• netstat -natp |grep 27017

www.cybexer.com 13
Graylog - MongoDB installation
verification
Another way to verify MongoDB is running, is to check
running MongoDB processes. We use 'ps' program to
check that

• ps auxf |grep mongo

www.cybexer.com 14
Graylog - MongoDB installation
verification
Third way is to use 'systemctl' service manager. This
way gives most of information about service and more
preferrable
• systemctl status mongod

www.cybexer.com 15
Graylog - OpenSearch installation
Before OpenSearch installation, it is advised to adjust some system
performance parameters
• cat > /etc/systemd/system/disable-transparent-huge-
pages.service << 'EOF'
• Description=Disable Transparent Huge Pages (THP)
• DefaultDependencies=no
• After=sysinit.target local-fs.target
• [Service]
• Type=oneshot
• ExecStart=/bin/sh -c 'echo never | tee
/sys/kernel/mm/transparent_hugepage/enabled > /dev/null'
• [Install]
• WantedBy=basic.target
• EOF

www.cybexer.com 16
Graylog - OpenSearch installation
To ensure, that performance tuning script is loaded and
system startup, we have to enable it
• systemctl daemon-reload
• systemctl enable disable-transparent-huge-
pages.service
• systemctl start disable-transparent-huge-
pages.service

www.cybexer.com 17
Graylog - OpenSearch installation
To install OpenSearch, we have to download
OpenSearch GPG key to local system
• curl -fsSL
https://artifacts.opensearch.org/publickeys/opense
arch.pgp | gpg --dearmor -o
/usr/share/keyrings/opensearch.pgp

www.cybexer.com 18
Graylog - OpenSearch installation
After GPG key was downloaded, we will add
OpenSearch repository
• echo "deb [signed-
by=/usr/share/keyrings/opensearch.pgp]
https://artifacts.opensearch.org/releases/bundle/o
pensearch/2.x/apt stable main" | tee -a
/etc/apt/sources.list.d/opensearch-2.x.list

www.cybexer.com 19
Graylog - OpenSearch installation
After adding new repositories to Ubuntu/Debian
systems, we have to update the list of available files
• apt-get update

www.cybexer.com 20
Graylog - OpenSearch installation
Now we can proceed with OpenSearch installation (be
aware, that installation file is around 750 MB, and it
might take some time)
• apt-get -y install opensearch

www.cybexer.com 21
Graylog - OpenSearch configuration
Before starting OpenSearch, we must adjust some
OpenSearch configurations manually in
'/etc/opensearch/opensearch.yml' file.
Three configuration options must be changed or added
if they are missing

• cluster.name: graylog
• node.name: graylog-node-01
• action.auto_create_index: false
• plugins.security.disabled: true

www.cybexer.com 22
Graylog - OpenSearch configuration
Before making any changes to configuration file, it's a good
practice to make backup of original configuration file
• cp /etc/opensearch/opensearch.yml
/etc/opensearch/opensearch.yml.bck

Now we can adjust OpenSearch configuration with one-

liners
• sed -i '/#cluster.name:/c\cluster.name: graylog'
/etc/opensearch/opensearch.yml
• sed -i '/#node.name:/c\node.name: graylog-node-01'
/etc/opensearch/opensearch.yml

www.cybexer.com 23
Graylog - OpenSearch configuration
Continue adjusting OpenSearch configuration with one-
liners
• echo 'action.auto_create_index: false' >>
/etc/opensearch/opensearch.yml
• echo 'plugins.security.disabled: true' >>
/etc/opensearch/opensearch.yml

www.cybexer.com 24
Graylog - OpenSearch configuration
Default installation of OpenSearch uses 1 GB of RAM,
which is sufficient for testing purposes.
For production or high-load environments you must
increase RAM amount dedicated to OpenSearch service
in '/etc/opensearch/jvm.options' file.

www.cybexer.com 25
Graylog - OpenSearch configuration
After OpenSearch installed, we must reload system
start-up scripts, enable OpenSearch automatic start-up
on boot and start OpenSearch. Run following
commands in you Linux terminal:
• systemctl daemon-reload
• systemctl enable opensearch.service
• systemctl restart opensearch.service

www.cybexer.com 26
Graylog - OpenSearch installation
verification
We must verify OpenSearch is running. There're several
ways how to do it.
By default, OpenSearch listens on port 9200, let's check
if this port is listening
• netstat -natp |grep 9200

www.cybexer.com 27
Graylog - OpenSearch installation
verification
Another way to verify OpenSearch is running, is to
check running OpenSearch processes. We use 'ps'
program to check that
• ps auxf |grep opensearch

www.cybexer.com 28
Graylog - OpenSearch installation
verification
Third way is to use 'systemctl' service manager. This
way gives most of information about service and more
preferrable
• systemctl status opensearch.service

www.cybexer.com 29
Graylog - OpenSearch installation
verification
OpenSearch allows to run so called 'health check query' against
it and check status of OpenSearch
• curl -XGET
'http://localhost:9200/_cluster/health?pretty=true'

It's import, that 'status' value is 'green'. If you see other colors,
then further investigation is needed.

www.cybexer.com 30
Graylog - OpenSearch installation
verification
If for some reasons OpenSearch is not running, then
you have to check log file '
/var/log/opensearch/opensearch.log ' for errors.

Usually, log files will show exactly why service is not

started.
Some of the reasons might be - run out of disk space,
wrong file and folder permissions, missing additional
software and broken/incorrect configuration file.

www.cybexer.com 31
Graylog installation
To install Graylog we have to download Graylog's
repository package. It's reasonable to do in separate
folder
• cd /opt
• wget
https://packages.graylog2.org/repo/packages/grayl
og-5.0-repository_latest.deb

www.cybexer.com 32
Graylog installation
Now let's install downloaded repository file. It will add
all required authentication keys and Graylog package
repositories
• dpkg -i graylog-5.0-repository_latest.deb

www.cybexer.com 33
Graylog installation
As usually, after adding new repository we have to
update the file list
• apt-get update
Now we can proceed with installation of Graylog
• apt-get install graylog-server

www.cybexer.com 34
Graylog configuration
Graylog configuration file must be adjusted before
starting Graylog service. Firstly, let's make backup of
original configuration file
• cp /etc/graylog/server/server.conf
/etc/graylog/server/server.conf.bck

By default, Graylog will not start if following

configuration options are not set (they are mandatory)
• password_secret
• root_password_sha2

www.cybexer.com 35
Graylog configuration
Generating new 'password_secret' value
• pwgen -N 1 -s 96

Automatically configure 'password_secret' value and

make corresponding changes in configuration file
• sed -i -e "s/password_secret =.*/password_secret =
$(pwgen -s 96 1)/" /etc/graylog/server/server.conf

www.cybexer.com 36
Graylog configuration
Next, we need to generate hashed password for
administrator user. This hash goes into
'root_password_sha2' option
• echo -n admin | shasum -a 256

Automatically configure 'root_password_sha2' value

• sed -i -e "s/root_password_sha2
=.*/root_password_sha2 = $(echo -n 'admin' |
shasum -a 256 | cut -d' ' -f1)/"
/etc/graylog/server/server.conf

www.cybexer.com 37
Graylog configuration
Verify, that required values are set in the configuration
file
• grep -E '(^password_secret|^root_password_sha2)'
/etc/graylog/server/server.conf

www.cybexer.com 38
Graylog configuration
Additional configuration options to change/review before finalizing
Graylog installation:

• root_username
• root_email
• root_timezone*
• http_bind_address
• elasticsearch_max_docs_per_index
• elasticsearch_index_prefix
• allow_highlighting
• allow_leading_wildcard_searches

* list of time zones- https://www.joda.org/joda-time/timezones.html

www.cybexer.com 39
Graylog configuration
Allow Graylog search results to be highlighted
• sed -i -e "s/allow_highlighting
=.*/allow_highlighting = true/"
/etc/graylog/server/server.conf

www.cybexer.com 40
Graylog configuration
After Graylog installed, we must reload system start-up
scripts, enable Graylog automatic start-up on boot and
start Graylog
• systemctl daemon-reload
• systemctl enable graylog-server.service
• systemctl restart graylog-server.service

Note! First start of Graylog may take some time.

The reason is - creation of required tables, indices etc.

www.cybexer.com 41
Graylog configuration verification
We must verify Graylog is running. There're several
ways how to do it.
By default, Graylog listens on port 9000, let's check if
this port is listening

• netstat -natp |grep 9000

www.cybexer.com 42
Graylog configuration verification
Another way to verify Graylog is running, is to check
running Graylog processes. We use 'ps' program to
check that

• ps auxf |grep graylog

www.cybexer.com 43
Graylog configuration verification
Third way is to use 'systemctl' service manager. This
way gives most of information about service and more
preferrable

• systemctl status graylog-server.service

www.cybexer.com 44
Graylog configuration verification
If for some reasons Graylog is not running, then you
have to check log file '/var/log/graylog-
server/server.log' for errors.
Usually, log files will show exactly why service is not
started.
Some of the reasons might be - run out of disk space,
wrong file and folder permissions, missing additional
software and broken/incorrect configuration file

www.cybexer.com 45
Graylog - setup complete

Congratulations!

You have now successfully setup MongoDB,

Elasticsearch and Graylog server.

www.cybexer.com 46
Graylog - advanced setup
By default, Graylog's web interface listens on loopback
interface (127.0.0.1) and you will not be able to access
web GUI remotely.

Also, Graylog is running on unencrypted HTTP protocol.

We will setup reverse proxy, which will use encrypted

HTTPS connection to Graylog.

www.cybexer.com 47
Graylog - advanced setup
To secure Graylog we will use Apache webserver to
proxy requests to Graylog.
Verify Ubuntu system is up-to-date
• apt-get -y update && apt-get -y upgrade

www.cybexer.com 48
Graylog - advanced setup
Install Apache web server from repository
• apt-get -y install apache2

www.cybexer.com 49
Graylog - advanced setup
Generate new self-signed private and public certificates
for Apache webserver
• openssl req -x509 -nodes -days 365 -newkey
rsa:2048 -keyout /etc/ssl/private/apache.key -out
/etc/ssl/certs/apache.crt

www.cybexer.com 50
Graylog - advanced setup
You can add certificate options during certificate
generation process
• openssl req -x509 -nodes -days 365 -newkey
rsa:2048 -keyout /etc/ssl/private/apache.key -out
/etc/ssl/certs/apache.crt --subj
'/C=EU/O=Cyber/OU=IT/CN=Graylog' -addext
"subjectAltName = DNS:graylog5-
server,IP:10.103.8.238"

www.cybexer.com 51
Graylog - advanced setup
Create /etc/apache2/sites-available/graylog-ssl.conf with following content:
<VirtualHost *:443>
ServerName 10.255.34.114
ProxyRequests Off
SSLEngine on
SSLCertificateFile /etc/ssl/certs/apache.crt
SSLCertificateKeyFile /etc/ssl/private/apache.key
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
<Location />
RequestHeader set X-Graylog-Server-URL "https://10.255.34.114/"
ProxyPass http://127.0.0.1:9000/
ProxyPassReverse http://127.0.0.1:9000/
</Location>
</VirtualHost>

Verify that IP addresses are correct. Use your Graylog server's IP address

www.cybexer.com 52
Graylog - advanced setup
Or download Apache configuration file from 'files.csirt.crp'
server
• wget http://files.csirt.crp/graylog-ssl.conf.txt -O
/etc/apache2/sites-available/graylog-ssl.conf

Be sure, to set correct IP address.

• sed -i 's/10.255.34.114/10.XX.32.4/' /etc/apache2/sites-

available/graylog-ssl.conf

www.cybexer.com 53
Graylog - advanced setup
Apache will be acting as a front-end server for Graylog.
To be able to proxy Apache requests to Graylog, we
have to enable several Apache modules.
In Linux terminal type following commands:
• a2ensite graylog-ssl.conf
'a2ensite' command will enable our Apache HTTPS
configuration in 'graylog-ssl.conf' file

www.cybexer.com 54
Graylog - advanced setup
Apache will be acting as a front-end server for Graylog.
To be able to proxy Apache requests to Graylog, we
have to enable several Apache modules.
In Linux terminal type following commands:
• a2enmod proxy
• a2enmod proxy_http
'a2enmod' command will enable proxy modules, which
allow to proxy client requests to Graylog server

www.cybexer.com 55
Graylog - advanced setup
Apache will be acting as a front-end server for Graylog.
To be able to proxy Apache requests to Graylog, we
have to enable several Apache modules.
In Linux terminal type following commands:
• a2enmod ssl
• a2enmod headers
'ssl' module enables Apache web server to run on
HTTPS protocol and 'header' module passes header
information to Graylog

www.cybexer.com 56
Graylog - advanced setup
To make new Apache modules available, we have to
restart Apache webserver
• systemctl restart apache2
After Apache web server is restated, verify, that it's up
and running
• systemctl status apache2

www.cybexer.com 57
Graylog - first login
After Graylog and Apache web server are restarted,
open your browser and enter your Graylog's IP address
in address bar. Since we have enabled HTTPS support,
be sure to use 'https://' scheme

https://10.XX.32.4/

Note! Be sure to use correct IP address.

www.cybexer.com 58
Graylog - first login
Since we created self-signed certificates, you may see,
that browser warns, that connection is not private.
Click on 'Advanced' button

www.cybexer.com 59
Graylog - first login
And now click on 'Proceed to .....' link

www.cybexer.com 60
Graylog - first login
Login to Graylog's web interface with username 'admin'
and password 'admin'

www.cybexer.com 61
Graylog - first login
Congratulations!

You have now Graylog running in a secure way.

Graylog is listening on localhost (127.0.0.1), which
means, that access to it is available only from server
itself.
Apache web server is running on HTTPS port (443) and
proxying all requests to Graylog's port.

www.cybexer.com 62
Graylog - first run
Main Graylog interface

www.cybexer.com 63
Graylog - adding users
By default, Graylog has only one user created - admin

It is good practice to create new, dedicated user for

Graylog system.

www.cybexer.com 64
Graylog - adding users
Navigate to System->Users and Teams

www.cybexer.com 65
Graylog - adding users
Click 'Create User'

www.cybexer.com 66
Graylog - adding users
Enter user details. Use your student ID 2-digit number.

www.cybexer.com 67
Graylog - adding users
Since log management is very important in any
organization, it's required to set correct time zone

www.cybexer.com 68
Graylog - adding users
Assign role 'Admin'

www.cybexer.com 69
Graylog - adding users
Remove 'Reader' role by clicking 'x'

www.cybexer.com 70
Graylog - adding users
'Admin' role must be selected

www.cybexer.com 71
Graylog - adding users
Required fields for new user are:
First Name, Last Name, Username, E-Mail Address and
Password

When all required fields set, click 'Create user'

www.cybexer.com 72
Graylog - adding users
If new user creation was successful, then you will see in
user table new username:

Verify, that new user has correct permissions role

www.cybexer.com 73
Graylog - adding users
Logout current 'Administrator' user

www.cybexer.com 74
Graylog - adding users
Login with new username and password

www.cybexer.com 75
Graylog - adding users
Login with new username and password

www.cybexer.com 76
Graylog - first run
Default installation of Graylog does not have any
running inputs.
Input - is a main function, which accepts/receives the
logs.

www.cybexer.com 77
Graylog - create input
Let's create new input for receiving logs
Navigate to System->Inputs

www.cybexer.com 78
Graylog - create input
Enter tcp in the search field, select 'Raw/Plaintext TCP'
and click 'Launch new input' button

www.cybexer.com 79
Graylog - create input
Enter Title for input name

www.cybexer.com 80
Graylog - create input
For testing purposes, 'Bind address' can be '0.0.0.0',
but in production environment you have to select
proper 'Bind address'

www.cybexer.com 81
Graylog - create input
Same goes for the 'Port' number. In production
environment set port number, based on internal
regulations and security requirements

www.cybexer.com 82
Graylog - create input
Create new input by clicking 'Launch Input'

www.cybexer.com 83
Graylog - create input
TCP input is configured and running

www.cybexer.com 84
Graylog - send data to input
Now we have to verify, that Graylog is able to receive
logs from the input we have previously configured.

From Linux command line send test data to Graylog

input

• echo Hello | nc localhost 5555

www.cybexer.com 85
Graylog - view events
Navigate to Search and check the events

www.cybexer.com 86
Graylog - view events
Navigate to Search and
check the events.

If you see in 'All

Messages' block 'Hello'
message, that means your
Graylog setup is correct,
and it receives the logs
through the input.

www.cybexer.com 87
Graylog - purge all events
By design, Graylog does not allow to delete processed
logs.

For testing purposes, you can delete all Elasticsearch

data. First, you have to find out correct indice name
• curl 'http://localhost:9200/_cat/indices?v'

www.cybexer.com 88
Graylog - purge all events
In our case, indice name is 'graylog_0'

To delete all data from 'graylog_0' we have to send

'DELETE' command to Graylog API
• curl -XDELETE 'http://localhost:9200/graylog_0'

Verify that '{"acknowledged":true}' message is shown

www.cybexer.com 89
Graylog - send event(s)
Now we can test that real logs are reaching Graylog
Run following command in Linux terminal
• echo '192.168.1.5 - - [18/May/2020:16:43:05
+0300] "GET /gettingstarted HTTP/1.1" 404 507 "-"
"Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/69.0.3112.113 Safari/537.36"' | nc
localhost 5555

www.cybexer.com 90
Graylog - view events
Navigate to Search and
check the events.

If you see in 'All Messages'

block previously sent
message, then it means your
Graylog setup is correct, and
it receives the logs through
the input.

www.cybexer.com 91
Graylog - view events
Click on event you have in
'All Message' block

You should see detailed

view of received event.

www.cybexer.com 92
Graylog - advanced setup

What wrong with that event?

www.cybexer.com 93
Graylog - advanced setup
All SIEM and log management systems do log
normalization. This helps to search the logs, create
correlations and do log aggregations.

Most popular way is to normalize the logs is to use

GROK.

Grok works by combining text patterns into something

that matches your logs.
The syntax for a grok pattern is %{SYNTAX:SEMANTIC}

www.cybexer.com 94
Graylog - advanced setup

The SYNTAX is the name of the pattern that will match

your text/data.

The SEMANTIC is the identifier you give to the piece of

text being matched.

www.cybexer.com 95
Graylog - advanced setup
Let's see sample log event:
55.3.244.1 GET /index.html 15824 0.043

Following data can be normalize as:

IP address - 55.3.244.1
HTTP request method - GET
HTTP request - /index.html
HTTP request size in bytes - 15824
HTTP response time - 0.043

www.cybexer.com 96
Graylog - advanced setup
Parsed with this GROK rule:
%{IP:client_ip} %{WORD:http_method}
%{URIPATHPARAM:http_request}
%{NUMBER:http_bytes} %{NUMBER:http_duration}

More GROK patterns can be found here:

https://github.com/hpcugent/logstash-
patterns/blob/master/files/grok-patterns

www.cybexer.com 97
Graylog - advanced setup
Let's add GROK pattern to
existing input

In Graylog web interface

navigate to System-
>Inputs

www.cybexer.com 98
Graylog - advanced setup

Find correct input and

click 'Manage extractors'

www.cybexer.com 99
Graylog - advanced setup

Since we don't have any

existing extractors, click
on 'Create extractor'

www.cybexer.com 100
Graylog - advanced setup

GROK extractor must be

created for existing log
event.
Click 'Load Message'

www.cybexer.com 101
Graylog - advanced setup
You should see log message to which we will be
creating GROK extractor

www.cybexer.com 102
Graylog - advanced setup
Open 'Select extractor type' dropdown next to event
and select 'Grok pattern'

www.cybexer.com 103
Graylog - advanced setup

www.cybexer.com 104
Graylog - advanced setup
Enter ip into filter pattern field and click 'Add'

www.cybexer.com 105
Graylog - advanced setup

www.cybexer.com 106
Graylog - advanced setup
Click 'Try against example' and check extractor preview
for results

www.cybexer.com 107
Graylog - advanced setup
Set IP extractor name to 'client_ip'

www.cybexer.com 108
Graylog - advanced setup
Click 'Try against example' and check extractor preview
for results

www.cybexer.com 109
Graylog - advanced setup
Let's examine our Apache server log

192.168.1.5 - - [18/May/2020:16:43:05 +0300] "GET

/gettingstarted HTTP/1.1" 404

First field is IP address of event - 192.168.1.5

With Grok pattern %{IP:client_ip} we were able to
extract IP address into separate field

www.cybexer.com 110
Graylog - advanced setup
Let's go further. Our test event looks like this
192.168.1.5 - - [18/May/2020:16:43:05 +0300] "GET
/gettingstarted HTTP/1.1" 404

We have already extracted IP address field with this Grok

pattern:
%{IP:client_ip}

In event, after IP address, we have static content:

" - - " - it is "space, dash, space, dash, space"

www.cybexer.com 111
Graylog - advanced setup

So, our Grok pattern with IP address and some static

content will look like this (without single quotes):

'%{IP:client_ip} - - '

www.cybexer.com 112
Graylog - advanced setup
Next part of our log is timestamp when event occurred,
which is between square brackets '[' and ']'
192.168.1.5 - - [18/May/2020:16:43:05 +0300] "GET
/gettingstarted HTTP/1.1" 404

Square brackets are also static content in the event, so

we can add them to Grok pattern. Important thing is,
that square brackets are special symbols, and they
must be escaped. Use backslash '\' as escape symbol
'%{IP:client_ip} - - \['

www.cybexer.com 113
Graylog - advanced setup
Next part of our log is timestamp when even occurred,
which is between square brackets '[' and ']'
192.168.1.5 - - [18/May/2020:16:43:05 +0300] "GET
/gettingstarted HTTP/1.1" 404

To correctly parse timestamps, we need to specify time

format, including time-zone.
Timestamp must be a special name 'timestamp'

www.cybexer.com 114
Graylog - advanced setup
In our example time of event looks like this:

18/May/2020:16:43:05 +0300

The Grok extractor will be following

%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:mm
:ss Z}

www.cybexer.com 115
Graylog - advanced setup
Timestamp formats in detail
18/May/2020:16:43:05 +0300

18 - is date. It's numerical and 2 digits

So Grok pattern for date will be 'dd'

www.cybexer.com 116
Graylog - advanced setup
Timestamp formats in detail
18/May/2020:16:43:05 +0300

May - is month. It's alphabetical and 3 letters

So Grok pattern for month will be 'MMM'

www.cybexer.com 117
Graylog - advanced setup
Timestamp formats in detail
18/May/2020:16:43:05 +0300

2020 - is year. It's numerical and 4 digits

So Grok pattern for year will be 'yyyy'

www.cybexer.com 118
Graylog - advanced setup
Timestamp formats in detail
18/May/2020:16:43:05 +0300

Since we have slashes '/' between date, month and

year we have to write them in Grok pattern as well

Final date Grok pattern is following

dd/MMM/yyyy

www.cybexer.com 119
Graylog - advanced setup
Timestamp formats in detail
18/May/2020:16:43:05 +0300

Similar approach is used for time parsing

16:43:05 will be HH:mm:ss

Final position in timestamp is time zone

+0300
Grok has special format for it
Z

www.cybexer.com 120
Graylog - advanced setup
Timestamp formats in detail
18/May/2020:16:43:05 +0300

Now let's put everything regarding timestamp together

dd/MMM/yyyy:HH:mm:ss Z

www.cybexer.com 121
Graylog - advanced setup

Some systems use timestamp with millisecond

precision "18:13:17.861", where .861 are milliseconds.

Grok pattern for this timestamp will be

HH:mm:ss.SSS

www.cybexer.com 122
Graylog - advanced setup
Final Grok pattern for following timestamp
[18/May/2020:16:43:05 +0300]

Will be
\[%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:m
m:ss Z}\]

www.cybexer.com 123
Graylog - advanced setup
• Add timestamp GROK pattern -
\[%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:
mm:ss Z}\]

www.cybexer.com 124
Graylog - advanced setup
Click 'Try against example' and check extractor preview
for results

www.cybexer.com 125
Graylog - advanced setup
Enable 'Named captures only'

www.cybexer.com 126
Graylog - advanced setup
Click 'Try against example' and check extractor preview
for results

Now parsed timestamp must be same as in event

www.cybexer.com 127
Graylog - advanced setup
Add next GROK pattern
• "%{WORD:http_method}

www.cybexer.com 128
Graylog - advanced setup
Click 'Try against example' and check extractor preview
for results

www.cybexer.com 129
Graylog - advanced setup
GROK pattern for parsing apache logs:

• %{IP:client_ip} - -
\[%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:
mm:ss Z}\] "%{WORD:http_method}
%{NOTSPACE:http_request}
HTTP/%{NUMBER:http_version}"
%{NUMBER:http_response}
%{NUMBER:http_bytes}

www.cybexer.com 130
Graylog - advanced setup
GROK pattern for parsing apache logs:

Be sure, you do not have any new lines at the end of GROK
pattern!

www.cybexer.com 131
Graylog - advanced setup
Click 'Try against example' and check extractor preview
for results

www.cybexer.com 132
Graylog - advanced setup
Set the name for extractor and save it by clicking
'Create extractor' button

www.cybexer.com 133
Graylog - advanced setup
New extractor will be listed in 'Configured extractors'

www.cybexer.com 134
Graylog - advanced setup
Now we need to test our new Grok extractor.
Since existing logs cannot be changed, we have to
remove all existing logs from Graylog/ElasticSearch.

Delete all events from Graylog

• curl -XDELETE 'http://localhost:9200/graylog_0'

www.cybexer.com 135
Graylog - advanced setup

In terminal send new test event to Graylog:

• echo '192.168.1.5 - - [18/May/2020:16:43:05
+0300] "GET /gettingstarted HTTP/1.1" 404 507 "-"
"Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/69.0.3112.113 Safari/537.36"' | nc
localhost 5555

www.cybexer.com 136
Graylog - viewing events
Navigate to Search and select 'Search in all messages'

www.cybexer.com 137
Graylog - viewing events
Click green 'search' button

www.cybexer.com 138
Graylog - viewing events
Message will appear in search results

www.cybexer.com 139
Graylog - detailed view
Expand message by clicking on it to see detailed view of
event

www.cybexer.com 140
Graylog - GROK patterns
We have GROK pattern ending with
'%{NUMBER:http_bytes}', but Apache log has more
data after bytes '"-" "Mozilla/5.0 (Windows NT 10.0;
Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/69.0.3112.113 Safari/537.36"'

First part '"-"' is HTTP referrer, which might present in

some logs and its value is quoted. GROK has special
name for quoted strings - QS

www.cybexer.com 141
Graylog - GROK patterns
Grok pattern for '"-"' will be %{QS:http_referrer}

Last part of the Apache log is HTTP User-Agent string,

which is also quoted. Using same %{QS} pattern we
have GROK pattern for User-Agent string

%{QS:http_agent}

www.cybexer.com 142
Graylog - GROK patterns
Full GROK pattern for Apache logs

• %{IP:client_ip} - -
\[%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:
mm:ss Z}\] "%{WORD:http_method}
%{NOTSPACE:http_request}
HTTP/%{NUMBER:http_version}"
%{NUMBER:http_response}
%{NUMBER:http_bytes} %{QS:http_referrer}
%{QS:http_agent}

www.cybexer.com 143
Graylog - GROK patterns
Let's finalize GROK pattern in Graylog

Navigate to System->Inputs and select 'Manage

Extractors' for TCP input

www.cybexer.com 144
Graylog - GROK patterns
Select 'Edit' next to 'Apache log parser'

www.cybexer.com 145
Graylog - GROK patterns
Paste latest GROK pattern. Be sure to remove 'new
lines' from pattern window

And click 'Update extractor'

www.cybexer.com 146
Graylog - GROK patterns
Let's verify final GROK pattern and Apache logs.

Remove all logs from Graylog

• curl -XDELETE 'http://localhost:9200/graylog_0'

Be sure you see '{"acknowledged":true}' in your

terminal window

www.cybexer.com 147
Graylog - GROK patterns
Send test Apache log to Graylog input from terminal

• echo '192.168.1.5 - - [18/May/2020:16:43:05

+0300] "GET /gettingstarted HTTP/1.1" 404 507 "-"
"Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/69.0.3112.113 Safari/537.36"' | nc
localhost 5555

www.cybexer.com 148
Graylog - viewing events
Navigate to 'Search' tab and select 'Search in all
messages'

www.cybexer.com 149
Graylog - viewing events
Click green 'search' button

www.cybexer.com 150
Graylog - viewing events
Message will appear in search results

www.cybexer.com 151
Graylog - detailed view
Expand message by clicking on it to see detailed view of
event

Now HTTP User-Agent string is in 'http_agent' field

www.cybexer.com 152
Graylog - detailed view
Congratulations!

Now you have fully working Grok extractor for Apache

access logs.

Each field of log is extracted and set to own field.

www.cybexer.com 153
Graylog - detailed view

Next, we will feed more Apache logs to Graylog.

www.cybexer.com 154
Graylog - importing logs
Empty Graylog database
• curl -XDELETE 'http://localhost:9200/graylog_0'

www.cybexer.com 155
Graylog - importing logs
Let's download real Apache logs
• wget http://files.csirt.crp/apache_access_log.gz -O
/opt/apache_access_log.gz

Import logs to Graylog (will take some time)

• gunzip -c /opt/apache_access_log.gz | nc localhost
5555

www.cybexer.com 156
Graylog - importing logs
During Apache log import, you will see in Graylog web
interface top-right bar changing numbers

In - number of events being received by TCP input

Out - number of events after being extracted and saved

to Graylog

In/out numbers are changing

www.cybexer.com 157
Graylog - importing logs
Wait until 'in/out' numbers are zeros

Speed of log processing is heavily depending on CPU

and RAM

www.cybexer.com 158
Graylog - viewing events
Navigate to 'Search' tab and select 'Search in all
messages'

www.cybexer.com 159
Graylog - viewing events
Click green 'search' button

www.cybexer.com 160
Graylog - viewing events
All imported messages will appear in search results

www.cybexer.com 161
Graylog - viewing events
By default, Graylog is not showing how many events are
matched the search query.
Let's add generic 'message counter'.
Navigate to 'Search' tab and select '+' sign from left
menu bar and click 'Message Count'

www.cybexer.com 162
Graylog - viewing events
Now you have 'message counter', which shows number
of events matched your search query.
Resize 'Message Count' window by dragging arrow in
right-bottom corner

www.cybexer.com 163
Graylog - viewing events
Let's try how 'Message Counter' works.
Navigate to 'Search' tab and select 'Search in all
messages'
Click green 'search' button

You should see the number

or event which match your
search query

www.cybexer.com 164
Graylog - viewing events
Check number of lines in plain-text Apache log file
• gunzip -c /opt/apache_access_log.gz | wc -l

Number should match the number in 'Message Count'

window

www.cybexer.com 165
Graylog - Search query
language

166
Graylog - search

The search syntax is very close to the Lucene (Google)

syntax. By default, all message fields are included in the
search if you don’t specify a message field to search in.

www.cybexer.com 167
Graylog - search syntax
Linux - this will search for all messages with string
'Linux' in it

www.cybexer.com 168
Graylog - search syntax
Linux windows - this will search for all messages with
string 'Linux' or 'windows' in it

www.cybexer.com 169
Graylog - search syntax
"Linux android" - this will search for all messages
where 'Linux' and 'android' are in message

www.cybexer.com 170
Graylog - search syntax
http_response:404 - this will search for all messages
where http_response field has exactly 404

www.cybexer.com 171
Graylog - search syntax
http_response:(400 OR 405) - this will search for all
messages where http_response field has value 400 or
405

www.cybexer.com 172
Graylog - search syntax
http_response:"ssh login" - this will search for all
messages where http_response field has exact phrase
"ssh login"

www.cybexer.com 173
Graylog - search syntax
_exists_:my_field - this will display all messages where
field with name 'my_field' exists

NOT _exists_:my_field - this will display all messages

where field with name 'my_field' does not exist

www.cybexer.com 174
Graylog - search syntax
AND OR operators are case sensitive
• "ssh login" AND source:example.org
• "ssh login" AND NOT source:example.org

Wildcard searches:
• source:*.org
• source:exam?le.org
• source:exam?le.*
NB! set "allow_leading_wildcard_searches = true"

www.cybexer.com 175
Graylog - search syntax
Numerical searches

• http_response_code:[500 TO 504] - inclusive

• http_response_code:{400 TO 404} - exclusive
• bytes:{0 TO 64]
• http_response_code:[0 TO 64}

www.cybexer.com 176
Graylog - search syntax
Numerical searches

• http_response_code:>400
• http_response_code:<400
• http_response_code:>=400
• http_response_code:<=400

Combine numerical searches

• http_response_code:(>=400 AND <500)

www.cybexer.com 177
Extending Graylog

178
Graylog - Geo Locations

Graylog lets you extract and visualize geolocation

information from public IP addresses in your logs

Graylog uses GeoIP (MaxMind) city database

www.cybexer.com 179
Graylog - Geo Locations
Download GeoIP MMDB databases
• mkdir /opt/geoip && cd /opt/geoip/
• wget 'http://files.csirt.crp/GeoLite2-
City_20230530.tar.gz'
• tar -zxf GeoLite2-City_20230530.tar.gz
Copy extracted mmdb file to Graylog folder
• cp GeoLite2-City_20230530/GeoLite2-City.mmdb
/etc/graylog/server/GeoLite2-City.mmdb

www.cybexer.com 180
Graylog - Geo Locations
Download GeoIP MMDB databases
• mkdir /opt/geoip && cd /opt/geoip/
• wget 'http://files.csirt.crp/GeoLite2-
ASN_20230530.tar.gz'
• tar -zxf GeoLite2-ASN_20230530.tar.gz
Copy extracted mmdb file to Graylog folder
• cp GeoLite2-ASN_20230530/GeoLite2-ASN.mmdb
/etc/graylog/server/GeoLite2-ASN.mmdb

www.cybexer.com 181
Graylog - enable GeoIP location
Next step, is to configure GeoIP Processor.
Navigate to System->Configurations, scroll to end of
page and click 'Edit configuration' on Geo-Location
Processor block.

www.cybexer.com 182
Graylog - enable GeoIP location
Enable Geo-Location processor, check path to GeoIP
database and click 'Update configuration'

www.cybexer.com 183
Graylog - enable GeoIP location
Now GeoIP support for Graylog is enabled

www.cybexer.com 184
Graylog - enable GeoIP location
Graylog uses different 'Message Processors' on all
incoming logs.
Order of message processors is important, and since
GeoIP Resolver is less important than message
processor, we will move it to the end.

www.cybexer.com 185
Graylog - enable geo location
Navigate to System->Configurations and
click 'Edit configuration' in Message Processor
Configuration

www.cybexer.com 186
Graylog - enable geo location
Drag with mouse 'GeoIP Resolver' plugin to the end of
list and click 'Update configuration'

www.cybexer.com 187
Graylog - enable geo location
GeoIP Resolver must be the last in the list of 'Message
Processors'

www.cybexer.com 188
Graylog - enable geo location

That’s it, at this point Graylog will start looking for fields
containing exclusively an IPv4 or IPv6 address and
extracting their geolocation into a <field>_geolocation
field.

Keep in mind, that GeoIP processor will work only for

new events. Existing events, will not have GeoIP fields

www.cybexer.com 189
Graylog - enable geo location
Delete all data from Graylog
• curl -XDELETE 'http://localhost:9200/graylog_0'

Send apache logs to the Graylog

• gunzip -c /opt/apache_access_log.gz | nc localhost
5555
Sending logs to Graylog will take some time. Wait until
both 'in/out' numbers are zero.

www.cybexer.com 190
Graylog - viewing events
Navigate to 'Search' tab and select 'Search in all
messages'

www.cybexer.com 191
Graylog - viewing events
Click green 'search' button

www.cybexer.com 192
Graylog - view geo data
Click 'Fields' button on left bar

www.cybexer.com 193
Graylog - view geo data
If you see 3 new fields 'client_ip_city_name',
'client_ip_country_code' and 'client_ip_geolocation'
then GeoIP message processing work correctly

www.cybexer.com 194
Graylog - view Geo data

Now, let's create world map visualization.

www.cybexer.com 195
Graylog - viewing events
Navigate to 'Search' tab and select 'Search in all
messages'

www.cybexer.com 196
Graylog - viewing events
Click green 'search' button

www.cybexer.com 197
Graylog - viewing events
Click on plus '+' on left menu bar and then create
generic 'Aggregation'

www.cybexer.com 198
Graylog - viewing events
Click 'Edit' button on new aggregation section

www.cybexer.com 199
Graylog - viewing events
Empty aggregation screen will appear

www.cybexer.com 200
Graylog - viewing events
Select 'World Map' in 'Visualization' drop-down

www.cybexer.com 201
Graylog - viewing events
Open 'Metrics' drop-down and 'Function' 'Count'

www.cybexer.com 202
Graylog - viewing events
Open 'Group By' drop-down and in 'Field' pick
'client_ip_geolocation' and set 'Limit' to 100

www.cybexer.com 203
Graylog - viewing events
Hit 'Update preview' button, to see World Map
preview. If map is populated with data, then click
'Apply Changes' to save it.

www.cybexer.com 204
Graylog - viewing events
If preview map is populated with data, click 'Update
widget' to save it.

www.cybexer.com 205
Graylog - viewing events
You can resize World map visualization window by
dragging it from right-bottom corner.

www.cybexer.com 206
Graylog - viewing events
To make visualization window to fit browser's window
size, click on 'Stretch width' icon.

www.cybexer.com 207
Graylog - viewing events
Now you have perfect size window for World map.

www.cybexer.com 208
Graylog - viewing events
Let's now present data in 'pie' view.
Create new 'Aggregation'

www.cybexer.com 209
Graylog - viewing events
Select 'Pie Chart' in 'Visualization' drop-down

www.cybexer.com 210
Graylog - viewing events
Select 'Count' in 'Metrics' drop-down

www.cybexer.com 211
Graylog - viewing events
Select 'client_ip' in 'Group By' drop-down and click
'Update Preview'

www.cybexer.com 212
Graylog - viewing events
You will see a pie chart of client IP addresses.

www.cybexer.com 213
Graylog - data types
Extracted data type can be changed:

%{NUMBER:http_bytes}

vs

%{NUMBER:http_bytes:int}

www.cybexer.com 214
Graylog - data types
Navigate to System->Inputs
Select Manager Extractors and click Edit

www.cybexer.com 215
Graylog - data types
Change %{NUMBER:http_bytes} to
%{NUMBER:http_bytes:int}

www.cybexer.com 216
Graylog - data types
If http response is cached, then 'http_response' byte
field can be '-'. To avoid errors, modify GROK expression
to following:

(?:%{NUMBER:http_bytes:int}|-)

www.cybexer.com 217
Graylog - data types
To be able to search in HTTP response ranges, we have
to make 'http_response' field as integer.
Change %{NUMBER:http_response} to
(?:%{NUMBER:http_response:int}|-) and click 'Update
extractor'.

www.cybexer.com 218
Graylog - data types
Final GROK pattern will be like this:

%{IP:client_ip} - -
\[%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:m
m:ss Z}\] "%{WORD:http_method}
%{NOTSPACE:http_request}
HTTP/%{NUMBER:http_version}"
%{NUMBER:http_response:int}
(?:%{NUMBER:http_bytes:int}|-) %{QS:http_referrer}
%{QS:http_agent}

www.cybexer.com 219
Graylog - data types
Empty Graylog database
• curl -XDELETE 'http://localhost:9200/graylog_0'

Import logs to Graylog (will take some time)

• gunzip -c /opt/apache_access_log.gz | nc localhost
5555

www.cybexer.com 220
Graylog - viewing events
Navigate to 'Search' tab and select 'Search in all
messages'

www.cybexer.com 221
Graylog - viewing events
Click green 'search' button

www.cybexer.com 222
Graylog - viewing events
Let's now calculate amount of traffic requested from
top 15 IP addresses.
Create new aggregation.

www.cybexer.com 223
Graylog - viewing events
Select 'client_ip' in 'Group By' drop-down.
Set Limit to 15.

www.cybexer.com 224
Graylog - viewing events
Select 'Sum' in 'Metrics' drop-down.
Set 'Field' to 'http_bytes' and click 'Update Preview'

www.cybexer.com 225
Graylog - viewing events
A table will show sum of bytes for each IP address.

www.cybexer.com 226
Graylog - Streams

• The Graylog streams are a mechanism to route

messages into categories in real-time while they are
processed.

• A message will be routed into every stream that has

all (or any) of its rules matching. This means that a
message can be part of many streams and not just
one.

www.cybexer.com 227
Graylog - Streams
Navigate to 'Streams' tab and click 'Create Stream'

www.cybexer.com 228
Graylog - Streams
Set 'Title' and 'Description' for new Stream.
Then 'Create stream' it.

www.cybexer.com 229
Graylog - Streams
Click on 'Manager Rules' next to new stream

www.cybexer.com 230
Graylog - Streams
Click on 'Add stream rule'

www.cybexer.com 231
Graylog - Streams
Fill out new stream rule field, type and value and save
the stream rule.

www.cybexer.com 232
Graylog - Streams
To save stream rules - click 'I'm done!'

www.cybexer.com 233
Graylog - Streams
By default, new streams are disabled. To enable the
stream - click 'Start Stream'

www.cybexer.com 234
Graylog - Streams
Now, If new incoming Apache logs have '404' in HTTP
response field, Stream rule will match and copy event
to corresponding stream.

www.cybexer.com 235
Graylog - alerts
• Alerts are always based on streams. You can define
conditions that trigger alerts.

• For example, whenever the stream All production

exceptions has more than 50 messages per minute or
when the field milliseconds had a too high standard
deviation in the last five minutes.

www.cybexer.com 236
Graylog - dashboards

Using dashboards allows you to build pre-defined views

on your data to always have everything important just
one click away.

Any search, graph can be linked to dashboards.

www.cybexer.com 237
Graylog - dashboards

www.cybexer.com 238
Filebeat
Filebeat is a log shipper belonging to the Beats family
— a group of lightweight shippers installed on hosts for
shipping different kinds of data into Graylog (or ELK).

www.cybexer.com 239
Filebeat - installation
To install Filebeat, we have to add ElasticSearch's GPG
key to local repository list

• curl -fsSL https://artifacts.elastic.co/GPG-KEY-

elasticsearch | gpg --dearmor -o
/usr/share/keyrings/elastic.gpg

www.cybexer.com 240
Filebeat - installation
Adding ElasticSearch v7 repository
• echo "deb [signed-
by=/usr/share/keyrings/elastic.gpg]
https://artifacts.elastic.co/packages/7.x/apt stable
main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list
After adding new repositories to Ubuntu/Debian
systems, we have to update the list of available files
• apt-get update

www.cybexer.com 241
Filebeat - installation
Install Filebeat from repository
• apt-get install filebeat

www.cybexer.com 242
Filebeat - configuration
Filebeat uses modules to collect and parse log data.
To list available modules, type following command
• filebeat modules list

www.cybexer.com 243
Filebeat - configuration
To add new Filebeat module, run following command
• filebeat modules enable system

You can recheck enabled modules with following

command
• filebeat modules list

www.cybexer.com 244
Filebeat - configuration
The 'system' module collects and parses logs created by
the system logging service of common Unix/Linux
based distributions.

www.cybexer.com 245
Filebeat - configuration
First, let's make backup of default configuration file
• cp /etc/filebeat/filebeat.yml
/etc/filebeat/filebeat.yml.bck

Now, let's view Filebeat's main configuration file

• cat /etc/filebeat/filebeat.yml

www.cybexer.com 246
Filebeat - configuration
Firstly, input block must be configured. It's important to
enable it, set unique ID and correct paths to log files

www.cybexer.com 247
Filebeat - configuration
Final log input configuration for /var/log/syslog logs

www.cybexer.com 248
Filebeat - configuration
Next important part, is to configure 'output' for logs.
Output defines destination of log management system,
which will be receiving logs from Filebeat.
Setup 'output.logstash' option, for testing purposes,
we will use port '5678'

www.cybexer.com 249
Filebeat - configuration
Be sure, you comment out other outputs.
E.g 'output.elasticsearch' and 'hosts' option

www.cybexer.com 250
Filebeat - configuration
Final preparation step is to create corresponding 'input'
in Graylog server.
Login to your Graylog server and navigate
to 'System->Inputs'

www.cybexer.com 251
Filebeat - configuration
In drop-down menu select 'Beats' and click 'Launch
new input'

www.cybexer.com 252
Filebeat - configuration
Fill out 'Title' field and set correct port number '5678'.
And then scroll down and click 'Launch input'

www.cybexer.com 253
Filebeat - configuration
If entered information was correct, then you should see
newly created input in 'running' state

www.cybexer.com 254
Filebeat - configuration
After all configuration changes are done, we must start
Filebeat service
• systemctl restart filebeat
Verify, that Filebeat service is running
• systemctl status filebeat

www.cybexer.com 255
Filebeat - configuration
Open Graylog web interface in your browser and check
recent logs

www.cybexer.com 256
Filebeat - configuration
If you want to add more logs files for processing, then
use following syntax

After modifications, restart Filebeat service

• systemctl restart filebeat

www.cybexer.com 257
Web server log analysis
Goaccess
GoAccess is an open-source real-time web log analyzer
and interactive viewer that runs in a terminal on Linux
systems or through your browser.
It provides fast and valuable HTTP statistics for system
administrators that require a visual server report on the
fly.

www.cybexer.com 259
Goaccess - installation
Goaccess installation can be accomplished in several
ways
1. It is easiest to install GoAccess on Linux using the
preferred package manager of your Linux
distribution. Please note that not all distributions
will have the latest version of GoAccess available.
2. GoAccess can be compiled from the source code.
This approach is less convenient but gives much
better understanding of compiling source code on
Linux systems.

www.cybexer.com 260
Goaccess - installation
Since most of the software is available on Github, it's
considered the best source to get latest versions of
software.
Before downloading any software from vendor sites, it's
advised to have dedicated folder to that
• cd /opt/

www.cybexer.com 261
Goaccess - installation
Let's fetch latest version of Goaccess from Github
• git clone https://github.com/allinurl/goaccess

www.cybexer.com 262
Goaccess - installation
Once software is downloaded, move to new folder
• cd goaccess/
Install required software
• apt-get -y install dh-autoreconf automake gcc
libncursesw5-dev libgeoip-dev libmaxminddb-dev
libssl-dev gettext make gcc autoconf automake

www.cybexer.com 263
Goaccess - installation
Rerun generation of configuration scripts for 'goaccess'
• autoreconf -fiv

Note. You may ignore warnings but pay attention to

output errors.

www.cybexer.com 264
Goaccess - installation
Prepare for compilation
• ./configure --with-openssl --enable-utf8 --enable-
geoip=legacy

www.cybexer.com 265
Goaccess - installation
If all dependencies are met and no errors occurred,
then you should see your build configuration

www.cybexer.com 266
Goaccess - installation
Start building
• make

No errors should be shown at the end of output.

www.cybexer.com 267
Goaccess - installation
Install compiled software
• make install

www.cybexer.com 268
Goaccess - installation
Verify installed software and its version
• goaccess -V

www.cybexer.com 269
Goaccess - installation
Install web server
• apt-get -y install apache2

www.cybexer.com 270
Goaccess - installation
Let's fetch some web server logs
• mkdir /opt/logs ; cd /opt/logs
• wget http://files.csirt.crp/access.log
• wget http://files.csirt.crp/access.log.1
• wget http://files.csirt.crp/access.log.2.gz
• wget http://files.csirt.crp/access.log.3.gz

www.cybexer.com 271
Goaccess - usage
First run
• goaccess access.log --log-format=COMBINED --
output=/var/www/html/logs_report.html

Start Apache web server

• service apache2 start
Now open log report in your browser
http://10.XX.YY.2/logs_report.html

www.cybexer.com 272
Goaccess - usage
Let's see what log files we have
• ls -la /opt/logs

www.cybexer.com 273
Goaccess - usage
With several non-compressed log files, you can run
following command:
• cat access.log access.log.1 | goaccess - --log-
format=COMBINED --
output=/var/www/html/logs_report.html

www.cybexer.com 274
Goaccess - usage
If you want to parse simultaneously compressed and
non-compressed log files, then run following:
• ( zcat access.log.*.gz && cat access.log.1 && cat
access.log ) > /tmp/biglog ; cat /tmp/biglog |
goaccess - --log-format=COMBINED --
output=/var/www/html/logs_report.html

www.cybexer.com 275
Log analysis in bash
Very often, you may not have any log management
systems available, and you need to do quick log
analysis.
You can use native Linux 'bash' commands/tools for
that.

www.cybexer.com 276
Log analysis in bash
Extract all User-Agents and sort by count:
• awk -F\" '{print $6}' access.log | sort | uniq -c | sort
-n

www.cybexer.com 277
Log analysis in bash
Sort by number of HTTP response codes:
• awk '{print $9}' access.log | sort | uniq -c | sort -k 2

www.cybexer.com 278
Log analysis in bash
Display all '404' resources sorted by count
• grep " 404 " access.log | cut -d ' ' -f 7 | sort | uniq -c
| sort -n

www.cybexer.com 279
Log analysis in bash
Another way to show 25 top '404' resources sorted by
count
• cat access.log | awk '($9 ~ /404/)' | awk '{ print $7
}' | sort | uniq -c | sort -rn | head -n 25

www.cybexer.com 280
Log analysis in bash
Display top 25 IP addresses
• cat access.log | awk '{ print $1 }' | sort | uniq -c |
sort -rn | head -n 25

www.cybexer.com 281
Log analysis in bash
To display country/region if IP address, you must install
IP lookup command line tool 'GeoIP'
• apt-get -y install geoip-bin

www.cybexer.com 282
Log analysis in bash
Show top 25 IP addresses with their country codes
• cat access.log | awk '{ print $1 }' | sort | uniq -c |
sort -rn | head -n 25 | awk '{ printf("%5d\t%-15s\t",
$1, $2); system("geoiplookup " $2 " | cut -d \\: -f2
") }'

www.cybexer.com 283
Log analysis in bash
Show HTTP request by days (sort by count)
• awk '{print $4}' access.log | sed 's/\[//' | cut -d: -f1
| uniq -c | sort -n

www.cybexer.com 284
Log analysis in bash
Show top 25 most requested URLs
• cat access.log | awk '{ print $7 }' | sort | uniq -c |
sort -rn | head -n 25

www.cybexer.com 285
Log analysis in bash
If you have compressed log file, you can use 'zcat'
command to view/analyze its content
• zcat access.log.2.gz | awk -F\" '{print $6}' | sort |
uniq -c | sort -n

www.cybexer.com 286
Log analysis in bash
You may have compressed (archived logs) and
uncompressed files in the folder
• ls -la

www.cybexer.com 287
Log analysis in bash
You can use following command to parse data from
compressed and uncompressed files
• ( zcat access.log.*gz && cat access.log && cat
access.log.1 ) | awk '{ print $1 }' | sort | uniq -c |
sort -rn | head -n 25

www.cybexer.com 288