Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

loggly.com

Analyzing Linux Logs - The Ultimate

Guide To Logging
10-13 minutos

There’s a great deal of information stored within your Linux logs,

but the challenge is knowing how to extract it. There are several
tools you can use to do this, from command-line tools to more
advanced analytics tools capable of searching specific fields,
calculating summaries, generating charts, and much more.

In this section, we’ll show you how to use some of these tools
and how log management solutions like SolarWinds® Loggly®
can help automate and streamline the log analysis process.

Searching With grep

One of the simplest ways to analyze logs is by performing plain

text searches using grep. grep is a command line tool capable
of searching for matching text in a file or output from other
commands. It’s included by default in most Linux distributions
and is also available for Windows and macOS.

To perform a simple search, enter your search string followed by

the file you want to search. Here, we search the authentication
log for lines containing “user hoover”:

$ grep "user hoover" /var/log/auth.log

pam_unix(sshd:session): session opened for user
hoover by (uid=0)

1 de 10 23/3/23, 19:00
Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

pam_unix(sshd:session): session closed for user

hoover

Note this returns lines containing the exact match. This makes it
useful for searches where you know exactly what you’re looking
for.

Regular Expressions

A regular expression (or regex) is a syntax for finding certain

text patterns within a file. Regular expressions are much more
flexible than plain text searches because they let you use
several techniques beyond simple string matching. They allow
for a high degree of control, but constructing an accurate pattern
can be difficult.

For example, let’s say we want to find authentication attempts

on port 4792. Simply searching “4792” would match the port, but
it could also match a timestamp, URL, or another number. In
this case, it matched an Apache log that happened to have 4792
in the URL:

$ grep "4792" /var/log/auth.log

Accepted password for hoover from 10.0.2.2 port
4792 ssh2
74.91.21.46 - - [06/Jun/2019:19:44:32 +0000]
"GET /scripts/samples/search?q=4792HTTP/1.1"
404 545 "-" "-"

To prevent this, we could use a regex only returning instances

of 4792 preceded by “port” and an empty space. We do this
using a technique known as positive lookbehind. Our expression
looks like this (the -P flag indicates we’re using the Perl regular
expression syntax):

2 de 10 23/3/23, 19:00
Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

$ grep -P "(?<=port\s)4792" /var/log/auth.log

Accepted password for hoover from 10.0.2.2 port
4792 ssh2

Surround Search

Using surround search returns several lines before or after a

match. This provides context for each event by letting you trace
the events leading up to or immediately following the event. The
-B flag specifies how many lines to return before the event, and
the -A flag specifies the number of lines after.

For example, let’s search for attempted logins with an invalid

username and show the surrounding results. We see users who
fail to log in also fail the reverse mapping check. This means the
client doesn’t have a valid reverse DNS record, which is
common with public Internet connections. This doesn’t mean
your SSH server is vulnerable, but it could mean attackers are
actively trying to gain access to it.

$ grep -B 3 -A 2 'Invalid user' /var/log

/auth.log
Apr 28 17:06:20 ip-172-31-11-241 sshd[12545]:
reverse mapping checking getaddrinfo for
216-19-2-8.commspeed.net [216.19.2.8] failed -
POSSIBLE BREAK-IN ATTEMPT!
Apr 28 17:06:20 ip-172-31-11-241 sshd[12545]:
Received disconnect from 216.19.2.8: 11: Bye
Bye [preauth]
Apr 28 17:06:20 ip-172-31-11-241 sshd[12547]:
Invalid user admin from 216.19.2.8
Apr 28 17:06:20 ip-172-31-11-241 sshd[12547]:
input_userauth_request: invalid user admin

3 de 10 23/3/23, 19:00
Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

[preauth]
Apr 28 17:06:20 ip-172-31-11-241 sshd[12547]:
Received disconnect from 216.19.2.8: 11: Bye
Bye [preauth]

Tail

Tail is another command line tool capable of displaying the

latest changes from a file in real time. This is useful for
monitoring ongoing processes, such as restarting a service or
testing a code change.

You can also use tail to print the last N lines of a file using
tail -n:

$ tail -n 5 /var/log/messages
Nov 11 23:15:40 rockyLinuxPepperAndEgg
DropletAgent[729407]: INFO:2022/11/11 23:15:40
creds_actioner.go:49: [Creds Actioner]
Attempting to update 1 dotty keys
Nov 11 23:15:40 rockyLinuxPepperAndEgg
DropletAgent[729407]: INFO:2022/11/11 23:15:40
creds_actioner.go:70: [Creds Actioner] Updating
1 keys
Nov 11 23:15:40 rockyLinuxPepperAndEgg
DropletAgent[729407]: INFO:2022/11/11 23:15:40
creds_actioner.go:75: [Creds Actioner] Keys
updated
Nov 11 23:15:41 rockyLinuxPepperAndEgg
systemd[1]: Started Session 4 of user root.
Nov 11 23:15:41 rockyLinuxPepperAndEgg systemd-
logind[790]: New session 4 of user root.

4 de 10 23/3/23, 19:00
Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

Additionally, the output of tail -f can be piped into grep to

filter the output from a log file.

$ tail -f /var/log/auth.log | grep 'Invalid

user'
Apr 30 19:49:48 ip-172-31-11-241 sshd[6512]:
Invalid user ubnt from 219.140.64.136
Apr 30 19:49:49 ip-172-31-11-241 sshd[6514]:
Invalid user admin from 219.140.64.136

If you pipe tail -f into grep and experience significant

delays, appending --line-buffered to grep may help. For
example, our previous command would become the following:

$ tail -f /var/log/auth.log | grep --line-

buffered 'Invalid user'

A full introduction to grep and regular expressions is outside the

scope of this guide, but you can find more resources at the end
of this section.

Cut

The cut command allows you to parse fields from delimited logs.
Delimiters are characters like equal signs or commas that break
up fields or key-value pairs.

Let’s say we want to parse the user from this log:

pam_unix(su:auth): authentication failure;

logname=hoover uid=1000 euid=0 tty=/dev/pts/0
ruser=hoover rhost= user=root

We can use the cut command to get the eighth match. This
example is on an Ubuntu system:

$ grep "authentication failure" /var/log

5 de 10 23/3/23, 19:00
Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

/auth.log | cut -d '=' -f 8

root

Filtering and Parsing With Awk

Filtering allows you to search on a specific field value instead of

doing a full text search. This makes your log analysis more
accurate because it will ignore undesired matches from other
parts of the log message. To search on a field value, you need
to parse your logs or at least have a way of searching based on
the event structure. To do this, we can use awk.

Awk is a powerful command-line tool providing a complete

scripting language, so you can filter and parse out fields more
effectively. For example, let’s say we want to extract the
username from all failed login attempts. Our logs have the
following format:

Mar 24 08:28:18 ip-172-31-11-241 sshd[32701]:

input_userauth_request: invalid user guest
[preauth]

Here’s how you can use the awk command. First, we’ll use the
regular expression /sshd.*invalid user/ to match the sshd invalid
user lines. Then, we’ll print the ninth field using the default
delimiter (a space character) using { print $9 }. This outputs the
usernames.

$ awk '/sshd.*invalid user/ { print $9 }'

/var/log/auth.log
guest

Filtering on Errors With Awk

One of the most common things people want to see in their logs

6 de 10 23/3/23, 19:00
Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

are errors. Unfortunately, the default syslog configuration

doesn’t output the severity of errors directly, making it difficult to
filter on them.

There are two ways you can solve this problem. First, you can
modify your rsyslog configuration to output the severity in the log
file to make it easier to read and search. In your rsyslog
configuration, you can add a template with pri-text:

"<%pri-text%> :
%timegenerated%,%HOSTNAME%,%syslogtag%,%msg%n"

This example gives you an output in the following format. You

can see the severity in this message is “err”:

<authpriv.err> : Mar 11 18:18:00,hoover-

VirtualBox,su[5026]:, pam_authenticate:
Authentication failure

You can use awk to search for just the error messages. In this
example, we’re including some surrounding syntax to match this
field specifically:

$ awk '/.err>/ {print}' /var/log/auth.log

<authpriv.err> : Mar 11 18:18:00,hoover-
VirtualBox,su[5026]:, pam_authenticate:
Authentication failure

Though command-line tools are useful for quick searches on

small files, they don’t scale well for large files or across multiple
systems. Log management systems are much more effective at
searching through large volumes of log data quickly. We’ll
discuss log management systems in the next section.

Searching With Log Management Systems

Log management systems help simplify the process of

7 de 10 23/3/23, 19:00
Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

analyzing and searching large collections of log files. They can

automatically parse common log formats like syslog events,
SSH logs, and web server logs. For example, SolarWinds
Papertrail supports a live tail for all logs in a single consolidated
view.

Log management systems can also index each field so you can
quickly search through gigabytes or even terabytes of log data.
They often use query languages like Apache Lucene to provide
more flexible searches than grep with an easier search syntax
than regex. This can save both time and effort since you don’t
have to create your own parsing logic for each unique search.

For example, here we’re collecting logs from a Debian server

using Loggly, a cloud-based log management service. Here’s an
example log message from sshd, which automatically parses
out the user field:

A failed login attempt parsed by Loggly.

You can also do custom parsing for nonstandard formats. Using

derived fields, we can parse the unparsed portion of the
message by defining its layout. This lets us index each
individual field in the unparsed data instead of treating it as a
single string. For example, we can create a new field called

8 de 10 23/3/23, 19:00
Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

“auth_stage” and use it to store the stage in the authentication

process where the error occurred, which in this example is
“preauth”:

Viewing a derived field in the Loggly Dynamic Field Explorer™.

Filtering on Errors With Log Management

Systems

Log management systems can make it easier to filter on errors

since they automatically parse logs for us. In many cases, you
can simply click on the desired field and enter a value to filter
the resulting logs. For example, we can view errors in Loggly by
clicking on the syslog severity field and selecting “Error”:

9 de 10 23/3/23, 19:00
Analyzing Linux Logs - The Ultimate Guide To Logging about:reader?url=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.loggly.co...

Filtering on syslog messages with severity “Error” in SolarWinds

Loggly.

Additional Resources

Grep and Regular Expressions! (Ryan’s Tutorials)

The GNU Awk User’s Guide (GNU)

Using Grep & Regular Expressions to Search for Text Patterns

in Linux (DigitalOcean)

Last updated: 2022

10 de 10 23/3/23, 19:00