Command Line Compromise

Detection for Linux

Craig H. Rowland
@CraigHRowland @SandflySecurity
www.sandflysecurity.com
Introduction

echo “Don’t Panic.”

www.sandflysecurity.com
 Simple First
“Common problems are surprisingly common.”

• Don’t worry about Advanced Persistent Threats

(APT)

• Worry about Commonly Run Attacks Preferred

(CRAP)

• Spotting common problems allows you to find

advanced attackers.

www.sandflysecurity.com
 1000:1 Rule
Defenders need to know thousands of ways for a
system to get compromised. Attackers need to be
right just once.

Attackers need to know about thousands of ways to

cover their tracks. Defenders need to spot something
wrong just once.

www.sandflysecurity.com
The Big Five
Processes
Directories
Files
Users
Logs

www.sandflysecurity.com
Suspicious Processes

www.sandflysecurity.com
 Suspicious Processes
• Named to look legit or very odd.
• Network activity you don’t recognize.
• High CPU/RAM.
• Deleted binary still running.
• Combination of the above.

www.sandflysecurity.com
 Suspicious Processes
Unusual Ports
root@ubuntu18-dirty:/lib# netstat -nalp TCP port 22222
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID
tcp 0 0 0.0.0.0:22222 0.0.0.0:* LISTEN 10580/cron
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1293/sshd
tcp 0 332 192.168.1.122 120.136.1.1:56639 ESTABLISHED 11022/2
tcp6 0 0 :::22 :::* LISTEN 1293/sshd
udp 0 0 0.0.0.0:555 0.0.0.0:* 32481/t
raw 0 0 0.0.0.0:1 0.0.0.0:* 7 10580/cron

Raw socket ICMP Protocol PID “cron”

www.sandflysecurity.com
 Suspicious Processes
Investigating a PID
root@ubuntu-18-dirty:/# ls -alR /proc/*/cwd 2> /dev/null | grep tmp
lrwxrwxrwx 1 root root 0 Nov 14 02:07 /proc/10580/cwd -> /tmp

PID of Running
interest from /tmp
root@ubuntu-18-dirty:/tmp# ls -alR /proc/*/exe 2> /dev/null | grep deleted
lrwxrwxrwx 1 root root 0 Nov 13 07:39 /proc/10580/exe -> /usr/bin/perl (deleted)

Binary deleted but still running?

www.sandflysecurity.com
 Suspicious Processes
Investigating a PID
PID of interest
root@ubuntu-18-dirty:~# ls -al /proc/10580/cwd Running
lrwxrwxrwx 1 root root 0 Nov 14 02:07 /proc/10580/cwd -> /tmp
root@ubuntu-18-dirty:/tmp# cat /proc/10580/comm from /tmp
/usr/sbin/cron Says it’s cron.
root@ubuntu-18-dirty:/tmp# cat /proc/10580/cmdline
/usr/sbin/cron
root@ubuntu-18-dirty:/tmp# ls -al /proc/10580/exe
Really, I’m cron!
lrwxrwxrwx 1 root root 0 Nov 13 07:39 /proc/10580/exe -> '/usr/bin/perl (deleted)'

You’re not cron.

www.sandflysecurity.com
Suspicious Directories

www.sandflysecurity.com
 Suspicious Directories
• Used to hide malicious binaries.

• Used to hide stolen data.

• Used to holding staging data for further incursion.

• Used to hold persistence mechanisms.

www.sandflysecurity.com
 Suspicious Directories
Commonly Targeted
/tmp, /var/tmp /lib*, /usr/lib*

/dev /etc

/dev/shm /var/log

/bin /var/spool

/sbin public_html

/usr/bin Privileged home directories

/usr/sbin Standard home directories

www.sandflysecurity.com
 Suspicious Directories
Weird Names
root@ubuntu18-dirty:~# ls -al /bin
total 17120
drwxr-xr-x 2 root root 4096 Jul 25 21:45
drwxr-xr-x 2 root root 4096 Sep 7 09:52 .
drwxr-xr-x 10 root root 12288 Sep 7 09:52 .
drwxr-xr-x 2 root root 4096 Mar 25 2017 .
drwxr-xr-x 2 root root 4096 Mar 25 2017 . .
drwxr-xr-x 24 root root 4096 Oct 11 04:01 ..
drwxr-xr-x 2 root root 4096 Jun 4 01:56 ..
drwxr-xr-x 2 root root 4096 Jun 4 02:25 ...
drwxr-xr-x 2 root root 4096 Jun 7 00:46 ..%
-rwxr-xr-x 1 root root 1037528 May 16 12:49 bash
-rwxr-xr-x 1 root root 520992 Jun 15 23:46 btrfs
-rwxr-xr-x 1 root root 249464 Jun 15 23:46 btrfs-calc-size

www.sandflysecurity.com
 Suspicious Directories
Weird Names

}
root@ubuntu18-dirty:~# ls -al /bin
total 17120
drwxr-xr-x 2 root root 4096 Jul 25 21:45
drwxr-xr-x 2 root root 4096 Sep 7 09:52 .
drwxr-xr-x 10 root root 12288 Sep 7 09:52 .
drwxr-xr-x 2 root root 4096 Mar 25 2017 .
drwxr-xr-x 2 root root 4096 Mar 25 2017 . . What is this?
drwxr-xr-x 24 root root 4096 Oct 11 04:01 ..
drwxr-xr-x 2 root root 4096 Jun 4 01:56 ..
drwxr-xr-x 2 root root 4096 Jun 4 02:25 ...
drwxr-xr-x 2 root root 4096 Jun 7 00:46 ..%
-rwxr-xr-x 1 root root 1037528 May 16 12:49 bash
-rwxr-xr-x 1 root root 520992 Jun 15 23:46 btrfs
-rwxr-xr-x 1 root root 249464 Jun 15 23:46 btrfs-calc-size

www.sandflysecurity.com
 Suspicious Directories
Weird Names
root@ubuntu18-dirty:~# ls -lap /bin “space”
total 17120 “space” dot
drwxr-xr-x 2 root root 4096 Jul 25 21:45 /
drwxr-xr-x 2 root root 4096 Sep 7 09:52 ./ dot “space”
drwxr-xr-x 10 root root 12288 Sep 7 09:52 ./
drwxr-xr-x 2 root root 4096 Mar 25 2017 . / dot dot “space”
drwxr-xr-x 24 root root 4096 Oct 11 04:01 ../ Trying to look
drwxr-xr-x 2 root root 4096 Jun 4 01:56 .. / legit
drwxr-xr-x 2 root root 4096 Jun 4 02:25 .../
drwxr-xr-x 2 root root 4096 Jun 7 00:46 ..%/ Special
-rwxr-xr-x 1 root root 1037528 May 16 12:49 bash characters
-rwxr-xr-x 1 root root 520992 Jun 15 23:46 btrfs
-rwxr-xr-x 1 root root 249464 Jun 15 23:46 btrfs-calc-siz

www.sandflysecurity.com
 Suspicious Directories
Hidden Directories
root@ubuntu-18:/# find / -type d -name ".*”
/root/.local
/root/.ssh
/lib/modules/4.15.0-34-generic/vdso/.build-id
/bin/. .
/dev/.blKb
/dev/shm/. . All suspicious!

www.sandflysecurity.com
Suspicious Files

www.sandflysecurity.com
 Suspicious Files
• Exploit traces left behind from attack.

• Files that are not what they claim to be or are out of place.

• Binaries that are modified or in strange locations.

www.sandflysecurity.com
 Suspicious Files
Exploit Traces Left Behind
root@ubuntu-18-dirty:~# ls -al /tmp Left by poorly
total 44 written or
rwxrwxrwt 8 root root 12288 Sep 5 00:12 .
drwxr-xr-x 23 root root 4096 Sep 5 00:03 .. crashed
drwxrwxrwt 2 root root 4096 Sep 5 00:03 .font-unix
drwxrwxrwt 2 root root 4096 Sep 5 00:03 .ICE-unix
log cleaner.
drwxrwxrwt 2 root root 4096 Sep 5 00:03 .Test-unix
-rw-r--r-- 1 root root 2304 Sep 5 00:12 utmp.bak
drwxrwxrwt 2 root root 4096 Sep 5 00:03 .X11-unix
drwxrwxrwt 2 root root 4096 Sep 5 00:03 .XIM-unix

www.sandflysecurity.com
 Suspicious Files
Immutable Files

root@ubuntu-18-dirty:/tmp# lsattr / -R 2> /dev/null | grep "\----i"

----i---------e--- /tmp/.t
----i---------e--- /bin/pss Suspicious
name
and immutable in
/tmp.

Binaries are not normally

set immutable.

www.sandflysecurity.com
 Suspicious Files
File Masquerading
root@ubuntu-18-dirty:~/public_html# file * -p
1.jpg: ELF 32-bit LSB executable, ARM, …statically linked, stripped
2.jpg: ELF 32-bit LSB executable, ARM, …statically linked, stripped
3.jpg: ELF 32-bit MSB executable, MIPS, …statically linked, stripped
4.jpg: ELF 32-bit LSB executable, MIPS, …statically linked, stripped
index.html: data
logo.jpg: PHP script, ASCII text
logo.png: PNG image data, 32 x 32, 8-bit/color RGBA, non-interlaced

JPGs are ELF executables.

HTML is unknown data.
Another JPG is PHP.

www.sandflysecurity.com
 Suspicious Files
Hidden Binaries

root@ubuntu-18-dirty:/tmp# find / -name ".*" -exec file -p '{}' \; | grep ELF

/var/tmp/.ICE-unix/.db: ELF 64-bit … stripped

Hidden binary in /tmp directory. Why?

www.sandflysecurity.com
 Suspicious Files
Named Pipes
root@ubuntu-18:/# find / -type p
/run/dmeventd-client
/run/dmeventd-server
…
/tmp/f

One character and in a /tmp directory?

www.sandflysecurity.com
 Suspicious Files
System Files Modified
[root@centos-6-2 ~]# rpm -Va | grep ^..5.
SM5....T. c /etc/ssh/sshd_config
S.5....T. c /etc/ssh/ssh_config Manually
S.5....T. c /root/.bashrc
inspect
these.

root@ubuntu16-dirty:/bin# debsums -c
/usr/sbin/nologin

Why was this replaced?

www.sandflysecurity.com
Suspicious Users

www.sandflysecurity.com
 Suspicious Users
History File Weirdness
Should www user have Creation
a history file? Date Anti-forensics

root@ubuntu-18:/# ls -alR | grep .*history

lrwxrwxrwx 1 www www 9 Nov 13 00:23 .bash_history -> /dev/null
-rw------- 1 root root 53083 Nov 12 23:49 .bash_history

www.sandflysecurity.com
 Suspicious Users
SSH Keys

root@ubuntu-18-dirty:/tmp# find / -name authorized_keys

/root/.ssh/authorized_keys User bin with
/bin/.ssh/authorized_keys
/home/jscott/.ssh/authorized_keys SSH key?
/home/www/.ssh/authorized_keys

Do you recognize all users

that should have
SSH credentials?

www.sandflysecurity.com
 Suspicious Users
Scheduler Persistence

root@ubuntu-18:/tmp# crontab -l
* * * * * /tmp/.d >/dev/null 2>&1

Weird name.
Cron script in tmp?

www.sandflysecurity.com
Suspicious Logs

www.sandflysecurity.com
 Suspicious Logs
Audit Log Tampering
High value and frequently targeted files:

/var/log/wtmp - All valid past logins

/var/log/lastlog - Last login for each user
/var/log/btmp - All bad logins
/var/run/utmp - All current logins
/var/log/* - Various logs

www.sandflysecurity.com
 Suspicious Logs
Zero Byte Logs
root@ubuntu16-dirty:~# ls -al /var/log
total 104
drwxrwxr-x 8 root syslog 4096 Oct 24 06:25 . Zero byte
drwxr-xr-x 17 root root 4096 Jul 25 23:18 .. audit logs?
-rw-r----- 1 syslog adm 0 Oct 25 00:55 auth.log
-rw-r----- 1 syslog adm 0 Oct 25 00:55 auth.log.1
-rw-r----- 1 syslog adm 0 Oct 25 00:55 auth.log.2.gz No bad logins?
-rw-rw---- 1 root utmp 0 Oct 25 00:55 btmp
-rw------- 1 root utmp 0 Oct 25 00:55 btmp.1 No kernel
… messages?
-rw-r—— 1 syslog adm 0 Oct 25 00:55 kern.log
Log rotate
-rw-r----- 1 syslog adm 0 Oct 25 00:55 kern.log.1
compressed a
-rw-r----- 1 syslog adm 0 Oct 25 00:55 kern.log.2.gz
zero byte file?
-rw-r--r-- 1 root root 292292 Oct 24 21:09 lastlog
-rw-r----- 1 syslog adm 0 Oct 25 00:55 syslog
-rw-r----- 1 syslog adm 0 Oct 25 00:55 syslog.1 Date/time all
-rw-r----- 1 syslog adm 0 Oct 25 00:55 syslog.2.gz identical?
…
www.sandflysecurity.com
 Suspicious Logs
Null Erased Logins
root@ubuntu16-dirty:~# utmpdump /var/run/utmp
Utmp dump of /dev/stdin
[2] [00000] [~~ ] [reboot ] [~ ] [4.4.0-93-generic ] [0.0.0.0 ] [Tue Sep 05 00:03:17 2017 UTC]
[1] [00053] [~~ ] [runlevel] [~ ] [4.4.0-93-generic ] [0.0.0.0 ] [Tue Sep 05 00:03:22 2017 UTC]
[6] [01391] [tyS0] [LOGIN ] [ttyS0 ][ ] [0.0.0.0 ] [Tue Sep 05 00:03:23 2017 UTC]
[6] [01388] [tty1] [LOGIN ] [tty1 ][ ] [0.0.0.0 ] [Tue Sep 05 00:03:23 2017 UTC]
[7] [01488] [ts/0] [root ] [pts/0 ] [120.136.1.1 ] [120.136.1.1 ] [Tue Sep 05 00:03:57 2017 UTC]
[0] [00000] [ ] [ ][ ][ ] [0.0.0.0 ][ ]

Type 0 (null) Entries are empty. No date.

Someone overwrote this entry with nulls.

www.sandflysecurity.com
 Suspicious Logs
Null Erased Bad Logins
root@ubuntu16-dirty:~# utmpdump /var/log/btmp
Utmp dump of /dev/stdin
[6] [23367] [ ] [cbm ] [ssh:notty ] [13.78.176.165 ] [13.78.176.165 ] [Mon Sep 11 20:52:56 2017 UTC]
[6] [23367] [ ] [cbm ] [ssh:notty ] [13.78.176.165 ] [13.78.176.165 ] [Mon Sep 11 20:52:58 2017 UTC]
[6] [23515] [ ] [cbm ] [ssh:notty ] [13.78.176.165 ] [13.78.176.165 ] [Mon Sep 11 20:55:30 2017 UTC]
[6] [23515] [ ] [cbm ] [ssh:notty ] [13.78.176.165 ] [13.78.176.165 ] [Mon Sep 11 20:55:33 2017 UTC]
[0] [00000] [ ] [ ][ ][ ] [0.0.0.0 ][ ]
[0] [00000] [ ] [ ][ ][ ] [0.0.0.0 ][ ]
[0] [00000] [ ] [ ][ ][ ] [0.0.0.0 ][ ]
[0] [00000] [ ] [ ][ ][ ] [0.0.0.0 ][ ]

Type 0 (null) Entries are empty. No date.

utmpdump works on wtmp, utmp, and btmp

www.sandflysecurity.com
 Suspicious Logs
Null Erased Bad Logins
root@ubuntu16-dirty:~# lastb
cbm ssh:notty 13.78.176.165 Mon Sep 11 20:58 - 20:58 (00:00)
cbm ssh:notty 13.78.176.165 Mon Sep 11 20:58 - 20:58 (00:00)
cbm ssh:notty 13.78.176.165 Mon Sep 11 20:52 - 20:52 (00:00)
cbm ssh:notty 13.78.176.165 Mon Sep 11 20:52 - 20:52 (00:00)
Thu Jan 1 00:00 - 00:00 (00:00)
Thu Jan 1 00:00 - 00:00 (00:00)

These two entries are nulled.

Overwritten after intruder logged in.

www.sandflysecurity.com
 Conclusions
• Focus on simple first.
• Remember the 1000:1 rule works in your
advantage once a host is compromised.

• Look for suspicious processes, directories,

files, users, and logs.

• Simple tools and careful attention can find

many problems.

www.sandflysecurity.com
 Linux Compromise Detection Command Cheat Sheet
The Big Five Directories Users
Processes • Directories • Files • Users • Logs Commonly targeted directories: Find all ssh authorized_keys files:
/tmp, /var/tmp, /dev/shm, /var/run,
 find / -name authorized_keys
Haste makes waste: /var/spool, user home directories Find history files for all uses:
echo "Don't Panic." List and delimit spaces, etc. in names: find / -name .*history
ls -lap History files linked to /dev/null:
Processes List all hidden directories: ls -alR / 2> /dev/null | grep .*history
Large amounts of CPU/RAM: find / -type d -name ".*" | grep null
top List UID 0/GID 0 users:
Process tree: Files grep ":0:" /etc/passwd
ps -auxwf Show all immutable files and directories: Check sudoers file:
Open network ports or raw sockets: lsattr / -R 2> /dev/null | grep "\----i" cat /etc/sudoers and /etc/group
netstat -nalp Find SUID/SGID files: Check scheduled tasks:
netstat -plant find / -type f \( -perm -04000 -o -perm crontab -l
-02000 \) -exec ls -lg {} \;
ss -a -e -i atq
lsof [many options] Files/dirs with no user/group name: systemctl list-timers --all
find / \( -nouser -o -nogroup \) -exec
Deleted binaries still running:
ls -lg {} \; Logs
ls -alR /proc/*/exe 2> /dev/null | grep
deleted List all file types in current dir: Check for zero size logs:
file * -p ls -al /var/log/*
Process command name/cmdline:
cat /proc/<PID>/comm Find executables anywhere, /tmp, /dev, etc.: Dump audit logs:
cat /proc/<PID>/cmdline find / -type f -exec file -p '{}' \; | utmpdump /var/log/wtmp
grep ELF
Real process path: utmpdump /var/run/utmp
Find all named pipes: utmpdump /var/log/btmp
ls -al /proc/<PID>/exe
find / -type p
Process environment: last
Find files modified/created within last day: lastb
cat /proc/<PID>/environ
find / -mtime -1
Process working directory: Logs with binary in them:
Persistence areas: grep [[:cntrl:]] /var/log/*.log
ls -alR /proc/*/cwd
/etc/rc.local, /etc/initd, /etc/rc*.d, /etc/
ls -alR /proc/*/cwd 2> /dev/null | grep modules, /etc/cron*, /var/spool/cron/*
tmp
Package commands to find changed files:
ls -alR /proc/*/cwd 2> /dev/null | grep
rpm -Va | grep ^..5.
dev

debsums -c

www.sandflysecurity.com
@SandflySecurity


2018-11-14
 Links

PDF Linux Compromise Detection Cheat Sheet

www.sandflysecurity.com/blog/compromised-linux-cheat-sheet/

www.sandflysecurity.com
www.sandflysecurity.com