Effectively Using Splunk

(Scenario 2)

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 1
The organization you work for (Wayne Enterprises) is using Splunk as a SIEM solution to
enhance its intrusion detection capabilities. Wayne Enterprises went through a red team
exercise and the red team provided you with technical details about some of their
exploitation activities (a.k.a Tactical Threat Intelligence). Your SOC manager tasked you
with first trying to identify successful exploitation attempts on your own through Splunk.
He then tasked you with translating the provided TTPs into Splunk searches, once the
initial investigation is complete.

Note: This lab is based on the Boss Of The SOC (BOTS) v1 dataset released by Splunk.

The learning objective of this lab, is to learn effective Splunk search writing and how to
translate attacker TTPs into Splunk searches.

Specifically, you will learn how to use Splunk’s capabilities in order to:
 Have better visibility over a network
 Respond to incidents timely and effectively
 Proactively hunt for threats

 Splunk

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 2
  Incident Responder’s Subnet: 172.16.72.0/24

 Splunk: 172.16.72.100:8000

 Connection Type: Web-based

o Use a Chrome or Firefox browser to connect to Splunk's web interface
(http://172.16.72.100:8000), as follows
 Username: admin
 Password: els@nalyst

As already mentioned the red team provided you with technical details about some of their
exploitation activities. Your SOC manager tasked with first trying to identify a successful
exploitation attempt on your own leveraging Splunk.

The red team performed a plethora of exploitation activities. Identifying one is enough to
complete this task.

Hints:

 Start your investigation by focusing on the stream:dns sourcetype. Then, keep

following leads until you identify what actually happened. Curious-looking domain
names are always of interest.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 3
The red team informed you that they used the following TTPs during the exercise.

1. Malicious USB

2. Computer-generated domain names (to speed the domain generation process up)

3. Malicious VBS

4. Mature Ransomware

5. Code Obfuscation

Translate the provided red team TTPs into Splunk searches.

Hints:

1. Removable media can be identified by the existence of drive letters in Sysmon logs
or the existence of the string friendlyname in Windows registry logs

2. https://www.splunk.com/blog/2017/11/03/you-can-t-hyde-from-dr-levenshtein-
when-you-use-url-toolbox.html

3. The CommandLine field of Sysmon logs can help you with that

4. Mature ransomware in addition to attempting to disable system restore try to delete

everything stored in the VSC using the Volume Shadow Copy Service (VSS)

5. Obfuscated code usually involves the execution of an overly long command

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 4
SOLUTIONS

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 5
Below, you can find solutions for every task of this lab. Remember though, that you can
follow your own strategy (which may be different from the one explained in the following
lab.

Once you are logged into Splunk’s web management interface, click the Search &
Reporting application that resides on the Apps column on your left. You should see
something similar to the below.

Before starting your investigation change the time range picker to All time.

Always identify the available sourcetypes before you begin your investigation. You can do
that as follows.

| metadata type=sourcetypes index="botsv1"

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 6
You should see the below.

As you can see, Splunk has ingested Windows event logs, Sysmon logs, Fortigate UTM logs,
Suricata logs etc.

If you want better granularity regarding the available sourcetypes, submit the search
below.

| metadata type=sources index="botsv1"

Notice that searches leveraging metadata are executed almost instantaneously.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 7
You should now see something similar to the below.

The results between the last two searches are the same. The second search will provide
you with a little more detail about the available sourcetypes.

If you look carefully enough you will notice that the firstTime, lastTime and recentTime
entries follow the epoch time representation. To convert epoch time to a human
understandable representation submit the following search.

| metadata type=sources index="botsv1" | convert ctime(firstTime) as

firstTime | convert ctime(lastTime) as lastTime | convert
ctime(recentTime) as recentTime

You should see the following.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 8
In case you want to identify all the available hosts in the dataset before you start your
investigation, you can do that through the following search.

| metadata type=hosts index="botsv1" | convert ctime(firstTime) as

firstTime | convert ctime(lastTime) as lastTime | convert
ctime(recentTime) as recentTime

You can sort the above by total count to gain a better understanding.

A great sourcetype to start with is stream:dns.

index=botsv1 sourcetype=stream:dns | fieldsummary

The results of the search above may be difficult to read, so create a table that will contain
field and values entries only. You can that by submitting the following search.

index=botsv1 sourcetype=stream:dns | fieldsummary | table field values

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 9
You should see something similar to the below.

You now need to determine which of the available fields is more important. dest could
provide you with useful information, but the most interesting field in these results is
query{}, since it can provide you with information related to interactions with remote (and
possibly malicious) servers.

To better analyze DNS query information, submit the following search.

index=botsv1 sourcetype=stream:dns record_type=A | stats count by

query{} | sort count

You should see something similar to the below.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 10
Going through all queries you will spot some curious-looking domain names. Such a
domain name is cerberhhyed5frqa.xmfir0.win

You can look into this curious-looking domain, as follows.

index=botsv1 sourcetype=stream:dns record_type=A

query{}=cerberhhyed5frqa.xmfir0.win | table _time src dest query{}

You will see the following results.

In the results (sorted by time) above you can see the 192.168.250.100 host making a DNS
query to 192.168.250.20. 192.168.250.20 in turn makes a number of external DNS
queries. From this behavior you can assume that 192.168.250.20 is a DNS server and
192.168.250.100 is probably a compromised machine.

Based on the time included in the results above, you can give 192.168.250.100 a look as
follows.

index=botsv1 sourcetype=stream:dns record_type=A src=192.168.250.100

earliest=08/24/2016:0:0:0 | table _time src dest query{} | dedup
query{}

You should see the following.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 11
Notice that the earliest events are located at the bottom of the table.

The possibly compromised 192.168.250.100 system is looking for isatap and wpad right
after visiting the curious-looking cerberhhyed5frqa.xmfir0.win domain. isatap is related
to IPv6 tunneling and wpad to proxying. This is quite suspicious…

What you should do next is investigate the behavior of the possibly compromised
192.168.250.100 system, by analyzing other logs for approximately the same period of
time as above. Sysmon logs are perfect for this.

First, change the time range picker as follows and click Apply.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 12
Then, submit the following search.

index=botsv1 source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
SourceIp="192.168.250.100"

You should see something similar to the below.

Those two spikes are certainly suspicious.

An important field to check is app.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 13
  The fact that osk.exe is in around 83% of the values is certainly suspicious

 osk.exe is usually related to on screen keyboard. The known osk.exe though doesn’t
reside in the C:\\Users\\<user>\\AppData\\Roaming\\ directory. This is also
suspicious.

 Notice a user named Bob Smith, who is a possible attack victim

You could have also identified this application, as follows.

index=botsv1 source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
SourceIp="192.168.250.100" | stats count by app

osk.exe definitely looks suspicious. So, give it a closer look by simply clicking on it.

You should see something similar to the below.

By inspecting the dest_port field. You will be presented with the below.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 14
  That’s an awful lot of network traffic for an application like on screen keyboard. This
is suspicious.

 Port 6892 corresponds to bit torrent and windows live messenger file transfer,
something also suspicious.

There is only one communication on port 80, click on it to learn more. You should see the
below.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 15
There’s a destination IP in the result 54.148.194.58, which is worth checking, but since
user Bob Smith is most probably a victim of an attack, consult with the available Sysmon
logs to identify what else is running on his machine. You can do that as follows.

index=botsv1 source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
app="C:\\Users\\bob.smith.WAYNECORPINC*"

Inspect the app field once again. You should see the following.

Notice the existence of another curious looking application

C:\Users\bob.smith.WAYNECORPINC\AppData\Roaming\121214.tmp. Give it a look by
clicking on it. You should see the following.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 16
Nothing curious-looking in the results, but there are important fields that could be added to
assist your investigation, such as the CommandLine or the ParentCommandLine one.

Submit the following search to see all the occurrences of 121214.tmp in the Sysmon logs
and also any entry/log that contains ParentCommandLine or CommandLine entries.

index=botsv1 source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
"121214.tmp" AND (ParentCommandLine=* OR CommandLine=*) | table _time
process process_id ParentProcessId ParentImage CommandLine
ParentCommandLine

You should see the following.

The earliest events are at the bottom of the table. If you start from the first (earliest) event
you will see that wscript.exe (parent) called cmd.exe (child). In addition to that, you can see
from ParentCommandLine that wscript.exe executed 20429.vbs.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 17
You can identify more about 20429.vbs by submitting the following search.

index=botsv1 source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
"20429.vbs" AND (ParentCommandLine=* OR CommandLine=*) | table _time
process process_id ParentProcessId ParentImage CommandLine
ParentCommandLine

You should see the following.

This is clearly obfuscated code. User Bob Smith is definitely victim of an attack.

Sysmon logs also contain MD5 hashes. If you would like to learn more about that
121214.tmp file you saw earlier, change time range picker to All time, submit the following
search and inspect the md5 field.

index=botsv1 source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
"121214.tmp"

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 18
You will come across the following.

If you submit the EE0828A4E4C195D97313BFC7D4B531F1 hash on a search engine,

you will identify that you are dealing with Cerber ransomware.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 19
 1. Identify a malicious USB

Removable media can be identified through the following Splunk searches.

index=botsv1 sourcetype="xmlwineventlog:microsoft-windows-
sysmon/operational" "d:\\" | stats count by Computer,CommandLine

You will have to include all possible drive letters. The search above is to test the existence
of a D: drive only.

index=botsv1 sourcetype=winregistry friendlyname | table host object

data

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 20
 2. Identify computer-generated domain names

The following search may uncover computer-generated domain names.

index=botsv1 sourcetype=stream:dns record_type=A | table query{} |

lookup ut_parse_extended_lookup url as query{} | search
ut_domain!=None NOT (ut_domain_without_tld=microsoft OR
ut_domain_without_tld=msn OR ut_domain_without_tld=akamaiedge OR
ut_domain_without_tld=akadns OR ut_domain=nsatc.net OR
ut_domain=quest.net OR ut_domain=windows.com OR ut_domain=arin.net) |
`ut_shannon(ut_subdomain)` | stats count by query{} ut_subdomain
ut_domain ut_domain_without_tld ut_tld ut_shannon | sort - ut_shannon

3. Identify malicious VBS

The following search may identify malicious VBS files

index=botsv1 source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
"*.vbs" AND (ParentCommandLine=* OR CommandLine=*) | table _time
process process_id ParentProcessId ParentImage CommandLine
ParentCommandLine

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 21
 4. Identify mature ransomware activity

The following search can possibly identify mature ransomware activity.

index="botsv1" source="wineventlog:microsoft-windows-
sysmon/operational" EventCode=1 process=*\\vssadmin.exe | search
CommandLine="*vssadmin*" CommandLine="*Delete *"
CommandLine="*Shadows*"

5. Identify code obfuscation

The following search can possibly identify attackers using code obfuscation.

index="botsv1" source="wineventlog:microsoft-windows-
sysmon/operational" | eval len=len(CommandLine) | table User, len,
CommandLine | sort - len

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 2) 22