Administering Splunk Enterprise Security

Administering Splunk Enterprise Security

turn data into doing™ 1 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Document Usage Guidelines
• Should be used only for enrolled students
• Not meant to be a self-paced document
• Do not distribute

May 16, 2012

Administering Splunk Enterprise Security
turn data into doing™ 2 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Course Prerequisites
• Prerequisites: – Creating Field Extractions
– What is Splunk? – Enriching Data with Lookups
– Intro to Splunk – Data Models
– Using Fields – Splunk Enterprise System
– Introduction to Knowledge Administration
Objects – Splunk Enterprise Data
– Creating Knowledge Objects Administration

Administering Splunk Enterprise Security

turn data into doing™ 3 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Course Prerequisites (cont.)
• Recommended:
– Scheduling Reports and Alerts
– Search Optimization
– Using Splunk Enterprise Security
– Splunk Enterprise Cluster Administration
– Architecting Splunk Deployments
– Splunk Cloud Administration

Administering Splunk Enterprise Security

turn data into doing™ 4 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Course Goals
• Overview of Enterprise Security (ES)
• Explain how an ES administrator can customize the Security Posture and
Incident Review dashboards
• Examine the ES Risk framework and risk-based alerting information
provided for risk notable events
• Discuss how an ES admin can customize the Investigation Workbench
• Perform initial ES installation and configuration
• Manage data intake and normalization in ES Important!
All labs must be completed for
• Create and tune correlation searches course credit

• Configure ES lookups
• Configure the different ES frameworks
including Assets & Identities and Threat Intelligence
Administering Splunk Enterprise Security
turn data into doing™ 5 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Course Outline
1. Introduction to ES 7. Validating ES Data
2. Security Monitoring 8. Custom Add-ons
3. Risk-Based Alerting 9. Tuning Correlation Searches
4. Incident Investigation 10. Creating Correlation Searches
5. Installation 11. Asset & Identity Management
6. Initial Configuration 12. Threat Intelligence Framework
Appendix A: Analyst Tools & Dashboards
Appendix B: Use Case Library
Appendix C: Event Sequencing Engine
Appendix D: ES On-prem Deployment
Appendix E: Using ES Overview
Administering Splunk Enterprise Security
turn data into doing™ 6 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 1:
Introduction to
Enterprise Security

Administering Splunk Enterprise Security

turn data into doing™ 7 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Review how ES functions
• Understand how ES uses data models
• Configure ES roles and permissions

Administering Splunk Enterprise Security

turn data into doing™ 8 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Flow in Enterprise Security
ES Searches for Threats
Raw Events are and Anomalies
Indexed Data is available for ES
ES creates notable events
| tstats queries and
Data is generated, which are stored in summary
forwarded, and indexed dashboards can now use
indexes and are searchable
into Splunk the data
by data models

Data Model Summary ES Background Searches

Searches Run (content) Process Data
CIM DM normalization is Correlation Searches, trackers, and
applied, CIM DM threat intelligence search data
key/value pairs are stored models
(acceleration) in DM
TSIDX

Administering Splunk Enterprise Security

turn data into doing™ 9 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 ES Data Flow
• Security-related data is acquired by add-ons in your enterprise from
servers, routers, etc.
– This data is forwarded to Splunk indexers and stored as events

Vulnerability Scanners
(port scanning, testing
Firewalls/Proxies Intrusion Detection System
vulnerabilities)
• cisco-pix (packet sniffing)
• mcafee
• pa-networks • snort
• nessus
• juniper-networks • dragon-ids
• bluecoat • mcafee

Production Servers
(any operating system)
Network Capture
(Stream) • microsoft-av
• stream:tcp • linux-secure
• stream:udp • windows:*
• stream:http Splunk ES • access-combined
(events, data models)

Administering Splunk Enterprise Security

turn data into doing™ 10 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Models
• ES depends heavily on accelerated data models
• ES uses the Common Information Model (CIM) that helps you to
normalize your data to match a common standard
• Data models show normalized data
• Acceleration provides a “speedup” factor
• Use | tstats searches with summariesonly = true to search
accelerated data

Administering Splunk Enterprise Security

turn data into doing™ 11 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Models
Splunk Enterprise > Settings > Data models
Tech add-ons normalize events based
on source types which associates
events with specific data models

Administering Splunk Enterprise Security

turn data into doing™ 12 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
tstats Search Example

• ES uses | tstats to create reports based on accelerated data models

– Use | tstats summariesonly=t to restrict results to accelerated data

• Use Search > Datasets to search datasets using ES data models

Administering Splunk Enterprise Security
turn data into doing™ 13 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Looking for Trouble
• ES runs real-time and with scheduled searches on accelerated Data
Model data, looking for indicators of threats, vulnerabilities, or attacks
– If a search discovers something that needs attention, ES displays it on
one or more dashboards
– You can then investigate the issue, track it, analyze it, and take the
appropriate action

Administering Splunk Enterprise Security

turn data into doing™ 14 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Correlation Searches
• Correlation Searches run continually in the background
looking for known types of threats and vulnerabilities such as anomalies
and suspicious/malicious behavior
– There are a number of built-in correlation searches in ES, and more in
the Use Case Library. You can also add your own searches
• When a correlation search detects any Indicators of Compromise (IOC), ES
raises an adaptive response. A frequently used adaptive response is a
notable event also called an incident
• ES enables you to track, update, and resolve incidents
– Security Posture dashboard provides a cross-domain SOC overview
– Incident Review dashboard is used to inspect and manage incidents

Administering Splunk Enterprise Security

turn data into doing™ 15 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Correlation Searches (cont.)
• Correlation searches run either in real-time or on a schedule
• Correlation searches can be modified and extended as needed
• Each search looks for a specific type of threat, vulnerability, or sign of malicious
attack
– Example Correlation Searches
• Activity from Expired User Identity
• Brute Force Access Behavior Detected
• Excessive Failed Logins
• Threat Activity Detected
• Generating a notable event (also referred to as an incident) is a typical AR, others
include sending email, running a script, and updating a risk score

Administering Splunk Enterprise Security

turn data into doing™ 16 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Content
Configure > Content > Content Management
Add new content

Filter by Type, App, Status, or text

Enable, disable, or export

the selected content Enable,
disable or
clone
content

Click a title to edit

Administering Splunk Enterprise Security

turn data into doing™ 17 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Active Correlation Searches
• View which correlation searches are enabled
• By default, only ES Admins can enable, disable, clone, modify, or add new
correlation searches
• Clone a correlation search, make changes, and save as a new search

Filter by Correlation Search and Enabled

Administering Splunk Enterprise Security

turn data into doing™ 18 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Notable Events
• Correlation searches create notable events in the notable index
• Notable events are created with fields, event types, and tags that
provide information necessary for incident investigation and a link to
the original source event(s)
• Search notable events in the notable index
– InES, select Search > Search to run a manual search
– Search index=notable for a given time period
– Event source field shows the correlation search that created the notable
event

Administering Splunk Enterprise Security

turn data into doing™ 19 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Notable Events Example
From ES > Search > Search, run a search for all events in the notable index

The source field shows which correlation

searches generated notable events

Administering Splunk Enterprise Security

turn data into doing™ 20 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Roles
ES Roles (required for ES login)

ES User ES Analyst ES Admin

ess_user ess_analyst ess_admin
Configures ES system-
Runs real-time searches Owns notable events
wide, including adding
and views all ES and performs notable
ES users, managing
dashboards event status changes
correlation searches, and
adding new data sources

User Power Admin

Standard Splunk Roles

Administering Splunk Enterprise Security
turn data into doing™ 21 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Enabling Role Capabilities
• Users should not be added
to the ES Admin role ES > Configure > General > Permissions
• Instead, enable or disable
ES component permissions
for the ess_analyst or
ess_user role

Administering Splunk Enterprise Security

turn data into doing™ 22 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Accessing ES
• Access Splunk Web using a URL similar to:
https://eshostname:8000
• To access ES a user must have an assigned ES
role on the ES server
(ess_admin, ess_analyst, ess_user)
• Once logged on, ES displays in the list of apps on
the Splunk home page
– Apps can be made ”visible” in Manage Apps.
Click Edit Properties for an app, and click the
Yes button for Visible

Administering Splunk Enterprise Security

turn data into doing™ 23 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Setting ES as the Default App
• Users can configure ES to be the default app to open in Splunk Web
– Click the username on the top menu bar and select Preferences

– SelectEnterprise Security from the

Default application drop-down, and
click Apply

Administering Splunk Enterprise Security

turn data into doing™ 24 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 1 Lab: Overview of Splunk ES
Time: 10 minutes
Tasks:
1. Log on to the lab Splunk server and navigate to the ES home page
2. Change the preferences for the administrator (admin) account
3. Examine the source events ES is using to monitor the security
environment and notable events

Administering Splunk Enterprise Security

turn data into doing™ 25 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 2:
Security Monitoring

Administering Splunk Enterprise Security

turn data into doing™ 26 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Customize the Security Posture dashboard
• Customize the Incident Review dashboard
• Create ad hoc notable events
• Suppress notable events

Administering Splunk Enterprise Security

turn data into doing™ 27 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Security Posture Dashboard
• Provides an overview of Notable Events

• Key Indicators (KI)

at the top provide
an at-a-glance view of
notable event status
over the last 24 hours

• The four panels

provide additional summary information categorized by urgency, time,
and most common notable event types and sources
Administering Splunk Enterprise Security
turn data into doing™ 28 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Configuring Key Indicators
• Key Indicators (KIs)
appear in many ES views
• By default, KIs do not have
a threshold set, so the current count is displayed in black
• You can configure thresholds for each KI
– If the count is above the threshold, the value is shown in red
– Green indicates a value below the threshold

• You can also re-order, delete, or add KIs

• Click Edit to display the edit tools
Save Add Cancel

Administering Splunk Enterprise Security

turn data into doing™ 29 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Editing Key Indicators
Save changes
Drag and drop to re-arrange

Remove KI
from display
Select KIs to add and
click Add Indicators

Add a new Key

Indicator

Administering Splunk Enterprise Security

turn data into doing™ 30 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Changing KI Thresholds
• You may want to use different threshold values
– For instance, if you have a very large
organization, you may expect a few
minor security threats per day, and
therefore would want to increase some
of the thresholds above their defaults
• Edit the Key Indicator panel
• Enter a value in the Threshold field
• Save the new panel settings

Administering Splunk Enterprise Security

turn data into doing™ 31 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Incident Review Dashboard
Use charts, filters, and search to
focus on specific notable events
Hide the donut
charts or filters

Expand for Add event(s) to an investigation

details Actions
Notable Events menu

Investigation bar

Administering Splunk Enterprise Security

turn data into doing™ 32 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Notable Event Urgency
• Each notable event has an Urgency field, ranging from
Unknown to Critical
• Urgency is a combination of
two factors:
– Severity
• Based on the severity added to
the notable event by the
correlation search
– Priority
• Assigned to the associated assets or identities—i.e., the server or user
• If more than one asset or identity is involved in a single notable event, the one
with the highest priority determines the urgency
Administering Splunk Enterprise Security
turn data into doing™ 33 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Calculating Urgency
• How urgency values are calculated in notable events by default
• Can be overwritten by modifying asset/identity priority and rank,
correlation search syntax, or Urgency Levels lookup
Event Severity
Informational Unknown Low Medium High Critical
Asset/Identity Priority

Unknown Informational Low Low Low Medium High

Low Informational Low Low Low Medium High
Medium Informational Low Low Medium High Critical
High Informational Medium Medium Medium High Critical
Critical Informational Medium Medium High Critical Critical

Asset/Identity Priority + Event Severity = Urgency

Administering Splunk Enterprise Security
turn data into doing™ 34 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Modifying Urgency Lookup
Configure > Content > Content Management > Managed Lookup > Urgency Levels
• ES Admins can edit the lookup to change the matrix that determines how
correlation severity
and asset/identity
priority combine to
set urgency
– Each row is a
Make changes and
combination of remember to save!

severity and priority

with the result
displaying in the
urgency column

Administering Splunk Enterprise Security

turn data into doing™ 35 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding a New Status Value
ES Admins can define new status values and assign values to different
roles for both notable events and ES investigations
Configure > Incident Management > Status Configuration

Click to add a
New status
End Status designates the label is the
final stage of notable examination

Enable or Disable a
Select a status from the
label from displaying in
Label column to edit
Incident Review

Administering Splunk Enterprise Security

turn data into doing™ 36 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Editing a Status
• Change the name or description of a status
• Check Default Status to
make this label the initial
status of all notable events
• Check End Status to make
this label the final status of
notable events

Administering Splunk Enterprise Security

turn data into doing™ 37 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Editing Status Transitions
• Change which roles can
transition a notable from
the selected status to
another status
• The example shows the
transitions for In Progress
• To restrict users with the
ess_analyst role from
transitioning a notable
to Closed from In Progress,
uncheck the ess_analyst box

Administering Splunk Enterprise Security

turn data into doing™ 38 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding a New Status
New Status
• Select the Status Type. Will this be
for notable events or for
investigations
• Enter a label and description
and configure the status options
and transitions
• Remember to save your changes

Administering Splunk Enterprise Security

turn data into doing™ 39 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Customizing Incident Review
Configure > Incident Management > Incident Review Settings
• Allow Overriding of Urgency – allows
analysts to change notable urgency
(default = on)
• Comments
– Required – requires comments when
changing status (default = off)
– Minimum Length – sets the minimum
length of the comment
• Default Time Range – set the default
time range used in Incident Review
(default is last 24 hours)
Administering Splunk Enterprise Security
turn data into doing™ 40 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Customizing Incident Review (cont.)
Configure > Incident Management >
Incident Review Settings

• Incident Review - Table Attributes

– Add, remove, or reorder columns in
Incident Review
• Incident Review - Event Attributes
– Addor remove fields that display in
notable event details in Incident
Review

Administering Splunk Enterprise Security

turn data into doing™ 41 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Add a Column to Incident Review
• Example: display the src (IP) field to the
right of the Title column
– Under Table Attributes click
+ Add Column
– Enter src for the field and Source
for the label
– Use the double ellipsis to move the
Source field under the Title field
and click Save

Administering Splunk Enterprise Security

turn data into doing™ 42 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Add a Field to Event Attributes
• Example: add a field called
Event Type to events
– Under Event Attributes click
+Add Field
– Enter eventtype for the field
and Event Type for the label
– Click Edit

Important!
The new attribute will only display for
notable events containing the field
referenced here. In this case, the
eventtype field.

Administering Splunk Enterprise Security

turn data into doing™ 43 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add a Field to Event Attributes (cont.)
The new Event Type field is added to notable events containing the
eventtype field

Administering Splunk Enterprise Security

turn data into doing™ 44 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Add a Workflow Action to Incident Review
• Enhances the data available for
a field in Incident Review
• Add a workflow action to a
field’s Action menu to display
workbench specific panel data
– For example, add a workbench panel
called $dest$ Installed OS
for the Destination field
– When selected, $dest$ Installed OS
displays the data for the
workbench_context_os_updates
workbench panel

Administering Splunk Enterprise Security

turn data into doing™ 45 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Workbench Panels
View workbench panels: Configure > Content > Content
Management. Filter Type as Panel and filter on workbench

For example, the search behind the

Click a panel name to view workbench_context_os_updates panel
the search behind the panel

Administering Splunk Enterprise Security

turn data into doing™ 46 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Create a Workflow Action
To apply a workbench panel to
a field’s Action menu, create a Name - for the Workflow action

new Workflow Action Label - How it will appear in the

field Action menu. In this example,
1. From Splunk Enterprise <dest name> Computer Inventory
Settings > Fields > Workflow
Actions Apply only to the following fields – which fields will
display the <dest name> Computer Inventory
2. Click New Workflow Action workflow action in Incident Review

3. Complete the new workflow Fields menus – workflow will appear in

the Action menu for an Incident Review
action as shown notable event as a link

URI –composed of tokens and query parameters. Must include the

name of the workbench panel.
… $&panel=workbench_context_computer_inventory& …

docs.splunk.com/Documentation/ES/latest/Admin/Embeddedworkbench
Administering Splunk Enterprise Security
turn data into doing™ 47 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Using Workflow Actions
From the Destination field Action menu, select
the <dest name> Installed OS link

Drill down on any field in the

table to view the details

Administering Splunk Enterprise Security

turn data into doing™ 48 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Creating Ad hoc Notable Events
• ess_admin and ess_analyst have the capability to create notable events
• Other roles can be given the Create New Notable Events permission
Configure > General > Permissions

• Why create an Ad hoc notable event?

There is an event in Splunk that has not been
detected by a correlation search, but you feel it
should be investigated
Administering Splunk Enterprise Security
turn data into doing™ 49 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Create a Notable Event
Steps:
1. From a Splunk search, expand an event
2. Select Event Actions
3. Select Create notable event
4. Enter the desired data for the
notable event
5. Click Save

Administering Splunk Enterprise Security

turn data into doing™ 50 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Suppressing Notable Events
• Event suppression hides notable events from appearing in
Incident Review
• It does not change count of notable events on Security Posture or
Audit dashboards
• Example suppression: remove events from a group of servers that
have been temporarily misconfigured
• By default, only ES Admins have the ability to suppress notable
events

Administering Splunk Enterprise Security

turn data into doing™ 51 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Permission to Suppress Events
• Grant users with the ess_analyst or ess_user role permission to
create and edit event suppressions

Configure > General > Permissions> Edit Notable Event Suppressions

Administering Splunk Enterprise Security

turn data into doing™ 52 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Administering Splunk Enterprise Security
turn data into doing™ 53 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Suppression Audit
Audit > Suppression Audit

Drill-down on any
dashboard value to view
the search and results

Administering Splunk Enterprise Security

turn data into doing™ 54 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 2 Lab: Monitoring with ES
Time: 30 minutes
Scenario: An expired user account has been detected attempting to
log on to high priority resources
Tasks:
1. Use the Security Posture dashboard
2. Use the Incident Review dashboard to research unauthorized network
access
3. Begin working the issue
4. Resolve the issue

Administering Splunk Enterprise Security

turn data into doing™ 55 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 3:
Risk-Based Alerting

Administering Splunk Enterprise Security

turn data into doing™ 56 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Give an overview of Risk-Based Alerting
• View Risk Notables and risk information on the Incident Review
dashboard
• Explain risk scores and how an ES admin can change an object’s
risk score
• Review the Risk Analysis dashboard
• Describe annotations

Administering Splunk Enterprise Security

turn data into doing™ 57 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Risk Score
• A risk score is a single metric that shows the relative risk of an object
(system, user, or other) in the network over time
• Risk is increased by the adaptive response associated with the
correlation search
• Risk scores simplify the threat investigation process by helping
prioritize suspicious behavior
• ES Admins can configure an object’s risk value:
– by creating an ad-hoc risk score for an object in the Risk Analysis dashboard
– by editing the Risk Analysis response action in a correlation search
– by creating a Risk Factor under Content Management

Administering Splunk Enterprise Security

turn data into doing™ 58 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Why Risk-Based Alerting?
• Address alert fatigue!
• Improve detection of sophisticated threats like low-and-slow attacks that traditional
SIEMs miss
• Seamlessly align to cyber security frameworks like MITRE ATT&CK, Kill Chain, CIS
20, and NIST
• Scale analyst resources to optimize SOC productivity and efficiency

Administering Splunk Enterprise Security

turn data into doing™ 59 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Risk Framework

Create risk rules to create risk Enrich risk attributions by When an entity’s risk score or
attributions for entities when appending relevant context behavioral pattern meets the
something suspicious happens. like a risk score or a MITRE predetermined threshold, a
Instead of triggering an alert, ATT&CK technique notable event is triggered
risk attributions are sent to the
risk index

Administering Splunk Enterprise Security

turn data into doing™ 60 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Risk Rules
• Risk Rules feed results (risk attributions) into the risk index
• Risk Rules are any
correlation search
that has the Risk
Analysis adaptive
response action
configured

Administering Splunk Enterprise Security

turn data into doing™ 61 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Risk Correlation Searches
• Risk Incident Rules are the “risk” correlation searches that run against
the risk index
• Risk Incident Rules create “Risk Notables”
• There are two out-of-the-box Risk Incident Rules
– ATT&CK Tactic Threshold Exceeded for Object Over Previous 7 days
• Creates a notable when the number of MITRE attacks exceeds 3 over the
last 7 days
– Risk Threshold Exceeded for Object Over 24 Hour Period
• Creates a notable when the risk score for an object exceeds 100 over the
last 24 hours
• Create custom Risk Incident Rules to suit your environment
Administering Splunk Enterprise Security
turn data into doing™ 62 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Risk Based Alerting Example
“Risk attributions” along the timeline
are put into the risk index

Risk Incident Rule creates a risk notable

Use Incident Review to
due to the risk score of the user
view the Risk Notable
exceeding 100 over a 24-hour period

Administering Splunk Enterprise Security

turn data into doing™ 63 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Risk Notables

Filter Incident Review to

show only Risk Notables

Fields display risk information for

risk objects

Administering Splunk Enterprise Security

turn data into doing™ 64 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Risk Notable Details

Click Risk Events to

view the details

Click an individual event

for the details

Expand for
details

Administering Splunk Enterprise Security

turn data into doing™ 65 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Risk Objects
• The default risk objects are system, user, and other
• ES Admins can create and edit risk objects to categorize anything as a risk
object so it can be assigned a risk score, for example a file or URL
• To add or change objects, edit the Risk Object Types lookup under
Content Management
Configure > Content> Content Management > Managed Lookup

Administering Splunk Enterprise Security

turn data into doing™ 66 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Changing Risk – Adaptive Response Action
Configure > Content> Content Management
• Change the risk score for an
object in a notable event that
is created by a correlation
search
• From a correlation search,
scroll to the bottom to edit
the Risk Analysis adaptive
response action
• Edit the risk score, object field, A field that exists in the correlation
search to apply the risk score
and object type
The name of an object
• Use the plus sign (+) to add (user, system, or other)

different risk scores to different fields

Administering Splunk Enterprise Security
turn data into doing™ 67 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Changing Risk – Ad-hoc Risk Entry

Add a message

• From the Risk Analysis Add a score (positive or negative)

dashboard, use Enter the object’s name (asset or identity)

Create Ad-Hoc Risk Entry
to change the score of a
specific object Select the object type

• This is a one-time adjustment

• The value entered is added (or
subtracted) to/from the
object’s overall risk score Remember to Save!

Administering Splunk Enterprise Security

turn data into doing™ 68 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Changing Risk - Risk Factors
• Use Risk Factors to specify conditions to dynamically adjust risk
scores for specific objects
– For
example: increase the risk score by a factor of five for a user that is a
contractor
• Risk Factors help the risk score to be more precise based on threat
• Adjust risk scores without creating new searches
• Risk Factor configuration is saved in the risk_factors.conf file of
the SA-ThreatIntelligence app
• Behind the scenes the Risk Analysis data model is updated with
“factor” calculated fields
Administering Splunk Enterprise Security
turn data into doing™ 69 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Risk Factor Example
Configure > Content > Content Management

Find and sort risk factors

Enable the risk factor

View the number of

events that match the
selected risk factor

Create a custom risk factor

Administering Splunk Enterprise Security

turn data into doing™ 70 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Risk Analysis Dashboard

Timeline of most active Annotations effecting risk

risk-increasing events

Object and risk score Risk scores by correlation search

Administering Splunk Enterprise Security

turn data into doing™ 71 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Risk Analysis Data Model
• The Risk data model is the data source
for the panels on the Risk Analysis
dashboard
• Each panel has its own search
• Use the spyglass to view the search
behind the panel

Administering Splunk Enterprise Security

turn data into doing™ 72 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Annotations
Note
• Use annotations to enrich correlation search
Only revise the .csv if you want to
results with the context from industry-standard display non-default info in the
Annotations dropdown field.
mappings
• Used as field labels in the Risk Analysis
dashboard
• Stored in savedsearches.conf under
action.correlationsearch.annotations
For Example:
[Identity - Activity from Expired User Identity - Rule]
action.correlationsearch.annotations =
{"mitre_attack":["T1546.004","T1003.008","T1558.004"]}

• MITRE ATT&CK definitions pre-populated in:

$SPLUNK_HOME/etc/apps/SA-ThreatIntelligence/lookups/
security_framework_annotations.csv

Administering Splunk Enterprise Security

turn data into doing™ 73 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Annotations (cont.)
ES includes the following annotations for common security
frameworks, or you can create custom annotations

Example industry-standard mappings:

Security
Mapping Examples
Framework
Center for Internet
CIS 3, CIS 9, CIS 11, CIS 7, CIS 12
Security (CIS) 20
Kill Chain Reconnaissance, Actions on Objectives, Exploitation, Delivery, Lateral Movement
MITRE ATT&CK T1015, T1138, T1084, T1068, T1085
Also contains MITRE technique IDs from the mitre_attack_lookup lookup definition
National Institute
For Standards and PR.IP, PR.PT, PR.AC, PR.DS, DE.AE
Technology (NIST)

Administering Splunk Enterprise Security

turn data into doing™ 74 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
View Annotation Details

Click an annotation to view

details in the Risk data model

Administering Splunk Enterprise Security

turn data into doing™ 75 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 View Annotation Details (cont.)
Events in the Risk Analysis data model show annotation details

Click “annotation” fields to view details such as

description or which platforms a MITRE
ATT&CK-pattern applies to

Administering Splunk Enterprise Security

turn data into doing™ 76 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Risk Permissions
ES Admins can give ess_analyst or ess_user the ability to edit the Risk Analysis
adaptive response action, or manage Risk Factors by giving the role the following
permissions
Configure > General > Permissions

Ability to edit the Risk

Analysis adaptive
response action

Ability to manage
Risk Factors

Administering Splunk Enterprise Security

turn data into doing™ 77 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 3 Lab: Risk-Based Alerting
Time: 15 minutes
Scenario:
Tasks:
1. Use the Incident Review dashboard to review Risk Notable details
including MITRE ATT&CK tactics and techniques
2. Examine the Risk Incident Rules (correlation searches) creating the Risk
Notables

Administering Splunk Enterprise Security

turn data into doing™ 78 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 4:
Incident Investigation

Administering Splunk Enterprise Security

turn data into doing™ 79 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Review the Investigations dashboard
• Customize the Investigation Workbench
• Give an overview of the Investigation Workbench
• Assign collaborators and update the status of an investigation

Administering Splunk Enterprise Security

turn data into doing™ 80 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Investigations
• By default, only ess_admin and ess_analyst have permission
to start investigations
• ess_analyst can only
manage investigations
they have created
• ess_admin can manage
all investigations
• ess_user cannot view or
manage investigations
• Give permission to roles to manage investigations

Administering Splunk Enterprise Security

turn data into doing™ 81 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Investigations (cont.)
• An investigation has an owner and any number of collaborators
– Owners and collaborators can work and modify the investigation
– Any user that has been granted the manage_all_investigations permission
by an ES admin can add an event to an open investigation, even if they are not
an owner or collaborator on the investigation
• Ways to start an investigation:
– from the Incident Review dashboard Actions menu
– on the Investigations dashboard
– when searching raw events, from the Event Actions menu
– Using the Investigation Bar at the bottom of ES windows

Administering Splunk Enterprise Security

turn data into doing™ 82 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Investigations Dashboard
Lists all investigations

Filter for a specific

investigation
Add investigations

Click an investigation to view its details

on the Investigation Workbench

Administering Splunk Enterprise Security

turn data into doing™ 83 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Investigation Workbench

Time range
Tabs

1
Select
Artifact(s)

2 Expand panel view

Use Explore to add
selected artifacts to the to
the workbench No
te
Workbench will be blank until
artifacts are added using Explore.

Administering Splunk Enterprise Security

turn data into doing™ 84 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Workbench Tabs & Panels

Context Panels Endpoint Data Panels Network Data Panels Risk Panels
• Risk Scores • File System Changes • Web Activity • Risk Scores
• IDS Alerts • Registry Activity • Email Data • Recent Risk Modifiers
• Notable Events • Process Activity • Network Traffic Data • MITRE ATT&CK
Techniques
• System Vulnerabilities • Service Activity • DNS Data
• MITRE ATT&CK
• Latest OS Updates • User Account Changes • Certificate Activity
tactics
• Computer Inventory • Port Activity • Network Session Data
• Authentication Data

Administering Splunk Enterprise Security

turn data into doing™ 85 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add a Tab to the Investigation
• Add other tabs to the investigation
– For example, add the Authentication tab
Content > Add single tab > Select a tab > Authentication
• Imports cloud-authentication-
related notable events into the
investigation
• Displays authentication related
data relevant to the investigation

• By default, the tabs do not persist,

you must add them each time you
view the investigation Click Save

Administering Splunk Enterprise Security

turn data into doing™ 86 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Update Investigation Status
• When you open an
investigation, the 1

status is New
• Investigations can only
be deleted by admins
• Analysts can delete
investigation entries Edit the Title, Status, and
Description of the investigation

Administering Splunk Enterprise Security

turn data into doing™ 87 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add Collaborators to an Investigation
Hover over a collaborator to
view the name, or click to edit

Click to add a
collaborator

Select a user to change

write permissions or remove
as a collaborator

Administering Splunk Enterprise Security

turn data into doing™ 88 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Investigation Overview Dashboard
Audit > Investigation Overview
• Dashboard gives insight into investigations
– Monitoring open investigations
– Shows time to completion
– Displays the number of collaborators

• ES analyst can only see investigations they created by default

• ES users will not see any data Configure > General > Permissions

on the Investigations dashboard

• Give users one of the manage
investigation permissions

Administering Splunk Enterprise Security

turn data into doing™ 89 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Investigation Overview Dashboard (cont.)
The Investigation Overview
dashboard has a lot of
information. Shown here are a
few of the panels

• Oldest Unclosed Investigations

• Total Time Spent on Investigations
• Investigations Unclosed Per Creator
• Investigations Unclosed Per Status

Administering Splunk Enterprise Security

turn data into doing™ 90 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Customizing the Workbench
• ES admins can customize the Investigation Workbench by:
– Creating new types of panels and tabs
– Creating investigation profiles that correspond to specialized
investigation types
– Applying profiles to notable events from correlation searches

• Example:
Create workbench tab named IDS / IPS Activity that displays detailed
information on panels focused on Cisco Sourcefire activity. The tab will
not display by default, analyst will have to add this content to the
investigation manually
docs.splunk.com/Documentation/ES/latest/Admin/Customizeinvestigations
Administering Splunk Enterprise Security
turn data into doing™ 91 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Creating Workbench Panels
• Add pre-defined panels to be used in a new Investigation Workbench
tab
– Follow this process for each panel you want to display in the new tab
Configure > Content > Content Management

Select a pre-defined panel from the

Panel Name drop-down, select
Enterprise Security as the App.

Optionally, define a Label to

replace the default panel title on
the workbench, and Description of
the panel data.

Administering Splunk Enterprise Security

turn data into doing™ 92 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Adding Workbench Panels (cont.)
• Add one or more tokens to the panel Enter a Token Name in the
search to limit the search results to the format of $token$.

artifacts investigated on the

workbench
• Use multiple tokens to substitute more
than one type of artifact
Choose the Type: Notable
Event, Investigation ID, Asset,
Identity, File, or URL.

Add a token to replace the

token in the panel search.

Administering Splunk Enterprise Security

turn data into doing™ 93 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding a Workbench Profile
Add a profile to display a specific set of tabs and panels

Enter a Profile Name. This becomes the stanza

name in es_investigations.conf and is
used as the label if the label is not specified.
Select Enterprise Security as the App.

Optionally, enter a Label if you want the

profile tab to named something different
than the profile name.

Optionally, enter a Description of

the profile.

Administering Splunk Enterprise Security

turn data into doing™ 94 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding a Workbench Tab

Name the tab that will display in the

workbench, and select Enterprise
Security as the App.

Add the Workbench Profile

created previously.

Add the Workbench Panels

to display under the tab.

Set Load By Default to True to have the tab

load in the Investigation Workbench by
default, or False to manually add the tab to
the investigation.

Administering Splunk Enterprise Security

turn data into doing™ 95 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Displaying a Workbench Profile

Click Add Content and select the newly

created profile. The new IDS/IPS Activity tab is
displayed with the configured panels.

Administering Splunk Enterprise Security

turn data into doing™ 96 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 4 Lab: Customizing the Investigation Workbench
Time: 30 minutes
Scenario: Customize the investigation workbench to add a new
workbench tab to view data specific to Cisco Sourcefire
Tasks:
1. Add a tab called IDS / IPS Activity that displays detailed information on
panels focused on Cisco Sourcefire activity
2. View the newly created tab and panels in an investigation

Administering Splunk Enterprise Security

turn data into doing™ 97 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 5:
Installation

Administering Splunk Enterprise Security

turn data into doing™ 98 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Explain the different add-ons and where they are installed
• List ES pre-installation requirements
• Identify steps for downloading and installing ES
• Describe the Splunk_TA_ForIndexers app
and where it is installed

Administering Splunk Enterprise Security

turn data into doing™ 99 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES App and Add-ons

Domain Add-ons (DA)

(views, UI components)

Tech Add-ons (TA)

(input, normalization)
Supporting Add-ons (SA)
(searches, macros,
data models, utilities)
Administering Splunk Enterprise Security
turn data into doing™ 100 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Included Add-ons
These add-ons are distributed with the ES installer and are only required to
be on Splunk search heads. Generally, you will not need to edit any of their
configuration files directly; most settings are available via the admin user
interface
Main ES Application Supporting add-ons
• SplunkEnterpriseSecuritySuite • SA-AccessProtection
• SA-AuditAndDataProtection
Domain add-ons • SA-EndpointProtection
• DA-ESS-AccessProtection • SA-IdentityManagement
• DA-ESS-EndpointProtection • SA-NetworkProtection
• DA-ESS-IdentityManagement • SA-ThreatIntelligence
• DA-ESS-NetworkProtection • SA-UEBA
• DA-ESS-ThreatIntelligence • SA-Utils
• Splunk_SA_CIM
• Splunk_ML_Toolkit

Administering Splunk Enterprise Security

turn data into doing™ 101 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Technology Add-ons
• Technology add-ons (TA’s) can configure inputs on forwarders, parsing on
indexers, and normalizing (CIM compliance) on search heads
• Only the User Behavior Analytics (UBA) add-on Splunk_TA_ueba
is included in the ES install
• Download and install other add-on’s from Splunkbase as needed for the
technologies in your environment (https://splunkbase.splunk.com/)

ES supported TA’s:
– Splunk Add-on for Blue Coat ProxySG – Splunk Add-on for RSA SecurID
– Splunk Add-on for Zeek (Bro) IDS – Splunk Add-on for Sophos
– Splunk Add-on for McAfee – Splunk Add-on for FireSIGHT
– Splunk Add-on for Juniper – Splunk Add-on for Symantec Endpoint Protection
– Splunk Add-on for Microsoft Windows – Splunk Add-on for Unix and Linux
– Splunk Add-on for Oracle Database – Splunk Add-on for Websense Content Gateway
– Splunk Add-on for OSSEC

Administering Splunk Enterprise Security

turn data into doing™ 102 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
What Gets Installed Where?
• Install the full ES app on the search head
– Includes
• DAs, SAs, and UBA TA
• Machine Learning Toolkit (MLTK)
• Install TAs on search head, and on forwarders if they perform input
phase actions
– See TA readme files and configuration files
• Create Splunk_TA_ForIndexers on search head
• Install Splunk_TA_ForIndexers on indexers and heavy forwarders
– Includes
all configurations from all enabled TAs, as well as
indexes.conf settings
Administering Splunk Enterprise Security
turn data into doing™ 103 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Typical Server Architecture
Forwarders
input-time TAs

Indexers & heavy forwarders

ES index configurations
And index-time TA configurations
(via Splunk_TA_ForIndexers.spl)

Universal
Forwarders gather
Search Head(s)
operational and
ES app + all DAs, SAs
security data and
and TAs
send to indexers
or heavy
forwarders

Administering Splunk Enterprise Security

turn data into doing™ 104 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Installation Checklist
• Follow these steps for a single server or distributed (non-clustered) site:
1. Confirm the environment meets the minimum system requirements for
Splunk Enterprise and ES
2. Increase the Splunk Web upload size limit in web.conf (version 6.0.0+)
3. Install ES app on search head
4. Install any required TA’s
5. Create Splunk_TA_ForIndexers and deploy to indexers
6. Deploy input-time technical add-ons (TAs) to forwarders

• If using deployment server to deploy ES-

Important!
installed apps and add-ons, disable it ES 6.0.x is the last major release that is compatible
before the installation, and re-enable after with Python 2 and with MLTK 4.0. ES 6.1+ is
compatible with Python 3 only. ES 6.1+ is compatible
installation with versions of Splunk Enterprise that ship with the
Python 3 interpreter only, as well as MLTK 5.0+ only.

Administering Splunk Enterprise Security

turn data into doing™ 105 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Increase the Splunk Web Upload Size
• The 6.0.0+ installer is larger than the default upload limit for
Splunk Web
• Increase the Splunk Web upload size in web.conf
• Create the $SPLUNK_HOME/etc/system/local/web.conf file and
add the following stanza (requires restart)
[settings]
max_upload_size = 1024

Administering Splunk Enterprise Security

turn data into doing™ 106 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Install ES on a Single Search Head
• Start with a clean basic Splunk installation
• ES functions best without the installation of additional apps on top of
the basic Splunk package

1
Click Install app from file

Important!
Do not uninstall any of the default
apps which are part of the basic
Splunk package, as they are
required by ES.

Administering Splunk Enterprise Security

turn data into doing™ 107 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Upload the ES App
Obtain the ES app from Splunk and upload the file on the designated
ES search head

2
Click Choose File and browse to
the ES .spl or .tar file

3
Click Upload

Note
If upgrading ES, select the Upgrade
app checkbox to install a new
version of ES but keep all
configurations.

Administering Splunk Enterprise Security

turn data into doing™ 108 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Setup the App
Once the file is installed, you are prompted to set up the app

4
Click Set up now

Administering Splunk Enterprise Security

turn data into doing™ 109 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
App Setup
5
Review the notices on the Post-Install
Configuration page and click
Start Configuration Process

6
Before the setup starts,
choose to keep SSL enabled,
or turn off SSL for Splunk

Administering Splunk Enterprise Security

turn data into doing™ 110 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Setup Complete
7
The Configuration Process goes
through the stages of the set up

8
When the process is
complete, Restart Splunk

Administering Splunk Enterprise Security

turn data into doing™ 111 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Splunk Web Restarts
• If SSL is enabled, Splunk Web restarts using an address similar to
https://<ip-address>:8000
– Port stays as default port 8000
– Can be changed in web.conf, or
Settings > Server Settings > General Settings

• If using HTTPS, the pre-loaded SSL certificates are self-signed

– Thiscauses a browser warning, but they are completely secure
– You can install your own externally signed certificates
docs.splunk.com/Documentation/Splunk/latest/Security/Howtogetthird-partycertificates

Administering Splunk Enterprise Security

turn data into doing™ 112 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Is Installed on the Search Head!
• ES also installs:
– Machine Learning Toolkit (MLTK)
– UBA add-on
ES
• The Stream app, if installed,
can be integrated with ES Add-on Builder

• If additional add-ons or apps are MLTK

needed like Splunk Add-on ES Content Update

Builder or ES Content Updates,
Stream App
install them from Splunkbase

Administering Splunk Enterprise Security

turn data into doing™ 113 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Technical Add-Ons
• There are several technical add-ons (TAs) for common security data
sources that can be installed with Enterprise Security
– For a complete list:
docs.splunk.com/Documentation/ES/latest/Install/InstallTechnologyAdd-ons

• Each TA is related to a specific vendor product or technology

• Each has a specific add-on name and one or more event
source types
• Some, like the *NIX, and Windows add-ons, are designed to input
OS data and will require configuration before use
• See the README file in each add-on to for configuration steps

Administering Splunk Enterprise Security

turn data into doing™ 114 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Disable Unused ES Add-ons
• Tech add-ons are intended for use with specific technologies
– For example, Splunk Add-on for Websense, Splunk Add-on for
Zeek (Bro) IDS, Splunk Add-on for Juniper
• If this is an upgrade, there may be add-ons that are no longer required.
They can be disabled under Apps > Manage Apps

Listed are some of the TAs installed on

the system. Notice you can disable
add-ons not being used

Administering Splunk Enterprise Security

turn data into doing™ 115 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Installing ES on a Search Head Cluster
• The installer will dynamically detect if you are installing in a single
search head environment or search head cluster environment
• Install ES on the Deployer
1. On the Splunk toolbar, select Apps > Manage Apps and click
Install app from file
2. Click Choose File and select the Splunk Enterprise Security file
3. Click Upload to begin the installation
4. Click Continue to app setup page
5. Click Start Configuration Process, and wait for it to complete
6. Use the Deployer to deploy ES to the cluster members. From the Deployer run:
splunk apply shcluster-bundle

Administering Splunk Enterprise Security

turn data into doing™ 116 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Configuration Page
Navigate to ES > Configure > All Configurations

Administering Splunk Enterprise Security

turn data into doing™ 117 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Distributed Configuration Management
ES > Configure > General > General Settings

Download Splunk_TA_ForIndexers Download Splunk_TA_AROnPrem

• Creates the Splunk_TA_ForIndexers.spl • Creates the Splunk_TA_AROnPrem
add-on add-on that is used when setting up
• Collects index-time configurations an adaptive response relay from an ES
and basic index definitions into one Cloud SH to an On-prem HF
package to simplify the deployment
of add-on configurations to
on-premises indexers
Administering Splunk Enterprise Security
turn data into doing™ 118 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Deploy Indexer Configurations
• To create the Splunk_TA_ForIndexers.spl, click Download
Splunk_TA_ForIndexers
• Select at least one option and
click Download the Package props.conf and
transforms.conf
– Include index time properties:
adds the props.conf and indexes.conf

transforms.conf files to the

package
– Include index definitions adds the indexes.conf file to the package

• Copy the downloaded .spl file to your indexers

Administering Splunk Enterprise Security

turn data into doing™ 119 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Integrity Control
• An ES Admin can enable data integrity control to ensure that the data ES
relies on in the indexes is not tampered with
– Data integrity applies hashes on indexed data Indexes.conf
[gia_summary]
• Ways to configure data integrity control: enableDataIntegrityControl=true

– In indexes.conf, create a stanza per index

– Settings > Indexes, enable
Data Integrity Check per index
– Re-start Splunk
• Only new inputs will be hashed
• Test integrity from the command line or script:
./splunk check-integrity -index <indexname>

Administering Splunk Enterprise Security

turn data into doing™ 120 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Protection Audit
Audit > Data Protection
• Displays status of data protection settings per index
• Also displays status for sensitive data if the Personally Identifiable
Information Detected correlation search is enabled

Administering Splunk Enterprise Security

turn data into doing™ 121 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Splunk Stream and ES
• ES can use wire data captures from
the Splunk Stream app
– Supports Protocol Intelligence
• Install the Splunk Stream app on the ES server
• Install the Stream add-on (Splunk_TA_stream) on machines where
you want to capture data
• Details on installing and configuring Stream:
docs.splunk.com/Documentation/StreamApp
• Details on integrating Stream with ES:
https://docs.splunk.com/Documentation/ES/latest/Install/IntegrateSplunkStream

Administering Splunk Enterprise Security

turn data into doing™ 122 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Stream Data Flow

Production Servers
with forwarders and
Stream add-on
Capture network data and
forward to indexers
Splunk ES Indexers
With Stream app Store captured
Execute and stream data
display search
results Captured data does not include
message content unless
specifically configured

Administering Splunk Enterprise Security

turn data into doing™ 123 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 5 Lab: Post-installation Tasks
Time: 15 minutes
Tasks:
1. Following an ES upgrade, disable un-needed add-ons
2. Create an app package for your indexer(s) (Splunk_TA_ForIndexers)

Administering Splunk Enterprise Security

turn data into doing™ 124 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 6:
Initial Configuration

Administering Splunk Enterprise Security

turn data into doing™ 125 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Set general configuration options
• Add external integrations
• Configure local domain information
• Customize navigation
• Configure Key Indicator searches

Administering Splunk Enterprise Security

turn data into doing™ 126 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
General ES Configurations
Some settings require that you
Configure > General > Save your changes, some auto-
save when you make changes
General Settings
Filter by app and/or text
• Set or modify various ES
parameters
• For example:
– Enable the Event Sequencing
Engine
– Set the size in bytes for the
Large Email Threshold
– Set HTTP Category Analysis
and HTTP User Agent
Analysis sparkline limits
Administering Splunk Enterprise Security
turn data into doing™ 127 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Configuring Local and Cloud Domains
• Some correlation searches need to differentiate between your local
domain vs. external domains
– For
instance, if you work at Acme Corp, you may have local domains
ending in acme.com, acmecorp.com, etc.
• Also, there are external cloud domains you may use frequently that
are not suspicious
– External vendors for accounting, expenses, document sharing, etc.
• Your email system may use different email domains from your
standard corporate domain
– Due to acquisitions, mergers, etc.

Administering Splunk Enterprise Security

turn data into doing™ 128 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Editing Domain Tables
• Select Configure > Content > Content Management and select
Type: Managed Lookup
• Edit any of the Domain
lookups:
– Corporate Web Domains
– Corporate Email Domains
– Cloud Domains (external
vendor sites) Right-click a row, select Insert row below
and add your domains.
Right-click and delete any sample rows.
Remember to click Save!

Administering Splunk Enterprise Security

turn data into doing™ 129 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Configuring Domain Analysis
• The New Domain Analysis dashboard, Security Intelligence > Web
Intelligence, relies on domain name lookup information retrieved via a
modular input from domaintools.com
1. Add your domaintools.com credentials in the Credentials Manager
2. Configure the settings for the Network Query input
3. Enable whois checking
4. Check for events in the whois index
docs.splunk.com/Documentation/ES/latest/User/ThreatListActivitydashboard#Configure_the_external_API_for_WHOIS_data

Administering Splunk Enterprise Security

turn data into doing™ 130 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding Credentials
• In ES, navigate to Configure > General > Credential Management
and click New Credential
• Enter the domaintools.com credentials,
an app, and click Save

Administering Splunk Enterprise Security

turn data into doing™ 131 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Configuring the whois_domaintools Input
• Select Configure > Data Enrichment >
Whois Management
• Edit the whois_domaintools entry:
– API Host: URI to your account’s server
– API User: domaintools.com username
(password is retrieved from credential
manager automatically)
– App: the app you stored the credentials in
– Leave other fields with default values unless
you have a proxy or want to alter defaults for
queue interval, etc
• Click Save
Administering Splunk Enterprise Security
turn data into doing™ 132 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Enabling the Domain Analysis Setting
• Modify the domain analysis setting Configure > General > General Settings

– Enable Domain Analysis

• The whois system is now enabled

– Domain name lookup happens when events with IP addresses are
indexed
– Domain info is stored in the whois index and used by the New Domain
Analysis dashboard

Administering Splunk Enterprise Security

turn data into doing™ 133 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES and User Behavior Analytics (UBA)
• Splunk User Behavior Analytics (UBA)
is a separate solution that extends your
ability to detect insider threats
– Sendthreats and anomalies from UBA to ES to adjust risk scores
and create notable events

– Send correlation search results from ES to UBA to be processed

for anomalies

– Retrieve user and device association data from UBA to view it in ES

Administering Splunk Enterprise Security

turn data into doing™ 134 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Integrating ES and SOAR
• Install the Splunk App for SOAR
Export (Formerly known as Phantom
App for Splunk) to be able to send
ES events to SOAR using an ES
Adaptive Response Action
– Send to SOAR sends
ES search results to SOAR
– Run Playbook in SOAR
Runs a SOAR playbook
on the ES event

Administering Splunk Enterprise Security

turn data into doing™ 135 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Incident Review KV Store
• All incident review status changes and comments are stored in the
incident_review KV Store collection
• Use the `incident_review` macro to retrieve information from
this lookup
• Example: you are working on a new incident that is similar to one
you worked on before and you want to search for comments related
to the incident
|`incident_review` | search comment = "*...text...*"

Administering Splunk Enterprise Security

turn data into doing™ 136 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Incident Review KV Store Maintenance
• Periodically clear data from the incident review KV Store:
| inputlookup incident_review_lookup
| eval age = (now()-time)/86400 | search age < 30
| fields - age
| outputlookup incident_review_lookup append=f

• Use the splunk clean command to completely clear out the

incident review collection:
splunk clean kvstore -app SA-ThreatIntelligence
-collection incident_review

• Splunk must be running to use splunk clean kvstore

• See additional KV store maintenance commands at:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/BackupKVstore

Administering Splunk Enterprise Security

turn data into doing™ 137 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Untriaged Incident Alert
• The Untriaged Notable Events correlation search can be configured
and customized for your site as needed

• By default, it prepares a list of all notable events in new status or

unassigned owner over the last 48 hours

• Configure its adaptive response actions to send email to a group,

run a script, or create a new notable event with a specific owner
responsible to assign incidents to analysts

Administering Splunk Enterprise Security

turn data into doing™ 138 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Configuration Health Audit
Audit > ES Configuration Health
• Compare the latest
installed version of ES to
prior releases and identify
configuration anomalies
• Useful to check ES status
after initial configuration
or upgrade

Administering Splunk Enterprise Security

turn data into doing™ 139 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Controlling and Customizing Views
• Set permissions on dashboards and
reports to control access
– Onlythe views that the current user
can access are displayed
in navigation
• Clone views and edit
to create custom
alternatives

Administering Splunk Enterprise Security

turn data into doing™ 140 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Customizing ES Menus
Configure > General > Navigation
• Customize menus by adding, removing, or moving items
• Change the default page when logging into ES (default is ES Home)
• Changes affect all users
• Customizing tools:
Edit item
Add Divider

Drag and drop items

to rearrange Make default view when Delete
launching ES item Add View

Administering Splunk Enterprise Security

turn data into doing™ 141 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Navigation Example
Example, move Content Management to the main ES menu

Add a new menu drop-down Revert changes Save

The check mark makes the item
the default view instead in ES

Add new view Drag and drop items to the

or dashboard top to add them to the
main ES menu

Rearrange items using drag

and drop

Administering Splunk Enterprise Security

turn data into doing™ 142 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Adding a Menu Item
Note
Add a New View adds a top-level menu item; to
add an item to a menu, use the icon.

Note
The view list scrolls—there are many to pick from.

Adding links to filtered Incident Review results:

docs.splunk.com/Documentation/ES/latest/Admin/Customizemenubar#Add_a_link_to_a_filtered_view_of_Incident_Review
Administering Splunk Enterprise Security
turn data into doing™ 143 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Edit Navigation Permissions
• By default, only users with the ess_admin role can edit
ES navigation
• Admins can give users with the ess_analyst and ess_user role
the Edit ES Navigation permission

Administering Splunk Enterprise Security

turn data into doing™ 144 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Editing Key Indicator Searches
• Configure > Content > Content Management and select
Type : Key Indicator
• Select a search name to edit indicator search definition
– Click Edit Acceleration to configure an acceleration search schedule

• To make a new Key Indicator search, click

Create New Content > Key Indicator Search

Click to edit

Administering Splunk Enterprise Security

turn data into doing™ 145 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding a Key Indicator Search: 1
• Enter Name, App, Title and Subtitle
Add a search that generates a current and
delta value
• Drilldown URL can be a search,
dashboard or view to open on click
• Add optional acceleration settings

Administering Splunk Enterprise Security

turn data into doing™ 146 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Example Key Indicator Search
• Example Key Indicator search:
– The `get_delta` macro looks for fields current_count and
historical_count and outputs delta
– The two counts should be based on the two previous 24-hour periods
– Use tstats `summariesonly` if possible for performance

| tstats `summariesonly` count as current_count

from datamodel=Risk.All_Risk
where All_Risk.risk_object_type="user" All_Risk.risk_score>60
earliest=-24h@h latest=+0s
| appendcols [|tstats `summariesonly` count as historical_count
from datamodel=Risk.All_Risk
where All_Risk.risk_object_type="user" All_Risk.risk_score>60
earliest=-48h@h latest=-24h@h ]
| `get_delta`

Administering Splunk Enterprise Security

turn data into doing™ 147 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding a Key Indicator Search: 2
• Value contains the current value for
the previous 24-hour period
• Delta contains the difference between
the value for the previous 24-hour
period and the preceding 24-hour
period
• Rendering Options for threshold
coloring, suffix notation, and inversion
• Click Save

Administering Splunk Enterprise Security

turn data into doing™ 148 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 6 Lab: Initial Configuration
Time: 45 minutes
Tasks:
1. Configure Key Indicators and examine the search behind a KI
2. Modify dashboard permissions
3. Customize navigation
4. Review the capabilities of the soc_analyst user account with the ess_user role
5. Create a SOC manager role and give it specific permissions
6. Confirm the capabilities of the new SOC manager account
7. Enable specific ES permissions for the ess_user and ess_analyst roles

Administering Splunk Enterprise Security

turn data into doing™ 149 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 7:
Validating ES Data

Administering Splunk Enterprise Security

turn data into doing™ 150 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Verify data is correctly configured for use in ES
• Validate normalization configurations
• Install additional add-ons

Administering Splunk Enterprise Security

turn data into doing™ 151 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Data Flow
ES uses Splunk events for all correlation and analytical searches using
the following process:
1. Data is input from its source, indexed into events and a
sourcetype is applied
2. Tech add-ons apply normalization configurations based on the source
types that assign the events to a data model
3. The data model events are accelerated and placed into accelerated
storage, with retention periods up to 1 year
4. Most ES correlation searches and dashboard searches are based on
accelerated data model events

Administering Splunk Enterprise Security

turn data into doing™ 152 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
From Input to Dashboard
Normalization
Search Time

Forwarders Parsing/Indexing Data Acceleration

Acceleration
Inputs

TA_ForIndexers

DM
Data Models
HPAS

DA+SA+ ES app
Storage
Index

TA apps
Technology Dashboards
Add-on Unaccelerated DM
Inputs

_raw searches

Notable events & summary indexes

Administering Splunk Enterprise Security

turn data into doing™ 153 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Data Models
• ES uses data models in the Common Information Model (CIM)
docs.splunk.com/Documentation/CIM/latest/User/Howtousethesereferencetables

• Each data model defines a standard set of field names for events
that share a logical context, such as:
– Malware: anti-virus logs
– Performance: OS metrics like CPU and memory usage
– Authentication: log-on and authorization events
– Network Traffic: network activity

• Data models are conceptual maps, not containers

Administering Splunk Enterprise Security

turn data into doing™ 154 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Data Normalization
• Normalization converts non-standard field names and values into a
uniform set of standardized fields within a data model
• Report designers can build report searches based on these standard
terms without knowing where the data originally came from

Example: one sourcetype has events with an ACCESS field, containing numeric
codes like 0 (access allowed) and 1 (access denied). Another sourcetype has an
Action field, with values “allowed” and “denied”. After normalization, both source
types will have the action field with the same values (success or failure), making it
easier to build reports

Administering Splunk Enterprise Security

turn data into doing™ 155 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Normalization Process
• Normalization is a search-time process based on event source types
and includes steps such as:
– Adding tags, which control which events are displayed
by which data models
– Changing field aliases and values to conform to
data model specifications
• Add-ons automatically normalize most common source types
• You may have to adjust normalization rules, or create new
normalization add-ons for custom data

Administering Splunk Enterprise Security

turn data into doing™ 156 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
CIM Setup
ES > Configure > CIM Setup
• Use the CIM add-on to change
data model settings like
acceleration, index whitelist, and
tag whitelist

• Enable acceleration for the data

model to return results faster for
searches, reports, and dashboard
panels that reference the data
model

Administering Splunk Enterprise Security

turn data into doing™ 157 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
CIM Setup (cont.)
ES > Configure > CIM Setup
• Indexes whitelist - improve
performance by constraining the
indexes that each data model
searches (default, is all indexes)
• Tags whitelist - restrict
the tag attribute of a data model
to specific tag values to improve
performance
– By default, whitelists use the tags
for the child datasets in the data
model
Administering Splunk Enterprise Security
turn data into doing™ 158 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Data Input Troubleshooting
• Ideally, after installing ES you will find that all the searches and
dashboards work automatically
• However, if any events have non-standard source types, the
normalization configurations in the tech add-ons won’t work
– Example: an admin created a sourcetype and the name is incorrect
– Fix: specify the correct sourcetype name in your configuration files

• If you have incoming data from a technology that requires a tech

add-on that does not ship with ES, you’ll have to install it
• If you have custom data to use in ES, you might have to create your
own TA (discussed later in the course)
Administering Splunk Enterprise Security
turn data into doing™ 159 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Confirming Normalization
• Match your enabled TAs to CIM data models and verify the events
are being added to the correct data models
– Use the dashboard requirements matrix to determine which data models
support each dashboard
docs.splunk.com/Documentation/ES/latest/User/DashboardMatrix
– Also useful: https://www.splunk.com/en_us/blog/tips-and-tricks/relating-
add-ons-to-cim.html

• If a sourcetype is not showing up in a data model:

– Check the sourcetype
– Make sure the TA is installed

Administering Splunk Enterprise Security

turn data into doing™ 160 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Steps for Initial Data Verification
1. Make a list of all source types required by ES
– This will be dependent on the exact set of technologies and security
products in use at your site
2. Map the sourcetype to the TA that normalizes it
3. Confirm that the correct sourcetype name is being used
– Verify against the TA documentation
4. Install additional TAs if needed
5. Verify that normalization is happening
– Make sure the sourcetype is appearing in the correct data model and
that all searches are executing as expected
Administering Splunk Enterprise Security
turn data into doing™ 161 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Map Source Types to Tech Add-ons
1. Match each sourcetype to the tech add-on that will normalize it
– Use add-on documentation to determine which source types
are supported
docs.splunk.com/Documentation/AddOns
2. Make sure the correct sourcetype name is being set
– Change the sourcetype setting to the correct one, or
– Edit the TA to use the local sourcetype name variant if necessary

3. Install (or create) any missing tech add-ons

4. Disable un-needed ES tech add-ons

Administering Splunk Enterprise Security

turn data into doing™ 162 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Finding More Add-ons
• Splunkbase has additional add-ons available for ES
https://splunkbase.splunk.com/
• Add-ons must be CIM-compliant to be compatible with ES
• Search Splunkbase and/or the add-on documentation for the vendor
or technology names related to the sourcetype you are trying to
normalize

Administering Splunk Enterprise Security

turn data into doing™ 163 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Examining Data Model Contents
• Use the datamodel command to examine the source types
contained in the data model
| tstats count from
datamodel=Network_Traffic.All_Traffic by sourcetype
• If the sourcetype is present, the events are correctly tagged and
fields can be checked for normalization
• If the sourcetype or fields are missing:
– Locate an add-on in Splunkbase that corresponds to the vendor or
technology for the sourcetype, or
– Build your own (discussed later in the course)

Administering Splunk Enterprise Security

turn data into doing™ 164 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Problem: Missing Cisco ASA Events
1. As you audit the data in Splunk, you find that you want to use
events from the Cisco router logs with the cisco:asa source type

2. You confirm the data is present in Splunk indexes, but ES is not

displaying it in any dashboards

3. The Network Traffic data model does not contain events with the
cisco:asa source type
– This
is because the events are not being tagged with the network and
communicate tags, and the fields are not being aliased to the proper
names required in the data model

Administering Splunk Enterprise Security

turn data into doing™ 165 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Solution: the Cisco Add-on
• In Splunkbase, the Splunk Add-on for Cisco ASA is:
– CIM-compliant
– Designed for use with ES
• Source type:
– cisco:asa:
Authentication, Change Analysis, Network Sessions,
Network Traffic, Malware

https://docs.splunk.com/Documentation/CIM/latest/User/NetworkTraffic

Administering Splunk Enterprise Security

turn data into doing™ 166 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Installing Add-ons on the ES Search Head
• Not all add-ons and apps require you to restart Splunk. Check the
add-on documentation on Splunkbase for individual instructions

• Knowledge objects in add-ons and apps that are installed on the

same search head as ES, and are exported to other apps or
exported system-wide (export = system) are automatically
visible in ES

• Check the TA Readme file for specific add-on information

– Ifit indicates it performs index-time actions, re-generate and re-deploy
the Splunk_TA_ForIndexers add-on
– Carry out any additional TA setup in the Readme

Administering Splunk Enterprise Security

turn data into doing™ 167 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Data Model Audit
Audit > Data Model Audit
Determine which data models are using the most storage or processor time

Easily view each data model’s

size, retention settings, and
current refresh status

Administering Splunk Enterprise Security

turn data into doing™ 168 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Forwarder Audit
Audit > Forwarder Audit
• Ensures hosts are properly
forwarding data to Splunk
• Detects forwarders that
have failed
• Can be set to monitor all
hosts, or only hosts
configured as is_expected
in the ES Assets lookup
table

Administering Splunk Enterprise Security

turn data into doing™ 169 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Indexing Audit
Audit > Indexing Audit
Summary of events indexed per day (EPD)

Time series shows trends Time series shows

trends

Summarized by index
(main, threat_activity, etc.)

Administering Splunk Enterprise Security

turn data into doing™ 170 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 7 Lab: Validate ES Data
Time: 25 minutes
Tasks:
1. Plan and verify inputs
2. Examine data model activity
3. Install a new Splunk technology add-on to automatically normalize
Cisco ASA events

Administering Splunk Enterprise Security

turn data into doing™ 171 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 8:
Custom Add-ons

Administering Splunk Enterprise Security

turn data into doing™ 172 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Use custom data in ES
• Create an add-on for a custom sourcetype
• Describe add-on troubleshooting

Administering Splunk Enterprise Security

turn data into doing™ 173 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Custom Data Input
• If you have custom data sources you want ES to recognize, create
an add-on to make your custom events CIM-compliant
• Your add-on should contain:
– Data inputs and parsing (if required)
– Field extractions (if required)
– A tagged event type that maps your sourcetype to the appropriate CIM
data model
– Field aliases to map non-standard field names to CIM field names
– Eval statements (calculated fields) or lookups to map non-standard field
values to CIM field values

Administering Splunk Enterprise Security

turn data into doing™ 174 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Models and the CIM
• Your custom events are referenced by CIM data models
– Seethe CIM documentation for a list of all the data models and their
contents (docs.splunk.com/Documentation/CIM)
• Once you determine which data model should reference your events,
plan which CIM fields relate to your custom fields
• Example:
– You want the Network Traffic data model to return your events
– At docs.splunk.com/Documentation/CIM/latest/User/NetworkTraffic, you
see the list of required and optional fields for this data model
– You make a mapping of your fields to CIM fields

Administering Splunk Enterprise Security

turn data into doing™ 175 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Normalization Strategy
• Not all of the source fields will match CIM fields
– You can ignore the extra source fields as appropriate
• Not all of the CIM fields will be present in the source events
– Use eval statements or regex-based field extractions to generate these
fields with valid values if possible, or with placeholder values if no valid
values can be determined
• Should you populate every CIM field in the target data model?
– You need to at least populate the fields used by ES dashboards and
correlation searches
– Mapping as many of the data model fields as possible will make your
events more robust for future use in new views, searches or reports
Administering Splunk Enterprise Security
turn data into doing™ 176 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Planning Normalization Requirements
• Determine the dashboards that will display your events
• Use the dashboard requirements matrix to determine the data
model(s) and field names the dashboard(s) require:
docs.splunk.com/Documentation/ES/latest/User/DashboardMatrix

Administering Splunk Enterprise Security

turn data into doing™ 177 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Model Definitions
• The data model names in the dashboard requirements matrix are
linked to the data model’s CIM documentation
• Use this documentation to determine the tags, field names and field
values your events must use to be CIM-compliant

Administering Splunk Enterprise Security

turn data into doing™ 178 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Mapping Original Fields to CIM Fields
• Plan your normalization settings Original CIM Procedure
using a table sender src alias

• List the required CIM-compliant receiver dest alias

method app alias
field names user user none

• Match them to corresponding account unused ignore

original source fields missing signature Use eval to create default value
SSID unused regex to mask all but last 4 digits
• Determine if normalization status action Use eval to translate source
numeric codes to CIM terms
is required for each ... ...
field’s name and value

Administering Splunk Enterprise Security

turn data into doing™ 179 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Splunk Add-on Builder
• Very fast way to build out the initial TA
• Use it to create source types,
extractions, and data model mapping
• TAs can:
– Automatically input data into Splunk
– Extract fields and map fields to the CIM
– Create alert actions
https://docs.splunk.com/Documentation/AddonBuilder/latest/UserGuide/Overview

Administering Splunk Enterprise Security

turn data into doing™ 180 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add-on Builder: Getting Started
• Install the Add-on Builder from Splunkbase
• Navigate to the Add-on Builder home page
• Click New Add-on

Administering Splunk Enterprise Security

turn data into doing™ 181 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add-on Builder: Create Add-on
• Enter a name for the add-on
– Thisfield becomes the name of the new app
– The builder adds a TA- prefix

• Add other optional project items

• Click Create
– This
creates a new add-on app on the local
Splunk server
• Your add-on home page is displayed
• You may see a system message to restart Splunk—you can defer
this until done with the new add-on
Administering Splunk Enterprise Security
turn data into doing™ 182 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add-on Builder Home Page
1 2 3
Manage Source Map to Data Validate and
Types Models Package

Note
The Add-on Builder can do a lot of
things, but for CIM normalization you
only need to add sample data and the
CIM mapping function.

Administering Splunk Enterprise Security

turn data into doing™ 183 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add-on Builder: Sample Data
• Select Manage Source Types
– May need to reboot first if add-on is newly created
• If your sample data is already in Splunk,
use Add > Import from Splunk 1

– Select from a sourcetype list and click Save

– You also specify event breaks, time-stamping and other settings

• You can add multiple source types if desired

3

Administering Splunk Enterprise Security

turn data into doing™ 184 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Model Mapping
• Select Map to Data Models
• Click New Data Model Mapping
• Enter a name for the new event type
• Select the source type you are mapping
• Click Save
• The Data Model
Mapping Details
view opens

Administering Splunk Enterprise Security

turn data into doing™ 185 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add-on Builder: Event Types
• Before you can add data model mappings, you must identify your
sourcetype(s) with an event type
– This
is used to generate the correct tags for your events to match the
CIM target data model’s constraints
• On the Data Model Mapping Details page, each sourcetype you
added in the sample data must map to one event type
– More than one sourcetype can map to the same event type
• You can also add search criteria to filter out unwanted events from
your data model mapping
– This excludes the events from the data model acceleration

Administering Splunk Enterprise Security

turn data into doing™ 186 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
CIM Mapping
1
Select one or 6
more target Click Done
Select FIELDALIAS or EVAL data models
from the New Knowledge 2
Object drop-down

3
Select a
source field 4
Select a
5
Click OK target field

Note
The source event type or expression field can be an
eval statement (to transform the source value to
the CIM required format).

Administering Splunk Enterprise Security

turn data into doing™ 187 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add-on Builder: Validate and Package
• You can validate your add-on for best practices, CIM mapping, and
field extractions
– Any errors indicate a problem that should be corrected
– Warnings are non-fatal but might need attention
– If you select App pre-certification, a Splunkbase login is required

• Use Download Package to create an SPL package you can deploy

to your production environment
– The add-on is already active on the local system

Administering Splunk Enterprise Security

turn data into doing™ 188 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add-on Builder: Validate and Package (cont.)
Click to
download
3

1 2
Select validations to apply Click to start
validation

Administering Splunk Enterprise Security

turn data into doing™ 189 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 8 Lab: Building a Custom Add-on
Time: 30 minutes
Tasks:
1. Plan a new add-on for custom data
2. Create the add-on with the Splunk Add-on Builder
3. Validate the new add-on

Administering Splunk Enterprise Security

turn data into doing™ 190 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 9:
Tuning Correlation Searches

Administering Splunk Enterprise Security

turn data into doing™ 191 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Describe correlation search operation
• Customize correlation searches
• Describe numeric vs. conceptual thresholds

Administering Splunk Enterprise Security

turn data into doing™ 192 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Plan, Install, Evaluate, Refine
• Start with a base level of enabled correlation searches
– Security
events in the enterprise
– Anomalous audit trails

• Adjust correlation search sensitivity

– False positives: returning results when none are actually there
– False negatives: returning no results when something is expected

• Revisit and adjust thresholds as needed

– New security data is added to your ES install
– The size of what is monitored shrinks or grows
– Decreased number of open issues (i.e., ES is working!!)

Administering Splunk Enterprise Security

turn data into doing™ 193 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Managed Content
• Correlation searches are one type of ES content
– Correlationsearches are stored as saved searches
– Content in ES is any search or view that can be shared and used
between multiple ES sites
• Examples:
– Correlation and Key Indicator searches
– Entity (asset or identity) swim lane searches
– Lookups
– Views (dashboards and panels)
– Saved searches

Administering Splunk Enterprise Security

turn data into doing™ 194 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Content Management Functions
Configure > Content > Content Management
Add new content

Filter by Type, App, Status, or text

Enable, disable, or export Enable,

the selected content disable or
clone
content

Click a title to edit

Administering Splunk Enterprise Security

turn data into doing™ 195 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Content Management Functions (cont.)
Configure > Content > Content Management

Expand the Information (i)

column to verify dependency
and usage information

The details for each type of

content, and each individual
knowledge object vary

http://docs.splunk.com/Documentation/ES/latest/Admin/Expandcontentmanagementsearches
Administering Splunk Enterprise Security
turn data into doing™ 196 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Enabling Correlation Searches
• Only enable correlation searches that make sense for
your environment
• Consider:
– Types of vulnerabilities or threats you have determined might exist
– Type of security operations you are focused on, i.e., malware, intrusion
detection, audit, change monitoring, etc.
– You may need to increase hardware specs if you have many correlation
searches running
– You can improve overall performance by making less critical correlation
searches scheduled instead of real-time

Administering Splunk Enterprise Security

turn data into doing™ 197 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Scheduling a Correlation Search
• By default, all correlation
searches run in indexed real-
time mode

• If changed to scheduled, it will

execute every 5 minutes by
default

• When editing the scheduled

search, you can change the
time range settings Start time,
End time, and Cron Schedule
Administering Splunk Enterprise Security
turn data into doing™ 198 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Tuning Correlation Searches
• Threshold: the criteria that causes a correlation search to trigger
• Scheduling and throttling: how often to run the search and how often
to generate notable events for the same type of incidents
• Adaptive Responses: list of actions to take, including possibly
creating a notable event or setting risk
– Notable event settings: severity, default owner, default status, etc.
– Risk: assigning, increasing, or decreasing the risk score for a given type
of threat or incident
– Other adaptive responses include sending email, running scripts

Administering Splunk Enterprise Security

turn data into doing™ 199 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Correlation Thresholds
• Some correlation searches may generate more (or fewer) notable
events than you want
• Examine the search string and look for comparison terms in search
or mltk models and modify as appropriate for your environment
• Two types of thresholds:
– Numeric
– Conceptual (Machine Learning Based)
docs.splunk.com/Documentation/ES/latest/User/ConfigureCorrelationSearches

Administering Splunk Enterprise Security

turn data into doing™ 200 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Numeric Thresholds
• Simple numeric comparisons
• Example: Excessive DNS Failures
• Note where command with numeric comparison
• Change the numeric value if you need to alter how
frequently notable events are generated in your environment

Administering Splunk Enterprise Security

turn data into doing™ 201 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Conceptual Thresholds
• Uses Machine Learning Tool Kit (MLTK) functions
• Note the mltk_apply_upper macro is using “high” as a threshold
• Macro arguments: Brute Force Access Behavior Detected correlation search

– model
Name of the model for applying data
and comparing against standards to
find outliers. For example:
app:failures_by_src_count_1h
– Qualitative_id
Default IDs that correspond to percentages of deviation, representing where on the
distribution curve to look for outliers. For example: high, medium, low
– field
Where to search for or count outliers, such as failure

Administering Splunk Enterprise Security

turn data into doing™ 202 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Correlation Search Throttling
• Once a correlation search has been triggered, you probably don’t
want it to immediately re-trigger again for the same issue
• Most OOTB correlation searches throttle alerts to once a day
• If you want to modify this, change the Window duration
• In most cases, leave the Fields to group by alone

Administering Splunk Enterprise Security

turn data into doing™ 203 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adaptive Response Actions
• When a correlation search detects an issue, it can initiate one or
more adaptive response actions
• The most common response is to create a notable event
• Many also add risk to the objects associated with the issue
• Other responses can include sending email, running a script, stream
capture, and sending data to UBA

Administering Splunk Enterprise Security

turn data into doing™ 204 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Customizing Notable Event Default Values
• Expand the notable adaptive response
• You can modify all the properties of the
notable event that is created by a triggered
correlation search—typically:
– Severity
– Default Owner
– Default Status

Administering Splunk Enterprise Security

turn data into doing™ 205 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 9 Lab: Tuning Correlation Searches
Time: 15 minutes
Tasks:
1. Identify thresholds in correlation searches

Administering Splunk Enterprise Security

turn data into doing™ 206 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 10:
Creating Correlation Searches

Administering Splunk Enterprise Security

turn data into doing™ 207 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Create a custom correlation search
• Manage adaptive responses
• Manage content import/export

Administering Splunk Enterprise Security

turn data into doing™ 208 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Creating a New Correlation Search
1. Determine a pattern of events that indicates an issue you want to
respond to with a notable event or other action
2. Create a new correlation search in the UI using
Configure > Content > Content Management and select Create
New Content > Correlation Search
– Use Guided Mode if desired
3. Configure scheduling and throttling
4. Configure the adaptive responses (notable event, etc.)

Administering Splunk Enterprise Security

turn data into doing™ 209 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Correlation Search Example: Risk
• This example creates a new
correlation search that generates
a notable event once a day for any
server with a risk score over 100
• On the Content Management
page, select Create New Content >
Correlation Search
• Enter the search name, App,
UI Dispatch Context, and Description
• Select Guided Mode to create the actual search

Administering Splunk Enterprise Security

turn data into doing™ 210 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Correlation Search Example: Risk (cont.)
• From the Guided Search Editor,
select the Risk Analysis data model
and the All_Risk dataset
• Set Summaries only to Yes
– The correlation search will only
search in accelerated data
– This is faster, but un-accelerated data
is ignored
• Select the time range for
the search
• Click Next
Administering Splunk Enterprise Security
turn data into doing™ 211 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding a Where Filter
• Next, add filter expressions to limit the source events the correlation
search retrieves
– Thiscould be used to focus
on high priority assets or
specific business units

Administering Splunk Enterprise Security

turn data into doing™ 212 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding Aggregate and Split-by Functions
• Next, add aggregate functions to
perform operations like count,
sum, or average on fields in the
data model
• Optionally, add split-by conditions
to aggregate values categorically
– The example takes the sum of all
risk per source (src)
• Click Next

Administering Splunk Enterprise Security

turn data into doing™ 213 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding Filters
• Define the logic to determine
what condition will trigger a
new notable event
• In this case, a notable event
is generated if the risk score
for any one source is greater
than 100
• Click Next

Administering Splunk Enterprise Security

turn data into doing™ 214 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Parsing the Search
• Finally, the search is parsed
and displayed
• After verifying the test, select
Done to save the correlation
search criteria and continue
configuring the rest of the
correlation search fields
• If you edit the search string manually later, you will not be able to use
guided mode to modify the search string

Administering Splunk Enterprise Security

turn data into doing™ 215 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Setting the Time Range
• Configure time range options
– Earliest Time and Latest Time
are relative to the scheduled
start time
– Cron schedule is how often to
run the search
• The default is ‘*/5 * * * *’
which is every five minutes
• This is overruled if the
correlation search is set to
real time

Administering Splunk Enterprise Security

turn data into doing™ 216 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Setting Annotations
• Annotations can enrich
correlation search results with
the context from industry-
standard mappings
– Enter any annotation attributes for
CIS 20, Kill Chain, or NIST
– For MITRE ATT&CK, choose the
attributes from the dropdown list

Administering Splunk Enterprise Security

turn data into doing™ 217 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Scheduling Settings
• Scheduling: real-time or continuous
– Manages real-time scheduling
– Typically, leave the default of real-time

• Schedule Window: seconds (or “auto”)

– Allow some flexibility in scheduling to
improve scheduling efficiency

• Scheduling Priority: higher-priority

searches will be selected first by scheduler if a conflict occurs
docs.splunk.com/Documentation/Splunk/latest/Admin/Savedsearchesconf#Scheduling_options

Administering Splunk Enterprise Security

turn data into doing™ 218 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Setting Trigger Conditions
• Normally, a correlation search will trigger its adaptive responses
(notable, etc.) if any results are found by the search
• You can use the Trigger Conditions to alter this default

Administering Splunk Enterprise Security

turn data into doing™ 219 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Setting Throttling
• Throttling: You should throttle based on a field’s value
– Example:no more than one notable event per host per day
(86,400 seconds)
• More than one field can be selected
– Throttling is based on all the field values ANDed together

Administering Splunk Enterprise Security

turn data into doing™ 220 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding Adaptive Responses

Note
Each time Response Actions are
modified, you must update the
Splunk_TA_AROnPrem app.

Select the Notable

response

Administering Splunk Enterprise Security

turn data into doing™ 221 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Configuring Notable Event Fields
• Configure notable event field values
– Title, description, security domain, severity
– Default owner and status
– Drill-down settings

• Embed field values in title, description, and

drill-down fields using $fieldname$ format
• Description fields support URLs to
external locations
– Useful for best practices documents,
investigation procedures, etc.

Administering Splunk Enterprise Security

turn data into doing™ 222 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Configuring Notable Event Fields (cont.)
• Control actions taken when a
notable is added to an investigation
– Select an investigation profile to
apply to the investigation
– Automatically extract artifcacts that will be added to
the investigation

Administering Splunk Enterprise Security

turn data into doing™ 223 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Configuring Notable Event Fields (cont.)
• You can control the “next steps”
and “recommended actions”
adaptive responses that appear in
Incident Review
– Next steps appear as links in the
notable event details
– Recommended Actions appear in
the notable event’s Actions menu

Administering Splunk Enterprise Security

turn data into doing™ 224 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Saving the Correlation Search
• Click Save to create the new
correlation search
• Click Close and navigate back
to the Content Management page
• Your new search will now display in the list of correlation searches
for the ES app
• You can enable, disable, and change to scheduled or
real-time as desired

Administering Splunk Enterprise Security

turn data into doing™ 225 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adaptive Response Actions
• Besides (or instead of) creating notable events, adaptive response
actions can automate other critical tasks
• One or more adaptive response actions can be added to each
correlation search
– The action will be executed if the correlation search finds any matches
• ES ships with a set of default adaptive responses
• You can also install additional adaptive responses, and control who
can access each adaptive response
docs.splunk.com/Documentation/ES/latest/Admin/Setupadaptiveresponse

Administering Splunk Enterprise Security

turn data into doing™ 226 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Default Adaptive Response Actions
Notable, Risk Create a notable event or add to an object’s risk score
Send email, create Splunk Send email to one or more people, or add a system message in the
message Splunk web interface
Run script Execute an automated script. Example: when a correlation search
indicates a host is infected with malware, run a script to quarantine the
target server
Stream capture Automatically begin collecting detailed network information
Nbstat, nslookup, ping Execute diagnostic command and attach output to the notable event to
assist in analysis
Send to UBA/output to If User Behavior Analytics is installed and integrated, send the notable
telemetry endpoint event to UBA for analysis/send to Splunk telemetry
Add Threat Intelligence Create a threat intel artifact. Example: a new type of infection is
discovered; add the characteristics of the infection (file name, source IP,
code hash, etc.) to the threat intel database so that future similar attacks
will be immediately alerted

Administering Splunk Enterprise Security

turn data into doing™ 227 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Edit Adaptive Responses
Settings > Alert actions
An ES Admin can use the Alert actions page in Splunk to make changes to
Adaptive Responses

Disable View usage

Change adaptive statistics and
permissions response log events

Administering Splunk Enterprise Security

turn data into doing™ 228 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Adaptive Response Permissions
Select Permission for a specific alert action to make changes

Select Read, Write, or both

permissions for each role

Administering Splunk Enterprise Security

turn data into doing™ 229 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adaptive Response Action Center
Audit > Adaptive Response Action Center

Administering Splunk Enterprise Security

turn data into doing™ 230 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Content Import/Export
• You can export any of the content types on the Content
Management page by selecting them in the custom search list and
choosing Export
• Enter an app name, prefix, label, version and build number, and click
Export
– The content will be downloaded to your workstation as an .spl file
– It can then be installed as a new app into another ES search head

• Import content by installing an app

Administering Splunk Enterprise Security

turn data into doing™ 231 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Example: Content Export

Note
DA-ESS is a recommended
prefix for content add-ons but is
not required.

Administering Splunk Enterprise Security

turn data into doing™ 232 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Content Export Best Practices
• The app name for the content export is uploaded to the etc/apps
directory of the receiving server
• Be careful when exporting updates to your content
– Example: you export correlation1, naming it correlations.spl,
and upload it to another ES server. Later you export correlation2,
again using correlations.spl as the export name. When you upload
correlations.spl to the second server, it overwrites the old version
of correlations.spl, deleting correlation1
• Either use new app names each time (which could be difficult to
manage) or make sure you always include all content (old and new)
each time you export
Administering Splunk Enterprise Security
turn data into doing™ 233 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 10 Lab: New Correlation Searches
Time: 20 minutes
Tasks:
1. Create a custom correlation search
• SSH logins are prohibited in your environment. Create a custom
correlation search that detects successful SSH logins and generates a
notable event to alert analysts

Administering Splunk Enterprise Security

turn data into doing™ 234 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 11:
Asset & Identity Management

Administering Splunk Enterprise Security

turn data into doing™ 235 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Review the Asset and Identity Management interface
• Describe Asset and Identity KV Store collections
• Configure and add asset and identity lookups to the interface
• Configure settings and fields for asset and identity lookups
• Explain the asset and identity merge process
• Describe the process for retrieving LDAP data for an asset or identity
lookup

Administering Splunk Enterprise Security

turn data into doing™ 236 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Assets & Identities Overview
• Asset and identity configuration enhances the information available for
users and systems in the notable index and ES dashboards
• SA-IdentityManagment is the supporting add-on that maintains
macros, lookups, knowledge objects, etc.
• CRUD (create, read, update, and delete) operations maintain the
attributes defined in the lookups
• View the contents of a lookup table. For example:
| inputlookup demo_identities.csv

Administering Splunk Enterprise Security

turn data into doing™ 237 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Asset & Identity Management Interface
Configure > Data Enrichment > Asset and Identity Management
Asset and identity lookups and settings are configured in this interface

Administering Splunk Enterprise Security

turn data into doing™ 238 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Assets & Identity KV Store Collections
• Asset and Identity lookups are stored as KV Store collections
– assets_by_str collection – asset_lookup_by_str lookup
– assets_by_cidr collection – asset_lookup_by_cidr lookup
– identities_expanded collection – identity_lookup_expanded lookup

• Prior to 6.2 assets and identities were only defined in lookup tables
• Using the KV Store allows for larger tables for assets and identities
• Lookups are defined in SA-IdentityManagement transforms.conf
[asset_lookup_by_cidr]
external_type = kvstore
match_type = CIDR(asset) Example lookup configuration
collection = assets_by_cidr
fields_list =
_delete,_key,_last_updated,_sources,asset,asset_tag,bunit,category,city,country,dns,ip,is_expected,lat,
long,mac,nt_host,owner,pci_domain,priority,requires_av,should_timesync,should_update max_matches = 1
case_sensitive_match = false
filter = NOT _delete="true"

Administering Splunk Enterprise Security

turn data into doing™ 239 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Uploading Assets and Identities
• Update the lookups provided in the management interface under Asset
Lookups and Identity Lookups with your corporate data (static_assets,
static_identities, administrative_identities)
• Or create lookup .csv files with the proper fields and add them to the
management interface
• Initially pull corporate asset and identity data using a Splunk add-on such
as LDAP Search or DB Connect
Important!
– The add-on pulls the information Do not include every piece of hardware or
to be used in the lookup files every person – focus on the ones with the
most significance.
– Periodically re-run to keep assets Not every column needs to be populated.
and identities updated
Administering Splunk Enterprise Security
turn data into doing™ 240 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Asset Lookups
• The Asset Lookups tab lists all configured asset lookups
• New lookups can be added here, and existing lookups can be edited
Reset Collections at any time, rather than
waiting for the automated process to clear out
the KV store collection

Add a new asset lookup

Rank files, lowest rank takes precedence

Click a Name to change Open and edit the Enable | Disable

the lookup settings Source lookup file a lookup

Administering Splunk Enterprise Security

turn data into doing™ 241 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add a New Asset Lookup
New > New Configuration
1. From the New Asset Manager window
select the lookup file from the Source
drop-down menu
2. Name is auto-filled with the file name
3. Complete the Category and Description
4. Type defaults to asset
5. From Field Exclusion List, select fields to
ignore when merging assets
Important!
To create a new asset lookup the file must
first be uploaded to Configure > Content >
Content Management > Create New
Content > Managed Lookup

Administering Splunk Enterprise Security

turn data into doing™ 242 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add a New Asset Lookup (cont.)
The new asset list displays in the
Asset Lookup Configuration tab

The lookup is added to

$SPLUNK_HOME/etc/apps/SA-IdentityManagement/local/inputs.conf
[identity_manager://foo_assets] Stanza Name
blacklist = true
category = FOO_domain
description = FOO servers and workstations
rank = 3
target = asset
url = lookup://FOO_assets CSV File
Name
Administering Splunk Enterprise Security
turn data into doing™ 243 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Remove an Asset Lookup

Click an added
lookup configuration
to edit or delete it

Note
The default configurations demo_assets
and static_assets cannot be removed

Administering Splunk Enterprise Security

turn data into doing™ 244 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Asset Fields
• Asset fields are column headers in .csv
lookup files used to customize data
• Asset field matching settings
– Name - which headers/fields in a lookup table to
match during the merge process
– Key - like ip (key), field is used in
merge process
– Tag - field can be used as an asset tag
– Multivalue - field can output multiple values
– Multivalue Limit - number of values in a multivalue
field merge

Administering Splunk Enterprise Security

turn data into doing™ 245 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Changing Default Asset Fields
• For default fields, these
settings can be changed:
– Key

– Tag

– Multivalue

– Multivalue Limit

Administering Splunk Enterprise Security

turn data into doing™ 246 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding Custom Asset Fields
Add up to 20 custom header fields Click +Add New Field and give the field a name.
Select Multivalue if the field can output multiple
values and set a limit. Check the Key box to use
the field in the merge process. Check the Tag box
if the field can be used as an asset tag.

Only manually added

fields can be deleted.

Administering Splunk Enterprise Security

turn data into doing™ 247 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Case Sensitive Matching
• Enable case sensitive matching
–A warning displays that there will be a full rewrite of the collection
– Click Update to continue

Administering Splunk Enterprise Security

turn data into doing™ 248 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Identity Lookups
• The Identity Lookups tab has the same features as the Asset
Lookups configuration tab
– Change lookup settings, open and edit the source lookup file, enable
and disable the lookups
– Change the Rank of the lookups using the double ellipsis

Administering Splunk Enterprise Security

turn data into doing™ 249 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Identity Lookups (cont.)
• Similar to the configuration of an asset
lookup, the exception is the
email convention
• If selected, email conventions can be
used to uniquely identify the identities in
the data
– Email address or Email Short
(username in email address) Email Short id un-checked
by default
– Custom Conventions – for example,
identify users by the first letter of their
first name and their last name
• Added configurations can be deleted

Administering Splunk Enterprise Security

turn data into doing™ 250 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Identity Fields
• Similar to the setup of asset fields
– Add up to 20 custom fields
– Key – field is used in merge process
– Tag - can be used as an asset tag
– Enable case sensitive matching
– Multivalue - field can output
multiple values
• By default, the only “Key” field is
identity - Multivalue Limit (1-
100). The default is 25
• Non-key fields - Multivalue Limit is
1 – 100. The default is 25

Administering Splunk Enterprise Security

turn data into doing™ 251 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Global Settings Overview
Settings are described in detail in the next few slides

Administering Splunk Enterprise Security

turn data into doing™ 252 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Enable Merge
• The default behavior is to merge rows
of source data based on a match in
any one of the “key” fields
• Enabled by default for assets
and identities
• Disable the merge process here if you
need to stop the process
– Forexample, if you have a source file with duplication in the “key” fields,
and you cannot groom the file to make sure that the information belongs
to the same asset or identity, disable the merge process

Administering Splunk Enterprise Security

turn data into doing™ 253 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Asset Merge Example
• Example: the source file has duplicates in the nt_hosts key field:
ip,mac,nt_hosts,dns,owner,priority,lat,long,city,country,bunit,category,pci_domain,is_expected,should
_timesync,should_update,requires _av
192.0.2.2,,host1,,,,,,,,,,,,,,
192.0.2.120,,host1,,,,,,,,,,,,,,
192.0.2.135,,host1,,,,,,,,,,,,,,
192.0.2.242,,host2,,,,,,,,,,,,,,
192.0.2.65,,host2,,,,,,,,,,,,,,

– By default, the three host1 entries defined by the nt_hosts field are merged
into one asset and the two host2 entries are merged into another
– With merge disabled, the collection remains the same as the source file
– When you do a lookup on a non-merged collection, there is no context for how
to resolve the overlapping key field values
• For example, the asset_lookup_by_str lookup in transforms.conf has
max_matches=1, so the first host it matches in the assets_by_str
collection is the only one you will see in the search results
Administering Splunk Enterprise Security
turn data into doing™ 254 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Enable Zones
• Create Zones for entries that overlap, like IP addresses
– For example, two companies are merging who use the same IP address scheme.
Assign all entries with a location of palo_alto to a zone called flowmill, and
entries with a location of boulder to a zone called victorops

Administering Splunk Enterprise Security

turn data into doing™ 255 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Ignore Values
• The default behavior in ES is to merge rows of source data based on a
match in any one of the key fields
• Source data may have placeholder values that span multiple rows,
causing them to merge into one large multivalue row
• Solution: define the placeholder values as null and clean them during
the merge process. Independent rows are maintained in the final
lookups
• Ignore values are case sensitive, you may have to enter more than one
value, like “unknown” and “Unknown”

Administering Splunk Enterprise Security

turn data into doing™ 256 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Ignore Values
• Define values to be ignored when creating assets and identities
– By default, null, n/a, unknown, and undefined are ignored
– Ignored values apply to any type of field (key, non-key, multivalue, single value)
– Strings are saved as ignored_values in
/SA-IdentityManagement/local/inputs.conf

Use +Add Row to define

custom ignore values

Administering Splunk Enterprise Security

turn data into doing™ 257 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Enforcements
• How ES verifies the following in SA-IdentityManagement each time the
identity manager runs (every 5 minutes):
– Enforce props – automatic lookups that are defined in props.conf
– Enforce macros – macros that read from the CSV files into
the KV Store collections. Best to always enforce them!
Settings > Advanced Search > Search macros > SA-IdentityManagement

– Enforce transforms – transforms.conf defines the

.csv filenames and settings like case sensitive matching
– Enforce replicate – collection replication settings defined
in collections.conf
– Enforce identityLookup – IdentityLookup conventions

Administering Splunk Enterprise Security

turn data into doing™ 258 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Miscellaneous Settings
• Settings specific to the Identity Manager framework
– Time(s) – how often Identity Manager runs
• Default is 300 seconds (5 minutes)
• Can be increased for better performance
– Master host – host where the Identity
Manager runs, defaults to SHC captain
– Overlay CIDR – asset_lookup_by_str
will include CIDR data
– Debug mode – enable debug logging for
Asset and Identity Management

Administering Splunk Enterprise Security

turn data into doing™ 259 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Correlation Setup
• Choose how to use asset and identity correlation to enrich events
– By Default, correlation is enabled

Note
Typically, this should not need to
be changed from the default:
”Enable for all sourcetypes”.

– Disabling correlation prevents events from

being enriched with asset and identity
information from the lookups
– Another option is to restrict correlation to
occur only for select source types
Administering Splunk Enterprise Security
turn data into doing™ 260 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Search Preview
Search Preview creates custom-built searches with what is currently in inputs.conf,
running a search displays the merge data from all asset or identity lookups

Test the merge process on your data

without actually performing the merge!

Administering Splunk Enterprise Security

turn data into doing™ 261 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Search Preview (cont.)
• Each search is dynamic and generates each time you refresh or load
the page
– For
example, the asset_lookup_by_str search changes when the new
FOO_assets table is added If nothing has changed in the
source files since the last merge,
you will not see any output

Administering Splunk Enterprise Security

turn data into doing™ 262 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Onboarding LDAP Data
• Use the Splunk Supporting Add-on for AD (SA-ldapsearch) to pull LDAP
data from an AD database and create a lookup of assets or identities
• From Asset Lookups or Identity Lookups, select LDAP Lookup from the
New menu

Administering Splunk Enterprise Security

turn data into doing™ 263 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Onboarding for LDAP
• Complete the Lookup Builder
– Notice the window says either LDAP
Lookup Builder (Asset) or (Identity)
– Give the search a name and enter the
name of the domain
– Give the lookup file a Label and
Lookup name
– Splunk populates the Lookup filename
field with the lookup name and .csv

Administering Splunk Enterprise Security

turn data into doing™ 264 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Data Onboarding for LDAP (cont.)
• Scroll down to enter a cron schedule
for how often the search should run
• Select the scheduling
– Real-time: run at the scheduled
time or not at all
– Continuous: if a report cannot run
now, it will run in the future, after
other reports finish
• PREVIEW displays the search that Creates the Saved Search found in Configure
will pull asset or identity data from > Content > Content Management
Creates the Lookup table and definition found
LDAP, create a lookup, and merge in Settings > Lookups
data into the KV Store

Administering Splunk Enterprise Security

turn data into doing™ 265 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Example: LDAP Search Identity Upload
• Example search for collecting identity data from Active Directory
– Use as a guide to construct and test a working search
– Rename the lookup to something appropriate for your environment
|ldapsearch domain=<domain_name> search="(&(objectclass=user)(!(objectClass=computer)))"
|makemv userAccountControl
|search userAccountControl="NORMAL_ACCOUNT"
|eval suffix=""
|eval priority="medium"
|eval category="normal"
|eval watchlist="false"
|eval endDate=""
|table sAMAccountName,personalTitle,displayName,givenName,sn,suffix,mail,telephoneNumber,mobile,manager,
priority,department,category,watchlist,whenCreated,endDate
|rename sAMAccountName as identity, personalTitle as prefix, displayName as nick, givenName as first, sn
as last, mail as email, telephoneNumber as phone, mobile as phone2, manager as managedBy, department as
bunit, whenCreated as startDate
|outputlookup my_identity_lookup

Administering Splunk Enterprise Security

turn data into doing™ 266 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Troubleshooting Assets and Identities
• Examine CSV files
$SPLUNK_HOME/etc/apps/SA-IdentityManagement/lookups
– Verify that all CSV files are properly formatted
• Examine lookup configuration
$SPLUNK_HOME/etc/apps/SA-IdentityManagement/default/transforms.conf

• Check log files (over all time)

index=_internal sourcetype=python_modular_input collection=assets_by_str OR
collection=identities_expanded

• Test an asset match:

| makeresults | eval src="1.2.3.4" | `get_asset(src)`

• Test an identity match:

| makeresults | eval user="hax0r" | `get_identity4events(user)`

Administering Splunk Enterprise Security

turn data into doing™ 267 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Asset Matching Algorithm
• For assets, ES takes the value from an event’s src, dvc, or dest
field and tries to match it to these columns in this order
Order Column Description
1 ip match the IP address or address range
2 mac match on a Media Access Control address
3 dns match on DNS name
4 nt_host match on Windows Machine Name (a.k.a. NetBIOS name)

• ES uses the above order to make its first match, then checks CIDR-
based matches for IP addresses

Administering Splunk Enterprise Security

turn data into doing™ 268 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
CIDR Asset Matching Algorithm
• For ip and mac field ranges, if more than one range matches, ES
matches on the smallest range

For example, host=1.2.3.4 matches both

the first and second IP ranges; however,
it only matches on the second one since
that’s the smaller range.

• Asset matching allows you to create large, catch-all categories on

MAC or IP ranges, yet still single out smaller groups or individual IPs
within the larger group
Administering Splunk Enterprise Security
turn data into doing™ 269 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Identity Matching
• For identities, ES takes a value from an event’s user, src_user,
email,or src_email field and tries to match it to a value in the
“identities” lookup in the order shown here
Order Column Description
1 identity Exact match on any one of a list of usernames in identity column
2 Email Exact match
3 Email First part of email, i.e. “htrapper” of “[email protected]”
4 Any Disabled by default—see “conventions” in identityLookup.conf.spec

Administering Splunk Enterprise Security

turn data into doing™ 270 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Watchlisting Assets and Identities
• Add identities and assets to a watchlist to highlight them in various
dashboards and searches
– Watchlisted assets or identities trigger the Watchlisted Event Observed
correlation search, if enabled
– Watchlisted users display on the User Activity dashboard

• Watchlist users by setting watchlist to true in the Identities lookup

• Add websites to watchlists

– Configure > General > General Settings
– Edit Website Watchlist Search and add asset IP or DNS
Administering Splunk Enterprise Security
turn data into doing™ 271 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Asset and Identity Investigators
• Used to search by an asset or identity over a specific time range
• Returns a time-sequenced set of swim lanes showing activity for the asset or identity

Selecting an individual bar in a swim lane

shows details on the right Event
details

Area graph shows activity over time period

Administering Splunk Enterprise Security

turn data into doing™ 272 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Configuring Swim Lanes
• Click Edit and select a collection
of swim lanes
• Use the Custom collection to
select specific swim lanes
• Customize swim lane colors Drag swimlanes up
and down into the
• ES Admins can add new swim lane order you prefer

searches and set overall defaults

and permissions per role

Administering Splunk Enterprise Security

turn data into doing™ 273 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Create a Swim Lane Search
From Content Management,
select Swim Lane Search from
the Create New Content menu
and complete the information

Administering Splunk Enterprise Security

turn data into doing™ 274 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 11 Lab: Working with Assets & Identities
Time: 30 minutes
Tasks:
1. Modify asset priority for PROD-MFS servers
2. Create two swim lane searches that track successful and failed logins for
identities
3. Use the Identity Investigator to confirm the newly created swim lanes

Administering Splunk Enterprise Security

turn data into doing™ 275 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Module 12:
Managing Threat Intelligence

Administering Splunk Enterprise Security

turn data into doing™ 276 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Understand and configure threat intelligence
• Use the Threat Intelligence Management interface to configure a new
threat list

Administering Splunk Enterprise Security

turn data into doing™ 277 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
The Threat Intelligence Framework
• Threat intel is downloaded regularly from external and internal sources by
the Threat Download Manager modular input
– Data is parsed into KV store collections with “_intel” suffixes
– Collections are used as lookups during threat generation searches
• Threat Gen searches run by default every 5 minutes and scan for threat
activity related to any of the threat collections
– When threat matches are found, events are generated in the threat_activity
index and appear in the Threat Intelligence data model
• The data model is scanned by the Threat Activity Detected correlation
search and new notables for threat activity are created
https://dev.splunk.com/enterprise/docs/devtools/enterprisesecurity/threatintelligenceframework/#H
ow-Splunk-Enterprise-Security-processes-threat-intelligence
Administering Splunk Enterprise Security
turn data into doing™ 278 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Threat Intelligence Administration
• ES Admins are tasked with managing ES threat intelligence
• Analysts and users can be given the Edit Intelligence Downloads
permission to manage threat intelligence downloads

• Threat Intelligence can be added to ES by

– downloading a feed from the Internet
– uploading a structured file
– inserting threat intelligence directly from events in ES
Administering Splunk Enterprise Security
turn data into doing™ 279 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Configuring Threat Intelligence
• ES can download the following threat intelligence types:
– Threat lists: IP addresses of known malicious sites
– STIX/TAXII: detailed information about known threats, including threat
type, source, etc.
– OpenIOC: additional detailed information about known threats

• Threat lists can also be configured locally

• Many intel sources require regular refresh from external sources
• This information is used by the Threat Activity Detected
correlation search

Administering Splunk Enterprise Security

turn data into doing™ 280 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Included Generic Intel Sources
• Generic intelligence included in ES by default
• These intelligence sources are not added to the threat intelligence KV
Store collections, but are used to enrich data in ES
Data List Data Provider URL
Cisco Umbrella Top 1 Million Sites Cisco https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip

ICANN Top-level Domains List IANA https://data.iana.org/TLD/tlds-alpha-by-domain.txt

MaxMind GeoIP ASN IPv4 MaxMind https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-

ASN-CSV&license_key=YOUR_LICENSE_KEY&suffix=zip
database
MaxMind GeoIP ASN IPv6 MaxMind https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-
database ASN-CSV&license_key=YOUR_LICENSE_KEY&suffix=zip

Mozilla Public Suffix List Mozilla https://publicsuffix.org/list/effective_tld_names.dat

MITRE ATT&CK framework Mitre https://raw.githubusercontent.com/mitre/cti/master/enterprise-

attack/enterprise-attack.json

Administering Splunk Enterprise Security

turn data into doing™ 281 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Threat Intelligence Audit
Audit > Threat Intelligence Audit

Display status and time for all downloads

View details of downloads including errors like

“No content returned” and “retrying download”

Administering Splunk Enterprise Security

turn data into doing™ 282 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Intelligence Management Interface
Configure > Data Enrichment > Threat Intelligence Management

The Threat Intelligence Management interface has

three tabs for creating and managing threat intel:
Sources, Threat Matching, and Global Settings

Administering Splunk Enterprise Security

turn data into doing™ 283 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Sources Tab
Configure > Data Enrichment > Threat Intelligence Management

The Sources tab lists all Create a new download by

configured intel downloads selecting the type

Click a download name to display the

Edit Intelligence Document form.
Allows you to edit the fields relevant
only to the selected document

Use Advanced Edit to

change all
Settings for the download

Administering Splunk Enterprise Security

turn data into doing™ 284 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add a Threat Download

Download name (no spaces)

Delimiting regex: how to split the lines in

Valid URL for the download location
the file, comma, semi colon, etc.
Weight to increase the risk score of
threat intel objects on this list

Interval of how often to download the information Ignoring regex: by default, ignores blank
lines and lines beginning with #

Max age for KV Store retention (default 30 days) Fields: comma separated list of fields in the
data (defaults to description and ip)

Skip header lines: does the data

have any header lines to ignore
Check Threat Intelligence to use
information as threat intel for the
_intel KV collections

Administering Splunk Enterprise Security

turn data into doing™ 285 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add a Threat Download (cont.)

Retries: number of attempts to download the file

Retry Interval: seconds to wait between download attempts

Timeout: seconds to wait before marking the download as failed

Remote site user and realm: only if the threat feed requires authentication.
Must be configured in Splunk Credential Management

Sinkhole: delete file after downloading

Enable debug logging

Administering Splunk Enterprise Security

turn data into doing™ 286 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Upload an Intel File

Upload or browse to the file and Manually upload a threat

complete the information on the intel file by selecting Upload
General, Parsing, and Advanced from the New menu
tabs as needed for the file

Administering Splunk Enterprise Security

turn data into doing™ 287 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Threat Matching
Edit the Threat Match settings that generate the SPL for threat
match searches and enrich data with threat intelligence

Source: type of threat match

sources enabled. Click to edit
Match Fields: what fields to match
against to generate threats

Interval: how often Earliest and Latest:

the search runs when the search starts
and completes

Expand a Source to
see the threat match
configuration

Administering Splunk Enterprise Security

turn data into doing™ 288 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Threat Matching (cont.)

Edit the times which the

search runs and completes,
and the interval which is
runs (Cron)

Edit, remove, or a add dataset

from a data model to the
threat match configuration

Administering Splunk Enterprise Security

turn data into doing™ 289 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Global Settings

If a proxy server is used to

send intel to ES, configure
that information here

Configure ES to extract fields and values

embedded in other fields, such as extracting
Note the domain name from a URL
Proxy user must be configured in
Splunk Credential Management

Administering Splunk Enterprise Security

turn data into doing™ 290 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Viewing Threat Intel Collections
• After download, threat intel data is stored in KV store collections with
the “_intel” suffix
Use Settings > Lookups to view the collections

• Use |inputlookup to examine the contents of a collection

• Use the Threat Artifacts dashboard to examine the overall contents
of the entire threat intelligence framework
Administering Splunk Enterprise Security
turn data into doing™ 291 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Threat Activity Dashboard
Security Intelligence > Threat Intelligence > Threat Activity
Display details related to known threat sites over a period of time

Panels:
• Threat activity over time by threat collection
• Most active threat collections and sources
• Threat activity detail

Administering Splunk Enterprise Security

turn data into doing™ 292 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Threat Artifacts
Security Intelligence > Threat Intelligence > Threat Artifacts
Displays the current content of the downloaded threat intel

Administering Splunk Enterprise Security

turn data into doing™ 293 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Module 12 Lab: Threat Intel Framework
Time: 10 minutes
Tasks:
Add a new threat list download

Administering Splunk Enterprise Security

turn data into doing™ 294 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
What’s Next?
Become a Splunk Enterprise Security Certified Admin
This certification demonstrates an individual's ability to install, configure, and
manage a Splunk Enterprise Security deployment

Prerequisite Certification(s): Splunk Enterprise Security Congratulations! You are a...

Certified Admin Exam
● None
Time to study! We suggest candidates
Prerequisite Course(s): looking to prepare for this exam complete
the following course:
● None
❏ Administering Splunk Enterprise
Security

Please note: all candidates are expected

to have working knowledge and Recommended Next Steps
experience as either Splunk Cloud or ● Splunk Phantom Certified
Splunk Enterprise Administrators. Admin

See here for registration assistance.

Administering Splunk Enterprise Security

turn data into doing™ 295 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Splunk Security Courses
• For more Splunk security training, please review these courses on
https://www.splunk.com/en_us/training.html

– Splunk User Behavior Analytics

– Administering Splunk SOAR
– Developing SOAR Playbooks
– Advanced SOAR Implementation

Administering Splunk Enterprise Security

turn data into doing™ 296 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Community
• Splunk Community Portal • Slack User Groups
splunk.com/en_us/community.html splk.it/slack
– Splunk Answers • Splunk Dev Google Group
answers.splunk.com groups.google.com/forum/#!forum/splunkdev
– Splunk Apps
• Splunk Docs on Twitter
splunkbase.com twitter.com/splunkdocs
– Splunk Blogs
splunk.com/blog/ • Splunk Dev on Twitter
twitter.com/splunkdev
– .conf
conf.splunk.com

Administering Splunk Enterprise Security

turn data into doing™ 297 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Splunk How-To Channel
• Check out the Splunk Education How-To channel on YouTube:
splk.it/How-To
• Free, short videos on a variety of Splunk topics

Administering Splunk Enterprise Security

turn data into doing™ 298 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Support Programs
• Web
– Documentation: dev.splunk.com and docs.splunk.com
– Wiki: wiki.splunk.com
• Splunk Lantern
Guidance from Splunk experts
– lantern.splunk.com
• Global Support
Support for critical issues, a dedicated resource
to manage your account – 24 x 7 x 365
– Web: splunk.com/index.php/submit_issue
– Phone: (855) SPLUNK-S or (855) 775-8657
• Enterprise Support
– Access customer support by phone and manage your
cases online 24 x 7 (depending on support contract)
Administering Splunk Enterprise Security
turn data into doing™ 299 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
MGM Grand, Las Vegas, NV | June 13–16
Virtual | June 14–15
Join us for a hybrid experience and learn why
data is key to achieving better outcomes.

“ .conf21 gave me the ability to immerse

myself in all things Splunk for two full
days, I learned so much.”
— John Whitefield
Progressive Insurance, IT DevOps Eng. Senior
Administering Splunk Enterprise Security
turn data into doing™ 300 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Thank You

Administering Splunk Enterprise Security

turn data into doing™ 301 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Appendix A:
Analyst Tools & Dashboards

Administering Splunk Enterprise Security

turn data into doing™ 302 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Troubleshoot missing dashboard data
• Explain dashboard dependencies including
data models and searches

Administering Splunk Enterprise Security

turn data into doing™ 303 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Dashboard Data Dependencies
• Each dashboard panel’s search pulls events from a data model
• If a panel is missing data, examine the panel’s search to see which
data model is used; this can help you understand why the data is
missing
• Causes:
– The data is not in Splunk: install and enable add-ons to input the data
– The data is present in Splunk but is not normalized correctly: modify
normalization settings
docs.splunk.com/Documentation/ES/latest/User/DashboardMatrix

Administering Splunk Enterprise Security

turn data into doing™ 304 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Example: DNS Activity Missing Data
• The example shows that the
Top Reply Codes By Unique
Sources panel in the DNS Activity
dashboard is empty
• Using Open in Search shows
that the data for this panel is taken
from the Network_Resolution
data model and DNS data set

Administering Splunk Enterprise Security

turn data into doing™ 305 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Example: DNS Activity Missing Data (cont.)
Things to check if a dashboard is empty
– Is your data normalized to the
data model?
– If collecting data from a
streaming app like Splunk
Stream, is it configured to collect
the correct type of data?
– Is your data tagged properly?

Administering Splunk Enterprise Security

turn data into doing™ 306 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Data Models
• Data models use tags to identify events relevant to the data model
Splunk Enterprise > Settings > Data models
– For example, the
Network_Resolution.DNS
datamodel uses the network,
dns, and resolution tags to
match relevant events
– If an event is not tagged with
these constraints, it will not be
referenced by the data model

Administering Splunk Enterprise Security

turn data into doing™ 307 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Searching Data Models

Use the |from command

to search all events from
the data model

Administering Splunk Enterprise Security

turn data into doing™ 308 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Example: Protocol Center Missing Data
• Protocol Center is missing information for juniper.idp,
junos_firewall, and netscreen
• Open in Search confirms the missing
events are not being referenced by the
Network_Traffic.All_Traffic data model

Administering Splunk Enterprise Security

turn data into doing™ 309 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Example: Protocol Center Missing Data (cont.)
• Apps > Manage Apps shows that the Splunk_TA_juniper
app is disabled

• Juniper data is now

referenced by the data
model and displays on
the dashboard

Administering Splunk Enterprise Security

turn data into doing™ 310 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Per-panel Filtering Dashboards
• Some ES dashboards allow highlighting or filtering of items on
dashboard views
– If it is determined that an event is not a threat, it can be added to a
whitelist to remove it from the dashboard view
– If an event is determined to be a threat, use the Per-panel filter button to
add the item to the blacklist of known threats
• For Example, on HTTP Category Analysis filter out expected
categories and highlight unwanted categories
• Permission for Per-panel filtering
can be granted for ess_analyst
and ess_user roles
Administering Splunk Enterprise Security
turn data into doing™ 311 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Creating Per-panel Filters

3
Choose to either filter
out or highlight the
events
4

2
Click Per-panel Filter

1
Select one or more events

Administering Splunk Enterprise Security

turn data into doing™ 312 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Filtered vs. Highlighted Events
• Filtered events are no longer displayed, though summary statistics
continue to calculate
• Highlighted events are marked yellow in the Per-panel Filter column
and are displayed at the top of the list by default

Administering Splunk Enterprise Security

turn data into doing™ 313 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Managing Per-panel Filtering Lookups
• Edit filters in the corresponding lookup table
• Access the lookup table by
– clicking View/edit existing filters in the Per-
panel Filter window
Or
– by selecting the lookup table under
Configure > Content > Content
Management

Administering Splunk Enterprise Security

turn data into doing™ 314 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Per-panel Filter Audit
Audit > Per-panel Filter Audit
Display data on per-
panel filter usage

See who creates per-panel

filters, and what data is
being filtered

Administering Splunk Enterprise Security

turn data into doing™ 315 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Splunk & Fraud Analytics

• Leverages Splunk Enterprise Security

– Analyst can work in a familiar
incident review tab
– Fraud Incident Review includes workflow link to Investigate dashboard
– Visual link analysis to make fraud investigations quick
– Leverages Risk-based Alerting (RBA) principles

• Extensible and configurable

– All fraud rules available as correlation searches and can be modified
– Application designed with data models as the source of all searches
– Macros used to define constraints (sources for data models)
Administering Splunk Enterprise Security
turn data into doing™ 316 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Appendix B:
ES On-Prem Deployment

Administering Splunk Enterprise Security

turn data into doing™ 317 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Objectives
• Identify on-prem deployment topologies
• Examine the deployment checklist
• Understand pre-deployment requirements

Administering Splunk Enterprise Security

turn data into doing™ 318 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Deployment Checklist
1. Determine size and scope of installation
2. Configure additional servers if needed
3. Obtain the ES software
4. Determine software installation requirements for search heads,
indexers, and forwarders
5. Install all ES apps on search head(s)
6. Deploy indexer configurations

Administering Splunk Enterprise Security

turn data into doing™ 319 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
ES Impact on Resources
• ES generally requires a new, dedicated search head or search head
cluster
– ES is only compatible with other CIM-compatible apps
– ES adds a large number of searches and search results

• Hardware must meet or exceed Splunk minimum requirements:

docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware
• ES increases some hardware requirements:
docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning#Splunk_E
nterprise_system_requirements

Administering Splunk Enterprise Security

turn data into doing™ 320 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Supported Architectures
• Single server (proof of concept, testing, dev)
• Distributed search (single search head, multiple indexers)
• Search head clustering
docs.splunk.com/Documentation/ES/latest/Install/InstallEnterpriseSecuritySHC
• Indexer clustering (including multi-site)
docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning

Administering Splunk Enterprise Security

turn data into doing™ 321 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Adding ES to an Existing Site
Before ES After ES
Pre-ES site with a single search head After ES install, ES increases
and 3 indexers supporting search requirements, adds an extra
~500GB/day of indexed data search head and an additional 5 indexers

Log on here for Log on here

Splunk search for ES
ES

Administering Splunk Enterprise Security

turn data into doing™ 322 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Search Head Requirements
• A dedicated server or cluster for the ES search head(s) with only CIM-
compliant apps installed
• 64-bit OS, minimum 32 GB RAM of memory and 16 processor cores
– Additional
memory and CPU capacity may be needed depending on
number of concurrent users, searches, etc.
• Configure search head forwarding:
docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

• If enabling Monitoring Console, do not use distributed mode

docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning#Monitoring_Console

Administering Splunk Enterprise Security

turn data into doing™ 323 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Indexer Requirements
• Increased search load in ES typically requires more indexers
– Assumeat most 100GB/day per indexer. For deployment planning use
80GB/day to be on the safe side
– Hardware minimum: 16 CPU cores, 32 GB RAM of memory

• The exact number of indexers required depends on:

– Types
and amounts of data being used by ES
– Number of active correlation searches
– Number of real-time correlation searches
https://docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning

Administering Splunk Enterprise Security

turn data into doing™ 324 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Indexer Cluster Requirements
• You can only enable ES on one search head or search head cluster
for each indexer cluster
• On a multisite indexer cluster:
– Enable summary replication to improve performance
docs.splunk.com/Documentation/Splunk/latest/Indexer/Clustersandsummaryreplication
– Disable search affinity
docs.splunk.com/Documentation/Splunk/latest/Indexer/Multisitesearchaffinity

• Deploy ES add-ons to the indexer cluster using the

Splunk_TA_ForIndexers app from cluster master

Administering Splunk Enterprise Security

turn data into doing™ 325 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Accelerated Data Model Storage
• In addition to index storage requirements, ES requires space for
accelerated data models
• Acceleration requires approximately 3.4 x (daily input volume) of
additional space per year, or more if replicated in an indexer cluster
• Example: input volume of 500 GB per day with one year retention
– 500 GB * 3.4 = 1700 GB additional space for accelerated data model storage
• Space is added across all indexers
– Example: if there are 5 indexers, 1700 GB / 5 = ~ 340GB per indexer
additional space is required

Administering Splunk Enterprise Security

turn data into doing™ 326 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
More About Accelerated Data Models
• Most ES searches are executed on accelerated data models
• The storage volumes allocated for acceleration should be tuned for
best performance and replicated if in a cluster
• By default, acceleration storage is allocated in the same location as
the index containing the raw events being accelerated
• Use the tstatsHomePath setting in indexes.conf if needed to
specify alternate locations for your accelerated storage
docs.splunk.com/Documentation/ES/latest/Install/Datamodels#Configuring_storage_volumes

Administering Splunk Enterprise Security

turn data into doing™ 327 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Indexed Real Time Search
• ES automatically configures Splunk to use indexed real time
searching
docs.splunk.com/Documentation/Splunk/latest/Search/Aboutrealtimesearches#In
dexed_real-time_search
• Improves concurrent real time search performance at the cost of a
small delay in delivering real time results from searches
• Leave turned on in ES for best performance

Administering Splunk Enterprise Security

turn data into doing™ 328 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Forwarder Requirements
• In general, forwarders are unaffected by ES installation
• However, some add-ons that ES depends on must be deployed to
forwarders to collect data
• Examples:
– Windows add-on
– *NIX add-on
– Splunk Stream add-on

Administering Splunk Enterprise Security

turn data into doing™ 329 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
App/Add-on Deployment Options
• Depending on your requirements, you may need to distribute add-
ons to other Splunk instances like search heads, indexers, and heavy
forwarders
• Use the appropriate app and add-on deployment methodology:
– Forwarders and non-clustered Indexers: use Forwarder Management
(Deployment Server)
– Indexer clusters: use the master node to deploy apps to peer nodes
– Search head clusters: use the deployer to deploy apps to cluster
members
docs.splunk.com/Documentation/ES/latest/Install/InstallTechnologyAdd-ons

Administering Splunk Enterprise Security

turn data into doing™ 330 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add-on Builder
• splunkbase.splunk.com/app/2962/
• Builds add-ons for custom ES data
• Normalizes custom data into the
Common Information Model
• Built-in validation
• Should not be used on
production servers

Administering Splunk Enterprise Security

turn data into doing™ 331 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Important Resources
• Splunk Education courses
– Splunk Data Administration
– Splunk System Administration
– Splunk Cluster Administration
– Architecting and Deploying Splunk

• Distributed Splunk overview:

docs.splunk.com/Documentation/Splunk/latest/Deploy/Distributedoverview
• Capacity planning:
docs.splunk.com/Documentation/Splunk/latest/Capacity/Accommodatemanysimult
aneoussearches

Administering Splunk Enterprise Security

turn data into doing™ 332 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Appendix C:
Use Case Library

Administering Splunk Enterprise Security

turn data into doing™ 333 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Use Case Library
Configure > All Configurations > Content > Use Case Library
• The Use Case Library contains analytic stories which are ready-to-
use examples of how to use ES to quickly identify the scope of
attacks, determine mitigation options, and take remedial action
• Analytic stories:
– Contain the searches needed to implement the story in your own ES
environment
– Provide an explanation of what the searches achieve and how to convert
a search into adaptive response actions, where appropriate
• Uses the Enterprise Security Content Update app

Administering Splunk Enterprise Security

turn data into doing™ 334 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Enterprise Security Content Update
• Splunk Enterprise Security Content Update (ESCU) add-on delivers
analytic stories to customers as
part of a content subscription service and
is updated often with new stories
• The app can be downloaded from Splunkbase
• Check ESCU app version: App > Manage Apps > ES Content Updates

Administering Splunk Enterprise Security

turn data into doing™ 335 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Use Case Library Permissions
• All ES users can view the analytic stories in the Use Case Library
• By default, ess_admin and ess_analyst roles have the ability to
edit the stories
• An admin can assign the ess_user role the Edit Analytic Story permission
Enterprise Security > Configure > General > Permissions

Administering Splunk Enterprise Security

turn data into doing™ 336 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Use Case Library (cont.)
Configure > All Configurations > Content > Use Case Library

Bookmark stories
specific to your duties

Choose a topic to focus

on related use cases

Administering Splunk Enterprise Security

turn data into doing™ 337 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Use Case Specifics

Expand an Analytic Story

Sourcetypes use by Lookups used by the
the detection detection searches for this
searches for this analytic story
analytic story

Detection Searches are Data Models used

correlation searches that by the detection
populate the story results searches for this
Recommended Data Sources analytic story
that are likely to provide
valuable data

Administering Splunk Enterprise Security

turn data into doing™ 338 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Analytic Story Details
Select a story from the library to view the details

Details on how to implement

the story

Correlation
Searches

Administering Splunk Enterprise Security

turn data into doing™ 339 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Appendix D:
Event Sequencing Engine

Administering Splunk Enterprise Security

turn data into doing™ 340 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Event Sequencing Engine
• The Event Sequencing Engine groups correlation searches into batches of
events, in a specific sequence, by specific attributes, or both
• Event sequencing is configured in Sequence Templates
• Sequence Templates:
– Define which Start, Transition, and End correlation
searches need to occur, and the match conditions
– Define if the transitional searches must occur in a
given order, or if they can occur in any order

• Templates run as a real-time searches and listen for incoming notable events
and risk modifiers that are triggered by the correlation searches
Administering Splunk Enterprise Security
turn data into doing™ 341 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Scenario
Give the template a
Create a template to detect high name and description,
and select the ES app.
priority hosts with multiple malware
infections, excluding test host
ACME-004. Then, detect if the host
has an abnormally high number of
HTTP method events, excluding
any “unknown” methods

2
Set the starting correlation search to
Endpoint – High Or Critical Priority Host With
Malware. Set the expression to detect all
destinations (dest) except ACME-004.

https://docs.splunk.com/Documentation/ES/latest/Admin/Sequencecorrelationsearches
Administering Splunk Enterprise Security
turn data into doing™ 342 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Scenario (cont.)
3 4 Set the ending correlation search to Web – Abnormally High
Add the transitional correlation search Endpoint – Number of HTTP Method Events By Src, and the expression
Host With Multiple Infections with the expression to
to detect all methods except unknown. Also, set the time
detect all destinations (dest) except ACME-004. limit for the template to run to 60 days.

Note 5 Add a title, urgency, and

In this template, the Enforce Ordering box security domain for the notable
has been unchecked. Therefore, the events that are created when
transitional searches do not have to the template is triggered.
happen in order, they just have to exist for
the template to trigger.

Administering Splunk Enterprise Security

turn data into doing™ 343 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Results
The results of the Sequence
Templates are Sequenced Events,
which are viewed in the
Incident Review dashboard

Transitions display the

correlation searches
matched in the template.

Administering Splunk Enterprise Security

turn data into doing™ 344 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Appendix E:
Using ES Overview

Administering Splunk Enterprise Security

turn data into doing™ 345 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
The Security Posture Dashboard

Key Indicators (KI) provide an at-a-

glance view of notable event status
over the last 24 hours

The panels provide additional summary

information categorized by urgency, time, and
most common notable event types and sources

Administering Splunk Enterprise Security

turn data into doing™ 346 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 KI Drilldown to Incident Review
1
From the Security
Posture dashboard,
click a KI total value

2
The details for the
KI opens in Incident
Review

Administering Splunk Enterprise Security

turn data into doing™ 347 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Drilldown Support
Hover over an item to
preview details about its
underlying notable events

1
Click an item to open the related notable
events in the Incident Review dashboard

2
From the Incident Review dashboard:
a. Drilldown into notables’ details
b. Take ownership
c. Work the issue

Administering Splunk Enterprise Security

turn data into doing™ 348 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Incident Review Dashboard
Use charts, filters, and search to
focus on specific notable events
Hide the donut
charts or filters

Expand for Add event(s) to an investigation

details Actions
Notable Events menu

Investigation bar

Administering Splunk Enterprise Security

turn data into doing™ 349 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Using the Incident Review Dashboard

• Search supports full SPL and wildcard search

• Adding one or more values per field, values are ORed together
• Urgency values can be toggled on and off
– Gray values are “off” and will not be searched
• If values are set for more than one field, the fields are ANDed together
• Status, Owner, Security Domain and Tag support multiple OR values
Administering Splunk Enterprise Security
turn data into doing™ 350 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Notable Event Details

Expand Notable event

the Actions menu
notable
event for
details
Fields for the
notable event, Field Action
with Action menus
menus for
each field

Note
You cannot expand an event until the
search is complete. Not all incidents
have all the same detail items.

Administering Splunk Enterprise Security

turn data into doing™ 351 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Create a Short ID from Event Details
Scroll to the bottom of the details for a notable event to see the Event
Details section and create a Short ID for the event

1
Click Create Short ID for ES to
automatically generate a short ID that
makes it easier to find and share a
notable event

2
The Short ID replaces
the Create Short ID link

Administering Splunk Enterprise Security

turn data into doing™ 352 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Create a Short ID: Notable Event Actions
1
From the
notable event
Actions drop-
down, creating a
Short ID is
possible using
Share Notable
Event

2
In addition to creating a Short ID, this enables sharing the event via a link:
• Click the Bookmark button to copy the link for sharing
or
• Click and drag the Bookmark button to your Bookmarks bar to save the link

Administering Splunk Enterprise Security

turn data into doing™ 353 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Search for a Short ID or Investigation

1
Select Associations from the Time or
Associations menu, and Short ID from
the Associations menu 2
Click inside the filter field and
enter all or part of a Short ID
(drop-down appears and filters as you type)
Or
Click and scroll to the Short ID

Note
You can search for one or
multiple Short IDs.
3
Click Submit

Administering Splunk Enterprise Security

turn data into doing™ 354 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Field Action Menu
• Each notable event field has an
Action menu allowing you to:
– Investigate the asset, set tags or
search Google. Depending on
the field type other options may
be available
• Risk scores for hosts or users
are displayed next to fields
– Click
a risk score to open the
Risk Analysis dashboard for that No
te
Scroll the menu to make sure you
asset or identity see all the available field actions.

Administering Splunk Enterprise Security

turn data into doing™ 355 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Notable Event Actions Menu
• Each notable event has an Actions menu with options related to the
event, such as:
– Adding the event to an investigation
– Suppressing the notable event
– Sharing the notable event with others
– Initiating further adaptive response actions

Administering Splunk Enterprise Security

turn data into doing™ 356 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Incident Workflow: Procedures
As needed, add
2
Click Edit selected event(s) to an
Selected investigation. It will
appear under Related
Investigations in the
1 event details 3
Select one or Set Status, Urgency, Owner,
more events and Disposition. Optionally,
add a Comment

4
Click Save changes
As needed, click the + icon on the
Investigation Bar to view an investigation,
add a new one, or click the spy glass to
perform a quick search

Investigation Bar

Administering Splunk Enterprise Security

turn data into doing™ 357 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Incident Review History

1
Select View all review activity for this Tip
Notable Event to open a new
The `incident_review` macro can
search showing all “review” events
for the current issue be used in custom searches and
reports for incident status tracking by
directly accessing the KV Store

2 The results show the reviewer,

urgency, status, and owner
changes for the event throughout
the review process

Administering Splunk Enterprise Security

turn data into doing™ 358 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Notable Event Adaptive Response
• Notable events may contain
further adaptive responses
that an analyst can initiate
(ping, nslookup, change risk,
run script, etc.)

• Depending on the type of

notable event, different actions
are available Adaptive Responses: Previously
executed actions
• Use Actions > Run Adaptive
Response Actions to trigger an
action Next Steps: If configured in the
correlation search, suggested
actions to trigger next

Administering Splunk Enterprise Security

turn data into doing™ 359 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Triggering Actions
Actions > Run Adaptive Response Actions
• Choose from a list of
actions to run

• This list is configured

by your ES admin Enter some, or all the action
name to filter
(list filters as you type)

• You may see different

options depending on
availability and
permissions

Administering Splunk Enterprise Security

turn data into doing™ 360 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Ping Example
As you investigate, you may need to see if the affected server is up
3
Host Field (event field with the host to
ping (i.e., dest, src, etc.)

2
4
Max Results: number of results for
the ping returns (default is 1)

Index and Worker Set are optional

Note
6 Find your action in the notable
If there is an investigation selected in the
event’s list of Adaptive Responses
Investigation Bar, Adaptive Responses will
and click Ping to view the results
display an Action column with the option to add
the response to the current investigation.

Administering Splunk Enterprise Security

turn data into doing™ 361 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Threat Intel Example
Similarly, you can add threat artifacts to a threat collection
(needs to be configured by your admin first)

Threat Group to attribute this artifact to

1 (i.e. iblocklist_logmein (threatlist))

Threat Collection to
2
add the threat artifact
to (i.e. ip_intel)

Field from event: a field in the

event containing the information
(i.e.dest)

Administering Splunk Enterprise Security

turn data into doing™ 362 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Send to UBA Example
Automatically send correlation
search results to Splunk User
Behavior Analytics (UBA)
Category in UBA

Severity sets the score

in UBA for the notable
1 event (optional)

Note
UBA must be installed on the ES
search head for this Response Action
to be available.

Administering Splunk Enterprise Security

turn data into doing™ 363 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Tagging Incidents
• Associate significant incidents with tags
– Example: quickly find all incidents
related to servers being used by
project “whammo”

• Add a tag to each server using Action

> Edit Tags for the dest, src or ip
field (for this example)
• Search for tag name “whammo” in the
Tag filter in Incident Review
• Now only notable events with this tag
value will display

Administering Splunk Enterprise Security

turn data into doing™ 364 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Audit > Incident Review Audit
• Overview of analyst
notable event handling
• Volume of incidents
reviewed and by whom
• Incident aging over last
48 hours, by status and
by reviewer
• Statistics on triage time
and closure time

Administering Splunk Enterprise Security

turn data into doing™ 365 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
The Executive Summary Dashboards
• Select Executive Summary or SOC Operations dashboard
• Provide summary of data over several time range options
• Action menus allow for search and refresh

Time Range

Action Menu

Administering Splunk Enterprise Security

turn data into doing™ 366 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Cloud Security Dashboards
Visualize the security of your Cloud infrastructure (AWS, Azure)
through several dashboards Important!
To onboard Cloud data sources and examine your
Cloud Security environment, you must install and
set up Splunk Add-on for Amazon Kinesis Firehose
and Splunk Add-on for Microsoft Office 365 from
Splunkbase.

Administering Splunk Enterprise Security

turn data into doing™ 367 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Using ES Investigations

Administering Splunk Enterprise Security

turn data into doing™ 368 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Investigation Artifacts
• Artifacts are assets or identities you may add to an investigation to
determine whether they are involved in the overall incident
• There are several ways to add an artifact to an investigation
– From a notable event (set up by an admin)
• Actions > Add Event to Investigation
– Manually
• Add Artifact button
• Add Artifact icon on the Investigation Bar
– From a workbench panel (select any item)
– From an investigation event (Timeline View > Details > click a value)

Administering Splunk Enterprise Security

turn data into doing™ 369 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Manually Add Artifacts
1. Click Add Artifact button or click
2. Select Add artifact or Add multiple artifacts and enter the artifact(s) (all
artifacts added must be the same type: assets or identities)
3. Select either Asset or Identity artifact
4. To separate multiple artifacts, click New Line or use a comma
5. Optionally, add a Description and Label(s) (separate labels with <Enter>
or <,>)
6. Optionally, Expand artifact (seeks correlated items from lookups)
7. Click Add to Scope

Administering Splunk Enterprise Security

turn data into doing™ 370 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add Artifacts within the Investigation

1
When exploring, click a value
to add it as an artifact

2
Enter details and click
Add to Scope

Administering Splunk Enterprise Security

turn data into doing™ 371 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Add Items to an Investigation
It is important to add items to investigations to document the purpose of the
steps you have taken to research the issue and to provide any details that
may be useful to your team’s future investigation work. You can add several
types of entries:
• Notes • Action History items:
- Dashboards viewed
• Search strings - Notable Event Updated
• Notable or source events - Notable Event Suppression Updated
- Panel Filtered
- Search Run
Enable Add Quick Add Action
Livefeed Artifact Search Notes History

Investigation Bar

Administering Splunk Enterprise Security

turn data into doing™ 372 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding a Note
1
Enter a title 2
2 Modify time as
Click to add a note needed
default = now

Note
If you create a standard note, and do 3
Enter comments
not check the Show on Timeline box,
the note will show under Notes as a
“draft” note.

4
Add attachments (text or binary
1
Click to view notes format). 4MB max per file and 5
are stored in KV Store.

Administering Splunk Enterprise Security

turn data into doing™ 373 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
 Adding an Action History Item
4
2 Filter search
Select type 3 as needed
Modify time
as needed

5
Select items
6

Administering Splunk Enterprise Security

turn data into doing™ 374 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding a Search String (Quick Search)
• Perform a search from the Investigation Bar and add the string to an
investigation 1

Click and drag to resize

the search window.
Double click to toggle full
screen to minimized 2 3
Enter search criteria

• Analyst can run the

saved search to
view the results 4
Determine whether the
while investigating results are useful to the
investigation

Administering Splunk Enterprise Security

turn data into doing™ 375 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding Events
There are several ways to add events to an investigation

Add notable events

from Incident Review
or
or
Add source events from
a search result

Administering Splunk Enterprise Security

turn data into doing™ 376 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Enabling Notable Event Livefeed
• Get a visual notification when a notable event occurs for assets or
identities included in the investigation
– Select an investigation, click the bell icon, and toggle Enable Notification
– Bell icon turns orange within five minutes of the next occurrence Enable
Livefeed

Review events and use the plus sign (+) to

add events to the investigation

Administering Splunk Enterprise Security

turn data into doing™ 377 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Adding Collaborators to an Investigation
1
Click + to add a
collaborator
3
Click a collaborator initial to remove
or change write permissions 2
Search and/or click a
username to add as a
collaborator

4 Select whether they have “write

permission” and click Done

Administering Splunk Enterprise Security

turn data into doing™ 378 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Updating Investigation Status
• When you open an
1
investigation, its status
is New
• Investigations can only
be deleted by admins
• Analysts can delete 2
Edit the Title, Status, and
investigation entries Description of the investigation

Administering Splunk Enterprise Security

turn data into doing™ 379 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Investigation Summary View

Expand for details

Examine the correlation search that

created the notable event

Click to open the event in Incident Review

Click to examine the

source notable event

Administering Splunk Enterprise Security

turn data into doing™ 380 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Timeline: Slide View

Filter by Type: Action History, Adaptive Response Action,

Search String, Notable Event, Note, Splunk Event Edit, delete, or open
in Incident Review

Scroll left Scroll right

(newer) (older)

Click an item to view its

details in upper panel

Administering Splunk Enterprise Security

turn data into doing™ 381 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Timeline Details View to Add Artifacts
1
Click Details for a detailed view of all fields and values

Add Artifacts view opens

and auto-populates

2
Click an item to add it
as an artifact

3 Enter a type,
description and
label as needed

Administering Splunk Enterprise Security

turn data into doing™ 382 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Timeline: List View
From Timeline, change
view to List View
Use the Action menu to
delete selected entries

View
details
Edit or delete entries or open in
Incident Review

Administering Splunk Enterprise Security

turn data into doing™ 383 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Edit Investigation Entry
1

Click Action and select Edit Entry to

change the title of the entry

2
Enter new title and Save

Administering Splunk Enterprise Security

turn data into doing™ 384 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022
Investigation Bar and Inline Timeline View
Select an investigation from the list
or click + to add a new one

Toggle the
Investigation
Timeline

Inline Investigation Timeline

Investigation
Timeline Zoom Entries

Jump to start

Administering Splunk Enterprise Security

turn data into doing™ 385 Copyright © 2022 Splunk, Inc. All rights reserved | 25 May 2022