Administering Splunk Enterprise Security Lab Exercises
Administering Splunk Enterprise Security Lab Exercises
Lab Environment
Welcome to the Splunk Education lab environment. Your instructor will give you the IP address to access your
Splunk server which has the Enterprise Security app installed. You will access the Splunk Web interface via
HTTP using a browser on your local computer.
The lab environment is running a testing app called SA-Eventgen that is generating simulated source events.
In a production environment, these events would be generated by Splunk forwarders, which would gather data
from your network’s servers, routers, and applications. Your lab event data only goes back as far as the time
the lab server was set up—probably only a day or so.
Important: Please disable popup blockers, ad blockers, and clear your cache (or use incognito mode).
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 1
Module 1 Lab Exercise – Overview of Splunk Enterprise Security
Description
In this exercise, you will get familiar with your lab environment and access the ES user interface.
Steps
Task 1: Log on to your lab Splunk server and navigate to the ES home page.
5. Click the splunk>enterprise button at the top of the window to return to the Splunk home page. Notice
which apps are visible on the home page.
11. From the Splunk main menu select the Administrator drop-down and select Preferences. Notice that
there are two Preferences tabs, Global and SPL Editor.
12. On the Global window, set your local time zone and select Enterprise Security as the Default
application.
13. On the SPL Editor window, toggle the Advanced editor button, then toggle the Line numbers button.
14. Click the Themes tab and select Dark Theme. Click Apply.
15. Click the splunk>enterprise button and notice that you are directed to the Enterprise Security
homepage because it was set has the default application above.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 2
Task 4: Examine the source events in Splunk that ES is using to monitor the security environment
and review the notable events in the notable index.
16. In ES, select Search > Search to run a search using Splunk Search Processing Language (SPL). This
page is very similar to the Search and Reporting app you have used before.
17. Begin a search for all events coming into the main index (index=main) over the Last 15 minutes.
Note: If the search runs for more than 30 seconds, you can stop it before it completes.
18. Examine the results. You will probably have many events. From the result count of this search you can
determine how many events Splunk is seeing per day. Also, look at the sources and source types—this will
give you an understanding of the type of systems being monitored.
19. Examine the variety of source (src) and destination (dest) IP addresses and hostnames. Other fields to
notice are host, signature, and eventtype.
20. Select All Fields and select the checkbox for a couple fields you would like to add to the search
results and close the Select Fields window. Notice that the new fields are added under SELECTED
FIELDS.
21. Run a new search for all events in the notable index over the Last 24 hours.
22. Note the number of results and compare this to the total number of indexed events in the main index over
the Last 15 minutes. This demonstrates how useful notable events are; you do not need to search
through all the data to find the events that need attention.
23. Examine the values in the source field. These are the correlation search names that created the notable
events.
24. Examine some of the other discovered fields. Note that they are extracted from the source events, so they
will be similar to what you saw in the main index.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 3
Module 2 Lab – Monitoring with ES
Description
In this lab exercise, you will use the Security Posture dashboard to monitor your organization for
unauthorized network access and the Incident Review dashboard to work an incident. Next, you will use the
Incident Review dashboard to find false positive events coming from a set of test servers, use the Adaptive
Response Ping action to check the status of the servers, and finally suppress the notable events coming from
the servers.
Steps
Scenario: You are investigating reports of unauthorized access to your network resources.
9. On the Incident Review dashboard, search for the username Hax0r for the last 24 hours. You should find
one or more notable events for this user.
Hint: Use the Search field to enter the user ID and remember to click Submit.
Note: Unless otherwise indicated, execute all subsequent searches using a time range of Last 24
hours.
10. In the results, view the details for one of the notable events.
11. Examine the details of the Original Event. Some of this data may be useful to determine the seriousness
of this vulnerability.
12. Click the View activity from Hax0r link to see source events associated with this username.
This opens a new search window that uses a custom 10-minute time range which references the creation
time of the notable event (5 minutes before and 5 minutes after). This allows you to see source events that
were indexed immediately before and after the notable event.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 4
13. Expand an event and note the fields Type=Failure Audit, Message=Logon Failure, UserName=Hax0r,
and Source Network Address. You may have to click the Show all n lines link to view all the fields.
Note: In this case, Type=Failure Audit and Message=Logon Failure indicate failed logon
attempts. So, Hax0r is not actually authenticating, but the fact that someone is attempting to use this
expired account is an issue.
14. Close the search window and return to the Incident Review dashboard.
15. In the Incident Review dashboard, make sure the search results are still filtered to the user Hax0r.
16. Click Edit All n Matching Events.
17. Set the status to In Progress and click Assign to me. Save the changes.
18. Notice the change in status and ownership.
Note: Now you can begin working with network analysts and others to research and resolve the issue.
While this takes place, you will still need to review new incidents.
19. Click Incident Review.
20. Set the Status filter to New, then click Submit.
Notice that you no longer see your in-progress Hax0r incidents. You would do this to see only New
incidents requiring attention. They would normally then be assigned to an owner and their status changed
to show they are In Progress or Resolved.
21. Reset the Incident Review dashboard.
22. In the Owner filter, select yourself (admin) and run the search again.
Now you only see your incidents. This is a typical way to view your queue of assigned incidents.
Task 4: You have resolved the Hax0r issue by hardening a firewall asset. You can now resolve your
incident.
23. Reset the Incident Review dashboard and search for all Hax0r incidents.
24. Click Edit All n Matching Events.
25. Change the Status to Resolved and enter a Comment of “Firewall hardened”, click Save changes.
In the future, you probably want to see only unresolved, open incidents.
26. Clear the dashboard and search for open incidents by selecting all status values except Resolved and
Closed in the Status field. The Hax0r events should not appear in the search results.
Tip: You can also use the Search field to find events by status, owner, etc. For instance, use the
status field (lower case) and the status values: 0 = “Unassigned”, 1 = “New”, 2 = “In Progress”, 3 =
“Pending”, 4 = “Resolved”, and 5 = “Closed.” For example, enter status<4 in the Search field to filter
out Resolved (4) or Closed (5) events.
Task 5: Configure the Incident Review dashboard to display a new column named Source with
information from the src field.
27. From the Configure menu, select Incident Management > Incident Review Settings.
28. From the Incident Review – Table Attributes list, select Add Column.
29. For the new column, enter src in the first field and Source in the second field.
30. Use the vertical dotted lines on the left of the new column to move it under the rule_title/Title column.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 5
31. Click Save at the bottom left of the Incident Review Settings window.
32. When the changes are successful, click Close.
33. Click Incident Review and confirm that the new Source column is displayed. You may have to click the
sort ascending/descending arrows to see source information.
Scenario: Several false positives have been generated and are coming from a set of servers named
PROD-MFS-XXX, which are a set of QA lab workstations used to test production security
configurations. You want to first determine the workstation’s status—are these
workstations still online? You will ping them to see.
Scenario: Now that you know the status of the test systems, you will close out the affected false
positive notable events.
48. In Incident Review, make sure you are still displaying the Endpoint events for the PROD-MFS-*
workstations.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 6
49. Click Edit All n Matching Events.
50. Change the status to Closed and in the comments, enter: False positive generated by testing process.
51. Click Save changes.
Scenario: You have closed the PROD-MFS-XXX false positives, but new notable events will still
occur. You would like to suppress them for the rest of the testing project.
52. Clear the Incident Review dashboard and search for PROD-MFS-* notable events. Several notable event
types appear, including malware infections and access anomalies. For our purposes, use an event from
the Host With Old Infection Or Potential Re-Infection correlation search as an example of a notable
event type that is a false positive and should be suppressed.
53. On the notable event, click the Actions menu and select Suppress Notable Events.
Review the fields of the Suppress Notable Events window, these are used to create an event type that
will be used by the correlation search engine to determine if a notable event should be generated or not. If
an event matches the event type’s time range and search criteria, it will be ignored by the correlation
search.
Notice the Search Preview. The suppression uses source=(correlation search name) as one of
its criteria. Therefore, the suppression only suppresses this type of correlation search. Also notice the
Selected Fields are dest and signature. Only events matching this destination and signature are
suppressed (i.e. PROD-MFS-006/LeakTest).
54. In the Suppress From … To fields, select a range from now until a date 6 months in the future.
55. Click Save to return to Incident Review.
Notable events will be hidden for this destination and signature for the next 6 months. You are
suppressing only one of several servers for the purpose of this exercise. If many servers were identified as
false positives in a production environment, you could locate this event type in Settings > Event types by
searching for notable_suppression across all apps and editing the event type’s search string to include
wildcards in the dest and/or signature field values. You can edit the event type in the future if necessary;
for instance, you could add additional selected fields to make the event type more restrictive, or you could
alter the search expression to use a wildcard instead of a specific hostname to make it generalize to a set
of servers.
56. From the Audit menu select Suppression Audit. View the data in the panels including suppressed events
over the last 24 hours, suppression history over the last 30 days, and suppression activity.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 7
Module 3 Lab – Investigating with ES
Description
In this lab exercise, you will create an investigation to monitor activity from expired user accounts.
Steps
Scenario: Looking at the events on the Incident Review dashboard over the last 24 hours, you notice
that there are a number of events from expired user accounts. You will open an
investigation for further research.
1. Click the Incident Review dashboard and make sure Correlation Search is selected and enter Activity
from Expired User Identity or select the search from the drop-down list. Select a Time of Last 24
hours and click Submit.
2. Select the checkboxes for all of the results found on the first page and click Add Selected to
Investigation.
3. From the Add Event to Investigation window review the events being added to the investigation and click
Create Investigation.
4. From the Create New Investigation window, name the investigation Activity from Expired User
Accounts.
5. Change the Status to In Progress and click Save.
6. Open the link for the new investigation. If prompted with the Artifact Extraction window, review the
information and click Ok. The warning shows that there are duplicate entries for these artifacts.
7. Under Artifacts on the investigation page, click the Select all link and click Explore.
8. Examine the tabs and panels on the right. Notice that areas were automatically populated with more
information for you to explore and document.
9. Select the Endpoint Data tab and scroll down to Authentication Data.
10. From the first entry under Authentication Data, add the following artifacts to your investigation: src,
src_user, and dest.
11. Complete the following for each new artifact:
12. Click each artifact and the Add Artifacts window opens.
13. The Artifact field is populated with the artifact name.
14. From the Type drop-down select Asset for src and dest, and Identity for src_user.
15. Click Add to Scope.
16. The new artifacts are added to the Artifacts window on the left. Select the radio button for each of the
newly added artifacts (or click Select all) and click Explore.
17. Review the new information that has been added under the Endpoint Data and Network Data tabs.
Task 2: Add a tab called IDS / IPS Activity that displays detailed information on panels focused on
Cisco Sourcefire activity
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 8
20. From the New Workbench Panel window enter the following information:
o Panel Name: filter on Cisco and select Cisco Sourcefire – Activity by Dest.
o Label: enter Cisco Sourcefire – Activity by Dest.
o App: select Enterprise Security.
o Description: enter Sourcefire activity by destination.
o Tokens: click Add Token and enter a Token Name of $dest$ and select a Type of Asset.
21. Click Apply.
22. Click Save.
23. Click the Create New Content button and select Workbench Panel to add a new workbench panel for
Cisco Sourcefire – Activity by Src.
o Panel Name: filter on Cisco and select Cisco Sourcefire – Activity by Src.
o Label: enter Cisco Sourcefire – Activity by Source.
o App: select Enterprise Security.
o Description: enter Sourcefire activity by source.
o Tokens: click Add Token and enter a Token Name of $src$ and select a Type of Asset.
24. Click Apply.
25. Click Save.
26. From the Content Management page, add a workbench profile by clicking on Create New Content and
selecting Workbench Profile. Enter the following information:
o Profile Name: enter Intrusion Detection.
This becomes the stanza name in es_investigations.conf and is used as the label if the
label is not specified.
o App: select Enterprise Security.
o Label: enter Intrusion Detection.
Enter a label name if you want the profile tab to named something different than the Profile Name.
o Description: enter Detects IDS/IPS activity.
27. Click Save.
28. From the Content Management page, add a workbench tab by clicking on Create New Content and
selecting Workbench Tab. Enter the following information:
o Tab Name: enter IDS / IPS Activity.
o App: select Enterprise Security.
o Label: enter IDS / IPS Activity.
o Workbench Profile: select the Intrusion Detection profile created above.
o Workbench Panels: select the Cisco Sourcefire – Activity by Dest and Cisco Sourcefire –
Activity by Src workbench panels created above.
o Load by Default: select False.
Set Load By Default to True to have the tab load in the Investigation Workbench by default. In this
case we are setting it to False to manually add the tab to the investigation.
o Description: enter Detects IDS/IPS activity from Cisco Sourcefire.
29. Click Save.
30. Navigate to Investigations and click the Activity from Expired User Accounts investigation.
31. In the investigation, click the Select all link under Artifacts and click the Explore button.
The Investigation Bar is displayed at the bottom of ES windows where investigations are referenced.
Icons include, notify on new related notables , add an investigation artifact , run a quick search ,
add a note, and add an action history item to an investigation .
36. In the Related Notable Event Livefeed for Last 48 Hours window, toggle Enable notification.
37. Any new events involving expired user accounts for the last 48 hours are listed here. Use the plus sign (+)
to select one or more of the events and add them to your investigation. You may add all or some of the
events.
38. If you are done reviewing and have left some events deselected, click Mark All as Seen.
39. Click the Close button.
40. The next time a notable event is generated for the Activity from Expired User Identity correlation
search the icon will change from gray to orange. You can then add the new events to your investigation if
needed.
41. From the Activity from Expired User Accounts investigation window, add a note by clicking the Notes
icon on the Investigation Bar.
42. On the Investigation notes window click +Add new Note.
43. In the Create Note window, enter the following values:
Title: Starting investigation
Note: Noticed an unusual number of events from expired user accounts under Incident Review
in the last 24 hours.
44. Click Add to investigation and notice that the new note is added under Notes. Close the Investigation
notes window.
45. From the Investigation Workbench, select Timeline > Slide View. From the Type: drop-down select
Note. Notice that no notes display under the Slide View. Click List View, and notice that the new note is
displayed but is prepended with Draft:. Let’s add this note to the Timeline Slide View.
46. Click the Notes icon , and click on the Starting Investigation note. The Edit Note window opens.
47. Check the Show on timeline box and click Save. Notice the Starting Investigation note is now under
Timeline Notes.
48. Close the Investigation notes window.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 10
49. From the Investigation Workbench, select Timeline > Slide View and notice that the Starting
Investigation note now displays in the Slide View. Click List View and note that the title is now
prepended with Note:.
Note: You can make a note visible on the Timeline by using the +Add new Timeline Note option in the
Investigation notes window, or by checking the Show on timeline checkbox in a standard note.
50. Add a search by clicking the Quick Search icon in the Investigation Bar.
51. Run the search index=main Type="Failure Audit" for the last 60 minutes.
52. Click Add Search String to Investigation.
53. Close the Quick Search window.
54. Navigate to Security Intelligence > User Intelligence > Asset Investigator.
55. Enter hostname HOST-001 in the Search for asset field and select a time range of Today, click Search.
Note: The Time Range Picker is below the list of swimlanes.
56. Click some of the colored bars and review the detailed information that is populated in the panel on the
right. Remember, the darker color bars mean more events.
57. Navigate to the Investigation Workbench by clicking Investigations.
58. Click the Activity from Expired User Accounts investigation.
59. From the Investigation Bar, click the Action History icon .
60. From the Select action history type drop-down, select Dashboard Viewed and Last 30 minutes, click
Search.
61. Select the checkbox for the asset_investigator dashboard viewed and click Add to Investigation.
62. From the Activity from Expired User Accounts investigation window, select Timeline and toggle
between List View and Slide View to review all of the entries you created during the investigation. Use the
Type: drop-down to filter the entries (Action History, Note, Search String, etc.).
Note: Any physical actions can also be logged in the timeline, as well as scans of any pertinent
documents, copies of files, etc.
63. For a higher-level view of your investigation, navigate to the Summary view.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 11
Module 4 Lab Exercise – Dashboard Data Sources
Description
As a security administrator, you need to be able to debug issues in ES when dashboards fail to display all
needed data.
Steps
Task 1: Examine dashboard data sources.
4. Click the spyglass icon to open the panel’s search in a new search window.
5. Examine the search SPL and the results.
6. Note the name of the data model and object: Web.Web.
7. Note that the search is based on |tstats `summariesonly`, which returns results only from accelerated
data. If the panel is not displaying all expected results, acceleration may be an area to examine.
8. To view or change the acceleration of a data model navigate to Configure > Content > Content
Management and select Type: Data Mode and filter on web.
9. Notice the yellow lightning bolt which means the Web data model is accelerated. A black lightning bolt
12. Take a closer look at the Web data model. From the Splunk Settings menu select Data models.
13. Filter the list for web. Click the Web data model. Review the following information:
o Data sets: A data model is a hierarchically structured search-time mapping of semantic
knowledge about one or more datasets. Each dataset has a parent and one or more child
datasets. Datamodel and datasets are referred to as datamodel.dataset like Web.Web.
What are the two datasets in the Web data model? _____________________________________
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 12
o Constraints: are used to filter out events that are not relevant to the dataset. In the Web.Web data
model dataset, what is the constraint search that filters out events?_________________________
Child datasets inherit constraints and fields from their parent datasets and have additional
constraints and fields of their own. For example, the Proxy dataset inherits the results from the
(`cim_Web_indexes`) tag=web constraint. What constraint does the Proxy dataset use to
further filter data? _______________________________
o Inherited: All datasets have at least a few inherited fields. Child datasets inherit fields from their
parent dataset, and these inherited fields always appear in the Inherited category.
o Extracted: A field extracted by Splunk at index time or search time. You can only add auto-
extracted fields to root datasets. Child datasets can inherit them, but they cannot add new auto-
extracted fields of their own.
o Calculated: If a dataset has eval expressions, regular expressions, lookups, or Geo IP field types,
they appear in this field category.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 13
Module 5 Lab Exercise – Plan a Deployment
Description
Scenario: You are working with a customer that has Splunk installed on a distributed site. There is
one search head and 4 indexers inputting 800 GB of parsed data per day, with a retention
period of 1 year. The customer is marginally happy with current performance. All servers
are at basic minimum Splunk hardware levels. No new data inputs are planned after
installing ES.
Steps
Task 1: Identify configuration changes to make before installing ES.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 14
Module 6 Lab Exercise – Post-installation Tasks
Description
In this lab exercise, you will examine how your ES server has been configured. You also verify access to your
server and perform initialization procedures.
Steps
Task 1: Disable un-needed apps.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 15
Module 7 Lab Exercise – Initial Configuration
Description
In this exercise, you will carry out some initial configuration tasks: customize Key Indicators, establish
dashboard permissions, and modify ES navigation.
Steps
Task 1: Configure key indicators.
7. Click Edit on the Key Indicator panel, then click the plus sign to add a new indicator. Select the IDS -
High Severity Attacks indicator and click Add indicators. Select the checkmark to save the
changes.
8. Observe the new indicator on the dashboard.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 16
20. Log out of Splunk Web and test user access by logging back in with user account analyst and password
b0ss0fth3s0c.
21. You are now logged in as an ES Analyst user.
22. Navigate to the ES application and open the Search menu. Note the Predictive Analytics menu item is
gone.
23. Select Search > Dashboards and look for the Predictive Analytics dashboard.
Because you removed Read access for all roles except admin, analysts do not even see the dashboard
listed on this page.
24. Log out and log back in as admin.
25. Click the logo in the upper-left corner. Notice that this takes you to the Splunk
Enterprise Security home page because that is the default app set for the administrator account
Preferences in Lab 1.
26. Your users would like to have the Security Posture dashboard open instead, so you will need to modify
the navigation settings.
27. Navigate to Configure > General > Navigation.
28. Notice that the Home item has a checkmark enabled. This indicates it is the default view for the ES app.
You want the Security Posture dashboard to be the default view.
29. Click the checkmark on the Security Posture item. The checkmark should dim on the Home item and
enable on the Security Posture item. Security Posture is now the default view for ES.
30. You decide the Identity dashboards will be frequently used, so you want to make them more accessible.
31. Locate the Identity sub-menu under the Security Domains menu. You will have to scroll down the list of
items under the Security Domains menu to the bottom to see the Identity menu.
32. Click and drag the Identity sub-menu out of the Security Domains menu. Move it so it appears between
the Audit and Search menus.
33. Click Save and OK to apply your changes and close the navigation editor.
The Identity menu should now appear on the app navigation bar.
31. Click or select Enterprise Security from the App: menu. Notice that the Security
Posture dashboard now displays as the default view.
Scenario: As an ES Admin you have a SOC manager that currently uses the soc_analyst user
account with the ess_user role. The manager needs the ability to edit notable events,
change ES navigation, and view all investigations which the current role does not support.
You decide to create a new role and user account for the SOC manager.
Task 5: Review the capabilities of the soc_analyst user account with the ess-user role.
32. Log out of Splunk and log in with username soc_analyst and password b0ss0fth3s0c.
33. Click on Incident Review and select the checkboxes for a couple of notable events. Notice that Edit
Selected is grayed out because the soc_analyst user account does not have the ability to edit notable
events.
34. Navigate to Configure > General > Navigation. Notice that the soc_analyst account does not have the
capability to edit ES navigation.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 17
35. Click on Investigations. Notice that the Activity from Expired User Accounts investigation created
earlier is not listed because the soc_analyst user account is not the owner of the investigation or a
collaborator of the investigation.
Task 6: Create a SOC manager role and give it specific permissions.
36. Log back into Splunk with username admin and password s3cur3m3.
37. Navigate to Settings > Roles.
38. Click New Role.
39. Enter soc_manager as the name of the role.
40. On the Resources tab select SplunkEnterpriseSecuritySuite as the Default app and leave the other
fields at the defaults.
41. On the Inheritance tab select the user, ess_analyst and ess_user roles.
42. On the Capabilities tab notice that the edit_notable_events capability is inherited. Select the checkboxes
for edit_es_navigation and manage_all_investigations.
Note: you can use the Capability Name filter to search by the capability name.
43. Click Create.
44. Navigate to Settings > Users.
45. Enter the name of the user as soc_manager.
46. Enter a password as p@ssw0rd.
47. Select a time zone for the user.
48. Set the Default app as SplunkEnterpriseSecuritySuite (Enterprise Security).
49. From the Assign to roles click soc_manager.
Remember: the soc_manager role inherits the ess_analyst and ess_user roles.
50. Uncheck Create a role for this user.
51. Uncheck Require password change on first login.
52. Click Save.
Task 7: Confirm the capabilities of the new SOC Manager account.
53. Log out of Splunk and log in using the new soc_manager account. Click Skip if prompted with a message
to continue the tour of Splunk.
54. Click on Incident Review and select the checkboxes for a couple of notable events. Notice that the
soc_manager user account has the ability to edit notable events.
55. Navigate to Configure > General > Navigation. Notice that the soc_manager account has the capability
to edit ES navigation.
56. Navigate to ES Investigations. Notice that there are no investigations under Investigations Assigned to Me
but this role now has a tab called All Investigations.
Hint: the new user has not created any investigations of their own.
57. Click on All Investigations and confirm that you can see the Activity from Expired User Accounts
investigation created earlier. Remember, this investigation was created by the admin user account.
Scenario: As an ES Admin you have the ability to assign specific permissions to the ess_user and
ess_analyst roles within Enterprise Security.
Task 8: Enable specific ES permissions for the ess_user and ess_analyst roles.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 18
60. Enable Edit Notable Events and Own Notable Events for the ess_user role.
61. Enable Edit Sequence Templates for the ess_analyst role.
62. Click Save.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 19
Module 8 Lab Exercise – Validate ES Data
Description
In this exercise, you will identify industry-standard data sources, Splunk add-ons, and source types. Then you will
normalize a set of indexed Cisco ASA events to the Common Information Model (CIM) using a Splunk add-on.
Steps
Task 1: Plan and verify inputs.
1. Assume you will be using the following vendors and technologies in your ES implementation. Use the
documentation at docs.splunk.com/Documentation/ES/latest/Install/InstallTechnologyAdd-ons to identify
the add-on and source type that is needed.
2. In the following table, for each vendor/technology listed, use the documentation above to determine each
required add-on and which source type applies to each item. If more than one source type applies to a
given vendor/technology, use a wildcard, such as acme:*.
Bluecoat ProxySG
3. Verify Splunk is inputting the above source types. For each source type, run a search in ES over the last
hour and confirm that there are events being generated for each source type.
Bluecoat ProxySG
4. Run this search over the last 60 minutes (searches that reference a data model take longer than normal to
complete):
| datamodel Network_Traffic All_Traffic search | stats count by sourcetype
5. Confirm that the Juniper, Sophos and Cisco source types are present in the search results. This confirms
normalization is working.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 20
7. Confirm that the cisco:sourcefire source type is present in the results.
8. From the Security Domains menu, confirm there is data in the following dashboards and review the data.
o Access > Access Center
o Endpoint > Malware Center (New Malware – Last 30 Days panel might be empty)
o Network > Intrusion Center (New Attacks – Last 30 Days panel might be empty)
o Network > Vulnerability Center (New Vulnerabilities might be empty)
9. Use the Data Model Audit dashboard to get an idea of the performance of the data models used by ES.
Navigate to Audit > Data Model Audit.
11. Which data model takes the most time to execute (runDuration (s))? ___________________________
12. Which data models are not contained in the CIM app?
Task 3: Install a new Splunk technology add-on to automatically normalize Cisco ASA events.
Out of the box, the ES app does not include an add-on to recognize log events from the Cisco ASA firewall
appliance. You will install the add-on for Cisco ASA and verify it is working properly.
13. Navigate to Search > Search and run the following search over the Last 60 minutes:
sourcetype=cisco:* | stats count by sourcetype
14. Note that you see cisco:asa, cisco:fwsm, cisco:pix, and cisco:sourcefire source types.
15. To check for normalization, run this search over the Last 60 minutes:
| datamodel Network_Traffic All_Traffic search | search sourcetype=cisco:*
| stats count by sourcetype
You see cisco:sourcefire, but not the others. This is because the Splunk Add-on for Cisco FireSIGHT
ships with ES, but not the Splunk Add-on for Cisco ASA. You need to install it to normalize the other
source types. Your instructor will provide the app package.
16. In the App menu, select Manage Apps, then click Install app from file.
17. Under File, click Choose File and browse to the Cisco ASA add-on (ta-cisco-asa.tgz) provided by your
instructor.
18. Click Upload to install the add-on.
19. After the add-on is installed, navigate to ES > Search > Search.
20. After a couple minutes, run the data model search again:
| datamodel Network_Traffic All_Traffic search | search sourcetype=cisco:*
| stats count by sourcetype
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 21
You should now see the cisco:asa, cisco:fwsm and cisco:pix source types. Because the
Network_Traffic data model powers the searches for most of the network-oriented dashboards and
several correlation searches in ES, you have successfully normalized log data from your Cisco ASA
appliance into ES.
22. To force the acceleration update, navigate to Settings > Data models, locate the Network Traffic data
model, expand the row to see the data model details, and click Update.
23. Wait a few minutes, then run the tstats search again—you should now see all 4 source types.
Any correlation search or dashboard searches using accelerated data will now see all the events.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 22
Module 9 Lab Exercise – Building a Custom Add-on
Description
Your environment has a locally-built audit system running to manage transactions between systems during
online purchase situations. This data is being ingested by Splunk but is currently not available in ES because it
is not being normalized to any CIM data models. You will plan and implement a custom add-on which maps
the events to the correct data model and normalizes the fields required by ES. The events you need to
normalize have a source type of bcg:accounting.
A discussion with a local sys admin determines the following information about the bcg:accounting source
type:
“The device field is the hostname of the accounting machine. policy is an internal code used by
accounting. service is the network protocol. action can be “allow” or “deny”—deny would only
happen if the transaction was invalid. sent and received show the number of bytes in the actual
transaction, and CRC is the checksum for the transaction. host_from and host_to are the
hostnames of the two machines involved in the transaction, along with their ports.”
Steps
Scenario: Based on this discussion, you decide to normalize this data into the Network Traffic data
model so that the events will be available in ES searches and views that depend upon that
data model.
1. Open the documentation for the Network Traffic CIM data model:
docs.splunk.com/Documentation/CIM/latest/User/NetworkTraffic
2. In the following table, identify each CIM field in the Network Traffic data model that corresponds to each
field in the bcg:accounting source. If the source field does not correspond to any CIM field, mark the CIM
Field column N/A. In the Action column, indicate which action is required for normalization: Field Alias
(field name change), val (value change or creation), or N/A (i.e., no action is required).
Source Field CIM Field Action
device
policy
service
action
sent
received
host_from
host_to
src_port
dst_port
CRC
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 23
3. From the ES app, select Search > Search and run the search over the Last 24 hours:
index=main sourcetype=bcg:accounting
Note there are many events being generated for this source type.
Also note there are no tags—so this source type is not being normalized to any of the CIM data models.
You will also see a system message to restart Splunk. This is not required at this time. You can delete the
system message.
8. From the app navigation bar, click Manage Source Types.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 24
20. Expand the Network Traffic data model and then check the checkbox to the right of the All Traffic data
model object. The fields in the Network Traffic/All Traffic object are displayed on the right.
21. Click Select (green button in the upper right).
Now you have the Network Traffic data model’s All Traffic object fields displayed on the right, and you
can begin mapping bcg:accounting fields to the corresponding CIM Network Traffic data model fields.
Note that the action and src_port fields are green—they do not need an alias, since their names already
match.
22. For each field in the table above that is marked as Field Alias do the following (ignore fields marked N/A
or Eval):
• Select New Knowledge Object > FIELDALIAS to add a new row.
• Click on a source field on the left to fill in the Event Type Field or Expression.
• Click on a target field on the right to fill in the Data Model Field.
• Click OK to save the new field alias mapping.
23. For each field in the table above marked Eval (the action field) do the following:
• Select New Knowledge Object > EVAL.
• Enter the following in Event Type Field or Expression:
case(action="allow","allowed",action="deny","blocked")
• For the Data Model Field, click the action target field on the right.
• Click OK to save the new eval mapping.
24. You should end up with a total of eight mappings, seven Field Aliases and one Eval object.
25. Click Done. On the Data Model Mapping page, you will now see a row for the new mapping you have
created.
26. Click Validate & Package from the app navigation bar.
27. Remove Field Extraction and App Precertification from the list of validations and click Validate.
The validation tests execute—this can take a couple minutes.
28. While the validation is processing, click Download Package to download your new add-on to your local
workstation.
This creates a .spl file on your workstation, which is actually a compressed .tgz file.
29. On your workstation, locate the TA-bcg_1_0_0.spl file.
30. Change the file extension to .zip or .tgz and double-click it to extract the contents into a directory.
31. Examine the TA-bcg directory contents.
The primary normalization configurations are in the default directory, in the props.conf,
eventtypes.conf and tags.conf.
You do not need to install the add-on on your lab server—the add-on builder automatically added it to
your search head, so you can test the add-on.
You can also use Splunk Web to view the contents of your TA-bcg add-on.
32. After the validation process is complete, you should see no errors or warnings.
33. Navigate to Settings > All configurations.
34. From the App drop-down menu, select bcg (TA-bcg).
35. Make sure Created in the App is selected.
You should see nine items; seven fieldaliases, an eventtype, and an fvtag object.
36. Navigate to a new search window and run this search over All Time:
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 25
| datamodel Network_Traffic All_Traffic search | search sourcetype=bcg:accounting
| stats count as Total_Normalized_Events
The count for Total_Normalized_Events should no longer be zero, indicating the bcg:accounting
events are being normalized. These events will now be available through the Network_Traffic data model
for all ES correlation searches and views.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 26
Module 10 Lab Exercise – Tuning Correlation Searches
Description
In this lab exercise, you will examine existing correlation searches to identify search thresholds, both explicit
and dynamic.
Steps
Task 1: Identify thresholds in correlation searches.
5. Find the portion of the correlation search that represents the “threshold”. You may have to scroll down in
the Search pane.
What part of the search expression defines the threshold for creating a notable event? _______________
6. How many DNS failures does it take to cause a notable event to be generated? ___________________
8. Examine the search expression for the Brute Force Access Behavior Detected correlation search.
Task 2: Create two swim lane searches that track successful and failed logins for an identity.
11. Click Create New Content and select Swim Lane Search.
12. Create a new swim lane search with the following information:
o Search Name: enter Failed Authentications
o App: select Enterprise Security
o Search Title: enter Failed Authentications
o Search:
| tstats `summariesonly` values(Authentication.action) as
action,values(Authentication.app) as app,values(Authentication.src) as
src,values(Authentication.dest) as dest,values(Authentication.user) as
user,count from datamodel=Authentication.Authentication where
Authentication.action=failure AND ( $constraints$ ) by _time span=$span$
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 27
o Drilldown Search:
| from datamodel:"Authentication"."Authentication" | search
Authentication.action=failure AND ( $constraints$ )
o Color: select a color for the swimlane
o Entity Type: select Identity
o Constraint Fields (click <Enter> or <Tab> for each entry): Authentication.src_user,
Authentication.user, src_user, user
13. Click Save.
14. Create another new swim lane search with the following information:
o Search Name: enter Successful Authentications
o App: select Enterprise Security
o Search Title: enter Successful Authentications
o Search:
| tstats `summariesonly` values(Authentication.action) as
action,values(Authentication.app) as app,values(Authentication.src) as
src,values(Authentication.dest) as dest,values(Authentication.user) as
user,count from datamodel=Authentication.Authentication where
Authentication.action=success AND ( $constraints$ ) by _time span=$span$
o Drilldown Search:
| from datamodel:"Authentication"."Authentication" | search
Authentication.action=success AND ( $constraints$ )
o Color: select a color for the swimlane
o Entity Type: select Identity
o Constraint Fields (click <Enter> or <Tab> for each entry): Authentication.src_user,
Authentication.user, src_user, user
15. Click Save.
Task 3: Confirm the newly created swim lane search in the Identity Investigator.
16. Navigate to Security Intelligence > User Intelligence > Identity Investigator.
17. In the Identity Investigator click Edit.
18. Under Collection, select the Custom radio button.
19. Under Individual Lanes, select the boxes for Successful Authentications and Failed Authentications.
(You can uncheck the boxes for unwanted lanes.)
20. Ensure the Time Range Picker is set to Last 24 hours.
Hint: The Time Range Picker is below the swimlane titles.
21. Enter username test in the Search for identity box and click Search.
22. Review the swimlane search results by clicking some of the colored bars and viewing the information on
the right. (Remember, the darker bars contain more results.)
23. Note the apps that user test has successfully accessed, and those they have failed at accessing.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 28
Module 11 Lab Exercise – New Correlation Searches
Description
In this lab exercise, you will build a custom correlation search to look for successful SSH login activity—which
is prohibited in your environment—to create notable events. Therefore, your analysts are proactively alerted to
the prohibited activity on the Incident Review dashboard.
Steps
Task 1: SSH logins are prohibited in your environment. Create a custom correlation search that
detects successful SSH logins and generates a notable event to alert analysts.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 29
Note: Fields to group by is a multi-value field. Enter each value and press Enter, otherwise you will
receive an error when saving the search.
This configures the correlation search to create no more than one notable event per host every 5 minutes.
8. Click + Add New Response Action, then Notable. Use the following settings, leaving all others default:
o Title: Successful SSH connection on host $host$
o Description: There has been an SSH connection on host $host$
o Security Domain: Access
o Severity: High
o Default Owner and Default Status: (leave as system default)
o Drill-down Name: View SSH events on $host$
o Drill-down Search: index=main host=$host$ *sshd*
o Drill-down Earliest Offset: 5m
o Drill-down Latest Offset: $info_max_time$
The above options create a drill-down search that returns all SSH events for the host(s) detected by the
correlation search in the 5 minutes leading up to the time of the correlation search (since the previous
iteration of the correlation search).
What is the full name of the correlation search (Hint: search_name field)?__________________________
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 30
Module 12 Lab Exercise – Adjusting Asset Priority
Description
In this lab exercise, you will adjust the priority level of selected assets, which in turn increases the severity
level of any notable events associated with the assets.
Steps
Scenario: You are about to go live with a new customer-facing website that has been developed on a
set of servers. These servers have been located behind the firewall up to now. However,
router tables are now being updated and these servers will be accessible to the public.
You need to change the server’s priority in ES.
1. Navigate to the Incident Review dashboard and search for Endpoint domain notable events for servers
that start with PROD-MFS (Hint: PROD-MFS*) over the last 24 hours.
You should see a few pages of notable events, and the urgency should be Medium. This is because the
servers’ priority is set to Low, and the severity of the correlation search(es) is High, causing the overall
urgency to be set to Medium.
Since you are going live with the PROD-MFS-XXX servers, you want to increase the priority as a mission-
critical asset.
2. Navigate to Configure > Data Enrichment > Asset and Identity Management.
3. From the Asset Lookup Configuration tab, click the simple_asset_lookup link under the Source
column.
Note: the demo_assets and demo_identities lookups contain sample data that can be used for testing
purposes.
Important: an ES admin can edit the identity and asset lookups though Configure > Content > Content
Management. Filter on Type: Managed Lookup and select either Assets or Identities.
4. Scroll through the rows until you find a list of servers with nt_host names PROD-MFS-001 to
PROD-MFS-006.
5. Change all PROD-MFS servers from low to critical. The background should change to red.
Tip: Type critical for the first server, then copy/paste to update the other five.
6. Click Save.
Note: It will take a few minutes for your changes to apply. Any changes you make are not validated for
formatting before being saved.
7. Wait several minutes, then navigate back to the Incident Review dashboard.
8. Search the Endpoint Security Domain again for the PROD-MFS* servers.
The Urgency value should now be Critical for all PROD-MFS* notable events. This is a retroactive
change and all new notables for these servers will also be Critical.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 31
Module 13 Lab Exercise – Threat Intelligence Framework
Description
As an ES administrator, you need to add new threat intel sources to your system, and confirm they are working
properly.
Steps
Scenario: You notice traffic from IP addresses that could belong to hijacked systems. Fortunately,
there is a threat list for hijacked systems available at http://www.iblocklist.com.
You will download the hijacked servers list, which looks like this:
Hijacked:2.56.0.0-2.59.255.255
Hijacked:5.34.242.0-5.34.243.255
Hijacked:5.72.0.0-5.75.255.255
Hijacked:5.180.0.0-5.183.255.255
Hijacked:14.129.0.0-14.129.255.255
Hijacked:14.192.48.0-14.192.59.255
...
Note that each entry is two fields: the description, followed by the IP address or range, separated by a colon.
You will use this in your threat list definition.
© 2020 Splunk Inc. All rights reserved. Administering Splunk Enterprise Security 2/21/20 Page 32