SOAR 6: Advanced Implementation Lab Exercise Manual

Introduction
This is the lab exercise manual for the Advanced SOAR Implementation course. There is one lab exercise for
each module of the course. You’ll be working mostly on your own in each exercise. The first lab exercise
focuses on getting settled in your lab server environment. After that, you’ll work through a sequence of lab
exercises beginning with Splunk-SOAR integration, then custom playbook coding, and finally REST API usage.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 1
Lab Exercise 1 – Prepare for Splunk-SOAR Integration
Description
In this lab exercise, you’ll access your lab servers, check on documentation resources, and examine SOAR
apps on SplunkBase.

Steps
Task 1: Examine lab environment.

1. Get the following information from your instructor and record it here:

SOAR server address https://

User name soardev

Password

2. Using the above information, log on to your lab server.

3. Configure Company Settings:
§ Company Name: Buttercup Games
§
§ Base URL: https:// + your server’s URL.
4. While logged on to the SOAR server, enable DEBUG logging for the Decide and Action daemons. This
will be handy for debugging playbooks later.
5. Get the following information from your instructor and record it here:

Splunk server address https://

User name admin

Password

6. Examine the configuration of this Splunk search head. Note the installed applications. This is an
Enterprise Security system.
7. Navigate to the ES Security Posture panel. Note the numbers and types of notable events being
generated.
8. Navigate to the Incident Review dashboard. Examine some details about the notables being
generated.
Task 2: Access online resources.

9. Navigate to https://docs.splunk.com and select Splunk SOAR. Examine the available documentation.
Note the docs for the Phantom App for Splunk and Splunk SOAR remote search.
© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 2
 10. Navigate to https://splunkbase.splunk.com.
Note: If you do not already have an account here, use the My Account menu and select Sign Up to
join. The next couple labs will instruct you to access Splunkbase to download apps; if your new
account has not been completed, your instructor can download the apps for you.
11. Search for "SOAR". Note several apps and add-ons are available to install on Splunk to interact with
SOAR. You’ll be learning about these as you go through the class.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 3
Lab Exercise 2 – Remote Search

Description
In this exercise, you’ll configure your SOAR server to use your Splunk server for search.

Steps

Task 1: Prepare the Splunk server.

1. Log on to your Splunk server.

2. Install the Splunk App for SOAR and restart the server.
Note: you will need an account at Splunkbase to download the app; if you do not have one, your
instructor can provide the app.
3. Add the splunk_app_soar role to the list of inherited roles for the Admin role.
4. Add the SOAR search and delete user accounts with the correct phantom* roles. Make sure to
uncheck the option to require password change on first login.
5. On the Splunk App for SOAR’s Configurations tab, in the Advanced Options section, create the
SOAR remote search indexes.
6. Edit the HTTP Event Collector (HEC) global settings and enable all tokens. Make a note of the port
being used.
7. Add a new HEC token for use by SOAR. Make sure to add all required indexes. Make a note of the
token value.
8. Determine and record the management port for Splunk, which is used for REST connections.
Hint: Settings > Server Settings > General Settings.
9. Run a search of all phantom* indexes. No results are returned.

Task 2: Test search on SOAR.

Before you change how SOAR is storing and searching for event data, check how it’s working now.
10. On your SOAR server, use global search to search for a few terms, like "splunk", "virus", "mail", etc.
Most of the returned results will be from app descriptions. This shows that search is functional.
11. Search for "sample event". Note the results; they’re all from apps.
12. Create a new event manually, named "Sample Event". It does not need any artifacts.
13. Re-run your "sample event" search. The event you created should be listed. This shows search is
indexing system events as they happen, using the default embedded Splunk instance on the SOAR
server.

Task 3: Configure remote search on SOAR.

14. Configure the SOAR server to use Splunk search.

• Enable SSL for the REST port and HTTP Event Collector.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 4
 • Do not enable certificate verification; these ports are using self-signed certificates for training and
will not validate.
15. Do not re-index SOAR search yet.
16. Test the connection.
17. Run some searches again in global search. There should be no results. Why is this?
18. In Administration/Administration Settings/Search Settings, Re-index all sections.
19. Try your searches on Splunk again. They should succeed now. Searches on SOAR are running on the
remote Splunk system.
20. On the Splunk server, search the audit log for events for action=search for the phantom search user.
You should see these events each time remote search is run from the SOAR server.
21. In Splunk, search the phantom indexes for "sample event". This was sent to Splunk by re-indexing.
What index is it stored in?
22. On SOAR, create a new event.
23. Search the phantom_container index and verify the container creation was recorded.
24. Examine some of the reports and dashboards available.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 5
Lab Exercise 3 – Exporting Splunk Events to SOAR

Description
In this exercise, you’ll configure the Splunk and SOAR servers to send notable events to SOAR.

Steps

Task 1: Prepare the SOAR server.

1. Create a new automation user on SOAR. Make sure to make its type "automation", put it in the
Automation and Observer roles, and allow connection from any IP address.
NOTE: best practice of course would be to only allow connections from IP addresses in your domain,
but for lab purposes, “any” is OK.
2. Record the authentication info from the new automation account.
3. You’ll be sending notable events to SOAR, so make a notable label to identify them separately from
the default events label.

Task 2: Prepare the Splunk server.

4. On your Splunk server, install the Splunk App for SOAR Export.
Note: this will require an account at Splunkbase; if you do not have one, your instructor can download
the app for you.
5. Add the phantom role to the inherited roles list for the admin role.
6. Create a new server instance in the SOAR Export app’s Configurations page for your SOAR server.
7. Sync the playbooks and severities from the SOAR instance.

Task 3: Test sending a notable event to SOAR.

8. In Splunk Enterprise Security, use the Actions menu on a notable event in Incident Review to send a
notable event to SOAR. Make sure to use the notable label.
9. In the Adaptive Responses section, examine the logged data from the action in the notable event’s
details.
10. On SOAR, examine the new event. The artifact(s) contain all the fields from the notable event in
Splunk. Duplicate artifacts are created when fields have multiple values, such as eventtype or tags.
Also note that very few if any of the CEF values have a data type; therefore, there are no context drop-
down menus for any of them. This is because sendtophantom does little if any CIM to CEF re-
mapping automatically, so only in the case of a CIM name being the same as a CEF name will a data
type be assigned—for instance, destinationAddress.

Task 4: Configure event forwarding on Splunk.

11. On Splunk, use the search window to test a search for notable events for malware detection. The
`notable` macro is useful here, and the source field will contain text useful to determine if the
notable is associated with malware.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 6
 12. The search should limit the output fields to only what is needed for the input requirements of your
design. This should include the notable event ID, the targeted server host name or IP address, and
any other fields that could contribute useful information to SOAR for analysis. It’s also a good idea for
lab/testing purposes to use the |head command to limit the amount of output from the search.
13. The search should not produce any multi-value fields, unless those values are joined into a single
delimited string.
14. As you work with the notable event search, you’ll probably see that there is a mix of both IP addresses
and host names. These should be passed to SOAR with a specific CEF type so SOAR knows what
type of data it’s dealing with. Structure your search to generate fields identifying the server by either IP
address or hostname. The goal is for each container in SOAR to have destinationAddress CEF
values that are either Null or valid IP addresses, and destinationHostName CEF values that are Null
or valid host names.
15. Save the search, make sure the permissions are set correctly for global access from other apps.
16. In SOAR, create a custom CEF field type for event_id, with data type “splunk notable event id”.
17. Create a new event forwarding configuration for the SOAR Export app to send notable events to
SOAR, labeled notable. The name of the created container should be the descriptive name of the
notable event, which is in the source field. Configure the forwarding to happen in real time.
18. Set the severity to Low and sensitivity to Green for forwarded events. Map all search fields to a
matching CEF field, or create a non-standard field name if no existing CEF field name applies. Make
sure the event_id field is included.
19. Use Save and Preview to verify notable events are being found.
20. After the forwarding has been active for about 10 minutes, examine the SOAR server’s Analyst Queue.
There should be some new events from Splunk. Make sure the artifact CEF data looks like you expect.
21. After a while you will have several notable events in SOAR. You can disable the forwarding search on
the Splunk server in the SOAR export app’s Event Forwarding tab after you have about 20 or so
events in SOAR to work with.

Task 5: Migrate old events to SOAR.

The Splunk App for SOAR Export is set up now and is sending new notable events to SOAR, but
existing notable events may need to be sent.
22. In Splunk, create a new search that is based on your previous malware notable search, but also calls
the sendtophantom alert action.
23. Use it to send all matching alerts to SOAR if the notable was created prior to the time you started
running the active event forwarding, for the previous 2 hours. Set the priority of these events to low.
24. Compare the containers you’ve forwarded using sendtophantom to the containers created by event
forwarding. Because you created the event_id CEF field type, your bulk events have the correct data
type for the event_id field. This will become important once you install the Splunk app on SOAR,
which has actions mapped to this data type.
25. Make sure you have some containers with destinationAddress IP values, and some with
destinationHostName values. You’ll need some of both for testing.

Once complete with this lab exercise, you may want to disable the event forwarding to avoid loading
your SOAR server with additional containers.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 7
Lab Exercise 4 – Searching Splunk from SOAR

Description
In this exercise, you’ll begin development of the playbook for the solution. Initially you want the playbook to use
the destinationAddress and/or destinationHostName values to use to run searches for events in the last
24 hours that have matching src or dest field values. The search should return a list of matching server
names with a count of the number of events for that server. This will help prioritize analysis; servers with high
contact values will be more at risk of infection.

Steps
Task 1: Design the solution.

This exercise is the first stage of a solution for your customer. The desired solution is automated malware
detection, analysis, and isolation. As an additional requirement, the customer wants to immediately identify any
other systems in its environment that may be at risk of infection. When Splunk detects a malware signature in
Enterprise Security, an event will be created automatically in SOAR, which triggers a playbook. This playbook
should use the infected host in a search to find other hosts that have communicated with the infected host in
the previous 24 hours. Store the list in a new custom list, including the number of times the infected host
communicated with the peer, and the priority of the peer. The solution should then create a new event in
SOAR for any peer with a "high" or "critical" priority.
1. To prepare for this, use I2A2 to design the solution. To summarize the requirements:
• Automatically start when a container with label "notable" is created
• Check to see if this notable is related to malware infection
o If so, search Splunk for peers of this server: defined as having the infected server as a src
or dest, within the previous 24 hours
• Store the peer list as a multi-column custom list, including the peer name, count, and priority. (add
this to the design; you will not implement this until a later lab exercise.)
• Update the notable event in Enterprise Security; set the status to Open.
• Process the new list as soon as it is created, and for any critical or high priority peers, create a
new event. (add this to the design; you will not implement this until a later lab exercise.)

2. Identify the required Inputs, Interactions, Actions, and Artifacts.

3. Draw out a flow plan of the proposed playbook.

In the next few steps, you’ll begin developing this solution.

Task 2: Configure and test an asset for the Splunk App on SOAR.

4. Create a new Splunk user account to use for running searches from SOAR. This user account should
be in the Admin role.
NOTE: for debugging, you can search in the _audit index for user=xxx to find actions by this user.
5. Add an asset for the Splunk app on SOAR to access your ES Splunk instance Using the user
credentials you created in the previous step to connect.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 8
 • Do NOT enable certificate validation; the ES lab servers do not have validated SSL certificates on
port 8089.

6. Test the asset by manually executing a run query action from any container. Use a test search like
"* | head", which will return the last 10 events ingested into Splunk.
7. Examine the results in the Investigation page. Drill down into the JSON result data structure. Notice
that no field extraction happened; this is because by default REST searches are run in fast mode with
no extraction.
8. Run another query: "* | fields * | head". Notice in the results the full field extraction has been
performed. Note as well, several metadata fields that begin with an underscore. You can omit these
from results by adding "|fields - _*" to a search.

Task 3: Design a Splunk search to find peer servers.

9. On Splunk, develop a search that returns a three-column table of host names, priorities and counts.
The hosts are servers that have been identified in a src or dest field from any events associated with
a suspected target system. For testing, you can use any common host name from the sample data in
Splunk—you can run a search first to find some candidate host names to use.
10. The search should be ordered by the most number of contacts first, and should include results for the
preceding 24 hours. The peer list should not contain the infected host itself.
11. The search should be designed to run as efficiently as possible.
12. Since the search is designed to determine the peers of any host, design it to accept a variable for the
original host name.
13. Save the search on Splunk. Make sure it is globally accessible.

Task 4: Run a search from a playbook.

For the following task you’ll be creating a playbook. Make sure you test it in the playbook debugger
and that all steps work. Use any containers for debugging but make sure you test it on containers that
have a destinationHostName and containers with a destinationAddress. For this version, do NOT
use any custom code other than for debugging. You will likely find that to avoid a 2-lane approach (one
for destinationAddress, one for destinationHostname), the Utility block list_merge custom function is
useful.

14. On SOAR, create a new playbook that runs on notable labels. Make sure logging is on.
15. In the playbook, call the saved search on Splunk. Use the value in either the destinationAddress or
destinationHostname as the host value to search for peers.
16. Add a comment to the container to show the total number of peer servers returned by the search.

Task 5: Update the notable event on ES.

17. You'll be using the Splunk app's update event action; this action requires an input event_id of type
"Splunk Notable Event ID". If you have not already done so, add this as a custom CEF type. Look for
CEF in the Administration menu under Administration Settings. Create a new CEF field named
event_id with the appropriate data type.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 9
 18. In your playbook, add an action block to update the related notable event in ES. Pass the event_id and
set the status to "in progress" (hint: look in the "app defined values" lookup when setting this
parameter) and add a comment to the notable event on Splunk that the notable event is being
processed by SOAR. The comment should include a URL to the related container in SOAR.
19. Test this new feature.

Lab Exercise 5 – Custom Coding

Description
In this exercise, you’ll store the results of the peer search in a custom list. While you could build this solution in
a single playbook, you’ll partition the solution into two playbooks: the second playbook will process suspected
peer lists. This modular approach will make it easier in the future to handle similar scenarios where peer
servers must be investigated. For any peer in the list with a high or critical priority, the new playbook will create
a new container with appropriate information, including the host name and priority, to begin analysis. You’ll
also be coding custom functions. While the custom code is only used in one playbook, it’s good practice to
learn to manage custom functions that can be leveraged in multiple playbooks in the future.

Steps

Task 1: Write a custom function to store the peer list.

1. Create a new custom function to parse the search results and store them into a new custom list.
• The custom function will require argument inputs for the current container ID and the run query
action_result.data element.
• The created list should contain 3 columns
• Each host should use one row in the list
• Use a custom function and not a code block; this feature is likely to be used in multiple playbooks
2. Your code should take into account the possibility the list already exists; in this case, it should be re-
written and not appended.
3. Your list name should start with "temp_" to identify it as a transient list, and should also include the
container ID.
4. Call your custom function from the playbook, passing the container ID and the results from the run
query action.
5. Test your playbook in debug mode. Confirm the new list is created correctly and is replaced (not
appended) during subsequent debug executions.
Task 2: Build the playbook.

6. Create a new input playbook and configure its inputs to receive the name of the peer list. Add the
necessary contents to:
• Retrieve the peer list created by the parent playbook
• Parse the list for high or critical peers
• Create a new event container for each high or critical peer, with an artifact identifying the peer
name
© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 10
 7. During testing, you might need to manually edit the custom list to change some peer priorities to high
or critical.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 11
Lab Exercise 6 – Working with REST

Description
In this exercise, you’ll use the SOAR REST API to execute some Django queries to learn more about how
objects are stored in SOAR. You’ll also create a new utility playbook that uses REST to delete temporary lists.

Steps
Task 1: Use Django queries.

Experiment with Django queries in your browser. You may want to install a JSON viewer plug-in to make this
easier. Work out queries for each of the following tasks and save those queries in a text file. Send them to your
instructor for review.
1. All containers with a name containing the string "malware".
2. For one of the malware containers, show the container details, with a list all of its artifacts.
3. Find all playbook runs for the most recent version of your notable event playbook.
4. For the most recent run, find all the actions.
5. Find all the run query action_runs.
6. From one of the run query action_runs, find the app_runs.
7. For that app_run, find the results_data.
8. For the playbook_run, find the log data.
9. Find all custom lists with names starting with "temp_"

Task 2: Use curl to create containers.

10. Using curl, create a simple container.

11. Also using curl, add an artifact to the container.

Task 3: Remove temporary lists.

12. Create a new label on your SOAR instance: "system".

13. Configure an asset for the HTTP app on your SOAR instance.
14. Configure an asset for the Timer app on your SOAR instance. It should create containers with label
"system". Don't configure an interval yet.
15. Create a new playbook for "system" labels.
16. In the new playbook, use the HTTP app's get_data action to get a list of all custom lists that have
names starting with "temp_".
17. Parse this list and use it to delete each of the temporary lists.
18. Configure the timer asset to run every 10 minutes and test your configuration.

In production, you could also restrict access to the "system" label to admins only; or this playbook
could be manually executed only when needed.

© 2023 Splunk Inc. All rights reserved. Advanced SOAR 6.0.1 Implementation 6/14/23 Page 12