Cisco Advanced Threat Solutions Team

Threat Hunting Workshop

Hand-On with Cisco Security Products
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

v. 2.1.1550862109

Table of Contents
Logging In to the Lab Environment ......................................................................................... 3
AMP for Endpoints .........................................................................................................................3
Cisco Threat Response ...................................................................................................................4
Threat Grid ....................................................................................................................................5
Umbrella Investigate ......................................................................................................................6
Lab 0 – Olympic Destroyer (walk through with instructor) ...................................................... 7
Scenario ........................................................................................................................................7
Steps: ............................................................................................................................................8
Lab 1 – VPNFilter ................................................................................................................. 15
Scenario ...................................................................................................................................... 15
Steps ........................................................................................................................................... 16
Lab 2 – Fish the Phish ........................................................................................................... 31
Scenario ...................................................................................................................................... 31
Steps ........................................................................................................................................... 32
Lab 3 – Bifrost ...................................................................................................................... 42
Scenario ...................................................................................................................................... 42
Steps ........................................................................................................................................... 42
Lab 4 – Poweliks .................................................................................................................. 60
Scenario ...................................................................................................................................... 60
Steps ........................................................................................................................................... 61
Lab 5 – Threat Hunting ........................................................................................................ 89
Scenario ...................................................................................................................................... 89
Tips for Threat Hunting ........................................................................................................ 91

Administrative Notes 2
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Logging In to the Lab Environment

This lab environment uses four Cisco cloud-based security applications: AMP for Endpoints,
Threat Response, Threat Grid, and Umbrella Investigate. The account credentials for these
applications are standardized and will be provided by the lab proctor. Of these four, one (AMP)
requires an additional step when logging in – that is, providing a 2-factor authentication (2FA)
code. The lab proctor will display this code on the screen during the self-guided lab section of
the agenda.

Here are the steps to log into each account in preparation for the self-guided labs:

AMP for Endpoints

1. Go to the AMP console at https://console.amp.cisco.com/
2. Log in with the credentials provided by your lab proctor.

3. Enter the six-digit 2FA code provided by the lab proctor.

Note: Check “Remember this computer for 30 days” to ensure that you will not
be prompted for this 2FA code again for the remainder of this lab session.

Administrative Notes 3
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Cisco Threat Response

You can access Cisco Threat Response directly by going to the console URL,
https://visibility.amp.cisco.com/#/investigate, or by pivoting from AMP for Endpoints using the
contextual menu that is accessible by clicking on any hash, IP address, or domain name within
the AMP console:

Administrative Notes 4
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

You will be redirected to the Threat Response login page where you can click the first blue
button to log in using your Cisco Security credentials:

NOTE: To keep the lab environment consistent, always log in using Cisco
Security credentials (not Threat Grid credentials).

Threat Grid

The Threat Grid console URL is https://panacea.threatgrid.com/login. The Threat Grid account
is already linked with AMP for Endpoints, so instead of typing in the username and password,
you can simply click on “Log In with Cisco Security” at the bottom of the login screen:

Administrative Notes 5
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Umbrella Investigate

The Umbrella Investigate URL is https://investigate.umbrella.com/. Use the account

credentials provided by your lab proctor. The username and password will be the same as the
AMP for Endpoints Console username and password.

Administrative Notes 6
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Lab 0 – Olympic Destroyer (walk through with instructor)

Olympic Destroyer
Threat Hunting Workshop: Hand-On with Cisco Security Products

Scenario: The CIO read a front-page news article on something called “Olympic Destroyer”,
which was recently used to disrupt the Winter Olympic Games in Pyeongchang. The news
article suggests that other threat actors may be able to reuse this malware in a commodity
attack against other targets. The CIO is asking if our security products are already blocking this
threat or if we need to update to be protected.

Lab 0
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Steps:

Our first step is to research “Olympic Destroyer”. The first hit on Google is a Cisco Talos
Intelligence blog entry, authored by Warren Mercer and Paul Rascagneres on February 12, 2018
(http://blog.talosintelligence.com/2018/02/olympic-destroyer.html):

This blog post details how the malware was identified, how it propagates, what it does once
executed (stealing credentials using a browser credential stealer and a system stealer via LSASS,
then destroying data using vssadmin, wbadmin, bcdedit, and wevtutil). The final section of the
blog post details several Indicators of Compromise (IOCs) for the Olympic Destroyer binary, the
browser and system stealer packages, the destroyer tool, Psexec, and additional samples, each
in SHA256 format.

Lab 0
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

To check whether we are protected against these threats, let’s start an investigation in Cisco
Threat Response (https://visibility.amp.cisco.com/) by copying and pasting the list of IOCs from
the Talos blog post:

There are 10 unique SHAs in the Talos IOC list for Olympic Destroyer, and we can see ten
observables in Threat Response – most of which are malicious:

The first SHA256, corresponding to Psexec, is colored green, showing that Threat Response has
queried for its reputation and it is known as a clean file as we expected. The rest of the
SHA256s are already marked as malicious with a variety of verdicts from sources such as AMP
File Reputation, AMP Global Intel, and Virus Total.

Looking further, we can see that one of our internal endpoints has actually seen one of the IOCs
associated with the Olympic Destroyer malware campaign investigation:

Lab 0
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Looking more closely at this hash and the nodes surrounding it in the Relations Graph, we can
see that the affected computer is Demo_AMP_Intel, the filename for the malicious hash is
OLD.exe, and it was launched from the user’s desktop (the file path is
\\?\C:\Users\johndoe\Desktop\OLD.exe).

We can see more detail about the AMP event that enriched this hash by looking at the
Observables pane on the right side of the screen. The Observables pane has two display
modes: one shows each observable as a card, and the other shows detailed information for
each of the observables with judgments, verdicts, sightings, and indicators, if applicable. We
can change between the card view and detailed view with a toggle switch in the top right
corner of the Observables pane:

Lab 0
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

(Card view)

(Detail view)

Looking at the card view, we can immediately see that one of the the ten observables from the
list of IOCs we copied out of the Talos blog has two sightings on one internal target:

Lab 0
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

If we click on this observable (SHA256 beginning with edb1ff2521f…), we will see all the
information that Threat Response has collected on this hash from sources including AMP File
Reputation, AMP Global Intel, and Virus Total, among others. We can see a timeline showing
how many times it was observed within our environment and compare this with a global
sightings timeline to see if it was a targeted attack or part of a larger malware campaign. To
find out specifically where it was seen in our local environment, click on the Sightings tab within
this Observables pane:

Lab 0
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

This Sightings tab shows the module that contributed the information (in this case AMP for
Endpoints), the confidence of the disposition, the description, the source – with a link to the
event itself in AMP for Endpoints, the relations (or what exactly happened during the event),
the target, and how long ago it was observed.

Let’s click on the AMP Event link to jump to Device Trajectory:

Which brings us to:

As we can see from this event, the file was quarantined on the endpoint, proving that AMP is
actively protecting our systems from the Olympic Destroyer threat.

Lab 0
THREAT HUNTING WORKSHEET
Name:
Date:

Kill Chain Analysis

Reconnaissance (Targeted System)
Patient Zero: System and User Information
Computer Name: Logged-in Username:
Operating System: MAC Address:
Vulnerable Applications: Host IP Address(es):

Delivery
Attacker Infrastructure and Dropper File
Malicious Domains: IP Addresses:
Dropper File Name: Dropper File Type:

Exploitation
Infection Vector
☐ Email: Attachment ☐ Web: Phishing Link ☐ Active Exploitation
☐ Email: Phishing Link ☐ Web: Exploit Kit ☐ Other:

Installation
Malware Binary
File Name: File Type:
Malware Family Name:

Command and Control

Command and Control (C2) Communications
Domains Contacted: IPs Contacted:
URLs Contacted:

Actions on Objective
Threat Behavior
☐ Process injection detected? (if so, which process)
☐ Escalation of privileges detected? (if so, which method)
☐ PowerShell / scripting commands detected? (if so, list)
Known Malware Capabilities:

Executive Summary
Date/Time of Patient Zero initial infection:
Total number of systems affected:
Timeline:

Security recommendations:

Sponsored by the Cisco Advanced Threat Solutions team and Cisco Incident Response Services. For assistance with an ongoing breach, contact
[email protected] or call 1-844-831-7715.
Lab 1 – VPNFilter
(Please note – Screenshots included in the lab may differ from actual results)

VPNFilter
Threat Hunting Workshop: Hands-On with Cisco Security Products

Scenario: The CIO saw a Twitter post mentioning a threat called “VPNFilter” that has infected
over half a million routers worldwide. While none of our corporate routers should be affected,
the CIO wants to know if there are any infected “Shadow IT” devices connected to our network
– and if so, if our security products are blocking this threat or not.

Lab 1 15
Steps:

The CIO mentioned this Twitter post:

The shortened link in the post takes us to the full Cisco Talos blog article announcing the
discovery and impact of this “VPNFilter” threat. Let’s navigate to cs.co/vpnfilter to learn more:

Lab 1 16
The TalosIntelligence.com blog is divided into several sections:

• Intro – summarizes the threat itself, mentions how it was discovered, and introduces
the content that will follow in the rest of the article
• Brief technical breakdown – discusses the three known stages of the malware platform
• Tradecraft discussion – includes background on the capabilities of the malware from an
operational perspective
• Observed activities of concern – describes the timeline of activity for this threat
• Defending against this threat – includes notes on how to guard against VPNFilter
• Recommendations – lists specific actions that should be taken by device operators
• Multi-Stage Technical Details – includes an exhaustive analysis of the malware based on
kill chain and stage, beginning with exploitation and ending with non-persistent plugin
modules loaded in stage 3
• Conclusion – summarizes the findings in the blog article, particularly the impact of this
threat to the global security community
• IOCs – contains detailed Indications of Compromise for each stage of the malware as of
the publish date for the blog article

Take a few minutes to read through the blog post to understand the threat and how it
operates. When you are finished, let’s continue our threat hunt by taking the observables
recorded for the first stage of the malware and querying our Incident Response platform (Cisco
Threat Response) for any recorded judgments, verdicts, or sightings.

There are 13 observables associated with the first stage of the VPNFilter malware, the
persistent loader:

Copy these 13 observables from the Talos blog and paste them into a new Threat Response
investigation (https://visibility.amp.cisco.com/):

Lab 1 17
Click the blue “Investigate” button to initiate the investigation. Threat Response will query
each configured source of threat intelligence to determine a disposition for each observable
and will also query its configured infrastructure modules to find any sightings of these
observables in our environment. We can quickly see that we have both dispositions and
sightings in our investigation:

Threat Response shows dispositions for both domains in our query, toknowall[.]com (which is
considered malicious) and photobucket[.]com (which is considered clean or benign). Let’s
consult the Observables pane to learn more about these dispositions – in particular, we want to
know why they are considered clean or malicious.

The Observables pane has two display modes: Tile View, which shows each observable as a
square tile, and List View, which shows detailed information for each of the observables
including judgments, verdicts, sightings, and indicators, if applicable. We can change between
Tile View and List View with a dropdown menu in the top right corner of the Observables pane:

(Tile View)

Lab 1 18
(List View)

Let’s open the List View for the malicious domain, toknowall[.]com. Click the red tile (if you are
in Tile View mode) or click the red tab on the left of the pane (if you are in List View mode) to
open the details for this observable:

Threat Response has three tabs for this observable’s entry: Judgments, Verdict, and Sightings.
Judgments are dispositions, and can be either clean, malicious, suspicious, common, or
unknown. An observable can have several judgments with conflicting dispositions; for instance,
VirusTotal (an antivirus engine aggregator) often returns 50-60 individual judgments for a file
hash from its queried AV products, and not all products return a malicious disposition. In this
case, we have two judgments for toknowall[.]com – one from Umbrella’s Investigate API and
the other from Talos Intelligence – and both agree that the domain is malicious. Threat
Response will take the highest-priority active judgment as its overall verdict for the observable,
and we can find out what it is by clicking on the second tab, named “Verdict”, within this
Observables pane:

The third tab, “Sightings”, contains instances where a Cisco Security product observed this
domain within our network. This information is brought into Threat Response via an
integration API and allows us to see exactly when and where the observable was sighted, as
well as the action that was taken for that event.

Lab 1 19
In this case, both of our sightings come from the Umbrella DNS security service, provided by the
Umbrella Reporting API integration. One sighting tells us that one of our organization’s Top
Identities had made a DNS request for the malicious domain, and the other sighting gives us the
actual requester IP address. We can click on the pivot link in the “Source” tab to find out more
details on this sighting from the Umbrella console. Click the link for the IP address sighting –
the description will read “DNS request for 'toknowall.com' made by 'Chicago Marketing Office'
(Sites)”.

This brings us to the Umbrella Security Activity Report, filtered to show only requests made to
this domain.

In this view we can quickly see that the request was blocked and which network the device is
on. From here we can use the menu from the ellipsis to dig in to this security event further.

Lab 1 20
One of the options in this context menu allows us to pivot into the Umbrella Investigate console
to learn more about the domain that was requested. Click the last option in the menu, “View
Domain in Investigate”, to bring up this detailed view into the domain toknowall[.]com:

The Investigate console offers a wealth of information about the queried domain. Three red
banners at the top of the report tell us that the domain is in Umbrella’s block list, and we can
see the overall risk score (which will vary over time); that it is associated with an Advanced
Persistent Threat or APT; and that it has a suspicious SecureRank 2 score (more on this attribute
later). We can see a graph of DNS queries over time that tell us how active this domain is in the
Internet – in this case the tens of thousands of queries per hour means that there are still tens
or possibly hundreds of thousands of infected devices with Stage 1 malware.

Immediately below the timeline graph is a summary of WHOIS data showing the domain
registrar and any registrant information that is available – including a link to display expanded
WHOIS information including the registrant contact name, email address, and mailing address.

Lab 1 21
Umbrella researchers use this information to track and block domains associated with malicious
registrants or registrant accounts that have been compromised:

Scrolling further displays a map highlighting the host IP locations as well as requester
distribution. We can see that most of the requests for toknowall[.]com originated from the
Ukraine, with China, the United States, Russia, and Italy rounding out the top requesters. The
geodistribution here is not a surprise given the global scale of VPNFilter, but tracking geo-
distributions can help identify targeted countries or be a tip off that a site is intended to be
malicious if it’s hosted in a country far from the majority of requesters.

The Umbrella Investigate report also shows us the current categorization for this domain –
“Malware” – with a timeline indicating when it received this categorization, and any other
categories it was given previously, helping admins confirm why and how long a particular site
has been blocked.

Lab 1 22
The next section of the Investigate report contains features that are used in various statistical
and machine learning models within Umbrella’s backend. The domain’s Time to Live (TTL)
values, hosting country history, Autonomous System Number history, number of prefixes,
geographic distance between hosting IP address locations, and other statistics help Umbrella’s
security researchers and automated systems identify fast flux domains and other risky Internet
infrastructure. The results are summarized in the “Security Features” section with scores for
algorithms and categories such as SecureRank 2, PageRank, ASN, Prefix, RIP, Popularity, and the
Kolomogorov-Smirnov test for requester geographic distributions.

Lab 1 23
Our investigated domain has few interesting features. The time-to-live (or TTL) values are
nominal, which is common for fast fluxing domains, but upon further inspection the domain has
not been hosted in different countries or with different Autonomous System Numbers (ASNs)
over its lifespan. Fast flux domains change ASNs or IP addresses rapidly to avoid being blocked
by reputation filters. Based on Investigate’s historical data we can see that this domain has
only been originated through AS 13335:

Clicking this hyperlinked AS Number will take us to another Umbrella report detailing who owns
this portion of the Internet – in this case, Cloudflare:

Further down we can see a list of prefixes on this ASN and the malicious domains they host.
This information is used to calculate the ASN and Prefix scores which we will review below – the
more malicious domains hosted the more negative the score will be:

Lab 1 24
Now, return to the Investigate report for toknowall[.]com by using the back button in your
browser. The Security Features summary table lists some useful metrics that can help us
understand where this domain falls in the spectrum between highly malicious and highly
reputable. We’ve mentioned how the ASN and Prefix scores were calculated, and many
different machine learning or artificial intelligence systems are involved in calculating these
other scores. While some are very good at identifying likely malicious sites, others are more
useful for identifying likely benign domains which helps avoid false positives. The combination
of all these scores helps us understand why toknowall[.]com is considered a dangerous domain.
The key indicator in this section is the SecureRank 2 score which, at negative 100 on a scale of -
100 to +100, is a very strong verdict against this domain.

Finally, at the end of the Umbrella Investigate report, we encounter a list of co-occurrences and
related domains. These features help us identify any other domains that are frequently looked
up around the same time as our investigated domain. A co-occurrence is a way of representing
relationships and interconnections between one or more domains being looked up by the same
client. Related domains are other domains that have been frequently requested around the
same time – up to 60 seconds before or after – the given domain name, but that are not
frequently associated with other domain names.

Lab 1 25
As we recall from the Talos blog on VPNFilter, the first stage of this malware reaches out to a
series of Photobucket albums to attempt to locate the IP address of a command and control
server. If unsuccessful, it will use its backup domain, toknowall[.]com. Umbrella Investigate
corroborates this analysis by featuring Photobucket domains in the related domains lists for
toknowall[.]com. This information helps Umbrella researchers identify domains that may be a
part of the killchain for this attack and also need to be blocked to prevent infection.

Having completed our analysis of the malicious domain, we now want to return to the Umbrella
console to find out more about the identity that attempted to reach out to these domains
associated with the VPNFilter malware. Let’s return to the Activity Search page to continue our
investigation.

NOTE: The Umbrella console may have timed out and returned to an overview page depending
on how long we spent reviewing the Investigate report. To return to our previous location, we
can go back to the Threat Response tab in our browser and click the “Umbrella Reporting API”
pivot link from the Sightings for the domain toknowall.com.

Our Activity Search currently has two constraints – it is filtered to only show requests for the
domain toknowall.com, and it is only showing requests from the last 24 hours. We can expand
this time window to see all requests made over the past 7 days, 30 days, or even a custom
range. Let’s change the time window to show the last 7 days of requests for this domain to
determine if there is a recurring pattern.

Lab 1 26
The pattern is clear: every 24 hours, a device with the IP address 192.168.4.213 within the
Chicago Marketing Office location makes a single request to resolve toknowall[.]com:

If we click on the link for the Chicago Marketing Office identity within one of these events, we
will see the Umbrella report for this site identity. The site in this case is a location that has an
Umbrella Virtual Appliance configured as the local DNS server to intercept all DNS requests
made from devices at this location. This configuration allows us to see – and block – suspicious
activity from devices that may not be able to run a traditional endpoint protection system such
as Advanced Malware Protection. In this case, we are likely dealing with a small home/office
router that would not be able to run any type of next-generation antivirus product, but
Umbrella still gives us the ability to observe the telltale signs of a VPNFilter infection. When
looking at the identity report page, we can see a timeline of all requests that were made at this
location over the past week, as well as the top destinations and top security categories from
this site (the IP traffic varies so your graph will not match exactly):

Lab 1 27
There is a high volume of allowed traffic from normal users at this location and a very small,
perhaps almost imperceptible, trend of blocks. Hovering over any point in the activity graph
shows us details for that day:

The other charts and tables in this report show us the top security destinations and top security
categories, as well as recent activity at this site:

Lab 1 28
This confirms that the only concerning events at this location are related to the infected device.
Our investigation produced the IP address of what is almost certain to be a compromised
home/office router, 192.168.4.213, which we can provide to the site IT administrator with
instructions to identify the device and remove it from the network. Alternatively, if this
location is equipped with a network access control solution such as Cisco Identity Services
Engine (ISE), we could execute a quarantine action to disable the device’s access to an access
switchport via a Radius Change of Authorization (CoA).

With our investigation now complete, we can quickly answer the CIO’s question: yes, we have
located a “Shadow IT” device that is infected with VPNFilter, but our security architecture was
able to contain the threat by blocking the first stage of the malware from resolving a command
and control server.

Lab 1 29
THREAT HUNTING WORKSHEET
Name:
Date:

Kill Chain Analysis

Reconnaissance (Targeted System)
Patient Zero: System and User Information
Computer Name: Logged-in Username:
Operating System: MAC Address:
Vulnerable Applications: Host IP Address(es):

Delivery
Attacker Infrastructure and Dropper File
Malicious Domains: IP Addresses:
Dropper File Name: Dropper File Type:

Exploitation
Infection Vector
☐ Email: Attachment ☐ Web: Phishing Link ☐ Active Exploitation
☐ Email: Phishing Link ☐ Web: Exploit Kit ☐ Other:

Installation
Malware Binary
File Name: File Type:
Malware Family Name:

Command and Control

Command and Control (C2) Communications
Domains Contacted: IPs Contacted:
URLs Contacted:

Actions on Objective
Threat Behavior
☐ Process injection detected? (if so, which process)
☐ Escalation of privileges detected? (if so, which method)
☐ PowerShell / scripting commands detected? (if so, list)
Known Malware Capabilities:

Executive Summary
Date/Time of Patient Zero initial infection:
Total number of systems affected:
Timeline:

Security recommendations:

Sponsored by the Cisco Advanced Threat Solutions team and Cisco Incident Response Services. For assistance with an ongoing breach, contact
[email protected] or call 1-844-831-7715.
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Lab 2 – Fish the Phish

(Please note – Screenshots included in the lab may differ from actual results)

Fish the Phish

Threat Hunting Workshop: Hands-On with Cisco Security Products

Scenario: One of our IT analysts noticed a phishing domain that was caught by Umbrella. We
have decided to investigate further in our environment to see if we can determine the source of
the offending URL. Unfortunately, we have not deployed AMP for Endpoints to this user’s
computer, so we don’t have visibility on his machine. We were able to identify the specific link
by looking at the user’s browser history, but unfortunately, the user has no recollection of
where the link came from. Now we will investigate to determine the source and to see if we
can identify any further steps to prevent this in the future!

Lab 2 31
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Steps:

We noticed the domain www.examplemalwaredomain.com was blocked in our Umbrella

dashboard, and then we were able to retrieve the following link from the user’s browser
history:

https://www.examplemalwaredomain.com/phishme.html

Now we are going to jump into Cisco Threat Response (which will be referred to as CTR going
forward) immediately to investigate this URL further. To do this, we will point our web browser
to https://visibility.amp.cisco.com, and once logged in, we paste the URL into the Investigation:

Now that we have pasted the complete link here, click on the “Investigate” button in order to
kick off the investigation. Please note that this process can take up to sixty seconds, while CTR
queries all available modules on this URL.

Once the query completes, note that a large amount of information is now available in our
investigation. Looking across the top column, we see that we have targets in our environment.
We will go ahead and click “Targets” to see where we have seen any observables (note that the
lab view may vary from this):

What is important here? Notice that we have one or more network targets identified by
Umbrella, which confirms what we already knew from the Umbrella Dashboard before the
investigation started.

Lab 2 32
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

More importantly in this situation, we see that we have an email target. Depending on which
lab environment you are using, it will either be [email protected] or
[email protected]. We will refer to the target email as bobc@acmematerial(s).com
from here on out, for brevity.

We can also see that we have two observables that CTR identified from the URL that we
provided:

Note that we have identified a domain and a URL as observables.

Now take a look further at the Relations Graph, which shows us the information that we have
collected from the available modules, and the relationships between those items. It may be
necessary to drag the view around, as well as zoom in and out in order to see all of the nodes.

Here we note the targets reported by Umbrella, which we know about (we may see more or
less targets depending on the lab environment):

If we re-arrange this view again by scrolling and zooming as necessary, we can see the graph
related to the email target (again noting, the lab view may see more than what is in this
screenshot):

Lab 2 33
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Notice that we have a lot of information about the email target now. We have the target email
address, the sending email address, the sending email domain, the subject line of the email,
and the sender IP address, amongst other pieces of information. We also have envelope icons
that represent any messages that matched the content – these are labeled “cisco_mid”, and
have a number that is the message ID on the Cisco Email Security appliance that processed the
message. There will be a cisco_mid icon for each message that contained any of our
observables, so there may well be more than one of these.

For now, we will click on the email subject, and then click on the downward arrow where the
resulting information window appears in the upper left corner of the relations graph window:

Take note of the subject line “Document you requested”, as we will use this to search for the
message in the next steps.

We now have all of the information needed in order to pivot to the Cisco Security Management
Appliance (SMA for short) in our Cloud Email Security solution (CES for short) in order to
investigate this message further. To do this, use the following shortened URL, replacing the X
with the Lab/Pod you are using (check with your proctor if you are unsure, it should be the
number on the end of the username you use to log in to the other tools):

Lab 2 34
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

http://cs.co/thw-email-X

The login prompt for the Cisco Security Management Appliance (the SMA) should appear. Use
the credentials provided for the lab in order to log in to the SMA.

Once logged in to the SMA, we will land on the Incoming Mail Flow Summary page. We can
take a moment to look at the summary information that our SMA is providing about the
incoming mail traffic. Clicking on “Time Range” in the top right corner of this view allows us to
toggle from the current day to whatever date range we prefer.

However, we have logged on to the SMA in order to investigate a specific message, so

afterward, we click on “Tracking” at the top of the screen, next to “Monitoring”:

This brings us to the message tracking screen, where we will now search using details of the
message we are looking for. There are any number of message details that we can search on,
and we can also restrict the time range for when the message was processed.

The default time range is 1 day, which we are going to change to cover the current month. To
do this, click on “Custom Range”, and change the “From” field to be 30 days previous to the

Lab 2 35
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

current day (the current day is automatically populated in the “To” field). This 30 day time
range also matches the length of time CTR will search for email targets.

Next, we will specify the subject line “Document you requested” in the “Subject” field. Note
that we have a number of options to choose from in searching for these text fields, but “Begins
with” will work for this situation because we are including the entire subject line. The resulting
message search options should look something like the following (adjusted for the current
date):

Now, we are ready to click on the “Search” button in the lower right-hand corner of the screen.
The SMA will search and return a list of the messages that match our criteria. There are any
number of criteria we could use to identify the message we are looking for, including the
message id of the message. For the sake of simplicity in this lab, we will simply look at the most

Lab 2 36
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

recent message that matches our search (which would also be guaranteed to have been
identified in CTR).

The first thing we notice is that the message appears to have been quarantined by our email
security. The notification “Quarantined by Content Filters” appears next to the details for our
message(s).

Other details are provided for the message in question which match what we saw in CTR,
including the sender, recipient, subject, and sender IP address.

A nagging question remains though – if the message was quarantined, then how did
bobc@acmemateriel(s).com hit this URL? We will look into the message details to see if we can
see anything that indicates how this happened. Click on “More Details” to switch to the
message details view. Once this view loads, we have significantly more details on how the
message was received and processed by our email security.

Lab 2 37
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

One thing to take note of here is that the message has an attachment – “Web_2_0_Draft.pdf”.
This file could be the source, if the URL was inside the PDF. But how did
bobc@acmemateriel(s).com get this if we quarantined it?

We have configured our CES to log URL’s in messages based on reputation. So, we will search
for the offending URL https://www.examplemalwaredomain.com/phishme.html in the message
tracking results, using standard web browser search capabilities (or our eyes).

Note that the logging lines can be scrolled horizontally, although scroll bars may not be
displayed depending on your browser. What do we see here? We can confirm that the
attachment did indeed contain the URL, from either of the log lines that include the URL:

Message 67 URL: https://www.examplemalwaredomain.com/phishme.html, URL

reputation: -6.3, Condition: URL Reputation Rule, Attachment
name:Web_2_0_Draft.pdf.

Something even more interesting pops up in the message tracking details just a few lines up
from our offending URL:

Message 67 Custom Log Entry: Message Recipients: bobc@acmemateriel(s).com

<bobc@acmemateriel(s).com>,"[email protected]" <[email protected]>

Might bobciscoman@gmail be the same person as bobc@acmemateriel(s).com? This seems

very likely, and we will follow up with Bob Ciscoman on this immediately.

Now we have two very important pieces of information for our investigation.

1. The URL was in a file attachment, “Web_2_0_Draft.pdf”

2. This file was also emailed to, we suspect, our internal user’s personal email address, and
this is how the file was accessed.

Lab 2 38
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

If we search the message tracking results from the top one more time, for the attachment
filename of “Web_2_0_Draft.pdf”, we can also find the SHA256 of the file on a log line similar
to the following:

Message 67 contains attachment 'Web_2_0_Draft.pdf' (SHA256

d58e26b26d370b6dee72ddff3996ebdd67219c4b954349f3eacddf6b5d9d2167).

We know that our email security protected us from this threat, but we would also like to make
sure that this file is blocked within our organization. To do this, we copy the above SHA256
value and add it to our investigation in CTR:

We then click again on “Investigate”, and wait for CTR to complete the query. Now, we will see
the SHA256 hash we added as part of the Relations Graph. We want to block this file using CTR,
and we can do so by clicking on the downward arrow next to the icon for the SHA256 in the
Relations Graph, and choosing “Add SHA256 to custom detections…”. Note that this block is a
toggle, so if someone else in the lab has already blocked this file, the option will be “Remove
SHA56 from custom detections…” instead:

Lab 2 39
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Now that we have blocked this file, it will be blocked on our Cisco NGFW, Cisco Cloud Email
Security (CES), AMP for Endpoints, and the Cisco WSA (Web Security Appliance) through the
integration of these products with AMP for Endpoints.

At this point we consider our investigation complete – we have identified the source of the
malicious domain, the malicious URL, and blocked the specific file across our business using the
integration of the Cisco Security portfolio and CTR.

Lab 2 40
THREAT HUNTING WORKSHEET
Name:
Date:

Kill Chain Analysis

Reconnaissance (Targeted System)
Patient Zero: System and User Information
Computer Name: Logged-in Username:
Operating System: MAC Address:
Vulnerable Applications: Host IP Address(es):

Delivery
Attacker Infrastructure and Dropper File
Malicious Domains: IP Addresses:
Dropper File Name: Dropper File Type:

Exploitation
Infection Vector
☐ Email: Attachment ☐ Web: Phishing Link ☐ Active Exploitation
☐ Email: Phishing Link ☐ Web: Exploit Kit ☐ Other:

Installation
Malware Binary
File Name: File Type:
Malware Family Name:

Command and Control

Command and Control (C2) Communications
Domains Contacted: IPs Contacted:
URLs Contacted:

Actions on Objective
Threat Behavior
☐ Process injection detected? (if so, which process)
☐ Escalation of privileges detected? (if so, which method)
☐ PowerShell / scripting commands detected? (if so, list)
Known Malware Capabilities:

Executive Summary
Date/Time of Patient Zero initial infection:
Total number of systems affected:
Timeline:

Security recommendations:

Sponsored by the Cisco Advanced Threat Solutions team and Cisco Incident Response Services. For assistance with an ongoing breach, contact
[email protected] or call 1-844-831-7715.
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Lab 3 – Bifrost
(Please note – Screenshots included in the lab may differ from actual results)

Scenario: One of your users was phished. The attacker was very careful, using a legitimate
email account belonging to an employee of a catering company that you’ve done business with
in the past. The email didn’t contain any active code or malicious attachments – just a link to a
website that looked very similar to a portal that is sometimes used for invoicing, but in this
case, the “invoice” was actually a powerful piece of malware. We were able to trace the name
of the file that was downloaded by querying our firewall, which intercepted the file and sent it
to the cloud sandbox for analysis. Unfortunately, the file was already on its way to the victim’s
computer when the alert came back for a malware detection.

Steps: First, we want to find out more about this malware. What is it? How does it work?
What is it capable of doing to its victim? To answer these questions, let’s turn to the Threat
Grid portal and look up the name of the file, invoice-1563.zip. First, go to the Search
(magnifying glass) icon in the top right of the Threat Grid portal:

This brings us to the Search Samples window. Enter the file name in the Query input field and
click Search:

Lab 3 42
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

We have a recent result:

Let’s click on the blue, hyperlinked name of the sample to view the analysis report.

Right away we can see that the sample is highly malicious. It received a Threat Score of 100 and
has two malicious Judgements, one from the AMP Protect Database and one from this Threat
Grid sample analysis report:

Lab 3 43
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

This top menu bar is the result of Threat Grid’s close integration with Cisco Threat Response.
Threat Grid can automatically query through the API to pull in contextual information about any
sample. We can click on each of the Metrics buttons to see data enriched from Threat
Response and other sources, but before we go to Threat Response, let’s spend some time
researching this malware here in the Threat Grid portal to understand its capabilities and
activity.

The left side of the Threat Grid sample report window shows us a navigation menu:

Lab 3 44
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

This menu allows us to jump to different sections of the report – from Metrics at the top, to
Indicators, HTTP Traffic, Processes, Artifacts, and more. Let’s get started by looking at the
Behavioral Indicators seen as this binary executed in the sandbox environment. Click the
Indicators link in the navigation menu, or simply scroll down a little to see the color-coded list
of indicators:

Lab 3 45
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

It’s an extensive list – there are 41 individual Behavioral Indicators reported for this one
malware sample alone! Each indicator is categorized and tagged based on its type (network,
forensics, persistence, evasion, and so on), and lists the number of hits or instances seen during
the sample’s runtime, as well as the score for each indicator. Some are very strong indicators
of malware, while others are less specific and might not raise the overall score of the sample
very much at all. For instance, the last few indicators (when sorted by default, which is by score
decreasing) involve odd values for fields in the file’s header and the use of a nonstandard port
for HTTP traffic – things that by themselves may not mean a lot, but when taken into account
together with all the other indicators Threat Grid found during the analysis, can paint a very
detailed picture of the malware’s behavior.

The highest-severity indicator is the first one we should explore – Bifrost Default
Mutex Detected. Click on the “>” icon to the left of the title to expand the indicator:

Threat Grid gives us a clear description of this indicator, showing what it means in plain English
that a specific mutex was detected. We now have a name for the malware binary: Bifrost, a
Remote Access Trojan or RAT. This malware has extensive capabilities, including the ability to
capture screenshots and keystrokes, record video and audio using the target’s camera and
microphone, monitor and manipulate processes, and manage files. Also included in the
indicator record is a specific pointer to the process identified as Bifrost, which in this case is
named SqGGuYXyy.exe:

NOTE: The Process ID number is dynamic and may differ from what is pictured
in the lab guide. However, the Process Name will remain the same.

This Behavioral Indicator also includes a direct link to the corresponding entry in the Process
section of the report. By clicking on the Process ID link within the Behavioral Indicator, we will
jump directly to Threat Grid’s record of this named process. If we expand this record by clicking
on the “>” button once more, we will find some very detailed and useful information about the
activity recorded by this specific process:

Lab 3 46
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

We can see its parent process – WScript.exe – and that it has four child processes, all
cmd.exe (also directly linked). The Artifacts section points us directly to the file on disk that
corresponds to the SqGGuYXyy.exe process, so let’s investigate it next by clicking on the
linked Artifact ID:

This brings us to the linked entry in the Artifacts section of the Threat Grid sample analysis
report:

Even without expanding the record for this artifact, we can see that SqGGuYXyy.exe is a
fairly large file (1550848 bytes on disk). It was created on disk as opposed to within memory or
through a network connection, it has imported 8 symbols from various DLLs (listed individually
farther down in the record), and it matches one antivirus service definition (also listed within
the record itself). Threat Grid also provides us with the SHA256 hash of this file which we can

Lab 3 47
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

use to pivot into other tools by clicking the dropdown context menu icon to the right of the
SHA256 hash:

Because Threat Grid is so closely integrated with Threat Response, we can access the same
pivots and enforcement actions directly from the Threat Grid report – and we can even build a
Casebook to keep track of any interesting observables we might discover from the analysis
report, and then pivot into Threat Response with all of the indicators at once. The Casebook
appears as a floating blue notepad icon that is found in the bottom right corner of the screen:

Clicking on the blue icon reveals the Casebook:

Lab 3 48
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

We can add observables to this casebook using pivot menus or by simply dragging an
observable into the casebook itself, and we can also add notes describing our work as we
progress through the investigation. However, this lab includes a casebook already created for
the investigation, so we will not need to add anything to it right now. Click the blue “Open” link
to bring up the “Lab -Bifrost Casebook” (you may need to scroll down to find the correct one):

This will bring up the prebuilt Casebook:

Lab 3 49
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Let’s return to our exploration of the Behavioral Indicators associated with this malware
sample. Click the “Indicators” link in the navigation pane on the left side of the screen to jump
back to our previous location:

Let’s take a look at another interesting indicator: “BITSAdmin Execution Detected”

which has a score of 95 and is tagged with “botnet”, “spreading”, “command and control”, and
“network”:

The description text tells us that the Background Intelligent Transfer Service, or BITS, is a
legitimate service that can be misused for malicious purposes. Examining the details for the
linked process, bitsadmin.exe, we can see the command line arguments passed to this
utility which tell us how the attacker used BITS to reach out to a Web server and download the
Bifrost malware. This raises the question, what process passed those command line arguments
to bitsadmin.exe and initiated this activity? The answer can be found from the process
details further in this Threat Grid report. Click on the Process ID to jump to the relevant
section:

Lab 3 50
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

The parent process is listed and linked immediately to the right of the original process name. In
this case, the parent process is WScript.exe. Click on the blue hyperlink to jump to this
process record:

Now we can expand the entry for WScript.exe and discover that it, in turn, was executed by
invoice-urgent.doc.js running from the C:\TEMP\ folder:

This shows us that the attacker leveraged JavaScript to initiate the attack sequence on the
endpoint. The filename of this script file is interesting, and appears to try to take advantage of
the default Windows behavior of hiding common file extensions to trick the user into believing
that it is an innocuous document file instead of a more suspicious JavaScript file.

Let’s continue to examine indicators that were surfaced in this file analysis report. Another
interesting Behavioral Indicator is “Downloaded Packed, Encrypted or Encoded
PE”:

This indicator tells us two things: first, that an executable was downloaded from a Web server,
and second, that the executable was obfuscated in some way (by being encrypted, encoded, or
packed). This is highly indicative of malware on its own – even more so when we look at the
domain that the executable was downloaded from, which are flagged as malicious by Cisco
Umbrella. This Behavioral Indicator provides quite a few observables that we want to see in
our upcoming Threat Response investigation, including the referenced domain

Lab 3 51
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

(getmalware.com) and the SHA256 of the downloaded executable (which we can assume to
be a secondary payload).

If this file was downloaded over the network, Threat Grid will show the raw HTTP connection
event in the Network section. Let’s jump to the HTTP Traffic link in the navigation pane:

There are five connection events using HTTP, all to the same server IP address: one HEAD
request, three GET requests, and one POST. The first two (HEAD and GET) are both part of TCP
stream 8. The HEAD method is identical to the GET method – used to retrieve information
identified by the Request-URI – except that the server will not return a message-body in the
response; this is used to test the link for accessibility and validity as all headers should be
identical to those returned by a GET request. If we compare the headers from Transaction ID
(TID) 0 and 1, we can verify that they are, in fact, identical:

Transaction ID 0, Method HEAD Transaction ID 1, Method GET

Request

Lab 3 52
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Response

The request headers (user-agent, connection, accept, accept-encoding, and host), as well as the
response headers (content-length, content-type, content-disposition, and date) show that a
request was made to http://getmalware.com:7777/payload by Microsoft BITS
version 7.5 and the server returned a status code 200, “OK”, along with a 1550848-byte
application file named payload.exe – which in this screenshot corresponds to the
packed/encrypted/encoded PE that we investigated previously.

There are many other Behavioral Indicators in this Threat Grid dynamic analysis report, and
each of them adds to our understanding of how this instance of the Bifrost malware functions.
Several indicators mention the use of JavaScript to call ActiveX objects and executables, while
others call attention to registry changes and code injection. Take a few minutes to read
through the rest of the indicators and note any interesting or potentially useful information.

At this point, we have added all relevant observables to our Casebook and are ready to pivot
into Threat Response for the next phase of the investigation. Check to ensure that the
observables in your Casebook match the following screenshot:

Lab 3 53
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Click the “Investigate” link in the upper right corner of the floating window to open a new
Threat Response investigation with all seven observables. You should see a Relations Graph
similar to this one:

We can immediately see that two internal hosts were affected. The Targets dropdown shows
us the details:

Lab 3 54
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Of our seven observables – four SHA256 hashes, two URLs, and one domain – two of the hashes
are clean files, two hashes are malicious, and both URLs as well as their domain are also
considered malicious by Threat Response.

Let’s take a closer look at these two clean files. Threat Response tagged most of the clean
SHA256 objects in the Relations Graph with file names and file paths, and we can see that
WScript.exe and bitsadmin.exe were both involved in some activity. The Observables
pane tells us more about these sightings:

These sightings and indicators explain how the Windows Script Host, or WScript.exe, executed a
JavaScript file inside a zip archive. This JavaScript file had a fake benign extension in an attempt
to trick the victim into opening it, and the result of this JavaScript execution by WScript.exe was

Lab 3 55
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

a set of command-line arguments passed to the Bitsadmin tool which is used to upload and
download files from remote servers even after the system is rebooted. Each sighting has a
pivot link into AMP for Endpoints that will jump directly to the relevant part of the Device
Trajectory. Let’s go there now:

Our Device Trajectory view in AMP for Endpoints is now filtered to show only events related to
the SHA256 for the Windows Script Host:

Clicking on each event icon, we can trace the relationships between the three processes: first,
explorer.exe launched wscript.exe (which we know came from the user opening the
JavaScript file). Then, the Windows Script Host launched bitsadmin.exe with the following
command line arguments:

Lab 3 56
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

AMP triggered three separate Indications of Compromise based on this activity, and we can
review them as well:

Lab 3 57
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Let’s look for this ZIP archive in Device Trajectory. The Threat Grid sample was named
invoice-1563; let’s clear out the SHA256 from the Search bar and look for this file name
instead:

We can now see the ZIP archive, created by Internet Explorer in a temporary folder:

We have now followed the chain all the way back to the beginning, when a user clicked on a
phishing link and received a ZIP archive containing a dropper JavaScript file which introduced
the malware binary into the environment. We also confirmed the Bitsadmin behavior detected
in the Threat Grid sandbox by reviewing AMP’s Cloud IOC events.

Lab 3 58
THREAT HUNTING WORKSHEET
Name:
Date:

Kill Chain Analysis

Reconnaissance (Targeted System)
Patient Zero: System and User Information
Computer Name: Logged-in Username:
Operating System: MAC Address:
Vulnerable Applications: Host IP Address(es):

Delivery
Attacker Infrastructure and Dropper File
Malicious Domains: IP Addresses:
Dropper File Name: Dropper File Type:

Exploitation
Infection Vector
☐ Email: Attachment ☐ Web: Phishing Link ☐ Active Exploitation
☐ Email: Phishing Link ☐ Web: Exploit Kit ☐ Other:

Installation
Malware Binary
File Name: File Type:
Malware Family Name:

Command and Control

Command and Control (C2) Communications
Domains Contacted: IPs Contacted:
URLs Contacted:

Actions on Objective
Threat Behavior
☐ Process injection detected? (if so, which process)
☐ Escalation of privileges detected? (if so, which method)
☐ PowerShell / scripting commands detected? (if so, list)
Known Malware Capabilities:

Executive Summary
Date/Time of Patient Zero initial infection:
Total number of systems affected:
Timeline:

Security recommendations:

Sponsored by the Cisco Advanced Threat Solutions team and Cisco Incident Response Services. For assistance with an ongoing breach, contact
[email protected] or call 1-844-831-7715.
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Lab 4 – Poweliks
(Please note – Screenshots included in the lab may differ from actual results)

Scenario: It’s early in the workday and you log into your AMP dashboard
(https://console.amp.cisco.com/) to check malware activity within your network. Right away,
you can see that there are a large number of affected systems listed in the Inbox tab:

Below the heat map – which is bright red, indicating that we have a high percentage of affected
systems – we can see the list of hostnames that are reporting security incidents. There are
quite a few, so how do we choose one to look at first?

One method would be to sort by date and work on the systems in chronological order. Another
method might involve working on the endpoints with the highest number of reported security

Lab 4 60
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

events. In either case, one system with the hostname Demo_AMP_Threat_Audit will be at
or near the top of our list. Why were 65 incidents reported on this single host? How can we
find out what happened on this endpoint, and how to protect it?

Steps: Let’s begin by analyzing the events surrounding the computer

Demo_AMP_Threat_Audit. From the Inbox, expand the record for this hostname:

We can see some basic information about the computer, such as the operating system,
connector version, install date, internal and external IP addresses, and the group and policy
associated with the AMP connector. In this case, we can tell right away that the Audit policy is
applied which may explain the large number of detection events, as AMP would be able to
detect malware but not quarantine it while in Audit mode. To confirm our theory, let’s open
Device Trajectory by clicking on one of the related events.

This brings up a new tab with the full Device Trajectory for this endpoint (you can close the
event detail window that opens):

Lab 4 61
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

There is quite a lot going on in the Device Trajectory. There is a filter and search pane at the
bottom which gives us the ability to view specific event types, including file create, file copy, file
move, file execute, and so on. We can filter by event dispositions to show only clean or
malicious files and processes, and we can filter to show only events that have a warning flag,
events from a connector in audit mode, or events that include captured command line
arguments. We can filter specific file types (executable, MS OLE2, PDF, etc), and we can search
for a file or process name – full or partial match – within the Device Trajectory view to isolate
our view to a specific interesting file or process. By default, all options are checked and the
view is unfiltered to show all processes that were observed on the system, excluding some
clean processes that only interact with other clean processes to avoid cluttering up the
trajectory unnecessarily:

Lab 4 62
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

The top part of the Device Trajectory view shows each of these processes and files and their
relationships in a graphical, chronological flow. In this case, we can see four processes and one
file – three of which are clean (rendered in green), one unknown (rendered in grey), and one
malicious (rendered in red):

Let’s click on one of the red event symbols to view the event record itself:

Just as we suspected, this file was not quarantined because the connector was in Audit mode,
and therefore the malware was able to execute repeatedly on the endpoint resulting in a large
number of individual detection events. Let’s trace back along the trajectory view using the
slider bar to find out how this malware was initially introduced to the endpoint:

We need to scroll back quite far, but eventually we will find the first detection event for the
red-colored process named ekjrngjker.exe [PE]. The event icon itself tells us quite a
lot about this initial event:

Lab 4 63
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

This was an execution event, shown by the triangle inside the circle that makes the first layer of
the icon. There is a command line capture flag just above this circled triangle that alerts us that
there was some type of argument passed to the executable, and the graphic of the eye at the
bottom right of the circled triangle tells us that this was an audit-only event. We can also see
that it was executed by a known clean process because there is a green line with an arrow
connecting to this file execution event. When we click on the event icon itself, we will see the
full description of this interesting event:

This confirms that the parent file was rundll32.exe, a Windows operating system utility.

There were a lot of processes executing at this time on the endpoint, and our Trajectory view is
quite busy. Let’s make life easier by filtering our view to show events surrounding one
particular process, and let’s begin with rundll32.exe. Type “rundll32” in the search bar at
the bottom of the Device Trajectory screen:

Lab 4 64
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Our Trajectory is immediately much cleaner:

We can not only see the execution of ekjrngjker.exe by rundll32.exe, but also the
Indications of Compromise (IOCs) that were triggered based on captured command line
arguments. These icons, which appear as a circled exclamation point and question mark, are
definitely worth a close look:

This command line argument string actually triggered two IOCs, one specifically looking for
evidence of malware called Poweliks and another, more generic IOC that looks for possible
abuse of rundll32 executing JavaScript code:

Lab 4 65
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Let’s refocus our Trajectory view to look at the malware process, ekjrngjker.exe. Type
the process name into the search bar to filter on this process and its parent and child processes:

We can still see rundll32.exe launching the malware process, but now we can also see the
malware launching a cmd.exe process to perform some malicious activity on the victim:

Let’s refocus again on cmd.exe by filtering on this process name to see all of these CMD
process launch events grouped together. Now our Trajectory view shows six CMD process
launches within a few minutes of each other:

If we click on each of these green CMD events, we will see the command line arguments passed
to the application that show us what the malware was trying to accomplish:

Lab 4 66
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Once we view all six events, we will see the following command line arguments (listed in no
particular order):

CMDC:\Windows\system32\cmd.exe /c netstat -an

CMDC:\Windows\system32\cmd.exe /c net user 'jmaldive' /add

CMDC:\Windows\system32\cmd.exe /c tasklist

CMDC:\Windows\system32\cmd.exe /c ipconfig

CMDC:\Windows\system32\cmd.exe /c net use

CMDC:\Windows\system32\cmd.exe /c net user

This malware apparently attempted to enumerate aspects of the system using commands such
as netstat, tasklist, and ipconfig, and added a local user for persistent access.

Now that we have a good understanding of the activities of this malware, let’s expand our
investigation to bring in the full threat intelligence research capabilities of Cisco Threat
Response by pivoting on the malware hash. Click on the dropdown arrow to the right of the
malicious (red) SHA-256 hash in one of the cmd.exe events to bring up the pivot menu:

Lab 4 67
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Click on the link titled “Investigate this SHA-256” to load this hash into Threat Response and
initiate a new investigation. This will bring up a Relations Graph centered on the malware file:

Lab 4 68
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Through the Sightings tab in the Observables pane of Threat Response, we can see information
collected from AMP for Endpoints each time the file was seen. We know the file name
(ekjrngjker.exe) and the file path that it runs from (the root of the C:\ drive). One of
AMP’s IOCs that we reviewed earlier suggested that this is a malware variant called Poweliks,
but let’s find out more about the malware sample by browsing Threat Grid to see if there are
any file analysis reports for this hash.

The Threat Grid Artifact Report shows details about this artifact, associated paths, and a list of
related samples with links to their analysis reports. Scroll down to the bottom of the page to
view the list of related samples:

Lab 4 69
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Threat Grid has seen this file several times with different filenames, but always with the same
SHA256 hash. Let’s click on one of the linked filenames to look at a sample report and find out
what Threat Grid knows about the file.

We will arrive at the Threat Grid sample report for this specific sandbox run:

The first behavioral indicator tells us that this is confirmed as a Poweliks variant. We can
expand the indicator to see additional information about this detection:

Lab 4 70
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

This tells us that Poweliks is a variant of a Trojan that does not require a file to be present on
the victim’s system. Instead, it uses Javascript embedded in the registry to store and launch
itself. It then uses PowerShell or .NET to carry out its instructions.

Let’s navigate back to the browser tab containing our Threat Response investigation. We can
see quite a few pieces of intelligence related to this malicious hash in our Relations Graph,
including two URLs, four IP addresses, and two clean files with metadata tags containing their
file names.

The domains and IP addresses came from events surrounding the initial observation of the
SHA256 hash when it was seen on the internal host. Our Relations Graph shows that the
malicious file connected to two URLs, both hosted on the retdemos.com domain. We can
add one of these URLs to our investigation by clicking on the context menu icon (the small
down-facing arrow next to the URL) and selecting “Add to Investigation”:

Lab 4 71
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

The observable icon will change to show a small blue magnifying glass above the icon itself,
which tells us that it is one of the observables we directly investigated (as opposed to an
observable brought in when Threat Response enriched, or queried threat intelligence sources
for, the observable(s) in the original query).

Threat Response tells us that this URL is malicious, but why? To find out, scroll down to the
Observables pane and examine the Judgment tab for the URL in List View:

We have one judgment against this observable, sourced from our Talos Intelligence module,
which carries the disposition of Malicious. The Reason column tells us why – it has a poor Talos
reputation score – and this disposition has both high severity and high confidence. We can
learn more by pivoting into the Talos Intelligence portal directly from the link in the Source
column of the Judgment tab:

We can see that the URL points to a server in Singapore, and that it has a poor reputation from
the Talos Web Reputation Center which is why it is labeled Malicious in Threat Response.

Lab 4 72
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Back in the Threat Response console, the Relations Graph shows us information about the
relationships between observables in this investigation. If we click on one of the suspicious
URLs, we can see that it was hosted by one of the IPs in our graph, 52.148.86.91:

Let’s look up this IP address in Umbrella Investigate to find out more about the domains that it
is hosting. Click on the context menu for the IP address and select “IP view” under the
Umbrella heading:

Lab 4 73
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

The Umbrella Investigate console shows us that this IP address has hosted two malicious
domains in the past week – legitmalware.xyz and retdemos.com:

We already knew that retdemos.com was hosted on this IP address, but the other domain –
legitmalware.xyz – is a new link in our investigation. Let’s view the Umbrella Investigate
domain report for this new domain by clicking on the link for legitmalware.xyz:

Lab 4 74
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

This console shows a wealth of information about the domain, including the Umbrella risk
score, a graph of the volume of DNS requests over time, WHOIS record data for the domain
showing the registrant’s email, name, and address, geolocation for the IPs hosting the domain
as well as its requesters, Threat Grid samples that have been associated with this domain, and
the current categorization of the domain.

Once we have finished researching this newly discovered domain in Umbrella Investigate, let’s
add it to our Threat Response investigation along with the server’s IP address,
52.148.86.91, and the original retdemos.com domain to see if any other endpoints in
our environment have been exposed to this malicious infrastructure. Go back to the Threat
Response tab and scroll to the top to see the Investigation pane which currently contains the
SHA256 of the Poweliks malware as well as the URL we previously added to the investigation:

Lab 4 75
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Then, add the two domains and the IP address from the Umbrella Investigate console
(legitmalware.xyz, retdemos.com, and 52.148.86.91):

After clicking “Investigate” to load the new query, we can immediately see that the
legitmalware.xyz domain, which was new information that we just surfaced using the
Umbrella Investigate console, has been contacted by two additional internal endpoints.

Our Relations Graph just got much busier! In addition to the two new internal targets, we also
have another malicious SHA256, two more clean SHA256s, a few more IP addresses, and quite a
few more URL nodes as well. Remember that we can move nodes in the Relation Graph to
organize our investigation and trace links between observables.

Lab 4 76
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

The two new clean SHA256s appear to have connected to several legitmalware.xyz URLs.
This is interesting, as any sign of a clean process communicating to a malicious Internet
destination could be an indicator of process hollowing or DLL injection. Let’s find out what
these clean SHA256s are by looking them up in the AMP for Endpoints File Trajectory:

When we expand the File Details for this File Trajectory page, we can see a familiar filename in
the “Known names” section. This clean SHA-256 is a version of Powershell.

Lab 4 77
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Now, let’s look up the other SHA256:

This SHA256 corresponds to the Firefox browser, version 41.0.0.0. It has a digital certificate
signed by Mozilla Corporation and it has been observed making outbound connections on ports

Lab 4 78
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

80, 443, 4444, and 8080 to a variety of IPs and URLs, consistent with behavior we would expect
to see from a browser application.

Lab 4 79
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Now, let’s look at the new malicious file that appeared in our Relations Graph, with a SHA256
beginning with fce5b678. We should add this file to our Threat Response investigation as
well:

This SHA256 now populates in our Observables pane and has been linked to one Target with 8
sightings. Let’s take a look at the Indicators linked to this SHA256 in the Observables pane:

Without even leaving Threat Response, we can see that Threat Grid has identified this file as a
Dorkbot variant. We can pivot into Threat Grid using the contextual menu to read the full
sandbox analysis report:

Lab 4 80
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

This link takes us to a page with all the information Threat Grid has collected about this
observable. The Details section shows hashes, the basic application type (file extension, MIME
type, and file magic type), as well as any flags or tags set during an analysis. The artifact report
also shows AV signatures that may trigger on the artifact, any associated file paths, and all
related Threat Grid sample submissions.

Lab 4 81
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

We can click the linked name of any related sample to view the full report, including all
Behavioral Indicators, a process tree, and a screen-capture video of the sample’s virtual
machine during the run.

Now we have identified the two variants of malware found on our three internal targets –
Poweliks and Dorkbot – and we can continue our threat hunting workflow by investigating what
happened on each of the affected systems.

Let’s take a look at the two systems that have connected with the malicious domain
legitmalware.xyz. One of them is the target of the Dorkbot variant that we just
investigated in Threat Grid – we can see this by the arrow connecting the target endpoint with
the red Malicious SHA256 icon; but there is no arrow connecting the malware to the other
target endpoint that had also connected to legitmalware.xyz. What could be the
difference between these two endpoints?

Lab 4 82
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

To find out, let’s pivot from Threat Response into the Device Trajectory for these two affected
hosts. To begin, select the Device Trajectory for the system that was connected to the Dorkbot
malware, Demo_AMP_Exploit_Prevention_Audit, by clicking on the any of the links
titled “AMP Event” within the Sightings for the Dorkbot malware file’s hash
(fce5b6784dc9f44cdc1d6214bb7b68d3029db049dcaf734edc9660bb3373bc79):

From here we can see that the original malware was dropped by powershell.exe:

Lab 4 83
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

We can also see that the file was detected and quarantined via a Retrospective Quarantine
event:

Further information can be gleaned from this Device Trajectory, namely that the network
connections generated by this sample are to the same URL that we researched earlier in Threat
Response:

Lab 4 84
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Right now, our Device Trajectory view is filtered to only show events related to this malware
hash. Let’s change the filter to show any events related to the domain legitmalware.xyz.
If the malware beaconing out to this domain, we may be able to find other processes that have
also reached out to the malicious domain. This information would help us discover how the
malware was loaded onto the system and executed.

Our Device Trajectory view is now filtered to only show network connection events to or from
this domain. There are three processes with these connection events – pp32.exe (the
malware file itself), powershell.exe, and firefox.exe. The first connection event
recorded was from Firefox:

Let’s try to match this activity with the other system that also reached out to the malicious
domain. Going back to our Threat Response investigation, we can pivot to the second machine
(Demo_AMP_Exploit_Prevention) which we saw had also contacted
legitmalware.xyz but was not linked to the malicious binary. We can access this
endpoint’s Device Trajectory by pivoting from its AMP Computer GUID:

Lab 4 85
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

This brings us to the Device Trajectory for this host:

From here we can see there was a vulnerability detected in Firefox running on this system:

Lab 4 86
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Scrolling back further we can see that an exploit was prevented within Firefox:

Clicking on the network connection right before the Exploit Prevention event occurred we can
see that this is the same domain and port that was contacted by our previous machine:

From this we can see that the Dorkbot attack relied on exploiting a weakness in the Firefox
browser, but because this system was protected by the AMP for Endpoints Exploit Prevention
technology, it was not infected.

Lab 4 87
THREAT HUNTING WORKSHEET
Name:
Date:

Kill Chain Analysis

Reconnaissance (Targeted System)
Patient Zero: System and User Information
Computer Name: Logged-in Username:
Operating System: MAC Address:
Vulnerable Applications: Host IP Address(es):

Delivery
Attacker Infrastructure and Dropper File
Malicious Domains: IP Addresses:
Dropper File Name: Dropper File Type:

Exploitation
Infection Vector
☐ Email: Attachment ☐ Web: Phishing Link ☐ Active Exploitation
☐ Email: Phishing Link ☐ Web: Exploit Kit ☐ Other:

Installation
Malware Binary
File Name: File Type:
Malware Family Name:

Command and Control

Command and Control (C2) Communications
Domains Contacted: IPs Contacted:
URLs Contacted:

Actions on Objective
Threat Behavior
☐ Process injection detected? (if so, which process)
☐ Escalation of privileges detected? (if so, which method)
☐ PowerShell / scripting commands detected? (if so, list)
Known Malware Capabilities:

Executive Summary
Date/Time of Patient Zero initial infection:
Total number of systems affected:
Timeline:

Security recommendations:

Sponsored by the Cisco Advanced Threat Solutions team and Cisco Incident Response Services. For assistance with an ongoing breach, contact
[email protected] or call 1-844-831-7715.
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Lab 5 – Threat Hunting

Scenario: John Doe from Human Resources is working on hiring additional security engineers
for your department. Unfortunately, this morning John let you know that he tried to open a
resume from an email attachment, but it did not open correctly - instead of a document, he saw
a command prompt window pop up on his desktop. John doesn't remember anything about
the email message subject, sender, or file attachment name, but he did take a screen capture of
his desktop:

Your mission is to find out what happened. Hunt the threat and fill out the Incident Response
Worksheet with your findings. When you are done, alert your lab proctor. Happy hunting!

Note: If you get stuck, there are some hints in the “Tips for Threat Hunting"
section at the end of the lab guide.

Lab 5 89
THREAT HUNTING WORKSHEET
Name:
Date:

Kill Chain Analysis

Reconnaissance (Targeted System)
Patient Zero: System and User Information
Computer Name: Logged-in Username:
Operating System: MAC Address:
Vulnerable Applications: Host IP Address(es):

Delivery
Attacker Infrastructure and Dropper File
Malicious Domains: IP Addresses:
Dropper File Name: Dropper File Type:

Exploitation
Infection Vector
☐ Email: Attachment ☐ Web: Phishing Link ☐ Active Exploitation
☐ Email: Phishing Link ☐ Web: Exploit Kit ☐ Other:

Installation
Malware Binary
File Name: File Type:
Malware Family Name:

Command and Control

Command and Control (C2) Communications
Domains Contacted: IPs Contacted:
URLs Contacted:

Actions on Objective
Threat Behavior
☐ Process injection detected? (if so, which process)
☐ Escalation of privileges detected? (if so, which method)
☐ PowerShell / scripting commands detected? (if so, list)
Known Malware Capabilities:

Executive Summary
Date/Time of Patient Zero initial infection:
Total number of systems affected:
Timeline:

Security recommendations:

Sponsored by the Cisco Advanced Threat Solutions team and Cisco Incident Response Services. For assistance with an ongoing breach, contact
[email protected] or call 1-844-831-7715.
Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Tips for Threat Hunting

1. What information can we get from the screenshot? Are there any file names or file
paths we could look up in one of our tools?

2. Threat Response will automatically parse out any IP addresses (v4 or v6), domains, file
hashes (SHA256, SHA1, or MD5), MAC addresses, and URLs. Other observables need to
be referenced using the format <type>:"<value>" where the type could be one of
the following:
• file_path • pki-serial • md5
• file_name • user • imei
• mac_address • email • imsi
• device • ip • amp_computer_guid
• hostname • ipv6 • amp-device
• domain • sha256
• url • sha1

As an example, to look up a file named test.pdf, type the following into the New
Investigation search box:

3. When you pivot into an AMP event from a sighting in Threat Response, your Device
Trajectory view will be filtered to only show events directly related to the observable
(typically a hash, IP address, or URL). You can clear the filter criteria to see all events, or
type something else into the filter bar to focus Device Trajectory on events related to a
different SHA256 hash, file or process name, IP, or domain.

4. Command line capture data in AMP Device Trajectory can be an invaluable resource
when you are looking for the reason why a file or process was
launched. If you see an icon that looks like the picture on the left,
this tells you that a benign (green) process was executed with some
command line arguments. You may need to scroll down to see the
command line section of the event within Device Trajectory
depending on your screen resolution; it will appear at the bottom of
the event, below the parent file certificate information.

Tips for Lab 5 91

Lab Guide – Threat Hunting Workshop Cisco Advanced Threat Solutions Team

Appendix 92