The Hunters Handbook
The Hunters Handbook
Handbook
Endgame’s Guide to Adversary Hunting
Publisher’s Acknowledgements
CyberEdge Group thanks the following individuals for their respective contributions:
Editor: Susan Shuttleworth
Designer: Debbi Stocco
Publishing Coordinator: Steve Piper
Table of Contents
Foreword................................................................................................................ v
Preface................................................................................................................ vii
Introduction..........................................................................................................ix
Chapters at a Glance......................................................................................... ix
Helpful Icons...................................................................................................... x
Investigation....................................................................................................43
Scoping the investigation.................................................................. 43
Gathering and analyzing information............................................... 45
Expanding the investigation.............................................................46
Reprioritizing the hunt...................................................................... 47
Adversary Removal.......................................................................................... 47
Synopsis of the Hunt........................................................................................49
Hunt Reporting.................................................................................................51
Glossary.............................................................................................................. 58
Foreword
Vikram Desai
Managing Director, Global Lead Security Analytics
Accenture
Introduction
Chapters at a Glance
Chapter 1, “The Power of Hunting,” explains the basic
concepts of hunting, the motivations for hunting, and the
benefits of hunting.
Helpful Icons
TIP
Tips provide practical advice that you can apply in your own
organization.
DON’T FORGET
When you see this icon, take note as the related content
contains key information that you won’t want to forget.
CAUTION
Proceed with caution because if you don’t it may prove costly
to you and your organization.
TECH TALK
Content associated with this icon is more technical in nature
and is intended for IT practitioners.
ON THE WEB
Want to learn more? Follow the corresponding URL to
discover additional content available on the Web.
Chapter 1
Advanced Threats
The threats against the data on our systems and networks are
increasingly adept and sentient, making them much harder
to stop. Let’s look at threats in terms of the individuals and
groups causing them, and the ways in which these individuals
and groups attack organizations.
Adversaries
Today a wide range of adversaries pose threats to organiza-
tions. Some are highly skilled and primarily rely on tools and
techniques they develop, while others rely mostly or entirely
on tools developed and distributed elsewhere. Some adversar-
ies are lone actors, but others are teams and groups such as
nation states, criminal organizations, and hacktivist groups.
CAUTION While most adversaries are external, many breaches are per-
formed by organizational insiders. This book focuses on hunt-
ing for external adversaries, but don’t forget about internal
adversaries and the threats they pose to their own organiza-
tions. Some insiders are obviously difficult to detect, such as a
colleague who steals unattended printouts containing sensi-
tive data. However, many insiders use some of the same tools
and techniques that external adversaries use, which means
that hunting can play an important role in detecting malicious
insider behavior.
What sets today’s adversaries apart from previous generations
is the sophistication of techniques. Adversaries are increas-
ingly employing never-before-seen tools and tactics, including
custom and polymorphic malware that defeats existing
security technologies. They are building evasion techniques
into their exploits and malware to disable or circumvent
traditional security tools and gain access to networks and the
assets connected to them.
Chapter 1: The Power of Hunting | 3
Weakening Defenses
As adversaries have become highly motivated by financial,
political, and ideological aims to pursue data breaches and
other compromises, they’ve realized the importance of avoid-
ing defenses. By using a stealthy attack approach that takes
advantage of defensive weaknesses, adversaries can maintain
their presence within an organization for weeks, months, or
even years.
One of the main reasons that organizations’ defenses have
weakened so much is the erosion of the traditional perimeter.
Organizations rely on firewalls, intrusion prevention systems,
and other network security controls at the perimeter to
prevent threats from entering their internal networks. Today,
with laptops and mobile devices operating and connecting
from outside the perimeter, and many servers and services
hosted in external clouds, the perimeter is porous, riddled
with additional, highly dynamic entry points provided by both
enterprise and personal devices.
As the perimeter dissolved, adversaries learned how to evade
detection by security technologies that rely on signature-
based methods. These technologies can’t keep up with highly
dynamic and customized exploits and malware. Indicators
of compromise – distinct characteristics that correspond to a
particular campaign or piece of malware – are often unique
for each instance of an attack, making detection through
known signatures unlikely or impossible.
Chapter 1: The Power of Hunting | 5
DON’T FORGET Although the focus of this book is hunting for external threats,
insider threats are also a major concern. They often involve an
employee’s abuse of privileges granted by the organization,
thus avoiding the need to use malware or other exploits. Most
security tools can’t differentiate an insider’s malicious actions
from benign actions. Many insiders simply copy data from their
computers to a USB flash drive or other removable media, an
action that may go completely unnoticed by security controls.
As the result of weaker defenses and stronger threats, prevent-
ing compromises is much harder and detecting them takes
much longer. On average, adversaries are present within an
organization for 146 days before they are detected, and it often
takes weeks or months more to fully remove them. The period
during which an adversary maintains a continuous presence
within an organization, generally achieved through malicious
processes running on one or more assets, is better known as
dwell time. (The source of the dwell time statistic above is
Mandiant’s M-Trends 2016 report.)
Hunting Basics
The primary goal for hunting is to reduce dwell time. This helps
the organization reduce the damage and loss it incurs from
compromises. Hunting accomplishes these goals by bringing
an offense-based approach to securing your organization’s
systems.
To understand what that means, let’s first look at the current
state of security. It relies almost completely on passive, reac-
tive defenses. Once security professionals become aware of a
compromise that’s been detected through standard, passive
security controls or reported by an external party, they react to
that compromise and execute incident response procedures to
recover.
Unfortunately, this leaves a great deal to be desired. When an
organization relies on passive, reactive defenses, less-skilled
adversaries may be stopped quickly, but more-sophisticated
adversaries can easily evade the defenses and reside undetected
within the organization’s systems and networks.
6 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
characteristics (i.e., signatures) and can’t identify attacks that don’t use
malware.
Survey Phase
The first phase of the hunt cycle is to survey the environment.
The purpose of this survey is to determine on which assets
in the environment hunting will take place. In addition, the
hunter must implement the necessary monitoring capabilities
on each of the selected assets and begin collecting data from
them. Let’s look at these components in detail.
Select assets
Selecting the assets to hunt on next may sound simple, but
it’s incredibly important. If a critical asset is overlooked or
mischaracterized, the hunters may not look for threats on that
asset in a timely manner, if at all. As a result, the dwell time
for any adversaries on the asset may be extremely long, and
additional loss of sensitive information may occur.
In an ideal world, hunters would identify every asset within
the organization through network-based scans or other means
and hunt on all of them. Alternatively, hunters could perform
the hunt on every asset that stores or processes sensitive
information, plus every asset that provides access to those
assets, and so on. Unfortunately, these approaches would end
up involving just about every asset.
TIP Few organizations, if any, have the resources to actively hunt
on all their assets all the time, so it’s vital that the assets per-
ceived as the greatest risk to the organization are prioritized
for hunting purposes.
Let’s take a step back and think about the factors hunters
should consider when deciding which assets to hunt on.
Examples include the following:
12 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
Monitor assets
Once the hunters have selected the assets to hunt on, the
next step is to ensure that the necessary monitoring and data
collection capabilities for each of the assets are in place. The
goal of monitoring the assets is visibility into the activities
occurring within them, as well as between an asset and any
others, in order to observe actions performed by adversaries
and maintain situational awareness for current threats.
Achieving full visibility into security events for the hunt gener-
ally requires both host and network monitoring. Network-
level monitoring is usually present in the enterprise through
intrusion prevention systems and other network-based
security technologies. Hunters should have access to the logs
from these technologies to help them verify the sources of
suspicious activity and correlate activity across assets.
Host-level monitoring is best accomplished by installing hunt
sensors on these assets. Chapter 3 contains much more on
hunt sensors, but basically a sensor performs a wide range
of hunt functions within an asset. An alternate approach to
installing a hunt sensor is to use a sensorless monitoring
method. There are limitations to a sensorless approach
because it relies on user mode and operating system data
retrieved through an application programming interface
(API). A sensorless approach can’t provide continuous moni-
toring, creating blind spots in the data. Another weakness is
its dependence on data from system log files, which can be
altered by the adversary, thus giving low confidence in the
data collected from them.
TECH TALK For the purposes of the hunt, host-level monitoring usually
focuses on certain types of activities, such as unexpected and
anomalous features of the filesystem, as well as particular
processes, network connections, and configuration settings
(e.g., registry keys on Microsoft Windows assets). Such anom-
alies are often evidence of a current or past compromise. See
Chapter 5, “The Hunt Experience,” for deeper technical
insights into this topic.
It’s important that monitoring and data collection be imple-
mented in such a way that adversaries can’t disable or evade
it. If given the opportunity, adversaries will shut off logging
14 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
Secure Phase
Many people don’t realize that prevention is a major compo-
nent of hunting. The unspoken assumption is that securing
assets—everything from applying patches and configuring
software securely to installing, monitoring, and maintaining
network and host-based enterprise security controls—is the
responsibility of system administrators, security administra-
tors, and other operational staff.
Unfortunately, this is a short-sighted approach based on a
reactive, passive defense. While operations should certainly
play the largest role in fundamental security activities for
assets, hunters are perfectly positioned to take asset security a
step further through the use of hunt sensors.
TECH TALK Some hunt sensors provide mechanisms that can prevent the
use of various exploit and attacker techniques. Examples of
these techniques include:
Detect Phase
At this point in the hunt cycle, the hunter has selected assets
for the hunt, established monitoring and protection capabili-
ties for those assets through hunt sensors, and ensured that
the hunt sensors are preventing any additional movement of
the adversaries to limit further damage and loss.
Detect attacks
At the heart of the hunt is the ability to detect attacks as early
as possible in the kill chain. Hunters must gather data that
could uncover suspicious activities and analyze that data to
find the adversaries hiding among all the noise. Indications
of malicious activity can appear within a single set of data or
can be found by looking for anomalies across many assets.
Hunters can leverage analytic capabilities provided by their
hunt platform, along with their own abilities to understand
the meaning and significance of the collected data, to ulti-
mately identify the compromises.
Finding anomalies in the collected data that indicate the
presence of attackers is far more effective at detecting the
most advanced threats than searching for known indicators of
compromise, which change rapidly and often aren’t useful by
the time they make it into detection tools. If technologies to
secure the hunting ground are in place, those capabilities can
provide high-confidence detections of malicious activity.
DON’T FORGET Remember that it’s not hunting if the hunter already knows
what to look for.
16 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
Perform analysis
The sheer volume of data captured by hunt sensors moni-
toring the assets can be overwhelming. Fortunately, hunt
technologies also offer analysis capabilities. Here are some
commonly offered capabilities:
Respond Phase
Once a hunter has found a compromised asset, the hunter
may choose to monitor the adversary for some time to collect
additional information on the adversary’s techniques, intent,
and goals. Ultimately, however, the hunter’s objective will be
to evict the adversary and stop any further damage or loss,
and report findings from the hunt.
18 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
Report findings
TIP Although reporting findings comes at the very end of the hunt
cycle, hunters must document their actions throughout the
course of the hunt. Trying to recall accurately all the signifi-
cant actions, the sequence and timeline of events, and other
details after the hunt is completed is often impossible. Partial
documentation may also be needed during the hunt, such as
briefing others within the organization on the status of a
potential compromise being investigated by the hunter.
CAUTION Hunters must be careful to strike the right balance between
hunt documentation and primary hunt goals. Spending an
hour documenting hunt actions while an adversary exfiltrates
sensitive data from the organization is not a wise use of time.
Instead, hunters should strive to document enough informa-
tion so they can fill in the blanks when time permits.
Automated tools may also be helpful in keeping track of
hunter actions.
20 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
Supplementing Signature-
Based Detection
Signatures have been used to automatically detect attacks for
decades. Although signatures can stop attacks that have been
seen before, the customized nature of current attacks means
that signatures can’t detect them.
Hunt platforms can use indicators of compromise and other
forms of signatures to detect some adversaries, but they must
also monitor attempts to use attack techniques within assets
to detect more sophisticated adversaries. Although most
instances of malware are unique, nearly all malware uses com-
mon attack techniques, such as process injection, credential
dumping, token stealing, and lateral movement. Non-malware
forms of attack rely on these techniques too.
A hunt platform can constantly look for and stop attempts
to use attack techniques by monitoring a relatively small
number of chokepoints within each asset’s operating system.
Executing most malicious actions requires adversaries to use
one of these chokepoints. Another helpful characteristic of a
hunt platform is that it can look for patterns across assets that
indicate malicious activity.
CAUTION Here are some tips for hiding from the adversary:
Endgame Advantages
Endgame offers a comprehensive related communications
hunt platform that automates between its components.
the hunt for never-before-seen
• The Endgame platform rapidly
adversaries before any damage
detects activity across all
and loss occurs. The Endgame hunt
stages of the kill chain to stop
platform has numerous advantages
adversaries from gaining a
over other hunt technologies.
foothold. The platform protects
Let ’s highlight a few of these
e n t e r p r i s e s a ga i n s t b o t h
advantages.
malware-based and malware-
• Endgame has extensive less attacks by focusing on
automation capabilities. detecting the attack techniques
The Endgame platform rather than only known
automates asset discovery, indicators of compromise.
sensor deployment, adversary
Endgame is focused on solving
detection, question-driven
t h e h a rd p ro b l e m o f f i n d i n g
investigation, and response
threats in innovative ways, instead
actions to reduce investigation
of reinventing the wheel and
time and stop advanced
creating yet another solution that
adversaries. Automation across
relies on observing filesystem
the hunt process improves
changes. Endgame’s approach
productivity of tier 1 and tier
enables instant detection and
2 analysts while enabling tier 3
characterization of new threats,
analysts to scale their hunting
which in turn enables immediate
across the enterprise.
and precise responses to those
• The Endgame platform threats. These responses evict
operates stealthily to evade adversaries from the enterprise
adversary detection. It hides while minimizing or completely
its presence from adversaries avoiding any business disruption.
in terms of deployment,
For more information on Endgame’s
execution, and communication.
advantages, check out the archive
Endgame hunt sensors have
of hunt webinars available at
a unique footprint on each
https://www.endgame.com/
asset, thus providing signature
resources.
diversity. Also, the Endgame
platform encrypts all hunt-
Chapter 4
Hunt Readiness
In this chapter
Know what to keep in mind when defining hunting roles and
responsibilities
Learn what’s involved in scoping the hunt, including developing
the cyber risk assessment report, hunt policy, and rules of
engagement documents
Understand how to build and maintain hunters’ capabilities
CAUTION Every organization should have its own structure and job defi-
nitions related to security in general and hunting in particular.
Each organization is unique in terms of its culture, security
requirements, and risk appetite and profiles. This section
illustrates the range of hunting responsibilities and suggests
one way of distributing them.
IT operations personnel
IT operations personnel are usually responsible for daily
deployment, monitoring, and maintenance tasks for the orga-
nization’s IT assets and networks. This work, which includes
security-related tasks such as installing patches and checking
configuration settings, is primarily performed by IT staff who
don’t specialize in security. However, it may also involve the
organization’s Security Operations Center (SOC) team—for
example, a security analyst who receives an off-hours noti-
fication of a new exploitable vulnerability that needs to be
patched immediately.
In terms of the hunt, the designated operations and/or SOC
personnel play a key role in ensuring that the IT assets are
well secured under both typical and emergency conditions.
They may also be called upon to help restore normal opera-
tions after an asset compromise.
Security team
The incident response personnel and the hunters are usually
part of a larger security team. This team has many respon-
sibilities, including identifying, documenting, and reporting
shortcomings in the organization’s security plans, policies,
procedures, and technologies, and recommending ways to
address them. The goal is to make compromises more difficult
for all adversaries while avoiding negative impact to the orga-
nization’s legitimate users.
TIP The individuals who manage the security team usually have
the ultimate responsibility for hunting. Duties include every-
30 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
Policy development
The hunt policy contains all the requirements for hunting
within the organization, without specifying step-by-step pro-
cedures or other low-level information. It’s also common for
the hunt policy to contain guidelines and recommendations.
These often give hunters flexibility in deciding what to do or
not do, while still encouraging certain actions and behaviors.
The contents of the hunt policy will vary from organization
to organization, but at a minimum the following should be
addressed:
Definitions
External Requirements
Personnel
Metrics
TIP The organization should also have metrics for measuring the
success of the hunt program. These aren’t to be confused with
hunt performance metrics, such as the average dwell time per
compromise and per adversary. Rather, these are metrics that
quantify the impact of hunting on the entire organization,
such as the annual reduction in damage or improvement in IT
availability. At first, hunt plan metrics may be highly subjec-
tive and thus have limited usefulness, but over time the hunt
program should mature and the metrics can mature
correspondingly.
Security knowledge
Hunters obviously must have the ability to use hunt tech-
niques and tools, including those typically used for red
teaming and penetration testing. In addition, it’s beneficial
for hunters to have strong knowledge of enterprise security
principles and how they are implemented through security
architectures and technologies, as well as operational and
management security controls. Hunters who understand how
all the different areas of security interact and fit together will
be more capable of finding adversaries and understanding
what they’re trying to do.
Other helpful areas of security knowledge for hunters include:
IT knowledge
Hunters should have a solid understanding of the organiza-
tion’s IT assets and networks. Much of this knowledge is
a prerequisite to grasping security. For example, a hunter
should be familiar with the organization’s enterprise architec-
tures, especially applications, to understand how parts of an
application (user interface, middleware, database server, etc.)
are divided among assets and interact with each other.
Another example of important IT knowledge is familiarity
with the internals – the inner workings – of the operating
systems used by the organization’s assets. Such knowledge
is essential for investigating events and understanding their
effect on each asset. Hunters should also be familiar with
Chapter 4: Hunt Readiness | 37
Hunting mindset
What distinguishes hunting from so many other aspects of
defense is the requirement to think like an adversary. At a
high level, this means that the hunter approaches the asset as
an adversary would, focusing the hunt on the aspects of the
asset that adversaries would focus on as well.
Decision-making
Decision-making is a vital skill for hunters. At key points in
the hunt, hunters must assess the current state of security.
Relying on their strong critical thinking skills, they must
consider many factors before deciding what to do next and
when to do it. Examples of decision-making points are pre-
sented earlier in this chapter within the “Hunters and Risk
Assessment” sidebar.
Communications skills
Another important skill for hunters is the ability to commu-
nicate effectively with others. Communication may involve a
wide range of audiences, from other hunters, security profes-
sionals, and system administrators to upper management, the
legal team, and human resources personnel.
38 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
;; Preparation
;; Investigation
;; Adversary Removal
;; Hunt Reporting
Preparation
Pat needs to perform some preparatory tasks before hunting.
These include determining hunt priorities, reviewing available
information on the organization’s IT assets and networks,
understanding what’s considered normal activity on the orga-
nization’s assets and networks, and configuring and deploying
the hunt sensor software on the assets selected for hunting.
CAUTION Most of these tasks aren’t meant to be performed each time
the hunter is getting ready to hunt. They’re general prepara-
tory activities that will make future hunts go more smoothly
Chapter 5: The Hunt Experience | 41
and be more effective. For example, the hunter can act imme-
diately when a likely compromise is discovered. Another
example is that the hunter can quickly reprioritize hunt activi-
ties as needed while gathering more information about a
potential compromise.
Investigation
Pat has completed all the preparatory activities. It’s time to
investigate the 25 selected assets for the presence of adversar-
ies. This investigation is documented in four parts: scoping
the investigation, gathering and analyzing information,
expanding the investigation, and reprioritizing the hunt.
Pat uses the hunt platform to review the results from the hunt
sensors, and Pat quickly discovers that the third server, which
is the primary web server that the company’s customers use,
has a running process that seems suspicious.
TECH TALK The IIS process (the Microsoft web server itself) has an
unbacked executable. A DLL was loaded into the process’s
memory space but no file corresponding to that DLL exists on
the server’s disk. Normally, all code executing in memory
comes from a file on disk with a filename ending in .exe, .dll.
.sys, etc. Two common causes for an unbacked executable are
46 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
Adversary Removal
All evidence points to an adversary exfiltrating sensitive data
from the company via the compromised database server. Pat
must act as quickly as possible to stop the compromise by
disrupting the exfiltration communications. Possible methods
for handling this situation include the following:
Key Insights
Shadowing Pat has provided you • Hunters must constantly
with key insights into hunting, reprioritize their actions based
including the following: on all available information.
• Hunt technologies enable • Hunters must think and react
hunters to detect advanced quickly in order to detect
adversaries, including those and evict adversaries while
already present within the m i n i m i z i n g t h e i m p a c t to
enterprise who have bypassed operations.
other security controls.
Hunt Reporting
Pat documented the findings and results for the hunt on
an ongoing basis, issuing a separate report for each major
compromise investigated through hunting. Pat’s report
for the database server compromise includes the following
information:
Stealth
An adversary who discovers an organization’s security controls
can disable or tamper with them, or simply alter attack plans
to avoid detection by those controls. These tactics illustrate
why it’s so important for the hunt solution to be stealthy, hid-
ing its presence from adversaries. Organizations should look
for hunt technologies that offer the following characteristics
supporting stealth:
54 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
Automation
The degree to which a hunt solution is automated can make
a huge difference in the effectiveness and productivity of
security analysts. Every action that hunt technologies can per-
form on behalf of a human hunter frees the hunter to execute
the most impactful and challenging parts of the hunt, such
as analyzing information provided by the hunt sensors, and
expanding hunting to include other assets.
Chapter 6: Hunt Technology Selection | 55
Workflow Support
A common problem of security software is poor workflow
support. For example, a security analyst investigating an issue
often has to access a separate console interface for each piece
of software. This forces analysts to alternate between several
consoles during a single investigation, creating significant
inefficiencies and delaying analysis.
In contrast, a hunt solution can optimize hunters’ efforts
by streamlining workflow. A holistic hunt solution gives
the hunter access to a wide range of information in a single
console. This greatly reduces the time it takes for hunters to
complete tasks, including deploying hunt sensors, analyzing
hunt data, and evicting adversaries.
Enterprise Integration
Hunt solutions must provide robust integration with existing
business processes and security infrastructure. An essential
component of maximizing return on investment (ROI) for
your hunt solution is to integrate it with the organization’s
existing security investments.
A great example of enterprise integration is the hunt solution
providing information to the organization’s SIEM. The hunt
solution can instruct the SIEM to enforce remediation actions,
from changing firewall rulesets to adding IPS signatures.
Integration with the SIEM allows analysts to use hunt data
when they look for potential security incidents. This integra-
tion also permits the SIEM to perform log management on
behalf of the hunt solution.
56 |The Hunter’s Handbook: Endgame’s Guide to Adversary Hunting
Scalability
A final key attribute of a hunt solution is its scalability. When
an organization is first adopting the hunt, scalability may not
be a major concern, but as the value of hunting becomes obvi-
ous, its scope will naturally expand. This expansion is likely to
be unmanageable and unsustainable unless the hunt solution
can support all of the organization’s IT assets and hunters.
TIP Organizations should ensure that the hunt solution they
acquire and deploy is not only sized reasonably for meeting
current requirements, but also is easily expandable on an as-
needed basis as requirements change.
Chapter 6: Hunt Technology Selection | 57