Chapter 6:

Advanced Malware Analysis

Techniques
Outlines
• Dynamic Analysis

• Kernel-Level Debugging

• Memory Forensics Techniques in malware analysis

• Fuzzing
 Introduction to Advanced Dynamic Analysis
Techniques
• Dynamic analysis allows malware analysts to observe the behavior of
malicious software while it executes in a controlled environment. This
technique is crucial for uncovering malware behavior that may not be
evident in static analysis, such as:
o Persistence mechanisms
o Network communications
o Process creation and manipulation.
 Importance of Dynamic Analysis in Malware
Detection
• Dynamic analysis plays a critical role in detecting:

o Malicious payloads that only execute under specific conditions.

o Malware designed to evade static analysis.

o Techniques for analysing different malware types such as trojans, ransomware,

and rootkits.
Introduction to Kernel-Level Debugging
• Kernel-level debugging involves
analysing the code that runs in
kernel mode.
• Kernel-mode code has unrestricted
access to system resources, making
it the primary target for advanced
malware such as rootkits.
• Kernel-level malware can modify
system behaviour, hide processes,
and avoid detection.
Why Kernel-Level Debugging is Important

• Kernel-level debugging is essential in malware analysis because:

o It allows detection of rootkits that operate deep within the system.
o Helps identify privileged malware techniques such as function hooking and
API interception.
o Enables understanding of how malware manipulates low-level system
components.
Key Concepts in Kernel-Level Debugging

1. Kernel Mode vs. User Mode:

o Kernel mode has full access to the system, while user mode is restricted.

2. Rootkits:
o Malware designed to hide its presence and persist in kernel mode.

3. Function Hooking:
o A technique to intercept function calls to system APIs.
Tools for Kernel-Level Debugging
1. WinDbg:
o A comprehensive debugger for analysing both user-mode and kernel-mode
applications.

2. KD (Kernel Debugger):
o Used for debugging kernel components from a remote machine.

3. Example Session:
o A typical analysis session involves attaching WinDbg to the system kernel to
examine system calls, memory access, and potential malware injection points.
 Common Malware Techniques in Kernel-
Level Malware
• Code Injection:
Malware injects code into other processes to remain hidden.
• Process Hiding:
Rootkits modify the kernel to hide processes from task managers and security
tools.
• Hooking System Functions:
Intercepting critical OS functions to alter behavior or hide malicious activity.
 Analyzing Network Communications of
Malware
• Network analysis is critical to understand how malware communicates
with Command and Control (C2) servers.

• Malware often communicates through standard protocols like HTTP,

DNS, or custom protocols to exfiltrate data or receive commands.

• Analysts track and block these communications to isolate malware

activities.
Network Traffic
Analysis Tools
1. Wireshark:
A widely-used network
packet analyser for
capturing and inspecting
traffic in real-time.
Example: Analyzing a Packet Capture
• In this example, a malware sample is found communicating with a
remote C2 server. Using Wireshark, we can:

• Filter HTTP traffic.

1.

• Identify suspicious domains and IP addresses.

2.
• Trace the exfiltration of data or the download of malicious
3. payloads.
2. tcpdump:
A command-line tool for
capturing network packets for
offline analysis.
Both tools are essential in
analysing malware traffic and
identifying malicious
communications.
Understanding Network Protocols Used by
Malware
1. HTTP:

• Malware uses HTTP to blend malicious traffic with regular web traffic.

2. DNS:

• Command and control instructions can be embedded within DNS queries.

3. SSL/TLS:

• Encrypted communications make it harder to detect malware traffic.

 Interpreting Network Packets

1. Filtering Packets:
§ Using tools like Wireshark, analysts filter traffic based on protocols or
suspicious IP addresses.

2. Detecting C2 Communications:
§ Indicators include repeated connections to the same IP/domain, encrypted
traffic, or unusual packet sizes.
 Memory Forensics Overview

• Memory forensics is a powerful technique to analyse malware that

resides only in memory or injects malicious code into other processes.
Analysts can find critical artifacts such as:
• Unusual process injections
• Hidden processes
• Suspicious network connections from memory dumps.
 Memory Forensics Techniques

Memory forensics is the process of analysing a system's memory

to uncover malware that resides solely in RAM.
•Purpose:
üDetect malware that may not leave traces on disk.
üRetrieve information on active processes, injected code, and network connections.
•Artifacts in Memory Forensics:
üHidden Processes: Malicious processes that attempt to conceal themselves.
üSuspicious Connections: IP addresses and domains involved in C2 communication.
Memory Forensics Tools and Techniques

ØTools:
qVolatility: A powerful framework for analysing memory dumps,
identifying injected code, and detecting hidden processes.
qFTK Imager: Useful for acquiring and imaging memory for forensic
analysis.
ØCommon Commands and Techniques in Volatility:
qpslist: Lists running processes to detect anomalies.
qnetscan: Identifies open network connections.
qmalfind: Scans for malicious injections in memory.
Memory Forensics Tools

1. Volatility:
• An open-source framework for
analysing memory dumps to detect
malware behaviors.

2. FTK Imager:
• A tool to capture and analyse memory
dumps from infected machines.
 Fuzzing Techniques Overview
•Definition:
Fuzzing is a software testing technique that involves providing invalid, unexpected, or random data as inputs to a
program.
The goal is to detect bugs or vulnerabilities by observing crashes, unexpected behavior, or security flaws in the
software.
•Purpose of Fuzzing:
•Uncover vulnerabilities that may not be easily identifiable through static analysis or manual testing.
•Test the robustness and security of software by simulating unusual inputs or user behavior.
•Help identify memory corruption issues, such as buffer overflows, use-after-free errors, and input validation failures.
Types of Fuzzing:
1.Generation-based Fuzzing:
1. Inputs are generated from scratch according to predefined rules, specifications, or
protocols.
2. Effective for well-documented file formats or network protocols.

2.Mutation-based Fuzzing:
1. Inputs are derived by altering existing valid inputs (e.g., flipping bits, removing
data).
2. Useful when you have a set of valid inputs and want to explore edge cases or
anomalies.
 Black-box vs. White-box Fuzzing
• Black-box Fuzzing:
• Definition: The tester has no knowledge of the internal workings of the
system being tested.
• Advantages:
• Simple to set up, as it does not require access to the codebase or knowledge of the system
internals.
• Can uncover vulnerabilities in the overall behavior and interactions of the system with
external inputs.
• Use Cases:
• Ideal for testing commercial off-the-shelf (COTS) software or third-party applications.
• Focused on identifying flaws in exposed interfaces like APIs or network protocols.
Black-box vs. White-box Fuzzing
• White-box Fuzzing:
• Definition: The tester has full access to the system’s source code, internal
structure, and functionality.
• Advantages:
• Enables more thorough testing by guiding the input generation process based on
knowledge of the system’s code paths.
• Can identify deep vulnerabilities by targeting specific code paths or complex logic.
• Use Cases:
• Effective for internal software development where the source code is available.
• Suitable for targeting sensitive areas like authentication modules, encryption
algorithms, or memory management.
• Comparison:
• Black-box:
• Requires no knowledge of internal code.

• Broad but shallow coverage.

• White-box:
• In-depth exploration of code paths.

• Requires access to code and often more computational resources.

Fuzzing Tools Overview
• AFL (American Fuzzy Lop):
• Description: A widely-used, open-source fuzzer that uses a combination of
generation-based and mutation-based techniques. It instruments target binaries
during compilation to monitor code coverage and guide input mutations.
• Features:
• Efficient at discovering memory corruption bugs (e.g., buffer overflows).
• Tracks code coverage to explore as many unique execution paths as possible.
• Highly customizable with support for multiple file formats and protocols.
• Peach Fuzzer:
• Description: A robust fuzzing framework that supports file format, network
protocol, and API fuzzing.
• Features:
• Allows users to define custom fuzzing scenarios using Peach Pit (a schema for defining
inputs and behaviors).
• Supports both black-box and white-box fuzzing.
• Suitable for enterprise-grade testing with scalability and automation features.
 Fuzzing Process
• Step-by-Step Guide:
• Define the Target:
• Select the software component to fuzz (e.g., a file parser, network protocol handler, or API).
• Identify the input format, which could be files, packets, or command-line inputs.
• Instrument the Target (for White-box Fuzzing):
• Use instrumentation tools like AFL or Peach to compile the target with feedback mechanisms that track code
coverage.
• Create a Fuzzing Campaign:
• Define the scope of the campaign: the types of inputs to generate, maximum time to run the campaign, and how
deep to explore input variations.
• Set up fuzzing environments, such as virtual machines, to prevent actual system damage in case of crashes.
• Monitor for Crashes:
• Run the fuzzing tool and monitor for any crashes, assertion failures, or abnormal outputs that may indicate a bug
or vulnerability.
• Analyse the Results:
• Review crash logs and core dumps generated during the fuzzing campaign to identify potential vulnerabilities.
• Reproduce crashes and understand the cause (e.g., buffer overflows, null pointer dereferences).
• Common Outcomes:
• Crash Dumps: A crash log is generated when a specific input causes the
program to fail. These logs are analysed to identify security flaws.
• Coverage Reports: Tools like AFL provide coverage reports indicating which
parts of the code were executed during fuzzing.

• Test Case Generation: A collection of test cases that caused the program to
behave abnormally are saved for further investigation.