Leveraging Recursive File Scanning

Frameworks to Amplify Reverse

Engineering Results
Joshua Acklin
Jason Batchelor

Software Engineering Institute

Carnegie Mellon University
Pittsburgh, PA 15213
Leveraging Recursive File Scanning Frameworks to
Amplify Reverse © Engineering
2016 Carnegie Results Mellon University
Date October 5th, 1
2016 Statement A] This material has been approved for public release and unlimited distribution.
[Distribution
© 2016 Carnegie Mellon University
[Distribution Statement A]REV-03.18.2016.0
This material has been approved for public release and unlimited distribution.
Distribution Statement
Copyright 2016 Carnegie Mellon University
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-
0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research
and development center.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the United States Department of Defense.
References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or
otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon
University or its Software Engineering Institute.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS
FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND,
EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF
FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE
MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO
FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see
Copyright notice for non-US Government use and distribution.
This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form
without requesting formal permission. Permission is required for any other use. Requests for permission should be directed
to the Software Engineering Institute at [email protected].
CERT® is a registered mark of Carnegie Mellon University.
DM-0003953

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
2
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Outline

Philosophy
• Motivation
• Paradigm Shift
• Implementation

Case Study
• Unrealized Analysis
• Capability
• Empowerment

Conclusion
• Takeaways
• Questions
• Resources

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
3
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Philosophy

Motivation Implementation

Paradigm Shift

Leveraging Recursive File Scanning Frameworks to

© 2016 Results
Amplify Reverse Engineering Carnegie Mellon University
Date October 5th, 2016 4
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
© 2016 Carnegie Mellon University
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Motivation

• Many organizations have become far too tool centric

• Personnel more reliant on vendor capability
• Less reliant on enhancing analytical tradecraft
• Who has more visibility?
- You vs MSSP?
• Fail to recognize the opportunity cost of automation for what it is
• Sometimes you need to struggle to learn
• Capture and codify tribal knowledge
• In many cases, capabilities of the tools exceed technical
capacity of the teams they support
• Can’t use the whole buffalo!
• Rebalance the landscape

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
5
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Paradigm Shift

Capabilities driving analysts

• Receive alert > block > reimage > update ticket
- Is my job done? No!
- Where is the analysis?
• It's okay to say I don't know. We should embrace absence of knowledge as
an opportunity to learn.
• It's not okay to substitute absence of knowledge with fear, hyperbole, false
claims, or an excuse not to learn.
Analysts driving capabilities
• Taking ownership for what they can see and cannot
• Strategic direction by taking a more analytically principled approach
• Performing root cause analysis
- Is it more expensive? Yes
- Is it worth it in the long run? Yes
• Innovate past limitations
- Empowerment
- Ownership

Source: Bro at Emerson Electric: https://www.bro.org/brocon2015/slides/batchelor_emerson.pdf

Leveraging Recursive File Scanning Frameworks to
Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
6
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Implementation

Principles are cornerstones upon which

several capabilities have been built
• Bro
• Yara
• Pattern Matching Swiss Army Knife
• Suricata

Case Study: File Scanning Framework

• Analysis driven framework
• Opportunities for each person in Incident
Response Chain
• Leverages open source solutions
Leveraging Recursive File Scanning Frameworks to
Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
7
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Case Study

Unrealized Analysis Design

Capability Empowerment

Leveraging Recursive File Scanning Frameworks to

© 2016 Results
Amplify Reverse Engineering Carnegie Mellon University
Date October 5th, 2016 8
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
© 2016 Carnegie Mellon University
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Unrealized Analysis

Common Deliverables for RE and MW Analysis

• Custom malware signatures (YARA)
• Configuration dumpers
• IDS signatures
• Capability enumeration
• Attack sequence

Work product is expensive!

• A few days to month or more depending on sophistication

Usefulness of results depends on a variety of things

• How well instrumented is the network?
• How proficient is your customer?
• How much of a moving target is the malware?
Leveraging Recursive File Scanning Frameworks to
Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
9
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Unrealized Analysis

The journey…
• I can write signatures on anything using YARA…
• I can dictate execution flow using my signatures…

What if…
• I did this recursively?
• We alert on certain YARA signatures that fire?
• I exposed useful metadata for every object I scanned?
• Results were structured data representing the object hierarchy?
• I could make all my generated data searchable?
• I could inform detections using the exposed metadata and
relationships?
Leveraging Recursive File Scanning Frameworks to
Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
10
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Leveraging Recursive File Scanning Frameworks to
Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
11
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Capability

File Scanning Framework

• Take custom YARA signatures you write
• Run modules returning metadata and/or new sub object
• Process new sub objects recursively
• Data returned as a nested JSON tree
• Represents the structure of file being scanned
• Enables detections on…
• YARA signatures that may have hit
• Structure of the file itself!

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
12
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Design

Languages
• Python for core framework and modules
• YARA for signatures identifying opportunities
• JQ to capture observations
Paradigm
• Client / Server
• Dual purpose as an analyst tool and perimeter defense tool
• Integration with Bro / Suricata for file extraction

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
13
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Design

Source: FSF GitHub: https://github.com/EmersonElectricCo/fsf

Leveraging Recursive File Scanning Frameworks to
Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
14
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Empowerment

Visual of zip file containing UPX packed malware

• Identify ZIP
ZIP •
•
Extract ZIP contents
Enumerate metadata

• Identify EXE
• Enumerate metadata
UPX • Identify UPX
• Unpack UPX

• Identify EXE
• Enumerate metadata
Raw EXE • Identify exposed malware
• Assign alert flag

YARA allows us to capture observations. Modules allow us to do something with them...

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
15
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Empowerment

Extend utility and agility of…

• Yara signatures
• Configuration dumpers
• Threat intelligence
Let’s consume publicly released NetTraveler Intel
• Source: NetTraveler Spear Phishing Targets Diplomat of Uzbekistan
Intel Module

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
16
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Empowerment

SFX RAR containing dropper we can identify and dump config

• Identify EXE
SFX •
•
Enumerate metadata
Extract embedded RAR

• Identify RAR
RAR •
•
Enumerate metadata
Decompress embedded files

• Identify EXE
• Enumerate metadata
Legit EXE
• Identify DLL
• Enumerate metadata
Bootstrap DLL • Identify NT Dropper
• Decode embedded config

Encoded Payload
• Enumerate metadata

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
17
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Empowerment

Use modules to return sub objects for static malware analysis…

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
18
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Empowerment
‘ELK Stack’ Integration
Expose new opportunities to
derive actionable intelligence…
• Have we seen this hash,
domain, etc…?
• What other data points might
be of interest?
• How well did we do here?
• Decoded C2
• Did we decode the RAT?
• Sandbox integration?
• Was the team notified?

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
19
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Empowerment

Using a sample set of FSF output

• Capture observations and possibly detect on them
• We want to make observations/detections on relationships
between objects, sub objects, and metadata

Example use cases:

• A Windows executable within the dataset was compiled less
than 24 hours ago
• An object, or sub object, had more than twenty AV hits and our
example AV solution was not one of them!

JQ integration allows us to capture and detect on additional insights once a scan is complete…

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
20
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
A Windows executable within the dataset was
compiled less than 24 hours ago…

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
21
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
An object, or sub object, had more than twenty AV hits and
our example AV solution was not one of them!

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
22
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Conclusion

Takeaways Questions

Resources

Leveraging Recursive File Scanning Frameworks to

© 2016 Results
Amplify Reverse Engineering Carnegie Mellon University
Date October 5th, 2016 23
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
© 2016 Carnegie Mellon University
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Takeaways

Rethink how security personnel are File Scanning Framework

enabled
• Is an example of an analysis
• Foster a sense of ownership driven framework
• Actualize • Not end all be all
• New opportunities to defend
• Increase utility of malware
the enterprise
reverse engineering
• The potential within
observations
themselves
• Start with Root Cause Analysis
• Empowers all members within
the chain of response
• The rest will take care of itself
• How does a product make my
• Cultivates a sense ownership
team better? and value
• More important than any
security feature it provides
Leveraging Recursive File Scanning Frameworks to
Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
24
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Takeaways

It's up to us…

• Ask questions that drive the right behaviors

- Of ourselves
- Of our leadership
- Of our subordinates

• It's up to us to make the adversary pay for every byte!

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
25
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Resources

File Scanning Framework

• https://github.com/EmersonElectricCo/fsf

Yara
• http://yara.readthedocs.io/en/v3.5.0/

JQ
• https://stedolan.github.io/jq/

Bro
• https://www.bro.org/

Suricata
• https://suricata-ids.org/

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
26
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Questions?

Thanks for your kind attention!

Leveraging Recursive File Scanning Frameworks to

Amplify Reverse Engineering Results
Date October 5th, 2016
© 2016 Carnegie Mellon University
27
[Distribution Statement A] This material has been approved for public release and unlimited distribution.