Visual Forensic Analysis and

Reverse Engineering of
Binary Data

Gregory Conti
Erik Dean

United States Military Academy

West Point, New York
[email protected]
[email protected]
 Outline

• The Problem – Tiny

y Windows
• Background and Motivation
• Related Work
• Moving Beyond Hex
• System Design
• Case Studies
• Demos
 data operated on by
applications
doc
xls
txt…
file
exe executed by OS
ELF
01010 PE...
10101 other special cases
01010
core dump
pagefile.sys
hiberfil sys
hiberfil.sys…

memory process memory

cache
cache…
network packets…
 Ida Pro
OllyDBG
high BinNavi (Zynamics)
insight BinDiff (Zynamics)…
(Zynamics)

Filemon
Regmon…
011

objdump
lower hex editors
insight h d
hexdump original
grep & diff application
g
strings

general purpose precise application

 strings /grep/diff
H:\Datasets>strings 20040517_homeISP.pcap | more

Strings v2.4
Copyright (C) 1999-2007 Mark Russinovich
Sysinternals - www.sysinternals.com

0hF
M@y
7bs
Z19Z
MICROSOFT NETWORKS
WINDOWS USER
Microsoft Security Bulletin MS03-043
Buffer Overrun in Messenger Service Could Allow Code Execution
(828035)
Affected Software:
Microsoft Windows NT Workstation
Microsoft Windows NT Server 4.0
Microsoft Windows 2000
...
011 Hex Editor
Hex Workshop
WinHex

http://www.x-ways.net/pics/winhex.gif
 Ida Pro
OllyDBG
high BinNavi (Zynamics)
insight BinDiff (Zynamics)…
(Zynamics)

Filemon
Regmon…
011

objdump
lower hex editors
insight h d
hexdump original
grep & diff application
g
strings

general purpose precise application

 Ida Pro
OllyDBG
high BinNavi (Zynamics)
insight BinDiff (Zynamics)…
(Zynamics)

Filemon
Regmon…
011

objdump
lower hex editors
insight h d
hexdump original
grep & diff application
g
strings

general purpose precise application

 SysInternals

FileMon

RegMon
g

Process Monitor
...
http://technet.microsoft.com/en-us/sysinternals/default.aspx
Wireshark

image: http://code.google.com/support/bin/answer.py?answer=71567
OllyDbg

http://www.ollydbg.de/
IDA Pro
v5.1
5

http://www.hex-rays.com/idapro/
 F-Secure Malware

http://www.f-secure.com/weblog/archives/00000662.html
Zynamics BinDiff

http://www.zynamics.com/content/_images/bindiff_scr2.gif
Zynamics BinNavi

http://www.zynamics.com/index.php?page=binnavi
 Ida Pro
OllyDBG
high BinNavi (Zynamics)
insight BinDiff (Zynamics)…
(Zynamics)

Filemon
Regmon…
011

objdump
lower hex editors
insight h d
hexdump original
grep & diff application
g
strings

general purpose precise application

 Framework

• File Independent Level

– Entropy
– Byte Frequency
– N-Gram
N Gram Analysis
– Strings
– Hex / Decimal / ASCII
– Bit Plot (2D/3D)
– File Statistics
• File Specific Level
– Complete or Partial Knowledge of File
Structure
– For Example, Metadata
Syntax Highlighting for Hex Dumps
(Kaminsky)

image: Dan Kaminsky, CCC2006

 nwdiff

http://computer.forensikblog.de/en/2006/02/compare_binary_files_with_nwdiff.html
http://www.geocities.jp/belden_dr/ToolNwdiff_Eng.html
 Dot Plots & Visual BinDiff
(Kaminsky)

Self-Similarity in Diffing Two Files

a single file. (.NET Assembly) images: Dan Kaminsky, CCC2006
 Traditional
Textual
Textual Graphical
Hex/ASCII
Utilities Displays
Detail View
(strings...)

Machine Assisted Mapping and Navigation

Hex Editor Core

 Towards a Visual Hex Editor
• Identify Unknown Binaries
• Malware Analysis
y
• Analyze Unknown/Undocumented File Format
• Locate Embedded Objects
– Encoding
E di /E
Encryption
ti
• Audit Files for Vulnerabilities
• Compare files (Diffing)
• Cracking
• Cryptanalysis
• Perform Forensic Analysis
• File System Analysis
• Reporting
• File Fuzzing
 Goals
• Handle Large Files
• Many Insightful Windows
• Big Picture Context
• Improved Navigation
• Data Files / Executable Files
• Hex Editor best practices is the
foundation
• Support Art & Science
 Design

• Robust extensible framework

• Open source
• Context Independent File Analysis
• Semantic File Analysis
• Useful
• Multiple
p coordinated views
• Combine Functionality of current
tools and extend with visuals
 Filtering + Encoding
• Identifying something
– REGEX Æ algorithmic
• Usingg this knowledge
g
to..
– highlight
g g
– fade
– remove
• Interactive or automated
• Textual: Text/ASCII,
Text/ASCII Strings,
Strings ByteCloud

• Graphical: Bitplot, BytePlot, RGBPlot, BytePresence, ByteFrequency,

Digram, Dotplot

• Interaction: VCR, Memory Map, Color Coding

 Traditional Views

Hex / ASCII View Strings

 Strange Attractors and TCP/IP
Sequence Number Analysis
(Michal Zalewski)

• http://lcamtuf.coredump.cx/oldtcp/tcpseq.html
• http://lcamtuf.coredump.cx/newtcp/
 Digraph View
black hat
bl (98,108)
la (108,97)
ac (97,99)
ck
k (99 107)
(99,107)
k_ (107,32)
_h (32,104)
ha (104 97)
(104,97)
at (97,116)
 Digraph View
0,1, ... 255

Byte 0
Byte 1

32,108

98,108
...

Byte 255
uuencoded
d d compression incrementing
encryption words

slashdot.org .txt constrained pairs

 Bit Plot
1 640

1
1
0
1
...

480
 Byte Plot
1 640

255
108
0
40
...

480
Byte Plot Example
(Word Document)
Byte Presence
255 RGB Plot
108
0
1 640
40 1
128
255

0
0
0

200
0
0 480
 Dot Plots

• Jonathan
Helfman’s
“Dotplot
p
Patterns: A
Literal Look at
Pattern
Languages.”
g g
• Dan Kaminsky,
CCC & BH 2006
 DotPlots
Byte 0, Byte 1, ... Byte N

Byte 0
Byte 1

O(N2)
...

Byte
y N
 Dynamic DotPlots
Byte 0, Byte 1, ... Byte N

Byte 0
Byte 1
500x500

O(N)
...

Byte
y N
 DotPlot Examples

Images: Jonathan Helfman, “Dotplot Patterns: A Literal Look at Pattern Languages.”

 DotPlot Examples

Images: Jonathan Helfman, “Dotplot Patterns: A Literal Look at Pattern Languages.”

 English Text Compressed Audio

Bitmap Image
 Byte Clouds

Tag Cloud
Smashing the Stack
for Fun and Profit
http://tagcrowd.com/

Byte Cloud
Neverwinter Nights Database File
Firefox .hdmp
Firefox .hdmp
Firefox .hdmp
Firefox .hdmp
Redacted
PDF...
 Weaknesses

• entire file may

y be extracted from
bit/byte/RGB
– May trigger AV or IDS
– 8bit/byte steg
• Screams for big monitor
Demos
 A Look to the Future…

• Visual Front Ends for Offensive Tools

• Visual Cryptanalysis Support
• Human Insights Passed to Machine Processors
• User centric Evaluation
User-centric
• More Inspiration from General InfoVis Community
• Visual Fingerprints
g p / Smart Books
• Web-based Visualization (AJAX)
• User-task Analyses
– True
T e Use Case Based Designs
– Engagement of Users Beyond Students
• Examination of Full Range of Security Data
– Merging Multiple Security Dataflows
 Future Work

• Plug-ins / Editable Config Files

– Visualizations
– Encodings
• Saving state
– Memory Maps
• Improving Interaction
– What works / What doesn’t
• Multiple Files / File Systems
• REGEX search
• Automated Memory Map Generation
 DAVIX
((Jan
a Monsch
o sc and
a d Raffy
a y Marty)
a y)

http://www.secviz.org/node/89
InfoVis Survey
Security Visualization Survey
 Communities
http://secviz.org/ http://vizsec.org/

“The place to share, discuss, “vizSEC is a research community for

challenge, and learn about security computer security visualization.”
visualization.”

Raffy Marty John Goodall

Splunk Secure Decisions
VizSEC 2008

http://www.vizsec.org/workshop2008/
 More Information
• “Visual Reverse
Engineering
g g of Binary
y
and Data Files.” Gregory
Conti, Erik Dean,
Matthew Sinda, Benjamin
Sangster. VizSEC 2008.
– Available
A ailable September
Septembe
• Security Data
Visualization
(No Starch Press)
• Applied Security
Visualization
(Addison-Wesley)
 Acknowledgements

Damon Becknell,, Jon Bentley, y, Jean

Blair, Sergey Bratus, Chris
Compton,
p , Tom Cross,, Ron Dodge,g ,
Carrie Gates, Chris Gates, Joe
Grand,, Julian Grizzard,, Toby
y
Kohlenberg, Oleg Kolesnikov,
Frank Mabry, y, Raffy
y Marty,
y, Brent
Nolan, Gene Ressler, Ben
Sangster,
g , Matt Sinda,, and Ed
Sobiesk
"In fact,, master reversers
like Fravia recommend
cracking while intoxicated
with a mixture of strong
alcoholic beverages.

While for health reasons

we cannot recommend
this method, you may find
that a relaxing cup of hot
tea unwinds your mind
and allows you to think in
reverse."

-from Security Warrior