Introduction to

Reverse Engineering

Inbar Raz
Malware Research Lab Manager
December 2011
©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved.
What is Reverse Engineering?

Reverse engineering is the process of

discovering the technological principles of a
device, object, or system through analysis of
its structure, function, and operation.

aka: Reversing, RE, SRE

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 2
2
Why do it?

Find Academic
Discover Vulnerabilities Research
Trade (Yeah, right…)
Secrets

Circumvent
[Copy]
Protection

Analyse
Patch Binary Pure Protocols
and Curiosity
Alter Behavior

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 3
3
 Sounds
awesome,
right?

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 4
4
So where’s the catch?

 Low-level is, well, low level…

00401000 push ebp
00401001 mov ebp, esp
00401003 push ecx
00401004 push ecx
00401005 and dword ptr [ebp-4], 0
00401009 push esi
0040100A mov esi, [ebp+8]
for (Serial = 0, i = 0; i < strlen(UserName);
0040100D i++) push { edi
CurChar = (int) UserName[i]; 0040100E push esi
Serial += CurChar; 0040100F call ds:[00402008h]
Serial = (((Serial << 1) && 0xFFFFFFFE)
00401015|| mov ((Serialedi,
>> 31)
eax && 1));
Serial = (((Serial * CurChar) + CurChar)
00401017 xor ^ CurChar);edx, edx
} 00401019 test edi, edi
UserSerial = ~((UserSerial ^ 0x1337C0DE) -
0040101B jle0xBADC0DE5); 00401047h
0040101D movsx ecx, byte ptr [edx+esi]
00401021 add [ebp-4], ecx
00401024 mov [ebp-8], ecx
00401027 rol dword ptr [ebp-4], 1
0040102A mov eax, ecx
0040102C imul eax, [ebp-4]
00401030 mov [ebp-4], eax
00401033 mov eax, [ebp-8]
00401036 add [ebp-4], eax
00401039 xor [ebp-4], ecx
0040103C inc edx
0040103D cmp edx, edi
©2011 Check Point0040103F jl Ltd. [PROTECTED]
Software Technologies 0040101Dh — All rights reserved. 5
5
So where’s the catch?

 Low-level is, well, low level…

 Needle in a haystack
– Average opcode size:
3 bytes
– Average executable size:
500KB (on WinXP)
– There are executables,
libraries, drivers….

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 6
6
So where’s the catch?

 Low-level is, well, low level…

 Needle in a haystack
 Sometimes, the code resists
– Packers and compressors
– Obfuscators

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 7
7
So where’s the catch?

 Low-level is, well, low level…

 Needle in a haystack
 Sometimes, the code resists
 Sometimes, the code fights back
– Detect reversing tools
– Detect VMs and emulators

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 8
8
A Battle of Wits

 Video clip: The Battle of Wits, “The Princess Bride”

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 9
9
A Battle of Wits

 Author writes code

 Reverser reverses it
 Author creates an anti-reversing technique
 Reverser bypasses it
 And so on…

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 10
10
So what do you need
in order to be
a good reverser?

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 11
11
We’ll come back
to this…

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 12
12
Tools of the Trade

 Debugger (Dynamic code analysis)

 Disassembler (Static code analysis)
 Hex Editor
 PE Analyzer
 Resource Editor
and more…

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 13
13
 Debuggers
‫באג בדיזיין – זין בדיבאג‬

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 14
14
First, there was DEBUG…

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 15
15
GUI and much more: Turbo Debugger

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 16
16
GUI and much more: Turbo Debugger

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 17
17
GUI and much more: Turbo Debugger

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 18
18
Next major step: Soft-ICE

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 19
19
And finally: OllyDbg

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 20
20
Disassemblers

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 21
21
The old world: Sourcer

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 22
22
The old world: Sourcer

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 23
23
Old ages: Sourcer

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 24
24
Old ages: Sourcer

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 25
25
Welcome to Windows: W32DASM

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 26
26
The Holy Grail: IDA-Pro

 Started as an Interactive Dis-Assembler, enabling user

interaction with the disassembler’s decisions.
 Slowly evolved into an automatic RE tool:
– Built-in full-control script language
– Library recognition (including user-generated)
– Function prototype information
– Display
– Propagate throughout the code
– Support for plug-ins
– Support for Python scripting
– Multi-architecture, cross-platform support
– Full incorporation with built-in and external debuggers

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 27
27
Hex-Editor

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 28
28
PE Analyzer

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 29
29
Resource Editor

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 30
30
 Let’s play
with them tools…

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 31
31
60 seconds on x86 registers

 General purpose registers:

32bit/16bit/8bit
 Index registers:
32bit/16bit
 Segment registers:
16bit
 Flags:
32bit/16bit

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 32
32
 Exercise 1:
Static Reversing

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 33
33
Exercise 1: Static Reversing

 Target: a 2004 “Crack-Me”

 Tools: IDA-Pro

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 34
34
 Exercise 2:
Dynamic Reversing

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 35
35
Exercise 2: Dynamic Reversing

 Target: a 2004 “Crack-Me”

 Tools: OllyDbg, IDA-Pro

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 36
36
 Exercise 3:
Simple Anti-Debugging

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 37
37
Exercise 3: Simple Anti Debugging

 Target: a 2006 “Crack-Me”

 Tools: OllyDbg

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 38
38
Reversing Malware

 Malware is comprised of the following building blocks:

– Infection Vector
– Concealment
– Operation
– Communications

 Check Point’s Anti-Malware Software Blade

sits at the gateway
 Therefore, communications interest us the most

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 39
39
Introducing: Spy Eye

 A CrimeWare ToolKit, originating in Russia.

 Used mostly for stealing financial information, but will settle
for any other identity information and key logging…
 Like any serious trojan, Spy Eye compresses its traffic and
encrypts it
– Compression is performed using a public library (LZO)
– Encryption algorithm is proprietary

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 40
40
 Act 1:
Encryption

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 41
41
 Act 2:
Configuration Download

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 42
42
 Act 3:
Another Encryption

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 43
43
So what do you need
in order to be
a good reverser?

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 44
44
What makes a good reverser?

Qualities Knowledge
• Patient • Assembly Language
• Curious • Some High-Level programming
• Best: origin of binary
• Persistent
• Outside-the-Box Thinking • Operating System Internals
• API
• Data Structures
• File Structures
• Good scripting skills
• Anti-Debugging Tricks
• Optional: Good lookin’

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 45
45
Outside-the-Box Thinking

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 46
46
And remember, kids:

Binary Reverse Engineer

+ =?

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 47
47
Which means…

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 48
48
Questions?

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 49
49
 Thank you!

[email protected]

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 50
50
Credits

All images and videos have their origin URL in the “Alt Text” property.
All rights belong to their respective owner.

©2011 Check Point Software Technologies Ltd. [PROTECTED] — All rights reserved. 51
51