RETURN ORIENTED OBFUSCATION

Vivek Balachandran1, Sabu Emmanuel2 and Ng Wee Keong3

1

Singapore Institute of Technology, Singapore

[email protected]
2

Kuwait University, Kuwait

[email protected]

Nanyang Technological University, Singapore

[email protected]

ABSTRACT
Software reverse engineering is an active threat against software programs. One of the popular
techniques used to make software reverse engineering harder is obfuscation. Among various
control flow obfuscations methods proposed in the last decade there is a lack of inter-functional
control flow obfuscation techniques. In this paper we propose an inter-functional control flow
obfuscation by manipulating return instructions. In our proposed method each function is split
into different units, with each unit ending with a return instruction. The linear order in which
functions appear in the program is obscured by shuffling these units there by creating an interfunctional control flow obfuscation. Experimental results show that the algorithm performs well
against automated reverse engineering attacks.

KEYWORDS
Software protection, Code obfuscation, Reverse engineering

1. INTRODUCTION
To develop high quality software, engineers use various software analysing tools to detect
vulnerabilities and loopholes in the program thereby facilitating them with an environment to
improve their software. However, software analysing tools are double-edged swords that can be
used to reverse engineer the software for malicious intents like intellectual property theft or
finding vulnerabilities to exploit. Tools and books on reverse engineering are readily available for
download on various Internet websites [1, 2].
A major factor that makes it harder to prevent software reverse engineering is that the attacker is a
user and has all the power of a user to control the software and its running environment. One of
the ways to provide some security to the distributed program is to incorporate a security
mechanism embedded within the program. Software obfuscation is one such effective mechanism
that hinders the process of software reverse engineering. Obfuscation is the process of translating
a software into a semantically equivalent obscure form, so that it is harder to understand the logic
of the program. Obfuscation can be applied to an entire program or partly to a section of the
program, like watermarked code [4]. Low performance overhead compared to other techniques
like encryption, is one of the desirable properties of obfuscation [5]
Software obfuscation can be applied to a program at different stages of its compilation. Source
code obfuscation refers to the application of obfuscation on the source code of the program [6-8].
Similarly binary level obfuscation refers to applying obfuscation algorithms on compiled binary
Natarajan Meghanathan et al. (Eds) : NETCOM, NCS, WiMoNe, GRAPH-HOC, SPM, CSEIT - 2016
pp. 25 36, 2016. CS & IT-CSCP 2016
DOI : 10.5121/csit.2016.61503

26

Computer Science & Information Technology (CS & IT)

programs [10]. Obfuscation can be applied on various intermediate levels such as on bytecode
representation in the case of Android applications [19] or Java programs.
Most of the modern reverse engineering tools, like IDAPro [1], are capable of constructing the
control flow graph of a program by converting a binary program to its equivalent assembly
representation.
A control flow graph shows the basic block [16] of instructions as vertices and the possible
control flow directions as edges, which enables an attacker to follow the program logic and find
possible points to attack. Thwarting the disassembly process, by not allowing the reverse
engineering tools to determine the correct program representation will result in an erroneous
assembly program generation, thereby making program analysis harder. This is the basic idea of
most of the binary obfuscation algorithms. In the past years, many binary obfuscation algorithms
have been designed to fool the reverse engineering tools. Signal based obfuscation [11], control
flow flattening [18], self-modification based obfuscation [14], double process obfuscation [12],
instruction embedding [13], are some of the binary level obfuscation algorithms.
One of the limitations of all these obfuscation techniques is that they are all trying to obfuscate
the instructions within a function. So, even though the obfuscation does a good job in obscuring
the program, the functions remain intact. A reverse engineering tool will still be able to find the
number of functions in the program and will be able to differentiate the instructions of one
function from the other.
In this paper we discuss an obfuscation technique where, we shuffle code fragments from
different functions disturbing the linear order of functions in the program. The reverse
engineering tool will identify more functions than the original program and each function will be
a small code fragment of the original function.
The paper is organized as follows. The proposed algorithm is explained in section 2. Section 3
discusses about the implementation details of the obfuscation method. In section 4, we analyses
the overhead created by our obfuscation on the program performance. Performance evaluation of
our obfuscated algorithm is discussed in section 5. The paper concludes with section 6.

2. PROPOSED METHOD
In this section, our new obfuscation method against software reverse engineering is discussed.
Our obfuscation algorithm takes an assembly program as input and split the functions in the
program and shuffles them, while maintaining the semantics of the program. The assembly
representation generated by any assembler maintains a functional structure of the program i.e., the
functions in the program are spatially arranged one after another. Each function starts with the
standard set of instructions, to set the stack, and ends with a return instruction(s). When a reverse
engineering tool disassembles a binary program to an assembly representation, it is thus capable
of identifying functions and could segregate them into different functional units. This can help the
reverse engineer, better analyse the program or creating a function call graph.
The basic idea of our technique is to disturb this normal representation of the program. In our
technique, a function is split into various code fragments by inserting return instruction at the end
of each code fragment. Each of these code fragments are then shuffled between functions, giving
a inter-functional mix as shown in Fig. 1. One of the advantages of this method is that the linear
arrangement of functions (one after another) is obscured. One of the challenges in implementing
such a technique is to maintain the semantics of the program. In a normal program, a return
instruction is used to return the control flow from a callee function to a caller function. Adding

Computer Science & Information Technology (CS & IT)

27

new return instructions could thus affect the behaviour of a function. In this section, we explain in
detail about inserting the return instructions into functions while maintaining the semantics of the
program.

Figure 1 Overview of the algorithm

2.1. Splitting the function

The first step of our algorithm is splitting the function into different segments. The input
assembly program is scanned for finding all the functions in the program. Once the functions are
identified, each function is split into different code fragments. The obfuscator has the option to
specify the number of splits in the function. In the default mode the obfuscator splits each
function into four code fragments. While splitting the function into code fragments, our
implementation put a constraint that the code fragment should contain at least five instructions.
For each function, the line numbers at which the function has to be split are identified and a
randomly generated unique label is inserted. The unique label refers to the entry point of a code
fragment. Fig. 2, shows the insertion of the labels to split the function into different segments.
In the example shown, two labels are inserted at the beginning of the code fragments.

Figure 2 Inserting labels at the splits

28

Computer Science & Information Technology (CS & IT)

2.2. Pushing the return address to the stack

We insert a ret (return) instruction at the end of each of the code fragment. This will make a
disassembler think that each code fragment is a separate function. The ret instruction takes the
return address from the stack and transfers the execution control to the particular address location.
Thus, to maintain semantics, the current return address should be saved in the stack for later use
and the address location of the next code fragment should be pushed into the stack as the new
return address.
In our obfuscation algorithm, at the end of each code fragment two assembly instructions are
added before inserting the ret instruction. One instruction stores the original return address in ebp
+ 4 to a register that has not been used. It is followed by another instruction which stores the
address of the next code fragment to the stack location ebp + 4.
In the example shown in Fig. 2, register edx is used to store the current return address in the
stack. The address location split_label_2 is then stored in the stack location ebp + 4. These two
instructions can be stored anywhere between split_label_1 and split_label_2 and not necessarily
at the end of the code fragment.

Figure 3 Inserting return address in the stack

2.3. Inserting return instruction

The next step in our technique is to insert the ret instructions in each of the code fragments. Like
a standard return instruction the stack pointer and base pointer are reset using the two
instructions, move sp, ebp and pop ebp which is then followed by the ret instruction. We add an
extra instruction to store the stack pointer value to a free register. In the example shown in Fig. 4,
the instruction is mov ecx, esp, is used to store the value of stack pointer to the register ecx.
We add an extra instruction to store the stack pointer value to a free register. In the example
shown in Fig. 4, the instruction is mov ecx, esp, is used to store the value of stack pointer to the
register ecx. The reason for this instruction is that we cannot reset the stack pointer value as the
function is not completely returning to its caller function and it is needed in the following code
fragments that will get executed.

Computer Science & Information Technology (CS & IT)

29

With the address location split_label_2 in stack and return address, the control, flows from the
first code fragment to split_label_2, the beginning of the second code fragment when the ret
instruction gets executed.

Figure 4 Inserting return instruction

2.4. Restoring the stack

During the execution, after ret instruction from one code fragment is executed; the program
execution control reaches the next code fragment. Since the stack pointer was reset during the
return instruction, the stack has to be restored at the beginning of the new code fragment.
The first address that has to be restored in the stack is the original return address, which is stored
in the register edx. By pushing the register edx, we can restore the original return address in the
stack. Instructions push ebp, and mov ebp, esp restores the ebp register. The original stack pointer
value is stored in ecx register as shown in the example in Fig. 4. Instruction mov esp, ecx, restores
the original stack pointer value.

2.5. Shuffling the code fragments

The obfuscation algorithm treats each code fragment as a separate function unit and shuffles them
randomly. The linear order of the function representation is disturbed and code fragments from
different functions will be interleaved together. This helps in inter-functional control flow
obfuscation.
Fig. 5 shows the function call graph of nqueens program generated by IDAPro, before and after
obfuscation. The obfuscation has clearly confused the IDAPro that it is unable to generate the
function calls from main after obfuscation. Fig. 6 shows the disassembled main function of the
program before and after obfuscation. It is clear from the figure that the control flow of the
function is completely obscured by the obfuscation.

30

Computer Science & Information Technology (CS & IT)

Figure 5 Function call graph of nqueens program

Figure 6 Main function of nqueens program

Computer Science & Information Technology (CS & IT)

31

3. IMPLEMENTATION
The proposed obfuscation method is implemented in python programming language. Our
implementation expects assembly level representation of the program to be obfuscated. We have
implemented the obfuscation algorithm for Microsoft Windows XP and Ubuntu Linux 13.04
operating systems. Our implementation accepts Microsoft Visual Studio 10.0 generated assembly
programs and assembly program generated by gcc 4.7.3 as input for obfuscation. Our algorithm
generates the obfuscated assembly program which is assembled using the corresponding assembly
program to generate obfuscated binary program.
The python code copies the input assembly program to a buffer. It analyses the buffer to find the
functions and the start and end of the functions. The number of code fragments for each function
is calculated according to the size of the function. The instructions to modify the stack for return
address and restoring the stack pointer are added to the beginning and end of each code
fragments. The line numbers of beginning and end of each code fragments are changed due to the
insertion of instructions. The code fragments are given numbers in sequential order and are
represented in a data structure with the number, starting line in the buffer and ending line in the
buffer. A simple shuffling algorithm is used to shuffle the code fragments as shown in the
following psuedocode. The code fragments are then stored into a new file in the shuffled order to
generate the obfuscated assembly file.
Shuffle_code_fragments (Code_Fragment_list [])
L = Length (Code_Fragment_list)
while (L > 1)
R = Random (1, L-1)
Temp = Code_Fragment_List [R]
Code_Fragment_List [R] = Code_Fragment_List [L]
Code_Fragment_List [L] = Temp
L = L -1
Return Code_Fragment_List

4. PERFORMANCE EVALUATION
In this section we perform experimental evaluation of our algorithm against reverse engineering.
We measure the efficacy of our algorithm by measuring the potency against IDAPro [1].
Instruction disassembly error which calculates the number of instructions that the reverse
engineering tool is unable to disassemble properly gives the potency of the obfuscation algorithm.
In this section, we also analyse the space and time overhead caused by the obfuscation. The
increase in space and time at different levels of obfuscation is analysed. We used the test
programs from the lcc 4.2 [17] compiler source as input test programs for our obfuscation
algorithm.

4.1. Instruction disassembly error

The potency of the obfuscation algorithm against reverse engineering tool, is measured by the
error in the disassembly of assembly instructions. IDAPro [1] was used to disassemble the
obfuscated test programs. We measured the total number of instructions in the original program
and the instructions recognized by IDAPro [1] after reverse engineering the obfuscated program.
Confusion factor is then calculated as the ratio of their differences, as defined by the following
equation,
CFinstr = |Ttotal Tdisasm| / Ttotal

32

Computer Science & Information Technology (CS & IT)

The total number of instruction addresses before obfuscation is represented by Ttotal. The number
of instruction addresses recognized by IDAPro [1] after disassembling the obfuscated binary
program is represented by Tdisasm.
Table 1 shows the confusion factor while disassembling obfuscated test pro-grams by IDAPro
[1]. The table shows varies levels of splitting the program. The first column represented by zero
splits is the original program and all the instructions are reverse engineered successfully. Column
2 represents the obfuscated program, where every function is split into two and the mean
disassembly error is 55.9% when functions are split into two. The instruction disassembly error
increases as the splitting of the program increases. The splitting of a program saturates after a
while. For instance, the program fields has the same instruction disassembly error for 16 splits
and 32 splits. This is because the program is split to the maximum possible split by 16 splits and
the program cannot be further split down.
Mean instruction disassembly error of 85.16% is obtained at level 8, where each function is split
into 8 code fragments.
Table 1. Instruction Disassembly Error
Splits

16

32

64

128

283

125

65

42

25

25

25

25

341

150

78

51

31

20

20

20

184

81

42

28

17

11

11

11

13786

6066

2068

2068

2068

2068

2068

2068

674

297

155

101

61

40

27

14

339

149

78

51

20

20

20

20

392

172

90

24

24

24

24

24

415

183

95

62

37

17

17

17

162

71

37

24

12

12

12

12

506

223

116

76

46

46

46

46

433

191

100

65

39

26

26

26

505

222

116

76

45

36

30

30

597

263

137

90

54

24

24

24

55.9%

82.9%

85.1%

86.6%

87.2%

87.3%

87.4%

Prog
8q
array
cf
cq
cvt
fields
incr
init
limits
sort
spill
struct
wf1
CFinstr

4.2. Space overhead

The insertions of the new instructions have significant effect on the size of the program. If a
function is split into 2 code fragments, then 10 new instructions are added into the program and
20 instructions are added if the function is split into 3 code fragments and so on. Let the number
of instructions in the program be Nbefore and the original program is split into n+1 code fragments.
The total number of instructions in the obfuscated program will be,
Nafter = Nbefore + 10n

Computer Science & Information Technology (CS & IT)

33

In the worst case, the entire program is split into code fragments with 5 instructions. Let the
program is split into n+1 code fragments, with each code fragment having four instructions, then
the total number of instructions in the program before obfuscation is,
Nbefore = 5(n + 1)
After the obfuscation, 10 instructions are added per code fragment and the total number of
instructions in the program after obfuscation is,
Nafter = 5(n + 1)+ 10n
Nafter = 3Nbefore - 10
So, in the worst case there are three times more instructions in the obfuscated program than the
original program. We can see in the experimental evaluation that this upper bound is held.
Spaceovh defines the increase in the size of the program.
Spaceovh = Spaceafter / Spacebefore
In Table 2, we show how the program size increases as the program is obfuscated. The size of the
program increases as the number of splits increase. In the worst case, the size increases to 2.2
times the original size. But on an average, the program size increases by 1.57 times the original
size with 128 splits, which is less than the theoretical upper bound.
Table 2 Space Overhead
0

16

32

64

128

8q

8743

8779

10256

13586

17843

17843

17843

17843

array

8678

8714

11324

14321

18545

18923

18923

18923

cf

8750

8786

9957

12985

13876

14178

14178

14178

cq

63599

75898

85699

85699

85699

85699

85699

85699

cvt

12983

13019

13091

13235

13523

14099

15251

21651

fields

8728

8764

12633

13589

13589

13589

13589

13589

incr

8594

8630

9987

12371

12371

12371

12371

12371

init

9134

9170

9242

9386

9674

9782

9782

9782

limits

8522

8558

8630

8774

12062

13458

13458

13458

sort

8856

8882

11954

12098

14350

14350

14350

14350

spill

8863

8899

10971

12115

16403

18799

18799

18799

struct

8843

8879

8951

9095

9383

14055

17127

17127

Wf1

13179

13215

13287

17521

18754

21875

21875

21875

177462

190193

215982

234775

256072

269021

273245

279645

1.07

1.21

1.32

1.44

1.51

1.53

1.57

Splits
Prog

Mean
Spaceovh

34

Computer Science & Information Technology (CS & IT)

4.2. Time overhead

Obfuscation does have an effect on the execution time. The execution time of the program will
increase, because of the execution of the additional instructions. We know that the size of the
program increases by 3 times in the worst case. The input parameter of the time complexity thus
increases by 3 times. If the original time complexity was T(n), then the new time complexity will
be 3T(n) in the worst case. The time complexity of the obfuscated program is thus between T(n)
and 3T(n).
Timebefore refers to the time taken by the program to execute without obfuscation and Timeafter is
the time taken by the obfuscated program to execute. We evaluate the effect of obfuscation on
execution speed with Timeovh defined as,
Timeovh = Timeafter / Timebefore
Table 3, shows the time overhead caused by our obfuscation on various binary programs. In the
worst case the time overhead is 2.36 times in the case of cvt with 128 levels of obfuscation. On an
average the worst case time overhead is 1.86 which is lower than the upper bound of 3T(N).
Table 3 Time Overhead
Splits

16

32

64

128

1300

1395

1534

1975

2589

2589

2589

2589

2777

3276

3588

4215

5014

5581

5581

5581

1077

1121

1154

1178

1223

1498

1892

2548

1089

1258

1685

1987

2406

2406

2406

2406

1271

1301

1354

1537

1537

1537

1537

1537

1174

1256

1325

1456

1658

2219

2219

2219

1342

1398

1469

1566

1689

1726

1754

1754

1245

1322

1391

1499

1785

1997

2015

2015

1011

1250

1347

1678

2001

2022

2500

2694

1025

1119

1243

1567

1874

2198

2198

2198

1198

1245

1376

1461

1653

1985

1985

1985

1037

1134

1256

1370

1523

1523

1523

1523

15546

17075

18722

21489

24952

27281

28199

29049

1.09

1.20

1.38

1.60

1.75

1.84

1.86

Prog
8q
cf
cvt
fields
incr
spill
struct
wf1
array
cq
init
sort
Mean
Spaceovh

5. CONCLUSIONS
In this paper we proposed an obfuscation algorithm to perform inter functional obfuscation. Our
method slices each function in the program into separate code fragments. Each fragment ends
with a return instruction and starts with stack allocation instructions, thereby appearing itself like
a function. The return instruction transfers the control flow to the next code fragment instead of
returning to a caller function. The code fragments are shuffled disturbing the linear order of the

Computer Science & Information Technology (CS & IT)

35

functions. Unlike other control flow obfuscation, our method adds more control flow instructions
(return instruction) to increase the control flow obscurity instead of removing the control flow
instructions. The experimental results show that obfuscating with 8 splits provides a good
obfuscation without too much overhead on the space and time requirements of the program.
Experimental analysis shows that that our method has an instruction disassembly error of 85.1 %
with 8 levels of splitting. An average time overhead of 1.38 and space overhead of 1.32 are
observed while obfuscating with 8 splits.

REFERENCES
[1]

"Data Rescue," Available at: http://www.hex-rays.com/ [Last accessed: October 14, 2016]

[2]

J. Miecznikowski and L. Hendren, "Decompiling java using staged encapsulation," in Reverse

Engineering, 2001, pp. 368-374.

[3]

W. Thompson, A. Yasinsac, and J. McDonald, "Semantic encryption transformation scheme," in

International Workshop on Security in Parallel and Distributed Systems, San Francisco, CA, 2004.

[4]

J. Hamilton and S. Danicic, A survey of static software watermarking, in World Congress on Internet
Security, pp.100-107, 2011.Lecture Notes in Computer Science: Authors' Instructions 13
B. Anckaert, M. Madou, B. De Sutter, B. De Bus, K. De Bosschere, and B. Preneel. Program
obfuscation: a quantitative approach, in ACM Workshop on Quality of Protection, ACM New York,
NY, USA, 2007, pp. 15-20.

[5]

[6]

W. F. Zhu, "Concepts and techniques in software watermarking and obfuscation," Ph.D. Thesis,
University of Auckland, 2007.

[7]

M. Sosonkin, G. Naumovich, and N. Memon, "Obfuscation of design intent in object oriented

applications," in ACM workshop on Digital Rights Management, 2003, pp.142-153.

[8]

C. Collberg and J. Nagra. "Surreptitious Software: Obfuscation, Watermarking, and Tamperproofing

for Software Protection (1st ed.)," Addison-Wesley Professional, 2009.

[9]

C. Collberg, C. Thomborson, and D. Low, "Manufacturing cheap, resilient, and stealthy opaque
constructs," in ACM Symposium on Principles of Programming Languages, 1998, vol. 25, pp. 184196.

[10] V. Balachandran and S. Emmanuel,Software Protection with Obfuscation and Encryption,

Information Security Practices and Experiences Conference, ISPEC, volume 7863 of Lecture Notes in
Computer Science, pp 309-320. Springer, 2013.
[11] I. V. Popov, S. K. Debray, and G. R. Andrews, "Binary obfuscation using signals," in USENIX
Security Symposium, 2007, pp. 1-16.
[12] J. Ge and S. Chaudhari, "Control flow based obfuscation," in ACM Digital Rights Management,
2005.
[13] C. LeDoux, M. Sharkey, B. Primeaux, and C. Miles "Instruction Embedding for Improved
Obfuscation," in Proceedings of the 50th Annual Southeast Regional Conference (ACM-SE
'12),pp.130-135, 2012.
[14] V. Balachandran and S. Emmanuel, "Software code obfuscation by hiding control flow information in
stack," in IEEE Workshop on Information Forensics and Security, 2011.

36

Computer Science & Information Technology (CS & IT)

[15] V. Balachandran and S. Emmanuel,"Potent and Stealthy Control Flow Obfuscation by Stack Based
Self-Modifying Code," in IEEE Transactions on Information Forensics and Security, vol.8, no.4,
pp.669-681, April 2013.
[16] "Basic Block," Available at: http://gcc.gnu.org/onlinedocs/gccint/Basic-Blocks.html [Last accessed:
October 14, 2016]
[17] C. W. Fraser, "lcc, A Retargetable Compiler for ANSI C," Available at:
https://sites.google.com/site/lccretargetablecompiler/downloads [Last accessed: October 14, 2016]
[18] L. Shan and S. Emmanuel, "Mobile agent protection with self-modifying code," in Journal of Signal
Processing Systems, vol. 65, pp. 105-116, 2010.
[19] V. Balachandran, Sufatrio, D.J.J. Tan, and V.L.L. Thing. "Control Flow Obfuscation for Android
Applications." in Computers and Security, vol.61,pp.72-93, Aug 2016.

AUTHORS
Dr. Vivek Balachandran holds a PhD in Computer Engineering from the Nanyang
Technological University, Singapore, where he worked with the Center for Strategic
Infocomm Technologies and Temasek Laboratories. After graduating he worked as a
Research Scientist with Institute for Infocomm Research, A*STAR, Singapore, where
he worked on mobile security and forensics. He has published numerous articles on
software security. He is currently a Lecturer at the Singapore Institute of Technology,
Singapore. His research interests includes software security, mobile forensics, program
analysis, and compiler optimization.
Dr. Wee Keong Ng is Associate Chair (Research) of the School of Computer Science
& Engineering, Nanyang Technological University, Singapore. He received his Ph.D.
from the University of Michigan at Ann Arbor, USA. His research areas are secure
data analytics, secure data storage, data monetization, and data security, and has
published more than 200 technical papers in these areas. Dr. Ng has served in
program/organizing committees of international conferences. In recent years, he is
General Co-chair of the International Conference on Information and Communications
Security 2016; Jury Member of the Second Dutch Cyber Security Research Award in
March 2016; Senior PC Member of the 20th, 19th, 18th, 17th Pacific-Asia Conference on Knowledge
Discovery and Data Mining. Dr. Ng has advised and graduated more than 20 Ph.D. students and 20 Master
students
Dr. Sabu Emmanuel received the B.E. degree in electronics and communication
engineering from Regional Engineering College, Durgapur, India, in 1988, the M.E.
degree in electrical communication engineering from the Indian Institute of Science,
Bangalore, in 1998, and the Ph.D. degree in computer science from the National
University of Singapore in 2002. He is currently an associate professor with Kuwait
University. Previously, he was an assistant professor in the School of Computer
Engineering, Nanyang Technological University (NTU), Singapore. He began his
career as a research and development engineer and then moved on to the academic
profession. He has taught engineering students of Mangalore University, National University of Singapore,
and NTU. His current research interests are in multimedia and software security and surveillance media
processing. He has been a reviewer for ACM Multimedia Systems. He is a member of the Technical
Program Committee of several conferences. Dr. Emmanuel has been a reviewer for IEEE Transactions on
Circuits and Systems for Video Technology, IEEE Transactions on Multimedia, and IEEE Transactions on
Information Forensics and Security.