Semantics-Aware Malware Detection

Mihai Christodorescu∗ Somesh Jha∗ Sanjit A. Seshia† Dawn Song Randal E. Bryant†
University of Wisconsin, Madison Carnegie Mellon University
{mihai, jha}@cs.wisc.edu {sanjit@cs., dawnsong@, bryant@cs.}cmu.edu

Abstract The goal of a malware writer (hacker) is to modify

or morph their malware to evade detection by a mal-
A malware detector is a system that attempts to de- ware detector. A common technique used by malware
termine whether a program has malicious intent. In or- writers to evade detection is program obfuscation [30].
der to evade detection, malware writers (hackers) fre- Polymorphism and metamorphism are two common ob-
quently use obfuscation to morph malware. Malware fuscation techniques used by malware writers. For ex-
detectors that use a pattern-matching approach (such ample, in order to evade detection, a virus can morph
as commercial virus scanners) are susceptible to obfus- itself by encrypting its malicious payload and decrypt-
cations used by hackers. The fundamental deficiency ing it during execution. A polymorphic virus obfus-
in the pattern-matching approach to malware detection cates its decryption loop using several transformations,
is that it is purely syntactic and ignores the semantics such as nop-insertion, code transposition (changing the
of instructions. In this paper, we present a malware- order of instructions and placing jump instructions to
detection algorithm that addresses this deficiency by in- maintain the original semantics), and register reassign-
corporating instruction semantics to detect malicious ment (permuting the register allocation). Metamor-
program traits. Experimental evaluation demonstrates phic viruses attempt to evade detection by obfuscat-
that our malware-detection algorithm can detect vari- ing the entire virus. When they replicate, these viruses
ants of malware with a relatively low run-time over- change their code in a variety of ways, such as code
head. Moreover, our semantics-aware malware detec- transposition, substitution of equivalent instruction se-
tion algorithm is resilient to common obfuscations used quences, change of conditional jumps, and register re-
by hackers. assignment [28, 35, 36].
Addition of new behaviors to existing malware is an-
other favorite technique used by malware writers. For
example, the Sobig.A through Sobig.F worm variants
1. Introduction (widespread during the summer of 2003) were devel-
oped iteratively, with each successive iteration adding
A malware instance is a program that has malicious or changing small features [25–27]. Each new vari-
intent. Examples of such programs include viruses, ant managed to evade detection either through the use
trojans, and worms. A classification of malware with of obfuscations or by adding more behavior. The re-
respect to its propagation method and goal is given cent recurrence of the Netsky and B[e]agle worms (both
in [29]. A malware detector is a system that attempts active in the first half of 2004) is also an example of
to identify malware. A virus scanner uses signatures how adding new code or changing existing code creates
and other heuristics to identify malware, and thus is an new undetectable and more malicious variants [9, 17].
example of a malware detector. Given the havoc that For example, the B[e]agle worm shows a series of “up-
can be caused by malware [18], malware detection is an grades” from version A to version C that include the
important goal. addition of a backdoor, code to disable local security
mechanisms, and functionality to better hide the worm
∗ This work was supported in part by the Office of Naval Research
within existing processes. A quote from [17] summa-
under contracts N00014-01-1-0796 and N00014-01-1-0708. The U.S.
Government is authorized to reproduce and distribute reprints for rizes the challenges worm families pose to detectors:
Governmental purposes, notwithstanding any copyright notices af-
fixed thereon. Arguably the most striking aspect of Beagle
The views and conclusions contained herein are those of the authors is the dedication of the author or authors to
and should not be interpreted as necessarily representing the official refining the code. New pieces are tested, per-
policies or endorsements, either expressed or implied, of the above
government agencies or the U.S. Government.
fected, and then deployed with great fore-
† This work was supported in part by Army Research Office grant thought as to how to evade antivirus scanners
DAAD19-01-1-0485. and how to defeat network edge protection
 devices. semantic rather than purely syntactic, it is resistant to
Commercial malware detectors (such as virus scan- common obfuscations used by hackers. Specifically,
ners) use a simple pattern matching approach to mal- this paper makes the following contributions:
ware detection, i.e., a program is declared as malware • Formal semantics: We formally define the problem
if it contains a sequence of instructions that is matched of determining whether a program exhibits a speci-
by a regular expression. A recent study demonstrated fied malicious behavior. In general, this problem is
that such malware detectors can be easily defeated us- undecidable. Our semantics is described in detail in
ing simple program obfuscations [8], that are already Section 2. We believe that the semantics presented in
being used by hackers. The basic deficiency in the pat- Section 2 can be used as a “reference semantics” for
tern matching approach to malware detection is that they other researchers working on malware detection.
ignore the semantics of instructions. Since the pattern-
matching algorithm is not resilient to slight variations, • A semantics-aware malware detection algorithm:
these malware detectors must use different patterns for Since the problem of determining whether a program
detecting two malware instances that are slight varia- exhibits a specified malicious behavior is undecid-
tions of each other. This is the reason that the signature able, we cannot hope to have a algorithm for the
database of a commercial virus scanner has to be fre- abovementioned problem. In Section 3 we present
quently updated. The goal of this paper is to design a an algorithm for handling a limited set of transfor-
malware-detection algorithm that uses semantics of in- mations used by hackers. However, our evaluation
structions. Such an algorithm will be resilient to minor (in Section 4) shows that our algorithm is very effec-
obfuscations and variations. tive in discovering malicious behavior, as it detects
Suppose a program P is transformed by a compiler multiple Netsky and B[e]agle variants with a single
phase (such as register allocation) to produce a program template, and its resilience to obfuscation is better
P ′ . The translation-validation problem is the prob- than that of commercial virus scanners.
lem of determining whether P ′ “simulates” P [31, 32].
Translation validation can be used to prove that vari-
ous compiler phases preserve the semantics of a spe- 2. Semantics of malware detection
cific program P . By viewing hacker obfuscations as
compiler phases, the malware-detection problem at the Before we present our formal semantics for malware
surface seems similar to the translation-validation prob- detection, we will provide an intuitive explanation of
lem. However, there are fundamental differences be- the underlying ideas. We will use the example shown
tween the two problems. Translation-validation tech- in Figure 1 as a running example. Figure 1(a) shows
niques determine whether the two programs are seman- the control-flow graph (CFG) of a specification of ma-
tically equivalent. However, variants of malware are licious behavior and Figure 1(b) shows the CFG of an
not equivalent because malware writers add additional instruction sequence (which is a fragment of larger pro-
functionality between variants. Moreover, translation- gram). Instructions in Figure 1 are in the intermediate
validation researchers make certain assumptions about form (IR) used by our tool. The instructions in our IR
the transformed program. For example, Necula [31] as- are intuitive, and we give a formal description of the IR
sumes that all branches in the target program must cor- in Appendix B. We describe various components of our
respond to branches in the source program. In the con- semantics.
text of malicious-code detection, such an assumption is Specifying the malicious behavior. In our frame-
not valid: an adversary is free to transform the mali- work, malicious behavior is described using templates,
cious code as they wish. Although we use some ideas which are instruction sequences where variables and
from the translation-validation literature in the context symbolic constants are used. Figure 1(a) describes
of malware detection (such as modeling semantics of a malicious behavior that is a simplified version of a
instructions and using decision procedures), our algo- decryption loop found in polymorphic worms. The
rithm is differs from those in the translation-validation malware specification described in Figure 1(a) de-
literature. crypts memory starting from address const addr1 and
We use the observation that certain malicious behav- writes the decrypted data to memory starting at address
iors (such as a decryption loop in a polymorphic virus const addr2. The decryption function and the termina-
or a loop to search for email addresses in a user’s mail tion condition are denoted by the function f(·) and pred-
folder) appear in all variants of a certain malware. The icate condition(·) respectively. By abstracting away the
problem is that the specified malicious behavior (such names of specific registers and symbolic constants in
as a decryption loop) appears in different guises in the the specification of the malicious behavior, our seman-
different variants. We formalize this problem and de- tics and algorithm are insensitive to simple obfusca-
scribe an algorithm for discovering specified malicious tions, such as register renaming and changing the start-
behaviors in a given program. Since our algorithm is ing address of a memory block.
 1

eax = 0x403000
2

ebx = 0x400000
1 const addr1 ← 0x403000
3
A = const_addr1 const addr2 ← 0x400000
edx = eax + 3
2 condition(X) ← X≥0x406000
4
B = const_addr2 true f(X) ← X≪2 + 1
eax >= 0x406000 ? c←4
3
5 false
d←4
condition(A) ? true mem[ebx] = mem[edx−3] << 2 + 1
4 false
6 (c) Execution context.
mem[B] = f(mem[A]) eax = eax + 4
5
7
A=A+c const addr1 : F(0)
edx = edx + 4
6
const addr2 : F(0)
8
c : F(0)
B=B+d ebx = ebx + 4
d : F(0)
7 9
f : F(1)
jump const_address2 jump 0x400000
condition : P(1)

(a) Template of malicious behavior. (b) Malware instance. (d) Symbolic constant types.

Figure 1. Malware instance (b) satisfies the template (a) according to our semantics.

When does an instruction sequence contain the constant is 0-ary function or has type F (0). A con-
malicious behavior? Consider the instruction se- stant c of type τ is written as c : τ . In the template
quence shown in Figure 1(b). Assume the symbolic shown in Figure 1(a) the variables VT are {A, B} and
constants in the template are assigned values shown the symbolic constants CT are shown in Figure 1(d).
in Figure 1(c). If the template and the instruction se- Let I be an instruction sequence or a program frag-
quence shown in Figure 1(a) and 1(b) are executed from ment. An example instruction sequence is shown in Fig-
a state where the contents of the memory are the same, ure 1(b). Memory contents are represented as a function
then after both the executions the state of the memory M : Addr → V alues from the set of addresses Addr
is the same. In other words, the template and the in- to the set of values V alues, where M [a] denotes the
struction sequence have the same effect on the mem- value stored at address a.
ory. This is because whenever memory accesses are An execution context for a template T , with T =
performed the addresses in the two executions are the (IT , VT , CT ), is an assignment of values of appropriate
same. Moreover, stores to memory locations are also types to the symbolic constants in the set CT . Formally,
the same. Thus, there is an execution of instruction se- an execution context ECT for a template T is a function
quence shown in Figure 1(b) that exhibits the behavior with domain CT , such that for all c ∈ CT the type of c
specified by the template given in Figure 1(a). In other and ECT (c) are the same. An execution context for the
words, the malicious behavior specified by the template template shown in Figure 1(a) is shown in Figure 1(c).
is demonstrated by the instruction sequence. Note that Given an execution context ECT for a template T , let
this intuitive notion of an instruction sequence demon- ECT (T ) be the template obtained by replacing every
strating a specified malicious behavior is not affected constant c ∈ CT by ECT (c).
by program transformations, such as register renaming, A state sT for the template T is a 3-tuple denoted
inserting irrelevant instruction sequences, and changing (valT , pcT , memT ), where valT : VT → V alues is
starting addresses of memory blocks. an assignment of values to the variables in VT , pcT is a
value for the program counter, and memT : Addr →
2.1. Formal semantics V alues gives the memory content. We follow the con-
vention that a state and its component referring to the
A template T = (IT , VT , CT ) is a 3-tuple, where template are superscripted by T . Given a template state
IT is a sequence of instructions and VT and CT are sT , val(sT ), pc(sT ) and mem(sT ) refer to the three
the set of variables and symbolic constants that appear components of the state. Similarly, a state for the in-
in IT . There are two types of symbolic constants: an struction sequence I is a 3-tuple (val, pc, mem), where
n-ary function (denoted as F (n)) and an n-ary predi- val : Reg → V alues is an assignment of values to
cate (denoted as P (n)). Notice that a simple symbolic the set of registers Reg, pc is a value for the program
counter, and mem : Addr → V alues gives the mem- • (Condition 3): If
ory contents. Let S T and S be the state space for the
pc(sTk ) ∈ affected (σ(T, ECT , sT0 )),
template and the instruction sequence respectively.
Assume that we are given an execution context ECT then pc(sr ) ∈ affected (σ(T, ECT , sT0 )). In other
for a template T , and the template is in state sT . If we words, if the program counter at the end of executing
execute an instruction i from the template ECT (T ) in the template T points to the affected memory area,
state sT , we transition to a new state sT1 and generate a then the program counter after executing I should
system event e (in our context, events are usually sys- also point into the affected memory area.
tem calls or kernel traps). We denote a state change
from sT to sT1 generating an event e as sT −→ sT1 .
e Consider the example shown in Figure 1. Assume
For uniformity, if an instruction i does not generate a that we use the execution context shown in Figure 1(c)
system event, we say that it generates the null event, for the template shown in Figure 1(a). Suppose we ex-
or e(i) = null. For every initial template state sT0 , ecute the template and instruction sequence shown in
executing the template T in an execution context ECT Figure 1 from states with the same memory contents.
generates a sequence as follows: The state of the memory is same in both executions,
so condition 1 is true. Condition 2 is trivially satisfied.
eT eT Since the jumps have the same target, condition 3 is triv-
σ(T, ECT , sT0 ) = sT0 −→
1
sT1 −→
1
··· , ially true.
where for i ≥ 1, sTi is the state after executing the i- Definition 2 A program P satisfies a template T (de-
th instruction from the template ECT (T ) and eTi is the noted as P |= T ) iff P contains an instruction sequence
event generated by the (i − 1)-th instruction. Notice I such that I contains a behavior specified by T . Given
that if the template does not terminate, σ(T, ECT , sT0 ) a program P and a template T , we call the problem of
can be infinite. Similarly, σ(I, s0 ) denotes the sequence determining whether P |= T as the template matching
when the instruction sequence I is executed from the problem or TMP.
initial state s0 . Defining a variant family. Definition 2 can be used
Definition 1 We say that an instruction sequence I to define a variant family. The intuition is that most vari-
contains a behavior specified by the template T = ants of a malware contain a common set of malicious
(IT , VT , CT ) if there exists a program state s0 , an ex- behavior, such as a decryption loop and a loop to search
ecution context ECT , and a template state sT0 such for email addresses. Let T be a set of templates (this set
that mem(sT0 ) = mem(s0 ) (the memory in the two contains specification of malicious behavior common to
states are the same), and the two sequences σ(I, s0 ) and a certain malware family). The set T defines a variant
σ(T, ECT , sT0 ) are finite, and the following conditions family as follows:
hold on the two sequences: {P | for all T ∈ T , P |= T }
• (Condition 1): Let the two execution sequences be In other words, the variant family defined by T contains
given as follows: all programs that satisfy all templates in the set T.
eT eT eT Theorem 1 TMP is undecidable.
σ(T, ECT , sT0 ) = sT0 −→
1
sT1 −→
2 k
· · · −→ sTk Proof: We will reduce the halting problem to TMP. Let
e
1 2 e r e
σ(I, s0 ) = s0 −→ s1 −→ · · · −→ sr M be a Turing machine, and PM be a program that uses
instructions in our IR that simulates M (since our IR
Let affected (σ(T, ECT , sT0 )) be the set of addresses is Turing complete, this can be accomplished). With-
a such that mem(sT0 )[a] 6= mem(sTk )[a], i.e., out loss of generality, assume that PM does not touch
affected (σ(T, ECT , sT0 )) is the set of memory ad- a special address sp addr while simulating the Turing
dresses whose value changes after executing the tem- machine M . Before starting to simulate M , PM sets
plate T from the initial state. We require that mem[sp addr] to 0. After simulating M , if PM halts,
mem(sTk )[a] = mem(sr )[a] holds for all a ∈ it sets mem[sp addr] to 1. Consider the template T
affected (σ(T, ECT , sT0 )), i.e. values at addresses shown below:
that belong to the set affected (σ(T, ECT , sT0 )) are mem[sp addr] = 0
the same after executing the template T and the in- mem[sp addr] = 1
struction sequence I.
It is easy to see that PM |= T iff M halts. 
• (Condition 2): Ignoring null events, the event se-
quence heT0 , · · · , eTk i is a subsequence of the event 2.2. A weaker semantics
sequence he0 , · · · , er i. In order for the two system
events e1 and e2 to match, their arguments and return In some scenarios the semantics described in defini-
values should be identical. tion 1 is too strict. For example, if a template uses cer-
tain memory locations to store temporaries, then these Program IDAPro
memory locations should not be checked for equality
in comparing executions of an instruction sequence and IR conversion
a template. Let σ(T, ECT , sT0 ) be the sequence gen-
erated when a template T is executed from a state sT0
using the execution context ECT . Define a set of core Template Program IR
memory locations core(σ(T, ECT , sT0 )) which is a sub-
set of affected (σ(T, ECT , sT0 )). Condition 1 in defini-
tion 1 can be changed as follows to give a weaker se- Yes/No Malware detector
mantics:
• (Modified condition 1): We require that for all
Decision procedures
a ∈ core(σ(T, ECT , sT0 )), we have mem(sTk )[a] =

Increasing precision and cost

mem(sr )[a], i.e., values in the addresses that belong Predefined patterns of
Nop library
irrelevant instructions.
to the set core(σ(T, ECT , sT0 )) are the same after ex-
ecuting the template T and the instruction sequence Randomized x86
interpreter. Random execution
I. In other words, only the memory locations in the
set core(σ(T, ECT , sT0 )) are checked for equality in Theorem prover. Simplify
the two executions. Decision procedure with
bit-vector arithmetic. UCLID
There are several ways to define core memory lo-
cations. We describe one possible definition of core
memory locations. Assume that each instruction in Figure 2. The architecture of the mali-
the template T is labeled as temp or persistent. Intu- cious code detector. Gray boxes repre-
itively if an instruction is labeled with temp, it performs sent existing infrastructure.
temporary computation. Let σ(T, ECT , sT0 ) be the se-
quence generated when a template T executed from a
state sT0 using the execution context ECT . Recall that starting with the input binary program on the left and
affected (σ(T, ECT , sT0 )) is the set of addresses a such ending with an output indicating whether the program
that mem(sT0 )[a] 6= mem(sTk )[a], where sTk is the satisfies the template.
last state in the sequence σ(T, ECT , sT0 ). Then the core We first disassemble the binary program, construct
memory locations core(σ(T, ECT , sT0 )) can be defined control flow graphs, one per program function (as iden-
as any address in affected (σ(T, ECT , sT0 )) that is a tar- tified by IDA Pro), and produce an intermediate rep-
get of a load or store instruction with label persistent. resentation (IR). The IR is generated using a library
of x86 instruction transformers, in such a way that the
IR is architecture-independent. The IR can still con-
3. Semantics-aware matching algorithm tain library and system calls that are, by definition,
platform-specific – we remove this last dependency by
We present an algorithm for checking whether a pro- using summary functions expressed in IR form (see Ap-
gram P satisfies a template T ; this algorithm is sound, pendix B for a description of the IR).
but not complete, with respect to the semantics pre- The rest of the toolkit takes advantage of the plat-
sented in Section 2.1, yet it can handle several classes form- and architecture-independent IR. We have ini-
of obfuscations (see Section 3.4). We have also im- tially focused our research on the Microsoft Windows
plemented our algorithm as a malware-detection tool. on Intel IA-32 platform, with no loss of generality. By
The tool takes as input a template and a binary program providing appropriate front-ends that translate the pro-
for the Microsoft Windows on Intel IA-32 (x86) plat- gram into the IR format, one can use our malware-
form, and determines whether a fragment of the sus- detector tool to process programs for different plat-
picious program contains a behavior specified by the forms.
template. The tool’s output describes the matched frag-
ment of the program together with information relating
template variables to program registers and memory lo- 3.2. Algorithm description
cations, and a list of “irrelevant” instruction sequences
that are part of the matched program fragment. Our malware detection algorithm AMD works by
finding, for each template node, a matching node in
3.1. Architecture the program. Individual nodes from the template and
the program match if there exists an assignment to vari-
The tool is built on top of the IDA Pro disassem- ables from the template node expression that unifies it
bler [13]. Figure 2 illustrates the flow through the tool, with the program node expression. Once two matching
nodes are found, we check whether the def-use relation- – The instruction I(n1 ) (at node n1 ) defines a vari-
ships true between template nodes also hold true in the able that I(nk ) uses, that is, def (n1 ) ∩ use(nk ) 6=
corresponding program (two nodes are related by a def- ∅.
use relationship if one node’s definition of a variable – None of the instructions at the intermediate nodes
reaches the other node’s use of the same variable). If all redefine the value of the variables in def (n1 ), i.e.,
the nodes in the template have matching counterparts def (n1 ) ∩ def (ni ) = ∅ for 1 < i < k.
under these conditions, the algorithm has found a pro-
gram fragment that satisfies the template and produces Two nodes n1 and n2 are def-use related (denoted by
a proof of this relationship. duse(n1 , n2 )) iff there exists a path from n1 to n2
Formally, given a malicious code template T and a that is a def-use path. In Figure 3, template nodes
program P , we define the malware detection algorithm 1 and 3 are def-use related, with the corresponding
AMD as a predicate on pairs of programs and templates, def-use path h1, 2, 3i.
AMD : Programs × Templates → {yes, ⊥}, such that
AMD (P, T ) returns “yes” when program P satisfies Consider two nodes n and n′ in T that are def-use
template T under the conditions below. We denote by ⊥ related. Let nodes n and n′ in T be matched to
(i.e. “don’t know”) the return value from the algorithm nodes m and m′ in program P , i.e. f (m) = n
when it cannot determine that the program satisfies the and f (m′ ) = n′ . Suppose X is a variable that is
template. defined in n and used in n′ , and B(X, n, m) = r
and B(X, n′ , m′ ) = r′ , i.e., X is bound to expres-
Consider the example in Figure 3, where a simpler
sion r program in node m and r′ in program node
version of the template T from Figure 1(a) uses expres-
m′ . In this case, we need to make sure that the
sion X≥const addr3 instead of the symbolic constant
value of expression r after executing the instruction
condition(X) and Xˆ5 (where symbol ‘ˆ’ represents the
I(m) is same as the value of expression r′ before
bit-vector xor operator) instead of f(X). Similarly, the
executing the instruction I(m′ ), for every program
program P in Figure 3 is a simpler version of the mal-
path corresponding (under the matching function f )
ware instance from Figure 1(b). For P to satisfy T in
to a template path from n to n′ . In Figure 3, pro-
algorithm AMD , i.e., AMD (P, T ) = yes, the following
gram nodes 1 and 5 match template nodes 1 and
two conditions have to hold:
3, with respective bindings B(A, 1, 1) = eax and
B(A, 3, 5) = (ecx − 1). The condition we need to
• Matching of template nodes to program nodes.
enforce is that the value of eax after program instruc-
First, each node n in the template has to unify with a
tion I(1) is same as the value of (ecx −1) before pro-
node m in P . In other words, there is an onto partial
gram instruction I(5). In our algorithm, these value
function f from the nodes of P to nodes in T , such
invariants are checked using a decision procedure.
that f (m) in T and m in P are unifiable. We will de-
note by B(X, n, m), where n = f (m), the binding Algorithm AMD implements a conservative approxima-
of variable X referred to in the template instruction tion of the formal semantics of malware detection from
at node n to an expression referred to in the program Section 2.1, as the following result demonstrates.
instruction at node m. In Figure 3, the gray arrows
connect unified template nodes and program nodes: Theorem 2 AMD is sound with respect to the TMP se-
for instance, program node 7 unifies with template mantics, i.e. AMD (P, T ) = yes implies that P |= T .
node 4. 
The proof is given in Appendix A. Our algorithm is thus
• Preservation of def-use paths. For each node n in sound with respect to the TMP semantics, but it is not
T , define def (n) as the set of variables defined in complete. We can show there exists a program P , and a
n. Similarly, use(n) is the set of variables used in template T , such that P |= T , but AMD (P, T ) returns
node n. In Figure 3, template node 3 uses variable A, ⊥. Consider the following example (in C-like syntax)
use(3) = {A}, while template node 6 both defines where the template captures the behavior of initializing
and uses variable B, def (6) = use(6) = {B}. De- an array of 10 elements to 0.
fine synthetic nodes npre as predecessor to the tem- Template T Program P
plate entry node and npost as successor to all template for ( i =0; i <10; i +=2)
f o r ( i =0; i <10; i ++)
exit nodes, such that def (npre ) = use(npost ) = VT . a[ i ]=0;
a[ i ]=0;
A def-use path is a sequence of nodes hn1 , ..., nk i for ( i =1; i <10; i +=2)
such that the following conditions are true: a[ i ]=0;
Both the program and the template initialize one array
– For 1 ≤ i < k, there is an edge from ni to ni+1 of 10 elements: the sets of affected memory locations
in T , i.e., hn1 , · · · , nk i is a valid path through the are the same in the final states of the template and the
template T . program, thus P |= T . Our algorithm AMD will not
 1 1

A = const_addr1 eax = 0x403000

2 2

B = const_addr2 ebx = 0x400000 valpre<I>(eax) = valpost<I>(ecx−1)

A
3

nop
4

ecx = eax + 1
3 5
true true
A > const_addr3 ? ecx − 1 > 0x406000
false 6
false

eax = ecx − 1
4 7

mem[B] = mem[A] ^ key mem[ebx] = mem[eax]^5

5 8

A=A+c eax = eax + 1

6 9

B=B+d ebx = ebx + 2

7 10

jump const_address2 jump 0x400000

Figure 3. Example of program (on the right) satisfying a template (on the left) according to
our algorithm AMD . Gray arrows connect program nodes to their template counterparts. The
dashed arrow on the left marks one of the def-use relations that does hold true in the template,
while the corresponding dashed arrow on the right marks the def-use relation that must hold true
in the program.

match the program with the template as the ordering of • A predefined operator in the template can only be
memory updates in the template loop is different from unified with the same operator in the program.
that in the program loops.
Local unification. The unification step addresses • An external function call in the template can only be
the first condition of algorithm AMD , by producing an unified with the same external function call in the
assignment to variables in an instruction at a template program.
node such that it matches a program node instruction (if Standard unification rules apply in all other cases: for
such an assignment exists). Since a program node ex- example, an operator expression in the template unifies
pression contains only ground terms, the algorithm uses with an expression in the program if the program oper-
one-way matching that instantiates template variables ator is the same as the template operator and the cor-
with program expressions. In Figure 3, template node responding operator arguments unify. Template node 1
3 matches program node 5, with the binding { A ← ecx (A = const addr1) can unify with program nodes 1
- 1, const addr3 ← 0x406000 }. In the prototype we
(eax = 0x403000) or 2 (ebx = 0x400000) but not
implemented, a template can contain expressions with with program nodes 8 or 9, since template node 1 ex-
variables, symbolic constants, predefined functions (see pression has a symbolic constant as the right-hand side.
the operators in Appendix B), and external function The result of the local unification step is a binding
calls. The matching algorithm takes these restrictions relating template variables to program expressions. The
into account: binding for the example in Figure 3 is shown in Table 1.
• A variable in the template can be unified with any Note that the bindings are different at various program
program expression, except for assignment expres- points (at program node 1, template variable A is bound
sions. to eax, while at program node 5, template variable A is
bound to (ecx - 1)). Requiring bindings to be consis-
• A symbolic constant in the template can only be uni- tent (monomorphic) seems like an intuitive approach,
fied with a program constant. but leads to an overly restrictive semantics – any obfus-
cation attack that reassigns registers in a program frag-
• The predefined function memory : F (1) (traditionally ment would evade detection. We want to check pro-
written in array notation memory[. . . ]) can only be gram expressions bound to the same template variable
unified with the program function memory : F (1). (e.g. eax and (ecx - 1) are both bound to A) and verify
they change value in the same way the template vari- lem: given the program fragment R, we want to check
able changes values. We employ a mechanism based whether it maintains the value predicate
φ ≡ ∀I ∈
on def-use chains and value preservation to answer this R . val prehIi (eax ) = val posthIi (ecx − 1) .
problem. The algorithm uses decision procedures to determine
whether a value predicate holds. We discuss these de-
Unified nodes Bindings cision procedures in Section 3.3; for now, we treat the
A ← eax decision procedures as oracles that can answer queries
T1 P1
const addr1 ← 0x403000 of the form “Does a program fragment R maintain an
B ← ebx invariant φ?” For each match considered between a pro-
T2 P2
const addr2 ← 0x400000 gram node and a template node, the algorithm checks
A ← ecx - 1 whether the def-use chain is preserved for each program
T3 P5
const addr3 ← 0x406000 expression corresponding to a template variable used at
A ← eax the matched node (see Appendix C for a listing of all
T4 P7
B ← ebx the def-use chains checked for the example shown in
A ← eax Figure 3). This approach eliminates a large number of
T5 P8
increment1 ← 1 matches that cannot lead to a correct template assign-
B ← ebx ment.
T6 P9
increment2 ← 2
T7 P10 const addr2 ← 0x400000 3.3. Using decision procedures to check value-
preservation
Table 1. Bindings generated from the uni-
fication of template and program nodes in
A value-preservation oracle is a decision procedure
Figure 3. Notation Tn refers to node n of
that determines whether a given program fragment is
the template, and Pm refers to node m of
a semantic nop with respect to certain program vari-
the program.
ables. Formally, given a program fragment P and pro-
gram expressions e1 , e2 , a decision procedure D deter-
Value preservation on def-use chains. The sec- mines whether the value predicate φ(P, e1 , e2 ) ≡ ∀I ∈
ond condition of algorithm AMD requires template vari- P . val prehIi (e1 ) = val posthIi (e2 ) holds.
ables and the corresponding program expressions to 
have similar update patterns (although not necessarily  true if P is a semantic nop,
the same values). For each def-use chain in the tem- D(P, e1 , e2 ) = i.e. φ(P, e1 , e2 ) holds
plate, the algorithm checks whether the value of the pro-  ⊥ otherwise
gram expression corresponding to the template-variable
use is the same as the value of the program expression Similarly, we can define decision procedures that de-
corresponding to the template-variable definition. Con- termine whether ¬φ(P, e1 , e2 ) holds (in this case, the
sider Figure 3, where template nodes 1 and 3 are def- result of D(P, e1 , e2 ) is “false” or ⊥). We denote by
use related. Program nodes 1 and 5 map to template D+ a decision procedure for φ(P, e1 , e2 ), and by D− a
nodes 1 and 3, respectively. Denote by R the program decision procedure for ¬φ(P, e1 , e2 ).
fragment between nodes 1 and 5, such that R contains As the value preservation queries are frequent in our
only program paths from program node 1 to node 5 that algorithm (possibly at every step during node match-
correspond to template paths between template nodes 1 ing), the prototype use a collections of decision proce-
and 3 (i.e., any path in R has nodes that either map to dures ordered by their precision and cost. Intuitively,
template paths between 1 and 3, or have no correspond- the most naı̈ve decision procedures are the least precise,
ing template node). A def-use chain for template vari- but the fastest to execute. If a D+ -style decision proce-
able A connects template node 1 with template node 3: dure reports “true” on some input, all D+ -style deci-
in the program, the expressions corresponding to tem- sion procedures following it in the ordered collection
plate variable A after program node 1 and before pro- will also report “true” on the same input. Similarly, if a
gram node 5 must be equal in value, across all paths in D− -style decision procedure reports “false on some in-
R. This condition can be expressed in terms of value put, all D− -style decision procedures following it in the
predicates: for each path I in program fragment R (e.g. ordered collection will also report “false” on the same
I = hP2, P3, P4i), val prehIi (eax ) = val posthIi (ecx − 1), input. As both D+ - and D− -style decision procedures
where val prehIi represents the variable-valuation func- are sound, we define the order between D+ and D− de-
tion for the program state before path I and val posthIi cision procedures based only on performance.
is the variable-valuation function for the program state This collection of decision procedures provides us
after path I. We can formulate this query about pre- with an efficient algorithm for testing whether a pro-
serving def-use chains as a value preservation prob- gram fragment P preserves expression values: iterate
through the ordered collection of decision procedures, Handled
Obfuscation transformation
querying each Di , and stop when one of them returns by AMD ?
“true”, respectively “false” for D− -style decision pro- Instruction reordering ✓
cedures. This algorithm provides for incrementally ex- Register renaming ✓
pensive and powerful checking of the program frag- Garbage insertion ✓
ment, in step with its complexity: program fragments Instruction replacement limited
that are simple semantic nops will be detected early by Equivalent functionality ✗
decision procedures in the ordered collection. Complex Reordered memory accesses ✗
value preserving fragments will require passes through
multiple decision procedures. We present, in order, four Table 3. Obfuscation transformations ad-
decision procedures that are part of our prototype. dressed by our malware detection algo-
Nop Library D+ rithm and some limitations.
NOP . This decision procedure iden-
tifies sequences of actual nop instructions, which are
processor-specific instructions similar to the skip state- fragment for a given number of steps and determines
ment in some high-level programming languages, as whether φ(P, e1 , e2 ) holds at the end of the simulation.
well as predefined instruction sequences known to be For illustration, consider Figure 3: the value preser-
semantic nops. Based on simple pattern matching, the vation problem consists of the program fragment R, cre-
decision procedure annotates basic blocks as nop se- ated from program nodes 2, 3, and 4, and the value pred-
quences where applicable. If the whole program frag- icate φ ≡ ∀I ∈ R . val prehIi (eax ) = val posthIi (ecx −1).
ment under analysis is annotated, then it is a semantic To use the Simplify theorem proving oracle, the formula
nop. The nop library can also act as a cache for queries shown in Table 2 is generated from program fragment
already resolved by other decision procedures. R.
Randomized Symbolic Execution D− RE . This ora-
cle is based on a D− -style decision procedure using ran- 3.4. Strengths and limitations
domized execution. The program fragment is executed
using a random initial state (i.e. the values in registers For the algorithm to be effective against real-life at-
and memory are chosen to be random). At completion tacks, it has to “undo” various obfuscations and other
time, we check whether it is true that ¬φ(P, e1 , e2 ): if program transformations that a malware writer might
true, at least one path in the program fragment is not a use. We list the strengths and weaknesses of our algo-
semantic nop, and thus the whole program fragment is rithm in Table 3. We discuss below in detail four classes
not a semantic nop. of obfuscations algorithm AMD can handle: code re-
Theorem Proving D+ ThSimplify . The value preser- ordering, equivalent instruction replacement, register
vation problem can be formulated as a theorem to be renaming, and garbage insertion.
proved, given that the program fragment has no loops. Code reordering is one of the simplest obfuscations
We use the Simplify theorem prover [14] to implement hackers use to evade signature matching. The obfus-
this oracle: the program fragment is represented as a cation changes the order of instructions on disk, in the
state transformer δ, using each program register and binary image, while maintaining the execution order us-
memory expression converted to SSA form. We then ing jump instructions. This obfuscation is handled by
use Simplify to prove the formula δ ⇒ φ(P, e1 , e2 ), in the use of control flow graphs. Register renaming is a
order to show that all paths through the program frag- simple obfuscation that reassigns registers in selected
ment are semantic nops under φ(P, e1 , e2 ). If Simplify program fragments. As a result, a signature matching
confirms that the formula is valid, the program fragment tool will fail to detect any obfuscated variant as long as
is a semantic nop. One limitation of the Simplify the- it searches for a signature with a specific register. Our
orem prover is the lack of bit-vector arithmetic, which template matching algorithm avoids this pitfall by us-
binary programs are based on. Thus, we can query Sim- ing templates. The uninterpreted variables are assigned
plify only on programs that do not use bit-vector opera- corresponding program registers and memory locations
tions. only during unification and, thus, the matching algo-
Theorem Proving D+ ThUCLID . A second theorem rithm can identify any program with the same behavior
proving oracle is based on the UCLID infinite-state as the template, irrespective of the register allocation.
bounded model checker [22]. For our purposes, the Garbage insertion obfuscates a program by inserting
logic supported by UCLID is a superset of that sup- instruction sequences that do not change the behavior
ported by Simplify. In particular, UCLID precisely of the program. Algorithm AMD tackles this class of
models integer and bit-vector arithmetic. We model the obfuscations through the use of decision procedures to
program fragment instructions as state transformers for reason about value predicates on def-use chains. Equiv-
each register and for memory (represented as an unin- alent instruction replacement uses the rich instruction
terpreted function). UCLID then simulates the program set available on some architectures, notably on the Intel
 Instruction sequence: Simplify formula:
2: ebx = 0x400000 ( IMPLIES ( AND ( EQ ebx 1 4194304)
3: nop (EQ ecx 4 ( + e a x p r e 1 ) )
4: ecx = eax + 1 (EQ e c x p o s t ecx 4 )
)
(EQ e a x p r e ( − e c x p o s t 1 ) )
Value predicate: )
∀I ∈ R . val prehIi (eax ) = val posthIi (ecx − 1)

Table 2. Example of Simplify query corresponding to program fragment R and value predicate
φ.

IA-32 (x86), to replace groups of consecutive instruc- gorithm’s resilience to obfuscation. We used malware
tions with other short instruction sequences that have instances in the wild both for developing behavior tem-
the same semantics. For example, to add 1 to register plates and as starting point for obfuscation transforma-
X in the x86 architecture, one can use the inc X (in- tions in the obfuscation resilience testing. Highlights of
crement) instruction, or the add X, 1 instruction, or the evaluation are:
the sub X, -1 instruction. We handle a limited kind
of instruction replacement by normalizing the code into • The template-based algorithm detects worms from
an intermediate representation (IR) with semantically- the same family, as well as unrelated worms, using
disjoint operations. a single template.
Limitations. Our tool has few limitations. First, • No false positives were generated when running our
the current implementation requires all of the IR in- malware detector on benign programs, illustrating the
structions in the template to appear in the same form soundness of our algorithm in its current implemen-
in the program. For example, if an IR node in the tem- tation.
plate contains “x = x * 2”, the same operation (an
assignment with a multiplication on the right hand side) • The algorithm exhibits improved resilience to obfus-
has to appear in the program for that node to match. cation when compared to commercial anti-virus tool
This means that we will not match the equivalent pro- McAfee VirusScan.
gram instruction “eax = eax << 1” (arithmetic left
shift). Attacks against this requirement are still possi- 4.1. Variant detection evaluation
ble, but are fairly hard, as the bar has been raised: the
attacker has to devise multiple equivalent, yet distinct,
We developed two templates and tested malware
implementations of the same computation, to evade de-
samples and benign programs against them. One tem-
tection. This is not an inherent limitation of the seman-
plate captures the decryption-loop functionality com-
tics, and can be handled using an IR normalization step.
mon in many malicious programs, while the other tem-
The second limitation comes from the use of def-use
plate captures the mass-mailing functionality common
chains for value preservation checking. The def-use re-
to email worms. While throughout this section we use
lations in the malicious template effectively encode a
only the decryption-loop template as example, the re-
specific ordering of memory updates – thus, our algo-
sults from using the mass email template were similar.
rithm AMD will detect only those program that exhibit
For malware samples we used seven variants of Netsky
the same ordering of memory updates. We note that au-
(B, C, D, O, P, T, and W), seven variants of B[e]agle (I,
tomatically generating functionally-equivalent variants
J, N, O, P, R, and Y), and seven variants of Sober (A, C,
is a hard problem. Handling obfuscations that reorder
D, E, F, G, and I). All of them are email worms, each
instructions to achieve a different ordering of memory
with many variants in the wild, ranging in size from
updates is one goal of our ongoing research.
12 kB to 75 kB.
Malware Template detection Running time
4. Experimental results Decryp- Mass-
family tion loop mailer
Avg. Std. dev.
Netsky 100% 100% 99.57 s 41.01 s
We evaluated our implementation of algorithm AMD B[e]agle 100% 100% 56.41 s 40.72 s
against real-world malware variants. The three major Sober 100% 0% 100.12 s 45.00 s
goals of our experiments were to develop malicious be-
havior templates that can match malware variants, to Table 4. Malware detection using algo-
measure the false positive rates the algorithm generates rithm AMD for 21 e-mail worm instances.
with these templates, and to measure the detection al-
 The decryption-loop template describes the behav- tion: programs are grouped by size, in 5 kB increments,
ior of programs that unpack or decrypt themselves to and the disassembly and detection rates are plotted for
memory: the template consists of (1) a loop that pro- each group. For example, the bar to the right of the
cesses data from a source memory area and writes data 71,680 B point represents programs between 71,680 B
to a destination memory area, and (2) a jump that targets and 76,799 B in size, and indicates that 97.78% of the
the destination area. Many binary worms use this tech- programs in this group were disassembled successfully
nique to make static analysis harder: the actual code of and were detected as benign, while 2.22% failed to dis-
the worm is not available for inspection until runtime. assemble (either because the disassembler crashed, or
To construct the template, we analyzed the Netsky.B because it failed to group program code into functions).
email worm and manually extracted the code that per- The running time per test case ranged from 9.12 s to
forms the decryption and the jump. The template was 413.78 s, with an average of 165.5 s.
then generalized from this code by replacing actual reg-
False Positive Evaluation of 2,000 Binaries
isters and memory addresses with variables. The mass-
mailing template was developed in a similar manner: it 100%

describes the custom SMTP functionality present in the

worm, including email composition and transmission. 80%

We applied our algorithm on each malware instance

and each decryption-loop template. We achieved 100% Disassembly rate 60%
detection of the Netsky and B[e]agle worm instances
using only the two templates (the mass-mailing tem- 40%
plate required minor tweaks to detect B[e]agle). The
Sober worm instances did not match the mass-mailing 20%
template, requiring a separate template. Due to limita-
tions in the current prototype implementation, match- 0%
ing calls into the Microsoft Visual Basic runtime li- 0B 35,840 B 71,680 B 107,520 B 143,360 B

brary (used by Sober worm instances) is not supported. Program size (grouped in 5 kB increments)

Nonetheless, we have shown that algorithm AMD can

detect multiple worm instances from the same family Figure 4. Evaluation of algorithm AMD on
using one template, in contrast with commercial virus a set of 2,000 benign Windows programs
scanners that require many signatures (McAfee VirusS- yielded no false positives. Gray bars indi-
can uses 13 signatures to detect the worms in our test cate the percentage of programs that dis-
set). Our detection results come with one caveat: 3 of assembled correctly and were detected as
the Netsky variants could not be processed by IDA Pro. benign; white bars indicate the percent-
Since we build our tool on the assumption that disas- age of programs that failed to disassem-
sembly can be performed successfully, we do not con- ble.
sider those 3 variants as false negatives. Running times
for the tool (listed in Table 4) are satisfactory for a pro-
totype and suggest that real-time performance can be 4.3. Obfuscation resilience evaluation
achieved with an optimized implementation.
To test resilience to obfuscation, we applied garbage
4.2. False positive evaluation insertion transformations of varying degrees of com-
plexity to worm variant B[e]agle.Y, and compared the
We evaluated the algorithm and the templates against detection rate of our malware detection tool against
a set of benign programs, in order to measure the false McAfee VirusScan. The garbage insertion transforma-
positive rate and to evaluate the templates we devel- tion adds code fragments to a program, such that each
oped. We used a set of 2,000 programs with sizes code fragment is irrelevant in its insertion context. We
between <1 kB and 150 kB, from standard Microsoft considered three types of garbage insertion: (1) nop in-
Windows system programs, to compiler tools, to Mi- sertion adds simple sequences of nop instructions; (2)
crosoft Office programs. We have also tried to test stack-op insertion adds sequences of stack operations;
larger benign binaries, with sizes up to 5 MB, but they and (3) math-op insertion adds sequences of arithmetic
did not disassemble successfully. For successfully dis- operations. We generated 20 variants for each type of
assembled programs, the false positive rate was 0%, obfuscation. To test the limits of our implementation,
meaning that our implementation AMD correctly clas- one of the math-op insertion transformations actually
sified all programs as benign (none of the test programs replaced single instructions of the form x = x + const
matched the templates). with equivalent sequences. The results are tabulated in
In Figure 4, we present the results of this evalua- Table 5: our tool catches all obfuscations with the ex-
ception of the math-op replacement transformation; in simple obfuscations can thwart the disassembly pro-
such a case, since the algorithm seeks to exactly match cess. The abovementioned research demonstrates the
a template instruction of the form x = x + const, any importance of handling obfuscations in malware detec-
equivalent instruction sequences are not detected. tion. Kruegel et al. [20] present techniques for disas-
sembling obfuscated executables. In this paper, we as-
Obfuscation Algorithm AMD McAfee sume that the malware being analyzed can be disassem-
type Average Detection VirusScan bled.
time rate The theoretical limits of malicious code detection
Nop 74.81 s 100% 75.0% (specifically of virus detection) have been the focus of
Stack op 159.10 s 100% 25.0% many researchers. A virus is a malware that replicates
Math op 186.50 s 95% 5.0% itself by copying its code to other program files. Co-
hen [10] and Chess-White [6] showed that in general
Table 5. Evaluation of algorithm AMD on the problem of virus detection is undecidable. Spinel-
a set of obfuscated variants of B[e]agle.Y. lis [34] proved that problem of reliably identifying a
For comparison, we include the detection bounded-length virus is NP-complete. These results are
rate of McAfee VirusScan. similar to the result presented in this paper (see Theo-
rem 1). However, our formal model of malware detec-
tion is different than the one used by these researchers:
we present a semantics and an algorithm for detecting
5. Related work specific malicious behaviors, avoiding the problem of
undecidability (as proven by Cohen) and the problem of
We compare our work to existing research in the undetectable malware classes (as introduced by Chess
areas of malware detection, translation validation, and and White).
software verification. Dynamic monitoring can also be used for malware
detection. Cohen [10] and Chess-White [6] propose
5.1. Malware detection a virus detection model that executes code in a sand-
box. Another approach rewrites the binary to introduce
In previous work [8], we demonstrated that cur- checks driven by an enforceable security policy [15]
rent commercial virus scanners can be easily evaded (known as the inline reference monitor or the IRM ap-
by using simple obfuscations used by hackers. Many proach). We believe static analysis can be used to im-
malware-detection techniques are based on static- prove the efficiency of dynamic analysis techniques,
analysis of executables. In this area, we previously de- e.g., static analysis can remove redundant checks in the
scribed a malware-detection algorithm called SAFE [7]. IRM framework. In the future we will explore hybrid
SAFE can only handle very simple obfuscations (only static-dynamic approaches to malware detection.
nops can appear between matching instructions), e.g.,
the example shown in Figure 1 cannot be handled by 5.2. Translation validation
SAFE. Moreover, the formal semantics of malware de-
tection was not explored by the earlier work. Static A translation-validation [12, 16, 31, 32] system at-
analysis of binaries is used by Kruegel et al. [21] to tempts to verify that the semantics of a program is not
detect kernel-level rootkits, which are attack tools that changed by an optimizing transformation. Differences
are used by hackers to hide their presence from system between malware detection and translation validation
administrators. They look for suspicious instruction se- were discussed in the introduction.
quences using symbolic execution. The detection algo-
rithm presented in this paper is more powerful because 5.3. Software verification
we use multiple decision procedures (symbolic execu-
tion being just one of them). Our specification of ma- Our work is closely related to previous results on
licious behavior is richer than the one used by Kruegel static analysis techniques for verifying security proper-
et al. Therefore, using our algorithm can result in a ties of software [1, 3–5, 19, 24]. In a larger context, our
more powerful tool for detecting kernel-level rootkits. work is similar to existing research on software verifica-
We will explore this interesting application of our algo- tion [2,11]. However, there are several important differ-
rithm in the future. Singh and Lakhotia [33] provide an ences. First, since these systems analyze benign code,
annotated bibliography of papers on malware analysis they do not have to handle obfuscations. Second, to
and detection. our knowledge, all existing work on static analysis tech-
An essential step in statically analyzing executables niques for verifying security properties analyzes source
is disassembly, which translates machine code to as- code. On the other hand, our analysis technique works
sembly code. Linn and Debray [23] demonstrate that on executables.
6. Conclusion [10] F. Cohen. Computer viruses: Theory and experiments.
Computers and Security, 6:22–35, 1987.
[11] J. Corbett, M. Dwyer, J. Hatcliff, C. Pasareanu, Robby,
We observe that certain malicious behaviors (such S. Laubach, and H. Zheng. Bandera: Extracting finite-
as decryption loops) appear in all variants of a certain state models from Java source code. In Proceedings of
malware. Based on this intuition, we gave a formal the 22nd International Conference on Software Engi-
semantics for malware detection. We also presented a neering (ICSE’00), pages 439–448. ACM Press, 2000.
malware-detection algorithm that is sound with respect [12] D. W. Currie, A. J. Hu, and S. Rajan. Automatic formal
to our semantics. Experimental evaluation demon- verification of DSP software. In Proceedings of the 37th
strated that our algorithm can detect all variants of cer- Annual ACM IEEE Conference on Design Automation
tain malware, has no false positives, and is resilient to (DAC’00), pages 130–135. ACM Press, 2000.
obfuscation transformations generally used by hackers. [13] DataRescue sa/nv. IDA Pro – interactive disassembler.
Published online at http://www.datarescue.
There are several opportunities for further work. In
com/idabase/. Last accessed on 3 Feb. 2003.
the future we will address the limitations discussed in [14] D. Detlefs, G. Nelson, and J. Saxe. The Simplify theo-
Section 3 (see Table 3). We also plan to optimize our rem prover. Published online at http://research.
tool to reduce the execution times. compaq.com/SRC/esc/Simplify.html. Last
Acknowledgments. We are thankful to our anony- accessed on 10 Nov. 2004.
mous reviewers for their invaluable comments. We [15] U. Erlingsson and F. B. Schneider. IRM enforce-
would also like to express our gratitude to Giovanni Vi- ment of Java stack inspection. In Proceedings of the
gna, our shepherd throughout the revision process, for 2000 IEEE Symposium on Security and Privacy (Oak-
his feedback. land’00), pages 246–255, May 2000.
[16] X. Feng and A. J. Hu. Automatic formal verification
for scheduled VLIW code. In Proceedings of the Joint
References Conference on Languages, Compilers and Tools for Em-
bedded Systems & Software and Compilers for Embed-
ded Systems (LCTES/SCOPES’02), pages 85–92. ACM
[1] K. Ashcraft and D. Engler. Using programmer-written
Press, 2002.
compiler extensions to catch security holes. In Proceed- [17] J. Gordon. Lessons from virus developers: The Beagle
ings of the 2002 IEEE Symposium on Security and Pri- worm history through April 24, 2004. In SecurityFocus
vacy (Oakland’02), pages 143–159, May 2002. Guest Feature Forum. SecurityFocus, May 2004. Pub-
[2] T. Ball and S. Rajamani. Automatically validating tem- lished online at http://www.securityfocus.
poral safety properties of interfaces. In Proceedings of com/guest/24228. Last accessed: 9 Sep. 2004.
the 8th International SPIN Workshop on Model Check- [18] L. A. Gordon, M. P. Loeb, W. Lucyshyn, and
ing of Software (SPIN’01), volume 2057 of Lecture R. Richardson. 2004 CSI/FBI computer crime and se-
Notes in Computer Science, pages 103–122, Toronto, curity survey. Technical report, Computer Security In-
Ontario, Canada, 2001. Springer-Verlag Heidelberg. stitute, 2004.
[3] M. Bishop and M. Dilger. Checking for race conditions [19] T. Jensen, D. Métayer, and T. Thorn. Verification of
in file accesses. Computing Systems, 9(2), 1996. control flow based security properties. In Proceedings
[4] H. Chen and D. Wagner. MOPS: an infrastructure for of the 1999 IEEE Symposium on Security and Privacy
examining security properties of software. In 9th ACM (Oakland’99), May 1999.
Conference on Computer and Communications Security [20] C. Kruegel, W. Robertson, F. Valeur, and G. Vigna.
(CCS’02). ACM Press, November 2002. Static disassembly of obfuscated binaries. In Proceed-
[5] B. Chess. Improving computer security using extending ings of the 13th USENIX Security Symposium (Secu-
static checking. In Proceedings of the 2002 IEEE Sym- rity’04), San Diego, CA, Aug. 2004.
posium on Security and Privacy (Oakland’02), pages [21] C. Kruegel, W. Robertson, and G. Vigna. Detecting
160–173, May 2002. kernel-level rootkits through binary analysis. In Pro-
[6] D. Chess and S. White. An undetectable computer virus. ceedings of the 20th Annual Computer Security Ap-
In Proceedings of the 2000 Virus Bulletin Conference plications Conference (ACSAC’04), Tucson, AZ, Dec.
(VB2000), 2000. 2004.
[7] M. Christodorescu and S. Jha. Static analysis of exe- [22] S. K. Lahiri and S. A. Seshia. The UCLID decision pro-
cutables to detect malicious patterns. In Proceedings cedure. In R. Alur and D. A. Peled, editors, Proceed-
of the 12th USENIX Security Symposium (Security’03), ings of the 16th International Conference on Computer
pages 169–186. USENIX Association, USENIX Asso- Aided Verification (CAV’04), volume 3114 of Lecture
ciation, Aug. 2003. Notes in Computer Science, pages 475–478, Boston,
[8] M. Christodorescu and S. Jha. Testing malware de- MA, USA, July 2004. Springer-Verlag Heidelberg.
tectors. In Proceedings of the ACM SIGSOFT Inter- [23] C. Linn and S. Debray. Obfuscation of executable code
national Symposium on Software Testing and Analysis to improve resistance to static disassembly. In Pro-
2004 (ISSTA’04), pages 34–44, Boston, MA, USA, July ceedings of the 10th ACM Conference on Computer and
2004. ACM Press. Communications Security (CCS’03), Oct. 2003.
[9] M. Ciubotariu. Netsky: a conflict starter? Virus Bul- [24] R. Lo, K. Levitt, and R. Olsson. MCF: A malicious code
letin, pages 4–8, May 2004. filter. Computers & Society, 14(6):541–566, 1995.
[25] LURHQ Threat Intelligence Group. Sobig.a and the such that pc(s) = m (in this proof we equate the pro-
spam you received today. Technical report, LURHQ, gram counter with the CFG node it corresponds to), and
2003. Published online at http://www.lurhq. B(·, n, m) be the binding. We define a template state
com/sobig.html. Last accessed on 16 Jan. 2004. corresponding to s (denoted by T (s)) as follows:
[26] LURHQ Threat Intelligence Group. Sobig.e - Evo-
lution of the worm. Technical report, LURHQ, • Let B(val(s)) be the assignment implied by the bind-
2003. Published online at http://www.lurhq. ing B, i.e., B(val(s))(X) is equal to the value of the
com/sobig-e.html. Last accessed on 16 Jan. 2004. expression B(X, n, m) in the state s.
[27] LURHQ Threat Intelligence Group. Sobig.f examined.
Technical report, LURHQ, 2003. Published online at • Define pc(T (s)) to be the program counter corre-
http://www.lurhq.com/sobig-f.html. Last sponding to node n.
accessed on 16 Jan. 2004.
[28] A. Marinescu. Russian doll. Virus Bulletin, pages 7–9, • Define mem(T (s)) as mem(s).
Aug. 2003.
[29] G. McGraw and G. Morrisett. Attacking malicious Let n0 be the initial node of the template and m0
code: report to the Infosec research council. IEEE Soft- be a node in the program such that f (m0 ) = n0 .
ware, 17(5):33 – 41, Sept./Oct. 2000. Let s0 be a state such that pc(s0 ) = m0 , and define
[30] C. Nachenberg. Computer virus-antivirus coevolution. sT0 as T (s0 ). Recall that I(n) denotes the instruction
Communications of the ACM, 40(1):46–51, Jan. 1997. corresponding to node n. Given a state s, I(n)(s) de-
[31] G. C. Necula. Translation validation for an optimiz- notes the state obtained by executing instruction I(n)
ing compiler. In Proceedings of the ACM SIGPLAN from state s, and e(I(n), s) denotes the event gener-
2000 Conference on Programming Language Design ated by executing instruction I(n) from state s. Let s′1
and Implementation (PLDI’00), pages 83–94. ACM be equal to I(m0 )(s0 ) and e1 be equal to e(I(m0 ), s0 ).
Press, June 2000. Let sT1 = I(n0 )(sT0 ) and eT1 = e(I(n0 ), sT0 ). Since n0
[32] A. Pnueli, M. Siegel, and E. Singerman. Translation
and m0 are unifiable, it is easy to see that eT1 = e1 . Let
validation. In Proceedings of the 4th International
Conference on Tools and Algorithms for Construction n1 be the template node corresponding to pc(sT1 ). As-
and Analysis of Systems (TACAS’98), volume 1384 of sume that in program P , from state s′1 there is a feasible
Lecture Notes in Computer Science, pages 151–166. path path(s′1 , m1 ) from pc(s′1 ) to a node m1 , such that
Springer-Verlag Heidelberg, Mar. 1998. f (m1 ) = n1 , i.e., executing path(s′1 , m1 ) from state
[33] P. Singh and A. Lakhotia. Analysis and detection of s′1 yields a state s1 , such that pc(s1 ) = m1 . We de-
computer viruses and worms: An annotated bibliogra- e
note this fact by s0 1 s1 , i.e., there is an execution
phy. ACM SIGPLAN Notices, 37(2):29–35, Feb. 2002. sequence α that starts at s0 and ends at s1 and the first
[34] D. Spinellis. Reliable identification of bounded-length event generated by α is e1 . Thus we obtain the follow-
viruses is NP-complete. IEEE Transactions on Infor-
ing execution sequences in P and T :
mation Theory, 49(1):280–284, Jan. 2003.
[35] P. Ször and P. Ferrie. Hunting for metamorphic. In e1
Proceedings of the 2001 Virus Bulletin Conference σ = s0 s1
(VB2001), pages 123 – 144, Sept. 2001. eT
σT = sT0 →1
sT1
[36] z0mbie. z0mbie’s homepage. Published online at
http://z0mbie.host.sk. Last accessed: 16 Jan. Recall that use(n1 ) is the set of variables used by in-
2004.
struction I(n1 ). For every variable X ∈ use(n1 ),
value of expression B(X, n1 , m1 ) is unchanged by
A. Proof of Theorem 2 path(s′1 , m1 ). Therefore, executing instruction I(n1 )
in the template from the states T (s1 ) and sT1 =
Let P be a program and T a template such that I(n0 )(s0 ) results in same memory accesses, generate
AMD (P, T ) returns yes. Let f be a function from the the same event, and transitions to the same node n2 in
set of nodes of P to nodes in T that satisfy the two the template. We continue augmenting the execution
conditions given in section 3 (such a function exists be- sequences σ and σT until we obtain a template state sTk
cause the AMD (P, T ) returned yes). We will prove that such that pc(sTk ) corresponds to the final node of the
P |= T . template. At the end, we have two execution sequences
There are two set of states - one for the program e1 e2 ek
σ = s0 s1 s2 · · · sk
(which we simply refer to as state) and other for the
eT eT eT
template (which we refer to as template state). Re- σT = sT0 → sT1 → sT2 · · · → sk
1 2 k

call that a state s of program is a 3-tuple comprising

of an assignment val(s) of values to registers, value where σ and σT are execution sequences in the program
pc(s) for the program counter, and the memory con- and the template. Moreover, for 1 ≤ i ≤ k, ei = eTi
tents mem(s). Consider a node n in the template T and and template states sTi and T (si ) satisfy the following
a node m in P such that f (m) = n. Let s be a state property:
• Executing instruction I(pc(si )) in the program P proven as semantic nops for the program in Figure 3
from the state si and instruction I(pc(sTi )) in the to satisfy the template in the same figure. In Table 7,
template from the state T (si ) results in same mem- we list the 10 def-use chains in the template, the de-
ory accesses and generate the same event. rived value predicates that relate program expressions,
and the corresponding program fragments. The nota-
• In the template T , executing instruction I(pc(sTi )) x
tion n → n′ describes the variable corresponding to the
from the state sTi and from the state T (si ) results in
def-use path, such that def (n) ∩ use(n′ ) = {x}.
same memory accesses and generate the same event.
Def-use chain and
The proof of abovementioned property uses the condi- value predicate
Program fragment
tion that def-use paths are preserved by the program. A
From the above two observations it is easy to see that T1 → T3 2: ebx = 0x400000
P |= T . 3: nop
val posthIi (ecx − 1) 4: ecx = eax + 1
= val prehIi (eax )
B. Intermediate representation language A
T5 → T3
9: ebx = ebx + 1
The intermediate representation (IR) of a program is val posthIi (ecx − 1) 4: ecx = eax + 1
structured as a collection of control flow graphs (CFGs), = val prehIi (eax )
A
each node in the graph corresponding to one IR instruc- T1 → T4 2: ebx = 0x400000
tion. Edges in a CFG are unlabeled, with the exception 3: nop
of the outgoing edges of a conditional control flow in- val posthIi (eax ) 4: ecx = eax + 1
struction. In such a case, the IR instruction is labeled = val prehIi (eax ) 6: eax = ecx - 1
A
jump, and the outgoing edges are labeled with the cor- T5 → T4 9: ebx = ebx + 1
responding condition that holds true along that branch. 4: ecx = eax + 1
IR instructions are elements in a language with sim- val posthIi (eax ) 6: eax = ecx - 1
ple rules, illustrated in Table 6. Each IR instruction (or = val prehIi (eax )
B
IR term) is either an assignment that updates state vari- T2 → T4 3: nop
ables or a call to other functions in the same program, 4: ecx = eax + 1
or to external functions (library or system routines). An val posthIi (ebx ) 6: eax = ecx - 1
IR expression uses arithmetic, bit-vector, and relational = val prehIi (ebx )
B
operators, as well as the special memory addressing op- T6 → T4
erator. 4: ecx = eax + 1
val posthIi (ebx ) 6: eax = ecx - 1
term :: expr1 ASSIGN expr2 = val prehIi (ebx )
| CALL expr A 2: ebx = 0x400000
T1 → T5 3: nop
expr :: expr1 op expr2 4: ecx = eax + 1
val posthIi (eax ) 6: eax = ecx - 1
| ident ( expr, . . . )
= val prehIi (eax ) 7: mem[ebx] = mem[eax] ˆ 5
| memory [ expr ]
A
| ident | number | string T5 → T5 9: ebx = ebx + 1
4: ecx = eax + 1
op :: OR | AND | XOR | NOT | BITAND val posthIi (eax ) 6: eax = ecx - 1
| BITOR | BITXOR | BITNOT = val prehIi (eax ) 7: mem[ebx] = mem[eax] ˆ 5
| EQUAL | LESSTHAN B 3: nop
T2 → T6 4: ecx = eax + 1
| GREATERTHAN | LEQ | GEQ
6: eax = ecx - 1
| BITSHIFTLEFT | BITSHIFTRIGHT val posthIi (ebx ) 7: mem[ebx] = mem[eax] ˆ 5
| PLUS | MINUS | STAR | DIV | MOD = val prehIi (ebx ) 8: eax = eax + 1
B
T6 → T6 4: ecx = eax + 1
Table 6. Term algebra for intermediate rep- 6: eax = ecx - 1
resentation (IR) expressions. 7: mem[ebx] = mem[eax] ˆ 5
val posthIi (ebx )
= val prehIi (ebx ) 8: eax = eax + 1

Table 7. Def-use paths, value predicates,

C. Def-use paths and value predicates and program paths for the example in Fig-
ure 3.
We list here the complete set of value predicates
and corresponding program fragments that have to be