454

Buffer-Overflow Protection: The Theory

Krerk Piromsopa, Member, IEEE, and Richard J. Enbody, Member, IEEE

done on any data and variables. This paper focuses on buffer-

Abstract- We propose a framework for protecting against overflow attacks on control data, but is not limited to control
buffer overflow attacks-the oldest and most pervasive attack data.
technique. The malicious nature of buffer-overflow attacks is the The goal ofthis paper is to provide a generic framework for
use of external data (input) as addresses (or control data). With
this observation, we establish a sufficient condition for preventing buffer-overflow attacks. We begin by defining
preventing buffer-overflow attacks and prove that it creates a buffer overflows in general. Next, we establish a sufficient
secure system with respect to buffer-overflow attacks. The condition for preventing buffer-overflow attacks and prove
underlying concept is that input is untrustworthy, and should not that it will create a secure system with respect to buffer-
be use as addresses (return addresses and function pointers.). If overflow attacks.
input can be identified, buffer-overflow attacks can be caught.
We used this framework to create an effective, hardware, buffer-
overflow prevention tool.
II. BACKGROUND
Index Terms-Buffer overflow, Buffer-Overflow Attacks, This section is intended to be a gentle introduction to buffer
Computer security, Function-Pointer Attacks, Intrusion overflows
Detection, Intrusion Prevention
A. Fundamental ofBuffer-Overflow Attacks
First, we define buffer overflow(from the Webopedia [18]).
I. INTRODUCTION
W5 IE create a theoretical foundation for a secure system Definition 1:
with respect to buffer-overflow attacks. The goal is to A buffer overflow is the condition wherein the data
use the foundation to provide a framework for implementing transferred to a buffer exceeds the storage capacity of the
buffer-overflow protection. buffer and some of the data "overflows" into another buffer,
Although they date back to the infamous MORRIS worm of one that the data was not intended to go into.
1988 [3], buffer-overflow attacks remain the most common.
Though skilled programmers should write code without buffer Since buffers can only hold a specific amount of data, when
overflows, no program is guaranteed free from bugs so it that capacity has been reached the data has to flow somewhere
cannot be considered completely secure against buffer- else, typically into another buffer, which can corrupt data
overflow attacks. The persistence of buffer-overflow already in that buffer.
vulnerabilities speaks to the difficulty of eliminating them. In Exploiting buffer overflow can lead to a serious system
addition, as buffer overflow vulnerabilities are eliminated in security breach (buffer-overflow attack) when necessary
operating systems, they are being found and exploited in conditions are met. The seriousness of buffer-overflow attacks
applications. When applications are run with root or ranges from writing into another variable, another process'
administrator privileges the impact of a buffer overflow is memory (segmentation fault), or redirecting the program flow
equally devastating. to execute malicious or unexpected code. Based on the
definition of buffer overflow, Definition 2 defines buffer-
In an effort to avoid relying on individual programming overflow attacks.
skill, a number of researchers have proposed a variety of
methods to protect systems from buffer-overflow attacks. Definition 2:
Most of them are not able to provide complete protection. For A buffer-overflow attack is an attack that (possibly
example, some only prevent the original stack-smashing implicitly) uses memory-manipulating operations to overflow
attack, but can be circumvented by more recent attacks. a buffer which results in the modification of an address to
It is worth clarifying that buffer-overflow attacks can be point to malicious or unexpected code.
In general, a buffer-overflow attack is an attack on any data
K. Piromsopa is a Ph.D. student in the Department of Computer Science (including variables and addresses). To make this paper
and Engineering, Michigan State University, East Lansing, MI 48824 USA
(corresponding author to provide phone: 517-353-3148; fax: 517-432-1061; e- readable, the term "buffer-overflow attacks" is used to refer to
mail: piromsop 0cse.msu.edu). attacks on control data.
R. J. Enbody is with the Department of Computer Science and
Engineering, Michigan State University, East Lansing, MI 48824 USA. (e-
mail: enbody a,cse.msu.edu). Observation: An analysis of buffer-overflow attacks
 455
indicates that a buffer of a process is always overflowed with a All execution flow
Exceptioiis Fiulction1
buffer passed from another domain (machine, process) hence Fuictioiis Return©
its malicious nature. 7i r t,i in I n Pt-t harlvs A r l r,- mrs

Assumption 1:
In an attack, a buffer is always overflowed using a buffer
passed from another domain.

This concept is not new. For example, Howard and LeBlanc v:oid ftiiic(ifhar ' p, ilmt, i)

state in their book "All input is evil until proven otherwise" I '~~~~~~~~1 CDiiitjy=IOiimy
i* local wariabule*
[9]. Accordingly, an intuitive way to prevent buffer-overflow
attacks is to detect and validate input, especially input which is . ___ ________ ___ _ ___ ___
UlYllYlydL
b11211X
char b1)2b1,p
eventually used for control. r,h,py bIp;

In early attacks, the attacked address was a return address,

but later other control data (e.g. function pointers, jump table)
were attacked. In all cases, the eventual access of that address
Figure 1 Stack Smashing
(e.g. by a return, function call or jump) will redirect the
program control flow to execute the malicious or unexpected
code. If the address was modified by something other than a To illustrate an advanced buffer-overflow attack, we
buffer overflow, it is a race condition, a Trojan horse, or other provide a multistage buffer-overflow attack [10] (a.k.a.
type of attack. Hannibal Exploit [6]) that can bypass most software buffer-
overflow solutions. Fundamental to a multistage buffer-
overflow attack is that there exists a vulnerable pointer to a
B. Sample Attacks and Variations buffer. That is, there is a user-writeable buffer sufficiently
There are two main targets of buffer-overflow attacks: near a useful pointer. First, the pointer is modified (by
control data and local variables. In the vast majority of attacks, overflowing) to point to a specific location (e.g. a jump slot or
control data is the target so prevention schemes have focused a function pointer). In the second stage of the attack, an input
on control data. Control data can be divided into several types: is stored at the pointer's target. These two steps allow
return addresses, function pointers, and branch slots. Return attackers to create a pointer to any location (first stage) and
addresses have been the primary target since their location can overwrite the pointer's target with a desired value (second
easily be guessed. More advanced buffer-overflow attacks stage). The next time some program jumps to that target, it
target other control data. Some literature refers to attacks on will be redirected based on the value inserted by the attacker.
return addresses as first-generation attacks, and those on In particular the program will be redirected to the attacker's
function pointers as second-generation attacks [5]. malicious code. For example, if the program is running in
The first step-by-step description of how to construct a privileged mode and the pointer points to shell code, the
buffer-overflow attack was written by Elias Levy (a.k.a. Aleph attacker will have created a privileged shell allowing free
One) in 1996 [ 1]. He first used the term "Stack Smashing" to reign. Figure 2 is an example of such a vulnerable program.
refer to plastering the stack with shell code and its address to Before examining the code, let's review how a jump table is
set up an eventual overflow of a return address on the stack. used. Consider the slot in the table for the pointer to the printf
executable. A call to printf indexes to that slot in the table and
Figure 1 shows a stack-smashing example. In this example, then jumps to the printf executable. To attack this type of
an attacker (shown in the black hat) passes a buffer containing program, the buffer-overflow is done in two stages. First, the
malicious code (e.g. shell) and multiple copies of the address ptr pointer is overflowed to point to a desired memory
of the target buffer as an argument to the vulnerable program location (1), e.g. the printf slot in the jump table. In particular,
(through parameter p). A buffer manipulation function (e.g. argv[1] controlled by the attacker will contain the address of
strcpy in this example) will overflow the function's return the printf slot in the jump table. The strcpy routine will copy
address with the address of the target buffer containing argv[1] into buffer, but overflows to overwrite ptr with the
malicious code. The eventual result is that the return printf address slot. In the figure, we see that the pointer ptr
instruction will use the address of the target buffer as a return originally pointed to 'buffer' (arc labeled 'Before') but now
address and return the program flow to execute that malicious points to the jump slot (arc labeled 'After). Now that ptr
code. Additional attack vectors are provided (but not used) in points to the printf slot in the jump table, we need to insert a
the code such as function pointer fp. As outlined above, it is desired value into that slot. Suppose, for illustration, that we
important to note that the critical component of this attack is also have determined the address of resident shell code (we'll
the modification of the return address (replaced by the address call it residentcode). Using our modified ptr we will overwrite
of the target buffer). the jump table slot with the residentcode address. We use the
second strcpy call (2) to write argv[2] (also controlled by the
attacker and whose value is residentcode) into the target ofptr
 456
which now points to the printf entry in the jump table. The
result of that second strcpy call is that we have placed the
address residentcode (resident shell code) into the printf slot
of the jump table. The attacker has achieved his goal. Now
when a program calls printf, control passes as usual to the
printf entry in the jump table, but now the attacker has
redirected control to residentcode, the address of the shell
code. Instead of printf a shell will be started. If the program
which called printf was operating in privileged mode, the
attacker will have succeeded in creating a privileged shell with
full system access. (See [13] for more details). As we will see
below, this multi-stage attack gets around most buffer-
overflow protection schemes. Less obvious is that we can use
a similar approach to circumvent some software solutions to
buffer-overflow attacks by modifying a handling vector which Figure 3 Buffer-Overflow Attacks on Pointers
can allow us to bypass the buffer-overflow handling routine.
Recently "integer overflows" (known more generically as
Before * After
"integer arithmetic" attacks) have emerged as a variation [2].
In these attacks guard data which protects buffers is attacked.
After defeating the guard data some buffer-overflow attack is
,-o used. Therefore, these attacks can be considered as a variation
of a buffer-overflow attack which the more robust schemes
can protect against.
An attack on local variables is exemplified by the classic
password attack from a 1987 paper [19]. Basically, a variable
is being overflowed allowing an arbitrary password to be
validated resulting in root or administrator access. No control
data is involved so no existing buffer-overflow protection
schemes protect against this type of attack.
Figure 2 An example of a vulnerable program
Another variation which most schemes cannot protect is an III. PREVENTION
attack on generic pointers. By modifying both the source and This section discusses the necessary conditions for
destination pointers, copying from one arbitrary memory preventing buffer-overflow attacks on control data.
location to another is possible. Figure 3 illustrates an attack
based on attacking pointers. In this figure, the first strepy will Postulate 1:
allow attackers to overflow buffer b so the src and des pointers In buffer-overflow attacks on control data, the generic
are replaced with two pointers of the attacker's choice: in this bufferememory-manipulating operations are used by the
case, valid control data and target address respectively. The vulnerable routine to overflow the address (e.g. a return
second strepy will then copy from the src location to the des address or a function pointer).
location. Using this approach, it is possible to modify any
memory entry with a known local entry while avoiding most From Definition 2, we observe that preserving the integrity
protection mechanisms. For example, an encoded function of the address is a sufficient condition to prevent this class of
pointer can be used to attack another function pointer (e.g. buffer-overflow attacks. To clarify, Definition 3 shows the
jump slot). meaning of the integity of an address in this context.
A related attack is worth noting here: printf vulnerabilities Defi'nition 3:
[17]. Malformed formatting instructions can allow arbitrary Maintaining the integrity of an address means that the
memory to be overwritten. It is not a buffer-overflow attack, address has not been modified by overflowing with a buffer
but with the ability to overwrite arbitrary memory the attack passed from another domain.
then proceeds like many buffer-overflow attacks by attacking
control data. Some buffer-overflow schemes can prevent some Consider the implication of Definition 3 in light of our
of those attacks. However, any variable can also be attacked, "Observation" about Definition 2 which noted the importance
and no buffer-overflow scheme protects arbitrary data. of attacks working across domains (machines, processes): in
Fortunately, simple static analysis of source code can identify order to preserve the integrity of the address (e.g. a return
printf vulnerabilities. As we will see below, static analysis is address or a function pointer), an address cannot be created
not sufficient for buffer overflows.
 457
from data passed across domains (e.g. machines, processes) fails. If modification of the address can be recognized, the
via buffer overflow. buffer-overflow attack can be recognized and stopped. On the
To maintain its integrity, the address created locally can be other hand, if the address can be validated, execution can
signed when it is created and is validated by associated proceed safely.
instructions (e.g. return, call, and jump instructions) before
they are completely executed. Implicitly, a signature
represents some metadata associated with the address. IV. SECURE SYSTEM
Necessarily, the signature must not be passed across domains. To claim that this framework can enforce the integrity of
If the signature could be passed across domains, a valid the addresses and result in a secure system, a validation will
address could be used for attacking a system. If we assume be discussed. Assuming that a computer system can be
that a signature only exists locally, the last condition is represented as a finite-state automation with a set of transition
enforced when a buffer is passed across a network/hardware functions, we can define a secure system (with Definitions 4
device where the signature cannot be passed. and 5).
If local data can be differentiated from data passed from
another domain, we can detect buffer-overflow attacks on Definition 4:
control data. Thus we may reverse the signature by signing A security policy is a statement that partitions the states of
data that passed across domains and leave the local data the system into a set of authorized, or secure, states and a set
unsigned. This scheme provides better backward- of unauthorized, or insecure, states [1].
compatibility, since no modification is required for legacy
processes. In the case of buffer-overflow attacks, the security policy is
With these definitions, Theorem 1, and its corollary are simply the statement:
introduced. The corollary is the key to the entire framework
presented in this paper since it defines a sufficient condition Protocol 1:
for buffer-overflow attacks. Overflowing a buffer cannot create a valid address
(e.g. a return address or a function pointer)
Theorem 1:
Modifying an address by replacing ("overflowing") it using which follows from Corollary 1.1. Before going further, we
a buffer passed from another domain is a necessary condition first define a secure system.
for a buffer-overflow attack on control data.
Definition 5:
Restatement: If there is to be a buffer-overflow attack on A secure system is a system that starts in an authorized
control data, an address must be modified using a buffer state and cannot enter an unauthorized state. [1].
passed from another domain.
Theorem 2:
Proof: A system which preserves the integrity of an address (e.g. a
Theorem 1 follows directly from Definitions 1 and 2. return addresses or a function pointer) is a secure system with
QED respect to buffer-overflow attacks.
Corollary 1.1:
Preserving the integrity of an address is a sufficient Restatement: A system that does not use input as a control
condition for preventing a buffer-overflow attack. data is a secure system with respect to buffer-overflow attacks
on control data.
Restatement: If the integrity of an address is preserved, that
is a sufficient condition for preventing a buffer-overflow Proof:
attack. Assume that a system is partitioned into two states: normal
operation and buffer-overflow attack. By the definition of
Proof: buffer-overflow attacks (Definition 2), only overwriting the
From Theorem 1, "If there is to be a buffer-overflow attack, address (e.g. a return address or a function pointer) with an
an address must be modified by manipulating a buffer from address passed as a buffer (input) to vulnerable programs will
another domain." The contrapositive of that statement is "If an result in the state of buffer-overflow attack. By the definition
address cannot be modified (or such modification can be of preservation of the address (Definition 3), if such
detected), then a buffer-overflow attack is not possible." We overflowing can be recognized and prevented, the system will
know that the contrapositive of a true statement is true. not result in the state of buffer-overflow attacks. With respect
QED to Definition 5, our system cannot enter an unauthorized state
Intuitively, from Definition 2, the attack is the ability to and is considered to be a secure system.
redirect the program flow to execute malicious or unexpected QED
code. To achieve this goal, the address must be modified. If
the address cannot be modified, the buffer-overflow attack
 458
addresses (return addresses and function pointers) with a
buffer passed from another domain (machine, and process). In
1. Overflow the a in
Normal
another domain this paper we developed a formal argument that "a necessary
Operation condition for preventing buffer-overflow attacks is the
preservation of the integrity of addresses across domains".
We then show how a protocol based on that statement
supports a variety of successful hardware-based methods to
Figure 4 State diagram of buffer-overflow attacks. prevent buffer overflow attacks. Our formalism lends credence
to their claims of success.
Hitherto, we will show that the enforcement of Protocol 1
(stated early) results in a secure system with respect to buffer-
overflow attacks. REFERENCES
[1] M. Bishop, Computer Security, Addison-Wesley, (Dec. 2002)
Theorem 3: [2] Blexim, "Basic Integer Overflow," 2002. Available:
A system that enforces Protocol 1 can preserve the integrity
[3] C. Schmidt, and T. Darby, "The What, Why, and How ofthe 1988
of an address, and result in a secure system with respect to Internet Worm,"
buffer-overflow attacks. [4] S. Chen, J. Xu, N. Nakka, Z. KalbarcZyk, and R. K. Iyer, "Defeating
Memory Corruption Attacks via Pointer Taintedness Detection," Proc.
Of IEEE International Conf on Dependable Systems and Networks
Proof: (DSN), Yokohama, Japan, June 28 - July 1, 2005
By enforcing Protocol 1, we can detect that an address (e.g. [5] E. Chien, and P. Szor, "Blended Attacks Exploits, Vulnerabilities and
a return address or a function pointer) is overflowed by a Buffer-Overflow Techniques in Computer Viruses," Proc. of Virus
Bulletin Conf, 2002
buffer passed from another domain (including input). If we [6] J.R. Crandall, and F.T. Chong, "Minos: Control Data Attack Prevention
can detect that an address is modified by a buffer from another Orthogonal to Memory Model," Intl. Sym. on Microarchitecture, 2004.
domain, we can preserve the integrity of the address. This [7] J.R. Crandall, and F.T. Chong, "A Security Assessment of the Minos
follows directly from Definition 3. Thus a system that can Architecture," ACM SIGARCH, Vol 33. No. 1, 2005.
[8] A. Glew, "Segments, Capabilities, and Buffer Overrun Attacks,"
enforce Protocol 1 preserves the integrity of the address and is Computer Architecture NEWS, ACM SIG Computer Architecture
a secure system with respect to buffer-overflow attacks. This Vol.3 1, No.4 - September 2003, pp. 26 - 31
follows directly from Theorem 2 [9] M. Howard, D. Leblance, Chapter 10:All Input Is Evil!. Writing Secure
Code, Microsoft Press, 2nd ed.(1965)
QED [10] S. Hsiangren, "Apache/mod_ssl (slapper) Worm," GIAC Certified
Incident Handler, 2002
[11] A. One, "Smashing stack for fun and benefit," Phrack Mag, 49(7), 1996.
[12] J. Pincus, and B. Baker, "Beyond Stack Smashing: Recent Advances in
V. IMPLEMENTATIONS Exploiting Buffer Overruns," IEEE Security & Privacy, Vol. 2, No. 4,
It was suggested that metadata is a key to preventing buffer- July/August 2004, pp. 20 - 27
[13] K. Piromsopa, and R. Enbody, "Buffer Overflow: Fundamental,"
overflow attacks in a 2003 article [8]. There exist at least four Technical Report #MSU-CSE-04-47, Dept. of Computer Science and
methods that follow our proposed framework by using Engineering, Michigan State University, 2004.
metadata for tracking input. These methods are Minos [6] [7], [14] K. Piromsopa, and R. Enbody, "Secure Bit2 Transparent, Hardware
Buffer-Overflow Protection," Tech.Report #MSU-CSE-05-9, Dept of
Tainted Pointer [3], Dynamic Flow Tracking [16] and Secure Computer Science and Engineering, Michigan State University, 2005.
Bit [14]. These methods share the same assumption that input [15] K. Piromsopa, and R. Enbody. "Survey of Buffer-Overflow
should not be used as control data. Protection,"Technical Reports #MSU-CSE-06-3, Department of
All methods required an additional bit augmented to each Computer Science and Engineering, Michigan State University, 2006.
[16] G. Suh, J. Lee, S. Devadas, "Secure program execution via dynamic
memory word (or byte). This bit is used for tracking input. information flow tracking," In ASPLOSXI (Oct, 2004.)
Whenever input is passed to a process, the bit is marked. Upon [17] U. Shankar, K. Talway, J.S. Foster, and D. Wagner, "Detecting Format
execution, Jump, Call or Return instructions validate that String Vulnerabilities with Type Qualifiers," In Proc. of the 10th
USENIXSecurity Symposium
control data is not derived from input. If input is about to be [18] Webopedia. What is buffer overflow?,
used as an address, an exception is raised. http://www.webopedia.com/TERM/B/buffer_overflow.html
However, the implementations differ. Minos views data [19] W.D. Young, "Coding for a Believable Specification to Implementation
Mapping," IEEE Symp on Security and Privacy 1987: pp. 140-149.
across segments as input. Tainted Pointer considers data
passed from the operating system as input. Dynamic Flow
Tracking relies on operating systems for marking input.
Secure Bit treats data passing between processes through the
kernel as input. More details of buffer-overflow attacks and
protection schemes can be found in [12] and [15] respectively.

VI. SUMMARY
Buffer-overflow attacks on control data require overflowing