Proceedings of 2014 Zone 1 Conference of the American Society for Engineering Education (ASEE Zone 1)

The Cost of Preventing a Buffer Overflow

Dr. Anatoliy S. Gordonov

Abstract - In the paper we have considered the main methods

of buffer overflows, mitigation strategies, and their influence on II. THE PROBLEM
the memory consumption. The analysis of various methods of As it was mentioned above, buffer overflows may be used
stack protection has given us an estimate of the additional in different flavors. Some of the exploits of program
memory required for the implementation of specific techniques.
vulnerabilities are more complex than just a buffer overflow,
The size of the additional memory depends on many factors
including computer architecture, OS environment, programming
but still based on it. Several types of buffer overflows are
languages used to create the program. For the protection known. They are also called buffer overflow generations [1].
methods considered in the paper, the cost may vary from the
insignificant amount for prevention purposes, based on the The first generation of buffer overflows (stack smashing) is
careful analysis of input data in the program, to the use of Guard described in [2]. This attack is based on the fact that a buffer
Pages when extra memory may include additional pages of the in a procedure is allocated in the stack where the return
memory. In many cases developers have to use various address is saved. Writing to the buffer more items than it can
mitigation strategies in order to make programs less vulnerable
to buffer overflows. The main contribution of this paper is the
hold will overwrite adjacent to the buffer areas in the stack
analysis and evaluation of the additional memory required for including the return address. If an attacker were able to
the various methods of protection from buffer overflow. The submit an executable code to the buffer and change the return
current paper allows readers to understand the cost of these address to the address of the beginning of this code, the code
methods more clearly, which, in turn, will result in more efficient would be automatically executed as soon as the function
and secure programs. The results of this paper are useful for executes the return command.
both software developers and the instructors who teach methods
of secure programming. One of the variations of the traditional stack overflow
attack can be used in the Windows environment, and it is
Index Terms- buffer overflows, mitigation strategies, memory based on exploiting Windows structured exception handling
consumption (SEH) [5]. This mechanism allows applications to register a
handle to act on errors. Then, if an error has occurred, an
I. INTRODUCTION application has a chance to catch an exception and recover.
Buffer overflow attacks are the most common types of All handlers are connected together into the chain, and this
intrusion attacks today. A buffer overflow is a software bug chain is traversed to find an exception with the required code.
that allows data to be copied to the locations in the memory, The chain is created for each thread and is allocated in the
which are positioned beyond the boundaries of the original stack area. The compiler reserves the space in the stack for
buffer, corrupting adjacent data or instructions. Not every local variables, which are immediately followed by the
buffer overflow leads to program vulnerability. However, it exception handler address. This gives the ground for exploit:
can be used by malicious users to get control over the by overflowing a local buffer, the address of exception can be
computer system. The possibility to exploit the buffer changed to the code submitted by the attacker.
overflow in the specific program is not an easy task and The second generation of buffer overflows is related to the
depends on many factors. These factors include computer mistake (called off-by-one) that can frequently be found in
architecture, OS environment, programming languages used to programs. By “one” we mean an element of an array. For
create the program, and, of course, the qualification of the example, very often, this error can be found in loop operators
attacker. One of the widely used buffer overflow variants is a where elements may start from 0 instead of 1, or by comparing
stack overflow, which can be exploited in computers that the end of the loop with “<=n” instead of “< n”. Changing
support stacks. Some runtime environments for languages, one element beyond the boundaries of the buffer will change
such as Java, Pascal, and C#, detect buffer overflows and the element that is located in the stack next to the end of the
generate an exception. However, very popular programming buffer. This can be saved “state information” that includes the
languages C and C++ that are widely used in programming saved frame pointer (EBP) and return address (EIP). In this
development do not have this kind of detection. This allows case, EBP would be changed, and EIP would most likely stay
the attackers to exploit stack overflows to get control and the same. The number of bytes to be changed in EBP depends
change the execution of the program. on the data type written into the buffer. The saved frame
pointer value is changed. The new EBP refers to another
location in the overwritten buffer where the dummy stack
frame has been created. This dummy stack frame usually has
a return address pointing to the shell code in the same buffer.

978-1-4799-5233-5/14/$31.00 ©2014 IEEE

This type of attack is indirect and can be used when only a be fully implemented in practice because of “the human
limited number of the bytes in the stack can be overwritten factor” involved. Developers make errors, and this source of
[3]. exploitable software flaws will not cease to exist. That does
The third generation of buffer overflows is Arc Injection [1, not allow us to use this strategy as the main instrument of
3, and 4]. Two types of buffer overflows considered above solving the input validation problem and requires using it
(namely, stack smashing and off-by-one) are based on together with other protection mechanisms, such as manual
overwriting part of the stack with a code specially created by secure code audits, automatic software tests of the source code
the attacker and passing control to this code. These type of (which are faster and more cost-effective than manual
attacks are not possible if the computer system has some form inspections), and binary audits (in cases when a source code is
of memory protection (for example, W^X mechanism, which not known).
prevents any area from being simultaneously writable and The use of the methods mentioned above can help find
executable). In this case, an attacker tries to find a way to run many bugs, but cannot provide a completely safe solution.
a code, existing in the memory, with specially prepared One of the examples is a situation when a technique used for
arguments. In many instances, this code is a standard library the attack is not known. In this case we can use techniques
function that already exists in the memory. Arc injection is that eliminate the flaws by reducing the damage to a program.
also called return-into-labc (where the labc is a standard Many types of stack-based buffer overflows can be
UNIX or LINUX system library) or Return to System Call. eliminated by using wrapping unsafe library functions.
The name Arc injection comes from the fact that the attack Libsafe [8] offers an approach of using dynamically loadable
includes a new arc (transfer of control) in the program flow libraries that catch and substitute calls to vulnerable library
chart but not a new node with a code. In many operating functions. Being very useful for eliminating various types of
systems, the standard C library is loaded for most processes at stack-based overflow attacks, unfortunately, this method does
a well-known address and allows an attacker to find the not make it possible to cover all potential unsafe library
addresses of the functions. A more advanced method of Arc functions.
injection allows an attacker to execute a chain of library calls, Another approach utilizes the compiler extensions with
one after another. specific safety mechanisms. A well-known extension is
All buffer overflow types mentioned above (as well as some StackGuard [9], which enhances the executable code produced
others not considered here) provide a basis for many security by a compiler so that it detects buffer-overflow attacks against
attacks. Various mitigation strategies used to prevent and stop the stack. To do so, StackGuard has to detect that the return
these attacks are known. All of them have their own price in address has been altered before the function returns. It does
terms of program performance. But if the evaluation of the this by placing a “canary” word (or a stack cookie) next to the
extra time required for the execution of additional security return address on the stack. StackGuard as well as Stack-
mechanisms may be found in literature, there is almost no Smashing Protector ”ProPolice” have been developed for
information about the extra memory required for this. Below, UNIX/LINUX systems. The stack buffers overrun detection
we are going to analyze the cost of different protection has also been developed for MS Visual Studio. The code
mechanisms from the point of view of memory usage. used for canaries may be predefined or randomly generated.
III. ANALISIS OF DIFFERENT MITIGATION A predefined canary may include four different termination
STRATAGIES characters (CR, LF, NULL, and -1) that would guard against
There are several directions of mitigation strategies: buffer overflows caused by an unbounded strcpy( ) call. The
prevention, detection, and recovery. The general runtime main disadvantage of the predefined canary is that it is known
protection strategies may be classified [6] by the various to the attacker. To avoid this, random generated canaries are
participants of the development process that provide these used. They are hard-to-spoof and are changed each time the
mechanisms: developers, compilers and associated runtime program is executed. Figure 1a shows the ordinary function
systems, the operating systems. There are also classifications stack frame, and figure 1b shows the same frame with the
of mitigation techniques [1, 7] that involve eliminating the added canary.
cause of buffer overflows or dealing with alleviating the
impact of buffer overflows by fixing the surrounding of a
vulnerable program.

The best way of protection is to prevent buffer overflows from

occurring. It can be done, for example, by carefully analyzing
the length of the input in order to prevent overflows. This
analysis, included in the program by a developer, introduces
some additional insignificant memory overhead, which is
definitely a good price for the protection from buffer
overflows. Data validation requires that the developers
correctly identify and check all the external inputs. Being
very “simple” and obvious, this mitigation strategy is hard to Fig. 1. Allocating a canary in the stack
The use of canaries includes the following operations: 01391441 8B EC mov ebp,esp
01391443 81 EC F8 00 00 00 sub esp,0F8h
Before calling a procedure:
01391449 53 push ebx
-generate a random canary; 0139144A 56 push esi
-save it in global variables; 0139144B 57 push edi
-save it into the stack immediately after the saved return 0139144C 8D BD 08 FF FF FF lea edi,[ebp-0F8h]
01391452 B9 3E 00 00 00 mov ecx,3Eh
address and saved frame pointer but before the local variables
01391457 B8 CC CC CC CC mov eax,0CCCCCCCCh
in each stack frame; 0139145C F3 AB rep stos dword ptr es:[edi]
Before returning from a procedure: 0139145E A1 00 70 39 01 mov eax,dword ptr [___security_cookie
-compare the canary from the stack with the saved value; (1397000h)]
01391463 33 C5 xor eax,ebp
-if they do not match, exhibit the exception code; 01391465 89 45 FC mov dword ptr [ebp-4],eax
-exit the program if the changing of the stack area is found, 17: // declare buffer that is bigger than expected
18: char large_buffer[] = "This string is longer than 10 characters!!";
or return back from the procedure. 01391468 B9 0A 00 00 00 mov ecx,0Ah
All these steps will be implemented for every procedure call 0139146D BE 3C 57 39 01 mov esi,offset string "This string is ]
longer than 10 ch"... (139573Ch)
and introduce both time and memory increase. Below is an 01391472 8D 7D CC lea edi,[ebp-34h]
example of the program, which produces a stack overflow. 01391475 F3 A5 rep movs dword ptr es:[edi],dword ptr
[esi]
01391477 66 A5 movs word ptr es:[edi],word ptr [esi]
01391479 A4 movs byte ptr es:[edi],byte ptr [esi]
19: vulnerable(large_buffer);
// compile with: /c /W1 0139147A 8D 45 CC lea eax,[ebp-34h]
#include <cstring> 0139147D 50 push eax
#include <stdlib.h> 0139147E E8 5E FC FF FF call vulnerable (13910E1h)
#pragma warning(disable : 4996) // for strcpy use 01391483 83 C4 04 add esp,4
20: }
// Vulnerable function 01391486 33 C0 xor eax,eax
void vulnerable(const char *str) { 01391488 52 push edx
char buffer[10]; 01391489 8B CD mov ecx,ebp
strcpy(buffer, str); // overrun buffer !!! 0139148B 50 push eax
0139148C 8D 15 B8 14 39 01 lea edx,[ (13914B8h)]
// use a secure CRT function to help prevent buffer overruns 01391492 E8 E6 FB FF FF call
// truncate string to fit a 10 byte buffer @ILT+120(@_RTC_CheckStackVars@8) (139107Dh)
// strncpy_s(buffer, _countof(buffer), str, _TRUNCATE); 01391497 58 pop eax
} 01391498 5A pop edx
int main() { 01391499 5F pop edi
// declare buffer that is bigger than expected 0139149A 5E pop esi
char large_buffer[] = "This string is longer than 10 characters!!"; 0139149B 5B pop ebx
vulnerable(large_buffer); 0139149C 8B 4D FC mov ecx,dword ptr [ebp-4]
} 0139149F 33 CD xor ecx,ebp
013914A1 E8 6E FB FF FF call
Fig. 2. Example of the program with a buffer overflow. @ILT+15(@__security_check_cookie@4) (1391014h)
013914A6 81 C4 F8 00 00 00 add esp,0F8h
013914AC 3B EC cmp ebp,esp
We compiled this program with and without /GS flag and 013914AE E8 83 FC FF FF call @ILT+305(__RTC_CheckEsp)
(1391136h)
Stack Frame Check set, then, compared the sizes of the 013914B3 8B E5 mov esp,ebp
assembly codes. Figure 3 shows an assembly listing with both 013914B5 5D pop ebp
/GS flag and Stack Frame Check having been set. The 013914B6 C3 ret
compiler includes extra operations that are intended to protect Fig. 3. Assembly code of the part of the program from
the stack by analyzing if the previously saved canary has been figure 2.
unchanged. The corresponding assembly instructions are As we can see from the above, stack canaries (or “stack
highlighted in figure 3. The program compiled without setting cookies” as they are also called) allow the program to detect
/GS flag and Stack Frame Check does not include buffer overflows in the stack. In terms of additional memory
highlighted instructions and is 68 bytes smaller. This gives us consumption, using stack canaries is not very expensive. It
an idea how much the stack protection costs in terms of depends on the program structure and for a program with
memory consumption. It is important to note that this amount many procedures may add a notable amount of memory.
of bytes is added to every procedure call in the application. Obviously, this mechanism can work as long as the canary
So, for instance, in our small example, this amount of memory value remains secret. Unfortunately, using stack canaries has
will be doubled. In general, about 70 bytes will be added to limitations. Neither the predefined nor a random canary offers
every procedure call in an application. For big applications, complete protection against exploits that overwrite the return
the memory cost may be noteworthy. address. Canaries do not protect against exploits that modify
variables adjacent to the buffer, data pointers, or function
pointers. Canaries cannot prevent buffer overflows from
occurring. They can only help detect buffer overflows after
16: int main() { they have already occurred. Neither are they able to prevent
01391440 55 push ebp
changing the return address in the situations when an attacker the cost may vary from an insignificant amount for the
has managed to write directly to the location of the return prevention purposes, based on the careful analysis of the input
address in the stack. data in the program, to the use of Guard Pages when extra
The goal of buffer overflows is to load an executable code memory may include many additional pages of the memory
(very often called shell-code) into the memory, overwrite the for each process. Not only does this memory increase the
return address to point to the shell-code, and pass control to it. space needed for an application, but it also affects the
One of the major challenges that hackers meet is finding the execution time. In many cases developers have to use various
beginning of the loaded shell-code. Many stack-based buffer mitigation strategies in order to make programs less
overflow exploits rely on the fact that the stack is allocated at vulnerable to buffer overflows. The main contribution of this
the known address in the memory. Being able to overwrite the paper is the analysis and evaluation of the additional memory
function return address, an attacker can run the shell-code. required for the various methods of protection from the buffer
Changing the address of the buffer in the memory by inserting overflow. The current paper allows us to more clearly
randomly sized space gaps before allocating memory to the understand the cost of these methods, which, in turn, will
stack of a thread can make it much more difficult for the result in more efficient and secure programs
intruder to find the return address in the stack. This allocated TABLE I
random space is called StackGap or Guard Page. The Random PAGE SIZE FOR DIFFERENT COMPUTER
Guard Page offsets the beginning of the stack, and it does not ARHITECTURES
allow the attacker to predict the stack address from one run of Architecture Page Size Large Page Size
a program to another: (KB)
-The Guard Page insertion is done by the operating system. i386 4 4M
Each Guard Page is flagged as illegal space and cannot be x86-64 4 2M, 1G
accessed without a process being aborted. The interesting IA-64 (Itanium) 4 8K, 64K, 256K, 1M,
question here is how much it costs in terms of additional 4M, 16M, 256M
memory. Each Guard Page has one memory page upper limit. Sparc 8 64K, 4M, 256M, 2G
-If the Guard Page is allocated one per thread, the size of
this page is the upper limit of the price that we pay. Table 1 REFERENCES
[1] Matthias Vallentin ” On the Evolution of Buffer Overflows”, 2007,
shows the examples of various page sizes for different
http://matthias.vallentin.net/course-work/buffer_overflows.pdf
computer architectures. [2] Aleph One. Smashing the stack for fun and profit. Phrack, 49–14,
-The standard page size is 4K (8K for Sparc). November1996. Available from
Nevertheless, as we see from table 1, much bigger pages can http://www.phrack.org/issues.html?issue=49&id=14
[3]. W. Stallings. L. Brown, “Computer Security”, Prentice Hall, 2008, ISBN-
be used which will considerably increase the memory 13: 978-0-13-600424
consumption. This may happen when threads allocate large [4] J. Pincus, B. Baker “Beyond Stack Smashing: Recent Advances in
data structures on stack and a large guard area may be needed Exploiting Buffer Overruns”, IEEE Security and Privacy, July/August 2004.
[5] M. Down, J. McDonald, J. Schun, “The Art of Software Assessment”,
to detect the stack overflow. Addison Wesley, 2007, ISBN 13:978-0-321-44442-4
-The application that creates a large number of threads will [6] R. Seacord “Secure Coding in C and C++”, Addison Wesley, 2013, ISBN-
include a Guard Page for each thread stack, which potentially 13:978-0-321-82213-0.
[7] David Evans and David Larochelle. Improving security using extensible
results in the wasted system resources.
lightweight static analysis. IEEE Software, 19(1):42–51, 2002.
-Including Guard Pages between stack frames makes this [8] A. Baratloo, N. Singh, and T. Tsai. Libsafe: Protecting critical elements of
mitigation strategy much stronger because it introduces more stacks, 1999.
difficulties for an attacker to find absolute addresses in the http://www.orkspace.net/secdocs/Unix/Protection/Description/Libsafe%20-
%20Protecting%20Critical%20Elements%20of%20Stacks.pdf
stack. But, on the other hand, the price becomes too big. [9] Crispan Cowan, Calton Pu, Dave Maier, Jonathan Walpole, Peat Bakke,
Guard Pages are typically used to prevent buffer overflow Steve Beattie, Aaron Grier, Perry Wagle, Qian Zhang, and Heather Hinton.
attacks on global data (such as global offset table). StackGuard: Automatic adaptive detection and prevention of buffer-overflow
attacks. In Proc. 7th USENIX Security Conference, pages 63–78. San Antonio,
Unfortunately, Guard Pages are not the complete solution to Texas, Jan 1998.
prevent buffer overflows. For example, they cannot prevent
exploits if an attacker is able to use relative addresses.

IV. CONCLUSION
In the paper we have considered the main methods of buffer
overflows, mitigations strategies, and their influence on the
memory consumption. The analysis of various methods of
stack protection has given us an estimate of the additional
memory required for the implementation of specific
techniques. The size of the additional memory use depends on
many factors including computer architecture, OS
environment, programming languages used to create the
program. For the protection methods considered in the paper,