Lecture 4 – Software Threats and

Vulnerabilities
CS Department
City University of Hong Kong

Slides partially adapted from lecture notes by M. Goodrich&R. Tamassia,

W. Stallings&L. Brown, Dan Boneh, and Dawn Song.
An Information Security Short Course
1
(Summer 2020)
 Why Software Security?
• You may have a perfect design, a perfect
specification, perfect algorithms, but still have
implementation vulnerabilities.

• After configuration errors, implementation errors are

probably the largest single class of security errors
exploited in practice.

• Flaws in software break certain assumptions

important for security

An Information Security Short Course

2
(Summer 2020)
 Operating Systems Concepts

An Information Security Short Course

3
(Summer 2020)
 A Computer Model
• An operating system has to deal with the fact
that a computer is made up of a CPU, random
access memory (RAM), input/output (I/O)
devices, and long-term storage.
0
1
2
3
I/O CPU 4
5 RAM Disk Drive
6
7
8
4
An Information Security Short Course 9
(Summer 2020)
.
 OS Concepts
• An operating system (OS) provides the interface
between the users of a computer and that
computer’s hardware.
– An operating system manages the ways applications access
the resources in a computer, including its disk drives, CPU,
main memory, input devices, output devices, and network
interfaces.
– An operating system manages multiple users.
– An operating system manages multiple programs.

An Information Security Short Course

5
(Summer 2020)
 Multitasking
• Give each running program a “slice” of the CPU’s time.
• The CPU is running so fast that to any user it appears that the
computer is running all the programs simultaneously.

An Information Security Short Course

6
(Summer 2020)
 The Kernel
• The kernel is the core component of
the operating system. It handles the
management of low-level hardware
resources, including memory, User Applications Userland
processors, and input/output (I/O)
devices, such as a keyboard, mouse, Non-essential OS
or video display. Applications
• Most operating systems define the Operating System

tasks associated with the kernel in The OS Kernel

terms of a layer metaphor, with the
hardware components, such as the CPU, Memory,
Hardware
Input/Output
CPU, memory, and input/output
devices being on the bottom, and
users and applications being on the
top.
An Information Security Short Course
7
(Summer 2020)
 Input/Output
• The input/output devices of a computer include things
like its keyboard, mouse, video display, and network
card, as well as other more optional devices, like a
scanner, Wi-Fi interface, video camera, USB ports, etc.
• Each such device is represented in an operating system
using a device driver, which encapsulates the details of
how interaction with that device should be done.
– The application programmer interface (API), which the
device drivers present to application programs, allows those
programs to interact with those devices at a fairly high level,
while the operating system does the “heavy lifting” of
performing the low-level interactions that make such devices
actually work.
An Information Security Short Course
8
(Summer 2020)
 System Calls
• User applications don’t communicate directly with
low-level hardware components, and instead
delegate such tasks to the kernel via system calls.
• System calls are usually contained in a collection of
programs, that is, a library such as the C library (libc),
and they provide an interface that allows applications
to use a predefined series of APIs that define the
functions for communicating with the kernel.
– Examples of system calls include those for performing file
I/O (open, close, read, write) and running application
programs (exec).
An Information Security Short Course
9
(Summer 2020)
 Processes
• A process is an instance of a program
that is currently executing.
• The actual contents of all programs are
initially stored in persistent storage,
such as a hard drive.
• In order to be executed, a program
must be loaded into random-access
memory (RAM) and uniquely identified
as a process.
• In this way, multiple copies of the same
program can be run as different
processes.
– For example, we can have multiple copies of
MS Powerpoint open at the same time.
An Information Security Short Course
10
(Summer 2020)
 Process IDs
• Each process running on a given computer is identified by a
unique nonnegative integer, called the process ID (PID).
• Given the PID for a process, we can then associate its CPU
time, memory usage, user ID (UID), program name, etc.

An Information Security Short Course

11
(Summer 2020)
 Memory Management
• The RAM memory of a computer is its address space.
• It contains both the code for the running program, its
input data, and its working memory.
• For any running process, it is organized into different
segments, which keep the different parts of the
address space separate.
• As we will discuss, security concerns require that we
never mix up these different segments.

An Information Security Short Course

12
(Summer 2020)
 Control Hijacking

Buffer-overflow and
Memory Safety

An Information Security Short Course

13
(Summer 2020)
An exploit is any input (i.e., a piece of software, an
argument string, or sequence of commands) that
takes advantage of a bug, glitch or vulnerability
in order to cause an attack

An Information Security Short Course

14
(Summer 2020)
 Control hijacking
• Attacker’s goal:
– Take over target machine
• Execute arbitrary code on target by hijacking application
control flow
– Webserver à server target
– Email reader , web browser à client target

• Examples.
– Buffer overflow attacks
– Integer overflow attacks
– Format string vulnerabilities
– Race condition vulnerabilities
An Information Security Short Course
15
(Summer 2020)
 Example 1: buffer overflows
• Extremely common bug in C/C++ programs.
– First major exploit: 1988 Internet Worm. fingerd. (a simple background
process on server)

# of vulnerable 600
software product
500 »20% of all vuln.
400 2005-2007: » 10%
300

200

100 Source: NVD/CVE

0
1995 1997 1999 2001 2003 2005
 What is needed
• Understanding C functions, memory manager, the stack, the heap.
• Know how system calls are made
• The execve() system call

• Attacker needs to know which CPU and OS used on the target

machine:
– Our examples are for x86 running Linux or Windows
– Details vary slightly between CPUs and OSs:
• Little endian vs. big endian (x86 vs. Motorola)
– Decimal 1000 (hexadecimal 3E8) in two bytes
» big endian 03 E8
» little endian E8 03
• Stack Frame structure (Unix vs. Windows)
An Information Security Short Course
18
(Summer 2020)
 structure that grows downwards
and keeps track of the activated
method calls, their arguments
Newly allocated variables and local variables
will stay below the previous
(existing) variables in stack.
e.g. libc

data dynamically generated

during the execution of a
process. e.g., malloc()
static program variables
initialized in the source code
prior to execution.

machine code of the program,

compiled from the source code 19
An Information Security Short Course
 Stack Frame
Every time a function call is executed, a new frame is pushed onto the stack.
high

Arguments from the

arguments function being called

The address control

return address flow is supposed to
return, when the
stack frame pointer function terminates.

exception handlers e.g.: Try-catch block

local variables Stack

Growth
callee saved registers
SP low
An Information Security Short Course
20
(Summer 2020)
An Information Security Short Course
21
(Summer 2020)
out[i] = ‘\0’;

An Information Security Short Course

22
(Summer 2020)
 What are buffer overflows?
Suppose a web server contains a function: void func(char *str) {
char buf[128];
When func() is called stack looks like: strcpy(buf, str);
do-something(buf);
}
argument: str
return address
stack frame pointer

char buf[128]

SP
An Information Security Short Course
23
(Summer 2020)
 What are buffer overflows?
• One of the most common OS bugs is a buffer overflow
– The developer fails to include code that checks whether an input
string fits into its buffer array
– An input to the running process exceeds the length of the buffer
– The input string overwrites a portion of the memory of the process
– Causes the application to behave improperly and unexpectedly

• Effect of a buffer overflow

– The process can operate on malicious data or execute malicious code
passed in by the attacker
– If the process is executed as root, the malicious code will be
executing with root privileges
An Information Security Short Course
24
(Summer 2020)
 What are buffer overflows?
What if *str is 136 bytes long? void func(char *str) {
After strcpy: char buf[128];
strcpy(buf, str);
do-something(buf);
}

argument: str
return address
stack frame pointer
*str Problem:
no length checking in strcpy()
char buf[128]

SP An Information Security Short Course

25
(Summer 2020)
Another Example with strcpy() Vulnerability
Top of
Memory
domain.c 0xFFFFFFFF
Main(int argc, char *argv[ ]) Stack
/* get user_input */ Fill
{ Direction
char var1[15];
char command[20];
strcpy(command, “whois "); var1 (15 char)
strcat(command, argv[1]);
strcpy(var1, argv[1]); command
printf(var1); (20 char)
system(command);
} ..
.
• Retrieves domain registration info Bottom of
• e.g., domain brown.edu Memory
An Information Security Short Course 0x00000000
26
(Summer 2020)
 Another Example with strcpy() Vulnerability
domain.c Top of
Main(int argc, char *argv[]) Memory
/*get user_input*/ 0xFFFFFFFF Stack
{ Fill
char var1[15]; Direction
char command[20];
strcpy(command, “whois "); argv[1]
var1argv[1]
(15 char)
strcat(command, argv[1]); (15 char)
(20 char)
strcpy(var1, argv[1]); Overflow
printf(var1); command
exploit
system(command); (20 char)
}
..
• argv[1] is the user input .
• strcpy(dest, src) does not check buffer Bottom of
• strcat(d, s) concatenates strings Memory
0x00000000
An Information Security Short Course
27
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

28
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

29
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

30
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

31
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

32
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

33
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

34
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

35
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

36
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

37
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

38
(Summer 2020)
A Deeper View of Buffer Overflow

An Information Security Short Course

39
(Summer 2020)
 Buffer Overflow Attack in a Nutshell
• First described in
Aleph One. Smashing The Stack For Fun And Profit. e-zine
www.Phrack.org #49, 1996
• The attacker exploits an unchecked buffer to perform a
buffer overflow attack
• The ultimate goal for the attacker is getting a shell that
allows to execute arbitrary commands with high
privileges
– Vulnerability scenarios: the targeted victim program
has root privileges (setuid)
An Information Security Short Course
40
(Summer 2020)
 Basic Stack Exploit
• Overwriting the return address allows an attacker
to redirect the flow of program control
• Instead of crashing, this can allow arbitrary code
to be executed
– Code segment is called “shellcode”
• Example, the execve system call is used to
execute a file
– With the correct permissions, execve(“/bin/sh”) can
be used to obtain a root level shell
An Information Security Short Course
41
(Summer 2020)
Basic stack exploit high

Suppose *str is such that Program P

after strcpy stack looks like:

Program P: execve(“/bin/sh”)

return address
When func() exits, the user gets shell !
Note: attack code P runs in stack.

char buf[128]

low
An Information Security Short Course
42
(Summer 2020)
 Shellcode of execve
How to develop shellcode that runs as execve(“/bin/sh”)?

(exact shell code by Aleph One)

An Information Security Short Course
43
(Summer 2020)
An Information Security Short Course
44
(Summer 2020)
An Information Security Short Course
45
(Summer 2020)
An Information Security Short Course
46
(Summer 2020)
An Information Security Short Course
47
(Summer 2020)
An Information Security Short Course
48
(Summer 2020)
An Information Security Short Course
49
(Summer 2020)
An Information Security Short Course
50
(Summer 2020)
 The NOP slide high

Problem: how does attacker

determine ret-address?

Solution: NOP slide

• Guess approximate stack state
when func() is called
– Install the program and perform random
fuzzy testing

• Insert many NOPs before program P:

nop ‘/x90’ , xor eax,eax , inc ax

low
An Information Security Short Course
51
(Summer 2020)
An Information Security Short Course
52
(Summer 2020)
 More on Stack Smashing
• Some complications:
– Shellcode should not contain the ‘\0’ character.
– Overflow should not crash program before func()
exists.

• Sample remote stack smashing overflows:

– (2007) Overflow in Windows animated cursors (ANI).
LoadAniIcon()
– (2005) Overflow in Symantec Virus Detection
test.GetPrivateProfileString "file", [long string]

An Information Security Short Course

53
(Summer 2020)
 Many unsafe libc functions
strcpy (char *dest, const char *src)
strcat (char *dest, const char *src)
gets (char *s)
scanf ( const char *format, … ) and many more.
• “Safe” libc versions strncpy(), strncat() are misleading
– Function strncpy() copies the string by specifying the number n of
characters to copy
• e.g., strncpy(dest, src, n); dest[n] = ‘\0’
– strncpy() may leave string unterminated.
• If source string is longer than the destination string, the overflow
characters are discarded automatically
– You have to place the null character manually

• Windows C run time (CRT):

– strcpy_s (*dest, DestSize, *src):
An Information Securityensures
Short Course proper termination
54
(Summer 2020)
 Buffer overflow opportunities
• Exception handlers: (Windows SEH attacks)
– Overwrite the address of an exception handler in stack
frame.

• Function pointers: (e.g. PHP 4.0.2, MS MediaPlayer Bitmaps)

Heap
buf[128] FuncPtr or
stack

– Overflowing buf will override function pointer.

• Longjmp buffers: longjmp(pos) (e.g. Perl 5.003)

– Overflowing bufAnnext to pos overrides value of pos.
Information Security Short Course
55
(Summer 2020)
 Corrupting Function Pointers in Heap
• Compiler generated function pointers (e.g. C++ code)
FP1
method #1

ptr FP2 method #2

FP3
method #3
data vtable

Object T
NOP shell
• After overflow of buf : slide code

data
buf[256] vtable

ptr
An Information Security Short Course 56
(Summer 2020) object T
 General Control Hijacking:
Corrupting Function Pointers in Heap

shellcode
shellcode
shellcode

Overwrite Step: Overwrite entries in a vtable for Object T

Activate Step: Call any method from Object T
An Information Security Short Course
57
(Summer 2020)
 General Control Hijacking:
Corrupting Function Pointers in Heap

Overwrite Step: Overwrite pointer to vtable on heap to point

to a crafted vtable.
An Information Security Short Course
Activate Step: Call any method from Object T (Summer 2020) 58
 General Control Hijacking

Overwrite Step: Find some way to modify a Control Flow

Pointer to point to your shellcode, library entry point, or
Other code of interest..
Activate Step: Find some way to activate that modified
Control Flow Pointer
An Information Security Short Course
59
(Summer 2020)
Instances of Control Hijacking

An Information Security Short Course

60
(Summer 2020)
 Finding buffer overflows
• To find overflow:
– Run web server on local machine
– Issue malformed requests (ending with “$$$$$” )
• Many automated tools exist (called fuzzers – next module)
– If web server crashes,
search core dump for “$$$$$” to find overflow location

• Construct exploit (not easy given latest defenses)

An Information Security Short Course

61
(Summer 2020)
 Control Hijacking

More Control
Hijacking Attacks

An Information Security Short Course

62
(Summer 2020)
 More Hijacking Opportunities

• Integer overflows: (e.g. MS DirectX MIDI Lib)

• Double free: double free space on heap.

– Can cause memory mgr to write data to specific location
– Examples: CVS server

• Format string vulnerabilities

An Information Security Short Course

63
(Summer 2020)
 Integer Overflows (see Phrack 60)

Problem: what happens when int exceeds max value?

int m; (32 bits) short s; (16 bits) char c; (8 bits)

c = 0x80 + 0x80 = 128 + 128 ⇒ c=0

s = 0xff80 + 0x80 ⇒ s=0
m = 0xffffff80 + 0x80 ⇒ m=0

Can this be exploited?

An Information Security Short Course

64
(Summer 2020)
 An example
void func( char *buf1, *buf2, unsigned int len1, len2) {
char temp[256];
if (len1 + len2 > 256) {return -1} // length check
memcpy(temp, buf1, len1); // cat buffers
memcpy(temp+len1, buf2, len2);
do-something(temp); // do stuff
}

What if len1 = 0x80, len2 = 0xffffff80 ?

⇒ len1+len2 = 0
Second memcpy() will overflow heap !!
An Information Security Short Course
65
(Summer 2020)
End of Segment

An Information Security Short Course

66
(Summer 2020)