CSC662

COMPUTER SECURITY

11 - SOFTWARE SECURITY

These slides are prepared from Prof Pavel Laskov‘s lecture slide Version 2.0

SOFTWARE SECURITY THREATS

Modi cation of program code

Viruses and self-replicating code
OS and API hooking
Control ow hijacking
Integer over ow
Buffer over ow
Heap over ow
Format string vulnerabilities
Code and data injection
Script injection (e.g. XSS)
SQL injection

1
 MALICIOUS SOFTWARE
(MALWARE)
The term malicious software denotes program code
executed without a user’s consent and carrying out
harmful functionality.

MALWARE CLASSIFICATION

Source: Aman Hardikar .M (2008), Malware 101 - Viruses, SANS Institute, doi https://www.sans.org/reading-room/whitepapers/incident/malware-101-viruses-32848

2
 SIMPLIFY MALWARE
CLASSIFICATION

Source: Hossein Rouhani Zeidanloo, S. Farzaneh Tabatabaei, Payam Vahdani Amoli and Atefeh Tajpour , All About Malwares (Malicious Codes) , University of
Technology Malaysia(UTM),doi: http://users.jyu.fi/~pavahdan/SAM10.pdf

AN EXAMPLE…

3
 AN EXAMPLE… THE CODE

MALWARE’S “THEORETICAL
FOUNDATIONS”
Von Neumann’s model (1948,1953)
Universal machine
Universal constructor
Information on the tape
Darwin/Core Wars: ghting programs (1966)
A special assembly language with 10 instructions (“Redcode”)
Two programs simultaneously running in the same memory
A program dies if it executes division by 0 or a null instruction
To increase their survival chances, programs can replicate
themselves
Cohen’s thesis (1984)
De nition of a virus
Mathematical description of virus propagation

4
 MALWARE PIONEERS
Elk Cloner (1982)
An Apple-II program written by a student Rich Skrenta
A program spread via boot sector infection
On every 50-th reset a short poem was is played by
hooking the reset handler
Brain Virus (1986)
First virus to spread in the wild
Written by Ashar brothers to prevent illegal copying of
software
Included in a boot sector of distributed software diskettes
Morris worm (1988)
Used a debugging feature of sendmail (remote execution)
Propagated in the Arpanet
Penetrated ca.6,000 computers (10% of Arpanet)

MALWARE CARRIERS: BOOT

SECTOR
Save the original MBR in a safe location
Overwrite the MBR with an infected one
Bootstrap a system using the new MBR

5
 MALWARE CARRIERS: COM
EXECUTABLES
Append a virus body to a program
Save an entry point to a program in a virus body
Replace a program entry point with a jump to a virus body
Virus code restores the original entry point and jumps to it
after its own execution

MALWARE CARRIERS: EXE

EXECUTABLES
Append a virus body to a program
Over write a program header to switch the entry point to a
virus
Jump to the original entry point during execution

6
MALWARE CARRIERS: MACROS
AND SCRIPTS
Malicious functionality is implemented in Visual Basic for
Applications (VBA)
If a document template are infected, so will be every
document on a system

OTHERS…
Trojan Horses: Named after the Trojan Horse in Greek
mythology, Trojan horses or simply trojans are malicious
programs that replicate themselves and steal information or
harm the host computer system. They usually masquerade as
helpful or harmless programs and begin to steal private data,
delete files, log keystrokes or provide an opening for further
malware to get installed.
Computer worms: Worms are another type of malicious
software that exploit a vulnerability in the computer system to
cause harm. They usually corrupt or delete critical files,
consume excess bandwidth and often put the computer under
control of the worm author. Once infected, computer systems
are used to send spam mails and perform many such unwanted
functions.

7
 OTHERS…
Rootkit: Rootkits are a type of malware that mask malicious
processes and software in the infected systems to ensure
privileged remote access. They are usually installed after a
vulnerability has been found and detected so that the infected
computer continues to be under the control of the malware author.
Rootkit detection is quite difficult as they are designed to stay
hidden from the very software used to detect them.
Spyware: As the name suggests, spyware are programs that spy
on the users. They steal user logins and passwords, document
sites visited, log keystrokes and even redirect browsers to
unwanted sites. They can result identity theft and sensitive bank
information can be compromised as well.
Adware: Adware as such are quite harmless, although they are
annoying. They generate ads in the form of popups and interrupt
other programs

MALWARE SUMMARY

Source: Aman Hardikar .M (2008), Malware 101 - Viruses, SANS Institute, doi https://www.sans.org/reading-room/whitepapers/incident/malware-101-viruses-32848

8
 BEYOND PURE VIRUSES
Triggered replication
Automatic replication using exploits
Download of malicious code
Modern botnets

MALWARE HANDLING STEPS

Source: Aman Hardikar .M (2008), Malware 101 - Viruses, SANS Institute, doi https://www.sans.org/reading-room/whitepapers/incident/malware-101-viruses-32848

9
CONTIGUOUS MEMORY (BUFFER)
ALLOCATION
C/C++
static: int x[20] declaration outside any function;
allocated in the static variables memory along the
program code
automatic: int x[20] declaration inside some
function; allocated on the stack
dynamic: int *x = new int[20]; must be followed
by delete to avoid memory leaks; allocated on the
heap
Java
dynamic: int x = new int[20]; gets allocated in on the
heap; automatic deallocation by garbage collector

BUFFER OVERRUNS

What does the following program do?

#include <stdio.h>
#define SIZE 10
main() {
int matrix[SIZE*SIZE];
int total_size = SIZE*SIZE;
int* row_ind[SIZE];
for (int i = 0; i <= total_size; i++)
matrix[i] = i;
for (int i = 0; i <= SIZE; i++)
row_ind[i] = &matrix[i*SIZE];
for (int i = 0; i <= SIZE; i++)
printf("a[%d] = %d\n", i, *row_ind[i]);
}

10
PROCESS MEMORY ORGANIZATION

Process memory is partitioned into

segments:
.text – program code
.data – initialized static data
.bss – uninitialized static data
heap – dynamically allocated memory
stack – program call stack
Each memory segment has
appropriate permissions
Access operations violating these
permissions cause the
“segmentation fault“ error

STACK ORGANIZATION

Stack is composed of
frames
Each frame comprises
functions arguments
return address
frame pointer: the address of
the start of the previous frame
local variables
Frames are pushed on the
stack during function
invocation and popped back
after the return

11
 OVERWRITING THE RETURN
ADDRESS
A local buffer is allocated
“bottom-up”, i.e. it starts at
lower and ends at higher stack
locations.
Without proper bound checking
a buffer content can overspill
into adjacent upper stack area.
By controlling buffer content, an
attacker can overwrite the
return address with an arbitrary
value and hijack the execution
ow.

WHAT ARE BUFFER OVERFLOWS?

Suppose a web server
contains a function:
void func(char *str)
{
char buf[128];

strcpy(buf, str);
do-something(buf);
}
When func()is called stack
looks like:

12
WHAT ARE BUFFER OVERFLOWS?
void func(char *str)
{
char buf[128];

strcpy(buf, str);
do-something(buf);
*str
}
What if *str is 136 bytes buf[128]
long? After strcpy…
strcpy() do not check the
string length

BASIC STACK EXPLOIT

Suppose *str is such that
after strcpy; stack looks
exec(“/bin/sh”)
like...
Program P:
exec(“/bin/sh”)
When func()exits, the *str
user gets shell! Note:
attack code P runs in stack. buf[128]

To determine return
address; guess position of
stack when func()is called

13
 SOFTWARE SECURITY
MECHANISMS
Data execution protection
mark certain areas in memory as non-executable
Address space layout randomization
choose stack memory allocation at random
makes it dif cult to guess the values to overwrite the
return address with
Canaries
preceed the return value with a special value
before following the return value, check if is content has
not changed after the call

CODE INJECTION

Goal: trick program into executing an

attacker’s code by clever input construction
that mixes code and data
Mixed code and data channels have special
characters that trigger a context change
between data and code interpretation
The attacker wants to inject these meta-
characters through some clever encoding or
manipulation, so supplied data is interpreted as
code

14
EXPLOITATION OF UNVALIDATED
INPUT - I
A CGI script mails a le to an address read from
a form:
cat $file | mail $address
The user inputs: user@host | rm -rf /
The following statement is executed:
cat $file | mail user@host | rm -rf /
Root directory is wiped out.

EXPLOITATION OF UNVALIDATED
INPUT - II
The following script validates username and password:
$login = Request.Form(“login“)
$password = Request.Form(“password“)
$sql_command = “SELECT user FROM database WHERE
Login='$login' AND Password='$password'“
db->prepare($sql_command)
The user inputs 'OR''=' for login and 'OR''=' for
password
The following SQL statement is executed
SELECT user FROM database WHERE
Login=“ OR “=“ AND Password=“ OR “=“
Always true (since “=“ is true); login is successful.

15
 DEFENDING AGAINST CODE
INJECTION
Input cleansing and validation
Model the expected input
Discard what doesn't fit (e.g., metacharacters)
Keep track of which data has been cleansed
e.g., Perl's taint mode
Keep track of all sources of inputs
Or cleanse as the input is received
Type and range verification, type casts
Separating code from data
Transmit, receive and manipulate data using
different channels than for code

KEY POINTS

Software insecurity stems from attacker’s ability to

modify system resources criticial for program
execution, e.g. instruction pointer, function call
addresses, interrupt addresses, etc.
One of the key sources for software insecurity is
failed validation of user input.
Buffer over ows are a most widely used
exploitation technique.
Special techniques for strengthening software
security exist, e.g. canaries, address space layout
randomization and data execution prevention.

16
Thank You

17