1

CSIT 561 – COMPUTER SECURITY

MODULE 5 : PROGRAM SECURITY

Bharath K. Samanthula
Department of Computer Science
Montclair State University

Slides are adopted from Chapter 12, Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043)..
 2

Objectives
• Learn about memory organization, buffer
overflows, and relevant countermeasures
• Common programming bugs, such as off-by-one
errors, race conditions, and incomplete mediation
• Survey of past malware and malware capabilities
• Virus detection
• Tips for programmers on writing code for security

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 3

Programs
• Programs and their computer code are the basis of computing
• Programs consist of elementary (primitive) machine
commands, such as move one data item or compare two data
items
• Example: Intel 32- and 64-bit instruction set has about 30 basic
primitives
• Primitive commands implement higher-level programming
constructs, such as conditional loops, arithmetic and string
operations
• Programmers often use code libraries to build complex
programs
• Errors or flaws in programs can range from insignificant to
catastrophic

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 4

Program Flaws
Program flaws can have two kinds of security implications:
• A program flaw can be a fault affecting the correctness of
the program’s result
• Incorrect operations is an integrity failure

• Even a flaw from a benign cause can be exploited by

someone malicious

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 5

Code and Data

• Code, data, instructions, the operating system, user
programs, photos, etc…. Everything in memory are just
strings of 0s and 1s
• In Memory, code is indistinguishable from data

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 6

Memory Allocation
High addresses
Stack

The key takeaways: code

and data separated, with
the heap growing up Heap
toward high addresses
and the stack growing Static data
down from the high
addresses.
Code
Low addresses
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 7

Data vs. Instructions

Store sum " 7178

0!1C0A

Execute instruction
“Jump forward 10
bytes”

Memory

The same hex value in the same spot in memory can either be a
meaningful data value or a meaningful instruction depending on
whether the computer treats it as code or data. This will be the basis
of the attacks in the following slides.

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 8

Buffer Overflows
• Occur when data is written beyond the space allocated for
it, such as a 10th byte in a 9-byte array
• In a typical exploitable buffer overflow, an attacker’s
inputs are expected to go into regions of memory
allocated for data, but those inputs are instead allowed to
overwrite memory holding executable code
• The trick for an attacker is finding buffer overflow
opportunities that lead to overwritten memory being
executed, and finding the right code to input

NOTE: Buffer overflows often come from innocent programmer

oversights or failures to document and check for excessive data

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 9

How Buffer Overflows Happen

char sample[10];

int i;

for (i=0; i<=9; i++)

sample[i] = ‘A’;

sample[10] = ‘B’;

NOTE: This is a very simple buffer overflow. Character B is placed in

memory that wasn’t allocated by or for this procedure.

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 10

Memory Organization
High addresses
Stack

Heap

Local Data

Program Code

System Data

System Code
Low addresses
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 11

A variation of Buffer Overflow

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 12

Where a Buffer Can Overflow

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 13

The Stack
Stack
P3
P2
Direction of
growth P1
Prog Ctr
Stack Ptr
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 14

The Stack after Procedure Calls

Procedure B

Stack
Procedure A P3 call C
P2
call B P1
Prog Ctr
Stack Ptr Procedure C
P2
P1
Prog Ctr
Stack Ptr
NOTE: When procedure A calls procedure B, procedure B gets added to the stack along with a pointer back to procedure
A. In this way, when procedure B is finished running, it can get popped off the stack, and procedure A will just continue
executing where it left off.

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 15

Compromised Stack Procedure B

Stack
Procedure A P3 call C
P2
call B P1
Prog Ctr
Stack Ptr Procedure C
code
code
Prog Ctr
Stack Ptr

NOTE: Instead of pointing at procedure B in this case, the program counter is pointing at code that’s been placed on
the stack as a result of an overflow.

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 16

Overwriting Memory for Execution

• Overwrite the program counter stored in the
stack
• Overwrite part of the code in low memory,
substituting new instructions
• Overwrite the program counter and data in
the stack so that the program counter
points to the stack

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 17

Harm from Buffer Overflows

• Overwrite:
• Another piece of your program’s data
• An instruction in your program
• Data or code belonging to another program
• Data or code belonging to the operating system

• Overwriting a program’s instructions gives attackers that

program’s execution privileges
• Overwriting operating system instructions gives attackers
the operating system’s execution privileges

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 18

Overflow Countermeasures
• Staying within bounds
• Check lengths before writing
• Confirm that array subscripts are within limits
• Double-check boundary condition code for off-by-one errors
• Limit input to the number of acceptable characters
• Limit programs’ privileges to reduce potential harm

• Many languages have overflow protections

• Code analyzers (e.g., static code analyzer – see the
reference below) can identify many overflow vulnerabilities
• Canary values in stack to signal modification

https://www.us-cert.gov/bsi

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 19

Incomplete Mediation
• Mediation: Verifying that the subject is
authorized to perform the operation on an
object
• Preventing incomplete mediation:
• Validate all input
• Limit users’ access to sensitive data and
functions

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 20

Time-of-Check to Time-of-Use
• Mediation performed with a “bait and switch” in the middle
• Between access check and use, data must be protected against change

File: Action:
my_file Change byte 4 to A

File: Action:
your_file Delete file
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 21

Race Conditions
A Seat available? Book seat
Yes

Reservation system

B Seat available? No

Time
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 22

Race Conditions
Situation in which program behavior depends on the order in
which two procedures execute

A Seat available? Book seat

Yes

Reservation system

B Seat available? Book seat

Yes

Time

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 23

Other Programming Oversights

• Undocumented access points (backdoors)
• During debugging or testing, programmers creates or
undocumented entry point or execution mode which he later forgot
to remove during migration
• Off-by-one errors
• Example: miscalculating the condition to end a loop, repeat while
i<=n or i<n?

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 24

Other Programming Oversights

• Integer overflows
• Storage location is fixed and finite size.
• Overflow depends on whether the data values are signed or not
• The excess digits on the most significant end of the data
item are lost or a program exception arises

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 25

Other Programming Oversights

• Unterminated null-terminated string
• Long strings are the source of man buffer overflows
• A program might mistakenly overwrite part of a string exceeding its
intended length
• Depends on how strings are stored (depends on programming
language + operating system involved)

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 26

Other Programming Oversights

• Parameter length, type, or number errors
• Too many parameters
• Wrong output type or size
• Too-long string
• Unsafe utility libraries
• Use safer functions from libraries
• Safer to use strnncpy rather than strcpy

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 27

Malware
• MALicious softWARE - Programs planted by an agent
with malicious intent to cause unanticipated or undesired
effects
• In May 2010, Roger Thompson from AVG detected
malicious code (a hidden call to Ukraine’s website) at the
website of the U.S. Bureau of Engraving and Printing
• Used the Eleonore attack toolkit – click and run application against
a targeted website

• Malicious code comes in many forms under many names

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 28

Malware
Three most popular forms
• Virus
• A program that can replicate itself and pass on malicious code to
other nonmalicious programs by modifying them
• Worm
• A program that spreads copies of itself through a network

• Trojan horse
• Code that, in addition to its stated effect, has a second,
nonobvious, malicious effect

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 29

Virus vs. Worm

• Worm operates through the network, but a virus mostly
spread through infected programs or files

• Worm spreads copies of itself whereas virus spreads

copies of itself as a program that attaches to

• A bot is a kind of worm used in vast numbers for search

engine hosts

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 30

Types of Malware

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 31

Types of Malware (cont.)

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 32

The Real Impact of Malware

• Companies are loath to report breaches except when
required by law
• The 2014 Verizon Breach Report shows that there is
increase of breaches for the purpose of espionage
although financial gain was still the main motive

• Refer to the following link for statistics on top infections

https://statistics.securelist.com/en

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 33

History of Malware

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 34

History of Malware (cont.)

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 35

Zero-Day Attack
• Active malware exploiting a product vulnerability for which
the manufacturer has no countermeasure available

• On 27 Dec 2005, a vulnerability was discovered in

windows metafile (.WMF) files
• Within hours hundreds of sites exploited this vulnerability
• Microsoft release a patch after nine days

• David Litchfield of Next Generation Software in the U.K.

founds vulnerabilities in Oracle software
• Oracle took astonishing 800 days to fix two of them and others
were not fixed for 650 days

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 36

Malicious code - Features

Four aspects of malicious code:
• Harm – how they affect users and systems
• OS, document applications, browsers, etc.
• Transmission – how they are transmitted and replicated
• Example: documents, music files, networks, flash media, etc.

• Activation – how they gain control and install themselves

• Stealth – how they hide to avoid detection

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 37

Harm from Malicious Code

• Harm to users and systems:
• Sending email to user contacts
• Deleting or encrypting files (Jerusalem or Ransomware virus)
• Modifying system information, such as the Windows registry
• Stealing sensitive information, such as passwords
• Attaching to critical system files
• Hide copies of malware in multiple complementary locations
• Harm to the world:
• Some malware has been known to infect millions of systems,
growing at a geometric rate
• Infected systems often become staging areas for new infections

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 38

Harm from Malicious Code

• Harm to the world:
• Attacker’s goal is infect as many people as possible
• Examples: Morris worm affected 3000 computers
• I Love You worm infected 100, 000 servers
• Code red is believed to have infected 3 million hosts

• Cost of recovery is often heavy

• Enumerate losses – tangile and intangible factors
• Damage of Code red range from $500 million to $2.6 billion
• Damage from Conficker was $9.2 billion, or approximately $1, 000 per
system

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 39

Transmission and Propagation

• Setup and installer program
• Attached file
• Document viruses (e.g., spreadsheet)
• Autorun (e.g., autoexec.bat in windows, .profile on Unix)
• Propagation using nonmalicious programs: It is easy to
distribute a 200 lines of virus code across as many
program as the attackers wants using jump statements
• Appended viruses
• Viruses that surround a program
• Integrated viruses and replacements

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 40

Interesting Question

Is it better to disclose a flaw and alert users that they are

vulnerable or conceal it until there is a countermeasure?

In July 2005, Michael Lynn (security researcher for ISS)

found a serious flaw in IOS on which Cisco based most of
its firewalls and router products. He went public followed by
legal suits and settlements.

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 41

Malware Activation
• One-time execution (implanting)
• User clicks to download a file or opens an attachment
• Boot sector viruses

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 42

Malware Activation
• Memory-resident viruses
• Attach virus to resident code (such as routine that interprets keys
pressed)
• Example: a boot sector virus after activation might attach itself to a
piece of resident code
• Application files
• If there is a flaw in pdf interpreter à execution of malicious code
• Virus macros in applications, such as spreadsheets, can adds itself
to the startup directives that are execute every time
• Code libraries
• Compilers, linkers, runtime debuggers are good candidates for
hosting viruses as they are widely shared

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 43

Malware Stealth
• Most viruses maintain stealth by concealing their action,
not announcing their presence, and disguising their
appearance – during installation, while executing, or even
at rest in storage
• Installation Stealth: for example, downloading as a result of loading
a web page
• Execution Stealth: OS supports dozens of concurrent processes
which have unrecognizable names and functions. Hard for users to
distinguish legitimate programs from malware code
• Stealth in Storage: produce unique copy for every user
• Rearrange order of modules/instructions
• Insert instructions that have no impact
• Insert random strings
• Replace instructions with others of equivalent effect
• Insert instructions that are never executed

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 44

Virus Effects
• Virus Effect How It Is Caused
Attach to executable • Modify file directory
program • Write to executable program file
Attach to data or • Modify directory
control file • Rewrite data
• Append to data
• Append data to self
Remain in memory • Intercept interrupt by modifying interrupt
handler address table
• Load self in non-transient memory area
Infect disks • Intercept interrupt
• Intercept operating system call (to format
disk, for example)
• Modify system file
• Modify ordinary executable program
Conceal self • Intercept system calls that would reveal
self and falsify result
• Classify self as “hidden” file
Spread infection • Infect boot sector
• Infect systems program
• Infect ordinary program
• Infect data ordinary program reads to
control its execution
Prevent deactivation • Activate before deactivating program and
block deactivation
• Store copy to reinfect after deactivation

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 45

Countermeasures for Users

• Use software acquired from reliable sources
• Test software in an isolated environment
• Only open attachments when you know them to be safe
• Treat every website as potentially harmful
• Create and maintain backups

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 46

Virus Detection
• Virus scanners look for signs of malicious code infection using
signatures in program files and memory
• Traditional virus scanners have trouble keeping up with new
malware—detect about 45% of infections
• Detection mechanisms:
• Known string patterns in files or memory
• Effective iff virus scanner is up-to-date with latest information on current
viruses
• Execution patterns – Code Analysis
• Often difficult as all the structure and documentation are lost during
compilation
• Storage patterns – where the virus code is located relative to the
infected program
• Changes to file sizes or program’s functionalities can also be used
• Look for suspicious patterns, such as JUMP instruction as the first
instruction – Possibly a virus

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 47

Virus Signatures
IF (--)
Attached Recognizable
JUMP
Virus Code signature elements

Original
Program
Original
Program

Separate
Virus
Module
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 48

Countermeasures for Developers

• Modular code: Each code module should be
• Single-purpose – Performs one function
• Small – ready to grasp both structure and content
• Simple – low degree of complexity
• Independent – performs tasks isolated from other modules
• Encapsulation – technique for defining what should be hidden
and what is visible inside a component
• Information hiding – describes what module does, not how
(black box)
• Mutual Suspicion – Programs are not trustworthy (Facebook
was shutdown for several hours in 2011 due to misplaced trust)
• Confinement – Limits damage from infected code
• Genetic diversity – reduces number of targets susceptible to
one attack type
• In 2014 Heartbleed malware affected OpenSSL implementations

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 49

Code Testing
Testing – goal is to make the product failure free (not really
possible, only reduces the likelihood of failures)
• Unit testing – Each module/component is tested separately
• Integration testing – Components work together!
• Function testing – evaluate for requirement specifications
• Performance testing – check for overall software and hardware
requirements
• Acceptance testing – system is checked against customer’s
requirements description
• Installation testing – testing under real environment
• Regression testing – done after a fix to ensure functions are
working and performance has not been degraded
• Penetration testing – look for possible vulnerabilities

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 50

Design Principles for Security

Multics project established secure computing principles below
• Least privilege – users/programs should operate using fewest
privileges
• Economy of mechanism – the design of protection system should
be small, simple and straightforward
• Open design – mechanism should be public
• Complete mediation – every access must be checked
• Permission based – default condition is denial of access
• Separation of privilege – higher privileges should depend on more
than one condition, such as authentication + crypto key
• Least common mechanism – physical and logical separation
reduce the risk from sharing
• Ease of use

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 51

Other Countermeasures
• Good
• Penetration Testing – experts trying to crack the system being tested
• Proofs of program correctness—where possible
• Initial assertions about the program’s inputs and see if the desired output is
generated, check the logical flow of the program
• Defensive programming
• Anticipate what could go wrong (e.g., appropriate data types, values out of
range, incorrect number of parameters)
• Design by contract
• Checking Preconditions, postconditions and invariants
• Bad
• Penetrate-and-patch – fixing one problem causes failure at other place
• Security by obscurity – assuming system is secure by hiding internal
mechanisms, such as hiding account passwords in binary files

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 52

Summary
• Buffer overflow attacks can take advantage of the fact that
code and data are stored in the same memory in order to
maliciously modify executing programs
• Programs can have a number of other types of
vulnerabilities, including off-by-one errors, incomplete
mediation, and race conditions
• Malware can have a variety of harmful effects depending
on its characteristics, including resource usage, infection
vector, and payload
• Developers can use a variety of techniques for writing and
testing code for security

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.