Program Security

Program Security

• Protecting programs is the heart

of computer security
 All kinds of programs, applications,
operating systems and database
systems etc.
 A program is secure if it provides
confidentiality, integrity and
availability.
 Secure programs
• Security implies some degree of trust that the
program enforces expected confidentiality,
integrity, and availability.
• An assessment of security can also be influenced by
someone's general perspective on software quality.
• For example, one person may decide that code is
secure because it takes too long to break through its
security controls. And someone else may decide code
is secure if it has run for a period of time with no
apparent failures.
 PROGRAM SECURITY
• Program security is an idea
implemented to
protect software/program against
malicious attack and other hacker
risks so that the software continues
to function correctly under such
potential risks. Security is necessary
to provide integrity, authentication
and availability.
 Program security is difficult
• Security often conflicts with usefulness
and performance.
• It is essentially impossible to ensure
that program does only what it is
designed to do and nothing else.
• Programming and software engineering
techniques change faster than
computer security techniques.
• Even a single flaw can be catastrophic.
 Program Security
Fault tolerance terminology:
• Error-may lead to a fault
• Fault-a deviation from intended functionality
• Failure- system malfunction caused by fault

Two categories of faults(classified by duration)

• Permanent faults
• Transient faults-can be much more difficult
to diagnose.
 Types of program flaws
 Intentional

Malicious

Nonmalicious

 Inadvertent

Validation error (incomplete or inconsistent)
 e.g., incomplete or inconsistent input data

Domain error
 e.g., using a variable value outside of its domain

Serialization and aliasing
 serialization – e.g., in DBMSs or OSs
 aliasing - one variable or some reference, when changed, has an
indirect (usually unexpected) effect on some other data

Inadequate ID and authentication

Boundary condition violation

Other exploitable logic errors
 Non malicious program errors

• Most of the mistakes made by the

programmers are unintentional and
non malicious.
• Many such errors will not lead to more
serious vulnerabilities but few will put
many security professionals in trouble.
• We look at three such classic error
types and explain why they are
relevant to security and how can they
be prevented.
 Buffer overflows

• Its like pouring 2 liters of water into a

1 liter jug.
Definition
• A buffer is a space in memory in
which data is held.
• As memory in finite => buffer
capacity is finite
• Therefore, in programming
languages the programmer must
declare the buffers maximum size.
 Buffer overflow example -C
• char sample[10];
// compiler sets 10 bytes to store this buffer.
• sample[10]=‘B’;
// out of bounds error, compiler detects this during
compilation.
• Now, what if we do
sample[i]=‘B’;
• In some programming languages, buffer sizes need not
be predefined.
• C does notperform array bounds checking.
• Similar problem caused by pointers--No reasonable way
to define limits for pointers
 Buffer overflows

for (i=0; i<=9; i++) sample[i] = 'A';

sample[10] = 'B'
• Where does ‘B’ go?
Depends on what is adjacent to ‘sample[10]’
• Affects user’s data- overwrites user’s data
• Affects users code- changes user’s
instruction
• Affects OSdata- overwrites OS data
• Affects OScode- changes OS instruction
Buffer overflows
 Implications of buffer overflow:

Attacker can insert malicious data

values/instruction codes into “overflow space”
• Buffer overflow affects OS code area
Attacker code executed as if it were OS code
 Attacker might need to experiment to see what
happens when he inserts B into OS code area
Can raise attacker’s privileges(to OS
privilege level)
 When B is an appropriate instruction
Attacker can gain full control of OS
 Incomplete mediation
 Incomplete mediation flaw — often inadvertent (=> nonmalicious)
but with serious security consequences
 Incomplete mediation:
Sensitive data are in exposed, uncontrolled condition
 Example
 URL to be generated by client’s browser to access server, e.g.:
http://www.things.com/order/
final&custID=101&part=555A&qy=20&price=10&ship=boat&shipc
ost=5&total=205
 Instead, user edits URL directly, changing price and total cost
as follows:
http://www.things.com/order/final&custID=101&part=555A&qy=2
0&price=1&ship=boat&shipcost=5&total=25
 User uses forged URL to access server
 The server takes 25 as the total cost
 Unchecked data are a serious vulnerability!
 Possible solution: anticipate problems
 Don’t let client return a sensitive result
(like total) that can be easily
recomputed by server
 Use drop-down boxes / choice lists for
data input

Prevent user from editing input directly
 Check validity of data values received
from client
 Do not rely solely on client-side
validation
Race condition
 Time-of-check to Time-of-use
Errors(TOCTTOU)
 Time-of-check to time-of-use flaw — often
inadvertent (=> nonmalicious) but with serious
security consequences
 synchronization flaw / serialization flaw
 TOCTTOU — mediation with “bait and switch” in
the middle

Non-computing example:

Swindler shows buyer real Rolex watch ( bait)

After buyer pays, switches real Rolex to a forged one

In computing:

Change of a resource (e.g., data) between
time access checked and time access used
Q: Any examples of TOCTTOU problems from
computing?

A: E.g., DBMS/OS: serialization problem:

pgm1 reads value of X = 10
pgm1 adds X = X+ 5
 pgm2 reads X = 10, adds 3 to X, writes
X = 13
pgm1 writes X = 15

X ends up with value 15 – should be X = 18

 Prevention of TOCTTOU
 errors
Be aware of time lags

Use digital signatures and certificates to “lock” data values
after checking them
 So nobody can modify them after check & before use

Q: Any examples of preventing TOCTTOU from
DBMS/OS areas

A1: E.g., DBMS: locking to enforce proper serialization
(locks need not use signatures—fully controlled by DBMS)
In the previous example:
will force writing X = 15 by pgm 1, before pgm2
reads X (so pgm 2 adds 3 to 15)
OR:
will force writing X = 13 by pgm 2, before pgm1
reads X (so pgm 1 adds 5 to 13)

• An intelligent attacker uses each of the previously

mentioned three flaws(buffer overflow, incomplete
mediation, TOCTTOU) as one step in a multistep attack.
 Penetrate and Patch
• A method of judging software security in which a Red
Team/Tiger Team intentionally tries to crack or break a
software program.
• Theory –if the program withstands the attack, then security
is adequate.
• Is this true?Rarely
• Developers often try to quicky fix problems discovered by the Tiger
Team
• Quick Patches often introduce new faults due to
1.The pressure to repair a specific problem encouraged a narrow
focus on the fault itself and not on its context. In particular, the
analysts paid attention to the immediate cause of the failure and
not to the underlying design or requirements faults.
2.The fault often had nonobvious side effects in places other
than the immediate area of the fault. Fixing one problem often
caused a failure somewhere else, or the patch addressed the
problem in only one place, not in other related places.
3.The fault could not be fixed properly because system
functionality or performance would suffer as a consequence
Viruses and other Malicious code
• Work done by a program is invisible to users and they
will not be aware of any malicious activity.
• Example:
1. When is the last time you saw a bit?
2.Do you know in what format a document file is stored?
3.If a document is stored on a disk, can you tell the exact
location where is it residing?
4.Which programs execute when we start our computer and
how they are executed?
5. Can you tell if a game program does anything in addition to
its expected interaction with you?
6. Which files are modified by a word processor when you
create a document?
• We cannot answer these question properly, since we
don’t see computer data directly. Malicious people can
make programs serve as vehicles to access and change data
and other programs.
 Malicious code

• Malicious code executes just like any other program on

the system. But, it is written to exploit the vulnerabilities
of a system/software.
• Malicious code can change: data and other
programs.
• Malicious can do anything like writing a message to
the screen, stopping a running program, erasing a
stored record etc. or sometimes malicious code will not
do anything at all and stay dormant in the system.
• Dormant malicious code just needs a trigger to become
active.
• Malicious codes are not new to computers, they have
been in existence for the past few decades.
 • Trapdoors
• Trojan Horses
•
X
• Fil
es

• Bacteria

• Worms
• Logic Bombs • Viruses
 Kinds of malicious code
• Malicious code or a rogue program is the general name
for unanticipated or undesired effects in programs or
program parts, caused by an agent intent on damage.
• Agent is the writer of the program or the person who
causes its distribution.
• Virus is a program that can replicate itself and pass onto
other non malicious programs. A hidden, self-replicating
section of computer software, usually malicious logic, that
propagates by infecting (i.e., inserting a copy of itself into and
becoming part of) another program. A virus cannot run by
itself; it requires that its host program be run to make the
virus active.
• Virus can be: transient or resident
Transient virus has a life that depends on the life of its
host.
Resident virus located itself in the memory and will be
active in the system even after the attached program ends.
• Trojan Horse
A computer program that appears to have a useful
function, but also has a hidden and potentially malicious
function that evades security mechanisms, sometimes by
exploiting legitimate authorizations of a system entity
that invokes the program
• Trojan horse gets installed along with an infected
legitimate program.
Effects of a Trojan horse:
• Deleting, editing files.
• Transmitting files to intruders.
• Installing malicious code that can gain network access.
• Privilege elevation attacks etc.

• Worm is a program that replicates itself and spreads

across a network of systems. Primary difference between
a worm and a virus is that, a worm operates through
networks whereas a virus spread through any medium.
• Logic bomb is a special class of malicious code that
“detonates” or goes off when a certain condition is met.
Time bomb is a logic bomb whose trigger is time or date.
• Trapdoor or backdoor is a feature in program, which
provides an alternate entry or access to the program
avoiding the direct calls and perhaps with special privileges.
• Rabbit is a virus or a worm that replicates itself without any
bound to exhaust the computing resources of a system.

• Often the term “Virus” is used to refer to any malicious code.

 How viruses work?
• Program containing virus must be executed to spread
virus or infect other programs
• Even one program execution suffices to spread virus
widely
• Virus actions: spread / infect

• Spreading–Example 1: Virus in a program on

installation CD
 User activates program containing virus when she runs
INSTALL or SETUP
 Virus installs itself in any/all executing programs
present in memory
 Virus installs itself in programs on hard disk
 From now on virus spreads whenever any of the
infected programs (from memory or hard disk) executes
• Spreading –Example 2:Virus in attachment to
e-mail message
 User activates program containing virus (e.g.
macro in MS Word)by just opening the attachment
 => Disable automatic opening of attachments!!!
 Virus installs itself and spreads

• Spreading –Example 3: Virus in downloaded

file
• File with program or document(.doc, .xls, .ppt,
etc.)
Kinds of viruses-based on their way of attaching

1.Appended Viruses
• Appends to program. Often virus code precedes
the program code execution by running its code
before the 1st program instruction in executable
file.
• Executes whenever program gets executed.
 2. Surrounding viruses
• Surrounds program
• Executes before and after infected program
• Intercepts its input/output
• Erases its tracks
• The “after” part might be used to mask virus
existence.
• For example, a virus writer might want to prevent
the virus from being detected. If the virus is stored on
disk, its presence will be given away by its file name,
or its size will affect the amount of space used on the
disk. The virus writer might arrange for the virus to attach
itself to the program that constructs the listing of files on the
disk. If the virus regains control after the listing
program has generated the listing but before the
listing is displayed or printed, the virus could
eliminate its entry from the listing and falsify space
counts so that it appears not to exist.
3. Integrating and replacing viruses
 Integrates into program code
• Spread within infected programs
 (Replacing) virus V gains control over target
program T by:
• Overwriting T on hard disk
OR
• Changing pointer to T with pointer to V
 OS has File Directory
 File Directory has an entry that points to file with
code for T
 Virus replaces pointer to T’s file with pointer to
V’s file
In both cases actions of V replace actions of T
when user executes what she thinks is “T”
 Integrating and
replacing viruses
 Document virus-one form of integrated
virus
• Spreads via picture, document, spreadsheet, slide
presentation, database.
• E.g., via .jpg, via MS Office documents .doc, .xls, .pptetc.
• It is implemented within a formatted document, such as a
written document, a database, a slide presentation, or a
spreadsheet.
• These documents are highly structured files that contain
both data (words or numbers) and commands (such as
formulas, formatting controls, links).
• Commands are part of rich programming language.
• Attacker uses these command portions to integrate his
virus code with the document.
• Ordinary user just sees the plain document but not the
virus code embedded in commands portion.
 Characteristics of a “Virus”

• Hard to detect
• Not easily destroyed or deactivated
• Spreads infection widely
• Can re-infect programs
• Easy to create
• Machine and OS independent
 Homes for viruses
• Most viruses are passed through e-mails or
drive-by-downloads.
• Attackers lure the victims to open the
emails / click the malicious links that enable
drive-by-download.
• Ways for virus to take control over program:
Overwriting the complete program
Changing the pointer to point to a virus
code instead of program on the disk.
• One-time execution: majority of the viruses
today execute only once, spreading their
effect in that once execution.
 Boot sector viruses

• When OS is started, firmware detects the hardware

components present, tests them and then transfers the
control to the OS.
• OS is invoked dynamically and not coded in the firmware.
• OS resides on the disk. It is fetched into memory by a
program called Bootstrap.
• Firmware reads fixed number of bytes from a fixed
location (boot sector) on the disk to a fixed location in the
memory and jumps to that address for execution.
• Often the boot sector size will be less than 512 bytes
whereas the bootstrap loader will be of larger size.
• To support this situation most of the hardware designers
support “chaining”.
• This chaining has both pros and cons.
• Virus writer will simply break the
chain at any point, inserts a pointer
to the virus code, and reconnects the
chain later .
 Memory resident viruses

• Most of the user programs will execute, terminate

and disappear making space for other programs.
• Few specialized programs are called very often and
loading them each time takes a long time. So, OS
keeps such programs and resident programs in the
memory.
Ex: resident code that interprets the keys pressed
on keyboard.
• Resident routines are also called as “terminate and
stay resident” TSR.
• Viruses attach with this programs in memory so that
virus gets control whenever this program is invoked.
• This viruses are also capable of modifying Windows
tables (registries).
 Other homes for viruses
• Other home is application programs, like
spreadsheets, word processors having “macro”
feature, by which user can record series of
commands and can repeat same by single
invocation.
• Libraries are also excellent places for virus to
reside. Often libraries are called from legitimate
code and also libraries are shared between
users.
• Compilers, loaders, linkers, runtime debuggers
and even virus control programs are good
candidates for hosting viruses as they are mostly
shared.
 Virus signatures

• Viruses executes in a particular way, using

certain methods leaving some patterns.
• These patterns of virus can be used to design
programs like “virus scanners”.
• Patterns can be:
1.Storage patterns
2.Execution patterns
3.Transmission patterns
• Symantec reports on viruses gives statistical
information on viruses.
 Storage pattern

• Often attached virus piece is invariant, so the

start of the virus codes becomes detectable.
• Virus attaches itself to a file, increasing the size
of the file.
• Else, virus can obliterate the actual code, which
will not increase the size of the code but
impacts the program functioning.
• Virus scanner can use a code or checksum to
detect changes to a file. It can also look for
suspicious statements like JUMP at the starting
instruction of the code.
 Execution pattern

• Most of the operations that a virus

does are the common operations like
removing directory, modifying files
etc. which are common in OS.
• Damage is bounded only by the
creativity of the virus’s writer.
 Transmission pattern

• Virus travel is not confined to any single

medium or execution pattern.
• A virus may come through a network,
reside in disk, may get attached to a
program in execution, while executing
may transfer a copy of itself to memory
staying there as a resident and etc.
• These transmissions have to be observed
in order to detect virus patterns in the
system.
 Polymorphic viruses

• Virus signatures or patterns are useful for a virus

scanner to detect their existence in the systems.
• Virus scanners look for such pre-defined patterns in
the application code.
• Intelligent virus writers can change these patterns
just by sprinkling some no-ops(jumps, adding 0 to a
num, comparing with itself) to distort the pattern.
• A virus that can change its pattern/appearance is
called as a polymorphic virus.
• Ex: if a virus writer has 100 bytes of code and 50
bytes of data; there can be ‘n’ arrangements of this
code using several jump statements.
 Prevention of virus infection

• Do not receive executable code from an

unknown source.
• But today, non executable file can have
executable code, like macro’s in docs.
• Hidden extension types are another
problem, which deceives the user with a fake
format.
• Hiding and making the files as read-
only will not prevent the attacks of virus.
• Some prevention steps possible are:
 Prevention of virus
infection
• Use only commercial software acquired from reliable, well-
established vendors. There is always a chance that you might
receive a virus from a large manufacturer with a name everyone
would recognize. However, such enterprises have significant
reputations that could be seriously damaged by even one bad
incident, so they go to some degree of trouble to keep their
products virus-free and to patch any problem-causing code right
away. Similarly, software distribution companies will be careful
about products they handle.
• Test all new software on an isolated computer. If you must
use software from a questionable source, test the software first on
a computer with no hard disk, not connected to a network, and
with the boot disk removed. Run the software and look for
unexpected behavior, even simple behavior such as unexplained
figures on the screen. Test the computer with a copy of an up-to-
date virus scanner, created before running the suspect program.
Only if the program passes these tests should it be installed on a
less isolated machine.
• Open attachments only when you know them
to be safe. What constitutes "safe" is up to you,
as you have probably already learned in this
chapter. Certainly, an attachment from an
unknown source is of questionable safety. You
might also distrust an attachment from a known
source but with a peculiar message.
• Make a recoverable system image and store
it safely. If your system does become infected,
this clean version will let you reboot securely
because it overwrites the corrupted system files
with clean copies. For this reason, you must keep
the image write-protected during reboot. Prepare
this image now, before infection; after infection it
is too late. For safety, prepare an extra copy of the
safe boot image.
• Make and retain backup copies of executable
system files. This way, in the event of a virus
infection, you can remove infected files and reinstall
from the clean backup copies (stored in a secure, offline
location, of course).
• Use virus detectors (often called virus scanners)
regularly and update them daily. Many of the virus
detectors available can both detect and eliminate
infection from viruses. Several scanners are better than
one, because one may detect the viruses that others
miss. Because scanners search for virus signatures,
they are constantly being revised as new viruses are
discovered. New virus signature files, or new versions of
scanners, are distributed frequently; often, you can
request automatic downloads from the vendor's web
site. Keep your detector's signature file up-to-date.
Truths and misconceptions about viruses

1. Viruses can infect only Microsoft Windows systems.

False
2. Viruses can modify "hidden" or "read only" files.
True.
3. Viruses can appear only in data files, or only in
Word documents, or only in programs. False.
4. Viruses spread only on disks or only in e-mail.
False
5. Viruses cannot remain in memory after a complete
power off/power on reboot. True.
6. Viruses cannot infect hardware. True.
7. Viruses can be malevolent, benign, or benevolent.
True
 Controls against program
threats
• Controlling against program threats
is most effectively accomplished
during the software development
process
• I.e. during the design,
implementation, and testing of the
software program.
 Modularity
• Modular software programs are composed of
many subcomponents(modules), which are
defined on a logical or functional basis.
• Each module should meet four criteria:
• It performs only one function
• It is small
• It is simple
• It performs the task in isolation
• By building software using a set of small,
independent modules, the likelihood of security
flaws is diminished.
 Encapsulation
• If a component is isolated from the effects of
other components, then it is easier to trace a
problem to the fault that caused it and to
limit the damage the fault causes.
• It is also easier to maintain the system,
since changes to an isolated component do not
affect other components.
• And it is easier to see where vulnerabilities
may lie if the component is isolated.
• We call this isolation encapsulation.
 Information hiding
• Information hiding is another
characteristic of modular software.
• When information is hidden, each
component hides its precise
implementation or some other design
decision from the others.
• Thus, when a change is needed, the
overall design can remain intact
while only the necessary changes are
made to particular components.
 Mutual suspicion and
Confinement
Mutual suspicion
• When two programs must interact, each should be suspicious
of the other
• A program should never fully trust another program
• A program should protect itself by:
Granting other programs only limited access
Checking/Validating input values supplied by other programs
Confinement
• Programs that are suspected of being untrustworthy should
be confined by the operating system in order to protect the
system and other programs.
• A confined program has very limited access to system
resources and data.
 Peer Reviews
• In order to limit security flaws in software, the computer code
written by one programmer may be reviewed by another
Non malicious security flaws
• A peer review process can allow unintentional security flaws to
be detected before they become a problem
• Often when writing software, a programmer can become blind to
his or her own mistakes- a peer review can catch these mistakes.
Malicious security flaws
• A peer review process also greatly limits the possibility of
intentional, malicious security flaws becoming part of a software
program.
• It is much more difficult for a programmer to embed a malicious
security flaw in a program if his or her work is being reviewed by
other programmers- unless there is collusion among all the
programmers
 Configuration Management
• During the software development process, it is
important to keep track of who is making which
changes to the program-it is accomplished using
configuration management.
• Changes to one part of a program can have
unintentional(or intentional) consequences for other
parts of the program.
• Configuration management allows undesirable changes
to be rolled back as necessary.
• Configuration management is particularly important for
large software development projects and for the
development of open-source software programs.
 Other Software Development
Security Controls
Hazard Analysis
• A set of systematic techniques designed to reveal hazardous system
states.
Prediction
• Predicting which security problems are most dangerous or most likely to
occur.
• These predictions allow us to manage risk by targeting our limited
resources to where they will be most useful.
Static Analysis
• Examining a program’s code and design specifications for evidence of
potential security threats.
• Conducted before a system is developed-focuses on control flows, data
flows, and data structure.
Analysis of mistakes
• Learning from security breaches in order to avoid making the same
mistakes in the future.
• Requires that we document our design decisions so that we can review
them when security breaches are identified.
 Testing
• Programs should extensively tested prior to development in order to
ensure quality, stability and accuracy.
• Unit testing
Testing(in isolation) the functionality and behavior of each
component that comprises the program
• Integration Testing
Testing the extent to which system components interact with one
another as intended.
• Function Testing
Evaluating the ability of the system to perform the functions for
which it was designed
• Performance Testing
Evaluating the ability of the system to achieve the performance
standards for which it was designed.
• Acceptance Testing
Evaluating the system capabilities against the requirements
provided by the customer
 Testing(continued)
Installation Testing
Evaluating whether the system functions correctly in its
final production environement
Black box testing
• Ensuring that a system generates correct output value in
light of known input values
• Does not consider the internal workings of the system.
White box testing
• Testing a program by simulating an attack from external or
internal threats
Regression testing
• Testing whether all system functionalities are still working
properly after change has been made.
 Covert Channels: Programs That Leak
Information

• So far, we have looked at malicious code that

performs unwelcome actions.
• Next, we turn to programs that communicate
information to people who should not receive
it.
• The communication travels unnoticed,
accompanying other, perfectly proper,
communications.
• The general name for these extraordinary
paths of communication is covert channels.
 Covert Channels
• In computer security, a covert channel is a type
of attack that creates a capability to transfer
information objects between processes that are
not supposed to be allowed to communicate by
the computer security policy. 
• A covert channel is so called because it is hidden from
the access control mechanisms of secure operating
systems since it does not use the legitimate data
transfer mechanisms of the computer system
(typically, read and write), and therefore cannot be
detected or controlled by the security mechanisms
that underlie secure operating systems.
 Human example of covert
channel
• Suppose a group of students is preparing for an exam
for which each question has four choices (a, b, c, d);
one student in the group, Sophie, understands the
material perfectly and she agrees to help the others.
• She says she will reveal the answers to the
questions, in order, by coughing once for answer "a,"
sighing for answer "b," and so forth.
• Sophie uses a communications channel that
outsiders may not notice; her communications
are hidden in an open channel. This
communication is a human example of a covert
channel.
• Storage channels - Communicate by
modifying a "storage location", such
as a hard drive.
• Timing channels - Perform operations
that affect the "real response time
observed" by the receiver.
 Timing Channels
• Other covert channels, called timing channels, pass
information by using the speed at which things
happen. Actually, timing channels are shared
resource channels in which the shared resource is
time.
• A service program uses a timing channel to communicate by
using or not using an assigned amount of computing time.
• In the simple case, a multiprogrammed system with two
user processes divides time into blocks and allocates blocks
of processing alternately to one process and the other.
• A process is offered processing time, but if the process is
waiting for another event to occur and has no processing to
do, it rejects the offer. The service process either uses its
block (to signal a 1) or rejects its block (to signal a 0).