CHAPTER FOUR

Operating System Security

1
 Outline
• Operating System overview
• Security features of an operating system
• Hardening the operating system
• Operating system Security attack types
• Protecting an OS from Malicious Software
• The Components of an OS Security Environment
• OS Authentication Methods
• Authorization
• Maintenance, backup, and restore
• Creating and Implementing Organizational Policies

2
 What does Operating System do?
• Manages all the resources in a
computer (including processor,
memory, i/o devices)
• Provides an interface between
the hardware and application
software.
• Three layers:
• Inner layer, computer hardware
• Middle layer, operating system
• Outer layer, different softwares

3
 Operating System Overview
• Key functions of an operating system:

– Multiprogramming, multiprocessor
– Computer resource management
– Provides a user interface
– Runs software utilities and programs
– Schedule jobs
– Provide tools to configure the operating system
and hardware
– Administers user actions and accounts
– Enforce security measures
4
Operating System functions

5
 Why need security at the OS level?
• No more standalone computer system environments.

• Any system can be globally accessible through a set of

vast inter and intra-network connections.

• Transition motivated by the need to work remotely,

– Remote login, online shopping etc.

• A single security loophole in the OS design and

implementation known to a malicious attacker could do
serious damage.

6
 Security in Operating System

• Security refers to providing a protection system to

computer system resources such as:
– CPU, memory, disk
– software programs and
– most importantly data/information stored in the computer
system.

• So a computer system must be protected against

– unauthorized access by users and
– malicious access to system including viruses, worms etc…

7
 Security features in Operating System
• An operating system manages and controls access to
hardware components
• Older operating systems focused on ensuring data
confidentiality
• Modern operating systems support four basic functions
– Positively identify a user
– Restrict access to authorized resources
– Record user activity
– Ensure proper communications with other computers and
devices (sending and receiving data)

8
 Security features in ordinary OS
• Authentication of users
– password comparison
• Protection of memory
– user space, paging, segmentations
• File and I/O device access control
– access control matrix
• Enforcement of sharing resources
– To preserve integrity, consistency (critical section)
9 9
 Security features in ordinary OS…
• Fair service
– no starvation and deadlock
• Inter-process communication & synchronization
– Shared variable (e.g, using semaphores)
• Protection of data
– encryption, isolation

10 10
 Security features of Trusted OS
 Identification and Authentication
 Mandatory (enforce multilevel security by classifying the
data and users into various security classes)
 Top Secret, secrete, confidential, unclassified
 Discretionary Access Control (grant privileges to users)
 Object use and reuse protection (Subject and object)
 Anti-virus scan
 Accountability and Audit (security log)
 Firewall
 Intrusion detection (patterns of normal system usages,
anomalies)
11 11
 Hardening the OS
• Default OS configurations are for ease of use
• Measures have to be done at all stages
– Installing and patching
– Configuring
• Remove unnecessary applications, services and protocols
• Users, groups, controls and privileges

– Install additional software (anti-virus, firewall, intrusion

detection system, etc.)
– Test security

12
Operating system Security attack types

13
 Name Description
Virus Attaches itself to a program and propagates copies of itself to
other programs
Worm Program that propagates copies of itself to other computers
Logic bomb Triggers action when condition occurs
Trojan horse Program that contains unexpected additional functionality
Backdoor (trapdoor) is a secret entry point into a program that allows someone
that is aware of the backdoor to gain access without going
through the usual security access procedures.
Auto-rooter Malicious hacker tools used to break into new machines
remotely
Kit (virus generator) Set of tools for generating new viruses automatically
Spammer programs Used to send large volumes of unwanted e-mail
Flooders Used to attack networked computer systems with a large
volume of traffic to carry out a denial of service (DoS) attack
Keyloggers Captures keystrokes on a compromised system
Zombie Program activated on an infected machine that is activated to
launch attacks on other machines
Rootkit Set of hacker tools used after attacker has broken into a
computer system and gained root-level access 14
Malware Attack…

15
Malware Attack…

16
 Malware Attack…
 Viruses
- A malicious code that replicates and hides itself inside
other programs usually without your knowledge.
- A virus is a piece of software that can "infect" other
programs by modifying them.
- Similar to biological virus: replicates and spreads
- Can do serious damage such as erasing file…

 Worms
• A worm is a program that can replicate itself and send
copies from computer to computer across network
connections.

17
 More on Virus
During its lifetime, a typical virus goes through the following four phases:
• Dormant phase: The virus is idle.
– The virus will eventually be activated by some event, such as a date, the presence of
another program or file, or the capacity of the disk exceeding some limit.
• Propagation phase: The virus places an identical copy of itself into other
programs or into certain system areas on the disk.
– Each infected program will now contain a clone of the virus, which will itself enter a
propagation phase.
• Triggering phase: The virus is activated to perform the function for which it
was intended.
– As with the dormant phase, the triggering phase can be caused by a variety of system
events

• Execution phase: The function is performed.

– The function may be harmless, such as a message on the screen, or
– damaging, such as the destruction of programs and data files.
18
 More on Virus…
Types of viruses
• Parasitic virus: The traditional and still most common form of
virus.
– A parasitic virus attaches itself to executable files and replicates
• Memory-resident virus: resides in main memory as part of a
resident system program.
– From that point on, the virus infects every program that executes.
• Boot sector virus: Infects a master boot record or boot record
and spreads when a system is booted from the disk containing the
virus.
• Stealth virus: A form of virus explicitly designed to hide itself
from detection by antivirus software.
– a virus that uses compression so that the infected program is exactly the
same length as an uninfected version.
19
 More on Virus…
Types of viruses…

 Polymorphic virus: A virus that mutates with every infection,

making detection by the "signature" of the virus impossible.

 Metamorphic virus: As with a polymorphic virus, a metamorphic

virus mutates with every infection.
– The difference is that:
– a metamorphic virus rewrites itself completely at each iteration,
increasing the difficulty of detection.
– Metamorphic viruses my change their behavior as well as their
appearance.

20
 Malware Attack…
• Logic bomb
 The logic bomb is code embedded in some legitimate program
that is set to "explode" when certain conditions are met.
 Examples of conditions that can be used as triggers for a logic
bomb are:
• the presence or absence of certain files,
• a particular day of the week or date, or
• a particular user running the application.

 Once triggered, a bomb may:

• alter or delete data or entire files,
• cause a machine halt, or
• do some other damage.

21
 Malware attack..
Trojan Horse
•Any malicious program which
misinterprets itself as useful, or
interesting in order to convince a
victim to install it.
•The program claims to do one thing
– (it may claim to be a game) but
instead does damage when you run it
(it may erase your Hard Disk).
•Trojan horse programs do not
replicate themselves like a virus,
•Such program traps user login
credentials and stores them to send to
malicious user 22
 Spyware
• Software placed on a computer
– typically without user’s knowledge
– reports back information about user’s activities
• Some operate through monitoring cookies
• A software that literally spies on what you do on
your computer.
– Example: Simple Cookies, mobile codes , web
crawlers, Xerox
– Types of information that is gathered includes the Websites
visited, browser and system information, and your computer
IP address.
23
 Spam (junk mail)
• Filling e-mail inboxes with unwanted junk mail.

• Anyone using e-mail is essentially guaranteed to

receive spam

• How spammers get your mail.

– Web search
– Sending test emails
– Exchange or buy from other spammers
24
 Malware attacks…
• Infection mechanisms
 First, the virus should search for and detect objects to infect
 Installation into the infectable object
• Writing on the boot sector
• Add some code to executable programs
• Add some code to initialization/auto-executable programs
• Trigger mechanism
 Date
 Number of infections
 First use

• Effects: It can be anything

 a message
 deleting files
 formatting disk
 overloading processor/memory
 etc...
25
 Suggestions to secure your computer
• Use anti-virus software.
• Depending on the vendor, the antivirus software may also
contain anti-spyware tools, anti-spam filtering, a personal
firewall, and more.
• Update your computer regularly.
• Be careful with the email attachments
– Safe: .jpg .bmp .pdf .txt ….
– Unsafe: .exe .doc .xls .ppt …
• Avoid email software by Microsoft (outlook, outlook
express…). They are mostly targeted.
• Use firewall to protect you from malware attack.
• Use IDS…
26
Protecting an OS from Malicious Software

• Install updates
• Use malicious software scanners
• Back up systems and create repair disks
• Create and implement organizational policies

27
 Installing Updates for Windows
• Windows Update
– Provides access to patches that are regularly issued/released
• Service packs
– Address security issues and problems affecting stability,
performance, or operation of features included with the OS
• Patch This fixes something small and is usually quick to
download and install.
• Rollup This might include a group of patches for a program.
• Update Updates might add or fix features in your program or
fix an earlier patch.
• Service Pack This is the biggie; the one you read about in the
news when Microsoft releases some big service pack.

28
Automatic Updates

29
Protecting an OS from Malicious Software

• Install updates
• Use malicious software scanners
• Back up systems and create repair disks
• Create and implement organizational policies

30
 Using Malicious Software Scanners

• Effective way to protect operating system

• Scan systems for virus, worms, and Trojan
horses
• Often Called Virus Scanners
• Functions of anti-viruses
- Identification of known viruses
- Detection of suspected viruses
- Blocking of possible viruses
- Disinfection of infected objects
- Deletion and overwriting of infected objects
31
 Malicious Software Scanners…

• Automatically runs at a scheduled time

• Manual run option
• Detects known and unknown malicious software
• Updates for new malicious software
• Scans files that are downloaded
• Uses protected or quarantined zones for
downloaded files

32
 Suggestions to fight spam
• Never reply junk emails
• Do not open any files or executable attachments
• Immediately DELETE the malicious email
• Don’t post your actual email address in the website.
• Norton, McAfee, and many more include spam as one
of the threats that they protect against.
• Can use spam filtering applications

33
Virus Scanning Software…

Depending on the vendor, the antivirus software

may also contain anti-spyware tools, anti-spam
filtering, a personal firewall, and
more.

continued… 34
 The Components of an OS Security
Environment

• Three components:
– Services
– Files
– Memory

35
 OS Security: Services

• Main component of operating system security

environment
• Used to gain access to the OS and its features
• Include
– User authentication
– Remote access
– Administration tasks
– Password policies

36
 OS Security: Files
• Common threats to file system:
– File permission
– File sharing

• Files must be protected from unauthorized reading

and writing actions

• Data resides in files;

– protecting files means protects data

37
 OS Security: File Permissions
• Read, write, and execute privileges
• In Windows:
– Change permission on the Security tab on a file’s Properties
dialog box
– Allow indicates grant;
– Deny indicates revoke
• In UNIX/Linux
– Three permission settings:
• owner; group to which owner belongs; all other users
– Each setting consist of rwx
• r for reading, w for writing, and x for executing
– CHMOD command used to change file permissions 38
 File Permissions

• One can easily view the permissions for a file by invoking a long
format listing using the command ls -l.

• For instance, if the user Abe creates an executable file named

test, the output of the command ls -l test would look like this:

rwxrwxr-x Abe student Sep 26 12:25 test.l

39
 Access Permissions
• This listing indicates that the file is readable, writable,
and executable by the user who owns the file (user
Abe)
• as well as the group owning the file (which is a group
named student).
• The file is also readable and executable, but not
writable by other users.

rwxrwxr-x Abe student Sep 26 12:25 test.l

40
 Access Permission of File/Directory
• The ownership of the file or directory can be changed using
the command
– chown <owner> <file/directory name>
• The group of the file or directory can be changed using the
command
– chgrp <group> <file/directory name>
• The permissions of the file can be changed using chmod
command
– chmod -R ### <filename or directory>
• -R is optional and when used with directories will traverse all
the sub-directories of the target directory changing ALL the
permissions to ###.
41
 Access Permission of File/Directory

The #'s can be:

0 = Nothing
1 = Execute
2 = Write
3 = Execute & Write (2 + 1)
4 = Read
5 = Execute & Read (4 + 1)
6 = Read & Write (4 + 2)
7 = Execute & Read & Write (4 + 2 + 1)

42
 OS Security: Memory

• Hardware memory available on the system can

be corrupted by badly written software
• Can harm data integrity
• Two options:
– Stop using the program
– Apply a patch (service pack) to fix it

43
 OS Authentication Methods
• Authentication:
– Verifies user identity; something a person is, has, or does.
– Permits access to the operating system
– Use of biometrics, passwords, passphrase, token, or other
private information.
– Strong Authentication is important
• Physical authentication:
– Allows physical entrance to company property
– Magnetic cards and biometric measures
• Digital authentication:
– verifies user identity by digital means
44
 OS Authentication Methods…
• Biometrics
• Verifies an identity by analyzing a unique person attribute or behavior
(e.g., what a person “is”).
• Most expensive way to prove identity, also has difficulties with user
acceptance.
• Most common biometric systems:
– Fingerprint
– Palm Scan
– Hand Geometry
– Iris Scan
– Voice Print
– Facial Scan

45
 OS Authentication Methods…
• Passwords
– User name + password most
common identification,
authentication scheme.
– Weak security mechanism, must
implement strong password
protections

• Passphrase
– Is a sequence of characters that is
longer than a password.
– Takes the place of a password.
– Can be more secure than a
password because it is more
complex. 46
 OS Authentication Methods…

• Digital certificates: digital passport that identifies

and verifies holder of certificate

• Kerberos:
– Developed by MIT
– Uses tickets for authentication purposes

47
 OS Authentication Methods…
• Digital card:
– Also known as a security card or smart card
– Similar to a credit card; uses an electronic circuit
instead of a magnetic strip
– Stores user identification information

48
 OS Authentication Methods…
Digital token (security token)
• They are similar to smart cards in
functionality as
– Key is generated inside the
token.
– Key is highly secured as it
doesn’t leave the token.

• Small electronic device – Highly portable.

• Displays a number unique to the – Machine Independent.

token holder; used with the
holder’s PIN as a password • iKEY is one of the most
commonly used token as it
doesn’t need a special reader and
can be connected to the system
using USB port.

Biometrics – adds another level of security to these tokens/cards

 OS Authentication Methods…
• Secure Socket Layer (SSL):
– authentication information is transmitted over the network
in an encrypted form

• Public Key Infrastructures (PKI):

– User keeps a private key
– Authentication firm holds a public key
– Encrypt and decrypt data using both keys

50
 Authorization
• Process that decides whether users are
permitted to perform the functions they
request
• Authorization is not performed until the user is
authenticated
• Deals with privileges and rights (User
administration)
• Create user accounts
• Grant privileges to users…
51
 Maintenance

• Maintenance involves:
– Monitoring and analyzing logging information
– Performing regular backups
– Recovering from security compromises
– Restoring systems to its previous point
– Regular testing of security
– Patch, update, and revise critical software

52
 Data Backup
• Backup is the act of creating copies of information
such that it may be recovered

• Archive is to keep these backups for a long period

of time

• Data may be lost accidentally (hardware failures,

human mistake) or intentionally

53
 Restore
• Restoring the computer system to an earlier point in
time
• System restore can resolve many system problems
• It is the best recovery methods to try first
• It undo recent system changes, but leave files such as
documents, pictures… unchanged
• System restore remove recently installed programs and
drives

54
 Creating and Implementing Organizational
Policies
• Provide users with training in security techniques
• Train users about common malicious software
• Require users to scan flush disks and CDs before
use
• Establish policies about types of media that can
be brought in from outside and how they can be
used
• Establish policies that discourage/prevent users
from installing their own software
55
 Creating and Implementing Organizational
Policies
• Define policies that minimize/prevent
downloading files;
• require users to use a virus scanner on any
downloaded files
• Create quarantine areas for files of uncertain
origin
• Use virus scanning on e-mail and attachments
• Discard e-mail attachments from unknown or
untrusted sources
56