Operating System Security

Prepared by:
Dr. Avita Katal
Assistant Professor (SG)
School of Computer Science
UPES, Dehradun
 Introduction-Operating System Security
 Operating System Security refers to the set of measures, protocols, and strategies implemented to
safeguard the integrity, confidentiality, and availability of an operating system. It involves the protection
of system resources, user data, and critical processes against unauthorized access, malicious activities,
and potential security threats.

 A typical commodity operating system has complete control of all (or almost all) hardware on the machine
and is able to do literally anything the hardware permits. That means it can control the processor, read and
write all registers, examine any main memory location, and perform any operation one of its peripherals
supports. As a result, among the things the OS can do are:
• examine or alter any process’s memory
• read, write, delete or corrupt any file on any writeable persistent storage medium, including hard
disks and flash drives
• change the scheduling or even halt execution of any process
• send any message to anywhere, including altered versions of those a process wished to send
• enable or disable any peripheral device
• give any process access to any other process’s resources
• arbitrarily take away any resource a process controls
• respond to any system call with a maximally harmful lie.
In essence, processes are at the mercy of the operating system.
 Introduction-Operating System Security
• It is nearly impossible for a process to ‘protect’ any part of itself from a malicious operating
system. We typically assume our operating system is not actually malicious , but a flaw that
allows a malicious process to cause the operating system to misbehave is nearly as bad,
since it could potentially allow that process to gain any of the powers of the operating
system itself.
• At a high conceptual level, they have defined three big security-related goals that are common
to many systems, including operating systems. They are:
 Confidentiality –For example, you don’t want someone to learn what your credit card
number is – you want that number kept confidential.
 Integrity –For example, if you’ve placed an online order for delivery of one pepperoni
pizza, you don’t want a malicious prankster to change your order to 1000 anchovy pizzas.
One important aspect of integrity is authenticity. It’s often important to be sure not only
that information has not changed, but that it was created by a particular party and not by an
adversary.
 Availability –For example, if your business is having a big sale, you don’t want your
competitors to be able to block off the streets around your store, preventing your customers
from reaching you
 Operating System Security
• Authentication
• One Time passwords
• Program Threats
• System Threats
• Computer Security
Classifications
 Operating System Security
Authentication
Authentication refers to identifying each user of the system and associating the executing
programs with those users. It is the responsibility of the Operating System to create a
protection system which ensures that a user who is running a particular program is
authentic. Operating Systems generally identifies/authenticates users using following three
ways:
• Username / Password - User need to enter a registered username and password
with Operating system to login into the system.

• User card/key - User need to punch card in card slot, or enter key generated by key
generator in option provided by operating system to login into the system.

• User attribute - fingerprint/ eye retina pattern/ signature - User need to pass his/her
attribute via designated input device used by operating system to login into the
system.
 Operating System Security
One Time Passwords

One time passwords provides additional security along with normal authentication. In One-
Time Password system, a unique password is required every time user tries to login into the
system. Once a one-time password is used then it can not be used again. One time password
are implemented in various ways.

Random numbers - Users are provided cards having numbers printed along with
corresponding alphabets. System asks for numbers corresponding to few alphabets randomly
chosen.

Secret key - User are provided a hardware device which can create a secret id mapped
with user id. System asks for such secret id which is to be generated every time prior to login.

Network password - Some commercial applications send one time password to user on
registered mobile/ email which is required to be entered prior to login.
 Operating System Security
Program Threats
Operating system's processes and kernel do the designated task as instructed. If a user program
made these process do malicious tasks then it is known as Program Threats. One of the common
example of program threat is a program installed in a computer which can store and send user
credentials via network to some hacker. Following is the list of some well known program threats:
• Trojan Horse - Such program traps user login credentials and stores them to send to malicious
user who can later on login to computer and can access system resources.

• Trap Door - If a program which is designed to work as required, have a security hole in its
code and perform illegal action without knowledge of user then it is called to have a trap door.

• Logic Bomb - Logic bomb is a situation when a program misbehaves only when certain
conditions met otherwise it works as a genuine program. It is harder to detect.

• Virus - Virus as name suggest can replicate themselves on computer system. They are highly
dangerous and can modify/delete user files, crash systems. A virus is generally a small code
embedded in a program. As user accesses the program, the virus starts getting embedded in other
files/ programs and can make system unusable for user.
 Operating System Security
System Threats
System threats refers to misuse of system services and network connections to put user in
trouble. System threats can be used to launch program threats on a complete network called as
program attack. System threats creates such an environment that operating system
resources/ user files are misused. Following is the list of some well known system threats.

Worm -Worm is a process which can choke down a system performance by using system
resources to extreme levels. A Worm process generates its multiple copies where each copy
uses system resources, prevents all other processes to get required resources. Worms processes
can even shut down an entire network.

Port Scanning - Port scanning is a mechanism or means by which a hacker can detects
system vulnerabilities to make an attack on the system.

Denial of Service - Denial of service attacks normally prevents user to make legitimate use
of the system. For example, user may not be able to use internet if denial of service attacks
browser's content settings.
 Term Description Characteristics Examples
- Intentionally inserted by developers or attackers. - Allows
Adding a hidden account
Secret backdoor unauthorized access without authentication. - Typically hidden
Trap with administrative
in software or from users and administrators. - Can be used for legitimate
Door privileges in a software
system purposes (e.g., debugging) or malicious activities (e.g.,
application.
unauthorized access).
Malicious - Appears to be benign or useful software to deceive users. -
A fake antivirus software
software When executed, performs malicious actions without user's
Trojan that, when installed, steals
disguised as knowledge. - Often used to gain unauthorized access, steal data,
Horse sensitive information from the
legitimate or damage the system. - Relies on social engineering to trick
user's system.
software users into executing it.
- Infects host files or programs by inserting its code. - Spreads
Self-replicating The "Melissa" virus, which
by attaching itself to executable files or documents. - Can
malware that spread via email attachments
Virus replicate and spread to other systems when infected files are
infects files or and replicated itself to other
shared or executed. - Can cause damage by corrupting files,
programs users' address books.
stealing data, or disrupting system operations.
- Spreads independently across networks without user
Self-replicating The "Conficker" worm,
intervention. - Exploits vulnerabilities in network services to
malware that which exploited vulnerabilities
Worm propagate. - Can spread rapidly and cause widespread damage.
spreads over in Windows systems to spread
- Does not require a host file or program to propagate. - Can
networks across networks worldwide.
carry payloads for various malicious activities.
 Term Description Characteristics Examples
- Code inserted into a system or application to
execute a malicious action when certain conditions
A programmer inserts code
Malicious code are met. - Often designed to activate after a
into a company's financial
Logic Bomb triggered by specific predefined event or time. - Can cause damage or
software to delete critical
conditions disruption to the system or data when triggered. -
data if they are terminated.
Typically hidden within legitimate code or
applications.
- Scans network hosts to discover open ports and
An attacker uses a port
services. - Used by attackers to identify potential
Probing a computer scanner to identify open ports
vulnerabilities and targets for exploitation. - Can be
Port Scanning system or network for on a target network and
conducted using various techniques, such as TCP
open ports exploits vulnerabilities in
SYN scanning, UDP scanning, or service version
exposed services.
detection.
- Overwhelms a target system or network with a
large volume of traffic or requests. - Aimed at Launching a DDoS attack
Flooding a target
Denial of disrupting normal operations or rendering a service against a website to make it
system with excessive
Service (DoS) unavailable to legitimate users. - Can be achieved unavailable to legitimate users
requests
using various methods, such as SYN flooding, UDP by flooding
flooding, or HTTP flooding.
 Operating System Security
Computer Security Classifications (https://
en.wikipedia.org/wiki/Trusted_Computer_System_Evaluation_Criteria)
As per the U.S. Department of Defense Trusted Computer System's Evaluation Criteria there are
four security classifications in computer systems: A, B, C, and D.
S. No Classification Description
Type
1 Type A Highest Level. Uses formal design specifications and verification techniques. Grants a
(Verified high degree of assurance of process security. Examples are: Honeywell's SCOMP, Aesec's
Protection) GEMSOS, and Boeing's SNS Server.
2 Type B Provides mandatory protection system. Have all the properties of a class C2 system.
(Mandatory Attaches a sensitivity label to each object. It is of three types.
Protection) B1-Labeled Security Protection - Maintains the security label of each object in the
system. Label is used for making decisions to access control.
B2-Structured Protection - Extends the sensitivity labels to each system resource,
such as storage objects, supports covert channels and auditing of events. Example:
Multics
B3-Security Domains - Allows creating lists or user groups for access-control to
grant access or revoke access to a given named object. ExampleXTS-300, a precursor
to the XTS-400
S. No Classification Description
Type
3 Type C Provides protection and user accountability using audit capabilities. It is of
(Discretionary two types:
Protection) C1- Discretionary Security Protection- Incorporates controls so that users can
protect their private information and keep other users from accidentally reading
/ deleting their data. UNIX versions are mostly Cl class.
C2 – Controlled Access Protection- Adds an individual-level access control to
the capabilities of a Cl level system Example-HP-UX
4 Type D Lowest level. Minimum protection. MS-DOS, Window 3.1 fall in this category.
(Minimal Reserved for those systems that have been evaluated but that fail to meet the
Protection) requirement for a higher division
Level Description Example
High-assurance systems that have undergone rigorous formal verification to ensure
A Verified Protection security. These systems are typically used in critical environments such as military or
financial institutions.
Systems that enforce mandatory access control (MAC) based on sensitivity labels attached
B1 Labeled Security Protection
to objects. Access decisions are made based on these labels.
Systems with controlled access and auditing capabilities. These systems provide a
B2 Structured Protection structured approach to security, including user authentication and logging of security-relevant
events.
Systems that allow finer-grained access control through access control lists (ACLs) or
B3 Security Domains user groups. Administrators can specify which users or groups are granted or denied access to
specific resources.
Systems with basic access controls where users have discretion over the access
Discretionary Security
C1 permissions of objects they own. Access decisions are typically based on the permissions set
Protection
by the owner.
Systems with enhanced access controls and auditing capabilities. These systems enforce
C2 Controlled Access Protection stricter controls over user access and include features such as login banners, password
protection, and auditing of security-relevant events.

Systems with minimal security controls, often lacking in authentication, access controls,
D Minimal Security Protection or auditing capabilities. These systems are typically single-user systems without network
connectivity.