Operating System Security

Vishal Patel Amlan Das

Bachelors of technology Bachelors of technology Varsha Yadav
(CSE) (CSE) Bachelors of technology
Lovely Professional University Lovely Professional University (CSE)
Jalandhar, India Jalandhar, India Lovely Professional University
[email protected] [email protected] Jalandhar, India
[email protected]
Muskaan Singh Gagandeep kaur
Bachelors of technology Bachelor of technology
(CSE) (CSE)
Lovely Professional University Lovely Profesional University
Jalandhar, India Jalandhar, India
[email protected] [email protected]

Abstract— Operating systems (OS) security is a very be bypassed. The OS should confine an application to a unique
important aspect of the protection of computer systems and their security domain.
data’s integrity, confidentiality and availability. Consequently, this
Applications with special privileges that perform security-
research manuscript seeks to provide a comprehensive analysis of
related functions are called trusted applications. Such applications
OS security in order to enhance understanding, establish challenges
should only be allowed the lowest level of privileges required to
and understand the best way forward. The research method proc-
perform their functions. For example, type enforcement is a
ess involves major areas of studies through the literature review,
mandatory security mechanism that can be used to restrict a trusted
data collection, case studies analysis, experimental and
application to the lowest level of privileges.
comparative evaluations and qualitative analyses. From the theatre
of experimentalism, various theoretical underpinnings are applied
in studying on the historical backgrounds and contemporary frames
of work on OS security with the most emphasized questions. Using
real-life data, the data acquisition process incorporates information
gathered through report surveys, security bulletins, and case-study
research as well as interviews with experts. The application of
security policies is demonstrated in the contextual studies when
case studies are presented; on the other hand, the efficiency of the
security measures is examined in the experimental evaluation of
the security controls. Qualitative comparisons with competitors and
industry segments illustrate what is occurring in OS security and
give insight into its vulnerabilities and future development. The
analysis and integration of results provide practical implications for
improving OS security, recommendations on policy, and Fig. 1
recommendations for future research. This research enriches the
knowledge about OS security and also assists in the development Enforcing mandatory security through mechanisms left to the
of the right measures to prevent future contingencies and risks in discretion of users could lead to a breach of security due not only
computer systems. [1][2]. to malicious intent but also carelessness or lack of understanding.
Discretionary mechanisms place the burden of security
I. INTRODUCTION on individual users. Moreover, an application may change a
An operating system (OS) allows multiple applications to share the carefully defined discretionary policy without the consent of the
hardware resources of a physical system, subject to a set of user, whereas a mandatory policy can only be changed by a system
policies. A critical function of an OS is to protect applications administrator.
against a wide range of malicious attacks such as unauthorized
Unfortunately, commercial operating systems do not
access to privileged information, tempering with executable code,
support multilayered security; such systems only distinguish
and spoofing. Such attacks can now target even single-user systems
between a completely privileged security domain and a completely
such as personal computers, tablets, or smartphones. Data brought
unprivileged one. Some operating systems, such as Windows NT,
into the system may contain malicious code; this could occur via a
allow a program to inherit all the privileges of the program
Java applet, or data imported by a browser from a malicious Web
invoking it, regardless of the level of trust in that program.
site [1].
The existence of trusted paths, mechanisms supporting user
The mandatory security of an OS is considered to be “any
interactions with trusted software, is critical to system security. If
security policy where the definition of the policy logic and the
such mechanisms do not exist, malicious software can impersonate
assignment of security attributes is tightly controlled by a system
trusted software. Some systems provide trust paths for a few
security-policy administrator” [209].Access
functions such as login authentication and password changing and
control, authentication usage, and cryptographic usage policies are
allow servers to authenticate their clients [2].
all elements of mandatory OS security. The first policy specifies
how the OS controls the access to different system objects, the The solution discussed in [209] is to decompose a complex
second defines the authentication mechanisms the OS uses to mechanism into several components with well-defined roles. For
authenticate a principal, and the last specifies the cryptographic example, the access control mechanism for the application space
mechanisms used to protect the data. A necessary but not sufficient could consist of enforcer and decider components. To access a
condition for security is that the subsystems tasked with protected object, the enforcer will gather the required information
performing security-related functions are temper-proof and cannot about the agent attempting the access and will pass this information
to the decider, together with the information about the object and

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

the elements of the policy decision. Finally, it will carry out the cannot determine whether a request comes from an authorized user
actions requested by the decider. or from a malicious program; in turn, a human user cannot
distinguish a response from a malicious program impersonating the
A trusted-path mechanism is required to prevent malicious service from the response provided by the service itself [4].
software invoked by an authorized application to tamper with the
attributes of the object and/or with the policy rules. A trusted path
is also required to prevent an impostor from impersonating the
decider agent. A similar solution is proposed for cryptography
usage, which should be decomposed into an analysis of the
invocation mechanisms and an analysis of the cryptographic
mechanism.
Another question is how an OS can protect itself and the
applications running under it from malicious mobile code
attempting to gain access to the data and the other resources and
compromise system confidentiality and/or integrity. Java Security
Manager uses the type-safety attributes of Java to prevent
unauthorized actions of an application running in a “sandbox.” Yet,
the Java Virtual Machineja (JVM) accepts byte code in violation
of language semantics; moreover, it cannot protect itself from
tampering by other applications.

Fig. 3

II. PRINCIPLES OF OPERATING SYSTEM SECURITY

Operating System (OS) Security is built on a set of
fundamental principles designed to protect the integrity,
confidentiality, and availability of computing systems.
Understanding these principles is crucial for anyone involved in the
development, administration, or use of operating systems [6].
Fig. 2
A. Least Privilege:
Even if all these security problems could be eliminated, good The principle of least privilege dictates that users and programs
security relies on the ability of the file system to preserve the should operate using the least amount of privilege necessary to
integrity of Java class code. The approach to require digitally complete a task. This limits the potential damage from accidents or
signed applets and accept them only from trusted sources could fail malicious actions by reducing the access and permissions to the
due to the all-or-nothing security model. A solution to securing minimum required.
mobile communication could be to confine a browser to a distinct
security domain. B. Defense In Depth:
Defense in depth is a strategy that employs multiple layers of
Specialized closed-box platforms such as the ones on security controls and defenses throughout the system. If one layer
some cellular phones, game consoles, and automated teller fails, others still stand, providing a comprehensive approach to
machines (ATMs) could have embedded cryptographic keys that security that addresses potential physical, technical, and
allow themselves to reveal their true identity to remote systems and administrative vulnerabilities.
authenticate the software running on them. Such facilities are not
available to open-box platforms, the traditional hardware designed
for commodity operating systems [3].
A highly secure operating system is necessary but not sufficient
unto itself; application-specific security is also necessary.
Sometimes security implemented above the operating system is
better. This is the case for electronic commerce that requires a
digital signature on each transaction.
We conclude that commodity operating systems offer low
assurance. Indeed, an OS is a complex software system consisting
of millions of lines of code, and it is vulnerable to a wide range of
malicious attacks. An OS poorly isolates one application from
another, and once an application is compromised, the entire
physical platform and all applications running on it can be affected. Fig. 4
The platform security level is thus reduced to the security level of
the most vulnerable application running on the platform. C. Fail-Safe Defaults:
Operating systems provide only weak mechanisms for Systems should be designed to default to a state of security if they
applications to authenticate to one another and do not have a fail. Access decisions should deny by default, and permissions
trusted path between users and applications. These shortcomings should not be granted unless explicitly allowed. This ensures that if
add to the challenges of providing security in a distributed a failure occurs, the system remains secure [3].
computing environment. For example, a financial application
 D. Economy Of Mechanism: to the owner's linked account is a challenging endeavour. It
The design of security mechanisms should be as simple and small necessitates the establishment of a robust monitoring system
as possible. Complex systems are harder to secure because they capable of detecting unauthorized access in real-time. Once an
potentially have more vulnerabilities. Simplification aids in the attempt is identified, the system must initiate a secure data transfer
verification and validation of security properties. protocol to relocate the data to a designated secure account, which
may be cloud-based or hosted on a different server. This process
E. Complete Mediation: must guarantee data integrity and confidentiality during transfer,
Every access to every resource must be checked for authorization. thereby preventing any risk of data leakage or corruption [14].
This principle ensures that attackers cannot bypass security
controls to access sensitive resources. It requires the system to
verify permission for an action each time that action is attempted.

III. METHOLOGY
Operating System (OS) security methodology involves a
structured approach to protecting the integrity, confidentiality, and
availability of an OS and its resources. Key components include:
A. Access Control:
Implementing strict user authentication and authorization
mechanisms ensures that only authorized users can access system
resources. This includes using passwords, biometric data, and role-
based access controls [3].

Fig. 6

Furthermore, the system should incorporate a mechanism to

authenticate the legitimacy of the access attempt to prevent
accidental data deletion due to false positives. This could involve
multi-factor authentication or other verification techniques to
validate the user's identity. Additionally, the system owner should
Fig. 5 receive immediate notifications regarding the unauthorized access
attempt and the subsequent data transfer, enabling them to take
B. Security Policies: further action if necessary.

Establishing clear security policies helps define acceptable use, Establishing a backup strategy is also essential prior to
data protection standards, and incident response strategies. Policies implementing such a security measure. This ensures that even if
guide users in maintaining security and compliance [7]. data is removed from the original system, it is not permanently lost
and can be restored if needed. The backup system should be secure
C. Regular Updates and Patch Management: and regularly updated to reflect the latest data.
Keeping the OS and applications up-to-date with the latest Finally, legal and privacy considerations must be addressed
security patches mitigates vulnerabilities that can be exploited by when designing and implementing this system. Depending on the
attackers [8]. jurisdiction, there may be regulations governing data breach
responses and the protection of personal information that must be
D. Intrusion Detection and Prevention Systems (IDPS): followed. It is advisable to consult with legal experts and
Deploying IDPS can monitor system activities for suspicious cybersecurity professionals to ensure compliance and to create a
behaviour and respond to potential threats in real-time [9]. system that effectively safeguards against unauthorized access while
respecting privacy rights and regulations. Implementing such a
E. Encryption: system requires meticulous planning, a comprehensive
Utilizing encryption for data at rest and in transit protects understanding of potential risks, and a commitment to preserving
the security and privacy of the involved data.
sensitive information from unauthorized access.
F. Auditing and Monitoring: V. WHAT ARE WE PROTECTING?
Regularly reviewing logs and system activities helps identify We aren’t likely to achieve good protection unless we have a fairly
anomalies and potential security breaches. comprehensive view of what we’re trying to protect when we say
our operating system should be secure. Fortunately, that question is
G. User Education and Training: easy to answer for an operating system, at least at the high level:
Educating users about security best practices and potential everything. That answer isn’t very comforting, but it is best to have
threats is crucial in fostering a security-aware culture. a realistic understanding of the broad implications of operating
system security [12]. A typical commodity operating system has
IV. EASE OF USE complete control of all (or almost all) hardware on the machine and
Implementing a security measure that automatically deletes is able to do literally anything the hardware permits. That means it
data from a system upon unauthorized access attempts and sends it can control the processor, read and write all registers, examine any
main memory location, and perform any operation one of its
peripherals supports. As a result, among the things the OS can do
are:

 Examine or alter any process’s memory

 Read, write, delete or corrupt any file on any writeable
persistent storage medium, including hard disks and flash
drives
 Change the scheduling or even halt execution of any
process
 Send any message to anywhere, including altered
Fig. 7
versions of those a process wished to send
 Enable or disable any peripheral device C. Defense In Depth:
Methodology
 This strategy advocates for a layered security
 Give any process access to any other process’s resources approach, combining multiple security mechanisms to
protect the system.
 Arbitrarily take away any resource a process control
 If one layer is compromised, others remain in place to
 Respond to any system call with a maximally harmful lie prevent complete system compromise.
In essence, processes are at the mercy of the operating system. It is  This approach encompasses diverse security controls,
nearly impossible for a process to ’protect’ any part of itself from a including firewalls, intrusion detection systems,
malicious operating system. We typically assume our operating access controls, encryption, and regular security
system is not actually malicious, but a flaw that allows a malicious audits.
process to cause the operating system to misbehave is nearly as
bad, since it could potentially allow that process to gain any of the  For instance, a secure system might employ firewall
powers of the operating system itself. This point should make you to block unauthorized network traffic, intrusion
detection to identify suspicious activities, and strong
think very seriously about the importance of designing secure
passwords to prevent unauthorized access.
operating systems and, more commonly, applying security patches
to any operating system you are running. Security flaws in your D. Fail-Safe Defaults:
operating system can completely compromise everything about the
machine the system runs on, so preventing them and patching any  This principle emphasizes the importance of
that are found is vitally important defaulting to a secure state in case of errors or
unexpected situations.
VI. FUNDAMENTAL PRINCIPLES OF OPERATING  Access decisions should deny by default, and
SYSYTEM SECURITY permissions should be explicitly granted only when
Operating system (OS) security relies on a set of core principles necessary.
that guide its design and implementation. These principles are  This approach ensures that security is maintained
crucial for establishing a robust security posture and mitigating even in the event of system failures or
potential threats [7]. misconfigurations.
A. Economy Of Mechanism:  For example, a newly created user account should
have minimal privileges by default, requiring explicit
 Security mechanisms should be designed with
authorization for accessing specific resources.
simplicity and minimality in mind.
 Complex systems are inherently more difficult to E. Complete Mediation:
secure due to increased potential for vulnerabilities  These principal mandates that every access attempt
and unforeseen interactions. to any resource must be verified for authorization.
 A simpler design facilitates easier understanding,  This prevents attackers from circumventing security
implementation, and verification of security controls and gaining unauthorized access to sensitive
properties. information.
 For instance, a straightforward access control model  Complete mediation requires continuous monitoring
is easier to audit and maintain compared to a and enforcement of access control policies.
convoluted system with numerous exceptions and
special cases.  For example, even if a user has previously accessed a
file, the system should re-validate their permissions
B. Least Privilege: every time they attempt to access it again.
 This principle restricts user and process privileges to By adhering to these fundamental principles, operating system
the minimum necessary for performing their tasks. developers and administrators can create a secure foundation for
protecting valuable data and maintaining system integrity. However,
 By limiting access rights, the potential damage from
it is crucial to recognize that security is an ongoing process that
errors, malfunctions, or malicious activities is
requires continuous adaptation and improvement to address the
significantly reduced.
evolving threat landscape.
 Implementation involves fine-grained access control
mechanisms, role-based permissions, and privilege VII. DESIGNIG SECURE SYSTEMS
separation techniques. Few of you will ever build your own operating system, nor even
 For example, a user account responsible for web make serious changes to any existing operating system, but we
server maintenance should not have access to expect many of you will build large software systems of some kind.
sensitive financial data. Experience of many computer scientists with system design has
shown that there are certain design principles that are helpful in
building systems with security requirements [6]. These principles H. Acceptability –
were originally laid out by Jerome Saltzer and Michael Schroeder in A critical property not dear to the hearts of many programmers. If
an influential paper [SS75], though some of them come from earlier your users won’t use it, your system is worthless. Far too many
observations by others. While neither the original authors nor later promising secure systems have been abandoned because they asked
commentators would claim that following them will guarantee that too much of their users.
your system is secure, paying attention to them has proven to lead to
more secure systems, while you ignore them at your own peril. These are not the only useful pieces of advice on designing secure
We’ll discuss them briefly here. If you are actually building a large systems out there. There is also lots of good material on taking the
software system, it would be worth your while to look up this paper next step, converting a good design into code that achieves the
(or more detailed commentaries on it) and study the concepts security you intended, and other material on how to evaluate
carefully. whether the system you have built does indeed meet those goals.
These issues are beyond the scope of this course, but are extremely
important when the time comes for you to build large, complex
systems. For discussion of approaches to secure programming, you
might start with Seacord [SE13], if you are working in C. If you are
working in another language, you should seek out a similar text
specific to that language, since many secure coding problems are
related to details of the language. For a comprehensive treatment on
how to evaluate if your system is secure, start with Dowd et al.’s
work [D+07].

VIII. THE BASICS OF OS SECURITY

In a typical operating system, then, we have some set of security
goals, centered around various aspects of confidentiality, integrity,
and availability. Some of these goals tend to be built in to the
operating system model, while others are controlled by the owners
Fig. 8 or users of the system. The built-in goals are those that are
extremely common, or must be ensured to make the more specific
A. Economy of mechanism — goals achievable [3][4]. Most of these built-in goals relate to
This basically means keep your system as small and simple as controlling process access to pieces of the hardware. That’s because
possible. Simple systems have fewer bugs and it’s easier to the hardware is shared by all the processes on a system, and unless
understand their behavior. If you don’t understand your system’s the sharing is carefully controlled, one process can interfere with the
security goals of another process. Other built-in goals relate to
behavior, you’re not likely to know if it achieves its security goals.
services that the operating system offers, such as file systems,
B. Fail-safe defaults — memory management, and communications. If these services are not
carefully controlled, processes can subvert the system’s security
Default to security, not insecurity. If policies can be set to goals. Clearly, a lot of system security is going to be related to
determine the behavior of a system, have the default for those process handling. If the operating system can maintain a clean
policies be more secure, not less. separation of processes that can only be broken with the operating
system’s help, then neither shared hardware nor operating system
C. Complete mediation –
services can be used to subvert our security goals [6]. That
This is a security term meaning that you should check if an action requirement implies that the operating system needs to be careful
to be performed meets security policies every single time the action about allowing use of hardware and of its services. In many cases,
is taken. the operating system has good opportunities to apply such caution.
For example, the operating system controls virtual memory, which
D. Open design – in turn completely controls which physical memory addresses each
Assume your adversary knows every detail of your design. If the process can access. Hardware support prevents a process from even
system can achieve its security goals anyway, you’re in good naming a physical memory address that is not mapped into its
shape. This principle does not necessarily mean that you actually virtual memory space. (The software folks among us should
tell everyone all the details, but base your security on the remember to regularly thank the hardware folks for all the great
assumption that the attacker has learned everything. He often has, stuff they’ve given us to work with.)
in practice. When a process performs a system call, then, the operating system
E. Separation of privilege – will use the process identifier in the process control block or
similar structure to determine the identity of the process. The OS
Require separate parties or credentials to perform critical actions. can then use access control mechanisms to decide if the identified
For example, two-factor authentication, where you use both a process is authorized to perform the requested action. If so, the OS
password and possession of a piece of hardware to determine either performs the action itself on behalf of the process or arranges
identity, is more secure than using either one of those methods for the process to perform it without further system intervention
alone.
[2]. If the process is not authorized, the OS can simply generate an
F. Least privilege – error code for the system call and return control to the process, if
the scheduling algorithm permits.
Give a user or a process the minimum privileges required to
perform the actions you wish to allow. The more privileges you
give to a party, the greater the danger that they will abuse those
privileges. Even if you are confident that the party is not malicious,
if they make a mistake, an adversary can leverage their error to use
their superfluous privileges in harmful ways.
G. Least common mechanism –
For different users or processes, use separate data structures or
mechanisms to handle them. For example, each process gets its own
page table in a virtual memory system, ensuring that one process
cannot access another’s pages.
 Fig. 9 awareness training, and continuous monitoring for
suspicious activities.
IX. FUNDAMENTAL SECURITY REQUIREMENTS OF These four security requirements are fundamental to building a
OPERATING SYSTEMS secure OS. Addressing them comprehensively requires a multi-
faceted approach that combines strong security mechanisms,
Operating system (OS) security is built upon a foundation of core
proactive threat mitigation strategies, and continuous monitoring
requirements that ensure the confidentiality, integrity, availability,
and adaptation to the evolving threat landscape.
and authenticity of system resources. These requirements are not
merely isolated objectives but rather interconnected aspects of a X. ENHANCING OPERATING SYSTEM SECURITY
comprehensive security strategy [13].
A Multi-Layered Approach Protecting an operating system (OS)
A. Confidentiality: from security threats requires a multi-faceted strategy that
 Beyond Secrecy: Confidentiality goes beyond simply encompasses both proactive and reactive measures. Here's a
preventing unauthorized access to information. It breakdown of essential methods [6]:
encompasses controlling all forms of disclosure,
including reading, printing, displaying, and even
revealing the existence of data or resources.
 Enforcement Mechanisms: Achieving confidentiality
requires a multi-layered approach. This includes access
control mechanisms like access control lists (ACLs) and
mandatory access control (MAC), encryption techniques
to protect data in transit and at rest, and secure storage
solutions to prevent unauthorized physical access.
 Challenges: Maintaining confidentiality in modern
computing environments is challenging due to factors
like insider threats, sophisticated malware, and the
increasing use of cloud storage and mobile devices.
B. Integrity:
 Preserving Data Fidelity: Integrity ensures that only
authorized parties can modify system assets,
encompassing actions like writing, changing, deleting,
and creating data or resources. This ensures data
accuracy and reliability.
 Protection Mechanisms: Integrity is enforced through
various mechanisms, including access controls, data
validation techniques, version control systems, and digital
signatures to detect unauthorized modifications.
 Challenges: Ensuring data integrity in the face of
malware attacks, software bugs, and accidental data
corruption requires robust error detection and correction
mechanisms, secure backup and recovery procedures, Fig. 10
and comprehensive data validation routines.
A. Data Backup and Recovery:
C. Availability:
 Regularly backing up critical data ensures recoverability
 Guaranteed Access: Availability guarantees that in case of data corruption, ransomware attacks, or system
authorized users have timely and reliable access to failures.
system resources. This includes ensuring the system's  Employing diverse backup strategies (e.g., local backups,
operational continuity and resilience against disruptions. cloud backups, offsite backups) enhances resilience
 Supporting Mechanisms: Achieving high availability against data loss.
involves measures like redundancy (e.g., RAID for  Implementing versioning allows restoration to previous
storage, failover clusters for servers), disaster recovery states, mitigating the impact of malware infections or
planning, load balancing, and robust network accidental modifications.
infrastructure.
 Challenges: Maintaining availability is increasingly B. Secure User Practices:
challenging due to threats like denial of-service attacks,  Exercise caution when interacting with emails and web
hardware failures, natural disasters, and the growing links, avoiding suspicious or unknown sources. • Employ
complexity of distributed systems. strong, unique passwords and utilize multi-factor
authentication wherever possible.
D. Authenticity:
 Regularly update software to patch vulnerabilities and
 Reliable Verification: Authenticity focuses on verifying minimize exposure to exploits.
the identity of users and the origin of information. This is
crucial for establishing trust and accountability within the C. Robust Access Control:
system.  Implement strong authentication and authorization
 Implementation Techniques: Authenticity is commonly mechanisms to restrict access to system resources.
achieved through authentication mechanisms like  Utilize access control lists (ACLs), role-based access
passwords, biometrics, multi-factor authentication, and control (RBAC), or mandatory access control (MAC) to
digital certificates. enforce granular access permissions.
 Challenges: Ensuring authenticity in the face of  Regularly review and update user privileges to ensure
sophisticated phishing attacks, identity theft, and social least privilege principal adherence.
engineering requires robust authentication protocols, user
 D. Network Security:.  Mechanism: Employ signature-based detection, heuristic
 Connect only to trusted and secure Wi-Fi networks, analysis, and behavioral monitoring to identify and
avoiding public or unsecured networks that can expose quarantine malicious code.
data to interception.  Considerations: Require regular updates to stay abreast
of emerging threats. Advanced malware may employ
 Utilize Virtual Private Networks (VPNs) to encrypt evasion techniques to bypass detection.
network traffic and enhance privacy, especially on public
Wi-Fi. B. Firewalls:
 Implement network segmentation to isolate sensitive data  Function: Act as gatekeepers, monitoring and controlling
and limit the impact of potential breaches. network traffic based on predefined rules.
E. Malware Protection:  Mechanism: Inspect incoming and outgoing network
packets, blocking or allowing traffic based on
 Implement network segmentation to isolate sensitive data source/destination IP addresses, ports, and protocols.
and limit the impact of potential breaches.  Considerations: Can be implemented in hardware or
 Keep anti-malware definitions updated to counter software. Effectiveness depends on well-defined rules
emerging threats. and regular updates to address new threats.
 Utilize sandboxing techniques to isolate potentially
harmful programs and limit their system impact.
F. Application Control:
 Carefully manage application permissions, granting only
necessary access to system resources. • Employ
application whitelisting to restrict execution to trusted
programs. Fig. 11
 Regularly monitor application activity and resource
usage to detect anomalies. C. Intrusion Detection/Prevention Systems (IDS/IPS):
G. Firewall Implementation:  Function: IDS passively monitors network traffic for
 Utilize firewalls to monitor and filter network traffic, suspicious patterns, alerting administrators to potential
blocking unauthorized access attempts. intrusions. IPS actively blocks malicious traffic.
 Configure firewall rules to allow only legitimate  Mechanism: Employ anomaly-based detection
connections and services. (deviations from normal behavior) and signature-based
 Employ intrusion detection and prevention systems detection (matching known attack patterns).
(IDPS) to identify and respond to malicious network  Considerations: IDS generates alerts requiring human
activity. analysis, while IPS takes automated action. Fine-tuning
is crucial to minimize false positives/negatives.
H. Data Encryption:
D. File Integrity Checkers:
 Encrypt sensitive data both in transit and at rest to protect
against unauthorized access.  Function: Monitor critical system files for unauthorized
modifications, ensuring their integrity.
 Utilize strong encryption algorithms and secure key  Mechanism: Create baseline hashes of files and
management practices. periodically compare them to detect any changes.
 Employ full-disk encryption to protect the entire system  Considerations: Effective for detecting tampering with
drive from unauthorized access. system files but may not detect sophisticated attacks that
I. Secure Information Handling: replace files with malicious versions having the same
hash.
 Exercise caution when sharing personal information
online, limiting exposure to potential phishing attacks or E. Encryption Tools:
social engineering.  Function: Encode sensitive data, rendering it unreadable
 Be aware of data privacy policies and practices of without the decryption key.
websites and online services.  Mechanism: Utilize various encryption algorithms (e.g.,
 Utilize privacy-enhancing tools and techniques to AES, RSA) to protect data at rest and in transit.
minimize online tracking and data collection.  Considerations: Key management is crucial for effective
By implementing these comprehensive security measures, users encryption. Loss of the decryption key can result in
and organizations can significantly enhance the protection of their permanent data loss.
operating systems and mitigate the risks associated with cyber
threats. F. Patch Management Software:
 Function: Automate the process of applying security
XI. ESSENTIAL TOOLS FOR OPERATING SYSTEM updates and patches to OS and applications.
SECURITY  Mechanism: Scan for missing patches, download
Maintaining a robust security posture for operating systems requires updates, and install them with minimal user intervention.
a multi-layered approach utilizing a variety of specialized tools.  Considerations: Regular patching is essential to mitigate
These tools address different facets of OS security, working in vulnerabilities. However, patch testing is crucial to avoid
concert to protect against diverse threats [5]. compatibility issues.

A. Antivirus/Anti-malware Software: G. Vulnerability Scanners:

 Function: These tools form the first line of defense,  Function: Proactively identify weaknesses and
proactively scanning for and eliminating malicious misconfigurations in the OS and software.
software like viruses, worms, Trojans, ransomware, and  Mechanism: Scan systems for known vulnerabilities,
spyware. outdated software, and insecure settings.
  Considerations: Provide valuable insights into potential access identification. So when an authorization process is not
security gaps but require regular updates to stay current secured correctly, it is effortless for a masquerade attack to happen.
with newly discovered vulnerabilities.
C. SQL Injection:
H. Log Management and Analysis Tools: SQL, otherwise known as Structured Query Language, is a
 Function: Collect, store, and analyze log data from programming language. It manages data from the database
various sources to detect and investigate security management system. In SQL injections, the hackers insert malicious
incidents. codes into the servers that use SQL, thereby stealing their sensitive
 Mechanism: Centralize log data, correlate events, and information. They are usually prevented by regularly monitoring the
identify suspicious patterns. user using the application with the help of white-listing and black-
listing. You can also use SQL injections for firewall protection.
 Considerations: Effective log analysis requires expertise
and knowledge of attack techniques. D. Zero-Day Attacks:
I. Authentication Tools: When there is an attack, the IT team announces the network
 Function: Verify user identities and control access to the vulnerability. The hackers try to exploit the vulnerability before the
system. security experts fix it. As a result, constant monitoring is required
 Mechanism: Employ various authentication methods (e.g., to protect against any cyberattack. So, before the hackers can find
passwords, multi-factor authentication, biometrics) to the vulnerabilities, infrastructure penetration testing can help
ensure only authorized users gain access. identify the vulnerabilities. Securing those vulnerabilities could
thereby reduce the risk of attacks.
 Considerations: Strong authentication is crucial to
prevent unauthorized access. Implementing robust E. Man-in-the-middle Attack:
password policies and multi-factor authentication A MITM (man-in-the-middle) attack happens when a hacker
enhances security. intercepts and reads all the messages between two users who are
The effectiveness of these tools hinges on proper configuration, interacting. They believe the data travels only between those two
regular maintenance, and integration into a comprehensive security users, but the hackers steal their information. It is also called an
strategy. Organizations must proactively assess their security needs “eavesdropping attack.” Once they are in the conversation, they
and deploy a combination of these tools to create a multi-layered can filter and manipulate sensitive data. So, data encryption is the
defense against evolving threats. only way to protect the organization from such attacks. Also,
regular auditing and monitoring can help prevent attacks.
XII. VARIOUS ATTACKS ON OPERATING SYSTEM
SECURITY F. Misconfiguration Attacks:
Operating systems are prone to cyberattacks because of their They are common among operating systems, mainly in their sector.
complex features, which include many programs. These attacks For example, the administrator must change the default
exploit them and get access to the target network. Buffer overflows, configurations when installing a new system. Still, if they are in the
bugs, and an unmatched operating design are some system default setting, any user can access them, which can lead to an
vulnerabilities that are easily affected. Cyber-attacks are also carried attack. As a result, it is critical to ensure that the systems are
out by cracking passwords and encryptions, breaching file system regularly updated. Usually, making the organization name a
security, and exploiting system authentication [8][9]. username and password makes it easy for hackers to attack, so
avoid such practices.
G. Shrink Wrap Code:
It is another way to access a system. In this attack, systems with
weak software are targeted. If it has a bug in the original software
version, it is more vulnerable to shrink-wrap code attacks.
Unpatched operating systems are always at risk and are highly
prone to cyberattacks. When the hacker attacks the operating
system software, it also affects the insecure test pages and
debugging scripts. So the developers need to be very careful when
removing all the vulnerabilities.
H. Password Attack:
It is yet another major threat that takes advantage of the user’s
authentication. The most common password attack methods
include DNS cache poisoning, URL hijacking, tab catching, UI
Fig. 12 redressing, clone phishing, brute force attacks, password spraying,
dictionary attacks, credential stuffing, and keylogger attacks.
A. Buffer Overflow: Usually, hackers use the software to crack or guess passwords.
It is one such major operating system attack. When an application Repeating passwords makes it easy for hackers to compromise the
lacks well-defined boundaries or restrictions about the data capacity system.
it can handle or the type of data needed, buffer overflow problems
I. Distributed Denial Of Service Attack:
like denial of service, rebooting, freezing, and unrestricted access
occur. Buffers are memory regions that temporarily store the data A distributed denial of service (DDoS) attack denies access to our
when it gets transferred from one location to another. They affect all website by overloading it, making it difficult to load, or completely
types of software. They usually result from failing to allocate space blocking it. The attackers build a botnet, a network of hacked
for the buffer. devices, to attack by sending remote instructions. DDoS attacks
can last for hours, days, or even weeks, depending on the intensity
B. Masquerade: of the attacks. It’s challenging to find a DDoS attack because the
A masquerade attack can occur when one entity pretends to be symptoms are very common, such as slow loading and network
another entity. It is one of the forms of active attack. A masquerade issues. A large-volume DDoS attack can generate traffic of nearly
attack always uses a fake identity, such as a network identity, to get ten gigabytes per second.
access to the system. It gets unauthorized access through legitimate
 XIII. SUMMARY  Provides a unified framework for managing access to all
system resources.
The security of the operating system is vital for both its own and its
applications’ sakes. Security failures in this software allow  Ensures that objects are accessed only in authorized
essentially limitless bad consequences. While achieving system ways, preventing misuse or interference.
security is challenging, there are known design principles that can D. User Authentication:
help. These principles are useful not only in designing operating
systems, but in designing any large software system. Achieving  Verifies the identity of users attempting to access the
security in operating systems depends on the security goals one has. system.
These goals will typically include goals related to confidentiality,  Employs mechanisms like passwords, biometrics, and
integrity, and availability. In any given system, the more detailed multi-factor authentication to confirm user credentials.
particulars of these security goals vary, which implies that different  Establishes accountability and prevents unauthorized
systems will have different security policies intended to help them access by impersonation.
meet their specific security goals. As in other areas of operating
system design, we handle these varying needs by separating the
specific policies used by any particular system from the general
mechanisms used to implement the policies for all systems. The
next question to address is, what mechanisms should our operating
system provide to help us support general security policies? The
virtualization of processes and memory is one helpful mechanism,
since it allows us to control the behavior of processes to a large
extent. We will describe several other useful operating system
security mechanisms in the upcoming chapters [10].
Fig. 14

However, simply providing these services is not enough. A truly

trusted operating system rests on four key pillars:
a) Policy: A well-defined security policy outlines the rules
and regulations governing access to system resources and
information. It establishes the security goals and the
permissible actions within the system.
b) Model: A security model provides a formal
representation of the security policy, enabling rigorous
analysis and verification of its properties. Common
models include Bell-LaPadula, Biba, and Clark-Wilson.
c) Design: The OS design should incorporate security
considerations from the outset, integrating security
mechanisms seamlessly into its architecture. This
includes secure coding practices, minimized attack
Fig. 13 surfaces, and defense-in-depth strategies.
d) Trust: Trustworthiness is built upon a foundation of
XIV. CONCLUSION assurance, demonstrating that the OS has been rigorously
evaluated and certified to meet its security requirements.
Foundations Of a Trusted Operating System. Ensuring the This involves formal verification, penetration testing, and
security of an operating system (OS) hinges on the effective code audits.
provision of fundamental security services. These services form the By adhering to these principles, operating system developers
bedrock of a trusted OS, one that instills confidence in its users by can create secure and trustworthy platforms that protect user data
reliably safeguarding their information and resources. Key and maintain system integrity in the face of evolving threats.
primitive security services provided by operating systems include:
REFERENCES
A. Memory Protection:
[1] Abraham Silberschatz, Greg Gange, and Peter B. Galvin:
 Isolates processes from each other, preventing Operating System Concept (1983)
unauthorized access or modification of another process’s [2] Andrew S. Tanenbaum and Herbert Bos: Modern Operating
memory space. Systems (1992)
 Employs mechanisms like virtual memory, address space [3] Computer Security and Principles and Practice (2008): William
layout randomization (ASLR), and memory management Stallings and Lawrie Brown
units (MMUs) to enforce memory boundaries. [4] Internals and Design Principles (1985): William Stallings
 Crucial for preventing malicious code from corrupting or [5] Guide to Operating Systems Security (2003): Michael Palmer
hijacking other processes.
[6] Security Engineering, A Guide to Building Dependable
B. File Protection: Distributed Systems (2001): Ross Anderson
[7] Josef Pieprzyk, Thomas Hardjono, and Jennifer Seberry:
 Controls access to files and directories, ensuring that Fundamentals of Computer Security (2002)
only authorized users or processes can read, write, or [8] Khraisat, A., Gondal, I., Vamplew, P., & Kamruzzaman, J.
execute them. (2019). Survey of intrusion detection systems: techniques,
 Utilizes access control lists (ACLs), permissions, and datasets and challenges. Cybersecurity, 2(20).
ownership attributes to manage file access [11]. [9] Scarfone, K., & Mell, P. (2007). Guide to Intrusion Detection
 Protects sensitive data from unauthorized disclosure, and Prevention Systems (IDPS). National Institute of
modification, or deletion. Standards and Technology (NIST).
[10] Walsh, T., & Healy, M. (2012)
C. General Object Access Protection: [11] Data Security on Mobile Devices: Current State of the Art,
 Extends access control principles to other system objects, Open Problems, and Proposed Solutions Maximilian Zinkus,
such as semaphores, shared memory segments, and I/O Tushar M. Jois, Matthew Green
devices.
[12] Data exfiltration: A review of external attack vectors and M Mubeen, M Arslan, G Anandhi
countermeasures [14] Fundamentals of Computer Security
F Ullah, M Edwards, R Ramdhany
[13] Strategies to Avoid Illegal Data Access