Operating System Security

Read Chapter 12

1
 OS Security Layers
 Computer system is viewed as layered system:
− physical hardware at the bottom
− OS including privileged kernel code APIs, and services; in the middle
− and finally user applications and utilities in the top layer
 BIOS and possibly other code that is external to, and largely not
visible from kernel but used when booting the system or to support
low-level hardware control

 Each layer is vulnerable to attack from below if the lower layers are not
secured appropriately

2
 OS Hardening Measures & Strategies
• The 2010 Australian Signals Directorate (ASD) lists the “Strategies to
Mitigate Cyber Security Incidents”.
• Over 85% of the targeted cyber intrusions investigated by ASD in 2009
could have been prevented if just the top four of these strategies are
implemented, which they are:
– White-list approved applications
– Patch third-party applications and operating system vulnerabilities
– Restrict administrative privileges
– Create a defense-in-depth system
• In 2017, the updated list “Essential Eight” added the following items:
• Configure Microsoft Office macro settings
• User application hardening
• Multifactor authentication
• Regular backups
• Collectively these assist in creating a defense-in-depth system
3
 Operating System Security
• It is possible for a system to be compromised during the installation
process before it can install the latest patches or implement other
hardening measures.
• Building and deploying a system should be a planned process
designed to counter this threat and to maintain security during its
operational lifetime
• Process must:
– assess risks and plan the system deployment
– secure the underlying operating system and then the key applications
– ensure any critical content is secured
– ensure appropriate network protection mechanisms are used
– ensure appropriate processes are used to maintain security

4
 System Security Planning
• The first step in deploying a
new system is planning

• Plan needs to identify • Planning should

appropriate personnel include a wide
and training to install security assessment
and manage the system of the organization

• Planning process needs • Aim is to

to determine security maximize
requirements for the security while
system, applications,
minimizing
data, and users
costs 5
 System Security Planning Process
• List of items that should be considered during the system security
planning process
– The purpose of the system, the type of information stored, the applications and
services provided, and their security requirements.
– The categories of users of the system, the privileges they have, and the types of
information they can access.
– How the users are authenticated.
– How access to the information stored on the system is managed.
– What access the system has to information stored on other hosts, such as file or
database servers, and how this is managed.
– Who will administer the system, and how they will manage the system (via local
or remote access).
– Any additional security measures required on the system, including the use of
host firewalls, anti-virus or other malware protection mechanisms, and logging.

6
 Operating Systems Hardening
• First critical step in securing a system is to secure the base OS
• Basic steps that should be used to secure an OS
– Install and patch the OS

– Harden and configure the OS to adequately address the identified security needs
of the system by:

• Removing unnecessary services, applications, and protocols

• Configuring users, groups, and permissions

• Configuring resource controls

– Install and configure additional security controls, such as anti-virus, host-based

firewalls, and intrusion detection system (IDS)

– Test the security of the basic OS to ensure that the steps taken adequately
address its security needs
7
Initial Setup Overall The integrity
and source of
boot
and Patching process
must also
any additional
device driver
be secured code must be
carefully
validated
System
security begins Initial
with the installation
installation of should install Critical that the
OS the minimum system be kept up
necessary for to date, with all
the desired critical security
system related patches
installed

Full installation
Ideally new and hardening
process should Should stage and
systems
occur before the validate all
should be
system is patches on the test
constructed
deployed to its systems before
on a protected
intended location deploying them in
network
production 8
 Remove Unnecessary Services
• if fewer software packages are available to run the risk is reduced
• system planning process should identify what is actually required
for a given system
• when performing the initial installation the supplied defaults
should not be used
– default configuration is set to maximize ease of use and functionality rather
than security

– if additional packages are needed later they can be installed when they are
required

9
 Configure Users and Privileges
• Not all users with access to a system will have the same access to all
data and resources on that system
• Elevated privileges should be restricted to only those users that
require them, and then only when they are needed to perform a task
• System planning process should consider:
– categories of users on the system
– privileges they have
– types of information they can access

• Default accounts included as part of the system installation should be

secured
– those that are not required should be either removed or disabled
– policies that apply to authentication credentials should be configured
• Authentication methods and details of the required length, complexity, and age
allowed for passwords 10
 Configure Resource Controls and Install Additional Security
Controls
• Once the users and groups are defined, appropriate permissions can be
set on data and resources
• Many of the security hardening guides provide lists of recommended
changes to the default access configuration
• Install Additional Security Controls: Further security possible by
installing and configuring additional security tools:
– Anti-virus software: should be considered for any system as part of its security
profile
– Host-based firewalls, IDS or IPS software: improve security limiting remote
network access to services on the system
• Firewalls are configured to limit access by port or protocol
• IDS and IPS software may include mechanisms such as traffic monitoring, or file
integrity checking
– Application white-listing: limits programs that can execute on the system to
just those in an explicit list
• can prevent an attacker installing and running their own malware 11
 Test the System Security
• Final step in the process of initially securing the base OS is security
testing
• Goal is to:
– Ensure the previous security configuration steps are correctly implemented
– Identify any possible vulnerabilities

• Suitable checklists are included in security hardening guides

• There are programs specifically designed to:
– Review a system to ensure that a system meets the basic security requirements
– Scan for known vulnerabilities and poor configuration practices

• This should be done following the initial hardening of the system

• Then repeated periodically as part of the security maintenance process

12
 Application Security- Application Configuration
• Once OS is installed and appropriately secured, services and
applications must next be installed and configured
• Application configuration may include:
– Creating and specifying appropriate data storage areas for application
– Making right changes to application or service default configuration details
• Some applications/services include default data, scripts, user accounts
– These should be reviewed, and only retained if required, and suitably secured.
• Of particular concern with remotely accessed services such as Web
and file transfer services
– should not be granted right to modify files, unless it is specifically required
– common configuration fault: all files supplied by the service owned by same
“user” account that server executes
• Attacker exploit some vulnerability in either server software or script executed (to
modify any of these files)
– Risk from this form of attack is reduced by ensuring that most of the files can
13
only be read, but not written, by the server
 Application Security- Encryption Technology
• Encryption is a key enabling technology that may be used to secure
data both in transit and when stored
• Required Encryption technology must be configured and appropriate
cryptographic keys created, signed, and secured
• If secure network services are provided using TLS or IPsec suitable
public and private keys must be generated for each of them
• If secure remote access is provided using SSH, appropriate server and
client keys must be created
• Cryptographic file systems are another use of encryption

14
 Security Maintenance
• Once the system is appropriately built, secured, and deployed, the
process of maintaining security is continuous
• Security maintenance includes:
– Monitoring and analyzing logging information

– Performing regular backups

– Recovering from security compromises

– Regularly testing system security

– Using appropriate software maintenance processes to patch and update all

critical software, and to monitor and revise configuration as needed

15
 Security Maintenance- Logging
• Logging reactive control that can only inform you about bad things
that have already happened
• Effective logging ensures that in event of a system breach or failure,
system administrators can more quickly identify what happened
• Key is to ensure you capture the correct data and then appropriately
monitor and analyze this data
• Information can be generated by the system, network and applications
• Range of data acquired should be determined during the system
planning stage
• Generates significant volumes of information and it is important that
sufficient space is allocated for them
• Automated analysis is preferred
16
 Security Maintenance- Data Backup and Archive
• Performing regular backups of data is a critical control that assists
with maintaining integrity of system and user data
– Also, there May be legal or operational requirements for the retention of data

• Backup: process of making copies of data at regular intervals

• Archive: process of retaining copies of data over extended periods of
time in order to meet legal and operational requirements to access past
data
• Needs and policy relating to backup and archive should be determined
during the system planning stage
– Kept online or offline
– Stored locally or transported to a remote site
• Trade-offs include ease of implementation and cost versus greater security and
robustness against different threats

17
 Linux/Unix Security: Patch/Configs
• Large range of resources are available to assist administrators
• Patch management: keeping security patches up to date is a widely
recognized and critical control for maintaining security
– Red Hat, Fedora, and CentOS include up2date or yum
– SuSE includes yast
– Debian uses apt-get

• Application and service configuration on Unix and Linux

– most commonly implemented using separate text files for each application and
service
– System-wide configuration details are generally located either in the /etc
directory or in the installation tree for a specific application
– individual user configurations that can override the system defaults are located
in hidden “dot” files in each user’s home directory
– most important changes needed to improve system security are to disable
services and applications that are not required 18
 Linux/Unix Security: Users, groups, permissions
• Users, groups, and permissions: Discretionary access control (DAC) is
used; access specified as granting read, write, and execute permissions to each of
owner, group, and others for each resource.
• Set using the chmod command, extended access rights set using getfacl and
setfacl
– Information on user accounts and group stored in /etc/passwd and
/etc/group
– Guides for hardening Unix/Linux systems recommend changing the access
permissions for critical directories and files further limit access to them
– Attackers target programs: setuid (set user to root), setgid (set to privileged
group)
– local exploit: software vulnerability that can be exploited by an attacker to gain
elevated privileges
– remote exploit : software vulnerability in a network server that could be triggered by a
remote attacker

19
 Linux/Unix Security
Remote access controls, Logging and log rotation, chroot jail
• Remote access controls
– Several host firewall programs (iptables to configure the netfilter
kernel module) or TCP Wrappers (tcpd daemon) may be used
– Most systems provide an administrative utility to select which services will be permitted
to access the system
• Logging and log rotation
– log levels of detail: ranging from debugging (maximum detail) to none
– Should not assume that the default setting is necessarily appropriate
– Applications allow to specify either a dedicated file or a syslog facility to use
when writing log data to /dev/log
• chroot jail: restricts the server’s view of file system to just a specified portion
– Many services (such as FTP) do not require access to full file-system
– uses chroot system call to confine a process by mapping the root of the
filesystem (“/”) to some other directory (e.g., /srv/ftp/public)
– file directories outside the chroot jail aren’t visible or reachable
– main disadvantage is added complexity 20
 Windows Security
Patch management, Users administration and access controls
• Patch management: “Windows Update” and “Windows Server Update
Service” assist with regular maintenance and should be used
– Third party applications also provide automatic update support
• Users administration and access controls
– Users and groups are defined with a Security ID (SID).
– Systems implement discretionary access controls (DAC) resources
– ACL has many entries that may grant or deny access rights to a specific SID
– Vista and later systems include mandatory integrity controls
– Objects are labeled as being of low, medium, high, or system integrity level
– System ensures the subject’s integrity is equal or higher than the object’s level
– Implements a form of the Biba Integrity model
– User Account Control (UAC)
• Assists with ensuring users with administrative rights only use them when required,
otherwise accesses the system as a normal user
– Low Privilege Service Accounts
• Used for long-lived service processes such as file, print, and DNS services 21
 Windows Security: Application and Service
Configuration, Other security controls, Security Testing
• Application and Service Configuration
– Much of configuration information is centralized in Registry, which forms
database of keys and values that may be queried and interpreted by applications
– Registry keys can be directly modified using the “Registry Editor”: useful for
making bulk changes; changes may also be recorded in a central repository
• Other security controls
– Essential that anti-virus, anti-spyware, personal firewall, and other malware and
attack detection and handling software packages are installed and configured
– Windows include basic firewall and malware countermeasure capabilities
– Important to ensure the set of products in use are compatible
– Windows systems also support a range of cryptographic functions:
• Encrypting files and directories using the Encrypting File System (EFS)
• Full-disk encryption with AES using BitLocker
• Security Testing: “Microsoft Baseline Security Analyzer”
• Free, easy to use tool that checks for compliance with Microsoft’s security
recommendation 22
 Virtualization
• Refers to a technology that provides an abstraction of the resources
used by some software which runs in a simulated environment called a
virtual machine (VM)
– provides support for multiple distinct operating systems and associated
applications on one physical system

• We are interested in full virtualization:

– allows multiple full operating system instances to execute on virtual hardware,
supported by a hypervisor that manages access to the actual physical hardware
resources.

• Benefits include better efficiency in the use of the physical system

resources than is typically seen using a single operating system
instance
– In virtualized systems, available hardware resources must be appropriately
shared between the various guest OSs. These include CPU, memory, disk,
network, and other attached devices

• Raises additional security concerns 23

 Hypervisor
• Software that sits between the hardware and the VMs
• Acts as a resource broker
• It allows multiple VMs to safely coexist on a single physical server host and
share that host’s resources
• Virtualizing software provides abstraction of all physical resources and
thus enables multiple computing stacks, called virtual machines, to be
run on a single physical host
• Each VM includes an OS, called the guest OS
– This OS may be the same as the host OS, if present, or a different one
• The principal functions performed by a hypervisor are:
– Execution management of VMs
– Devices emulation and access control
– Execution of privileged operations by hypervisor for guest VMs
– Management of VMs (also called VM lifecycle management)
– Administration of hypervisor platform and hypervisor software 24
 Full Virtualization Variations
• Full virtualization systems divided into:
– Native virtualization
– hosted virtualization
• Native virtualization systems
– hypervisor executes directly on the underlying hardware
– seen in servers, with the goal of improving the execution efficiency of the
hardware
– More secure: as they have fewer additional layers

25
 Full Virtualization Variations
• Hosted virtualization systems
– hypervisor executes as just another application on a host OS that is running on
the underlying hardware,
– Seen in clients, where they run along side other applications on the host OS
– Used to support applications for alternate operating system versions or types
Hosted OS run along other apps

• Adds additional layers: increased security concerns

26
Container virtualization or application virtualization
• In this recent approach, software known as a virtualization
container, runs on top of the host OS kernel and provides an isolated
execution environment for applications
• Unlike hypervisor-based VMs, containers do not aim to emulate
physical servers
• All containerized applications on a host share a common OS kernel
• For containers, only a small container engine is required as support
for the containers
• Containerization sits in between the OS
and applications and incurs lower
overhead, but potentially introduces
greater security vulnerabilities

27
 Virtualization Systems
• In virtualized systems, the available hardware resources must be
appropriately shared among the various guest OS’s

– These include CPU, memory, disk, network, and other attached devices

– CPU and memory are generally partitioned between these, and scheduled as
required

– Disk storage may be partitioned, with each guest having exclusive use of some
disk resources

– Alternatively, a “virtual disk” may be created for each guest, which appears to
it as a physical disk with a full file-system, but is viewed externally as a single
”disk image” file on the underlying file-system

• Attached devices such as optical disks, or USB devices are generally

allocated to a single guest OS at a time
28
 Virtualization Security Issues
• Security concerns that result from use of virtualized systems include
– Guest OS isolation: ensuring that programs executing within a guest OS may
only access and use the resources allocated to it
– Guest OS monitoring by the hypervisor: hypervisor has privileged access to
programs and data in each guest OS and must be trusted as secure from
subversion and compromised use of this access.
– Virtualized environment security: particularly image and snapshot
management which attackers may attempt to view or modify
• Above security concerns may be regarded as an extension of concerns
we have already discussed with securing OSes and applications
• Organizations using virtualization should:
– Carefully plan the security of the virtualized system
– Secure all elements of a full virtualization solution and maintain their security
– Ensure that the hypervisor is properly secured
– Restrict and protect administrator access to the virtualization solution
29
 Hypervisor Security
• Hypervisor should be
– secured using a process similar to securing an operating system
– installed in an isolated environment
– configured so that it is updated automatically
– monitored for any signs of compromise
– accessed only by authorized administration
• May support both local and remote administration so must be configured
appropriately
• Remote administration access should be considered and secured in the
design of any network firewall and IDS capability in use
• Ideally administration traffic should use a separate network with very
limited access provided from outside the organization

30
• Virtualized systems manage access to hardware resources such as
disk storage and network interfaces.
– This access must be limited to just the appropriate guest OSs that use any
resource

• Configuration of network interfaces and use of an internal virtual

network may present issues for organizations that wish to monitor all
network traffic between systems.
– This should be designed and handled as needed

• Access to VM images and snapshots must be carefully controlled,

since these are another potential point of attack

31
 32

• Provides firewall capabilities for the network traffic flowing between

systems hosted in a virtualized or cloud environment that does not
require this traffic to be routed out to a physically separate network
supporting traditional firewall services
– VM Bastion Host
▪ Where a separate VM is used as a bastion host supporting the same firewall
systems and services that could be configured to run on a physically
separate bastion, including possibly IDS and IPS services
– VM Host-Based Firewall
▪ Where host-based firewall capabilities provided by the guest OS running on
the VM are configured to secure that host in the same manner as used in
physically separate systems
– Hypervisor Firewall
▪ Where firewall capabilities are provided directly by the hypervisor
 Hosted Virtualization Security 33

• Hosted virtualized systems pose additional security concerns that

result from the host OS under the hypervisor and its guest Oss
• Users have access to configure the hypervisor and to any VM images
and snapshots
• Possible to design a host system and virtualization solution that is
more protected from access and modification by the users
• There will be security concerns from possible compromise unless it is
adequately secured and managed
 Summary

• System security planning • Linux/Unix security

• operating systems hardening – patch management
– initial setup and patching – application configuration
– remove unnecessary services – users, groups, permissions
– configure users and groups – remote access
– test system security – security testing
• Application security • Windows security
– application configuration – patch management
– encryption technology – users administration and access
– security maintenance controls
– data backup – application and service
– virtualization security configuration
• virtualization alternatives – security testing

34