Computer Security:

Principles and Practice

Fourth Edition

By: William Stallings and Lawrie Brown

Chapter 12
Operating System Security
 Strategies
• The 2010 Australian Signals Directorate (ASD) lists the
“Top 35 Mitigation Strategies”
• Over 85% of the targeted cyber intrusions investigated
by ASD in 2009 could have been prevented
• The top four strategies for prevention are:
• White-list approved applications
• Patch third-party applications and operating system vulnerabilities
• Restrict administrative privileges
• Create a defense-in-depth system

• These strategies largely align with those in the “20

Critical Controls” developed by DHS, NSA, the
Department of Energy, SANS, and others in the United
States
 Operating System
Security
• Possible for a system to be compromised during the
installation process before it can install the latest patches
• Building and deploying a system should be a planned
process designed to counter this threat
• Process must:
• Assess risks and plan the system deployment
• Secure the underlying operating system and then the key applications
• Ensure any critical content is secured
• Ensure appropriate network protection mechanisms are used
• Ensure appropriate processes are used to maintain security
System Security Planning
The first step in
Plan needs to deploying a new system
identify is planning
appropriate
personnel and Planning should
training to install include a wide
and manage the security
system assessment of the
organization

Planning process needs

to determine security Aim is to
requirements for the maximize security
system, applications, while minimizing
data, and users costs
 System Security Planning
Process
The purpose of the
Who will administer the
system, the type of Any additional security
system, and how they
information stored, the measures required on the
will manage the system
applications and services system, including the use
(via local or remote
provided, and their of host firewalls, anti-
access)
security requirements virus or other malware
protection mechanisms,
and logging

The categories of users of What access the system

the system, the privileges has to information stored
they have, and the types on other hosts, such as
of information they can file or database servers,
access and how this is managed

How access to the

How the users are
information stored on the
authenticated
system is managed
 Operating Systems
Hardening
• First critical step in securing a system is to secure the
base operating system
• Basic steps
• Install and patch the operating system
• Harden and configure the operating system to adequately
address the indentified security needs of the system by:
• Removing unnecessary services, applications, and protocols
• Configuring users, groups, and permissions
• Configuring resource controls
• Install and configure additional security controls, such as anti-
virus, host-based firewalls, and intrusion detection system (IDS)
• Test the security of the basic operating system to ensure that the
steps taken adequately address its security needs
 Initial Setup and Patching
Overall The integrity
boot and source of
process any additional
must also device driver
code must be
be secured
carefully
validated

System
security begins Initial
Should stage
installation
with the should install and validate all
installation of the minimum Critical that the patches on the
system be kept
the operating necessary for
up to date, with
test systems
system the desired before
all critical
system deploying them
security related
patches installed in production

Full installation
Ideally new and hardening
systems process should
should be occur before the
constructed system is
on a protected deployed to its
network intended location
 Remove
Unnecessary
Services, • When performing the
Applications, initial installation the
Protocols supplied defaults should
not be used
• Default configuration is set
to maximize ease of use and
• If fewer software packages functionality rather than
security
are available to run the risk
is reduced • If additional packages are
• System planning process needed later they can be
installed when they are
should identify what is required
actually required for a
given system
 • System planning process
should consider:
Configure
• Categories of users on the system
Users, Groups,
and • Privileges they have

Authentication • Types of information they can

access
• How and where they are defined
and authenticated

• Default accounts included as

• Not all users with access to part of the system installation
a system will have the should be secured
same access to all data and • Those that are not required
resources on that system should be either removed or
• Elevated privileges should disabled
be restricted to only those • Policies that apply to
users that require them, authentication credentials
and then only when they configured
are needed to perform a
task
 Install
Configure
Additional
Resource
Security
Controls
Controls

• Once the users and groups are

defined, appropriate • Further security possible by
permissions can be set on data installing and configuring
and resources additional security tools:
• Anti-virus software
• Many of the security • Host-based firewalls
hardening guides provide lists • IDS or IPS software
of recommended changes to • Application white-listing
the default access
configuration
 • Checklists are included in
security hardening guides
Test the • There are programs
System specifically designed to:
Security • Review a system to
ensure that a system
meets the basic security
requirements
• Scan for known
• Final step in the process of vulnerabilities and poor
initially securing the base configuration practices
operating system is
security testing • Should be done following
• Goal: the initial hardening of the
system
• Ensure the previous security
configuration steps are correctly
implemented • Repeated periodically as
• Identify any possible
part of the security
vulnerabilities maintenance process
 Application
Configuration
• May include:
• Creating and specifying appropriate data storage areas for application
• Making appropriate changes to the application or service default
configuration details
• Some applications or services may include:
• Default data
• Scripts
• User accounts
• Of particular concern with remotely accessed services
such as Web and file transfer services
• Risk from this form of attack is reduced by ensuring that most of the
files can only be read, but not written, by the server
 Encryption Technology
Is a key
enabling
technology
that may be If secure network Cryptographic
services are provided file systems are
used to secure another use of
using TLS or IPsec If secure network
data both in Must be encryption
services are
transit and configured and suitable public and provided using
when stored appropriate private keys must be SSH, appropriate
cryptographic generated for each of server and client
keys created, them keys must be
signed, and created
secured
 Security Maintenance
• Process of maintaining security is continuous
• Security maintenance includes:
• Monitoring and analyzing logging information
• Performing regular backups
• Recovering from security compromises
• Regularly testing system security
• Using appropriate software maintenance processes to patch and
update all critical software, and to monitor and revise
configuration as needed
 Logging
In the event of a system
Key is to ensure you
Can only inform you about breach or failure, system
capture the correct data and
bad things that have administrators can more
then appropriately monitor
already happened quickly identify what
and analyze this data
happened

Generates significant
Range of data acquired
Information can be volumes of information
should be determined
generated by the system, and it is important that
during the system planning
network and applications sufficient space is allocated
stage
for them

Automated analysis is
preferred
 Data Backup and Archive
Performing regular Needs and policy
backups of data is Backup Archive relating to
a critical control backup and
that assists with archive should be
maintaining the The process of The process of determined
integrity of the making copies of retaining copies of
during the system
data over extended
system and user data at regular
periods of time in planning stage
intervals
data order to meet legal
and operational
requirements to
May be legal or access past data
operational Kept online or
requirements for offline
the retention of
data

Stored locally or
transported to a
remote site
• Trade-offs include
ease of
implementation and
cost versus greater
security and
robustness against
different threats
 Linux/Unix Security
• Patch management
• Keeping security patches up to date is a widely recognized and critical
control for maintaining security

• Application and service configuration

• Most commonly implemented using separate text files for each
application and service
• Generally located either in the /etc directory or in the installation tree
for a specific application
• Individual user configurations that can override the system defaults are
located in hidden “dot” files in each user’s home directory
• Most important changes needed to improve system security are to
disable services and applications that are not required
 Linux/Unix Security
• Users, groups, and permissions
• Access is specified as granting read, write, and
execute permissions to each of owner, group, and
others for each resource
• Guides recommend changing the access permissions
for critical directories and files
• Local exploit
• Software vulnerability that can be exploited by an attacker to
gain elevated privileges
• Remote exploit
• Software vulnerability in a network server that could be
triggered by a remote attacker
 Linux/Unix Security

Remote access controls Logging and log rotation

•Several host firewall programs may •Should not assume that the default
be used setting is necessarily appropriate
•Most systems provide an
administrative utility to select which
services will be permitted to access
the system
 Linux/Unix Security
• chroot jail
• Restricts the server’s view of the file system to just a
specified portion
• Uses chroot system call to confine a process by mapping
the root of the filesystem to some other directory
• File directories outside the chroot jail aren’t visible or
reachable
• Main disadvantage is added complexity
 Windows Security

Users administration
Patch and access controls
management •Systems implement
discretionary access controls
• “Windows Update” and resources
“Windows Server
•Vista and later systems include
Update Service” assist mandatory integrity controls
with regular
•Objects are labeled as being of
maintenance and should
low, medium, high, or system
be used integrity level
• Third party applications •System ensures the subject’s
also provide automatic integrity is equal or higher than
update support the object’s level
•Implements a form of the Biba
Integrity model
 Windows Security
Users Administration and Access
Controls
Windows systems also define Combination of share and
privileges NTFS permissions may be
•System wide and granted to user used to provide additional
accounts security and granularity when
accessing files on a shared
resource

User Account Control (UAC) Low Privilege Service

•Provided in Vista and later systems Accounts
•Assists with ensuring users with •Used for long-lived service processes
administrative rights only use them such as file, print, and DNS services
when required, otherwise accesses
the system as a normal user
 Windows Security
Application and service
configuration

•Much of the configuration information is

centralized in the Registry
• Forms a database of keys and values that may be
queried and interpreted by applications
•Registry keys can be directly modified
using the “Registry Editor”
• More useful for making bulk changes
 Windows Security
Other security controls
• Essential that anti-virus, anti-spyware, personal firewall, and other malware
and attack detection and handling software packages are installed and
configured
• Current generation Windows systems include basic firewall and malware
countermeasure capabilities
• Important to ensure the set of products in use are compatible

Windows systems also support a range of cryptographic

functions:
• Encrypting files and directories using the Encrypting File System (EFS)
• Full-disk encryption with AES using BitLocker

“Microsoft Baseline Security Analyzer”

• Free, easy to use tool that checks for compliance with Microsoft’s security
recommendations
 Virtualization
• A technology that provides an abstraction of the
resources used by some software which runs in a
simulated environment called a virtual machine (VM)
• Benefits include better efficiency in the use of the
physical system resources
• Provides support for multiple distinct operating systems
and associated applications on one physical system
• Raises additional security concerns
 Hypervisor
• Software that sits between the hardware and the VMs
• Acts as a resource broker
• It allows multiple VMs to safely coexist on a single
physical server host and share that host’s resources
• Virtualizing software provides abstraction of all physical
resources and thus enables multiple computing stacks,
called virtual machines, to be run on a single physical
host
• Each VM includes an OS, called the guest OS
• This OS may be the same as the host OS, if present, or a
different one
 Hypervisor Functions
• Execution management of VMs
• Devices emulation and access control
• Execution of privileged operations by
hypervisor for guest VMs
• Management of VMs (also called VM
The principal lifecycle management)
functions performed • Administration of hypervisor platform and
by a hypervisor are: hypervisor software
 Virtualized Systems
• In virtualized systems, the available hardware resources must
be appropriately shared among the various guest OS’s
• These include CPU, memory, disk, network, and other
attached devices
• CPU and memory are generally partitioned between these,
and scheduled as required
• Disk storage may be partitioned, with each guest having
exclusive use of some disk resources
• Alternatively, a “virtual disk” may be created for each guest,
which appears to it as a physical disk with a full file-system,
but is viewed externally as a single ”disk image” file on the
underlying file-system
• Attached devices such as optical disks, or USB devices are
generally allocated to a single guest OS at a time
 Software Defined
Networks (SDNs)
SDNs enable network segments to logically span multiple servers
within and between data centers, while using the same underlying
physical network

There are several possible approaches to providing SDNs,

including the use of overlay networks

• These abstract all layer 2 and 3 addresses from the underlying physical network into
whatever logical network structure is required
• This structure can be easily changed and extended as needed
• The IETF standard DOVE (Distributed Overlay Virtual Network) which uses
VXLAN (Virtual Extended Local Area Network) can be used to implement such an
overlay network
• With this flexible structure, it is possible to locate virtual servers, virtual IDS, and
virtual firewalls anywhere within the network as required
 Containers
• A recent approach to virtualization is known as container
virtualization or application virtualization
• In this approach, software known as a virtualization container,
runs on top of the host OS kernel and provides an isolated
execution environment for applications
• Unlike hypervisor-based VMs, containers do not aim to
emulate physical servers
• All containerized applications on a host share a common OS
kernel
• For containers, only a small container engine is required as
support for the containers
• Containerization sits in between the OS and applications and
incurs lower overhead, but potentially introduces greater
security vulnerabilities
 Virtualization Security
Issues
• Security concerns include:
• Guest OS isolation
• Ensuring that programs executing within a guest OS may
only access and use the resources allocated to it
• Guest OS monitoring by the hypervisor
• Which has privileged access to the programs and data in
each guest OS
• Virtualized environment security
• Particularly image and snapshot management which
attackers may attempt to view or modify
Securing Virtualization
Systems
• Carefully plan the
Organizations security of the
virtualized system
using • Secure all elements of
a full virtualization
virtualization solution and maintain
their security
should:
• Ensure that the
hypervisor is properly
secured

• Restrict and protect

administrator access
to the virtualization
solution
 Hypervisor Security
• Should be
• Secured using a process similar to securing an operating system
• Installed in an isolated environment
• Configured so that it is updated automatically
• Monitored for any signs of compromise
• Accessed only by authorized administration

• May support both local and remote administration so must be

configured appropriately
• Remote administration access should be considered and secured in the
design of any network firewall and IDS capability in use
• Ideally administration traffic should use a separate network with very
limited access provided from outside the organization
 Virtualized
Infrastructure Access to VM
image and

Security snapshots must

be carefully
controlled

Access must
be limited to
just the
appropriate
guest OSs
Systems
manage access
to hardware
resources
 Virtual Firewall
Provides firewall capabilities for the network traffic flowing
between systems hosted in a virtualized or cloud
environment that does not require this traffic to be routed
out to a physically separate network supporting traditional
firewall services

VM Host-Based
VM Bastion Host Hypervisor Firewall
Firewall

Where a separate VM is used as a Where host-based firewall

bastion host supporting the same capabilities provided by the Guest Where firewall capabilities are
firewall systems and services that OS running on the VM are provided directly by the
could be configured to run on a configured to secure that host in hypervisor
physically separate bastion, including the same manner as used in
possibly IDS and IPS services physically separate systems
 Summary
• Introduction to operating • Linux/Unix security
system security • Patch management
• System security planning • Application and service configuration
• Operating systems •
•
Users, groups, and permissions
Remote access controls
hardening • Logging and log rotation
• Operating system installation: • Application security using a chroot jail
initial setup and patching
• Remove unnecessary services, • Security testing
applications and protocols • Windows security
• Configure users, groups, and • Patch management
authentications • Users administration and access
• Configure resource controls controls
• Install additional security controls • Application and service configuration
• Test the system security • Other security controls
• Application security • Security testing
• Application configuration • Virtualization security
• Encryption technology • Virtualization alternatives
• Security maintenance •
•
Virtualization security issues
• Logging Securing virtualization systems
• Data backup and archive