System Security

Content
 The Security Problem
Program Threats
System and Network Threats
Cryptography as a Security Tool
User Authentication
Implementing Security Defenses
 The Security Problem
System secure if resources used and accessed as intended under all
circumstances
Unachievable
Intruders (crackers) attempt to breach security
Threat is potential security violation
Attack is attempt to breach security
Attack can be accidental or malicious
Easier to protect against accidental than malicious misuse
 Security Violation Categories
Breach of confidentiality
Unauthorized reading of data
Breach of integrity
Unauthorized modification of data
Breach of availability
Unauthorized destruction of data
Theft of service
Unauthorized use of resources
Denial of service (DOS)
Prevention of legitimate use
 Security Violation Methods
Masquerading (breach authentication)
Pretending to be an authorized user to escalate privileges
Replay attack
As is or with message modification
Man-in-the-middle attack
Intruder sits in data flow, masquerading as sender to receiver and vice
versa
Session hijacking
Intercept an already-established session to bypass authentication
Privilege escalation
Common attack type with access beyond what a user or resource is
supposed to have
Standard Security Attacks
 Security Measure Levels
Impossible to have absolute security, but make cost to perpetrator
sufficiently high to deter most intruders
Security must occur at four levels to be effective:
Physical
Data centers, servers, connected terminals
Application
Benign or malicious apps can cause security problems
Operating System
Protection mechanisms, debugging
Network
Intercepted communications, interruption, DOS
Security is as weak as the weakest link in the chain
Humans a risk too via phishing and social-engineering attacks
But can too much security be a problem?
 Program Threats
Many variations, many names
Trojan Horse
Code segment that misuses its environment
Exploits mechanisms for allowing programs written by users to be
executed by other users
Spyware, pop-up browser windows, covert channels
Up to 80% of spam delivered by spyware-infected systems
Trap Door
Specific user identifier or password that circumvents normal security
procedures
Could be included in a compiler
How to detect them?
 Program Threats Cont..
Many variations, many names
Trojan Horse
Code segment that misuses its environment
Exploits mechanisms for allowing programs written by users to be
executed by other users
Spyware, pop-up browser windows, covert channels
Up to 80% of spam delivered by spyware-infected systems
Trap Door
Specific user identifier or password that circumvents normal security
procedures
Could be included in a compiler
How to detect them?
 Program Threats Cont..
Malware - Software designed to exploit, disable, or damage computer
Trojan Horse – Program that acts in a clandestine manner
 Spyware – Program frequently installed with legitimate software to display
adds, capture user data
 Ransomware – locks up data via encryption, demanding payment to unlock
it
Others include trap doors, logic boms
All try to violate the Principle of Least Privilege

Goal frequently is to leave behind Remote Access Tool (RAT) for repeated
access
 Program Threats Cont..
Viruses
Code fragment embedded in legitimate program
Self-replicating, designed to infect other computers
Very specific to CPU architecture, operating system, applications
Usually borne via email or as a macro
Visual Basic Macro to reformat hard drive
Sub AutoOpen()
Dim oFS
Set oFS = CreateObject(’’Scripting.FileSystemObject’’)
vs = Shell(’’c:command.com /k format c:’’,vbHide)
End Sub
 Program Threats Cont..
Attacks still common, still occurring
Attacks moved over time from science experiments to tools of organized
crime
Targeting specific companies
Creating botnets to use as tool for spam and DDOS delivery
Keystroke logger to grab passwords, credit card numbers
Why is Windows the target for most attacks?
Most common
Everyone is an administrator
Licensing required?
Monoculture considered harmful
Four-layered Model of Security
A Boot-sector Computer Virus
A Boot-sector Computer Virus
 System and Network Threats
Some systems “open” rather than secure by default
Reduce attack surface
But harder to use, more knowledge needed to administer
Network threats harder to detect, prevent
Protection systems weaker
More difficult to have a shared secret on which to base access
No physical limits once system attached to internet
Or on network with system attached to internet
Even determining location of connecting system difficult
IP address is only knowledge
 System and Network Threats Cont..
Worms – use spawn mechanism; standalone program
Internet worm
Exploited UNIX networking features (remote access) and bugs in
finger and sendmail programs
Exploited trust-relationship mechanism used by rsh to access friendly
systems without use of password
Grappling hook program uploaded main worm program
99 lines of C code
Hooked system then uploaded main code, tried to attack connected
systems
Also tried to break into other users accounts on local system via
password guessing
If target system already infected, abort, except for every 7 th time
Worm Vs Virus Vs Trojan Horses
 System and Network Threats Cont..
Port scanning
Automated attempt to connect to a range of ports on one or a range of
IP addresses
Detection of answering service protocol
Detection of OS and version running on system
nmap scans all ports in a given IP range for a response
nessus has a database of protocols and bugs (and exploits) to apply
against a system
Frequently launched from zombie systems
To decrease trace-ability
 System and Network Threats Cont..
Denial of Service
Overload the targeted computer preventing it from doing any useful work
Distributed Denial-of-Service (DDoS) come from multiple sites at once
Consider the start of the IP-connection handshake (SYN)
How many started-connections can the OS handle?
Consider traffic to a web site
How can you tell the difference between being a target and being really
popular?
Accidental – CS students writing bad fork() code
Purposeful – extortion, punishment
Port scanning
Automated tool to look for network ports accepting connections
Used for good and evil
 Cryptography as a Security Tool
Broadest security tool available

Internal to a given computer, source and destination of messages

can be known and protected
OS creates, manages, protects process IDs, communication
ports
Source and destination of messages on network cannot be trusted
without cryptography
Local network – IP address?
Consider unauthorized host added
WAN / Internet – how to establish authenticity
Not via IP address
Cryptography
Means to constrain potential senders (sources) and / or receivers
(destinations) of messages
◦ Based on secrets (keys)
◦ Enables
◦ Confirmation of source
◦ Receipt only by certain destination
◦ Trust relationship between sender and receiver
Encryption
Constrains the set of possible receivers of a message

Encryption algorithm consists of

◦ Set K of keys
◦ Set M of Messages
◦ Set C of ciphertexts (encrypted messages)
◦ A function E : K → (M→C). That is, for each k  K, Ek is a
function for generating ciphertexts from messages
◦ Both E and Ek for any k should be efficiently computable
functions
◦ A function D : K → (C → M). That is, for each k  K, Dk is a
function for generating messages from ciphertexts
◦ Both D and Dk for any k should be efficiently computable
functions
Symmetric Encryption
Same key used to encrypt and decrypt
◦ Therefore k must be kept secret

DES was most commonly used symmetric block-encryption algorithm (created by US Govt)
◦ Encrypts a block of data at a time
◦ Keys too short so now considered insecure

Triple-DES considered more secure

◦ Algorithm used 3 times using 2 or 3 keys
◦ For example

2001 NIST adopted new block cipher - Advanced Encryption Standard ( AES)
◦ Keys of 128, 192, or 256 bits, works on 128 bit blocks

RC4 is most common symmetric stream cipher, but known to have vulnerabilities
◦ Encrypts/decrypts a stream of bytes (i.e., wireless transmission)
◦ Key is a input to pseudo-random-bit generator
◦ Generates an infinite keystream
Secure Communication over
Insecure Medium
Asymmetric Encryption
Public-key encryption based on each user having two keys:
◦ public key – published key used to encrypt data
◦ private key – key known only to individual user used to decrypt data

Must be an encryption scheme that can be made public without making

it easy to figure out the decryption scheme
◦ Most common is RSA block cipher
◦ Efficient algorithm for testing whether or not a number is prime
◦ No efficient algorithm is know for finding the prime factors of a number
Asymmetric Encryption (Cont.)
Formally, it is computationally infeasible to derive kd,N from ke,N, and so ke
need not be kept secret and can be widely disseminated
◦ ke is the public key
◦ kd is the private key
◦ N is the product of two large, randomly chosen prime numbers p and q (for
example, p and q are 512 bits each)
◦ Encryption algorithm is Eke,N(m) = mke mod N, where ke satisfies kekd mod (p−1)
(q −1) = 1
◦ The decryption algorithm is then Dkd,N(c) = ckd mod N
Encryption using
RSA Asymmetric Cryptography
Authentication
Constraining set of potential senders of a message
◦ Complementary to encryption
◦ Also can prove message unmodified

Algorithm components
◦ A set K of keys
◦ A set M of messages
◦ A set A of authenticators
◦ A function S : K → (M→ A)
◦ That is, for each k  K, Sk is a function for generating authenticators from messages
◦ Both S and Sk for any k should be efficiently computable functions
◦ A function V : K → (M × A→ {true, false}). That is, for each k  K, Vk is a
function for verifying authenticators on messages
◦ Both V and Vk for any k should be efficiently computable functions
 Authentication – Hash Functions
Basis of authentication

Creates small, fixed-size block of data message digest (hash value) from
m
Hash Function H must be collision resistant on m
◦ Must be infeasible to find an m’ ≠ m such that H(m) = H(m’)

If H(m) = H(m’), then m = m’

◦ The message has not been modified

Common message-digest functions include MD5, which produces a 128-

bit hash, and SHA-1, which outputs a 160-bit hash
Not useful as authenticators
◦ For example H(m) can be sent with a message
◦ But if H is known someone could modify m to m’ and recompute H(m’) and
modification not detected
◦ So must authenticate H(m)
Authentication - MAC
Symmetric encryption used in message-authentication code (MAC)
authentication algorithm
Cryptographic checksum generated from message using secret key
◦ Can securely authenticate short values

If used to authenticate H(m) for an H that is collision resistant, then

obtain a way to securely authenticate long message by hashing them
first
Note that k is needed to compute both Sk and Vk, so anyone able to
compute one can compute the other
Authentication – Digital
Signature
Based on asymmetric keys and digital signature algorithm

Authenticators produced are digital signatures

Very useful – anyone can verify authenticity of a message
In a digital-signature algorithm, computationally infeasible to derive ks
from kv
◦ V is a one-way function
◦ Thus, kv is the public key and ks is the private key

Consider the RSA digital-signature algorithm

◦ Similar to the RSA encryption algorithm, but the key use is reversed
◦ Digital signature of message Sks (m) = H(m)ks mod N
◦ The key ks again is a pair (d, N), where N is the product of two large, randomly
chosen prime numbers p and q
◦ Verification algorithm is Vkv(m, a) (akv mod N = H(m))
◦ Where kv satisfies kvks mod (p − 1)(q − 1) = 1
Digital Certificates
Proof of who or what owns a public key

Public key digitally signed a trusted party

Trusted party receives proof of identification from entity and certifies
that public key belongs to entity
Certificate authority are trusted party – their public keys included with
web browser distributions
◦ They vouch for other authorities via digitally signing their keys, and so on
Man-in-the-middle Attack on
Asymmetric Cryptography
Authentication Vs Authorization
Authentication Vs Authorization Cont..
 Upcoming topics

System Protection