Computer System Security

(040613601)

Lecture 9 : Network Security (II)

IDS, Firewall, IPS

By Assoc. Prof. Benchaphon Limthanmaphon, PhD.

 Objective

 Students be able to understand:

 Network intrusion
 Intrusion detection
 how firewalls work
 how to use the Honey Pot.
 how a VPN works
 Students can apply to protect the network.

2
 Contents
 introduced intruders & intrusion  introduced need for & purpose of
detection firewalls
 hackers, criminals, insiders  types of firewalls
 intrusion detection approaches  packet filter
 host-based (single and  stateful inspection
distributed)  application
 network  circuit gateways
 distributed adaptive  Next-gen firewall
 exchange format  Firewall location
 SNORT example  honeypots
 VPN
 IPSec
 Intrusion Prevention Systems
 UTM
3
 Network Security Attacks
 classify as passive or active
 passive attacks are eavesdropping
 release of message contents
 traffic analysis
 are hard to detect so aim to prevent
 active attacks modify/fake data
 masquerade
 replay
 modification
 denial of service
 hard to prevent so aim to detect

4
 Intruders
 significant issue hostile/unwanted trespass
 from benign to serious
 user trespass
 unauthorized logon, privilege abuse
 software trespass
 virus, worm, or trojan horse
 classes of intruders:
 Masquerader: An individual, likely an outsider, not authorized to use
the computer and who penetrates a system's access controls to
exploit a legitimate user's account.
 Misfeasor: A legitimate user, generally an insider, who accesses
data, programs, or resources for which such access is not
authorized, or who misuses authorized access.
 Clandestine user: An individual, either an outsider or an insider,
who seizes supervisory control of the system and uses this control to
evade auditing and access controls or to suppress audit collection
5
 Examples of Intrusion
 remote root compromise
 web server defacement
 guessing / cracking passwords
 copying viewing sensitive data / databases
 running a packet sniffer
 distributing pirated software
 impersonating a user to reset password
 using an unattended workstation

6
 Security Intrusion & Detection
Security Intrusion
a security event, or combination of multiple security events, that
constitutes a security incident in which an intruder gains, or
attempts to gain, access to a system (or system resource)
without having authorization to do so.
Intrusion Detection
a security service that monitors and analyzes system events for
the purpose of finding, and providing real-time or near real-time
warning of attempts to access system resources in an
unauthorized manner.

The definitions shown from RFC 2828 (Internet Security Glossary)

7
 Hackers
 motivated by thrill of access and status
 hacking community a strong meritocracy
 status is determined by level of competence
 benign intruders might be tolerable
 do consume resources and may slow performance
 can’t know in advance whether benign or malign
 IDS / IPS / VPNs can help counter
 awareness led to establishment of CERTs (Computer Emergency
Response Team)
 collect / disseminate vulnerability info / responses

8
 Hacker Behavior Example

1. select target using IP lookup tools

2. map network for accessible services
3. identify potentially vulnerable services
4. brute force (guess) passwords
5. install remote administration tool
6. wait for admin to log on and capture password
7. use password to access remainder of network

9
 Criminal Enterprise
 organized groups of hackers now a threat
 corporation / government / loosely affiliated gangs
 typically young
 often Eastern European or Russian hackers
 common target credit cards on e-commerce server
 criminal hackers usually have specific targets
 once penetrated act quickly and get out
 IDS / IPS help but less effective
 sensitive data needs strong protection

10
 Criminal Enterprise Behavior

1. act quickly and precisely to make their activities harder to detect

2. exploit perimeter via vulnerable ports
3. use trojan horses (hidden software) to leave back doors for re-
entry
4. use sniffers to capture passwords
5. do not stick around until noticed
6. make few or no mistakes.

11
 Insider Attacks
 among most difficult to detect and prevent
 employees have access & systems knowledge
 may be motivated by revenge / entitlement
 when employment terminated
 taking customer data when move to competitor
 IDS / IPS may help but also need:
 least privilege, monitor logs, strong authentication, termination
process to block access & mirror data

12
 Insider Behavior Example

1. create network accounts for themselves and their friends

2. access accounts and applications they wouldn't normally use for their
daily jobs
3. e-mail former and prospective employers
4. conduct furtive instant-messaging chats
5. visit web sites that cater to disgruntled employees
6. perform large downloads and file copying
7. access the network during off hours.

13
 Intrusion Techniques
 objective to gain access or increase privileges
 initial attacks often exploit system or software vulnerabilities to
execute code to get backdoor
 e.g. buffer overflow
 or to gain protected information
 e.g. password guessing or acquisition

14
 Intrusion Methods
 Port Scan to find, for the target, Ex. Nmap - which ports/services are running
 Toolkits provided by manufacturers to make products compatible with their
products. These may be used to discover the vulnerabilities of the product.
 Impersonation Methods - Guess the ID and password of an authorized user:
 by guessing passwords / default passwords given with a system by its
manufacturer
 by overflow - in some ill designed systems, authentication may be foiled
by ‘overflow’ of password (if the password overflows, the system may
assume authentication)
 by non-existent authentication. In Unix, the file
- .rhosts lists the trusted hosts
- rlogin lists trusted users, who can access without authentication
A user may login one system as a guest- to access public information
and through this host, he may connect to a trusted host
- /etc/hosts.equiv file 15
 Impersonation Examples

 Masquerade of a site: An example:

Thus xxx.com bank may be the official site.
A hacker registers x_xx.com and asks clients to visit the site. Thus
passwords and pin numbers may be collected for misuse.
 Session Hijacking: An example:
A customer may select books on Amazon.com. When it comes to
taking the order and making the payment, Amazon.org may hijack the
session.

Man-in-the-middle Attack vs. Session Hijacking

Man-in-the-middle is wire-tapping actively from the beginning,
whereas a session-hijacker takes over after part of the session is over.

16
 Intrusion Detection Systems
 is a hardware device or software app that monitors inbound and outbound
network traffic to detect vulnerability exploits, policy violations, and
malicious activity.
 The system places sensors or network devices like servers, firewalls, and
routers to analyze traffic activity continuously and detect abnormal changes
in patterns.
 In case it detects unusual behavior, the IDS will notify administrators
immediately. The administrator can then review alarms and take action
to eliminate threats right away.
 an IDS can:
 Monitor online user behavior
 Recognize attack patterns within network packets
 Detect abnormal traffic activity and raise notifications
 Ensure user and system activity compliance with security policies
 IDS usually looks for two suspicious cybercrime clues:
 Signatures or patterns of known attacks
17
 Abnormal deviations from regular activity
 IDS
 logical components:
 sensors - collect data
 analyzers - determine if intrusion has occurred
 user interface - manage / direct / view IDS
 classify intrusion detection systems (IDSs) as:
 Host-based IDS (HIDS): monitor single host activity
 Network-based IDS (NIDS): monitor network traffic
 Distributed or hybrid IDS: combines info from a number of sensors

18
 IDS Principles
 assume intruder behavior differs from legitimate users
 expect overlap as shown
 observe deviations from past history

 problems of:
 false positives authorized
users identified as
intruders
 false negatives intruders
not identified as intruders
 must compromise
 the effectiveness of
some IDSs can be
misinterpreted due to a
statistical error known
as the base-rate fallacy.
19
 IDS Requirements
 run continually
 be fault tolerant
 resist subversion
 impose a minimal overhead on system
 configured according to system security policies
 adapt to changes in systems and users
 scale to monitor large numbers of systems
 provide graceful degradation of service
 allow dynamic reconfiguration

20
 Analysis Approaches
 Anomaly Detection
 defines normal/expected/legitimate user behavior
 Signature or Heuristic Detection
 Uses a set of known malicious data patterns (signatures) or attack
rules (heuristics)
 Compared with current behavior
 Also known as misuse detection
 Can only identify known attacks for which it has patterns or rules

21
 Host-Based IDS

 specialized software to monitor system activity to detect suspicious

behavior
 primary purpose is to detect intrusions, log suspicious events,
and send alerts
 can detect both external and internal intrusions
 two approaches, often used in combination:
 anomaly detection - defines normal/expected behavior
 threshold detection
 profile based
 signature detection - defines proper behavior

22
 Anomaly Detection
 threshold detection
 checks excessive event occurrences over time
 alone a crude and ineffective intruder detector
 must determine both thresholds and time intervals
 profile based
 characterize past behavior of users / groups
 then detect significant deviations
 based on analysis of audit records
 gather metrics: counter, guage, interval timer, resource utilization
 analyze: mean and standard deviation, multivariate, markov
process, time series, operational model

23
 Signature Detection
 observe events on system and applying a set of rules to decide if intruder
 approaches:
 rule-based anomaly detection
 analyze historical audit records for expected behavior to identify
usage patterns & auto-generate rules for them
 Then observe current behavior & match against rules to see if
conforms
 rule-based penetration identification
 rules identify known penetrations / weaknesses
 often by analyzing attack scripts from Internet
 supplemented with rules from security experts

24
 Distributed Intrusion Detection

 Traditional focus is on single systems

 But typically have networked systems
 More effective defense has these working together to detect
intrusions
 Issues:
 Dealing with varying audit record formats
 Integrity & confidentiality of networked data
 Centralized or decentralized architecture

25
 Distributed Host-Based IDS
 Host agent module: An audit collection
module operating as a background
process on a monitored system. Its
purpose is to collect data on security-
related events on the host and transmit
these to the central manager.

 LAN monitor agent module: Operates in the same fashion as a host agent
module except that it analyzes LAN traffic and reports the results to the
central manager.
 Central manager module: Receives reports from LAN monitor and host
agents and processes and correlates these reports to detect intrusion. 26
 Distributed Host-Based IDS
 The agent captures each audit record produced
by the native audit collection system.
 A filter is applied that retains only those records
that are of security interest. These records are
then reformatted into a standardized format
referred to as the host audit record (HAR).
 template-driven logic module analyzes the
records for suspicious activity.
 the agent scans for notable events that are of
interest independent of any past events.
 the agent looks for anomalous behavior of an
individual user based on a historical profile of
that user
 When suspicious activity is detected,
an alert is sent to the central manager.
 The central manager includes an expert system that can draw inferences
from received data
27
 Network-Based IDS
 network-based IDS (NIDS)
 monitor traffic at selected points on a network
 in (near) real time to detect intrusion patterns
 may examine network, transport and/or application level protocol
activity directed toward systems
 comprises a number of sensors
 inline (possibly as part of other net device)
 passive (monitors copy of traffic)

28
NIDS Sensor Deployment

29
 NIDS :Intrusion Detection Techniques
 signature detection
 Application, transport, network layers; unexpected application
services, policy violations
 anomaly detection
 denial of service attacks, scanning, worms
 when potential violation detected sensor sends an alert and logs
information
 used by analysis module to refine intrusion detection parameters and
algorithms
 by security admin to improve protection

30
Distributed Adaptive Intrusion Detection

31
 Intrusion Detection Exchange Format
 functional components are the key elements of any IDS
 Data source: raw data an IDS uses to detect unauthorized or undesired
activity
 Sensor: collects data from the data source & forwards events to the
analyzer
 Analyzer: process analyzing data
collected for unauthorized /
undesired activity
 Administrator: human with overall
responsibility for setting security
policy of org
 Manager: process from which
operator manages components of
ID system
 Operator: human that is the
primary user of the IDS manager
32
 SNORT

 lightweight IDS
 real-time packet capture and rule analysis
 passive or inline
 four logical components
 Packet Decoder: efficiently processes each captured packet to identify
and isolate protocol headers at the data link, network, transport, and
application layers.
 Detection Engine: does actual work of intrusion detection, analyzing
each packet using rules defined for this configuration of Snort by the
security administrator.
 Logger: of each packet that matches a rule, if specified. The security
administrator can then use the log file for later analysis.
 Alerter: can be sent for each detected packet to a file, a UNIX socket,
or a database. 33
 SNORT Rules
 use a simple, flexible rule definition language
 with fixed header and zero or more options
 header includes: action, protocol, source IP, source port, direction,
dest IP, dest port
 many options
 example rule to detect TCP SYN-FIN attack:
Alert tcp $EXTERNAL_NET any -> $HOME_NET any \
(msg: "SCAN SYN FIN"; flags: SF, 12; \
reference: arachnids, 198; classtype: attempted-recon;)

34
 Firewalls and IPS
 effective means of protecting LANs
 A firewall is an integrated collection of security measures designed to
prevent unauthorized electronic access to a networked computer system.
 acting as a barrier between your network and the outside world.
 They control network traffic, allowing authorized access while blocking
unauthorized attempts.

35
 Firewall Characteristics
 All traffic from inside to outside and vice versa, must pass through the
firewall – achieved by physically blocking all access to the local network
 Only authorized traffic, as defined by the local security policy, will be
allowed to pass.
 Packets flowing through a firewall can have one of three outcomes:
Accepted: permitted through the firewall
Dropped: not allowed through with no
indication of failure
Rejected: not allowed through,
accompanied by an attempt to inform
the source that the packet was rejected

36
 Firewall Characteristics (2)
 Policies used by the firewall to handle packets are based on several
properties of the packets being inspected, including
 the protocol used, TCP or UDP
 the source and destination IP addresses
 the source and destination ports
 the application-level payload of the packet (e.g.whether it contains virus).
 The firewall itself is immune to penetration.
 4 general techniques that firewalls use to control access and enforce the
site’s security policy:
 Service control : firewall filter traffic, interprets service request
 Direction control: determines direction in which particular service
requests
 User control: control access to a service according to which user is
attempting to access it
 Behavior control: control how particular services are used.
37
 Firewall Capabilities & Limits
 capabilities:
 defines a single choke point
 provides a location for monitoring security events
 convenient platform for some Internet functions such as NAT,
usage monitoring
 IPSEC VPNs
 limitations:
 cannot protect against attacks bypassing firewall
 may not protect fully against internal threats
 improperly secure wireless LAN
 laptop, PDA, portable storage device infected outside then used
inside

38
 Types of Firewalls
 A firewall may act as a packet filter. It can operate as
 a positive filter, allowing to pass only packets that meet specific
criteria
 a negative filter, rejecting any packet that meets certain criteria.
 Depending on the type of firewall, it may examine one or more
protocol headers in each packet, the payload of each packet, or the
pattern generated by a sequence of packets.

39
 Types of Firewalls
principal types of firewalls include:
 Packet Filtering Firewall
 Stateful Inspection Firewalls
 Application-Level Gateway
 Circuit-Level Gateway
 Next-Generation Firewalls

40
 Packet Filtering Firewall
 applies a set of rules to each incoming and outgoing IP packet and then
forwards or discards the packet
 based on information in packet header
 src/dest IP addr & port, IP protocol, interface
 typically a list of rules of matches on fields
 if match rule says if forward or discard packet
 Pros: Simple, fast, and low-cost.
 Cons: Limited in functionality, only examines packet headers, vulnerable
to IP spoofing.

41
 Stateful Inspection Firewall
 reviews packet header information but also keeps
info on TCP connections
 maintain tables containing information on each
active connection, including the IP addresses, ports,
and sequence numbers of packets.
 can tell when packets are part of legitimate sessions
originating within a trusted network.
 keep track of TCP
sequence numbers to
prevent attacks that
depend in some fashion on
the sequence number,
such as session hijacking.

42
 Application-Level Gateway
 acts as a relay of application-level traffic - as intermediaries between your
network and external networks.
 user contacts gateway with remote host name
 authenticates themselves
 gateway contacts application on remote host and relays TCP segments
between server and user
 inspect the content of packets at the application level
gateway-to-remote
host-to-gateway host telnet session Also called Application proxy
telnet session Proxy Server : A program that
deals with external servers on
behalf of internal clients.
application
gateway router and filter

43
 Circuit-Level Gateway
 sets up two TCP connections, to an inside user and to an outside host
 relays TCP segments from one connection to the other without
examining contents
 hence independent of application logic
 just determines whether relay is permitted
 typically used when inside users trusted
 may use application-level gateway inbound and circuit-level gateway
outbound
 hence lower overheads
 Also called Circuit-level proxy

 Pros: Efficient and fast, good for

basic connection control.
 Cons: Doesn't inspect the content of packets,
vulnerable to attacks that mimic legitimate
connections
44
 Next-Generation Firewalls (NGFWs)
 the most advanced type of firewalls.
 combine traditional firewall capabilities with advanced features like
 Deep packet inspection
 Identify and control application (apply specific rules)
 Intrusion prevention systems (IPS)
 Detect advanced threats – NGFWs often integrate with threat intelligence
feeds, which provide up-to-date information about the latest threats and
attack techniques.
 Malware protection - Some NGFWs include sandboxing capabilities,
where suspicious files are detonated in a safe environment to analyze
their behavior. This helps identify and block zero-day malware
 Pros: Comprehensive security, can identify and block sophisticated attacks,
provides detailed visibility into network traffic.
 Cons: Can be expensive and complex to manage.

45
 Firewall Basing
 It is common to base a firewall on a standalone machine running a
common operating system, such as UNIX, windows.
 On Linux, Iptables is used to provide firewall function
 On Windows, use “control panel” “Windows Firewall”
 Firewall functionality can also be implemented as a software module
in a router or LAN switch.

 several options for locating firewall:

 bastion host
 individual host-based firewall
 personal firewall

46
 Bastion Hosts
 hosts application/circuit-level gateways
 critical strongpoint in network

 common characteristics:
 runs secure OS
 only essential services are installed
 may require user auth to access proxy or host
 each proxy can restrict features, hosts accessed
 each proxy small, simple, checked for security
 each proxy is independent
 each proxy runs as a non-privileged user in a private and secured
directory on host.
 limited disk use, hence read-only code

47
 Host-Based Firewalls
 a software module used to secure individual host
 available in/add-on for many OS
 filter packet flows
 often used on servers
 advantages:
 tailored filter rules for specific host needs
 protection from both internal / external attacks
 additional layer of protection – a new type of server can be added
to the network, with its own firewall, without necessity of altering
the network firewall configurations

48
 Personal Firewall
 controls traffic flow to/from PC/workstation
 for both home or corporate use
 may be software module on PC
 or in home cable/DSL router/gateway
 typically much less complex
 primary role to deny unauthorized access
 may also monitor outgoing traffic to detect/block worm/malware
activity

49
 Firewall Locations
 A firewall is positioned to provide a protective barrier
between an external, potentially untrusted source of
traffic and an internal network.

 With that general principal

in mind, a security
administrator must decide
on the location and on the
number of firewalls needed

 An external (or perimeter)

firewall is placed at the
edge of a local or enterprise
network.
 One or more internal
firewalls protect the bulk of
the enterprise network.
50
 Firewall Locations (2)
 Between these two types of firewalls are one or more networked devices
in a region referred to as a DMZ (demilitarized zone) network.
 Systems that are externally accessible but need some protections are
usually located on DMZ networks.
 The external firewall provides a measure of access control and protection
for the DMZ systems consistent with their need for external connectivity.

Internet

DMZ
Firewall

Firewall
Intranet
WEB EMAIL PROXY
SERVER SERVER SERVER 51
 Firewall Locations (3)
 Internal firewalls serve three purposes:
 adds more stringent filtering capability in order to protect enterprise
servers and workstations from external attack.
 provides two-way protection with respect to the DMZ, protecting the
remainder of the network from attacks launched from DMZ and
protecting DMZ systems from attack from the internal protected
network.
 Multiple internal firewalls can be used to protect portions of the
internal network from each other.

52
 Honeypots
 are decoy systems
 filled with fabricated info
 instrumented with monitors / event loggers
 divert and hold attacker to collect activity info
 without exposing production systems
 initially were single systems
 more recently are/emulate entire networks

53
Honeypot Deployment

Location of Honeypot
 outside external
Firewall
 In DMZ (demilitarized
Zone)
 Internal Honeypot

54
 Virtual Private Networks
 A secure connections, or tunnels, over the Internet
 is a technology that allows private networks to be safely extended over
long physical distances by making use of a public network, such as
the Internet, as a means of transport.

Encrypted Data

55
 Virtual Private Networks (2)
 VPN consists of a set of computers that interconnect by means of a
relatively unsecure network.
 VPN counter the use of a public network exposes corporate traffic to
eavesdropping and provides an entry point for unauthorized users
 VPN uses encryption and authentication in the lower protocol layers to
provide a secure connection through an otherwise insecure network,
typically the Internet.
 The encryption may be performed by firewall software or possibly by
routers.
 The most common protocol is at the IP level and is known as IPSec.
 A logical means of implementing an IPSec is in a firewall
 If IPSec is implemented in a separate box behind (internal to) the
firewall, then VPN traffic passing through the firewall in both directions
is encrypted. In this case, the firewall is unable to perform its filtering
function or other security.
 IPSec could be implemented in the boundary router, outside the
firewall. However, this device is likely to be less secure than the
firewall and thus less desirable as an IPSec platform.
56
Virtual Private Networks

57
 IPSec (IP Security Protocol Suite)
 IPv6 solves IP addressing problem
 Common IP attacks: spoofing, eavesdropping, and session hijacking
 IPSec defines a standard means for handling encrypted data
 IPSec is implemented at the IP layer, so it affects all layers above
 The basic of IPSec is a security association, which is essentially the
set of security parameters for a secured communication channel,
includes:
 Encryption algorithm and mode, key and parameters
 Authentication protocol
 Lifespan of the association, permit long-running sessions choose
a new cryptographic key
 Address of the opposite end of association
 Sensitivity level of protected data (usable for classified data)

58
 IPSec
 Supports two encryption modes:
 Transport - encrypts only the data portion (payload) of each packet, but
leaves the header untouched.

 Tunnel mode (more secure) - encrypts both the header and the payload.
On the receiving side, an IPSec-compliant device decrypts each packet

59
 Intrusion Prevention Systems (IPS)

 recent addition to security products which

 An IPS is an inline network-based IDS (NIDS) that has the
capability to block traffic by discarding packets as well as simply
detect suspicious traffic.
 For host-based systems, an IPS is a host-based IDS that can
discard incoming traffic.
 An IPS is a functional addition to a firewall that adds IDS types of
algorithms to the repertoire of the firewall.
 can block traffic like a firewall
 using IDS algorithms
 may be network or host based

60
IDS vs IPS

61
 Host-Based IPS
 identifies attacks using both:
 signature techniques
 malicious application packets
 anomaly detection techniques
 behavior patterns that indicate malware
 can be tailored to the specific platform
 e.g. general purpose, web/database server specific
 can also sandbox applets to monitor behavior
 may give desktop file, registry, I/O protection

62
 Network-Based IPS
 inline NIDS that can discard packets or terminate TCP connections
 uses signature and anomaly detection
 may provide flow data protection
 monitoring full application flow content
 can identify malicious packets using:
 Pattern matching: Scans incoming packets for specific byte sequences
(the signature) stored in a database of known attacks.
 Stateful matching: Scans for attack signatures in the context of a traffic
stream rather than individual packets.
 Protocol anomaly: Looks for deviation from standards set forth in RFCs.
 Traffic anomaly: Watches for unusual traffic activities, such as a flood of
UDP packets or a new service appearing on the network.
 Statistical anomaly: Develops baselines of normal traffic activity and
throughput, and alerts on deviations from those baselines.
 cf. SNORT inline can drop/modify packets
63
Unified Threat Management Products
 One approach to reducing the
administrative and performance
burden is to replace all inline
network products (firewall, IPS, IDS,
VPN, antispam, antisypware, and so
on) with a single device, a unified
threat management (UTM) system,
that integrates a variety of
approaches to dealing with network-
based attacks

64
 Unified Threat Management Products
1. Inbound traffic is decrypted if necessary before its initial inspection. If the
device functions as a VPN boundary node, then IPSec decryption would
take place here.
2. An initial firewall module filters traffic, discarding packets that violate rules
and/or passing packets that conform to rules set in the firewall policy.
3. A number of modules analyze individual packets and flows of packets at
various protocols levels. A data analysis engine is responsible for keeping
track of packet flows and coordinating the work of antivirus, IDS, and IPS
engines.
4. Data analysis engine reassembles multipacket payloads for content
analysis by the antivirus engine and the web filtering and antispam
modules.
5. some incoming traffic may need to be re-encrypted to maintain internal
security
6. all detected threats are reported to the logging and reporting module, which
is used to issue alerts for specified conditions and for forensic analysis.
7. the bandwidth-shaping module can use various priority and quality of
65
service (QoS) algorithms to optimize performance.