Computer Security:

Principles and Practice

Chapter 8: Intrusion Detection

Classes of intruders: criminals
• Individuals or members of an organized
crime group with a goal of financial reward
– Identity theft
– Theft of financial credentials
– Corporate espionage
– Data theft
– Data ransoming
• Meet in underground forums to trade tips
and data and coordinate attacks

2
Classes of intruders: activitists
• Are either individuals, usually working as
insiders, or members of a larger group of
outsider attackers, who are motivated by
social or political causes
• Also know as hacktivists
– Skill level is often quite low
• Aim of their attacks is often to promote and
publicize their cause typically through:
– Website defacement
– Denial of service attacks
– Theft and distribution of data that results in
negative publicity or compromise of their targets

3
Intruders: state-sponsored
• Groups of hackers sponsored by
governments to conduct espionage or
sabotage activities
• Also known as Advanced Persistent
Threats (APTs) due to the covert nature
and persistence over extended periods
involved with any attacks in this class
• Widespread nature and scope of these
activities by a wide range of countries
from China to the USA,
UK, and their intelligence allies

4
Intruders: others
• Include classic hackers or crackers who are
motivated by technical challenge or by peer-
group esteem and reputation
• Many of those responsible for discovering
new categories of buffer overflow
vulnerabilities could be regarded as
members of this class
• Given the wide availability of attack toolkits,
there is a pool of “hobby hackers” using
them to explore system and network security

5
Intruders: another
classification
• Masquerader: unauthorized individuals
who penetrates a system
• Misfeasor: legit user who accesses
unauthorized data
• Clandestine: seizes supervisory control

6
User and software trespass
• User trespass: unauthorized logon,
privilege abuse
• Software trespass: virus, worm, or Trojan
horse

7
Example of intrusion
• Remote root compromise
• Web server defacement
• Guessing/cracking passwords
• Copying databases containing credit card numbers
• Viewing sensitive data without authorization
• Running a packet sniffer
• Distributing pirated software (via FTP server, P-2-P)
• Using an unsecured modem to access internal network
• Impersonating an executive to get information
• Using an unattended workstation (without permission)

8
Hackers
• motivated by thrill of access and status
– hacking community a strong meritocracy
– status is determined by level of competence
• benign intruders might be tolerable
– do consume resources and may slow
performance
– can’t know in advance whether benign or
malign
• IDS / IPS / VPNs can help counter
• Conferences: DEFCON, BlackHat,
DerbyCon, HOPE, SummerCon, THOTCON
9
Hacker behavior example

1. Select target using IP lookup tools

2. Map network for accessible services
– study physical connectivity (via NMAP)
3. Identify potentially vulnerable services
4. Brute force (guess) passwords
5. Install remote administration tool
6. Wait for admin to log on and capture password
7. Use password to access remainder of network

10
Criminal intruder behavior

1. Act quickly and precisely to make their

activities harder to detect
2. Exploit perimeter via vulnerable ports
3. Use Trojan horses (hidden software) to leave
back doors for re-entry
4. Use sniffers to capture passwords
5. Do not stick around until noticed

11
Insider attacks

• Among most difficult to detect and prevent

• Employees have access & systems knowledge
• May be motivated by revenge/entitlement
– When employment terminated
– Taking customer data when move to competitor
• IDS/IPS may help but also need
– Least privilege, monitor logs, strong authentication,
termination process to block access & take mirror image
of employee’s HD (for future purposes)

12
Security intrusion & detection
(RFC 2828)
• Security intrusion: a security event, or
combination of multiple security events, that
constitutes a security incident in which an
intruder gains, or attempts to gain, access to
a system (or system resource) without having
authorization to do so.
• Intrusion detection: a security service that
monitors and analyzes system events for the
purpose of finding, and providing real-time or
near real-time warning of attempts to access
system resources in an unauthorized manner.

13
Intrusion techniques
• Objective to gain access or increase
privileges
• Initial attacks often exploit system or
software vulnerabilities to execute code to
get backdoor
– e.g. buffer overflow
• Or to gain protected information
– Password guessing or acquisition (or via social
engineering)

14
Intrusion detection systems
• Host-based IDS: monitor
single host activity
• Network-based IDS:
monitor network traffic
• Distributed or hybrid:
Combines information from a
number of sensors, often both
host and network based, in a
central analyzer that is able
to better identify and respond
to intrusion activity

15
IDS principles
• Assumption: intruder behavior differs from
legitimate users loose vs tight interpretation:
catch more (false +) or catch less (false -)
– Expect overlap as shown
– for legit users:
Observe major deviations
from past history
– Problems of:
• false positivesvalid user identified as intruder
intruder not identified
• false negatives
• must compromise

16
IDS requirements
• Run continually with minimal human supervision
• Be fault tolerant: recover from crashes
• Resist subversion: monitor itself from change by
intruder
• Impose a minimal overhead on system
• Configured according to system security policies
• Adapt to changes in systems and users
• Scale to monitor large numbers of systems
• Provide graceful degradation of service: if one
component fails, others should continue to work
• Allow dynamic reconfiguration

17
Detection techniques
• Anomaly (behavior) detection
• Signature/heuristic detection

18
IDS: Anomaly (behavior)
detection
• Involves the collection of data relating to
the behavior of legitimate users over a
period of time
• Current observed behavior is analyzed to
determine whether this behavior is that of
a legitimate user or that of an intruder

19
Anomaly detection
• Threshold detection
– checks excessive event occurrences over time
– alone a crude and ineffective intruder detector
– must determine both thresholds and time intervals
– lots of false positive/false negative may be possible
• Profile based
– characterize past behavior of users/groups
– then detect significant deviations
– based on analysis of audit records:
• gather metrics: counter, gauge, resource utilization, interval
time
• analyze: mean, standard deviation, multivariate

20
Example of metrics
• Counters: e.g., number of logins during
an hour, number of times a cmd executed
• Gauge: e.g., the number of outgoing
messages [pkts]
• Interval time: the length of time between
two events, e.g., two successive logins
• Resource utilization: quantity of
resources used (e.g., number of pages
printed)

21
Signature/heuristic detection
• Uses a set of known malicious data patterns
or attack rules that are compared with
current behavior
• Also known as misuse detection
• Can only identify known attacks for which it
has patterns or rules (signature)
– Very similar to anti-virus (requires frequent
updates)
– Rule-based penetration identification
• rules identify known penetrations/weaknesses
• often by analyzing attack scripts from Internet
• rules are specific to the machine or OS

22
Example of rules in a signature
detection IDS
• Users should not be logged in more than
one session
• Users should not read in other users’
directories
• Users must not write other users’ files
• Users who log after hours often access the
same files they used earlier
• Users do not generally open disk devices
but rely on high-level OS utils

23
Host-based IDS: signature vs
anomaly detection

Anomaly Detection Signature

Detection

24
Host-based IDS

• Specialized software to monitor system activity to

detect suspicious behavior
– primary purpose is to detect intrusions, log suspicious
events, and send alerts
– can detect both external and internal intrusions
• Two approaches, often used in combination:
– Anomaly detection: consider normal/expected
behavior over a period of time; apply statistical tests to
detect intruder
• threshold detection: for various events (#/volume of
copying)
• profile based (time/duration of login)
– Signature detection: defines proper (or bad) behavior
(rules)
25
Audit records
• A fundamental tool for intrusion detection
• Two variants:
– Native audit records: provided by O/S
• always available but may not be optimum
– Detection-specific audit records: IDS specific
• additional overhead but specific to IDS task
• often log individual elementary actions
• e.g. may contain fields for: subject, action, object,
exception-condition, resource-usage, time-stamp
• possible overhead (two such utilities)

26
Network-Based IDS
• Network-based IDS (NIDS)
– Monitor traffic at selected points on a network
– In (near) real time to detect intrusion patterns
– May examine network, transport and/or
application level protocol activity directed
toward systems
• Comprises a number of sensors
– Inline (possibly as part of other net device) –
traffic passes through it
– Passive (monitors copy of traffic)

27
Passive sensors

• Passive sensors are more efficient as packet delay is less

• The sensor has a NIC that connects to the n/w with a IP address

28
 2. monitor and documents
NIDS Sensor Deployment unfiltered packets;
more work to do

3. protect major backbones;

monitor internal/external attacks

1. monitor attacks from outsid

(see attacks to servers)

4. Special IDS to provide additional protection

for critical systems (e.g., bank accounts)

29
NIDS intrusion detection
techniques
• Signature detection
– at application (FTP), transport (port scans),
network layers (ICMP); unexpected application
services (host running unexpected app), policy
violations (website use)
• Anomaly detection
– of denial of service attacks, worms (significant
traffic increase)
• When potential violation detected, sensor
sends an alert and logs information
– Used by analysis module to refine intrusion
detection parameters and algorithms

30
Logging of alerts (for all
types)
• Typical information logged by a NIDS sensor
includes:
– Timestamp
– Connection or session ID
– Event or alert type
– Rating
– Network, transport, and application layer protocols
– Source and destination IP addresses
– Source and destination TCP or UDP ports, or ICMP types and codes
– Number of bytes transmitted over the connection
– Decoded payload data, such as application requests and
responses
– State-related information

31
Honeypots
• Decoy systems
– Filled with fabricated info (which appear authentic
and valuable)
– Instrumented with monitors/event loggers
– Lure a potential attacker away from critical systems
– Collect information about the attacker’s activity
– Encourage the attacker to stay on the system long
enough for administrators to respond
– Divert and hold attacker to collect activity info
without exposing production systems
• Initially were single systems
• More recently are/emulate entire networks

32
Honeypot classification
Honeypots differ in the way that they’re deployed and
the sophistication of the decoy.
•Low interaction honeypot
– Give an attacker very limited access to the operating
system
– Emulates services or systems well enough to provide a
realistic initial interaction, but does not execute a full
version of those services or systems
– Provides a less realistic target, unable to capture complex
threats such as zero-day exploits.
•High interaction honeypot
– A real system, with a full operating system, services and
applications, which are instrumented and deployed where
they can be accessed by attackers
– the biggest downside to a high interaction honeypot is the
time and effort it takes to build the decoy system at the
start

33
 1. Tracks attempts to connect
to an unused IP address; can’t

Honeypot help with inside attackers

deployment

3. Full internal
honeypot; can detect
internal attacks; detect
misconfigured firewall

2. In DMZ; must make sure the other

systems in the DMZ are secure; firewalls
may block traffic to the honeypot

34
Snort IDS
• Lightweight IDS
– Open source (rule-based)
– Real-time packet capture and rule analysis
– Passive or inline
– Components: decoder, detector, logger, alerter

processes captured intrusion

packets to identify detection
packet headers work

35
SNORT Rules
• Use a simple, flexible rule definition language
• Header includes: action, protocol, source IP,
source port, direction, dest IP, dest port
• Example rule to detect TCP SYN-FIN attack:
alert tcp $EXTERNAL_NET any -> $HOME_NET any \
(msg: "SCAN SYN FIN"; flags: SF, 12; \
reference: arachnids, 198; classtype: attempted-recon;)

– detects an attack at the TCP level; $strings are variables with defined
values; any source or dest port is considered; checks to see if SYN and
FIN bits are set

36
Other Open Source IDS

• OSSEC (Open Source Security Event

Correlator).
• Tripwire
• Wazuh
• Samhain
• Security Onion

37
Summary
• Introduced intruders & intrusion detection
– Hackers, criminals, insiders
• Intrusion detection approaches
– Host-based (single and distributed)
– Network
– Distributed adaptive
• Honeypots
• Snort example

38