MODULE III

Intrusion Detection & Malicious Software

Intruders

One of the key threats to security is the use of some form of hacking by an intruder, often referred to as a
hacker or cracker. three classes of intruders are there.

• Masquerader – unauthorized user who penetrates a system exploiting a authorized user’s

account (outside)
• Misfeasor – authorized user who makes unauthorized accesses or misuses his privileges (inside)
• Clandestine user – Takes control to auditing and access controls
lists the following examples of intrusion:

• Performing a remote root compromise of an e-mail server

• Defacing a Web server

• Guessing and cracking passwords

• Copying a database containing credit card numbers

• Viewing sensitive data, including payroll records and medical information, without authorization

Intrusion Detection classification

Intrusion Detection: A security service that monitors and analyzes system events for the purpose of
finding, and providing real-time or warning of attempts to access system resources in an unauthorized
manner.

IDSs are often classified based on the source and type of data analyzed, as:

• Host-based IDS (HIDS): Monitors the characteristics of a single host and the events occurring within
that host for evidence of suspicious activity.

• Network-based IDS (NIDS): Monitors network traffic for particular network segments or devices
and analyzes network, transport, and application protocols to identify suspicious activity.

An IDS comprises three logical components:

• Sensors: Sensors are responsible for collecting data. The input for a sensor may be any part of a system
that could contain evidence of an intrusion.
• Analyzers: Analyzers receive input from one or more sensors or from other analyzers. The analyzer is
responsible for determining if an intrusion has occurred.
• User interface: The user interface to an IDS enables a user to view output from the system or control
the behavior of the system

Requirements of an IDS

lists the following as desirable for an IDS. It must

• Run continually with minimal human supervision.

• Be fault tolerant in the sense that it must be able to recover from system crashes

• Impose a minimal overhead on the system where it is running.

• Be able to adapt to changes in system and user behavior over time.

• Be able to scale to monitor a large number of hosts.

• Provide graceful degradation of service in the sense that if some components of the IDS stop working
for any reason, the rest of them should be affected as little as possible.

Host based Intrusion detection

Host-based IDSs (HIDSs) add a specialized layer of security software to vulnerable or sensitive
systems; such as database servers and administrative systems. The HIDS monitors activity on the system
in a variety of ways to detect suspicious behavior. Its main purpose is to detect intrusions, log suspicious
events, and send alerts.

Host based IDSs follow one of two general approaches to intrusion detection:

1. Anomaly detection: Involves the collection of data relating to the behavior of legitimate users
over a period of time. Then current observed behavior is analyzed to determine whether this
behavior is that of a legitimate user or that of an intruder.
2. Signature or Heuristic detection: Uses a set of known malicious data patterns (signatures) or
 attack rules (heuristics) that are compared with current behavior to decide if is that of an intruder.

Audit records

A fundamental tool for intrusion detection is audit record. Some record of ongoing activity by users must
be maintainted as input to an IDS. Basically two plans are used.

1. Native Audit Records:

It is an Accounting software that collects information on user activity. Advantage is that no

additional collection software is required.

2. Detection Specific Audit Records

A collection facility to generate audit records containing required information used by IDS.
Each audit records contains the following fields.

1. Subject : Initiators of actions. Eg:user

2. Action : Operation performed by the subject on or with an object eg. Login,read,perform
I/O,execute etc..
3. Object : Receptors of actions. Eg. Files, programs,records,messages, printers etc..
4. Exception Condition: Denotes which, if any , exception condition is raised on return.
5. Resource Usage : A list of quantitative elements
6. Time-Stamp : Unique time and date stamp identifying when the action took place.

Anomaly Detection

Anomaly detection technique fall into two broad catogories:

Threshold Detection and Profile Based System.

Threshold Detection: It involve counting the number of occurance of a specific event type over an
interval of time.If the count passes, Then intrusion is assumed.

Profile Based detection

It focuses on charaterizing the past behaviour of individual users or related groups of users and then
detecting significant deviations.

Signature Detection

Signature or heuristic techniques detect intrusion by observing events in the system and applying
either a set of signature patterns to the data, or a set of rules that characterize the data, leading to a
decision regarding whether the observed data indicates normal or attacker behavior.

Signature approaches match a large collection of known patterns of malicious data against data
stored on a system . This approach is widely used in antivirus products. The advantages of this approach
include the relatively low cost in time and resource use, and its wide acceptance.

Rule-based heuristic identification involves the use of rules for identifying attackers. Typically,
the rules used in these systems are specific to the machine and operating system.
Distributed host based intrusion detection
It is used to detect intrusion in a distributed collection of hosts supported by a LAN or
internetwork.

A good example of a distributed IDS is one developed at the University of California at Davis.

Figure 8.2 shows the overall architecture, which consists of three main components:

1. Host agent module: An audit collection module operating as a background process on a monitored
system. Its purpose is to collect data on securityrelated events on the host and transmit these to the central
manager. Figure 8.3 shows details of the agent module architecture.

2. LAN monitor agent module: Operates in the same fashion as a host agent module except that it
analyzes LAN traffic and reports the results to the central manager.

3. Central manager module: Receives reports from LAN monitor and host agents and processes and
correlates these reports to detect intrusion.

When suspicious activity is detected, an alert is sent to the central manager. The central manager
includes an expert system that can draw inferences from received data. The LAN monitor agent also
supplies information to the central manager. The LAN monitor agent audits host-host connections,
services used, and volume of traffic. It searches for significant events, such as sudden changes in network
load, the use of security-related services, and suspicious network activities.
 Network based intrusion detection

A network-based IDS (NIDS) monitors traffic at selected points on a network or interconnected

set of networks. The NIDS examines the traffic packet by packet in real time, or close to real time, to
attempt to detect intrusion patterns.

A typical NIDS facility includes a number of sensors to monitor packet traffic, one or more
servers for NIDS management functions, and one or more management consoles for the human interface.

Types of Network Sensors

Sensors can be deployed in one of two modes: Inline and passive.

An Inline sensor is inserted into a network segment so that the traffic that it is monitoring must
pass through the sensor. One way to achieve an inline sensor is to combine NIDS sensor logic with
another network device, such as a firewall or a LAN switch. This approach has the advantage that no
additional separate hardware devices are needed;

A passive sensor monitors a copy of network traffic; the actual traffic does not pass through the device.
From the point of view of traffic flow, the passive sensor is more efficient than the inline sensor, because it
does not add an extra handling step that contributes to pack et delay.

Intrusion Detection Exchange Format

It is used to facilitate the development of distributed IDSs that can function across a wide range of
platforms and environments, standards are needed to support interoperability.
Figure illustrates the key elements of the model on which the intrusion detection message exchange
approach is based.

The functional components are as follows:

• Data source: The raw data that an IDS uses to detect unauthorized or undesired activity. Common data
sources include network packets, operating system audit logs, application audit logs, and system-
generated checksum data.
• Sensor: Collects data from the data source. The sensor forwards events to the analyzer.

• Analyzer: The ID component or process that analyzes the data collected by the sensor for signs of
unauthorized or undesired activity .
• Administrator: The human with overall responsibility for setting the security policy of the
organization, and, thus, for decisions about deploying and configuring the IDS.
• Manager: The ID component or process from which the operator manages the various components of
the ID system
• Operator: The human that is the primary user of the IDS manager.
 Honeypots

Honeypots are systems that are designed to remove a potential attacker away from critical systems.
Honeypots are designed to:

• Divert an attacker from accessing critical systems.

• Collect information about the attacker’s activity.

• Encourage the attacker to stay on the system long enough for administrators to respond.

These systems are filled with fabricated information designed to appear valuable but that a legitimate user
of the system would not access. Thus, any access to the honeypot is suspect.

Honeypots are typically classified as being either low or high interaction.

• Low interaction honeypot: Consists of a software package that emulates particular IT services or
systems well enough to provide a realistic initial interaction, but does not execute a full version of those
services or systems.
• High interaction honeypot: Is a real system, with a full operating system, services and applications,
which are instrumented and deployed where they can be accessed by attackers.

Honeypots can be deployed in a variety of locations.

Figure illustrates some possibilities.

Example System: Snort

Snort is an open source, highly configurable and portable host-based or network-based IDS. Snort is
referred to as a lightweight IDS, which has the following characteristics:

• Easily deployed on most nodes (host, server, router) of a network.

• Efficient operation that uses small amount of memory and processor time.

• Easily configured by system administrators who need to implement a specific security solution in a short
amount of time.

Snort can perform real-time packet capture, protocol analysis, and content searching and
matching.
Snort Architecture

A Snort installation consists of four logical components

• Packet decoder: The packet decoder processes each captured packet to identify and isolate
protocol headers at the data link, network, transport, and application layers.

• Detection engine: The detection engine does the actual work of intrusion detection. This
module analyzes each packet based on a set of rules defined for this configuration of Snort by the security
administrator.If no rule matches the packet, the detection engine discards the packet.

• Logger: The security administrator can then use the log file for later analysis.

• Alerter: For each detected packet, an alert can be sent.

Snort Rules

Snort uses a simple, flexible rule definition language that generates the rules used by the detection
engine

• Action: The rule action tells Snort what to do when it finds a packet that matches the rule
criteria
• Protocol: Snort proceeds in the analysis if the packet protocol matches this field.
• Source IP address: Designates the source of the packet.
• Source port: This field designates the source port for the specified protocol
• Direction: This field takes on one of two values: unidirectional or bidirectional.
• Destination IP address: Designates the destination of the packet.

• Destination port: Designates the destination port.

Malicious Software

Malicious software, or malware, arguably constitutes one of the most significant categories of
threats to computer systems.
Definition: “a program that is inserted into a system, usually covertly, with the intent of compromising the
confidentiality, integrity, or availability of the victim’s data, applications, or operating system or otherwise
annoying or disrupting the victim.

Types of Malicious Software (Malware)

Backdoor (trapdoor)

Any mechanism that bypasses a normal security check; it may allow unauthorized access to functionality
 in a program, or onto a compromised system.
It is a secret entry point into a program that allows someone who is aware of the backdoor to gain access
without going through the usual security access procedures.

Logic bomb

Code inserted into malware by an intruder. A logic bomb lies dormant until a predefined condition is met;
the code then triggers an unauthorized act.

Trojan horse

A computer program that appears to have a useful function, but also has a hidden and potentially
malicious function that tolerates security mechanisms.

Mobile code

Software (e.g., script, macro, etc) that can be shipped unchanged to a heterogeneous collection of
platforms and execute with identical semantics.

Multiple Threat Malware

1. Blended attack:

A blended threat is an exploit that combines elements of multiple types of malware

A blended threat typically includes:

More than one means of propagation -- for example, sending an email with a hybrid virus/worm that
will self-replicate and also infect a Web server so that contagion will spread through all visitors to a
particular site.
• The intent to cause real harm, for example, by launching a denial of service (DOS) attack against a
target or delivering a Trojan horse that will be activated at some later date.

2.Multiparatite Virus:

A multipartite virus is a computer virus that infects and spreads in multiple ways.

Viruses

The Nature of Viruses

A computer virus is a piece of software that can “infect” other programs, or indeed any type of
executable content, by modifying them. The modification includes injecting the original code with a
routine to make copies of the virus code, which can then go on to infect other content.
 A computer virus has three parts:

• Infection mechanism: The means by which a virus spreads or propagates, enabling it to replicate. The
mechanism is also referred to as the infection vector.

• Trigger: The event or condition that determines when the payload is activated or delivered, sometimes
known as a logic bomb.

• Payload: What the virus does, besides spreading. The payload may involve damage or may involve
benign but noticeable activity.

During its lifetime, a typical virus goes through the following four phases:

• Dormant phase: The virus is idle. The virus will eventually be activated by some event, such as a date,
the presence of another program or file.

• Propagation phase: The virus places a copy of itself into other programs or into certain system areas
on the disk.

• Triggering phase: The virus is activated to perform the function for which it was intended.
• Execution phase: The function is performed. The function may be harmless, such as a message on the
screen, or damaging, such as the destruction of programs and data files.

Virus Structure

The infected program begins with the virus code and works as follows. The first line of code is a special
marker that is used by the virus to determine whether or not a potential victim program has already been
infected with this virus. When the program is invoked, control is immediately transferred to the main
action block containing the virus code. The virus may first seek out uninfected executable files and infect
them. Next, the virus may execute its payload if the required trigger conditions, if any, are met. Finally,
the virus transfers control to the original program.
Viruses Classification

A virus classification by target includes the following categories:

• Boot sector infector: Infects a master boot record or boot record and spreads when a system is booted
from the disk containing the virus.

• File infector: Infects files that the operating system or shell consider to be executable.

• Macro virus: Infects files with macro or scripting code that is interpreted by an application.

• Multipartite virus: Infects files in multiple ways. Typically, the multipartite virus is capable of
infecting multiple types of files, so that virus eradication must deal with all of the possible sites of
infection.

A virus classification by concealment strategy includes the following categories:

• Encrypted virus: A form of virus that uses encryption to obscure it’s content
• Stealth virus: A form of virus explicitly designed to hide itself from detection by anti-virus software.

• Polymorphic virus: A form of virus that creates copies during replication that are functionally
equivalent but have distinctly different bit patterns, in order to defeat programs that scan for viruses. In
this case, the “signature” of the virus will vary with each copy.

• Metamorphic virus: As with a polymorphic virus, a metamorphic virus mutates with every infection.
Antivirus approaches

• Detection: Once the infection has occurred, determine that it has occurred and locate the malware.

• Identification: Once detection has been achieved, identify the specific malware that has infected the
system

• Removal: Once the specific malware has been identified, remove all traces of malware virus from all
infected systems so that it cannot spread further.

four generations of anti-virus software:

• First generation: simple scanners- A first-generation scanner requires a malware signature to
identify the malware.

• Second generation: heuristic scanners- the scanner uses heuristic rules to search for probable
malware instances.

• Third generation: activity traps- memory-resident programs that identify malware by its actions
rather than its structure in an infected program

• Fourth generation: full-featured protection- Fourth-generation products are packages consisting of

a variety of anti-virus techniques.

Antivirus techniques

• Generic Decryption

Generic decryption (GD) technology enables the antivirus program to easily detect even the most
complex polymorphic viruses and other malware, while maintaining fast scanning speeds .
• CPU emulator: A software-based virtual computer. Instructions in an executable file are interpreted by
the emulator rather than executed on the underlying processor. Virus signature scanner: A module that
scans the target code looking for known malware signatures.

• Emulation control module: Controls the execution of the target code.

BOTS

A Bot(robot) is also known as a zombie or drone, is a program that secretly takes over another
Internet-attached computer and then uses that computer to launch or manage attacks that are difficult to
trace to the bot’s creator. The bot is typically planted on hundreds or thousands of computers belonging to
unsuspecting third parties. The collection of bots often is capable of acting in a coordinated manner; such
a collection is referred to as a botnet.

Uses of Bots

• Distributed denial-of-service (DDoS) attacks:

A DDoS attack is an attack on a computer system or network that causes a loss of service to users.

• Spamming:

With the help of a botnet and thousands of bots, an attacker is able to send massive amounts of bulk e-
mail (spam).

• Sniffing traffic: Bots can also use a packet sniffer to watch for interesting clear text data passing by a
compromised machine Keylogging:

• Spreading new malware:

Botnets are used to spread new bots.

• Installing advertisement add-ons and browser helper objects (BHOs):

Botnets can also be used to gain financial advantages. This works by setting up a fake Web site with some
advertisements

• Manipulating online polls/games:

Online polls/games are getting more and more attention and it is rather easy to manipulate them with
botnets.
Remote Control Facility(RCF)

The remote control facility is what distinguishes a bot from a worm. A worm propagates itself and
activates itself, whereas a bot is controlled by some form of commandand-control (C&C) server network.
This contact does not need to be continuous, but can be initiated periodically when the bot observes it has
network access.
Rootkit
A rootkit is a set of programs installed on a system to maintain covert access to that system with
administrator (or root) privileges, while hiding evidence of its presence to the greatest extent possible.
This provides access to all the functions and services of the operating system. The rootkit alters the host’s
standard functionality in a malicious and stealthy way.
A rootkit can make many changes to a system to hide its existence, making it difficult for the
user to determine that the rootkit is present and to identify what changes have been made.

A rootkit can be classified using the following characteristics:

• Persistent: Activates each time the system boots. The rootkit must store code in a persistent
store, such as the registry or file system, and configure a method by which the code executes without user
intervention. This means it is easier to detect, as the copy in persistent storage can potentially be scanned.

• Memory based: Has no persistent code and therefore cannot survive a reboot. However,
because it is only in memory, it can be harder to detect.

• User mode: Intercepts calls to APIs (application program interfaces) and modifies returned
results. For example, when an application performs a directory listing, the return results do not include
entries identifying the files associated with the rootkit.

• Kernel mode: Can intercept calls to native APIs in kernel mode. The rootkit can also hide the
presence of a malware process by removing it from the kernel’s list of active processes.

Rootkit Installation

The following sequences is representative of a hacker attack to install a rootkit.

1. The attacker uses a utility to identify open ports or other vulnerabilities.

2. The attacker uses password cracking ,malware or system vulnerability to gain initial
access and root access.
3. The attacker uploads the rootkit to the victims machine.
4. The attacker can add a virus,DOS etc..to the rootkit’s payload
5. The attacker then run the rootkit installation script.
6. The rootkit replaces the binaries,files,commands or system utilities to hide it’s presence.
7. The rootkit listens at a port in the target server, install sniffers,or keyloggers or take other
steps to compromise the victim.
Worms
A worm is a program that actively seeks out more machines to infect, and then each infected machine
serves as an automated launching pad for attacks on other machines. Worm programs exploit software
vulnerabilities in client or server programs to gain access to each new system. They can use network
connections to spread from system to system. They can also spread through shared media, such as USB
drives or CD and DVD data disks.
To replicate itself, a worm uses some means to access remote systems

. These include the following:

• Electronic mail or instant messenger facility: A worm e-mails a copy of itself to other
systems

• File sharing: A worm either creates a copy of itself or infects other suitable files as a virus on
removable media such as a USB drive.

• Remote execution capability: A worm executes a copy of itself on another system

• Remote file access or transfer capability: A worm uses a remote file access or transfer service
to another system to copy itself from one system to the other.

• Remote login capability: A worm logs onto a remote system as a user and then uses commands
to copy itself from one system to the other, where it then executes.
A worm typically uses the same phases as a computer virus: dormant, propagation, triggering, and
execution.

Worm Propagation Model

A well-designed worm can spread rapidly and infect massive numbers of hosts. It is useful to have a
general model for the rate of worm propagation. Computer viruses and worms exhibit similar self-
replication and propagation behavior to biological viruses.

A simplified, classic epidemic model can be expressed as follows:

dI(t) /dt = bI(t)S(t)

where I(t) = number of individuals infected as of time t

S(t) = number of susceptible individuals (susceptible to infection but not yet infected) at time t

b = infection rate

N = size of the population,

N = I(t) + S(t)
 Figure shows the dynamics of worm propagation using this model. Propagation proceeds through
three phases. In the initial phase, the number of hosts increases exponentially. After a time, infecting hosts
waste some time attacking already infected hosts, which reduces the rate of infection. During this middle
phase, growth is approximately linear, but the rate of infection is rapid. When most vulnerable computers
have been infected, the attack enters a slow finish phase as the worm seeks out those remaining hosts that
are difficult to identify.

Requirements for Worm Countermeasures

• Generality: The approach taken should be able to handle a wide variety of attacks.

• Timeliness: The approach should respond quickly so as to limit the number of infected
programs or systems and the consequent activity.

• Resiliency: The approach should be resistant to evasion techniques employed by attackers to

hide the presence of their worms
.

• Minimal denial-of-service costs: The approach should result in minimal reduction in capacity
or service due to the actions of the countermeasure software.

• Transparency: The countermeasure software and devices should not require modification to
 existing (legacy) OSs, application software, and hardware.
Global and local coverage: The approach should be able to deal with attack sources both from outside and
inside the enterprise network
.
.