Module 6

Malware Analytics
Intrusion Detection System
An Intrusion Detection System( IDS) is a
wall of defense to confront the attacks of
computer systems on the internet.
The main assumption of the IDS is that the
behavior of intruders is different from legal
users.
What is an intrusion detection system?

 IDS are used for IDS key components

◦ Detect intrusions and Audit Source
intrusion attempts
◦ Give alarms
◦ Stop on-going attacks
◦ Trace attackers
Models Detection
◦ Investigate and assess the
damage
◦ Gather information for Response
recovery actions
Where does intrusion detection fit into the
security landscape?
Security

Prevention Detection Recovery

Intrusion
Encryption Firewalls Detection
Systems (IDS)

Network Virus
…
IDS Scanners

 Principle: Defense in Depth (layered mechanisms)

Prevent Detect React/Survive
Classification
 Intrusion Detection methods
◦ Signature-based intrusion detection systems
(SIDS)
◦ Anomaly-based intrusion detection system
(AIDS)
 Data Source
◦ Host-based IDS (HIDS)
◦ Network-based IDS (NIDS)

IT 6823 – LM9 IDS 5

Types of Cyber attacks
Cyber-attacks can be categorized based on the activities and
targets of the attacker. Each attack type can be classified into
one of the following four classes :
 Denial-of-Service (DoS) attacks have the objective of
blocking or restricting services delivered by the network,
computer to the users.
 Probing attacks have the objective of acquisition of
information about the network or the computer system.
 User-to-Root (U2R) attacks have the objective of a non-
privileged user acquiring root or admin-user access on a
specific computer or a system on which the intruder had
user-level access.
 Remote-to-Local (R2L) attacks involve sending packets to
the victim machine. The cybercriminal learns the user's
activities and obtains privileges which an end-user could have
on the computer system.
Sub categories of computer attacks
Types of IDS
 Anomaly approaches: Determine whether
deviations from normal usage patterns
can be flagged as intrusions
 Misuse or Signature detection approaches:
This kind of approach uses patterns of
well-known attacks to identify intrusions.
Clearly Machine Learning is well suited for
the first kind of approach.
Signature-based intrusion detection
systems (SIDS)
 Signature intrusion detection systems (SIDS) are
based on pattern matching techniques to find a
known attack;
 these are also known as Knowledge-based
Detection or Misuse Detection
 In SIDS, matching methods are used to find a
previous intrusion, when an intrusion signature
matches with the signature of a previous intrusion
that already exists in the signature database, an
alarm signal is triggered.
 For SIDS, host's logs are inspected to find
sequences of commands or actions which have
previously been identified as malware
Conceptual working of SIDs

The main idea is to build a database of intrusion signatures and

to compare the current set of activities against the existing
signatures and raise an alarm if a match is found. For example, a
rule in the form of "if: antecedent -then: consequent" may lead to
"if (source IP address=destination IP address) then label as an
attack ".
SIDS- Pros and cons
 SIDS usually gives an excellent detection accuracy for
previously known intrusions
 However, SIDS has difficulty in detecting zero-day attacks for
the reason that no matching signature exists in the database
until the signature of the new attack is extracted and stored
 Traditional approaches to SIDS examine network packets
and try matching against a database of signatures. But these
techniques are unable to identify attacks that span several
packets. As modern malware is more sophisticated it may be
necessary to extract signature information over multiple
packets. This requires the IDS to recall the contents of
earlier packets.
 With regards to creating a signature for SIDS, generally, there
have been a number of methods where signatures are
created as state machines ,formal language string patterns or
semantic conditions .
 The increasing rate of zero-day attacks has
rendered SIDS techniques progressively less
effective because no prior signature exists for
any such attacks.
 Polymorphic variants of the malware and the
rising amount of targeted attacks can further
undermine the adequacy of this traditional
paradigm.
 A potential solution to this problem would be
to use AIDS techniques, which operate by
profiling what is an acceptable behavior rather
than what is anomalous
Anomaly-based intrusion detection
system (AIDS)
 Signature intrusion detection systems (SIDS) are
based on pattern matching techniques to find a
known attack; these are also known as Knowledge-
based Detection or Misuse Detection
 In SIDS, matching methods are used to find a
previous intrusion. In other words, when an intrusion
signature matches with the signature of a previous
intrusion that already exists in the signature database,
an alarm signal is triggered.
 For SIDS, host's logs are inspected to find sequences
of commands or actions which have previously been
identified as malware. SIDS have also been labelled in
the literature as Knowledge-Based Detection or
Misuse Detection
Techniques for implementing AIDS

 AIDS methods can be categorized into three main

groups:
 Statistics-based ; knowledge-based; machine learning-
based
 The statistics-based approach involves collecting and
examining every data record in a set of items and
building a statistical model of normal user behavior.
 knowledge-based approach tries to identify the
requested actions from existing system data such as
protocol specifications and network traffic instances
 Machine-learning methods acquire complex pattern-
matching capabilities from training data.
Classification of ADS methods
AIDS approaches based on machine
learning
 Machine learning is the process of extracting
knowledge from large quantities of data. Machine
learning models comprise of a set of rules, methods,
or complex "transfer functions" that can be applied to
find interesting data patterns, or to recognize or
predict behavior
 Machine learning techniques have been applied
extensively in the area of AIDS.
 Several algorithms and techniques such as clustering,
neural networks, association rules, decision trees,
genetic algorithms, and nearest neighbour methods,
have been applied for discovering the knowledge from
intrusion datasets
Feature selection for IDS
 Feature selection is helpful to decrease the computational difficulty,
eliminate data redundancy, enhance the detection rate of the
machine learning techniques, simplify data and reduce false alarms.
In this line of research, some methods have been applied to develop
a lightweight IDSs.
 Feature selection techniques can be categorized into wrapper and
filter methods.
 Wrapper methods estimate subgroups of variables to identify the
feasible interactions between variables. There are two main
drawbacks of these techniques: accumulative overfitting when the
amount of data is insufficient and the importance calculation time
when the amount of variables is big.
 Filter methods are normally applied as a pre-processing stage. The
selection of features is separate of any machine learning techniques.
As an alternative, features are nominated on the basis of their
scores in several statistical tests for their correlation with the
consequence variable.
 Refer to Naïve Bayes theorem material
Unsupervised learning in intrusion
detection system
 Unsupervised learning is a form of machine learning
technique used to obtain interesting information from input
datasets without class labels
 The input data points are normally treated as a set of
random variables.
 A joint density model is then created for the data set.
 In supervised learning, the output labels are given and used
to train the machine to get the required results for an
unseen data point, while in unsupervised learning, no labels
are given, and instead, the data is grouped automatically into
various classes through the learning process.
 In the context of developing an IDS, unsupervised learning
means, use of a mechanism to identify intrusions by using
unlabeled data to train the model.
K- means Clustering
 K-means: The K-means techniques are one of the
most prevalent techniques of clustering analysis that
aims to separate 'n' data objects into 'k' clusters in
which each data object is selected in the cluster with
the nearest mean.
 It is a distance-based clustering technique and it does
not need to compute the distances between all
combinations of records.
 It applies a Euclidean metric as a similarity measure.
 The number of clusters is determined by the user in
advance.
◦ Typically several solutions will be tested before accepting
the most appropriate one.
 Once records are clustered, all of the cases that appear in small clusters
are labelled as an intrusion because the normal occurrences should
produce sizable clusters compared to the anomalies. In addition, malicious
intrusions and normal instances are dissimilar, thus they do not fall into the
identical cluster
IDS evasion techniques

 These techniques are employed by a

cybercriminal to avoid detection by IDS
 IDS Evasion techniques: Fragmentation,
Flooding, Obfuscation, and Encryption.
 These techniques pose a challenge for the
current IDS as they circumvent existing
detection methods.
Fragmentation
 A packet is divided into smaller packets. The fragmented
packets are then be reassembled by the recipient node at the
IP layer before forwarding it to the Application layer.
 To examine fragmented traffic correctly, the network
detector needs to assemble these fragments similarly as it
was at fragmenting point.
 The restructuring of packets needs the detector to hold the
data in memory and match the traffic against a signature
database. Methods used by attackers to escape detection by
hiding attacks as legitimate traffic are fragmentation overlap,
overwrite, and timeouts
 Fragmentation attack replaces information in the constituent
fragmented packets with new information to generate a
malicious packet.
Packet Fragment 3 is generated by the attacker. The network intrusion
detector must retain the state for all of the packets of the traffic which
it is detecting.

The duration of time that the detector can maintain a state of traffic
might be smaller than the period that the destination host can maintain
a state of traffic
Flooding
 The attacker begins the attack to overwhelm the
detector and this causes a failure of control
mechanism.
 When the detector fails, all traffic would be
allowed
 A popular method to create a flooding situation is
spoofing the legitimate User Datagram Protocol
(UDP) and Internet Control Message Protocol
(ICMP).
 The traffic flooding is used to disguise the
abnormal activities of the cybercriminal.
Therefore, IDS would have extreme difficulty to
find malicious packets in a huge amount of traffic.
Obfuscation
 Obfuscation techniques can be used to evade detection, which are the
techniques of concealing an attack by making the message difficult to
understand
 The terminology of obfuscation means changing the program code in a way
that keeps it functionally identical with the aim to reduce detectability to
any kind of static analysis or reverse engineering process and making it
obscure and less readable. This obfuscation of malware enables it to evade
current IDS.
 Obfuscation attempts to utilize any limitations in the signature database
and its capability to duplicate the way the computer host examines
computer's data
 An effective IDS should be supporting the hexadecimal encoding format
or having these hexadecimal strings in its set of attack signatures
 Unicode/UTF-8 standard permits one character to be symbolized in
several various formats. Cybercriminals may also use double-encoded data,
exponentially escalating the number of signatures required to detect the
attack.
 SIDS relies on signature matching to identify malware where the signatures
are created by human experts by translating a malware from machine code
into a symbolic language such as Unicode. However, the use of code
obfuscation is very valuable for cybercriminals to avoid IDSs.
Encryption
 Generally, encryption offers a number of security services,
such as data confidentiality, integrity, and privacy.
 Malware authors employ these security attributes to escape
detection and conceal attacks that may target a computer
system.
 For example, attacks on encrypted protocols such as
HyperText Transfer Protocol Secure (HTTPS) cannot be read
by an IDS
 The IDS cannot match the encrypted traffic to the existing
Database signatures if it doesn't interpret the encrypted
traffic.
 Therefore, examining encrypted traffic makes it difficult for
detectors to detect attacks
 For example, packet content-based features have been
applied extensively to identify malware from normal traffic,
which cannot readily be applied if the packet is encrypted.
Challenge of IDS on intrusion
evasion detection
 Detecting attacks masked by evasion techniques is a
challenge for both SIDS and AIDS.
 The ability of evasion techniques would be determined by
the ability of IDS to bring back the original signature of the
attacks or create new signatures to cover the modification of
the attacks.
 Robustness of IDS to various evasion techniques still needs
further investigation. For example, SIDS in regular
expressions can detect the deviations from simple mutation
such as manipulating space characters, but they are still
useless against a number of encryption techniques.
Summary
 Cybercriminals are targeting computer users by using
sophisticated techniques as well as social engineering
strategies.
 Cybercriminals have shown their capability to obscure their
identities, hide their communication, distance their identities
from illegal profits, and use infrastructure that is resistant to
compromise.
 Therefore, it becomes increasingly important for computer
systems to be protected using advanced intrusion detection
systems that are capable of detecting modern malware.
 In order to design and build such IDS systems, it is necessary
to have a complete overview of the strengths and limitations
of contemporary IDS research.