How to Configure IPS on FortiGate Firewall

Introduction

Intrusion Prevention Systems (IPS) play a critical role in network security by identifying and
blocking threats before they can cause harm. FortiGate's IPS uses both anomaly-based and
signature-based techniques to protect networks from a variety of attacks. This guide provides a
detailed overview of IPS concepts, configuration steps, and best practices.

Key Concepts

Vulnerability and Exploit

• Vulnerability: A weakness in an IT system (OS or Application) that can be exploited by an

attacker to deliver a successful attack.

• Exploit: A program or piece of code designed to find and take advantage of a security
flaw or vulnerability in an application or system, typically for malicious purposes such as
installing malware. An exploit is not malware itself but rather a method used by
cybercriminals to deliver malware.

Vulnerabilities vs. Anomalies

• Vulnerabilities: Known attacks with identifiable patterns that can be matched by IPS,
WAF, or antivirus signatures.

• Anomalies: Unusual behaviors in the network, such as higher-than-normal CPU usage or

network traffic, which may indicate new attacks (Zero-day attacks). Detected through
behavioral analysis such as DoS policies and protocol constraint checks.

IPS concepts

Purpose

IPS protects your network from external attacks using two main techniques:

• Anomaly-based Defense: Detects and mitigates attacks based on unusual patterns in

network traffic. For example, it can block a host that is being flooded with more traffic
than it can handle (e.g., DoS attacks).

• Signature-based Defense: Detects known attacks by matching traffic against a database

of known attack signatures. These signatures are specific sequences of commands or
patterns that are known to be used in attacks.

IPS components

IPS Signatures

• Signatures: These are specific patterns or sequences of commands associated with

known attacks. Signatures include details about the attack, such as the network protocol,
vulnerable operating system, and application.

___________________________________________________________________________
How to Configure IPS on FortiGate Firewall | By: Emad Hegazi
Page 1 of 9 #support_team247 #FortiGate | https://www.linkedin.com/in/emadhegazi/
 • Protocol Decoders: These identify the protocols used in network traffic before checking
for attacks, allowing the IPS engine to efficiently scan only the relevant protocols.

IPS Engine

• Function: The IPS engine examines network traffic for attack signatures once the protocol
decoders have identified the protocols in use.

• Updates: The IPS engine and signatures are updated regularly to improve detection and
performance. FortiGuard provides frequent updates to ensure the IPS can detect new
vulnerabilities.

FortiGuard IPS updates

• IPS signatures updated most frequently.

By default, each FortiGate firmware release includes an initial set of IPS signatures. Used
to upgrade the IPS signature database. In this way, the IPS is still effective for new
vulnerabilities.

• Protocol decoders are rarely updated unless the protocol specification or RFC changes
(which doesn't happen often).

• The IPS engine itself changes more frequently, but still infrequently.

___________________________________________________________________________
How to Configure IPS on FortiGate Firewall | By: Emad Hegazi
Page 2 of 9 #support_team247 #FortiGate | https://www.linkedin.com/in/emadhegazi/
 • The default automatic update schedule for FortiGuard packages has been updated.
Previously, the frequency was a recurring random interval within two hours.

Starting with FortiOS 7.0, the frequency is automatic and the update interval is calculated
based on the model and percentage of active subscriptions. The update interval is within
1 hour.

• The FortiGuard research team identifies and builds new signatures, just like antivirus
signatures. So, if your FortiGuard service contract expires, you can still use IPS.

However, just like anti-virus scanning, the efficiency of IPS scanning will increase with the
extension of signature time, and old signatures will not be able to defend against new
attacks.

Types of IPS signature Databases

Regular signature database:

contains signatures of some common attacks that cause few or no false positives. It is a
smaller database and its default action is to block detected attacks.

___________________________________________________________________________
How to Configure IPS on FortiGate Firewall | By: Emad Hegazi
Page 3 of 9 #support_team247 #FortiGate | https://www.linkedin.com/in/emadhegazi/
The extended signature database:

• contains additional signatures for attacks that have a high-performance impact, or that
by their nature do not support blocking.

• In fact, due to its big size, the extended database is not suitable for FortiGate models with
smaller disks or memory. However, for high security networks, you may need to enable
the extended signature database.

• the extended database package may be disabled by default on some models, such as
desktop models.

You can only enable the extended IPS database by using the CLI.

To enable the extended IPS database:

conf

ig ips global

set database extendedend

• FortiGate models with the CP9 SPU receive the IPS full extended database, and the other
physical FortiGate models receive a slim version of the extended database.

The slim-extended DB is a smaller version of the full extended DB that contains top active
IPS signatures. It is designed for customers who prefer performance.

Creating IPS Security Profiles (IPS Sensors)

1. Navigate to Security Profiles > Intrusion Prevention.

2. Create a new IPS sensor by adding predefined signatures or using filters.

___________________________________________________________________________
How to Configure IPS on FortiGate Firewall | By: Emad Hegazi
Page 4 of 9 #support_team247 #FortiGate | https://www.linkedin.com/in/emadhegazi/
Adding IPS Signatures and Filters

There are two ways to add predefined signatures on IPS sensors.

• By Signature:

Choose signatures individually. When a signature is selected in the list, that signature is
added to the sensor as a default action. You can then right click on the signature and
change the action.

Rate-based signatures

You can also add rate-based signatures to block specific traffic when it exceeds a
threshold for a configured period of time.

Rate-based signatures are a subset of the signatures that are found in the database that
are normally set to monitor.

This group of signatures is for vulnerabilities that are normally only considered a serious
threat when the targeted connections come in multiples, a little like DoS attacks.

This saves system resources and prevents repeated attacks. FortiGate does not track
statistics for this client, and it is temporarily blocked.

• By Filters:

Add signatures to sensors by using filters. FortiGate adds all signatures that match the
filter.

IPS Inspection Sequence

Rules are similar to firewall policy matches; the engine first evaluates the filters and
signatures at the top of the list and applies the first match. The engine skips subsequent
filters.

Avoid doing too many filters, as this increases computation and CPU usage.

Also, avoid making very large signature groups in each filter, which increases memory
usage.

___________________________________________________________________________
How to Configure IPS on FortiGate Firewall | By: Emad Hegazi
Page 5 of 9 #support_team247 #FortiGate | https://www.linkedin.com/in/emadhegazi/
 Exempt IPs

Sometimes it is desirable to exempt specific source or destination IP addresses from

specific signatures.

This feature is very useful during false positive virus outbreaks. You can temporarily bypass
affected endpoints until you investigate and correct the false positive issue.

IP exemptions can only be configured on a single signature. Each signature can have
multiple IP exemptions.

Managing IPS Actions

• Allow: Permit traffic to continue to its destination.

• Monitor: Allow traffic and log the activity.

• Block: Silently drop the traffic.

• Reset: Generate a TCP RST message to terminate the connection.

• Default: Use the signature's default action.

• Quarantine: Isolate the attacker's IP for a specified period.

• Packet Logging: Save copies of packets that match the signature for analysis.

DDoS attack

In DDoS attacks, an attacker directs a large number of computers to attempt normal

access of the target system. If enough access attempts are made, the target is

___________________________________________________________________________
How to Configure IPS on FortiGate Firewall | By: Emad Hegazi
Page 6 of 9 #support_team247 #FortiGate | https://www.linkedin.com/in/emadhegazi/
 overwhelmed and unable to service genuine users. The attacker does not gain access to
the target system, but it is not accessible to anyone else.

Besides protecting against threats and exploitation of vulnerabilities, the IPS engine is also
responsible for mitigating DDoS attacks using anomaly-based defense,

Since the botnet database is part of the FortiGuard IPS contract, administrators can
enable scanning for botnet connections to maximize its internal security.

{A botnet refers to a group of computers which have been infected by malware and
have come under the control of a malicious actor. The term botnet is from the word’s
robot and network and each infected device is called a bot.}

Botnets and C&Cs Actions:

• Disabled: do not scan connections to botnet servers

• Block: Block the connection with the botnet server
• Monitoring: record the log of connecting to the botnet server

Best Practices

✓ Before implementing an IPS, you need to analyze your network needs.

• Enabling the default profile across all policies can quickly lead to problems, the
fewest of which are false positives.
• Performing unnecessary inspections on all network traffic can result in high
resource utilization, which can hinder FortiGate's ability to handle regular traffic.
✓ You must assess the applicable threats. If your organization only runs Windows, you don't
need to scan for Mac OS vulnerabilities.

✓ It is also important to consider the direction of traffic. There are many IPS signatures that
only apply to clients, and many signatures that only apply to servers.
___________________________________________________________________________
How to Configure IPS on FortiGate Firewall | By: Emad Hegazi
Page 7 of 9 #support_team247 #FortiGate | https://www.linkedin.com/in/emadhegazi/
 ✓ Create IPS sensors specific to the resources you wish to protect. This ensures that FortiGate
does not scan insignificantly signed traffic.
✓ IPS is not a fixed implementation. You need to monitor the logs regularly to find unusual
traffic patterns and adjust the configuration of the IPS profile according to the actual
situation. You should also regularly audit your internal resources to determine if certain
vulnerabilities still apply to your organization.
✓ Certain vulnerabilities apply only to encrypted connections. In some of these cases,
FortiGate cannot reliably identify threats without being able to parse the payload. For this
reason, if you want to get the most out of your IPS and WAF capabilities, you must use SSL
inspection profiles.
✓ Usually, the traffic that needs to be inspected, such as anti-virus and IPS, is processed by
the CPU on the FortiGate. However, there are dedicated chips on certain FortiGate
models that offload these inspection tasks. This frees up CPU cycles to manage other
tasks, and also speeds up sessions that require security checks.

Troubleshooting

✓ IPS update requests Send to update.fortiguard.net on TCP port 443.

✓ You should periodically check for the most recent update timestamp.

• If there are any indications that the IPS definitions haven't been updated, you
should investigate {Always make sure that FortiGate has proper DNS resolution for
update.fortiguard.net}.
✓ The high CPU usage of the IPS engine is abnormal and needs to be checked. You can
troubleshoot these issues by using CLI, {diagnose test application ipsmonitor} command

___________________________________________________________________________
How to Configure IPS on FortiGate Firewall | By: Emad Hegazi
Page 8 of 9 #support_team247 #FortiGate | https://www.linkedin.com/in/emadhegazi/
 • Option 2: completely disable the IPS engine
• Option 5: enables IPS bypass mode. In this mode, the IPS engine is still running, but
no traffic inspection is performed.
If CPU usage drops after this, it usually indicates that the traffic being inspected is
too high for that FortiGate model.
If CPU usage remains high after enabling IPS bypass mode, it usually indicates a
problem with the IPS engine and needs to be reported to Fortinet Support.
• Option 99: ensures that all IPS-related processes restart gracefully.

✓ When there is no enough memory in the IPS socket buffer to accommodate new
packets, IPS will enter fail open mode.

What happens in this state depends on the IPS configuration.

• If the fail-open setting is enabled, some new packets (depending on system load)
will pass through without being inspected.
• If this setting is disabled, new packets will be dropped.

Frequent IPS failures to open events usually indicate that the IPS cannot meet the
traffic requirements. So, try to recognize patterns {Has traffic increased recently?
Has throughput demand increased? Are fail-opens triggered at specific times of
the day?}.

___________________________________________________________________________
How to Configure IPS on FortiGate Firewall | By: Emad Hegazi
Page 9 of 9 #support_team247 #FortiGate | https://www.linkedin.com/in/emadhegazi/