Performance Evaluation and Comparative Analysis of

Network Firewalls
Chirag Sheth Rajesh Thakker
Tata Consultancy Services Limited, Electronics & Commu Dept,
Info-Tower-1, Infocity, Gandhinagar – 382009, India Vishwakarma Govt. Engg. College,
[email protected] Chandkheda, Gandhinagar – 382424, India
[email protected]

Abstract— Firewalls are no longer just perimeter devices for the is to examine each packet that passes through the entrance and
data center, but should be weaved into the fabric of the network decide whether to accept the packet and allow it to proceed or
from edge to edge such as to offer security layered in-depth and to discard the packet. A firewall's basic task is to regulate
ubiquitous. The next evolution of the firewall has to combine some of the flow of traffic between computer networks of
dynamic policy-based security with performance, rapid scaling, different trust levels. Typical examples are the Internet which
high availability and application intelligence. Today, increasing
is a zone with no trust and an internal network which is a zone
attention is paid to network firewall design quality due to
regulations such as the Sarbanes-Oxley act, CobiT framework, of higher trust. A zone with an intermediate trust level,
the Payment-Card Industry Data Security Standard (PCI DSS) situated between the Internet and a trusted internal network, is
and the NIST standard. All these regulations include specific often referred to as a "perimeter network" or Demilitarized
sections dealing with firewall configuration, management and zone (DMZ).
audit. A firewall’s configuration contains a large set of access
This paper is a humble attempt to examine various control rules, each specifying source addresses, destination
types of firewalls operational as on today and cross reference addresses, source ports, destination ports, one or multiple
each firewall operation with causes and effects of weaknesses in protocol ids, and an appropriate action. The action is typically
their operation. In addition, we analyze reported problems with
“accept” or “deny.” Some firewalls can support other types of
existing firewalls. Detailed analysis and comparison is done in
terms of cost, security, operational ease and implementation of actions such as sending a log message, applying a proxy, and
Open source packet filter (PF) firewall, Checkpoint SPLAT and passing the matched packets into a VPN tunnel. For most
Cisco ASA in a testing environment with laboratory generated firewalls, the rule set is order-sensitive. An incoming packet
traffic. Various throughputs and connections statistics were used will be checked against the ordered list of rules. The rule that
as benchmark for performance comparison. The results matches first decides how to process the packet. Due to the
indicated that Cisco ASA outperforms its peers in most multidimensional nature of the rules (including
performance criterions. Checkpoint SPLAT and OpenBSD PF source/destination addresses and ports), the performance of a
also provides reasonably good and competitive performance. The firewall degrades as the number of rules increases.
results reported in this paper will also be useful in comparing
Commercially deployed firewalls often carry tens of
vendors to procure firewall based on one’s own organizational
business requirements. thousands of rules, creating performance bottlenecks in the
network. More importantly, the empirical fact shows that the
Keywords— Network Security, Distributed Firewall, Checkpoint
number of configuration errors on a firewall increases sharply
NGX, Cisco ASA, OpenBSD PF along with the size of the rule set. A complex rule set can
easily lead to mistakes and mal-configuration [1-3].
I. INTRODUCTION Despite their critical role, firewalls have traditionally been
“Cyber Attacks Hit 75% of Global Enterprises in 2009” tested without well-defined and effective methodologies.
[Symantec, Feb-2010] Currently, a diverse set of firewalls are being used. Because it
As enterprises take on the Internet as a new business tool - is infeasible to examine each firewall separately for all
whether to sell, to collaborate or to communicate - web potential problems, a general mechanism is required to
applications have become the new weakest link in the understand firewall vulnerabilities in the context of firewall
organization's security strategy. Technological innovations are operations. The firewall data flow model presented in this
fundamentally changing the way people live, work, play, report gives an overall description of firewalls by detailing the
share information and communicate with each other. operations they perform (depicted in figure 1). When a packet
Network Firewalls protect a trusted network from an un- is received by a firewall, it first undergoes link layer filtering.
trusted network by filtering traffic according to a specified Then, it is checked against a dynamic rule set. The packet then
security policy. A firewall is often placed at the entrance of undergoes packet legality checks, and IP and port filtering.
each private network in the Internet. The function of a firewall Finally, network/port address translation is performed.

978-1-4244-9190-2/11/$26.00 ©2011 IEEE

 discovery are common to all. Some of the criteria in Audit
checklist which demand efficient firewall configuration and
management [4-6] are given in Table I.
TABLE I
MAJOR CRITERIONS IN AUDIT CHECKLIST FOR FIREWALL
CONFIGURATION IN VARIOUS REGULATORY COMPLIANCE

Major Regulatory Standards

Payment Card ISO – 27002 Control
Industry – Data Information Objectives for
Security Security Information and
Standards (PCI- Standard related
DSS) Requirements Technology
Requirements (COBIT)
Requirements
1. Install and 1. Network (where 1. Plan and
maintain a firewall business partner’s organize effective,
configuration to and/ or third robust and scalable
protect cardholder parties need access Network Security
data to information policy.
system) should be
Fig. 1 Firewall operations and data flow. 2. Encrypt segregated using 2. Define the level
Sophisticated firewalls also reassemble packets and transmission of perimeter security of security and
perform application level analysis. After a routing decision is cardholder data mechanisms such control that is
across open and as firewalls. necessary to
made on the packet, out-bound filtering may also be
public networks protect companies’
performed. Each of these operations is optional, and the order 2. Information assets through the
in which the packet traverses may also differ in different 3. Use and security policy development of an
firewalls. regularly update should be IT governance
anti-virus software approved by the model.
II. NETWORK FIREWALL – REGULATORY COMPLIANCE or programs management,
Managing the security of critical information has proven a published and 3. All users and
challenge for businesses and organizations of all sizes. Even 4. Track and communicated to their activity
companies that invest in the latest security infrastructure and monitor all access all employees. should be uniquely
to network identifiable and
tools soon discover that these technology-based solutions are resources and 3. Should have monitored.
short-lived. Primary objective of information security is to cardholder data controls such as
protect information from unauthorized access and maintain firewalls, 4. User access
confidentiality, integrity and availability of information. 5. Maintain a Operating system rights to systems
policy that hardening, any and data should be
addresses Intrusion detection in line with
information type of tools used defined and
security for to monitor the documented
employees and system etc. business needs and
contractors job requirements.
4. Encryption
6. Develop and techniques should 5. Define and
maintain secure be used to protect implement
systems and the data. procedures to
applications ensure integrity
5. Regular and consistency of
7. Do not use assessments should all data stored in
vendor-supplied be conducted to electronic form,
Fig. 2 Network security objectives. defaults for system analyze the such as databases,
passwords and sensitivity of the data warehouses
Information technology (IT) security is indispensable to an other security data and the level and data archives.
organization’s ability to conduct business and achieve its parameters. of protection
objectives. Security requirements affect almost every business needed.
process and system, and successful security measures help
protect a business’ brand value, stakeholder confidence, risk III. PERFORMANCE COMPARISIONS OF VARIOUS FIREWALLS
management strategies, and compliance status. Requirements In spite of exponential rise in firewall deployment, no
vary among industries, geographies, and regions, but the need standard method of firewall performance evaluation is
to protect privacy, retain important data, and facilitate e- prevalent in market. The primary reason for the same is that
firewall implementations vary widely making it difficult to TABLE IIII
PERFORMANCE TEST RESULTS
carry out direct performance comparisons. As more and more
organizations deploy firewalls on their networks, question System Under Test – Firewall
arises whether the products they buy will stand up and sustain Key Performance Products
to relatively heavy loads. Indicators (KPI) Cisco CP OpenB
ASA SPLAT SD PF
A. Performance Testing Setup Firewall Licensing Proprietary Proprietary BSD
In order to characterize performance of firewall, the testing Application Intelligence Yes Yes No
environment setup shown in Fig. 3 is used to compare Firewall Management Local Centralize Local
performance of three most operational firewalls in market. HTTP Throughput
10.6 5.6 4.5
(Gbps)
TCP Throughput (Gbps)
18.6 14.2 10.2
(Object size = 512 KB)
Concurrent Connections 200K 250K 500K
UDP Throughput (Gbps)
8 4 7
(Object size = 512 KB)
Connections per Second 160K 68K 180K

Fig. 3 Setup diagram for performance testing.

Test traffic is generated using Open-Source tool called

Curl-Loader. The Curl-Loader is capable of simulating
application behaviour of hundreds of thousands of
HTTP/HTTPS clients, each with its own source IP-address.
Performance of System under Test (SUT) is tested with
necessary laboratory traffic generated with gateway in the
middle and under various scenarios.
TABLE II
SUT CONFIGURATIONS

System Under Test – Firewall Products

Firewall Checkpoint
OpenBSD
Configurations Cisco ASA (CP)
PF
SPLAT
Cisco ASA -
Platform HP DL 380 HP DL 380
5580
SPLAT 2.4
Operating ASA V 8.2.2 Checkpoint
Free BSD
System ASDM 6.2.5 NGX Fig.4- Throughput and connections comparison.
R65 HFA 50
1) Firewall Licensing: Purpose of licensing is to ensure
Product Multi-processor, authorized use of the product. The license structure of Cisco is
Architecture Multi-core mainly based on per-connection whereas the license structure
Processing of Checkpoint is mainly based on per protected host
8
Cores Compared with both, OpenBSD PF comes with BSD licence
Gigabit which is most cost effective.
Ethernet 0
Interfaces 2) Firewall Management: Criteria for the comparison is Ease
10 Gigabit of firewall management in distributed firewall environment.
Ethernet 4 Checkpoint has very functional GUI Interface and is more
Interfaces user-friendly than Cisco. Checkpoint provides centralized
policy management which can be applied to set of firewalls
B. Performance Testing Results with similar accesses. Whereas Cisco and PF firewalls can be
Some of the major Key Performance Indicators (KPI) given only locally managed.
in Table III are explored in order to compare performance of 3) Application Intelligence: It is the ability of firewall to
three of the most widely used firewall products in market as filter packets based on Application layer Intelligence. Cisco
on today [7-10]. and Checkpoint both uses stateful inspection technologies for
filtering traffic, whereas PF firewall provides limited layer 7 our testing setup, we are able to achieve 5.6 Gbps throughput
intelligence. with HP Hardware Platform.
OpenBSD also mentions that performance of PF firewall
4) HTTP Throughput: It is the maximum offered HTTP load,
varies with hardware platform, performance of system bus,
expressed in either bits per second or packets per second, at
efficiency of network card and type of application [13]. PF is
which no packet loss is detected. The goal of this test is to
a kernel-based process and therefore, it will not use swap
characterize the performance of the SUT when deployed to
spare and will degrade in performance with lesser RAM size.
protect a high performance Web-based application. Cisco
In our laboratory setup, we are able to achieve 4.5 Gbps
outperformed other firewalls in real-world HTTP performance
HTTP throughput performances with HP hardware platform.
tests (See Fig. 4).
However, to the best of author’s knowledge, exact UDP
5) TCP Throughput: It is the maximum offered TCP load, throughput and TCP throughput were not specified in the
expressed in either bits per second or packets per second, at product literature. In testing setup, attempt was made to
which no packet loss is detected. This test will allow better generate different traffic patterns to measure and compare
understanding of how firewall will perform in highly performance. UDP throughput and concurrent connections
transactional environments sch as streaming media. Test supported by PF firewall obtained were impressive and saw
included opening a TCP connection, transferring an object good rise as compared with its HTTP throughput performance.
using HTTP and closing the TCP connection with object size
512 KB. Cisco again outperforms other firewalls with highest IV. SECURITY COMPARISIONS OF VARIOUS FIREWALLS
obtained TCP throughput (See Fig. 4). The testing objective in this study is to observe and
compare the behaviour of each firewall to compare behaviour
6) Concurrent Connection: It is the aggregate number of
shown under adapted and common attack methods. We have
simultaneous connections between hosts across SUT. It gives
broken down attack into two types groups: discovery and
number of connections handled by firewall simultaneously.
penetration. The discovery group establishes or verifies the
This is critical aspect of data centre firewall implementations,
actual location of the target device. The penetration group
as it directly affects the ability to deploy large scale
observes the defensive measures of each firewall [14-16].
applications.
7) UDP Throughput: Though it seems paradoxical, A. Security Testing Setup
connectionless protocols such as UDP may also involve The overall testing setup is developed from the perspective
connections, at least for the purposes of firewall performance of an outside intruder and shown in Fig. 5. Because of this,
measurement. For example, one host may send UDP packets our protected network provided public access to itself as a
to another across a firewall. If the destination host is listening means of establishing a gateway. We placed an FTP server
on the correct UDP port, it receives the UDP packets, inside the network and gave the outside world (Source: Any))
otherwise we will have host unreachable message. For the access to it. This gave the outside intruder a legitimate means
purposes of firewall performance measurement, this is of knowing the IP address of the FTP server.
considered a connection. Test included performance
measurement of UDP throughput using 512KB UDP packets
using QCheck application tool. Cisco and PF stands at par
showing good performance.
8) Connections per Second: It is number of connections
handled per second. It is very important criterion and may
render an otherwise high throughput firewall useless in
production environments. It is also necessary when defending
against denial of service (DoS) attacks.
Fig. 5 Setup diagram for security testing.
C. Deviation from Data Sheets
For the configuration used in testing setup, Cisco mentions The aim of this setup is to simulate a condensed, real-world,
of achieving up to 10 Gbps of real-world HTTP firewall corporate network layout. The access list of all three SUT
throughput performance and 1,50,000 connections per second firewalls permit WWW traffic to pass out on port 80 while at
[11]. The results shown in Fig. 4 agree with the same. In fact, the same time allowing for FTP to pass in on port 21. For
we are able to achieve 10.2 Gbps HTTP throughput with real performing security testing, we used application called
world traffic and around 160, 000 connections per second. It is NetWag. Respective tests are performed by opening NetWag,
slightly better compared to performance reported by Cisco. selecting respective test and entering the target IP Address of
In comparison, Checkpoint mentions of achieving upto 12 our SUT. Firewall configurations taken are similarly used for
Gbps of maximum HTTP throughput performance [12]. performance testing as shown in TABLE II. The brief
However, Checkpoint recommends IBM systems for description of various tests conducted to verify ability of
achieving maximum throughput, and also mentions that firewall to identify common attach types and block intruder
throughput performance may vary depending on hardware. In are listed in Table IV.
 TABLE IV centralized policy management and better user interface than
SECURITY TEST DESCRIPTION
Cisco and PF. Security Testing results indicated that all the
Test Group Test Type Purpose three firewall demonstrated basic intrusion detection
Network Discover target IP address and capability and blocked transmission against common attack
Sniffer protocols used on target network. types. Choosing the right firewall depends on the needs of
Discovery Locate target device and all business and network. Cisco ASA is one of the best choices
Trace route intermediate routers, switches and for large corporate networks. For complex production
systems. networks which demand high protection, Checkpoint SPLAT
Check if firewall can overcome a provides an upper edge over Cisco. PF is one of the best
Synflood repeated open connection request
and also log attack.
inexpensive open source solutions which provides equally
Check if firewall can overcome competent performance as compared with proprietary
Garbage random data packets on random products, but lacks on application layer intelligence.
ports. Author also recommends that the benchmark that counts
Penetration system performance is the network environment. A
Check if firewall can overcome a
UDP Ping
large UDP ping packet sent to it. benchmark that does not replicate exact network environment
TCP Ping
Check if firewall can overcome will not properly help in planning firewall system. The best
large TCP ping packet sent to it. benchmark is to setup network conditions as close as possible
Ping of Check if firewall can overcome a that the actual firewall would experience.
Death single over-sized packet sent to it.
REFERENCES
B. Security Testing Results
[1] M.G. Gouda and A.X. Liu, “A Model of Stateful Firewalls and Its
Regardless of which port the attack has used, with the Properties,” Proc. IEEE Int’l Conf. Dependable Systems and
stateful packet inspection activated, all the three test firewalls Networks (DSN), pp. – 128-137, June 2005.
demonstrated good resistance and blocked all transmissions [2] MyungKeun Yoon, Shigang Chen, and Zhan Zhang, “Minimizing
the Maximum Firewall Rule Set in a Network with Multiple
on every test conducted in this study. All of them also Firewalls” Proc. IEEE Transactions on Computers, Vol. 59,Issue.
continued to allow the proper connections that were not 2,pp. – 218-230, Feb. 2010
considered as attacks during the tests, and effectively blocked [3] Ritchey, R. O'Berry, B. Noel, S., “Representing TCP/IP
the outgoing and incoming packets. OpenBSD PF has all connectivity for topological analysis of network security”, In Proc.
IEEE Computer Security Applications Conference, 2002.
required features which one might expect from an open source Proceedings, pp.-25-31, 2002.
firewall. It has the power and potential of Cisco ASA and [4] PCI data security standard (PCI DSS). Payment Card Industry -
Checkpoint SPLAT with lower cost. It also keeps detailed text Security Standards Council, 2006 www.pcisecuritystandards.org/
log files for each attack, which are fairly easy to read, but it [5] Control objectives for information and related technology (CobiT).
IT Governance Institute (ITGI), 1992-2008 www.cobit.org
does not provide any type of graphical result analysis of the [6] The International Organization for Standards(ISO)
logged attacks which Cisco and Checkpoint provide. www.27000.org/iso-27002.htm
OpenBSD PF also does stateful or stateless packet inspections, [7] D. Newman, Benchmarking Terminology for Firewall
remembers sessions, and modulates the session to assist in Performance, IETF RFC2647, August 1999
[8] M.G. Gouda and A.X. Liu, “Firewall Design: Consistency,
prevention of the data connections from being hijacked. Completeness and Compactness,” Proc. Int’l Conf. Distributed
Computing Systems (ICDCS ’04), pp. 320-327, Mar. 2004.
V. CONCLUSIONS [9] A.X. Liu and M.G. Gouda, “Diverse Firewall Design,” Proc. IEEE
Int’l Conf. Dependable Systems and Networks (DSN ’04), pp. 595-
In the paper, we have attempted to evaluate performance of 604, June 2004.
major operational firewalls in the market today. With [10] H. Hamed, A. El-Atawy, and E. Al-Shaer, “On Dynamic
increasing importance of Network Security in any Optimization of Packet Matching in High Speed Firewalls,” IEEE
organization, we have summarized major regulatory J. Selected Areas in Comm., vol. 24, no. 10, pp. 1817-1830, Oct.
2006
compliance and need to have efficient, clean and robust [11] Cisco Systems Inc., Cisco ASA 5500 Series Security Appliances
firewall configuration and management. To the best of http://www.cisco.com/en/US/products/ps6120/prod_models_comp
author’s knowledge, most currently undertaken and reported arison.html
research work on firewall have been carried out theoretically [12] Check Point Technologies Ltd., NGX R65 Security Platform
http://www.checkpoint.com/press/2007/r65unifiedsecurity.html
and lacks practical implementation. We have attempted to [13] PF: The OpenBSD Packet Filter http://www.openbsd.org/faq/pf/
compare performance of various firewalls based on practical [14] Luis Martin Garcin, “Programming with Libpcap – Sniffing the
implementation. Comparison of various firewalls presented in network from our own application”, Hakin9 Magazine, pp.38-46,
this paper will also help in selecting right vendor at time of February 2008
[15] Bremler-Barr, A. Levy, H, “Spoofing Prevention Method”, in
procurement. proceedings of 24th Annual Joint Conference of IEEE computer
The performance testing results clearly indicated that Cisco and communications societies , Infocom V1, pages 536-547, 2005
ASA provides better performance and Checkpoint SPLAT [16] L. Yuan, J. Mai, Z. Su, H. Chen, C.-N. Chuah, and P. Mohapatra,
provides better functionality. OpenBSD PF also proves to be “FIREMAN: A toolkit for Firewall Modeling and Analysis”, In
Proc. IEEE Symposium on Security and Privacy, pages 199–213,
best open source solution if cost is the deciding factor. 2006.
Checkpoint provides better Firewall Management with