A Stateful CSG-based Distributed Firewall

Architecture for Robust Distributed Security

v. Ramsurrun, and K. M. S. Soyjaudah

Electrical & Electronic Engineering Department
University of Mauritius (UoM)
Reduit, Mauritius
[email protected], [email protected]

Abstract-Distributed firewalls have been developed in order to is provided at the very cluster level, the whole of the network
provide networks with a higher level of protection than will become more secure as we can reduce the occurrence of
traditional firewalling mechanisms like gateway and host-based both insider & external attacks, and limit their spread & effects
firewalls. Although distributed firewalls provide higher security, more readily. A 2-active-node stateful CSG is used for
they too have limitations. This work presents the design & protecting each end-user cluster in our working prototype.
implementation of a new distributed firewall model, based on
In this paper, we perform the following:
stateful Cluster Security Gateway (CSG) architecture, which
addresses those shortcomings. This distributed security model 1. Review of the strengths & limitations of distributed
adopts a bottom-up approach such that each cluster of end-user firewalls.
hosts is first secured using the CSG architecture. These different
2. Use of the stateful CSG to implement a new robust
CSGs are then centrally managed by the Network Administrator.
distributed firewall model.
A file-based firewall update mechanism is used for dynamic real-
time security. IPsec is used to secure the firewall policy update 3. Qualitative comparison of its strengths & weaknesses
distribution while X.509 certificates cater for sender/receiver with other major software-based & hardware-based
authentication. The major benefits of this approach to distributed distributed firewall architectures available.
security include tamper resistance, anti-spoofing, anti-sniffing,
secure real-time firewall updating, low overall network load, high II. DISTRIBUTED FIREWALLS
scalability and low firewall convergence times.
Pioneered by Steven Bellovin [2] in 1999, distributed
Keywords-stateful CSG architecture, distributed firewall, firewalls have been created in response to the limitations of
distributed cluster security, Layer 2 per-packet load balancing both gateway & host-based firewalls, and more specifically, in
order to prevent insider attacks. According to Ioannidis et al.
[3], a distributed firewall is a mechanism that enforces a
I. INTRODUCTION
centralized security policy but the latter is applied at the edges.
Distributed firewalls have been devised with a view to Distributed firewalls are basically centrally managed host-
address the problems of traditional firewalls like gateway and resident security software applications that protect a network's
host-based firewalls. Although distributed firewalls achieve critical endpoints against unwanted intrusions. The conceptual
their purpose, they, too, are not free of shortcomings (Section design of distributed firewalls rests upon three elements:
2). Problems like increase in processing load on end-user hosts
due to the packet filtering strain, decrease in overall network 1. A general policy language that is used for defining
performance because of dynamic firewall updating, and user security policies that are distributed to the firewall
tampering in particular, hamper the deployment and usage of endpoints forming the distributed firewall. Examples of
distributed firewalling solutions. In light of the shortcomings of general policy languages include KeyNote [4] and
distributed firewalls, a new approach to distributed firewalling, Firmato [5].
based on the stateful CSG architecture, has been designed and 2. Network-wide mechanisms for the distribution and
implemented in order to overcome those limitations. The CSG application of the security policy files to the distributed
architecture [1] provides a methodology for grouping together firewall endpoints.
multiple networking elements such as routers, security
gateways, and switches in order to create more secure, more 3. IPsec: security protocol that provides network-level
reliable switched network clusters. The motivation behind the encryption for the secure transmission of the security
CSG-based distributed firewall design is that if robust security policy.

The financial support of the Tertiary Education Commission (TEC) of

Mauritius is gratefully acknowledged.

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.
A. Major Strengths • High reconfiguration time ofhost-resident components
• Centralized management ofdistributedfirewalls

Security policies are formulated centrally and then Since distributed firewalls allow for dynamic updating of
distributed to the different endpoints for enforcement. security policies, the bigger the size of a network, the more
Coherence of security policies over the network and control time it takes to re-deploy security policies. The convergence
over their deployment is enhanced and maintained [3]. time of the network hosts and their firewalls is much higher as
it is directly dependent on the number of hosts found on the
• Defense in depth network.
When used together with the gateway firewall, distributed
firewalls provide multiple layers of defense that an attacker has III. DESIGN OF THE STATEFUL CSG-BASED DISTRffiUTED
to pierce through. This makes the task of the attacker much FIREWALL
more difficult, allows time to other defense mechanisms to The stateful CSG-based distributed firewall is as shown in
counter the threat effectively, and thus, delay & prevent its Fig. 1. This novel distributed firewalling architecture consists
spread in the network [6]. of four main components, namely the Network Administrator
machine, the Cluster Security Manager (CSM), the stateful
B. Major Limitations CSG & the CSG-based gateway firewall, and several sub-
• User tampering systems like the Policy Repository, the Policy Distributor & the
Policy Handler. In our test implementation, each end-user
According to Wei Li [7], this represents the biggest
cluster is protected by a 2-active-node stateful CSG. Each
problem in distributed firewalls. Users requiring administrator
CSG-protected end-user cluster possesses a dedicated CSM,
privileges to work, can modify host-based firewall rules at will
whose main job is to receive firewall updates from the Network
or completely remove the firewall, thereby exposing those
Administrator and forward them to the CSG firewall nodes
hosts to attacks. Hackers can, in tum, use those hosts as base
falling under its responsibility. The workings of these different
for launching attacks from inside the network. Both internal &
components are described in greater detail below:
remote hosts can be attacked.
• Decrease in network performance A. The Network Administrator Machine

The utilization of real-time security policy updates will add This machine is used by the Network Administrator for
considerable strain on the network with all the traffic that is managing the various network components. It is from this
being generated by the distributed firewall. As a result, the computer that the Network Administrator updates CSG firewall
network becomes more vulnerable to DoS attacks [7]. nodes. This machine contains two major components - the
Policy Repository and the Policy Distributor.
• Increase in host load
1) The Policy Repository: The Policy Repository is a
There is degradation in host performance. The host-level central database where all the firewall scripts deployed in the
packet filtering adds considerable load on hosts with limited network are stored. All the firewall updates are also stored
resources. In addition, with the implementation of other there. The Network Administrator can thus consult the
security tools at the host level like real-time host-based existing firewall scripts in order to create new firewall update
intrusion detection systems and Portsentry, as in the security files when the network is under attack. The firewall scripts and
model devised by M. Gangadharan and K. Hwang [6], [8] hosts update files are stored in usable forms (for example, as .sh
will be heavily taxed. files) so that they can be directly applied onto the firewall
nodes. All scripts and updates pertaining to a particular cluster
are stored together for easy referencing. File versioning and
creation details are also kept.
CSG-baltd
Ga~W8Y Firtwall
2) The Policy Distributor: The Policy Distributor is used
by the Network Administrator for sending firewall updates to
Cluster Security Managers (CSMs). The Policy Distributor
establishes end-to-end connections with the appropriate
.__ ... :"
CSMs. These connections are authenticated & encrypted for
"'.CSl\f
/,..---~:------, secure transmission of firewall updates across the network.
a
/ KEY
Erxt-t-host
This helps in preventing threats like man-in-the-middle attack,
_ IPgec-protected2-way
('8M Nttwolk AdmiDisn~~/ commWlication replay attack and IP address spoofing. Firewall updates are
distributed to firewall nodes via CSMs because direct updating
"-"----------------~~
•••• statefulCSllfrewallnode
pallcyupdatlng

of firewall nodes will require secure connections (for instance,

Figure 1. The stateful CSG-based distributed firewall architecture.

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.
using IPsec) between the Policy Distributor and the firewall • Port security [9] so as to prevent source MAC address
nodes themselves. This will significantly increase the spoofing.
processing strain on the firewall nodes as they will then be
1) The Policy Handler: The Policy Handler runs on each
acting as IPsec gateways. IPsec packet processing, together
of the firewall nodes. It receives updates from its CSM and
with Ebtables & IPTables packet filtering, will considerably
integrates them in the current firewall ruleset. Since updates
reduce the efficiency and throughput of the firewall nodes.
are in directly usable format, firewall rules can be inserted or
The firewall update distribution mode is determined by the
deleted easily.
number of secure connections to be established. Unicast
transmission mode is preferred over multicast as not all the D. The Stateful CSG-based Gateway Firewall
CSMs will need updating at a particular point in time. There The gateway firewall, which is the first line of access
may be firewall updates that are meant only for one cluster control & protection against external attacks, needs dynamic
and not for the rest of the clusters in the network. If all the updating as well in the face of emerging threats. In our security
CSMs are made part of a single multicast group, then all of model, a CSG-based gateway firewall is used, where load
them will have to accept the firewall updates. This will cause balancing and failover techniques not only help in eliminating
their respective rulesets to increase in size unnecessarily, and the single point of failure, but also help boost firewall
this will potentially affect firewall performance. throughput and reliability. Like the end-user clusters, the CSG-
B. The Cluster Security Manager (CSM) based gateway firewall, too, has a CSM for receiving firewall
updates from the Network Administrator.
The CSM is the first and foremost recipient of firewall
updates from the Policy Distributor. It is the endpoint of the
secure connections established by the Policy Distributor. Each IV. IMPLEMENTATION DETAILS

end-user cluster has exactly one CSM. The CSM consists of a The high-level architecture of the stateful CSG-based
user-level process that waits for firewall updates from the distributed firewall is as shown:
Policy Distributor and then distributes them to the stateful CSG Adllll1l1:~trator
firewall nodes falling under its responsibility.

C. The Stateful CSG

It comprises of multiple active firewall nodes working in
parallel to filter traffic (intra-cluster, inter-cluster and remote
communication traffic) travelling to/from the end-user hosts of
a particular cluster. The CSG architecture uses a different type
IPSK-protK'trd pelity
of load balancing - the Ebtables distributed sender-initiated updat~ m~s«Iisdub
MAC-based per-packet load balancing (PPLB) scheme, where IPSK-PI'Ot~tttd
flTol'lolS
the load balancing is done by the end-user nodes themselves. FIrewall Node
pol.,- opcla~ tiM
distribndonu.mr;
PPLB helps make optimum usage of network links by allowing unitastTCP
('oon~ctions
Clu~ter Secllnty
for equal distribution of traffic along those links. This Layer 2 FIrewall :Mallager
PPLB scheme has been developed primarily for a seamless flTor 10' SfIlt to
integration in load balancing setups involving stealth firewalls, I...- .........J AdmiDi.~fntor\ia('~·M ~ ---J

especially where IP addressing is not used. It load balances Figure 2. High-level implementation details of the stateful CSG-based
distributed firewall.
network traffic onto MAC addresses rather than IP addresses.
However, it can be successfully utilized in IP-based networks The different software components are implemented as
as well. This scheme is advantageous as it prevents the creation follows:
of single points of failure by removing the need for a dedicated
load balancer, and it integrates well in already-in-place A. The Policy Distributor
switched networks so that no major network re-design is The Policy Distributor (/etc/djw/pol_d. c) is implemented as
required. The CSG architecture deployed for each end-user a user-space program written in C. The program takes as
cluster provides the following security mechanisms: argument the full pathname of firewall update file and the IP
• Layer 2 and Layer 3 packet filtering using Ebtables and addresses of the appropriate CSMs. TCP is used for
IPTables respectively. distributing the firewall updates to the CSMs as it provides
reliable delivery of packets, which is crucial to the delivery of
• Network Access Control (NAC) using MAC ACLs firewall updates. The Policy Distributor uses the
applied on specific switch ports [1] to ensure that end- pd_updatehandlerO function to read the specified firewall
user hosts communicate only via the firewall nodes. update file, and the pd_sendupdateO function sends the update
out to each specified IP address.

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.
B. The Cluster Security Manager (CSM) C. The Stateful CSG
The CSM is made up of two user-space parts - a firewall For our working prototype, a 2-active-node stateful CSG is
update receiving part and a firewall update sending part. Both utilized to secure each end-user cluster as shown in Fig. 3. The
parts make use of TCP sockets. The receiving & sending parts MAC, IP & virtual IP (VIPs) addresses for the above setup are
are implemented in the /etc/dfw/csm.c file. In the receiving part, as shown in Table 1. The components of the stateful CSG
the packets from the Policy Distributor are read by the architecture & their configurations are given below:
csm_updatehandlerO function. The firewall update file is
• End-user node configuration
reconstructed on the CSM so as to ensure that the file is
received error-free and in its entirety. If an error occurs, it is All IP traffic emanating from any end-user node (for
logged and the Network Administrator is notified. In the example, hosts A & B) is forced to pass through the active
sending part, the csm_sendupdateO function is invoked, which firewall hosts M_I and M_2. The traffic is load balanced onto
then sends the firewall update out to each of the CSG firewall M_I and M_2 using the new lbdnat_rr target of Ebtables [12]
nodes via unicast TCP connections. The IP addresses of CSG that we created. lbdnat_rr performs Layer 2 per-packet load
firewall nodes are kept in a file (/etc/dfw/fw_list.txt) that is balancing and makes use of the Round-Robin algorithm in
created by the Network Administrator on the CSM. The order to load balance IP traffic to any number of firewall hosts.
csm_sendupdateO function reads fw_list. txt in order to Software packages like an enhanced version of Ebtables-
determine the IP addresses to which the updates have to be v2.0.8-rc3, and bridge-utils-l.2 [13] are installed. A I-port
sent. Both active and backup firewall nodes are updated. Any bridge is created on each end-user host, with the stp feature
communication errors between the CSM and the firewall nodes turned off. The destination MAC address of all the frames
are logged and the Network Administrator is notified of them. leaving any end-user host is changed to that of the firewall
For example, if ever one or more of the firewall nodes becomes hosts M_l and M_2, that is, M_l_ethO and M_2_ethO, in a
inaccessible to the CSM when csm_sendupdateO tries to send a round robin manner on a per-packet basis. The Ebtables rule
particular firewall update, say because of damaged cable or that is used looks as follows:
NIC failure on the firewall node(s), the connection error
root# ebtables -t nat -A OUTPUT -p ipv4 -0 ethO -j
thereby generated is caught and recorded in a log file, which is
Ibdnat_rr --to-lbdst_rr M_l_ethO,M_2_ethO
eventually sent to the Network Administrator. Multicasting is
• Stateful CSG firewall node configurations
not used in the sending part as UDP does not provide reliability
of packet delivery. Reliability can, nonetheless, be added to a The CSG firewall setup is made up of four PCs, two active
UDP multicast application by incorporating features like master firewall nodes, M_I & M_2, and two standby backup
positive acknowledgements, lost packet retransmission, use of firewall nodes, S_l & S_2. Several software packages are
sequence numbers and packet re-ordering. However, adding installed on the firewall nodes, namely bridge-utils-I.2,
reliability increases the program complexity [10], which is not ebtables-v2.0.8-rc3, iptables-I.3.7, keepalived-I.I.I3-
warranted when it comes to updating such small numbers of 6.fc5.i386.rpm, macchanger-1.5.0 (only on backup nodes),
firewall nodes per cluster. The CSM is deployed in a failover conntrackd-0.9.2 (enhanced code), libnetfilter_conntrack-
configuration such that if it fails, a backup machine will take up 0.0.50 and libnfnetlink. Each firewall node has three NICs
its place. Keepalived [11] is used for this purpose. installed. Bridge-utils is used on each firewall node to create a
bridge device, brO, to which interfaces ethO and ethl are added.
Besides their normal IP addresses, VIP addresses are also

TABLE!. MAC, IP & VIP ADDRESSING FOR THE STATEFUL CSG TEST
SETUP
vrrp
sync Node NIC IP address VIP address
group
- Host A A ethO 192.168.10.2/24 -
- HostB B ethO 192.168.10.3/24 -
Gl S 1 S 1 ethO 192.168.10.100/24 -
S 1 ethl 192.168.0.100/24 -
S 1 eth2 192.168.100.100/24 -
Gl Ml M 1 ethO 192.168.10.110/24 192.168.10.10/24
M 1 ethl 192.168.0.110/24 192.168.0.10/24
M_l_eth2 192.168.100.110/24 -
G2 M2 M 2 ethO 192.168.10.120/24 192.168.10.20/24
M 2 ethl 192.168.0.120/24 192.168.0.20/24
M 2 eth2 192.168.100.120/24 -
G2 S 2 S 2 ethO 192.168.10.130/24 -
Figure 3. The 2-active-node stateful CSG as when applied to one end-user S 2 ethl 192.168.0.130/24 -
cluster. S 2 eth2 192.168.100.130/24 -

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.
assigned to these interfaces but only on the master firewall • Switch Configuration
nodes M I & M 2. Interface eth2 is not enslaved to brO, and
The stateful CSG architecture requires the use of three
hence, is-kept separate from the travel path of end-user traffic.
switches. Switches I, 2 & 3, each connect to one of the NICs of
eth2 is made inaccessible to all nodes except the CSM and the
each firewall node. While switches 2 & 3 can be ordinary
other firewall nodes. It is through this interface that the CSM
unmanageable switches, Switch I has to be a manageable one
nodes update their respective CSG firewall nodes. Ebtables and
with support for MAC ACLs. This feature is used for Network
IPTables provide Layer 2 & Layer 3 packet filtering
Access Control so that traffic from end-user hosts is forced to
respectively. Keepalived [11] caters for high availability in the
pass through the firewall nodes for inspection. To this end, the
event that one of the active master firewall nodes fails. For
switch used as Switch I is a Cisco Catalyst 2970 switch. A
instance, if M_l fails, S_1 automatically takes up the role of
named MAC ACL [18], used for filtering traffic, is applied on
master and becomes active. The VIP addresses on the ethO and
a per-port basis on all switch ports for inbound direction except
eth1 interfaces of M_I are also re-assigned to the
those to which the firewall nodes are connected.
corresponding interfaces on S_I, thus ensuring continuity of
the services offered. The master & backup nodes monitor each Switch (config)# mac access-list extended mac1
other by sending regular multicast advertisements [14]. The Switch (config-ext-mac1)# permit any host M_1_ethO
multicast traffic generated by Keepalived is kept separate from Switch (config-ext-mac1)# permit any host M_2_ethO
that generated by the end-user nodes for security reasons and The above ACL rules are used to permit only frames that
for minimizing the loss of update messages and vrrp have the MAC addresses M_1_ethO or M_2_ethO as destination
advertisements. This multicast traffic is sent via the eth2 MAC address to be forwarded by Switch I. Any other
interface of the firewall nodes and is restricted to a separate destination MAC address will cause the frames to be dropped.
network on Switch 3. GNU Mac Changer [15] helps in
1) The Policy Handler: The Policy Handler
performing MAC address takeover. It is installed only on
(/etc/dfw/fwJJolicy_handler.c) is implemented as a user-space
backup firewall nodes, with appropriate notify_master &
TCP application that runs on the firewall nodes. It waits for
notify_backup scripts defined on them [16]. It is used in
connections from the CSM. The fw_updatehandlerO function
conjunction with Keepalived as the latter allows us to define
reconstructs the firewall update file received from the CSM
scripts that can be called during state transitions (from backup
and applies it to the current firewall ruleset using the systemO
to master and vice-versa). For example, the notify_master
system call. Since the firewall update file contains IPTables
keyword defines a script that runs on the backup node every
rules, it is already in usable form and can thus be applied
time the latter becomes master. Since the Ebtables per-packet
directly. Any error is caught & logged. The Policy Handler
load balancing is a Layer 2 load balancing scheme, network
notifies the Network Administrator of errors via the CSM so
traffic is load balanced onto MAC addresses rather than IP
that any communication between the firewall nodes and the
addresses. Hence, in case of a failover, the backup node has to
Network Administrator machine remains secure.
have the same MAC addresses as the failed master node for
per-packet Layer 2 load balancing to continue to work. This is D. Router Configuration
achieved by changing the MAC addresses of the backup node In order to obtain the full benefit of per-packet firewall load
to that of the failed master node during the state transition from balancing, the load balancing must be supported on both sides
backup to master using a notify_master script defined on the of the firewall nodes. Hence, traffic going towards the end-user
backup node. Also, a modified version of the Conntrackd [17] cluster from the router is also load balanced onto the active
package is used for connection state synchronization on all four stateful CSG firewall nodes. Since the Ebtables Layer 2 PPLB
firewall nodes, which is required for stateful firewalling and scheme cannot be used on the router, the router is configured
PPLB to work together. When a conntrack entry is created in accordingly in order to perform PPLB. A Cisco 2600 series
the internal cache of Conntrackd in one of the master (active) router is used. Static routing, enabling CEF, and using the trip
firewall nodes, updates are sent to the other three firewall load-sharing per-packet" command on the router interface
nodes, both active & standby ones, of that particular CSG via FaO/O help achieve round-robin PPLB. The following
multicast. All four PCs therefore have the same conntrack entry configuration is used to load balance incoming Internet traffic:
created in their respective Conntrackd internal cache.
Consequently, in the case of a failover taking place in one of # Adding static routes
the vrrp synchronization groups, the new master will continue router(config)# ip route 192.168.10.0 255.255.255.0
to forward traffic pertaining to already established connections VIP_M_1_eth1 10
as it already has the necessary connection state information. router(config)# ip route 192.168.10.0 255.255.255.0
The multicast traffic generated by Conntrackd is kept separate VIP_M_2_eth1 10
from that generated by the communicating end-user hosts for # Enabling CEF
the same reasons as that for Keepalived. It is sent via eth2 and router(config)# ip cef
is restricted to Switch 3. # Enabling PPLB on router's Ethernet interface
router(config-if)# ip load-sharing per-packet

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.
 Two equal-cost static routes are installed on the router for generate those certificates. For.testing purposes, a Certificate
the end-user subnet 192.168.1 0.0/24. Each static route uses the Authority (CA) is assumed.
VIP address applied to the eth1 interface of one the active
1) Benchmark results: Two user-land tools, Iperf [20] and
(master) firewall nodes as next-hop IP address. The VIP
Netio [21], are used to provide some throughput estimates of
addresses are used as next-hop IP addresses such that if a
the secure connections between the Network Administrator
master firewall node fails, its VIP will go to the backup firewall
machine and a CSM. Iperf generates TCP connections, which
node and load balancing will continue.
involve requests and replies. Netio, too, generates TCP
connections but sends packets of varying sizes. Two Linux
E. The Stateful CSG-based Gateway Firewall
machines are used in the test setup and they have the
A 4-active-node stateful CSG is used to implement the following configurations:
gateway firewall. The main difference between the CSG used • Development platform - Fedora Core 5 with an
for protecting an end-user cluster and the CSG for the gateway upgraded kernel of2.6.20.4.
firewall is that, in the latter implementation, the four active
firewall nodes are sandwiched between two routers which are • Pentium 4, 1.60 GHz, 256MB RAM
responsible for load balancing network traffic on a per-packet • 10/100 Network Interface Cards
basis onto the firewall. Static routing, enabling CEF, and using
the flip load-sharing per-packet" command on the router The test parameters for Iperf are:
interfaces help achieve per-packet round-robin load balancing. Window size: 214KB
Maximum Segment Size (MSS): the default for Iperf - 1460
F. IPsec The results obtained with Iperf and Netio respectively for
In our design, IPsec is used for securing the distribution of different IPsec transforms are as follows:
firewall updates from the Network Administrator machine to
CSMs, and for securing error reports from CSMs to the TABLE II. PERFORMANCE RESULTS FROM IPERF FOR DIFFERENT
IPSEC TRANSFORMS
Network Administrator machine. The paths between a CSM
and its cluster's CSG firewall nodes are not IPsec-protected as Transform Bandwidth (Mb/s)
w/o IPsec 94.1
they are inherently secure. This is because the firewall nodes 3DES&MD5 46.7
accept firewall updates destined for their local process only DES&MD5 89.8
from the CSM of the cluster to which they belong. This traffic 3DES&SHAI 35.4
is only accepted from the eth2 interface on the firewall nodes.
This interface is not enslaved to the bridge device on the TABLE III. PERFORMANCE RESULTS FROM NETIO FOR DIFFERENT
IPSEC TRANSFORMS
firewall nodes such that it can only be accessed by the CSM.
Each CSM communicates only with Network Administrator Packet Bandwidth w/o Bandwidth with IPsec (KB/s)
size IPsec (KB/s) 3DES& DES & 3DES&
machine and the CSG firewall nodes falling under its
MD5 MD5 SHAI
responsibility. The Network Administrator machine's identity lIrn 11474 5767 10386 4159
is checked via the utilization of digital certificates. Direct IPsec 2KB 11511 5803 10588 4465
connections are not established with the firewall nodes so as 4KB 11512 5835 10918 4466
8KB 11511 5854 10991 4485
not to strain them with IPsec packet processing. This
16KB 11507 5848 10983 4471
processing is handled by the CSM. The latest Linux kernels 32KB 11508 5812 10861 4457
provide native support for IPsec. The IPsec-tools 0.6.7 package
is installed for a 2.6.20.4 Linux kernel in which all the V. THREAT MODEL
necessary kernel IPsec options and cryptographic algorithms in
the CryptoAPI have been selected. Transport mode is used to The effectiveness of the design of the new stateful CSG-
secure the host-to-host connections between the Network based distributed firewall has been assessed qualitatively
Administrator machine and the CSMs as both types of node are against various types of threats originating from both inside
actual participating source/destination pairs - the CSMs receive and outside the network. The way in which the new security
the firewall updates while the Network Administrator machine model counters these threats is described below:
receives error logs. The IPsec-tools package contains the IKE 1. Insider attacks
daemon, racoon, and the setkey utility [19]. racoon is used for
the setting up of automatically keyed IPsec connections while The insider attacks come in two flavors - intra-cluster &
the setkey utility is used for manipulating parameters stored in inter-cluster attacks. If a malicious end-user wants to attack
the Security Association Database (SAD) and Security Policy another machine found on the same cluster (intra-cluster
Database (SPD). X.509 certificates are used for authentication attacks), the traffic emanating from the malicious end-user's
as digital certificates are difficult to forge and represent the machine will be forced to pass through the CSG firewall nodes
most secure method to manage keys. openssl is used to for inspection. Direct communication between end-user hosts is
prevented by MAC ACLs placed on the switch connecting

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.
them. In the case of inter-cluster attacks, the traffic from the handled quite well since IP spoofing is difficult to realize
attacker undergoes packet filtering several times along the way within the CSG architecture as described above. Moreover,
to its destination. The packet filtering is performed once by the egress filtering is performed at the very cluster level in order to
CSG of each of the clusters involved in the communication. throttle these types of attacks as close as possible to the source
Also, since each CSG has cluster-specific firewall rules defined as recommended in RFC 2827 [23].
for both ingress and egress packet filtering in addition to the
4. Packet Sniffing
general network-wide security policy, the fine-grain access
control thus achieved, together with successive packet filtering, Sniffing is one of the main ways attackers use to gather
help limit inter-cluster attacks. Hence, all these mechanisms network-related information. The MAC ACLs on the switch
help contain insider attacks as close as possible to the source. help restrict multicastlbroadcast traffic on a cluster. No end-
user node is allowed to send multicastlbroadcast traffic. This is
2. IP & MAC address spoofing
because the only destination MAC addresses allowed in
An attacker may also try to impersonate different nodes in outgoing packets from end-user nodes are those of the firewall
the cluster like end-user nodes, firewall nodes or CSMs, using nodes. Hence, there is significantly less traffic on the switch
IP/MAC address spoofing. This is prevented by several that attackers can sniff. Moreover, putting the NIC of an end-
security mechanisms that have been put in place. Port security, user node in promiscuous mode will not allow sniffing of
one of the in-built security mechanisms provided by the 2970 unicast traffic of other end-user nodes as the switch makes use
series Catalyst switch, is used to prevent a malicious end-user of virtual circuits. Thus, that traffic is restricted only to the two
node from spoofing the source MAC address of its outgoing communicating nodes.
packets. Port security can be set on each individual switch
5. Rule tampering
interface (port). Both static and dynamic secure MAC
addresses can be configured. Also, the first address Rule tampering by malicious end-users is prevented as the
dynamically learned by the switch port can be converted into filtering rules are not found on the end-user nodes, but rather
the secure address for that port by enabling "sticky learning". on dedicated CSG firewall nodes. Hence, even if an end-user
Sticky secure MAC addresses can then be saved in the has root access on his machine, he will still have to comply
configuration file of the switch so that the switch interface does with the security policy defined for the network if he wants to
not need to dynamically reconfigure them when the switch send traffic. The only rules found on an end-user machine are
restarts. The number of MAC addresses that can access a Ebtables rules that are used for load balancing outgoing traffic.
particular port can be limited to one. Hence, if a malicious end- Hence, the maximum an un-cooperating "insider" can do is to
user tries to use a source MAC address other than the change the load balancing rule.
authorized one, a security violation will occur and a syslog
message will be logged. The advantage of port security is that it VI. RELATED WORK
discards packets with spoofed source MAC addresses such that
Several distributed firewall implementations have been
the firewall nodes do not have to waste CPU cycles in
developed over the years. These implementations fall in two
processing them.
broad categories, namely software-based and hardware-based.
In order to guard against IP address spoofing, Ebtables Some of the major software-based distributed firewalls are as
rules have been formulated and installed on the CSG firewall follows:
nodes. The --among-src match of Ebtables allows several
The concept of distributed firewall, as expounded by
MAC/IP source address pairs to be defined, and packet headers
Bellovin, was implemented by Ioannidis et al. [3]. That
are checked against these defined pairs. Spoofed packets from
implementation later evolved into the Strongman distributed
end-user nodes can thus be identified and discarded.
firewall Architecture [24]. KeyNote was used as an
ebtables -A FORWARD -p IPV4 --among-src intermediate common language for translating different high-
00:01:02:03:04:05=192.168.10.2,11:12:13:14:15:16=19 level policy languages into KeyNote policies/credentials, and
2.168.10.3,21:22:23:24:25:26=192.168.10.4 -j ACCEPT helped manage access control in large heterogeneous networks
3. Denial ofService attacks [25] that made use of diverse security mechanisms. IPsec was
used for userlhost authenticator, secure distribution of
DoS attacks generally come under two main categories -
credentials and firewall traffic protection.
bandwidth depletion attacks and resource depletion attacks.
These two types involve consumption of resources like Smokey [26] is a user-based distributed firewall system that
bandwidth and CPU cycles respectively [22]. The use of load was responsible for putting in place a centralized security
balancing techniques in the CSG spreads the packet filtering policy on member nodes of a distributed system. The policy
strain over multiple firewall nodes and prevents the latter nodes was distributed on a per-user basis and provided as much
from quickly becoming chokepoints. There are a variety of access as was required. A local policy manager retrieved user-
DoS attacks and not all can be handled by distributed firewalls. specific security policies from a policy server. It then passed
DoS attacks, which rely on IP spoofing mechanisms, can be them to the local policy handler, which deployed them on the

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.
host. The security policies consisted of directly usable Network Access Control is provided by switch MAC ACLs
IPChains rules. and port security. The Strongman, Smokey and Micro-firewall
architectures too provide robust packet filtering and access
Gangadharan and Hwang [6], [8] made use of a distributed
control mechanisms. The hardware-based models, however, do
micro-firewall approach in order to protect nodes of a Linux
not score high in this area as they have limited packet filtering
cluster. The micro-firewall module was made up of three
capability due to limited processing power and memory on the
components - a packet filter (IPChains), an anomaly detector
NIC. The NIC can be easily overloaded by network traffic even
(LIDS) & access logging facility (LogCheck), and it was built
when small firewall rulesets are used [27].
on each node. Each cluster formed a policy domain managed
by a policy manager. The individual anomaly detectors • Firewall tamper resistance
communicated with the policy manager using mobile agents,
Since the firewall rulesets are not found on the end-user
thus forming a DIDS (distributed intrusion detection system).
hosts but rather on dedicated CSG firewall nodes, malicious
When an intrusion was detected by the anomaly detector of the
end-users cannot change or delete packet filtering rules. They
micro-firewall on an end-user host, it was reported to the policy
can only comply with the security policy defined. Only the
manager, which generated a policy update that was sent to all
CSG-based and hardware-based distributed firewall models
cluster nodes using Java-based RMI. The policy manager also
exhibit this characteristic. This is not the case for the other
notified other policy managers so that the whole network got
software-based distributed firewall models, where malicious
updated to prevent the spread of the threat.
end-users working with root privileges on end-user hosts can
As for hardware-based distributed firewalls, the known modify/delete firewall rules.
ones include the Distributed Embedded Firewall (EFW), the
Autonomic Distributed Firewall (ADF) and the Network Edge
• High scalability
Security Distributed Firewall. The Ebtables Layer 2 load balancing scheme allows
additional end-user hosts to be added very easily to the CSG
The EFW [27]-[29] is a NIC-based host-aS-independent
architecture. Firewall nodes, too, can be added easily. This
firewall that filtered IP traffic to/from a particular host. It was
helps in preventing the overloading of the firewall nodes as the
managed by a central policy server and all policy server/EFW-
end-user cluster grows in size. All the other models also
enhanced NIC communication was authenticated by 3DES.
provide high scalability.
Each policy server managed exactly one policy domain, within
which EFW NICs were grouped into device sets by virtue of • Anti-spoofing
their function. That helped ease manageability and policy
In the CSG-based model, packet spoofing by end-users is
deployment. The EFW NIC had a stateless packet filtering
prevented by using features like port security on switches, and
engine provided in the EFW-enhanced firmware of the NIC,
the --among-src match of Ebtables on the firewall nodes. Any
which enforced the policy rules. The 3Com 3CR990 family of
packet with spoofed MAC or IP addresses is dropped at the
NICs was used for the EFW as these NICs have on-board
very cluster level. In the hardware-based models, anti-spoofing
processor, memory & cryptographic engine, which allowed the
is achieved through the inaccessibility of the NIC to end-users.
EFW to operate independently of the host as.
All the other models use some kind of user authentication
The ADF [27], [30] implementation finds its origins in the mechanism. For instance, cryptographic certificates are used in
same codebase as the EFW. EFW was created first and was the Strongman and the Micro-firewall models, while Smokey
commercialized by 3Com. ADF was later developed by uses tickets, uids and secure authenticated channels in order to
Adventium Labs, with added capabilities like Virtual Private prevent user identity spoofing and the spoofing of entities like
Groups (VPGs). Policy Server and Policy Manager.
The Network Edge Security distributed firewall [31], as • Anti-sniffing
well, was derived from the same design as EFW & ADF.
In the CSG-based model, switch MAC ACLs make it
impossible for end-user hosts to send multicastlbroadcast
A. System Evaluation
traffic. Moreover, end-users cannot sniff unicast traffic
The stateful CSG-based distributed firewall model has been between two communicating hosts as switches make use of
compared to the above-mentioned software-based and virtual circuits. In the case of the hardware-based distributed
hardware-based distributed firewalls in order to determine the firewall models, end-users are unable to put the end-user host
various desirable characteristics that it possesses and to see NIC in promiscuous mode in order to sniff multicastlbroadcast
how well it fares with respect to the others: traffic because the NIC operates independently of the host as.
• Fine-grained security As for the software-based implementations, sniffing is possible
by malicious end-users working with root privileges on the
The CSG firewall nodes provide both robust Layer 2/3 host, and if the network in which they are deployed does not
packet filtering using Ebtables and IPTables respectively. limit multicastlbroadcast traffic.

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.
 • Low overall network load and increases with the number of protected end-user hosts
involved.
In our distributed firewall model, the number of secure
(IPsec) connections to be made is very low as compared to • Low end-user host processing strain
those in the other distributed firewalling schemes which have
In our model, the packet filtering strain is removed from the
to establish secure connections with every end-user host
end-user hosts. Only two Ebtables rules are placed on each
present on the network. In those models, the strain on the
end-user host for load balancing purposes. This characteristic is
overall network increases with the number of protected end-
supported by the hardware-based models as well since the
user nodes present, especially during policy update broadcasts.
packet filtering process is carried out independently by the
This can be deduced easily from the bandwidth estimates
NIC's on-board processor. On the other hand, the end-user
obtained for one IPsec connection in Tables II & III. In our
nodes in the other software-based models are heavily taxed by
model, secure connections are made only with CSMs, which
the packet filtering process, which is done directly on the end-
amount to one per cluster. Since fewer IPsec connections and
user hosts themselves.
fewer copies of firewall updates are needed, the overall
network load is much lower than with the other schemes. • Context knowledge
• Secure & rapid real-time updating Context knowledge helps achieve even more robust traffic
filtering, especially at OSI Application Layer level. The
In the CSG-based model, upon detection of a threat, the
firewall nodes forming part of the CSG-based distributed
Network Administrator can rapidly create new firewall rules so
firewall model do not possess end-user host context
as to counter it. These new rules are sent as firewall updates to
knowledge. This is because the packet filtering process is done
the affected clusters or to all clusters in the network if required.
independently of the end-user hosts, on dedicated firewall
All the models, except Smokey, support this feature. However,
nodes. This also holds true for all the hardware-based models
the CSG-based model goes one step further than all the other
as the NIC operates independent of the host's operating system.
models in the sense that end-user nodes that were previously
Only the Strongman, Smokey and Micro-firewall models are
offline or that have freshly been added to the cluster do not
able to use context knowledge during the packet filtering
need to fetch firewall updates from servers as they are instantly
process as the latter is done directly on the end-user hosts
protected by the CSG firewall nodes, which are online and
themselves.
updated all the time. Moreover, increase in the number of
protected end-user hosts in the CSG-based model does not The results of this comparison are summarized in Table IV.
affect the speed of the updating process, as is the case in the
other models. This is because the number of firewall nodes VII. CONCLUSION
remains the same.
This paper highlights a new approach to distributed cluster
• Low convergence time and network security - the stateful CSG-based distributed
firewall architecture. This distributed security model
Since we have few IPsec connections to establish and a
successfully addresses the limitations plaguing distributed
small number of CSG firewall nodes per cluster, the
firewalls and adds a few more desirable characteristics of its
convergence time of the firewalls across the network is much
own.
less compared to updating every single end-user host in the
network as may be required by the other models. This becomes Future work will look at the creation of load balancing
even more evident when we have to deploy network-wide network interface cards (LB-NICs). In order to prevent the
security policies. In the other models, the convergence time of tampering of the Ebtables load balancing rules that are found
the network with regard to the re-synchronization of all the on the end-user hosts by malicious users, the Ebtables MAC-
host-resident parts of the distributed firewall tends to be high based load balancing scheme can be incorporated into a

TABLE IV. COMPARISON OF THE STATEFUL CSG-BASED DISTRIBUTED FIREWALL WITH OTHER MAJOR DISTRIBUTED FIREWALL SCHEMES

Characteristics Strongman Smokey M-F EFW ADF NES Stateful CSG-based

Fine-grained security ./ ./ ./ x x x ./
Firewall tamper resistance x x x ./ ./ ./ ./
High scalability ./ ./ ./ ./ ./ ./ ./
Anti-spoofing ./ ./ ./ ./ ./ ./ ./
Anti-sniffing x x x ./ ./ ./ ./
Low overall network load x x x x x x ./
Secure real-time updating ./ x ./ ./ ./ ./ ./
Low convergence time x x x x x x ./
Low end-user host processing strain x x x ./ ./ ./ ./
Context knowledge ./ ./ ./ x x x x
KEY: M-F = Micro-frrewalls ; EFW = Distributed Embedded Firewall; ADF = Autonomic Distributed Firewall; NES = Network Edge Security

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.
tamper-resistant network interface card with on-board [7] W. Li (2000). Distributed Firewall [Online]. Available:
http://citeseer.ist.psu.edu/liOOdistributed.html
processing engines. This approach adopts a similar line of [8] M. Gangadharan, and K. Hwang, "Intranet security with micro-firewalls
thought as that used in the implementation of hardware-based and mobile agents for proactive intrusion response," in IEEE Int. Con!
distributed firewalls like EFW [28], [29] and ADF [30]. These Computer Networks and Mobile Computing, 2001.
[9] W. Odom, "CCENT/CCNA ICNDI Official Exam Certification Guide,"
load balancing cards will have to register with a central policy 2nd ed. Indianapolis, USA: Cisco Press, 2008, pp. 253-256.
server first in order to be able to function. The central policy [10] W. R. Stevens, B. Fenner, and A. M. Rudoff, Unix Network
server will perform LB-NIC group management, where each Programming: The Sockets Networking API Volume 1, 3rd ed., Addison-
LB-NIC group will consist of all the end-user node NICs Wesley, 2004, pp. 595-598.
[11] Keepalived website (2007). Keepalived for Linux - Linux High
forming part of a particular end-user cluster. Thus, each LB- Availability [Online]. Available: http://www.keepalived.org/index.html
NIC group will have an appropriate load balancing policy [12] Ebtables website (2007). Who's behind Ebtables? [Online]. Available:
created for it. The load balancing policy will simply consist of http://ebtables.sourceforge.net/about.html#behind
[13] Bridge website (2007). Bridge [Online]. Available: http://linux-
the Ebtables load balancing rules that will indicate the firewall net.osdl.orgjindex.php/Bridge
nodes to which traffic from end-user nodes of a particular [14] P. Hollenback (2005). Improving Network reliability with Keepalived
cluster will be load balanced to. The potential advantages of [Online]. Available: http://www.linuxdevcenter.com/lpt/a/6162
[15] L. Ortega (2007). GNU Mac Changer [Online]. Available:
this scheme include: http://www.alobbs.com/macchanger/
[16] ClarkConnect website (2007). Howtos - Clustering with LVS [Online].
1. Load Balancing rule tamper resistance Available: http://www.clarkconnect.comlwiki/index.php?title=Howtos_-
_ Clusterin~with_LVS
Tamper resistance of the load balancing rules will be
[17] P. N. Ayuso (2007). conntrack-tools: Connection tracking userspace
provided as the LB-NIC will be protected from direct tools for Linux [Online]. Available:
manipulation. This can be ensured in the following manner: http://people.netfilter.orglpablo/conntrackd/
[18] Cisco Systems Inc. (2005). Configuring Network Security with ACLs
• Only the policy server can add/delete/update the load [Online]. Available:
balancing policy on the LB-NIC. http://www.cisco.com/en/US/products/hw/switches/ps628/products conf
iguration guide chapter09186a00800d84c8.html
• Only the policy server can disable the LB-NIC. [19] R. Spenneberg (2007). IPsec HOWTO (Revision 0.9.96) [Online].
Available: http://www.ipsec-howto.orgj
• All policy server/LB-NIC communication is [20] M. Gates, A. Tirumala, 1. Dugan, and K. Gibbs (2003). Iperf version
1.7.0: Iperf User Docs [Online]. Available:
authenticated and secured by means of cryptography. http://www.itl.ohiou.edu/iperf-doc/
[21] K. U. Rommel (2005). NETIO - Network Benchmark, Version 1.26
2. Easy addition/removal offirewall nodes in the CSG [Online]. Available: http://www.ars.de/ars/ars.nsf/docs/netio
[22] S. M. Specht, and R. B. Lee, "Distributed Denial of Service:
The centralized management of LB-NICs will also allow Taxonomies of Attacks, Tools, and Countermeasures," in 2004 Proc.
new firewall nodes to be added/removed in the CSG very 17th Int. Con! Parallel and Distributed Computing Systems, pp. 543-
easily. Upon such changes, the policy server can update the 550.
[23] P. Ferguson, and D. Senie, ''Network Ingress Filtering: Defeating Denial
load balancing policy on the LB-NICs by multicasting new
of Service Attacks which employ IP Source Address Spoofing," RFC
load balancing rules to the appropriate LB-NIC groups. 2827,2000.
[24] A. D. Keromytis, S. loannidis, M. B. Greenwald, and 1. M. Smith, "The
3. No needfor a dedicated load balancer STRONGMAN Architecture," in Proc. DARPA Information
Survivability Conference and Exposition, vol. 1, 2003.
Since load balancing is performed by the LB-NICs [25] A. D. Keromytis, K. G. Anagnostakis, S. Ioannidis, M. B. Greenwald,
themselves, there is no need for a dedicated software-based or and 1. M. Smith, "Managing Access Control in Large-Scale
hardware-based load balancer. Also, network latency decreases Heterogeneous Networks," in Proc. NATO C3 Symposium Interoperable
Networks for Secure Communications (INSC'03), 2003.
as the number of intermediate components decreases. [26] R. Rubin (2002). Smokey: A User-Based Distributed Firewall System
[Online]. Available: http://www.cs.berkeley.edu/-daw/teachingjcs261-
REFERENCES fU2/reports/rubin. pdf
[27] M. Ihde, and W. H. Sanders, "Barbarians in the Gate: An Experimental
[1] V. Ramsurrun, and K. M. S. Soyjaudah, "Efficient cluster security Validation of NIC-based Distributed Firewall Performance and Flood
gateway architecture for per-packet load balanced IP filtering on Tolerance," in 2006 Proc. Int. Conj. Dependable Systems and Networks
switched clusters," in 2006 Proc. CSNDSP'06 Conj., pp. 256-261. (DSN 2006), pp. 209-216.
[2] S. M. Bellovin, "Distributed firewalls," ;login: magazine, special issue [28] T. Markham, L. Meredith, and C. Payne, "Distributed embedded
on security, 1999. firewalls with virtual private groups," in 2003 Proc. 3,d DARPA
[3] S. loannidis, A. D. Keromytis, S. M. Bellovin, and 1. M. Smith, Information Survivability Conj., vol. 2, pp. 81-83.
"Implementing a distributed firewall," in Proc. 7th ACM Conj. [29] C. Payne, and T. Markham, "Architecture and applications for a
Computer and communications security, Athens, 2000, pp. 190-199. distributed embedded firewall," in Proc. 17th Annual Computer Security
[4] M. Blaze, J. Feigenbaum, J. loannidis, and A. Keromytis, "The KeyNote Applications Conference (ACSAC 2001), 200 1, pp. 329-336.
Trust-Management System Version 2," RFC (Informational) 2704, [30] L. M. Meredith, "A summary of the autonomic distributed firewalls
Internet Engineering Task Force, 1999. (ADF) project," in 2003 Proc. 3,d DARPA Information Survivability
[5] Y. Bartal, A. Mayer, K. Nissim, and A. Wool, "Firmato: A Novel Conj., vol. 2, pp. 260-265.
Firewall Management Toolkit," in Proc. IEEE Symposium Security and [31] T. Markham, and C. Payne, "Security at the network edge: A distributed
Privacy, 1999, pp. 17-31. firewall architecture," in 2001 Proc. 2nd DARPA Information
[6] M. Gangadharan, and K. Hwang, "Micro-firewalls for dynamic network Survivability Conference and Exposition (DISCEX II).
security with distributed intrusion detection," in IEEE Int. Symposium
Network Computing and Applications (NCA'01), 2001.

Authorized licensed use limited to: Middlesex University. Downloaded on October 08,2023 at 10:19:46 UTC from IEEE Xplore. Restrictions apply.