26 September 2018

CLUSTERXL

R80.20

Administration Guide
Classification: [Protected]
© 2018 Check Point Software Technologies Ltd.
All rights reserved. This product and related documentation are protected by copyright and
distributed under licensing restricting their use, copying, distribution, and decompilation. No part
of this product or related documentation may be reproduced in any form or by any means without
prior written authorization of Check Point. While every precaution has been taken in the
preparation of this book, Check Point assumes no responsibility for errors or omissions. This
publication and features described herein are subject to change without notice.
RESTRICTED RIGHTS LEGEND:
Use, duplication, or disclosure by the government is subject to restrictions as set forth in
subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS
252.227-7013 and FAR 52.227-19.
TRADEMARKS:
Refer to the Copyright page http://www.checkpoint.com/copyright.html for a list of our
trademarks.
Refer to the Third Party copyright notices http://www.checkpoint.com/3rd_party_copyright.html
for a list of relevant copyrights and third-party licenses.
Important Information
Latest Software
We recommend that you install the most recent software release to stay up-to-date
with the latest functional improvements, stability fixes, security enhancements and
protection against new and evolving attacks.

Certifications
For third party independent certification of Check Point products, see the Check Point
Certifications page
https://www.checkpoint.com/products-solutions/certified-check-point-solutions/.

Check Point R80.20

For more about this release, see the R80.20 home page
http://supportcontent.checkpoint.com/solutions?id=sk122485.

Latest Version of this Document

Open the latest version of this document in a Web browser
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_
ClusterXL_AdminGuide/html_frameset.htm.
Download the latest version of this document in PDF format
http://supportcontent.checkpoint.com/documentation_download?ID=60445.
To learn more, visit the Check Point Support Center
http://supportcenter.checkpoint.com.

Feedback
Check Point is engaged in a continuous effort to improve its documentation.
Please help us by sending your comments
mailto:[email protected]?subject=Feedback on ClusterXL R80.20
Administration Guide.

Revision History
Date Description
26 September 2018 First release of this document
Contents
Important Information................................................................................................... 3
Terms ............................................................................................................................ 8
Introduction to ClusterXL ............................................................................................ 18
The Need for Clusters ............................................................................................. 18
ClusterXL Solution .................................................................................................. 18
How ClusterXL Works ............................................................................................. 19
The Cluster Control Protocol .........................................................................................19
ClusterXL Requirements and Compatibility ................................................................ 20
Check Point Appliances and Open Servers ............................................................. 20
Supported Number of Cluster Members ................................................................. 20
Hardware Requirements for Cluster Members ...................................................... 20
Software Requirements for Cluster Members ........................................................ 21
Hardware Requirements for Switches and Routers ............................................... 21
High Availability and Load Sharing Unicast Modes ........................................................21
Load Sharing Multicast Mode ........................................................................................22
VMAC Mode....................................................................................................................23
Supported Topologies for Synchronization Network .............................................. 24
Clock Synchronization in ClusterXL ........................................................................ 26
IPv6 Support for ClusterXL ..................................................................................... 26
IPv6 in ClusterXL High Availability .................................................................................26
Synchronized Cluster Restrictions .......................................................................... 27
Check Point Software Compatibility ........................................................................ 28
ClusterXL Compatibility with IPS ...................................................................................28
ClusterXL Compatibility (Excluding IPS) ........................................................................28
Example Configuration of a Cisco Catalyst Routing Switch ..................................... 29
High Availability and Load Sharing in ClusterXL ......................................................... 31
Introduction to High Availability and Load Sharing ................................................. 31
High Availability .............................................................................................................31
Load Sharing .................................................................................................................32
Example ClusterXL Topology .................................................................................. 33
Defining the Cluster Member IP Addresses ...................................................................34
Defining the Cluster Virtual IP Addresses .....................................................................34
Defining the Synchronization Network ..........................................................................34
Configuring Cluster Addresses on Different Subnets ....................................................34
Implementing Planning Considerations .................................................................. 36
High Availability or Load Sharing...................................................................................36
Choosing the Load Sharing Mode ..................................................................................36
IP Address Migration .....................................................................................................36
ClusterXL Modes ..................................................................................................... 37
High Availability Mode ...................................................................................................37
Load Sharing Multicast Mode ........................................................................................39
Load Sharing Unicast Mode ...........................................................................................40
Mode Comparison Table ................................................................................................41
Cluster Failover ...................................................................................................... 43
When Does a Failover Occur? ........................................................................................43
What Happens When a Cluster Member Recovers? .......................................................44
How a Recovered Cluster Member Obtains the Security Policy .....................................44
Synchronizing Connections in the Cluster .................................................................. 45
The Check Point State Synchronization Solution .................................................... 45
The Synchronization Network ........................................................................................45
How State Synchronization Works .................................................................................46
Configuring Services to Synchronize After a Delay ........................................................46
Configuring Services not to Synchronize .......................................................................47
Sticky Connections ........................................................................................................49
Non-Sticky Connections ................................................................................................52
Synchronizing Clusters on a Wide Area Network...........................................................53
Synchronized Cluster Restrictions ................................................................................54
Configuring ClusterXL................................................................................................. 55
Installing Cluster Members .................................................................................... 55
Configuring Routing for Client Computers .............................................................. 56
Selecting the CCP Transport Mode on the Cluster Members .................................. 57
Configuring the Cluster Object and Members ......................................................... 59
Overview ........................................................................................................................59
Using the Wizard Mode ..................................................................................................59
Using the Manual Configuration ....................................................................................62
Configuring a ClusterXL in Bridge Mode ................................................................. 67
Configuring ClusterXL in Bridge Mode - Active/Standby with Two Switches..................67
Configuring ClusterXL in Bridge Mode - Active/Active with Two Switches .....................80
Configuring a ClusterXL in Bridge Mode - Active/Active with Four Switches .................93
Advanced Features and Procedures ........................................................................... 96
Working with VPNs and Clusters ............................................................................ 96
Configuring VPN and Clusters .......................................................................................96
Defining VPN Peer Clusters with Separate Security Management Servers ...................97
Working with NAT and Clusters .............................................................................. 98
Cluster Fold and Cluster Hide .......................................................................................98
Configuring NAT on the Cluster .....................................................................................98
Configuring NAT on a Cluster Member ..........................................................................98
Working with VLANS and Clusters ........................................................................ 100
VLAN Support in ClusterXL..........................................................................................100
Connecting Several Clusters on the Same VLAN .........................................................101
Monitoring the Interface Link State ...................................................................... 106
Enabling Interface Link State Monitoring ....................................................................106
Bonding and Clusters ............................................................................................ 107
Bond Interfaces (Link Aggregation) .............................................................................108
Bonding - High Availability Mode .................................................................................110
Bonding - Load Sharing Mode......................................................................................116
Group of Bonds ............................................................................................................118
Performance Guidelines for Bonding ..........................................................................124
Troubleshooting Issues with Bonded Interfaces ..........................................................125
Advanced Cluster Configuration ........................................................................... 127
How to Configure Reboot Survival ...............................................................................127
Controlling the Clustering and Synchronization Timers .............................................. 127
Blocking New Connections Under Load .......................................................................128
Reducing the Number of Pending Packets...................................................................129
Defining Non-Monitored Interfaces ...................................................................... 130
Configuring Policy Update Timeout ....................................................................... 131
Enhanced 3-Way TCP Handshake Enforcement .................................................... 132
Cluster IP Addresses on Different Subnets .......................................................... 133
Introduction .................................................................................................................133
 Configuring Cluster Addresses on Different Subnets ..................................................133
Example of Cluster IP Addresses on Different Subnets ...............................................135
Limitations of Cluster Addresses on Different Subnets ...............................................136
Converting a Security Gateway to a ClusterXL ...................................................... 138
Converting a Standalone Deployment to ClusterXL .....................................................138
Creating the New Member ...........................................................................................140
Creating the ClusterXL Object .....................................................................................140
In SmartConsole, for Computer 'B' ..............................................................................140
Preparing Computer 'A' ...............................................................................................141
In SmartConsole, for Computer 'A' ..............................................................................141
Adding Another Member to an Existing Cluster .................................................... 142
Removing a Member from an Existing Cluster ..................................................... 144
Configuring ISP Redundancy on a Cluster ............................................................ 145
Configuring the ISP Links ............................................................................................146
Configuring Security Gateway as DNS .........................................................................147
Configuring the Firewall ..............................................................................................148
Configuring with VPN ..................................................................................................149
Force ISP Link State ....................................................................................................149
Editing the ISP Redundancy Script...............................................................................150
Enabling Dynamic Routing Protocols in a Cluster Deployment ............................ 151
Components of the System ..........................................................................................151
Wait for Clustering ......................................................................................................152
ClusterXL Configuration Commands......................................................................... 153
Configuring the Cluster Member ID Mode in Local Logs....................................... 157
Registering a Critical Device ................................................................................. 158
Unregistering a Critical Device ............................................................................. 159
Reporting the State of a Critical Device ................................................................ 160
Registering Critical Devices Listed in a File .......................................................... 161
Unregistering All Critical Devices ......................................................................... 162
Configuring the Cluster Control Protocol (CCP) Mode .......................................... 163
Initiating Manual Cluster Failover......................................................................... 164
Monitoring and Troubleshooting Clusters ................................................................ 166
ClusterXL Monitoring Commands ......................................................................... 167
Monitoring Cluster State .............................................................................................171
Monitoring Critical Devices..........................................................................................176
Monitoring Cluster Interfaces......................................................................................182
Monitoring Bond Interfaces .........................................................................................187
Monitoring Cluster Failover Statistics .........................................................................191
Monitoring MAC Magic and MAC Forward Magic Values .............................................. 192
Monitoring Delta Synchronization................................................................................193
Viewing IGMP Status ....................................................................................................202
Viewing Cluster Delta Sync Statistics for Connections Table ....................................... 203
Viewing Cluster IP Addresses ......................................................................................204
Viewing the Cluster Member ID Mode in Local Logs ....................................................205
Viewing Interfaces Monitored by RouteD .....................................................................206
Viewing Roles of RouteD Daemon on Cluster Members...............................................207
Viewing Cluster Correction Statistics ..........................................................................208
Monitoring Cluster Status Using SmartConsole Clients ....................................... 210
SmartConsole Logs View .............................................................................................210
Working with SNMP Traps .................................................................................... 213
cphastart and cphastop ......................................................................................... 214
How to Initiate Cluster Failover ............................................................................ 215
 Troubleshooting Dynamic Routing (routed) Pnote ................................................ 217
Standard RouteD Pnote Behavior ................................................................................217
Basic Troubleshooting Steps .......................................................................................217
ClusterXL Error Messages .................................................................................... 219
Appendix A - The clusterXL_admin Script ................................................................ 220
Appendix B - The clusterXL_monitor_ips Script ....................................................... 222
Appendix C - The clusterXL_monitor_process Script ............................................... 224
 Member_A will assume the Standby
state).
Terms Active(!)
In ClusterXL, state of the Active cluster
3rd party Cluster member that suffers from a failure. A
Cluster of Check Point Security Gateways problem was detected, but the cluster
that work together in a redundant member still forwards packets, because it is
configuration. These Check Point Security the only member in the cluster, or because
Gateways are installed on X-Series XOS, or there are no other Active members in the
IPSO OS. VRRP Cluster on Gaia OS is also cluster. In any other situation, the state of
considered a 3rd party cluster. The 3rd party the member is Down.
cluster handles the traffic, and Check Point • ACTIVE(!) - See above.
Security Gateways perform only State
Synchronization. • ACTIVE(!F) - See above. Cluster
member is in the freeze state.
Active • ACTIVE(!P) - See above. This is the
State of a cluster member that is fully Pivot cluster member in Load Sharing
operational: Unicast mode.
• In ClusterXL, this applies to the state of • ACTIVE(!FP) - See above. This is the
the Security Gateway component Pivot cluster member in Load Sharing
• In 3rd party / OPSEC cluster, this applies Unicast mode and it is in the freeze state.
to the state of the cluster State Active/Active Mode
Synchronization mechanism
See Load Sharing Mode.
Active Member
Active/Standby Mode
A cluster member that handles network
connections that pass through the cluster. See High Availability Mode.
• In a High Availability deployment, only ARP Forwarding
one cluster member is Active and can
handle connections. See sk111956
http://supportcontent.checkpoint.com/soluti
• In a Load Sharing deployment, all cluster ons?id=sk111956.
members are Active and can handle
connections. Backup
Active Up • In VRRP on Gaia cluster:
ClusterXL in High Availability mode that was State of a cluster member that is ready to
configured as Maintain current active be promoted to Master state (if Master
Cluster Member in the cluster object in member fails).
SmartConsole: • In VSX cluster configured in Virtual
• If the current Active member fails for System Load Sharing mode with three or
some reason, or is rebooted (for example, more cluster members:
Member_A), then failover occurs between State of a Virtual System on a third (and
cluster members - another Standby so on) VSX cluster member.
member will be promoted to be Active
A cluster member or Virtual System in this
(for example, Member_B).
state does not process any traffic passing
• When former Active member (Member_A) through cluster.
recovers from a failure, or boots, the
former Standby member (Member_B) will
remain to be in Active state (and
Blocking Mode The CCL provides connections stickiness by
Cluster operation mode, in which cluster "correcting" the packets to the correct
member does not forward any traffic (for cluster member:
example, caused by a failure). • In most cases, the CCL makes the
correction from the CoreXL SND.
Bond
• In some cases (like Dynamic Routing, or
A virtual interface that contains (enslaves) VPN), the CCL makes the correction from
two or more physical interfaces for the Firewall or SecureXL.
redundancy and load sharing. The physical
interfaces share one IP address and one MAC Cluster Interface
address. See Link Aggregation. An interface on a cluster member, whose
Network Type was set as Cluster in
Bonding SmartConsole in cluster object. This
See Link Aggregation. interface is monitored by cluster, and failure
on this interface will cause cluster failover.
Bridge Mode
Cluster Member
A Security Gateway or Virtual System that
works as a Layer-2 bridge device for easy A Security Gateway that is part of a cluster.
deployment in an existing topology.
Cluster Mode
Cluster Configuration of cluster members to work in
1. Two or more Security Gateways that work these redundant modes:
together in a redundant configuration - High • One cluster member processes all the
Availability or Load Sharing. traffic - High Availability or VRRP mode
2. In a virtualized environment - a set of • All traffic is processed in parallel by all
VMware ESX/i hosts used for High Availability cluster members - Load Sharing
or Load Sharing.
Cluster Topology
Cluster Control Protocol (CCP) Set of interfaces on all members of a cluster
Proprietary Check Point protocol that runs and their settings (Network Objective, IP
between cluster members on UDP port 8116, address/Net Mask, Topology, Anti-Spoofing,
and has the following roles: and so on).
• State Synchronization (Delta Sync) ClusterXL
• Health checks (state of cluster members Cluster of Check Point Security Gateways
and of cluster interfaces): that work together in a redundant
• Health-status Reports configuration. The ClusterXL both handles
• Cluster-member Probing the traffic and performs State
• State-change Commands Synchronization.
• Querying for Cluster Membership These Check Point Security Gateways are
installed on Gaia OS:
Note: CCP is located between the Check
Point Firewall kernel and the network • Up to 5 cluster members are supported in
interface (therefore, only TCPdump should ClusterXL.
be used for capturing this traffic). • Up to 2 cluster members are supported in
VRRP cluster.
Cluster Correction Layer (CCL)
• Up to 13 cluster members are supported
Proprietary Check Point mechanism that
in VSX VSLS cluster.
deals with asymmetric connections in Check
Point cluster. Note - In ClusterXL Load Sharing mode,
configuring more than 4 members
significantly decreases the cluster
Delta Sync
performance due to amount of Delta Sync.
Synchronization of kernel tables between all
CPHA working cluster members - exchange of CCP
packets that carry pieces of information
General term that stands for Check Point
about different connections and operations
High Availability (historic fact: the first
that should be performed on these
release of ClusterXL supported only High
connections in relevant kernel tables. This
Availability) that is used only for internal
Delta Sync process is performed directly by
references (for example, inside kernel
Check Point kernel. While performing Full
debug) to designate ClusterXL infrastructure.
Sync, the Delta Sync updates are not
Creating an Interface Bond in Load processed and saved in kernel memory. After
Sharing Mode Full Sync is complete, the Delta Sync packets
stored during the Full Sync phase are applied
Follow the instructions in the R80.20 Gaia
by order of arrival.
Administration Guide
https://sc1.checkpoint.com/documents/R80. Delta Sync Retransmission
20_GA/WebAdminGuides/EN/CP_R80.20_Gai
a_AdminGuide/html_frameset.htm - Chapter It is possible that Delta Sync packets will be
Network Management - Section Network lost or corrupted during the Delta Sync
Interfaces - Section Bond Interfaces (Link operations. In such cases, it is required to
Aggregation). make sure the Delta Sync packet is re-sent.
The cluster member requests the sending
Critical Device cluster member to retransmit the
lost/corrupted Delta Sync packet.
Also known as a Problem Notification, or
Each Delta Sync packet has a sequence
pnote. A special software device on each number.
cluster member, through which the critical
The sending member has a queue of sent
aspects for cluster operation are monitored.
Delta Sync packets.
When the critical monitored component on a
Each cluster member has a queue of packets
cluster member fails to report its state on
sent from each of the peer cluster members.
time, or when its state is reported as
If, for any reason, a Delta Sync packet was
problematic, the state of that member is
not received by a cluster member, it can ask
immediately changed to Down. The complete for a retransmission of this packet from the
list of the configured critical devices (pnotes) sending member.
is printed by the cphaprob -ia list The Delta Sync retransmission mechanism is
command or show cluster pnotes all somewhat similar to a TCP Window and TCP
command. retransmission mechanism.
When a member requests retransmission of
Dead
Delta Sync packet, which no longer exists on
State reported by a cluster member when it the sending member, the member prints a
goes out of the cluster (due to cphastop console messages that the sync is not
command (which is a part of cpstop), or complete.
reboot).
Down
Decision Function State of a cluster member during a failure
A special cluster algorithm applied by each when one of the Critical Devices reports its
cluster member on the incoming traffic in state as "problem":
order to decide, which cluster member • In ClusterXL, applies to the state of the
should process the received packet. Each Security Gateway component
cluster members maintains a table of hash
values generated based on connections tuple • In 3rd party / OPSEC cluster, applies to
(source and destination IP addresses/Ports, the state of the State Synchronization
and Protocol number). mechanism
A cluster member in this state does not Flush and ACK
process any traffic passing through cluster. Also, FnA, F&A. Cluster member forces the
Delta Sync packet about the incoming packet
Dying
and waiting for acknowledgments from all
State of a cluster member as assumed by other Active members and only then allows
peer members, if it did not report its state for the incoming packet to pass through.
0.7 sec.
In some scenarios, it is required that some
Failback information, written into the kernel tables,
will be Sync-ed promptly, or else a race
Also, Fallback. Recovery of a cluster member condition can occur. The race condition may
that suffered from a failure. The state of a occur if a packet that caused a certain
recovered cluster member is changed from change in kernel tables left cluster
Down to either Active, or Standby Member_A toward its destination and then
(depending on Cluster Mode). the return packet tries to go through cluster
Member_B.
Failed Member
In general, this kind of situation is called
A cluster member that cannot send or accept asymmetric routing. What may happen in this
traffic because of a hardware or software scenario is that the return packet arrives at
problem. cluster Member_B before the changes
induced by this packet were Sync-ed to this
Failover
Member_B.
Also, Fail-over. Transferring of a control over
Example of such a case is when a SYN packet
traffic (packet filtering) from a cluster
goes through cluster Member_A, causing
member that suffered a failure to another
multiple changes in the kernel tables and
cluster member (based on internal cluster
then leaves to a server. The SYN-ACK packet
algorithms).
from a server arrives at cluster Member_B,
Failure but the connection itself was not Synced yet.
In this condition, the cluster Member_B will
A hardware or software problem that causes drop the packet as an Out-of-State packet
a Security Gateway to be unable to serve as a (First packet isn't SYN). In order to
cluster member (for example, one of cluster prevent such conditions, it is possible
interface has failed, or one of the monitored tousethe"FlushandAck"(F&A)mechanism.
daemon has crashed). Cluster member that
suffered from a failure is declared as failed, This mechanism can send the Delta Sync
and its state is changed to Down (a physical packets with all the changes accumulated so
interface is considered Down only if all far in the Sync buffer to the other cluster
configured VLANs on that physical interface members, hold the original packet that
are Down). induced these changes and wait for
acknowledgment from all other (Active)
Flapping cluster members that they received the
information in the Delta Sync packet. When
Consequent changes in the state of either all acknowledgments arrived, the
cluster interfaces (cluster interface flapping), mechanism will release the held original
or cluster members (cluster member packet.
flapping). Such consequent changes in the
state are seen in the Logs & Monitor > This ensures that by the time the return
Logs (if in SmartConsole >cluster object, the packet arrived from a server at the cluster,
cluster administrator set the Track all the cluster members are aware of the
changes in the status of cluster connection.
members to Log). F&A is being operated at the end of the
Inbound chain and at the end of the Outbound
chain (it is more common at the Outbound).
Forwarding Packets that are sent on the Forwarding
Process of transferring of an incoming traffic Layer use a special source MAC address to
from one cluster member to another cluster inform the receiving cluster member that
member for processing. There are two types they have already been inspected by another
of forwarding the incoming traffic between cluster member. Thus, the receiving cluster
cluster members - Packet forwarding and member can safely hand over these packets
Chain forwarding. Also see Forwarding Layer to the local Operating System, without
and ARP Forwarding. further inspection.

Forwarding Layer Full High Availability Mode

The Forwarding Layer is a ClusterXL Also, Full HA Mode. A special Cluster Mode
mechanism that allows a cluster member to (supported only on Check Point appliances
pass packets to peer cluster members, after running Gaia OS or SecurePlatform OS,
they have been locally inspected by the where each cluster member also runs as a
firewall. This feature allows connections to Security Management Server. This provides
be opened from a cluster member to an redundancy both between Security Gateways
external host. (only High Availability is supported) and
between Security Management Servers (only
Packets originated by cluster members are High Availability is supported). See sk101539
hidden behind the Cluster Virtual IP address. http://supportcontent.checkpoint.com/soluti
Thus, a reply from an external host is sent to ons?id=sk101539 and sk39345
the cluster, and not directly to the source http://supportcontent.checkpoint.com/soluti
cluster member. This can pose problems in ons?id=sk39345.
the following situations:
• The cluster is working in High Availability Full Sync
mode, and the connection is opened from Process of full synchronization of applicable
the Standby cluster member. All packets kernel tables by a cluster member from the
from the external host are handled by the working cluster member(s) when it tries to
Active cluster member, instead. join the existing cluster. This process is
• The cluster is working in a Load Sharing meant to fetcha"snapshot" of the applicable
mode, and the decision function has kernel tables of already Active cluster
selected another cluster member to member(s).
handle this connection. This can happen Full Sync is performed during the
since packets directed at a Cluster IP initialization of Check Point software (during
address are distributed between cluster boot process, the first time the cluster
members as with any other connection. member runs policy installation, during
If a cluster member decides, upon the cpstart, during cphastart). Until the Full
completion of the firewall inspection process, Sync process completes successfully, this
that a packet is intended for another cluster cluster member remains in the Down state,
member, it can use the Forwarding Layer to because until it is fully synchronized with
hand the packet over to that cluster member. other cluster members, it can not function as
a cluster member.
In High Availability mode, packets are
forwarded over a Synchronization network Meanwhile, the Delta Sync packets continue
directly to peer cluster members. It is to arrive, and the cluster member that tries
important to use secured networks only, as to join the existing cluster, stores them in the
encrypted packets are decrypted during the kernel memory until the Full Sync
inspection process, and are forwarded as completes.
clear-text (unencrypted) data. The whole Full Sync process is performed by
In Load Sharing mode, packets are fwd daemons on TCP port 256 over the Sync
forwarded over a regular traffic network. network (if it fails over the Sync network, it
tries the other cluster interfaces). The
information is sent by fwd daemons in other Cluster Members still run their
chunks, while making sure they confirm Software Blades in the kernel space
getting the information before sending the (represented by the fw_worker processes). In
next chunk. the Hybrid Mode, Cluster Members are able
to synchronize the required information.
Also see Delta Sync.
Init
HA not started
State of a cluster member in the phase after
Output of the cphaprob <flag> command
the boot and until the Full Sync completes. A
or show cluster <option> command on
cluster member in this state does not
the cluster member. This output means that
process any traffic passing through cluster.
Check Point clustering software is not
started on this Security Gateway (for IP Tracking
example, this machine is not a part of a
cluster, or cphastop command was run, or Collecting and saving of Source IP addresses
some failure occurred that prevented the and Source MAC addresses from incoming IP
ClusterXL product from starting correctly). packets during the probing. IP tracking is a
useful for members within a cluster to
High Availability Mode determine whether the network connectivity
of the member is acceptable.
A redundant cluster mode, where only one
cluster member (Active member) processes IP Tracking Policy
all the traffic, while other cluster members
(Standby members) are ready to be promoted Setting that controls, which IP addresses
to Active state if the current Active member should be tracked during IP tracking:
fails. • Only IP addresses from the subnet of
In the High Availability mode, the Cluster cluster VIP, or from subnet of physical
Virtual IP address (that represents the cluster interface (this is the default)
cluster on that network) is associated: • All IP addresses, also outside the cluster
• With physical MAC Address of Active subnet
member Link Aggregation
• With virtual MAC Address (see sk50840 A technology that joins multiple physical
http://supportcontent.checkpoint.com/sol interfaces together into one virtual interface,
utions?id=sk50840) known as a bond interface. Also known as
HTU Interface Bonding.

Stands for "HA Time Unit". All internal time Load Sharing Mode
in ClusterXL is measured in HTUs (the times
Also, Load Balancing mode. A redundant
in cluster debug also appear in HTUs).
cluster mode, where all cluster members
Formula in the Check Point software: 1 HTU
process all incoming traffic in parallel. See
= 10 x fwha_timer_base_res = 10 x 10
milliseconds = 100 ms Load Sharing Multicast Mode and Load
Sharing Unicast Mode.
Hybrid Mode
Load Sharing Multicast Mode
Starting in R80.20, on Security Gateways with
40 or more CPU cores, Software Blades run Load Sharing Cluster Mode, where all cluster
in the user space (as fwk processes). The members process all traffic in parallel. Each
Hybrid Mode refers to the state when you cluster member is assigned the equal load of
upgrade Cluster Members from R80.10 (or [ 100% / number_of_members ].
below) to R80.20 (or above). The Hybrid Mode The Cluster Virtual IP address (that
is the state, in which the upgraded Cluster represents the cluster on that network) is
Members already run their Software Blades associated with Multicast MAC Address
in the user space (as fwk processes), while 01:00:5E:X:Y:Z (which is generated based on
last 3 bytes of cluster Virtual IP address on interface state appears in the output of the
that network). cphaprob -a if command.
A ClusterXL decision algorithm (Decision
Function) on all cluster members decides, Non-Sticky Connection
which cluster member should process the A connection is called non-sticky, if the reply
given packet. packet returns via a different cluster
member, than the original packet (for
Load Sharing Unicast Mode
example, if network administrator has
Load Sharing Cluster Mode, where one configured asymmetric routing). In Load
cluster member (called Pivot) accepts all Sharing mode, all cluster members are
traffic. Then, the Pivot member decides to Active, and in Static NAT and encrypted
process this traffic, or to forward it to other connections, the Source and Destination IP
non-Pivot cluster members. addresses change. Therefore, Static NAT and
The traffic load is assigned to cluster encrypted connections through a Load
members based on the hard-coded formula Sharing cluster may be non-sticky.
per the value of Pivot_overhead attribute
(see sk34668 Packet Selection
http://supportcontent.checkpoint.com/soluti Distinguishing between different kinds of
ons?id=sk34668). packets coming from the network, and
The Cluster Virtual IP address (that selecting, which member should handle a
represents the cluster on that network) is specific packet (Decision Function
associated with: mechanism):
• Physical MAC Address of Pivot member • CCP packet from another member of this
• Virtual MAC Address (see sk50840 cluster
http://supportcontent.checkpoint.com/sol • CCP packet from another cluster or from
utions?id=sk50840) a cluster member with another version
Management Server (usually older version of CCP)

A Security Management Server or a • Packet is destined directly to this

Multi-Domain Server. member
• Packet is destined to another member of
Master
this cluster
State of a cluster member that processes all
• Packet is intended to pass through this
traffic in cluster configured in VRRP mode.
cluster member
Network Objective • ARP packets
Defines how the cluster will configure and Pingable Host
monitor an interface - Cluster, Sync,
Some host (that is, some IP address) that
Cluster+Sync, Monitored Private,
cluster members can ping during probing
Non-Monitored Private. Configured in
mechanism. Pinging hosts in an interface's
SmartConsole > cluster object > Topology
subnet is one of the health checks that
pane - Network Objective. ClusterXL mechanism performs. This
pingable host will allow the cluster members
Non-Blocking Mode
to determine with more precision what has
Cluster operation mode, in which cluster failed (which interface on which member).
member keeps forwarding all traffic. On Sync network, usually, there are no hosts.
In such case, if switch supports this, an IP
Non-Monitored Interface address should be assigned on the switch
An interface on a cluster member, whose (for example, in the relevant VLAN).
Network Type was set as Private in The IP address of such pingable host should
SmartConsole, in cluster object. This be assigned per this formula:
IP_of_pingable_host = member with the lowest priority appears
IP_of_physical_interface_on_membe at the bottom of the table.
r + ~10
Assigning the IP address to pingable host • The cluster member with the highest
that is higher than the IP addresses of priority will assume the Active state.
physical interfaces on the cluster members • If the current Active cluster member with
will give some time to cluster members to the highest priority (for example,
perform the default health checks. Member_A), fails for some reason, or is
Example: rebooted, then failover occurs between
• IP address of physical interface on a given subnet cluster members. The cluster member
on Member_A is 10.20.30.41 with the next highest priority will be
• IP address of physical interface on a given subnet promoted to be Active (for example,
on Member_B is 10.20.30.42 Member_B).
• IP address of pingable host should be at least
10.20.30.5 • When the cluster member with the
highest priority (Member_A) recovers
Pivot Member
from a failure, or boots, then additional
A cluster member in the Unicast Load failover occurs between cluster
Sharing cluster that receives all packets. members. The cluster member with the
Cluster Virtual IP addresses are associated highest priority (Member_A) will be
with Physical MAC Addresses of this cluster promoted to Active state (and Member_B
member. This Pivot cluster member will return to Standby state).
distributes the traffic between other cluster
Private Interface
members.
An interface on a cluster member, whose
Pnote Network Type was set as Private in
See Critical Device. SmartConsole in cluster object. This
interface is not monitored by cluster, and
Preconfigured Mode failure on this interface will not cause any
changes in cluster member's state.
Cluster Mode, where cluster membership is
enabled on all members to be. However, no Probing
policy had been yet installed on any of the
members - none of them is actually If a cluster member fails to receive status for
configured to be primary, secondary, and so another member (does not receive CCP
on. The cluster cannot function if one packets from that member) on a given
machinefails.Inthisscenario,the segment, cluster member will probe that
"preconfigured mode" takes place. The segment in an attempt to illicit a response.
preconfigured mode also comes into effect The purpose of such probes is to detect the
when no policy is yet installed, right after the nature of possible interface failures, and to
machines came up after boot, or when determine which module has the problem.
running the cphaconf init command. The outcome of this probe will determine
what action is taken next (change the state of
Primary Up an interface, or of a cluster member).

ClusterXL in High Availability mode that was Problem Notification

configured as Switch to higher
See Critical Device.
priority Cluster Member in the cluster
object in SmartConsole: Ready
• Each cluster member is given a priority State of a cluster member during after
(SmartConsole > cluster object > initialization and before promotion to the
Cluster Members pane). Cluster next required state - Active / Standby / VRRP
member with the highest priority appears Master / VRRP Backup (depending on Cluster
at the top of the table, and cluster Mode). A cluster member in this state does
not process any traffic passing through Standby
cluster. A member can be stuck in this state State of a cluster member that is ready to be
due to several reasons - see sk42096 promoted to Active state (if the current Active
http://supportcontent.checkpoint.com/soluti member fails). Applies only to ClusterXL High
ons?id=sk42096. Availability Mode.
Security Gateway State Synchronization
A Check Point computer that runs Check Technology that synchronizes the relevant
Point software to inspect traffic and enforces information about the current connections
Security Policies for connected network (stored in various kernel tables on Check
resources. Point Security Gateways) among all cluster
members over Synchronization Network. Due
Security Management Server
to State Synchronization, the current
A Check Point computer that runs Check connections are not cut off during cluster
Point software to manage the objects and failover.
policies in Check Point environment.
Sticky Connection
Selection
A connection is called sticky, if all packets
The packet selection mechanism is one of are handled by a single cluster member (in
the central and most important components High Availability mode, all packets reach the
in the ClusterXL product and State Active cluster member, so all connections
Synchronization infrastructure for 3rd party are sticky).
clustering solutions. Its main purpose is to
decide (to select) correctly what has to be Subscribers
done to the incoming and outgoing traffic on User Space processes that are made aware
the cluster machine. of the current state of the ClusterXL state
• In ClusterXL, the packet is selected by machine and other clustering configuration
cluster member(s) depending on the parameters. List of such subscribers can be
cluster mode: obtained by running the cphaconf
• In HA modes - by Active member debug_data command (see sk31499
http://supportcontent.checkpoint.com/soluti
• In LS Unicast mode - by Pivot member
ons?id=sk31499).
• In LS Multicast mode - by all
members Sync Interface
Then the cluster member applies the
Also, Secured Interface, Trusted Interface.
Decision Function (and the Cluster
An interface on a cluster member, whose
Correction Layer (CCL)).
Network Type was set as Sync or
• In 3rd party / OPSEC cluster, the 3rd Cluster+Sync in SmartConsole in cluster
party software selects the packet, and object. This interface is monitored by cluster,
Check Point software just inspects it (and and failure on this interface will cause
performs State Synchronization). cluster failover. This interface is used for
SmartConsole State Synchronization between cluster
members.
Check Point main GUI client used to create The use of more than one Sync Interfaces for
and manage the security policy. redundancy is not supported because the
CPU load will increase significantly due to
SmartDashboard
duplicate tasks performed by all configured
A legacy Check Point GUI client used to Synchronization Networks. See sk92804
create and manage the security policy in http://supportcontent.checkpoint.com/soluti
R77.30 and below. ons?id=sk92804.
Synchronization Network
Also, Sync Network, Secured Network,
Trusted Network. A set of interfaces on
cluster members that were configured as
interfaces, over which State Synchronization
information will be passed (as Delta Sync
packets ). The use of more than one
Synchronization Network for redundancy is
not supported because the CPU load will
increase significantly due to duplicate tasks
performed by all configured Synchronization
Networks. See sk92804
http://supportcontent.checkpoint.com/soluti
ons?id=sk92804.

Traffic
The flow of data between network devices.

VLAN
Virtual Local Area Network. Open servers or
appliances connected to a virtual network,
which are not physically connected to the
same network.

VLAN Trunk
A connection between two switches that
contains multiple VLANs.

VMAC
Virtual MAC address. When this feature is
enabled on cluster members, all cluster
members in High Availability mode and Load
Sharing Unicast mode associate the same
Virtual MAC address with Virtual IP address.
This allows avoiding issues when Gratuitous
ARP packets sent by cluster during failover
are not integrated into ARP cache table on
switches surrounding the cluster. See
sk50840
http://supportcontent.checkpoint.com/soluti
ons?id=sk50840.
CHAPTE R 1

Introduction to ClusterXL
In This Section:
The Need for Clusters ..................................................................................................18
ClusterXL Solution ........................................................................................................18
How ClusterXL Works ..................................................................................................19

The Need for Clusters

Security Gateways and VPN connections are business critical devices. The failure of a Security
Gateway or VPN connection can result in the loss of active connections and access to critical data.
The Security Gateway between the organization and the world must remain open under all
circumstances.

ClusterXL Solution
ClusterXL is a Check Point software-based cluster solution for Security Gateway redundancy and
Load Sharing. A ClusterXL Security Cluster contains identical Check Point Security Gateways.
• A High Availability Security Cluster ensures Security Gateway and VPN connection redundancy
by providing transparent failover to a backup Security Gateway in the event of failure.
• A Load Sharing Security Cluster provides reliability and also increases performance, as all
members are active

Item Description
1 Internal network

2 Switch for internal network

3 Security Gateways with ClusterXL Software Blade

4 Switch for external networks

5 Internet

ClusterXL Administration Guide R80.20 | 18

 Introduction to ClusterXL

How ClusterXL Works

ClusterXL uses State Synchronization to keep active connections alive and prevent data loss when
a member fails. With State Synchronization, each member "knows" about connections that go
through other members.
ClusterXL uses virtual IP addresses for the cluster itself and unique physical IP and MAC
addresses for the cluster members. Virtual IP addresses do not belong to physical interfaces.
Note - This guide contains information only for Security Gateway clusters. For additional
information about the use of ClusterXL with VSX, see the R80.20 VSX Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_VSX_Admin
Guide/html_frameset.htm.

The Cluster Control Protocol

The Cluster Control Protocol (CCP) is the glue that links together the members in the Security
Cluster. CCP traffic is distinct from ordinary network traffic and can be viewed using any network
sniffer.
CCP runs on UDP port 8116, and has the following roles:
• It allows cluster members to report their own states and learn about the states of other
members by sending keep-alive packets (this only applies to ClusterXL clusters).
• State Synchronization.
The Check Point CCP is used by all ClusterXL modes.
Note - There is no need to add a rule to the Security Policy Rule Base that accepts CCP.
For more information about the CCP packets, see sk25977
http://supportcontent.checkpoint.com/solutions?id=sk25977.

ClusterXL Administration Guide R80.20 | 19

 ClusterXL Requirements and Compatibility

ClusterXL Requirements and

Compatibility
Check Point Appliances and Open Servers
You can install ClusterXL on Check Point appliances in one of these configurations:
• A distributed configuration - the cluster members and the Security Management Server are
installed on different computers
• A Full High Availability configuration - the cluster members and the Security Management
Servers are installed on the same computers (each computer runs a Standalone configuration)
You can install ClusterXL on Open Servers only in a distributed configuration - the cluster
members and the Security Management Server are installed on different computers.
To see the ClusterXL supported platforms, see the R80.20 Release Notes
http://downloads.checkpoint.com/dc/download.htm?ID=65044.
For installation instructions, see the R80.20 Installation and Upgrade Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Installation
_and_Upgrade_Guide/html_frameset.htm.

Supported Number of Cluster Members

• ClusterXL in High Availability mode supports up to 5 cluster members.
• ClusterXL in Load Sharing mode supports up to 5 cluster members.
Note - Configuring more than 4 members significantly decreases cluster performance due to
amount of Delta Sync.
• VRRP cluster on Gaia OS supports only 2 cluster members (see sk105170
http://supportcontent.checkpoint.com/solutions?id=sk105170).
• Virtual System Load Sharing (VSLS) mode supports up to 13 cluster members.

Hardware Requirements for Cluster Members

ClusterXL operation completely relies on internal timers and calculation of internal timeouts,
which are based on hardware clock ticks.
Therefore, in order to avoid unexpected behavior, ClusterXL is supported only between machines
with identical CPU characteristics.
In addition, in order to avoid unexpected fail-overs due to issues with CCP packets on cluster
interfaces, we strongly recommend to pair only identical physical interfaces as cluster interfaces -
even when connecting the cluster members via a switch. For example:
• Intel 82598EB on Member_A with Intel 82598EB on Member_B
• Broadcom NeXtreme on Member_A with Broadcom NeXtreme on Member_B

ClusterXL Administration Guide R80.20 | 20

 ClusterXL Requirements and Compatibility

Note - There is no requirement for throughput of Sync interface to be identical to, or larger than
throughput of traffic interfaces (although, to prevent a possible bottle neck, a good practice for
throughput of Sync interface is to be at least identical to throughput of traffic interfaces).

Software Requirements for Cluster Members

ClusterXL is supported only between identical operating systems - all cluster members must be
installed on the same operating system).
ClusterXL is supported only between identical Check Point software versions - all cluster
members must be installed with identical Check Point software, including OS build and hotfixes.
All Check Point software components must be the same on all cluster members. Meaning that the
same Software Blades and features must be enabled on all cluster members:
• SecureXL status on all cluster members must be the same (either enabled, or disabled)
• Number of CoreXL FW instances on all cluster members must be the same
• Advanced Dynamic Routing status and configuration on all cluster members must be the same
Otherwise, traffic might not be processed as expected and/or state of cluster members might
change expectedly. In addition, Full Sync will fail.

Hardware Requirements for Switches and Routers

The Cluster is usually located in an environment having other networking devices, such as
switches and routers. These devices and the cluster members must interact to assure network
connectivity. This section outlines the requirements imposed by ClusterXL on surrounding
networking equipment.

High Availability and Load Sharing Unicast Modes

When using CCP in multicast mode, configure the following settings on the switch:
By default, when ClusterXL is configured in High Availability mode or Load Sharing Unicast Mode,
cluster members send the Cluster Control Protocol (CCP) packets in Multicast mode (the Layer 2
destination MAC address in the CCP packets is a multicast MAC address 01:00:5e:X:X:X). To let the
CCP packets pass between cluster members, configure the settings below on the surrounding
switches.

Switch Setting Explanation

IGMP and IGMP registration (also known as IGMP Snooping) is enabled by default on
Static CAMs cluster members. You can disable IGMP registration on cluster members.
In scenarios, where disabling IGMP registration is problematic, you can
configure Static CAMs on switches to allow multicast traffic on specified switch
ports.

ClusterXL Administration Guide R80.20 | 21

 ClusterXL Requirements and Compatibility

Switch Setting Explanation

Disabling Certain switches have an upper limit on the number of broadcasts and
multicast multicasts that they can pass, in order to prevent broadcast storms. This limit is
limits usually a percentage of the total interface bandwidth.
It is possible to either turn off broadcast storm control, or to allow a higher level
of broadcasts or multicasts through the switch.
If the connecting switch is incapable of having any of these settings configured, it
is possible, though less efficient, for the switch to use broadcast to forward
traffic, and to configure the cluster members to use broadcast CCP (see
"Selecting the CCP Transport Mode on the Cluster Members" on page 57).

When using CCP in multicast mode, configure the following settings on the router:
By default, when ClusterXL is configured in High Availability mode or Load Sharing Unicast Mode,
the unicast Cluster Virtual IP addresses are mapped to unicast MAC addresses of the physical
interfaces on the Active or Pivot cluster member. To let the traffic reach the cluster, configure the
settings below on the surrounding routers.

Router Setting Explanation

Unicast MAC The router needs to be able to learn this MAC through regular ARP messages.

Load Sharing Multicast Mode

When working with Load Sharing in Multicast mode, configure the following settings on the
switch:
By default, when ClusterXL is configured in Load Sharing Multicast Mode, cluster members send
the Cluster Control Protocol (CCP) packets in Multicast mode (the Layer 2 destination MAC
address in the CCP packets is a multicast MAC address 01:00:5e:X:X:X).

Switch Setting Explanation

Port Mirroring ClusterXL in Load Sharing Multicast mod does not support the use of unicast
MAC addresses with Port Mirroring.

When working with Load Sharing in Multicast mode, configure the following settings on the
router:
By default, when ClusterXL is configured in High Availability mode or Load Sharing Multicast
Mode, the unicast Cluster Virtual IP addresses are mapped to multicast MAC addresses (these are
generated automatically by the Management Server based on the configured Cluster Virtual IP
addresses). To let the traffic reach the cluster, the router must support sending unicast IP packets
with multicast MAC addresses.

Router Setting Explanation

Static MAC Most routers can add ARP entries with a unicast IP address and a multicast
MAC address automatically. If you have a router that is not able to learn this
type of mapping dynamically, you need to configure static MAC entries to map
unicast Cluster Virtual IP addresses to multicast MAC addresses.

IGMP and Some routers require disabling of IGMP snooping, or configuration of static
Static CAMs CAMs to support sending of unicast IP packets with multicast MAC addresses.

ClusterXL Administration Guide R80.20 | 22

 ClusterXL Requirements and Compatibility

Router Setting Explanation

Disabling Certain routers have an upper limit on the number of broadcasts and multicasts
multicast limits that they can pass, in order to prevent broadcast storms. This limit is usually a
percentage of the total interface bandwidth.
It is possible to either turn off broadcast storm control, or to allow a higher limit
of broadcasts or multicasts through the router.

Disabling Some routers send multicast traffic to the router itself. This may cause a
forwarding of multicast storm through the network. Disable such configuration on your
multicast router.
traffic to the
router itself

VMAC Mode
When ClusterXL is configured in HA mode or Load Sharing Unicast mode (not Multicast) a single
cluster member is associated with the Cluster Virtual IP address. In a High Availability
environment, the single member is the Active member. In a Load Sharing environment, the single
member is the Pivot.
After fail-over, the new Active member (or Pivot member) broadcasts a series of Gratuitous ARP
Requests (G-ARPs). The G-ARPS associate the Virtual IP address of the cluster with the physical
MAC address of the new Active member or the new Pivot. When this happens:
• A member with a large number of Static NAT entries can transmit too many GARPs
Switches may not integrate these GARP updates quickly enough into their ARP tables.
Switches continue to send traffic to the physical MAC address of the member that failed. This
results in traffic outage until the switches have fully updated ARP cache tables.
• Network components, such as VoIP phones, ignore GARPs
These components continue to send traffic to the MAC address of the failed member.
To minimize possible traffic outage during a fail-over, configure the cluster to use a virtual MAC
address (VMAC).
By enabling Virtual MAC in ClusterXL High Availability mode, or Load Sharing Unicast mode, all
cluster members associate the same Virtual MAC address with all Cluster Virtual Interfaces and
the Virtual IP address. In Virtual MAC mode, the VMAC that is advertised by the cluster members
(through G-ARP Requests) keeps the real MAC address of each member and adds a Virtual MAC
address on top of it.
For local connections and sync connections, the real physical MAC address of each cluster
member is still associated with its real IP address.
Note - Traffic originating from the cluster members will be sent with the VMAC Source address.
You can enable VMAC in SmartConsole, or on the command line. See sk50840
http://supportcontent.checkpoint.com/solutions?id=sk50840.
Failover time in a cluster with enabled VMAC mode is shorter than a failover in a cluster that uses
a physical MAC addresses.

To configure VMAC Mode using SmartConsole:

1. Double-click the Cluster object to open its Properties window.
2. On the ClusterXL and VRRP page, select Use Virtual MAC.
ClusterXL Administration Guide R80.20 | 23
 ClusterXL Requirements and Compatibility

3. Click OK.
4. Install the Access Control Policy on this cluster object.

To configure VMAC Mode using the command line:

On each cluster member, set the same value for the global kernel parameter
fwha_vmac_global_param_enabled.
1. Connect to the command line on each cluster member.
2. Log in to the Expert mode.
3. Get the current value of this kernel parameter. Run:
fw ctl get int fwha_vmac_global_param_enabled
4. Set the new value for this kernel parameter temporarily (does not survive reboot). Run:
To enable VMAC mode: fw ctl set int fwha_vmac_global_param_enabled 1
To disable VMAC mode: fw ctl set int fwha_vmac_global_param_enabled 0
5. Make sure the state of the VMAC mode was changed. Run on each cluster member:
In Gaia Clish: show cluster interfaces all
In Expert mode: cphaprob -a if
When VMAC mode is enabled, output of this command shows the VMAC address of each virtual
cluster interface.
6. To set the new value for this kernel parameter permanently:
Follow the instructions in sk26202
http://supportcontent.checkpoint.com/solutions?id=sk26202 to add this line to the
$FWDIR/boot/modules/fwkern.conf file:
fwha_vmac_global_param_enabled=<value>

Supported Topologies for Synchronization Network

Topology 1
• Sync interface is a Bond of several physical slave interfaces.
To work with this topology, you can configure the Bond interface in High Availability or Load
Sharing mode.
• All physical slave interfaces on all cluster members connect to the same switch.

Topology 2
• Sync interface is a Bond of several physical slave interfaces.

ClusterXL Administration Guide R80.20 | 24

 ClusterXL Requirements and Compatibility

To work with this topology, you can configure the Bond interface in High Availability or Load
Sharing mode.
• On each cluster member, physical slave interfaces of the same Bond connect to different
switches.
• The switches must connect to each other through a cable.

Topology 3 (new in R80.20: Enhanced Active/Backup Bond)

• Sync interface is a Bond of several physical slave interfaces.
To work with this topology, you must configure the Bond interface in High Availability mode.
• On each cluster member, physical slave interfaces of the same Bond connect to different
switches.
• The switches do not need to connect to each other through a cable.

Topology 4 (new in R80.20: Enhanced Active/Backup Bond)

• Sync interface is a Bond of several physical slave interfaces.
To work with this topology, you must configure the Bond interface in High Availability mode.
• On each cluster member, physical slave interfaces of the same Bond connect to different pairs
of switches.

ClusterXL Administration Guide R80.20 | 25

 ClusterXL Requirements and Compatibility

• These pairs of switches connect to each other (chained together).

Clock Synchronization in ClusterXL

When using ClusterXL, make sure to synchronize the clocks of all of the cluster members. You
can synchronize the clocks manually, or using a protocol such as NTP. Features, such as VPN,
only function properly when the clocks of all of the cluster members are synchronized.

IPv6 Support for ClusterXL

R80.20 ClusterXL supports High Availability clusters for IPv6. IPv6 status information is
synchronized and the IPv6 clustering mechanism is activated during failover.

You can define IPv6 addresses for:

• Cluster virtual interfaces
• Member physical interfaces

Limitations
• IPv6 is not supported for Load Sharing clusters.
• You cannot define IPv6 address for synchronization interfaces.

IPv6 in ClusterXL High Availability

During failover, a cluster sends gratuitous ARP request packets to update hosts and routers
connected to cluster interfaces. It does this by advertising the new MAC address for the virtual
cluster IPv4 addresses.
ClusterXL updates the IPv6 network during failovers. ClusterXL sends Neighbor Advertisement
messages to update the neighbor cache (which is equivalent to the ARP cache in IPv4) by
advertising the new MAC address for the virtual cluster IPv6 address. In addition, ClusterXL will
reply to any Neighbor Solicitation with a target address equal to the Virtual Cluster IPv6 address.
Note - ClusterXL failover event detection is based on IPv4 probing. During state transition the IPv4
driver instructs the IPv6 driver to reestablish IPv6 network connectivity to the HA cluster.

ClusterXL Administration Guide R80.20 | 26

 ClusterXL Requirements and Compatibility

Synchronized Cluster Restrictions

The following restrictions apply when you synchronize cluster members:
• The use of more than one dedicated physical interface for synchronization redundancy is not
supported. You can use Bonding ("Sync Redundancy" on page 113) for synchronization
interface redundancy.
Synchronization interface redundancy is not supported for VRRP clusters. See sk92804
http://supportcontent.checkpoint.com/solutions?id=sk92804.
• All cluster members must run on identically configured hardware platforms.
• If a cluster member goes down, user-authenticated connections through that member are
lost. Other cluster members cannot restore the connection. Client-authenticated or
session-authenticated connections are maintained.
The reason for these restrictions is that the user authentication state is maintained by a
process on the Security Gateway. It cannot be synchronized on cluster members in the same
way as kernel data is synchronized. However, the states of Session Authentication and Client
Authentication are saved in kernel tables, and can be synchronized.
• The connection statutes that use system resources cannot be synchronized for the same
reason that user-authenticated connections cannot be synchronized.
• Accounting information for connections is accumulated on each cluster member, sent to the
Security Management Server, and aggregated. In the event of a cluster failover, the accounting
information that is not yet sent to the Security Management Server, is lost. To minimize this
risk, you can reduce the time interval when accounting information is sent. To do this, in the
cluster object > Logs > Additional Logging pane, set a lower value for the Update Account Log
every attribute.

ClusterXL Administration Guide R80.20 | 27

 ClusterXL Requirements and Compatibility

Check Point Software Compatibility

ClusterXL Compatibility with IPS
The following IPS features are supported by ClusterXL, with the limitations listed in the notes.

IPS Feature Load Sharing High Availability

Fragment Sanity Check Yes (1, 3) Yes (1)

Pattern Matching Yes (2, 3) Yes (2)

Sequence Verifier Yes (2, 4) Yes (2)

FTP, HTTP and SMTP Security Servers Yes (2, 5) Yes (2)

Notes:
1. If there is a cluster failover when fragments are being received, the packet will be lost.
2. Does not survive cluster failover.
3. Requires unidirectional stickiness. This means that the same cluster member must receive all
external packets, and the same cluster member must receive all internal packets, but the
same cluster member does not have to receive both internal and external packets.
4. Requires bidirectional connection stickiness.
5. Uses the cluster Forwarding Layer.

ClusterXL Compatibility (Excluding IPS)

The following table presents ClusterXL Load Sharing and High Availability compatibility. Some
Check Point products and features are not supported, or are only partially supported for use with
ClusterXL.

Feature or Product Feature LS HA

Security Management No No

Security Gateway Yes Yes

Firewall Authentication / Security Servers Yes (1) Yes (1)

Firewall ACE servers and SecurID Yes Yes

Firewall Application Intelligence protocol inspection (2) Yes (3) Yes

Firewall Sequence Verifier Yes (4) Yes (1)

Firewall UDP encapsulation Yes Yes

Firewall Suspicious Activity Monitoring (SAM) Yes Yes

ClusterXL Administration Guide R80.20 | 28

 ClusterXL Requirements and Compatibility

Feature or Product Feature LS HA

Firewall ISP Redundancy Yes Yes

VPN Third party VPN peers Yes Yes

Endpoint Security Client Software Distribution Server (SDS) No No

Endpoint Security Client IP per user in Office Mode Yes Yes

SecureXL Yes Yes

QoS Yes (4, 5) Yes

SmartProvisioning SmartLSM Security Gateway No No

Notes:
1. If there is a cluster failover when fragments are being received, the packet will be lost.
2. Does not survive cluster failover.
3. Requires unidirectional stickiness. This means that the same cluster member must receive all
external packets, and the same cluster member must receive all internal packets, but the
same cluster member does not have to receive both internal and external packets.
4. Requires bidirectional connection stickiness.
5. Uses the cluster Forwarding Layer.

Example Configuration of a Cisco Catalyst Routing

Switch
The following examples show how to perform the configuration needed to support ClusterXL on a
Cisco Catalyst 6500 Series routing switch. For more details, or instructions for other networking
devices, always refer to the device vendor documentation.
To disable IGMP snooping:
Cisco(config)# no ip igmp snooping

To disable multicast limits:

Cisco(config)# no storm-control multicast level

To define Static CAM entries:

To determine the MAC addresses that must be set:
1. On a network that has a Cluster Virtual IP address of x.y.z.w:
• If y<=127, the multicast MAC address would be 01:00:5e:y:z:w
For example: 01:00:5e:5A:0A:64 for 192.90.10.100
• If y>127, the multicast MAC address would be 01:00:5e:(y-128):z:w
For example: 01:00:5e:28:0A:64 for 192.168.10.100 (168 - 128 = 40 in dec = 28 in hex)

ClusterXL Administration Guide R80.20 | 29

 ClusterXL Requirements and Compatibility

2. For a network x.y.z.0 that does not have a Cluster Virtual IP address, such as the Sync, you
use the same procedure and substitute fa instead of 0 for the last octet of the MAC address.
• For example: 01:00:5e:00:00:fa for the 10.0.0.X network
To add a permanent CAM entry for the multicast MAC address for module 1 - port 1, and module 2
- ports 1, 3, and 8 through 12:
Cisco> (enable) set cam permanent 01-40-5e-28-0a-64 1/1,2/1,2/3,2/8-12
Permanent multicast entry added to CAM table.
Cisco> (enable)

To prevent multicast packets from reaching the router:

1. Determine the MAC addresses that must be set (see above).
2. Define a static CAM entry (entry will remain in the CAM table until the switch is reset). Run:
Cisco> (enable) set cam static <MAC address> module/port
Static unicast entry added to CAM table.
Cisco> (enable)

To define a static ARP entry:

1. Determine the MAC addresses that must be set (see above).
2. Define a static ARP entry. Run:
Cisco(config)# arp <MAC address> arpa

ClusterXL Administration Guide R80.20 | 30

CHAPTE R 2

High Availability and Load Sharing in

ClusterXL
In This Section:
Introduction to High Availability and Load Sharing ....................................................31
Example ClusterXL Topology .......................................................................................33
Implementing Planning Considerations ......................................................................36
ClusterXL Modes...........................................................................................................37
Cluster Failover ............................................................................................................43

Introduction to High Availability and Load Sharing

ClusterXL is a software-based Load Sharing and High Availability solution that distributes network
traffic between clusters of redundant Security Gateways.
ClusterXL has these High Availability features:
• Transparent failover in case of member failures
• Zero downtime for mission-critical environments (when using State Synchronization)
• Enhanced throughput (in Load Sharing modes)
• Transparent upgrades
All members in the cluster are aware of the connections passing through each of the other
members. The cluster members synchronize their connection and status information across a
secure synchronization network.
The glue that binds the members in a ClusterXL cluster is the Cluster Control Protocol (CCP),
which is used to pass synchronization and other information between the cluster members.

High Availability
In a High Availability cluster, only one member is active (Active/Standby operation). In the event
that the active cluster member becomes unavailable, all connections are re-directed to a
designated standby without interruption. In a synchronized cluster, the standby cluster members
are updated with the state of the connections of the active cluster member.
In a High Availability cluster, each member is assigned a priority. The highest priority member
serves as the Security Gateway in normal circumstances. If this member fails, control is passed to
the next highest priority member. If that member fails, control is passed to the next member, and
so on.
Upon Security Gateway recovery, you can maintain the current active Security Gateway (Active Up),
or to change to the highest priority Security Gateway (Primary Up).
ClusterXL High Availability mode supports both IPv4 and IPv6.

ClusterXL Administration Guide R80.20 | 31

 High Availability and Load Sharing in ClusterXL

Load Sharing
ClusterXL Load Sharing distributes traffic within a cluster so that the total throughput of multiple
members is increased. In Load Sharing configurations, all functioning members in the cluster are
active, and handle network traffic (Active/Active operation).
If any member in a cluster becomes unreachable, transparent failover occurs to the remaining
operational members in the cluster, thus providing High Availability. All connections are shared
between the remaining Security Gateways without interruption.
ClusterXL Load Sharing modes do not support IPv6.

ClusterXL Administration Guide R80.20 | 32

 High Availability and Load Sharing in ClusterXL

Example ClusterXL Topology

ClusterXL uses unique physical IP and MAC addresses for each cluster member, and a virtual IP
addresses for the cluster itself. Cluster interface addresses do not belong to any real member
interface.
The following diagram illustrates a two-member ClusterXL cluster, showing the cluster Virtual IP
addresses and members physical IP addresses. This sample deployment is used in many of the
examples presented in this chapter.

Item Description
1 Internal network

2 Internal switch (internal cluster IP address 10.10.0.100)

3 Security Gateway - Cluster member A

3a Virtual interface to the internal network (10.10.0.1)

3b Interface to the Cluster Sync network (10.0.10.1)

3c Virtual interface to the external network (192.168.10.1)

4 Security Gateway - Cluster member B

4a Virtual interface to the internal network (10.10.0.2)

4b Interface to the Cluster Sync network (10.0.10.2)

4c Virtual interface to the external network (192.168.10.2)

5 External switch (external cluster IP address 192.168.10.100)

6 Internet

Each cluster member has three interfaces: one external interface, one internal interface, and one
for synchronization. Cluster member interfaces facing in each direction are connected via a hub or
switch.

ClusterXL Administration Guide R80.20 | 33

 High Availability and Load Sharing in ClusterXL

All cluster member interfaces facing the same direction must be in the same network. For
example, there must not be a router between cluster members.
The Security Management Server can be located anywhere, and connection should be established
to either the internal or external cluster IP addresses.
These sections present ClusterXL configuration concepts shown in the example.
Note - In these examples, RFC 1918 private addresses in the range 192.168.0.0 to
192.168.255.255 are treated as public IP addresses.

Defining the Cluster Member IP Addresses

The guidelines for configuring each cluster member are as follows:
All members within the cluster must have at least three interfaces:
• An interface facing the external network (that, for example, faces the Internet).
• An interface facing the internal network.
• An interface used for synchronization.
All interfaces pointing in a certain direction must be on the same network.
For example, in the previous illustration, there are two cluster members, Member_A and
Member_B. Each has an interface with an IP address facing the Internet through a hub or a
switch. This is the external interface with IP address 192.168.10.1 on Member_A and IP address
192.168.10.2 on Member_B.
Note - This release presents an option to use only two interfaces per member, one external and
one internal, and to run synchronization over the internal interface. We do not recommend this
configuration. It should be used for backup only (see "Synchronizing Connections in the Cluster"
on page 45).

Defining the Cluster Virtual IP Addresses

In the previous illustration, the IP address of the cluster is 192.168.10.100.
The cluster has one external virtual IP address and one internal virtual IP address. The external IP
address is 192.168.10.100, and the internal IP address is 10.10.0.100.

Defining the Synchronization Network

The previous illustration shows a synchronization interface with a unique IP address on each
cluster member - IP 10.0.10.1 on Member_A and IP 10.0.10.2 on Member_B.

Configuring Cluster Addresses on Different Subnets

Only one public IP address is required in a ClusterXL cluster, for the virtual cluster interface that
faces the Internet. Physical IP addresses of all cluster members can be private.
Configuring different subnets ("Cluster IP Addresses on Different Subnets" on page 133) for the
cluster IP addresses and the members IP addresses is useful in order to:
• Configure a cluster to replace one Security Gateway in a pre-configured network, without the
need to allocate new IP addresses to the cluster members.

ClusterXL Administration Guide R80.20 | 34

 High Availability and Load Sharing in ClusterXL

• Allow organizations to use only one public IP address for the ClusterXL Cluster. This saves
public IP addresses.

ClusterXL Administration Guide R80.20 | 35

 High Availability and Load Sharing in ClusterXL

Implementing Planning Considerations

High Availability or Load Sharing
Whether to choose a Load Sharing (Active / Active) or a High Availability (Active / Standby) mode
depends on the need and requirements of the organization.
A High Availability cluster mode ensures fail-safe connectivity for the organization.
A Load Sharing cluster mode provides the additional benefit of increased performance.
See Mode Comparison Table (on page 41).

Choosing the Load Sharing Mode

Load Sharing Multicast mode is an efficient way to handle a high traffic load, because the load is
distributed optimally between all Active cluster members.
However, not all switches can be used for Load Sharing Multicast mode. Load Sharing Multicast
mode associates a multicast Cluster MAC addresses with a unicast Cluster Virtual IP addresses.
This ensures that traffic destined for the cluster is received by all cluster members.
In response to ARP Request packets for Cluster Virtual IP address, cluster members send ARP
Replies that contain a unicast Cluster Virtual IP address and a multicast MAC address. Some
switches do not accept such ARP Replies. For some switches, adding a static ARP entry for the
unicast Cluster Virtual IP address and the multicast MAC address will solve the issue. Other
switches do not accept this type of static ARP entry.
Another consideration is whether your deployment includes networking devices with interfaces
operating in a promiscuous mode. If on the same network segment there exist two such
networking devices, and a ClusterXL in Load Sharing Multicast mode, traffic destined for the
cluster that is generated by one of the networking device could also be processed by the other
networking device.
For these cases, use Load Sharing Unicast mode, which does not require the use of multicast MAC
address for the Cluster Virtual IP addresses.

IP Address Migration
If you wish to provide High Availability or Load Sharing to an existing Security Gateways
configuration, we recommend taking the existing IP addresses from the Active Security Gateway,
and make these the Cluster Virtual IP addresses, when feasible. Doing so will avoid altering
current IPsec endpoint identities, as well keep Hide NAT configurations the same in many cases.

ClusterXL Administration Guide R80.20 | 36

 High Availability and Load Sharing in ClusterXL

ClusterXL Modes
ClusterXL has four working modes. This section briefly describes each mode and its relative
advantages and disadvantages.
• High Availability Mode
• Load Sharing Multicast Mode
• Load Sharing Unicast Mode
Note - Many examples in the section refer to the sample deployment shown in the ClusterXL
example ("Example ClusterXL Topology" on page 33).

High Availability Mode

The ClusterXL High Availability mode provides basic High Availability capabilities in a cluster
environment. This means that the cluster can provide Firewall services even when it encounters a
problem, which on a regular Security Gateway results in a complete loss of connectivity. When
combined with Check Point State Synchronization, ClusterXL High Availability can maintain
connections through failover events, in a user-transparent manner, allowing a flawless
connectivity experience. As a result, High Availability provides a backup mechanism, which
organizations can use to reduce the risk of unexpected downtime, especially in a mission-critical
environment (such as money transactions).
To achieve this, ClusterXL High Availability mode designates one of the cluster members as the
Active member, while the other cluster members remain in Standby mode. The cluster associates
the Virtual IP addresses with the physical MAC addresses of the physical interfaces on the Active
member (by matching the cluster Virtual IP address with the unique MAC address of the
appropriate physical interface). Therefore, all traffic directed at the cluster Virtual IP addresses, is
actually routed (and filtered) by the Active cluster member.
The role of each cluster member is chosen according to its cluster state and cluster member
priority. Cluster member priorities correspond to the order, in which they appear in the Cluster
Members page of the cluster object in SmartConsole. The top-most cluster member has the
highest priority. You can modify this ranking at any time (requires policy installation and causes a
failover).
In addition to its role as a Firewall, the Active cluster member is also responsible for informing
the Standby cluster members of any changes to its cluster state and kernel tables. This keeps the
peer cluster members up-to-date with the current traffic passing through the cluster.
Whenever the the Active cluster member detects a problem that is severe enough to prevent this
cluster member from working correctly, failover occurs in the cluster. One of the Standby cluster
members (the cluster member with the next highest priority) assumes the role of the Active
cluster member. If State Synchronization is enabled, the new Active cluster member recognizes
any open connections and handles them according to their last known state.
Upon the recovery of a failed former Active cluster member with a higher priority, the role of the
Active cluster member may or may not be switched back to that cluster member. This depends
on the cluster object configuration - Maintain current active Cluster Member, or Switch
to higher priority Cluster.
It is important to note that Standby cluster members may encounter problems as well. In this
case, in the event of a cluster failover, the Standby cluster members are not considered for the
role of a new Active cluster member.
ClusterXL Administration Guide R80.20 | 37
 High Availability and Load Sharing in ClusterXL

Example
This scenario describes a connection from a Client computer on the external network to a Web
server behind the Cluster (on the internal network). The cluser of two members is configured in
High Availability mode.
Example topology:
[Client on]
[external network]
{IP 192.168.10.78/24}
{DG 192.168.10.100}
|
|
{VIP 192.168.10.100/24}
/ \
| |
{IP 192.168.10.1/24} {IP 192.168.10.2/24}
| |
{Active} {Standby}
[Member1]-----sync-----[Member2]
| |
{IP 10.10.0.1/24} {IP 10.10.0.2/24}
| |
\ /
{VIP 10.10.0.100/24}
|
|
{DG 10.10.0.100}
{IP 10.10.0.34/24}
[Web server on]
[internal network]

Chain of events:
1. The user requests tries to connect from his Client computer 192.168.10.78 to the Web server
10.10.0.34.
2. The Default Gateway on Client computer is 192.168.10.100 (the cluster Virtual IP address).
3. The Client computer issues an ARP Request for IP 192.168.10.100.
4. The Active cluster member (Member1) handles the ARP Request for IP 192.168.10.100.
5. The Active cluster member sends the ARP Reply with the MAC address of the external
interface, on which the IP address 192.168.10.1 is configured.
6. The Client computer sends the HTTP request packet to the Active cluster member - to the VIP
address 192.168.10.100 and MAC address of the corresponding external interface.
7. The Active cluster member handles the HTTP request packet.
8. The Active cluster member sends the HTTP request packet to the Web server 10.10.0.34.
9. The Web server handles the HTTP request packet.
10. The Web server generates the HTTP response packet.
11. The Default Gateway on Web server computer is 10.10.0.100 (the cluster Virtual IP address).
12. The Web server issues an ARP Request for IP 10.10.0.100.
13. The Active cluster member handles the ARP Request for IP 10.10.0.100.
14. The Active cluster member sends the ARP Reply with the MAC address of the internal
interface, on which the IP address 10.10.0.1 is configured.
15. The Web server sends the HTTP response packet to the Active cluster member - to VIP
address 10.10.0.100 and MAC address of the corresponding internal interface.
ClusterXL Administration Guide R80.20 | 38
 High Availability and Load Sharing in ClusterXL

16. The Active cluster member handles the HTTP response packet.
17. The Active cluster member sends the HTTP response packet to the Client computer
192.168.10.78.
18. From now, all traffic between the Client computer and the Web server is routed through the
Active cluster member (Member1).
19. If a failure occurs on the current Active cluster member (Member1), the cluster fails over.
20. The Standby cluster member (Member2) assumes the role of the Active cluster member.
21. The Standby cluster member sends Gratuitous ARP Requests to both the 192.168.10.x and the
10.10.0.x networks.
These GARP Requests associate the Cluster Virtual IP addresses with the MAC addresses of
the physical interfaces on the new Active cluster member (former Standby cluster member):
• Cluster VIP address 192.168.10.100 and MAC address of the corresponding external
interface, on which the IP address 192.168.10.2 is configured.
• Cluster VIP address 10.10.0.100 and MAC address of the corresponding internal interface,
on which the IP address 10.10.0.2 is configured.
22. From now, all traffic between the Client computer and the Web server is routed through the
new Active cluster member (Member2).
23. The former Active member (Member1) is now considered to be "down". Upon the recovery of a
former Active cluster member, the role of the Active cluster member may or may not be
switched back to that cluster member, depending on the cluster object configuration.

Load Sharing Multicast Mode

Load Sharing lets you distribute network traffic between cluster members. In contrast to High
Availability, where only a single member is active at any given time, all cluster members in a Load
Sharing solution are active. The cluster members decide, which cluster member handles which
packet. The cluster decision function performs this task - examining each packet going through
the cluster, and determining which cluster member should handle it. As a result, a Load Sharing
cluster utilizes all cluster members, which usually leads to an increase in its total throughput.
It is important to understand that ClusterXL Load Sharing, provides a full High Availability solution
as well. When all cluster members are active, traffic is evenly distributed between the cluster
members. In case of a failover event, caused by a problem in one of the cluster members, the
processing of all connections handled by the faulty member is immediately taken over by the other
cluster members.
ClusterXL offers two separate Load Sharing solutions: Multicast and Unicast. The two modes
differ in the way cluster members receive the packets sent to the cluster. This section describes
the Load Sharing Multicast mode.
The Multicast mechanism, which is provided by the Ethernet network layer, allows several
interfaces to be associated with a single physical (MAC) address. Unlike Broadcast, which binds all
interfaces in the same subnet to a single MAC address, Multicast enables grouping within
networks. This means that it is possible to select the interfaces within a single subnet that will
receive packets sent to a given MAC address.
ClusterXL uses the Multicast mechanism to associate the cluster Virtual IP addresses with
Multicast MAC addresses. This makes sure that all packets sent to the cluster, acting as a default
gateway on the network, will reach all cluster members. Each cluster member then decides,
whether it should process the packets or not. This decision is the core of the Load Sharing
mechanism: it has to assure that at least one cluster member will process each packet (so that

ClusterXL Administration Guide R80.20 | 39

 High Availability and Load Sharing in ClusterXL

traffic is not blocked), and that no two cluster members will handle the same packets (so that
traffic is not duplicated).
An additional requirement of the decision function, is to route each connection through a cluster
member, to ensure that packets that belong to a single connection will be processed by the same
cluster member. Unfortunately, this requirement cannot always be enforced, and in some cases,
packets of the same connection will be handled by different cluster members. ClusterXL handles
these situations using its State Synchronization mechanism, which mirrors connections on all
cluster members.

Example
This scenario describes a user logging from the Internet to a Web server behind the cluster that is
configured in Load Sharing Multicast mode.
1. The user requests a connection from 192.168.10.78 (his computer) to 10.10.0.34 (the Web
server).
2. A router on the 192.168.10.x network recognizes 192.168.10.100 (the cluster Virtual IP
address) as the default gateway to the 10.10.0.x network.
3. The router issues an ARP Request for IP 192.168.10.100.
4. One of the Active cluster members processes the ARP Request, and responds with the
Multicast MAC assigned to the cluster Virtual IP address of 192.168.10.100.
5. When the Web server responds to the user requests, it recognizes 10.10.0.100 as its default
gateway to the Internet.
6. The Web server issues an ARP Request for IP 10.10.0.100.
7. One of the Active members processes the ARP Request, and responds with the Multicast MAC
address assigned to the cluster Virtual IP address of 10.10.0.100.
8. All packets sent between the user and the Web server reach every cluster member, which
decide whether to process each packet.
9. When a cluster failover occurs, one of the cluster members goes down. However, traffic still
reaches all of the Active cluster members. Therefore, there is no need to make changes in the
network ARP routing. The only thing that changes, is the cluster decision function, which takes
into account the new state of the cluster members.

Load Sharing Unicast Mode

Load Sharing Unicast mode provides a Load Sharing solution adapted to environments, where
Multicast Ethernet cannot operate. In this mode, a single cluster member, referred to as Pivot, is
associated with the cluster Virtual IP addresses. This Pivot cluster member is the only cluster
member to receive all packets sent to the cluster. The Pivot cluster member is then responsible
for propagating the packets to other cluster members, creating a Load Sharing mechanism.
Distribution of packets is performed by applying a decision function on each packet, the same way
it is done in Load Sharing Multicast mode. The difference is that only one cluster member
performs this selection: any non-Pivot member that receives a forwarded packet will handle it,
without applying the decision function. Note that non-Pivot cluster members are still active,
because they perform routing and Firewall tasks on a share of the traffic (although they do not
perform decisions).
Even though the Pivot cluster member is responsible for the decision process, it still acts as a
Security Gateway that processes packets (for example, the decision it makes, can be to handle a
packet by itself). However, since its additional tasks can be time consuming, it is usually assigned
a smaller share of the total load.
ClusterXL Administration Guide R80.20 | 40
 High Availability and Load Sharing in ClusterXL

When a failure occurs in a non-Pivot member, its handled connections are redistributed between
active non-Pivot cluster members, providing the same High Availability capabilities of High
Availability and Load Sharing Multicast. When the Pivot cluster member encounters a problem, a
regular failover event occurs, and, in addition, another active non-Pivot member assumes the role
of the new Pivot. The Pivot member is always the active member with the highest priority. This
means that when a former Pivot recuperates, it will retain its previous role.

Example
In this scenario, we use a Load Sharing Unicast cluster as the Security Gateway between the end
user computer and the Web server.
1. The user requests a connection from 192.168.10.78 (his computer) to 10.10.0.34 (the Web
server).
2. A router on the 192.168.10.x network recognizes 192.168.10.100 (the cluster Virtual IP
address) as the default gateway to the 10.10.0.x network.
3. The router issues an ARP Request for IP 192.168.10.100.
4. The Pivot cluster member handles the ARP Request, and responds with the MAC address that
corresponds to its own unique IP address of 192.168.10.1.
5. When the Web server responds to the user requests, it recognizes 10.10.0.100 as its default
gateway to the Internet.
6. The Web server issues an ARP Request for IP 10.10.0.100.
7. The Pivot cluster member handles the ARP Request, and responds with the MAC address that
corresponds to its own unique IP address of 10.10.0.1.
8. The user request packet reaches the Pivot cluster member on interface 192.168.10.1.
9. The Pivot cluster member decides that the second non-Pivot cluster member should handle
this packet, and forwards it to 192.168.10.2.
10. The second cluster member recognizes the packet as a forwarded packet, and handles it.
11. Further packets are processed by either the Pivot cluster member, or forwarded and
processed by the non-Pivot cluster member.
12. When a failover occurs from the current Pivot cluster member, the second cluster member
assumes the role of Pivot.
13. The new Pivot member sends Gratuitous ARP Requests to both the 192.168.10.x and the
10.10.0.x networks. These G-ARP requests associate the cluster Virtual IP address of
192.168.10.100 with the MAC address that corresponds to the unique IP address of
192.168.10.2, and the cluster Virtual IP address of 10.10.0.100 with the MAC address that
correspond to the unique IP address of 10.10.0.2.
14. Traffic sent to the cluster is now received by the new Pivot cluster member, and processed by
it (as it is currently the only active cluster member).
15. When the former Pivot cluster member recovers, it re-assumes the role of Pivot, by
associating the cluster Virtual IP addresses with its own unique MAC addresses.

Mode Comparison Table

This table summarizes the similarities and differences between the ClusterXL modes.

High Availability Load Sharing Load Sharing

Multicast Unicast
High Availability Yes Yes Yes

ClusterXL Administration Guide R80.20 | 41

 High Availability and Load Sharing in ClusterXL

High Availability Load Sharing Load Sharing

Multicast Unicast
Load Sharing No Yes Yes

Performance Good Excellent Very Good

Hardware Support All routers Not all routers are All routers
supported

SecureXL Support Yes Yes Yes

State Synchronization Optional Mandatory Mandatory

VLAN Tagging Support Yes Yes Yes

ClusterXL Administration Guide R80.20 | 42

 High Availability and Load Sharing in ClusterXL

Cluster Failover
Failover is a cluster redundancy operation that automatically occurs if a cluster member is not
functional. When this occurs, other cluster members take over for the failed cluster member.
In a High Availability mode:
• If the Active cluster member detects that it can not function as a cluster member, it notifies the
peer Standby cluster members that it must go down. One of the Standby cluster members
(with the next highest priority) will promote itself to the Active state.
• If one of the Standby cluster members stops receiving Cluster Control Protocol (CCP) packets
from the current Active cluster member, that Standby cluster member can assume that the
current Active cluster member failed. As a result, one of the Standby cluster members (with
the next highest priority) will promote itself to the Active state.
• If you do not use State Synchronization in the cluster, existing connections are interrupted
when cluster failover occurs.
• If a cluster member detects that it can not function as a cluster member, it notifies the peer
cluster members that it must go down. Traffic load will be redistributed between the working
cluster members.
• If the cluster members stop receiving Cluster Control Protocol (CCP) packets from one of their
peer cluster member, those working cluster members can assume that their peer cluster
member failed. As a result, traffic load will be redistributed between the working cluster
members.
• Because by design, all cluster members are always synchronized, current connections are not
interrupted when cluster failover occurs.

To tell each cluster member that the other cluster members are alive and functioning, the
ClusterXL Cluster Control Protocol (CCP) maintains a heartbeat between cluster members. If
after a predefined time, no CCP packets are received from a cluster member, it is assumed that
the cluster member is down. As a result, cluster failover can occur.
Note that more than one cluster member may encounter a problem that will result in a cluster
failover event. In cases where all cluster members encounter such problems, ClusterXL will try to
choose a single cluster member to continue operating. The state of the chosen member will be
reported as Active Attention. This situation lasts until another cluster member fully recovers. For
example, if a cross cable connecting the sync interfaces on cluster members malfunctions, both
cluster members will detect an interface problem. One of them will change to the Down state, and
the other to Active Attention state.

When Does a Failover Occur?

A failover takes place when one of the following occurs in a cluster:
• Any Critical Device (on page 10) reports its state as problem (see Monitoring Critical Devices
(on page 176)).
For example, fwd process failed, or Security Policy is uninstalled on a cluster member.
• Cluster members do not receive Cluster Control Protocol (CCP) packets from their peer
cluster member.

ClusterXL Administration Guide R80.20 | 43

 High Availability and Load Sharing in ClusterXL

For more on failovers, see sk62570 http://supportcontent.checkpoint.com/solutions?id=sk62570.

What Happens When a Cluster Member Recovers?

In a High Availability mode:
• If cluster object is configured as Maintain current active Cluster Member, it means any
cluster member that becomes Active, remains Active.
If the cluster member with highest priority fails, cluster failover occurs. A cluster member with
the next highest priority becomes Active.
If the cluster member with highest priority recovers, cluster failover does not occurs again,
and that cluster member becomes Standby.
• If cluster object is configured as Switch to higher priority Cluster Member, it means that
cluster member with the highest priority always has to be Active.
Cluster member with the highest priority is the cluster member that appears at the top of the
list in Cluster object > Cluster Members pane.
If the cluster member with the highest priority fails, cluster failover occurs. A peer cluster
member in Standby state, with the next highest priority, becomes Active.
If the cluster member with the highest priority recovers, cluster failover occurs again. The
cluster member with the highest priority becomes Active again. The cluster member with the
next highest priority that was Active, returns to the Standby state.
• When the failed cluster member recovers, all connections are redistributed between all Active
cluster members.

How a Recovered Cluster Member Obtains the Security Policy

The Administrator installs the Security Policy on the cluster object, rather than separately on
individual cluster members. The policy is automatically installed on all cluster members. The
policy is sent to the IP addresses defined in the General Properties page of the cluster member
object.
When a failed cluster member recovers, first it tries to fetch a policy from one of the peer Active
cluster members. The assumption is that the other cluster members have a more up to date
policy. If fetching a policy from peer cluster member fails, the recovered cluster member
compares its own local policy to the policy on its Security Management Server or Domain
Management Server. If the policy on the Security Management Server or Domain Management
Server is more up to date than the one on the recovered cluster member, the policy is fetched
from the Security Management Server or Domain Management Server. If the cluster member
does not have a local policy, it retrieves one from the Security Management Server or Domain
Management Server. This ensures that all cluster members use the same policy at any given
moment.

ClusterXL Administration Guide R80.20 | 44

CHAPTE R 3

Synchronizing Connections in the

Cluster
In This Section:
The Check Point State Synchronization Solution ........................................................45

The Check Point State Synchronization Solution

A failure of a firewall results in an immediate loss of active connections in and out of the
organization. Many of these connections, such as financial transactions, may be mission critical,
and losing them will result in the loss of critical data. ClusterXL supplies an infrastructure that
ensures that no data is lost in case of a failure, by making sure each cluster member is aware of
the connections going through the other members. Passing information about connections and
other Security Gateway states between the cluster members is called State Synchronization.
Every IP-based service (including TCP and UDP) recognized by the Security Gateway, is
synchronized.
Members of a ClusterXL in Load Sharing mode must be synchronized.
Members of a ClusterXL High Availability mode do not have to be synchronized. Although if they
are not, current connections are interrupted during cluster failover.

The Synchronization Network

The Synchronization Network is used to transfer synchronization information about connections
and other Security Gateway states between cluster members.
The synchronization network carries the most sensitive Security Policy information in the
organization. Therefore, it is critical that you protect it against both malicious and unintentional
threats.
We recommend that you secure the synchronization interfaces using one of the following
strategies:
• Use a dedicated synchronization network.
• Connecting the physical network interfaces of the cluster members directly using a
cross-cable. In a cluster with three or more members, use a dedicated hub or switch.
Notes:
• See Supported Topologies for Synchronization Network (on page 24).
• You can synchronize members across a WAN. To do this, do the steps in Synchronizing
Clusters on a WAN (see "Synchronizing Clusters on a Wide Area Network" on page 53).
• In ClusterXL, the synchronization network is supported on the lowest VLAN tag of a VLAN
interface. For example, if three VLANs with tags 10, 20 and 30 are configured on interface eth1,
only interface eth1.10 may be used for synchronization.

ClusterXL Administration Guide R80.20 | 45

 Synchronizing Connections in the Cluster

How State Synchronization Works

Synchronization works in two modes:
• Full Sync transfers all Security Gateway kernel table information from one cluster member to
another. The fwd daemon handles the Full Sync using an encrypted TCP connection on port
256.
• Delta Sync transfers changes in the kernel tables between cluster members. The Security
Gateway kernel handles the Delta Sync using UDP connections on port 8116.
Full Sync is used for initial transfers of state information, when a cluster member joins the
cluster. If a cluster member is brought up after being down, it performs the Full Sync with the
Active peer cluster member(s). After all cluster members are synchronized, only updates are
transferred using the Delta Sync, because the Delta Sync is quicker than the Full Sync.
State Synchronization traffic typically makes up around 90% of all Cluster Control Protocol (CCP)
traffic. Cluster members distinguish the State Synchronization packets from the rest of CCP
traffic based on the opcode in the UDP data header.
Note - You can change (see "Synchronizing Clusters on a Wide Area Network" on page 53) the
source MAC address for CCP packets. See sk25977
http://supportcontent.checkpoint.com/solutions?id=sk25977.

Configuring Services to Synchronize After a Delay

Some TCP services (for example, HTTP) are characterized by connections with a very short
duration. There is no point to synchronize these connections, because every synchronized
connection consumes resources on cluster members, and the connection is likely to have finished
by the time a cluster failover occurs.
For short-lived services, you can use the Delayed Notifications feature to delay telling the cluster
member about a connection, so that the connection is only synchronized, if it still exists X seconds
after the connection was initiated. The Delayed Notifications feature requires SecureXL to be
enabled on all cluster members.

Procedure:
1. In SmartConsole, click Objects > Object Explorer.
2. In the left tree, click the small arrow on the left of the Services to expand this category
3. In the left tree, select TCP.
4. Search for the applicable TCP service.
5. Double-click the applicable TCP service.
6. In the TCP service properties window, click Advanced page.
7. At the top, select Override default settings (on Domain Management Server: Override global
domain settings).
8. At the bottom, in the Cluster and synchronization section, select Start synchronizing and
enter the desired value.
Important - This change applies to all policies that use this service.
9. Click OK.
10. Close the Object Explorer.
11. Publish the session.
12. Install the Access Control Policy on the cluster object.

ClusterXL Administration Guide R80.20 | 46

 Synchronizing Connections in the Cluster

Note - The Delayed Notifications setting in the service object is ignored, if Connection Templates
are not offloaded by the Firewall to SecureXL. For additional information about the Connection
Templates, see the R80.20 Performance Tuning Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Performanc
eTuning_AdminGuide/html_frameset.htm.

Configuring Services not to Synchronize

Synchronization of connections incurs a performance cost. Not all connections that go through a
cluster must be synchronized:
• Protocols that run solely between cluster members need not be synchronized. Although, you
can synchronize them, you go not gain any benefit. This synchronization information does not
help during a cluster failover.
• You can decide not to synchronize TCP, UDP and other service types. By default, cluster
members synchronize all these services.
• The VRRP and the IGMP protocols are not synchronized by default (but you can choose to turn
on synchronization for these protocols).
• Broadcast and multicast connections are not, and cannot be, synchronized.
You may choose not to synchronize a service if these conditions are true:
• A significant amount of traffic goes through the cluster. Not synchronizing the service reduces
the amount of synchronization traffic, and so enhances cluster performance.
• The service typically opens short connections, whose loss may not be noticed. DNS (over UDP)
and HTTP are typically responsible for most connections, frequently have short life, and
inherent recoverability in the application level. Services that open long connections, such as
FTP, should always be synchronized.
• Configurations that ensure bi-directional stickiness for all connections, do not require
synchronization to operate (only to maintain High Availability). Such configurations include:
• Any cluster in High Availability mode (for example, ClusterXL High Availability, or VRRP on
Gaia).
• ClusterXL in a Load Sharing mode with clear connections (no VPN, or Static NAT).
• VPN and Static NAT connections passing through a ClusterXL cluster in a Load Sharing
mode (either multicast, or unicast) may not maintain bi-directional stickiness. State
Synchronization must be turned on for such environments.
You can have a synchronized service and a non-synchronized definition of a service, and use them
selectively in the Rule Base. For more information, see the R80.20 Security Management
Administration Guide http://downloads.checkpoint.com/dc/download.htm?ID=65846.

To configure a service not to synchronize in a cluster

1. In SmartConsole, click Objects > Object Explorer.
2. In the left tree, select Services.
3. Double-click the applicable existing synchronized service, for which you need to create a
non-synchronized counterpart service.
4. Write down all the settings from both the General and Advanced pages, and click OK.
5. Click New > Service > desired service type.
6. Enter the desired name that distinguishes the new non-synchronized counterpart service from
the existing synchronized service.
ClusterXL Administration Guide R80.20 | 47
 Synchronizing Connections in the Cluster

7. On the General page, configure the same settings as in the existing synchronized service.
8. On the Advanced page:
a) Configure the same settings as in the existing synchronized service.
b) In the Cluster and synchronization section, clear Synchronize connections if State
Synchronization is enabled on the cluster.
Important - This change applies to all policies that use this service.
9. Click OK.
10. Close the Object Explorer.
11. Use the synchronized service and the non-synchronized counterpart service in the applicable
rules in the applicable Access Control Policies.
12. Publish the session.
13. Install the Access Control Policy on the cluster object.

ClusterXL Administration Guide R80.20 | 48

 Synchronizing Connections in the Cluster

Sticky Connections
Introduction to Sticky Connections
A connection is considered sticky, when all of its packets are handled, in either direction, by a
single cluster member.
This is the case in High Availability mode, where all connections are routed through the same
cluster member, and hence, are sticky.
This is also the case in Load Sharing mode, when there are no VPN peers, Static NAT rules, or SIP
traffic.
In Load Sharing mode, there are cases, where it is necessary to ensure that a connection that
starts on a specific cluster member will continue to be processed by the same cluster member in
both directions. Certain connections can be made sticky by enabling the Sticky Decision Function
in the cluster object in SmartConsole.
The Check Point Cluster Correction Layer (CCL) deals with asymmetric connections in Check
Point cluster.

VPN Tunnels with 3rd Party Peers and Load Sharing

Check Point provides interoperability with third-party vendor gateways by enabling them to peer
with Check Point gateways. A special case occurs when certain third-party peers (for example,
Microsoft LT2P, Cisco gateways) attempt to establish VPN tunnels with ClusterXL in Load Sharing
mode. These VPN peers are limited in their ability to store SAs, which means that a VPN session
that begins on one cluster member and, due to Load Sharing, is routed on the return trip through
another cluster member, is unrecognized and dropped.

Item Description
1 Internal network

2 Switch for internal network

3 ClusterXL in Load Sharing mode - Cluster Members "A" and "B"

4 Switch for external networks

5 Internet

6 3rd party peer VPN gateway

7 3rd party peer laptop with VPN client

ClusterXL Administration Guide R80.20 | 49

 Synchronizing Connections in the Cluster

In this scenario:
• A third-party peer (gateway or client) attempts to create a VPN tunnel.
• Cluster Members "A" and "B" belong to a ClusterXL in Load Sharing mode.
The third-party peers, lacking the ability to store more than one set of SAs, cannot negotiate a
VPN tunnel with multiple cluster members, and therefore the cluster member cannot complete
the routing transaction.
This issue is resolved for certain third-party peers or gateways that can save only one set of SAs
by making the connection sticky. The Cluster Correction Layer (CCL) makes sure that a single
cluster member processes all VPN sessions, initiated by the same third-party gateway.

Third-Party Gateways in Hub and Spoke VPN Deployments

Another case, where Load Sharing mode requires the connection stickiness, which the the Cluster
Correction Layer (CCL) provides, is when integrating certain third-party gateways into a Hub and
Spoke deployment. Without the ability to store more than one set of SAs, a third-party gateway
must maintain its VPN tunnels on a single cluster member in order to avoid duplicate SAs.

Item Description
1 Security Gateway - Cluster member A

2 Security Gateway - Cluster member B

3 Switch for external networks

4 Internet

5 Gateway - Spoke A

6 Gateway - Spoke B

In this sample deployment:

• The intent of this deployment is to enable hosts that reside behind Spoke A to communicate
with hosts behind Spoke B.
• The ClusterXL in Load Sharing mode, is composed of Cluster Members "A" and "B", and serves
as a VPN Hub.
• Spoke A is a third-party gateway, and is connected by a VPN tunnel that passes through the
VPN Hub to Spoke B.
• Spoke B can be either another third-party gateway, or a Check Point Security Gateway.

ClusterXL Administration Guide R80.20 | 50

 Synchronizing Connections in the Cluster

Spokes A and B must be set to always communicate using the same cluster member. The Cluster
Correction Layer (CCL) solves half of this problem, in that a single cluster member processes all
VPN sessions initiated by either third-party gateway.
To make sure that all communications between Spokes A and B are always using the same cluster
member, you must make some changes to the applicable user.def file (see sk98239
http://supportcontent.checkpoint.com/solutions?id=sk98239). This second step ensures that both
third-party gateways always connect to the same cluster member (see "Configuring a Third-Party
Gateway in a Hub and Spoke VPN Deployment" on page 51).

Configuring a Third-Party Gateway in a Hub and Spoke VPN Deployment

To configure a third-party gateway as a spoke in a Hub and Spoke VPN deployment, perform the
following on the Security Management Server:
1. Create a Tunnel Group to handle traffic from specific peers. Edit the applicable user.def file
(see sk98239 http://supportcontent.checkpoint.com/solutions?id=sk98239), and add a line
similar to the following:
all@{member1,member2} vpn_sticky_gws = {<10.10.10.1;1>, <20.20.20.1;1>};
The elements of this configuration are as follows:
• all - All cluster interfaces
• member1,member2 - Names of cluster members in SmartConsole
• vpn_sticky_gws - Table name
• 10.10.10.1 - IP address of Spoke A
• 20.20.20.1 - IP address of Spoke B
• ;1 - Tunnel Group Identifier, which indicates that the traffic from these IP addresses should
be handled by the same cluster member
2. Other VPN peers can be added to the Tunnel Group by including their IP addresses in the same
format as shown above. To continue with the example above, adding Spoke C would look like
this:
all@{member1,member2} vpn_sticky_gws = {<10.10.10.1;1>, <20.20.20.1;1>,
<30.30.30.1;1>};
The Tunnel Group Identifier ;1 stays the same, which means that the listed peers will always
connect through the same cluster member.
Note - More tunnel groups than cluster members may be defined.
This procedure turns off Load Sharing for the affected connections. If the implementation is to
connect multiple sets of third-party Security Gateways one to another, a form of Load Sharing
can be accomplished by setting Security Gateway pairs to work in tandem with specific cluster
members. For instance, to set up a connection between two other spokes (C and D), simply add
their IP addresses to the line and replace the Tunnel Group Identifier ;1 with ;2. The line would
then look something like this:
all@{member1,member2} vpn_sticky_gws = {<10.10.10.1;1>, <20.20.20.1;1>,
<192.168.15.5;2>, <192.168.1.4;2>,};
Note that there are now two peer identifiers: ;1 and ;2. Spokes A and B will now connect
through one cluster member, and Spokes C and D through another.
Note - The tunnel groups are shared between active cluster members. In case of a change in
cluster state (for example, failover or cluster member attach/detach), the reassignment is
performed according to the new state.

ClusterXL Administration Guide R80.20 | 51

 Synchronizing Connections in the Cluster

Non-Sticky Connections
A connection is called sticky if a single cluster member handles all packets of that connection. In
a non-sticky connection, the response packet of a connection returns through a different cluster
member than the original request packet.
The cluster synchronization mechanism knows how to handle non-sticky connections properly. In
a non-sticky connection, a cluster member can receive an out-of-state packet, which Firewall
normally drops because it poses a security risk.
In Load Sharing configurations, all cluster members are active. In Static NAT and encrypted
connections, the source and destination IP addresses change. Therefore, Static NAT and
encrypted connections through a Load Sharing cluster may be non-sticky. Non-stickiness may
also occur with Hide NAT, but ClusterXL has a mechanism to make it sticky.
In High Availability configurations, all packets reach the Active member, so all connections are
sticky. If failover occurs during connection establishment, the connection is lost, but
synchronization can be performed later.
If the other members do not know about a non-sticky connection, the packet will be out-of-state,
and the connection will be dropped for security reasons. However, the Synchronization
mechanism knows how to inform other members of the connection. The Synchronization
mechanism thereby prevents out-of-state packets in valid, but non-sticky connections, so that
these non-sticky connections are allowed.
Non-sticky connections will also occur if the network Administrator has configured asymmetric
routing, where a reply packet returns through a different Security Gateway than the original
packet.

Non-Sticky Connection Example: TCP 3-Way Handshake

The 3-way handshake that initiates all TCP connections can very commonly lead to a non-sticky
(often called asymmetric routing) connection. This diagram shows a sample scenario:

Item Description
1 Client

2 Security Gateway - Cluster member A

3 Security Gateway - Cluster member B

4 Server

The client initiates a connection by sending a SYN packet to the server. The SYN passes through
cluster member A, but the SYN/ACK reply returns through cluster member B. This is a non-sticky

ClusterXL Administration Guide R80.20 | 52

 Synchronizing Connections in the Cluster

connection, because the reply packet returns through a different Security Gateway than the
original packet.
The synchronization network notifies cluster member B. If cluster member B is updated before
the SYN/ACK packet sent by the server reaches it, the connection is handled normally. If, however,
synchronization is delayed, and the SYN/ACK packet is received on cluster member B before the
SYN flag has been updated, then the Security Gateway treats the SYN/ACK packet as out-of-state,
and drops the connection.
You can configure enhanced 3-Way TCP Handshake (see "Enhanced 3-Way TCP Handshake
Enforcement" on page 132) enforcement to address this issue.

Synchronizing Non-Sticky Connections

The synchronization mechanism prevents out-of-state packets in valid, but non-sticky
connections. The way it does this is best illustrated with reference to the 3-way handshake that
initiates all TCP data connections. The 3-way handshake proceeds as follows:
1. SYN (client to server)
2. SYN/ACK (server to client)
3. ACK (client to server)
4. Data (client to server)
To prevent out-of-state packets, the following sequence (called "Flush and Ack") occurs
1. Cluster member receives first packet (SYN) of a connection.
2. Suspects that it is non-sticky.
3. Hold the SYN packet.
4. Send the pending synchronization updates to all cluster members (including all changes
relating to this packet).
5. Wait for all the other cluster members to acknowledge the information in the sync packet.
6. Release held SYN packet.
7. All cluster members are ready for the SYN-ACK.

Synchronizing Clusters on a Wide Area Network

Organizations sometimes need to locate cluster members in geographical locations that are
distant from each other. A typical example is a replicated Data Center, whose locations are widely
separated for disaster recovery purposes. In such a configuration, it is clearly impractical to use a
cross cable for the synchronization network.
The synchronization network can be spread over remote sites, which makes it easier to deploy
geographically distributed clustering. There are two limitations to this capability:
1. The synchronization network must guarantee no more than 100ms latency and no more than
5% packet loss.
2. The synchronization network may only include Layer 2 networking devices - switches and
hubs. No Layer 3 routers are allowed on the synchronization network, because routers drop
Cluster Control Protocol (CCP) packets.
You can monitor and troubleshoot geographically distributed clusters using the command line
interface.

ClusterXL Administration Guide R80.20 | 53

 Synchronizing Connections in the Cluster

Synchronized Cluster Restrictions

The following restrictions apply when you synchronize cluster members:
• The use of more than one dedicated physical interface for synchronization redundancy is not
supported. You can use Bonding ("Sync Redundancy" on page 113) for synchronization
interface redundancy.
Synchronization interface redundancy is not supported for VRRP clusters. See sk92804
http://supportcontent.checkpoint.com/solutions?id=sk92804.
• All cluster members must run on identically configured hardware platforms.
• If a cluster member goes down, user-authenticated connections through that member are
lost. Other cluster members cannot restore the connection. Client-authenticated or
session-authenticated connections are maintained.
The reason for these restrictions is that the user authentication state is maintained by a
process on the Security Gateway. It cannot be synchronized on cluster members in the same
way as kernel data is synchronized. However, the states of Session Authentication and Client
Authentication are saved in kernel tables, and can be synchronized.
• The connection statutes that use system resources cannot be synchronized for the same
reason that user-authenticated connections cannot be synchronized.
• Accounting information for connections is accumulated on each cluster member, sent to the
Security Management Server, and aggregated. In the event of a cluster failover, the accounting
information that is not yet sent to the Security Management Server, is lost. To minimize this
risk, you can reduce the time interval when accounting information is sent. To do this, in the
cluster object > Logs > Additional Logging pane, set a lower value for the Update Account Log
every attribute.

ClusterXL Administration Guide R80.20 | 54

CHAPTE R 4

Configuring ClusterXL
In This Section:
Installing Cluster Members .........................................................................................55
Configuring Routing for Client Computers .................................................................56
Selecting the CCP Transport Mode on the Cluster Members ....................................57
Configuring the Cluster Object and Members ............................................................59
Configuring a ClusterXL in Bridge Mode .....................................................................67

This procedure describes how to configure the Load Sharing Multicast, Load Sharing Unicast, and
High Availability modes from scratch. Their configuration is identical, apart from the mode
selection in SmartConsole Cluster object or Cluster creation wizard.

Installing Cluster Members

Important - See Hardware Requirements for Cluster Members (on page 20) and Software
Requirements for Cluster Members (on page 21).

To install new cluster members for ClusterXL:

1. Install and configure Check Point Security Gateways that will be configured as cluster
members.
• For installation and initial configuration procedures, see the R80.20 Installation and
Upgrade Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Insta
llation_and_Upgrade_Guide/html_frameset.htm.
• During the Gaia First Time Configuration Wizard, enable ClusterXL.
• You must run cpconfig from the command line and select Enable cluster membership
for this gateway. This change requires reboot.
2. Using Gaia Portal or Gaia Clish, define an IP address on each interface on all cluster
members.
Note - Do not define IPv6 addresses for synchronization interfaces.
3. On cluster members that will participate in a VPN community, you must synchronize clocks
accurately to within one second of each other. If these cluster members are constantly up and
running, it is usually enough to set the time once. More reliable synchronization can be
achieved using NTP or some other time synchronization services supplied by the operating
system.
4. Connect the cluster members to each other and to the networks through switches. For the
synchronization interfaces, you can use a cross cable, or a dedicated switch. Make sure that
each network (internal, external, synchronization, DMZ, and so on) is configured on a separate
VLAN, or network segment.
Note - You can also perform synchronization over a WAN (see "Synchronizing Clusters on a Wide
Area Network" on page 53).

ClusterXL Administration Guide R80.20 | 55

 Configuring ClusterXL

Configuring Routing for Client Computers

Example topology:
[internal network 10.10.2.0/24] --- (VIP 10.10.2.100/24) [Cluster] (VIP 192.168.2.100/24) ---
[external network 192.168.2.0/24]

To configure routing for client computers:

1. Computers on the internal network 10.10.2.0/24 should be configured with Default Gateway IP
10.10.2.100
2. Computers on the external network 192.168.2.0/24 should be configured with Default Gateway
IP 192.168.2.100
3. For Proxy ARP configuration, see sk30197
http://supportcontent.checkpoint.com/solutions?id=sk30197
4. Also see Configuring Cluster Addresses on Different Subnets (on page 133)

ClusterXL Administration Guide R80.20 | 56

 Configuring ClusterXL

Selecting the CCP Transport Mode on the Cluster

Members
In R80.20, the Cluster Control Protocol (CCP) has four modes:

Mode Description
Automatic The CCP mode changes automatically between Multicast, Broadcast, and Unicast
to find the optimized CCP mode according to network state.
These CCP mode changes are independent for individual interfaces.
This CCP mode prevents unnecessary cluster failovers and interface state
changes when CCP packets are not received because of networking issues.

Unicast This CCP mode is applicable to clusters of two members only.

In this CCP mode, the CCP packets are sent to a unicast Layer 2 destination MAC
address of a peer cluster member.
This CCP mode decreases unneeded network traffic and overcomes network
barriers (switches that does not pass multicast traffic, and so on).

Multicast In this CCP mode, the CCP packets are sent to a multicast Layer 2 destination
MAC address (01:00:5E:xx:yy:zz). See sk25977
http://supportcontent.checkpoint.com/solutions?id=sk25977.
This is the default CCP mode for non-Sync interfaces.

Broadcast In this CCP mode, the CCP packets are sent to a broadcast Layer 2 MAC address
(FF:FF:FF:FF:FF:FF). See sk25977
http://supportcontent.checkpoint.com/solutions?id=sk25977 and sk36644
http://supportcontent.checkpoint.com/solutions?id=sk36644.
This is the default CCP mode for Sync interface.
This is the only supported CCP mode on Bridge interfaces.
Use this CCP mode if the connecting switches do not pass CCP multicast packets.

To set the CCP mode:

• In Gaia Clish, run:
set cluster ccp {auto|unicast|multicast|broadcast}
• In Expert mode, run:
cphaconf set_ccp {auto|unicast|multicast|broadcast}
This configuration applies immediately and survives reboot.

To monitor the CCP mode:

• In Gaia Clish, run:
show cluster interfaces virtual
• In Expert mode, run:
cphaprob -a if

ClusterXL Administration Guide R80.20 | 57

 Configuring ClusterXL

Example output:
[Expert@Member2:0]# cphaprob -a if

CCP mode: Automatic

Required interfaces: 3
Required secured interfaces: 1

eth0 UP non sync(non secured), multicast

eth1 UP sync(secured), unicast
eth2 UP non sync(non secured), multicast

Virtual cluster interfaces: 2

eth0 192.168.3.55 VMAC address: 00:1C:7F:00:36:70

eth2 192.250.3.55 VMAC address: 00:1C:7F:00:36:70

[Expert@Member2:0]#
[Expert@Member2:0]# cphaconf set_ccp auto
[Expert@Member2:0]#
[Expert@Member2:0]# cat $FW_BOOT_DIR/ha_boot.conf
ha_installed 1
ccp_mode automatic
[Expert@Member2:0]#
[Expert@Member2:0]# cphaconf set_ccp broadcast
[Expert@Member2:0]#
[Expert@Member2:0]# cphaprob -a if

CCP mode: Manual (Broadcast)

Required interfaces: 3
Required secured interfaces: 1

eth0 UP non sync(non secured), broadcast

eth1 UP sync(secured), broadcast
eth2 UP non sync(non secured), broadcast

Virtual cluster interfaces: 2

eth0 192.168.3.55 VMAC address: 00:1C:7F:00:36:70

eth2 192.250.3.55 VMAC address: 00:1C:7F:00:36:70

[Expert@Member2:0]#
[Expert@Member2:0]# cat $FW_BOOT_DIR/ha_boot.conf
ha_installed 1
ccp_mode broadcast
[Expert@Member2:0]#
[Expert@Member2:0]# grep ccp /config/active
cluster:ccp:broadcast t
[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 58

 Configuring ClusterXL

Configuring the Cluster Object and Members

Overview
You can use one of these procedures to define a cluster object and its members:
• Simple Mode (Wizard) ("Using the Wizard Mode" on page 59) - Lets you quickly create a new
cluster and configure some basic cluster properties:
• Cluster properties and Virtual IP addresses
• Properties and Topology of cluster members
• Synchronization interfaces and IP addresses
• Classic Mode ("Using the Manual Configuration" on page 62) - Opens the Cluster Gateway
Properties window, where you manually create a cluster and configure its properties.
The Cluster Gateway Properties window lets you:
• Manually create a new cluster
• Enable and configure Software Blades for the cluster
• Configure other cluster properties that you cannot configure with the wizards
• Change the properties of an existing cluster

Using the Wizard Mode

This version includes two wizards:
• Check Point Appliances and Open Servers
• Check Point Small Office Appliances

Wizard for Check Point Appliances or Open Servers

The Cluster Wizard is recommended for all Check Point Appliances (for example, 23800) except
Small Office, and for Open Server platforms.

To create a new cluster using Wizard Mode:

1. In SmartConsole, click Objects menu > More object types > Network Object > Gateways and
Servers > Cluster > New Cluster.
2. In Check Point Security Gateway Cluster Creation window, click Wizard Mode.
3. In the Cluster General Properties window:
a) In the Cluster Name field, enter unique name for the cluster object.
b) In the Cluster IPv4 Address, enter the unique Cluster Virtual IPv4 addresses for this
cluster. This is the main IPv4 address of the cluster object.
c) In the Cluster IPv6 Address, enter the unique Cluster Virtual IPv6 addresses for this
cluster. This is the main IPv6 address of the cluster object.
Important - You must define a corresponding IPv4 address for every IPv6 address. This
release does not support pure IPv6 addresses.
d) In the Choose the Cluster's Solution field, select the applicable option and click Next:
 Check Point ClusterXL and then select High Availability or Load Sharing

ClusterXL Administration Guide R80.20 | 59

 Configuring ClusterXL

 Gaia VRRP
4. In the Cluster member's properties window do these steps for each cluster member and click
Next:
We assume you create a new cluster object from the scratch.
a) Click Add > New Cluster Member to configure each cluster member.
b) In the Cluster Name field, enter unique name for the cluster member object.
c) In the Cluster IPv4 Address, enter the unique Cluster Virtual IPv4 addresses for this
cluster member. This is the main IPv4 address of the cluster member object.
d) In the Cluster IPv6 Address, enter the unique Cluster Virtual IPv6 addresses for this
cluster member. This is the main IPv6 address of the cluster member object.
Important - You must define a corresponding IPv4 address for every IPv6 address. This
release does not support pure IPv6 addresses.
e) In the Activation Key and Confirm Activation Key fields, enter a one-time password that
you entered in First Time Configuration Wizard during the installation of this cluster
member.
f) Click Initialize.
Management Server will try to establish SIC with each cluster member. The Trust State
field should show Trust established.
g) Click OK.
5. In the Cluster Topology window, define a network type (network role) for each cluster
interface and define the Cluster Virtual IP addresses.
The wizard automatically calculates the subnet for each cluster network and assigns it to the
applicable interface on each cluster member. The calculated subnet shows in the upper
section of the window.
The available network objectives are:
• Cluster Interface - A cluster interface that connects to an internal or external network.
Enter the Cluster Virtual IP addresses for each network (internal or external). Also see
Cluster IP Addresses on Different Subnets (on page 133).
• Cluster Sync Interface - A cluster Synchronization interface. In Load Sharing mode, you
must define a synchronization interface.
 For Check Point Appliances (for example, 23800) or Open Servers: The Synchronization
Network is supported only on the lowest VLAN tag of a VLAN interface.
 For Small Office appliances (for example, 1100, 1200R, 1400): You can only select 1st
sync and only for the LAN2/SYNC interface. You cannot configure VLANs on the
Synchronization interface.
Important Note - Make sure that you do not define IPv6 address for sync interfaces. The
wizard does not let you define an interface with an IPv6 address as a sync interface.
• Private - An interface that is not part of the cluster. Cluster does not monitor this interface.
Cluster failover does not occur if a fault occurs on this interface. This option is
recommended for the dedicated management interface.
Click Next.
6. In the Cluster Definition Wizard Complete window, click Finish.

ClusterXL Administration Guide R80.20 | 60

 Configuring ClusterXL

After you complete the wizard, we recommend that you open the cluster object and complete the
configuration:
• Define Anti-Spoofing properties for each interface
• Change the Topology settings for each interface, if necessary
• Define the Network Type
• Configure other Software Blades, features and properties as necessary

Wizard for Small Office Appliances

The Small Office Cluster wizard is recommended for these Centrally Managed Check Point
appliances:
• 1100 appliances
• 1200R appliances
• 1400 appliances

To create a new Small Office cluster using Wizard Mode:

1. In SmartConsole, click Objects menu > More object types > Network Object > Gateways and
Servers > Cluster > New Small Office Cluster.
2. In Check Point Security Gateway Cluster Creation window, click Wizard Mode.
3. In the Cluster General Properties window:
a) Enter a unique name for the cluster object.
b) Select the correct hardware type.
c) Click Next.
4. In the Cluster Members window:
a) Enter the member name and IPv4 addresses for each cluster member.
b) Enter the one-time password for SIC trust.
c) Click Next.
d) Management Server will try to establish SIC with the Primary cluster member.
5. In the Configure WAN Interface page, configure the Cluster Virtual IPv4 address.
6. Define the Cluster Virtual IPv4 addresses for the other cluster interfaces.
7. Click Next, and then Finish to complete the wizard.
After you complete the wizard, we recommend that you open the cluster object and complete the
configuration:
• Define Anti-Spoofing properties for each interface
• Change the Topology settings for each interface, if necessary
• Define the Network Type
• Configure other Software Blades, features and properties as necessary

ClusterXL Administration Guide R80.20 | 61

 Configuring ClusterXL

Using the Manual Configuration

The Cluster Gateway Properties window contains many different ClusterXL properties, as well as
other properties related to Security Gateway and Software Blades functionality. This section
includes only the properties and procedures directly related to ClusterXL.

Configuration Steps
Configuring General Properties...................................................................................62
Adding a New Cluster Member to the Cluster Object ................................................62
Adding an Existing Security Gateway as a Cluster Member to the Cluster Object ...63
Deleting a Cluster Member from Cluster Object ........................................................64
Working with Cluster Topology ....................................................................................64
Completing the Cluster Definition ...............................................................................65
Changing the Synchronization Interface .....................................................................65

Configuring General Properties

To configure the general properties of a cluster:
1. In the Name field, enter a unique name for this cluster object.
2. In the IPv4 Address field, enter the unique Cluster Virtual IPv4 addresses for this cluster. This
is the main IPv4 address of the cluster object.
3. In the Cluster IPv6 Address field, enter the unique Cluster Virtual IPv6 addresses for this
cluster. This is the main IPv6 address of the cluster object.
Important - You must define a corresponding IPv4 address for every IPv6 address. This release
does not support pure IPv6 addresses.
4. In the Hardware field, select the correct hardware platform.
5. In the Version field, select the correct Check Point version.
6. In the OS field, select the correct operating system.
7. Configure the desired cluster type:
• To work with ClusterXL, select ClusterXL.
Go to ClusterXL and VRRP pane and configure the applicable settings.
• To work with VRRP on Gaia cluster, clear ClusterXL.
Go to VRRP pane and configure the applicable settings.
8. Enable other Network Security Software Blades as necessary.

Adding a New Cluster Member to the Cluster Object

To add a new cluster member to the Cluster object:
1. In SmartConsole, open the cluster object.
2. Go to the Cluster Members page.
3. Click Add > New Cluster Member.
The Cluster Members Properties window opens.
4. Click the General tab.
5. In the Name field, enter a cluster member name.

ClusterXL Administration Guide R80.20 | 62

 Configuring ClusterXL

6. In the IPv4 Address field, enter a physical IPv4 addresses.

The Security Management Server must be able to connect to the cluster member at this IPv4
address. This IPv4 address can be an internal, or external. You can use a dedicated
management interface on the cluster member.
Important - You must define a corresponding IPv4 address for every IPv6 address. This release
does not support the configuration of only IPv6 addresses.
7. In the IPv6 Address field, enter a physical IPv6 address, if you need to use IPv6.
The Security Management Server must be able to connect to the cluster member at this IPv6
address. This IPv6 address can be an internal, or external. You can use a dedicated
management interface on the cluster member.
Important - You must define a corresponding IPv4 address for every IPv6 address. This release
does not support the configuration of only IPv6 addresses.
8. Click Communication, and initialize Secure Internal Communication (SIC) trust.
Enter the same key you entered during First Time Configuration Wizard on each cluster
member.
9. Click the NAT tab to configure the applicable NAT settings.
10. Click the VPN tab to configure the applicable VPN settings.
11. Click OK.

Adding an Existing Security Gateway as a Cluster Member to the Cluster

Object
To add an existing Security Gateway as a cluster member to the Cluster object:
Before doing these steps, we recommend exporting a complete management database with
migrate export command.
1. In SmartConsole, open the cluster object.
2. Go to the Cluster Members page.
3. Click Add > Add Existing Gateway.
4. Select a Security Gateway from the list and click OK.
5. Read the warning is displayed and click Yes:
If you add <Name_of_Security_Gateway_object> to the cluster, it will be converted
to a cluster member.
Some settings will be lost.
The following settings will still remain:
-SIC
-VPN
-NAT (except for IP Pools)
In order to revert the conversion, session must be discarded.
Are you sure you want to continue?
6. In the list of cluster members, select the new cluster member and click Edit.
7. Click the NAT tab to configure the applicable NAT settings.
8. Click the VPN tab to configure the applicable VPN settings.
9. Click OK.

ClusterXL Administration Guide R80.20 | 63

 Configuring ClusterXL

Deleting a Cluster Member from Cluster Object

To delete an existing cluster member:
Before doing these steps, we recommend exporting a complete management database with
migrate export command.
1. In SmartConsole, open the cluster object.
2. Go to the Cluster Members page.
3. Click Remove > Delete Cluster Member.
4. Click OK.
Important - This cluster member object will be deleted from the cluster object and from the
management database.

Working with Cluster Topology

IPv6 Considerations
To activate IPv6 functionality for an interface, define an IPv6 address for the applicable interface
on each cluster member and in the cluster object. All interfaces configured with an IPv6 address
must also have a corresponding IPv4 address. If an interface does not require IPv6, only the IPv4
definition address is necessary.
Note - You must configure synchronization interfaces with IPv4 addresses only. This is because
the synchronization mechanism works using IPv4 only. All IPv6 information and states are
synchronized using this interface.
1. In SmartConsole, open the cluster object.
2. Go to Network Management page.
3. Select a cluster interface and click Edit.
4. From the left navigation tree, click General page:
a) In the General section, configure these settings for Cluster Virtual Interface:
 Network Type - one of these: Cluster, Sync, Cluster + Sync, Private
The available network types (network objectives) are:
Network Type Description
Cluster An interface that connects to an internal or external network.

Cluster + Sync A cluster interface that also works as a Synchronization interface.

We do not recommend this configuration because it adds the Delta
Sync traffic to the interface.

Sync An interface used exclusively for cluster state synchronization.

Private An interface that is not part of the cluster. ClusterXL does not
monitor the state of this interface. As a result, there is no cluster
failover if a fault occurs with this interface. This option is
recommended for the management interface.

 Virtual IPv4 - Virtual IPv4 address assigned to this Cluster Virtual Interface
 Virtual IPv6 - Virtual IPv6 address assigned to this Cluster Virtual Interface

ClusterXL Administration Guide R80.20 | 64

 Configuring ClusterXL

Important - You must define a corresponding IPv4 address for every IPv6 address. This
release does not support the configuration of only IPv6 addresses.
b) In the Member IPs section, click Modify and configure these settings:
 Physical IPv4 address and Mask Length assigned to the applicable physical interface on
each cluster member
 Physical IPv6 address and Mask Length assigned to the applicable physical interface on
each cluster member
Important - You must define a corresponding IPv4 address for every IPv6 address. This
release does not support the configuration of only IPv6 addresses.
See also: Configuring Cluster Addresses on Different Subnets.
c) In the Topology section, click Modify and configure these settings:
 Leads To - one of these: Internet (External), This Network (Internal)
 Security Zone - one of these: User defined, According to topology (ExternalZone,
InternalZone)
 Anti-Spoofing - whether to perform the Anti-Spoofing, and how to do it (Detect,
Prevent)
5. From the left navigation tree, click QoS page:
a) In the Bandwidth section, configure these settings:
 Inbound Active - rate limit for inbound traffic
 Outbound Active - rate limit for outbound traffic
b) In the DiffServ and Low Latency classes section, configure the applicable classes.
6. From the left navigation tree, click Advanced page:
a) In the Multicast Restrictions section, configure the applicable settings for dropping
multicast packets
b) In the Interfaces Names section, configure the names of applicable interfaces
7. Click OK.

Completing the Cluster Definition

1. Configure other Software Blades and options in the cluster object as required (NAT, VPN,
Remote Access, and other advanced options).
2. Install the Access Control Policy on this cluster object.

Changing the Synchronization Interface

Important - Schedule a maintenance window, because changing the synchronization interface can
impact the traffic.

To change the IPv4 address on the synchronization interface on cluster members:

1. On each cluster member, change the IPv4 address on the Sync interface.
Use Gaia Portal, or Gaia Clish.
2. In SmartConsole, open the cluster object.
3. In the Gateway Cluster Properties window, click Network Management page.
4. Click Get Interfaces > Get Interfaces With Topology.

ClusterXL Administration Guide R80.20 | 65

 Configuring ClusterXL

5. Make sure the settings are correct.

6. Select the Sync interface and click Edit.
7. From the left navigation tree, click General page.
8. In the General section, in the Network Type field, select Sync.
9. Click OK.
10. In SmartConsole, install the Access Control Policy on this cluster object.

To change the synchronization interface on cluster members to a new interface:

1. On each cluster member, configure a new interface that you will use as a new Sync interface.
Use Gaia Portal, or Gaia Clish.
2. On each cluster member, delete the IPv4 address from the old Sync interface.
3. Use Gaia Portal, or Gaia Clish.
4. In SmartConsole, open the cluster object.
5. In the Gateway Cluster Properties window, click Network Management page.
6. Click Get Interfaces > Get Interfaces With Topology.
7. Make sure the settings are correct.
8. Right-click on the old Sync interface and click Delete Interface.
9. Select the new interface and click Edit.
10. From the left navigation tree, click General page.
11. In the General section, in the Network Type field, select Sync.
12. Click OK.
13. In SmartConsole, install the Access Control Policy on this cluster object.

ClusterXL Administration Guide R80.20 | 66

 Configuring ClusterXL

Configuring a ClusterXL in Bridge Mode

You can configure ClusterXL in Bridge Mode in different cluster deployments:

ClusterXL in Bridge Mode Number of Supported Switches

Active / Standby Two only

Active / Active Two , or Four

Configuring ClusterXL in Bridge Mode - Active/Standby with Two

Switches

Notes:
• This procedure applies to both Check Point Appliances and Open Servers.
• ClusterXL in Bridge Mode deployed in Active/Standby supports only two switches.
The Bridge Active/Standby mode is the preferred mode in topologies that support it.
In the Bridge Active/Standby mode, ClusterXL works in High Availability mode. For more
information, see the R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_ClusterXL_
AdminGuide/html_frameset.htm.
Example:

ClusterXL Administration Guide R80.20 | 67

 Configuring ClusterXL

Item Description
1 Network, which an administrator needs to divide into two Layer 2 segments.
The ClusterXL in Bridge Mode connects between these segments.

2 First network segment.

3 Switch that connects the first network segment to one bridged slave interface (4) on the
ClusterXL in Bridge Mode.

4 One bridged slave interface (for example, eth1) on the Cluster Members in Bridge
Mode.

5 Dedicated Gaia Management Interface (for example, eth0) on the Cluster Members.

6 First Cluster Member in Bridge Mode (for example, in the Active cluster state).

7 Network that connects dedicated synchronization interfaces (for example, eth3) on the
ClusterXL in Bridge Mode.

8 Second Cluster Member in Bridge Mode (for example, in the Standby cluster state).

9 Another bridged slave interface (for example, eth2) on the Cluster Members in Bridge
Mode.

10 Switch that connects the second network segment to the other bridged slave interface
(9) on the ClusterXL in Bridge Mode.

11 Second network segment.

Workflow:
1. Install the two cluster members.
2. Configure the ClusterXL in High Availability mode in SmartConsole - in Wizard Mode, or
Classic Mode.
3. Configure the applicable policy for the ClusterXL Cluster in SmartConsole.
4. Enable the Bridge Active/Standby mode on both cluster members.
Best practice - If you configure Bridge Active/Standby mode, disable STP, RSTP, and MSTP on the
adjacent switches. See the applicable documentation for your switches.

Step 1 of 4: Install the two ClusterXL members

Step Description
1 Install the Gaia Operating System:
• Installing the Gaia Operating System on a Check Point Appliance
• Installing the Gaia Operating System on an Open Server
2 Run the First Time Configuration Wizard.

ClusterXL Administration Guide R80.20 | 68

 Configuring ClusterXL

Step Description
3 During the First Time Configuration Wizard, you must configure these settings:
• In the Installation Type window, select Security Gateway and/or Security
Management.
• In the Products window:
a) In the Products section, select Security Gateway only.
b) In the Clustering section, select these two options:
 Unit is a part of a cluster
 ClusterXL
• In the Secure Internal Communication window, enter the desired Activation Key
(between 4 and 127 characters long).

Step 2 of 4: Configure configure the ClusterXL in High Availability mode in

SmartConsole - Wizard Mode
Step Description
1 Connect with the SmartConsole to the Security Management Server or Domain
Management Server that should manage this ClusterXL.

2 Create a new Cluster object in one of these ways:

• From the top toolbar, click the New ( ) > Cluster > Cluster.
• In the top left corner, click Objects menu > More object types > Network Object >
Gateways and Servers > Cluster > New Cluster.
• In the top right corner, click Objects Pane > New > More > Network Object > Gateways
and Servers > Cluster > Cluster.
3 In the Check Point Security Gateway Cluster Creation window, click Wizard Mode.

4 On the Cluster General Properties page:

a) In the Cluster Name field, enter the desired name for this ClusterXL object.
b) Configure the main Virtual IP address(es) for this ClusterXL object.
In the Cluster IPv4 Address section, enter the main Virtual IPv4 address for this
ClusterXL object.
In the Cluster IPv6 Address section, enter the main Virtual IPv6 address for this
ClusterXL object.
c) In the Choose the Cluster's Solution field, select Check Point ClusterXL and High
Availability.
d) Click Next.

ClusterXL Administration Guide R80.20 | 69

 Configuring ClusterXL

Step Description
5A On the Cluster members' properties page, add the objects for the cluster members.
a) Click Add > New Cluster Member.
The Cluster Member Properties window opens.
b) In the Name field, enter the desired name for this cluster member object.
c) Configure the main physical IP address(es) for this cluster member object.
In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the cluster
member's First Time Configuration Wizard. Make sure the Security Management
Server or Multi-Domain Server can connect to these IP addresses.
d) In the Activation Key and Confirm Activation Key fields, enter the same Activation
Key you entered during the cluster member's First Time Configuration Wizard.
e) Click Initialize.
f) Click OK.
g) Repeat Steps a-f to add the second cluster member, and so on.

5B If the Trust State field does not show Established, perform these steps:
a) Connect to the command line on the cluster member.
b) Make sure there is a physical connectivity between the cluster member and the
Management Server (for example, pings can pass).
c) Run: cpconfig
d) Enter the number of this option: Secure Internal Communication.
e) Follow the instructions on the screen to change the Activation Key.
f) In the SmartConsole, click Reset.
g) Enter the same Activation Key you entered in the cpconfig menu.
h) Click Initialize.

6 On the Cluster Topology pages, configure the roles of the cluster interfaces:
a) Examine the IPv4 Network Address at the top of the page.
b) Select the applicable role:
 For cluster traffic interfaces, select Representing a cluster interface and
configure the Cluster Virtual IPv4 address and its Net Mask.
 For cluster synchronization interfaces, select Cluster Synchronization and
select Primary only. Check Point cluster supports only one synchronization
network. For more information, see the R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.
 For interfaces that do not pass the traffic between the connected networks,
select Private use of each member (don't monitor members interfaces).
c) Click Next.

ClusterXL Administration Guide R80.20 | 70

 Configuring ClusterXL

Step Description
7 On the Cluster Definition Wizard Complete page:
a) Examine the Configuration Summary.
b) Select Edit Cluster's Properties.
c) Click Finish.
The Gateway Cluster Properties window opens.

8 On the General Properties page > Machine section:

a) In the Name field, make sure you see the configured desired name for this
ClusterXL object.
b) In the IPv4 Address and IPv6 Address fields, make sure you see the configured IP
addresses.

9 On the General Properties page > Platform section, select the correct options:
a) In the Hardware field:
If you install the cluster members on Check Point Appliances, select the correct
appliances series.
If you install the cluster members on Open Servers, select Open server.
b) In the Version field, select R80.20.
c) In the OS field, select Gaia.

10 On the General Properties page > Network Security tab:

a) Make sure the ClusterXL Software Blade is selected.
b) Enable the additional applicable Software Blades.
Important:
• See the Supported Software Blades in Bridge Mode and Limitations in Bridge Mode.
• Do not select anything on the Management tab.

ClusterXL Administration Guide R80.20 | 71

 Configuring ClusterXL

Step Description
11A On the Cluster Members page:
a) Click Add > New Cluster Member.
The Cluster Member Properties window opens.
b) In the Name field, enter the desired name for this cluster member object.
c) Configure the main physical IP address(es) for this cluster member object.
In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the cluster
member's First Time Configuration Wizard. Make sure the Security Management
Server or Multi-Domain Server can connect to these IP addresses.
d) Click Communication.
e) In the One-time password and Confirm one-time password fields, enter the same
Activation Key you entered during the cluster member's First Time Configuration
Wizard.
f) Click Initialize.
g) Click Close.
h) Click OK.
i) Repeat Steps a-h to add the second cluster member, and so on.

11B If the Trust State field does not show Established, perform these steps:
a) Connect to the command line on the cluster member.
b) Make sure there is a physical connectivity between the cluster member and the
Management Server (for example, pings can pass).
c) Run: cpconfig
d) Enter the number of this option: Secure Internal Communication.
e) Follow the instructions on the screen to change the Activation Key.
f) In the SmartConsole, click Reset.
g) Enter the same Activation Key you entered in the cpconfig menu.
h) Click Initialize.

ClusterXL Administration Guide R80.20 | 72

 Configuring ClusterXL

Step Description
12 On the ClusterXL and VRRP page:
a) In the Select the cluster mode and configuration section, select High Availability
and ClusterXL.
b) In the Tracking section, select the desired option.
c) In the Advanced Settings section:
 Optional: Select Use State Synchronization. We recommend to select this
option.
 Optional: Select Use Virtual MAC (for more information, see sk50840
http://supportcontent.checkpoint.com/solutions?id=sk50840).
 Select the High Availability recovery - Maintain current active Cluster Member,
or Switch to higher priority Cluster Member. For more information, see the
R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.

ClusterXL Administration Guide R80.20 | 73

 Configuring ClusterXL

Step Description
13 On the Network Management page:
a) Select each interface and click Edit. The Network: <Name of Interface> window
opens.
b) From the left navigation tree, click the General page.
c) In the General section, in the Network Type field, select the applicable type:
 For cluster traffic interfaces, select Cluster. Make sure the Cluster Virtual IPv4
address and its Net Mask are correct.
 For cluster synchronization interfaces, select Sync or Cluster+Sync (we do not
recommend this configuration). Check Point cluster supports only one
synchronization network. For more information, see the R80.20 ClusterXL
Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.
 For interfaces that do not pass the traffic between the connected networks,
select Private.
d) In the Member IPs section, make sure the IPv4 address and its Net Mask are
correct on each Cluster Member.
Note - For cluster traffic interfaces, you can configure the Cluster Virtual IP address
to be on a different network than the physical IP addresses of the cluster members.
In this case, you must configure the required static routes on the cluster members.
For more information, see the R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.2
0_ClusterXL_AdminGuide/html_frameset.htm.
e) In the Topology section:
 Make sure the settings are correct in the Leads To and Security Zone fields.
 Make sure to enable the Anti-Spoofing.
Important:
• Make sure the Bridge interface and Bridge slave interfaces are not in the Topology.
• You cannot define the Topology of the Bridge interface. It is External by default.
14 Configure other applicable settings for this ClusterXL object.

15 Click OK.

16 Publish the SmartConsole session.

Step 2 of 4: Configure configure the ClusterXL in High Availability mode in

SmartConsole - Classic Mode
Step Description
1 Connect with the SmartConsole to the Security Management Server or Domain
Management Server that should manage this ClusterXL.

ClusterXL Administration Guide R80.20 | 74

 Configuring ClusterXL

Step Description
2 Create a new Cluster object in one of these ways:
• From the top toolbar, click the New ( ) > Cluster > Cluster.
• In the top left corner, click Objects menu > More object types > Network Object >
Gateways and Servers > Cluster > New Cluster.
• In the top right corner, click Objects Pane > New > More > Network Object > Gateways
and Servers > Cluster > Cluster.
3 In the Check Point Security Gateway Creation window, click Classic Mode.
The Gateway Cluster Properties window opens.

4 On the General Properties page > Machine section:

a) In the Name field, enter the desired name for this ClusterXL object.
b) In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the cluster
member's First Time Configuration Wizard. Make sure the Security Management
Server or Multi-Domain Server can connect to these IP addresses.

5 On the General Properties page > Platform section, select the correct options:
a) In the Hardware field:
If you install the cluster members on Check Point Appliances, select the correct
appliances series.
If you install the cluster members on Open Servers, select Open server.
b) In the Version field, select R80.20.
c) In the OS field, select Gaia.

6 On the General Properties page > Network Security tab:

a) Make sure the ClusterXL Software Blade is selected.
b) Enable the additional applicable Software Blades.
Important:
• See the Supported Software Blades in Bridge Mode and Limitations in Bridge Mode.
• Do not select anything on the Management tab.

ClusterXL Administration Guide R80.20 | 75

 Configuring ClusterXL

Step Description
7A On the Cluster Members page:
a) Click Add > New Cluster Member.
The Cluster Member Properties window opens.
b) In the Name field, enter the desired name for this cluster member object.
c) Configure the main physical IP address(es) for this cluster member object.
In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the cluster
member's First Time Configuration Wizard. Make sure the Security Management
Server or Multi-Domain Server can connect to these IP addresses.
d) Click Communication.
e) In the One-time password and Confirm one-time password fields, enter the same
Activation Key you entered during the cluster member's First Time Configuration
Wizard.
f) Click Initialize.
g) Click Close.
h) Click OK.
i) Repeat Steps a-h to add the second cluster member, and so on.

7B If the Trust State field does not show Established, perform these steps:
a) Connect to the command line on the cluster member.
b) Make sure there is a physical connectivity between the cluster member and the
Management Server (for example, pings can pass).
c) Run: cpconfig
d) Enter the number of this option: Secure Internal Communication.
e) Follow the instructions on the screen to change the Activation Key.
f) In the SmartConsole, click Reset.
g) Enter the same Activation Key you entered in the cpconfig menu.
h) Click Initialize.

ClusterXL Administration Guide R80.20 | 76

 Configuring ClusterXL

Step Description
8 On the ClusterXL and VRRP page:
a) In the Select the cluster mode and configuration section, select High Availability
and ClusterXL.
b) In the Tracking section, select the desired option.
c) In the Advanced Settings section:
 Optional: Select Use State Synchronization. We recommend to select this
option.
 Optional: Select Use Virtual MAC (for more information, see sk50840
http://supportcontent.checkpoint.com/solutions?id=sk50840).
 Select the High Availability recovery - Maintain current active Cluster Member,
or Switch to higher priority Cluster Member. For more information, see the
R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.

ClusterXL Administration Guide R80.20 | 77

 Configuring ClusterXL

Step Description
9 On the Network Management page:
a) Select each interface and click Edit. The Network: <Name of Interface> window
opens.
b) From the left navigation tree, click the General page.
c) In the General section, in the Network Type field, select the applicable type:
 For cluster traffic interfaces, select Cluster. Make sure the Cluster Virtual IPv4
address and its Net Mask are correct.
 For cluster synchronization interfaces, select Sync or Cluster+Sync (we do not
recommend this configuration). Check Point cluster supports only one
synchronization network. For more information, see the R80.20 ClusterXL
Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.
 For interfaces that do not pass the traffic between the connected networks,
select Private.
d) In the Member IPs section, make sure the IPv4 address and its Net Mask are
correct on each Cluster Member.
Note - For cluster traffic interfaces, you can configure the Cluster Virtual IP address
to be on a different network than the physical IP addresses of the cluster members.
In this case, you must configure the required static routes on the cluster members.
For more information, see the R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.2
0_ClusterXL_AdminGuide/html_frameset.htm.
e) In the Topology section:
 Make sure the settings are correct in the Leads To and Security Zone fields.
 Make sure to enable the Anti-Spoofing.
Important:
• Make sure the Bridge interface and Bridge slave interfaces are not in the Topology.
• You cannot define the Topology of the Bridge interface. It is External by default.
10 Configure other applicable settings for this ClusterXL object.

11 Click OK.

12 Publish the SmartConsole session.

Step 3 of 4: Configure the applicable policy for the ClusterXL in SmartConsole

Step Description
1 Connect with the SmartConsole to the Security Management Server or Domain
Management Server that manages this ClusterXL.

2 From the left navigation panel, click Security Policies.

ClusterXL Administration Guide R80.20 | 78

 Configuring ClusterXL

Step Description
3 Create a new policy and configure the applicable layers:
a) At the top, click the + tab (or press CTRL T).
b) On the Manage Policies tab, click Manage policies and layers.
c) In the Manage policies and layers window, create a new policy and configure the
applicable layers.
d) Click Close.
e) On the Manage Policies tab, click the new policy you created.

4 Create the applicable Access Control rules.

Important - See the Supported Software Blades in Bridge Mode and Limitations in Bridge
Mode.

5 Install the Access Control Policy on the ClusterXL Cluster object.

6 Examine the cluster configuration:

a) Connect to the command line on each cluster member.
b) Run:
• In Gaia Clish:
show cluster state
• In Expert mode:
cphaprob state

Step 4 of 4: Enable the Bridge Active/Standby mode on both cluster members

Item Description
1 Connect to the command line on each cluster member.

2 Run: cpconfig

3 Select Enable Check Point ClusterXL for Bridge Active/Standby.

4 Enter y to confirm.

5 Reboot the cluster member.

6 Connect with the SmartConsole to the Security Management Server or Domain

Management Server that manages this ClusterXL.

7 In SmartConsole, install the Access Control Policy on this cluster object.

ClusterXL Administration Guide R80.20 | 79

 Configuring ClusterXL

Item Description
8 Examine the cluster configuration on each cluster member.
• In Gaia Clish, run:
show cluster state
• In Expert mode, run:
cphaprob state
Example output:
Cluster Mode: High Availability (Active Up, Bridge Mode) with IGMP Membership
Number Unique Address Firewall State (*)
1 (local) 2.2.2.3 Active
2 2.2.2.2 Standby

Configuring ClusterXL in Bridge Mode - Active/Active with Two

Switches

When you define a Bridge interface on a cluster member, Bridge Active/Active mode is enabled by
default.
Notes:
• This procedure applies to both Check Point Appliances and Open Servers.
• This procedure describes ClusterXL in Bridge Active/Active Mode deployed with two switches.
In the Bridge Active/Active mode, ClusterXL works in Load Sharing mode. For more information,
see the R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_ClusterXL_
AdminGuide/html_frameset.htm.

ClusterXL Administration Guide R80.20 | 80

 Configuring ClusterXL

Example:

Item Description
1 Network, which an administrator needs to divide into two Layer 2 segments.
The ClusterXL in Bridge Mode connects between these segments.

2 First network segment.

3 Switch that connects the first network segment to one bridged slave interface (4) on the
ClusterXL in Bridge Mode.

4 One bridged slave interface (for example, eth1) on the Cluster Members in Bridge
Mode.

5 Dedicated Gaia Management Interface (for example, eth0) on the Cluster Members.

6 First Cluster Member in Bridge Mode (in the Active cluster state).

7 Network that connects dedicated synchronization interfaces (for example, eth3) on the
ClusterXL in Bridge Mode.

8 Second Cluster Member in Bridge Mode (in the Active cluster state).

9 Another bridged slave interface (for example, eth2) on the Cluster Members in Bridge
Mode.

10 Switch that connects the second network segment to the other bridged slave interface
(9) on the ClusterXL in Bridge Mode.

ClusterXL Administration Guide R80.20 | 81

 Configuring ClusterXL

Item Description
11 Second network segment.

Workflow:
1. Install the two cluster members.
2. Configure the Bridge interface on both cluster members - in Gaia Portal, or Gaia Clish.
3. Configure the ClusterXL in High Availability mode in SmartConsole - in Wizard Mode, or
Classic Mode.
4. Configure the applicable policy for the ClusterXL Cluster in SmartConsole.

Step 1 of 4: Install the two ClusterXL members

Step Description
1 Install the Gaia Operating System:
• Installing the Gaia Operating System on a Check Point Appliance
• Installing the Gaia Operating System on an Open Server
2 Run the First Time Configuration Wizard.

3 During the First Time Configuration Wizard, you must configure these settings:
• In the Installation Type window, select Security Gateway and/or Security
Management.
• In the Products window:
a) In the Products section, select Security Gateway only.
b) In the Clustering section, select these two options:
 Unit is a part of a cluster
 ClusterXL
• In the Secure Internal Communication window, enter the desired Activation Key
(between 4 and 127 characters long).

Step 2 of 4: Configure the Bridge interface in Gaia Portal:

For more information, see the R80.20 Gaia Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Gaia_Admin
Guide/html_frameset.htm.

Step Description
1 In your web browser, connect to the Gaia Portal on the Security Gateway.

2 In the left navigation tree, click Network Management > Network Interfaces.

3 Make sure that the slave interfaces, which you wish to add to the Bridge interface, do not
have IP addresses.

4 Click Add > Bridge.

To configure an existing Bridge interface, select the Bridge interface and click Edit.

ClusterXL Administration Guide R80.20 | 82

 Configuring ClusterXL

Step Description
5 On the Bridge tab, enter or select a Bridge Group ID (unique integer between 1 and 1024).

6 Select the interfaces from the Available Interfaces list and then click Add.
Notes:
• A Bridge interface in Gaia can contain only two slave interfaces.
• Do not select the interface that you configured as Gaia Management Interface.
7 On the IPv4 tab, do not enter the IPv4 address.

8 On the IPv6 tab (optional), do not enter the IPv6 address.

9 Click OK.

Step 2 of 4: Configure the Bridge interface in Gaia Clish:

For more information, see the R80.20 Gaia Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Gaia_Admin
Guide/html_frameset.htm.

Step Description
1 Connect to the command line on the Security Gateway.

2 Log in to Gaia Clish.

3 Make sure that the slave interfaces, which you wish to add to the Bridge interface, do not
have IP addresses. Run:
show interface <Name of Interface> ipv4-address
show interface <Name of Interface> ipv6-address

4 Add a new bridging group. Run:

add bridging group <Bridge Group ID 0 - 1024>

5 Add a slave interfaces to the new bridging group:

add bridging group <Bridge Group ID> interface <Name of First Slave Interface>
add bridging group <Bridge Group ID> interface <Name of Second Slave Interface>

6 Do not assign an IP address to the bridging group.

7 Save the configuration. Run:

save config

Step 3 of 4: Configure configure the ClusterXL in High Availability mode in

SmartConsole - Wizard Mode
Step Description
1 Connect with the SmartConsole to the Security Management Server or Domain
Management Server that should manage this ClusterXL.
ClusterXL Administration Guide R80.20 | 83
 Configuring ClusterXL

Step Description
2 Create a new Cluster object in one of these ways:
• From the top toolbar, click the New ( ) > Cluster > Cluster.
• In the top left corner, click Objects menu > More object types > Network Object >
Gateways and Servers > Cluster > New Cluster.
• In the top right corner, click Objects Pane > New > More > Network Object > Gateways
and Servers > Cluster > Cluster.
3 In the Check Point Security Gateway Cluster Creation window, click Wizard Mode.

4 On the Cluster General Properties page:

a) In the Cluster Name field, enter the desired name for this ClusterXL object.
b) Configure the main Virtual IP address(es) for this ClusterXL object.
In the Cluster IPv4 Address section, enter the main Virtual IPv4 address for this
ClusterXL object.
In the Cluster IPv6 Address section, enter the main Virtual IPv6 address for this
ClusterXL object.
c) In the Choose the Cluster's Solution field, select Check Point ClusterXL and High
Availability.
d) Click Next.

5A On the Cluster members' properties page, add the objects for the cluster members.
a) Click Add > New Cluster Member.
The Cluster Member Properties window opens.
b) In the Name field, enter the desired name for this cluster member object.
c) Configure the main physical IP address(es) for this cluster member object.
In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the cluster
member's First Time Configuration Wizard. Make sure the Security Management
Server or Multi-Domain Server can connect to these IP addresses.
d) In the Activation Key and Confirm Activation Key fields, enter the same Activation
Key you entered during the cluster member's First Time Configuration Wizard.
e) Click Initialize.
f) Click OK.
g) Repeat Steps a-f to add the second cluster member, and so on.

ClusterXL Administration Guide R80.20 | 84

 Configuring ClusterXL

Step Description
5B If the Trust State field does not show Established, perform these steps:
a) Connect to the command line on the cluster member.
b) Make sure there is a physical connectivity between the cluster member and the
Management Server (for example, pings can pass).
c) Run: cpconfig
d) Enter the number of this option: Secure Internal Communication.
e) Follow the instructions on the screen to change the Activation Key.
f) In the SmartConsole, click Reset.
g) Enter the same Activation Key you entered in the cpconfig menu.
h) Click Initialize.

6 On the Cluster Topology page, configure the roles of the cluster interfaces:
a) Examine the IPv4 Network Address at the top of the page.
b) Select the applicable role:
 For cluster traffic interfaces, select Representing a cluster interface and
configure the Cluster Virtual IPv4 address and its Net Mask.
 For cluster synchronization interfaces, select Cluster Synchronization and
select Primary only. Check Point cluster supports only one synchronization
network. For more information, see the R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.
 For interfaces that do not pass the traffic between the connected networks,
select Private use of each member (don't monitor members interfaces).
c) Click Next.

7 On the Cluster Definition Wizard Complete page:

a) Examine the Configuration Summary.
b) Select Edit Cluster's Properties.
c) Click Finish.
The Gateway Cluster Properties window opens.

8 On the General Properties page > Machine section:

a) In the Name field, make sure you see the configured desired name for this
ClusterXL object.
b) In the IPv4 Address and IPv6 Address fields, make sure you see the configured IP
addresses.

ClusterXL Administration Guide R80.20 | 85

 Configuring ClusterXL

Step Description
9 On the General Properties page > Platform section, select the correct options:
a) In the Hardware field:
If you install the cluster members on Check Point Appliances, select the correct
appliances series.
If you install the cluster members on Open Servers, select Open server.
b) In the Version field, select R80.20.
c) In the OS field, select Gaia.

10 On the General Properties page > Network Security tab:

a) Make sure the ClusterXL Software Blade is selected.
b) Enable the additional applicable Software Blades.
Important:
• See the Supported Software Blades in Bridge Mode and Limitations in Bridge Mode.
• Do not select anything on the Management tab.
11A On the Cluster Members page:
a) Click Add > New Cluster Member.
The Cluster Member Properties window opens.
b) In the Name field, enter the desired name for this cluster member object.
c) Configure the main physical IP address(es) for this cluster member object.
In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the cluster
member's First Time Configuration Wizard. Make sure the Security Management
Server or Multi-Domain Server can connect to these IP addresses.
d) Click Communication.
e) In the One-time password and Confirm one-time password fields, enter the same
Activation Key you entered during the cluster member's First Time Configuration
Wizard.
f) Click Initialize.
g) Click Close.
h) Click OK.
i) Repeat Steps a-h to add the second cluster member, and so on.

ClusterXL Administration Guide R80.20 | 86

 Configuring ClusterXL

Step Description
11B If the Trust State field does not show Established, perform these steps:
a) Connect to the command line on the cluster member.
b) Make sure there is a physical connectivity between the cluster member and the
Management Server (for example, pings can pass).
c) Run: cpconfig
d) Enter the number of this option: Secure Internal Communication.
e) Follow the instructions on the screen to change the Activation Key.
f) In the SmartConsole, click Reset.
g) Enter the same Activation Key you entered in the cpconfig menu.
h) Click Initialize.

12 On the ClusterXL and VRRP page:

a) In the Select the cluster mode and configuration section, select High Availability
and ClusterXL.
b) In the Tracking section, select the desired option.
c) In the Advanced Settings section:
 Optional: Select Use State Synchronization. We recommend to select this
option.
 Optional: Select Use Virtual MAC (for more information, see sk50840
http://supportcontent.checkpoint.com/solutions?id=sk50840).
 Select the High Availability recovery - Maintain current active Cluster Member,
or Switch to higher priority Cluster Member. For more information, see the
R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.

ClusterXL Administration Guide R80.20 | 87

 Configuring ClusterXL

Step Description
13 On the Network Management page:
a) Select each interface and click Edit. The Network: <Name of Interface> window
opens.
b) From the left navigation tree, click the General page.
c) In the General section, in the Network Type field, select the applicable type:
 For cluster traffic interfaces, select Cluster. Make sure the Cluster Virtual IPv4
address and its Net Mask are correct.
 For cluster synchronization interfaces, select Sync or Cluster+Sync (we do not
recommend this configuration). Check Point cluster supports only one
synchronization network. For more information, see the R80.20 ClusterXL
Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.
 For interfaces that do not pass the traffic between the connected networks,
select Private.
d) In the Member IPs section, make sure the IPv4 address and its Net Mask are
correct on each Cluster Member.
Note - For cluster traffic interfaces, you can configure the Cluster Virtual IP address
to be on a different network than the physical IP addresses of the cluster members.
In this case, you must configure the required static routes on the cluster members.
For more information, see the R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.2
0_ClusterXL_AdminGuide/html_frameset.htm.
e) In the Topology section:
 Make sure the settings are correct in the Leads To and Security Zone fields.
 Make sure to enable the Anti-Spoofing.
Important:
• Make sure the Bridge interface and Bridge slave interfaces are not in the Topology.
• You cannot define the Topology of the Bridge interface. It is External by default.
14 Configure other applicable settings for this ClusterXL object.

15 Click OK.

16 Publish the SmartConsole session.

Step 3 of 4: Configure configure the ClusterXL in High Availability mode in

SmartConsole - Classic Mode
Step Description
1 Connect with the SmartConsole to the Security Management Server or Domain
Management Server that should manage this ClusterXL.

ClusterXL Administration Guide R80.20 | 88

 Configuring ClusterXL

Step Description
2 Create a new Cluster object in one of these ways:
• From the top toolbar, click the New ( ) > Cluster > Cluster.
• In the top left corner, click Objects menu > More object types > Network Object >
Gateways and Servers > Cluster > New Cluster.
• In the top right corner, click Objects Pane > New > More > Network Object > Gateways
and Servers > Cluster > Cluster.
3 In the Check Point Security Gateway Creation window, click Classic Mode.
The Gateway Cluster Properties window opens.

4 On the General Properties page > Machine section:

a) In the Name field, enter the desired name for this ClusterXL object.
b) In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the cluster
member's First Time Configuration Wizard. Make sure the Security Management
Server or Multi-Domain Server can connect to these IP addresses.

5 On the General Properties page > Platform section, select the correct options:
a) In the Hardware field:
If you install the cluster members on Check Point Appliances, select the correct
appliances series.
If you install the cluster members on Open Servers, select Open server.
b) In the Version field, select R80.20.
c) In the OS field, select Gaia.

6 On the General Properties page > Network Security tab:

a) Make sure the ClusterXL Software Blade is selected.
b) Enable the additional applicable Software Blades.
Important:
• See the Supported Software Blades in Bridge Mode and Limitations in Bridge Mode.
• Do not select anything on the Management tab.

ClusterXL Administration Guide R80.20 | 89

 Configuring ClusterXL

Step Description
7A On the Cluster Members page:
a) Click Add > New Cluster Member.
The Cluster Member Properties window opens.
b) In the Name field, enter the desired name for this cluster member object.
c) Configure the main physical IP address(es) for this cluster member object.
In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the cluster
member's First Time Configuration Wizard. Make sure the Security Management
Server or Multi-Domain Server can connect to these IP addresses.
d) Click Communication.
e) In the One-time password and Confirm one-time password fields, enter the same
Activation Key you entered during the cluster member's First Time Configuration
Wizard.
f) Click Initialize.
g) Click Close.
h) Click OK.
i) Repeat Steps a-h to add the second cluster member, and so on.

7B If the Trust State field does not show Established, perform these steps:
a) Connect to the command line on the cluster member.
b) Make sure there is a physical connectivity between the cluster member and the
Management Server (for example, pings can pass).
c) Run: cpconfig
d) Enter the number of this option: Secure Internal Communication.
e) Follow the instructions on the screen to change the Activation Key.
f) In the SmartConsole, click Reset.
g) Enter the same Activation Key you entered in the cpconfig menu.
h) Click Initialize.

ClusterXL Administration Guide R80.20 | 90

 Configuring ClusterXL

Step Description
8 On the ClusterXL and VRRP page:
a) In the Select the cluster mode and configuration section, select High Availability
and ClusterXL.
b) In the Tracking section, select the desired option.
c) In the Advanced Settings section:
 Optional: Select Use State Synchronization. We recommend to select this
option.
 Optional: Select Use Virtual MAC (for more information, see sk50840
http://supportcontent.checkpoint.com/solutions?id=sk50840).
 Select the High Availability recovery - Maintain current active Cluster Member,
or Switch to higher priority Cluster Member. For more information, see the
R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.

ClusterXL Administration Guide R80.20 | 91

 Configuring ClusterXL

Step Description
9 On the Network Management page:
a) Select each interface and click Edit. The Network: <Name of Interface> window
opens.
b) From the left navigation tree, click the General page.
c) In the General section, in the Network Type field, select the applicable type:
 For cluster traffic interfaces, select Cluster. Make sure the Cluster Virtual IPv4
address and its Net Mask are correct.
 For cluster synchronization interfaces, select Sync or Cluster+Sync (we do not
recommend this configuration). Check Point cluster supports only one
synchronization network. For more information, see the R80.20 ClusterXL
Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R8
0.20_ClusterXL_AdminGuide/html_frameset.htm.
 For interfaces that do not pass the traffic between the connected networks,
select Private.
d) In the Member IPs section, make sure the IPv4 address and its Net Mask are
correct on each Cluster Member.
Note - For cluster traffic interfaces, you can configure the Cluster Virtual IP address
to be on a different network than the physical IP addresses of the cluster members.
In this case, you must configure the required static routes on the cluster members.
For more information, see the R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.2
0_ClusterXL_AdminGuide/html_frameset.htm.
e) In the Topology section:
 Make sure the settings are correct in the Leads To and Security Zone fields.
 Make sure to enable the Anti-Spoofing.
Important:
• Make sure the Bridge interface and Bridge slave interfaces are not in the Topology.
• You cannot define the Topology of the Bridge interface. It is External by default.
10 Configure other applicable settings for this ClusterXL object.

11 Click OK.

12 Publish the SmartConsole session.

Step 4 of 4: Configure the applicable policy for the ClusterXL in SmartConsole

Step Description
1 Connect with the SmartConsole to the Security Management Server or Domain
Management Server that manages this ClusterXL.

2 From the left navigation panel, click Security Policies.

ClusterXL Administration Guide R80.20 | 92

 Configuring ClusterXL

Step Description
3 Create a new policy and configure the applicable layers:
a) At the top, click the + tab (or press CTRL T).
b) On the Manage Policies tab, click Manage policies and layers.
c) In the Manage policies and layers window, create a new policy and configure the
applicable layers.
d) Click Close.
e) On the Manage Policies tab, click the new policy you created.

4 Create the applicable Access Control rules.

Important - See the Supported Software Blades in Bridge Mode and Limitations in Bridge
Mode.

5 Install the Access Control Policy on the ClusterXL Cluster object.

6 Examine the cluster configuration:

a) Connect to the command line on each cluster member.
b) Run:
• In Gaia Clish:
show cluster state
• In Expert mode:
cphaprob state
Example output:
Cluster Mode: High Availability (Active Up, Bridge Mode) with IGMP Membership
Number Unique Address Firewall State (*)
1 (local) 2.2.2.3 Active
2 2.2.2.2 Active

Configuring a ClusterXL in Bridge Mode - Active/Active with Four

Switches
When you define a Bridge interface on a cluster member, Bridge Active/Active mode is enabled by
default.
Notes:
• This procedure applies to both Check Point Appliances and Open Servers.
• This procedure describes ClusterXL in Bridge Active/Active Mode deployed with four switches.
In the Bridge Active/Active mode, ClusterXL works in Load Sharing mode. For more information,
see the R80.20 ClusterXL Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_ClusterXL_
AdminGuide/html_frameset.htm.

ClusterXL Administration Guide R80.20 | 93

 Configuring ClusterXL

Example:

Item Description
1 Network, which an administrator needs to divide into two Layer 2 segments.
The ClusterXL in Bridge Mode connects between these segments.

2 First network segment.

3 Switch that connects the first network segment to one bridged slave interface (6) on the
ClusterXL in Bridge Mode.

4 Switch that connects between one switch (that directly connects to the first network
segment) and one bridged slave interface (6) on the ClusterXL in Bridge Mode.

5 Dedicated Gaia Management Interface (for example, eth0) on the Cluster Members.

6 One bridged slave interface (for example, eth1) on the Cluster Members in Bridge
Mode.

7 First Cluster Member in Bridge Mode (in the Active cluster state).

8 Network that connects dedicated synchronization interfaces (for example, eth3) on the
ClusterXL in Bridge Mode.

9 Second Cluster Member in Bridge Mode (in the Active cluster state).

10 Another bridged slave interface (for example, eth2) on the Cluster Members in Bridge
Mode.

ClusterXL Administration Guide R80.20 | 94

 Configuring ClusterXL

Item Description
11 Switch that connects the second network segment to the other bridged slave interface
(10) on the ClusterXL in Bridge Mode.

12 Switch that connects between one switch (that directly connects to the second network
segment) and the other bridged slave interface (10) on the ClusterXL in Bridge Mode.

13 Second network segment.

Workflow:
1. Install the two cluster members.
2. Configure the Bridge interface on both cluster members - in Gaia Portal, or Gaia Clish.
3. Configure the ClusterXL in High Availability mode in SmartConsole - in Wizard Mode, or
Classic Mode.
4. Configure the applicable policy for the ClusterXL Cluster in SmartConsole.
The workflow and detailed instructions are the same as in the Configuring ClusterXL in Bridge
Mode - Active/Active with Two Switches (on page 80).

ClusterXL Administration Guide R80.20 | 95

CHAPTE R 5

Advanced Features and Procedures

In This Section:
Working with VPNs and Clusters .................................................................................96
Working with NAT and Clusters ...................................................................................98
Working with VLANS and Clusters ............................................................................100
Monitoring the Interface Link State ...........................................................................106
Bonding and Clusters .................................................................................................107
Advanced Cluster Configuration ................................................................................127
Defining Non-Monitored Interfaces ...........................................................................130
Configuring Policy Update Timeout ...........................................................................131
Enhanced 3-Way TCP Handshake Enforcement .......................................................132
Cluster IP Addresses on Different Subnets ..............................................................133
Converting a Security Gateway to a ClusterXL .........................................................138
Adding Another Member to an Existing Cluster .......................................................142
Removing a Member from an Existing Cluster .........................................................144
Configuring ISP Redundancy on a Cluster ................................................................145
Enabling Dynamic Routing Protocols in a Cluster Deployment ...............................151

Working with VPNs and Clusters

Configuring VPN and Clusters
Configuring a cluster using SmartConsole is very similar to configuring a single Security Gateway.
All attributes of the VPN are defined in the Cluster object, except for two attributes that are
defined per cluster member.
1. In SmartConsole, open the cluster object.
2. In the left navigation tree, go to Cluster Members page.
3. Select each cluster member and click Edit.
The Cluster Member Properties window opens.
4. Go the VPN tab:
• In the Office Mode for Remote access section:
If you wish to use Office Mode for Remote Access, select Offer Manual Office Mode and
define the IP pool allocated to each cluster member.
• In the Certificate List with keys stored on the Security Gateway section:
If your cluster member supports hardware storage for IKE certificates, define the
certificate properties. In that case, Security Management Server directs the cluster
member to create the keys and supply only the required material for creation of the
certificate request. The certificate is downloaded to the cluster member during policy
installation.
5. Click OK to close the Cluster Member Properties window.
6. In the left navigation tree, go to ClusterXL and VRRP page.
ClusterXL Administration Guide R80.20 | 96
 Advanced Features and Procedures

7. Make sure to select Use State Synchronization.

This is required to synchronize IKE keys.
8. In the left navigation tree, go to Network Management > VPN Domain page.
9. Define the encryption domain of the cluster.
Select one of the two possible settings:
• All IP addresses behind Cluster Members based on Topology information. This is the
default option.
• Manually defined. Use this option if the cluster IP address is not on the member network,
in other words, if the cluster virtual IP address is on a different subnet than the cluster
member interfaces. In that case, select a network or group of networks, which must
include the virtual IP address of the cluster, and the network or group of networks behind
the cluster.
10. Click OK to close the Gateway Cluster Properties window.
11. Install the Access Control Policy on this cluster object.

Defining VPN Peer Clusters with Separate Security Management

Servers
When working with a VPN peer that is a Check Point Cluster, and the VPN peer is managed by a
different Security Management Server, do NOT define another cluster object. Instead, do the
following:
1. In SmartConsole, go to Objects menu > More object types > Network Object > Gateways and
Servers > More > New Externally Managed VPN Gateway.
The Externally Managed Check Point Gateway window opens.
2. In the General Properties page, configure the name and the IP address.
3. In the Topology page, click New to add the external and internal cluster interfaces on the VPN
peer.
4. In the VPN Domain section of the Topology page, define the encryption domain of the
externally managed Security Gateway to be behind the internal Virtual IP address of the
Security Gateway.
If the encryption domain is just one subnet, select All IP addresses behind Gateway based on
Topology information.
If the encryption domain includes more than one subnet, select Manually defined.
5. Click OK.
6. Install the Access Control Policy on this cluster object.

ClusterXL Administration Guide R80.20 | 97

 Advanced Features and Procedures

Working with NAT and Clusters

Cluster Fold and Cluster Hide
Network Address Translation (NAT) is a fundamental aspect of the way ClusterXL works.
• When a cluster member establishes an outgoing connection towards the Internet, the source
address in the outgoing packets, is the physical IP address of the cluster member interface.
The source IP address is changed using NAT to that of the external virtual IP address of the
cluster. This address translation is called "Cluster Hide".
When working with VRRP on Gaia cluster, this corresponds to the default setting in the
ClusterXL and VRRP page of the cluster object of Hide Cluster Members outgoing traffic
behind the Cluster IP address being selected.
When working with VRRP on IPSO cluster, this corresponds to the default setting in the 3rd
Party Configuration page of the cluster object of Hide Cluster Members' outgoing traffic
behind the Cluster's IP address being selected.
• When a client establishes an incoming connection to external (virtual) address of the cluster,
ClusterXL changes the destination IP address using NAT to that of the physical external
address of one of the cluster members. This address translation is called "Cluster Fold".
When working with VRRP on Gaia cluster, this corresponds to the default setting in the
ClusterXL and VRRP page of the cluster object of Forward Cluster incoming traffic to Cluster
Members IP address being selected.
When working with IPSO IP Clustering cluster, this corresponds to the default setting in the
3rd Party Configuration page of the cluster object of Forward Cluster incoming traffic to
Cluster Members' IP addresses being selected.

Configuring NAT on the Cluster

Network Address Translation (NAT) can be performed on a Cluster, in the same way as it is
performed on a Security Gateway. This NAT is in addition to the automatic "Cluster Fold" and
"Cluster Hide" address translations.
To configure NAT, edit the Cluster object, and in the Cluster Properties window, select the NAT
page. Do NOT configure the NAT tab of the cluster member object.

Configuring NAT on a Cluster Member

It is possible to perform Network Address Translation (NAT) on a non-cluster interface of a cluster
member.
A possible scenario for this is if the non-Cluster interface of the cluster member is connected to
another (non-cluster) internal Security Gateway, and you wish to hide the address of the
non-Cluster interface of the cluster member.
Performing this NAT means that when a packet originates behind or on the non-Cluster interface
of the cluster member, and is sent to a host on the other side of the internal Security Gateway, the
source address of the packet will be translated.

To configure NAT on a non-cluster interface of a cluster member Security Gateway:

1. Edit the Cluster object.

ClusterXL Administration Guide R80.20 | 98

 Advanced Features and Procedures

2. In the Cluster Member page of the Cluster Properties window, edit the Cluster Member
object.
3. In the Cluster Member Properties window, click the NAT tab.
4. Configure Static or Hide NAT as desired.

ClusterXL Administration Guide R80.20 | 99

 Advanced Features and Procedures

Working with VLANS and Clusters

VLAN Support in ClusterXL
A VLAN switch tags packets that originate in a VLAN with a four-byte header that specifies, which
switch port it came from. No packet is allowed to go from a switch port in one VLAN to a switch
port in another VLAN, apart from ports ("global" ports) that are defined so that they belong to all
the VLANs.
The cluster member is connected to the global port of the VLAN switch, and this logically divides a
single physical port into many VLAN ports each associated with a VLAN tagged interface (VLAN
interface) on the cluster member.
When defining VLAN tags on an interface, cluster IP addresses can be defined only on the VLAN
interfaces (the tagged interfaces). Defining a cluster IP address on a physical interface that has
VLANs is not supported. This physical interface has to be defined with the Network Type Private.
ClusterXL (including VSX) supports the Synchronization Network (CCP packets that carry Delta
Sync information) only on the lowest VLAN ID (VLAN tag). For example, if three VLANs with IDs 10,
20 and 30 are configured on interface eth1, then you can use only the VLAN interface eth1.10
for the State Synchronization.
This is the default interface monitoring in Check Point cluster:

Interface type ClusterXL (non-VSX) VSX Cluster

Physical interfaces Monitors all cluster interfaces Monitors all cluster interfaces

VLAN interfaces Monitors only lowest VLAN ID VSX High Availability (non-VSLS):
configured on a physical interface
Monitors only lowest and highest
VLAN IDs configured on a physical
interface

Monitors only lowest and highest Virtual System Load Sharing:

VLAN IDs configured on a physical
Monitors all VLAN IDs configured on
interface
a physical interface

You can customize the default monitoring of VLAN IDs:

Need to monitor In ClusterXL (non-VSX) In VSX Cluster

Only the lowest This is the default behavior Need to disable the default behavior
VLAN ID (*)

Only the lowest and This is the default behavior VSX High Availability (non-VSLS):
highest VLAN IDs This is the default behavior

All VLAN IDs Not supported Virtual System Load Sharing: This is
the default behavior

Only specific VLAN See sk92784 See sk92784

IDs http://supportcontent.checkpoint.co http://supportcontent.checkpoint.co
m/solutions?id=sk92784 m/solutions?id=sk92784

ClusterXL Administration Guide R80.20 | 100

 Advanced Features and Procedures

(*) In a VSX cluster, to disable the default VLAN monitoring behavior, you need to set the value of

the kernel parameter fwha_monitor_all_vlan to 0 (zero). Follow the instructions in sk26202

http://supportcontent.checkpoint.com/solutions?id=sk26202.

Connecting Several Clusters on the Same VLAN

It is not recommended to connect the non-secured interfaces (the internal or external cluster
interfaces, for example) of multiple clusters to the same VLAN. A separate VLAN, and/or switch is
needed for each cluster.
Connecting the secured interfaces (the synchronization interfaces) of multiple clusters is also not
recommended for the same reason. Therefore, it is best to connect the secured interfaces of a
given cluster via a crossover link when possible, or to an isolated VLAN.
If there is a need to connect the secured or the non-secured interfaces of multiple clusters to the
same VLAN you need to make changes to:
• The destination MAC address, to enable communication between the cluster and members
outside the cluster (for ClusterXL Load Sharing Multicast Mode clusters only).
• The source MAC address of the cluster, to enable Cluster Control Protocol communication
between cluster members.

Changes to the Destination MAC Address

This section applies to ClusterXL Load Sharing Multicast Mode only.

How the Destination Cluster MAC Address is Assigned in Load Sharing Multicast Mode
When a member that is outside the cluster wishes to communicate with the cluster, it sends an
ARP query with the cluster (virtual) IP address. The cluster replies to the ARP request with a
multicast MAC address, even though the IP address is a unicast address.
This destination multicast MAC address of the cluster is based on the unicast IP address of the
cluster. The upper three bytes are 01.00.5E, and they identify a Multicast MAC in the standard way.
The lower three bytes are the same as the lower three bytes of the IP address. An example MAC
address based on the IP address 10.0.10.11 is shown below.

10. 0. 10. 11 Destination unicast IP address for the cluster

01 00 5E 00 0A 0B Destination multicast MAC address for the cluster

Upper 3 bytes Lower 3 bytes

Identify a Multicast MAC From IP address

Duplicate Multicast MAC Addresses: The Problem

When more than one cluster is connected to the same VLAN, the last three bytes of the IP
addresses of the cluster interfaces connected to the VLAN must be different. If they are the same,
then communication from outside the cluster that is intended for one of the clusters will reach
both clusters, which will cause communication problems.

ClusterXL Administration Guide R80.20 | 101

 Advanced Features and Procedures

For example, it is OK for the cluster interface of one of the clusters connected to the VLAN to have
the address 10.0.10.11, and the cluster interface of a second cluster to have the address
10.0.10.12. However, the following addresses for the interfaces of the first and second clusters
will cause complications: 10.0.10.11 and 20.0.10.11.

Duplicate Multicast MAC Addresses: The Solution

The best solution is to change to the last three bytes of the IP address of all but one of the cluster
interfaces that share the same last three bytes of their IP address.
If the IP address of the cluster interface cannot be changed, you must change the automatically
assigned multicast MAC address of all but one of the clusters and replace it with a user-defined
multicast MAC address. Proceed as follows:
1. In SmartConsole, open the cluster object.
2. In the left navigation tree, go to ClusterXL and VRRP page.
3. In the Select the cluster mode and configuration section, select Load Sharing and select
Multicast.
4. In the left navigation tree, go to Network Management page.
5. Select the cluster interface that is connected to same VLAN as the other cluster and click Edit.
6. Go to Advanced page.
7. In the Multicast Address section, select User Defined.
8. Enter the new user-defined MAC address. It must be of the form 01:00:5e:xy:yy:yy where x is
between 0 and 7 (in hex) and y is between 0 and f (in hex).
9. Click OK to close the Network interface properties window.
10. Click OK to close the Gateway Cluster properties window.
11. Install the Access Control Policy on this cluster object.

ClusterXL Administration Guide R80.20 | 102

 Advanced Features and Procedures

Changes to the Source MAC Address

This section applies to all ClusterXL modes, both High Availability and Load Sharing.

How the Source Cluster MAC Address is Assigned

Cluster members communicate with each other using the Cluster Control Protocol (CCP). CCP
packets are distinguished from ordinary network traffic by giving CCP packets a unique source
MAC address.
• The first four bytes of the source MAC address are all zero: 00.00.00.00
• The fifth byte of the source MAC address is a magic number. Its value indicates its purpose
Default value of fifth byte Purpose

0xfe CCP traffic

0xfd Forwarding layer traffic

• The sixth byte is the ID of the sending cluster member

Duplicate Source Cluster MAC Addresses: The Problem
When more than one cluster is connected to the same VLAN, if CCP and Forwarding Layer traffic
uses Multicast MAC address for the destination, this traffic reaches only the intended cluster.
If the Broadcast MAC address is used for Destination for CCP and for Forwarding Layer traffic
(and in certain other cases), cluster traffic intended for one cluster is seen by all connected
clusters. If this traffic is processed by the wrong cluster, it will cause communication problems.

Duplicate Source Cluster MAC Addresses: The Solution

To resolve the issue, change the source MAC address (MAC magic ID) of the cluster interfaces
connected to the broadcast domain in all but one of the clusters.
MAC magic has two modes, manual and automatic. Automatic is the default and the
recommended mode. Do not use manual mode unless Check Point Support tells you to use it.
Note - For more details, see sk25977
http://supportcontent.checkpoint.com/solutions?id=sk25977.

To change the MAC magic value:

1. Close all SmartConsole windows. To make sure, run the cpstat mg command on Security
Management Server or in the context of each Domain Management Server.
2. Connect with GuiDBedit Tool (see sk13009
http://supportcontent.checkpoint.com/solutions?id=sk13009) to Security Management Server
or Domain Management Server.
3. In the upper left pane, go to Table > Network Objects > network_objects.
4. In the upper right pane, select the applicable Cluster object.
5. Press CTRL+F (or go to Search menu > Find) > paste cluster_magic > click Find Next.
6. In the lower pane, right-click on the cluster_magic > select Edit.
7. Delete the current value.
8. Enter the desired value and click OK:
• To work in automatic mode (recommended mode), enter 254.

ClusterXL Administration Guide R80.20 | 103

 Advanced Features and Procedures

254 is the default value and should already be set. If duplicate Source MAC addresses of
CCP packets appear on the network even though automatic mode is set, then enter unique
values for each cluster (manual mode).
• To work in manual mode (only if instructed so by Check Point Support), enter an integer
value between 1 and 253.
Enter a unique value for each cluster in the domain.
9. Repeat steps 4-8 for each cluster.
10. Save the changes: go to File menu > click Save All.
11. Close the GuiDBedit Tool.
12. Connect with SmartConsole to Security Management Server or Domain Management Server.
13. Install the policy on the Cluster object.
14. Examine the MAC magic value on each cluster member:
cphaprob mmagic
All cluster members of the same cluster should have the save MAC magic value.
Example:
[Expert@MemberB:0]# cphaprob mmagic

Configuration mode: Automatic

Configuration phase: Stable

MAC magic: 100

MAC forward magic: 254

Used MAC magic values: None.

To change the MAC magic ID during a Connectivity Upgrade (R80.10 and higher):
Before the upgrade, find out the current configuration mode.
1. Connect to one of the cluster members.
2. Run cphaprob mmagic
The Configuration Mode field will show manual or automatic.
If the configuration field is automatic, upgrade the cluster. The upgraded cluster member
will learn the MAC magic value from a member that has not yet been upgraded (if a value
exists). Select a value if no previous value exists.
Note - If the configuration field shows manual, and you want to continue to use manual
configuration, the same MAC magic value must be reused.
3. Before the upgrade, find the configured MAC magic value:
• For R80.10 and higher, run: cphaprob mmagic
• For R77.30, run: cphaconf cluster_id get
• For R77.20 and below, run: fw ctl get int fwha_mac_magic
4. If you are working in manual mode, connect to the management server using GuiDBedit Tool:
a) Connect with GuiDBedit Tool (see sk13009
http://supportcontent.checkpoint.com/solutions?id=sk13009) to Security Management
Server or Domain Management Server.
b) In the upper left pane, go to Table > Network Objects > network_objects.
c) In the upper right pane, select the applicable Cluster object.
d) Press CTRL+F (or go to Search menu > Find) > paste cluster_magic > click Find Next.
ClusterXL Administration Guide R80.20 | 104
 Advanced Features and Procedures

e) In the lower pane, right-click on the cluster_magic > select Edit.

f) Delete the current value.
g) Enter the previous value and click OK.
h) Save the changes: go to File menu > click Save All.
i) Close the GuiDBedit Tool.
5. Upgrade the cluster members. See the R80.20 Installation and Upgrade Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Installat
ion_and_Upgrade_Guide/html_frameset.htm.
Note - When working in manual mode, the MAC magic value must be configured using
GuiDBedit Tool before the first policy installation.
6. Connect with SmartConsole to Security Management Server or Domain Management Server.
7. Install policy on the cluster object.
8. Examine the MAC magic value on each cluster member:
cphaprob mmagic
All cluster members of the same cluster should have the save MAC magic value.

ClusterXL Administration Guide R80.20 | 105

 Advanced Features and Procedures

Monitoring the Interface Link State

Enabling Interface Link State Monitoring shortens the time it takes ClusterXL to detect an
interface failure. By monitoring the link state (i.e. the electrical state) of an interface, ClusterXL is
immediately alerted to connectivity issues concerning a certain network interface, such as a
disconnected cable, or an electrical failure (real or simulated) on a switch.
Interface Link State Monitoring requires an interface device driver that supports link state
detection. The device driver reports the link state as either connected or disconnected.
Monitoring the interface link state is particularly useful in scenarios where a monitored interface
(an interface with Network Type Cluster, or Private) sends ICMP ECHO probe requests, which are
not answered by hosts or routers on the connected subnet.
When enabled, ClusterXL immediately detects when an interface goes down. When disabled,
ClusterXL determines whether an interface is malfunctioning by watching subsecond timeout
expiration.
Monitoring Interface Link State is enabled by default.
Note - Interface Link State Monitoring requires an interface device driver that supports link state
detection. This feature was tested on two cluster interfaces connected directly with a cross-cable.
However, it also works for interfaces connected to a switch. See sk31336
http://supportcontent.checkpoint.com/solutions?id=sk31336.

Enabling Interface Link State Monitoring

The global parameter fwha_monitor_if_link_state accepts these values:
• 0 – Disable Interface Link State Monitoring. This is the default setting for version R75.40VS and
earlier.
• 1 – Enable Interface Link State Monitoring. This is the default setting for version R76 and
higher.
To enable or disable Interface Link State Monitoring, run this command:
fw ctl set int fwha_monitor_if_link_state {0|1}
When you run this command, the change is not permanent. The Interface Link State Monitoring
setting goes back to its default state when you reboot the member.
To enable or disable Interface Link State Monitoring permanently, see sk26202
http://supportcontent.checkpoint.com/solutions?id=sk26202.

ClusterXL Administration Guide R80.20 | 106

 Advanced Features and Procedures

Bonding and Clusters

In This Section
Bond Interfaces (Link Aggregation)...........................................................................108
Bonding - High Availability Mode ...............................................................................110
Bonding - Load Sharing Mode ...................................................................................116
Group of Bonds ...........................................................................................................118
Performance Guidelines for Bonding ........................................................................124
Troubleshooting Issues with Bonded Interfaces ......................................................125

ClusterXL Administration Guide R80.20 | 107

 Advanced Features and Procedures

Bond Interfaces (Link Aggregation)

Check Point security devices support Link Aggregation, a technology that joins multiple physical
interfaces into one virtual interface, known as a bond interface. The bond interface share the load
among many interfaces, which gives fault tolerance and increases throughput. Check Point
devices support the IEEE 802.3ad Link Aggregation Control Protocol (LCAP) for dynamic link
aggregation.

Item No. Description

1 Security Gateway

1A Interface 1

1B interface 2

2 Bond Interface

3 Router

A bond interface (also known as a bonding group or bond) is identified by its Bond ID (for
example: bond1) and is assigned an IP address. The physical interfaces included in the bond are
called slaves and do not have IP addresses.
You can define a bond interface to use one of these functional strategies:
• High Availability (Active/Backup): Gives redundancy when there is an interface or a link
failure. This strategy also supports switch redundancy. Bond High Availability works in
Active/Backup mode - interface Active/Standby mode. When an Active slave interface is down,
the connection automatically fails over to the primary slave interface. If the primary slave
interface is not available, the connection fails over to a different slave interface.
• Load Sharing (Active/Active): All slave interfaces in the UP state are used simultaneously.
Traffic is distributed among the slave interfaces to maximize throughput. Bond Load Sharing
does not support switch redundancy.
Note - Bonding Load Sharing mode requires SecureXL to be enabled on Security Gateway or
each cluster member.
You can configure Bond Load Sharing to use one of these modes:
• Round Robin - Selects the Active slave interfaces sequentially.
• 802.3ad - Dynamically uses Active slave interfaces to share the traffic load. This mode uses
the LACP protocol, which fully monitors the interface link between the Check Point Security
Gateway and a switch.

ClusterXL Administration Guide R80.20 | 108

 Advanced Features and Procedures

• XOR - All slave interfaces in the UP state are Active for Load Sharing. Traffic is assigned to
Active slave interfaces based on the transmit hash policy: Layer 2 information (XOR of
hardware MAC addresses), or Layer 3+4 information (IP addresses and Ports).
For Bonding High Availability mode and for Bonding Load Sharing mode:
• The number of bond interfaces that can be defined is limited by the maximal number of
interfaces supported by each platform. See the R80.20 Release Notes
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_RN/html
_frameset.htm.
• Up to 8 slave interfaces can be configured in a single High Availability or Load Sharing bond
interface.

ClusterXL Administration Guide R80.20 | 109

 Advanced Features and Procedures

Bonding - High Availability Mode

In This Section
Simple Redundant Topology ......................................................................................110
Fully Meshed Redundancy .........................................................................................111
Bond Failover in High Availability Mode ....................................................................112
Creating an Interface Bond in High Availability Mode ..............................................112
Verifying that the Bond is Functioning Properly .......................................................112
Failover Support for VLANs .......................................................................................112
Sync Redundancy ........................................................................................................113
Configuring Bond High Availability in VRRP Cluster ................................................114

When dealing with mission-critical applications, an enterprise requires its network to be highly
available.
Clustering provides redundancy, and thus, High Availability, at the Security Gateway level. Without
Bonding, redundancy of Network Interface Cards (NICs) or of the switches on either side of the
Security Gateway are only possible in a cluster, and only by failover of the Security Gateway to
another cluster member.

Simple Redundant Topology

You can have redundancy of clustering without Bonding. If a switch or cluster member fails, a High
Availability cluster solution provides system redundancy. For example, you can have a redundant
system with two synchronized cluster members deployed in a simple redundant topology.

Item Description
1 Cluster member GW1 with interfaces connected to the external switches (items 5 and
6)

2 Cluster member GW2 with interfaces connected to the external switches (items 5 and
6)

3 Interconnecting network C1

4 Interconnecting network C2

5 Switch S1

6 Switch S2

If cluster member GW1 (item 1), its NIC, or switch S1 (item 5) fails, cluster member GW2 (item 2)
becomes the only Active member, connecting to switch S2 (item 6) over network C2 (item 4). If any

ClusterXL Administration Guide R80.20 | 110

 Advanced Features and Procedures

component fails (cluster member, NIC, or switch), the result of the failover is that no further
redundancy exists. A further failure of any active component completely stops network traffic
through this cluster.

Fully Meshed Redundancy

The Bonding High Availability mode, when deployed with ClusterXL, enables a higher level of
reliability by providing granular redundancy in the network. This granular redundancy is achieved
by using a fully meshed topology, which provides for independent backups for both NICs and
switches.
A fully meshed topology further enhances the redundancy in the system by providing a backup to
both the interface and the switch, essentially backing up the cable. Each cluster member has two
external interfaces, one connected to each switch.

Item Description
1 Cluster member GW1 with interfaces connected to the external switches (items 4 and
5)

2 Cluster member GW2 with interfaces connected to the external switches (items 4 and
5)

3 Interconnecting network

4 Switch S1

5 Switch S2

In this scenario:
• GW1 and GW2are cluster members in the High Availability mode, each connected to the two
external switches
• S1 and S2 are external switches
• Item 3 are the network connections
If any of the interfaces on a cluster member that connect to an external switch fails, the other
interface continues to provide the connectivity.
If any cluster member, its NIC, or switch fails, the other cluster member, connecting to switch S2
over network C2. If any component fails (cluster member , NIC, or switch), the result of the
failover is that no further redundancy exists. A further failure of any active component completely
stops network traffic.
Bonding provides High Availability of NICs. If one fails, the other can function in its place.

ClusterXL Administration Guide R80.20 | 111

 Advanced Features and Procedures

Bond Failover in High Availability Mode

In Bonding High Availability mode, when the Security Gateway is part of a cluster, bond internal
failover can occur in one of these cases:
• An active interface detects a failure in the link state, in a monitored interface.
• ClusterXL detects a failure in sending or receiving Cluster Control Protocol (CCP) keep-alive
packets.
Either of these failures will induce a failover within the interface bond, or between cluster
members, depending on the circumstances.
When a failure is detected, a log is recorded. You can see it in SmartConsole > Logs & Monitor >
Logs.

Creating an Interface Bond in High Availability Mode

Follow the instructions in the R80.20 Gaia Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Gaia_Admin
Guide/html_frameset.htm - Chapter Network Management - Section Network Interfaces - Section
Bond Interfaces (Link Aggregation).

Verifying that the Bond is Functioning Properly

Verify that the bond interface is UP, by displaying the bond information.
1. Run this command:
• In Gaia Clish:
show cluster interfaces all
• In Expert mode:
cphaprob -am if
Make sure that the bond status is reported as UP.
2. Run this command:
• In Gaia Clish:
show cluster bond {all | name <Name of Bond>}
• In Expert mode:
cphaprob show_bond <Name of Bond>
Check that the bond is correctly configured.

Failover Support for VLANs

In Bonding High Availability mode, ClusterXL monitors VLAN IDs for connectivity failure or
miscommunication, and initiate a failover when a failure is detected.
In a VLAN-enabled switched environment, ClusterXL monitors the VLAN with the lowest ID
number. The monitoring is conducted by sending ClusterXL Control Protocol (CCP) packets on
round-trip paths at a set interval. The lowest VLAN ID indicates the status of the physical
connection. This VLAN ID is always monitored, and a connectivity failure causes ClusterXL to
initiate a failover. ClusterXL will not detect a VLAN configuration problem on the switch.

ClusterXL Administration Guide R80.20 | 112

 Advanced Features and Procedures

Sync Redundancy
The use of more than one physical synchronization interface (1st sync, 2nd sync, 3rd sync) for
synchronization redundancy is not supported.
For synchronization redundancy, you can use bond interfaces.

Requirements and Limitations:

• The bond slave interfaces on each cluster member must connect to the same switch or VLAN
(for example, physical interface eth1 on all cluster members must connect to the same
switch).
• We recommend that interfaces and other network hardware support the IEEE 802.3 bond
mode.
• If you use a High Availability bond, you must add slave interfaces to the bonding group in the
same order on all cluster members.
Important - See Supported Topologies for Synchronization Network (on page 24).

To configure bond interfaces for sync High Availability:

1. Define a bond interface ("Creating an Interface Bond in High Availability Mode" on page 112) on
each cluster member with unused slave interfaces.
2. In SmartConsole, open the cluster object.
3. In the left navigation tree, go to Network Management page.
4. Click Get Interfaces > Get Interfaces With Topology to get the members' IP addresses.
5. Select the applicable interface and click Edit.
6. In General section, in the Network Type field, select Sync.
7. Click OK.
8. Install the Access Control Policy on this cluster object.
9. Make sure that the Sync interfaces are in the bond. Run this command on all cluster
members:
• In Gaia Clish: show cluster interfaces all
• In Expert mode: cphaprob -am if

ClusterXL Administration Guide R80.20 | 113

 Advanced Features and Procedures

Configuring Bond High Availability in VRRP Cluster

R80.20 introduces an improved Active/Backup Bond mechanism (Enhanced Bond) when working
in ClusterXL. If you work with ClusterXL, the Enhanced Bond feature is enabled by default, and no
additional configuration is required.
If you change your cluster configuration from ClusterXL to VRRP (MCVR & VRRP), or configure the
VRRP (MCVR & VRRP) cluster from scratch, the Enhanced Bond feature is disabled by default. If
you change your cluster configuration from VRRP to ClusterXL, you must manually enable the
Enhanced Bond feature.
To enable the Enhanced Bond feature in VRRP cluster, set the value of the kernel parameter
fwha_bond_enhanced_enable to 1 on each VRRP cluster member. You can set the value of the
kernel parameter temporarily, or permanently.

To set the value of the kernel parameter temporarily (does not survive reboot):
Step Description
1 Connect to the command line on each VRRP cluster member.

2 Log in to the Expert mode.

3 Set the value of the kernel parameter fwha_bond_enhanced_enable to 1:

[Expert@Member_HostName:0]# fw ctl set int fwha_bond_enhanced_enable
1

4 Make sure the value of the kernel parameter fwha_bond_enhanced_enable was set
to 1:
[Expert@Member_HostName:0]# fw ctl get int fwha_bond_enhanced_enable

To set the value of the kernel parameter permanently (survives reboot):

Step Description
1 Connect to the command line on each cluster member.

2 Log in to the Expert mode.

3 Back up the existing $FWDIR/boot/modules/fwkern.conf file:

[Expert@Member_HostName:0]# cp -v
$FWDIR/boot/modules/fwkern.conf{,_BKP}

4 Edit the existing $FWDIR/boot/modules/fwkern.conf file:

[Expert@Member_HostName:0]# vi $FWDIR/boot/modules/fwkern.conf

5 Add this line to the file (spaces and comments are not allowed):
fwha_bond_enhanced_enable=1

6 Save the changes in the file and exit from Vi editor.

7 Reboot the cluster member.

ClusterXL Administration Guide R80.20 | 114

 Advanced Features and Procedures

Step Description
8 Make sure the value of the kernel parameter fwha_bond_enhanced_enable was set
to 1:
[Expert@Member_HostName:0]# fw ctl get int fwha_bond_enhanced_enable

Important - If you change your cluster configuration from VRRP to ClusterXL, you must remove
the kernel parameter configuration from each cluster member.

ClusterXL Administration Guide R80.20 | 115

 Advanced Features and Procedures

Bonding - Load Sharing Mode

In This Section
Bond Failover in Load Sharing Mode.........................................................................116
Creating an Interface Bond in Load Sharing Mode...................................................116
Setting Critical Required Interfaces ..........................................................................116
Verifying that the Bond is Functioning Properly .......................................................117

In Bonding Load Sharing mode:

• All slave interfaces are active, and connections are balanced between the bond slave
interfaces, similar to the way ClusterXL Load Sharing balances connections between cluster
members.
• Each connection is assigned to a specific slave interface. For the individual connection, only
one slave interface is active. On failure of that interface, the bond fails over the connection to
one of the other slave interfaces, which adds the failed interface connection to the connections
it is already handling.
• All the slave interfaces of a bond must be connected to the same switch. The switch itself must
support and be configured for Bonding, by the same standard (for example, 802.3ad, or XOR)
as the Security Gateway bond.
Note - Bonding Load Sharing mode requires SecureXL to be enabled on each cluster member.

Bond Failover in Load Sharing Mode

In Bonding Load Sharing mode, when the Security Gateway is part of a cluster, bond internal
failover can occur in one of these cases:
• An active interface detects a failure in the link state, in a monitored interface.
• ClusterXL detects a failure in sending or receiving Cluster Control Protocol (CCP) keep-alive
packets.
Either of these failures will induce a failover within the interface bond, or between cluster
members, depending on the circumstances.
When a failure is detected, a log is recorded. You can see it in SmartConsole > Logs & Monitor >
Logs.

Creating an Interface Bond in Load Sharing Mode

Follow the instructions in the R80.20 Gaia Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Gaia_Admin
Guide/html_frameset.htm - Chapter Network Management - Section Network Interfaces - Section
Bond Interfaces (Link Aggregation).

Setting Critical Required Interfaces

Note - The Critical Required Interfaces feature is supported for ClusterXL only.
A bond in Load Sharing mode is considered to be down when fewer than a critical minimal number
of slave interfaces remain up. When not explicitly defined, the critical minimal number of slave
interfaces, which must remain up, in a bond of n interfaces is n-1. Failure of an additional slave
ClusterXL Administration Guide R80.20 | 116
 Advanced Features and Procedures

interface (when n-2 slave interfaces remain up) will cause the entire bond interface to be
considered down, even if the bond contains more than two slave interfaces.
If a smaller number of slave interfaces will be able to handle the expected traffic, you can increase
redundancy by explicitly defining the critical minimal number of slave interfaces. Divide your
maximal expected traffic speed by the speed of your slave interfaces and round up to a whole
number to determine an appropriate number of critical slave interfaces.
To define the critical number of slave interfaces explicitly, create and edit the following file:
$FWDIR/conf/cpha_bond_ls_config.conf
Each line of the file should be written in the following syntax:
<Name_of_Bond> <critical_minimal number_of_slave>
For example, if bond0 has 7 slave interfaces, and bond1 has 6 slave interfaces, file contents
could be:
bond0 5
bond1 3

In this example:
• bond0 would be considered down when 3 of its slave interfaces have failed.
• bond1 would be considered down when 4 of its slave interfaces have failed.

Verifying that the Bond is Functioning Properly

Verify that the bond interface is UP, by displaying the bond information.
1. Run this command:
• In Gaia Clish:
show cluster interfaces all
• In Expert mode:
cphaprob -am if
Make sure that the bond status is reported as UP.
2. Run this command:
• In Gaia Clish:
show cluster bond {all | name <Name of Bond>}
• In Expert mode:
cphaprob show_bond <Name of Bond>
Check that the bond is correctly configured.

ClusterXL Administration Guide R80.20 | 117

 Advanced Features and Procedures

Group of Bonds
Introduction
Group of Bonds, which is a logical group of existing Bond interfaces, provides additional link
redundancy.
Example topology with Group of Bonds:
• There is one router - R
• There are two switches that connect to the router R - SW-1 and SW-2
• There are two cluster members GW-A (Active) and GW-B (Standby)
• There are two Bond interfaces on each cluster member - Bond-1 and Bond-2
• On the cluster member GW-A:
• Bond-1 interface connects to the switch SW-1
• Bond-2 interface connects to the switch SW-2
• On the cluster member GW-B:
• Bond-1 interface connects to the switch SW-2
• Bond-2 interface connects to the switch SW-1

Chain of events without Group of Bonds:

1. The cluster member GW-A is the Active and the cluster member GW-B is the Standby.
2. On the cluster member GW-A, the Bond-1 interface fails.

ClusterXL Administration Guide R80.20 | 118

 Advanced Features and Procedures

3. On the cluster member GW-A, the Critical Device Interface Active Check reports its state as
"problem".
4. The cluster member GW-A changes its cluster state from Active to Down.
5. The cluster fails over - the cluster member GW-B changes its cluster state from Standby to
Active.
This is not the desired behavior, because the cluster member GW-A connects not only to the
switch SW-1, but also to the switch SW-2. In our example topology, there is no actual reason to
fail-over from the cluster member GW-A to the cluster member GW-B.
In order to overcome this problem, cluster members use the Group of Bonds consisting of Bond-1
and Bond-2. The Group of Bonds fails only when both Bond interfaces fail on the cluster member.
Only then the cluster fails over.
Chain of events with configured Group of Bonds:
1. The cluster member GW-A is the Active and the cluster member GW-B is the Standby.
2. On the cluster member GW-A, the Bond-1 interface fails.
3. On the cluster member GW-A, the Critical Device Interface Active Check reports its state as
"problem".
4. The cluster member GW-A does not change its cluster state from Active to Down.
5. On the cluster member GW-A, the Bond-2 interface fails as well.
6. The cluster member GW-A changes its cluster state from Active to Down.
7. The cluster fails over - the cluster member GW-B changes its cluster state from Standby to
Active.

To create a new Group of Bonds

Important - You must configure all cluster members in the same way..
1. Connect to the command line on the cluster member.
2. Log in to the Expert mode.
3. In VSX Cluster, switch to the context of the applicable Virtual System:
[Expert@Member:0]# vsenv <VSID>
4. Modify the current $FWDIR/boot/modules/fwkern.conf file:
a) Backup the current file:
[Expert@Member:0]# cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}
b) Edit the current file:
[Expert@Member:0]# vi $FWDIR/boot/modules/fwkern.conf
c) Add these two lines at the bottom of the file (spaces or comments are not allowed):
 fwha_group_of_bonds_str=<Name for Group of Bonds>:<List of all Bonds in this
Group separated by the comma>
Example:
fwha_group_of_bonds_str=GoB0:bond0,bond1;GoB1:bond2,bond3
 fwha_arp_probe_method=1
This parameter configures the cluster member to use the Virtual IP address as the
Source IP address in the ARP Requests during the probing of the local network.
d) Save the changes and exit from the Vi editor.
5. Change the value of the kernel parameter fwha_group_of_bonds_str to add the desired
ClusterXL Administration Guide R80.20 | 119
 Advanced Features and Procedures

Group of Bonds on-the-fly:

[Expert@Member:0]# fw ctl set str fwha_group_of_bonds_str '<Name for Group
of Bonds>:<List of all Bonds in this Group separated by the comma>'
Notes:
• The apostrophe characters are mandatory part of the syntax.
• Spaces are not allowed in the value of the kernel parameter
fwha_group_of_bonds_str.
Example:
fw ctl set str fwha_group_of_bonds_str 'GoB0:bond0,bond1;GoB1:bond2,bond3'
6. Change the value of the kernel parameter fwha_arp_probe_method on-the-fly:
[Expert@Member:0]# fw ctl set int fwha_arp_probe_method 1
7. Make sure the cluster member accepted the new configuration:
[Expert@Member:0]# fw ctl get str fwha_group_of_bonds_str
[Expert@Member:0]# fw ctl get int fwha_arp_probe_method
8. In SmartConsole, install the Access Control Policy on the cluster object.

To add a Bond interface to the existing Group of Bonds

Important - You must configure all cluster members in the same way.
1. Connect to the command line on the cluster member.
2. In VSX Cluster, switch to the context of the applicable Virtual System:
[Expert@Member:0]# vsenv <VSID>
3. Log in to the Expert mode.
4. Modify the current $FWDIR/boot/modules/fwkern.conf file:
a) Backup the current file:
[Expert@Member:0]# cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}
b) Edit the current file:
[Expert@Member:0]# vi $FWDIR/boot/modules/fwkern.conf
c) Edit the value of the kernel parameter fwha_group_of_bonds_str to add the desired
Bond interface to the existing Group of Bonds.
Example:
fwha_group_of_bonds_str=GoB0:bond0,bond1;GoB1:bond2,bond3,bond4
d) Save the changes and exit from the Vi editor.
5. Get the current value of the kernel parameter fwha_group_of_bonds_str and copy it:
[Expert@Member:0]# fw ctl get str fwha_group_of_bonds_str
6. Reset the current value of the kernel parameter fwha_group_of_bonds_str:
[Expert@Member:0]# fw ctl set str fwha_group_of_bonds_str ''
7. Make sure the cluster member reset the value of the kernel parameter
fwha_group_of_bonds_str:
[Expert@Member:0]# fw ctl get str fwha_group_of_bonds_str
8. Change the value of the kernel parameter fwha_group_of_bonds_str to add the desired
Bond interface to the existing Group of Bonds on-the-fly:
[Expert@Member:0]# fw ctl set str fwha_group_of_bonds_str '<Name for Group
of Bonds>:<List of all Bonds in this Group separated by the comma>'
ClusterXL Administration Guide R80.20 | 120
 Advanced Features and Procedures

Notes:
• The apostrophe characters are mandatory part of the syntax.
• Spaces are not allowed in the value of the kernel parameter
fwha_group_of_bonds_str.
Example:
fw ctl set str fwha_group_of_bonds_str 'GoB0:bond0,bond1;GoB1:bond2,bond3,bond4'
9. Make sure the cluster member accepted the new configuration:
[Expert@Member:0]# fw ctl get str fwha_group_of_bonds_str
10. In SmartConsole, install the Access Control Policy on the cluster object.

To remove a Bond interface from the existing Group of Bonds

Important - You must configure all cluster members in the same way.
1. Connect to the command line on the cluster member.
2. Log in to the Expert mode.
3. In VSX Cluster, switch to the context of the applicable Virtual System:
[Expert@Member:0]# vsenv <VSID>
4. Modify the current $FWDIR/boot/modules/fwkern.conf file:
a) Backup the current file:
[Expert@Member:0]# cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}
b) Edit the current file:
[Expert@Member:0]# vi $FWDIR/boot/modules/fwkern.conf
c) Edit the value of the kernel parameter fwha_group_of_bonds_str to remove the
desired Bond interface from the existing Group of Bonds.
Example:
fwha_group_of_bonds_str=GoB0:bond0,bond1;GoB1:bond2,bond3
d) Save the changes and exit from the Vi editor.
5. Get the current value of the kernel parameter fwha_group_of_bonds_str and copy it:
[Expert@Member:0]# fw ctl get str fwha_group_of_bonds_str
6. Reset the current value of the kernel parameter fwha_group_of_bonds_str:
[Expert@Member:0]# fw ctl set str fwha_group_of_bonds_str ''
7. Make sure the cluster member reset the value of the kernel parameter
fwha_group_of_bonds_str:
[Expert@Member:0]# fw ctl get str fwha_group_of_bonds_str
8. Change the value of the kernel parameter fwha_group_of_bonds_str to remove the
desired Bond interface from the existing Group of Bonds on-the-fly:
[Expert@Member:0]# fw ctl set str fwha_group_of_bonds_str '<Name for Group
of Bonds>:<List of all Bonds in this Group separated by the comma>'
Notes:
• The apostrophe characters are mandatory part of the syntax.
• Spaces are not allowed in the value of the kernel parameter
fwha_group_of_bonds_str.
Example:
fw ctl set str fwha_group_of_bonds_str 'GoB0:bond0,bond1;GoB1:bond2,bond3
ClusterXL Administration Guide R80.20 | 121
 Advanced Features and Procedures

9. Make sure the cluster member accepted the new configuration:

[Expert@Member:0]# fw ctl get str fwha_group_of_bonds_str
10. In SmartConsole, install the Access Control Policy on the cluster object.

To delete a Group of Bonds

Important - You must configure all cluster members in the same way.
1. Connect to the command line on the cluster member.
2. Log in to the Expert mode.
3. In VSX Cluster, switch to the context of the applicable Virtual System:
[Expert@Member:0]# vsenv <VSID>
4. Modify the current $FWDIR/boot/modules/fwkern.conf file:
a) Backup the current file:
[Expert@Member:0]# cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}
b) Edit the current file:
[Expert@Member:0]# vi $FWDIR/boot/modules/fwkern.conf
c) Delete these two lines in the file:
 fwha_group_of_bonds_str=<Name for Group of Bonds>:<List of all Bonds in this
Group separated by the comma>
 fwha_arp_probe_method=1
d) Save the changes and exit from the Vi editor.
5. Reset the current value of the kernel parameter fwha_group_of_bonds_str:
[Expert@Member:0]# fw ctl set str fwha_group_of_bonds_str ''
6. Make sure the cluster member reset the value of the kernel parameter
fwha_group_of_bonds_str:
[Expert@Member:0]# fw ctl get str fwha_group_of_bonds_str
7. In SmartConsole, install the Access Control Policy on the cluster object.

Monitoring
To see the configured Groups of Bonds, run the cphaprob show_bond_groups ("Monitoring
Bond Interfaces" on page 187) command.

Logs
Cluster members generate some applicable logs, which you can see in:
• In non-VSX Cluster: In the /var/log/messages files and output of the dmesg command.
• In VSX Cluster: In the $FWDIR/log/fwk.elg file in the context of the applicable Virtual
System.
To see the full logs, you need to enable the kernel debug:
• In the kernel module fw, debug flags error and ioctl
• Kernel module cluster, debug flag if
The full logs show:
• A physical slave interface goes down/up.

ClusterXL Administration Guide R80.20 | 122

 Advanced Features and Procedures

• A Bond interface goes down/up (regardless of a change in the cluster member's state).
• A Group of Bonds goes down/up.
Note - These logs are generated only once per event.

Limitations
• The maximal length on the text string <Name for Group of Bonds> is 16 characters.
• The maximal length on this text string is 1024 characters:
<Name for Group of Bonds>:<List of all Bonds in this Group separated by the comma>
• You can configure the maximum of five Groups of Bonds on a cluster member or Virtual
System.
• You can configure the maximum of five Bond interfaces in each Groups of Bonds.
• Group of Bonds feature does support Virtual Switches and Virtual Routers. Meaning, do not
configure Groups of Bonds in the context of these Virtual Devices.
• Group of Bonds feature supports only Bond interfaces that belong to the same Virtual System.
You cannot configure bonds that belong to different Virtual Systems into the same Group of
Bonds.
You must perform all configuration in the context of the applicable Virtual System.
• Group of Bonds feature does support Sync interfaces (an interface on a cluster member,
whose Network Type was set as Sync or Cluster+Sync in SmartConsole in cluster object).
• Group of Bonds feature does support Bridge interfaces.
• If a Bond interface goes down on one cluster member, the cphaprob show_bond_groups
("Monitoring Bond Interfaces" on page 187) command on the peer cluster members also
shows the same Bond interace as DOWN.
This is because the peer cluster members stop receiving the CCP packets on that Bond
interface and cannot probe the local network to determine that their Bond interface is really
working.
• After you add a Bond interface to the existing Group of Bonds, you must install the Access
Control Policy on the cluster object.
• After you remove a Bond interface from the existing Group of Bonds, you must install the
Access Control Policy on the cluster object.

ClusterXL Administration Guide R80.20 | 123

 Advanced Features and Procedures

Performance Guidelines for Bonding

To get the best performance, configure the SecureXL Affinity in static mode for the slave
interfaces.

Setting Affinities
If you are running SecureXL in a multi-core system, after you define bonds, set affinities manually.
Use the sim affinity -s command.
Note - The sim affinity commands take effect only if the SecureXL is enabled and actually
running. SecureXL begins running when you install a Policy for the first time.

For optimal performance, set affinities according to the following guidelines:

1. Run sim affinity -s.
2. Whenever possible, dedicate one processing core to each interface. See sk33520
http://supportcontent.checkpoint.com/solutions?id=sk33250.
3. If there are more interfaces than CPU cores, one or more CPU cores handle two interfaces.
Use interface pairs of the same position with internal and external bonds.
a) To view interface positions in a bond, run:
cat /proc/net/bonding/<bond name>.
b) Note the sequence of the interfaces in the output, and compare this for the two bonds
(external bond and its respective internal bond). Interfaces that appear in the same
position in the two bonds are interface pairs and set to be handled by one processing core.
For example, you might have four processing cores (0-3) and six interfaces (0-5), distributed
among two bonds:
bond0 bond1
eth0 eth3

eth1 eth4

eth2 eth5

Two of the CPU cores will need to handle two interfaces each. An optimal configuration can be:
bond0 bond1
eth0 core 0 eth3 core 0

eth1 core 1 eth4 core 1

eth2 core 2

eth5 core 3

ClusterXL Administration Guide R80.20 | 124

 Advanced Features and Procedures

Troubleshooting Issues with Bonded Interfaces

In This Section
Troubleshooting Workflow .........................................................................................125
Connectivity Delays on Switches................................................................................125

Troubleshooting Workflow
1. Examine the status of the bond ("Verifying that the Bond is Functioning Properly" on page 112).
2. If there is a problem, see if the physical link is down:
a) Run this command:
 In Gaia Clish:
show cluster bond name <bond_name>
 In Expert mode:
cphaprob show_bond <bond_name>
b) Look for a slave interface that reports the status of the link as no.
c) Examine the cable connections and other hardware.
d) Examine the port configuration on the switch.
3. See if a cluster member is down:
• In Gaia Clish:
show cluster state
• In Expert mode:
cphaprob state
If any of the cluster members has a firewall State other than Active, continue with the
cphaprob state troubleshooting ("ClusterXL Monitoring Commands" on page 167).
4. View the logs in SmartConsole > Logs & Monitor > Logs.
In a VSX cluster member, reboot is needed after these actions on a bond interface:
1. Changing a bond mode.
2. Adding a slave interface into a bond.
Note - Removing a slave interface does not require reboot.

Connectivity Delays on Switches

When using certain switches, connectivity delays may occur during some internal bond failovers.
With the various features that are now included on some switches, it can take close to a minute for
a switch to begin servicing a newly connected interface. These are suggestions for reducing the
startup time after link failure.
1. Disable auto-negotiation on the relevant interface.
2. On some Cisco switches, enable the PortFast feature.
3. Disable STP on the ports.

ClusterXL Administration Guide R80.20 | 125

 Advanced Features and Procedures

Warnings about PortFast

The PortFast feature should never be used on ports that connect to switches or hubs. It is
important that the Spanning Tree complete the initialization procedure in these situations.
Otherwise, these connections may cause physical loops where packets are continuously
forwarded (or even multiply) in such a way that can cause the network to fail.

Sample Configuration of PortFast Feature on a Cisco Switch

The following are the commands necessary to enable PortFast on a Gigabit Ethernet 1/0/15
interface of a Cisco 3750 switch running IOS.
1. Enter configuration mode:
cisco-3750A# conf t
2. Specify the interface to configure:
cisco-3750A(config)# interface gigabitethernet1/0/15
3. Set PortFast on this port:
cisco-3750A(config-if)# spanning-tree portfast
cisco-3750A(config-if)# end
4. Save the configuration:
cisco-3750A# write

ClusterXL Administration Guide R80.20 | 126

 Advanced Features and Procedures

Advanced Cluster Configuration

A number of synchronization and ClusterXL capabilities are controlled by means of Security
Gateway configuration parameters. Run these commands on the Security Gateway as follows:

fw ctl set int Parameter <value>

Parameter is any of the kernel parameters described in the following sections.

Changes to their default values must be implemented on all cluster members. Setting different
values on cluster members can cause configuration problems and possibly connection failures.
Note - All these configuration parameters can be configured to survive a reboot.

How to Configure Reboot Survival

Security Gateway configuration parameters that are changed using the fw ctl set int command, do
not survive reboot. To make them survive a reboot, you need to add the kernel parameter and its
value to the $FWDIR/boot/modules/fwkern.conf file (see sk26202
http://supportcontent.checkpoint.com/solutions?id=sk26202).
In the following instructions, Parameter is any of the parameters described in the following
sections.

To add or change a kernel parameter permanently:

1. Open $FWDIR/boot/modules/fwkern.conf in a text editor. Create this file, if it does not
exist yet.
2. Add or change the line (spaces or comments are forbidden):
Parameter=<value>
3. Save the changes in the text file.
4. Reboot the cluster member.
Note - This can cause a fail-over.
Do these steps for each cluster member.

Controlling the Clustering and Synchronization Timers

The following Security Gateway configuration parameters are used to control the clustering and
synchronization timers. Changing the default values is not recommended.
Clustering and Synchronization timers

Parameter Meaning Default

Value
fwha_timer_cpha_res The frequency of ClusterXL operations on the 1
cluster.

Operations occur every:

10 multiplied by fwha_timer_cpha_res multiplied
by fwha_timer_base_res milliseconds

ClusterXL Administration Guide R80.20 | 127

 Advanced Features and Procedures

Parameter Meaning Default

Value
fwha_timer_sync_res The frequency of sync flush operations on the 1
cluster.

Operations occur every:

10 multiplied by fwha_timer_sync_res multiplied
by fwha_timer_base_res milliseconds

fwha_timer_base_res Must be divisible by 10 with no remainders. 10

Blocking New Connections Under Load

This section applies only to Load Sharing modes. The reason for blocking new connections is that
new connections are the main source of new Delta Synchronization traffic. Delta Synchronization
may be at risk, if new traffic continues to be processed at high rate.
A related error message in cluster logs and in the /var/log/messages file is:
"FW-1: State synchronization is in risk. Please examine your synchronization network to avoid
further problems!".
Reducing the amount of traffic passing through the cluster member protects the Delta
Synchronization mechanism. See sk43896: Blocking New Connections Under Load in ClusterXL
https://supportcontent.checkpoint.com/solutions?id=sk43896.
• The kernel parameter fw_sync_block_new_conns allows cluster member to detect heavy
loads and start blocking new connections. Load is considered heavy when the synchronization
transmit queue of the firewall starts to fill beyond the fw_sync_buffer_threshold.
• To enable blocking new connections under load, set to 0.
• To disable blocking new connections under load, set to -1 (0xFFFFFFFF hex) (default).
Note that blocking new connections when sync is busy is only recommended for Load Sharing
ClusterXL deployments. While it is possible to block new connections in High Availability mode,
doing so does not solve inconsistencies in sync, as High Availability mode precludes that from
happening. This parameter can be set to survive boot using the mechanism described in How
to Configure a Kernel Parameter to Survive Reboot (see "How to Configure Reboot Survival" on
page 127).
• The kernel parameter fw_sync_buffer_threshold configures the maximum percentage of the
buffer that may be filled before new connections are blocked. By default it is set to 80, with a
buffer size of 512. By default, if more than 410 consecutive packets are sent without getting an
ACK on any one of them, new connections are dropped. When new connection blocking starts,
fw_sync_block_new_conns is set to 0. When the situation is stable, it is set back to -1.
• The kernel parameter fw_sync_allowed_protocols is used to determine the type of
connections that can be opened while the system is in a blocking state. Thus, the user can have
better control over the system behavior in cases of unusual load. The
fw_sync_allowed_protocols variable is a combination of flags, each specifying a different type
of connection. The required value of the variable is the result of adding the separate values of
these flags. For example, the default value of this variable is 24, which is the sum of
TCP_DATA_CONN_ALLOWED (8) and UDP_DATA_CONN_ALLOWED (16), meaning that the
default allows only TCP and UDP data connections to be opened under load.

ClusterXL Administration Guide R80.20 | 128

 Advanced Features and Procedures

Summary table:

ICMP_CONN_ALLOWED 1

TCP_CONN_ALLOWED 2 (except for data connections)

UDP_CONN_ALLOWED 4 (except for data connections)

TCP_DATA_CONN_ALLOWED 8 (the control connection should be established or allowed)

UDP_DATA_CONN_ALLOWED 16 (the control connection should be established or allowed)

Reducing the Number of Pending Packets

ClusterXL prevents out-of-state packets in non-sticky connections. It does this by holding packets
until a Sync ACK is received from all other active cluster members. If for some reason a Sync ACK
is not received, the Security Gateway on the cluster member will not release the packet, and the
connection will not be established.
To find out if held packets are not being released, run the fw ctl pstat command. If the output of
the command shows that the Number of Pending Packets is large under normal loads (more than
100 pending packets), and this value does not decrease over time, use the
fwldbcast_pending_timeout parameter to reduce the number of pending packets.
Change the value of fwldbcast_pending_timeout from the default value of 50 to a value lower
than 50.
The value is in ticks units, where each tick is equal to 0.1 sec, so that 50 ticks is 5 seconds.
The value represents the time, after which packets are released even if Sync ACKs are not
received.

ClusterXL Administration Guide R80.20 | 129

 Advanced Features and Procedures

Defining Non-Monitored Interfaces

Non-Monitored interfaces are cluster member interfaces that are not monitored by the ClusterXL
mechanism.
You may wish to define an interface as non-monitored, if the interface is down for a long time, and
you wish the cluster member to continue to be active.
1. In SmartConsole, open the cluster object.
2. Go to Network Management pane.
3. Select the interface, which is down for a long time.
4. Click Edit.
5. In the Network Type section, select Private.
6. Click OK.
7. Install the Access Control Policy on this cluster object.

ClusterXL Administration Guide R80.20 | 130

 Advanced Features and Procedures

Configuring Policy Update Timeout

When policy is installed on a Cluster, the cluster members undertake a negotiation process to
make sure all of them have received the same policy before they actually apply it. This negotiation
process has a timeout mechanism, which makes sure a cluster member does not wait indefinitely
for responses from other cluster members, which is useful in cases when another cluster
member goes down when policy is being installed (for example).
In configurations, on which policy installation takes a long time (usually caused by a policy with a
large number of rules), a cluster with more than two members, and slow members, this timeout
mechanism may expire prematurely.
It is possible to tune the timeout by setting the following kernel parameter:
fwha_policy_update_timeout_factor
The default value is 1, which should be sufficient for most configurations. For configurations
where the situation described above occurs, setting this parameter to 2 should be sufficient. Do
NOT set this parameter to a value larger than 3.

ClusterXL Administration Guide R80.20 | 131

 Advanced Features and Procedures

Enhanced 3-Way TCP Handshake Enforcement

The standard enforcement for a 3-way handshake that initiates a TCP connection provides
adequate security by guaranteeing one-directional stickiness. This means that it ensures that the
SYN-ACK will always arrive after the SYN. However, it does not guarantee that the ACK will always
arrive after the SYN-ACK, or that the first data packet will arrive after the ACK.
If you wish to have stricter policy that denies all out-of-state packets, you can configure the
synchronization mechanism so that all the TCP connection initiation packets arrive in the right
sequence (SYN, SYN-ACK, ACK, followed by the data). The price for this extra security is a
considerable delay in connection establishment.
To enable enhanced enforcement, use the GuiDBedit Tool (see sk13009
http://supportcontent.checkpoint.com/solutions?id=sk13009) to change the
sync_tcp_handshake_mode property from minimal_sync (default value) to complete_sync.

ClusterXL Administration Guide R80.20 | 132

 Advanced Features and Procedures

Cluster IP Addresses on Different Subnets

Introduction
You can configure cluster Virtual IP addresses in different subnets than the physical IP addresses
of the cluster members.
The network "sees" the cluster as one Security Gateway that operates as a network router. The
network is not aware of the internal cluster structure and physical IP addresses of cluster
members.

Advantages of using different subnets:

• You can create a cluster in an existing subnet that has a shortage of available IP addresses.
• You use only one Virtual IP address for the cluster. All other IP addresses can be on other
subnets.
• You can "hide" physical cluster members' IP addresses behind the cluster Virtual IP address.
This security practice is almost the same as NAT.
Note - This capability is available only for ClusterXL clusters.
Traffic sent from cluster members to internal or external networks is hidden behind the cluster
Virtual IP addresses and cluster MAC addresses. The cluster MAC address assigned to cluster
interfaces is:

Cluster Mode MAC Address

High Availability MAC address of the Active cluster member's
interface

Load Sharing Multicast Multicast MAC address of the cluster Virtual IP

Address

Load Sharing Uncast MAC address of the Pivot cluster member's interface

The use of different subnets with cluster objects has some limitations (see "Limitations of Cluster
Addresses on Different Subnets" on page 136).

Configuring Cluster Addresses on Different Subnets

These are the steps necessary to configure a cluster with IP addresses on different subnets:
1. On each cluster member, define these static routes for each cluster Virtual IP address:
next hop gateway for network of cluster Virtual IP address is applicable local member's
interface
2. Configure the cluster topology ("Working with Cluster Topology" on page 64).
Usually, cluster Virtual IP addresses are automatically related to an interface based on
membership in the same subnet. When the subnets are different, you must explicitly define the
relationship between a cluster member's interface and a cluster Virtual IP address.

ClusterXL Administration Guide R80.20 | 133

 Advanced Features and Procedures

Defining the Cluster Members Network

When using a cluster, in which the Cluster Virtual IP address and physical IP addresses of cluster
members are on different subnets, it is necessary to define the settings manually.

To define the member's network manually:

1. In SmartConsole, use the Classic Mode ("Using the Manual Configuration" on page 62) to
manually create a new cluster.
2. Define the cluster members and their physical interfaces.
3. Go to the Network Management page.
4. Select each cluster interface and click Edit.
5. In the General section, in the Virtual IPv4 field, enter the IPv4 address.
6. In the Member IPs, make sure the IP addresses are correct.
7. Click OK.
8. Install the Access Control Policy on this cluster object.
For more details, see the Configuring Cluster Object ("Configuring the Cluster Object and
Members" on page 59) chapter.

Configuring a Static Route

On each cluster member, define these static routes for each cluster Virtual IP address:
next hop gateway for network of cluster Virtual IP address is applicable local member's interface

If you do not define the static routes correctly, it will not be possible to connect to the cluster
members and pass traffic through them.
Note - It is not necessary to configure static routes manually on VSX cluster members. This is
done automatically when you configure routes in SmartConsole.
For configuration instructions, see the R80.20 Gaia Administration Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Gaia_Admin
Guide/html_frameset.htm - Chapter Network Management - Sections IPv4 Static Routes and IPv6
Static Routes.

ClusterXL Administration Guide R80.20 | 134

 Advanced Features and Procedures

Example of Cluster IP Addresses on Different Subnets

In this example, a one-Security Gateway firewall separating network 172.16.6.0 (Side "A") from
network 172.16.4.0 (Side "B") is to be replaced with a ClusterXL. The cluster members, however,
will use networks 192.168.1.0 for Side "A", 192.168.2.0 for Side "B" and 192.168.3.0 for the
synchronization network (all network addresses given in this example are of class "C"). The
addresses in italics are the cluster Virtual IP addresses.
In this example, cluster protects two networks, Network A and Network B.

Configuring Static Routes on the Cluster Members

Configure each cluster member with these static routes:
• next hop gateway for network 172.16.6.0 is local interface with IP address 192.168.1.x
• next hop gateway for network 172.16.4.0 is local interface with IP address 192.168.2.x
For more on static route configuration, see sk32073
http://supportcontent.checkpoint.com/solutions?id=sk32073.

Configuring Cluster IP Addresses in SmartConsole

1. In SmartConsole, open the cluster object.
2. In the left navigation tree, go to Network Management.
3. Select each cluster interface and click Edit.
4. Configure the settings as follows:
Interface Properties Cluster Interface "A" Cluster Interface "B"
IP address IP address
General section - 172.16.6.100 / 24 172.16.4.100 / 24
Virtual IPv4 field

Member IPs section 192.168.1.1 / 24 192.168.2.1 / 24

192.168.1.2 / 24 192.168.2.2 / 24

5. Click OK.
6. Install the Access Control Policy on this cluster object.

ClusterXL Administration Guide R80.20 | 135

 Advanced Features and Procedures

Limitations of Cluster Addresses on Different Subnets

This new feature does not yet support all the capabilities of ClusterXL. Some features require
additional configuration to work properly, while others are not supported.

Connectivity Between Cluster Members

Since ARP requests issued by cluster members are hidden behind the cluster IP and MAC
addresses, requests sent by one cluster member to the other may be ignored by the destination
computer. To allow cluster members to communicate with each other, a static ARP should be
configured for each cluster member, stating the MAC addresses of all other cluster members. IP
packets sent between cluster members are not altered, and therefore no changes should be made
to the routing table.
Note - Static ARP is not required in order for the cluster members to work properly as a cluster,
since the cluster synchronization protocol does not rely on ARP.

Load Sharing Multicast Mode with "Semi-Supporting" Hardware

Although not all types of network hardware work with multicast MAC addresses, some routers
can pass such packets, even though they are unable to handle ARP Replies containing a multicast
MAC address. Where a router semi-supports Load Sharing Multicast mode, it is possible to
configure the cluster MAC address as a static ARP entry in the router internal tables, and thus
allow it to communicate with the cluster.
When different subnets are used for the cluster IP addresses, static ARP entries containing the
router MAC address need to be configured on each cluster member. This is done because this
kind of router will not respond to ARP Requests containing a multicast source MAC address.
These special procedures are not required when using routers that fully support multicast MAC
addresses.

Manual Proxy ARP

When using Static NAT, the cluster can be configured to automatically recognize the hosts hidden
behind it, and issue ARP replies with the cluster MAC address, on their behalf. This process is
known as Automatic Proxy ARP.
However, if you use the ClusterXL VMAC mode or different subnets for the cluster IP addresses,
this mechanism will not work, and you must configure the proxy ARP manually. To do so, in
SmartConsole, select Menu > Global Properties > NAT Network Address Translation, and disable
Automatic ARP Configuration. Then create a file $FWDIR/conf/local.arp.
For instructions, see sk30197 http://supportcontent.checkpoint.com/solutions?id=sk30197.

Connecting to the Cluster Members from the Cluster Network

Since the unique IP addresses may be chosen arbitrarily, there is no guarantee that these
addresses are accessible from the subnet of the cluster IP address. In order to access the cluster
members through their unique IP addresses, you must configure routes on the accessing cluster
member, such that the cluster IP is the Default Gateway for the subnet of the unique IP addresses.

ClusterXL Administration Guide R80.20 | 136

 Advanced Features and Procedures

Configuring Anti-Spoofing
1. In SmartConsole, define a Group object, which contains the objects of both the external
network and the internal network.
In the example ("Example of Cluster IP Addresses on Different Subnets" on page 135), suppose
Side "A" is the external network, and Side "B" is the internal network.
You must define a Group object, which contains both network 172.16.4.0 and network
192.168.2.0.
2. Open the cluster object.
3. Go to Network Management pane.
4. Select the cluster interface and click Edit.
5. On the General page, in the Topology section, click Modify.
6. Select Override.
7. Select This Network (Internal).
8. Select Specific and select the Group object that contains the objects of both the external
network and the internal network.
9. Click OK.
10. Install the Access Control Policy on this cluster object.

ClusterXL Administration Guide R80.20 | 137

 Advanced Features and Procedures

Converting a Security Gateway to a ClusterXL

This section tells you how convert a Security Gateway to a ClusterXL. The source Security Gateway
becomes one of the members and you add one or more new members to the cluster. To help you
identify the members of the new ClusterXL, the procedures use these names:
• Standalone Computer - An existing computer that is configured as both a Security Gateway
and a Security Management Server.
• Source Security Gateway - The physical Security Gateway that will be converted into a
member of the new ClusterXL.
• Source Member - The member created from the Source Security Gateway.
• New Member - A newly created cluster member. There can be more than one new cluster
member.
You must have sufficient available IP address for the source Security Gateway and new members.
If not, see Configuring Cluster Addresses on Different Subnets.

Converting a Standalone Deployment to ClusterXL

Before you can convert a Standalone Deployment to ClusterXL, you must first migrate the Security
Gateway and the Security Management Server to two different computers. We recommend that
you keep the existing Standalone Computer available until you complete and test the new
ClusterXL environment.

Notes and Cautions:

• We recommend that you do the conversion during scheduled maintenance downtime.
• The new Security Management Server and Security Gateway must have the same version as
the existing Standalone Computer. Do version upgrades and/or install hotfixes before you start
the conversion process.
• We recommend that you keep the same interface IP addresses for your internal and external
networks on the new Security Gateway. This can minimize the necessity to reconfigure
gateway topologies.
• We recommend that you run cpinfo -z -o Standalone.cpinfo in the Expert mode on the
Standalone Computer and the new Security Management Server. This gives you "before" and
"after" status and debugging information that can help resolve migration issues.
Copy these files to another computer or external storage.
• For more detailed information about these procedures, see sk61681
http://supportcontent.checkpoint.com/solutions?id=sk61681.

To prepare the Standalone Computer for migration:

1. Backup the Standalone Computer. Use one of the procedures included in the Backing Up
section of the R80.20 Installation and Upgrade Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Installat
ion_and_Upgrade_Guide/html_frameset.htm. Copy the backup file to another computer or
external storage.
2. Disconnect the Standalone Computer from the network.

ClusterXL Administration Guide R80.20 | 138

 Advanced Features and Procedures

3. Disable all Security Gateway functionality:

a) Connect with SmartConsole and open the Standalone Computer object.
b) On the General Properties > Network Properties tab, clear all Software Blades including
Firewall. Click OK to continue.
c) Save the changes (Menu > File > Save).
d) Go to Menu > Policy > Install Database.
e) In the Install Database window, select the Standalone Computer object and click OK.
This operation must complete successfully.
f) Close SmartConsole and all other SmartConsole clients.

To Export the Management Database:

1. Connect with the CLI to the Standalone Computer in the Expert mode.
2. Export the management databases, run:
# cd $FWDIR/bin/upgrade_tools/
# ./upgrade_export /var/<export_file_name>

To Create the new Security Management Server:

Important - The new Security Management Server must have the same host name as the
existing Standalone Computer.

1. Do a clean Security Management Server installation based on the procedures in the R80.20
Installation and Upgrade Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Installat
ion_and_Upgrade_Guide/html_frameset.htm. Make sure that you only select Management
Server options.
Make sure that you install all Hotfixes and plug-ins that were installed in the existing
Standalone computer.
2. Close all of the Expert mode shells. Log into the regular shell.
3. Copy the exported database files to a temporary folder on the new Security Management
Server.
4. Import the management databases, from the Expert mode, run:
# cd $FWDIR/bin/upgrade_tools/
# ./upgrade_import /<path_to>/<export_file_name>
Important - If the import fails with the Database migration between standalone and
management only machines is not supported error, see sk61681
http://supportcontent.checkpoint.com/solutions?id=sk61681 for a workaround.
5. Connect with SmartConsole to the new Security Management Server and make sure that all
settings are correct.
6. Close SmartConsole and reboot the computer.

To Create the New Security Gateway:

1. Do a clean Security Gateway installation based on the procedures in the R80.20 Installation
and Upgrade Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Installat
ion_and_Upgrade_Guide/html_frameset.htm.

ClusterXL Administration Guide R80.20 | 139

 Advanced Features and Procedures

Make sure that you only select Network Security tab options.
Make sure that you install all Hotfixes and plug-ins that were installed in the existing
Standalone Computer.
2. In SmartConsole, create and configure the Security Gateway object.
Make sure that you establish SIC trust.
3. In SmartConsole, install the Access Control Policy on this Security Gateway object.
4. Connect the systems to the network.
5. Thoroughly test and debug the deployment.
Make sure that the rules for all Software Blades work correctly.
This Security Gateway will become the Source Member for the new ClusterXL cluster.

Creating the New Member

To create and configure a new cluster member:
1. Install a new Security Gateway.
2. Use the standard procedure to create a new cluster member ("Creating the ClusterXL Object"
on page 140).
3. Make sure that the cluster object definition and all applicable settings are the same as for the
Source Security Gateway. For example:
• Interface, topology and Anti-Spoofing definitions
• Authentication types
• IPsec VPN settings, including Link Selection
• Office mode settings
• Firewall rules settings
• Software Blade selections and configuration

Creating the ClusterXL Object

To create the ClusterXL object:
1. In SmartConsole, create a new cluster object ("Configuring the Cluster Object and Members"
on page 59).
2. Make sure that the cluster object definition and all applicable settings are the same as for the
Source Security Gateway. For example:
• Interface, topology and Anti-Spoofing definitions
• Authentication types
• IPsec VPN settings, including Link Selection
• Office mode settings
• Firewall rules settings
• Software Blade selections and configuration
3. If you assign Office Mode IP address from a pool, create a new pool

In SmartConsole, for Computer 'B'

1. Create a ClusterXL object.

ClusterXL Administration Guide R80.20 | 140

 Advanced Features and Procedures

2. In the Cluster Members page, click Add > Add Existing Gateway.
3. Connect to computer 'B', and define its topology.
4. Define the Synchronization networks for the cluster.
5. Define the cluster topology. To avoid reconfiguring network devices, the cluster IP addresses
should be on the same subnet as the IP addresses of computer 'A', on its proposed cluster
interfaces.
6. Install the Access Control Policy on this cluster object, currently including member 'B' only.

Preparing Computer 'A'

1. Disconnect all proposed cluster and Synchronization interfaces. New connections now open
through the cluster, instead of through computer 'A'.
2. Change the addresses of these interfaces to some other unique IP address, which is on the
same subnet as computer B.
3. Connect each pair of interfaces of the same subnet using a dedicated network. Any hosts or
Security Gateways previously connected to the Security Gateway must now be connected to
both members, using a hub/switch.
Note - It is possible to run synchronization across a WAN. For details, see Synchronizing Clusters
over a Wide Area Network (see "Synchronizing Clusters on a Wide Area Network" on page 53).

In SmartConsole, for Computer 'A'

1. Update the topology of Security Gateway A, either manually or by clicking Get Interfaces.
If the IP address of the management interface was changed, the Get Interfaces action will fail.
If this happens, manually change the main IP address in the Security Gateway object and save
the policy prior to performing an automatic topology fetch.
2. In the Cluster Members page, click Add > Add Existing Gateway.
3. Select computer 'A' in the window.
4. In the Network Management page, determine which interface is a cluster interface, and which
is an internal or an external interface.
5. Install the Access Control Policy on this cluster object.

ClusterXL Administration Guide R80.20 | 141

 Advanced Features and Procedures

Adding Another Member to an Existing Cluster

Important - Schedule a full maintenance window to perform this procedure.
1. Install the Security Gateway you plan to add to the existing cluster as another cluster member.
See the R80.20 Installation and Upgrade Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Installat
ion_and_Upgrade_Guide/html_frameset.htm.
2. On the Security Gateway computer you plan to add to the existing cluster:
a) Configure or change the IP addresses on the applicable interfaces to match the cluster
topology.
b) Configure or change the applicable static routes to match the cluster topology.
c) Connect to the command line.
d) Log in to Gaia Clish or Expert mode.
e) Run:
cpconfig
f) Select Enable cluster membership for this gateway > enter y to confirm.
g) Reboot the Security Gateway.
3. In SmartConsole:
a) Open the existing cluster object.
b) In the Cluster Members page:
 If the object of the Security Gateway you plan to add to the existing cluster is not defined
yet, click Add > New Cluster Member
 If the object of the Security Gateway you plan to add to the existing cluster is already
defined, click Add > Add Existing Gateway
Follow the instructions on the screen to configure this new cluster member.
c) In the Network Management page:
 Make sure all interfaces are defined correctly.
 Make sure all IP addresses are defined correctly.
d) Click OK.
e) Install the Access Control Policy on the cluster object.
4. On each cluster member (existing and the newly added):
a) Connect to the command line.
b) Log in to Gaia Clish or Expert mode.
c) Make sure all cluster members detect each other and agree on their cluster states. Run:
cphaprob state
5. If cluster members do not detect each other, or do not agree on their cluster states:
a) Connect to the command line on each cluster member (existing and the newly added).
b) Log in to Gaia Clish or Expert mode.

ClusterXL Administration Guide R80.20 | 142

 Advanced Features and Procedures

c) Restart the clustering on each cluster member.

Important - This temporarily causes the cluster member not to be a part of the cluster. As
a result, cluster failover can occur.
Run:
cphastop
cphastart
d) Make sure all cluster members detect each other and agree on their cluster states. Run:
cphaprob state

ClusterXL Administration Guide R80.20 | 143

 Advanced Features and Procedures

Removing a Member from an Existing Cluster

Important - Schedule a full maintenance window to perform this procedure.
1. In SmartConsole:
a) Open the existing cluster object.
b) In the Cluster Members page, click Remove > Delete Cluster Member. Confirm when
prompted.
Important:
 This operation deletes the object.
 There must be at least two cluster members in the cluster object.
c) In the Network Management page:
 Make sure all interfaces are defined correctly.
 Make sure all IP addresses are defined correctly.
d) Click OK.
e) Install the Access Control Policy on the cluster object.
2. On each existing cluster member:
a) Connect to the command line.
b) Log in to Gaia Clish or Expert mode.
c) Restart the clustering.
Important - This temporarily causes the cluster member not to be a part of the cluster. As
a result, cluster failover can occur.
Run:
cphastop
cphastart
d) Make sure all cluster members detect each other and agree on their cluster states. Run:
cphaprob state
3. On the Security Gateway computer you removed from the existing cluster:
a) Connect to the command line.
b) Log in to Gaia Clish or Expert mode.
c) Run:
cpconfig
d) Select Disable cluster membership for this gateway > enter y to confirm.
e) Select Secure Internal Communication > enter y to confirm > enter the new
Activation Key. Make sure to write it down.
f) Exit from the cpconfig menu.
g) Reboot the Security Gateway.

ClusterXL Administration Guide R80.20 | 144

 Advanced Features and Procedures

Configuring ISP Redundancy on a Cluster

Make Internet connectivity more reliable with ISP Redundancy. This connects a Security Gateway
or cluster member to the Internet through redundant Internet Service Provider (ISP) links. ISP
Redundancy monitors the ISP links and chooses the best current link.
R80.20 supports two ISPs.
If you have a ClusterXL Cluster, connect each cluster member to the two ISPs through a LAN with
two interfaces. The member interfaces must be on the same subnet as the cluster external
interfaces. Configure ClusterXL in the usual way.

• ISP A link address - 192.168.1.100

• ISP B link address - 172.16.2.100
Item Description
1 Internal network

2 VLAN switches

3 Security Gateway - Cluster member A

3a Virtual interface to the internal network (10.10.0.1)

3b Interface to the Cluster Sync network (10.0.10.1)

3c Virtual interface to ISP B (172.16.2.2)

3d Virtual interface to VLAN switch that connects to ISP A (192.168.1.3)

4 Security Gateway - Cluster member B

4a Virtual interface to the internal network (10.10.0.2)

4b Interface to the Cluster Sync network (10.0.10.2)

4c Virtual interface to ISP A (192.168.1.2)

4d Virtual interface to VLAN switch that connects to ISP B (172.16.2.3)

ClusterXL Administration Guide R80.20 | 145

 Advanced Features and Procedures

Item Description
5 Router that connects to ISP A (192.168.1.1)

6 Router that connects to ISP B (172.16.2.1)

7 Internet

You can configure the ISP preference to be for Load Sharing or Primary/Backup.
Load Sharing - Uses the two links with a distributed load of connections going out from the
Security Gateway. Connections coming in are alternated. You can configure best relative loads for
the links (set a faster link to handle more load). New connections are randomly assigned to a link.
If one link fails, the other takes the load.
Primary/Backup - Uses one link for connections going out from the Security Gateway and coming
in. It switches to the backup if the primary link fails. When the primary link is restored, new
connections are assigned to it. Existing connections continue on the backup link until they are
complete.
Note: ISP Redundancy settings override VPN Link Selection settings.

To enable ISP Redundancy:

1. Open the network object properties of the Security Gateway or cluster.
2. Click Other > ISP Redundancy.
3. Select Support ISP Redundancy.
4. Select Load Sharing or Primary/Backup.
5. Configure the links ("Configuring the ISP Links" on page 146).
6. Configure the Security Gateway to be the DNS server ("Configuring Security Gateway as DNS"
on page 147).
7. Configure the policy for ISP Redundancy ("Configuring the Firewall" on page 148).

Configuring the ISP Links

Before you begin, make sure you have the ISP data - the speed of the link and next hop IP address.
If the Security Gateway has only one external interface, configure two subnets on this interface.
You will need routers and a switch.
If the Security Gateway has two external interfaces in the Network Management page of the
gateway object, you can configure the links automatically.
If the gateway is a ClusterXL cluster member, configure the two cluster members to the two ISP.
Use a LAN with two interfaces. Make sure the member interfaces are on the same subnet as the
cluster external interfaces.

To configure ISP links automatically:

1. In the Security Gateway object go to the Other > ISP Redundancy page.
2. Click Set initial configuration.
The ISP Links are added automatically.
3. For Primary/Backup, make sure the primary interface is first in the list. Use the arrows to
change the order.

ClusterXL Administration Guide R80.20 | 146

 Advanced Features and Procedures

To configure ISP links manually:

1. In the Security Gateway object go to the Other > ISP Redundancy page.
2. Click Add.
3. In the ISP Link window, give the link a Name.
Note the names you give here. They are used in the ISP Redundancy script and commands.
4. Select the Interface of the Security Gateway for this ISP link.
• If the Security Gateway has two external interfaces, set each link to a different interface. If
one of the ISP links is dialup connection to a backup ISP, configure the ISP Redundancy
Script ("Editing the ISP Redundancy Script" on page 150).
• If the Security Gateway has only one external interface, set each ISP link to connect to this
interface.
5. Configure the Next hop IP Address.
• If the Security Gateway has two external interfaces, leave this field empty and click Get
from routing table. The next hop is the default gateway.
• If the Security Gateway has one external interface, set each ISP link to a different next hop
router.
6. For Load Sharing, enter the Weight. For equal weight distribution, enter 50. If one link is
faster, raise this value and lower it for the other link, so that the two equal 100.
7. Define hosts to be monitored, to make sure the link is working. Open the Advanced tab of the
ISP Link window, and add Selected hosts.

Configuring Security Gateway as DNS

The Security Gateway, or a DNS server behind it, must respond to DNS queries. It resolves IP
addresses of servers in the DMZ (or another internal network).
Get a public IP address from each ISP. If public IP addresses are not available, register the
domain to make the DNS server accessible from the Internet.

To enable DNS on the Security Gateway or cluster:

1. In SmartConsole, open the Security Gateway or cluster object
2. Go to Other > ISP Redundancy page and select Enable DNS Proxy.
The Security Gateway or cluster intercepts Type A DNS queries for the web servers in its
domain that come from external hosts. If the Security Gateway or cluster recognizes the
external host, it replies:
• In ISP Redundancy Load Sharing mode, the Security Gateway or cluster replies with two
addresses, alternating their order.
• In ISP Redundancy Primary/Backup mode, the Security Gateway or cluster replies with the
addresses of the active ISP link.
If the Security Gateway or cluster does not recognize the host, it passes the DNS query on to
the original destination, or to the domain DNS server.
3. Click Configure.
4. Add your DMZ or web servers. Give each two public IP addresses, one for each ISP.
5. Enter a number of seconds in DNS TTL.
This sets a Time To Live for each DNS reply. DNS servers in the Internet cannot cache your
DNS data in the reply for longer than the TTL.

ClusterXL Administration Guide R80.20 | 147

 Advanced Features and Procedures

6. Configure Static NAT to translate the public IP addresses to the real server's IP address.
External clients use one of the two IP addresses.
Note - If the servers use different services (for example, HTTP and FTP), you can use NAT for
only two public IP addresses.
7. Define an Access Control Policy rule: allow DNS traffic through the Security Gateway or
cluster using the domain_udp service.

To register the domain and get IP addresses:

1. Register your domain with the two ISP.
2. Tell the ISP the two IP addresses of the DNS server that respond to DNS queries for the
domain.
3. For each server in the DMZ, get two public IP addresses, one from each ISP.
4. In SmartConsole, click Menu > Global properties.
5. Go to NAT - Network Address Translation pane.
6. In the Manual NAT rules section, select Translate destination on client side.
7. Click OK.
8. Install the Access Control Policy on this cluster object.

Configuring the Firewall

The Firewall must allow connections through the ISP links, with Automatic Hide NAT on network
objects that start outgoing connections.

To configure the firewall for ISP Redundancy:

1. In the properties of the object for an internal network, select NAT > Add Automatic Address
Translation Rules.
2. Select Hide behind the gateway.
3. Click OK.
4. Define rules for publicly reachable servers (web servers, DNS servers, DMZ servers).
If you have one public IP address from each ISP for the Security Gateway, define Static NAT.
Allow specific services for specific servers. For example, make NAT rules so that incoming
HTTP connections from the two ISP reach a Web server, and DNS traffic from the ISP reach
the DNS server.
Example: Manual Static Rules for a Web Server and a DNS Server

Original Original Original Original Translated Translated Comment

Source Destination Service Source Destination Services
Any IP of web http = 10.0.0.2 = Incoming Web - ISP A
server (Static)

Any IP of web http = 10.0.0.2 = Incoming Web - ISP B

server (Static)

Any IP of DNS domain = 10.0.0.3 = Incoming DNS - ISP A

server _udp (Static)

Any IP of DNS domain = 10.0.0.3 = Incoming DNS - ISP B

server _udp (Static)

ClusterXL Administration Guide R80.20 | 148

 Advanced Features and Procedures

If you have a public IP address from each ISP for each publicly reachable server (in addition to
the Security Gateway), define NAT rules:
a) Give each server a private IP address.
b) Use the public IP addresses in the Original Destination.
c) Use the private IP address in the Translated Destination.
d) Select Any as the Original Service.
Note - If using Manual NAT, automatic ARP does not work for the NATed IP addresses. You need
to configure the local.arp file as described in sk30197
http://supportcontent.checkpoint.com/solutions?id=sk30197.
When done, install the Access Control Policy on this cluster object.

Configuring with VPN

When ISP Redundancy is enabled, VPN encrypted connections survive a failure of an ISP link. The
settings in the ISP Redundancy page override settings in the IPsec VPN > Link Selection page.

To configure ISP Redundancy with VPN on cluster:

1. In SmartConsole, open the cluster object.
2. In the left navigation tree, go to Other > ISP Redundancy.
3. Select Apply settings to VPN traffic.
4. In the left navigation tree, go to IPsec VPN > Link Selection.
5. Make sure that Use ongoing probing. Link redundancy mode shows the mode of the ISP
Redundancy: High Availability (for Primary/Backup) or Load Sharing.
VPN Link Selection now only probes the ISP configured in ISP Redundancy.

To configure for VPN with a third-party peer:

If the VPN peer is not a Check Point Security Gateway, the VPN may fail, or the third-party device
may continue to encrypt traffic to a failed ISP link.
• Make sure the third-party VPN peer recognizes encrypted traffic from the secondary ISP link
as coming from the Check Point cluster.
• Change the configuration of ISP Redundancy to not use these Check Point technologies:
• Use Probing - Make sure that Link Selection uses another option.
• Load Sharing, Service Based Link Selection, Route based probing - Work only on Check
Point Security Gateways. If used, the Security Gateway uses one link to connect to the
third-party VPN peer. The link with the highest prefix length and lowest metric is used.

Force ISP Link State

Use the fw isp_link command to force the ISP link state to Up or Down. Use this to test
installation and deployment, or to force the Security Gateway to recognize the true link state if it
cannot (the ISP link is down but the gateway sees it as up).
You can run this command on the Security Gateway or the Security Management Server: fw
isp_link [target-gw] <link_name> {up|down}
<link_name> is the name in the ISP Link window.

ClusterXL Administration Guide R80.20 | 149

 Advanced Features and Procedures

Editing the ISP Redundancy Script

When the Security Gateway starts, or an ISP link state changes, the
$FWDIR/bin/cpisp_update script runs. It changes the default route of the Security Gateway.
For example, you can force the Security Gateway to change the state of a dialup interface to match
that state of its ISP link.
Edit this script to enable a dialup connection for one of the ISP links.

To configure a dialup connection:

1. In the script on the Security Gateway, enter the command to change the dialup interface state:
• If the ISP link goes down: fw isp_link <link_name> down
• If the ISP link goes up: fw isp_link <link_name> up
2. If you use PPPoE or PPTP xDSL modems, in the PPPoE or PPTP configuration, the Use Peer as
Default Gateway option must not be selected.

ClusterXL Administration Guide R80.20 | 150

 Advanced Features and Procedures

Enabling Dynamic Routing Protocols in a Cluster

Deployment
ClusterXL supports Dynamic Routing (Unicast and Multicast) protocols as an integral part of Gaia.
As the network infrastructure views the clustered Security Gateway as a single logical entity,
failure of a cluster member will be transparent to the network infrastructure and will not result in
a ripple effect.

Components of the System

Virtual IP Integration
All cluster members use the cluster IP address(es).

Routing Table Synchronization

Routing information is synchronized among the cluster members using the Forwarding
Information Base (FIB) Manager process. This is done to prevent traffic interruption in case of
failover, and used for Load Sharing and High Availability modes. The FIB Manager is the
responsible for the routing information.
The FIB Manager is registered as a critical device (Pnote), and if the slave goes out of sync, a
Pnote will be issued, and the slave member will go down until the FIB Manager is synchronized.

Failure Recovery
Dynamic Routing on ClusterXL avoids creating a ripple effect upon failover by informing the
neighboring routers that the router has exited a maintenance mode. The neighboring routers then
reestablish their relationships to the cluster, without informing the other routers in the network.
These restart protocols are widely adopted by all major networking vendors. The following table
lists the RFC and drafts compliant with Check Point Dynamic Routing:

Protocol RFC or Draft

OSPF LLS draft-ietf-ospf-lls-00

OSPF Graceful restart RFC 3623

BGP Graceful restart draft-ietf-idr-restart-08

ClusterXL Administration Guide R80.20 | 151

 Advanced Features and Procedures

Wait for Clustering

When you use any of the Advanced Routing protocols, enable the Wait for Clustering option in the
Gaia Portal. See sk92322 http://supportcontent.checkpoint.com/solutions?id=sk92322.

ClusterXL Administration Guide R80.20 | 152

 ClusterXL Configuration Commands

ClusterXL Configuration Commands

Description
These commands let you configure internal behavior of the Clustering Mechanism.

Important - We do not recommend that you run these commands. These commands
must be run automatically only by the Security Gateway or the Check Point Support. The
only exception to this rule is to changing the CCP mode, as described below.

Important - You must configure all the cluster members in the same way.

Syntax
Notes:
• In Gaia Clish:
Enter the set cluster<ESC><ESC> to see all the available commands.
• In Expert mode:
Run the cphaconf command see all the available commands.
You can run the cphaconf commands only from the Expert mode.
• Syntax legend:
a) Curly brackets or braces {}:
Enclose a list of available commands or parameters, separated by the vertical bar |, from
which user can enter only one.
b) Angle brackets <>:
Enclose a variable - a supported value user needs to specify explicitly.
c) Square brackets or brackets []:
Enclose an optional command or parameter, which user can also enter.
• You can can include these commands in scripts to run them automatically.
The meaning of each command is explained in the next sections.
Description Command in Command in
of Command Gaia Clish Expert Mode
set cluster member idmode cphaconf mem_id_mode
Configure how to show the id id
cluster member in local name name
ClusterXL logs - by its
Member ID or its Member
Name ("Configuring the
Cluster Member ID Mode
in Local Logs" on page
157)

ClusterXL Administration Guide R80.20 | 153

 ClusterXL Configuration Commands

N/A cphaconf set_pnote -d

Register a single Critical <Name of Device> -t
Device (Pnote) on the <Timeout in Sec> -s
cluster member {ok|init|problem} [-p]
[-g] register
("Registering a Critical
Device" on page 158)
N/A cphaconf set_pnote -d
Unregister a single Critical <Name of Device> [-p] [-g]
Device (Pnote) on the unregister
cluster member
("Unregistering a Critical
Device" on page 159)
N/A cphaconf set_pnote -d
Report (change) a state in <Name of Device> -s
a single Critical Device {ok|init|problem} [-g]
(Pnote) on the cluster report
member ("Reporting the
State of a Critical Device"
on page 160)
N/A cphaconf set_pnote -f
Register several Critical <Name of File> [-g]
Devices (Pnotes) from a register
file on the cluster member
("Registering Critical
Devices Listed in a File" on
page 161)
N/A cphaconf set_pnote -a
Unregister all Critical [-g] unregister
Devices (Pnotes) on the
cluster member
("Unregistering All Critical
Devices" on page 162)
set cluster member ccp cphaconf set_ccp
Configure the Cluster auto auto
Control Protocol (CCP) broadcast unicast
mode on the cluster multicast multicast
unicast broadcast
member ("Configuring the
Cluster Control Protocol
(CCP) Mode" on page 163)
set cluster member forwarding cphaconf forward
Configure the Cluster on on
Forwarding Layer (on page off off
12) on the cluster member
(controls the forwarding of
traffic between cluster
members)
Note - For Check Point use
only.

ClusterXL Administration Guide R80.20 | 154

 ClusterXL Configuration Commands

N/A cphaconf debug_data

Print the current cluster
configuration as loaded in
the kernel on the cluster
member (for details, see
sk93306
http://supportcontent.chec
kpoint.com/solutions?id=s
k93306)
N/A cphaconf failover_bond
Start internal failover <bond_name>
between slave interfaces
of specified bond interface
- only in Bond High
Availability mode (for
details, see sk93306
http://supportcontent.chec
kpoint.com/solutions?id=s
k93306)
N/A cphaconf
Configure what happens enable_bond_failover
during a failover after a <bond_name>
Bond already failed over
internally (for details, see
sk93306
http://supportcontent.chec
kpoint.com/solutions?id=s
k93306)
set cluster member admin clusterXL_admin
Initiate manual cluster down down
failover ("Initiating Manual up up
Cluster Failover" on page
164)

List of the Gaia Clish set cluster member commands

set cluster member
admin
down
up
ccp
auto
broadcast
multicast
unicast
forwarding
off
on
idmode
id
name

List of the cphaconf commands

Note - Some commands are not applicable to 3rd party clusters.
cphaconf [-D]

ClusterXL Administration Guide R80.20 | 155

 ClusterXL Configuration Commands

[-c <Cluster Size>]

[-i <Member ID>]
[-n <Cluster ID>]
[-p <Policy ID>]
[-m {1|service} | {2|balance} | {3|primary-up} | {4|active-up}]
[-R a | <Number of Required IF>]
[-t <Sync IF 1>...]
[-d <Non-Monitored IF 1>...]
[-M {0|multicast} | {1|pivot}]
[-l <Cluster Failover Track Mode 0-7>]
[-M multicast|pivot]
[-N <MAC Magic value>]
[-u <Member_Name1,Member_Name2,...>]
start

cphaconf stop

cphaconf [-t <Sync IF 1>...] [-d <Non-Monitored IF 1>...] add

cphaconf clear-secured

cphaconf clear-non-monitored

cphaconf set_ccp {auto|unicast|multicast|broadcast}

cphaconf debug_data

cphaconf delete_link_local [-vs <VSID>] <IF name>

cphaconf set_link_local [-vs <VSID>] <IF name> <Cluster IP>

cphaconf mem_id_mode {id | name}

cphaconf failover_bond <bond_name>

cphaconf [-s] {set|unset|get} var <Kernel Parameter Name> [<Value>]

cphaconf set_pnote -d <Device> -t <Timeout in sec> -s {ok|init|problem} [-p] [-g]

register

cphaconf set_pnote -f <File> [-g] register

cphaconf set_pnote -d <Device> [-p] [-g] unregister

cphaconf set_pnote -a [-g] unregister

cphaconf set_pnote -d <Device> -s {ok|init|problem} [-g] report

ClusterXL Administration Guide R80.20 | 156

 ClusterXL Configuration Commands

Configuring the Cluster Member ID Mode in Local Logs

Important - You must configure all the cluster members in the same way.

Description
You can configure how to show the cluster member in the local ClusterXL logs - by its Member ID
(default) or its Member Name.
This configuration affects these local logs:
• /var/log/messages
• dmesg
• $FWDIR/log/fwd.elg
Note - See Viewing the Cluster Member ID Mode in Local Logs (on page 205).

Syntax
Shell Command
set cluster member idmode
Gaia Clish id
name
cphaconf mem_id_mode
Expert mode id
name

Example
[Expert@Member2:0]# cphaprob names
Current member print mode in local logs is set to: ID
[Expert@Member2:0]#

[Expert@Member2:0]# cphaconf mem_id_mode name

Member print mode in local logs: NAME
[Expert@Member2:0]#

[Expert@Member2:0]# cphaprob names

Current member print mode in local logs is set to: NAME
[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 157

 ClusterXL Configuration Commands

Registering a Critical Device

Important - You must configure all the cluster members in the same way.

Description
You can add a user-defined critical device to the default list of critical devices. Use this command
to register <device> as a critical process, and add it to the list of devices that must run for the
cluster member to be considered active. If <device> fails, then the cluster member is seen as
failed.
If a Critical Device fails to report its state to the cluster member in the defined timeout, the
Critical Device, and by design the cluster member, are seen as failed.
Define the status of the Critical Device that is reported to ClusterXL upon registration. This initial
status can be one of these:
• ok - Critical Device is alive.
• init - Critical Device is initializing. The cluster member is Down. In this state, the cluster
member cannot become Active.
• problem - Critical Device failed. If this state is reported to ClusterXL, the cluster member
immediately goes Down. This causes a failover.

Syntax
Shell Command
Gaia Clish N/A

Expert mode cphaconf set_pnote -d <Name of Critical Device> -t <Timeout in

Sec> -s {ok | init | problem} [-p] [-g] register

Notes:
• The name of the Critical Device must have no more than 15 characters, and must not include
white spaces.
• For no timeout, use the value 0.
• The -p flag makes these changes permanent. After you reboot the cluster member, the status
of critical devices that were registered with this flag is saved.
• The -g flag applies the command to all configured Virtual Systems.
Restrictions:
• Total number of critical devices (pnotes) on cluster member is limited to 16.
• Name of any critical device (pnote) on cluster member is limited to 16 characters.

ClusterXL Administration Guide R80.20 | 158

 ClusterXL Configuration Commands

Unregistering a Critical Device

Important - You must configure all the cluster members in the same way.

Description
Unregistering a user-defined Critical Device (Pnote) means that this device is no longer
considered critical. If a Critical Device was registered with a state problem, before you ran this
command, then after you run this command, the status of the cluster member depends only on
the states of the remaining Critical Devices.

Syntax
Shell Command
Gaia Clish N/A

Expert mode cphaconf set_pnote -d <Name of Critical Device> [-p] [-g]

unregister

Notes:
• The -p flag makes these changes permanent. This means that after you reboot, these Critical
Devices remain unregistered.
• The -g flag applies the command to all configured Virtual Systems.

ClusterXL Administration Guide R80.20 | 159

 ClusterXL Configuration Commands

Reporting the State of a Critical Device

Important - You must configure all the cluster members in the same way.

Description
Use this command to report (change) the state of a Critical Device to ClusterXL.
The reported state can be one of these:
• ok - Critical Device is alive.
• init - Critical Device is initializing. The cluster member is Down. In this state, the cluster
member cannot become Active.
• problem - Critical Device failed. If this state is reported to ClusterXL, the cluster member
immediately goes Down. This causes a failover.
If a Critical Device fails to report its state to the cluster member within the defined timeout, the
Critical Device, and by design the cluster member, are seen as failed. This is true only for Critical
Devices with timeouts. If a Critical Device is registered with the -t 0 parameter, there is no
timeout. Until the Critical Device reports otherwise, the state of the Critical Device is considered to
be the last reported state.

Syntax
Shell Command
Gaia Clish N/A

Expert mode cphaconf set_pnote -d <Name of Critical Device> -s {ok | init

| problem} [-g] report

Notes:
• The -g flag applies the command to all configured Virtual Systems.
• If the <Name of Critical Device> reports its state as "problem", then the cluster member
reports its state as failed.

ClusterXL Administration Guide R80.20 | 160

 ClusterXL Configuration Commands

Registering Critical Devices Listed in a File

Important - You must configure all the cluster members in the same way.

Description
Register all the user-defined Critical Devices listed in the specified file.
This file must be an ASCII file, with each Critical Device defined on a separate line.
Each definition much contain three parameters, which must be separated by a space or a tab
character:
<device> <timeout> <status>

Where:

Parameter Description

<device> The name of the Critical Device.

• Maximal name length is 15 characters
• The name must not include white spaces (space or tab characters).
<timeout> If the Critical Device <device> fails to report its state to the cluster member
within this specified number of seconds, the Critical Device (and by design the
cluster member), are seen as failed.
For no timeout, use the value 0 (zero).

<status> The Critical Device <device> reports one of these statuses to the cluster
member:
• ok - Critical Device is alive.
• init - Critical Device is initializing. The cluster member is Down. In this
state, the cluster member cannot become Active.
• problem - Critical Device failed. If this state is reported to ClusterXL, the
cluster member immediately goes Down. This causes a failover.

Syntax
Shell Command
Gaia Clish N/A

Expert mode cphaconf set_pnote -f <Name of File> [-g] register

Note - The -g flag applies the command to all configured Virtual Systems.

ClusterXL Administration Guide R80.20 | 161

 ClusterXL Configuration Commands

Unregistering All Critical Devices

Important - You must configure all the cluster members in the same way.

Description
Unregisters all critical devices from the cluster member.

Syntax
Shell Command
Gaia Clish N/A

Expert mode cphaconf set_pnote -a [-g] unregister

Notes:
• The -a flag specifies that all Pnotes must be unregistered
• The -g flag applies the command to all configured Virtual Systems

ClusterXL Administration Guide R80.20 | 162

 ClusterXL Configuration Commands

Configuring the Cluster Control Protocol (CCP) Mode

Important - You must configure all the cluster members in the same way.

Description
You can configure the Cluster Control Protocol (CCP) mode ("Selecting the CCP Transport Mode
on the Cluster Members" on page 57) on the cluster members

Syntax
Shell Command
set cluster member ccp
Gaia Clish auto
broadcast
multicast
unicast
cphaconf set_ccp
Expert mode auto
unicast
multicast
broadcast

ClusterXL Administration Guide R80.20 | 163

 ClusterXL Configuration Commands

Initiating Manual Cluster Failover

Description
This command lets you initiate a manual cluster failover ("How to Initiate Cluster Failover" on
page 215) (see sk55081 http://supportcontent.checkpoint.com/solutions?id=sk55081).

Syntax
Shell Command
set cluster member admin
Gaia Clish down
up
clusterXL_admin
Expert mode down
up

Example
Member2> show cluster state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 192.168.20.176 0% STANDBY Member1

2 (local) 192.168.20.177 100% ACTIVE Member2

Active PNOTEs: None

Last member state change event:

Event Code: CLUS-11482
State change: STANDBY -> ACTIVE
Reason for state change: No other ACTIVE member has been found in the cluster
Event time: Sun Jun 3 20:24:35 2018

Last cluster failover event:

Transition to new ACTIVE: Member 1 -> Member 2
Reason: Interface eth1 is down (Cluster Control Protocol packets are not
received)
Event time: Sun Jun 3 20:24:35 2018

Cluster failover count:

Failover counter: 261
Time of counter reset: Sun Jun 3 20:24:35 2018 (reboot)

Member2>

Member2> set cluster member admin down

Setting member to administratively down state ...
Member current state is DOWN
Member2>

Member2> show cluster state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 192.168.20.176 100% ACTIVE Member1

2 (local) 192.168.20.177 0% DOWN Member2

Active PNOTEs: ADMIN

Last member state change event:

Event Code: CLUS-11144
State change: ACTIVE -> DOWN
Reason for state change: ADMIN_DOWN PNOTE
Event time: Sun Jun 3 20:27:19 2018

Last cluster failover event:

ClusterXL Administration Guide R80.20 | 164

 ClusterXL Configuration Commands

Transition to new ACTIVE: Member 2 -> Member 1

Reason: ADMIN_DOWN PNOTE
Event time: Sun Jun 3 20:27:19 2018

Cluster failover count:

Failover counter: 262
Time of counter reset: Sun Jun 3 20:27:19 2018 (reboot)

Member2>

Member2> set cluster member admin up

Setting member to normal operation ...
Member current state is STANDBY
Member2>

Member2> show cluster state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 192.168.20.176 100% ACTIVE Member1

2 (local) 192.168.20.177 0% STANDBY Member2

Active PNOTEs: None

Last member state change event:

Event Code: CLUS-11490
State change: DOWN -> STANDBY
Reason for state change: There is already an ACTIVE member in the cluster (member 1)
Event time: Sun Jun 3 20:27:44 2018

Last cluster failover event:

Transition to new ACTIVE: Member 2 -> Member 1
Reason: ADMIN_DOWN PNOTE
Event time: Sun Jun 3 20:27:19 2018

Cluster failover count:

Failover counter: 262
Time of counter reset: Sun Jun 3 20:27:44 2018 (reboot)

Member2>

ClusterXL Administration Guide R80.20 | 165

CHAPTE R 6

Monitoring and Troubleshooting

Clusters
In This Section:
ClusterXL Monitoring Commands .............................................................................167
Monitoring Cluster Status Using SmartConsole Clients ..........................................210
Working with SNMP Traps .........................................................................................213
cphastart and cphastop ..............................................................................................214
How to Initiate Cluster Failover .................................................................................215
Troubleshooting Dynamic Routing (routed) Pnote ...................................................217
ClusterXL Error Messages .........................................................................................219

ClusterXL Administration Guide R80.20 | 166

 Monitoring and Troubleshooting Clusters

ClusterXL Monitoring Commands

Description
Use the monitoring commands to make sure that the cluster and the cluster members work
properly, and to define Critical Devices. A Critical Device (also known as a Problem Notification, or
pnote) is a special software device on each cluster member, through which the critical aspects for
cluster operation are monitored. When the critical monitored component on a cluster member
fails to report its state on time, or when its state is reported as problematic, the state of that
member is immediately changed to 'Down'.

Syntax
Notes:
• In Gaia Clish:
Enter the show cluster<ESC><ESC> to see all the available commands.
• In Expert mode:
Run the cphaprob command see all the available commands.
You can run the cphaprob commands from Gaia Clish as well.
• Syntax legend:
a) Curly brackets or braces {}:
Enclose a list of available commands or parameters, separated by the vertical bar |, from
which user can enter only one.
b) Angle brackets <>:
Enclose a variable - a supported value user needs to specify explicitly.
c) Square brackets or brackets []:
Enclose an optional command or parameter, which user can also enter.
• You can can include these commands in scripts to run them automatically.
The meaning of each command is explained in the next sections.
Description Command in Command in
of Command Gaia Clish Expert Mode
show cluster state cphaprob [-vs <VSID>]
Show states of cluster state
members and their names
("Monitoring Cluster State"
on page 171)
show cluster members pnotes cphaprob [-l] [-ia] [-e]
Show Critical Devices all list
(Pnotes) and their states problem
on the cluster member
("Monitoring Critical
Devices" on page 176)

ClusterXL Administration Guide R80.20 | 167

 Monitoring and Troubleshooting Clusters

Description Command in Command in

of Command Gaia Clish Expert Mode
show cluster members interfaces cphaprob [-vs all]
Show cluster interfaces on all [-a][-m] if
the cluster member (see secured
"Monitoring Cluster virtual
vlans
Interfaces" on page 182)
show cluster bond cphaprob show_bond
Show cluster bond all [<bond_name>]
configuration on the name <bond_name>
cluster member
("Monitoring Bond
Interfaces" on page 187)
N / A cphaprob
Show groups of bonds on show_bond_groups
the cluster member
("Monitoring Bond
Interfaces" on page 187)
show cluster failover [reset] cphaprob [-reset]
Show (and reset) cluster show_failover
failover statistics on the
cluster member
("Monitoring Cluster
Failover Statistics" on
page 191)
show cluster mmagic cphaprob [-vs <VSID>]
Show configuration of MAC [-k] mmagic
Magic and MAC Forward
Magic on the cluster
member ("Monitoring MAC
Magic and MAC Forward
Magic Values" on page
192)
show cluster statistics sync cphaprob [-reset]
Show Delta Sync statistics [reset] syncstat
on the cluster member
("Monitoring Delta
Synchronization" on page
193)
show cluster statistics cphaprob [-reset] ldstat
Show Delta Sync statistics transport [reset]
for the Connections table
on the cluster member
("Viewing Cluster Delta
Sync Statistics for
Connections Table" on
page 203)

ClusterXL Administration Guide R80.20 | 168

 Monitoring and Troubleshooting Clusters

Description Command in Command in

of Command Gaia Clish Expert Mode
show cluster members interfaces cphaprob [-vs all] -a if
Show the Cluster Control virtual
Protocol (CCP) mode on
the cluster member (see
"Monitoring Cluster
Interfaces" on page 182)
show cluster members igmp cphaprob igmp
Show the IGMP
membership of the cluster
member ("Viewing IGMP
Status" on page 202)
show cluster members ips cphaprob tablestat
Show cluster unique IP's
table on the cluster
member ("Viewing Cluster
IP Addresses" on page
204)
show cluster members idmode cphaprob names
Show the Cluster Member
ID Mode in local logs - by
Member ID (default) or
Member Name ("Viewing
the Cluster Member ID
Mode in Local Logs" on
page 205).
show ospf interfaces [detailed] cphaprob routedifcs
Show interfaces, which the
RouteD monitors on the
cluster member when you
configure OSPF ("Viewing
Interfaces Monitored by
RouteD" on page 206)
show cluster roles cphaprob roles
Show roles of RouteD
daemon on cluster
members ("Viewing Roles
of RouteD Daemon on
Cluster Members" on page
207)
N / A cphaprob corr
Show Cluster Correction cphaprob -c {a | d |f}
Statistics ("Viewing Cluster
Correction Statistics" on
page 208)

List of the Gaia Clish show cluster commands

show cluster
bond
all
name <Name of Bond>
failover [reset]
ClusterXL Administration Guide R80.20 | 169
 Monitoring and Troubleshooting Clusters

members
idmode
igmp
interfaces
all
secured
virtual
vlans
ips
pnotes
all
problem
mmagic
roles
state
statistics
sync [reset]
transport [reset]

List of the cphaprob commands

Note - Some commands are not applicable to 3rd party clusters.
cphaprob [-vs <VSID>] state
cphaprob [-reset] show_failover
cphaprob [-vs <VSID>][-k][-S] mmagic
cphaprob names
cphaprob [-reset] [-a] syncstat
cphaprob [-reset] ldstat
cphaprob [-l] [-i[a]] [-e] list
cphaprob [-vs all] [-a] [-m] if
cphaprob show_bond [<bond_name>]
cphaprob show_bond_groups
cphaprob igmp
cphaprob tablestat
cphaprob routedifcs
cphaprob roles
cphaprob {corr | -c {a | d |f}}

ClusterXL Administration Guide R80.20 | 170

 Monitoring and Troubleshooting Clusters

Monitoring Cluster State

Description
Run this command to monitor the cluster status (after you set up the cluster).

Syntax
Shell Command
Gaia Clish 1. set virtual-system <VSID>
2. show cluster state

Expert mode cphaprob [-vs <VSID>] state

Example
MEM2> cphaprob state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 (local) 150.150.150.2 0% STANDBY MEM2

2 150.150.150.1 100% ACTIVE MEM1

Active PNOTEs: None

Last member state change event:

Event Code: CLUS-111490
State change: DOWN -> STANDBY
Reason for state change: There is already an ACTIVE member in the cluster (member 2)
Event time: Sun Jun 3 09:50:46 2018

Last cluster failover event:

Transition to new ACTIVE: Member 1 -> Member 2
Reason: Interface eth1 is down (Cluster Control Protocol packets are not
received)
Event time: Sun Jun 3 09:50:18 2018

Cluster failover count:

Failover counter: 5
Time of counter reset: Sun Jun 3 09:50:46 2018 (reboot)

MEM2>

Description of the output fields:

Field Description
Cluster Mode Can be one of these:
• Load Sharing (Multicast).
• Load Sharing (Unicast).
• High Availability (Primary Up).
• High Availability (Active Up).
• Virtual System Load Sharing
• For third-party clustering products: Service, refer to Clustering
Definitions and Terms, for more information.

ClusterXL Administration Guide R80.20 | 171

 Monitoring and Troubleshooting Clusters

Field Description
ID • In the High Availability mode - indicates the cluster member
priority, as configured in the cluster object in SmartConsole.
• In Load Sharing mode - indicates the cluster member ID, as
configured in the cluster object in SmartConsole.
Unique Address Usually, shows the IP addresses of the Sync interfaces.
In some cases, can show IP addresses of other cluster interfaces.

Assigned Load • In the ClusterXL High Availability mode - shows the Active
cluster member with 100% load, and all other Standby cluster
members with 0% load.
• In ClusterXL Load Sharing modes (Unicast and Multicast) -
shows allActive cluster members with 100% load.
State • In the ClusterXL High Availability mode, only one cluster
member in a fully-functioning cluster must be ACTIVE, and the
other cluster members must be in the STANDBY state.
• In the ClusterXL Load Sharing modes (Unicast and Multicast),
all cluster members in a fully-functioning cluster must be
ACTIVE.
• In 3rd-party clustering configuration, all cluster members in a
fully-functioning cluster must be ACTIVE. This is because this
command only reports the status of the Full Synchronization
process.
See the summary table below.

Name Shows the names of cluster members objects as configured in

SmartConsole.

Active PNOTEs Shows the Critical Devices ("Monitoring Critical Devices" on page
176) that report theirs states as "problem".

Last member state change Shows information about the last time this cluster member
event changed its cluster state.

Event Code Shows an event code.

For information, see sk125152
http://supportcontent.checkpoint.com/solutions?id=sk125152.

State change Shows the previous cluster state and the new cluster state of this
cluster member.

Reason for state change Shows the reason why this cluster member changed its cluster
state.

Event time Shows the date and the time when this cluster member changed
its cluster state.

Last cluster failover event Shows information about the last time a cluster failover occurred.

ClusterXL Administration Guide R80.20 | 172

 Monitoring and Troubleshooting Clusters

Field Description
Transition to new ACTIVE Shows which cluster member became the new Active.

Reason Shows the reason for the last cluster failover.

Event time Shows the date and the time of the last cluster failover.

Cluster failover count Shows information about the cluster failovers.

Failover counter Shows the number of cluster failovers since the boot.
Notes:
• This value survives reboot.
• This counter is synchronized between cluster members.
Time of counter reset Shows the date and the time of the last counter reset, and the
reset initiator.

When you examine the state of the cluster member, consider whether it forwards packets, and
whether it has a problem that prevents it from forwarding packets. Each state reflects the result
of a test on critical devices. This table shows the possible cluster states, and whether or not they
represent a problem.

Cluster Description Forwarding Is this

State packets? state a
problem?
ACTIVE Everything is OK. Yes No

ACTIVE(!) A problem was detected, but the cluster member Yes Yes
still forwards packets, because it is the only member
ACTIVE(!F)
in the cluster, or because there are no other Active
ACTIVE(!P) members in the cluster. In any other situation, the
ACTIVE(!FP) state of the member is Down.
• ACTIVE(!) - See above.
• ACTIVE(!F) - See above. Cluster member is in
the freeze state.
• ACTIVE(!P) - See above. This is the Pivot
cluster member in Load Sharing Unicast mode.
• ACTIVE(!FP) - See above. This is the Pivot
cluster member in Load Sharing Unicast mode
and it is in the freeze state.
DOWN One of the Critical Devices ("Monitoring Critical No Yes
Devices" on page 176) reports its state as
"problem".

LOST The peer cluster member lost connectivity to this No Yes

local cluster member (for example, while the peer
cluster member is rebooted).

ClusterXL Administration Guide R80.20 | 173

 Monitoring and Troubleshooting Clusters

Cluster Description Forwarding Is this

State packets? state a
problem?
READY State Ready means that the cluster member No No
recognizes itself as a part of the cluster and is
literally ready to go into action, but, by design,
something prevents it from taking action. Possible
reasons that the cluster member is not yet Active
include:
• Not all required software components were
loaded and initialized yet and/or not all
configuration steps finished successfully yet.
Before a cluster member becomes Active, it
sends a message to the rest of the cluster
members, to check if it can become Active. In
High Availability mode it checks if there is already
an Active member and in Load Sharing Unicast
mode it checks if there is a Pivot member
already. The member remains in the Ready state
until it receives the response from the rest of the
cluster members and decides which, which state
to choose next (Active, Standby, Pivot, or
non-Pivot).
• Software installed on this cluster member has a
higher version than all the other cluster
members. For example, when a cluster is
upgraded from one version of Check Point
Security Gateway to another, and the cluster
members have different versions of Check Point
Security Gateway, the cluster members with the
new version have the Ready state, and the cluster
members with the previous version have the
Active/Active Attention state.
• If the software installed on all cluster members
includes CoreXL, which is installed by default on
multi-CPU computers, a member in Ready state
can have a higher number of CoreXL instances
than other members.
See sk42096
http://supportcontent.checkpoint.com/solutions?id=s
k42096 for a solution.

STANDBY Applies only to a High Availability mode. Means that No No

the cluster member waits for an Active cluster
member to fail in order to start packet forwarding.

ClusterXL Administration Guide R80.20 | 174

 Monitoring and Troubleshooting Clusters

Cluster Description Forwarding Is this

State packets? state a
problem?
BACKUP Applies only to a VSX cluster in Virtual System Load No No
Sharing mode with three or more cluster members
configured.
State of a Virtual System on a third (and so on) VSX
cluster member.

INIT The cluster member is in the phase after the boot No No

and until the Full Sync completes.

ClusterXL Administration Guide R80.20 | 175

 Monitoring and Troubleshooting Clusters

Monitoring Critical Devices

Description
When a Critical Device fails, the cluster member is considered to have failed. To see the list of
Critical Devices on a cluster member, and of all the other cluster members, run the commands
listed below on the cluster member.
There are a number of built-in Critical Devices, and the Administrator can define additional
Critical Devices.
The Critical Devices are:

Critical Device Description Meaning of "OK" state Meaning of "problem"

state
Problem Monitors all the None of the Critical At least one of the
Notification Critical Devices. Devices on this cluster Critical Devices on this
member report its cluster member
state as problem. reports its state as
problem.

Init Monitors if "HA This cluster member

module" was initialized receives cluster state
successfully. See information from peer
sk36372 cluster members.
http://supportcontent.c
heckpoint.com/solutio
ns?id=sk36372.

Interface Active Monitors the state of All cluster interfaces At least one of the
Check cluster interfaces. on this cluster cluster interfaces on
member are up (CCP this cluster member is
packets are sent and down (CCP packets are
received on all cluster not sent and/or
interfaces). received on time).

Load Balancing Pnote is currently not

Configuration used (see sk36373
http://supportcontent.c
heckpoint.com/solutio
ns?id=sk36373).

Recovery Delay Monitors the state of a State of a Virtual State of a Virtual

Virtual System (see System can be System can not be
sk92353 changed on this changed yet on this
http://supportcontent.c cluster member. cluster member.
heckpoint.com/solutio
ns?id=sk92353).

ClusterXL Administration Guide R80.20 | 176

 Monitoring and Troubleshooting Clusters

Critical Device Description Meaning of "OK" state Meaning of "problem"

state
CoreXL Monitors CoreXL Number of configured Number of configured
Configuration configuration for CoreXL FW instances CoreXL FW instances
inconsistencies on all on this cluster on this cluster
cluster members. member is the same member is different
as on all peer cluster from peer cluster
members. members.

Fullsync Monitors if Full Sync This cluster member This cluster member
on this cluster completed Full Sync was not able to
member completed successfully. complete Full Sync.
successfully.

Policy Monitors if the Security This cluster member Security Policy is not
Policy is installed. successfully installed currently installed on
Security Policy. this cluster member.

fwd Monitors the Security fwd daemon on this fwd daemon on this
Gateway process cluster member cluster member did
called fwd. reported its state on not report its state on
time. time.

cphad Monitors the ClusterXL cphamcset daemon cphamcset daemon

process called on this cluster on this cluster
cphamcset. member reported its member did not report
also see the state on time. its state on time.
$FWDIR/log/cphamc
set.elg file.

routed Monitors the Gaia routed daemon on routed daemon on

process called this cluster member this cluster member
routed. reported its state on did not report its state
time. on time.

cvpnd Monitors the Mobile cvpnd daemon on this cvpnd daemon on this
Access back-end cluster member cluster member did
process called cvpnd. reported its state on not report its state on
This pnote appears if time. time.
Mobile Access
Software Blade is
enabled.

ted Monitors the Threat ted daemon on this ted daemon on this
Emulation process cluster member cluster member did
called ted. reported its state on not report its state on
time. time.

ClusterXL Administration Guide R80.20 | 177

 Monitoring and Troubleshooting Clusters

Critical Device Description Meaning of "OK" state Meaning of "problem"

state
VSX Monitors all Virtual On VS0, means that Minimum of blocking
Systems in VSX states of all Virtual states of all Virtual
cluster. Systems are not Down Systems is not "active"
on this cluster (the VSIDs will be
member. printed on the line
Problematic
On other Virtual
VSIDs:) on this
Systems, means that
cluster member.
VS0 is alive on this
cluster member.

Instances This pnote appears in The number of CoreXL There is a mismatch

VSX HA mode (not FW instances in the between the number of
VSLS) cluster. received CCP packet CoreXL FW instances
matches the number of in the received CCP
loaded CoreXL FW packet and the number
instances on this VSX of loaded CoreXL FW
cluster member or this instances on this VSX
Virtual System. cluster member or this
Virtual System (see
sk106912
http://supportcontent.c
heckpoint.com/solutio
ns?id=sk106912).

Hibernating This pnote appears in This Virtual System is

VSX VSLS mode cluster in "Backup"
with 3 and more (hibernated) state on
cluster members. This this cluster member.
pnote shows if this
Virtual System is in
"Backup" (hibernated)
state. Also see
sk114557
http://supportcontent.c
heckpoint.com/solutio
ns?id=sk114557.

admin_down Monitors the Critical User ran the

Device admin_down. clusterXL_admin
down command on this
cluster member.
See Appendix A - The
clusterXL_admin
Script (on page 220).

ClusterXL Administration Guide R80.20 | 178

 Monitoring and Troubleshooting Clusters

Critical Device Description Meaning of "OK" state Meaning of "problem"

state
host_monitor Monitors the Critical All monitored IP At least one of the
Device addresses on this monitored IP
host_monitor. cluster member addresses on this
replied to pings. cluster member did
User executed the
$FWDIR/bin/cluste not reply to at least
rXL_monitor_ips one ping.
script.
See Appendix B - The
clusterXL_monitor_ips
Script (on page 222).

a name of a user space User executed the All monitored user At least one of the
process (except fwd, $FWDIR/bin/cluste space processes on monitored user space
routed, cvpnd, ted) rXL_monitor_proce this cluster member on this cluster
ss script. are running. member processes is
See Appendix C - The not running.
clusterXL_monitor_pro
cess Script (on page
224).

Syntax
Shell Command
Gaia Clish show cluster pnotes {all | problem}

Expert mode cphaprob [-l] [-ia] [-e] list

Where:

Command Description
show cluster pnotes all Shows cluster full list of Critical Devices

show cluster pnotes problem Prints the list of all the "Built-in Devices" and the
"Registered Devices"

cphaprob -l Prints the list of all the "Built-in Devices" and the
"Registered Devices"

cphaprob -i list When there are no issues on the cluster member, shows:
There are no pnotes in problem state
When a Critical Device reports a problem, prints only the
Critical Device that reports its state as "problem".

cphaprob -ia list When there are no issues on the cluster member, shows:
There are no pnotes in problem state
When a Critical Device reports a problem, prints the
Critical Device "Problem Notification" and the
Critical Device that reports its state as "problem"

ClusterXL Administration Guide R80.20 | 179

 Monitoring and Troubleshooting Clusters

Command Description
cphaprob -e list When there are no issues on the cluster member, shows:
There are no pnotes in problem state
When a Critical Device reports a problem, prints only the
Critical Device that reports its state as "problem"

Example
Critical Device fwd reports its state as problem because the fwd process is not up.
[Expert@Member2:0]# cphaprob -l list

Built-in Devices:

Device Name: Interface Active Check

Current state: OK

Device Name: Recovery Delay

Current state: OK

Device Name: CoreXL Configuration

Current state: OK

Registered Devices:

Device Name: Fullsync

Registration number: 0
Timeout: none
Current state: OK
Time since last report: 1221.5 sec

Device Name: Policy

Registration number: 1
Timeout: none
Current state: OK
Time since last report: 1221.5 sec

Device Name: routed

Registration number: 2
Timeout: none
Current state: OK
Time since last report: 1277.6 sec

Device Name: cphad

Registration number: 3
Timeout: 30 sec
Current state: OK
Time since last report: 1554.4 sec
Process Status: UP

Device Name: Init

Registration number: 4
Timeout: none
Current state: OK
Time since last report: 1522.7 sec

Device Name: fwd

Registration number: 5
Timeout: 30 sec
Current state: problem
Time since last report: 45.3 sec
Process Status: NOT UP
ClusterXL Administration Guide R80.20 | 180
 Monitoring and Troubleshooting Clusters

Device Name: ted

Registration number: 6
Timeout: 600 sec
Current state: OK
Time since last report: 2 sec

Device Name: cvpnd

Registration number: 7
Timeout: none
Current state: OK
Time since last report: 1.4 sec

[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 181

 Monitoring and Troubleshooting Clusters

Monitoring Cluster Interfaces

Description
This command lets you see the state of the cluster member interfaces and the virtual cluster
interfaces.
ClusterXL treats the interfaces as Critical Devices. ClusterXL makes sure that interfaces can send
and receive CCP packets.
ClusterXL also sets the required minimal number of functional interfaces to the largest number of
functional interfaces ClusterXL detected since the last reboot. If the number of functional
interfaces is less than the required number, ClusterXL declares the cluster member as failed and
starts a failover. The same applies to the synchronization interfaces, where only good
synchronization interfaces are counted.
When an interface is DOWN, it means that the interface cannot receive or send CCP packets, or
both. An interface may also be able to receive, but not send CCP packets. The time you see in the
command's output is the number of seconds that elapsed since the interface was last able to
receive or send a CCP packet.

Syntax
Shell Command
Gaia Clish 1. set virtual-system <VSID>
2. show cluster members interfaces {all | secured |
virtual | vlans}

Expert mode cphaprob [-vs all] [-a] [-m] if

Where:

Command Description
show cluster members interfaces all Shows full list of all cluster interfaces:
• including the number of required
interfaces
• including Network Objective
• including VLAN monitoring mode, or
list of monitored VLAN interfaces
show cluster members interfaces secured Shows only cluster interfaces (Cluster and
Sync) and their states:
• without Network Objective
• without VLAN monitoring mode
• without monitored VLAN interfaces

ClusterXL Administration Guide R80.20 | 182

 Monitoring and Troubleshooting Clusters

Command Description
show cluster members interfaces virtual Shows full list of cluster virtual interfaces
and their states:
• including the number of required
interfaces
• including Network Objective
• without VLAN monitoring mode
• without monitored VLAN interfaces
show cluster members interfaces vlans Shows only monitored VLAN interfaces

cphaprob if Shows only cluster interfaces (Cluster and

Sync) and their states:
• without Network Objective
• without VLAN monitoring mode
• without monitored VLAN interfaces
cphaprob -a if Shows full list of cluster interfaces and
their states:
• including the number of required
interfaces
• including Network Objective
• without VLAN monitoring mode
• without monitored VLAN interfaces
cphaprob -a -m if Shows full list of all cluster interfaces and
cphaprob -am if their states:
• including the number of required
interfaces
• including Network Objective
• including VLAN monitoring mode, or
list of monitored VLAN interfaces

Output
The output of these commands must be identical to the configuration in the cluster object's
Network Management page.

Example
Member2> show cluster members interfaces all

CCP mode: Automatic

Required interfaces: 3
Required secured interfaces: 1

eth0 Non-Monitored non sync(non secured)

eth3 UP non sync(non secured), unicast
eth4 UP non sync(non secured), unicast
bond0 UP sync(secured), unicast, bond High Availability

ClusterXL Administration Guide R80.20 | 183

 Monitoring and Troubleshooting Clusters

Virtual cluster interfaces: 2

eth3 192.168.151.7
eth4 192.168.1.5

No VLANs are monitored on the member

Member2>

Description of the output fields:

Field, or Text Description

CCP mode Shows the CCP mode (see "Selecting the CCP Transport Mode
on the Cluster Members" on page 57) that administrator
configured with the set cluster member ccp <mode>
command:
• Automatic
• Broadcast
• Multicast
• Unicast
Required interfaces Shows the total number of monitored cluster interfaces,
including the Sync interface.
This number is based on the configuration of the cluster object >
Network Management page.

Required secured interfaces Shows the total number of the required Sync interfaces.
This number is based on the configuration of the cluster object >
Network Management page.

Non-Monitored This means that cluster member does not monitor the state of
this interface.
In <tp_con, in the cluster object > Network Management page,
administrator configured the Network Type Private for this
interface.

UP This means that cluster member monitors the state of this

interface.
The current cluster state of this interface is UP, which means
this interface can send and receive CCP packets.
In <tp_con, in the cluster object > Network Management page,
administrator configured one of these Network Types for this
interface: Cluster, Sync, or Cluster + Sync.

ClusterXL Administration Guide R80.20 | 184

 Monitoring and Troubleshooting Clusters

Field, or Text Description

DOWN This means that cluster members monitors the state of this
interface.
The current cluster state of this interface is DOWN, which
means this interface cannot send CCP packets, receive CCP
packets, or both.
In <tp_con, in the cluster object > Network Management page,
administrator configured one of these Network Types for this
interface: Cluster, Sync, or Cluster + Sync.

sync(secured) This interface role means that this is a Sync interface.

In <tp_con, in the cluster object > Network Management page,
administrator configured one of these Network Types for this
interface: Sync, or Cluster + Sync.

non sync(non secured) This interface role means that this interface does not transfer
Delta Sync packets.
In <tp_con, in the cluster object > Network Management page,
administrator configured the Network Type Cluster for this
interface.

unicast This is the current CCP mode on this cluster interface.

This is the current CCP mode on this cluster interface.

This is the current CCP mode on this cluster interface.

Virtual cluster interfaces Shows the total number of the configure virtual cluster
interfaces.
This number is based on the configuration of the cluster object >
Network Management page.

No VLANs are monitored on Shows the VLAN monitoring mode - there are no VLAN
the member interfaces configured on the cluster interfaces.
For more information, seeVLAN Support in ClusterXL (on page
100).

Monitoring mode is Monitor all Shows the VLAN monitoring mode - there are some VLAN
VLANs: All VLANs are interfaces configured on the cluster interfaces, and cluster
monitored member monitors all VLAN IDs.
For more information, seeVLAN Support in ClusterXL (on page
100).

Monitoring mode is Monitor Shows the VLAN monitoring mode - there are some VLAN
specific VLAN: Only specified interfaces configured on the cluster interfaces, and cluster
VLANs are monitored member monitors only specific VLAN IDs.
For more information, seeVLAN Support in ClusterXL (on page
100).

ClusterXL Administration Guide R80.20 | 185

 Monitoring and Troubleshooting Clusters

ClusterXL Administration Guide R80.20 | 186

 Monitoring and Troubleshooting Clusters

Monitoring Bond Interfaces

Description
Shows the configuration of bond interfaces and their slave interfaces.

Syntax
Shell Command
Gaia Clish 1. show cluster bond {all | name <bond_name>}
2. show bonding groups

Expert mode cphaprob show_bond [<bond_name>]

cphaprob show_bond_groups

Where:

Command Description
show cluster bond all Shows configuration of all configured bond
show bonding groups interfaces
cphaprob show_bond

show cluster bond name <bond_name> Shows configuration of the specified bond
interface
cphaprob show_bond <bond_name>

cphaprob show_bond_groups Shows the configured Groups of Bonds ("Group of

Bonds" on page 118) and their settings.

Example 1
[Expert@Member2:0]# cphaprob show_bond

|Slaves |Slaves |Slaves

Bond name |Mode |State |configured |link up |required
-----------+-------------------+------+-----------+--------+--------
bond1 | High Availability | UP | 2 | 2 | 1

Legend:
-------
UP! - Bond interface state is UP, yet attention is required
Slaves configured - number of slave interfaces configured on the bond
Slaves link up - number of operational slaves
Slaves required - minimal number of operational slaves required for bond to be UP

[Expert@Member2:0]#

Member2> show bonding groups

Bonding Interface: 1
Bond Configuration
xmit-hash-policy Not configured
down-delay 200
primary Not configured
lacp-rate Not configured
mode active-backup
up-delay 200
mii-interval 100
Bond Interfaces
eth3
eth4
Member2>

ClusterXL Administration Guide R80.20 | 187

 Monitoring and Troubleshooting Clusters

Description of the output fields for the "cphaprob show_bond" and

"show cluster bond all" commands:

Field Description
Bond name Name of the Gaia bonding group.

Mode Bonding mode of this Gaia bonding group.

One of these:
• High Availability
• Load Sharing
State State of the Gaia bonding group:
• UP - Bond interface is fully operational
• UP! - Bond interface state is UP, yet attention is required
• DOWN - Bond interface failed
Slaves configured Total number of physical slave interfaces configured in this Gaia
bonding group.

Slaves link up Number of operational physical slave interfaces in this Gaia

bonding group.

Slaves required Minimal number of operational physical slave interfaces

required for the state of this Gaia bonding group to be UP.

Example 2
[Expert@Member2:0]# cphaprob show_bond bond1

Bond name: bond1

Bond mode: High Availability
Bond status: UP

Configured slave interfaces: 2

In use slave interfaces: 2
Required slave interfaces: 1

Slave name | Status | Link

----------------+-----------------+-------
eth4 | Active | Yes
eth3 | Backup | Yes

[Expert@Member2:0]#

Description of the output fields for the "cphaprob show_bond <bond_name>" and
"show cluster bond name <bond_name>" commands:

Field Description
Bond name Name of the Gaia bonding group.

Bond mode Bonding mode of this Gaia bonding group. One of these:
• High Availability
• Load Sharing

ClusterXL Administration Guide R80.20 | 188

 Monitoring and Troubleshooting Clusters

Field Description
Bond status Status of the Gaia bonding group. One of these:
• UP - Bond interface is fully operational
• UP! - Bond interface state is UP, yet attention is required
• DOWN - Bond interface failed
Configured slave interfaces Total number of physical slave interfaces configured in this Gaia
bonding group.

In use slave interfaces Number of operational physical slave interfaces in this Gaia
bonding group.

Required slave interfaces Minimal number of operational physical slave interfaces

required for the state of this Gaia bonding group to be UP.

Slave name Names of physical slave interfaces configured in this Gaia

bonding group.

Status Status of physical slave interfaces in this Gaia bonding group.

One of these:
• Active - In High Availability or Load Sharing bonding mode.
This slave interface is currently handling traffic.
• Backup - In High Availability bonding mode only. This slave
interface is ready and can support internal bond failover.
• Not Available - In High Availability or Load Sharing
bonding mode. The physical link on this slave interface is
lost, or this Cluster member is in status Down. The bond
cannot failover internally in this state.
Link State of the physical link on the physical slave interfaces in this
Gaia bonding group. One of these:
• Yes - Link is present
• No - Link is lost

Example 3
[Expert@Member2:0]# cphaprob show_bond_groups

| Required | Bonds | Bonds

Group of bonds name | State | active bonds | in group | status
--------------------+-----------+--------------+----------+--------+
GoB0 | UP | 1 | |
| | | bond1 | UP
| | | bond2 | UP

Legend:
---------
Bonds in group - a list of the bonds in the bond group
Required active bonds - number of required active bonds
[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 189

 Monitoring and Troubleshooting Clusters

Description of the output fields for the "cphaprob show_bond_groups" command:

Field Description
Group of bonds name Name of the Group of Bonds (on page 118).

State State of the Group of Bonds. One of these:

• UP - Group of Bonds is fully operational
• DOWN - Group of Bonds failed
Required active bonds Number of required active bonds in this Group of Bonds.

Bonds in group Names of the Gaia bond interfaces configured in this Group of
Bonds.

Bonds status State of the Gaia bond interface. One of these:

• UP - Bond interface is fully operational
• DOWN - Bond interface failed

ClusterXL Administration Guide R80.20 | 190

 Monitoring and Troubleshooting Clusters

Monitoring Cluster Failover Statistics

Description
This command lets you see the cluster failover statistics on the cluster member - number of
failovers that happened, reason, and the time of the last failover event.

Syntax
Shell Command
Gaia Clish show cluster failover

Expert mode cphaprob show_failover

Example
[Expert@Member2:0]# cphaprob show_failover

Last cluster failover event:

Transition to new ACTIVE: Member 2 -> Member 1
Reason: Available on member 2
Event time: Mon Apr 23 14:38:44 2018

Cluster failover count:

Failover counter: 2
Time of counter reset: Mon Apr 23 13:14:41 2018 (reboot)

[Expert@Member2:0]#
[Expert@Member2:0]# clusterXL_admin down
Setting member to administratively down state ...
Member current state is Down
[Expert@Member2:0]#
[Expert@Member2:0]# cphaprob show_failover

Last cluster failover event:

Transition to new ACTIVE: Member 1 -> Member 2
Reason: ADMIN_DOWN PNOTE
Event time: Mon Apr 23 16:20:23 2018

Cluster failover count:

Failover counter: 3
Time of counter reset: Mon Apr 23 13:14:41 2018 (reboot)

[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 191

 Monitoring and Troubleshooting Clusters

Monitoring MAC Magic and MAC Forward Magic Values

Description
This command lets you see the values of the MAC Magic and MAC Forward Magic parameters on
the cluster member (for details, see sk25977
http://supportcontent.checkpoint.com/solutions?id=sk25977).

Syntax
Shell Command
Gaia Clish 1. set virtual-system <VSID>
2. show cluster mmagic

Expert mode cphaprob [-vs <VSID>][-k] mmagic

Example 1
[Expert@Member2:0]# cphaprob mmagic

Configuration mode: Automatic

Configuration phase: Stable

MAC magic: 1
MAC forward magic: 254

Used MAC magic values: None.

[Expert@Member2:0]#

Example 2
[Expert@Member2:0]# cphaprob mmagic

Configuration mode: Automatic

Configuration phase: Stable

MAC magic: 2
MAC forward magic: 1

Used MAC magic values:

0x01(001)

[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 192

 Monitoring and Troubleshooting Clusters

Monitoring Delta Synchronization

Heavily loaded clusters and clusters with geographically separated members pose special
challenges. High connection rates, and large distances between the members can lead to delays
that affect the operation of the cluster.
Monitor the operation of the State Synchronization mechanism in highly loaded and distributed
clusters.
Perform these troubleshooting steps:
1. Shows the Delta Sync statistics counters:
Shell Command
Gaia Clish show cluster statistics sync

Expert mode cphaprob syncstat

2. Examine and understand the the Delta Sync statistics.

3. Tune the relevant synchronization global configuration parameters.
4. Reset the Delta Sync statistics counters:
Shell Command
Gaia Clish show cluster statistics sync reset

Expert mode cphaprob -reset syncstat

5. Examine the Delta Sync statistics to see if the problem is solved (see "Output example" on
page 194).
6. Solve any identified problem (see "Synchronization Troubleshooting Options" on page 199).

ClusterXL Administration Guide R80.20 | 193

 Monitoring and Troubleshooting Clusters

Output example
This section describes and explains the output parameters of the show cluster statistics
sync and cphaprob syncstat commands.
Example output from a cluster member:
Delta Sync Statistics

Sync status: OK

Drops:
Lost updates................................. 0
Lost bulk update events...................... 0
Oversized updates not sent................... 0

Sync at risk:
Sent reject notifications.................... 0
Received reject notifications................ 0

Sent updates:
Total generated sync messages................ 12316
Sent retransmission requests................. 0
Sent retransmission updates.................. 0
Peak fragments per update.................... 1

Received updates:
Total received updates....................... 12
Received retransmission requests............. 0

Queue sizes (num of updates):

Sending queue size........................... 512
Receiving queue size......................... 256
Fragments queue size......................... 50

Timers:
Delta Sync interval (ms)..................... 100

Reset on Sun Jun 3 14:37:26 2018 (triggered by fullsync).

The "Sync status:" section

This section shows the status of the Delta Sync mechanism. One of these:
• Sync status: OK
• Sync status: Off - Full-sync failure
• Sync status: Off - Policy installation failure
• Sync status: Off - Cluster module not started
• Sync status: Off - SIC failure
• Sync status: Off - Full-sync checksum error
• Sync status: Off - Full-sync received queue is full
• Sync status: Off - Release version mismatch
• Sync status: Off - Connection to remote member timed-out
• Sync status: Off - Connection terminated by remote member
• Sync status: Off - Could not start a connection to remote member

ClusterXL Administration Guide R80.20 | 194

 Monitoring and Troubleshooting Clusters

• Sync status: Off - cpstart

• Sync status: Off - cpstop
• Sync status: Off - Manually disabled sync
• Sync status: Off - Was not able to start for more than X second
• Sync status: Off - Boot
• Sync status: Off - Connectivity Upgrade (CU)
• Sync status: Off - cphastop
• Sync status: Off - Policy unloaded
• Sync status: Off - Hibernation
• Sync status: Off - OSU deactivated
• Sync status: Off - Sync interface down
• Sync status: Fullsync in progress
• Sync status: Problem (Able to send sync packets, unable to receive sync
packets)
• Sync status: Problem (Able to send sync packets, saving incoming sync
packets)
• Sync status: Problem (Able to send sync packets, able to receive sync
packets)
• Sync status: Problem (Unable to send sync packets, unable to receive sync
packets)
• Sync status: Problem (Unable to send sync packets, saving incoming sync
packets)
• Sync status: Problem (Unable to send sync packets, able to receive sync
packets)

The "Drops:" section

This section shows statistics for drops on the Delta Sync network.

Field Description
Lost updates Shows how many Delta Sync updates this cluster member considers as lost
(based on sequence numbers in CCP packets).
If this counter shows a value greater than 0, this cluster member lost Delta
Sync updates.
Possible mitigation:
Increase the size of the Sending Queue and the size of the Receiving Queue:
• Increase the size of the Sending Queue (see "Increasing the Size of the
Sending Queue" on page 199), if the counter Received reject notification is
increasing.
• Increase the size of the Receiving Queue (see "Increasing the Size of the
Receiving Queue" on page 199), if the counter Received reject notification
is not increasing.

ClusterXL Administration Guide R80.20 | 195

 Monitoring and Troubleshooting Clusters

Field Description
Lost bulk update Shows how many times this cluster member missed Delta Sync updates.
events
(bulk update = twice the size of the local receiving queue)
This counter increases when this cluster member receives a Delta Sync
update with a sequence number much greater than expected. This probably
indicates some networking issues that cause massive packet drops.
This counter increases when the amount of missed Delta Sync updates is
more than twice the local Receiving Queue Size.
Possible mitigation:
• If the counter's value is steady, this might indicate a one time
synchronization problem that can be resolved by running manual Full
Sync. See sk37029
http://supportcontent.checkpoint.com/solutions?id=sk37029.
• If the counter's value keeps increasing, probable there are some
networking issues. Increase the sizes of both the Receiving Queue (see
"Increasing the Size of the Receiving Queue" on page 199) and Sending
Queue (see "Increasing the Size of the Sending Queue" on page 199).
Oversized Shows how many oversized Delta Sync updates were discarded before
updates not sent sending them.
This counter increases when Delta Sync update is larger than the local
Fragments Queue Size.
Possible mitigation:
• If the counter's value is steady, increase the size of the Sending Queue
(see "Increasing the Size of the Sending Queue" on page 199).
• If the counter's value keeps increasing, contact Check Point Support
https://www.checkpoint.com/support-services/contact-support/.

The "Sync at risk:" section

This section shows statistics that the Sending Queue (see "Increasing the Size of the Sending
Queue" on page 199) is at full capacity and rejects Delta Sync retransmission requests.

Field Description
Sent reject Shows how many times this cluster member rejected Delta Sync
notifications retransmission requests from its peer cluster members, because this cluster
member does not hold the requested Delta Sync update anymore.

Received reject Shows how many reject notifications this cluster member received from its
notification peer cluster members.

The "Sent updates:" section

This section shows statistics for Delta Sync updates sent by this cluster member to its peer
cluster members.

ClusterXL Administration Guide R80.20 | 196

 Monitoring and Troubleshooting Clusters

Field Description
Total generated Shows how many Delta Sync updates were generated. This counts the Delta
sync messages Sync updates, Retransmission Requests, Retransmission Acknowledgments,
and so on).

Sent Shows how many times this cluster member asked its peer cluster members
retransmission to retransmit specific Delta Sync update(s).
requests
Retransmission requests are sent when certain Delta Sync updates (with a
specified sequence number) are missing, while the sending cluster member
already received Delta Sync updates with advanced sequences.
Note - Compare the number of Sent retransmission requests to the Total
generated sync messages of the other cluster members.
A large counter's value can imply connectivity problems. If the counter's value
is unreasonably high (more than 30% of the Total generated sync messages
of other cluster members), contact Check Point Support
https://www.checkpoint.com/support-services/contact-support/ equipped
with the entire output and a detailed description of the network topology and
configuration.

Sent Shows how many times this cluster member retransmitted specific Delta
retransmission Sync update(s) at the requests from its peer cluster members.
updates

Peak fragments Shows the peak amount of fragments in the Fragments Queue on this cluster
per update member (usually, should be 1).

The "Received updates:" section

This section shows statistics for Delta Sync updates that were received by this cluster member
from its peer cluster members.

Field Description
Total received Shows the total number of Delta Sync updates this cluster member received
updates from its peer cluster members.
This counts only Delta Sync updates (not Retransmission Requests,
Retransmission Acknowledgments, and others).

Received Shows how many retransmission requests this cluster member received from
retransmission its peer cluster members.
requests
A large counter's value can imply connectivity problems. If the counter's value
is unreasonably high (more than 30% of the Total generated sync messages
on this cluster member), contact Check Point Support
https://www.checkpoint.com/support-services/contact-support/ equipped
with the entire output and a detailed description of the network topology and
configuration.

The "Queue sizes (num of updates):" section

This section shows the sizes of the Delta Sync queues.

ClusterXL Administration Guide R80.20 | 197

 Monitoring and Troubleshooting Clusters

Field Description
Sending queue Shows the size of the cyclic queue, which buffers all the Delta Sync updates
size that were already sent until it receives an acknowledgment from the peer
cluster members.
This queue is needed for retransmitting the requested Delta Sync updates.
Each cluster member has one Sending Queue.
Default: 512 Delta Sync updates, which is also the minimal value.

Receiving queue Shows the size of the cyclic queue, which buffers the received Delta Sync
size updates in two cases:
• When Delta Sync updates are missing, this queue is used to hold the
remaining received Delta Sync updates until the lost Delta Sync updates
are retransmitted (cluster members must keep the order, in which they
save the Delta Sync updates in the kernel tables).
• This queue is used to re-assemble a fragmented Delta Sync update.
Each cluster member has one Receiving Queue.
Default: 256 Delta Sync updates, which is also the minimal value.

Fragments Shows the size of the queue, which is used to prepare a Delta Sync update
queue size before moving it to the Sending Queue.
Notes:
• This queue must be smaller than the Sending Queue.
• This queue must be significantly smaller than the Receiving Queue.
Default: 50 Delta Sync updates, which is also the minimal value.

The "Timers:" section

This section shows the Delta Sync timers.

Field Description
Delta Sync Shows the interval at which this cluster member sends the Delta Sync
interval (ms) updates from its Sending Queue.
The base time unit is 100ms (or 1 tick).
Default: 100 ms, which is also the minimum value.
See Increasing the Sync Timer (on page 199).

The "Reset on XXX (triggered XXX)" section

Shows the date and the time of last statistics reset.
In parentheses, it shows how the last statistics was triggered - manually, or by fullsync.

ClusterXL Administration Guide R80.20 | 198

 Monitoring and Troubleshooting Clusters

Synchronization Troubleshooting Options

These are the available troubleshooting options. Each option involves editing a global system
configurable parameter to reconfigure the system with different value than the default.

Increasing the Size of the Sending Queue

The Sending Queue on the cluster member stores locally generated Delta Sync updates. Updates
in the Sending Queue are replaced by more recent updates. In a highly loaded cluster, updates are
therefore kept for less time. If a cluster member is asked to retransmit an update, it can only do
so if the update is still in its Sending Queue. The default (and minimum) size of this queue is 512.
Each cluster member has one sending queue.

To increase the size of the Sending Queue:

1. Change the value of the global kernel parameter fw_sync_sending_queue_size. See
Advanced Cluster Configuration (on page 127). You can change the value of this kernel
parameter only permanently. Change takes effect only after reboot.
2. You must also make sure that the required queue size survives boot. See How to Configure a
Kernel Parameter to Survive Reboot (see "How to Configure Reboot Survival" on page 127).
Enlarging this queue allows the cluster member to save more updates from other cluster
members. However, be aware that each saved Delta Sync update consumes memory. When
changing value of this kernel parameter, you should consider carefully the memory implications.

Increasing the Size of the Receiving Queue

The Receiving Queue on the cluster member keeps the updates from each cluster member until it
has received a complete sequence of updates. The default (and minimum) size of this queue is
256. Each cluster member keeps a Receiving Queue for each of the peer cluster members.

To increase the size of the Receiving Queue:

1. Change the value of the global kernel parameter fw_sync_recv_queue_size. See Advanced
Cluster Configuration (on page 127). You can change the value of this kernel parameter only
permanently. Change takes effect only after reboot.
2. You must also make sure that the required queue size survives boot. See How to Configure a
Kernel Parameter to Survive Reboot (see "How to Configure Reboot Survival" on page 127).
Enlarging this queue means that the cluster member can save more updates from other cluster
members. However, be aware that each saved Delta Sync update consumes memory. When
changing the value of this kernel parameter, you should carefully consider the memory
implications.

Increasing the Sync Timer

The Sync Timer performs Delta Sync related actions every fixed interval. By default, the Sync
Timer interval is 100ms. The base time unit is 100ms (or 1 tick), which is therefore the minimal
value.

To increase the Sync Timer:

Change the value of the global kernel parameter fwha_timer_sync_res. See Advanced Cluster
Configuration (on page 127). You can change the value of this kernel parameter on-the-fly - while
the system is working. A reboot is not needed.

ClusterXL Administration Guide R80.20 | 199

 Monitoring and Troubleshooting Clusters

To make sure that the new Sync Timer interval value survives boot, see How to Configure a Kernel
Parameter to Survive Reboot (see "How to Configure Reboot Survival" on page 127).
By default, fwha_timer_sync_res has a value of 1, meaning that the Sync Timer operates every
base time unit (every 100ms). If you configure the value of this kernel parameter to N, the Sync
Timer operates every N*100ms.

Increasing the CPHA Timer

The CPHA Timer performs cluster related actions every fixed interval. By default, the CPHA Timer
interval is 100ms. The base time unit is 100ms (or 1 tick), which is also the minimum value.
If the cluster members are geographically separated from each other, set the CPHA Timer to be
around 10 times the round-trip delay of the sync network.
Enlarging the CPHA Timer interface increases the time it takes to detect a failover. For example, if
detecting interface failure takes 0.3 seconds, and you double the CPHA Timer interval to 200ms,
then the time needed to detect an interface failure is doubled to 0.6 seconds.

To increase the CPHA Timer:

Change the value of the kernel parameter fwha_timer_cpha_res. See Advanced Cluster
Configuration (on page 127). You can change the value of this kernel parameter on-the-fly - while
the system is working. A reboot is not needed.
To make sure that the new CPHA Timer interval value survives boot, see How to Configure a
Kernel Parameter to Survive Reboot (see "How to Configure Reboot Survival" on page 127).
By default, fwha_timer_cpha_res has a value of 1, meaning that the CPHA Timer operates every
base time unit (every 100ms). If you configure the value of this kernel parameter to N, the CPHA
Timer operates every N*100ms.

Reconfiguring the Acknowledgment Timeout

A cluster member deletes updates from its Sending Queue on a regular basis. This frees up space
in the queue for more recent updates.
The cluster member deletes updates from this queue if it receives an ACK about the update from
the peer member.
The peer cluster member sends an ACK in one of two circumstances - on condition that the Block
New Connections mechanism (described in Blocking New Connections Under Load (on page 128))
is active:
• After receiving a certain number of updates.
• If it did not send an ACK for a certain time. This is important if the sync network has a
considerable line delay, which can occur if the cluster members are geographically separated
from each other.
To reconfigure the timeout, after which the member sends an ACK:
Change the value of the global kernel parameter fw_sync_ack_time_gap. See Advanced
Cluster Configuration (on page 127). You can change the value of this kernel parameter
on-the-fly - while the system is working. A reboot is not needed.
To make sure that the new configured value survives boot, see How to Configure a Kernel
Parameter to Survive Reboot (see "How to Configure Reboot Survival" on page 127).
The default value for this variable is 10 ticks (10 * 100ms). Thus, if a member did not send an ACK
for a whole second, it will send an ACK for the updates it received.

ClusterXL Administration Guide R80.20 | 200

 Monitoring and Troubleshooting Clusters

ClusterXL Administration Guide R80.20 | 201

 Monitoring and Troubleshooting Clusters

Viewing IGMP Status

Description
This command lets you view the IGMP membership status.

Syntax
Shell Command
Gaia Clish show cluster members igmp

Expert mode cphaprob igmp

Example
Member2> show cluster members igmp
IGMP Membership: Enabled
Supported Version: 2
Report Interval [sec]: 60

IGMP queries are replied only by Operating System

Interface Host Group Multicast Address Last ver. Last Query[sec]

------------------------------------------------------------------------------
eth0 224.168.204.33 01:00:5e:28:cc:21 N/A N/A
eth1 224.10.10.250 01:00:5e:0a:0a:fa N/A N/A
eth2 224.20.20.33 01:00:5e:14:14:21 N/A N/A
Member2>

ClusterXL Administration Guide R80.20 | 202

 Monitoring and Troubleshooting Clusters

Viewing Cluster Delta Sync Statistics for Connections Table

Description
This command lets you see Delta Sync statistics about the operations performed in the
Connections Kernel Table (id 8158).
The output shows operations such as creating a new connection (SET), updating a connection
(REFRESH), deleting a connection (DELETE), and so on.

Syntax
Shell Command
Gaia Clish show cluster statistics transport [reset]

Expert mode cphaprob [-reset] ldstat

The reset flag resets the kernel statistics, which were collected since the last reboot or reset.

Example
Member2> show cluster statistics transport
Operand Calls Bytes Average Ratio %
----------------------------------------------------------
ERROR 0 0 0 0
SET 2035 106444 52 99
RENAME 0 0 0 0
REFRESH 0 0 0 0
DELETE 0 0 0 0
SLINK 1 64 64 0
UNLINK 0 0 0 0
MODIFYFIELDS 0 0 0 0
RECORD DATA CONN 0 0 0 0
COMPLETE DATA CONN 0 0 0 0

Total bytes sent: 114652 (0 MB) in 429 packets. Average 267

Member2>

ClusterXL Administration Guide R80.20 | 203

 Monitoring and Troubleshooting Clusters

Viewing Cluster IP Addresses

Description
This command lets you see the IP addresses and interfaces of the cluster members.

Syntax
Shell Command
Gaia Clish show cluster members ips

Expert mode cphaprob tablestat

Example
Member1> show cluster members ips

---- Unique IP's Table ----

Member Interface IP-Address

------------------------------------------

(Local)
0 1 172.23.88.176
0 2 1.0.0.176
0 3 2.0.0.176
0 4 3.0.0.176

1 2 1.0.0.177
1 3 2.0.0.177
1 4 3.0.0.177

------------------------------------------
Member1>

Member2> show cluster members ips

---- Unique IP's Table ----

Member Interface IP-Address

------------------------------------------
0 2 1.0.0.176
0 3 2.0.0.176
0 4 3.0.0.176

(Local)
1 1 172.23.88.177
1 2 1.0.0.177
1 3 2.0.0.177
1 4 3.0.0.177

------------------------------------------
Member2>

ClusterXL Administration Guide R80.20 | 204

 Monitoring and Troubleshooting Clusters

Viewing the Cluster Member ID Mode in Local Logs

Description
This command lets you see how the local ClusterXL logs show the cluster member - by its
Member ID (default), or its Member Name.
Note - See Configuring the Cluster Member ID Mode in Local Logs (on page 157).

Syntax
Shell Command
Gaia Clish show cluster members idmode

Expert mode cphaprob names

Example
[Expert@Member2:0]# cphaprob names

Current member print mode in local logs is set to: ID

[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 205

 Monitoring and Troubleshooting Clusters

Viewing Interfaces Monitored by RouteD

Description
This command lets you see the interfaces, which the RouteD daemon monitors on the cluster
member when you configure OSPF.
The idea is that if you configure OSPF; cluster member monitors these interfaces and does not
bring up the cluster member unless RouteD daemon says it is OK to bring up the cluster member.
This is used mainly in ClusterXL High Availability Primary Up configuration to avoid premature
failbacks.

Syntax
Shell Command
Gaia Clish show ospf interfaces [detailed]

Expert mode cphaprob routedifcs

Example 1
[Expert@Member2:0]# cphaprob routedifcs

No interfaces are registered.

[Expert@Member2:0]#

Example 2
[Expert@Member2:0]# cphaprob routedifcs

Monitored interfaces registered by routed:

eth0
[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 206

 Monitoring and Troubleshooting Clusters

Viewing Roles of RouteD Daemon on Cluster Members

Description
This command lets you view on which cluster member the RouteD daemon runs as a Master.
Notes:
• In ClusterXL High Availability, the RouteD daemon must run as a Master only on the Active
cluster member.
• In ClusterXL Load Sharing, the RouteD daemon must run as a Master only on one of the Active
cluster members and as a Non-Master on all other cluster members.
• In VRRP cluster, the RouteD daemon must run as a Master only on the VRRP Master cluster
member.

Syntax
Shell Command
Gaia Clish show cluster role

Expert mode cphaprob roles

Example
[Expert@Member2:0]# cphaprob roles

ID Role

1 Non-Master
2 (local) Master

[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 207

 Monitoring and Troubleshooting Clusters

Viewing Cluster Correction Statistics

Description
This command lets you view the Cluster Correction Statistics on each cluster member.
From R80.20, ClusterXL adds a new mechanism to deal with asymmetric connections - Cluster
Correction Layer (CCL).
The CCL provides connections stickiness by "correcting" the packets to the correct cluster
member:
• In most cases, the CCL makes the correction from the CoreXL SND.
• In some cases (like Dynamic Routing, or VPN), the CCL makes the correction from the Firewall
or SecureXL.
In some cases, ClusterXL needs to send some data along with the corrected packet (currently,
only in VPN). For such packets, the output shows "with metadata".
Note - For more information about CoreXL, see the R80.20 Performance Tuning Admin Guide
https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_Performanc
eTuning_AdminGuide/html_frameset.htm.

Syntax
Shell Command
Gaia Clish N/A
cphaprob corr
Expert mode cphaprob -c {a | d |f}

Where:

Command Description
cphaprob corr Shows Cluster Correction Statistics for all traffic.

cphaprob -c a Shows Cluster Correction Statistics for all traffic.

cphaprob -c d Shows Cluster Correction Statistics for CoreXL SND corrections only.

cphaprob -c f Shows Cluster Correction Statistics for CoreXL Firewall instances and
SND.

Example
[Expert@Member2:0]# cphaprob corr

Cluster Correction Stats (All traffic):

------------------------------------------------------
Sent packets: 0 (0 with metadata)
Sent bytes: 0
Received packets: 0 (0 with metadata)
Received bytes: 0
Send errors: 0
Receive errors: 0
Local asymmetric conns: 0
[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 208

 Monitoring and Troubleshooting Clusters

[Expert@Member2:0]# cphaprob -c a

Cluster Correction Stats (All traffic):

------------------------------------------------------
Sent packets: 0 (0 with metadata)
Sent bytes: 0
Received packets: 0 (0 with metadata)
Received bytes: 0
Send errors: 0
Receive errors: 0
Local asymmetric conns: 0
[Expert@Member2:0]#

[Expert@Member2:0]# cphaprob -c d

Cluster Correction Stats (SND corrections only):

------------------------------------------------------
Sent packets: 0 (0 with metadata)
Sent bytes: 0
Received packets: 0 (0 with metadata)
Received bytes: 0
Send errors: 0
Receive errors: 0
[Expert@Member2:0]#

[Expert@Member2:0]# cphaprob -c f

Cluster Correction Stats (Firewall instances and SND):

------------------------------------------------------
Sent packets: 0 (0 with metadata)
Sent bytes: 0
Received packets: 0 (0 with metadata)
Received bytes: 0
Send errors: 0
Receive errors: 0
Local asymmetric conns: 0
[Expert@Member2:0]#

ClusterXL Administration Guide R80.20 | 209

 Monitoring and Troubleshooting Clusters

Monitoring Cluster Status Using SmartConsole Clients

SmartConsole Logs View
Every change in status of a cluster member is recorded in the Logs tab of the Logs & Monitor of
SmartConsole. This is based on what you select for the Fail-Over Tracking option of the cluster
object ClusterXL page.

ClusterXL Log Messages

The following conventions are used in this section:
1. Square brackets are used to indicate place holders, which are substituted by relevant data
when an actual log message is issued (for example, [NUMBER] will be replaced by a numeric
value).
2. Angle brackets are used to indicate alternatives, one of which will be used in actual log
messages. The different alternatives are separated with a vertical line (for example,
<up|down> indicates that either "up" or "down" will be used).
3. These place holders are frequently used:
• ID: A unique cluster member identifier, starting from "1". This corresponds to the order, in
which members are sorted in the cluster object.
• IP: Any unique IP address that belongs to the member.
• MODE: The cluster mode (for example, New HA, LS Multicast, and so on).
• STATE: The state of the member (for example, active, down, standby).
• DEVICE: The name of a pnote device (for example, fwd, Interface Active Check).

General logs
Starting <ClusterXL|State Synchronization>.
Indicates that ClusterXL (or State Synchronization, for 3rd party clusters) was successfully started
on the reporting member. This message is usually issued after a member boots, or after an
explicit call to cphastart.
Stopping <ClusterXL|State Synchronization>.
Informs that ClusterXL (or State Synchronization) was deactivated on this member. The member
will no longer be a part of the cluster (even if configured to be so), until ClusterXL is restarted.
Unconfigured cluster Computers changed their MAC Addresses. Please reboot
the cluster so that the changes take affect.
This message is usually issued when a member is shut down, or after an explicit call to
cphastop.

State logs
Mode inconsistency detected: member [ID] ([IP]) will change its mode to
[MODE]. Please re-install the security policy on the cluster.
This message should rarely happen. It indicates that another cluster member has reported a
different cluster mode than is known to the local member. This is usually the result of a failure to
install the Access Control Policy on all cluster members. To correct this problem, install the
Access Control Policy again.

ClusterXL Administration Guide R80.20 | 210

 Monitoring and Troubleshooting Clusters

Note - The cluster will continue to operate after a mode inconsistency has been detected, by
altering the mode of the reporting member to match the other cluster members. However, it is
highly recommended that the policy will be re-installed as soon as possible.
State change of member [ID] ([IP]) from [STATE] to [STATE] was cancelled,
since all other members are down. Member remains [STATE].
When a member needs to change its state (for example, when an active member encounters a
problem and needs to bring itself down), it first queries the other members for their state. If all
other members are down, this member cannot change its state to a non-active one (or else all
members will be down, and the cluster will not function). Thus, the reporting cluster member
continues to function, despite its problem (and will usually report its state as "Active Attention").
member [ID] ([IP]) <is active|is down|is stand-by|is initializing>
([REASON]).
This message is issued whenever a cluster member changes its state. The log text specifies the
new state of the member.

Pnote logs
Pnote log messages are issued when a pnote device changes its state.
• [DEVICE] on member [ID] ([IP]) status OK ([REASON]).
The pnote device is working normally.
• [DEVICE] on member [ID] ([IP]) detected a problem ([REASON]).
Either an error was detected by the pnote device, or the device has not reported its state for a
number of seconds (as set by the "timeout" option of the pnote)
• [DEVICE] on member [ID] ([IP]) is initializing ([REASON]).
Indicates that the device has registered itself with the pnote mechanism, but has not yet
determined its state.
• [DEVICE] on member [ID] ([IP]) is in an unknown state ([STATE ID]) ([REASON]).
This message should not normally appear. Contact Check Point Support.

Interface logs
• interface [INTERFACE NAME] of member [ID] ([IP]) is up.
Indicates that this interface is working normally, meaning that it is able to receive and transmit
packets on the expected subnet.
• interface [INTERFACE NAME] of member [ID] ([IP]) is down (receive <up|down>, transmit
<up|down>).
This message is issued whenever an interface encounters a problem, either in receiving or
transmitting packets. Note that in this case the interface may still be working properly, as far
as the OS is concerned, but is unable to communicate with other cluster members due to a
faulty cluster configuration.
• interface [INTERFACE NAME] of member [ID] ([IP]) was added.
Notifies users that a new interface was registered with the Security Gateway (meaning that
packets arriving on this interface are filtered by the firewall). Usually this message is the
result of activating an interface (such as issuing an ifconfig up command on Unix systems).
The interface is now included in the ClusterXL reports (in the output of the Gaia Clish
command show cluster interfaces all, or Expert mode command cphaprob -am if).
Note that the interface may still be reported as "Disconnected", in case it was configured as
such for ClusterXL.

ClusterXL Administration Guide R80.20 | 211

 Monitoring and Troubleshooting Clusters

• interface [INTERFACE NAME] of member [ID] ([IP}) was removed.

Indicates that an interface was detached from the Security Gateway, and is therefore no longer
monitored by ClusterXL.

SecureXL logs
• SecureXL device was deactivated since it does not support CPLS.
This message is the result of an attempt to configure a ClusterXL in Load Sharing Multicast
mode over Security Gateways using an acceleration device that does not support Load Sharing.
As a result, acceleration will be turned off, but the cluster will work in Check Point Load
Sharing mode (CPLS).

Reason Strings
• member [ID] ([IP]) reports more interfaces up.
This text can be included in a pnote log message describing the reasons for a problem report:
Another member has more interfaces reported to be working, than the local member does.
This means that the local member has a faulty interface, and that its counterpart can do a
better job as a cluster member. The local member will therefore go down, leaving the member
specified in the message to handle traffic.
• member [ID] ([IP]) has more interfaces - check your disconnected interfaces configuration
in the <discntd.if file|registry>.
This message is issued when members in the same cluster have a different number of
interfaces. A member having less interfaces than the maximal number in the cluster (the
reporting member) may not be working properly, as it is missing an interface required to
operate against a cluster IP address, or a synchronization network. If some of the interfaces on
the other cluster member are redundant, and should not be monitored by ClusterXL, they
should be explicitly designated as "Disconnected". This is done using the file
$FWDIR/conf/discntd.if file.
• [NUMBER] interfaces required, only [NUMBER] up.
ClusterXL has detected a problem with one or more of the monitored interfaces. This does not
necessarily mean that the member will go down, as the other members may have less
operational interfaces. In such a condition, the member with the highest number of operational
interfaces will remain up, while the others will go down.

ClusterXL Administration Guide R80.20 | 212

 Monitoring and Troubleshooting Clusters

Working with SNMP Traps

You can configure and see SNMP traps for ClusterXL High Availability.

To configure an SNMP trap:

1. Connect to the Security Management Server with an SSH connection or a serial console.
2. Go to the Expert mode.
3. Run: threshold_config
4. Select (9) Configure Thresholds from the Threshold Engine Configuration Options menu.
5. Select (2) High Availability from the threshold Engine Configuration Options menu.
6. Select the desired trap(s) from the High Availability Thresholds menu.
7. Select and configure these actions for the specified trap:
• Enable/Disable Threshold
• Set Severity
• Set Repetitions
• Configure Alert Destinations
8. From the Threshold Engine Configuration Options menu, select (7) Configure alert
destinations.
9. Configure your alert destinations.
10. From the Threshold Engine Configuration Options menu, select (3) Save policy.
You can optionally save the policy to a file.
11. In SmartConsole, install the Access Control Policy on this cluster object.

To see all defined SNMP traps, run threshold_config and select (8) View thresholds overview
from the Threshold Engine Configuration Options menu.
You can download the most recent Check Point MIBs file
http://supportcontent.checkpoint.com/solutions?id=sk90470 from the Support Center.

ClusterXL Administration Guide R80.20 | 213

 Monitoring and Troubleshooting Clusters

cphastart and cphastop

Running the cphastart command on a cluster member activates ClusterXL on the member. It also
initiates the Full Synchronization with peer cluster member(s). The cpstart command is the
recommended way to start a cluster member.
Running the cphastop command on a cluster member stops the cluster member from passing
traffic. State Synchronization also stops. It is still possible to open connections directly to this
cluster member.
These commands should only be run by the Security Gateway, and not directly by the user.

ClusterXL Administration Guide R80.20 | 214

 Monitoring and Troubleshooting Clusters

How to Initiate Cluster Failover

For more information on initiating manual cluster failovers, see sk55081
http://supportcontent.checkpoint.com/solutions?id=sk55081.

Method 1 (recommended)
Change in the Command in Gaia Clish Command in Expert Notes
state of the mode
cluster member
Change the state of the set cluster member clusterXL_admin • Does not disable
cluster member to admin down down Delta Sync.
DOWN
• See Appendix A -
The
clusterXL_admin
Script (on page
220).

Change the state of the set cluster member clusterXL_admin • Does not initiate
cluster member to UP admin up up Full Sync.
• See Appendix A -
The
clusterXL_admin
Script (on page
220).

Method 2 (not recommended)

Change in the Command in Expert mode Notes
state of the
cluster member
Change the state of the 1. cphaconf set_pnote -d <Name of Does not disable Delta
cluster member to Critical Device> -t 0 -s ok register Sync.
DOWN 2. cphaconf set_pnote -d <Name of
Critical Device> -s problem report

Change the state of the 1. cphaconf set_pnote -d <Name of Does not initiate Full
cluster member to UP Critical Device> -s ok report Sync.
2. cphaconf set_pnote -d <Name of
Critical Device> unregister

ClusterXL Administration Guide R80.20 | 215

 Monitoring and Troubleshooting Clusters

Notes:
• In Load Sharing mode, the cluster distributes the traffic load between the remaining Active
members.
• In High Availability mode, the cluster fails over to a Standby cluster member with the highest
priority.

ClusterXL Administration Guide R80.20 | 216

 Monitoring and Troubleshooting Clusters

Troubleshooting Dynamic Routing (routed) Pnote

In R76, Check Point added a new ClusterXL Pnote called routed that monitors the state of the
Dynamic Routing for Gaia clusters. This Pnote makes sure that traffic is not assigned to a cluster
member before it is ready to handle the traffic. The Gaia RouteD daemon handles all routing
(static and dynamic) operations.
There can be an issue with Dynamic Routing that shows one or more of these symptoms:
• Cluster IP address connectivity problems
• Unexpected failovers
• The Gaia Clish command show cluster pnotes (or Expert mode command cphaprob
list) shows that the current state of the Critical Device routed is problem:
Device Name: routed
Registration number: 4
Timeout: none
Current state: problem
These are some of the common causes of this issue:
• Cluster misconfiguration
• Traffic on TCP port 2010 between cluster members is blocked by the Firewall
• The RouteD daemon did not get all of its routes
• The RouteD daemon did not start correctly

Standard RouteD Pnote Behavior

Typically, the routed Pnote reports its current state as Problem when:
• A cluster member fails over
• A cluster member reboots
• There is an inconsistency in the Dynamic Routing configuration on cluster members
The routed Pnote reports its state as Ok when:
• A ClusterXL member tells the RouteD daemon that it is a Master
• The RouteD daemon gets the entire routing state from the Master

Basic Troubleshooting Steps

1. To make sure that your cluster and member interfaces are configured correctly, run:
• In Gaia Clish:
show cluster interfaces {all | secured | virtual | vlans}
Or
• In Expert mode:
cphaprob [-a] [-m] if
2. Generate RouteD cluster messages. Run in Expert mode:
dbset routed:instance:default:traceoptions:traceoptions:Cluster
Examine the /var/log/routed/log file.
ClusterXL Administration Guide R80.20 | 217
 Monitoring and Troubleshooting Clusters

3. Make sure that Firewall rules do not block traffic on TCP port 2010 between the cluster
members.
4. Make sure that the RouteD daemon is running on the Active member.
5. Look for a router-id mismatch in the OSPF configuration.
6. Make sure that the OSPF interface is up on the Standby member.
For advanced troubleshooting procedures and more information, see sk92787
http://supportcontent.checkpoint.com/solutions?id=sk92787.
For troubleshooting OSPF and the RouteD daemon, see sk84520
http://supportcontent.checkpoint.com/solutions?id=sk84520.

ClusterXL Administration Guide R80.20 | 218

 Monitoring and Troubleshooting Clusters

ClusterXL Error Messages

Each important cluster event that affects the Cluster Members has a unique code that appears in
the /var/log/messages file and dmesg.
Each cluster event message starts with the prefix CLUS-XXXXXX-Y.
Example: CLUS-214802-1

Part of the message Description

CLUS- Constant string.

XXXXXX A six-digit error code.

These error codes specify the events:
• Events related to Critical Devices
• Events related to the states of Cluster Members
• Events related to cluster synchronization
• Events related to policy installation
The first digit shows which Cluster Member generated the event:
• 1 - local member
• 2 - remote member
The second digit represents the cluster type event
The meaning of the other digits depends on the cluster event type.

Y Shows the ID or the NAME of the local Cluster Member that generated this
log message.
See Configuring the Cluster Member ID Mode in Local Logs (on page 157).

For more information, see sk125152

http://supportcontent.checkpoint.com/solutions?id=sk125152.

ClusterXL Administration Guide R80.20 | 219

APPENDIX A

Appendix A - The clusterXL_admin

Script
You can use the clusterXL_admin script to initiate a manual fail-over from a cluster member.
Location of this script on your cluster members is: $FWDIR/bin/clusterXL_admin
This shell script does one of these:
• Registers a Critical Device called admin_down and reports the state of that Critical Device as
problem. This gracefully changes the state of the cluster member to Down.
• Reports the state of the registered Critical Device admin_down as ok. This gracefully changes
the state of the cluster member to Up. Then, the script unregisters the Critical Device
admin_down.
For more information, see sk55081 http://supportcontent.checkpoint.com/solutions?id=sk55081.
Example:
#! /bin/csh -f
#
# The script will cause the machine to get into down state, thus the member will not filter packets.
# It will supply a simple way to initiate a failover by registering a new device in problem state when
# a failover is required and will unregister the device when wanting to return to normal operation.
# USAGE:
# clusterXL_admin <up|down>

set PERSISTENT = ""

# checking number of arguments

if ( $#argv > 2 || $#argv < 1 ) then
echo "clusterXL_admin : Invalid Argument Count"
echo "Usage: clusterXL_admin <up|down> [-p]"
exit 1
else if ( $#argv == 2 ) then
if ( "$2" != "-p" ) then
echo "clusterXL_admin : Invalid Argument ($2)"
echo "Usage: clusterXL_admin <up|down> [-p]"
exit 1
endif
set PERSISTENT = "-p"
endif

#checking if cpha is started

$FWDIR/bin/cphaprob stat | grep "Cluster" > /dev/null
if ($status) then
echo "HA is not started"
exit 1
endif

if ( $1 == "up" ) then
echo "Setting member to normal operation ..."
$FWDIR/bin/cphaconf set_pnote -d admin_down $PERSISTENT unregister > & /dev/null
if ( `uname` == 'IPSO' ) then
sleep 5
else
sleep 1
endif

set stateArr = `$FWDIR/bin/cphaprob stat | grep "local"`

$FWDIR/bin/cphaprob stat | egrep "Sync only|Bridge Mode" > /dev/null

#If it's third party or bridge mode, use column 4 , otherwise 5
if ($status) then
set state = $stateArr[5]
else
set state = $stateArr[4]
endif

ClusterXL Administration Guide R80.20 | 220

 Appendix A - The clusterXL_admin Script

echo "Member current state is $state"

if (($state != "Active" && $state != "Standby") && ($state != "ACTIVE" && $state != "STANDBY"
&& $state != "ACTIVE(!)")) then
echo "Operation failed: member is still down, run 'cphaprob list' for further details"
endif
exit 0
endif

if ( $1 == "down" ) then
echo "Setting member to administratively down state ..."
$FWDIR/bin/cphaconf set_pnote -d admin_down -t 0 -s problem $PERSISTENT register > & /dev/null
sleep 1

set stateArr = `$FWDIR/bin/cphaprob stat | grep "local"`

$FWDIR/bin/cphaprob stat | egrep "Sync only|Bridge Mode" > /dev/null

#If it's third party or bridge mode, use column 4 , otherwise 5
if ($status) then
set state = $stateArr[5]
else
set state = $stateArr[4]
endif

echo "Member current state is $state"

if ( $state == "Active attention" || $state == "ACTIVE(!)" ) then
echo "All the members within the cluster have problem/s and the local member was chosen
to become active"
else
if ( $state != "Down" && $state != "DOWN" ) then
echo "Operation failed: member is not down, run 'cphaprob list' for further
details"
endif
endif
exit 0
else
echo "clusterXL_admin : Invalid Option ($1)"
echo "Usage: clusterXL_admin <up|down> [-p]"
exit 1
endif

ClusterXL Administration Guide R80.20 | 221

APPENDIX B

Appendix B - The
clusterXL_monitor_ips Script
You can use the clusterXL_monitor_ips script to ping a list of predefined IP addresses and
change the state of the cluster member to Down or Up based on the replies to these pings. For this
script to work, you must write the IP addresses in the $FWDIR/conf/cpha_hosts file - each IP
address on a separate line. This file does not support comments or spaces.
Location of this script on your cluster members is: $FWDIR/bin/clusterXL_monitor_ips
This shell script does these:
1. Registers a Critical Device called host_monitor with status ok.
2. Starts to send pings to the list of predefined IP addresses in the $FWDIR/conf/cpha_hosts
file.
3. While the script receives responses to its pings, it does not change the status of that Critical
Device.
4. If the script does not receive a response to even one ping, it reports the state of that Critical
Device as problem. This gracefully changes the state of the cluster member to Down. If the
script receives responses to its pings again, it changes the status of that Critical Device to ok
again.
For more information, see sk35780 http://supportcontent.checkpoint.com/solutions?id=sk35780.
Important - You must do these changes on all cluster members.
Example:
#!/bin/sh
#
# The script tries to ping the hosts written in the file $FWDIR/conf/cpha_hosts. The names (must be
resolveable) ot the IPs of the hosrs must be written in seperate lines.
# the file must not contain anything else.
# We ping the given hosts every number of seconds given as parameter to the script.
# USAGE:
# cpha_monitor_ips X silent
# where X is the number of seconds between loops over the IPs.
# if silent is set to 1, no messages will appear on the console
#
# We initially register a pnote named "host_monitor" in the problem notification mechanism
# when we detect that a host is not responding we report the pnote to be in "problem" state.
# when ping succeeds again - we report the pnote is OK.

silent=0

if [ -n "$2" ]; then
if [ $2 -le 1 ]; then
silent=$2
fi
fi
hostfile=$FWDIR/conf/cpha_hosts
arch=`uname -s`
if [ $arch = "Linux" ]
then
#system is linux
ping="ping -c 1 -w 1"
else
ping="ping"
fi
$FWDIR/bin/cphaconf set_pnote -d host_monitor -t 0 -s ok register
TRUE=1
while [ "$TRUE" ]
do
result=1

ClusterXL Administration Guide R80.20 | 222

 Appendix B - The clusterXL_monitor_ips Script

for hosts in `cat $hostfile`

do
if [ $silent = 0 ]
then
echo "pinging $hosts using command $ping $hosts"
fi
if [ $arch = "Linux" ]
then
$ping $hosts > /dev/null 2>&1
else
$ping $hosts $1 > /dev/null 2>&1
fi
status=$?
if [ $status = 0 ]
then
if [ $silent = 0 ]
then
echo " $hosts is alive"
fi
else
if [ $silent = 0 ]
then
echo " $hosts is not responding "
fi
result=0
fi
done
if [ $silent = 0 ]
then
echo "done pinging"
fi
if [ $result = 0 ]
then
if [ $silent = 0 ]
then
echo " Cluster member should be down!"
fi
$FWDIR/bin/cphaconf set_pnote -d host_monitor -s problem report
else
if [ $silent = 0 ]
then
echo " Cluster member seems fine!"
fi
$FWDIR/bin/cphaconf set_pnote -d host_monitor -s ok report
fi
if [ "$silent" = 0 ]
then
echo "sleeping"
fi
sleep $1
echo "sleep $1"
done

ClusterXL Administration Guide R80.20 | 223

APPENDIX C

Appendix C - The
clusterXL_monitor_process Script
You can use the clusterXL_monitor_process script to monitor if the specified user space
processes run, and cause cluster fail-over if these processes do not run. For this script to work,
you must write the correct case-sensitive names of the monitored processes in the
$FWDIR/conf/cpha_proc_list file - each process name on a separate line. This file does not
support comments or spaces.
Location of this script on your cluster members is:
$FWDIR/bin/clusterXL_monitor_process
This shell script does these:
1. Registers Critical Devices (with status ok) called as the names of the processes you specified
in the $FWDIR/conf/cpha_proc_list file.
2. While the script detects that the specified process runs, it does not change the status of the
corresponding Critical Device.
3. If the script detects that the specified process do not run anymore, it reports the state of the
corresponding Critical Device as problem. This gracefully changes the state of the cluster
member to Down. If the script detects that the specified process runs again, it changes the
status of the corresponding Critical Device to ok again.
For more information, see sk92904 http://supportcontent.checkpoint.com/solutions?id=sk92904.
Important - You must do these changes on all cluster members.
Example:
#!/bin/sh
#
# This script monitors the existance of processes in the system. The process names should be written
# in the $FWDIR/conf/cpha_proc_list file one every line.
#
# USAGE :
# cpha_monitor_process X silent
# where X is the number of seconds between process probings.
# if silent is set to 1, no messages will appear on the console.
#
#
# We initially register a pnote for each of the monitored processes
# (process name must be up to 15 charachters) in the problem notification mechanism.
# when we detect that a process is missing we report the pnote to be in "problem" state.
# when the process is up again - we report the pnote is OK.

if [ "$2" -le 1 ]
then
silent=$2
else
silent=0
fi
if [ -f $FWDIR/conf/cpha_proc_list ]
then
procfile=$FWDIR/conf/cpha_proc_list
else
echo "No process file in $FWDIR/conf/cpha_proc_list "
exit 0
fi

arch=`uname -s`

for process in `cat $procfile`

do
$FWDIR/bin/cphaconf set_pnote -d $process -t 0 -s ok -p register > /dev/null 2>&1

ClusterXL Administration Guide R80.20 | 224

 Appendix C - The clusterXL_monitor_process Script

done

while [ 1 ]
do

result=1

for process in `cat $procfile`

do
ps -ef | grep $process | grep -v grep > /dev/null 2>&1

status=$?

if [ $status = 0 ]
then
if [ $silent = 0 ]
then
echo " $process is alive"
fi
# echo "3, $FWDIR/bin/cphaconf set_pnote -d $process -s ok report"
$FWDIR/bin/cphaconf set_pnote -d $process -s ok report
else
if [ $silent = 0 ]
then
echo " $process is down"
fi

$FWDIR/bin/cphaconf set_pnote -d $process -s problem report

result=0
fi

done

if [ $result = 0 ]

then
if [ $silent = 0 ]
then
echo " One of the monitored processes is down!"
fi
else
if [ $silent = 0 ]
then
echo " All monitored processes are up "
fi

fi
if [ "$silent" = 0 ]
then
echo "sleeping"
fi

sleep $1

done

ClusterXL Administration Guide R80.20 | 225