ADVANCED TROUBLESHOOTING

& DEBUGGING

Name | Diamond Services Engineer

©2020 Check Point Software Technologies Ltd. 1

Agenda
• Basic Concepts

• Check Point Infrastructure

• Check Point Tools

• Debug

©2020 Check Point Software Technologies Ltd. 2

Basic Concepts

©2020 Check Point Software Technologies Ltd. 3

Basic Concepts

• Kernel - A bridge between hardware and applications. Defines how

the OS and applications interact with the hardware and vice versa.
Allows multiple applications to effectively share the hardware by
controlling access to CPU, memory, disk I/O (Input / Output) and
networking.

©2020 Check Point Software Technologies Ltd. 4

Basic Concepts

• Kernel Space - A space in memory where the kernel executes (i.e.,

runs) and provides its services. Kernel’s memory map is fixed.
• User Space - A space in memory in which user processes run.
Kernel space can be accessed by user processes only through the
use of system calls.
• System calls - requests in a Unix-like operating system
by an active process for a service performed by
the kernel, such as input/output (I/O) or process creation
• I/O is any program, operation or device that transfers
data to or from a CPU and to or from a peripheral device
(such as disk drives, keyboards, mice and printers).
©2020 Check Point Software Technologies Ltd. 5
Basic Concepts

• Module - A piece of code which can be loaded into the kernel to

extend it’s functionality. To see which modules are enabled on the
environment:
# lsmod
• Get more information about a module:
# modinfo <module name>

©2020 Check Point Software Technologies Ltd. 6

Basic Concepts

• Driver - A module which allows the kernel to communicate and

control a certain hardware. Drivers for most common hardware
devices (such as memory, most keyboards and mice, most video
cards etc.) are built into the kernel (plug and play).
Other devices require a driver to be installed to function / add extra
functionality.
• Some drivers also provide a utility to control the driver / display
parameters of the driver.
For example, to display the parameters of a NIC (network interface
controllers):
# ethtool <interface>
©2020 Check Point Software Technologies Ltd. 7
[admin@shell] # lsmod
Module Size Used by
vrrp_lkmmod 265282 0
cppcap 13547 0
adpmod 14103880 0
8021q 33122 0
fw_2 44880582 22
fw_1 44880582 22
fw_0 44880582 38
ext3 217010 1
cdrom 42612 1 sr_mod
psmouse 121418 0
e1000 141192 0
libahci 31979 1
ahcicciss 122796 0
©2020 Check Point Software Technologies Ltd. 8
[admin@shell] # modinfo cdrom
filename: /lib/modules/3.10.0-
957.21.3cpx86_64/kernel/drivers/cdrom/cdrom.ko`
license: GPL
depends:
retpoline: Y
intree: Y
name: cdrom
vermagic: 5.4.0-kali4-amd64 SMP mod_unload modversions
parm: debug:bool
parm: autoclose:bool
parm: autoeject:bool
parm: lockdoor:bool
parm: check_media_type:bool
parm: mrw_format_restart:bool
©2020 Check Point Software Technologies Ltd. 9
[admin@shell] # ethtool eth1
Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised pause frame use: No
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
…

©2020 Check Point Software Technologies Ltd. 10

Check Point Infrastructure

©2020 Check Point Software Technologies Ltd. 11

Check Point Modules

• Check Point implements it’s own kernel modules in order to handle

the traffic.
• The modules are located at:
Firewall modules - $FWDIR/modules/
Performance Pack (SecureXL) modules -
/etc/ppk.boot/boot/modules/
(Locations may vary depending on the version of the OS).
• To see all active modules in the firewall kernel, type:
# fw ctl debug -m
• To see all active modules in the SecureXL kernel, type:
# fwaccel dbg
©2020 Check Point Software Technologies Ltd. 12
[admin@shell] # fw ctl debug -m | grep ‘Module’
Module: APPI
Module: CPAS
Module: MALWARE
Module: PSL
Module: RAD_KERNEL
Module: UP
Module: WS
Module: fw
Module: h323
Module: kiss
Module: kissflow
Module: synatk
…

©2020 Check Point Software Technologies Ltd. 13

[admin@shell] # fwaccel dbg | grep ‘Module’
Module: default (default)
Module: db
Module: api
Module: pkt
Module: infras
Module: tmp
Module: vpn
Module: nac
Module: cpaq
Module: synatk
Module: adp
Module: dos

©2020 Check Point Software Technologies Ltd. 14

Check Point Modules Parameters

• Each module have some parameters which can be adjusted.

Those are also called ‘Kernel Parameters’.
• Those parameters are responsible for various firewall functionality
and behavior.
• For example, the parameter ‘fw_allow_out_of_state_icmp’ will
allow the firewall to accept icmp response packets without previous
icmp request packets
• A list of parameters can be seen using the command ‘modinfo’ on
the module file:
# modinfo -p $FWDIR/modules/fw_kern_64.o
# modinfo -p /etc/ppk.boot/boot/modules/sim_kern_64.o
©2020 Check Point Software Technologies Ltd. 15
[admin@shell] # modinfo -p $FWDIR/modules/fw_kern_64.o
accel_pm_context_simulate_job:string param
accel_pm_simulate_result_collection:int param
cml_av_demo_percent:int param
cmi_lite_print_policy:int param
av_lite_always_f2h:int param
av_lite_default_hash:int param
av_lite_override_policy_default_tasks:int param
cphwd_sxl_reset_state_timer:int param
cphwd_sxl_print_load_balancing_stats:int param
cphwd_print_device_state:int param
cphwd_reset_device_state:int param
cphwd_print_device_state_full:int param
cphwd_print_device_queue_state:int param
…
©2020 Check Point Software Technologies Ltd. 16
Check Point Modules Parameters

• Each parameter has a type description after the ‘:’ sign. This will
indicate us what kind of input it can receive: string param (text), int
param (whole numbers only) etc.
• For example, The parameter ‘accel_pm_context_simulate_job’ can
have the value ‘Hello World’ while the parameter
‘accel_pm_simulate_result_collection’ will have a whole number
value ‘1’.
• In order to read or write from / to a parameter we will use a system
call command:
• ‘ fw ctl get / set <type> <parameter name> <optional value> -a ‘

©2020 Check Point Software Technologies Ltd. 17

Starting R80.20, SecureXL parameters can be also be
modified using System Calls. The ‘-a’ option will
modify the parameter in both FW and SecureXL
modules (if exists).

©2020 Check Point Software Technologies Ltd. 18

[admin@shell] # fw ctl get int psl_strip_syn_with_data -a
FW: psl_strip_syn_with_data = 1
PPAK 0: psl_strip_syn_with_data = 1
[admin@shell] # fw ctl set int psl_strip_syn_with_data 0 -a
PPAK 0: Get before set operation succeeded of psl_strip_syn_with_data
[admin@shell] # fw ctl get int psl_strip_syn_with_data -a
FW: psl_strip_syn_with_data = 0
PPAK 0: psl_strip_syn_with_data = 0

©2020 Check Point Software Technologies Ltd. 19

Check Point Modules Parameters

Do not change Kernel parameters

during production
(unless we know exactly what they do and what they can affect).
If unsure, Please consult TAC.

©2020 Check Point Software Technologies Ltd. 20

Check Point Packet Flow

• Packets which arrive to the gateway are being handled by the

kernel before they are forwarded to the OS (Inbound) and after the
packet returns to the Kernel (Outbound).
• Those directions also called Inbound chain and outbound chain
and can be also represented with ‘i - I’ and ‘o - O’
• Each chain contains a set of logical steps to inspect the packet.
• The number of steps in the chains of each firewall is different.
It depends on which blades/features are enabled on the firewall.
• To see All active chains in the firewall, type:
# fw ctl chain
©2020 Check Point Software Technologies Ltd. 21
[admin@shell] # fw ctl chain
in chain (20):
0: -7fffffff (0000000000000000) (00000000) SecureXL stateless check
(sxl_state_check)
1: -7ffffffe (0000000000000000) (00000000) SecureXL VPN before
decryption (vpn_in_before_decrypt)
2: -7ffffffd (0000000000000000) (00000000) SecureXL VPN after
decryption (vpn_in_after_decrypt)
…
out chain (14):
0: -7f800000 (ffffffff92676db0) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: - 1fffff0 (ffffffff922f6960) (00000001) TCP streaming (out) (cpas)
2: - 1ffff50 (ffffffff91b230e0) (00000001) passive streaming (out) (pass_str)
…
©2020 Check Point Software Technologies Ltd. 22
 Check Point Packet Flow illustration
The packet re-enters the Kernel and goes through the outbound chain (o - O)
The packet goes through the OS (Routing is also performed here)
The packet leaves the Kernel and goes to the NIC
User Mode / OS

I o
The packet enters the Kernel and goes through the inbound chain (i - I)
Kernel

i O
The packet leaves the Gateway
Hardware
The packet is being forwarded to the Kernel

The packet arrives to the gateway and enters the NIC

©2020 Check Point Software Technologies Ltd. 23

Check Point Stateful Inspection

• Stateful inspection is a mechanism which allows the gateway to

enforce RFC communication flow.
• For TCP, it enforces the 3-way-handshake, Connection termination
and Sequences. Also, it makes sure the only the relevant flags for
current connection state will pass.
• For UDP / ICMP. It enforces the state of the conversation (For
example, DNS reply can arrive only after a DNS query has been
seen etc.).
• Enforces Data connections (along with Handlers which will be
discussed now.)

©2020 Check Point Software Technologies Ltd. 24

Handlers

• A hander is a piece of code which tells the firewall how

to inspect traffic based on the Application protocol
• It enforces the RFC of each protocol
• It tells the firewalls which modules to activate
when inspecting a packet.

©2020 Check Point Software Technologies Ltd. 25

©2020 Check Point Software Technologies Ltd. 26
Handlers

• Starting R80.20, Handlers can be changed / removed only in a

custom created service.
• Handlers allow us to configure application protocol inspection on
custom ports. For example, HTTP traffic on port 123 etc.
• Leaving the handler field empty will allow any kind of traffic to pass
on a specific port.
• Some handlers are responsible for creating a Data connection (For
example, FTP) and allowing traffic on it (We did not specifically
allow this in the rule base).

©2020 Check Point Software Technologies Ltd. 27

Inspection Settings

• Inspection settings is a set of configurable options which tells the

firewall which RFC points to enforce and which not. In other words,
under which RFC conditions to allow or drop certain traffic.
• For example, the ‘TCP out of sequence’ setting allow the firewall to
drop / allow TCP packets which are out of sequence.
• Inspection Settings require only the firewall blade to be active.

©2020 Check Point Software Technologies Ltd. 28

©2020 Check Point Software Technologies Ltd. 29
Check Point Kernel Tables

• The Kernel writes almost all of it’s actions to the tables.

• There are many Kernel tables, each for a different Kernel function.
• We can say, the tables are the memory of the kernel.
• For example, There is a table for DNS cache, URL reputation,
actions performed on packets, NAT translations etc.
• Every table has a name and an ID.

©2020 Check Point Software Technologies Ltd. 30

Check Point Kernel Tables

• To view the kernel tables, type the following command:

# fw tab
• To view a certain table, type the following command:
# fw tab -t <table name/table ID>
• To view all entries in the table:
# fw tab -t <table name/table ID> -u
• To view the table translated from HEX to DEC:
# fw tab -t <table name/table ID> -u –f
• To export a table to a file:
# fw tab -t <table name/table ID> -u > filename.txt
• To view the summary of the table:
# fw tab -t <table name/table ID> -s
• To delete the kernel table:
# fw tab -t <table name/table ID> -x -y {!don’t use on production!}
• To delete a specific entry from a kernel table:
# fw tab -t <table name/table ID> -x -e <entry ID>

©2020 Check Point Software Technologies Ltd. 31

Check Point Connection Table

• One of the most used kernel tables is ‘8158 – the connections

table’.
every connection that passed the firewall is written into the
connection table.
• You are initiating a connection to the internet.
The first connection that will be written to the connections kernel
table is a DNS request to resolve the IP of the website you wish to
surf to.
# fw tab -t 8158
OR
# fw tab -t connections
©2020 Check Point Software Technologies Ltd. 32
Check Point Connection Table
Representation of a connection in the connection able:
<6-tuple-key;r_ctype,r_cflags,rule,service_id,handler,uuid1,uuid2,uuid3,uuid4,
ifncin,ifncout,ifnsin,ifnsout,bits1,bits2,connection_module_kbufs@ttl/timeout>
Example:
<00000000, c21d2137, 0000c6d7, ac1ea22f, 00000016, 00000006; 0001c001,
00044080, 00000001, 000001c5, 00000000, 5732f6a3, 00000000, 2fa21eac,
0000ffff, 00000000, 00000000, ffffffff, ffffffff, 02000000, 00000000, 00000000,
00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000,
00000000, 00000000, 00000000, 00000000, 00000000, 6cf0a800, 00000000,
00000000; 3599/3600>

©2020 Check Point Software Technologies Ltd. 33

<00000000, c0a80023, 0000db57, c0a801d2, 00000035, 00000011>
this is the main part of the first connection which is written into the connection
table.
this representation of the first six values of the connection is called a “six
tuple”, or "connection table Entry ID".
00000000 = first value of the tuple indicates on the direction (Inbound).
c0a80023 = 192.168.0.35 – firewall IP – source IP.
0000db57 = 56151 – source port
c0a801d2 = 192.168.1.210 – DNS server IP – destination IP.
00000035 = 53 – destination port
00000011 = 17 - UDP

©2020 Check Point Software Technologies Ltd. 34

Check Point Connection Table

• The “six tuple” is the basic information of the connection that is

written into the connections table. The kernel also writes a lot of
additional information to the connection entries.
• In addition to the “real” (original) connection entry which is written
into the connection table, a symbolic entry for each variation of the
“real” connection is also written into the connection table.
• Those symbolic entries are called ‘S-Links’ – Symbolic Links.
• For a basic connection, a “real” entry and 3 ‘S-Links’ are created:
Inbound: Client -> Server (“real” connection).
Outbound: Client -> Server

©2020 Check Point Software Technologies Ltd. 35

Check Point Connection Table

• And reply:
Inbound: Server -> Client
Outbound: Server -> Client
• Those links will help the firewall to identify the connection on each
of the VM chains.
• For NATed connections, additional S-Links are created (amount of
S-Links depend on NAT method and whether source / destination /
both are NATed).

©2020 Check Point Software Technologies Ltd. 36

Check Point Connection Table

• The reply entry (S-Link) of the DNS request connection will be:

<00000000, c0a801d2, 00000035, c0a80023, 0000db57, 00000011>

00000000 = Inbound.
c0a801d2 = 192.168.1.210 – DNS server IP – source IP.
00000035 = 53 – source port
c0a80023 = 192.168.0.35 – firewall IP – destination IP.
0000db57 = 56151 – destination port
00000011 = 17 - UDP
=========================================================
And the other two S-Links:
<00000001, c0a801d2, 00000035, c0a80023, 0000db57, 00000011>
<direction ,src_ip ,src_port ,dst_ip ,dst_port ,protocol>
<00000001, c0a80023, 0000db57, c0a801d2, 00000035, 00000011>
<direction ,src_ip ,src_port ,dst_ip ,dst_port ,protocol>

©2020 Check Point Software Technologies Ltd. 37

Notes • Connection table - sk65133

©2020 Check Point Software Technologies Ltd. 38

Check Point Tools

©2020 Check Point Software Technologies Ltd. 39

CPVIEW

• CPView Utility is a text based built-in utility that can be run ('cpview'
command) on Security Gateway / Security Management Server /
Multi-Domain Security Management Server.
• CPView Utility shows statistical data that contain both general
system information (CPU, Memory, Disk space) and information for
different Software Blades (only on Security Gateway).
• This data is also saved in history ( a few months back ) and can be
accessed at any time.

©2020 Check Point Software Technologies Ltd. 40

CPPCAP

• A similar tool to TCPUDMP by Check Point.

• The tool is adjusted to Gaia operating system yet requires
installation of an applicable RPM (embedded in R80.40).
• The tool is supported only on 64 bit OS.
• Can reduce CPU when collecting traffic captures.
• Has additional filters for Check Point OS (Such as specifying a
VS).

©2020 Check Point Software Technologies Ltd. 41

FW MONITOR

• Captures packets while they are traversing the Check Point Kernel.
• Allows tracking of packets as they go though the Chains.
• Allows to understand which specific chain is causing a potential
issue.
• Allows capturing and displaying packets going through SecureXL
or VPN.
• Compatible with WireShark.

©2020 Check Point Software Technologies Ltd. 42

CONNSTAT

• An analysis of the Connections Table is required in order to

optimize the rulebase, e.g.: for SecureXL to accelerate more
packets.
• On Windows OS, this analysis can be performed by the Check
Point utility 'connstat'.
• Provides statistical data from the connection table for: Connection
state info (Started, established, closed etc.), Top X Services used,
Top X Rules used, Top X Least Used Rules and Top Clients and
Servers,

©2020 Check Point Software Technologies Ltd. 43

 • Support Debug Tools - sk97443
• CPVIEW - sk101878

Notes • CPPCAP - sk141412

• FW MONITOR - sk30583
• CONNSTAT - sk85780

©2020 Check Point Software Technologies Ltd. 44

Debug

©2020 Check Point Software Technologies Ltd. 45

Debug facts

• For each kernel / SecureXL module, there are various flags which
provide us with different information.
• For example, the ‘conn’ flag of the ‘fw’ module will provide us an
output for each packet which enters and leaves the kernel.
• Each flag adds to CPU load, some flags require low CPU usage
(like ‘drop’ flag) and some have high CPU impact (like ‘ld’ flag).
• Both Firewall and SecureXL modules can be debugged.
• All flags can be seen running the command:
# fw ctl debug –m
# fwaccel dbg
©2020 Check Point Software Technologies Ltd. 46
Debug Buffer

• Each debug requires a buffer.

• Default buffer is 50Kb. This is because critical kernel messages are
constantly written to the /var/log/messages files etc.
• Kernel debug buffer can be increased up to 8192Kb.
• This can be done running the command:
# fw ctl debug -buf <buffer size>

©2020 Check Point Software Technologies Ltd. 47

Debug Buffer

• Each debug requires a buffer.

• Default buffer is 50Kb. This is because critical kernel messages are
constantly written to the /var/log/messages files etc.
• Kernel debug buffer can be increased up to 8192Kb.
• This can be done running the command:
# fw ctl debug -buf <buffer size>
• To write buffer content to a file, we will use the command:
# fw ctl kdebug > <file>

©2020 Check Point Software Technologies Ltd. 48

Debug Commands

• Defaulting all kernel debug properties (Use before and after debug):
# fw ctl debug 0
• Define a kernel debug buffer:
# fw ctl debug -buf <buffer size>
• Check which debug flags are on:
# fw ctl debug -m <module name>
• Add the flags you want to debug on the required FW module:
# fw ctl debug -m <module name> + flag1 flag2 flag3
* Adding the flag ‘all’ will add all flags for this module
• Remove flag:
# fw ctl debug -m <module name> - flag1
©2020 Check Point Software Technologies Ltd. 49
Debug Commands

• Write the buffer into a file (continuously):

# fw ctl kdebug -f > filename.dbg
• Timestamp (-T):
# fw ctl kdebug -T -f > filename.dbg
• Dividing the file multiple file limited to certain size:
# fw ctl kdebug -f -o <file_name> -m <number> -s <size>
• If we would like to make a short debug with only one module, we
can use the command:
# fw ctl zdebug -T -m fw + flag1 flag2 flag3
* The buffer will be set automatically to 1024Kb, the CPU load will still be flag
dependent and remain the same as with regular debug

©2020 Check Point Software Technologies Ltd. 50

Debug Commands

• If we would like to debug SecureXL also, we should use the

following.
• Add the flags you want to debug on the required SecureXL
module:
# fwaccel -m <module name> + flag1 flag2 flag3
• Add all flags for the required SecureXL module:
# fwaccel -m <module name> all
• Remove the flags you don’t want to debug on the required
SecureXL module:
# fw ctl debug -m <module name> - flag1 flag2 flag3

©2020 Check Point Software Technologies Ltd. 51

Debug Filters

• Starting R80.20. It is possible to filter the debug by IPs / Ports /

Protocols.
• Set one or more of the following kernel parameters:
# fw ctl set str simple_debug_filter_saddr_x <Source-IP> -a
# fw ctl set str simple_debug_filter_daddr_x <Destination-IP> -a
# fw ctl set int simple_debug_filter_sport_x <Source-Port> -a
# fw ctl set int simple_debug_filter_dport_x <Destination-Port> -a
# fw ctl set int simple_debug_filter_proto_x <IP-Protocol> -a
* Where “x” can be a value of 1-5, up to 5 concurrent filters.
NOTE: source address will become destination address in the response, if we
would like to capture 2 way communication we must set it for both. Also for
ports.
©2020 Check Point Software Technologies Ltd. 52
Debug Filters

• Filter by IP address (regardless of source / destination):

# fw ctl set str simple_debug_filter_addr_x <ip> -a
* Where “x” can be a value of 1-3, up to 5 concurrent filters.
• Filter by IP address (regardless of source / destination):
# fw ctl set str simple_debug_filter_vpn_x <ip> -a
* Where “x” can be a value of 1-2
• To disable the filters, run:
# fw ctl set int simple_debug_filter_off 1 -a

©2020 Check Point Software Technologies Ltd. 53

Debug Procedure

©2020 Check Point Software Technologies Ltd. 54

 • Kernel debug modules / flags
Notes details can be found in the
administrator guide.

©2020 Check Point Software Technologies Ltd. 55

THANK YOU

©2020 Check Point Software Technologies Ltd. 56