11/09/2019 Kernel Debug

+1-866-488-6691 Contact Us checkpoint.com CheckMates FAQs Blog Welcome: Don Paterson | Sign Out

SELL MARKET LEARN SUPPORT MY CHECK POINT

Support Center > Search Results > SecureKnowledge Details

Search Support Center

Kernel Debug

Rate This My Favorites Email Print

Solution ID sk98799
Product Security Gateway
Version R76, R77, R77.10, R77.20, R77.30, R80.10, R80.20, R80.30
Platform / Model All
Date Created 06-Mar-2014
Last Modified 24-Jun-2019

Solution
The kernel is the bridge between the hardware and the OS. In the Check Point kernel, packets are inspected both in Inbound (ingress) and Outbound (egress) directions. Each direction
has its own modules and order of inspection.
Handlers (INSPECT code) decide which modules will inspect the packet.

The inspection operations in the Check Point kernel are divided into modules, and the modules are divided into chains. The number of chains on every Security Gateway is different. It
depends on which blades/features are enabled on the Security Gateway.

To see all active chains in the Security Gateway, run:

[Expert@HostName]# fw ctl chain

Example:

in chain (17):
0: -7f800000 (f206df90) (ffffffff) IP Options Strip (in) (ipopt_strip)
1: - 2000000 (f149dd70) (00000003) vpn decrypt (vpn)
2: - 1fffff8 (f14a8b20) (00000001) l2tp inbound (l2tp)
3: - 1fffff6 (f206f290) (00000001) Stateless verifications (in) (asm)
4: - 1fffff2 (f14c4940) (00000003) vpn tagging inbound (tagging)
5: - 1fffff0 (f149bc10) (00000003) vpn decrypt verify (vpn_ver)
6: - 1000000 (f20c4980) (00000003) SecureXL conn sync (secxl_sync)
7: 0 (f201df50) (00000001) fw VM inbound (fw)
8: 1 (f2087ed0) (00000002) wire VM inbound (wire_vm)
9: 10 (f202f610) (00000001) fw accounting inbound (acct)
10: 2000000 (f149eaf0) (00000003) vpn policy inbound (vpn_pol)
11: 10000000 (f20ca740) (00000003) SecureXL inbound (secxl)
12: 7f600000 (f20646b0) (00000001) fw SCV inbound (scv)
13: 7f730000 (f21b11f0) (00000001) passive streaming (in) (pass_str)
14: 7f750000 (f231c540) (00000001) TCP streaming (in) (cpas)
15: 7f800000 (f206e320) (ffffffff) IP Options Restore (in) (ipopt_res)
16: 7fb00000 (f22de3b0) (00000001) HA Forwarding (ha_for)

out chain (15):

0: -7f800000 (f206df90) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: - 1ffffff (f149cea0) (00000003) vpn nat outbound (vpn_nat)
2: - 1fffff0 (f231c3c0) (00000001) TCP streaming (out) (cpas)
3: - 1ffff50 (f21b11f0) (00000001) passive streaming (out) (pass_str)
4: - 1ff0000 (f14c4940) (00000003) vpn tagging outbound (tagging)
5: - 1f00000 (f206f290) (00000001) Stateless verifications (out) (asm)
6: 0 (f201df50) (00000001) fw VM outbound (fw)
7: 1 (f2087ed0) (00000002) wire VM outbound (wire_vm)
8: 2000000 (f149e5a0) (00000003) vpn policy outbound (vpn_pol)
9: 10000000 (f20ca740) (00000003) SecureXL outbound (secxl)
10: 1ffffff0 (f14a87b0) (00000001) l2tp outbound (l2tp)
11: 20000000 (f149d110) (00000003) vpn encrypt (vpn)
12: 7f000000 (f202f610) (00000001) fw accounting outbound (acct)
13: 7f700000 (f231e5b0) (00000001) TCP streaming post VM (cpas)
14: 7f800000 (f206e320) (ffffffff) IP Options Restore (out) (ipopt_res)

When troubleshooting and trying to understand which chain is causing a problem on the Security Gateway, use the following command:

[Expert@HostName]# fw monitor -e "accept;" -p all

The '-p all' flag will show all the chains, through which the traffic passed.

To see it in kernel debug, use module fw with the flags monitorall, vm, chain and chainfwd

Note: These flags generate large number of messages, hence will cause high load on CPU.

To see all active modules on the Security Gateway, run:

https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eventSubmit_doGoviewsolutiondetails=&solu… 1/6
11/09/2019
[Expert@HostName]# fw ctl debug -m Kernel Debug

Debugging Modules

Module Explanation

Kiss Kernel I/s (Kernel Infrastructure)

Kissflow Kernel I/s communication with other modules and policy enforcement

fw Firewall, the most common module

h323 H323 VoIP traffic module

Multik CoreXL module

uc UserCheck module

dlpk Data Loss Prevention module

cluster ClusterXL module

CPAS Check Point Active Streaming (unlike PSL)

cmi_loader Signature loading for Security blades

NRB Next Rule Base, security blades rulebase

SGEN Security Gateway Enforcement (stateful)

Very important I/S, DNS resolving, string matching, md5sum check against database and 3rd party, resource
RAD_KERNEL
categorization

WS Web Security module - HTTP inspection by all blades

APPI Application Control module

CI Content Inspection (Anti-Virus, Anti-Bot )

RTM Real Time Monitoring

VPN VPN module

SFT Secure File Type

UP Unified Policy module (For more information, refer to sk120964 - ATRG: Unified Policy)

When debugging the kernel, it is important to first understand the flow of the packets and connections and understand the failure point; it is important to understand which module
you need to debug.

Every module has its own flags that were designed to print specific debug information. It is important to know the common flags to be more accurate when debugging the kernel.

Common traffic flags in fw module

Module Explanation

drop the connection that was dropped, the function in the firewall that dropped it and the reason

information from the connections table (everything that can be seen with this command #fw tab -t 8158) . Direction of packets in
conn
connections, source/destination of IP and ports, service type, handler, timeout, the rule that this packet is matched on, ETC

this flag refers to the kernel dynamic tables infrastructure. It reads and write everything that is written to any kernel table. Adding this
ld flag to the debug syntax may cause the machine high CPU and can even cause the machine to hang. Check load before performing this
debug on production environment.

nat basic NAT information

xlate the firewall NAT infrastructure - basic NAT information + NAT cache

xltrc NAT additional information, print NAT rulebase information and more information from the NAT kernel table (fwx_alloc)

packet information on actions performed on packet - like accept, drop, fragment, inspection

advanced information of the packet's header. stateless verifications - sequences, fragments, translations and other header changes and
packval
verifications.

shows actions of all the virtual chains that can be seen with the command: # fw ctl chain on the traffic that go through the chain handle
vm
function fw_filter_chain

tcpstr TCP streaming mechanism, organize TCP packets by order and checks protocol messages. debug the PSL I/S. (passive streaming layer)

chain chain cookie information, chain modules

chainfwd chain forwarding - related to fwha_perform_chain_forwarding global kernel variable (clusterXL only)

the kernel saves information and holds connections for inspection, this flag will show virtual de-fragmentation and cookie issues (cookies
cookie
in the data structure holding the packets)

hold holding mechanism and all packets being held / released

https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eventSubmit_doGoviewsolutiondetails=&solu… 2/6
11/09/2019 Kernel Debug

Common IPS flags in fw module

Module Explanation

aspii Accelerated Stateful Protocol Inspection Infrastructure (INSPECT streaming)

spii Stateful Protocol Inspection Infrastructure

cmi Context Management Infrastructure - IPS signature manager

advp IPS and Application Control signatures, contexts and patterns

synatk IPS protection "SYN Attack" (SYNDefender)

Kernel debug command line usage

Defaulting all kernel debug properties:

[Expert@HostName]# fw ctl debug 0

Define a kernel debug buffer:

[Expert@HostName]# fw ctl debug -buf <buffer_size>

Note: For R80.20 and above, kernel debug buffer is defined per CoreXL instance and it is limited to 8200 bytes.

If the machine has enough memory, it is recommended to set the maximum buffer of 32768.
The fact that the buffer size is 8000 or 32768 will not affect the file size.
The kernel writes the debug messages to the buffer. A user mode process named 'kdebug' writes the messages from the buffer to the file.

Check which debug flags are enabled:

[Expert@HostName]# fw ctl debug -m

Check which debug flags are on for specific module:

[Expert@HostName]# fw ctl debug -m <module_name>

Set the debugging flags for the required module:

[Expert@HostName]# fw ctl debug -m <module_name> + flag1 flag2 flag3

UnSet the debugging flags from the module:

[Expert@HostName]# fw ctl debug -m <module_name> - flag1 flag2 flag3

Add Timestamp (necessary to match the traffic capture to kernel debug):

[Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt

Dividing the debug output file into multiple file limited to certain size:

[Expert@HostName]# fw ctl kdebug -f -o <file_name> -m <number> -s <size in KB>

Kernel tables

The kernel writes all of its actions to relevant tables. Every kernel table has an assigned name and assigned kernel ID, and it holds specific information.
The forwarding, handling and decisions will be done based on informatin that is stored in different kernel tables.

Kernel tables command line usage

To view the kernel tables, run:

[Expert@HostName]# fw tab

To view a certain table, run:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID>

To view all entries (unlimited number) in the table:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -u

To view the table entries translated from HEX to DEC format:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -u -f

To export the table contents to a file:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -u > /var/log/file_name.txt

To delete all entries from a certain kernel table (all involved connections will be immediately lost and disconnected):
https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eventSubmit_doGoviewsolutiondetails=&solu… 3/6
11/09/2019
[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -x -y Kernel Debug

Important Note: This operation deletes all information from the table. Do NOT use this command in production environment, as it might cause a traffic outage.

To delete specific entry:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -x -e <Entry_ID | Tuple>

Display a summary about entries in certain table:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -s

Display a summary about entries in all tables:

[Expert@HostName]# fw tab -s

Connections Table

The most used kernel table is the Connections Table. Every connection that passed the firewall is written to the Connections Table. Timeouts, rule numbers, symbolic links, state
decisions, all are written and decided by this table.

Real entry contains the following:

<DIRECTION,6-tuple-
key;r_ctype,r_cflags,rule,service_id,handler,uuid1,uuid2,uuid3,uuid4,ifncin,ifncout,ifnsin,ifnsout,bits1,bits2,connection_module_kbufs@ttl/timeout>

Tuple is the basic information on the connection that is written to the connections table.

<direction,src_ip,src_port,dst_ip,dst_port,protocol>

Refer to sk65133 - Connections Table Format.

Example 1:

<00000001, C21D2B59, 0000AF6B, C21D24D9, 00000035, 00000011> - This is the first connection that was written to the Connections Table. The culmination of all six values
named "six tuple" or - connections table EntryID

00000001 - first value of the tuple indicates the direction (00000001 - outbound)

C21D2B59 - Source IP - in our example, Security Gateway (in Dec = 194.29.43.89)

0000AF6B - Source port (in Dec = 44907)

C21D24D9 - Destination IP - in our example, DNS Server (in Dec = 194.29.36.217)

00000035 - Destination port (in Dec = 53)

00000011 - Protocol (in Dec = 11 = UDP)

Example 2:

The reply of the DNS request connection will be:

<00000000, C21D24D9, 00000035, C21D2B59, 0000AF6B, 00000011>

00000000 - first value of the tuple indicates the direction (00000000 - inbound)

C21D24D9 - Source IP- in our example, DNS Server (in Dec = 194.29.36.217)

00000035 - Source port (in Dec = 53)

C21D2B59 - Destination IP - in our example, Security Gateway (in Dec = 194.29.43.89)

0000AF6B - Destination port (in Dec = 44907)

00000011 - Protocol (in Dec = 11 = UDP)

Kernel debug analysis

In terms of analysis for the debug, the ability to analyze the debug depends on how familiar you are with the expected behavior.

Here is a basic flow of accepted inbound handling of a connection:

1. fwconn_lookup: connection lookup, the Security Gateway wants to see if there is a match for the connection that just arrived via the interface

[-- Stateful VM inbound: Entering (1244447195) --];

Before VM:

https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eventSubmit_doGoviewsolutiondetails=&solu… 4/6
 <192.168.121.1:4461 -> 10.0.121.13:23 IPP 6> (len-48) TCP flags-0x2 (SYN), seq-55a92b89,
11/09/2019 Kernelack-0,
Debugdata end-55a92b8a (ifn-0) (first seen) (looked up)

2. Since the connection is "not found in connections table", the next step is to record it

fwconn_record_conn: record conn

3. Stateful inspection - SYN:

fwconn_record_conn: SYN packet. Turn on SYN_SENT flag;

Since we have enabled the syn_sent flag, only RST or SYN-ACK are allowed. In the phase for example, if any other packet besides RST or SYN-ACK, it will be dropped on "out of
state".

4. Getting further information from the Security Gateway connections table:

fwconn_get_service_timeout: Getting default protocol

5. Setting partial_handle bit: loading further code handling.

fwconn_record_conn: setting PARTIAL_SETUP flag

6. Creating REAL entry

fwconn_record_conn: created real

<dir 0, 192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>

7. After we checked that stateful inspection is ok, rulebase matching:

fw_handle_first_packet: Rulebase returned

8. NAT rulebase match:

fw_xlate_match conn-<192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>;

fw_xlate_find_all_matches_old: No match for conn

9. Creating symbolic links (after NAT decision)

fwconn_init_links: Creating links

fwconn_init_links: Creating links (inbound). One way links-0, Replies from any-0;

;fwconn_set_links_inbound: create link cls_o <dir 1, 10.0.121.13:23 -> 192.168.121.1:4461 IPP 6> -> <dir 0, 192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>(0x5);

fwconn_set_links_inbound: create link srs_o <dir 1, 192.168.121.1:4461 -> 10.0.121.13:23 IPP 6> -> <dir 0, 192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>(0x2);

10. Handling inbound is finished:

fwconn_init_links: connection completed, unset PARTIAL_SETUP bit;

11. fw_tcp_state_update: first SYN

12. VM Final action-ACCEPT;

13. ----- Stateful VM inbound Completed -----

14. fwconn_lookup (outbound.......)

<10001,44000,1,1ae,0,4a2cc1db,0,4779a8c0,ffff,0,ffffffff,ffffffff,ffffffff,2000000,0,0,0,0,0,0,0,0,0,0,0,0,7c9c0800,0,0>

found in connections table dir-1;

After this phase, the packet is entering the outbound chains and the handling will continue in accordance with the chain forwarding.

15. [-- Stateful VM outbound: Entering (1244447195) --];

16. This point represents the chains between I and o.

Before VM:

<192.168.121.1:4461 -> 10.0.121.13:23 IPP 6> (len-48) TCP flags-0x2 (SYN), seq-55a92b89, ack-0, data end-55a92b8a (ifn-1) (looked up) ;

https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eventSubmit_doGoviewsolutiondetails=&solu… 5/6
 fwconn_chain_update_anti_spoofing_cache: updating anti-spoofing interface info. [cin-0,
11/09/2019 Kernelcout--1,
Debug sin--1, sout--1], dir-1, cdir-1, new ifn-1, spoof offset-3, previous ifn--1;

17. NAT rulebase match, cache and Slinks, after we have routing info.

fw_xlate_match: conn-<192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>; ; 8Jun2009 10:46:35.420110;fw_xlate_match: cache hit!;

18. fw_conn_post_inspect: Setting SRS_OUTBOUND_SEEN flag;

19. Syn accepted, written, inspected by stateful inspection, rulebase and anti-spoofing.

fw_filter_chain: Final switch, action-ACCEPT;

After VM:

<192.168.121.1:4461 -> 10.0.121.13:23 IPP 6> (len-48) TCP flags-0x2 (SYN), seq-55a92b89, ack-0, data end-55a92b8a

VM Final action-ACCEPT
----- Stateful VM outbound Completed -----

Currently as we see, the timeout is "25", as the handshake was not completed.

The SYN-ACK will go through the same inspection process, however the timeout will be also be "25". It will change only after the 3-way handshake will be completed.

In terms of stateful inspection, the only allowed flags according to the current state (SYN_SENT) are SYN-ACK and RST.

Every other packet will be dropped for "out of state".

This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.

Give us Feedback Please rate this document [1=Worst,5=Best]

Enter your comment here

Comment

©1994-2019 Check Point Software Technologies Ltd. All rights reserved.

Copyright | Privacy Policy

https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eventSubmit_doGoviewsolutiondetails=&solu… 6/6