ATRG - Application Control
ATRG - Application Control
Solution ID sk73220
Technical Level
Solution
Table of Contents
Introduction
Process Flow Diagrams and Description
Application Control Troubleshooting Flow Chart
Application Control Debug Procedures
Next Rule Base (NRB) Debugging Instructions
Application Control and URL Filtering Update Process Debugging
Related Solutions
Introduction
The Need for Application Control
The use of internet applications comes with risks that administrators must know about:
Malware threats: Application use can open networks to threats from malware. Popular applications like Twitter, Facebook, and YouTube can cause users to dow
viruses unintentionally. File sharing can easily cause malware to be downloaded into your network.
Bandwidth hogging: Applications that use a lot of bandwidth, for example, streaming media, can limit the bandwidth that is available for important business
applications.
Loss of Productivity: Employees can spend time on social networking and other applications that can seriously decrease business productivity. Employers do no
what employees are doing on the internet and how that really affects them.
Check Point's latest firewall innovation brings the industry's strongest application and identity control to organizations of all sizes. You can easily create policies which
detect or block thousands of applications.
Granular Application Control: Identify, allow, or block thousands of applications. This provides protection against the increasing threat vectors and malware
introduced by internet applications.
Largest application library with AppWiki: Comprehensive application control that uses the industry's largest application library. It scans for and detects more th
8,411 applications and more than 255,736 Web 2.0 widgets. Checkpoint data base is updated frequently with world wide Apps and widgets.
Note: The CMI Loader is an underlying I/S that is responsible for signatures handling for Application Control and IPS blades.
In R75.20, we expand Application Control to also include custom applications located in the Check Point site, for specific customer use. Also, we introduce an addition
source:
Application control and Custom applications signatures are kept in the Application Database (APP DB), based on two files, appi_db.C and appi_custom_db.C, located o
Security Gateway. User defined URLs and HTTPS Inspection User defined URLs data is kept in the Security Gateway database.
We integrated URL Filtering together with Application Control, and both features are set in the Application Control engine (Application Control) rulebase.
URL Filtering is represented by a new CMI application client: URL Filtering. This client behaves like an Application Control client. It uses Application Control DB for
metadata, but instead of using signatures, it uses the Online application detection engine (RAD) service. (RAD (Short for Resource Advisor) is responsible for the detec
Social Network widgets, e.g Mafia Wars).)
In addition to signatures, application metadata (such as description, tags, etc.) is read from appi_db.C.
The final stage is serializing the data to the FireWall kernel. This completes the install policy process.
Updates Flow
FWD is the process that runs the update mechanism on the Security Gateway. It uses a dynamic library named CIU that is responsible for the execution of the update
process. The update process is only executed, if it is enabled in the policy, and it is executed according to the user defined schedule (every 2 hours, by default).
1. CIU fetches a Version file from the download site. The version of the new file is checked to find out if the Security Gateway is up to date or should be updated.
2. CIU fetches an update package, from the download site, that includes the databases of the Application Control (APPI) & URL Filtering (URLF) blades.
3. CIU executes the "fw load_sigs" command as a separate process to perform the actual commit of the new database.
4. The flow of CMI Loader during the update process is similar to the policy installation process: CMI Loader collects signatures from multiple CMI apps and comp
them together into unified Pattern Matchers (one for each context - such as URL, Host header etc.).
5. The signatures and the applications metadata are read from the new APP DB, fetched in Step 2 of the update (appi_db.C.tmp on the Security Gateway).
6. The data is serialized
7. At this point, CIU collects the status of the "fw load_sigs" command execution
8. If the status is OK, CIU executes another process: "rad_admin update" that loads the updated local database (LDB) of the URL Filtering blade. See URL Filterin
Engine.
9. If the second process is successful as well, CIU replaces the previous version and database files in the $FWDIR/appi/update/ directory with new ones.
10. CIU updates the $FWDIR/appi/update/appi_status.C file with the new update status. See Application Control and URL Filtering Update Process Debugging.
The parsers divide the connection to contexts (e.g. HTTP Host Header, TCP Client to Server Data) and then call CMI that invokes execution of Pattern Matchers on the
different contexts, or the parsers call directly to relevant blades.
After a match by the Pattern Matcher, the CMI Loader notifies the relevant blade that its signature was matched. Application Control retrieves all the data regarding th
application (its categories, priority, etc.), based on the signature that was matched, or goes to the online service for social network widget detection. URL Filtering ret
the URL context from CMI and sends it the URL Filtering Online Service for categorization.
The data regarding the matched application and/or the URL category is transferred to the Application Control connection, managing the connection state. The connec
holds a list of transactions that may include an HTTP transaction, representing a single HTTP request-response, or a connection transaction, representing a whole TC
client-server connection.
The data is transferred to the relevant (current) transaction that aggregates it to the point, where a decision must be made (usually the first response from the server
At this point, the transaction invokes a rulebase execution, passing all the data detected on it (applications, categories etc.). It is the rulebase's responsibility to
communicate with the user-awareness unit in order to retrieve the current user.
Application Control is responsible for enforcing the rulebase policy according to the matched rule. It can be understood from the above that Application Control and U
Filtering are managed and enforced together, meaning that each blade is responsible for collecting its own data and transferring it to the shared state (Application Co
connection).
In R80.x:
The data regarding the matched application and/or the URL category is transferred to the Unified Policy connection, managing the connection state. The connection h
list of transactions that may include an HTTP transaction, representing a single HTTP request-response, or a connection transaction, representing a whole TCP/UDP
server connection.
The data is transferred to the relevant (current) transaction that aggregates it to the point, where a decision must be made (usually the first response from the server
At this point, the transaction invokes a rulebase execution, passing all the data detected on it (applications, categories etc.). It is the rulebase's responsibility to
communicate with the user-awareness unit in order to retrieve the current user.
General
In order to inspect HTTPS encrypted traffic, we implement man-in-the-middle on the SSL protocol.
The Security Gateway poses as a client with the server, and as a server with the client (for this purpose the Security Gateway creates a fake server certificate with all fi
equals to the origin except the public key, issuer, issuer signature and CRL DP). Thus the Security Gateway creates two different sets of keys: one (CF key) to encrypt/d
client side traffic, another (SF key) to encrypt/decrypt server side traffic.
In this case first two messages between Client and Proxy are not SSL messages, but HTTP request and response: Client sends "CONNECT domain:port HTTP/1.1" req
Proxy to notify it about SSL tunneling that should be established. Proxy replies with "Connection Established" response. From this point, all traffic between Client and
is common SSL traffic.
The HTTPS Inspection RuleBase runs mostly in the kernel. The enforcement is done completely in the kernel, while policy installation is done partially in the kernel an
partially in user space. There is no dedicated process. Policy installation code is part of the "regular" policy installation of the Security Gateway, and is executed under
"fw" process (e.g., fw fetch localhost).
Matching
SSL Inspection requires CPAS streaming to be initialized. CPAS (active) streaming is a much more resource-consuming infrastructure than PSL (passive) streaming,
therefore we try to avoid CPAS streaming initialization, as soon as possible. In order to avoid CPAS streaming initialization, when we do not really need it, we run HTTP
Inspection rulebase twice on the same connection.
Notes:
Check Point's Active Streaming infrastructure (CPAS) supplies applications above it the means of handling the TCP data stream, without having to care about th
implementation below (e.g. 3 way handshake enforcement, retransmissions, checksums, etc.). CPAS, as opposed to PSL, supplies means of modifying the data
passed, such as enabling the changing of data, removing parts of it, and even injecting completely new data. These capabilities where once achieved through th
Security Servers. This included folding the connection to the firewall and having context switches to and from user-space. Since CPAS is implemented complete
the kernel, there is no more need for context switches, hence a performance increase.
PSL is the passive streaming infrastructure used in R70 that replaced the TCPSTR infrastructure that existed pre-R70.
The first time it is run on the SYN packet, as part of the Security Rulebase execution (fw_https_inspection_exe_rulebase_SYN function). Thus, if HTTPS Inspection Rul
returns NO MATCH, then SSL Inspection will not influence the steaming type decision (PSL will be used in most cases), but if HTTPS Inspection Rulebase returns
MATCH/POSSIBLE MATCH (we cannot make final decision), then active (CPAS) streaming will be initialized.
The second time it is run on the Client Hello packet (https_inspection_exe_rulebase function). This time, we make the final match (URL is taken from DN cache or from
previous CONNECT request), calculate the action that should be performed (according to action from the matched rule, blades defined on the matched rule and excep
and perform the action (INSPECT/BYPASS) on the connection.
Beside the common columns in RuleBase we have several non standard ones:
Site Category - Behaves as Application Control / URL Filtering Application/Site column, but here we use DN instead of full URL. We get the DN either from CONNECT
request (proxy) or from DN cache that maps server's IP to DN from server's certificate. In order to create the mapping of IP<->DN, the HTTPS inspection rulebase per
a one-time inspection for every new host accessed, regardless of the policy.
Blades - Determines which blades will be activate after SSL termination. Note that there may be exceptional defenses that will not be activated, although the blade is
because they are located before termination. An example of that is Advanced Patterns.
Connection could be held during HTTPS Inspection rulebase execution due to Identity Awareness (IDA) or RAD/URL Filtering request for a-sync execution.
Rulebase Matching
From R80.10: Refer to the "Rule Matching in the Access Control Policy" section in the Next Generation Security Gateway R80.10 Guide.
Pre-R80.10
Application Control and URL Filtering uses Transaction objects in order to manage the connection parsing events, detecting applications/URL Filtering and enforcing
Transaction object represents a single set of client request and server response in a connection.
One of the important phases in connection flow is calling Application Control/URL Filtering rulebase for matching with the detected applications/categories (and their
properties) and the connection data.
Every RB based on our NRB (New Rule Base) I/S includes common columns I/S (src, dst, service, action, track etc...) with additional column - Applications/Sites.
Each column maintain its data in a kernel table. The tables are built as follows: key = specific match data (application id, source ip etc...) and value = index to another t
(called appi_rb_intvl_kbufs_table) whose value is an array of rule number ranges.
The matched rules of every column are intersected to create the final matched rules list.
RB is integrated with Identity Awareness for matching the Src/Dst column and UserCheck for Action.
Application Control/URL Filtering RB is "first found" RB, therefore the first rule of the final matched rules list.
After RB match, Transaction may enforce the connection according to match result and additional action information from RB, such as action, tracking or rule UID.
Note that HTTPS Inspection RB is based on the same NRB I/S and adds a Site Category and Blade column. Same behaviour also here.
Application Control Troubleshooting Flow Chart
The flow chart above lists issues that are commonly encountered at various stages, such as Installation, Update DB and Operation:
Installation: You may encounter licensing issues and issues caused by the fact that the 61000 appliance does not specify Application Control as a blade.
Update Database: You may encounter cases where update of the Application Control database does not work.
Operation: You may encounter cases where:
General Description
NRB is responsible for executing the Application Control blade rule base. The rule base is activated upon application detection and returns the result to the Applicatio
Control Enforcement Manager for actual enforcement (log, connection handling, etc.).
Processes
The main part of the "Install Policy" process of the rule base is done as part of code generation in the Security Management machine. This results in kernel tables tha
be viewed in the $FWDIR/conf/<policy_name>.pf file.
Main Concepts
From R80.10: Refer to "Creating an Access Control Policy" in the Next Generation Security Gateway R80.10 Guide.
Pre-R80.10
Match
The tables are built as follows: key = specific match data (application ID, source IP, etc) and value = index to another table (called appi_rb_intvl_kbufs_table) in which i
value is an array of rule number ranges. An example array is [PROJECTINFO:1,2,3,5,7,8], meaning the matched rules are 1,2,3,5,7,8.
The matched rules of every column are intersected to create the final matched rules list.
The Application Rule base creates the following kernel tables:
The "Internet" option, in the destination column, is implemented using Check Point zones. Internet connection is defined as a connection to an External or DMZ zone w
source zone is not the Local zone.
fw ctl debug 0
export TDERROR_ALL_ALL=0
Management User Space Install policy - code generation
fwm load <policy_name> > /var/log/debug_policy_load.txt
From R80.10:
GW/
Kernel/
fw ctl debug 0
export TDERROR_ALL_ALL=0
Management User Space Install policy - code generation
Troubleshooting
If you know one matching parameter, such as the connection five tuple, the ID of the application that was or should be matched, or the matched rule number, you can
in the debug output and follow the lines above/below to try to figure out the problem. Look for the following lines:
"setting conn parameters on handle <address>" following your connection five tuple.
"setting app id <application id> on handle <address>" - this line should contain the id of the application. This way you can either know which application was rea
matched on the connection, or if you already know the application ID (using application database), you can track the matching section.
"matching column Application (handle <address>)" - following this line, you should see "matching app id" and "matching cat id <tag/category id>" messages. Fr
this, you can conclude what are the tag/category IDs that were received during enforcement.
"rulebase "match" (index <rule number>)" - this line contain the matched rule number
"Internet" is never matched A. Turn on kernel debug of NRB module (UP module in R80.x) and Topology is not defined on the Define Security Gateway
APPI module and follow Internet calculation according to Security Gateway topology. At least one inte
zones. should be defined as "Exte
B. Search for line "received INzone = <name> OUTzone = <name> or "DMZ".
(SYN 0)" and make sure it suits the Internet definition (see
"Main Concepts" section).
Wrong rule number 1. Turn on kernel debug of NRB module (UP module in R80.x) and "Internet" in destination Update Managemen
APPI module. (see issue above). Gateway application
2. Identify your desired match section in the debug output. database.
3. Note which application was matched and what are its The application was not Check Identity Awa
tags/categories. This way you can find out if there is a problem identified. configuration.
with the application identification engine, or maybe a No synchronization Define Security Gat
synchronization problem between Management and Gateway.
between GUI applications topology.
1. Check that the application ID and its tags/categories database and
match your expectations. enforcement applications
2. If the expected rule has a specific application (not "Any"), database.
open $FWDIR/conf/shadow_objects.C to make sure the Identity Role was not
ID exists in this file and was received during identified.
enforcement.
3. Compare information with application database on the
Security Gateway.
This was a suspected false positive packet. We had to know which rule it matched, in order to understand this false positive.
---------------------------
This was a suspected false positive packet. We had to know which rule it matched, in order to understand this false positive.
;[cpu_1];[fw4_0];1504195152:[SID: 73661] {rulebase} up_rulebase_execute_match: [SUMMARY ffffc200205e6228] MATCH dir 0, 192.168.109.58:64040 -> 172.16.4.45:22
6;
General Description
Application Control (APPI) and URL Filtering (URLF) Blades have a unified update logic. Updates are divided to Management Server updates and Security Gateway upd
Management package includes all the application information, such as application name, category, description, tags, risk etc. It is used for policy definition in the rule
Gateway package contains the applications info for logging purposes, application signatures, used as an input to the Application Control engine (Application Control) f
detection, and some large URL groups for use of the URL Filtering engine. For the blade to work optimally, it is highly recommended that the management and the Se
Gateway will be up to date. Updates are distributed meaning that the Management Server and each Security Gateway (and even each cluster member) is responsible f
own database package update. Updates are performed periodically according to the updates configuration of the Application Control and URL FIltering tab in
SmartDashboard.
Processes
FWD is the process that runs the update mechanism on the Security Gateway. It uses a dynamic library named CIU that is responsible for the entire execution of the u
process. Eventually, if a package is downloaded successfully, it will be passed to the kernel for final deployment.
Important Files
Application Control and URL Filtering update files are located in $FWDIR/appi/update/:
1. The main database files are appi_db.C and urlf_db.bin. The files are zipped and encrypted.
2. next_update: a clear text file that contains the time of the next update try. There are two lines in the file, each line is a number interpreted as seconds since 197
can use the following site in order to convert this number to a human readable date and time: http://www.onlineconversion.com/unix_time.htm. The top line is
time of the next update, in case the last update try succeeded. It is set according to the last update try time and the update interval configuration. The second li
the update in case the last update try failed. It is set to an hour from the last update try. This number is expected to be larger than the first, when the system is
working properly.
3. appi_status.C: This file is a clear set file that also contains the current status of update of the Application Control & URL Filtering Blades on the Security Gatew
These are the relevant lines in the file:
app_update_status: the status of the last update try, can be 'failed', 'up-to-date' or 'new'.
app_update_description: a detailed description of the last update result.
app_next_update_description: a description of when the next update should happen.
app_db_version: the version of the current database.
4. Version: This file is a set file that contains details about the current database. It looks as follows:
:pkg_file_name ("appi_urlf_db_pkg.tar")
:md5sum ("a14e72a1b93742f7fe8558ab3cd91662")
:pkg_version (290611_1)
:pkg_timestamp (1309331058)
:appi_filename ("appi_db.C.tmp")
:urlf_filename ("urlf_db.bin.tmp")
The first line contains the file name as stored on the server. The second line contains the md5 sum of the file. The third line contains the date of the package an
running number used for supporting a release of more than one package a day. In this case, the package was released on 29 June 2011 and it is the first packag
released that day. The fourth line contains the release date and time of the package in seconds since 1970. Final 2 lines are the contents of the package.
The version of the package is taken from the second line, and in this case it is 11062901.
Note: The file is created only after the first successful update.
fw debug fwd on TDERROR_ALL_CIU=5 GW User Space Start the debug of FWD daemon (per sk86321) before an update should happen.
Troubleshooting
Issue Debugging Probable Causes Resolution
Force update 1. Change the current directory to
$FWDIR/appi/update/.
2. If a file named "Version" exists, then
rename it: # mv Version Version.backup
3. Edit the next_update file: # vi
next_update
4. Change the top line to have the value 100.
5. The size of next_update is now 15 bytes,
wait until it will change again to 21 (can
be seen using "watch ls -l" command).
This should take no more than 3 minutes.
Updates fail - "...Internal error occurred" Follow Step 5 above ("Force and debug an
message appears in the update status in the update").
Security Gateway's page of the Application
Control and URL Filtering blades (it is a tooltip)
or in the system query of Application Control
and URL Filtering in SmartView Tracker.
Updates fail - "...Failed to reload database" Follow Step 5 above ("Force and debug an Corrupted Update Signature Development Group peop
message appears in the update status in the update") twice using the following syntax in database.
Security Gateway's page of the Application debug of FWD daemon:
Control and URL Filtering blades (it is a tooltip) once with "fw debug fwd on
or in the system query of Application Control TDERROR_ALL_policy=5" and once with "fw
and URL Filtering in SmartView Tracker. debug fwd on TDERROR_ALL_module=5".
Tips
Update failure might be temporary. Try to force an update before troubleshooting further.
Related Solutions
sk112249 - Best Practices - Application Control
sk90422 - How to modify URL Filtering cache size?
sk115652 - Users are still able to post and chat on various web sites, although Application & URL Filtering policy is configured to block those web sites
sk120349 - How to export the list of URLs contained in custom Application/Site object