App Note

SRX Troubleshooting Guide

Introduction
With the introduction of the SRX Gateway, Juniper has merged the legendary routing capabilities of the JUNOS
network operating system, with the hardened security functionality of ScreenOS. The new security
functionality which has been brought into the JUNOS platform has created many new facilities for
troubleshooting, which even those familiar with JUNOS may need some introduction to. The goal of this
document is to discuss troubleshooting some of the common components of the SRX platform.

Scope
This document will go in depth into troubleshooting components of the platform to the extent that such troubleshooting is
supported for field purposes. This document will not go into troubleshooting at the developer level, as this will require
JTAC involvement. This document will focus primarily on troubleshooting the SRX itself, and will not going into details of
troubleshooting other platforms, or the use of non JUNOS utilities. A fundamental understanding of JUNOS and the SRX
is expected in this document to gain the best understanding of the content presented.

Troubleshooting Components Table of Contents

Basic System Health Checks
Basic JUNOS Debugging
Troubleshooting Traffic Flows
Troubleshooting Virtual Private Networks
Troubleshooting SRX Intrusion Detection and Prevention
Troubleshooting SRX High Availability
Advanced Insight Solutions

Basic System Health Checks

Often, when a problem arises the first thing that a network administrator will want to do is perform some basic checks of
the network equipment to ensure that everything is functioning as expected. Obviously the checks performed can vary
widely depending on the nature of the issue, however our goal here is to cover some of the most common commands to
and show the output. Common utilities such as Ping and Traceroute are expected knowledge, and won’t be covered here,
however we will discuss JUNOS commands that will give you very valuable information.
Checking Interfaces

  ƒ‰‡ͳ ‘ˆͶͻ

 App Note

The “show interfaces” commands will give you important detail around the state of the interfaces on the
JUNOS platform itself.

show interfaces <interface> terse

Sometimes you simply want to see the state of an interface of interfaces. You can specify the interface which
you would like to see the state of, or omit the interface to see all interfaces. Note that if you do not specify the
unit, you will see the physical interface and all of its units. We can also see the L3 Protocols configured and
respective addressing.

root@SRX210> show interfaces ge-0/0/0 terse

Interface Admin Link Proto Local Remote
ge-0/0/0 up up
ge-0/0/0.0 up up inet 192.168.224.3/24

show interfaces <interface> extensive

There are varying degrees of detail that you can gather for the interface output (including brief, detail, and extensive) but
we are going to discuss the extensive option here. The extensive option gives an in depth look into the state of the
interface, interface errors, protocols enabled, addressing, interface security settings , queues, traffic rates, traffic
processing, and other important factors that may help resolve issues. If we specify the physical interface but not the unit,
we will see the physical state of the interface, and the respective units.
root@SRX210> show interfaces ge-0/0/0
Physical interface: ge-0/0/0, Enabled, Physical link is Up
Interface index: 131, SNMP ifIndex: 119
Link-level type: Ethernet, MTU: 1514, Link-mode: Full-duplex, Speed: 100mbps,
BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
Source filtering: Disabled, Flow control: Enabled, Auto-negotiation: Enabled,
Remote fault: Online
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x0
Link flags : None
CoS queues : 8 supported, 8 maximum usable queues
Current address: 00:24:dc:d4:e5:00, Hardware address: 00:24:dc:d4:e5:00
Last flapped : 2009-07-30 17:45:03 EDT (3d 18:17 ago)
Input rate : 808 bps (1 pps)
Output rate : 1344 bps (0 pps)
Active alarms : None
Active defects : None

  ƒ‰‡ʹ‘ˆͷ͸

 App Note

Logical interface ge-0/0/0.0 (Index 71) (SNMP ifIndex 120)

Flags: SNMP-Traps Encapsulation: ENET2
Input packets : 583637
Output packets: 124217
Security: Zone: trust
Allowed host-inbound traffic : bgp ospf dhcp http https ike ping ssh telnet
Protocol inet, MTU: 1500
Flags: Is-Primary, Sample-input, Sample-output
Addresses, Flags: Is-Default Is-Preferred Is-Primary
Destination: 192.168.224/24, Local: 192.168.224.3,
Broadcast: 192.168.224.255

Checking Packet Forwarding

show arp [no-resolve]

When interface troubleshooting shows that the interface is up, however we still cannot reach a peer device on the same
L3 network, then it is often helpful to check the ARP table. Checking the ARP table can help to show if we are getting
resolution between the Layer 3 and Layer 2 addresses. Since even transparent firewalls and IPS’ should not ordinarily
block ARP packets. The absence of an ARP entry may indicate that the peer host is not up (if there is an intermediate
Layer 1 Network) that something is blocking the ARP’s (such as Dynamic ARP Inspection on Ethernet Switches) or that
they may be an IP Address conflict on the network. Some High Availability and Load Balancing products also use ARP
manipulation to trigger traffic failovers, so checking which MAC address is present (if it changes) will be helpful in these
cases as well. We can use the “no-resolve” option to bypass name resolution (which will happen by default) that may
sometimes take a while to print the ARP table. In the output we can see the MAC addresses which are associated with
the respective IP Addresses, and which interface they are seen on.

root@SRX210> show arp no-resolve

MAC Address Address Interface Flags
00:10:db:ad:4b:82 192.168.224.1 ge-0/0/0.0 none
00:0b:78:66:a7:17 192.168.224.5 ge-0/0/0.0 none
00:10:db:a7:b1:82 192.168.224.6 ge-0/0/0.0 none
00:0c:29:a5:14:82 192.168.224.27 ge-0/0/0.0 none
00:1d:7d:0b:4c:28 192.168.224.30 ge-0/0/0.0 none
00:0c:29:c3:1d:b2 192.168.224.35 ge-0/0/0.0 none
00:0c:29:fc:2f:70 192.168.224.60 ge-0/0/0.0 none
Total entries: 7

show route
Checking the routing table of a platform can provide some very good insight into why the firewall is processing traffic in an
unexpected way. Just like with ScreenOS, when a packet arrives on an interface, we know what the source interface, and
therefore what the source zone is, we then do a route lookup (within that VR) and determine what the outgoing interface is,

  ƒ‰‡͵‘ˆͷ͸

 App Note

and therefore it’s outgoing zone (in Transparent Mode we typically ARP for the destination if we do not know where it is,
although this is configurable.) With the source and destination zones known, we can then do a policy lookup to find the
appropriate rule. Therefore if we have an incorrect route, it can lead to traffic not being processed in an expected manner.
When viewing the routing table, JUNOS will show us all valid routes that we know about, even if they are not selected as
being active. This is very helpful when you get routing information from multiple different sources, such as for different
dynamic routing protocols. We can view the active routes only with the “show route forwarding-table” command. Here we
will show the routing table as is. This will show all routing instances along with active and inactive (but valid) routes.
Routes that are labeled with an * show that they are valid and in the forwarding table. If using Equal Cost Multiple Path,
then you can have multiple active routes for the same prefix.

root@SRX210> show route

inet.0: 10 destinations, 11 routes (9 active, 0 holddown, 1 hidden)

+ = Active Route, - = Last Active, * = Both

0.0.0.0/0 *[Static/5] 3d 20:28:30

> to 192.168.224.1 via ge-0/0/0.0
[OSPF/150] 3d 20:27:53, metric 2, tag 0
> to 192.168.224.1 via ge-0/0/0.0
172.19.254.0/24 *[OSPF/10] 3d 04:45:04, metric 11
> to 192.168.224.1 via ge-0/0/0.0
172.31.0.0/16 *[Static/5] 3d 20:31:42
to table Lab.inet.0
192.168.100.13/32 *[OSPF/10] 3d 20:27:53, metric 2
> to 192.168.224.1 via ge-0/0/0.0
192.168.100.19/32 *[OSPF/10] 3d 20:27:53, metric 2
> to 192.168.224.6 via ge-0/0/0.0
192.168.224.0/24 *[Direct/0] 3d 20:28:30
> via ge-0/0/0.0
192.168.224.3/32 *[Local/0] 3d 20:28:34
Local via ge-0/0/0.0
192.168.225.0/24 *[OSPF/10] 3d 20:27:53, metric 2
> to 192.168.224.6 via ge-0/0/0.0
224.0.0.5/32 *[OSPF/10] 3d 20:31:43, metric 1
MultiRecv

Lab.inet.0: 4 destinations, 5 routes (4 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

  ƒ‰‡Ͷ‘ˆͷ͸

 App Note

172.19.254.2/32 *[OSPF/10] 3d 19:37:56, metric 1

> via st0.0
172.31.254.0/24 *[Direct/0] 3d 20:27:56
> via st0.0
[OSPF/10] 3d 20:27:56, metric 1
> via st0.0
172.31.254.1/32 *[Local/0] 3d 20:31:42
Local via st0.0
224.0.0.5/32 *[OSPF/10] 3d 20:31:43, metric 1
MultiRecv

Checking Firewall Sessions

show security flow session

Viewing the contents of the session table can be a very useful tool to determine the current state of connections. Through
the CLI we can filter on sessions in a combination of different arguments so that we can view the exact information in the
session table. There are a number of different arguments, such as defining the source prefix, port, destination
prefix/port/application, and others. You can also use the summary option to see the summary view of the session table
root@SRX210> show security flow session ?
Possible completions:
<[Enter]> Execute this command
application Show session for specified application or application set
destination-port Show each session that uses specified destination port
destination-prefix Show each session that matches destination prefix
idp Show IDP sessions
interface Show each session that uses specified interface
protocol Show each session that uses specified IP protocol
resource-manager Show resource-manager sessions
session-identifier Show session with specified session identifier
source-port Show each session that uses specified source port
source-prefix Show each session that matches source prefix
summary Show summary of sessions
tunnel Show tunnel sessions
Here we show the individual components of a session. We can see the individual session ID, the policy and rule that it is
applied to (in this case it is SSH traffic terminated on the SRX) we also count the timeout in seconds, which is different
then ScreenOS which we count the timeout in Ticks, with 1 tick being ten seconds. So here we see the timeout is 1800
seconds, or 30 minutes (the standard TCP timeout.) Next, we see the individual components of the connection, the
source address/port, destination address port, protocol both for the inbound traffic, as well as the outbound traffic. Of

  ƒ‰‡ͷ‘ˆͷ͸

 App Note

course, this may differ if there is NAT applied to it when discussing transit sessions. We can also see what interface the
traffic arrives and leaves on.
Session ID: 13284, Policy name: self-traffic-policy/1, Timeout: 1800
In: 192.168.225.100/2192 --> 192.168.224.3/22;tcp, If: ge-0/0/0.0
Out: 192.168.224.3/22 --> 192.168.225.100/2192;tcp, If: .local..0

Checking System Hardware / System State

show chassis hardware details

When troubleshooting a device remotely, you may want to check what components are in the device, along with serial
numbers for support purposes:
root@SRX3400-1> show chassis hardware detail
Hardware inventory:
Item Version Part number Serial number Description
Chassis AA4508AD0012 SRX 3400
Midplane REV 04 710-015748 TV3775 SRX 3400 Midplane
PEM 1 rev 06 740-020226 E182C9000A06P AC Power Supply
CB 0 REV 06 750-021914 TV3844 SRX3k RE-12-10
Routing Engine BUILTIN BUILTIN Routing Engine
ad0 977 MB SanDisk SDCFJ-1024 010607C0108V0536 Compact Flash
ad2 38154 MB ST940817SM 5RQ03MY6 Hard Disk
CPP BUILTIN BUILTIN Central PFE Processor
Mezz REV 05 710-021035 TV3805 SRX3k HD Mezzanine Card
FPC 0 REV 02 750-021882 TV3888 SRX3k SFB 12GE
PIC 0 BUILTIN BUILTIN 8x 1GE-TX 4x 1GE-SFP
FPC 1 750-020324 TV3685 SRX3k 16xGE TX
PIC 0 BUILTIN BUILTIN 16x 1GE-TX
FPC 3 750-020324 TV3674 SRX3k 16xGE TX
PIC 0 BUILTIN BUILTIN 16x 1GE-TX
FPC 5 REV 05 750-016077 TV3938 SRX3k SPC
PIC 0 BUILTIN BUILTIN SPU Cp-Flow
FPC 6 REV 04 750-017866 TV3971 SRX3k NPC
PIC 0 BUILTIN BUILTIN NPC PIC
Fan Tray 0 REV 04 750-021896 TV3818 SRX 3400 Fan Tray

show chassis fpc pic-status

  ƒ‰‡͸‘ˆͷ͸

 App Note

In the SRX HE platforms you may want to check whether a PIC is online or not, particularly when it is booting up, since
even after you can log into the system, a individual PIC may not yet be operational for a few minutes. This can easily be
done by issuing the “show chassis fpc pic-status” command. We can even see the type of PIC including which one is the
CP and what mode the SPU is in (CP, Flow, or Combo) along with which slot it is installed in.
root@SRX5800-1> show chassis fpc pic-status
node0:
--------------------------------------------------------------------------
Slot 0 Online SRX5k SPC
PIC 0 Online SPU Cp
PIC 1 Online SPU Flow
Slot 1 Online SRX5k SPC
PIC 0 Online SPU Flow
PIC 1 Online SPU Flow
Slot 11 Online SRX5k DPC 40x 1GE
PIC 0 Online 10x 1GE RichQ
PIC 1 Online 10x 1GE RichQ
PIC 2 Online 10x 1GE RichQ
PIC 3 Online 10x 1GE RichQ

node1:
--------------------------------------------------------------------------
Slot 0 Online SRX5k SPC
PIC 0 Online SPU Cp
PIC 1 Online SPU Flow
Slot 1 Online SRX5k SPC
PIC 0 Online SPU Flow
PIC 1 Online SPU Flow
Slot 11 Online SRX5k DPC 40x 1GE
PIC 0 Online 10x 1GE RichQ
PIC 1 Online 10x 1GE RichQ
PIC 2 Online 10x 1GE RichQ
PIC 3 Online 10x 1GE RichQ

show security monitoring fpc <number> [node <node>]

In the SRX 3k and 5k systems we have the ability to look at the resource utilization for the individual SPU’s on an SPC
(remember there is only 1 per SPC on the 3k, and 2 per SPC on the 5k.) Today we can only view the status of individual
FPC’s, but there is a JUNOScript that will show all FPC’s in the platform which is available here:
https://confluence.jnpr.net/confluence/display/sltfwidp/SRX+Monitor

  ƒ‰‡͹‘ˆͷ͸

 App Note

{primary:node0}
root@SRX3400-1> show security monitoring fpc 2
node0:
--------------------------------------------------------------------------
FPC 2
PIC 0
CPU utilization : 0 %
Memory utilization : 51 %
Current flow session : 0
Max flow session : 524288
Current CP session : 1
Max CP session : 1048576

node1:
--------------------------------------------------------------------------
FPC 2
PIC 0
CPU utilization : 0 %
Memory utilization : 51 %
Current flow session : 0
Max flow session : 524288
Current CP session : 0
Max CP session : 1048576

show chassis routing-engine

When you want to check the utilization on the control plane, this is the command to use. One thing to point out, on the
SRX, the memory utilization is a bit misleading, since we preallocate most of the memory on the platform, even on the
control plane. It is best to look at individual issues with memory utilization to see if the feature is actually over utilized or if
the system is just counting the allocated memory as actually contributing to the memory utilization. Remember that the
control plane is not involved with packet forwarding, and therefore will not directly impact the performance seen in data
forwarding (even with CPU percentages.)
root@SRX210> show chassis routing-engine
Routing Engine status:
Temperature 43 degrees C / 109 degrees F
DRAM 1024 MB
Memory utilization 90 percent
CPU utilization:

  ƒ‰‡ͺ‘ˆͷ͸

 App Note

User 2 percent
Background 0 percent
Kernel 3 percent
Interrupt 0 percent
Idle 95 percent
Model RE-SRX210-HIGHMEM
Serial ID AAAH2299
Start time 2009-07-30 17:37:21 EDT
Uptime 3 days, 23 hours, 56 minutes, 49 seconds
Last reboot reason 0x1:power cycle/failure
Load averages: 1 minute 5 minute 15 minute
0.04 0.01 0.00

show chassis alarms

When there is a major issue affecting the platform itself (such as a linecard or power supply failure) we can check in 1
place to find out what that issue is. We will typically see the Alarm LED on the craft interface, so we can login to collect
more information. In this case there are no alarms active. If there are alarms, we will want to dive more into the
respective log files for detail.

root@SRX210> show chassis alarms

No alarms currently active
show chassis environment
The temperature and fan speeds can be checked with this command. Should there be an alarm related to power,
temperature, or fan speed, you will want to take a look at the output here:

root@SRX3400-1> show chassis environment

Class Item Status Measurement
Temp PEM 0 Absent
PEM 1 OK
Routing Engine 0 OK
Routing Engine 1 Absent
CB 0 Intake OK 27 degrees C / 80 degrees F
CB 0 Exhaust A OK 31 degrees C / 87 degrees F
CB 0 Mezz OK 30 degrees C / 86 degrees F
CB 1 Intake Absent
CB 1 Exhaust A Absent
CB 1 Mezz Absent
FPC 0 Intake OK 32 degrees C / 89 degrees F

  ƒ‰‡ͻ‘ˆͷ͸

 App Note

FPC 0 Exhaust A OK 30 degrees C / 86 degrees F

FPC 0 Exhaust B OK 35 degrees C / 95 degrees F
FPC 0 SF OK 35 degrees C / 95 degrees F
FPC 1 Intake OK 28 degrees C / 82 degrees F
FPC 1 Exhaust A OK 33 degrees C / 91 degrees F
FPC 3 Intake OK 30 degrees C / 86 degrees F
FPC 3 Exhaust A OK 37 degrees C / 98 degrees F
FPC 5 Intake OK 32 degrees C / 89 degrees F
FPC 5 Exhaust A OK 34 degrees C / 93 degrees F
FPC 5 XLR OK 41 degrees C / 105 degrees F
FPC 6 Intake OK 28 degrees C / 82 degrees F
FPC 6 Exhaust A OK 32 degrees C / 89 degrees F
Fans Fan 1 OK Spinning at high speed
Fan 2 OK Spinning at high speed
Fan 3 OK Spinning at high speed
Fan 4 OK Spinning at high speed
show chassis craft-interface
Even if you are not in front of the SRX, you can still check the values of the LED’s (although there are numerous other
methods to get more detail within the SRX itself.)
root@SRX3400-1> show chassis craft-interface

Front Panel System LEDs:

Routing Engine 0 1
--------------------------
OK * .
Fail . .
Master * .

Front Panel Alarm Indicators:

-----------------------------
Red LED .
Yellow LED .

Front Panel FPC LEDs:

FPC 0 1 2 3 4 5 6 7
------------------------------------
Red . . . . . . . .

  ƒ‰‡ͳͲ‘ˆͷ͸

 App Note

Green * * . * . * * .

CB LEDs:
CB 0 1
-------------
OK * .
Fail . .
Master * .

PS LEDs:
PS 0 1
------------
Red . .
Green . *

Fan Tray LEDs:

FT 0
--------
Red .
Green *

HA LED:
HA 0
--------
Red .
Green .
Amber .

SFB LED:
SFB 0
--------
Red .
Green *
Amber .
Blink .

CFM Ok/Fail LED:

  ƒ‰‡ͳͳ‘ˆͷ͸

 App Note

--------
Red .
Green *
Amber .

CFM Service LED:

--------
Red .
Green *
Amber .

Checking System Log Messages

show log <logfile>

The SRX platform has a very rich and granular logging architecture, where we log information in great detail on the
JUNOS platform. Sometimes you will need to consult the log files based on troubleshooting. There are a number of
different log files, which sit in the /var/log directory. Some log files will have multiple archives, which depends on the
archive settings for the logs, once they get too big (configurable,) they are rolled over into another log, with a certain
number (configurable) of archives before they are deleted.

root@SRX210> start shell

root@SRX210% cd /var/log
root@SRX210% ls
.snap httpd.log.old nsd_chk_only
__jsrpd_commit_check__ idpd nstraced_chk_only
authd_sdb.log idpd_err.20090730 pf
autod idpinfo_err.20090728 pfed_trace.log
chassisd install pgmd
cosd install.0.gz rtlogd
dcd install.1.gz sampled
dfwc install.2.gz sdxd
dfwd install.3.gz snapshot
eccd interactive-commands stl_trace_508.log
ext inventory stl_trace_510.log
flowc jsrpd stl_trace_511.log
ggsn kmd stl_trace_513.log
gres-tp license stl_trace_976.log
hostname-cached mastership stl_trace_985.log
httpd.log messages utmd-av

  ƒ‰‡ͳʹ‘ˆͷ͸

 App Note

root@SRX210% pwd
/cf/var/log
The most command logs to check are the following:
• chassisd: Events relating to hardware, chassis control
• idpd: Events relating to the IDP daemon, events and failures
• interactive-commands: By default all commands entered on the platform will be stored in this log, so it is very
useful for checking other’s work. This includes operational commands that are not entered in the configuration.
• jsrpd: Events relating to High Availability
• kmd: Events relating to IKE negotiation
• messages: This would be the first place to look for device events, it is the general log message file, and often has
lots of useful information. While you can often find more detail for other individual specialized files, this is a great
starting point.
• utmd: For platforms that support UTM features, detailed event logs are stored here

When you are looking through these event log files, it is usually very helpful to use the powerful output filtering tools that
JUNOS has to offer. Specifically, you can use “show log <log file> | match <expression>”, show log <log file> | find
<expression>”, show log <log file> | last <show last X number of lines>” , and “show log <log file> | trim <show log after X
number of lines>”.

Checking VPN Status

show security ike security-associations <peer-address> detail

You can quickly check the status of a Phase 1 security association with an IKE peer with this command. If you omit the
peer address, you will see the status of all VPN tunnels, while if you omit the detail, then you will only see a single line
defining the state and Phase 1 mode (Main or Aggressive). In this command output we see the properties of the tunnel,
the time remaining, and even the bytes and packets sent and received.
root@SRX210> show security ike security-associations 192.168.224.1 detail
IKE peer 192.168.224.1, Index 115,
Role: Initiator, State: UP
Initiator cookie: c346b9206016bff1, Responder cookie: 57aa732dc1ce2b27
Exchange type: Main, Authentication method: Pre-shared-keys
Local: 192.168.224.3:500, Remote: 192.168.224.1:500
Lifetime: Expires in 3267 seconds
Algorithms:
Authentication : sha1
Encryption : aes-cbc (256 bits)
Pseudo random function: hmac-sha1
Traffic statistics:

  ƒ‰‡ͳ͵‘ˆͷ͸

 App Note

Input bytes : 812

Output bytes : 992
Input packets: 4
Output packets: 5
Flags: Caller notification sent
IPSec security associations: 1 created, 0 deleted
Phase 2 negotiations in progress: 0

show security ipsec security-associations / show security ipsec security-associations index <ID>
You can quickly check to see if Phase 2 (which also implies Phase 1) has established by issuing the “show security ipsec
security-associations” command. This will give you a high level overview of the connected gateways, protocols, and
lifetimes of the established Security Associations. You can gather more in depth information by taking the Index ID, and
issuing the second command, which will show more information about that individual SA. In this example, we can see
that the SRX has renegotiated another pair of Security Associations, since the top two are so close to expiring. This is
done to limit disruption of service if you have the “establish-tunnels-immediately” option enabled.
root@SRX210> show security ipsec security-associations
Total active tunnels: 1
ID Gateway Port Algorithm SPI Life:sec/kb Mon vsys
<131073 192.168.224.1 500 ESP:aes-256/sha1 d6393645 26/ unlim - 0
>131073 192.168.224.1 500 ESP:aes-256/sha1 153ec235 26/ unlim - 0
<131073 192.168.224.1 500 ESP:aes-256/sha1 f9a2db9a 3011/ unlim - 0
>131073 192.168.224.1 500 ESP:aes-256/sha1 153ec236 3011/ unlim - 0

Here we see the detailed output of each security association

root@SRX210> show security ipsec security-associations detail index 131073

Virtual-system: Root
Local Gateway: 192.168.224.3, Remote Gateway: 192.168.224.1
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=192.168.0.0/16)
DF-bit: clear
Direction: inbound, SPI: d6393645, AUX-SPI: 0
Hard lifetime: Expires in 4 seconds
Lifesize Remaining: Unlimited
Soft lifetime: Expired
Mode: tunnel, Type: dynamic, State: installed, VPN Monitoring: -
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: enabled, Replay window size: 64

  ƒ‰‡ͳͶ‘ˆͷ͸

 App Note

Direction: outbound, SPI: 153ec235, AUX-SPI: 0

Hard lifetime: Expires in 4 seconds
Lifesize Remaining: Unlimited
Soft lifetime: Expired
Mode: tunnel, Type: dynamic, State: installed, VPN Monitoring: -
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: enabled, Replay window size: 64

Direction: inbound, SPI: f9a2db9a, AUX-SPI: 0

Hard lifetime: Expires in 2989 seconds
Lifesize Remaining: Unlimited
Soft lifetime: Expires in 2359 seconds
Mode: tunnel, Type: dynamic, State: installed, VPN Monitoring: -
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: enabled, Replay window size: 64

Direction: outbound, SPI: 153ec236, AUX-SPI: 0

Hard lifetime: Expires in 2989 seconds
Lifesize Remaining: Unlimited
Soft lifetime: Expires in 2359 seconds
Mode: tunnel, Type: dynamic, State: installed, VPN Monitoring: -
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: enabled, Replay window size: 64

Checking IDP and Screen Status

show security screen statistics interface <interface>

You can check the screen tables to determine if you are seeing an increase in a particular screen attack on the system.
This can be done with either the interface option as shown, or using the zone <zone> option instead of interface
<interface>
root@SRX210> show security screen statistics interface ge-0/0/0
Screen statistics:

IDS attack type Statistics

ICMP flood 5
UDP flood 26

  ƒ‰‡ͳͷ‘ˆͷ͸

 App Note

TCP winnuke 0
TCP port scan 12
ICMP address sweep 15
IP tear drop 0
TCP SYN flood 0
IP spoofing 0
ICMP ping of death 0
IP source route option 0
TCP land attack 3
TCP SYN fragment 0
TCP no flag 0
IP unknown protocol 156
IP bad options 0
IP record route option 0
IP timestamp option 0
IP security option 0
IP loose source route option 0
IP strict source route option 0
IP stream option 0
ICMP fragment 0
ICMP large packet 210
TCP SYN FIN 0
TCP FIN no ACK 0
Source session limit 0
TCP SYN-ACK-ACK proxy 0
IP block fragment 0
Destination session limit 0

show security idp memory

When using the IDP engine, it can be helpful to check the memory usage on the system. This is especially true when
using very large IDP policy sets. We can see a breakdown of the current memory utilization here:
root@SRX210> show security idp memory

IDP data plane memory statistics:

Total IDP data plane memory : 188 MB

  ƒ‰‡ͳ͸‘ˆͷ͸

 App Note

Used : 3 MB ( 3072 KB )
Available : 185 MB ( 189440 KB )

show security idp status

To get an up to date listing of the flows that are being processed in the system, you can use the “show security idp status”
command, which is similar to sctop subscriber info in the standalone IDP. Since we refer flows to the IDP engine in the
firewall policy, the number of active flows you see here may be quite different then what you see in the “show security flow
session” table, however, you can use the “show security flow session idp <options>” command to view the sessions that
are being inspected by IDP.
root@SRX210> show security idp status
Status of IDP: s0, Up since: 2009-07-30 17:41:54 EDT (3d 23:13 ago)

Packets/second: 0 Peak: 0 @ 2009-07-30 17:41:54 EDT

KBits/second : 0 Peak: 0 @ 2009-07-30 17:41:54 EDT
Latency (microseconds): [min: 0] [max: 0] [avg: 0]

Packet Statistics:
[ICMP: 0] [TCP: 0] [UDP: 0] [Other: 0]

Flow Statistics:
ICMP: [Current: 0] [Max: 0 @ 2009-07-30 17:41:54 EDT]
TCP: [Current: 0] [Max: 0 @ 2009-07-30 17:41:54 EDT]
UDP: [Current: 0] [Max: 0 @ 2009-07-30 17:41:54 EDT]
Other: [Current: 0] [Max: 0 @ 2009-07-30 17:41:54 EDT]

Session Statistics:
[ICMP: 0] [TCP: 0] [UDP: 0] [Other: 0]
Policy Name : Basic

How to Mount a USB Key

Attaching a USB key can be useful to transfer files for troubleshooting or upgrading code when direct network access is
not available, but physical access is. Plug the USB key into the system. You should see some info appear on the CLI
about discovering the USB key. This does not automatically mount the USB, just recognizes the device. Perform the
following steps:

1. Log into the JUNOS OS Shell with the “start shell” command
2. At the “%” prompt, create a new directory with the command “mkdir /var/tmp/usb” this will create a directory
called “usb” in “/var/tmp”. Technically you can call it something else, or even put it in another location if desired.
3. Next mount the USB key with the command “mount_msdosfs /dev/da1 /var/tmp/usb”

  ƒ‰‡ͳ͹‘ˆͷ͸

 App Note

4. Now you can perform the operations desired on the USB

5. When done exit out of the /var/tmp/usb directory (e.g. “cd /”) and issue the “umount /dev/da1” command to
umount the USB
6. Exit the CLI with the “exit” command.

Basic JUNOS Debugging

While the show commands themselves offer some great insight into the operating status of the SRX, there is
another level of detail that we can gather from the SRX using the JUNOS debugging features. In JUNOS we
call the debugging features Traceoptions, and the setup is slightly different in JUNOS than in ScreenOS.

In ScreenOS what we know as debugging, is done by issuing the respective debug command, such as “debug
flow basic” some commands such as debug flow basic and debug ike basic allow us to define filters, to only
capture information on certain flows (which is much more powerful than having to look at all flows, which can
have a performance impact on the platform. The information that is logged from the ScreenOS debugs is
captured to a debug buffer, which is stored in memory. It can be limited to 4MB of space before it starts to
rewrite the logs. Also, debugs are typically disabled when the user logs out of the CLI.

In JUNOS, the concept of setting Traceoptions requires setting the actual trace in the configuration itself, rather
than as an operational mode command. When you set the trace in the configuration, you need to define the
“flags” for the actual debug that you want to perform (there are usually different options for each type
depending on what type of information you are looking for) as well as define the file that the trace information
should be written to. This is a very powerful feature because it gives you a lot more flexibility and more storage
space to capture the output. This also means that there are a few extra steps compared to ScreenOS, but it is
quite simple to setup, and remove, and you can also deactivate the trace, so that you can simply enable or
disable it in the future. With the power of JUNOScript, you could even write an event script that could trigger a
debug or collect information when certain events occur. Later in this document, we’ll discuss the power of the
Advanced Insight Solutions (AIS) component which leverages this power.

Configuring a Trace
Here we will configure a basic trace to monitor interfaces going up and down in the platform.

1. Log into the device and enter the configuration mode.

2. Configure the flag: set interfaces traceoptions flag config-states
a. This sets the flag to monitor any events when an interface changes state
3. Configure the file to log to: set interfaces traceoptions file Interface.txt size 1M files 5
a. This sets the file that we will log to, and optionally defines that the max file size should be 1M,
and we can keep up to 5 instances of the file when a file goes beyond 1M, it will roll that into a
new file. The active file will always be Interface.txt. We can also apply a few other commands,
such as the world-readable option to format the file in a world-readable ASCII output, as well as
defining the “match” option to only log certain messages into this file. Note that you do not use

  ƒ‰‡ͳͺ‘ˆͷ͸

 App Note

the pipe with this option, however after match, you define the regular expression that you would
like to match on.
4. Issue the commit to apply the configuration exit configuration and exit the configuration mode: commit
and-quit
5. Now you can take a look at the log file. You can do this by either using the “show log Interfaces.txt”
command (even using additional options to match, trim, find, and others by using the pipe output) OR
we can also view the file in realtime, using the command “monitor start Interfaces.txt” which is the
equivalent of the tail –f Unix/Linux command. If you would like to stop the realtime output just type
“monitor stop” even if you do not have an actual prompt (because the text is scrolling) the command
will be accepted (assuming no typos.)
6. Lastly, you can stop the actual trace in a few different ways. If you have not configured any other
changes in the meantime, you can enter the configuration mode and just issue the “rollback 1” and then
the “commit” command. You can also either use the “delete interfaces traceoptions” command to
delete the trace, or you can just deactivate the command with the “deactivate interfaces traceoptions”
command (both delete and deactivate will need to be followed by a “commit” to actually apply the
change. Using the deactivate is a great option if you may need to activate the configuration in the future.
7. You can remote the log with the operational mode command “file delete Interfaces.txt” or you can
clear the file with the command “clear log Interfaces.txt”.

root@SRX210# set interfaces traceoptions flag config-states

root@SRX210# set interfaces traceoptions file Interface.txt size 1M files 5
root@SRX210# commit and-quit
root@SRX210> monitor start Interface.txt

Aug 3 14:58:51 SRX210mib2d[759]: SNMP_TRAP_LINK_DOWN: ifIndex 227, ifAdminStatus up(1),

ifOperStatus down(2), ifName fe-0/0/7
Aug 3 14:58:51 SRX210 mib2d[759]: SNMP_TRAP_LINK_UP: ifIndex 226, ifAdminStatus up(1),
ifOperStatus up(1), ifName fe-0/0/7

root@SRX210> monitor start Interface.txt

root@SRX210> edit
Entering configuration mode

[edit]
root@SRX210# delete interfaces traceoptions

[edit]
root@SRX210# commit and-quit

  ƒ‰‡ͳͻ‘ˆͷ͸

 App Note

*** Interface.txt ***

Aug 4 13:09:13 Received SIGHUP, time to reparse.
Aug 4 13:09:13 Pending config request now being serviced
commit complete
Exiting configuration mode

root@SRX210> monitor stop

## You can also issue the ESC-Q sequence to pause a monitoring session
root@SRX210> clear log Interface.txt

Making Output More Readable (match, last, and trim)

Sometimes when you have a lot of information that you need to dig through simply displaying the output can be
burdensome. Luckily JUNOS has an excellent set of output modifiers that can make this task much easier. We
are only going to focus on three in this section, but you should experiment with the different output modifiers to
gain more experience working with them.

To trigger an output modifier with any show command (operational or configuration mode) you simply issue
the “|” character after the command. You can then issue the “?” for a list of the different output modifiers.
Note that you can actually pipe out multiple modifiers (for instance, you can view the last 50 lines of text with
the “| last 50” modifier followed by the “| match flow” modifier to match any line for display with the word
flow in it. In the next three examples, we will use the output of the “show log <log>” command to show how
you can view this information. As mentioned, we can view something other than a log file (e.g. show route |
match 1.1.1.0 for example, anything that displays content.) In this case, we are looking at the output of flow
debugging, explained later in the document. You can ignore the specific content for the moment, but it is
helpful to see the output and how to use the modifiers.

root@SRX210> show log DebugTraffic

Aug 14 19:08:56 SRX210 clear-log[20177]: logfile cleared
Aug 14 19:09:13 19:07:34.1473792:CID-0:RT:traceflag 0x0

Aug 14 19:09:13 19:09:12.808200:CID-0:CTRL:flow0: Rate limit changed to 0

Aug 14 19:09:13 19:09:12.808200:CID-0:CTRL:flow0: Destination ID set to 2
Aug 14 19:09:13 19:09:12.808200:CID-0:RT:filter 0 name MatchInbound is set

Aug 14 19:09:13 19:09:12.808200:CID-0:RT:filter 1 name MatchOutbound is set

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:<1.1.1.100/61232->192.168.222.50/53;17> matched

filter MatchTraffic:

  ƒ‰‡ʹͲ‘ˆͷ͸

 App Note

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:packet [50] ipid = 29214, @422eb81e

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type

13, common flag 0x0, mbuf 0x422eb680

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: flow process pak fast ifl 69 in_ifp fe-0/0/7.0

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: find flow: table 0x4dd31658, hash

14530(0xffff), sa 1.1.1.100, da 192.168.222.50, sp 61232, dp 53, proto 17, tok 448

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: flow_first_create_session

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: flow_first_in_dst_nat: in <fe-0/0/7.0>, out

<N/A> dst_adr 192.168.222.50, sp 61232, dp 53

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: chose interface fe-0/0/7.0 as incoming nat if.

Match: The match command will allow us to only display the specific information that we would like to see.
In this case we may only want to see any line with the word route in it. To do so we would use the command
“show log <logfile> | match route” Note that the match statement uses regular expressions so it is looking for
any line with the word route in it. It would match a line with route, routes, router &c. If we only want to match
route, then we would say {show log <logfile> | match “ route “} with the regular expression in the quotes and a
space preceeding and following the route keyword. Viewing the output of “show log <logfile> | match route”
displays this example:

root@SRX210> show log DebugTraffic | match route

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:flow_first_routing: call flow_route_lookup():
src_ip 1.1.1.100, x_dst_ip 192.168.222.50, in ifp fe-0/0/7.0, out ifp N/A sp 61232, dp 53,
ip_proto 17, tos 0

Last: The last command will only show the last “x” number of lines in a file. This is very useful to see the
latest logs in the log file. For instance, if we issue the command “show log messages | last 20” we will get the
following output:

root@SRX210> show log messages | last 20

Aug 28 13:59:30 SRX210 sshd[31551]: Accepted password for root from 192.168.225.100 port
4344 ssh2
Aug 28 19:57:18 SRX210 sshd[31993]: Accepted password for root from 192.168.225.100 port
1358 ssh2
Aug 28 20:05:17 SRX210 sshd[32248]: Accepted password for root from 192.168.225.100 port
1397 ssh2
Aug 28 20:05:17 SRX210 sshd[32248]: subsystem request for sftp

  ƒ‰‡ʹͳ‘ˆͷ͸

 App Note

Aug 28 20:21:24 SRX210 init: network-security-trace (PID 29949) terminate signal sent
Aug 28 20:21:24 SRX210 init: can not access /usr/sbin/ipmid: No such file or directory
Aug 28 20:21:24 SRX210 init: ipmi (PID 0) started
Aug 28 20:21:24 SRX210 init: network-security-trace (PID 29949) exited with status=0
Normal Exit
Aug 28 20:27:26 SRX210 init: network-security-trace (PID 33234) started
Aug 28 20:27:26 SRX210 init: can not access /usr/sbin/ipmid: No such file or directory
Aug 28 20:27:26 SRX210 init: ipmi (PID 0) started
Aug 31 13:51:11 SRX210 sshd[34799]: Accepted password for root from 192.168.225.100 port
3197 ssh2
Aug 31 16:08:41 SRX210 sshd[34847]: Accepted password for root from 192.168.225.100 port
1273 ssh2
Sep 1 07:39:32 SRX210 /kernel: twsi0: Device timeout on unit 0 (chassisd, Read, 0x48)
Sep 2 14:05:45 SRX210 sshd[35472]: Accepted password for root from 192.168.225.100 port
3652 ssh2
Sep 2 17:46:23 SRX210 utmd[35538]: AV_PATTERN_KL_CHECK_FAILED: AntiVirus: db file
signature mismatch:
/var/db/kav/kav_db/fa001.avcMFhMU3pucGRJNzFmQjMwMGU3VXdqMVdRWEtjbEVKdUVUUTh2T0Z6cDlQYnluRz
VtdW5Pcmx1QTVBeQ==#284825.
Sep 2 17:46:27 SRX210 utmd[35538]: AV_PATTERN_KL_CHECK_FAILED: AntiVirus: db file
signature mismatch:
/var/db/kav/kav_db/fa001.avcMFhMU3pucGRJNzFmQjMwMGU3VXdqMVdRWEtjbEVKdUVUUTh2T0Z6cDlQYnluRz
VtdW5Pcmx1QTVBeQ==#284825.
Sep 2 17:46:50 SRX210 last message repeated 4 times
Sep 2 17:47:04 SRX210 utmd[35538]: AV_PATTERN_KL_CHECK_FAILED: AntiVirus: db file
signature mismatch:
/var/db/kav/kav_db/dailyc.avcMFhMU3pucGRJNzFmQjMwMGU3VXdqMUpFN0NyNy81UzRNV25XeEZnWDdhZm03S
WFtM3dvYzFkYlc3YQ==#23026.
Sep 2 17:47:17 SRX210 last message repeated 5 times
Sep 3 14:38:56 SRX210 sshd[35914]: Accepted password for root from 192.168.225.100 port
3090 ssh2
Sep 3 19:44:07 SRX210 sshd[35959]: Accepted password for root from 192.168.225.100 port
3865 ssh2

Trim: Some information that is added into the prefix of the line can make it difficult to read. The trim
command will omit the first “x” number of characters in the line. So for instance, if the output is:

root@SRX210> show log DebugTraffic

Aug 14 19:08:56 SRX210 clear-log[20177]: logfile cleared
Aug 14 19:09:13 19:07:34.1473792:CID-0:RT:traceflag 0x0

Aug 14 19:09:13 19:09:12.808200:CID-0:CTRL:flow0: Rate limit changed to 0

Aug 14 19:09:13 19:09:12.808200:CID-0:CTRL:flow0: Destination ID set to 2
Aug 14 19:09:13 19:09:12.808200:CID-0:RT:filter 0 name MatchInbound is set

  ƒ‰‡ʹʹ‘ˆͷ͸

 App Note

Aug 14 19:09:13 19:09:12.808200:CID-0:RT:filter 1 name MatchOutbound is set

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:<1.1.1.100/61232->192.168.222.50/53;17> matched

filter MatchTraffic:

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:packet [50] ipid = 29214, @422eb81e

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type

13, common flag 0x0, mbuf 0x422eb680

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: flow process pak fast ifl 69 in_ifp fe-0/0/7.0

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: find flow: table 0x4dd31658, hash

14530(0xffff), sa 1.1.1.100, da 192.168.222.50, sp 61232, dp 53, proto 17, tok 448

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: flow_first_create_session

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: flow_first_in_dst_nat: in <fe-0/0/7.0>, out

<N/A> dst_adr 192.168.222.50, sp 61232, dp 53

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: chose interface fe-0/0/7.0 as incoming nat if.

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:flow_first_rule_dst_xlate: DST no-xlate:

0.0.0.0(0) to 192.168.222.50(53)

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:flow_first_routing: call flow_route_lookup():

src_ip 1.1.1.100, x_dst_ip 192.168.222.50, in ifp fe-0/0/7.0, out ifp N/A sp 61232, dp 53,
ip_proto 17, tos 0

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT:Doing DESTINATION addr route-lookup

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: routed (x_dst_ip 192.168.222.50) from untrust

(fe-0/0/7.0 in 0) to ge-0/0/0.0, Next-hop: 192.168.224.1

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: policy search from zone untrust-> zone trust

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: app 16, timeout 60s, curr ageout 60s

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: packet dropped, denied by policy

  ƒ‰‡ʹ͵‘ˆͷ͸

 App Note

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: packet dropped, policy deny.

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: flow didn't create session, code=-1.

Aug 14 19:09:13 19:07:34.1234876:CID-0:RT: ----- flow_process_pkt rc 0x7 (fp rc -1)

Then if we use the “| trim 42” option, we can omit all of the timestamp and redundant info in each entry,
making it much more readable:

root@SRX210> show log DebugTraffic | trim 42

ogfile cleared
traceflag 0x0

:flow0: Rate limit changed to 0

:flow0: Destination ID set to 2
ilter 0 name MatchInbound is set

ilter 1 name MatchOutbound is set

<1.1.1.100/61232->192.168.222.50/53;17> matched filter MatchTraffic:

packet [50] ipid = 29214, @422eb81e

---- flow_process_pkt: (thd 1): flow_ctxt type 13, common flag 0x0, mbuf 0x422eb680

flow process pak fast ifl 69 in_ifp fe-0/0/7.0

find flow: table 0x4dd31658, hash 14530(0xffff), sa 1.1.1.100, da 192.168.222.50, sp 61232, dp 53, proto 17,
tok 448

flow_first_create_session

flow_first_in_dst_nat: in <fe-0/0/7.0>, out <N/A> dst_adr 192.168.222.50, sp 61232, dp 53

chose interface fe-0/0/7.0 as incoming nat if.

flow_first_rule_dst_xlate: DST no-xlate: 0.0.0.0(0) to 192.168.222.50(53)

flow_first_routing: call flow_route_lookup(): src_ip 1.1.1.100, x_dst_ip 192.168.222.50, in ifp fe-0/0/7.0, out
ifp N/A sp 61232, dp 53, ip_proto 17, tos 0

  ƒ‰‡ʹͶ‘ˆͷ͸

 App Note

Doing DESTINATION addr route-lookup

routed (x_dst_ip 192.168.222.50) from untrust (fe-0/0/7.0 in 0) to ge-0/0/0.0, Next-hop: 192.168.224.1

policy search from zone untrust-> zone trust

app 16, timeout 60s, curr ageout 60s

packet dropped, denied by policy

packet dropped, policy deny.

flow didn't create session, code=-1.

----- flow_process_pkt rc 0x7 (fp rc -1)

Troubleshooting Traffic Flows

One of the most important troubleshooting features is the ability to troubleshoot the decisions that the SRX
platform makes on the traffic itself. Often just looking at the firewall logs will provide enough detail to
understand what the firewall is doing to a packet (permit, deny, the firewall policy applied, NAT, VPN &c.) but
sometimes we do not have enough information for why the firewall is making a decision on a packet itself, and
that’s when we may need to debug a packet flow. In ScreenOS, our primary feature that allowed us to
accomplish this was to use the “debug flow basic” command to record every decision that the firewall makes on
a packet. In the SRX, our primary method of capturing this information is called “set security flow
traceoptions basic-datapath” while we also have the ability to filter only certain packets for advanced
debugging using the “set security flow traceoptions packet-filter”

In the SRX, we can only set 1 expression per packet filter, so if you want to capture more than 1 flow direction,
then you would need to define multiple packet filters per capture (one for each direction). You will always
want to make sure that you enable packet-filters when tracing traffic, not only because there is often too much
information in the file if you don’t, but also because the box will take a performance hit if you are trying to
trace ALL traffic going through it. If you set the appropriate filter and you are not passing lots of data through
the filter, then it shouldn’t have any noticeable impact.

In this example, we have a machine 1.1.1.100 initiating a Microsoft RDP connection (TCP port 3389) to a
server with the IP address 1.1.1.30 (which is NAT’d to the real server address 192.168.224.30.) 192.168.224.30
does not actually have knowledge of the 1.1.1.0/24 network, so we need to account for that in our Packet Filters.

Interface fe-0/0/7.0: IP Address 1.1.1.50/24 (Untrust)

Interface ge-0/0/0.0: IP Address 192.168.224.3/24 (Trust)

  ƒ‰‡ʹͷ‘ˆͷ͸

 App Note

Our trace configuration is:

set security flow traceoptions file DebugTraffic

set security flow traceoptions flag basic-datapath
set security flow traceoptions packet-filter MatchTraffic source-prefix 1.1.1.0/24
destination-prefix 1.1.1.30/32
set security flow traceoptions packet-filter MatchTrafficReverse source-prefix 1.1.1.30/32
destination-prefix 192.168.224.3/32

Note that we set the packet filters as the addresses that the packets will arrive on the SRX as, not as the
translated IP Addresses in NAT. Comments will follow the statements starting with ## in bold.
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:<1.1.1.100/57650->1.1.1.30/3389;6> matched filter
MatchTraffic:

## Packet from source address 1.1.1.100 (source port 57650) captured by the filter, destined to 1.1.1.30
port 3389, with the protocol # 6 for TCP

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:packet [48] ipid = 885, @422ec91e

##IPID of the packet shows its 885, which is very useful for comparing packets across multiple capture
points (e.g. viewing packet on originating machine, on FW, and on destination to make sure the packet
arrives)

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type 13, common flag
0x0, mbuf 0x422ec780

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow process pak fast ifl 69 in_ifp fe-0/0/7.0

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: fe-0/0/7.0:1.1.1.100/57650->1.1.1.30/3389, tcp, flag 2 syn

## This packet arrived on port fe-0/0/7.0

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: find flow: table 0x4dd31658, hash 32955(0xffff), sa 1.1.1.100,
da 1.1.1.30, sp 57650, dp 3389, proto 6, tok 448

## Perform has on the 5 tuple of the packet and do a flow lookup

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: no session found, start first path. in_tunnel - 0, from_cp_flag -
0

##No existing session found in the table lookup, so we must do a policy lookup

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow_first_create_session

  ƒ‰‡ʹ͸‘ˆͷ͸

 App Note

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow_first_in_dst_nat: in <fe-0/0/7.0>, out <N/A> dst_adr

1.1.1.30, sp 57650, dp 3389

## First process destination to check if it should be NAT’d

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: chose interface fe-0/0/7.0 as incoming nat if.

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:flow_first_rule_dst_xlate: packet 1.1.1.100->1.1.1.30 nsp2

0.0.0.0->192.168.224.30.

## We find that the destination should be NAT’d from 1.1.1.30 to 192.168.224.30

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:flow_first_routing: call flow_route_lookup(): src_ip 1.1.1.100,

x_dst_ip 192.168.224.30, in ifp fe-0/0/7.0, out ifp N/A sp 57650, dp 3389, ip_proto 6, tos 0

##Do a route lookup to determine what the destination interface and zone should be so we can do a policy
lookup.

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:Doing DESTINATION addr route-lookup

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: routed (x_dst_ip 192.168.224.30) from untrust (fe-0/0/7.0 in 0)

to ge-0/0/0.0, Next-hop: 192.168.224.30

##We find the route lookup that points us out interface ge-0/0/0.0 with a next hop which is the destination
on the attached 192.168.224.0/24 network.

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: policy search from zone untrust-> zone trust

## Now we do a policy lookup since we determined that the destination zone on ge-0/0/0.0 is trust, and the
packet arrived on fe-0/0/7.0 which is untrust.

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: app 0, timeout 1800s, curr ageout 20s

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: packet dropped, denied by policy

## Policy lookup shows that we do not find a match and the traffic is dropped due to no matching policy.

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: packet dropped, policy deny.

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow find session returns error.

Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: ----- flow_process_pkt rc 0x7 (fp rc -1)

  ƒ‰‡ʹ͹‘ˆͷ͸

 App Note

Now we will perform the same test, but we will have the appropriate policy in place do permit the traffic:

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:<1.1.1.100/51303->1.1.1.30/3389;6> matched

filter MatchTraffic:
## Packet arrives which is matched by the filter MatchTraffic. It has a source address of
1.1.1.100 (source port 51303) with a destination address 1.1.1.30, destination port 3389
with IP Protocol number 6, which is TCP.

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:packet [48] ipid = 5015, @423d7e9e

## Packet has the IPID 5015, this is very useful when comparing the packet to a PCAP on a
from a source or destination machine to track each packets to see if any did not arrive.

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type

13, common flag 0x0, mbuf 0x423d7d00

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow process pak fast ifl 72 in_ifp fe-0/0/7.0

## Packet Arrived on the fe-0/0/7.0 interface

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: fe-0/0/7.0:1.1.1.100/51303->1.1.1.30/3389,

tcp, flag 2 syn
## Deeper look at the packet, shows that it is the syn packet of the connection,
indicating the first packet of the flow

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: find flow: table 0x5258d7b0, hash

17008(0xffff), sa 1.1.1.100, da 1.1.1.30, sp 51303, dp 3389, proto 6, tok 448
## Packet Lookup with the information above, where sa=source address, da=destination
address, sp=source port, dp=destination port, proto=protocol

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: no session found, start first path. in_tunnel

- 0, from_cp_flag – 0
## First packet in flow, so no session is found

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow_first_create_session

## Create new session
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow_first_in_dst_nat: in <fe-0/0/7.0>, out
<N/A> dst_adr 1.1.1.30, sp 51303, dp 3389

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: chose interface fe-0/0/7.0 as incoming nat if.

  ƒ‰‡ʹͺ‘ˆͷ͸

 App Note

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_rule_dst_xlate: packet 1.1.1.100-

>1.1.1.30 nsp2 0.0.0.0->192.168.224.30.
## Before we do a policy lookup, we need to look and see if we need to do destination NAT,
in this case it is static NAT, which NAT’s the 1.1.1.30 address to 192.168.224.30. We
need to do this lookup BEFORE we do a route lookup, since if there is a destination NAT
change, then we need to do a route lookup on the translated destination address.
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_routing: call flow_route_lookup():
src_ip 1.1.1.100, x_dst_ip 192.168.224.30, in ifp fe-0/0/7.0, out ifp N/A sp 51303, dp
3389, ip_proto 6, tos 0
## Begin Routing Lookup
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:Doing DESTINATION addr route-lookup
## Perform Route Lookup
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: routed (x_dst_ip 192.168.224.30) from untrust
(fe-0/0/7.0 in 0) to ge-0/0/0.0, Next-hop: 192.168.224.30
## Performing the route lookup on destination IP address 192.168.224.30 shows that it
should go out interface ge-0/0/0.0, since ge-0/0/0.0 IP address is 192.168.224.3/24, the
network is directly attached, and therefore the next-hop is the destination itself
192.168.224.30. If the destination was not directly attached, then our Next-Hop would be
the Next-Hop of the route.
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: policy search from zone untrust-> zone trust
## Since we know that the ingress interface is fe-0/0/7.0 which is zone untrust, and we do
a route lookup, which determines that the outgoing interface is going to be ge-0/0/0.0,
which is zone trust. Therefore we can now do a policy lookup.
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: policy has timeout 900

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: app 0, timeout 1800s, curr ageout 20s

## The session timeout is set to 1800 seconds, which is based on the standard TCP
application timeout of 30 minutes (note that we don’t use Ticks like we do in ScreenOS,
just seconds.) We also have a curr ageout of 20 seconds, which is the initial TCP
timeout, if we don’t have the session established within 20 seconds, then the session will
be dropped.
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_src_xlate: src nat 0.0.0.0(51303) to
192.168.224.30(3389) returns status 1, rule/pool id 1/2.
## Here we see that we have found the SRX NAT pool for source translation
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: dip id = 2/0, 1.1.1.100/51303-
>192.168.224.3/48810
## We will use DIP id 2/0, DIP is a legacy term from ScreenOS, but we can see here that
we are translating the source from 1.1.1.100 (source port 51303) to 192.168.224.3 (source
port 48810) in this example we need to do this translation because the destination host
192.168.224.30 does not know about our 1.1.1.0/24 network.

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: choose interface ge-0/0/0.0 as outgoing phy if

## We select ge-0/0/0.0 as our outgoing interface

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:is_loop_pak: No loop: on ifp: ge-0/0/0.0, addr:

192.168.224.30, rtt_idx:0

  ƒ‰‡ʹͻ‘ˆͷ͸

 App Note

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:policy is NULL (wx/pim scenario)

## Message references other service processing not applicable for this rule

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:sm_flow_interest_check: app_id 0, policy 9,

app_svc_en 0, flags 0x2. not interested

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:sm_flow_interest_check: app_id 1, policy 9,

app_svc_en 0, flags 0x2. not interested
## We match the traffic to Policy 9, we can see what policy 9 is by looking through the
policy list with the command “show security policies” or we can find it with the command
“show security policies | find "Index\: 9"” which uses the regular expression to search
for Index: 9 in the policy and will show us the first instance. We can then make sure
that the traffic is matching the correct rule:
root@SRX210# run show security policies | find "Index\: 9"
Policy: Allow-RDP-Server, State: enabled, Index: 9, Sequence number: 1
Source addresses: 1.1.1.0/24
Destination addresses: 192.168.224.30
Applications: RDP
Action: permit, log

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_service_lookup(): natp(0x51ee4680):

app_id, 0(0).

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: service lookup identified service 0.

Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow_first_final_check: in <fe-0/0/7.0>, out

<ge-0/0/0.0>
## Send the Packet out interface ge-0/0/0.0

Here is the reverse direction capture:

Aug 14 19:11:40 19:11:40.192283:CID-0:RT:<192.168.224.30/3389->192.168.224.3/48810;6> matched filter

MatchTrafficReverse:

## Here the traffic arrives from 192.168.224.30 source port 3389, and is destined to our NAT’d interface
192.168.224.3 destination port 48810

Aug 14 19:11:40 19:11:40.192283:CID-0:RT:packet [40] ipid = 8174, @42308b9e

  ƒ‰‡͵Ͳ‘ˆͷ͸

 App Note

Aug 14 19:11:40 19:11:40.192283:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type 13, common flag
0x0, mbuf 0x42308a00

Aug 14 19:11:40 19:11:40.192283:CID-0:RT: flow process pak fast ifl 67 in_ifp ge-0/0/0.0

Aug 14 19:11:40 19:11:40.192283:CID-0:RT: ge-0/0/0.0:192.168.224.30/3389->192.168.224.3/48810, tcp,

flag 10

Aug 14 19:11:40 19:11:40.192283:CID-0:RT: find flow: table 0x4dd31658, hash 44831(0xffff), sa

192.168.224.30, da 192.168.224.3, sp 3389, dp 48810, proto 6, tok 384

## Do a hash lookup on the session to find it in the table

Aug 14 19:11:40 19:11:40.192283:CID-0:RT: flow got session.

Aug 14 19:11:40 19:11:40.192283:CID-0:RT: flow session id 12535

## Since this is return traffic, we have found it in the session table, as id 12535

Aug 14 19:11:40 19:11:40.192283:CID-0:RT: tcp seq check.

Aug 14 19:11:40 19:11:40.192283:CID-0:RT: post addr xlation: 1.1.1.30->1.1.1.100.

## Translate the packet to the appropriate translation for outbound flow

Aug 14 19:11:40 19:11:40.192283:CID-0:RT:mbuf 0x42308a00, exit nh 0x80010

## Place packet in buffer to be sent out

Aug 14 19:11:40 19:11:40.192283:CID-0:RT: ----- flow_process_pkt rc 0x0 (fp rc 0)

Troubleshooting Firewall Session Establishment

1. Is the packet arriving on the SRX? The first thing that we need to make sure is that the traffic is
arriving on the SRX itself. We should be able to see the traffic by just viewing the output of the debug
itself. This may not be the case if stateless firewall filters are being used, which may drop the packets
before they are processed by the flow engine. If you don’t see the traffic arriving on the SRX, you
should check the routing table of the peer device, as well as checking to see if the peer device has an
ARP entry for the IP address we are trying to reach. This is particularly important if you are using static
NAT, as Proxy ARP MAY be required.

  ƒ‰‡͵ͳ‘ˆͷ͸

 App Note

2. Is the packet being blocked by a Screen? If a packet is an invalid packet, and screens are enabled on
the source zone, it may be dropped. An example would be a packet that has a SYN, ACK, and FIN bits
set.
3. Is the packet destined for the firewall itself? This is most common with management traffic such as
telnet, ssh, and SNMP, but also for protocol traffic such as BGP, OSPF, RIP, and PIM. In this case, we
must make sure that these respective services are running on that interface. In JUNOS, this is
configured at either the zone or interface level under [edit security zones security-zone <zone> host-
inbound-traffic] if this is not configured then the firewall will drop the packet.
4. Is the session asymmetric? Stateful SYN checking is on by default, and if the firewall does not see the
SYN packet, it will drop the subsequent packets and the session will not establish. This should be clear
from the debug. In this situation you must either ensure that the SRX is in the initial packet path so that
it sees the SYN packet (recommended!) or turn off SYN checking. If the SRX cannot see both
directions of the flow then some features such as IDP may not be able to work at the full capacity.
5. Is the packet from an existing session? If the packet matches an existing session then we will not
include all debugging information in that output, just some basics, and we refer it to the session ID to
cross reference. You can still examine the session table to track this session with the “show security
flow session” command.
6. Is the packet arriving on the correct interface/zone? When looking at the debug output you should
be able to tell what interface the packet has arrived on, and subsequently, what the source zone is. This
is important for policy lookup, and if it is not matched properly, then most likely a problem will occur.
7. Is the appropriate route / L2 forwarding path being selected? Depending on if you are using Layer
3 more or Transparent mode, we will need to ensure that the appropriate forwarding decision is chosen.
With layer 3 mode, the firewall will do a route lookup based on the destination of the route, which will
determine what the egress interface will be (which also determines the to-zone,) if the device is in
transparent mode, the device will either do an ARP, or will broadcast the packet out all interfaces
(except that from which it came) in the same bridging domain. If the wrong route is set (in transparent
mode, the forwarding decision should happen without intervention, unless other mechanisms such as
Dynamic ARP inspection or L2 Broadcast filtering in place to block this) then we may chose the wrong
outbound interface to send the traffic out, as well as the wrong NAT decision. Currently, we do not give
a route-id in the output of the debug like we did in ScreenOS. This will probably change in the future.
You may need to do a route lookup with the “show route <prefix>” command to make sure that the
correct route is chosen and that the appropriate outbound interface and next-hop is selected. In
transparent mode, you can check the L2 forwarding table with the “show arp” and “show ethernet-
switching table” if using the branch SRX, while the High End SRX use a different command “show l2-
learning interface” to see what entries are known by the system.
8. Is the correct policy being selected by the firewall? Now that we have determined what the
appropriate egress interface / zone are supposed to be, assuming that those are correct, we need to make
sure the correct firewall policy is being selected by the firewall. Today that number is referenced in the
debug logs as “policy <#>” and as shown in the debug above, we can run the “show security policies |
find "Index\: <#>" command to make sure that the policy is correct. If it is not the correct policy, you
will need to take a look at the source/destination address and service. It is often prudent to look into the
address and service objects themselves to make sure that they are really matching what you are looking
for, and not just the names of the address objects themselves as shown in the policy. For instance, we

  ƒ‰‡͵ʹ‘ˆͷ͸

 App Note

could name an object 1.1.1.0/24, but the actual address that it matches if we omit the subnet mask would
be 1.1.1.0/32. Just by looking at the object name we may have it wrong. Also, see the next step about
NAT configuration as it may impact your policy.
9. Is the correct NAT being applied by the policy? Today we have some improvements that are going to
be put into NAT debugging, but for now, the best thing to check is to look at the output of the debug and
see what translation takes place. Is it what you expect (e.g. is the source translated or not translated as
expected? Is the destination translated or not translated as expected?) Remember that static translations
happen before policy lookup, while source and destination translations occur AFTER policy lookup, so
you need to make sure that you have factored this in when configuring your policy. If the wrong action
is being taken on your policy, you may want to review the rulebase configuration to visually inspect the
match and action criteria. There can be many NAT rulebases depending on the configuration, so be sure
to take that into account.
10. Does the debug show that the packet is being processed by “Application Services” or is destined to
a VPN tunnel? If the packet is being processed by application services then further debugging may be
necessary, for instance, if you have IDP enabled on that policy, and the IDP detects that packet as
malicious, it may drop it based upon policy (so even though the FW flow looked OK, it could still be
dropped based upon policy) same is true for other UTM features, so you need to keep this in mind. If
the packet is destined to a tunnel, then you should see this output. If a VPN is referenced, the VPN must
be active, or else the packet will not have anywhere to go.

Troubleshooting IPSec VPN’s

VPN troubleshooting is a very common task, which is exacerbated by the fact that there may be interoperability
issues when working with other vendors. It’s not so much that vendors use proprietary features (although this
does come up from time to time) with standard site to site VPN tunnels, but that at least in IKEv1, there was
some components such as negotiating proxy-ID’s and key lifetimes which were left up to the imagination of the
developers, and since (in the case of proxy-ID’s) they must match, this causes issues with interoperability.
There are also a number of settings which must be coordinated between the two endpoints (often made more
challenging when these are not under 1 administrative domain.) The SRX provides mechanisms to debug the
actual VPN traffic in the event that a VPN does not come up properly. To start, you can easily check the status
of the VPN’s using the following commands:

show security ike security-associations [brief | detail | index]

This command will show the Phase 1 adjacencies that have been established
root@SRX210> show security ike security-associations detail
IKE peer 192.168.224.1, Index 92,
Role: Initiator, State: UP
Initiator cookie: 4a4b83a4bef11802, Responder cookie: ba56a54c3cac6380
Exchange type: Main, Authentication method: Pre-shared-keys
Local: 192.168.224.3:500, Remote: 192.168.224.1:500

  ƒ‰‡͵͵‘ˆͷ͸

 App Note

Lifetime: Expires in 26120 seconds

Algorithms:
Authentication : sha1
Encryption : aes-cbc (256 bits)
Pseudo random function: hmac-sha1
Traffic statistics:
Input bytes : 856
Output bytes : 1144
Input packets: 5
Output packets: 7
Flags: Caller notification sent
IPSec security associations: 1 created, 1 deleted
Phase 2 negotiations in progress: 0

show security ipsec security-associations [brief | detail | index]

This command will show the Phase 2 adjacencies that have been established
root@SRX210> show security ipsec security-associations detail
Virtual-system: Root
Local Gateway: 192.168.224.3, Remote Gateway: 192.168.224.1
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=192.168.0.0/16)
DF-bit: clear
Direction: inbound, SPI: 7936e70e, AUX-SPI: 0
Hard lifetime: Expires in 884 seconds
Lifesize Remaining: Unlimited
Soft lifetime: Expires in 244 seconds
Mode: tunnel, Type: dynamic, State: installed, VPN Monitoring: -
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: enabled, Replay window size: 64

Direction: outbound, SPI: 153ec5f5, AUX-SPI: 0

Hard lifetime: Expires in 884 seconds
Lifesize Remaining: Unlimited
Soft lifetime: Expires in 244 seconds
Mode: tunnel, Type: dynamic, State: installed, VPN Monitoring: -
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: enabled, Replay window size: 64

  ƒ‰‡͵Ͷ‘ˆͷ͸

 App Note

Starting with those two commands you can get a good feel for the state of the VPN connections. Absence of the
relationships means that they are not currently established. By default, the SRX will log IKE information to the
log /var/log/kmd, which you can view the information by issuing the command:

show log kmd

Without a debug enabled, this may not have very much information. There are two places
where you can enable debug information as it pertains to troubleshooting VPN
establishment. The first is under the IKE configuration, while the second is under the
IPSec configuration. These pertain to Phase 1 and Phase 2 establishment. By default,
they will log to /var/log/kmd, although you can define your own destination for each debug
to log to. The following commands will enable debugs:

Phase 1:

set security ike traceoptions flag ike

The above command enables IKE tracing, which will automatically send it to the log kmd.
You can also specify your own destination using the file option.

Phase 2:
set security ipsec traceoptions flag all
The above command enables IPSec tracing (all flag facilities) and will automatically send
it to kmd. There is not an option to send it elsewhere like there is for ike.

Example of Tracing VPN traffic:

For this example we have enabled both IKE and IPSec debugging with the commands below:

set security ike traceoptions flag ike

set security ipsec traceoptions flag all
root@SRX210> show log kmd
Aug 14 21:05:53 SRX210 clear-log[21533]: logfile cleared
Aug 14 21:07:38 ike_get_sa: Start, SA = { d3622295 fb67acf0 - 00000000 00000000
} / 00000000, remote = 1.1.1.100:500
Aug 14 21:07:38 ike_sa_allocate: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb47
3562 }
Aug 14 21:07:38 ike_init_isakmp_sa: Start, remote = 1.1.1.100:500, initiator = 0
Aug 14 21:07:38 ike_decode_packet: Start
Aug 14 21:07:38 ike_decode_packet: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb
473562} / 00000000, nego = -1

  ƒ‰‡͵ͷ‘ˆͷ͸

 App Note

Aug 14 21:07:38 ike_decode_payload_sa: Start

Aug 14 21:07:38 ike_decode_payload_t: Start, # trans = 1
Aug 14 21:07:38 ike_st_i_vid: VID[0..44] = 47bbe7c9 93f1fc13 ...
Aug 14 21:07:38 ike_st_i_vid: VID[0..8] = da8e9378 80010000 ...
Aug 14 21:07:38 ike_st_i_vid: VID[0..16] = afcad713 68a1f1c9 ...
Aug 14 21:07:38 ike_st_i_vid: VID[0..8] = 09002689 dfd6b712 ...
Aug 14 21:07:38 ike_st_i_vid: VID[0..16] = 4485152d 18b6bbcd ...
Aug 14 21:07:38 ike_st_i_vid: VID[0..16] = 90cb8091 3ebb696e ...
## Here we receive the first IKE packet and begin the IKE decode
Aug 14 21:07:38 ike_st_i_id: Start
Aug 14 21:07:38 ike_st_i_sa_proposal: Start
Aug 14 21:07:38 ike_isakmp_sa_reply: Start
Aug 14 21:07:38 ike_st_i_nonce: Start, nonce[0..32] = 35c16d86 b96558c9 ...
Aug 14 21:07:38 ike_st_i_cert: Start
Aug 14 21:07:38 ike_st_i_hash_key: Start, no key_hash
Aug 14 21:07:38 ike_st_i_ke: Ke[0..128] = 817323c0 1a4eefbe ...
Aug 14 21:07:38 ike_st_i_cr: Start
Aug 14 21:07:38 ike_st_i_private: Start
Aug 14 21:07:38 ike_st_o_sa_values: Start
Aug 14 21:07:38 ike_st_o_ke: Start
Aug 14 21:07:38 ike_st_o_nonce: Start
Aug 14 21:07:38 ike_policy_reply_isakmp_nonce_data_len: Start
Aug 14 21:07:38 ike_st_o_id: Start
Aug 14 21:07:38 ike_policy_reply_isakmp_id: Start
Aug 14 21:07:38 ike_st_o_certs_base: Start
Aug 14 21:07:38 ike_st_o_sig_or_hash: Start, auth_method = 4
Aug 14 21:07:38 ike_st_o_hash: Start
Aug 14 21:07:38 ike_find_pre_shared_key: Find pre shared key key for 1.1.1.1:500, id =
ipv4(udp:500,[0..3]=1.1.1.1) -> 1.1.1.100:500, id =
usr@fqdn(udp:500,[0..18][email protected])
Aug 14 21:07:38 ike_policy_reply_find_pre_shared_key: Start
Aug 14 21:07:38 ike_calc_mac: Start, initiator = false, local = true
Aug 14 21:07:38 ike_policy_reply_isakmp_vendor_ids: Start
Aug 14 21:07:38 ike_st_o_status_n: Start
Aug 14 21:07:38 ike_st_o_private: Start
Aug 14 21:07:38 ike_policy_reply_private_payload_out: Start
Aug 14 21:07:38 ike_policy_reply_private_payload_out: Start
Aug 14 21:07:38 ike_policy_reply_private_payload_out: Start

  ƒ‰‡͵͸‘ˆͷ͸

 App Note

Aug 14 21:07:38 ike_st_o_calc_skeyid: Calculating skeyid

Aug 14 21:07:38 ike_encode_packet: Start, SA = { 0xd3622295 fb67acf0 - 8980eab2 eb473562 }
/ 00000000, nego = -1
Aug 14 21:07:38 ike_send_packet: Start, send SA = { d3622295 fb67acf0 - 8980eab2
eb473562}, nego = -1, src = 1.1.1.1:500, dst = 1.1.1.100:500, routing table id = 0

## In the above statements we determine that the IKE packet is from 1.1.1.100 destined to
1.1.1.1 (our interface) with an IKE ID of type user at hostname (or UFQDN)
[email protected]. We also check the value in the preshared key to make sure that it
matches properly.
Aug 14 21:07:38 ike_get_sa: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb473562 } /
00000000, remote = 1.1.1.100:500
Aug 14 21:07:38 ike_sa_find: Found SA = { d3622295 fb67acf0 - 8980eab2 eb473562 }
Aug 14 21:07:38 ike_decode_packet: Start
Aug 14 21:07:38 ike_decode_packet: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb473562} /
00000000, nego = -1
Aug 14 21:07:38 ike_st_i_hash: Start, hash[0..20] = 52e59171 a54dad00 ...
Aug 14 21:07:38 ike_calc_mac: Start, initiator = false, local = false
Aug 14 21:07:38 ike_st_i_cert: Start
Aug 14 21:07:38 ike_st_i_status_n: Start, doi = 1, protocol = 1, code = Replay status
notification (24577), spi[0..16] = d3622295 fb67acf0 ..., data[0..4] = 00000000 00000000
...
Aug 14 21:07:38 ike_st_i_status_n: Start, doi = 1, protocol = 1, code = Initial contact
notification (24578), spi[0..16] = d3622295 fb67acf0 ..., data[0..0] = 00000000 00000000
...
Aug 14 21:07:38 ike_st_i_private: Start
Aug 14 21:07:38 ike_st_o_wait_done: Marking for waiting for done
Aug 14 21:07:38 ike_st_o_all_done: MESSAGE: Phase 1 { 0xd3622295 fb67acf0 - 0x8980eab2
eb473562 } / 00000000, version = 1.0, xchg = Aggressive, auth_method = Pre shared keys
with XAuth (initiator), Responder, cipher = aes-cbc, hash = sha1, prf = hmac-sha1, life =
0 kB /
Aug 14 21:07:38 1.1.1.1:500 (Responder) <-> 1.1.1.100:500 { d3622295 fb67acf0 - 8980eab2
eb473562 [-1] / 0x00000000 } Aggr; MESSAGE: Phase 1 version = 1.0, auth_method = Pre
shared keys with XAuth (initiator), cipher = aes-cbc, hash = sha1, prf = hmac-sha1, life =
0 kB /
## The above statements show that we are using Aggressive Mode, with Pre-Shared Keys and
XAuth for client negotiation. The proposed Cipher is AES128-Sha1 (128 is implied) and
that the key lifetime is not defined. This negotiation is clearly for a remote client
VPN, typical IPSec Site to Site would show Main Mode, and would not typically have XAuth.
Aug 14 21:07:38 ike_send_notify: Connected, SA = { d3622295 fb67acf0 - 8980eab2 eb473562},
nego = -1
Aug 14 21:07:38 jnp_ike_connect_cfg: Start, remote_name = 1.1.1.100:500, flags = 00008000
Aug 14 21:07:38 ike_sa_find_ip_port: Remote = 1.1.1.100:500, Found SA = { d3622295
fb67acf0 - 8980eab2 eb473562}
Aug 14 21:07:38 ike_alloc_negotiation: Start, SA = { d3622295 fb67acf0 - 8980eab2
eb473562}

  ƒ‰‡͵͹‘ˆͷ͸

 App Note

Aug 14 21:07:38 jnp_ike_connect_cfg: SA = { d3622295 fb67acf0 - 8980eab2 eb473562}, nego =

0
Aug 14 21:07:38 ike_init_cfg_negotiation: Start, initiator = 1, message_id = 3fdb5840
Aug 14 21:07:38 ike_st_o_gen_hash: Start
Aug 14 21:07:38 ike_st_o_cfg_attr: Start
Aug 14 21:07:38 ike_st_o_private: Start
Aug 14 21:07:38 ike_st_o_encrypt: Marking encryption for packet
Aug 14 21:07:38 ike_encode_packet: Start, SA = { 0xd3622295 fb67acf0 - 8980eab2 eb473562 }
/ 3fdb5840, nego = 0
Aug 14 21:07:38 ike_send_packet: Start, send SA = { d3622295 fb67acf0 - 8980eab2
eb473562}, nego = 0, src = 1.1.1.1:500, dst = 1.1.1.100:500, routing table id = 0
## We respond to the IKE message from the client
Aug 14 21:07:43 ike_retransmit_callback: Start, retransmit SA = { d3622295 fb67acf0 -
8980eab2 eb473562}, nego = 0
Aug 14 21:07:43 ike_send_packet: Start, retransmit previous packet SA = { d3622295
fb67acf0 - 8980eab2 eb473562}, nego = 0, src = 1.1.1.1:500, dst = 1.1.1.100:500, routing
table id = 0
Aug 14 21:07:44 ike_get_sa: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb473562 } /
3fdb5840, remote = 1.1.1.100:500
Aug 14 21:07:44 ike_sa_find: Found SA = { d3622295 fb67acf0 - 8980eab2 eb473562 }
Aug 14 21:07:44 ike_st_o_done: ISAKMP SA negotiation done
Aug 14 21:07:44 ike_send_notify: Connected, SA = { d3622295 fb67acf0 - 8980eab2 eb473562},
nego = -1
Aug 14 21:07:44 ike_free_negotiation_isakmp: Start, nego = -1
Aug 14 21:07:44 ike_free_negotiation: Start, nego = -1
Aug 14 21:07:44 ike_decode_packet: Start
Aug 14 21:07:44 ike_decode_packet: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb473562} /
3fdb5840, nego = 0
Aug 14 21:07:44 ike_decode_payload_attr: Start
Aug 14 21:07:44 ike_st_i_encrypt: Check that packet was encrypted succeeded
Aug 14 21:07:44 ike_st_i_gen_hash: Start, hash[0..20] = f431a256 ac900477 ...
Aug 14 21:07:44 ike_st_i_cfg_attr: Start
Aug 14 21:07:44 ike_st_i_private: Start
Aug 14 21:07:44 ike_st_o_cfg_wait_done: Marking for waiting for done
Aug 14 21:07:44 ike_st_o_cfg_wait_done: MESSAGE: CFG Mode wait done
Aug 14 21:07:44 1.1.1.1:500 (Initiator) <-> 1.1.1.100:500 { d3622295 fb67acf0 - 8980eab2
eb473562 [0] / 0x3fdb5840 } CFG; MESSAGE: CFG Mode wait done
## At this point Phase 1 establishment is completed, we now begin Phase 2 negotiation
Aug 14 21:07:44 ike_send_notify: Connected, SA = { d3622295 fb67acf0 - 8980eab2 eb473562},
nego = 0
Aug 14 21:07:44 jnp_ike_connect_cfg: Start, remote_name = 1.1.1.100:500, flags = 00008000

  ƒ‰‡͵ͺ‘ˆͷ͸

 App Note

Aug 14 21:07:44 ike_sa_find_ip_port: Remote = 1.1.1.100:500, Found SA = { d3622295

fb67acf0 - 8980eab2 eb473562}
Aug 14 21:07:44 ike_alloc_negotiation: Start, SA = { d3622295 fb67acf0 - 8980eab2
eb473562}
Aug 14 21:07:44 jnp_ike_connect_cfg: SA = { d3622295 fb67acf0 - 8980eab2 eb473562}, nego =
1
Aug 14 21:07:44 ike_init_cfg_negotiation: Start, initiator = 1, message_id = 7e56b115
Aug 14 21:07:44 ike_st_o_gen_hash: Start
Aug 14 21:07:44 ike_st_o_cfg_attr: Start
Aug 14 21:07:44 ike_st_o_private: Start
Aug 14 21:07:44 ike_st_o_encrypt: Marking encryption for packet
Aug 14 21:07:44 ike_encode_packet: Start, SA = { 0xd3622295 fb67acf0 - 8980eab2 eb473562 }
/ 7e56b115, nego = 1
Aug 14 21:07:44 ike_send_packet: Start, send SA = { d3622295 fb67acf0 - 8980eab2
eb473562}, nego = 1, src = 1.1.1.1:500, dst = 1.1.1.100:500, routing table id = 0
Aug 14 21:07:44 ike_get_sa: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb473562 } /
7e56b115, remote = 1.1.1.100:500
Aug 14 21:07:44 ike_sa_find: Found SA = { d3622295 fb67acf0 - 8980eab2 eb473562 }
Aug 14 21:07:44 ike_decode_packet: Start
Aug 14 21:07:44 ike_decode_packet: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb473562} /
7e56b115, nego = 1
Aug 14 21:07:44 ike_decode_payload_attr: Start
Aug 14 21:07:44 ike_st_i_encrypt: Check that packet was encrypted succeeded
Aug 14 21:07:44 ike_st_i_gen_hash: Start, hash[0..20] = dfcaa88c 40f3697e ...
Aug 14 21:07:44 ike_st_i_cfg_attr: Start
Aug 14 21:07:44 ike_st_i_private: Start
Aug 14 21:07:44 ike_st_o_cfg_wait_done: Marking for waiting for done
Aug 14 21:07:44 ike_st_o_cfg_wait_done: MESSAGE: CFG Mode wait done
Aug 14 21:07:44 1.1.1.1:500 (Initiator) <-> 1.1.1.100:500 { d3622295 fb67acf0 - 8980eab2
eb473562 [1] / 0x7e56b115 } CFG; MESSAGE: CFG Mode wait done
Aug 14 21:07:44 ike_send_notify: Connected, SA = { d3622295 fb67acf0 - 8980eab2 eb473562},
nego = 1
Aug 14 21:07:44 ike_get_sa: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb473562 } /
c8ef4ee3, remote = 1.1.1.100:500
Aug 14 21:07:44 ike_sa_find: Found SA = { d3622295 fb67acf0 - 8980eab2 eb473562 }
Aug 14 21:07:44 ike_alloc_negotiation: Start, SA = { d3622295 fb67acf0 - 8980eab2
eb473562}
Aug 14 21:07:44 ike_init_qm_negotiation: Start, initiator = 0, message_id = c8ef4ee3
## In the above lines we have received the first packet for Phase 2 Quick Mode
negotiation, we decrypt the packet (which matches a SA negotiated in Phase 1) and parse
out the fields to begin checking whether the Phase 2 Proposal matches a configured
proposal
Aug 14 21:07:44 ike_decode_packet: Start

  ƒ‰‡͵ͻ‘ˆͷ͸

 App Note

Aug 14 21:07:44 ike_decode_packet: Start, SA = { d3622295 fb67acf0 - 8980eab2 eb473562} /

c8ef4ee3, nego = 2
Aug 14 21:07:44 ike_decode_payload_sa: Start
Aug 14 21:07:44 ike_decode_payload_t: Start, # trans = 1
Aug 14 21:07:44 ike_st_i_encrypt: Check that packet was encrypted succeeded
Aug 14 21:07:44 ike_st_i_qm_hash_1: Start, hash[0..20] = f5431c9c 5a41ae04 ...
Aug 14 21:07:44 ike_st_i_qm_nonce: Nonce[0..32] = 847152b6 e30c41d0 ...
Aug 14 21:07:44 ike_st_i_qm_ke: Ke[0..128] = 5591055a 49022ca1 ...
Aug 14 21:07:44 ike_st_i_qm_sa_proposals: Start
Aug 14 21:07:44 KMD_INTERNAL_ERROR: Phase2 finish: No sa_cfg found!
Aug 14 21:07:44 KMD_PM_P2_POLICY_LOOKUP_FAILURE: Policy lookup for Phase-2 [responder]
failed for p1_local=ipv4(udp:500,[0..3]=1.1.1.1)
p1_remote=usr@fqdn(udp:500,[0..18][email protected])
p2_local=ipv4_subnet(any:0,[0..7]=192.168.100.0/24) p2_remote=ipv4(any:0,[0..3]=1.1.1.100)

## Here we see that the Phase 2 proposal is failing policy lookup. Please see the end of
this debug for steps to take to determine what the root cause can be. In this above
output, we see what the remote end is proposing. In our case, the lookup failure has to
do with the fact that the VPN is attempting to terminate on the wrong interface. Below is
the output that you should see if the connection succeeds Phase 2:
Aug 25 17:09:22 1.1.1.1:500 (Responder) <-> 1.1.1.100:500 { cdf1c372 0f946928 - ad6b9fbb
4594cb35 [2] / 0xf4f71bbc } QM; MESSAGE: Phase 2 connection succeeded, No PFS, group = 0
Aug 25 17:09:22 ike_qm_call_callback: MESSAGE: Phase 2 connection succeeded, No PFS, group
= 0
Aug 25 17:09:22 1.1.1.1:500 (Responder) <-> 1.1.1.100:500 { cdf1c372 0f946928 - ad6b9fbb
4594cb35 [2] / 0xf4f71bbc } QM; MESSAGE: SA[0][0] = ESP 3des, life = 0 kB/0 sec, group =
0, tunnel, hmac-sha1-96, key len = 0, key rounds = 0
Aug 25 17:09:22 ike_qm_call_callback: MESSAGE: SA[0][0] = ESP 3des, life = 0 kB/0 sec,
group = 0, tunnel, hmac-sha1-96, key len = 0, key rounds = 0
Aug 25 17:09:22 ike_st_o_qm_wait_done: Marking for waiting for done

Steps for Troubleshooting VPN Establishment Issues:

1. Are you seeing the IKE packets from the peer device (either initiated from the peer, or responses
from the peer?) If not make sure that you are allowing IKE packets to enter the firewall, which is
configured under [edit security zones security-zones <zone> host-inbound-traffic system-services] and
IKE. This can also be configured on an interface by interface basis under the zone config. If it is not
enabled, the IKE engine will not see the traffic, eventhough it will be able to send the IKE packets out.
This is a departure from how we supported it on ScreenOS so make sure that you keep this in mind.
2. Is your VPN terminating on the correct interface? Like ScreenOS, we must define which interface
the VPN will listen on for the VPN connection. In the example above, we showed that there was a
proposal mismatch, which was due to the fact that the VPN was terminating on interface fe-0/0/7.0 when
the configuration had originally stated that it was to terminate on interface ge-0/0/0.0 (configuration not

  ƒ‰‡ͶͲ‘ˆͷ͸

 App Note

shown for brevity.) Assuming that IKE is enabled on the interface, we will process the traffic and will
not fully define that it fails until Phase 2 if the interface is not correct. This behavior may change in
future JUNOS releases.
3. Do your Phase 1 Negotiation Types match? IKE Phase 1 has two different modes, Aggressive and
Main Mode. The difference has to do with how the IKE negotiation is established. Aggressive mode
negotiates the identity in clear text because it does not have a fixed IP Address (e.g. a client which may
move around) while Main Mode does have fixed configurations (or may use DNS hostnames.) You
cannot mix and match the Phase 1 types, if one side uses Main mode and the other uses aggressive mode,
then the Phase 1 negotiations will fail, with a message stating that there is no matching Phase 1 policy
and will list what the client is attempting to negotiate.
4. Does your Pre-Shared Keys Match? You will see messages in phase one relating to INVALID
Preshared Key if you have a PSK mismatch in the negotiation, this will happen in Phase 1. Make sure
that both sides have the same pre shared key (Case Matters!)
5. If using Certificates, is the appropriate CA certificate loaded on the SRX, and is it configured to
do CRL checking? If you use certificate based authentication, you must make sure that you load the
CA certificate which signed the client certificate onto the SRX. Furthermore, it is a good idea to check
for Revocation status of the certificate. Today this can be done via CRL (OSCP is not yet supported on
the SRX.) If the certificate has been revoked, then it will not be allowed to establish a VPN connection.
6. Do your Phase 1 Proposals match? Phase 1 negotiates what encryption and authentication algorithms
will be used to secure the Phase 2 proposal. The proposals must match on both sides. Juniper allows
you to have a proposal-set, which consists of multiple proposals, but at least one of those must match the
proposal from the peer. If a proposal does match (including bit-length [e.g. AES128 is different from
AES256] then the phase 1 negotiation will fail. Lastly, the Diffie-Hellman group being used must match
(e.g. group2) if these don’t match then phase 1 will fail.
7. Does the Phase 1 Identity match what is being proposed? This is particularly a concern when
matching identities to a client VPN rather than site to site. In site to site, the identity is usually an IP
Address. If using client based connections, there are several different types of IKE ID: Distinguished
Names, hostname (or FQDN), inet (or IP Address), or user-at-hostname (or email Address.) Also, you
must be mindful of the connection limit defined for the IKE Gateway, you cannot go over what the
number of concurrent connections are defined for the VPN.
8. If XAuth is being used, are the credentials provided valid? In the SRX, when using XAuth (which
takes place between Phase 1 and Phase 2) we must use external authentication, we do not support local
authentication at this point like we did in ScreenOS (this will be supported in the future.) You can
determine if authentication is processing properly by using the traceoption “set system processes
general-authentication-service traceoptions file Auth flag all” which will instruct the system to log
authentication requests to the file Auth. You can also check the Radius server logs as well to determine
if this is passing or failing authentication.
9. Is Perfect Forward Secrecy configured, and if so does the Diffie Hellman Group match? Since
Phase 1 negotiation in Aggressive Mode takes place in part in clear text, Perfect Forward Secrecy can
re-negotiate Phase 1 in the security association formed in the initial Phase 1 negotiation so that the
identity is hidden. This can also be performed in Main Mode as well. If using Perfect Forward Secrecy,
we must perform it on the peer device as well, and the Diffie Hellman group must match as well.

  ƒ‰‡Ͷͳ‘ˆͷ͸

 App Note

10. In Phase 2, does the Encryption and Authentication Algorithms match properly? Just like Phase 1,
we must match the encryption and authentication algorithms in Phase 2, if these do not match, then there
will be a failure in the negotiation, and a message stating that there is No Proposal Chosen will occur in
Quick Mode (Phase 2)
11. In Phase 2, does the Proxy ID’s match? This is one of the most common issues for IPSec VPN
negotiation failure, particularly when interoperating with another VPN vendor. The reason for this is
that the IKEv1 standard specificies that Proxy ID’s (Local/Remote/Service) must be specified, but
doesn’t say how they should be compiled, so therefore, each vendor may do it differently. In Policy
Based VPN’s, unless defined by the IPSec Policy, we will use the policy as the template for the proxy
ID.
a. So for instance, if we have a source address of 192.168.1.0/24 to destination address
192.168.2.0/24, with a service of any, we will negotiate the proxy ID as:
i. Local ID: 192.168.1.0/24
ii. Remote ID: 192.168.2.0/24
iii. Service: Any
b. While the remote side will negotiate it as:
i. Local ID: 192.168.2.0/24
ii. Remote ID: 192.168.1.0/24
iii. Service: Any
c. Note that if you use address-sets or multiple addresses in a policy match, we will negotiate that
as 0.0.0.0/0 since we cannot set multiple proxy-id’s in the proposal.
d. In Route Based VPN’s you must specify what the Proxy ID is since it will not take it from a
policy. In reality, the proxy ID’s don’t really mean very much when it comes to what traffic can
exist in the tunnel, that is still controlled by policy, but rather proxy ID’s are used to define a
tunnel uniquely. You will be able to see what the other side is proposing in the debug output.

Other factors which can cause a VPN negotiation to fail

1. Is the other side negotiating the correct version of IKE? This will not be that common, but we do
not yet support IKEv2 in SRX, only in ScreenOS. If the other end attempts to use IKEv2 then the
negotiations will fail.
2. Does the other end care about the key lifetimes? SRX does not particularly care about if the
negotiated value for the Phase 1 or Phase 2 key lifetimes match, but some other vendors do care and
require that the key lifetimes match on both sides.

Troubleshooting Intrusion Detection and Prevention

When using the Intrusion Detection and Prevention there are a few things that should be taken into account
when issues arise with the IPS engine. Some of these issues require JTAC involvement for resolution, but the
basic troubleshooting steps can be performed to catch many issues. We’ll start with the basics:

  ƒ‰‡Ͷʹ‘ˆͷ͸

 App Note

1. If using the branch products, is the hardware the correct hardware to support IDP? In the IDP
100/210 you must be using the high memory versions of the product to use IDP
2. Do you have the IDP License applied to your SRX (or both SRX if performing in HA cluster?)
You must be running the IDP license to be able to download attack signatures from Juniper. If just
using customer signatures you should not need an IDP license, however, there is a bug in 9.5 and 9.6
that requires that you have the license (you will get a warning on commit) so make sure that you have
the license enabled, or use the patched SRX version. You can import them into the WEBUI, NSM, or
do so in the CLI with the command: “request system license add <filename>” You can show the
licenses on the system with the “show system license” command.
3. Have you downloaded the attack signatures successfully? If using the Juniper defined attack
signatures you must download them first from Juniper. If you do not have internet access to do so, then
you can simply transfer the “/var/db/idpd” folder from another SRX to your SRX. To download the
attack objects, you can issue the “request security idp security-package download” command, and issue
the “request security idp security-package download status” to view the status of the download. You
may also want to download the policy templates using the “request security idp security-package
download policy-templates” command.
4. Have you installed the attack signatures on the device? You must install the attack signatures on the
device after you have downloaded them or they will not be used for IDP inspection. To install the attack
signatures use the “request security idp install” for attacks, “request security idp install policy-
templates” for the templates; or check the install status with “request security idp install status”
5. Have you set an active policy for the SRX to detect attacks against? After you have downloaded the
attacks, you must create a policy, or use an existing policy template as you active attack policy. Today,
only 1 policy can be active at a time, and this policy can be configured with the command “set security
idp active-policy <policy>”
6. Have you checked the IDP status to make sure that it is up and running properly? You should
check to make sure that the IDP engine is running properly. You can do this with the following
commands: “show security idp status” and the command “show security idp security-package-version”
7. Have you enabled IDP processing for the firewall rule which you wish to perform IDP processing
on? IDP processing is not on for firewall rules unless it is explicitly configured to be enabled. You can
check to see whether the IDP is enabled on a policy by using the command “show security policies” to
view if application services is enabled—although you will still need to delve further into the
configuration to determine if IDP is enabled:
root@SRX210# run show security policies from-zone vpn to-zone trust
From zone: vpn, To zone: trust
Policy: VPN-Policy, State: enabled, Index: 9, Sequence number: 1
Source addresses: 172.31.0.0/16
Destination addresses: 192.168.0.0/16
Applications: any
Action: permit, application services, log
8. Have you classified traffic in the exempt rulebase, terminal match, or ignore action on the IDP
policy? If the IDP is not detecting the appropriate attacks, make sure that you have not configured these
sessions to match under the exempt rulebase, which will not log or match attacks based upon the
configuration. The ignore action will ignore all other attacks that are matched within that flow, so it will

  ƒ‰‡Ͷ͵‘ˆͷ͸

 App Note

not drop that connection, and will not futher inspect it, and finally, did you enable terminal match in the
IDP rulebase? Terminal rulebase will not evaluate rules any further when enabled.

Further Troubleshooting Steps

Currently the IDP doesn’t have as in depth debugging as the firewall flow processing, however development is
in the works to improve this. Below are some additional command output steps to verify that this is working
properly.

show security idp status

root@SRX210> show security idp status
Status of IDP: s0, Up since: 2009-08-11 16:34:11 UTC (2w2d 22:12 ago)
## Here we see that the IDP engine is up and running

Packets/second: 56 Peak: 400 @ 2009-08-14 19:11:35 UTC

KBits/second : 10456 Peak: 111129 @ 2009-08-14 19:11:40 UTC
Latency (microseconds): [min: 0] [max: 0] [avg: 0]
## above we can see how many packets the IDP engine is currently processing along with the
throughput. We also have information based on the maximum numbers and when they occurred.

Packet Statistics:
[ICMP: 0] [TCP: 15] [UDP: 0] [Other: 0]

Flow Statistics:
ICMP: [Current: 0] [Max: 0 @ 2009-08-11 16:34:11 UTC]
TCP: [Current: 15] [Max: 2 @ 2009-08-14 19:11:35 UTC]
UDP: [Current: 0] [Max: 0 @ 2009-08-11 16:34:11 UTC]
Other: [Current: 0] [Max: 0 @ 2009-08-11 16:34:11 UTC]
## Here we see the current breakdown of the IDP sessions on the platform based on protocol

Session Statistics:
[ICMP: 0] [TCP: 15] [UDP: 0] [Other: 0]
Policy Name : IDP-Policy v0
Running Detector Version : 9.2.160090324
## Finally, we see which policy is active and the active detector engine

The next example shows how to see which attack db version and detector engines are active.

show security idp security-package-version

  ƒ‰‡ͶͶ‘ˆͷ͸

 App Note

root@SRX210> show security idp security-package-version

Attack database version:N/A(N/A)
Detector version :9.2.160090324
Policy template version :N/A
## Here we see that we have a detector engine, but we have not installed the attack
database version. The lack of policy templates is not necessarily an issue in and of
itself unless you are trying to use policy templates. This output could also mean that
you haven’t installed the security package.

If your IDP is running, but you are experiencing performance issues, there are a few things that you’ll want to
look into. First, how many sessions/pps is the IDP engine processing? This can have an impact on
performance, and can be checked as show above with the “show security idp status” command. Next, you may
want to check what the memory utilization is of the IDP platform. We pre-allocate a certain amount of memory
for the IDP, but if it is highly utilized then this can cause performance issues. Lastly, having all signatures
enabled is not recommended if the platform is going to approach the performance limits of the box, as this can
severely impact processing. You can check the output of the memory with the command:

show security idp memory

root@SRX210> show security idp memory

IDP data plane memory statistics:

Total IDP data plane memory : 188 MB

Used : 7 MB ( 7168 KB )
Available : 181 MB ( 185344 KB )
## Here we see that we still have plenty of IDP memory available for processing

Tracing idpd

The IDPD process can be traced to gain some insight into the IDPD if there is a problem with it. However,
currently, there isn’t a good way to debug IDP flows, although development for such a capability is in progress.
If debugging IDP flows must be performed, this should be done by JTAC. The information located within the
IDPD debug is primarily around the downloads/updates to the IDP engine, as well as commit information.

set security idp traceoptions flag all

set security idp traceoptions file <filename>

Below is the output of enabling a trace and performing an update/commit

root@SRX210> show log idpd

  ƒ‰‡Ͷͷ‘ˆͷ͸

 App Note

Aug 14 19:07:05 idpd_config_read: called: check: 0

Aug 14 19:07:05 idpd commit in progres ...
## IDP update called from Attack Object update
Aug 14 19:07:05 Entering enable processing.
Aug 14 19:07:05 Enable value (default)
Aug 14 19:07:05 IDP processing default.
Aug 14 19:07:05 idp config knob set to (2)
Aug 14 19:07:05 Compiling policy IDP-Policy....
Aug 14 19:07:05 Apply policy configuration, policy ops bitmask = 41
Aug 14 19:07:06 Starting policy(IDP-Policy) compile...
Aug 14 19:07:10 policy compilation memory estimate: 9768
Aug 14 19:07:21 ...Passed
## We check to make sure that we have enough memory to perform this IDP policy
compilation, which we do.
Aug 14 19:07:21 Starting policy package...
Aug 14 19:07:26 ...Policy Packaging Passed
Aug 14 19:07:26 idpd_policy_apply_config idpd_policy_set_config()
Aug 14 19:07:26 Reading sensor config...
Aug 14 19:07:26 sensor/idp node does not exist, apply defaults
Aug 14 19:07:26 sensor conf saved
Aug 14 19:07:26 idpd_dev_add_ipc_connection called...
Aug 14 19:07:26 idpd_dev_add_ipc_connection: done.
Aug 14 19:07:26 idpd_policy_apply_config: IDP state (2) being set
Aug 14 19:07:26 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:26 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:26 Apply policy configuration, policy ops bitmask = 4
Aug 14 19:07:26 Starting policy load...
## We begin the policy compilation in this step, we unpack the current IDP policy (IDP-
Policy) and begin to update the policy.
Aug 14 19:07:26 Loading policy(/var/db/idpd/bins/IDP-Policy.bin.gz.v + /var/db/i
dpd/sec-repository/installed-detector/libidp-detector.so.tgz.v + /var/db/idpd/bi
ns/compressed_ai.bin)...
Aug 14 19:07:26 idpd_dev_add_ipc_connection called...
Aug 14 19:07:26 idpd_dev_add_ipc_connection: done.
Aug 14 19:07:31 idpd_policy_load: creating temp tar directory '/var/db/idpd//bin
s/2f238f1'
Aug 14 19:07:31 sc_policy_unpack_tgz: running addver cmd '/usr/bin/addver -r /va
r/db/idpd/sec-repository/installed-detector/libidp-detector.so.tgz.v /var/db/idp
d//bins/2f238f1/__temp.tgz'
## Unpack and update new detector engine

  ƒ‰‡Ͷ͸‘ˆͷ͸

 App Note

Aug 14 19:07:32 sc_policy_unpack_tgz: running tar cmd '/usr/bin/tar -C /var/db/i

dpd//bins/2f238f1 -xzf /var/db/idpd//bins/2f238f1/__temp.tgz'
Aug 14 19:07:36 idpd policy load: running cp cmd 'cp /var/db/idpd//bins/2f238f1/
detector4.so /var/db/idpd//bins/detector.so'
Aug 14 19:07:38 idpd_policy_load: running chmod cmd 'chmod 755 /var/db/idpd//bin
s/detector.so'
Aug 14 19:07:38 idpd_policy_load: running rm cmd 'rm -fr /var/db/idpd//bins/2f23
8f1'
Aug 14 19:07:40 idpd_policy_load: detector version: 9.2.160090324
## Begin to load the new IDP policy with the new detector engine
Aug 14 19:07:40 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:40 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:40 idp_policy_loader_command: sc_klibs_subs_policy_pre_compile() returned 0
(EOK)

Aug 14 19:07:40 idpd_policy_load: IDP_LOADER_POLICY_PRE_COMPILE returned EAGAIN,

retrying... after (5) secs
Aug 14 19:07:45 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:45 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:45 idp_policy_loader_command: sc_klibs_subs_policy_pre_compile() returned 0
(EOK)

Aug 14 19:07:45 idpd_policy_load: idp policy parser pre compile succeeded, after (1)
retries
Aug 14 19:07:45 idpd_policy_load: policy parser compile subs s0 name
/var/db/idpd/bins/IDP-Policy.bin.gz.v.1 buf 0x0 size 0zones 0xb14542 z_size 102 detector
/var/db/idpd//bins/detector.so ai_buf 0x0 ai_size 0 ai /var/db/idpd/bins/compressed_ai.bin
## Above we see that we successfully compiled the policy, if there is an error compiling
the policy then we would see such an event here.
Aug 14 19:07:45 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:45 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:47 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:47 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:47 idpd_policy_load: idp policy parser compile succeeded
Aug 14 19:07:47 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:47 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:47 idpd_policy_load: idp policy pre-install succeeded
Aug 14 19:07:47 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:47 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:48 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:48 idpd_comm_server_get_event:530: evDispatch OK

  ƒ‰‡Ͷ͹‘ˆͷ͸

 App Note

Aug 14 19:07:48 idpd_policy_load: idp policy install succeeded

Aug 14 19:07:48 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:48 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:48 idpd_policy_load: idp policy post-install succeeded
Aug 14 19:07:49 IDP policy[/var/db/idpd/bins/IDP-Policy.bin.gz.v] and
detector[/var/db/idpd/sec-repository/installed-detector/libidp-detector.so.tgz.v] loaded
successfully.
## IDP Policy install completed and loaded new detector engine
Aug 14 19:07:50 Applying sensor configuration
Aug 14 19:07:50 idpd_dev_add_ipc_connection called...
Aug 14 19:07:50 idpd_dev_add_ipc_connection: done.
## Link in IDP connection
Aug 14 19:07:50 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:50 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:50 idpd_comm_server_get_event:522: evGetNext got event.
Aug 14 19:07:50 idpd_comm_server_get_event:530: evDispatch OK
Aug 14 19:07:50
...idpd commit end
## IDP update/compilation/installation complete

Troubleshooting SRX High Availability

When working with Chassis cluster configurations the most common SRX high availability issues are due to
basic configuration or architectural issues, so we will start with the common clustering issues, followed by
various commands that can be used to check the HA state, then delve into the debugging facilities.

Establishing Chassis Cluster

1. Is chassis clustering enabled? Check the output of the “show chassis cluster status” to determine the
status of a chassis cluster. The output below shows a valid cluster:
root@SRX5800-1> show chassis cluster status
Cluster ID: 1
Node name Priority Status Preempt Manual failover

Redundancy group: 0 , Failover count: 1

node0 1 primary no no
node1 1 secondary no no

Redundancy group: 1 , Failover count: 1

node0 254 primary no no

  ƒ‰‡Ͷͺ‘ˆͷ͸

 App Note

node1 1 secondary no no

Here is the output of a node where chassis clustering is not enabled

root@SRX210> show chassis cluster status

error: Chassis cluster is not enabled.

2. Is there like hardware in both chassis members? In a chassis cluster environment you must have the
same hardware in both members. On the SRX 5k, it doesn’t strictly matter which slots are used for the
different cards (however recommended that they match for simplicity sake) so long as you have the
same number and type of cards in both chassis, while the SRX 3k does require the same hardware in
both clusters, in the same slots on both chassis.
3. Is the JUNOS version the same on both members? We require that each member run the same
version of JUNOS in Chassis Clustering. In 9.6 and above, with Low Impact Cluster Upgrades, we can
have RTO sync while the second platform is upgraded to the new version to minimize failover, however
we do not support running the two members indefinitely on different JUNOS versions today.
4. Have both nodes in the chassis cluster been rebooted? In order setup a chassis cluster, each chassis
cluster node must be rebooted (today, this may change in the future) before they can join the cluster. If
you do not reboot the nodes, then the member will not be activated. Also, if you RMA a device (or
routing engine) you will need to issue this command again on the replaced unit, since the setting is
stored in NVRAM and not in the configuration itself, when the device is replaced, it will not have this
setting.
5. Is the control link in the appropriate port on the SRX? On the SRX 3k the control port is fixed to
HA port 0, in 10.0 with dual control links supported (requires 2 RE’s) control port 1 will also be
supported. On the 3k, you can use copper or fiber SFP’s. On the SRX 5k, you can put the control link
ports on any SPC (preferably not the lowest SPC since that has the CP on it) however you must use
Fiber as the interface type, not a copper SFP. On the 5k, you must also configure which ports are used
for the control ports (this is not required on the 3k)
6. If using dual control links are you running the correct version, and do you have dual routing
engines? On the SRX High End, in 10.0 and beyond, we support dual control links, however you must
have two routing engines in each cluster member. The second RE is not used for backup routing today,
but is used to activate the control link port in the internal switch.
7. Is the data link properly configured on the SRX? Unlike ScreenOS, we require separate links for the
control and datalink, along with different connections to the control/dataplane. For the data fabric ports,
this is done by using a dataplane port (revenue port) for the dataplane. This must be configured
manually to specify which interface will be the dataplane port. If using active/passive, 1Gbps will be
more than enough for the function, 10Gbps has no advantage, however in Active/Active, if you are
going to have data arrive on an interface on one chassis member, and cross the datalink to exit an egress
interface on the other member, then a 10Gbps link should be used for this configuration to maximize the
throughput. The datalink must be established for the HA communication to be fully supported, since it
is responsible for synchronizing the real time objects to the other member.

  ƒ‰‡Ͷͻ‘ˆͷ͸

 App Note

8. If multiple SRX clusters exist on the same L2 broadcast domain, is the same cluster ID used? If
you have multiple SRX clusters on the same L2 broadcast domain, you must use different cluster ID
numbers, because the cluster ID is used to form the virtual MAC address that is used for the RETH
interface. Therefore if you use the same cluster ID then you will have a MAC address overlap and
forwarding problems will occur.

Failover Behavior

9. Have the appropriate redundancy groups been configured on the chassis with the appropriate
priorities? Redundancy Group 0 must be configured for the control plane, and the lower priority is
preferred over high priority. Redundancy Group 1 and higher are used for the dataplane. In order for
proper failover to occur, we must make sure that the appropriate priorities are configured for the
appropriate node.
10. Is preempt configured for redundancy groups? If you enable preempt (which can be done on a
redundancy group by redundancy group basis) the lower priority redundancy group will become the
active member if the other member of the redundancy group has a higher priority and is active. If
preempt is not enabled, then when a higher priority member becomes active (after being disabled) it will
not seize control of the redundancy group.
11. Is the redundant Ethernet configuration configured within the proper redundancy group, is that
group configured with the correct priority on the correct node? When using redundant Ethernet,
you must make sure that the redundant Ethernet is applied to the correct redundancy group (must be
RG1 or higher) and that the redundancy group has the appropriate priority values to be active on the
correct node. When using active/passive, the control plane is RG0, while the dataplane is RG1. In
Active/Active, you can have multiple redundancy groups (RG1, RG2, RG3 &c) which can be active on
different members, controlled by the priority setting for the nodes.
12. After a control link failure, was a reboot performed on the disabled node to reactivate it in the
cluster? Unless “control-link-recovery” is enabled, you will need to manually reboot the disabled node
for it to become the secondary node in the cluster. If you do not reboot the disabled member, then it will
remain in the disabled state.
13. After a data link failure, was a reboot performed on the disabled node to reactivate it in the
cluster? If a data link failure occurs, then a reboot must be performed on the disabled member in order
for it to become active again. If you do not do a reboot, then the disabled member will not be able to
become active. There is no command at this point to automatically reboot when a datalink failure
occurs.
14. After a manual failover of a redundancy group, was the manual failover flag cleared? When you
use the command “request chassis cluster failover <redundancy-group> node <new master node>
command to failover a redundancy group, you must clear the failover with the command “request
chassis cluster failover “request chassis cluster failover reset redundancy-group <redundancy-group>”
15. Is the feature you are trying to use supported in High Availability clusters? Today there are many
features that are only supported in standalone mode and not in HA. These features will change over
time, and this document is not the place to review all the features that are or are not enabled in HA,

  ƒ‰‡ͷͲ‘ˆͷ͸

 App Note

however, be cognizant that some features may not be enabled in HA, so check with the documentation,
JTAC, or PLM if you run into an issue.

Further Troubleshooting Steps

It is not recommended that you enable Chassis Cluster debugs without the assistance of JTAC as these can have
performance and stability impacting results if you are not careful. There are some basic troubleshooting steps
that you can perform (in addition to what is mentioned above.)

1. Check the status of show chassis cluster status which will display what the current status of the chassis:

root@SRX5800-1> show chassis cluster status

Cluster ID: 1
Node name Priority Status Preempt Manual failover

Redundancy group: 0 , Failover count: 1

node0 1 primary no no
node1 1 secondary no no

Redundancy group: 1 , Failover count: 1

node0 254 primary no no
node1 1 secondary no no
2. Check what the status of the participating physical and logical interfaces are. You can do this by using
the commands “show interfaces <interface> terse” as well as “show chassis cluster interface”

root@SRX5800-1> show interfaces terse

Interface Admin Link Proto Local Remote
gr-0/0/0 up down
ip-0/0/0 up down
mt-0/0/0 up down
pd-0/0/0 up down
pe-0/0/0 up down
ge-11/0/0 up up
ge-11/0/0.0 up up inet 200.200.200.1/24
multiservice
ge-11/0/1 up up

  ƒ‰‡ͷͳ‘ˆͷ͸

 App Note

root@SRX5800-1> show chassis cluster interfaces

Control link name: em0

Redundant-ethernet Information:
Name Status Redundancy-group
reth0 Down 1
reth1 Down 1
reth2 Down 1

3. Check the status of the control plane to see if you are receiving heartbeats and messages (for both
control and data links) using the command “show chassis cluster control-plane statistics”

root@SRX5800-1> show chassis cluster control-plane statistics

Control link statistics:
Heartbeat packets sent: 692386
Heartbeat packets received: 692352
Fabric link statistics:
Probes sent: 692381
Probes received: 692100

4. Lastly, there are two files which you may want to examine to see if there is some sort of hardware
failure. The first is “show log messages” which will contain lots of general purpose log messages, as
well as the “show log chassisd” logs to determine if there are any other hardware chassis failures.

Advanced Insight Solutions

One of the best ways to troubleshoot JUNOS products when issues occur is to use the Advanced Insight
Solutions application which utilizes JUNOScript to initiate event-scripts which will automatically gather all of
the necessary information for JTAC to troubleshoot the issue. While this feature will not troubleshoot issues
like policy or VPN troubleshooting, it is very good at identifying failures, potential failures (such as memory
leaks which may impact other components.) Since this automates the process to collect the information, it
makes it much faster to resolve issues, with less back and forth communication between JTAC and end
customers. Also, Advanced Insight Solutions does not require any knowledge of JUNOScript, so it makes it
very easy to install. Advanced Insight Solutions is available to any JUNOS customer, and is free (unless the
customer is using advanced features such automatically opening JTAC cases. Eventually this will be
implemented into JNAP/JMP. While this guide is not going into the actual features below is a sample
configuration for the AIM on the SRX, with explanation of the SRX configuration. See the AIS configuration
guide for installing the AIM application on a server.

  ƒ‰‡ͷʹ‘ˆͷ͸

 App Note

Step 1: Transfer the JAIS package to the SRX. This can be done with SCP, FTP, or other file transfer.
Step 2: Issue the “request system scripts add <JAIS package>” operational mode command to install the
JUNOScripts into the system.

Step 3: Under the configuration mode, configure the AIM settings on the device:

set groups juniper-ais system scripts commit allow-transients

set groups juniper-ais system scripts commit file jais-activate-scripts.slax optional
## Define the script that activates the AIS commit scripts

set groups juniper-ais event-options destinations juniper-aim archive-sites

"scp://[email protected]:/var/aim/device" password "$9$xKX-bYJGifT3goT369OBxNd"

## The above statement defines that the location that we should transfer the AIM Incident
Messages to. In this example, we are using SCP as the transfer protocol, but we can use
any supported protocol. Next, “aim” is the user that we use to authenticate to the system
with, where aim is a linux user on that system. 172.19.101.103 is the hostname of the AIM
server, and where /var/aim/device is the folder that we transfer the messages to. This is
the directory where AIM looks for updated messages from the monitored hosts. Lastly, the
password is the password used to authenticate AIM to the system.

set apply-groups juniper-ais

## Apply the juniper-ais configuration

  ƒ‰‡ͷ͵‘ˆͷ͸