CONFIGURATION................... 5 COMMANDS............................ 6 LOG FILES ............................ 12 DATABASE ........................... 16 PROCESSES......................... 18 DATA COLLECTION............. 18 TOOLS................................... 21 ISSUES .................................. 27 APPENDIX.............................

40

NetXplorer and NetEnforcer x7.x.x System and Troubleshooting Guide

NetXplorer NX7.x.x and NetEnforcer S/E7.x.x This document describes the system and troubleshooting techniques for the following products: NetXplorer Software Version NX7.x.x NetEnforcer Software Version S7.x.x NetEnforcer Software Version E7.x.x

Customer Support Only

Confidentiality Notice
This document contains Proprietary Trade Secrets of Allot Communications LTD and its receipt or possession does not convey any right to reproduce, disclose its contents or to manufacture, use or sell anything that it may describe. Reproduction, disclosure or use without specific authorization from Allot Communications is forbidden. Allot reserves the right to make changes, add, remove or change the schedule of any element of the plan.

Document Version: 7.3 Date: 25-JUN-07

NetXplorer and NetEnforcer Troubleshooting Guide

Table of Contents
CONFIGURATION ..................................................................................................................................... 5 PORTS.......................................................................................................................................................... 5 NetXplorer Client and Server ................................................................................................................. 5 NetXplorer Server to NetEnforcer .......................................................................................................... 5 Additional ............................................................................................................................................... 5 ACCESSING SYBASE .................................................................................................................................... 5 Problems Uninstalling Sybase................................................................................................................ 6 COMMANDS................................................................................................................................................ 6 NETENFORCER ............................................................................................................................................ 6 OTHER NETENFORCER TOOLS..................................................................................................................... 7 ACSTAT ....................................................................................................................................................... 7 NICSTAT ...................................................................................................................................................... 8 ACTHRUPUT ................................................................................................................................................. 8 ACMODE ...................................................................................................................................................... 9 ACMON ........................................................................................................................................................ 9 HWADMIN ................................................................................................................................................. 10 LINKADMIN .............................................................................................................................................. 10 GO CONFIG NIC .......................................................................................................................................... 11 LOG FILES................................................................................................................................................. 12 NETXPLORER SERVER ............................................................................................................................... 12 C:\Allot\bin........................................................................................................................................... 12 C:\Allot\log........................................................................................................................................... 12 C:\Allot\conf ......................................................................................................................................... 13 C:\Allot\netxplorer\jboss-3.2.6\server\allot\log ................................................................................... 13 C:\Allot\netxplorer\jboss-3.2.6\server\allot\deploy.............................................................................. 14 C:\Allot\netxplorer\jboss-3.2.6\server\allot\conf.................................................................................. 14 NETXPLORER CLIENT ............................................................................................................................... 14 C:\Documents and Settings\<user name>............................................................................................ 14 NETENFORCER .......................................................................................................................................... 14 $SWGL.................................................................................................................................................. 14 /tmp/...................................................................................................................................................... 15 /var/log/apache..................................................................................................................................... 15 $SWGC ................................................................................................................................................. 16 DATABASE ................................................................................................................................................ 16 NETENFORCER .......................................................................................................................................... 16 $SWGD................................................................................................................................................. 16 $SWGD/data......................................................................................................................................... 17 NETXPLORER ............................................................................................................................................ 17 C:\Allot\data\db.................................................................................................................................... 17 Performing a Backup............................................................................................................................ 17 PROCESSES............................................................................................................................................... 18 NETENFORCER .......................................................................................................................................... 18 NETXPLORER ............................................................................................................................................ 18 DATA COLLECTION............................................................................................................................... 18 NETENFORCER .......................................................................................................................................... 18 $SWGE/httpd/htdocs/bucket ................................................................................................................. 18

www.allot.com

NetXplorer and NetEnforcer Troubleshooting Guide

$SWGE/httpd/htdocs/bucket/30 (same content for 300) ....................................................................... 18 Understanding the Manifest ................................................................................................................. 19 NETXPLORER ............................................................................................................................................ 19 C:\Allot\data\bucket\stc\<device ID> .................................................................................................. 19 C:\Allot\data\bucket\ltc_export\ ........................................................................................................... 20 C:\Allot\data\bucket\ltc_export\<device ID>....................................................................................... 20 Allot/data/bucket/ltc/device_ID ............................................................................................................ 20 TOOLS ........................................................................................................................................................ 21 Upgrading NX Server Version.............................................................................................................. 21 Enabling Compression ......................................................................................................................... 21 CHANGE ADMIN PASSWORD ..................................................................................................................... 22 MANAGING REPORTING DATABASES ........................................................................................................ 22 Recreating Default (ST and LT) Databases.......................................................................................... 22 Improving Database Performance ....................................................................................................... 22 CHANGING REPORTING DATABASE PROFILES ........................................................................................... 23 Changing LT Reduction Profile............................................................................................................ 23 Changing ST Profile Options ............................................................................................................... 23 CHANGING REPORTING DATABASE PARAMETERS ..................................................................................... 24 Disabling External Hosts Reporting..................................................................................................... 24 INCREASING THE NUMBER OF BUCKETS SENT PER TIME SLICE ................................................................... 24 Changing number of buckets in the NetEnforcer..................................... Error! Bookmark not defined. Changing number of buckets in the NetXplorer ................................................................................... 25 ENABLING TAP MODE .............................................................................................................................. 25 PORT MIRROR ........................................................................................................................................... 26 STEP 1 ...................................................................................................................................................... 26 STEP 2 ...................................................................................................................................................... 26 ISSUES ........................................................................................................................................................ 27 NTP/TIME ISSUES ...................................................................................................................................... 27 Synchronization issues between Client and Server............................................................................... 27 Synchronization issues between Server and NetEnforcer..................................................................... 27 Problem: GUI does not start ................................................................................................................ 29 CREATING A SNAPSHOT............................................................................................................................. 29 NetXplorer ............................................................................................................................................ 29 NetEnforcer .......................................................................................................................................... 29 TAKING A SNAPSHOT ................................................................................................................................ 29 The Manual Snapshot ........................................................................................................................... 29 The Automatic Snapshot ....................................................................................................................... 30 Sending the Snapshot............................................................................................................................ 30 HTTP SNAPSHOT ...................................................................................................................................... 30 ADD DEVICE ............................................................................................................................................. 32 CHANGE IP................................................................................................................................................ 34 Defined Behavior.................................................................................................................................. 34 Current Behavior.................................................................................................................................. 35 In-Band/Out of Band Definitions.......................................................................................................... 35 PROVISIONING CHANGES .......................................................................................................................... 36 Add Host ............................................................................................................................................... 36 CONFIGURATION CHANGES ....................................................................................................................... 36 Process ................................................................................................................................................. 36 Troubleshooting.................................................................................................................................... 36 DATABASES NOT SYNCHRONIZED ............................................................................................................. 37 Symptoms.............................................................................................................................................. 37 Explanation .......................................................................................................................................... 37 Troubleshooting.................................................................................................................................... 37 To Generate a Full Export.................................................................................................................... 37

www.allot.com

NetXplorer and NetEnforcer Troubleshooting Guide

RMA/BOX REPLACEMENT ........................................................................................................................ 38 COLLECTION PROBLEMS ........................................................................................................................... 38 STC Problems Related to Software....................................................................................................... 38 Data Collection Stops Due to NTP Issues ............................................................................................ 39 DEMO INSTALLATION ISSUES .................................................................................................................... 39 Installing NetEnforcer version 7.1.0 on a NetEnforcer AC-202/302.................................................... 39 Skipping installation hardware requirements....................................................................................... 39 APPENDIX ................................................................................................................................................. 40 APPENDIX I ............................................................................................................................................... 40 Host output from $SWGL/nedbg.DataSrv.log ...................................................................................... 40 APPENDIX II .............................................................................................................................................. 42 Host output from $SWGL/nedbg.AllSnmpAgent.log............................................................................. 42

www.allot.com

NetXplorer and NetEnforcer Troubleshooting Guide

Configuration
Ports
NetXplorer Client and Server
Port Number TCP:80 Description HTTP for initial access to Server. Once applet is downloaded, this is not required RMI (Java J2EE protocol) RMI (Java J2EE protocol) JNP (Java J2EE protocol)

TCP:1099 TCP:4444 TCP:1098

NetXplorer Server to NetEnforcer

Port Number TCP:80 UDP:161 UDP:161 UDP:123 TCP:123 Description Data sampling SNMP Configuration updates SNMP Traps (Events) NTP NTP

Additional
Port Number TCP:50000 TCP:50001 TCP:50002 Description For troubleshooting access to configuration database on NetXplorer Server For troubleshooting access to short term database on NetXplorer Server For troubleshooting access to long term database on NetXplorer Server

Accessing Sybase
Database access on the Server may be required in order to troubleshoot certain issues, regarding configuration, data accuracy, data collection (and many more). To access the database, Sybase Central must be installed on the local PC. This can be downloaded from ftp://support:[email protected]/Sybase.

To access the database open Sybase Central and perform the following: 1. Right click on ASA9 2. Select new connection

www.allot.com

NetXplorer and NetEnforcer Troubleshooting Guide

3. Enter user details under the Identification tab a. ID nms b. Password allot 4. Enter database (location and database) under the Database tab a. Localhost:db_port - if database resides on local server b. IP:db_port if database resides on different server (need to ensure access to specific server, i.e. firewall issues etc.) It is possible to open all databases simultaneously.

Problems Uninstalling Sybase

At times, the uninstall procedure does not completely uninstall the Sybase application. Deleting the Allot directory and registry entries still does not complete the uninstall process. If this is the case, go to the Environmental Variables and delete the reference to the Allot folder. This will complete the uninstall process. The environmental variables can be accessed as follows: Right click on My Computer and select Properties. Click on the Advanced Tab and then click on the Environment Variables button. Under System Variables at the bottom are various entries which will show the Allot folder as the value. For additional information on uninstalling Sybase, please see KB item #6976.

Commands
NetEnforcer
acstat acthruput clientTest clientTest is an application used to get statistical data on the box (client) sent to the server. Usage: clientTest -s <statistic type 5-lines;6-pipes;0-vcs;1-conv;7-ne;> -t <time interval> (30/300 seconds) -v <specific vcs id separated with space (Max 10)> -p <specific pipes id separated with space (Max 10)> -l <specific lines id separated with space (Max 10)> Example (for VC statistics every 30 seconds): clienttest s 0 t 30 Output (for VC statistics): [Output can be found in the nedbg.clienttest.log file]
03-14 15:27:02(201) <DL_USER1>: StatisticClient::handleNewSample, dataLen:126, sampleObject:Collection id:270195024 StartTime:1142342814 EndTime:1142342822 Number of slices:24 Number of rows:1 Schema: SM_LINE_ID(1),SM_PIPE_ID(2),SM_PIPE_INST(3),SM_VC_ID(4),SM_VC_INST(5),SM_DIVIDED_BYTES _IN(19),SM_DIVIDED_BYTES_OUT(20),SM_PACKETS_IN(15),SM_PACKETS_OUT(16),SM_LIVE_CONNECTI ONS(13),SM_NEW_CONNECTIONS(12),SM_DROPPED_CONN(14),

swgadmin l Output:
lcd 175 DataSrv 176 SessionDispatcher 9286 coll 180 StatisticMgr 181 AllSnmpAgent 182

go config view (see CLI document for full list of CLI commands)

www.allot.com

NetXplorer and NetEnforcer Troubleshooting Guide

Other NetEnforcer Tools acstat

acstat shows information about the current connections running through the NetEnforcer.
Usage: acstat [ [ [ [ -l -t -u -a -n -c -r -i -s -f -F -x -m -N -B -R -I -A and dst ip -l -t -s -I {session/vc/pipe/h} ] / -u /-a / -n / -c / -r / -i ] ] [ -f ] [ -F ] [-x ] [ -m <max_sessions> ] [ -N ] [ -B ] [ -R <file> ] <pipe_id>,<vc_id>] [ -A <src_ip_addr>,<dst_ip_addr>]

<LIST_TYPE> : List session/vc/pipe/hierarchy [session] : display TCP connections : display UDP connections : display any IP connections (other than TCP and UDP) : display non IP connections : display ICMP connections : display ARP connections : display all connections : display connection allocation summary (single option, default) : display extended view : display extended view - advanced : display internal/external (instead of client/server) <NUMBER> : display up to NUMBER of sessions (max 500k) : don't resolve names : dump binary data to file <FILE> : read binary data from FILE (single option) <PIPE>,<VC> : display hierarchy all connections of pipe and vc (zero means all) <SRC IP ADDR>,<DST IP ADDR> : display connections of specific src ip address address (zero means all)

acstat with no flags shows connection allocation summary Sessions are represented in the following format: Protocol Client Server State VC Client IF Protocol Client Server State

TTL

VLAN Tag

Tos

St

VC (Virtual Channel) Client IF TTL (Time to live) VLAN Tag ToS St (Session Status)

Name of the protocol. If the name is unknown, the hexadecimal number of the protocol is shown. Raw TCP is shown as TCP-r. IP of the host which initiated the session (for TCP and UDP sessions also the port). IP of the host to which the client send its request (for TCP and UDP sessions - also the port). Prisma Session State. Can be one of the following: OPENED, CONNECTED, WIRED, TO BE CLOSED, CLOSED, REJECTED, DROP or NONALLOCATED (the last one should never appear; if it does, there is probably a bug). If the client-server and the server-client sides of a session are in different states, both states are shown, e.g. WI-2b for WIRED - TO BE CLOSED. VC to which the session belongs. If the client-server and the serverclient sides of a session belong to different VCs, both VCs are shown. NetEnforcer interface that the client is connected to. Time left (in seconds) until the session expires if no traffic arrives. Indicates if the connection is VLAN tagged and to which VLAN. ToS marked value. If the number displayed is 0, then there are no ToS markings on the packets. Possible options are Raw, Half, Dbl, Chng, Loop or NA. Raw indicates if the session is raw i.e. the connection was classified after it had been established. Dbl indicates a double session.

www.allot.com

NetXplorer and NetEnforcer Troubleshooting Guide

nicstat
Displays the mode and speed of network interfaces. It is not the speed and duplex defined in the GUI configuration, but the actual values. The command is used for troubleshooting access links related problems and for verification that the links are compatible with the adjacent router or switch. Certain networking related problems are coming from NICs definitions that are mis-configured. Checking the nicstat and comparing it to the router/switch definition is a useful tool in troubleshooting problems like packet loss, synchronization and network slowness issues.
Command nicstat Output

+-----------+------+-------+--------+ | Interface | Link | Speed | Duplex | +-----------+------+-------+--------+ | eth0 | up | 10 | half | | eth1 | down | n/a | n/a | | eth2 | up | 10 | half | +-----------+------+-------+--------+

acthruput
Prints the amount of bits that have passed through each Interface, active pipe and active VC during one time slice (one second). The output of the command shows the bandwidth consumption of each of the active pipes/vcs and for the entire interface. It can be used also to determine if theres a need to alter the bandwidth definition of the pipe/vc and to troubleshoot bandwidth and traffic related problems. Usage: acthruput [ -b ] [ -B ] [ -c ] [ -v ] [ -d DIR ] -b : display throughput in bits (default) -B : display throughput in bytes -c : display throughput per connection -t : display total link throughput including IgnoreQoS -d DIR : analyze data in DIR instead of / e.g. acthruput -d $W/stat/last - to analyze the last snapshot

Command acthruput

Output --------------------------------------------------------Entity Name Bits/sec --------------------------------------------------------INTERFACE Internal 0 --------------------------------------------------------INTERFACE External 2896 PIPE 1 1024 VC 8 512 VC 1 512

Note: The actrhuput command should only be used for AC-x0x devices. For AC-1000 devices, please use the acmon command (see next page).

www.allot.com

NetXplorer and NetEnforcer Troubleshooting Guide

acmode
Switches between various NetEnforcer software modes. Shows, saves and restores modes and makes the NetEnforcer enter/exit software or hardware bypass. Examples: enable/disable QoS, TCP, UDP, etc.
acmode [ [ [ [ +/-endvcs ] [ +/-srcmac ] [ +/-ignoremom ] +/-verbose ] [ +/-mtu ] [ +/-noweight ] [ +/-novc ] [ +/-wnyfast ] save ] [ restore ] [ default ] [ show ] hwbp ]

+endvcs - enable ended vcs -endvcs - disable ended vcs +srcmac - enable source mac handling -srcmac - disable source mac handling +ignoremom - enable ignore monitoring only mode on dkm -ignoremom - disable ignore monitoring only mode on dkm +verbose - enable dkm verbose -verbose - disable dkm verbose +mtu - enable Check and Fragment IP packet according to MTU size -mtu - disable Check and Fragment IP packet according to MTU size +noweight - enable counting traffic with Ignore QoS Policy for monitoring/accounting purposes -noweight - disable counting traffic with Ignore QoS Policy for monitoring/accounting purposes +novc - enable counting traffic that passes through NE prior to policy assignment -novc - disable counting traffic that passes through NE prior to policy assignment +wnyfast - enable winny fast identify method -wnyfast - disable winny fast identify method (default) save restore default show hwbp save current settings restore saved settings restore default settings show current settings go into hardware bypass

Note: you can run acmode with a number of arguments, e.g. acmode +qos -tcp. The arguments are processed one by one in the order of appearance, with two exceptions: - hwbp (go into hardware bypass) is processed last.

acmon
Used to get statistics (ONLY for AC-1000 units).
Usage: acmon { -p <pipe id> / -v <vc id> / -s <service id> / -d [ -t <seconds> ] -p -v -s -d -l -r -t <PIPE> <VC> <SERVICE> : : : : : / -r / -l <count>}

<SECONDS>

monitor specific pipe rate monitor specific vc rate monitor specific service rate monitor dmu packet distribution run acmon limited count number : monitor octet rx : time to wait between samples in seconds [1 seconds]

Example:
[i ] 10:10:02 >> 0 conn ps [0] rate inbound: 0.000 bps outbound: 0.000 bps [1] rate inbound: 202.772 Kbps outbound: 0.000 bps

www.allot.com

NetXplorer and NetEnforcer Troubleshooting Guide

HwAdmin
Controls the bypass mechanism. This command can be used to send the box to hardware bypass. Usage: HwAdmin -s -H

: displays system status : displays hardware (AC, MACH, FULL, OEM) version information.
Output Status register = 0x3 Local machine is STAND_ALONE and in ACTIVE mode Local bypass is CONNECTED Remote machine not detected Hardware version - 402 Firmware version - 2 OEM version 0

Command HwAdmin s

HwAdmin -H

LinkAdmin
Changing the NIC configuration on the AC-X02 and AC-1000 series: LinkAdmin will give you various options: LinkAdmin -[dsuc] <interface name> -c <interface number> [autoneg on|off] [speed 10|100|1000] [duplex half|full] -d - link down -u - link up -s - show link status -f - show supported link speed and duplex optional interface name eth1 eth0 nic1 nic0 etc. If we want to set the internal interface to full 100, you can use either of the commands: LinkAdmin 0 autoneg off speed 100 duplex full LinkAdmin -c 0 autoneg off speed 100 duplex full LinkAdmin -c eth0 autoneg off speed 100 duplex full The command needs to be followed by a reboot. Please note that these commands are for the AC-X02 and AC-1000 only.

www.allot.com

10

NetXplorer and NetEnforcer Troubleshooting Guide

go config nic
The NIC settings on the NetEnforcer AC-404, AC-804, and AC-808 can be configured using the go config nic CLI command.
AC:~# go config nic Command: go config nic Usage: go config nic {<Label:Mode:Speed[:FailureAction]>,...} Acceptable Labels are: INTERNAL1, EXTERNAL1, MGMNT, INTERNAL2, and EXTERNAL2 Acceptable values of Mode are: half, full, and auto Acceptable values of Speed are: 10, 100, 1000, and auto (according to box type) Acceptable values of Failure Action are: none, fail_pair, fail_all, and bypass

Example: go config nic INTERNAL1:full:100:fail_pair Important Note: The AC-404 does not support 1000Mbps speed, although it is possible to run the go config nic command with 1000Mbps as a speed value. Labels: For the AC-808, the acceptable labels are: INTERNAL1, EXTERNAL1, INTERNAL2, EXTERNAL2, MGMNT, INTERNAL3, EXTERNAL3, INTERNAL4, and EXTERNAL4

Acceptable value of Speed: 1000 - the interfaces are capable of working with 1 Gbps physically (be connected to 1Gbps interfaces). All of the AC-808 interfaces support 1000 Gbps physical speed.
Values: Acceptable values of Failure Action: fail_pair: if one interface within a pair (INTERNAL x - EXTERNAL x) is down, the system will disable its peer. fail_all: if one interface is down, the system will disable all other interfaces. bypass: : if one interface is down, the system will move to bypass. Management port As of version 7.1.0 build 24, only the management port can be configured via the admin menu. The AC-80x (the new AC-802 platform, AC-804, and AC-808) management port supports 10/100/1000 (physical speed).

Speed:

www.allot.com

11

NetXplorer and NetEnforcer Troubleshooting Guide

Log Files
NetXplorer Server
All logs are stored under Allot\. This is usually located under C:\.

C:\Allot\bin
All batch and executable files are located here, including all processes (e.g. poller, keeper). File Name Create_snapshot_logs.bat Start_<db name>.bat Stop_<db_name>.bat reduction_profile_upd.bat check_<db name>_db.bat check_db.bat conf_assist.exe Explanation Snapshot generator Batch file initializing specified database Batch file stopping specified database Batch file that copies selected reduction cfg file from \allot\conf\Reduction to \allot\conf Checks if specific database (CFG, STC, LTC) alive mechanism used check_db.bat file Check database alive mechanism Prepare database password for \allot\conf stc_collect.cfg and \allot\conf ltc_collect.cfg files (Not in use for users) Used for Sybase install ,database create and recreate

db_install.exe

C:\Allot\log
File Name poller.log converter.log loader.log ltc_poller.log ltc_loader.log keeper.log allot_<db name>.txt allot_<db name>_stop.txt Explanation Poller log Converter log Loader log Long Term Poller (lt_poller) log Long Term Loader (lt_loader) log Keeper Server log file Database work process log file Database stop process log file

www.allot.com

12

NetXplorer and NetEnforcer Troubleshooting Guide

C:\Allot\conf
File Name nedbg.conf reduction.cfg stc_collect.cfg ltc_collect.cfg hosts.cfg Reduction MIB XML db swkeeper.ini static.ini directory directory directory directory file file Explanation Configuration file for keeperServer.exe and LTreducer.exe Configuration file for reduction process used by LTreducer.exe Configuration file for stc collector processes (poller, converter, loader, manifest_manager) Configuration file for ltc collector processes (ltc_poller, ltc_loader) Hosts list used by LTreducer.exe Optional reduction configurations MIB files for MIB modules supported by the agent XML schemas for interfacing with the agent Data files for static loading of certain tables Process and database initialization file including log level configuration (similar to swgrun.ini on the NetEnforcer) Database parameters and ports

C:\Allot\netxplorer\jboss-3.2.6\server\allot\log
File Name NMS.log Explanation Application Server log. Example messages: [EAR Deployment] Init J2EE application:. Implication: application loading Subsequent messages: loading of each module [NamingService] Started jndi bootstat1099 Implication: connecting to server Note: this port must be open otherwise system will not load [RARMetaData] Loading Jboss Resource Adapter Implication: loading connection to database (will appear after above message) Subsequent messages: loading of each module, look out for [Deploy] messages. Stacked traces indicate problems Older versions of nms.log (can be up to 40 before original one is overwritten) Jboss log Jboss log Jboss log including some application server exceptions

NMS.log.n boot.log jsr77.log server.log

www.allot.com

13

NetXplorer and NetEnforcer Troubleshooting Guide

C:\Allot\netxplorer\jboss-3.2.6\server\allot\deploy
File Name NMS.ear sybase-ds.xml Explanation This is the NetXplorer software application. A software upgrade can theoretically be performed by replacing this file. Contains configuration (allot_cfg) database and password

C:\Allot\netxplorer\jboss-3.2.6\server\allot\conf
File Name log4j.xml Explanation Contains configuration parameters for NMS.log including debug level and number of instances of log file. o maxfilesize - log size o maxbackupindex - max number of logs

NetXplorer Client
C:\Documents and Settings\<user name>
File Name NMS.log Explanation Application client log. The contents of this file are not the same as NMS.log located on the Server.

NetEnforcer
$SWGL
File Name ac_reboot.log badCCBs bt coll_dump counters.swg dbchanges.swg dkmdump errorlog.swg hwu.HwAdmin.log hwu.lcd.log kpc.SessionDispatch.log log.SWG Explanation Log of ac_reboot command Not in use. Directory that contains all backtrace files. Various counters from collector process that can be printed upon user request. nedbg.keeper.log takes information from this file. Policy changes accepted by DKM. Various counters from DKM process that can be printed upon user request. DKM log HwAdmin utility log LCD log Log created by every process that uses the KPC library (IPC between user and kernel) Obsolete - not used.

www.allot.com

14

NetXplorer and NetEnforcer Troubleshooting Guide

nedbg.acstat.log nedbg.AllSnmpAgent.log nedbg.AllSnmpAgent.log.old nedbg.Collector.log

Log of acstat process Log of SNMP agent/process (communication between Server and NetEnforcer) Old SNMP log Log of Collector process Log of DataSrv process. Issues with applying database changes and changes applied logged. In debug mode, this shows complete database update including XML command received from server, changed performed, counter ID updated and ok sent to Server. Obsolete not used. CLI log Log of Keeper (hardware keeper)process Log of lcd process Log of StatisticMgr (Statistics Manager) process. Problems with buckets will be logged. Log of swKeeper (software keeper) process Old log of nedbg.swKeeper.log Log of last installation process DoS attack reported by DKM Log of ntp process. Can identify problems with NTP synchronization. Various counters from Stat Mgr process that can be printed upon user request.

nedbg.DataSrv.log nedbg.default.log nedbg.go.log nedbg.keeper.log nedbg.lcd.log nedbg.StatisticMgr.log nedbg.swKeeper.log nedbg.swKeeper.log.old ne-instl.<date>.log notice.SWG ntp.log StatisticMgr_dump

/tmp/
File Name nedbg.ProvisionCli.log Explanation check whether content was received from the Apache Server View full XML content

/var/log/apache
File Name access_log Explanation check whether Apache received change Look for POST to ProvisionCli.exe

www.allot.com

15

NetXplorer and NetEnforcer Troubleshooting Guide

$SWGC
File Name reduction.conf SNMP actype addnsParameters dataCli.conf dkm.conf hosts.conf keeper.ini lcd_version memwatch.conf nedbg.conf provisioncli.conf reduction.conf Reduction.* statisticmgr_boot_counter swKeeper.ini Type File Directory File File File File File File File File File File Link to file File File File NetEnforcer version and type DNS refreshment parameters Internal config file dkm and prisma configuration parameters List of hosts referred to during the reduction of statistic data. HWKeeper ini file managing initialization parameters of all modules controlled by the HW Keeper Displays lcd version Memory consumption levels indicated memory issues Debug level of all nedbg log files Internal config file. Link to selected reduction configuration file All optional reduction configuration files Counter of restarts of statistic manager process. SWKeeper ini file managing initialization parameters of all processes controlled by Keeper Explanation Short Term reduction configuration parameters

Database
NetEnforcer
$SWGD
Name backup data schema lastSnmpUpdate Type directory directory directory file Explanation Location of most recent successful policy update (schema and data directories and their content) Location of policy and configuration database Location of policy and configuration database schema Maintains timestamp of last policy update received by SNMP. Used to report on synchronization status of device against the server.

www.allot.com

16

NetXplorer and NetEnforcer Troubleshooting Guide

$SWGD/data
Name allotConfig.xml Explanation Database of NetEnforcer configuration parameters. Including: device capabilities (modes), registration parameters, device limits (e.g. Lines, VCs, Pipes, bandwidth), data collection and reduction parameters. Network parameters are not included in this file. Policy and Catalog database. This is one file including all of the Catalog definitions and the Policy configuration. Maintains timestamp of the last full policy export to the device. Used to report on synchronization status of device against the server. Maintains timestamp of last policy update distributed by data server to internal clients. Used to report on synchronization status of device against the server.

allotProvision.xml lastPolicyFullExport lastPolicyUpdate

NetXplorer
C:\Allot\data\db
Name cfg ltc stc Type directory directory directory Explanation Location of configuration database, allot_cfg.db Location of long term data database, allot_ltc.db Location of short term data database, allot_stc.db

Performing a Backup
Please note that there are two kinds of database backups for the NX server. Cold backup done when services can be stopped. Hot backup done when services are running. Cold backup 1. Stop NetXplorer Service by going to Windows Services and stopping NetXplorer Server. 2. The following lines should appear in the allot_ltc.txt and allot_stc.txt files: Disable all events End of current events 3. Backup the database by copying the following folder: c:\Allot\data\db to a different location, preferably a different disk. 4. Start the NetXplorer Service. Hot backup In order to perform a hot backup, please see KB item 6269: "NetXplorer Backup and Restore Database". Please note that this should only be given to customers in exceptional cases.

www.allot.com

17

NetXplorer and NetEnforcer Troubleshooting Guide

Processes
NetEnforcer
There are several processes that should always be running on the NetEnforcer. These processes can be identified using several different commands, as follows: swgadmin -l lcd DataSrv SessionDispatcher coll StatisticMgr AllSnmpAgent ps awx|grep ntp or ntpq p (or use ps-ax) ntp client HTTP

NetXplorer
There are several processes that should be running on the NetXplorer Server. These processes can be identified using several different tools: Windows Services (Start>Control Panel>Administrative Tools>Services) o NetXplorer Server Windows Task Manager (CTRL+ALT+DEL and select Task Manager) o Poller.exe o Converter.exe o Loader.exe o ltc_poller.exe o ltc_Loader.exe o ltreducer (runs periodically therefore may not be seen) o manifest_manager.exe (runs periodically therefore may not be seen) o KeeperService.exe o Dbsrv9.exe (3 instances) o ntpd.exe

Data Collection
NetEnforcer
$SWGE/httpd/htdocs/bucket
Name 30 300 Type directory directory Explanation Location of 30 seconds buckets data Location of 300 (5 minutes) second buckets data

$SWGE/httpd/htdocs/bucket/30 (same content for 300)

Name conv_stat Type directory Explanation Location of conversation buckets (binary format)

www.allot.com

18

NetXplorer and NetEnforcer Troubleshooting Guide

vc_stat line_burst pipe_burst vc_burst manifest manifest<n>

directory directory directory directory Link file

Location of rules buckets (binary format) Not in use Not in use Not in use Link to current manifest The manifest file containing a list of buckets that need to be collected by the Poller on the NetXplorer

Understanding the Manifest

The manifest can be accessed through the web, by browsing to: http://<IP of NetXplorer>/bucket/<bucket type (30 or 300)>/manifest Example: http://192.123.234.56/bucket/30/manifest Format

Boot number, bucket index, bucket type (0=vc_stat, 1=conv_stat), statistic type, start time, end time, bucket duration, actual bucket duration, compression (0=no, 1-yes). Bucket duration is not always exactly 30/300 seconds. There may be a fluctuation of 1 or 2 seconds either way (for example, 299 or 301 seconds).

NetXplorer
C:\Allot\data\bucket\stc\<device ID>
Name conv_stat vc_stat line_burst pipe_burst vc_burst Type directory directory directory directory directory Explanation Contains conversations buckets in binary and then ascii format before import to short term database Contains rules buckets in binary and then ascii format before import to short term database Not in use Not in use Not in use

www.allot.com

19

NetXplorer and NetEnforcer Troubleshooting Guide

C:\Allot\data\bucket\ltc_export\
Name <Device ID> manifest Type directory file Explanation Multiple folders representing each device managed by the NetXplorer Server Manifest file containing list of buckets that need to be imported into the long term database

C:\Allot\data\bucket\ltc_export\<device ID>
Name conv_stat vc_stat line_burst pipe_burst vc_burst Type directory directory directory directory directory Explanation Contains conversations buckets in ascii format exported from the short term database Contains rules buckets in ascii format exported from the short term database Not in use Not in use Not in use

Allot/data/bucket/ltc/device_ID
Name conv_stat vc_stat line_burst pipe_burst vc_burst Type directory directory directory directory directory Explanation Contains conversations buckets in ascii format before import to long term database Contains rules buckets in ascii format before import to long term database Not in use Not in use Not in use

For details about the data collection procedure, refer to the SE training presentation.

www.allot.com

20

NetXplorer and NetEnforcer Troubleshooting Guide

Tools
Upgrading NX Server Version

Stop NetXplorer Service by going to Windows Services and stopping NetXplorer Server.
Open the Windows Task Manager by pressing <CTRL + ALT + DEL> and clicking the Task Manager button. Select the Processes tab and confirm that DbSrv9.exe does not appear in the list. Download the software version desired from the Allot ftp site by completing the following steps: 1. Log into the ftp site with your personal support login account (download\username) and password. Access will only be allowed if a valid license for NetXplorer has been purchased. 2. Type cd NetXplorer/NetXplorer_Server/Current_Versions/NetXplorer_NX7xx.zip 3. Please note that the NetXplorer files are approximately 460MB and will take some time to download. They are compressed and must be opened with WinZip or another utility.

For complete instructions and full installation procedures, see the NetXplorer Quick Install Guide and NetXplorer Operation Guide from http://www.allot.com.

There is no need to remove a previous installation. It will be detected automatically by the Installation Wizard. The NetXplorer Service will be stopped automatically when the upgrades starts. It will resume operation after the server is rebooted following the upgrade. At the end of the upgrade procedure you will be asked to reboot the NetXplorer Server.

Please note that if the NetXplorer Server will be down for more than 25 minutes, Real Time (Short Term) data after this period will be lost and data collection will be continued only after the server is up again. Therefore it is recommended to perform the upgrade during low traffic hours.

Enabling Compression
Toggling bucket compression on/off By default, compression is turned off (i.e. regular buckets). To toggle bucket compression: 1. Edit $SWGD/data/allotConfig.xml 2. The parameter data_collection/bucket_type should be set to 1 for compression or 0 for no compression. 3. Reboot the NetEnforcer. Note: Compression is not recommended as a default configuration, but only in situations where it is absolutely necessary. Enabling compression places additional heavy load on the NetEnforcer.

www.allot.com

21

NetXplorer and NetEnforcer Troubleshooting Guide

Change Admin Password

If the admin password has been lost, it is possible to replace it with the original password allot. In the SYSTEM_USERS table of the allot_cfg database, replace the admin password with: 53xXk0LYvZI=

Managing Reporting Databases

Recreating Default (ST and LT) Databases
It is possible to recreate empty (default) collector databases (STC and LTC). Data for the Device table will be loaded from Application Server (CFG database) as soon as the NetXplorer Server service is initialized after running the procedure. This utility replaces the current database files with clean databases (according to the configuration files c:\Allot\conf\static.ini and c:\Allot\conf\dynamic.ini created during installation process). Procedure 1. Stop the NetXplorer Server 2. Open MSDOS command window (Start>Run> type cmd). 3. c:\Allot\bin\recreate_default_db.bat <STC| LTC>. a. STC recreate STC database; b. LTC recreate LTC database. The following message appears in the command window - Recreate database <STC|LTC> successful or failed. 4. If the process has been successful restart the NetXplorer Server service. Note: There is more chance that the ST DB will get stuck, as it is in use approximately every 10 seconds, while the LT DB is only updated every hour. For problems with LT DB, please contact Escalation for additional assistance.

Improving Database Performance

To ensure better performance for complex NetEnforcer deployments managed by a single NetXplorer server, the following post-install changes for STC and LTC databases may be considered: Change temporary file location Change transaction log location Change dbspaces location (rename DBspace) Allocate additional disk space for DBspaces Deployment: 4 (four) files located in the directory \allot\bin: run_post_install_stc.bat; post_install_stc.vbs; - for STC database; run_post_install_ltc.bat; post_install_ltc.vbs for LTC database; Usage: First - NetXplorer Server service should be stopped. Before running, the VBscript files post_install_stc.vbs, post_install_stc.vbs should be manually edited. Carefully read all remarks,

www.allot.com

22

NetXplorer and NetEnforcer Troubleshooting Guide

comment unnecessary commands, set real paths for database files and necessary sizes for dbspaces. Recommendations for all post-install steps are available in the mentioned VBscript files. In case dbspaces file locations (paths) are changed, it is necessary to change (manually edit) the dbspaces locations in \allot\conf\dynamic.ini file. Open a command window (cmd.exe). From the command-line, run: \allot\bin\ run_post_install_stc.bat or run_post_install_ltc.bat. The following message will appear after the command has completed successfully: See post installation log in -\allot\tmp\install\post_install_stc.log

Changing Reporting Database Profiles

Changing LT Reduction Profile
Change the reduction.cfg file for the LTreducer application. The installation copies enterprise normal profile file into directory \allot\conf. The mentioned profile then becomes active (file name is reduction.cfg). All reduction profile files are located in the \allot\conf\Reduction directory. This utility will copy the active reduction profile file in \allot\conf from the \allot\conf\Reduction directory. The possible reduction profile types are: ent_normal; ent_accuracy; ent_history; isp_normal; isp_accuracy; and isp_history. Please note that ent = enterprise and isp = Internet Service Provider. Usage: Open command window (cmd.exe). From the command-line, run: \allot\bin\ reduction_profile_upd.bat <profile type>. Profile types are: ent_normal; ent_accuracy; ent_history; isp_normal; isp_accuracy; and isp_history. Example: \allot\bin\ reduction_profile_upd.bat isp_accuracy For more information on profiles, see the Excel chart on profiles in the knowledge base (http://support.allot.com) (item #6423), the SE Internal Training (item #6059), and item #6836.

Changing ST Profile Options

Purpose: Change data aging parameters in STC database PARAM table for second, minute and hour statistical data. Server Usage: Open command window (cmd.exe). From the command-line, run: \allot\bin\ stc_profile_upd.bat <profile type>. Profile types are: ent_normal; ent_accuracy; ent_history; isp_normal; isp_accuracy; and isp_history. NetXplorer Server service (or STC database) should be restarted. Example: \allot\bin\ stc_profile_upd.bat isp_accuracy NetEnforcer Change collection profile: go config data_collect <environment:profile> Acceptable values of Reduction Environment are: ent and isp Acceptable values of Reduction Profile are: normal, accuracy and history

www.allot.com

23

NetXplorer and NetEnforcer Troubleshooting Guide

Changing Reporting Database Parameters

Disabling External Hosts Reporting
To disable external host collection, use the following CLI command: go config data_collect -no_ext_host enable The NetEnforcer will reboot after 5 seconds. Please note that by default, the AC-1000 does not include external hosts as part of the collection key and the AC-400/AC-800 does.

Increasing the number of buckets sent per time slice

Changing number of buckets in the NetEnforcer
Note: This should only be used in situation where the need for increasing the buckets is critical. The default number of buckets sent is 5. There is an option to increase this number, to a maximum of 48 buckets (on x0x devices). In the AC-10x0/AC-25x0 devices there is no HDD and it is not recommended to increase this number at all. Increasing the number of buckets should be followed by enabling compression on the device (see page 21 on how to enable compression). This is done as follows: 1. CD to $SWGD/data 2. Vi to allotConfig.xml 3. Modify the line marked in bold below from 5 to the new number: <data_collection> <sample_interval>30</sample_interval> <bucket_type>1</bucket_type> <max_emb_rec>0</max_emb_rec> <max_st_bkts>5</max_st_bkts> <max_lt_bkts>5</max_lt_bkts> <bkt_mgmt_enable>0</bkt_mgmt_enable> <service_statistics>4. Restart the StatisticMgr module in order to include the modification. Note: Increasing this parameter would increase the number of buckets for 30 second as well as 300 second. 48 buckets is equal to 4 hours of 5 minute resolution, and 24 minutes of 30 second resolution.

www.allot.com

24

NetXplorer and NetEnforcer Troubleshooting Guide

Changing number of buckets in the NetXplorer

Every bucket has a time stamp. When the server receives a bucket, it checks the timestamp. If the timestamp is older than UTC time minus delta, it discards the buckets. In order to increase this delta, it is necessary to do the following: 1. Enter Sybase Central. 2. Enter the STC database. 3. Go to the PARAM(nms) table in the Table folder. 4. Choose the Data tab. 5. Go to line 66 The max time for a 30 seconds bucket time to be before of the current UTC. 6. Change the INT_VAL value from 180 to a value larger than 30sec x selected number of buckets. 7. Do the same on line 67 The max time for a 300 seconds bucket time to be before of the current UTC. 8. Change the INT_VAL value from 1800 to a value larger than 300sec x selected number of buckets.

Enabling TAP Mode

To enable TAP mode, right-click on a NetEnforcer and select configuration. On the Networking tab, check TAP Mode and save. TAP mode will now be enabled. Note: TAP Mode is not supported on the NetEnforcer AC-1040.

www.allot.com

25

NetXplorer and NetEnforcer Troubleshooting Guide

Port Mirror
Many customers do not wish to install a NetEnforcer inline between the LAN switch and the WAN router, even in monitoring-only mode, since they need to disconnect the line when installing the NetEnforcer. Therefore they wish to install the NetEnforcer on the switch mirror port, or span port, instead and monitor the traffic in that way. The switch mirror port mirrors the traffic received and transmitted on the port to the WAN router. The NetEnforcer is used as a simple monitoring probe and the Internal or External port is connected to the switch mirror port. Therefore only one port is connected. The NetEnforcer can still monitor traffic in this case, however there are two modifications needed for the NetEnforcer to operate properly.

Procedure
Step 1 Bridge learning must be disabled in order to prevent the NetEnforcer from learning and maintaining a bridge forwarding table for the port connected to the switch mirror port. 1. Connect to the NetEnforcer console via the Console port or a Telnet/SSH session. Login as user root with password bagabu (unless changed). 2. Open the file /usr/local/SWG/bin/init_modules for editing using the vi editor by entering the following command: vi /usr/local/SWG/bin/init_modules 3. Change the line prisma_args="stree=${STREE_MODE} to prisma_args="nolearn=1 stree=${STREE_MODE} 4. Save the changes by entering the following command :wq 5. Reboot the NetEnforcer for the change to take effect. Step 2 When the NetEnforcer has rebooted and has become active again, the handling of double sessions must be changed as follows: 1. Connect to the NetEnforcer console again via the serial port or a Telnet/SSH session. Login as user root with password bagabu (unless changed). 2. Type the following command: acmode +dbs 3. Type the following command acmode qos 4. The QoS software will restart automatically, no need to reboot.

Conclusion
Traffic between the LAN switch and the WAN router may now be monitored from the switch mirror port. All the different monitoring graphs should work with the exception of the Connections graphs. NetAccountant and the Long Term Monitoring may also be used.

www.allot.com

26

NetXplorer and NetEnforcer Troubleshooting Guide

Issues
NTP/Time issues
Synchronization issues between Client and Server
The NetXplorer Client and NetXplorer Server have a tolerance of 10 minutes time difference. The devices may be on different time zones. For example, if the Server is set to 10:03, and the device is set to 10:05, then this is acceptable. The same goes if the time zone difference is +2:00 (12:05). Note: Daylight savings time may cause an issue with the time zones. Symptoms If the clocks are out of sync, the graphs/logs times are inconsistent. Troubleshooting After login to the client, there is always a log of the time (UTC time dump). Check c:\Documents and Settings\<User name>\NMS.log to view this time dump.

Synchronization issues between Server and NetEnforcer

Symptoms If the clocks become out of sync, then there can be many issues including data collection. When statistics are gathered by the NetEnforcer, a bucket is created with a timestamp based on the NetEnforcer clock. Periodically, these buckets are collected by the poller process (on the NetXplorer). The NetXplorer compares the time of the bucket with its internal clock. If the NetEnforcer and NetXplorer Server have a time difference larger than 180 seconds for 30 second buckets and 1800 seconds for 300 second buckets, it will discard the bucket. When the user tries to generate real time monitoring graphs, no real-time data will be displayed, and an error message will appear in red displaying: No data for the time selected. The following alarm/event is received if data collection is stopped (can be found in the poller.log file located in C:\Allot\log): 'invalid bucket time on device NetEnforcer404' (id 208 - Current bucket time is older that current UTC minus delta) Cause How does the synchronization functionality work? The ntpdate command is initiated once at startup. It connects to the NTP server(s) and sets system time according to the time value received from the first server that responds. The ntpd process is initiated once the time is set by ntpdate. It is the daemon that keeps the unit time properly synchronized. If ntpdate fails to synchronize, ntpd will not be started. ntpd does not update the time at regular intervals. The update intervals are based on certain calculations to determine when synchronization is required. Typically, this is once every 30 to 60 minutes.

www.allot.com

27

NetXplorer and NetEnforcer Troubleshooting Guide

ntpdate may not initiate at startup for the following reasons: The NetXplorer Server is rebooted at the same time the NetEnforcer is booting up. The NetEnforcer does not manage to synchronize with the NTP Server because: o The server is down. o There are communication issues. Troubleshooting It is important to check the NetXplorer server first, then continue to the NetEnforcer if the problem has not been solved. NTP/NetXplorer Server Verify that the NTP service is running. By default, this runs on the NetXplorer server. If this is the case, run the following command: C:\Allot\ntp-server\ntpq -p ntpq:read:Connection refused This error indicates that the NTP service is not running on the NX Server. To initiate the NTP service on the NetXplorer server, do the following: 1. Go to Services in Administrative Tools on the PC, and start the Network Time Protocol Service. 2. To verify that the service is running, run Task Manager and search for the process ntpd.exe. If this process is found, run the ntpq -p command, as described above. 3. Reboot the NetEnforcer to see if the synchronization will take place after reboot. NetEnforcer Ensure that the NTP service is running on the NetXplorer server before continuing. Verify that the NTP process is running on the NetEnforcer: ps awx|grep ntp
89 ? SL 0:00 /usr/sbin/ntpd -l /usr/local/SWG/logs/ntp.log

The above line shows that the NTP process is running. If the process is not found, initiate the NTP Daemon by rebooting the NetEnforcer. Verify that synchronization is against the NTP server IP (NX or ext. NTP server):
AC-202:~# ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== LOCAL(0) LOCAL(0) 14 l 59 64 377 0.000 0.000 0.008 *10.4.70.1 LOCAL(1) 11 u 4 64 377 0.624 -2.455 0.291

Status 16 indicates failure to sync against NTP server. Verify that synchronization is against the NTP server, and not the internal (local) clock of the NetEnforcer. This is marked by an asterisk (*) at the beginning of the line with the NTP server. Verify that the Windows firewall is not enabled on the server (this is enabled by default) which could block the NTP requests.

www.allot.com

28

NetXplorer and NetEnforcer Troubleshooting Guide

For more information, the NTP manuals may be found at http://ntp.isc.org/bin/view/Main/DocumentationIndex. A document describing NTP and NTP on the NetEnforcer in general (for version 5.x) can be found at KB item 4723.

Problem: GUI does not start

To solve this issue, go to control panel on the machine that cannot access the NetXplorer and choose Java. 1. On the General tab, under Temporary Internet Files, click on delete and then OK. 2. Open browser with NX server IP address (http://NXServer-IP) and launch the application. Note: If this does not solve the problem, run javaws.exe from the Java 1.5 environment. This may typically be located at a location similar to: C:\Program Files\Java\jre1.5.0_06\bin. Delete anything shown on this screen (this will clear the cache).

Creating a Snapshot
NetXplorer
o \allot\bin contains a batch file called create_snapshot_logs.bat. This file takes all the relevant logs and prepares a snapshot file that can be sent via e-mail. Please note that this file can be large at times (approx. 9MB). The snapshot will be created under \allot\tmp\snapshot_<date>.tar.gz

NetEnforcer
The snapshot procedure is the same as in previous NetEnforcer versions. To generate a snapshot run snapshot.

Taking a Snapshot
The Snapshot File is a file used to help Allot Customer Support in the troubleshooting process. The file itself is a zip file that contains files which provide Allot Customer Support with a precise picture of what was happening inside the NetEnforcer when a particular event occurred. These files include log files, policy definitions, system settings, etc. The Snapshot is an essential support tool that is vital in solving any support issues. There are two ways of taking the Snapshot: Manually and Automatically.

The Manual Snapshot

The Snapshot can be run manually. If an Allot Customer Support Engineer requests you take a Snapshot of the box, it is best to run the Snapshot process manually. To run the Snapshot manually, simply login into the NetEnforcer as root, and from the command prompt, run the command snapshot. This will create a Snapshot file in the directory, /usr/local/SWG/snapshots/. The Snapshot file is created with the name snapshot.date_time.tgz. Core Snapshot While taking a regular snapshot, core files (all files under /usr/local/SWG/logs/core) will also be included in it. In some cases the core files might be very big. In cases where the size of the

www.allot.com

29

NetXplorer and NetEnforcer Troubleshooting Guide

snapshot is more than 15M, the NetEnforcer will create an additional snapshot with core files only. Example:
core.snapshot.07.05.02_09.27.00.tgz

The Automatic Snapshot

There may be some specific cases where Customer Support requests that you run an Automatic Snapshot. This process configures the Snapshot to run automatically every four hours. The snapshot files are deposited in the /usr/local/SWG/snapshots/ directory. To start the Automatic Snapshot, type snap_on_cron To stop the Automatic Snapshot, type snap_off_cron Prisma Snapshot An automated snapshot that is generated after DKM or Collector restarts. The Prisma Snapshot is a short version of the regular snapshot and contains only /proc/prisma directory and /usr/local/SWG/logs directory.

Sending the Snapshot

Note: This script does exist in the box, but there is a bug. Do not use this script for now. Normally, this takes a snapshot but currently cannot send it. Current Snapshot A utility is included on the NetEnforcer for sending the Snapshot files directly to Allot Customer Support. The utility is called send_snapshot and the syntax is send_snapshot. This utility will automatically take a snapshot of the units current state and log into the Allot Customer Support FTP Server. It will then open a directory (named with box number of the NetEnforcer) and send the snapshot. The file/s are copied into the opened directory. Saved Snapshot A snapshot which has been taken previously and saved may be sent using the syntax Send_snapshot_file(s). For example, if you have a saved Snapshot file, snapshot.01.03.00_09.54.39.tgz and you would like to send it to the Customer Support for analysis; you would type the following command from the command prompt:
send_snapshot snapshot.01.03.00_09.54.39.tgz

This will contact the Allot Customer Support FTP server, log in, create a numerical directory and copy in the snapshot file selected.

HTTP Snapshot
Some NetEnforcer and NetXplorer units do not have access to FTP. Therefore, it is not possible to send a snapshot directly from the box. If the unit does not have a public address or Internet access, use this workaround: 1. Create the snapshot by typing: snapshot The snapshot file is saved to the following directory: /usr/local/SWG/snapshots/ 2. Copy the snapshot file to the /usr/local/SWG/etc/httpd/htdocs directory: cp /usr/local/SWG/snapshots/snapshot.15.03.06_16.08.33.tgz

www.allot.com

30

NetXplorer and NetEnforcer Troubleshooting Guide

/usr/local/SWG/etc/httpd/htdocs (in this example, the file is named snapshot.15.03.06_16.08.33.tgz). 3. Point the browser to the NetEnforcer URL: http://<NetEnforcer IP>/snapshot name For example: http://192.1.1.2/snapshot.15.03.06_16.08.33.tgz. 4. This will start an HTTP download of the snapshot file to the PC. It is now possible to email this snapshot, or place it on an FTP server for access to Allot personnel. Note: If an FTP Server is available, it is also possible to connect to the NetEnforcer using the FTP, browse to where the snapshot is located, and use the mget command to get the snapshot (using bin mode).

www.allot.com

31

NetXplorer and NetEnforcer Troubleshooting Guide

Add Device
When adding a device to the NetXplorer NX730, there are 10 stages that need to be completed. Therefore, when adding a device and getting a "failed to create topology device" error, it is important to know on which stage it failed. Stage 1: configuration : create device topology Stage 2: event : create device event counter entry Stage 3: configuration : check device software version Stage 4: import configuration : set configuration from Device to DB Stage 5: catalog : export (deviceTopology) Stage 6: policy : export default policy (deviceTopology) Stage 7: register to snmp trap : register AS To Snmp Tables Listeners Stage 8: collector : assign device to collector Stage 9: configuration : set admin and oper to 1 - ON Stage 10: get the latest topology object To do this, go to the NMS.log, located under Allot_Home:\Allot\netxplorer\jboss3.2.6\server\allot\log and search for the word "CREATE":
2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(1/9) [admin/122.122.4.32] create device topology to DB - started 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(1/9) [admin / 122.122.4.32 #2] create device topology to DB - finished 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(2/9) [admin / 122.122.4.32 #2] create device event counter entry - started 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(2/9) [admin / 122.122.4.32 #2] create device event counter entry - finished

The first two stages almost always complete successfully. Keep track of the CREATE (by searching) until the failed stage is found. Fail on stage 4 - set configuration from device to database In this stage, the server reads IP configuration from rc.conf. The following indication will probably be found:
2006-04-01 02:07:22 [RMI TCP Connection(171)-122.122.4.101] ERROR management.ejb.ConfigurationFacadeEJB - failed to setConfigurationFromDeviceToDB null; CausedByException is: Device 122.122.4.101/161 is unreachable when trying to send pdu

This indicates that the probe could not send the configuration updates to the server on port 161. In this case, check the following:

Run netstat -an on the NetEnforcer or Server and check whether a connection on port 161 is established.

www.allot.com

32

NetXplorer and NetEnforcer Troubleshooting Guide

Check that nothing is blocking SNMP traffic along the way. Check that the database is up and available.

Fail on stage 5 - exporting catalogs from the Server to the NetEnforcer In this stage, the Application Server connects to the Apache Server (using CGI on port 80) on the NetEnforcer using the following link: http://122.122.4.32:80/cgi-bin/ProvisionCli.exe. In the NMS.log the following will be seen:
2006-04-01 02:06:48 [RMI TCP Connection(169)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(5/9) [admin / 122.122.4.32 #6] export Catalogs - started 2006-04-01 02:06:52 [RMI TCP Connection(169)-122.122.4.101] DEBUG catalog.synch.SynchUtils - send to device= http://122.122.4.32:80/cgi-bin/ProvisionCli.exe is name=<message id="5" type="req">

Potential problems: Authentication failure may also result from incorrect password. Another indication to that would appear in $SWGL/nedbg.DataSrv.log on the NetEnforcer. Make sure the correct admin password was entered. Try to reset the admin password. Communication exception: Indication:
2006-04-04 11:52:16 [RMI TCP Connection(52)-10.254.48.100] DEBUG catalogs.ejb.CatalogFacadeEJB - EXCEPTION = com.allot.nms.common.net.CommunicationException

Check for access lists (on the NetEnforcer, routers, firewalls, etc). Check with netstat -an that a connection from the NetEnforcer to the Server on port 80 was established. Try to connect the NetEnforcer to a different switch (this has worked in the past).

According to the Troubleshooting Guide. Please note that these problems have never been encountered: Check that DataSrv and ProvisionCli.exe are running. Check in $SWGL/nedbg.DataSrv.log whether DataSrv received the changes (check for full export). /tmp/nedbg.ProvisionCli.log - check whether content was received from the Apache Server (view full XML content). /var/log/apache/access_log - check whether Apache received change (look for POST to ProvisionCli.exe).

Fail on stage 6 - exporting default policy from the Server to the NetEnforcer Failing on stage 6 may be a result due to large catalogs on the server that need to be added to the NetEnforcer. The NetXplorer server has a timeout of 1 minute to complete the add process. If the process takes longer, it may reach step 6 before stopping. There is no workaround to solve this on site. R&D involvement is needed in order to reduce the processing time on the NetEnforcer to less than the 1 minute limitation. Fail on stage 7 - Register AS to SNMP Tables

www.allot.com

33

NetXplorer and NetEnforcer Troubleshooting Guide

Failing on stage 7 is most likely to happen when adding a device while management traffic goes through the box. The NetEnforcer reboots and the addition fails. The workaround is to switch the NetEnforcer to bypass, and then add the device. Stage 8 (assign device to collector), 98 (set admin and oper to 1 ON), and 10 (return topology object) may fail if the Application Server cannot connect to the database. The only workaround for this is to stop and start the service and ensure that the 3 databases: CFG, STC and LTC are up and running. If one of the databases are stuck, it must be recreated before the device can be added again. Indications that databases are up and running: In allot_cfg.log, look for the following: 02/26 11:59:14. Running on Windows XP Build 2600 Service Pack 2 I. 02/26 11:59:14. Database server started at Sun Feb 26 2006 11:59 I. 02/26 11:59:14. Trying to start SharedMemory link... I. 02/26 11:59:14. SharedMemory link started successfully I. 02/26 11:59:14. Trying to start TCPIP link... I. 02/26 11:59:14. Starting on port 50000 I. 02/26 11:59:19. TCPIP link started successfully I. 02/26 11:59:19. Now accepting requests In allot_stc.log and allot_ltc.log, look for Enable all events: I. 04/03 09:15:33. Running on Windows XP Build 2600 Service Pack 2 I. 04/03 09:15:37. Database server started at Mon Apr 03 2006 09:15 I. 04/03 09:15:37. Trying to start SharedMemory link... I. 04/03 09:15:37. SharedMemory link started successfully I. 04/03 09:15:37. Trying to start TCPIP link... I. 04/03 09:15:37. Starting on port 50001 I. 04/03 09:15:42. TCPIP link started successfully I. 04/03 09:15:42. Now accepting requests I. 04/03 09:16:08. Enable all events

Change IP
Defined Behavior
There are three locations where the IP of the NetEnforcer can be changed: The NetEnforcer itself, using the LCD, CLI or Admin menu The IP Properties tab within the Configuration Menu of the NetXplorer Server for a specific NetEnforcer The properties window of a specific NetEnforcer within the NetXplorer Server GUI Note: If the IP address cannot be changed for any reason, manually edit the IP address in the rc.conf file, located in the /etc/rc.d directory. The NetEnforcer Changing the IP address via the NetEnforcer does not impact the NetXplorer Server. The purpose of this is to enable a user to change the IP address of the NetEnforcer and move it to another Server, without affecting the configuration properties of the NetEnforcer within the Server. This will therefore allow another NetEnforcer to be installed in place of this NetEnforcer (using the same model and version) while maintaining the original policy configuration.

www.allot.com

34

NetXplorer and NetEnforcer Troubleshooting Guide

An event will be sent to the NetXplorer Server indicating an IP change on the NetEnforcer. An alarm may be assigned to this event within the Event Types Configuration window. To complete an IP address change, the address will also need to be configured within the device properties within the NetXplorer Server.

IP Properties tab within the Configuration Menu Changing the IP address of the NetEnforcer within the IP Properties of the Configuration Menu will change the address of the device itself and the Properties of the device within the Network tree. Properties window of a specific NetEnforcer Changing the IP address of the NetEnforcer within the Device Properties menu (accessed by right clicking on the device within the Network tree and selecting Properties) does not change any IP definitions on the NetEnforcer. This change will point the NetXplorer Server to connect to the specified IP address. To effect a change on the actual IP address of the device, the address must be defined within either the Configuration Menu or on the device itself.

Current Behavior
Please note the differences below: Changing the device IP address via the configuration menu will update the device properties (topology tree). This process will take effect approximately 30 seconds after entered.

In-Band/Out of Band Definitions

The NetXplorer and NetEnforcer do not support In Band IP configuration. Currently, the GUI displays both in and out of band. The in-band option is be grayed out. The option will remain since it is a feature that will be available in future versions. On the NetEnforcer itself, currently the CLI enables definition of an in-band address. The LCD does not have this option.

www.allot.com

35

NetXplorer and NetEnforcer Troubleshooting Guide

Provisioning Changes
Add Host
Process 1. Server sends XML command to NetEnforcer. 2. NetEnforcer performs changes and updates counters. 3. NetEnforcer sends trap to Server. Troubleshooting Server: C:\Allot\netxplorer\jboss-3.2.6\server\allot\log\NMS.log - check whether changes have been sent. a. send to device = location b. XML changes. c. result from device = location o <status>err <error_msg> explanation (development not complete). o <status>ok. Note: Asynchronous messages may not be displayed together. NetEnforcer: $SWGL/nedbg.DataSrv.log - check whether DataSrv received changes. Identify receipt, change applied and confirmation. Example successful output, see Appendix I. $SWGD/data/allotProvision.xml check counter ID and new catalog entry $SWGL/nedbg.AllSnmpAgent.log check for trap sent. Example successful output, see Appendix II.

Configuration Changes
Process
1. SNMP config changes sent. 2. SNMP config changes applied.

Troubleshooting
1. Check NMS.log on Server. 2. $SWGL/nedbg.AllSnmpAgent.log check for SET command.

3. $SWGL/nedbg.swKeeper.log (system changes) check for set_conf.

www.allot.com

36

NetXplorer and NetEnforcer Troubleshooting Guide

4. $SWGL/nedbg.DataSrv.log (application changes) look for XML.

Databases Not Synchronized

Symptoms
Full export of database from Server.

Explanation
This can occur due to manual XML changes CLI changes made when SNMP agent down NetEnforcer in rescue And others

Troubleshooting
1. $SWGL/nedbg.AllSnmpAgent.log Check for PolModifyTag=3 (bad database) 1 = good 2. $SWGL/nedbg.DataSrv.log Check for Full Export and complete XML

To Generate a Full Export

Touch $SWGD/data/allotProvision.xml.

www.allot.com

37

NetXplorer and NetEnforcer Troubleshooting Guide

RMA/Box Replacement
Important note: If there is no unit to replace, do not delete the unit from the server until you have another unit to replace it. Unit A is connected to the server. Unit B should replace unit A. 1. 2. 3. 4. 5. 6. Connect unit B and add it to the server with a different IP address than unit A. After unit B is reachable, disconnect both units (A and B) through the management port. Set unit B with the original IP address that was defined in unit A. Reconnect management port to unit B. Delete the IP address that was used to define unit B. Perform touch to allotProvision.xml.

Collection Problems
STC Problems Related to Software
There may be problems with the STC database due to software running on the NetXplorer PC which may be interrupting the database processes. Symptom The short term collector is stuck. no monitoring reports NX server reports event/alarm on STC_DEF Troubleshooting The following message can be found in allot_stc.txt:
E. E. I. I. I. I. I. 10/28 10/28 10/28 10/28 10/28 10/28 10/28 01:19:12. 01:19:12. 01:19:12. 01:19:12. 01:19:12. 01:19:12. 01:19:12. *** ERROR *** Assertion failed: 100909 (9.0.2.3137) Error deleting transaction log file *** ERROR *** Assertion failed: 100909 (9.0.2.3137) Error deleting transaction log file Attempting to save dump file at 'C:\WINDOWS\TEMP\sa_dump.dmp' Dump file saved

Explanation The first error, assertion failed error 100909: Error deleting transaction log file is usually caused the transaction log is locked. This indicates that there is another software application currently using the transaction log, preventing the NetXplorer databases from accessing it. Since the NetXplorer databases cannot access the log, the database is shut down. Potential software applications that may lock up the transaction log are: system backup software anti-virus software defragmentation tools or others.

www.allot.com

38

NetXplorer and NetEnforcer Troubleshooting Guide

Workaround The identified application must be configured not to access specific Sybase files (.db and .log files). Go to http://www.sybase.com/detail?id=1025501 for information on ASA, anti-virus and backup software. It is highly recommended NOT to run such programs on folders where the databases reside. After disabling such programs, it may be necessary to recreate the database. For details on this procedure, see the Recreating Default (ST and LT) Databases section on page 22.

Data Collection Stops Due to NTP Issues

Symptoms The following event/alarm is received on the NetXplorer Server: invalid bucket time on device <device name> (id 208 - Current bucket time is older that current UTC minus delta). Troubleshooting See section on Troubleshooting NTP Issues on page 27.

Demo Installation Issues

When installing the NetXplorer for demo or training purposes there are a couple of tricks to avoid the installation requirements. Note: These tools are internal and should only be used in exceptionally specific internal situations. This information should NEVER be distributed to anyone outside of Allot CS.

Installing NetEnforcer version 7.1.0 on a NetEnforcer AC-202/302

This should only be done for training purposes. The NetXplorer only supports NetEnforcer models AC-40x (AC-80x, AC-10x0 and AC-25x0 in the future). To disable the check for the NetEnforcer model, create the file /tmp/nocheck.

Skipping installation hardware requirements

From NetXplorer Version 25.27 (the current version is 23.25) it is possible to avoid hardware requirements such as memory and available ports. To disable the check, create the file c:\nocheck.

www.allot.com

39

NetXplorer and NetEnforcer Troubleshooting Guide

Appendix
Appendix I
Host output from $SWGL/nedbg.DataSrv.log
09-12 06:36:15(163) <DL_TRACE>: Message received from AS: <message id="4" type="req"> <check>53xXk0LYvZI=</check> <owner>168427883</owner> <change_id>4</change_id> <ops> <op id="1"> <opcode>create</opcode> <location>//catalogs/*/host/parent::*</location> <ID/> <data> <host a_right="1" id="2" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host> </data> </op> </ops> </message> 09-12 06:36:15(163) <DL_NOTIFY>: Create element, location: //catalogs/*/host/parent::* 09-12 06:36:15(163) <DL_TRACE>: Created element: <host a_right="1" id="2" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host> 09-12 06:36:15(163) <DL_TRACE>: PmChangeValidator::buildValidNewHostEntry. Validating. 09-12 06:36:15(163) <DL_TRACE>: Returned ID: 2 09-12 06:36:15(163) <DL_TRACE>: Set catalogs update counter to 4 09-12 06:36:15(163) <DL_TRACE>: Set update owner to 168427883 09-12 06:36:15(163) <DL_NOTIFY>: touch file data/lastPolicyUpdate : 1126506975

www.allot.com

40

NetXplorer and NetEnforcer Troubleshooting Guide

09-12 06:36:15(163) <DL_TRACE>: Sending notification to clients. 09-12 06:36:15(163) <DL_TRACE>: Update counter 4, number of changed catalogs 1 09-12 06:36:15(163) <DL_TRACE>: Changed catalog: type host_cat, name Host 09-12 06:36:15(163) <DL_TRACE>: Deleted entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_TRACE>: New entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 1 09-12 06:36:15(163) <DL_TRACE>: QuadID: 2. 09-12 06:36:15(163) <DL_TRACE>: Entry: <host a_right="1" id="2" level="trace" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host> 09-12 06:36:15(163) <DL_TRACE>: Modified entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_TRACE>: Tracked entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_NOTIFY>: CatSvr notify (0x83d89a0) [0] 09-12 06:36:15(163) <DL_TRACE>: Sending notification to clients. 09-12 06:36:15(163) <DL_TRACE>: Update counter 4, number of changed catalogs 1 09-12 06:36:15(163) <DL_TRACE>: Changed catalog: type host_cat, name Host 09-12 06:36:15(163) <DL_TRACE>: Deleted entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 0 09-12 06:36:15(163) <DL_TRACE>: New entries. 09-12 06:36:15(163) <DL_TRACE>: Number of entries: 1 09-12 06:36:15(163) <DL_TRACE>: QuadID: 2. 09-12 06:36:15(163) <DL_TRACE>: Entry: <host a_right="1" id="2" level="trace" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host> 09-12 06:36:15(163) 09-12 06:36:15(163) 09-12 06:36:15(163) 09-12 06:36:15(163) 09-12 06:36:15(163) 09-12 06:36:15(163)

<DL_TRACE>: Modified entries. <DL_TRACE>: Number of entries: 0 <DL_TRACE>: Tracked entries. <DL_TRACE>: Number of entries: 0 <DL_NOTIFY>: CatSvr notify (0x84b40d8) [0] <DL_TRACE>: Message returned to AS:

www.allot.com

41

NetXplorer and NetEnforcer Troubleshooting Guide

<message id="4" type="res"> <check>53xXk0LYvZI=</check> <owner>168427883</owner> <change_id>4</change_id> <ops> <op id="1"> <status>ok</status> <ID>2</ID> </op> </ops> </message> 09-12 06:36:15(163) <DL_NOTIFY>: Detach session (0x83c3cc8), client

Appendix II
Host output from $SWGL/nedbg.AllSnmpAgent.log
09-12 06:36:14(169) <DL_NOTIFY>: PolModifyTag= 1. snmp_t= 1126506686, pol_xml_t= 1126506686, pol_upd_t= 1126506686, rescue_t= 0, full_t= 1126506686 09-12 06:36:15(169) <DL_NOTIFY>: PmRegisterUser updates [260] 09-12 06:36:15(169) <DL_NOTIFY>: handleDataUpdates(): id= 168427883, count=4 09-12 06:36:15(169) <DL_NOTIFY>: Catalog instance 2 is changed :[ <host a_right="1" id="2" level="trace" name="10.1.1.1" scope="0" type="0"> <entries> <host_entry id="2" type="2"> <ip>167837953</ip> </host_entry> </entries> <queries/> </host>] 09-12 06:36:15(169) <DL_TRACE>: Trap counter = 3, index in NotifyLog= [7.99.97.116.95.108.111.103.3] 09-12 06:36:15(169) <DL_TRACE>: Trap counter = 3, index in NotifyLog= [1.3.6.1.2.1.92.1.3.1.1.9.7.99.97.116.95.108.111.103.3] 09-12 06:36:15(169) <DL_TRACE>: New trap counter = 4, index in NotifyLog= [1.3.6.1.2.1.92.1.3.1.1.9.7.99.97.116.95.108.111.103.3] 09-12 06:36:15(169) <DL_NOTIFY>: Trap sent, oid= [1.3.6.1.4.1.2603.0.2], prev_count= 3, cur_count= 4 09-12 06:36:15(169) <DL_NOTIFY>: Set [1.3.6.1.4.1.2603.5.2.203.0] counter to 4

www.allot.com

42