0% found this document useful (0 votes)

830 views149 pages

Control and Forwarding Plane

JUNOS CLI basics Space bar to complete a command Command: Show log messages file-name at more prompt, use forward slash( / ) to search or use --h to get a context help screen

Uploaded by

Mohsin Hafeez

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PPT, PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

830 views149 pages

Control and Forwarding Plane

JUNOS CLI basics Space bar to complete a command Command: Show log messages file-name at more prompt, use forward slash( / ) to search or use --h to get a context help screen

Uploaded by

Mohsin Hafeez

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PPT, PDF, TXT or read online on Scribd

You are on page 1/ 149

Control and Forwarding plane

Synchronization 1) 100-Mbps fxp1 Ethernet link is used between RE and PFE 2) For M320 case, 100-Mbps Ethernet switch is being used to provide a dedicated link to each FPC. For RE, these links are presented at bcm0 3) Fxp0: management interface 4) Fxp2: communication between Primary RE and backup RE 3) Forwarding table (FT) can hold over 800,000 routes.
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Difference between M7i and M10i

1. Redundant RE: M10i support, not M7i 2. Built-in Adaptive Service: M7i. M10i needs an external AS PIC. 3. RE: the same

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

System storage
3 types of storages: 1) Compact Flash(ad0) : built-in at the board. 2) Hard Drive(ad1) 3) External storage -PCMCIA card(da0??) -USB(da1??)

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

JUNOS CLI basics

Space bar to complete a command Command :Help topic <command> for general concepts Command: help reference <command to look> for configuration syntax Rebooting system: request system reboot Shut down system: request system halt Log and Trace files are located at /var/log Command: Show log | messages | file-name At more prompt, use forward slash(/) to search or use h to get a context help screen Log commands examples: - show log messages | match so-0/3/1 | match TRAP --- AND -- show log messages | mach fpc | sfm | kernel --- OR --Monitor log/trace in real time: monitor start file-name | match fail Stop monitoring in real time: monitor stop Enable/disable real-time output to screen: Esc-Q Stop traceing operation: delete flag open Truncate(clear) log/trace files: clear log file-name Delete log/trace files: file delete file-name

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

JUNOS CLI basics

Entering configuration: Type configure or edit Exclusive configuration (configure exclusive) and Private configuration (configure private??) Moving within the configuration hierarchy: edit (equivalent to cd), up, top, exit (to previous location in the hierarchy) Show command at configuration mode vs. show command at operational-mode Relative configuration commands Starting with JUNOS5.3: top - top show system login (show system login no matter where you are. Examples: - top edit protocols ospf ( to enter protocols ospf no matter where you are) Viewing configuration in operational mode: show configuration < configuration path> View configuration with set: show xxx | display set Viewing candidate configuration: show chassis alarm, show (at the current sub-hierarchy)

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

JUNOS CLI basics

Change the candidate configuration. Examples: - set alarm sonet lol red - delete alarm sonet pll Display difference between the candidate and active configurations: At the current statement-path, show | compare Viewing difference in files. Example: - file show filename1 | compare file filename2 - show configuarion | compare rollback number Removing statements: delete Delete the statements and all its subordinate statements and identifieres.
Wildcard delete. Example: wildcard delte interfaces fe-* Ignore portion of the configuration hierarchy: deactivate / activate Disable an interface: set disable interface Delete and disabled interface: delete interface <interface-name> disable

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

JUNOS CLI basics

Activate a configuration commit ----- candidate file is checked, actived and marked as the current operational sofware configuration file. commit check ----- only validate a candidate configuration without placing it into effect. rollback n -------- recover the previous configuration. And then commit rollback 0 is current configuration First 3 roll back (1-3) are stored in solid-state flash disk /config/juniper.conf.n (n=1-3) rest roll back (4-49) are stored in hard disk /var/db/config commit confirmed time-out ---- temporarily activate a configuration (default is 10 minutes). If the final commit is not executed, the system will performs a rollback 1, commit commands. commit synchronize ---- after committed on the master RE internally copied and committed on the backup RE automatically. commit at time ----- commit at some time clear system commit ---- cancel a pending commit

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

JUNOS CLI basics

Save a configuration
save filename save terminal -- for copy and paste into other others show | display set create configuration for simplifying configuration editing.

Loading configuration files ( load and then commit)

load override filename override the current config with the loaded one. Do it at the root of the configuration hierarchy. load merge filename - combine the new and old load merge terminal (then copy/paste hierarchical configuration) load replace filename statements with replace tag will replace the statements with the same name load relative load at where it is current at the configuration hierarchy.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Junos CLI Basics

Only save the configuration under certain hierarchy. To save the whole configure, issue this command at the top of the hierarchy. #Save <filenam> Display the contents of the file you saved #Run file show <filename> To load a configuration after clear the current configuration # delete #show #load override <filename> To recover a mistake made previously after committing. #rollback 1
Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

Junos CLI Basics

show log messages | last Show log interactive-commands | match restart Use sysctl a to display kernel parameters. sysctl a | grep icmp (under shell prompt)

show chassis 0 pic slot 1 information.

Show chassis pic fpc-slot 0 pic-slot 1

Master switchover Request chassis cfeb master switch Request chassis routing-engine master switch
Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

Junos CLI Basics

Find out who logins the system and kick out some particular users. show system users reequest system logout help syslog <log strings> Example: lab@santro-re0> help syslog ACCT_ACCOUNTING_FERROR Name: ACCT_ACCOUNTING_FERROR Message: Unexpected error <error-code> from file <filename> Help: Error occurred during file processing Description: An error prevented the accounting statistics process from processing the indicated file. Type: Error: An error occurred Severity: warning
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Junos CLI Basic

show configuration with inheritance show configuration interfaces ge-4/3/3 | display inheritance

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Syslog set system syslog file messages any notice

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Hardware troubleshooting process

Show chassis alarms Show chassis craft-interface Show log messages Show log chassid Monitor start [message | chassid] Show chassis hardware Show chassis fpc Show pfe stat error Show interface terse Show interface detail Show log <log-file-name>
Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

Display PIC status

Show chassis pic fpc-slot 0 pic-slot 1 Example: lab@santro-re0> show chassis pic fpc-slot 0 pic-slot 1
FPC slot 0, PIC slot 1 information: Type 10x 1GE(LAN), 1000 BASE ASIC type H chip State Online PIC version 1.13 Uptime 1 day, 22 hours, 25 minutes, 17 seconds
PIC port information: Fiber Xcvr vendor Port Cable type type Xcvr vendor part number Wavelength 0 GIGE 1000SX SM FINISAR CORP. FTRJ8519P1BNL-J2 850 nm

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Boot image
If you need to reboot from PCMCIA card, you need to copy a special image called jinstallmediaxxxx. Interrupt normal boot Hit space when the system is rebooting until it goes to either boot: or OK prompt. If you get boot: prompt, the loader is not run yet. You need to do this: Boot: /boot/loader Change a boot device at OK prompt Ok nextboot compact-flash Ok reboot
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Interfaces
Disable(admin down) an interface Admin Link So-0/1/1 down up So-0/1/1.0 up down

Deactivate an interface Admin So-0/1/1 up

Link up

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

RE overview (Q: how to find out RE <-> Platform compatibility list?)

Primary coopy of JUNOS resides on the flash memory. Use this command to create a backup copy:
request system snapshot

Mgd manages CLI RE has different versions: RE-333, RE-400, RE-600, RE-1600. Each RE is supported by certain platforms. RE uses Intel processor from P III to P IV. Use this command to find out what RE is being used: show chassis hardware. Hard disk monitoring: Self-Monitoring Analysis and Reporting Technology System(SMART). From 5.5, SMART is enabled by default. To disable:
set system processes disk-monitoring disable Configuration file compression: default starting Release 7.0 (maybe). To enable: set system compress-configuration-file

RE versions RE5(RE-400): only supported in M7i and M10i RE4(RE-600): All M and T series. Except M7i/M10i/M320. The only RE to have flash memory upgrade RE3 (RE-333): M5/10/20/40/40e, and M160 RE-1600: M320 and T320/T640. Using Broadcom chipset for Ethernet connectivity to PFE. While used on M320, the GE link is supported as bcm0. While on T-series, 100Mbps is supported(???)

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

PFE overview on M-series

1. 2. 3. 4. 5. Different names but referring to the route lookup module: M40 System Control Board (SCB) M20 System Switch Board (SSB) M5/10 FPC and SCB are combined into a single board called the Forwarding Engine Board (FEB) M7i/10i Compact FEB (CFEB) M40e and M160 Switching and Forwarding Module (SFM). 4 SFM on M160, each one provides 25% of lookup capability. 2 SFM on M40e, only one can be active. Special stuff on M40e and M160 platform: MCS card (Miscellaneous Control Subsystem): provide control and monitoring functions for the various components in the chassis PCG (PFE clock generation): 125-MHZ signal. Redundant PCGS

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

PFE on T-series and M320

M320 is different than T and M-series. It is a combio of two using I and J chips. T640 PFC2 has single PFE, PFC3 has two PFE T-Series nonblocking cross-bar switch fabric Switch Interface Boards(SIBs).

T320: 3 SIBs with 2 are active. SIB 1 and 2 are active, SIB0 is standby. SIB0 has only one high-speed line (HSL) connected to FPC. SIB1 and SIB2 has 2 HSL. So when SIB0 becomes active, system performance is degraded.
T640: 5 switch fabric cards or SIBs, 4 are active, 1 standby. Something like Ciscos GSR. M320: 4 SIBs. M320 FPC1: use single I chip M320 FPC2: dual I chip, thus two PFE M320 FPC3: dual J chip, thus two PFE

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Physical Interface Cards (PIC)

IP service PIC is to hardware assist complex packet processing and has no physical ports. IP service PIC include: 1)Tunnel service PIC for IP-IP, GRE tunnel and PIM-SM tunnel. 2)Multlink PIC: Multilink Point-to-Point (MLPPP) and Multilink Frame Relay (MLFR, FRF 1.5) Hot-Pluggable except M20 and M40 which need to remove FPC. Take PIC offline before physically removing it. Otherwise would cause system damage or PFE reset. Packet loss is expected on M-serials except M320 because of FPC reset.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Flexible PIC Concentrator (FPC)

Support 1 to 4 PICs. M160 OC-192 has an FPC support only one PIC. Each FPC on M-serial pooled to create shared memory switch fabric. So hot-swap FPC cause system to repartition the shared memory pool; 200 ms packet loss. FPC is hot-swappable in all platforms except M5 and M10 which is using FEB. However M7i and M10i are OK even using CFEB. Build-in FPC at some high-speed quad-wide PICs such as OC48c/STM-16 for M20/40. OC-192c/STM-64 SONET/SDH on M160. New FPC to support reuse of old PICs: M160 FPC1: intend to reuse M20/40 PIC M160 FPC2: design to support M160 only PIC, such as OC-48c FPC3: support native T-series PICs. T640 only support FPC2 and FPC3. How to power off FPC? set chassis fpc power off

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

M-series System Board

1. 2. 3. 4. 5.

General functions Names very by platforms

M40 System Control Board (SCB) M20 System Switch Board (SSB) M5/10 FPC and SCB are combined into a single board called the Forwarding Engine Board (FEB) M7i/10i Compact FEB (CFEB) M40e and M160 Switching and Forwarding Module (SFM). 4 SFM on M160, each one provides 25% of lookup capability. 2 SFM on M40e, only one can be active.

Enhanced System Boards: - 2nd generation Internet Processor II ASIC (not on M5/10 and M7i/10i) - support 840K routing entries, double from old board 420K. - Double on-chip memory to 16MB on IP II - CPU memory 128 M for M40, 256M for M20, M40e and M160. - Increased CPU speed to 256 MHZ. - First shipped with JUNOS 5.5 Sep 2002.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

IP II ASIC
Performance: 40 Mpps, 40 byte with 80K prefixes at routing table. Packet processing features: Filtering, sampling, logging, counting, load balancing All M-series have enhanced S-board which as IP II ASIC. M5/10 doesnt have enhanced S-board.

T-series might contain as many as 16 IP II ASIC. Each FPC has one or two PFE which contains its own IP II ASIC.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Craft Interface
What is it? Collection of mechanisms on M-series and T-series View System status messages Trouble shooting Where is it? On the front of the chassis What does it have? System status LEDs FPC/PIC online/offline buttons. LCD screen provide status reporting for the entire system. What alternatives on other platforms? M7i: FIC (Fixed Interface Card)provide PIC offline/online buttons M10i: HCM (High-Availability Chassis Manager) Card provide PIC offline/online bottons.
Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

Password recovery
Connect to console Power cycle the RE and watch it booting up Enter a space character at the boot loader quick help manue to get a command prompt (dont enter space too quickly) Enter boot s When system boots up, answer recovery to recover password Follow the on-screen steps to change password Commit the change Reboot the system again.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using syslog message

Step 1: Get the stack trace from syslog messages
lab@hissy> show log messages | find "machine check" Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 machine check caused by error on the PC I Bus Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error detect register 1: 0x08, 2: 0x00 Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error ack count = 0 Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error address: 0x08004014 Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 PCI bus error status register: 0x02 Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 was the PCI master Dec 5 01:51:17 hissy tnp_sfm_3 C/BE bits: I/O read [0b0010] Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error detection reg1: PCI cycle Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 PCI status reg: parity error Dec 5 01:51:17 hissy tnp_sfm_3 ^B Dec 5 01:51:17 hissy tnp_sfm_3 last message repeated 7 times Dec 5 01:51:17 hissy tnp_sfm_3 Registers: Dec 5 01:51:17 hissy tnp_sfm_3 R00: 0x000e8c4c R01: 0x0775dad4 R02: 0x0000334 4 R03: 0x00000000 Dec 5 01:51:17 hissy tnp_sfm_3 R04: 0x0775dae0 R05: 0x00142e34 R06: 0x06006b3 6 R07: 0x00006b36 Dec 5 01:51:17 hissy tnp_sfm_3 R08: 0x00142e4c R09: 0x88000000 R10: 0x0000000 0 R11: 0x00000000 Dec 5 01:51:17 hissy tnp_sfm_3 R12: 0x00100004 R13: 0x000cc411 R14: 0x0000c43 0 R15: 0x00040000

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using syslog message

Dec 5 01:51:17 hissy tnp_sfm_3 R16: 0x00000000 R17: 0x00041410 R18: 0x0004c42 0 R19: 0x8004c618 Dec 5 01:51:17 hissy tnp_sfm_3 R20: 0x0002c490 R21: 0x00110000 R22: 0x0000000 Juniper Confidential. For Internal use only. 0 R23: 0x001151cc Dec 5 01:51:17 hissy tnp_sfm_3 R24: 0x00000001 R25: 0x00000000 R26: 0x0775db1 4 R27: 0x06006b36 Dec 5 01:51:17 hissy tnp_sfm_3 Stack Traceback: Dec 5 01:51:17 hissy tnp_sfm_3 Frame 01: sp = 0x0775dad4, pc = 0x000e8c4c Dec 5 01:51:17 hissy tnp_sfm_3 Frame 02: sp = 0x0775db0c, pc = 0x0005cd9c Dec 5 01:51:17 hissy tnp_sfm_3 Frame 03: sp = 0x0775db34, pc = 0x00108914 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 04: sp = 0x0775db4c, pc = 0x00108888 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 05: sp = 0x0775db54, pc = 0x000eec84 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 06: sp = 0x0775db5c, pc = 0x00037e78 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 07: sp = 0x0775dc1c, pc = 0x000380f8 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 08: sp = 0x0775dcfc, pc = 0x000eeadc Dec 5 01:51:17 hissy tnp_sfm_3 Frame 09: sp = 0x0775dd2c, pc = 0x000eefd0 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 10: sp = 0x0775dd3c, pc = 0x000f0184 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 11: sp = 0x0775dd74, pc = 0x000b28cc Dec 5 01:51:17 hissy tnp_sfm_3 Frame 12: sp = 0x0775dd84, pc = 0x000b29f4 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 13: sp = 0x0775ddac, pc = 0x000b2a8c Dec 5 01:51:17 hissy tnp_sfm_3 Frame 14: sp = 0x0775ddcc, pc = 0x000b2c80 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 15: sp = 0x0775ddec, pc = 0x000b2d5c Dec 5 01:51:17 hissy tnp_sfm_3 Frame 16: sp = 0x0775de04, pc = 0x0002665c

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using syslog message

What do I want? I will copy the following into a file called stack
single% cat stack
Dec 5 01:51:17 hissy tnp_sfm_3 Stack Traceback: Dec 5 01:51:17 hissy tnp_sfm_3 Frame 01: sp = 0x0775dad4, pc = 0x000e8c4c Dec 5 01:51:17 hissy tnp_sfm_3 Frame 02: sp = 0x0775db0c, pc = 0x0005cd9c Dec 5 01:51:17 hissy tnp_sfm_3 Frame 03: sp = 0x0775db34, pc = 0x00108914 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 04: sp = 0x0775db4c, pc = 0x00108888 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 05: sp = 0x0775db54, pc = 0x000eec84 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 06: sp = 0x0775db5c, pc = 0x00037e78 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 07: sp = 0x0775dc1c, pc = 0x000380f8 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 08: sp = 0x0775dcfc, pc = 0x000eeadc Dec 5 01:51:17 hissy tnp_sfm_3 Frame 09: sp = 0x0775dd2c, pc = 0x000eefd0 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 10: sp = 0x0775dd3c, pc = 0x000f0184 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 11: sp = 0x0775dd74, pc = 0x000b28cc Dec 5 01:51:17 hissy tnp_sfm_3 Frame 12: sp = 0x0775dd84, pc = 0x000b29f4 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 13: sp = 0x0775ddac, pc = 0x000b2a8c Dec 5 01:51:17 hissy tnp_sfm_3 Frame 14: sp = 0x0775ddcc, pc = 0x000b2c80 Dec 5 01:51:17 hissy tnp_sfm_3 Frame 15: sp = 0x0775ddec, pc = 0x000b2d5c Dec 5 01:51:17 hissy tnp_sfm_3 Frame 16: sp = 0x0775de04, pc = 0x0002665c

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using syslog message

Step2: Find out which version and build of the image.
So it is on M160, 4.4B3.2 and build 4.4-20010408-b20191 lab@hissy> show version brief Hostname: hissy Model: m160 JUNOS base [4.4B3.2] (Export restricted edition) JUNOS Kernel Software Suite [4.4-20010408-b20191] JUNOS Routing Software Suite [4.4-20010408-b20191] JUNOS Packet Forwarding Engine Support [4.4-20010408-b20191] JUNOS Online Documentation Files [4.4-20010408-b20191]

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using syslog message

Step 3: Find out which symbol file to use.
debug package for the crashing code if the crash is in the kernel or routing, or the normal package for the PFE. The perl script jemsym can be used to decode the stack. Recent dailies; single% cd /volume/build single% ls 20010201-0805@ 20010217-0805@ 20010305-0805@ 20010320-0910@ 200104050810@ 20010202-0805@ 20010218-0805@ 20010306-0805@ 20010321-0910@ 200104060810@ older dailies for released versions; single% cd /volume/ftp/private/unregressed/ single% ls 3.4/ 4.0/ 4.1/ 4.2/ 4.3/ 4.4/ 5.0/ released code; single% cd /volume/ftp/private/junos/ single% ls 4.0B1/ 4.0R5/ 4.1R4/ 4.3B1.2/ 4.4B2.1/ 4.0B2/ 4.1B1.1/ 4.2B1.1/ 4.3B2.1/ 4.4B3.2/

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using syslog message

single% cp /volume/build/20010408-0810/jpfe-4.4-20010408-b20191-debug.tgz . single% tar zxfv jpfe-4.4-20010408-b20191-debug.tgz +CONTENTS +COMMENT fpc.sym - M20/M40 fpc stack traces +DESC +INSTALL fpc160.sym -- M160 fpc stack traces +REQUIRE usr/share/pfe/scb.jbf sbr.sym -- M5/M10 stack traces usr/share/pfe/scb.sym usr/share/pfe/scb.elf scb.sym -- M40/M20 S-Board traces usr/share/pfe/fpc.jbf usr/share/pfe/fpc.sym sfm.sym --M160 SFM traces. usr/share/pfe/fpc.elf usr/share/pfe/sfm.jbf usr/share/pfe/sfm.sym usr/share/pfe/sfm.elf usr/share/pfe/fpc160.jbf usr/share/pfe/fpc160.sym usr/share/pfe/fpc160.elf usr/share/pfe/sbr.jbf usr/share/pfe/sbr.sym usr/share/pfe/sbr.elf
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Coredump analysis using syslog message

What is Jemsym file?
#!/usr/local/bin/perl ## $Id: jemsym,v 1.7 1998/04/21 01:15:33 jim Exp $ ## This file takes a Juniper panic stack trace and turns it # into a user-readable output from the symbol table file # for the running micro-kernel. Juniper Confidential. For Internal use only. ## By default, gmake produces a symbol table file for each # target, and then you run the text of the panic stack trace, # perhaps saved to a temporary file, as follows: ## cat temp-backtrace_file | jemsym target.sym

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using syslog message

Step 4: Do the stack trace
single% cat stack | ~dbovis/bin/jemsym usr/share/pfe/sfm.sym 0x000e8c4c cchip_ab_pio (0x000e8b2c) +0x120 0x0005cd9c pfe_bmemchip_pio_write (0x0005cd44) +0x58 0x00108914 bchip_write_sram_opaque (0x00108898) +0x7c 0x00108888 bchip_write_sram_hton (0x00108878) +0x10 0x000eec84 bchip_write_sram_mem_val (0x000eec64) +0x20 0x00037e78 diags_pfe_mem_address_test (0x00037dfc) +0x7c 0x000380f8 diags_pfe_mem_test (0x0003802c) +0xcc 0x000eeadc bchip_mem_test (0x000eea08) +0xd4 0x000eefd0 bchip_diags_sram_test (0x000eef30) +0xa0 0x000f0184 bchip_probe_diag (0x000f00fc) +0x88 0x000b28cc cm_probe_slot (0x000b284c) +0x80 0x000b29f4 cm_probe_slots (0x000b297c) +0x78 0x000b2a8c cm_probe_chassis (0x000b2a64) +0x28 0x000b2c80 cm_probe_event_loop (0x000b2b98) +0xe8 0x000b2d5c cm_probe_thread_init (0x000b2ca8) +0xb4 0x0002665c thread_suicide (0x0002665c) +0x0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using syslog message

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using core files

Where to get coredump files? 1) Coredump files are stored at: /volume/ftp/pub/incomfing/<case_number>/<core_filenma> For Example:
/volume/ftp/pub/incoming/2008-0104-0511

2) For some freaking .tgz file, you need to do this gunzip < cosd.core-tarball.0.tgz.2 | tar -xvf Using GUI
http://jtac-tools.juniper.net/crashdecode/coredump.html

Using Manual methods: Step 1: Using Jdebug to find out the stack traces. jdebug='/volume/buildtools/bin/jdebug /volume/buildtools/bin/jdebug <core_file name> Examples: The core file is saved at /volume/ftp/pub/incoming/2008-0104-0511/core-SSB0.core.0 Step 2: Use query-pr to find out the possible PRs based on the stack trace. query-pr -m "thread_debug" -m "sched_suspend_thread" summary

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis using core (continued)

-bash-2.05b$ /volume/buildtools/bin/jdebug core-SSB0.core.0 GNU gdb 6.5 juniper_2006a_411 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "--host=i386-unknown-freebsd4.11 --target=powerpc-juniper-eabi". #0 0x000330a0 in panic ( format_string=0x25f204 "CCHIP: Too many SRAM parity errors; restart required\n") at ../ukern/cpu-ppc/ppc603e_panic.c:63 63 asm volatile ("sc"); (gdb) bt #0 0x000330a0 in panic ( format_string=0x25f204 "CCHIP: Too many SRAM parity errors; restart required\n") at ../ukern/cpu-ppc/ppc603e_panic.c:63 #1 0x0018bf7c in cchip_error_hardware (C=0x35, hwerror=402653184) at ../common/drivers/cchip/cchip_int.c:238 #2 0x0018c158 in cchip_error_scan () at ../common/drivers/cchip/cchip_int.c:352 #3 0x0006baec in pfe_error_scan (info=0x0) at ../common/toolkits/pfe/pfe_scb.c:172 #4 0x000da8c8 in cm_handle_pfe_error (rate_limit=FALSE) at ../common/applications/cm/cm_pfe_restart.c:1463 #5 0x000dabc0 in cm_restart_handle_timer_event (timer=0x35) at ../common/applications/cm/cm_pfe_restart.c:1652 #6 0x000daff0 in cm_restart_event_loop () at ../common/applications/cm/cm_pfe_restart.c:1898 #7 0x00026fa0 in thread_wake (thread=0x210000) at ../ukern/common/thread.c:572 (gdb)
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Coredump analysis core file from special image

Step 1: to find out the image path using what on image or core file.
scb release 8.2I20071212_2313_pgoyette built by pgoyette on 2007-12-12 23:14:53 UTC jtac-bbuild01.juniper.net:/b/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb -bash-2.05b$ cd /volume/nfsbuild40 -bash-2.05b$ ls jcano pgoyette ramanathan sdoshi yuris

-bash-2.05b$ what core-SSB0\[1\].core.3 core-SSB0[1].core.3:

So the whole path is: /volume/nfsbuild40/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb Step 2: Find out the *.elf file. In the above case, it is scb.elf under the above path.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis core file from special image

Soemtimes it take more trouble to untar the compressed jpfe file to get the elf file.

lab@iggy> show version brief | grep packet JUNOS Packet Forwarding Engine Support [4.0-20000608s22432] (From above number I dont know where to get the jpfe file) single% tar zxfv jpfe-4.0-20000608-regressed-debug.tgz +CONTENTS +COMMENT +DESC +INSTALL +REQUIRE usr/share/pfe/scb.jbf usr/share/pfe/scb.sym usr/share/pfe/scb.elf fpc.sym M20/M40 fpc stack traces usr/share/pfe/fpc.jbf fpc160.sym M160 fpc stack traces usr/share/pfe/fpc.sym sbr.sym M5/M10 stack traces usr/share/pfe/fpc.elf usr/share/pfe/sfm.jbf scb.sym M40/M20 S-Board traces usr/share/pfe/sfm.sym sfm.sym M160 SFM traces. usr/share/pfe/sfm.elf usr/share/pfe/fpc160.jbf usr/share/pfe/fpc160.sym usr/share/pfe/fpc160.elf
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Coredump analysis core file from special image

-bash-2.05b$ /volume/cross/cygnus-i386-ppc/bin/gdb-core.ppc -nw /volume/nfsbuild40/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb/scb.elf coreSSB0[1].core.3
GNU gdb 4.16-97r2a Copyright 1997 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are This GDB was configured as "--host=i386-unknown-freebsd2.2.5 --target=powerpc-eabi"... #0 topo_connect (topo=0xd5af08, next=0x28, reconnect=FALSE) at ../common/toolkits/topo/topo.c:428 ../common/toolkits/topo/topo.c:428: No such file or directory. (gdb) bt ----------------------------------------------------------#0 topo_connect (topo=0xd5af08, next=0x28, reconnect=FALSE) at ../common/toolkits/topo/topo.c:428 #1 0x155a84 in nh_indirect_add_sub (nh=0x2163a3c, unilist=0x0, indirect_elementpp=0x2163a98) at ../common/applications/nh/nh_indirect.c:193 #2 0x155a84 in nh_indirect_add_sub (nh=0x2163a3c, unilist=0x0, indirect_elementpp=0x2163a98) at ../common/applications/nh/nh_indirect.c:193 # at ../common/applications/pfeman/pfeman_rt.c:413 #11 0x276cc in thread_suicide () at ../ukern/common/thread.c:951

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis Kernel core of special image

Find out where is the symbol file by using what. Ex: /volume/nfsbuild40/pgoyette/VZ8.2I20071212_2313/ship/ jkernel8.2I20080311_1541_jtac-builder-debug.tgz

copy the jkernel file to your home directory and unzip it. Ex: gunzip < jkernel-8.2I20080311_1541_jtacbuilder-debug.tgz | tar -xvf Debug the vmcore.0 file Ex: gdb -k kernel.debug vmcore.0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis daemon crash

1) uncompress the freaking core *.tgz file
gunzip < cosd.core-tarball.2.tgz | tar -xvf cosd.core.0 juniper.conf messages cosd.info.0 juniper.conf.1.gz
2) Where is the symbol file by doing what bash-2.05b$ what cosd.core.0 cosd.core.0: COSD release 7.3R3.6 built by builder on 2006-02-01 08:03:43 UTC xathanon.juniper.net:/build/xathanon-c/7.3R3.6/obji386/juniper/usr.sbin/cosd getsubopt.c 8.1 (Berkeley) 6/4/93 Copyright (c) 1994 Powerdog Industries. All rights reserved.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis daemon crash

3) Decode the core file
-bash-2.05b$ gdb /build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd/cosd cosd.core.0 GNU gdb 4.18 (FreeBSD) Copyright 1998 Free Software Foundation, Inc. -bash-2.05b$ gdb /build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd/cosd cosd.core.0 GNU gdb 4.18 (FreeBSD) Copyright 1998 Free Software Foundation, Inc. Core was generated by `cosd'. Program terminated with signal 11, Segmentation fault. /usr/lib/libisc.so.2: No such file or directory. #0 0x806d6f2 in cos_ifd_configure (dop=0x81e4300, conf=0x81ba000, name=0xbfbff850 "ge-0/3/0", match_len=10, wc_match=0 '\000', ifd_has_ieee_classifier=1 '\001', errmsg=0xbfbffc70 "", errmsglen=256) at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:2705 2705 cos_ifd->if_flags |= COS_IFD_CONF_F_IEEE_CLASSIFIER; (gdb) bt #0 0x806d6f2 in cos_ifd_configure (dop=0x81e4300, conf=0x81ba000, name=0xbfbff850 "ge-0/3/0", match_len=10, wc_match=0 '\000', ifd_has_ieee_classifier=1 '\001', errmsg=0xbfbffc70 "", errmsglen=256) at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:2705 #1 0x806f851 in cos_config_interfaces (dop=0x81e4280, conf=0x81ba000, errmsg=0xbfbffc70 "", errmsglen=256) at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:3944
#2 0x807bb53 in cos_config (conf=0x81ba000, errmsg=0xbfbffc70 "", errmsglen=256) at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:10816 #3 0x807be0e in cosd_parse_config (cos_conf=0x81ba000, check_only=0 '\000') at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:10924 #4 0x8069ac4 in main (argc=1, argv=0xbfbffe0c) at ../../../../src/juniper/usr.sbin/cosd/cosd_main.c:330 (gdb) l 2700 2701 } else { cos_ifd = cos_pat_to_ifd(pnode);

2702
2703 2704

}
if (ifd_has_ieee_classifier) {

2705 cos_ifd->if_flags |= COS_IFD_CONF_F_IEEE_CLASSIFIER; 2706 2707 2708 2709 /* * in commit check, cosd hasn't built its interface data }

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis Software or Hardware issues?

Case #1 Panic, TLB Data miss, Data access etc type of system exceptions:most probably software related. What you should do is to enable the coredump on the chassisd and gather all the base information mentioned above. Case #2: pci parity error being reported on the CPU DRAM address space, this means that this is a bogus pci error. The reason is, there is no pci bus connected to the CPU DRAM. Action: In this case, we have to enable the coredump on chassisd and get the coredump of the PFE component along with the base information. No RMA should be issued. Example: mpc106 machine check caused by error on the PCI Bus mpc106 error detect register 1: 0x08, 2: 0x00 mpc106 error ack count = 2 mpc106 error address: 0x001d0048 < belongs to CPU DRAM mpc106 PCI bus error status register: 0x02 mpc106 was the PCI master C/BE bits: I/O read [0b0010] mpc106 error detection reg1: PCI cycle mpc106 PCI status reg: parity error < parity error.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Coredump analysis Software or Hardware issues?

Case #3: There is parity protection enabled (ECC is disabled) on the CPU DRAM, if a hw failure occurs here, the message that you should see is: "memory parity/ECC error". Action: Run the memory diagnostics tests and RMA.

Example: mpc106 machine check caused by error on the Processor Bus < reported by Processor Bus mpc106 error detect register 1: 0x04, 2: 0x00 mpc106 error ack count = 0 mpc106 error address: 0x02f39e18 mpc106 Processor bus error status register: 0x72 transfer type 0b01110, transfer size 2 mpc106 error detection reg1: memory parity/ECC error < parity error. mpc106 PCI status reg: parity error
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Coredump analysis Software or Hardware issues?

Monitoring - logs
Step 1: configure logging file Example: isis { traceoptions { file mike-isis; flag state; flag error; flag spf; flag lsp receive detail; } Step 2: monitor start <log-file-name>

Step 3: monitor start message Example:

lab@falcons> monitor start mike-isis lab@falcons> monitor start messages lab@falcons> *** mike-isis *** Feb 5 20:05:53.517506 Updating LSP falcons.00-00 in database Feb 5 20:05:53.517654 Updating L2 LSP falcons.00-00 in TED
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Booting up system
request system snapshot partition as-primary request system media usb request system reboot media usb - when reboot from another media, all
the file systems will be under this media.

request system snapshot part as-primary media compact-flash request system reboot media compact request system software add /var/tmp/junojseries-8.4R2.4domestic.tgz no-validate Request system snapshot -- make a image at another storage(if you request system software delete backup

are using disk, this will mirror the image to CF. If you are using CF, this will makes an image at disk.

request system storage cleanup

To remove swap space at the compact-flash: http://www.juniper.net/techpubs/software/junos/jun os85/rn-sw-85

Tools and quick reference http://clie.juniper.net /volume/build - junos releases and source code. After 8.4, go to extra hierarchy /volume/build/junos. For example: /volume/build/junos/8.4/release/8.4R2.4/ship http://jam.jnpr.net http://www-in.juniper.net/eng/cvs_pdf/ https://deepthought.juniper.net/app/ http://cvs/cgi-bin/viewcvs.cgi/ http://confluence.jnpr.net/ /volume/current - cvs functional specs /volume/labcores http://rogers.jtacemea.jnpr.net/wiki/index.php?title=Enginee
Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

How to find out what syslog means?

[email protected]> help syslog SNMPD_SUBAGENT_NO_RESOURCES Name: SNMPD_SUBAGENT_NO_RESOURCES Message: No resources available for subagent (<subagent-name>): <error-message> Help: Subagent resources were temporarily exhausted Description: The SNMP agent process (snmpd) uses certain resources for communication with subagents. Resources were not available for communication with the indicated subagent. Type: Error: An error occurred Severity: notice Cause: An internal software failure occurred. Action: Contact your technical support representative.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

How to find out the data between 2 proc sockets?

1. Find out the processes ID (use snmpd and mib2d as example)
4Feb08 0:12.24 /usr/sbin/snmpd -N 4Feb08 0:10.35 /usr/sbin/mib2d N root@Kelly_RE0% ps -aux | egrep -i "snmpd|mib2d" root 8322 0.0 0.2 5036 3932 ?? S root 8302 0.0 0.2 4464 3892 ?? I 2. Find out socket stream. root@Kelly_RE0% fstat -p 8302

USER
..... root USER .....

CMD
mib2d CMD

PID FD MOUNT

INUM MODE

SZ|DV R/W

8302 17* local stream faab6c80 <-> fab03e60 PID FD MOUNT INUM MODE SZ|DV R/W

root@Kelly_RE0% fstat -p 8322

root

snmpd

8322 15* local stream fab03e60 <-> faab6c80

3. Then, check the socket data. root@Kelly_RE0% netstat -Aan | egrep -i "mib2d|snmpd|Send" PCB PCB Proto Recv-Q Send-Q Local Address Proto Recv-Q Send-Q Local Address Inode Conn 0 0 0 0 0 0 0 0 0 0 faad35a0 0 faa47aa0 0 fab67dc0 Foreign Address Foreign Address (state) (state)

Address Type Recv-Q Send-Q f5f4e6c0 stream f5f4b300 stream f5f4fc20 stream

Refs Nextref Addr 0 /var/run/snmpd_stream 0 /var/run/snmpd_stream 0

Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

How to do RMA?
1. Logistics
csr-apac(emea, usa)

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Trouble shoot T-series

show chassis hardware

show pfe statistics traffic show interface [int] extensive start shell su vty fpc[x] show sys mess show nvram show lchip ifd show ifl brief show lchip [x] error

show lchip [x] lout hw nlif show lchip [x] stream [stream_#] show lchip [x] lout registers lsif lsif [stream_#]

( where [stream_#] is the stream you found which corresponds to the

interface that has the problem using the show lchip ifd output above ) show lchip [x] lout registers nlif nlif

show lchip [x] lout stat

show lchip [x] lout sw lsif show lchip [x] lout sw desrd show lchip [x] lout sw hdrf show lchip [x] lout sw nlif show lchip [x] lout hw lsif show lchip [x] lout hw nlif show lchip [x] lout hw hdrf

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Trouble shoot T-series

start shell
su vty fpc[x] show sys mess show nvram show lchip ifd show ifl brief show lchip [x] error show lchip [x] lout stat show lchip [x] lout sw lsif show lchip [x] lout sw desrd

(where [stream_#] is the stream you have seen on the "show lchip ifd"

output under the lchip [x])

show lchip [x] lout registers nlif nlif show lchip [x] lout reg nlif dbufpart show lchip [x] lout reg nlif bdispmon Wait a little, hopefully after a few more errors go by. show nchip [x] all show mq [x] wan stat show mq [x] wan stream active stat

show lchip [x] lout sw hdrf

show lchip [x] lout sw nlif show lchip [x] lout hw lsif show lchip [x] lout hw nlif show lchip [x] lout hw hdrf show lchip [x] lout hw nlif

Show chassis fabric topology Show chassis show lchip [x] stream [stream_#] fabric sibs Show chassis fabric fpcs show lchip [x] lout registers lsif lsif [stream_#]
Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

How to trouble shoot SNMP and MIB2d

rtsockmon -c mib2d rtsockmon -ge mib2d show snmp statistics extensive netstat an show system virtual-memory [edit snmp] lab@Johnny-re1# show community public; traceoptions { file test size 10m; flag all; }
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

How to trouble shoot routing and forwarding issues?

FPC7(FED1DSRJ01-LAB-re0 vty)# show route ip prefix 192.12.1.2 IPv4 Route Table 0, default.0, 0x0: Destination NH IP Addr Type NH ID Interface --------------------------------- --------------- -------- ----- -------192.12.1.2 Hold 716 ge-7/0/4.0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

How to trouble shoot routing and forwarding issues?

install@FED1DSRJ01-LAB-re0> show route forwarding-table destination 192.12.1.2 Routing table: inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif 192.12.1.2/32 dest 1 192.12.1.2 hold 716 2 ge-7/0/4.0
Routing table: __juniper_private1__.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 116 1 Routing table: __juniper_private2__.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 196 1 Routing table: FED1J1MIS.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 521 1 Routing table: TEST-L3VPN.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 530 1

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

How to trouble shoot routing and forwarding issues?

install@FED1DSRJ01-LAB-re0> show arp MAC Address Address Name Flags 02:01:00:00:00:05 10.0.0.5 10.0.0.5 none 00:04:80:9d:b5:00 10.1.1.1 10.1.1.1 none 00:0c:29:9a:e5:38 10.1.1.115 10.1.1.115 none 00:05:85:9b:5d:f5 31.1.1.2 31.1.1.2 none 00:14:f6:56:b8:7e 68.1.0.204 68.1.0.204 none 02:01:00:00:00:05 128.0.0.5 128.0.0.5 none 00:00:c0:10:01:02 192.16.1.2 192.16.1.2 none Total entries: 7
Interface em0.0 fxp0.0 fxp0.0 ge-7/0/3.493

ge-7/1/0.0
em0.0 ge-7/0/5.0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

How to trouble shoot routing and forwarding issues?

ge-7/1/0.0
em0.0 ge-7/0/5.0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

How to trouble shoot routing and forwarding issues?

install@FED1DSRJ01-LAB-re0> show route protocol ospf

inet.0: 260 destinations, 387 routes (186 active, 0 holddown, 77 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both 0.0.0.0/0 *[OSPF/10] 09:25:03, metric 16777215 Discard 3.1.1.0/24 *[OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.1.0.0/16 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.1.1.0/24 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.1.200.0/28 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.99.0.0/16 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.99.99.0/24 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 24.234.6.0/24 *[OSPF/10] 00:54:30, metric 182 > to 68.1.0.204 via ge-7/1/0.0 24.234.6.0/27 *[OSPF/10] 00:54:30, metric 166 > to 68.1.0.204 via ge-7/1/0.0 24.248.129.0/27 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

How to trouble shoot routing and forwarding issues?

FFPC7(FED1DSRJ01-LAB-re0 vty)# show route ip prefix 192.12.1.2 IPv4 Route Table 0, default.0, 0x0: Destination NH IP Addr Type NH ID Interface --------------------------------- --------------- -------- ----- --------192.12.1.2 192.12.1.2 Unicast 716 ge-7/0/4.0 FFPC7(FED1DSRJ01-LAB-re0 vty)# show route ip lookup 192.12.1.2 Route Information (192.12.1.2): interface : ge-7/0/4.0 (87) Nexthop prefix : 192.12.1.2 Nexthop ID : 716 MTU : 1514 Class ID :0
FFPC7(FED1DSRJ01-LAB-re0 vty)#
Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

How to trouble shoot routing and forwarding issues?

install@FED1DSRJ01-LAB-re0> show interfaces filters ge-7/0/4 Interface Admin Link Proto Input Filter Output Filter ge-7/0/4 up up ge-7/0/4.0 up up inet multiservice FFPC7(FED1DSRJ01-LAB-re0 vty)# show nhdb interface ge-7/0/4 ID Type Interface Next Hop Addr Protocol Encap MTU ----- -------- ------------- --------------- ---------- ------------ ---625 Bcast ge-7/0/4.0 IPv4 Ethernet 0 626 Receive ge-7/0/4.0 192.12.1.0 IPv4 Ethernet 0 628 Resolve ge-7/0/4.0 IPv4 Ethernet 0 716 Unicast ge-7/0/4.0 192.12.1.2 IPv4 Ethernet 1514

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Agilent Router Tester. Remote access: Top 3 chassis: 172.19.59.28 Bottom 3 chassis: 172.19.58.12 User name: Administrator Password: n2x Launch pad Create new session For FE, need to config SFP
IXIA: VNC 172.19.58.2 (SV) 172.25.84.219(HD) ixia-2.jtac-west IXIA application server: 172.19.58.17
Proprietary and Confidential www.juniper.net

Lab stuff

Copyright 2007 Juniper Networks, Inc.

How to trouble shoot EOAM?

http://www.juniper.net/techpubs/software/junos /junos82/swconfig82-networkinterfaces/html/interfaces-ethernetconfig50.html#1272612
http://www.juniper.net/techpubs/software/junos /junos82/swconfig82-networkinterfaces/html/interfacessummary298.html#11618684

Known PRs: -PR81057

How to trouble shoot EOAM?

protocols { oam { ethernet { link-fault-management { interfaces { [xge/ge/fe]-<fpc>/<pic>/<port> { pdu-interval <value>; link-discovery <active|passive>; pdu-threshold <count>; remote-loopback; } } } } } }
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

How to trouble shoot EOAM?

How to Manually mount a USB/CF storage?

http://kb.juniper.net/KB8017
First upload the desired JUNOS image to the router via ftp to /var/tmp. Connect the USB mass storage device. Format the USB device by dropping to shell (start shell) then enter "dd if=/dev/zero of=/dev/da0 bs=128k" (root access required). Note this step can take several minutes to complete with no output to the CLI window. Label the device by entering "disklabel -r -w da0 auto". (!! if you move the USB/CF around, you need to execut this command before mounting) Create the file system with "newfs -U /dev/da0c". Create a dir to be used as a mount point with "mkdir /var/tmp/usb". Mount the USB device using "mount /dev/da0c /var/tmp/usb". df -h can be used to verify the mount. Copy the JUNOS install image to the USB device. cp /var/tmp/junos-jseries-8.0R2.8-domestic.tgz /var/tmp/usb Delete the original image to free up space on the CF. rm /var/tmp/junos-jseries-8.0R2.8-domestic.tgz Use the "request system software add /var/tmp/usb/junos-jseries8.0R2.8-domestic.tgz" command to install the new JUNOS version.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

How to do tcpdump at Junos?

You have to login as root You have to know which incoming interface? Command: root@bananas-re0% tcpdump -xvf -i so-1/1/0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ethernet OAM
Ethernet OAM types
In short, there are two types of Ethernet OAM: 1. Ethernet OAM as defined by 802.3ah This is referred as LFM (Link Fault Management) and are identified by the ether-type 0x8809 (slow protocol type packets), sub-type 3. 2. Ethernet OAM as defined by IEEE 802.1ag This is referred as CFM (Connectivity Fault Management) and can be by the ether-type 0x8902.

Ethernet OAM implementation in JunOS Ethernet OAM is implemented using the RE user space daemons "lfmd" and "cfmd". Also, both "lfmd" and "cfmd" use the "ppmd" daemon on the PFE for some periodic packet processing. There is a packet processing path in the RE kernel as well in addition to the daemons mentioned above.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ethernet OAM
Ethernet OAM for regular Ethernet interfaces
Both 802.3ah (LFM) and 802.1ag (CFM) type Ethernet OAMs are supported in JunOS for the regular Ethernet interfaces with the following restrictions. 802.3ah (LFM) type OAM can be configured only on the Ethernet IFDs and NOT on the Ethernet IFLs. Also, these packets are always VLAN untagged.

However, 802.1ag (CFM) type OAM can be configured either on an Ethernet IFD or IFL. If this is configured on an IFD, the packets will be always VLAN untagged. If this is configured on an IFL, it will be either VLAN tagged or untagged based on the "vlantagging" keyword configuration on an Ethernet IFD.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ethernet OAM
Link Monitoring Link monitoring in Ethernet OAM detects and indicates link faults under a variety of conditions. Link monitoring uses the event notification OAMPDU and sends events to the remote OAMentity when there are problems detected on the link. The error events include the following: Error Symbol Period (error symbols per second)The number of symbol errors that occurred during a specified period exceeded a threshold. These errors are coding symbol errors. Error Frame (error frames per second)The number of frame errors detected during a specified period exceeded a threshold. Error Frame Period (error frames per n frames)The number of frame errors within the last n frames has exceeded a threshold. Error Frame Seconds Summary (error seconds per m seconds)The number of error seconds (1-second intervals with at least one frame error) within the last m seconds has exceeded a threshold. Since IEEE 802.3ah OAM does not provide a guaranteed delivery of any OAM PDU, the event notification OAM PDU may be sent multiple times to reduce the probability of a lost notification. A sequence number is used to recognize duplicate events
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ethernet OAM one scenario (2008-0401-0623)

Scenario: Two T640s with JUNOS 8.2SR are connected together through an optical transport network (e.g., Fujitsu 7500/7600), using LAN-PHY on 10GE IQ2 PICs.
Question: If there is a link failure in the transport network and the 10GE links between the Fujitsu switches and the T640s stay up, will the Local T640 send out Ethernet 802.3ah OAMPDUs with the Flags for Critical Link Events(1) and the Link Event TLVs(2) to the Remote T640? Answer: No. None of that will happen. What will happen is, the OAM Discovery INFO PDUs will timeout and both sides will detect that and mark a failure on their respective links. If only one direction of the link is down, one side will be in "Active Send Local" state and the other side will be in "Send Local Remote" state. There is no reason to send Link Event TLVs in the above situation as it's a link fault, not a framing error. The reason we do not send Link-Fault or Dying Gasp is, by the time we detect a Rx fault, the ifd is marked down and the Tx is also brought down. The Critical Event is not defined in the 802.3ah for any specific purposes,and is implementation dependant. In Juniper implementation, we use Critical event to simulate RDI functionality. We only send Critical event in case we have a CCC-DOWN on the ifls on the interface marked by RPD and an action profile to send a critical event is defined.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ethernet OAM one scenario (2008-0401-0623)

syslog { archive { files number; size size; (world-readable | no-world-readable); } console { facility severity; } file filename { facility severity; explicit-priority; match "regularexpression"; archive { files number; size size; (world-readable | noworld-readable); } } host (hostname | otherrouting-engine | scc-master) { facility severity; explicit-priority; facility-override facility; log-prefix string; match "regularexpression"; } source-address source-address;timeformat (year | millisecond | year millisecond); user (username | *) { facility severity; match "regularexpression"; }}
Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

CoS configuration (2008-0523-0448)

http://www.juniper.net/techpubs/software/junos/junos90/swconfig-cos/frameset.html
In the following classifier example, packets with EXP bits 000 are assigned to the data-queue forwarding class with a low loss priority, and packets with EXP bits 001 are assigned to the data-queue forwarding class with a high loss priority. [edit class-of-service] classifiers { exp exp_classifier { forwarding-class data-queue { loss-priority low code-points 000; loss-priority high code-points 001; } } } In the following drop-profile map example, the scheduler includes two drop-profile maps, which specify that packets are evaluated by the low-drop drop profile if they have a low loss priority and are from any protocol. Packets are evaluated by the high-drop drop profile if they have a high loss priority and are from any protocol. [edit class-of-service] schedulers { best-effort { drop-profile-map loss-priority low protocol any drop-profile low-drop; drop-profile-map loss-priority high protocol any drop-profile high-drop; } } In the following rewrite rule example, packets in the be forwarding class with low loss priority are assigned the EXP bits 000, and packets in the be forwarding class with high loss priority are assigned the EXP bits 001. [edit class-of-service] rewrite-rules { exp exp-rw { forwarding-class be { loss-priority low code-point 000; loss-priority high code-point 001;

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

How to verify packages are corrupted?

root@% mount /altroot root@% mount /altconfig root@% cd /altroot/packages/
root@% sha1 j*8.5R3.4 SHA1 (jbase-8.5R3.4) = 51a9f2cfe95a53d1dbda2daedd6b5dd6dd66213c SHA1 (jdocs-8.5R3.4) = c56296f2016d5ddbf8b22c00cb8c06dc5c664271 SHA1 (jkernel-8.5R3.4) = fedc82d6e8edb6b5ff972ac4c0f22885841ee48e SHA1 (jpfe-T-8.5R3.4) = f8ea2b28cf27a168a1023b0e544cdfb047ac2f0e ---> corrupted SHA1 (jpfe-common-8.5R3.4) = 0034ccbd5bd1b2bbd9b9ee41d3b42c50443e5562 ---> corrupted SHA1 (jroute-8.5R3.4) = 5c22ca387a78d4a3cb47af79ef6bdcfa0e0bc26f root@% sha1 /packages/j*8.5R3.4 SHA1 (/packages/jbase-8.5R3.4) = 51a9f2cfe95a53d1dbda2daedd6b5dd6dd66213c SHA1 (/packages/jdocs-8.5R3.4) = c56296f2016d5ddbf8b22c00cb8c06dc5c664271 SHA1 (/packages/jkernel-8.5R3.4) = fedc82d6e8edb6b5ff972ac4c0f22885841ee48e SHA1 (/packages/jpfe-T-8.5R3.4) = f14de1eb8e537a35088864192d6838bb24804492 SHA1 (/packages/jpfe-common-8.5R3.4) = 270c4f4cc9c0afb6ba52c6916c2213eeba851ddc SHA1 (/packages/jroute-8.5R3.4) = 5c22ca387a78d4a3cb47af79ef6bdcfa0e0bc26f
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Class-of-Service trouble shooting

There is bug in Gimlet FPC where the PLP high defined at classifier will *NOT* be copied to notification. Thus if egress FPC might have rewrite rule messed up.
Gimlet FPC to Gimlet FPC has no problem. Gimble FPC to Stoli FPC has problem Gimlet FPC to Gimlet FPC with drop-profile has problem. To work around this problem for scenario 2 & 3:

1. 2. 3.

lab@slayer-re1# set class-of-service copy-plp Default forwarding class: Queue Forwarding-class 0 best-effort 1 Assured-forwarding 2 expedited-forwarding 3 network-control

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Class-of-Service trouble shooting

http://www.juniper.net/techpubs/software/junos/jun os90/swconfig-cos/swconfig-cos.pdf Table 43: Default MPLS EXP Rewrite Table(P230) -----------------------------------------------Forwarding Class
best-effort(0) best-effort expedited-forwarding(1) expedited-forwarding assured-forwarding(2) assured-forwarding network-control(3) network-control

Loss Priority CoS Value

low high low high low high low high 000 001 010 011 100 101 110 111
#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Class-of-Service trouble shooting

http://www.juniper.net/techpubs/software/junos/jun os90/swconfig-cos/swconfig-cos.pdf
PLP Value Map to DSCP/DSCP IPv6/ EXP/IEEE/IP low high low high low high low high ef ef af11 af12 (DSCP/DSCP IPv6/EXP) be be nc1/cs6 nc2/cs7

Table 42: Default Packet Header Rewrite Mappings (p225)

Map from Forwarding Class expedited-forwarding expedited-forwarding assured-forwarding assured-forwarding best-effort best-effort network-control network-control

The mapping of alias to EXP code point is at next slide. Same thing to look up alias to DSCP code point.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Class-of-Service trouble shooting

lab@slayer-re1> show class-of-service code-point-aliases exp Code point type: exp Alias Bit pattern af11 100 af12 101 be 000 be1 001 cs6 110 cs7 111 ef 010 ef1 011 nc1 110
nc2 111

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

PLP Treatment on LMNR Platforms Overview

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Problem
Customer Cox was seeing an increase of NonReal-Time class traffic in the network when replacing IQ2 10GE PICs by 10GE XENPAK (nonIQ2) PICs. Hard to isolate as there was a mix of traffic from different sources. Initially though the problem was due to missclasification.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Topology

LSP

IP unlabeled Traffic

xe-0/1/0
IP unlabeled Traffic
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Configuration: Forwarding Classes

> ...service forwarding-classes queue 0 BEST-EFFORT; queue 1 NON-REAL-TIME; queue 2 INTERACTIVE; queue 3 REAL-TIME; queue 4 VIDEO; queue 5 VOICE; queue 6 NETWORK-CONTROL;

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Configuration: IP-Prec. Classifier

forwarding-class BEST-EFFORT { loss-priority high code-points BEST-EFFORT-be; } forwarding-class NON-REAL-TIME { loss-priority high code-points NON-REAL-TIME-af11; } forwarding-class INTERACTIVE { loss-priority low code-points INTERACTIVE-af21; } forwarding-class REAL-TIME { loss-priority low code-points REAL-TIME-af31; } forwarding-class VIDEO { loss-priority low code-points VIDEO-af41; } forwarding-class VOICE { loss-priority low code-points VOICE-ef; } forwarding-class NETWORK-CONTROL { loss-priority low code-points NETWORK-CONTROL-nc1; }

inet-precedence { BEST-EFFORT-be 000;

NON-REAL-TIME-af11 001;
INTERACTIVE-af21 010; REAL-TIME-af31 011; VIDEO-af41 100; VOICE-ef 101; NETWORK-CONTROL-nc1 110; }

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Configuration: EXP Classifier

BEST-EFFORT-be 000; NON-REAL-TIME-af11 001; INTERACTIVE-af21 010; REAL-TIME-af31 011;

VIDEO-af41 100;
VOICE-ef 101; NETWORK-CONTROL-nc1 110;

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Configuration: Rewrite Rules, EXP

exp WRITE-EXP { forwarding-class BEST-EFFORT { loss-priority low code-point BEST-EFFORT-be; loss-priority high code-point BEST-EFFORT-be; } forwarding-class NON-REAL-TIME { loss-priority low code-point NON-REAL-TIME-af11; loss-priority high code-point NON-REAL-TIME-af11;

}
forwarding-class INTERACTIVE { loss-priority low code-point INTERACTIVE-af21; loss-priority high code-point INTERACTIVE-af21; } forwarding-class REAL-TIME { loss-priority low code-point REAL-TIME-af31; loss-priority high code-point REAL-TIME-af31; } forwarding-class VIDEO { loss-priority low code-point VIDEO-af41; loss-priority high code-point VIDEO-af41; } forwarding-class VOICE { loss-priority low code-point VOICE-ef; loss-priority high code-point VOICE-ef; }

forwarding-class NETWORK-CONTROL {
loss-priority low code-point NETWORK-CONTROL-nc1; loss-priority high code-point NETWORK-CONTROL-nc1; } Copyright 2007 Juniper Networks, Inc.
Proprietary and Confidential www.juniper.net

PLP handling
IQ2 PIC

Simple Filter

Lin

Jtree Lookup

Lout

MF Classifier

Rewrite Rule

BA Classifier

Non-IQ2 PIC

PIC

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Which PLP ?
The L to N notification cell contains two bits (three with tri-color marking) of interest: The pseudo-plp bit: This is bit 2 of the QoS field (6-bits), and its used by the Lin BA Classifier and Rewrite rules The real plp bit: this is a separate bit, see the Lin functional description for location.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

PLP On LMNR

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Example: IP packet, precedence 000, nonIQ2 PIC

Lets say we receive a packet with IP-Prec bits 000. Lets say we have a BA Classifier that classifies IP-Prec: 000 as Best-Effort (queue 0) and plp=high: # show class-of-service code-point-aliases inet-precedence BEST-EFFORT-be 000; NON-REAL-TIME-af11 001; INTERACTIVE-af21 010; REAL-TIME-af31 011; VIDEO-af41 100; VOICE-ef 101; NETWORK-CONTROL-nc1 110;

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Contd
# show class-of-service classifiers inet-precedence CLASSIFY-IPP forwarding-class BEST-EFFORT { loss-priority high code-points 000; } # show class-of-service forwarding-classes queue 0 BEST-EFFORT; queue 1 NON-REAL-TIME; queue 2 INTERACTIVE; queue 3 REAL-TIME; queue 4 VIDEO; queue 5 VOICE; queue 6 NETWORK-CONTROL;
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Ctd
Because this packets real-plp bit will remain 0, RED will treat it as such. If we have the following rewrite rule: apena@austinp-re0# show class-of-service rewrite-rules exp WRITE-EXP { forwarding-class BEST-EFFORT { loss-priority low code-point 000; loss-priority high code-point 000; <<<< }
#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Will this work ?

The answer is:
It depends on the incoming PIC. By default we OR the LSB of EXP and DSCP with the real PLP (see flow chart):
EXP 000 ORed with plp=1 gives EXP=001 This produces incorrect classification at next hop router With IQ2 PIC, Lin can write proper real PLP thanks to cookie. Without IQ2, Lin cant write real plp, just pseudo plp.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Workaround:
Use compatible markings Enable copy-plp hidden knob. Enable tri-color marking

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Multicast trouble shooting

lab@ 320_1> show pim rps extensive Instance: PIM.master Address family INET RP: 198.140.33.2 Learned from 198.140.33.7 via: auto-rp Time Active: 17w5d 05:03:53 Holdtime: 150 with 139 remaining Device Index: 134 Subunit: 32780 Interface: pe-2/0/0.32780 Group Ranges: 224.0.2.64/32, 139s remaining 224.0.2.65/32, 139s remaining 224.0.2.66/32, 139s remaining 224.0.2.67/32, 139s remaining Active groups using RP: 233.43.202.9 233.43.202.8
#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

IPSec configuration and troubleshooting

This is a wiki for a very bad Google IPSeT defrag case.
http://confluence.jnpr.net/confluence/dis play/IPGE/Google+2009-0106+IPSec+Fragmentation+Issue++PR+414885

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

IPSec configuration and troubleshooting

lab@kings-re0# show services
service-set ny2ny02jt-payload { max-flows 2m; next-hop-service { inside-service-interface sp-0/0/0.1; outside-service-interface sp-0/0/0.2; } ipsec-vpn-options { local-gateway 200.1.1.2; } ipsec-vpn-rules ny2ny02jt-payload; } ipsec-vpn { rule ny2ny02jt-payload { term 1 { then { remote-gateway 200.1.1.1; dynamic { ike-policy ny2ny02jt-payload; ipsec-policy stream; } tunnel-mtu 9188; } establish-tunnels immediately; } } } } ike { proposal rivlet { authentication-method pre-shared-keys; dh-group group1; authentication-algorithm md5; encryption-algorithm 3des-cbc; } } policy stream { proposals brook; ipsec { proposal brook {

protocol esp;
authentication-algorithm hmac-md5-96; encryption-algorithm 3des-cbc;

policy ny2ny02jt-payload {
mode main; proposals rivlet; pre-shared-key ascii-text "$9$O4v9BEyleWXxd"; ## SECRET-DATA

anti-replay-window-size 1024;
} } match-direction input; }
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

IPSec configuration and troubleshooting

On T640 or other platforms where you Direct traffic to the IPSec tunnel. have service PIC, you need to 1) Static route configure lab@kings-re0# show routing-options the SP interfaces. static {
route 172.0.0.0/8 {

lab@kings-re0# show interfaces sp-0/0/0 description ipsec-vpn;

next-hop 172.25.44.1; retain; no-readvertise; } route 0.0.0.0/0 { next-hop sp-0/0/0.1;

mtu 9192;
unit 1 { description ipsec-vpn-inside; family inet; service-domain inside; } unit 2 { description ipsec-vpn-outside; family inet; service-domain outside; }
} }

retain;

2) IGP 3) BGP

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

IPSec configuration and troubleshooting

lab@kings-re0# run ping 111.0.0.1 PING 111.0.0.1 (111.0.0.1): 56 data bytes

lab@jazz-re0> monitor traffic interface sp-0/0/0.1

verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay. Address resolution timeout is 4s. Listening on sp-0/0/0.1, capture size 96 bytes Reverse lookup for 111.0.0.1 failed (check DNS reachability). Other reverse lookup failures will not be reported. Use <no-resolve> to avoid reverse lookups on IP addresses.

64 bytes from 111.0.0.1: icmp_seq=0 ttl=64 time=1.335 ms

64 bytes from 111.0.0.1: icmp_seq=1 ttl=64 time=1.026 ms 64 bytes from 111.0.0.1: icmp_seq=2 ttl=64 time=1.050 ms 64 bytes from 111.0.0.1: icmp_seq=3 ttl=64 time=1.065 ms 64 bytes from 111.0.0.1: icmp_seq=4 ttl=64 time=1.032 ms 64 bytes from 111.0.0.1: icmp_seq=5 ttl=64 time=0.869 ms 64 bytes from 111.0.0.1: icmp_seq=6 ttl=64 time=1.078 ms

64 bytes from 111.0.0.1: icmp_seq=7 ttl=64 time=0.905 ms

64 bytes from 111.0.0.1: icmp_seq=8 ttl=64 time=1.073 ms 64 bytes from 111.0.0.1: icmp_seq=9 ttl=64 time=1.084 ms 64 bytes from 111.0.0.1: icmp_seq=10 ttl=64 time=0.885 ms 64 bytes from 111.0.0.1: icmp_seq=11 ttl=64 time=1.095 ms 64 bytes from 111.0.0.1: icmp_seq=12 ttl=64 time=0.948 ms 64 bytes from 111.0.0.1: icmp_seq=13 ttl=64 time=0.912 ms
19:03:10.506267 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 6, length 64
19:03:10.506285 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 6, length 64 19:03:11.507050 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 7, length 64 19:03:11.507061 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 7, length 64 19:03:12.507977 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 8, length 64 19:03:12.507988 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 8, length 64 19:03:13.508794 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 9, length 64 19:03:13.508802 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 9, length 64 19:03:14.509561 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 10, length 64

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

IPSec configuration and troubleshooting

lab@jazz-re0# run show log kmd
Jul 17 18:32:20 jazz-re0 clear-log[8331]: logfile cleared Jul 17 18:33:26 Initialising the KMD ipsec-interface-id pool Jul 17 18:33:26 Deleted SA pair with index=0 tunnel index=1 to kernel Jul 17 18:33:26 Initializing certificate manager Jul 17 18:33:26 Added SA pair with index=0 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel Jul 17 18:34:06 Added SA pair with index=1 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel Jul 17 18:34:11 Added SA pair with index=2 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel Jul 17 18:57:25 Initialising the KMD ipsec-interface-id pool Jul 17 18:57:38 Initialising the KMD ipsec-interface-id pool Jul 17 18:58:53 Initialising the KMD ipsec-interface-id pool Jul 17 19:31:56 Deleted SA pair with index=1 tunnel index=1 to kernel Jul 17 19:31:56 Added SA pair with index=3 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel Jul 17 19:34:11 Deleted SA pair with index=2 tunnel index=1 to kernel

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

IPSec configuration and troubleshooting

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

How to compare rollback?

rprivette@CHRL-HAGG-03> show system rollback compare 0 2
[edit interfaces ge-3/3/1 unit 3478] + + + + + + + } description "16/VLXX/010009/TWCS - FREEMAN WHITE # 255277 [ENLAN]"; description "16/KDFN/010010/TWCS - Freeman White # FW115671"; encapsulation vlan-vpls; encapsulation vlan-ccc; family ccc { policer { input LIMIT_10M; output LIMIT_10M;

+
-

}
family vpls { policer { input LIMIT_10M; output LIMIT_10M; } }

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

MX VLAN configuration what are the new stuff?

STPs: original 802.1D 1) MSTP: based on 802.1s 2) RSTP: based on 802.1w 3) MISTP: Cisco Proprietary Multiple Instance STP 4) PVST+: Per-VLAN spanning-tree plus 5) Rapid PVST+
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

MX VLAN Trunking configuration General guideline

Generally, there are four things that you must configure in an L2 environment: Interfaces and virtual LAN (VLAN) tagsL2 interfaces are usually various type of Ethernet links with VLAN tags used to connect to customer devices or other bridges or routers. Bridge domains and virtual switchesBridge domains limit the scope of media access control (MAC) learning (and thereby the size of the MAC table) and also determine where the device should propagate frames sent to broadcast, unknown unicast, and multicast (BUM) MAC addresses. Virtual switches allow for the configuration of multiple, independent bridge domains. Spanning Tree Protocols (xSTP, where the x represents the STP type)Bridges function by associating a MAC address with an interface, similar to the way a router associates an IP network address with a next-hop interface. Just as routing protocols use packets to detect and prevent routing loops, bridges use xSTP frames to detect and prevent bridging loops. (L2 loops are more devastating to a network because of the broadcast nature of Ethernet LANs.) Integrated bridging and routing (IRB)Support for both Layer 2 bridging and Layer 3 routing on the same interface. Frames are bridged if they are not sent to the router's MAC address. Frames sent to the router's MAC address are routed to other interfaces configured for Layer 3 routing.
Proprietary and Confidential www.juniper.net

Copyright 2007 Juniper Networks, Inc.

MX VLAN Trunking configuration vlan tagging

interfaces ge-2/2/6 {
encapsulation flexible-ethernet-services; vlan-tagging; # Customer interface uses singly-tagged frames unit 200 {
encapsulation vlan-bridge; vlan-id 200;

} interfaces ae1 {
encapsulation extended-vlan-bridge; vlan-tagging; unit 100 {
vlan-id 100;

} unit 200 {
vlan-id 200;

}
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

MX VLAN Trunking configuration bridge domain

Configure the virtual switches and bridge domains on all three routers. There is always a default virtual switch in the router for L2 functions; however, if there is only one L2 network, then the virtual switch instance type is not needed. Configure a bridge domain on Router 1: [edit] bridge-domains {
vlan100 {

domain-type bridge; vlan-id 100; interface ge-2/2/1.100; interface ae1.100; interface ae2.100;
} vlan200 {

domain-type bridge; vlan-id 200; interface ge-2/2/1.200; interface ge-2/2/6.200; interface ae1.200; interface ae2.200;
}

}
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

MX VLAN Trunking configuration MSTP-1

Key words: MSTI: Multiple Spanning Tree Instances CIST: Common and Internal Spanning Tree MSTP: Multiple Spanning Tree Protocol Configuration name: The names must match to

be in the same region

Revision Level: must be the same across the same region. VLAN-to-MSTI mapping: vlans mapped to this MSTP instance.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

MX VLAN Truncking configuration MSTP-2

protocols {
mstp {
configuration-name mstp-for-R1-2-3; # The names must match to be in the same region revision-level 3; # The revision levels must match bridge-priority 0; # This bridge acts as root bridge for VLAN 100 and 200 interface ae1; interface ae2; msti 1 { vlan100; # This VLAN corresponds to MSTP instance 1 } msti 2 { vlan200; # This VLAN corresponds to MSTP instance 2 }

}
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

MX VLAN Truncking configuration IRB-1

You configure IRB in two steps: (1) Configure the IRB interface using the irb statement. (2) Reference the IRB interface at the bridge domain level of the configuration. IRB supports Layer 2 bridging and Layer 3 routing on the same interface. If the MAC address on the arriving frame is the same as that of the IRB interface, then the packet inside the frame is routed. Otherwise, the MAC address is learned or looked up in the MAC address database.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

MX VLAN configuration IRB-2

edit interfaces] xe-2/1/0 {
unit 0 {
family inet { address 10.0.10.2/24; # Routing interface }

bridge-domains { vlan-100 { domain-type bridge; vlan-id 100; interface gefamily inet { address 10.0.1.2/24 { vrrp-group 1 { virtual-address 10.0.1.51; priority 254; } }

} irb {
unit 0 {

2/2/2.100; interface ae1.100; interface ae3.100 routing-interface irb.0; } vlan-200 { domain-type bridge; vlan-id 200; interface ge3/3/3.200; interface ae1.200;
#
virtual-address 10.0.2.51; priority 100;

}
unit 1 {
family inet { address 10.0.2.2/24 { vrrp-group 2 {

} }

Proprietary and Confidential

www.juniper.net

MX VLAN configuration- host interface

New CLI introduced at in the fix of PR 299511

lab@Atlas_re0# show interfaces ge-5/0/4 encapsulation ethernet-bridge; unit 0 { family bridge; } [edit] lab@Atlas_re0# show interfaces ge-0/0/4 encapsulation ethernet-bridge; unit 0 { family bridge; } Bridge-domain{ vlan333 { domain-type bridge; vlan-id 333; interface ge-5/0/4.0; interface ge-0/0/4.0; } }
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Firewall Troubleshooting
lab@slayer-re1> show firewall filter logas0.0-i
Filter: log-as0.0-i Counters: Name Packets rsvp-as0.0-i 0 ospf-as0.0-i 0 bgp-as0.0-i 0 all-as0.0-i 99975614
Copyright 2007 Juniper Networks, Inc.

Bytes 0

0
0

149963421000
#

Proprietary and Confidential

www.juniper.net

Firewall Troubleshooting -temp

lab@slayer-re1> show firewall filter logas0.0-i
Filter: log-as0.0-i Counters: Name Packets rsvp-as0.0-i 0 ospf-as0.0-i 0 bgp-as0.0-i 0 all-as0.0-i 99975614
Copyright 2007 Juniper Networks, Inc.

Bytes 0

0
0

149963421000
#

Proprietary and Confidential

www.juniper.net

Firewall Troubleshooting -temp

lab@slayer-re1> show firewall filter logas0.0-i
Filter: log-as0.0-i Counters: Name Packets rsvp-as0.0-i 0 ospf-as0.0-i 0 bgp-as0.0-i 0 all-as0.0-i 99975614
Copyright 2007 Juniper Networks, Inc.

Bytes 0

0
0

149963421000
#

Proprietary and Confidential

www.juniper.net

Firewall Troubleshooting -temp

lab@slayer-re1> show firewall filter logas0.0-i
Filter: log-as0.0-i Counters: Name Packets rsvp-as0.0-i 0 ospf-as0.0-i 0 bgp-as0.0-i 0 all-as0.0-i 99975614
Copyright 2007 Juniper Networks, Inc.

Bytes 0

0
0

149963421000
#

Proprietary and Confidential

www.juniper.net

Firewall Troubleshooting -temp

lab@slayer-re1> show firewall filter logas0.0-i
Filter: log-as0.0-i Counters: Name Packets rsvp-as0.0-i 0 ospf-as0.0-i 0 bgp-as0.0-i 0 all-as0.0-i 99975614
Copyright 2007 Juniper Networks, Inc.

Bytes 0

0
0

149963421000
#

Proprietary and Confidential

www.juniper.net

MX-960 pegasus DPC auto-nego

https://tools.online.juniper.net/cm/case_note_detail.jsp?cid=Up9%2FoWPEU 57FR9OFIsO0vQ%3D%3D&type=WQDDoTj%2Bp28%3D&num=fF6aYIY jhYCr4QBubu3%2BXg%3D%3D&isInternal=false

http://cvs.juniper.net/cgi-bin/viewcvs.cgi/swprojects/platform/atlas/pegasus/pegasus_unit_tes t_plan.txt?rev=1.3&view=markup
7. Speed/Duplex selection from RE CLI - 100m/full-duplex Goal: Test configuration of speed, link-mode from RE CLI Test Steps: 1. Issue the below command on RE CLI -> set interfaces ge-x/y/z speed 100m link-mode full-duplex -> commit 2. Issue the below command on DPC console -> "show bcm5466 registers y z" 3. Compare the values from "MII Control Register" with Broadcom 5466 data sheet. 4. Issue the below command on DPC console -> "show npez y rgmii z" Success Criteria: Description in the Data sheet should match with the values read. From output of step 4 verify rgmii rate Result: PASS Output: Step 2: MII Control Register (0x00) : 0x3100 Step 4: The rate of the RGMII port is 100Mb

Proprietary and Confidential

www.juniper.net

How to trouble shoot RSVP/LSP issues?

RSVP related operational mode commands: - clear rsvp session - show rsvp session - clear mpls lsp - show mpls lsp - show rsvp interface - show ted database extensive -

Proprietary and Confidential

www.juniper.net

How to trouble shoot RSVP/LSP issues?

[email protected]> show ted database 168.215.52.177 extensive TED database: 0 ISIS nodes 671 INET nodes NodeID: 168.215.52.177 Type: Rtr, Age: 271072 secs, LinkIn: 2, LinkOut: 2 Protocol: OSPF(0.0.0.0) To: 66.192.245.116-1, Local: 66.192.245.126, Remote: 0.0.0.0 Local interface index: 0, Remote interface index: 0 Color: 0 <none> Metric: 100 Static BW: 1000Mbps Reservable BW: 700Mbps Available BW [priority] bps: [0] 699.07Mbps [1] 699.07Mbps [2] 699.07Mbps [3] 699.07Mbps [4] 699.07Mbps [5] 699.07Mbps [6] 699.07Mbps [7] 699.07Mbps Interface Switching Capability Descriptor(1): Switching type: Packet Encoding type: Packet

To: 66.192.245.68-1, Local: 66.192.245.78, Remote: 0.0.0.0

Local interface index: 0, Remote interface index: 0 Color: 0 <none> Metric: 100 Static BW: 1000Mbps Reservable BW: 700Mbps Available BW [priority] bps:

[0] 699.21Mbps [1] 699.21Mbps [2] 699.21Mbps [3] 699.21Mbps

[4] 699.21Mbps [5] 699.21Mbps [6] 699.21Mbps [7] 699.21Mbps Interface Switching Capability Descriptor(1): Switching type: Packet Encoding type: Packet Maximum LSP BW [priority] bps: [0] 699.21Mbps [1] 699.21Mbps [2] 699.21Mbps [3] 699.21Mbps

Maximum LSP BW [priority] bps:

[0] 699.07Mbps [1] 699.07Mbps [2] 699.07Mbps [3] 699.07Mbps [4] 699.07Mbps [5] 699.07Mbps [6] 699.07Mbps [7] 699.07Mbps

[4] 699.21Mbps [5] 699.21Mbps [6] 699.21Mbps [7] 699.21Mbps

Proprietary and Confidential

www.juniper.net

How to trouble shoot commit problem?

Commit synch | display details Show log ksyncd, same as the /var/log/ksyncd Roll back configuration of backup RE and sych up from RE0 Copy configuration from master RE to backup RE: Configure files are saved under /config. The running config is juniper.conf.gz.(execute this command from master RE, be careful of the permission on backup REs directory) rcp T juniper.config.gz re1:/var/tmp will copy the file to backup RE1s /var/tmp directory # commit check [email protected]> show system commit

Proprietary and Confidential

www.juniper.net

Trouble shoot PFE CPU high

start shell vty fpc6 sh nvram sh syslog messages FFPC4(cer-core-01 vty)# show pfe statistics traffic FFPC4(cer-core-01 vty)# show pfe statistics notification FFPC4(cer-core-01 vty)# show icmp statistics Show chassis fpc (to find out fpc cpu utilization)
Proprietary and Confidential www.juniper.net

Trouble shoot PFE CPU high

6PE trouble shooting

PE configuration
lab@Magenta# show protocols rsvp { interface as0.0; } mpls { ipv6-tunneling; label-switched-path to_PE2 { to 4.4.4.4; } interface as0.0; } bgp { group purple { type internal; local-address 2.2.2.2; family inet6 { } interface lo0.0; } fe-0/1/0 { unit 0 { family inet { } } } } family iso { address 49.0001.0005.0005.0005.00; } isis { } } lo0 { unit 0 { family inet { address 2.2.2.2/32; } peer-as 300; neighbor 8002::2; } gr-1/2/0 { // GSR tunnel group to_CE2 { type external; local-address 8002::1; family inet6 { unicast; } family inet6 { address 8002::1/126; unit 100 { tunnel { source 99.1.1.1; destination 99.1.1.2;

interface as0.0 {
level 2 metric 10;

labeled-unicast {
explicit-null; } } peer-as 100; neighbor 4.4.4.4; }

address 99.1.1.1/24;
} } }

Proprietary and Confidential

www.juniper.net

6PE trouble shooting

CE configuration
interfaces { fe-0/1/0 { unit 0 { family inet { address 99.1.1.2/24; } } } gr-1/2/0 { unit 100 { tunnel { source 99.1.1.2; destination 99.1.1.1; } family inet6 { address 8002::2/126; } protocols { bgp { } autonomous-system 300; } routing-options { static { route 172.0.0.0/8 { next-hop 172.19.58.1; no-readvertise;

group to_PE2 {
type external; local-address 8002::2; family inet6 { unicast; } export policy1; peer-as 100;

}
} } lo0 { unit 0 { family inet { address 127.0.0.1/32; } family inet6 { address 9001::5/128; } } }
Copyright 2007 Juniper Networks, Inc.

neighbor 8002::1;
} } }

Proprietary and Confidential

www.juniper.net

MPLS Auto-bandwidth
Auto-bandwidth configuration
mpls { apply-groups [ lspHigh-common lspStnd-common lsp-optimize-timer ]; path-mtu { rsvp mtu-signaling; label-switched-path lspStndT6toT1 {

to 166.34.95.71;
optimize-timer 60; node-link-protection; adaptive; auto-bandwidth { adjust-interval 300; adjust-threshold 10; minimum-bandwidth 100k; maximum-bandwidth 10g; adjust-threshold-overflow-limit 5; } primary use-ge-620; } path use-ge-620 { 192.100.36.37; }

}
statistics { file mpls.stat size 300k files 20 world-readable; interval 300; auto-bandwidth; display-id;

}
traceoptions { file mpls.log size 10m files 21 world-readable; flag error; flag state; flag cspf;

flag connection;
flag graceful-restart; } }
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

MPLS Auto-bandwith trouble shooting

lab@Magenta> file to_PE2

show /var/log/mpls.stat
139233752 Byte

Oct 30 15:41:21 trace_on: Tracing to "/var/log/mpls.stat" started 132491 pkt Oct 30 15:41:21 2008 UTC Total 2 sessions: 1 success, 0 fail, 1 ignored Oct 30 15:43:09 trace_on: Tracing to "/var/log/mpls.stat" started to_PE2 auto-bw 132491 pkt 0 pkt 139233752 Byte 0 Byte 0 pps 0 Bps

Oct 30 15:43:09 2008 UTC Total 3 sessions: 2 success, 0 fail, 1 ignored Oct 30 15:44:19 trace_on: Tracing to "/var/log/mpls.stat" started auto-bw 0 pkt 0 Byte 0 pps 0 Bps Util 0.00%

lab@Magenta> file show /var/log/mpls.log Oct 30 15:48:20 trace_on: Tracing to "/var/log/mpls.log" started Oct 30 16:03:09.172425 RPD_MPLS_PATH_BANDWIDTH_CHANGE: MPLS path (lsp auto-bw) bandwidth changed, path bandwidth 4140760 bps Oct 30 16:03:10.173337 RPD_MPLS_LSP_BANDWIDTH_CHANGE: MPLS LSP auto-bw bandwidth changed, lsp bandwidth 4140760 bps Oct 30 16:08:09.173234 RPD_MPLS_PATH_BANDWIDTH_CHANGE: MPLS path (lsp auto-bw) bandwidth changed, path bandwidth 1000 bps Oct 30 16:08:10.174771 RPD_MPLS_LSP_BANDWIDTH_CHANGE: MPLS LSP auto-bw bandwidth changed, lsp bandw

Proprietary and Confidential

www.juniper.net

MPLS Auto-bandwith trouble shooting

edit protocols mpls statistics] lab@Magenta# run show mpls lsp extensive Ingress LSP: 1 sessions 4.4.4.4 From: 2.2.2.2, State: Up, ActiveRoute: 0, LSPname: auto-bw Description: test2 ActivePath: (primary)

Node/Link protection desired

LoadBalance: Random Autobandwidth MinBW: 1000bps MaxBW: 10Gbps AdjustTimer: 300 secs AdjustThreshold: 10% Max AvgBW util: 0bps, Bandwidth Adjustment in 5 second(s). Overflow limit: 5, Overflow sample count: 0 Encoding type: Packet, Switching type: Packet, GPID: IPv4 *Primary Priorities: 7 0 Bandwidth: 1.824kbps OptimizeTimer: 60 SmartOptimizeTimer: 180 Reoptimization in 18 second(s). Computed ERO (S [L] denotes strict [loose] hops): (CSPF metric: 10) 5.5.5.1 S Received RRO (ProtectionFlag 1=Available 2=InUse 4=B/W 8=Node 10=SoftPreempt): State: Up

5.5.5.1(Label=3)
90 Oct 30 17:27:24.553 CSPF: computation result ignored[5 times] 89 Oct 30 17:23:09.175 Record Route: 5.5.5.1(Label=3) 88 Oct 30 17:23:09.175 Up 87 Oct 30 17:23:09.175 Automatic Autobw adjustment succeeded

Proprietary and Confidential

www.juniper.net

NAT stuff
To enable random port allocation, user has to configure "set services nat pool <pool-name> port automatic randomallocation" or "set services nat pool <pool-name> port range low <low-portnum> high <high-port-num> random-allocation".

Proprietary and Confidential

www.juniper.net

How to look up RE CPU and Memory?

lab@jazz-re0> show chassis routing-engine Routing Engine status: Slot 0: Current state Election priority Temperature CPU temperature DRAM Memory utilization CPU utilization: User 0 percent Master Master (default) 41 degrees C / 105 degrees F 43 degrees C / 109 degrees F 3584 MB 13 percent

Background
Kernel Interrupt Idle Model Serial ID Start time Uptime Load averages:

0 percent
2 percent 0 percent 97 percent RE-A-2000 9009002764 2008-11-18 08:15:10 PST 8 hours, 54 minutes, 29 seconds 1 minute 5 minute 15 minute 0.06 0.10 0.05
Proprietary and Confidential www.juniper.net

Translate Cisco ATM to Juniper ATM

interface ATM1/0/0 description ### Google DEDICADA### bandwidth 155000 no ip address no ip directed-broadcast no ip proxy-arp no ip mroute-cache load-interval 30 atm sonet stm-1 atm uni-version 3.1 no atm ilmi-keepalive no atm enable-ilmi-trap no snmp trap link-status

!
interface ATM1/0/0.1 point-to-point description Link Google_Akwan (50Mbps)*5531004003 bandwidth 50000 ip address 200.162.89.161 255.255.255.252 no ip redirects no ip unreachables no ip directed-broadcast no ip proxy-arp no atm enable-ilmi-trap snmp trap link-status pvc 5531004003 2/901 vbr-nrt 55209 55209 1 no ilmi manage oam-pvc manage oam retry 10 5 1 encapsulation aal5snap ! !----------------------------

Proprietary and Confidential

www.juniper.net

Translate Cisco ATM to Juniper ATM

chassis { fpc 0 { pic 3 { framing sdh; } } }

interfaces {
at-0/3/0 { atm-options { pic-type atm2; vpi 2; } unit 1 { encapsulation atm-snap; point-to-point; no-traps; vci 2.901; shaping { vbr peak 55209000 sustained 55209000 burst 1; } oam-period 10;

oam-liveness {
up-count 10; down-count 5; } family inet { address 200.162.89.162/30;
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Translate Cisco ATM to Juniper ATM

http://www.juniper.net/techpubs/software/junos/junos90/swconf ig-network-interfaces/frameset.html

Proprietary and Confidential

www.juniper.net

T1 / T3 trouble shooting
1. Loopback testing http://www.juniper.net/techpubs/software/erx/erx41x/swconfigphysical-link/html/t1-e1-ji-config8.html Either Local loopback or remote loopback can be configured at any given time. For local loopback, best use an external loopback plug because it can also tests the PICs transmit and receive circuitry.

SONET, T1/DS1 type P-T-P interfaces support remote loopback

Configuring remote loopback only results in a line loop on local router.

Configuration:
sonet-options { loopback local/remote;
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

A good status write up

[Action] Spoke with Bob Walsh and Mark Rippe. [Issue summarized] The issue was they were seeing physical layer T1 issues as well as intermittent ping loss. [Issue details] For T1 errors they were seeing BEE and LOF errors. When looking at the ping loss issue, [Start of cause analysis top layer of root cause] I determined that the reason for network outage was due to PPP going down and renegotiating over and over again. [ real root cause] This was due to the T1 error condition. [ here is why the real root cause is] Setting t1-0/0/3 holdtime up 0 down 100 stabilized the PPP connection. But that does not resolve the underlying issue with the T1 errors. BEE and LOF indicates a problem with upstream provider equipment. BEE is typically triggered when upstream switch has a problem in TX side and then notifies the upstream equipment of the problem. LOF implies that we are not seeing frames on the link for a period of time. Bob had also tested same J2300 router and cable on Verizon T1 circuit and observed no errors. So not likely a J2300 hardware issue. [address possible doubt to prove the root cause] Cox testing with end-to-end loopback and all zeroes testing indicated no errors. However, it is possible that the testing equipment sensitivity may not be great enough detect the failure compared to Juniper router T1 interfaces which tend to be very sensitive to any errors on the line. Going forward we [workaround recommendation] recommend keeping hold-time configured on the T1 interface for this very reason. But ultimately it would be up to provider to correct any line defects. [game plan] Current action plan is to wait for new ATM circuit to be installed to bypass the Amica equipment that this J2300 connects to. That will likely occur within the next several days. Will keep case open in the interim.
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Juniper Smartd Issues

PSN-2008-10-046 apparently covers multiple hdd related PRs. I looked at these PRs. If smartd is off, it may help PR/288011. However, I don't see how it would help PR/278580, PR/389540 and PR/390306.

Proprietary and Confidential

www.juniper.net

VPLS tagging configuration

Got a case with vpls tagging. Customer closed this case immediately for the reason of mis-configuration. Might worth for reference in the future
****Old Way to config**** unit 25 { description "DSH - ubr02 : 28/GCXG/061828//COXC"; encapsulation vlan-ccc;

*****New Way to config***** unit 4000 { description "Lab - Todd SPN Test 1"; encapsulation vlan-ccc; vlan-tags outer 4000 inner-range 1-4094; input-vlan-map { swap; vlan-id 1101; } output-vlan-map swap; }

vlan-id 25;
input-vlan-map { swap; vlan-id 1212; } output-vlan-map swap; }

Proprietary and Confidential

www.juniper.net

Juniper interface trouble shooting

To disable keepalive on a point-to-point interface. This is a tricky one as I have kept forgot it. set no-keepalive

Proprietary and Confidential

www.juniper.net

Platform code name

Atlas - The MX960, 14-slot carrier-class Ethernet platform, part of Harry. ATLAS Alexander - M40e ALEXANDER Autobahn - JUNOS upgrade to FreeBSD 6.1 Bellini - Bellini - Fine-grained (per VLAN) queuing for DPC (Dense Port Cards) on ATLAS Bombay - T320 BOMBAY Callypso - 7-slot chassis Ethernet switch MX480 Matrix takes Atlas cards, part of Harry.(IPG) Calvin - M7i CALVIN Chaser - M5 / M10 CHASER Cosmo - M 20 COSMO Dr Pepper - JUNOS on Saipan Flamingo - M320 FPCs Gibson - T640 GIBSON-LLC GIBSON-SHMC Gimlet - LMNR chipset GIMLET Greyhound - SONET OC768 PIC Haddock - HGE-PIC qpp HADDOCK Harry - Ethernet switch/router platforms HARRY Havana HAVANA
Copyright 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

Platform code name

Heavy Metal - T640 based platform (IPG) Hobbes - M10i Hobson - TX platform HOBSON Hurricane - Hardware Stackable switch - Java Fixed configuration switches: - Espresso (Fixed configuration switch) - Latte (Virtual chassis Switch) - Caffeine : - Biscotti (Software) - Grande (8 slot 1.6Tbps chassis Switch) - Venti (16 slot 3.2Tbps chassis switch)

Proprietary and Confidential

www.juniper.net

Jsim Procedure (M120)

lab@blackjack-re0> show chassis fpc-feb-connectivity lab@blackjack-re0> start shell pfe network feb0 RFEB0(blackjack-re0 vty)# show ichip ifd RFEB0(blackjack-re0 vty)# show ichip 0 r counters RFEB0(blackjack-re0 vty)# show ichip 0 iif statistics RFEB0(blackjack-re0 vty)# jsim reset full 0 (must reset) RFEB0(blackjack-re0 vty)# show ifl brief RFEB0(blackjack-re0 vty)# set jsim iif 73 (must bind intf) RFEB0(blackjack-re0 vty)# set jsim ipsrc 201.1.1.2 RFEB0(blackjack-re0 vty)# set jsim ipdst 200.1.1.2

RFEB0(blackjack-re0 vty)# jsim lookup verbose

Proprietary and Confidential

www.juniper.net

Jsim Procedure (M120)

1) Find out which FPC (cFPC) is connected to which FEB

lab@blackjack-re0> show chassis fpc-feb-connectivity FPC FPC type FPC state 0 1 2 cFPC cFPC Type 3 Online Online Online 1 0 Connected FEB FEB state Online Online OK OK Link status None

3
4 5

Type 2
Type 2 Empty

Online
Online 5

3
4 Online

Online
Online

OK
OK

2) Console to the corresponding FEB (FEB 0 is connected to FPC3 @ slot 2)

lab@blackjack-re0> start shell pfe network feb0 RFEB platform (666Mhz MPC 8541 processor, 512MB memory, 512KB flash) RFEB0(blackjack-re0 vty)# exit

Proprietary and Confidential

www.juniper.net

Jsim Procedure (M120)

3) Find out which iCHIP is being used (from here, we know ICHIP 0 is being used)

RFEB0(blackjack-re0 vty)# show ichip ifd I-chip global information: ICHIP 0: Initialized, Version 2,

STREAM 32 (wan stream 0) has 1 IFDs.

IFD 191: so-2/0/0 ICHIP 1: Not Initialized,

ICHIP 2: Not Initialized,

ICHIP 3: Not Initialized,

Proprietary and Confidential

www.juniper.net

Jsim Procedure (M120)

4) Collect some statistics of iCHIP 0
RFEB0(blackjack-re0 vty)# show ichip 0 r counters
Traffic stats: Counter Name rcp_input_ucast (BYTE) (BYTE) rcp_output_ucast Total 167035601285 164600940855 Rate Peak Rate 39270060 1832927823 39270077 1832926045 ---------------------- ---------------- -------------- -------------31638906 1265556255 31638902 1265556088 6868449722474 6771063304262

RFEB0(blackjack-re0 vty)# show ichip 0 iif statistics

Traffic stats: Counter Name Total Rate Peak Rate 592351311 0 784316693 ---------------------- ---------------- -------------- -------------GFAB_BCNTR 91405146968728 KA_PCNTR 0 0

KA_BCNTR
Discard counters: Counter Name WAN_DROP_CNTR FAB_DROP_CNTR KA_DROP_CNTR HOST_DROP_CNTR

0
Total

0
Rate

0
Peak Rate 7582075 0 0 0 0 0 # 11888478

---------------------- ---------------- -------------- -------------2194246089959 15144376205 0 194 2380431

Proprietary and Confidential

www.juniper.net

Jsim Procedure (M120)

5) Reset JSIM ( everytime you change something, you need to reset JSIM)

RFEB0(blackjack-re0 vty)# jsim reset full 0 6) Find out the interface ifl ( here it is 73) we will bind to JSIM lookup

RFEB0(blackjack-re0 vty)# show ifl brief

Index Name Type Encapsulation Flags ----- -------------------- ------------- -------------- -----71 ge-4/2/0.0 73 so-2/0/0.0 72 ge-4/2/0.32767 64 lo0.0 VLAN Tagged Cisco HDLC Ethernet Cisco HDLC Ethernet 0x000000000000c000 0x0000000000008010 0x000000000000c000

VLAN Tagged

Unspecified

0x0000000000000052

Proprietary and Confidential

www.juniper.net

Jsim Procedure (M120)

7) Bind iif to jsim and setup stream lookup key RFEB0(blackjack-re0 vty)# set jsim iif 73 RFEB0(blackjack-re0 vty)# set jsim ipsrc 201.1.1.2 RFEB0(blackjack-re0 vty)# set jsim ipdst 200.1.1.2 8) Finally, do the lookup (this is the data we are looking for)
RFEB0(blackjack-re0 vty)# jsim lookup verbose Step Kp Address Data Description ---- -- ----------- -------- ----------[ 1] 16 reg 000000 0000a679 nh: TID itable tid=10 offset=-7 itid 00000a 00040000 itable address (seg 0) 04000010 itable descriptor addr=0x000100 size=65536 idx_bits=16 bit_offset=0 lookup index=73 [ 2] 9 sram 00014b 10292f28 nh: extended buff-modify intermediate-nh addr=0x040a4a sram 040a4a 7840b2ab Buffer Translate: write kb(8), off 42, bits 12, data 0xffffc40 [ 3] 9 sram 040a4b 44060b61 nh: multiple SER(no SE) hops=1 addr=0x110182

Proprietary and Confidential

www.juniper.net

Tethereal to decode ixia packets.

-bash-2.05b$ tethereal -r cap.enc -V Frame 1 (70 bytes on wire, 70 bytes captured) Arrival Time: Feb 4, 2017 16:03:16.453824000 Time delta from previous packet: 0.000000000 seconds Time relative to first packet: 0.000000000 seconds Frame Number: 1 Packet Length: 70 bytes Capture Length: 70 bytes

Ethernet II, Src: 00:1f:12:23:e6:02, Dst: 00:00:c8:01:01:64

Destination: 00:00:c8:01:01:64 (AltosCom_01:01:64) Source: 00:1f:12:23:e6:02 (00:1f:12:23:e6:02) Type: IP (0x0800) Internet Protocol, Src Addr: 100.4.4.3 (100.4.4.3), Dst Addr: 200.1.1.100 (200.1.1.100) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable Transport (ECT): 0 .... ...0 = ECN-CE: 0

Proprietary and Confidential

www.juniper.net

IPSec SP-MTU and Tunnel-MTU(M/J series)

On m-series: with sp-mtu of 1440, the max IP payload size that is 8 byte aligned is 1416, adding 20 bytes of IP header len results in 1436.
On j-series: with mtu of 1446 (tunnel-mtu-ipsec overheads), the max IP payload size that is 8 byte aligned is 1424, adding 20 bytes of IP header len becomes 1444.

Proprietary and Confidential

www.juniper.net

Proprietary and Confidential

www.juniper.net

149 #

Juniper Training 1 BGP
No ratings yet
Juniper Training 1 BGP
20 pages
Segment Routing
No ratings yet
Segment Routing
1,475 pages
JNCIE SP v1.1 Technology Focused Workbook 2018
No ratings yet
JNCIE SP v1.1 Technology Focused Workbook 2018
28 pages
Jncia Lab Guide
100% (1)
Jncia Lab Guide
10 pages
Cisco-to-Juniper Show Commands
No ratings yet
Cisco-to-Juniper Show Commands
4 pages
Juniper Commands v2
No ratings yet
Juniper Commands v2
29 pages
Spare Parts Catalog: Machine Type: Vip Iii 46 / Vip Iii 47 Gaming Platform: Various Game Software: Various
No ratings yet
Spare Parts Catalog: Machine Type: Vip Iii 46 / Vip Iii 47 Gaming Platform: Various Game Software: Various
14 pages
Juniper Networks Virtual MX Series Router (VMX) With VMware Fusion
No ratings yet
Juniper Networks Virtual MX Series Router (VMX) With VMware Fusion
19 pages
Junos Pyez Developer Guide PDF
No ratings yet
Junos Pyez Developer Guide PDF
314 pages
LTRDCN 3077
No ratings yet
LTRDCN 3077
58 pages
Applying Junos Configuration Automation
No ratings yet
Applying Junos Configuration Automation
102 pages
Implementing Cisco NX-OS Switches and Fabrics in The Data Center (DCNX) v1.0
No ratings yet
Implementing Cisco NX-OS Switches and Fabrics in The Data Center (DCNX) v1.0
3 pages
SRX Juniper PDF
0% (1)
SRX Juniper PDF
19 pages
The Word "Computer" Usually Refers To The Central Processing Unit Plus? (A) External Memory (B) Internal Memory (C) Input Devices (D) Output Devices
No ratings yet
The Word "Computer" Usually Refers To The Central Processing Unit Plus? (A) External Memory (B) Internal Memory (C) Input Devices (D) Output Devices
4 pages
Routing JNASC Juniper - Curso
No ratings yet
Routing JNASC Juniper - Curso
160 pages
Juniper
No ratings yet
Juniper
68 pages
ConfigGuide Arista
No ratings yet
ConfigGuide Arista
742 pages
BRKRST-2930 - Implementing QoS With Nexus and NX-OS
No ratings yet
BRKRST-2930 - Implementing QoS With Nexus and NX-OS
88 pages
Day One - Junos For IOS Engineers
100% (1)
Day One - Junos For IOS Engineers
78 pages
Apstra User Guide
No ratings yet
Apstra User Guide
1,254 pages
Cisco Catalyst QoS Simplified Presentation
No ratings yet
Cisco Catalyst QoS Simplified Presentation
58 pages
CLI Commands Cisco VS Juniper PDF
No ratings yet
CLI Commands Cisco VS Juniper PDF
2 pages
SRX SNMP Monitoring Guide - v1.2
No ratings yet
SRX SNMP Monitoring Guide - v1.2
15 pages
Juniper IJSEC Chapter1 Course Outline
No ratings yet
Juniper IJSEC Chapter1 Course Outline
6 pages
Devops
100% (1)
Devops
13 pages
Cisco Sd-Wan Vmanage Cluster Creation and Troubleshooting
No ratings yet
Cisco Sd-Wan Vmanage Cluster Creation and Troubleshooting
27 pages
Comparison of Cisco, Huawei and Juniper Command Line
No ratings yet
Comparison of Cisco, Huawei and Juniper Command Line
6 pages
Cisco Asr 9k
No ratings yet
Cisco Asr 9k
9 pages
ASR903 Config PDF
No ratings yet
ASR903 Config PDF
186 pages
DB2 For ZOS Course 2
No ratings yet
DB2 For ZOS Course 2
654 pages
Not For Reproduction: Troubleshooting JUNOS Platforms
No ratings yet
Not For Reproduction: Troubleshooting JUNOS Platforms
464 pages
Juniper Basic Config
No ratings yet
Juniper Basic Config
12 pages
Juniper Commands v4 CLI
No ratings yet
Juniper Commands v4 CLI
2 pages
Cisco and Juniper Devices Compared Command
No ratings yet
Cisco and Juniper Devices Compared Command
4 pages
alcatel Lucent Configuring Packet Ip Filters
No ratings yet
alcatel Lucent Configuring Packet Ip Filters
5 pages
Day One Apply Junos Ops Automation - Juniper Networks PDF
No ratings yet
Day One Apply Junos Ops Automation - Juniper Networks PDF
107 pages
A5 byCCIEMAN2016 - Nonvrf
100% (1)
A5 byCCIEMAN2016 - Nonvrf
60 pages
Guide For Junos Upgrade
No ratings yet
Guide For Junos Upgrade
19 pages
ACI-3456 Openstack
No ratings yet
ACI-3456 Openstack
95 pages
Resumen Cisco Vs Juniper Commands
No ratings yet
Resumen Cisco Vs Juniper Commands
5 pages
LABDCT 2001 (Guide) Nexus.7000
No ratings yet
LABDCT 2001 (Guide) Nexus.7000
44 pages
Intro Juniper 2
No ratings yet
Intro Juniper 2
53 pages
Juniper Networks - (M - MX - T-Series) Troubleshooting Checklist - Routing Engine High CPU
No ratings yet
Juniper Networks - (M - MX - T-Series) Troubleshooting Checklist - Routing Engine High CPU
5 pages
Intro Juniper 1
No ratings yet
Intro Juniper 1
45 pages
CCIE Enterprise Infrastructure (v1.0) Exam Topics - Practical Exam
No ratings yet
CCIE Enterprise Infrastructure (v1.0) Exam Topics - Practical Exam
8 pages
Cisco Juniper Commands
No ratings yet
Cisco Juniper Commands
2 pages
Cisco Catalyst 9300 Series Datasheet
No ratings yet
Cisco Catalyst 9300 Series Datasheet
74 pages
C++ Header Files
No ratings yet
C++ Header Files
5 pages
JSEC EX InstructorNotes 30may2010
No ratings yet
JSEC EX InstructorNotes 30may2010
9 pages
Labspg 2442 LG
No ratings yet
Labspg 2442 LG
37 pages
Juniper Comandos
No ratings yet
Juniper Comandos
4 pages
BRKCRT 3100
No ratings yet
BRKCRT 3100
71 pages
Cisco Press - Configuring BGP
No ratings yet
Cisco Press - Configuring BGP
52 pages
IJOS Lab Guide - Lab3.Ready
No ratings yet
IJOS Lab Guide - Lab3.Ready
23 pages
Cisco Juniper Commands
No ratings yet
Cisco Juniper Commands
1 page
Auto Export Understanding
No ratings yet
Auto Export Understanding
12 pages
10.a-R LD
No ratings yet
10.a-R LD
14 pages
Junos Bootcamp Mod 1
No ratings yet
Junos Bootcamp Mod 1
13 pages
Aruba Switching Fundamentals For Mobility Rev 1641 - Lab Guide - Book 1 of 2 - PD60260
No ratings yet
Aruba Switching Fundamentals For Mobility Rev 1641 - Lab Guide - Book 1 of 2 - PD60260
365 pages
BRKSPG 3334
No ratings yet
BRKSPG 3334
170 pages
CCNP Route (300-101)
No ratings yet
CCNP Route (300-101)
219 pages
Zhone Dslam
No ratings yet
Zhone Dslam
410 pages
DCNI-2 Implementing Cisco Data Centre Network Infrastructure 2 SGvol1 Ver3.0
No ratings yet
DCNI-2 Implementing Cisco Data Centre Network Infrastructure 2 SGvol1 Ver3.0
418 pages
Abdul Mateen Middleware Devops Resume
No ratings yet
Abdul Mateen Middleware Devops Resume
6 pages
User Manual
No ratings yet
User Manual
37 pages
Android Spinner Example - Javatpoint
No ratings yet
Android Spinner Example - Javatpoint
4 pages
Controller Form: 4. Building The User Interface Layer
No ratings yet
Controller Form: 4. Building The User Interface Layer
5 pages
RFID Based Student Recording and Monitoring Over IOT
No ratings yet
RFID Based Student Recording and Monitoring Over IOT
17 pages
Electronic Jacquard Slave Board
No ratings yet
Electronic Jacquard Slave Board
2 pages
IT Data-Center Facilities Manager/Asset Management Hardware Tech
No ratings yet
IT Data-Center Facilities Manager/Asset Management Hardware Tech
2 pages
CST202 Computer Organization and Architecture, December 2024
No ratings yet
CST202 Computer Organization and Architecture, December 2024
2 pages
Dabur's Spoonful of The Cloud - As Sweet As It's Honey! - CIO Research Center
No ratings yet
Dabur's Spoonful of The Cloud - As Sweet As It's Honey! - CIO Research Center
6 pages
Chapter 4
No ratings yet
Chapter 4
3 pages
Adpcm-Hco PB 1 5 2
No ratings yet
Adpcm-Hco PB 1 5 2
2 pages
WM Recorder Manual
No ratings yet
WM Recorder Manual
15 pages
Reverse Engineering For Beginners by Dennis Yurichev - August 2016
No ratings yet
Reverse Engineering For Beginners by Dennis Yurichev - August 2016
987 pages
MC 10153695 9999
No ratings yet
MC 10153695 9999
6 pages
Suse Linux Enterprise Server SAP & HA & HANA Platform: Eduardo Diaz
No ratings yet
Suse Linux Enterprise Server SAP & HA & HANA Platform: Eduardo Diaz
30 pages
Normalization Activity
No ratings yet
Normalization Activity
4 pages
Big Mart Sales Analysis DOCUMENT
No ratings yet
Big Mart Sales Analysis DOCUMENT
58 pages
WinPADS200-EK BriefManual en PDF
No ratings yet
WinPADS200-EK BriefManual en PDF
21 pages
A Beginner's Guide To "C" Programming
No ratings yet
A Beginner's Guide To "C" Programming
56 pages
Cyb130 v2 Syllabus
No ratings yet
Cyb130 v2 Syllabus
2 pages
Mad Output
No ratings yet
Mad Output
5 pages
SST25VF016B: 16 Mbit SPI Serial Flash
No ratings yet
SST25VF016B: 16 Mbit SPI Serial Flash
30 pages
Comandos para Nuevo Networker
No ratings yet
Comandos para Nuevo Networker
4 pages
Extração Da Bios Do Arquivo .Exe
No ratings yet
Extração Da Bios Do Arquivo .Exe
1 page
Kushal Vijay Resume
No ratings yet
Kushal Vijay Resume
1 page
JUNOS OS For Dummies
From Everand
JUNOS OS For Dummies
Walter J. Goralski
No ratings yet
Next-Generation switching OS configuration and management: Troubleshooting NX-OS in Enterprise Environments
From Everand
Next-Generation switching OS configuration and management: Troubleshooting NX-OS in Enterprise Environments
Mamta Devi
No ratings yet