Troubleshooting GETVPN Deployments

Session Objectives and Prerequisites

 Session Objectives
Provide Overview of GETVPN
Deployment Considerations
Illustrate Troubleshooting Methodology and Techniques

 Prerequisites
Knowledge of IP Routing
Knowledge of IPSec VPN Technologies
Basic Understanding of GETVPN is a Plus

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Agenda

 GETVPN Solution Overview

What Is GETVPN and Where Does It Fit?

 Introduction to GETVPN
Technology Overview

 GETVPN Deployment
Configuration and Scalability

 Troubleshooting
Common Show Commands, Debugs and Syslogs
Common Troubleshooting Scenarios

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
GETVPN Solution Overview

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
Cisco Group Encrypted Transport - GETVPN
What Is GETVPN?
Cisco GETVPN delivers a revolutionary solution for tunnel-less,
any-to-any and confidential branch communication

 Large-scale any-to-any
encrypted communication
Any
Any--to
to --Any
Any  Native routing without
Connectivity
Connectivity
tunnel overlay
 Optimal for QoS and Multicast
support—improves application
Cisco GET
performance
VPN
 Transport agnostic—private
Scalable Real Time LAN/WAN, FR/ATM, IP, MPLS

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
Tunnel-Less VPN—
A New Security Model
Any-to-Any Encryption: Before and After GET VPN
Before: IPSec P2P Tunnels After: Tunnel-Less VPN

WAN

Multicast

 Scalability—an issue (N^2 problem)  Scalable architecture for any-to-any

 Overlay routing connectivity and encryption
 Any-to-any instant connectivity can’t  No overlays—native routing
be done to scale  Any-to-any instant connectivity
 Limited QoS  Enhanced QoS
 Inefficient Multicast replication  Efficient Multicast replication
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
VPN Technology Positioning
Data Center Core

Internet
Edge

IPSec

GM GM

KS KS

WAN
Edge
Remote
Access
Internet/
Shared GET MPLS/Private
Encrypted Network
Network

EzVPN
Spoke
DMVPN DMVPN
Spoke Spoke GET GM GET GM GET GM

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
VPN Technology Positioning (Cont.)

EzVPN DMVPN GET VPN

Infrastructure  Public Internet  Public Internet  Private IP

Network Transport Transport Transport

 Hub-Spoke and
 Hub-Spoke;  Any-to-Any;
Network Style Spoke-to-Spoke;
(Client to Site) (Site-to-Site)
(Site-to-Site)

 Reverse-route  Dynamic routing  Dynamic routing

Routing
Injection on tunnels on IP WAN

 Route  Route
Failover  Stateful Hub
Distribution Distribution
Redundancy Crypto Failover
Model Model + Stateful

 Peer-to-Peer  Peer-to-Peer
Encryption Style  Group Protection
Protection Protection

 Multicast
 Multicast  Multicast
IP Multicast replication in IP
replication at hub replication at hub
WAN network

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Introduction to
GETVPN Technology

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Group Encrypted Transport

 Uses three main components

Secure group keys
Header preservation
Key service

 Based on open standards with patented Cisco

technology
 Leverages existing IKE and IPSec and multicast
technologies
 Takes advantage of the existing routing
infrastructure

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
 Group Security Functions
Routing Member
Key Server  Forwarding
Key Server  Replication
 Validate Group Members  Routing
 Manage Security Policy
 Create Group Keys
 Distribute Policy/Keys
Group
Member
Routing
Members

Group
Member
Group
Group Member Member
 Encryption Devices
 Route Between Secure/ Unsecure
Regions Group
 Multicast Participation Member

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
Group Security Elements
KS Cooperative
Key Servers
Group Policy Protocol

Key Encryption Key

(KEK)

Traffic Encryption
Key (TEK) Group
Member
Routing
Members

Group
Member
Group
Member
RFC3547:
Group Domain of
Interpretation Group
(GDOI) Member

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
Basic GET VPN Architecture

Step 1: Group Members (GM) “Register”

via GDOI with the Key Server (KS)
 KS authenticates and authorizes the GM
 KS returns a set of IPSec SAs
for the GM to use
GM3
GM4
GM2

GM5
GM1

GM6

GM9 KS
GM8 GM7

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
Basic GET VPN Architecture

Step 2: Data Plane Encryption

 GM exchange encrypted traffic using the group keys
 The traffic uses IPSec Tunnel Mode with “Header
Preservation”

GM3
GM4
GM2

GM5
GM1

GM6

GM9 KS
GM8 GM7

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
Basic GET VPN Architecture

Step 3: Periodic Rekey of Keys

 KS pushes out replacement IPSec keys before current IPSec
keys expire; this is called a “rekey”

GM3
GM4
GM2

GM5
GM1

GM6

GM9 KS
GM8 GM7

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
 Header Preservation
IPSec Tunnel Mode vs. GETVPN
IP Header IP Payload
IP Packet

IPSec New IP Header ESP IP Header IP Payload

Tunnel Mode
 IPSec header inserted by VPN Gateway
 New IP Address requires overlay routing

IP Header IP Payload
IP Packet

Group Preserved Header ESP IP Header IP Payload

Encrypted
Transport  IP header preserved by VPN Gateway
 Preserved IP Address uses original routing plane

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
 GETVPN Data Path
Host2 GM 2 GM1 Host1

Encrypted/Authenticated Using Group SA

Original Src Original IP

and Dst ESP Header Data
Addresses

Encrypted

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
Rekey Methodology: Multicast Rekey

 Rekey Message sent from key server to all group members

 IP multicast message provides very efficient distribution
 Rekeys resulting from configured KEK and TEK intervals
or KS policy change
Single rekey
packet sent
Key
Server
to multicast
enabled core
GM2
GM1
S =10. 1. 1. 1
D= 192. 1. 1. 1
PE1 2

S= 10. 1. 1. 1 S =10. 1.1. 1

D= 192. 1.1. 1 D = 192. 1. 1. 1

P1 P2

Core
P3
P4 replicates the
packet to all
S= 10. 1. 1. 1 the GMs
D= 192. 1. 1. 1

PE3

GM4
GM3

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
Rekey Methodology: Unicast Rekey
 Key Server maintains state of active group members
 Group Member sends ACK in response to the rekey
messages
 Remove Group Member if the GM does not acknowledge
three rekeys
Key
Server

GM2
GM1
S =10. 1. 1. 1
D= 192. 1. 1. 1
PE1
S= 10. 1. 1. 1 S =10. 1.1. 1
D= 192. 1.1. 1 D = 192. 1. 1. 1

P1 P2

P3
P4

S= 10. 1. 1. 1
D= 192. 1. 1. 1

PE3

GM4
GM3

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
Requirement for Time-Based Anti-
Replay

 Sequence number based anti-replay only works with

single sender
 Need method to work for all senders using same IPSec
SA
Key Server downloads Relative pseudotime and window size to
all the GMs
GMs calculate pseudo-timestamp based on downloaded
pseudotime and sends out packet
Receiving GM verifies packet within window size
KS periodically refreshes GMs with pseudotime/window size—this
means clock does not need to be synchronized between GMs

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
 Time-Based Anti-Replay

 If Sender’s pseudotime falls in the below Receiver window,

packet accepted

Reject Accept Reject

Initial PTr - W PTr PTr + W

pseudotime

Anti-replay window

 Packet 1 and Packet 2 have pseudotime T0, providing loose

anti-replay protection (unlike counter-based)

T0 T1 T20
0
Packet1
Packet2

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
 Cooperative Key Servers—HA
 Single KS is a single point of failure
 Two or more KSs known as COOP KSs manage a common set of
keys and security policies for GETVPN group members
 Group members can register to any one of the available KSs

Cooperative KS1 Cooperative KS2

Subnet 1
Subnet 2

GM 1
GM 2

IP Network
Subnet 3
Subnet 4

GM 4 GM 3

Cooperative KS3
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
 Cooperative Key Servers—
Introduction (Cont.)
 One KS is elected as the Primary KS
 Cooperative KSs periodically exchange and synchronize group’s
database, policy and keys
 Primary KS is responsible to generate and distribute group keys

Cooperative KS1 Cooperative KS2 (Secondary)

(Primary)

Subnet 1
Subnet 2

GM 1
GM 2

IP Network
Subnet 4 Subnet 3
GM 4
GM 3
Announcement Messages
Rekey Messages
Cooperative KS3 (Secondary)
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
GETVPN Deployment Configuration
and Scalability

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
COOP Server Exportable RSA Keys

RSA Keys (to Be Generated only on KSs) Are Required for

Rekey Authentication

 Exporting RSA Key from Key Server to Group Member:

Public key generated in the RSA key pair, is sent to the GM at
the registration
The re-keys are signed by the private key of the KS and GM
verifies the signature in the re-key with the public key of the KS

 Exporting RSA Key between Key Servers:

One of the key server in the redundancy group should generate
the exportable RSA keys and copy those keys to other
key servers

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
 KS Configuration
crypto keyring gdoi1
Pre-shared Key pre-shared-key address 0.0.0.0 0.0.0.0 key nsite123
!
crypto isakmp policy 10
ISAKMP Policy
encr 3des
authentication pre-share
IPSec Transform !
crypto ipsec transform-set 3DES-SHA esp-3des
esp-sha-hmac
IPSec Profile
!
crypto ipsec profile gdoi1
Access-List used for set security-association lifetime seconds 7200
defining set transform-set 3DES-SHA
rekey (useful in !
multicast rekeys only) access-list 150 permit ip any host 225.1.1.1
!
Access-list denying access-list 160 deny eigrp any any
encryption for access-list 160 deny pim any any
ISAKMP/GDOI/EIGRP access-list 160 deny udp any any eq isakmp
packets and permitting access-list 160 deny udp any any eq 848
encryption for all IP traffic access-list 160 permit ip any any
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
 KS Configuration (Cont.)
GDOI Group ID crypto gdoi group dgvpn1
Rekey Address mapping identity number 101
to ACL 150 (only for server local
multicast rekeys) ! rekey address ipv4 150 !
Lifetime for Key rekey lifetime seconds 14400
Encryption Key rekey retransmit 10 number 2
Rekey Retransmission rekey authentication mypubkey rsa dgvpn1
rekey transport unicast
RSA Key to authenticate sa ipsec 1
rekeys profile gdoi1
Unicast Rekey match address ipv4 160

Encryption ACL address ipv4 130.23.1.1

redundancy
Source address for rekeys local priority 10
Coop Server Config peer address ipv4 130.1.2.1
!
Coop Server priority
Coop Server address

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
 GM Configuration
crypto keyring gdoi
Pre-Shared key pre-shared-key address 0.0.0.0 0.0.0.0 key nsite123
!
crypto isakmp policy 10
ISAKMP Policy encr 3des
authentication pre-share
group 2
!
crypto gdoi group dgvpn
GDOI Group identity number 101
server address ipv4 130.23.1.1
KS Address
!
!
GDOI configuration crypto map dgvpn 10 gdoi
mapped to crypto map set group dgvpn
!
Crypto map on interface FastEthernet0/0
the interface crypto map dgvpn

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
GETVPN Platform Support

Platform Group Member Key Server

Software Yes Not supported
870 Yes Not supported
1821 Yes Not supported
1841/1900 Yes Yes
2800 (AIM/SSL)/2900 Yes Yes
3800 (AIM-II/AIM-III)/3900 Yes Yes
7200 NPEG1, VAM2+ Yes Yes
7301 NPEG1, VAM2+ Yes Yes
7200 NPEG2, VAM2+ Yes Yes
7200 NPEG2, VSA Yes Yes
Cisco ASR 1000 Yes Not supported

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
Scalability and Performance

 GETVPN Provides complete segregation of control

and data plane
 Key Server is responsible to maintain the control
plane (key management) and GM is responsible to
handle the data plane (actual user traffic)
 KS and GM can not be configured on same IOS
device
 KS should be properly sized for number of
branches (scale) in the network
 GM should be properly sized for traffic throughput
at each branch

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
GETVPN Policy Considerations

What should not be protected with Group Security?

 Control Plane
Internet Key Exchange/Group Domain of Interpretation
Routing Exchanges with Service Provider routers (OSPF, BGP)
Diagnostic or troubleshooting traffic (ICMP)

What needs to be protected with Group Security?

 Data Plane
Enterprise Transactions
Enterprise Sensitive Multicast Streams

What may already be protected?

 Management Plane
SSH, TACACS, HTTPS

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
Deployment Best Practices

IKE/IPSec
 Use specific pre-shared keys for all the GMs and KSs instead of using default key
KS
 Always use COOP KSs
 Set the huge buffer to 65535 and add 10 buffers to permanent buffer list
 Configure periodic DPDs between the COOP KSs
 Enable GM authorization
Policy
 Aggregate the permit access-list entries to reduce the entries
 Enable Time-Based Anti-Replay
 Avoid re-encrypting traffic which is already encrypted (SSH, HTTPS)
Registration
 Distribute GM registration to multiple KSs by arranging the KS order in configuration
Rekey Timers
 Set TEK lifetime to 7200 Seconds
 Set KEK lifetime to 86400 Seconds

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
Troubleshooting

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
Troubleshooting methodology overview
 Assess
What’s going on
Ask the right questions to better define and clarify the problem

 Acquire
What information do we need but we don’t have?
How to get the information

 Analyze
Understand the flow
What’s supposed to happen vs. What actually happened

 Act
Test assumptions
Deploy changes

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
“A problem well stated is a problem half solved.”

Charles F. Kettering

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
Troubleshooting GETVPN
 Ultimately all problems manifest at the data plane -
“my user application is not working over GETVPN!”
 But where really is the problem?
 Control Plane
GM registration issues
Policy download issues
COOP issues
Rekey failures

 Data plane
Policy downloaded with SAs installed but traffic is not flowing
Which device is the culprit, encrypting or decrypting router?
In which direction is the problem happening, ingress or egress?
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
Control Plane Troubleshooting Tools

 GETVPN provides enhanced set of show

commands for functionality verification
 IOS also provided wide variety of syslog messages
to verify proper GETVPN operations
 Syslog messages provide an early insight into a
potential problem
 Use of syslog server strongly recommended when
deploying firewall solutions
 IPSec and GETVPN related debugs can then be
enabled for further troubleshooting

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
Best practices when using the debug
commands
 Turn off console logging
 Use NTP to sync up times on all devices
 Enable msec timestamping debug and log messages
service timestamps debug datetime msec
service timestamps log datetime msec

 Send the debugs to a syslog server

 If no syslog server isavailable, use the logging buffer
with an increased buffer size
logging buffered 1000000 debugging

 terminal exec prompt timestamp when using the show

commands to correlate show commands with the
debug output

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
Data Plane Troubleshooting Tools

 Interface counters
 Encryption/decryption counters
 Netflow
 IP Accounting
 ACL
 DSCP packet coloring
 Embedded Packet Capture (EPC)

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
 GETVPN Control Plane Verification
Common Show Commands

Commands on Key Server: Commands on Group Member:

show crypto gdoi show crypto gdoi

show crypto gdoi ks show crypto gdoi gm acl

show crypto gdoi ks acl show crypto gdoi gm rekey

show crypto gdoi ks coop show crypto gdoi gm replay

show crypto gdoi ks members show crypto gdoi ipsec sa

show crypto gdoi ks policy show crypto gdoi gm

show crypto gdoi ks rekey Show crypto isa sa

show crypto gdoi ks replay Show crypto ipsec sa

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
 Show crypto gdoi (on KS)
Group Name : GET
Group Identity : 101
Group Members : 3
Registered GMs IPSec SA Direction : Both
Active Group Server : Local
coop configuration Redundancy : Configured
Local Address : 130.23.1.1
Local Priority : 10
Local KS Status : Alive
Key server role Local KS Role : Primary
Group Rekey Lifetime : 1800 secs
Group Rekey
KEK lifetime remaining Remaining Lifetime : 88 secs
Rekey Retransmit Period : 10 secs
Rekey Retransmit Attempts: 3
Group Retransmit
Remaining Lifetime : 0 secs

IPSec SA Number : 1
IPSec SA Rekey Lifetime: 900 secs
Profile Name : gdoi1
Replay method : Count Based
Replay Window Size : 64
SA Rekey
TEK lifetime remaining Remaining Lifetime : 446 secs
ACL Configured : access-list 160

Group Server list : Local

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
 Show crypto gdoi KS policy (on KS)
For group GET(handle: 2147483650) server 130.23.1.1 :

# of teks : 1 Seq num : 0

KEK POLICY (transport type : Unicast) Rekey Transport
spi : 0x82B7EEDAC165482E9CAB8DAF94F08EB4
management alg : disabled encrypt alg : 3DES
crypto iv length : 8 key size : 24
Remaining life(sec): 1459 orig lifetime(sec): 1800 KEK Lifetimes
sig hash algorithm : enabled sig key length : 162
sig size : 128
sig key name : dgvpn1

TEK POLICY (encaps : ENCAPS_TUNNEL)

spi : 0x7A1B5D52 access-list : 160 Traffic ACL
# of transforms : 0 transform : ESP_3DES
hmac alg : HMAC_AUTH_SHA
alg key size : 24 sig key size : 20
orig life(sec) : 900 remaining life(sec) : 200 TEK Lifetime
override life (sec): 0 antireplay window size: 64

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
 Show crypto gdoi KS member (on KS)

Number of rekeys sent for group GET: 5

GM’s IP address Group Member ID : 131.1.1.1
Group ID : 101
Group Name : dgvpn1
KS GM is registered with Key Server ID : 130.2.1.1
Rekeys sent : 4
Rekey Acks Rcvd : 4

Sent seq num : 1 0 0 0

Rcvd seq num : 1 0 0 0

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
 Show crypto gdoi (on GM)
GROUP INFORMATION access-list deny ip any 224.0.0.0 0.255.255.255
Active access-list deny ip 224.0.0.0 0.255.255.255 any
Group Name : GET KS access-list deny ip any host 230.1.1.1
Group Identity : 101
Rekeys received : 270 KEK POLICY:
IPSec SA Direction : Both Rekey Transport Type : Unicast
Active Group Server : 134.50.0.1 Lifetime (secs) : 12295
Group Server list : 134.50.0.1 Encrypt Algorithm : 3DES
Key Size : 192
GM Reregisters in : 5187 secs Sig Hash Algorithm : HMAC_AUTH_SHA
Rekey Received(hh:mm:ss) : 00:02:30 Sig Key Length (bits) : 1024

Rekeys received TEK POLICY:

Cumulative : 270 FastEthernet0/0:
When was
After registration : 270
last rekey
Rekey Acks sent : 270 IPSec SA:
received
sa direction:outbound
ACL Downloaded From KS 134.50.0.1: spi: 0x7C45C74A(2084947786)
access-list deny eigrp any any transform: esp-aes esp-sha-hmac
access-list deny tcp any any port = 179 sa timing:remaining key lifetime
access-list deny udp any port = 848 (sec): (5246)
any port = 848 Anti-Replay(Time Based) : 2 sec interval
access-list permit ip any any

Time Based
Anti Replay
Value

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44
 Show crypto gdoi GM acl (on GM)
Group Name: GET
ACL downloaded ACL Downloaded From KS 130.2.1.1:
from KS access-list deny eigrp any any
access-list deny udp any any port = 500
access-list deny udp any any port = 848
Locally access-list permit ip any any
configured ACL ACL Configured Locally:
if present

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
Show crypto IPSec SA
PHEONIX-GM# sh crypto ipsec sa
interface: FastEthernet0/0
Crypto map tag: dgvpn, local addr 131.1.1.1
protected vrf: (none)
local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
current_peer port 848
<SNIP>
inbound esp sas:
spi: 0x1DA9D3E2(497669090)
transform: esp-3des esp-sha-hmac ,
<SNIP>
outbound esp sas:
spi: 0x1DA9D3E2(497669090)
transform: esp-3des esp-sha-hmac ,

RALEIGH-GM# sh crypto ipsec sa

interface: GigabitEthernet0/1
Crypto map tag: dgvpn, local addr 131.3.1.1
protected vrf: (none) Same inbound and
local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0) outbound SPI on all
remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0) Group Members
current_peer port 848
<SNIP>
inbound esp sas:
spi: 0x1DA9D3E2(497669090)
transform: esp-3des esp-sha-hmac ,
<SNIP>
outbound esp sas:
spi: 0x1DA9D3E2(497669090)
transform: esp-3des esp-sha-hmac ,
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 46
 GETVPN Verification
Common KS Syslog Messages
Syslog Messages Explanation
COOP_CONFIG_MISMATCH The configuration between the primary key server and secondary key
server are mismatched.
COOP_KS_ELECTION The local key server has entered the election process in a group.
COOP_KS_REACH The reachability between the configured cooperative key servers is
restored.
COOP_KS_TRANS_TO_PRI The local key server transitioned to a primary role from being a
secondary server in a group.
COOP_KS_UNAUTH An authorized remote server tried to contact the local key server in a
group. Could be considered a hostile event.
COOP_KS_UNREACH The reachability between the configured cooperative key servers is lost.
Could be considered a hostile event.
KS_GM_REVOKED During rekey protocol, an unauthorized member tried to join a group.
Could be considered a hostile event.
KS_SEND_MCAST_REKEY Sending multicast rekey.
KS_SEND_UNICAST_REKEY Sending unicast rekey.
KS_UNAUTHORIZED During GDOI registration protocol, an unauthorized member tried to join
a group. Could be considered a hostile event.
UNAUTHORIZED_IPADDR The registration request was dropped because the requesting device
was not authorized to join the group.

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 47
 GETVPN Verification
Common GM Syslog Messages

Syslog Messages Explanation

GM_CLEAR_REGISTER The clear crypto gdoi command has been executed by
the local group member.
GM_CM_ATTACH A crypto map has been attached for the local group
member.
GM_CM_DETACH A crypto map has been detached for the local group
member.
GM_RE_REGISTER IPSec SA created for one group may have been expired
or cleared. Need to reregister to the key server.
GM_RECV_REKEY Rekey received.
GM_REGS_COMPL Registration complete.
GM_REKEY_TRANS_2_MULTI Group member has transitioned from using a unicast
rekey mechanism to using a multicast mechanism.
GM_REKEY_TRANS_2_UNI Group member has transitioned from using a multicast
rekey mechanism to using a unicast mechanism.
PSEUDO_TIME_LARGE A group member has received a pseudotime with a
value that is largely different from its own pseudotime.
REPLAY_FAILED A group member or key server has failed an anti-replay
check.
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 48
Syslog Messages Example (GM)

Registration:
 CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2
for group G1 using address 10.1.13.2
 GDOI-5-GM_REKEY_TRANS_2_UNI: Group G1 transitioned
to Unicast Rekey
 GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2
complete for group G1 using address 10.1.13.2

Rekey:
 GDOI-5-GM_RECV_REKEY: Received Rekey for group G1
from 10.1.11.2 to 10.1.13.2 with seq # 3

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 49
Syslog Messages Example (KS)

Rekey:
 GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast
Rekey for group G1 from address 101.1.1.1 with seq # 1

COOP:
 GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.0.9.1
Unreachable in group G1
 GDOI-5-COOP_KS_ELECTION: KS entering election mode in
group G1 (Previous Primary = NONE)
 GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.0.8.1 in group
G1 transitioned to Primary (Previous Primary = NONE)

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 50
 Troubleshooting Methodology
crypto gdoi group G1 crypto gdoi group G1
identity number 3333 identity number 3333
server local server local
rekey lifetime seconds 86400 rekey lifetime seconds 86400
rekey authen mypubkey rsa get rekey authen mypubkey rsa get
rekey transport unicast rekey transport unicast
KS1 KS2
sa ipsec 1 sa ipsec 1
profile gdoi-p profile gdoi-p
match address ipv4 ENCPOL match address ipv4 ENCPOL
replay counter window-size 64 replay time window-size 5
address ipv4 10.1.11.2 Ser 1/0: 10.1.12.2 address ipv4 10.1.12.2
Ser 1/0: 10.1.11.2
redundancy redundancy
local priority 10 local priority 2
peer address ipv4 10.1.12.2
MPLS/Private IP peer address ipv4 10.1.11.2

Ser 1/0: 10.1.20.2 Ser 1/0: 10.1.21.2

crypto gdoi group G1
identity number 3333 GM1 GM2 crypto gdoi group G1
server address ipv4 10.1.11.2 identity number 3333
server address ipv4 10.1.12.2 server address ipv4 10.1.12.2
! server address ipv4 10.1.11.2
crypto map gm_map 10 gdoi Eth 0/0: 192.168.20.1/24
!
set group G1 Eth 0/0: 192.168.21.1/24
! crypto map gm_map 10 gdoi
interface Serial1/0 set group G1
! crypto map gm_map !
interface Serial1/0
! crypto map gm_map
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 51
 GETVPN Control Plane Setup Steps

 COOP KS IKE Setup

 COOP Election and Policy Creation

 GM-KS IKE Setup

 GM Authorization and Registration
 GM Encryption Keys and Policy download

Periodic Key Renewal and Distribution (Rekeys)

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 52
GETVPN Common Issues – Control Plane

COOP Setup and Policy Creation

IKE Setup

Authorization and Registration

Encryption Policy

Key Renewal—Rekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 53
 COOP KS Setup and Election
 IKE comes up as soon as COOP servers boot up
KS1# show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id slot status
10.1.11.2 10.1.12.2 GDOI_IDLE 1078 0 ACTIVE

KS2#sh cry isa sa

IPv4 Crypto ISAKMP SA
dst src state conn-id slot status
10.1.11.2 10.1.12.2 GDOI_IDLE 1023 0 ACTIVE

 Based on configured priority, one of the KSs transitions to Primary KS and

other becomes Secondary

%GDOI-5-COOP_KS_ELECTION: KS entering election mode in group G1

(Previous Primary = NONE)

%GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.1.11.2 in group G1 transitioned to

Primary (Previous Primary = NONE)

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 54
 COOP Configuration Mismatch

 COOP configuration MUST be manually synchronized on

all the KSs
 Due to mismatched configuration, COOP KSs can exhibit
unexpected behavior
 Syslog Messages indicates an error condition
%GDOI-3-COOP_CONFIG_MISMATCH: WARNNING: replay method configuration
between Primary KS and Secondary KS are mismatched
KS2#

KS1: KS2:
crypto gdoi group G1 crypto gdoi group G1
server local server local
sa ipsec 1 sa ipsec 1
replay counter window-size 64 replay time window-size 5

 Fixing the mismatched configuration parameters on both KSs will

fix this issue
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 55
GETVPN Common Issues – Control Plane

COOP Setup and Policy Creation

IKE Setup

Authorization and Registration

Encryption Policy

Key Renewal—Rekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 56
 IKE Setup Between KS and GM
 First step in GM registration is IKE setup
 On successful negotiation of the IKE process, GM proceeds
with the GDOI group registration
 IKE SA is established at the time of registration eventually
times out as its no longer needed after registration

KS1#sh cry isa sa

IPv4 Crypto ISAKMP SA
dst src state conn-id status
10.1.11.2 10.1.20.2 GDOI_IDLE 1013 ACTIVE
10.1.12.2 10.1.11.2 GDOI_IDLE 1004 ACTIVE
10.1.21.2 10.1.11.2 GDOI_REKEY 0 ACTIVE

Expires
after IKE
GM1#sh cry isa sa lifetime
IPv4 Crypto ISAKMP SA
dst src state conn-id slot status
10.1.11.2 10.1.20.2 GDOI_IDLE 1073 0 ACTIVE
10.1.20.2 10.1.11.2 GDOI_REKEY 1074 0 ACTIVE

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 57
IKE Setup—Pre-shared Key Mismatch
Symptoms

 If a GM fails to register with the KS, following syslog

message will appear on the KS
%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have
expired/been cleared, or didn't go through. Re-register to KS.

%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1

using address 10.1.20.2

 Possible causes:
Network issues between the GM and KS KS1 KS2

IKE negotiation failure

MPLS/Private IP
KS policy issues

GM2
GM1

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 58
 Pre-Shared Key Mismatch
Troubleshooting

 Verify routing information on KS and GM and try pinging KS

from the GM
 After ruling out the connectivity issues, check the IKE SA on
the GM

GM1#sh crypto isa sa

IPv4 Crypto ISAKMP SA
dst src state conn-id status
10.1.11.2 10.1.20.2 MM_KEY_EXCH 1038 ACTIVE

IPv6 Crypto ISAKMP SA

IKE SA not getting established;

can’t get to GDOI_IDLE state

 Verify the logs on the Key Server

KS1#
%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 10.1.20.2 failed its
sanity check or is malformed

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 59
 Pre-Shared Key Mismatch
Solution

 Syslog pointing to a mismatched pre-shared key

configuration
 Can be verified using “debug crypto isakmp”
KS Config crypto isakmp key cicso address 10.1.20.2

GM Config crypto isakmp key cisco address 10.1.11.2

Mismatch!

 Correct the pre-shared key configuration

KS1(config)#no crypto isakmp key cicso address 10.1.20.2

KS1(config)#crypto isakmp key cisco add 10.1.20.2
KS1(config)#^Z

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 60
GETVPN Common Issues – Control Plane

COOP Setup and Policy Creation

IKE Setup

Authorization and Registration

Encryption Policy

Key Renewal—Rekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 61
 GM Authorization and Registration
 When the IKE session is successfully established, GM is
authorized (if configured) and GM registers with the KS
GM1#show crypto gdoi
GROUP INFORMATION

Group Name : G1
Group Identity : 3333
Rekeys received : 221
IPSec SA Direction : Both
Active Group Server : 10.1.11.2

<SNIP>

KS1#show crypto gdoi ks members | in Group

Group Member Information :

Group Member ID : 10.1.20.2

Group ID : 3333
Group Name : G1

Group Member ID : 10.1.21.2

Group ID : 3333
Group Name : G1

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 62
 KS1 KS2
Unauthorized GM
Symptoms MPLS/Private IP

 IKE is established between the GM and GM1

GM2

the KS
GM1#sh crypto isa sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
10.1.11.2 10.1.20.2 GDOI_IDLE 1054 ACTIVE

 Even after IKE is established, GM fails to register with the KS

%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1

using address 10.1.20.2

Registration is not complete; following

message is not displayed

%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for group

G1 using address 10.1.20.2

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 63
 Unauthorized GM
Troubleshooting Steps

 Following Syslog Message appears on the KS:

%GDOI-1-UNAUTHORIZED_IPADDR: Group G1 received registration from
unauthorized ip address: 10.1.20.2

 Verify the KS authorization policy to identify

the authorization list:

KS1#sh run | section crypto gdoi

crypto gdoi group G1 Address not
identity number 3333 in ACL
server local
<snip>
authorization address ipv4 gm-author-list
...
KS1#sh access-lists gm-author-list
Standard IP access list gm-author-list
10 permit 10.1.21.2

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 64
Unauthorized GM
Solution

 Add the GM to the authorization list

KS1(config)#ip access-list standard gm-author-list

KS1(config-std-nacl)#permit host 10.1.20.2
KS1#

 Verify the GM can now register with the KS successfully

%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1

using address 10.1.20.2
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for
group G1 using address 10.1.20.2

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 65
GETVPN Common Issues – Control Plane

COOP Setup and Policy Creation

IKE Setup

Authorization and Registration

Encryption Policy

Key Renewal—Rekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 66
GM Policy Download
 As part of the registration process, KS pushes down the
encryption policies and keying material to the GM:
GM1#show cry gdoi

<SNIP>
ACL Downloaded From KS 10.1.11.2:
access-list deny eigrp any any
access-list deny ip 224.0.0.0 0.0.0.255 any
access-list deny ip any 224.0.0.0 0.0.0.255
access-list deny udp any port = 848 any port = 848
access-list permit ip any any

KEK POLICY:
Rekey Transport Type : Unicast
Lifetime (secs) : 2954

<SNIP>

TEK POLICY:
Serial1/0:
IPSec SA:
sa direction:inbound
spi: 0x2113F73B(554956603)
transform: esp-3des esp-sha-hmac
sa timing:remaining key lifetime (sec): (99)
Anti-Replay(Time Based) : 5 sec interval
<SNIP>

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 67
 KS1 KS2
KS Policy Issues
Routing Control Plane Traffic Failure MPLS/Private IP

BGP

 In most environments, GETVPN runs on GM1 GM2

the CE devices and PE devices do not

participate in GETVPN
 Failure to deny control plane traffic (such as routing protocol)
on the PE-CE link will cause routing protocol to go down as
soon as GM successfully registers
 To identify, look at the ACL downloaded at GM:
GM1#show crypto gdoi gm acl
BGP is not denied in
Group Name: G1 the ACL downloaded
ACL Downloaded From KS 10.1.11.2: from the KS
access-list deny eigrp any any
access-list deny ip 224.0.0.0 0.0.0.255 any
access-list deny ip any 224.0.0.0 0.0.0.255
access-list deny udp any port = 848 any port = 848
access-list permit ip any any
ACL Configured Locally:

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 68
 KS Policy Issues
Control Plane Traffic—Solution

 If most of the CEs are running BGP with the PE routers,

configure a global KS policy to deny BGP

KS1&2(config)# ip access-list extended ENCPOL

KS1&2(config-ext-nacl)# deny tcp any any eq bgp
KS1&2(config-ext-nacl)# deny tcp any eq bgp any

 If only a handful of CEs are running BGP with the PE routers,

configure a local GM policy to deny BGP

GM1# GM1#sh cry gdoi gm acl

! Group Name: G1
access-list 150 deny tcp any any eq bgp ACL Downloaded From KS 10.1.11.2:
access-list 150 deny tcp any eq bgp any access-list deny eigrp any any
! access-list deny ip 224.0.0.0 0.0.0.255 any
crypto map gm_map 10 gdoi access-list deny ip any 224.0.0.0 0.0.0.255
set group G1 access-list deny udp any port = 848 any port = 848
match address 150 access-list permit ip any any
! ACL Configured Locally:
Map Name: gm_map
access-list 150 deny tcp any any port = 179
access-list 150 deny tcp any port = 179 any

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 69
 KS Policy Issues
Data Plane Traffic Failure

 Encryption policies (what needs to be encrypted)

are defined centrally at the KS
 Symmetrical ACLs should be defined to either permit
or to deny traffic from getting encrypted
 If the traffic is not being encrypted or being blocked,
verify we have symmetrical ACL

MPLS/Private IP

GM1 GM2
Eth 0/0: 192.168.20.0/24 Eth 0/0: 192.168.21.0/24

KS Access-list
ip access-list extended ENCPOL
permit ip 192.168.20.0 0.0.0.255 192.168.21.0 0.0.0.255
permit ip 192.168.21.0 0.0.0.255 192.168.20.0 0.0.0.255
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 70
GETVPN Common Issues – Control Plane

COOP Setup and Policy Creation

IKE Setup

Authorization and Registration

Encryption Policy

Key Renewal—Rekey

Control Plane Replay Check

Control Plane Fragmentation Issue

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 71
GETVPN Rekeys
 Once the GETVPN network is properly setup and is
working, KS is responsible for sending out rekey
messages to all the GMs
 KS can use unicast or multicast rekeys
 Following syslog messages will appear in the log:
PRIMARY KS:

%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from

address 10.1.11.2 with seq # 11

All the GMs:

%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to

10.1.20.2 with seq # 11

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 72
 Missing RSA Key KS1 KS2

Symptoms
MPLS/Private IP

 When GM registers to the KS, the following GM1 GM2

message shows up in the syslog:

%GDOI-1-KS_NO_RSA_KEYS: RSA Key - get : Not found, Required for group G1

 If the above message goes undetected, second

symptom is KS will not send rekey messages and
GM will re-register when the keys expire

%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have

expired/been cleared, or didn't go through. Re-register to KS.

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 73
Missing RSA Key on the KS
Troubleshooting Steps

 Check whether KS is sending out the rekeys or not:

KS1#sh crypto gdoi ks rekey

Group G1 (Multicast)
Number of Rekeys sent : 0
Number of Rekeys retransmitted : 0
KEK rekey lifetime (sec) : 86400
Retransmit period : 10
Number of retransmissions : 2
IPSec SA 1 lifetime (sec) : 3600
Remaining lifetime (sec) : 166
Number of registrations after rekey : 22

 KS needs RSA keys to sign the rekey messages;

check logs for clues and/or verify the RSA keys

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 74
 Missing RSA Key on the KS
Troubleshooting Steps (Cont.)

 Verify RSA key configuration on the KS:

KS1#sh run | section gdoi group
crypto gdoi group G1
identity number 3333
server local
rekey address ipv4 102
rekey lifetime seconds 86400
rekey authentication mypubkey rsa get
sa ipsec 1
profile gdoi-p
match address ipv4 ENCPOL
no replay
address ipv4 10.1.11.2 RSA key not present or names do not match

 Verify the RSA key pair name on the router:

KS1#sh crypto key mypubkey rsa | include name

Key name: key1
Key name: key1.server

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 75
 Missing RSA Key on the KS
Solution

 Generate the required RSA key pair

KS1(config)#crypto key generate rsa label get exportable modulus 1024
The name for the keys will be: getvpn-rsa-key

% The key modulus size is 1024 bits

% Generating 1024 bit RSA keys, keys will be exportable...[OK]

 Verify rekey messages are now being sent on the KS

%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from

address 10.1.11.2 with seq # 1

KS1#sh crypto gdoi ks rekey

Group G1 (Unicast)
Number of Rekeys sent : 1
<SNIP>

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 76
Multicast Rekey Issues
Multicast Rekeys Failing - Symptom

 GM is not getting the multicast rekey messages and

therefore continue to re-register with the KS
 Rekey starts to work when switched from multicast
rekey to unicast rekey
 Possible Causes
Packet delivery issue within the multicast routing
infrastructure
End-to-end multicast routing enabled?
mVPN service provided by the MPLS core provider?

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 77
 KS1 KS2

Multicast Rekey Failing

Troubleshooting Multicast
Network
10.1.20.2
10.1.21.2

GM1
GM2
 Check KS to verify multicast rekey messages
are being sent
%GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group G1
from address 10.1.11.2 to 226.1.1.1 with seq # 6

 Make sure ICMP is excluded from the KS encryption

policy and is used as a tool to test multicast
KS1#ping 226.1.1.1

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 226.1.1.1, timeout is 2
seconds:

Reply to request 0 from 10.1.21.2, 44 ms

No response from
GM1 (10.1.20.2)

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 78
 Multicast Rekey Failing
Troubleshooting

 Check the multicast forwarding path

WAN#sh ip mroute 226.1.1.1

<snip>
(10.1.11.2, 226.1.1.1), 00:13:18/00:02:56, flags: T
Incoming interface: Serial0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Serial3/0, Forward/Sparse-Dense, 00:13:18/00:00:00
Verify the OIL

 Check the PIM neighbor

WAN#sh ip pim neighbor
PIM Neighbor Table
Neighbor Interface Uptime/Expires Ver DR
Address
Prio/Mode
10.1.11.2 Serial0/0 01:03:54/00:01:16 v2 1 / S
10.1.21.2 Serial3/0 01:13:06/00:01:26 v2 1 / S

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
Multicast Rekey Failing
Solution

 Enable PIM on the WAN router towards the GM

WAN(config)#int s2/0
WAN(config-if)#ip pim sparse-dense-mode
WAN(config-if)#end

%PIM-5-NBRCHG: neighbor 10.1.20.2 UP on interface Serial2/0

(vrf default)

 Check multicast routing path again

 Re-test with multicast ping
 Verify GM now receives the multicast rekey
messages

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 80
Unicast Rekey Failing
Transient Network Issues

 Due to transient changes in the network, unicast

rekey packets might not make it to the GM(s)
 If the GMs does not receive the rekey, it will have
to re-register
Symptoms:

Missing Following syslog on GM:

%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from

10.1.11.2 to 10.1.21.2 with seq # 3

GM shows re-registration syslog:

%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may

have expired/been cleared, or didn't go through. Re-register
to KS.
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for
group G1 using address 10.1.20.2

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 81
Unicast Rekey Failing
Troubleshooting and Solution
 Verify whether the rekeys are not being sent, not
being received or not being processed
KS: GM:
show crypto gdoi ks members show crypto gdoi gm rekey

Group Member Information : Group G1 (Unicast)

Number of rekeys sent for group G1 : 380 Number of Rekeys received (cumulative) : 0
Number of Rekeys received after registration : 0
Group Member ID : 10.1.20.2 Number of Rekey Acks sent : 0
Group ID : 3333 Rekey (KEK) SA information :
Group Name : G1 dst src conn-id my-cookie his-
Key Server ID : 10.1.11.2 cookie
Rekeys sent : 1 New : 10.1.20.2 10.1.11.2 1098 44F7FC32
Rekeys retries : 0 8302AC61
Rekey Acks Rcvd : 0 Current : 10.1.20.2 10.1.11.2 1098 44F7FC32
Rekey Acks missed : 0 8302AC61
Previous: --- --- --- --- ---

 Always configure retransmissions to overcome

temporary transient issues
rekey retransmit 30 number 3
 Make sure UDP port 848 is not blocked in the data path
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 82
Rekey Fails Signature Validation
 Primary KS fails, GM receives rekey from
secondary KS, but receives error:
*Apr 27 18:18:19.511: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI
mode failed with peer at 10.1.12.2

 Syslog is not conclusive, let’s see what we can get

with some debugs
GM1# debug crypto isakmp
Crypto ISAKMP debugging is on
GM1#
GM1# debug crypto gdoi Signature validation failed!
GDOI Generic Debug level: (Error, Terse)
*Apr 27 18:18:19.251: ISAKMP (0:1014): received packet from 10.1.12.2 dport 848 sport
848 Global (R) GDOI_REKEY
*Apr 27 18:18:19.251: GDOI:INFRA:(G1:0:1014:HW:0):Received Rekey Message!
*Apr 27 18:18:19.259: GDOI:INFRA:(G1:0:1014:HW:0):Signature Invalid! status = 13
*Apr 27 18:18:19.259: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed
with peer at 10.1.12.2
*Apr 27 18:18:19.259: ISAKMP: Receive GDOI rekey: Processing Failed. IKMP error = 6

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 83
 Rekey Fails Signature Validation
Solution

 Problem:
Secondary KS has its own RSA key pair instead of the
exported key pair from the primary
To verify, compare the RSA key pairs KS1 KS2

KS#show crypto key mypubkey rsa MPLS/Private IP

 Solution: GM1 GM2

Generate exportable RSA key pair on the primary KS

KS1(config)#crypto key generate rsa modulus 1024 exportable label key1

Export RSA key pair to all secondary KSs

KS2(config)#crypto key import rsa key1 pem terminal <passphrase>

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 84
GETVPN Common Issues

COOP Setup and Policy Creation

IKE Setup

Authorization and Registration

Encryption Policy

Key Renewal—Rekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 85
 Control Plane Replay Check Detection

 Control Plane messages can carry time sensitive

information and therefore require replay protection
Rekey messages from KS to GM
COOP Announcement messages between KSs
 Sequence number check to protect against replayed
messages
 Pseudotime check to protect against delayed messages
with TBAR enabled
 Control Plane Replay check added in IOS version
12.4(15)T10, 12.4(22)T3, 12.4(24)T2, 15.0(1)M,
12.2(33)XND2, 12.2(33)XNE and later

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 86
Control Plane Replay Check
Code interoperability issue

 Problem: customer upgraded IOS on a GM to

15.0(1)M for a bug fix, and started to experience
KEK rekey failures
 The following errors are observed in the syslog

*Apr 6 15:23:38.424: %GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to

process rekey seq # 1 in seq payload for group G1, last seq # 11
*Apr 6 15:23:38.424: %GDOI-3-GDOI_REKEY_FAILURE: Processing of
REKEY payloads failed on GM 10.1.13.2 in the group G1, with peer at
10.1.11.2
*Apr 6 15:23:38.424: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of
GDOI mode failed with peer at 10.1.11.2

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 87
 Control Plane Replay Check
Code interoperability issue - solution

 KS does not support control plane replay detection, and

resets the rekey sequence # for KEK rekey
 GM interprets that as a replayed rekey message
 Solution is to upgrade the KS to an IOS version that
also supports the control plane replay detection
 New behavior
KEK Rekey

*Apr 6 15:41:26.932: %GDOI-5-GM_RECV_REKEY: Received Rekey for

group G1 from 10.1.11.2 to 10.1.13.2 with seq # 8
GM1#
*Apr 6 15:42:01.940: %GDOI-5-GM_RECV_REKEY: Received Rekey for
group G1 from 10.1.11.2 to 10.1.13.2 with seq # 1

TEK Rekey with seq# reset

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 88
Control Plane Replay Check – IOS
Upgrade procedure
 Recommended IOS releases
IOS: 12.4(15)T12
IOS-XE: 12.2(33)XND3

 IOS upgrade procedure

Step 1. Upgrade a secondary KS first, wait until COOP KS
election is completed
Step 2. Repeat step 1 for all secondary KS
Step 3. Upgrade primary KS
Step 4. Upgrade Group Members

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 89
GETVPN Common Issues

COOP Setup and Policy Creation

IKE Setup

Authorization and Registration

Encryption Policy

Key Renewal—Rekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 90
Control Plane Fragmentation Issues
COOP Announcement Packets

 In a large network (1500+ GMs), COOP update

packet becomes larger than the default maximum
buffer size
 Default huge buffer size is 18024 bytes
 Syslog message appears on the KSs:
Symptoms:

%SYS-2-GETBUF: Bad getbuffer, bytes= 18872 -Process= "Crypto IKMP", ipl= 0, pid= 183

 Tune buffers to increase huge buffers and add

buffers to permanent list:
buffers huge permanent 10
buffers huge size 65535

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 91
GETVPN Data Plane

 IPSec tunnel mode just like IPSec classic so most IPSec

troubleshooting techniques still apply, however…
 Unique challenges with Header Preservation
PMTUD

 Time Based Anti-Replay

Extra encapsulation overhead – Fragmentation boundary condition
calculation
Timer Based Anti-Replay failure

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 92
Generic IPSec Data Plane Troubleshooting
 Need to have complete understanding of the forwarding
path and how to checkpoint it
 Some syslogs may help reveal data plane drops
Data plane errors are typically rate limited
Common errors include replay, authentication failures

 Heavily dependent upon show commands and counters to

trace the packet path
 Sniffer capture of limited use due to encryption, however
ESP-NULL – same crypto processing except packets not encrypted
DSCP coloring of packets to uniquely identify a flow

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 93
IPSec Data Plane Packet Flow Checkpoints
3 4
Traffic Direction 1 6

Private WAN
GM1 GM2

Client Server
2 5

 Encrypting GM  Decrypting GM
1. Ingress LAN interface 4. Ingress WAN interface
Input ACL Input ACL
Ingress Netflow Ingress Netflow
Embedded Packet Capture Embedded Packet Capture
2. Crypto engine Input IP precedence accounting
show crypto ipsec sa 5. Crypto engine
show crypto session detail show crypto ipsec sa
3. Egress WAN interface show crypto session detail
Egress Netflow 6. Egress WAN interface
Embedded Packet Capture Egress Netflow
Output IP precedence accounting Embedded Packet Capture

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 94
Encrypting GM Data Plane Flow
 Verify clear traffic being received with Ingress Netflow
interface Ethernet0/0
ip address 192.168.13.1 255.255.255.0
ip flow ingress
!
GM1#sh ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Et0/0 192.168.13.2 Se1/0 192.168.14.2 06 E443 0017 170

TCP port 23 = telnet

 Verify encryption operation performed
 Lack of per-flow granularity
GM1#show crypto session detail
<snip>
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 4, origin: crypto map
Inbound: #pkts dec'ed 162 drop 0 life (KB/Sec) 0/146
Outbound: #pkts enc'ed 170 drop 0 life (KB/Sec) 0/146

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 95
Encrypting GM Data Plane Flow – Cont.
 Verify encrypted traffic existing GM with egress Netflow
interface Serial/0
ip address 10.1.13.2 255.255.255.252
ip flow egress
!
GM1#sh ip cache flow Protocol 50 = ESP
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Et0/0 192.168.13.2 Se1/0* 192.168.14.2 32 EE5B 2BEF 170

GM1#show crypto ipsec sa Active IPSec SA SPI

interface: Serial1/0
<snip>
current outbound spi: 0xEE5B2BEF(3998952431)

 If per L4 flow granularity is desired, can use inbound

precedence coloring and egress precedence accounting

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 96
 Decrypting GM Data Plane Flow
 Verify encrypted traffic arriving on GM with Netflow
Protocol 50 = ESP
GM2#sh ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Se1/0 192.168.13.2 Et0/0 192.168.14.2 32 EE5B 2BEF 170

 Verify traffic decryption Inbound IPSec SA SPI

GM2#show crypto session detail
<snip>
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 10, origin: crypto map
Inbound: #pkts dec'ed 170 drop 0 life (KB/Sec) 0/150
Outbound: #pkts enc'ed 162 drop 0 life (KB/Sec) 0/150

 Verify clear traffic forwarding post decryption

GM2#sh ip cache flow TCP port 23 = telnet
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Se1/0 192.168.13.2 Et0/0* 192.168.14.2 06 E6CC 0017 170

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 97
GETVPN Common Issues – Data Plane

Network/Path MTU

Other data plane issues common to IPSec

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 98
 Fragmentation Issues
PMTU Discovery

 Large packets with the DF bit set may get black-

holed in the GETVPN network

MTU 1500 MTU 1500

GM1
MTU 1000 GM2

1400B 1460B

ICMP 3/4

 Server sends a large packet with the DF bit set in an attempt to perform
network PMTUD

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 99
 PMTUD and GETVPN

 Encrypting GM adds IPSec overhead and forwards it

 Intermediate router drops the packet and sends back
icmp3/4 to perform PMTUD, two possibilities
This ICMP dropped by the encrypt GM because it’s not
encrypted based on the encryption policy
This ICMP gets forwarded to the end host but gets dropped due
to unauthenticated payload

 Bottom line: PMTUD does not work with the current

header preservation implementation of GETVPN

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 100
 PMTUD and GETVPN
Solution
 Implement ip tcp adjust-mss to reduce the TCP packet
segment size
 Clear the DF bit in the encapsulating header User Traffic
Encrypting GM

DF=0 DF=0 Data

DF=1 Data

interface Ethernet0/0
ip address 192.168.13.1 255.255.255.0
ip policy route-map clear-df-bit
!
route-map clear-df-bit permit 10
match ip address 111
set ip df 0
!
access-list 111 permit tcp any any

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 101
GETVPN Common Issues – Data Plane

Network/Path MTU

Other data plane issues common to IPSec

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 102
 IPSec drop due to packet corruption
 IPSec integrity check makes IPSec packets a lot
more sensitive to packet corruption in the network
 Packet corruption symptoms
*Apr 6 20:57:09.171: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify
failed for connection id=695 local=192.168.14.2 remote=192.168.13.2
spi=7C4E759F seqno=00000001

 How to prove packets are corrupted in the network?

Enable EPC to capture packets into a circular buffer on both GMs
Use EEM (Embedded Event Manager) to
Synchronize and stop the capture on both routers when the
RECVD_PKT_MAC_ERR message is logged

Notify the network operator by email

Retrieve both captures to examine for packet corruption

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 103
 GETVPN Troubleshooting Summary

 Have a clear and concise problem description

 Try to break the problem down to either control or
data plane
 Understand the expected protocol flow on the control
plane and how to check for them
 Understand where/how to checkpoint the data plane
 Syslog is your friend
 There is always TAC! 

BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 104
BRKSEC-3011 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 105