Advanced Concepts of

DMVPN (Dynamic Multipoint VPN)

Mike Sullenberger – Distinguished Engineer

BRKSEC-4054
Cisco Spark

Questions?
Use Cisco Spark to communicate
with the speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

Cisco Spark spaces will be cs.co/ciscolivebot#BRKSEC-4054

available until July 3, 2017.
VPN and IWAN Breakout Sessions
• BRKRST-2362 - IWAN - Implementing Performance Routing (PfRv3) Monday
• BRKCRS-2000 - Intelligent WAN (IWAN) Architecture
• BRKSEC-3001 - Advanced IKEv2 Protocol
• BRKSEC-2054 - GET Your VPN's Secured with ESON
• BRKRST-2042 - Highly Available Wide Area Network Design
• BRKRST-2041 - WAN Architectures and Design Principles Tuesday

• BRKCRS-2002 - IWAN Design and Deployment Workshop

• BRKSEC-3054 - IOS FlexVPN Remote Access, IoT and Site-to-Site advanced Crypto VPN Designs
• BRKCRS-2007 - Migrating Your Existing WAN to Cisco’s IWAN
• BRKSEC-3052 - Demystifying DMVPN
• BRKRST-3413 - IWAN Serviceability: Deploying, Monitoring, and Operating
• BRKRST-2557 - IWAN and NFV Orchestration for Managed Service Providers
• BRKRST-3018 - Understanding and Troubleshooting Intelligent Path Control in IWAN Wednesday

• BRKSEC-4054 - Advanced Concepts of DMVPN

• BRKSEC-3005 - Cryptographic Protocols and Algorithms - a review Thursday
Agenda
• DMVPN Design Overview
• DMVPN Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• DMVPN Network Segmentation
• VRF-lite over DMVPN
• MPLSoDMVPN
DMVPN Design Overview
Agenda
• DMVPN Design Overview
• DMVPN Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• DMVPN Network Segmentation
• VRF-lite over DMVPN
• MPLSoDMVPN
What is Dynamic Multipoint VPN?

DMVPN is a Cisco IOS software solution

for building IPsec+GRE VPNs in an
easy, dynamic and scalable manner

• Uses two proven technologies

• Next Hop Resolution Protocol (NHRP)
• Creates a distributed mapping database of VPN (tunnel int.) to real (public int.) addresses
• Multipoint GRE Tunnel Interface
• Single GRE interface to support multiple GRE/IPsec tunnels and endpoints
• Simplifies size and complexity of configuration
• Supports dynamic tunnel creation
DMVPN Philosophy
• Distributed NHRP database
• No single point must have all the NHRP information for the DMVPN
• No single point limits the overall size of the DMVPN
• Don’t drop packets while building dynamic tunnels
• Pre-build (hierarchical) hub-and-spoke network
• Forward data packets via pre-built path until direct tunnel is ready
• Dynamic Mesh versus Full Mesh
• Small nodes participate in large DMVPNs up to their capabilities
• Doesn’t limit the participation of other (larger) nodes in the DMVPN
DMVPN Major Features
• Configuration reduction and no-touch deployment
• Supports:
• Passenger protocols (IP(v4/v6) unicast, multicast and dynamic Routing Protocols)
• Transport protocols (NBMA) (IPv4 and IPv6)
• Remote peers with dynamically assigned transport addresses.
• Spoke routers behind dynamic NAT; Hub routers behind static NAT.
• Dynamic spoke-spoke tunnels for partial/full mesh scaling.
• Can be used without IPsec Encryption
• Works with MPLS; GRE tunnels and/or data packets in VRFs and MPLS
switching over the tunnels
• Wide variety of network designs and options.
DMVPN Phases
Phase 2 – 12.3(4)T Phase 3 – 12.4.(6)T
Phase 1 – 12.2(13)T
(Phase 1 +) IWAN 1.0 (Phase 2 +) IWAN 2.0
• Hub and spoke functionality • Spoke to spoke functionality • More network designs and
• mGRE or p-pGRE interface • mGRE interface on spokes greater scaling
on spokes, mGRE on hubs • Direct spoke to spoke data • Same Spoke to Hub ratio
• Simplified and smaller traffic reduces load on hubs • No hub daisy-chain
configuration on hubs • Hubs must interconnect in • Spokes don’t need full routing
• Support dynamically daisy-chain table – can summarize
addressed CPEs (NAT) • Spoke must have full routing • Spoke-spoke tunnel triggered
• Support for routing table – no summarization by hubs
protocols and multicast • Spoke-spoke tunnel • Remove routing protocol
• Spokes don’t need full triggered by spoke itself limitations
routing table – can • Routing protocol limitations • NHRP routes/next-hops in
summarize on hubs RIB (15.2(1)T)
• No touch deployment
DMVPN How it works
• Spokes build a dynamic permanent GRE/IPsec tunnel to the hub, but not to
other spokes. They register as clients of the NHRP server (hub).
• When a spoke needs to send a packet to a destination (private) subnet behind
another spoke, it queries via NHRP for the real (outside) address of the
destination spoke.
• Now the originating spoke can initiate a dynamic GRE/IPsec tunnel to the target
spoke (because it knows the peer address).
• The dynamic spoke-to-spoke tunnel is built over the mGRE interface.
• When traffic ceases then the spoke-to-spoke tunnel is removed.
DMVPN Example
192.168.0.0/24
Static Spoke-to-hub tunnels
.1 LANs can have
private addressing
Dynamic Spoke-to-spoke tunnels

Physical: 172.17.0.1
Tunnel0: 10.0.0.1
Static known
IP address

Physical: dynamic
Tunnel0: 10.0.0.12

Dynamic
unknown
IP addresses Spoke B
.1

192.168.2.0/24

Physical: dynamic
Tunnel0: 10.0.0.11
Spoke A
.1 ...
192.168.1.0/24
DMVPN and IPv6
• IPv6 Passenger over DMVPN (IPv4 or IPv6) Transport
• IPv6 Passenger Addresses:
• NHRP requires IPv6 Unicast Global
• Routing Protocol requires IPv6 Link-local
• NHRP automatically registers both Unicast Global and Link-local Addresses
• IPv4 or IPv6 infrastructure transport network (separate mGRE tunnel interfaces)
• Both IPv4 and IPv6 (dual stack) can be over the same DMVPN mGRE tunnel
• (IPv4 and/or IPv6) Passenger over DMVPN IPv6 Transport
• Use IKEv2 for IPsec encryption key management
• Standard IPv6 configuration on Outside (WAN) interface
• IPv4 and IPv6 transports require separate DMVPNs (mGRE tunnels)
• DMVPN IPv4  DMVPN IPv6 spoke to spoke via hub
• WAN interface may support both IPv4 and IPv6 (dual stack)
DMVPN and IPv6 – Configuration
crypto ikev2 keyring DMVPN interface Tunnel0
peer DMVPNv6 ip address 10.0.0.11 255.255.255.0
address ::/0 ...
pre-shared-key cisco123v6 ip nhrp network-id 100000
peer DMVPNv4 ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
address 0.0.0.0 0.0.0.0 ...
preshared-key cisco123v4 ipv6 address 2001:DB8:0:100::B/64
! ...
crypto ikev2 profile DMVPNv6 ipv6 nhrp network-id 100006
match identity remote address ::/0 ipv6 nhrp nhs 2001:DB8:0:100::1 nbma 172.17.0.1 multicast
authentication local pre-share ...
authentication remote pre-share tunnel source Serial1/0
keyring DMVPN tunnel mode gre multipoint
dpd keepalive 30 5 on-demand tunnel protection ipsec profile DMVPNv4
crypto ikev2 profile DMVPNv4 !
match identity remote address 0.0.0.0 0.0.0.0 interface Tunnel1
authentication local pre-share ip address 10.0.6.11 255.255.255.0
authentication remote pre-share ...
keyring DMVPN ip nhrp network-id 100000
dpd keepalive 30 5 on-demand ip nhrp nhs 10.0.6.1 nbma 2001:DB8:0:FFFF:1::1 multicast
! ...
crypto ipsec profile DMVPNv6 ipv6 address 2001:DB8:0:106::B/64
set transform-set DMVPN ...
set ikev2-profile DMVPNv6 ipv6 nhrp network-id 100006
! ipv6 nhrp nhs 2001:DB8:0:106::1 nbma 2001:DB8:0:FFFF:1::1 multicast
crypto ipsec profile DMVPNv4 ...
set transform-set DMVPN tunnel source Serial1/0
set ikev2-profile DMVPNv4 tunnel mode gre multipoint ipv6
… tunnel protection ipsec profile DMVPNv6
interface Serial1/0 !
ip address 172.16.1.1 255.255.255.252 ip route 0.0.0.0 0.0.0.0 Serial1/0
ipv6 address 2001:DB8:0:FFFF:0:1:0:1/126 ipv6 route ::/0 Serial1/0
DMVPN and IPsec
• IPsec integrated with DMVPN, but not required
• Packets Encapsulated in GRE, then Encrypted with IPsec
• Supports both IKEv1 (ISAKMP) and IKEv2
• NHRP controls the tunnels, IPsec does the encryption
• Bringing up a tunnel
• NHRP signals IPsec to setup encryption
• ISAKMP/IKEv2 authenticates peer, generates SAs
• IPsec responds to NHRP and the tunnel is activated
• All NHRP and data traffic is Encrypted
• Bringing down a tunnel
• NHRP signals IPsec to tear down tunnel
• If encryption is cleared or lost IPsec can signal NHRP to clear the tunnel
• ISAKMP/IKEv2 Keepalives monitor remote crypto peers*
* BFD over DMVPN
DMVPN Encryption Scaling
Throughput depends on number
SLB Design and types of hub platforms

(45 GB) ASR1002-HX, ASR1006+/RP3/ESP200

(18 GB) ASR1006+/RP2/ESP100

(7 GB) ASR1006+/RP2/ESP40

(6 GB) ASR1004+/RP2/ESP20

(4 GB) ASR100[1,2]-X/Integrated

(2.5 GB) ASR1004+/RP2/ESP10

(1 GB) 4451-X IMIX Encryption

Throughput at
4351 70% Max CPU

500 M 1.0 G 2.0 G 4.0 G 8.0 G 16.0 G 32.0 G

Routing over DMVPN
• Supports all routing protocols, except ISIS
• Hubs are routing neighbors with spokes
• Receive spoke network routes from spokes
• Advertise spoke and local networks to all spokes
• Phase 1 & 3: Can Summarize (except OSPF)
• Phase 2: Cannot summarize (OSPF limited to 2 hubs)
• Hubs are routing neighbors with other hubs
• Phase 1: Can use different interface and routing protocol than hub-spoke tunnels
• Phase 2: Must use same tunnel interface and routing protocol as hub-spoke tunnels
• Phase 3: Can use different tunnel interface and routing protocol than hub-spoke tunnels
• Spokes are only routing neighbors with hubs, not with other spokes
• Phase 3: Spoke-spoke NHRP routes are added by NHRP directly to routing table (15.2(1)T)
Routing Protocols over DMVPN
EIGRP
• Distance Vector style matches with DMVPN NBMA network style
• Feasible successor for quick spoke-to-hub convergence
• Good scaling with reasonably fast convergence (hello 5, hold 15)
• Good metric control (automatic and/or manual)
• Change metrics, route tagging, filtering or summarization at hub and/or spoke
• Can be used to control load-balancing of spoke  hub(s) traffic
• Automatic metric increase per DMVPN hop
• Feature additions – spoke-spoke load-balance support
• Equal Cost MultiPath (15.2(3)T, 15.2(1)S)
• Add-path (15.3(1)S)
Routing Protocols over DMVPN
BGP
• Base Distance Vector style matches with DMVPN NBMA network style
• iBGP (recommended)
• Dynamic Neighbors, MED to control/compare routes;
• May need iBGP local-as (15.2(2)T, 15.1(3)S)
• eBGP (okay)
• AS-Path length to control/compare routes
• Good scaling but with slower convergence (hello 15+, hold 45+)
• Good metric control (manual)
• Change metrics, route tagging, filtering or summarization at hub and/or spoke
• Can be used to control load-balancing of spoke  hub(s) traffic
• Only manual metric increase per DMVPN hop
• Some issues with Equal Cost multi-path (ECMP) route selection
• Between multiple DMVPNs and preserving correct next-hop
• Spoke-spoke tunnel load-balancing for spoke sites with multiple spoke routers
Routing Protocols over DMVPN
OSPF
• Link-state style doesn’t match as well with DMVPN NBMA network style
• Area issues – DMVPN requires single Area
• Area 0 over DMVPN – spoke sites can be in different areas
• But, area 0 is extended over WAN – possible stability issues for Area 0
• Non-Area 0 over DMVPN – all spoke sites in this same single area
• Multi-subnet DMVPN can be used to have multiple OSPF areas
• Increase in complexity of DMVPN and OSPF design
• More difficult metric control
• Can only change metrics, filter or summarize at area boundaries
• Automatic metric increase per DMVPN hop
• Issue for failover path between multiple DMVPNs – slightly reduce Hub vs. Spoke cost
• No issues with Equal Cost multi-path (ECMP) route selection
Dynamic Routing Protocols
Routing Scaling
Converge
Protocol Route/Metric Control (ASR1k Notes
(hello/hold)
Type neighbors per hub)

Fair
• Summarize and metric control
Low Single Area over DMVPN
Link-State1 Faster
OSPF (multicast) (5/15) to (20/60)
only at area border (1500-2000) • Area = 0 (spokes in different areas)
• Automatic per hop metric (dynamic) • Area  0 (all spokes in same area)
increase

Good
Distance • Summarize and metric control
Medium
Faster Spokes: Stub/Stub-site
EIGRP Vector
(5/15) to (20/60)
at any node (4000-6000)
• Suppress EIGRP Queries
(multicast) • Automatic per hop metric (dynamic)
increase

iBGP
Good High • Hubs: route-reflector; iBGP local-AS;
Distance
Slower • Summarize and metric control (6000-10000) Dynamic neighbors;
BGP Vector
at any node • Metric: MED, Local-pref
(15/45) to (60/180) (iBGP: dynamic)
(unicast) • Manual metric control (eBGP: static) eBGP
• Metric: AS Path-length, Local-pref
1 Link-State is not a good match for NBMA style (hub-and-spoke) networks like DMVPN
Routing Protocol?
• Which routing protocol should I use?
• In general you would use the same routing protocol over DMVPN that you use in the
rest of your network, or over other WAN networks (like MPLS).
• BUT...
• EIGRP being an advanced distance vector protocol matches really well with DMVPN
network topologies
• BGP, specifically iBGP, runs well over DMVPN, but is more complicated to setup to
have it act more like an IGP than an EGP
• OSPF can run over DMVPN, BUT lower scaling and Area 0 issues can complicate the
network
• RIP can be used, but has longer hold time and limited metric values
• IS-IS cannot be used since it doesn’t run over IP
 Routing Protocol Scaling
Estimate

ASR100x-(H)X
OSPF 4451-X ASR1004+/ESP20+
Estimate

ASR100x-(H)X ASR1009
EIGRP 4451-X ASR1004+/ESP40+ RP3/ESP200 Estimate

ASR100x-X ASR100x-HX
BGP 4451-X ASR1004+/ESP20+ ASR1004+/ESP100+ ASR1009/RP3/ESP200

SLB design using BGP or EIGRP

2000 4000 6000 8000 10000

Number of Branches
Redundancy
• Active-active redundancy model – two or more hubs per spoke
• All configured hubs are active and are routing neighbors with spokes
• Can use Backup NHS feature to activate a subset of configured hubs
• Can use ‘if-state nhrp’ and ‘backup interface ...’ to disable/enable a backup tunnel interface
• Routing protocol routes are used to determine traffic forwarding
• Single route: one tunnel (hub) at a time – primary/backup mode
• Multiple routes: multiple tunnels (hubs) – load-balancing mode (CEF, PfR)
• (ISAKMP/IKEv2)/IPsec
• Cannot use IPsec Stateful failover (NHRP isn’t supported)
• Invalid SPI recovery is not useful with DMVPN
no crypto isakmp invalid-spi-recovery
• ISAKMP/IKEv2 keepalives on spokes for DPD
• BFD over DMVPN for quicker spoke-hub and spoke-spoke failure discovery
Redundancy (cont)
• Can use single or multiple DMVPNs for redundancy
• Each mGRE interface is a separate DMVPN network using
• Same: Tunnel source (optional).
• Different: NHRP network-id and IP subnet, Tunnel key
• When using same tunnel source  different tunnel keys, same IPsec profile (name) and shared
tunnel protection ipsec profile name shared
• Can “glue” mGRE interfaces into same DMVPN network (Phase 3 only)
• Same: NHRP network-id and authentication, Tunnel key (optional)
• Different: Tunnel source and IP subnet
• Spokes – two or more hubs (NHSs)
• Phase 1: (Hub-and-spoke)
• p-pGRE interfaces  two or more DMVPN networks, one hub (NHS) on each
• Phase 1, 2 or 3: (Hub-and-spoke or Dynamic Mesh)
• mGRE interface  one DMVPN network, two or more hubs (NHSs)
Redundancy (cont.)
• Hubs – interconnect and routing
• Phase 1: (Hub and spoke only)
• Interconnect hubs directly over physical link, p-pGRE or mGRE tunnel
• Can exchange routing through any of these paths
• Same or different routing protocol as with spokes
• Phase 2: (Dynamic Mesh)
• Must interconnect hubs over same mGRE tunnel as spokes, daisy-chain as NHSs
• Must exchange routing over DMVPN network
• Must use same routing protocol as with spokes
• Phase 3: (Dynamic Mesh)
• Interconnect hubs over same or different mGRE tunnel (same NHRP Network-id)
• Must exchange routing over a DMVPN network
• Same or different routing protocol as with spokes
Spoke-Spoke and Spoke-Hub Tunnels
Considerations
• Resiliency
• BFD over DMVPN for quick spoke-hub and/or spoke-spoke tunnel recovery
• Can also use ISAKMP/IKEv2 keepalives, but doesn’t test data channel – Spokes only
crypto {isakmp keepalive | ikev2 dpd} initial retry [on-demand | periodic] (Recommend: initial=30, retry=5)
crypto {isakmp | ikev2} nat keepalive interval (Recommend: interval=30)

• Path Selection
• NHRP will always try to build spoke-spoke tunnel
• No bandwidth/latency measurement of spoke-spoke vs. spoke-hub-spoke paths
• Can do interesting things with Smart-spoke feature
• Overloading routers
• CPU or memory  IKE Call Admission Control (CAC) – Hubs
crypto call admission limit ike {sa | in-negotiation} max-SAs (Default: no-limit)
crypto ikev2 limit max-in-negotiation-sa max-SAs {inbound | outbound} (Default: inbound: 40, outbound: 400)
show crypto call admission statistics
• Bandwidth  Design for expected traffic
• Hub-spoke versus Spoke-spoke; Spoke-spoke availability is best effort
Best Practices
• mGRE Tunnel configuration
• Both Hubs and Spokes
tunnel source interface-name
bandwidth <from WAN-interface> (as starting point, may adjust)
ip mtu 1400; ip tcp adjust-mss 1360
• NHRP
• Spokes
ip nhrp holdtime 600*
ip nhrp shortcut*
ip nhrp nhs {hub-tunnel-ip | dynamic} nbma {hub-nbma-ip | hub-fqdn} multicast (12.4(20)T)
• Hubs
ip nhrp redirect * Default in 16.3
ip nhrp map multicast dynamic*
ip nhrp server-only
Best Practices (cont)
• Routing
• Phase 2 – RP advertises routes with remote spoke as the next-hop
• EIGRP: (hubs) no ip [next-hop-self | split-horizon] eigrp as, (all) use delay to adjust metric
• OSPF: (all) ip ospf network broadcast; (spokes only) ip ospf priority 0
• BGP: iBGP (hubs) route-reflectors; (spokes) neighbor hub next-hop-self

• Phase 1 & 3 – RP advertises routes with the hub as the next-hop

• EIGRP: (hubs) no ip split-horizon eigrp <as>
• OSPF: (all) ip ospf network point-multipoint; prefix-suppression (suppress /32 routes)
• BGP: iBGP (hubs) route-reflectors; (all) neighbor [hub | spoke] next-hop-self all

• To manipulate path selection through DMVPN use:

• EIGRP: delay not bandwidth; OSPF: cost; iBGP: MED, Local-pref
Cisco IOS Code and Platform Support
* Recommended

• 3900(E), 2900, 1900, 890, 819, 880

• 15.3.3M9*, 15.4.3M7*, 15.5.3M5*, 15.6.3M2+
+ N/A for 881-887
• 15.4.2T4, 15.5.2T4+, 15.6.2T2+

• ASR1002-X, ASR100[4,6,6-X,9-X,13](RP2), 4451-X, 4431, 4300

• (3.13.7S)154-3.S7*, (3.15.4S)155-2.S4, (3.16.5S)155-3.S5*, (3.17.3S)156-1.S3
• Denali: 16.3.3, Everest: 16.4.2, 16.5.1b

• ASR100[1,2]-HX, ASR100[6,9]-X(RP3), ASR1013(RP3), 4221

+ N/A for 4221
• Denali: 16.3.3+, Everest: 16.4.2, 16.5.1b
• CSR1000V
• (3.13.7S)154-3.S7, (3.15.4S)155-2.S4, (3.16.5S)155-3.S5*, (3.17.3S)156-1.S3
• Denali: 16.3.3, Everest: 16.4.2, 16.5.1b
Basic DMVPN Designs
• Hub-and-spoke – Order(n)
• Spoke-to-spoke traffic via hub
• Phase 1: Hub bandwidth and CPU limit VPN
• SLB: Many “identical” hubs; increases CPU and bandwidth limits
• Spoke-to-spoke – Order(n) « Order(n2)
• Control traffic; Hub and spoke; Hub to hub
• Phase 2: (single)
• Phase 3: (hierarchical)
• Unicast Data traffic; Dynamic mesh
• Spoke routers support spoke-hub and spoke-spoke tunnels currently in use.
• Hub supports spoke-hub traffic and overflow from spoke-spoke traffic.
• Network Virtualization
• VRF-lite; Multiple DMVPNs (one per VRF)
• MPLS over DMVPN (2547oDMVPN); Single DMVPN (many VRFs)
Basic DMVPN Designs
Dual DMVPN Single Hub Single DMVPN Dual Hub
Single mGRE tunnel on Hub, Single mGRE tunnel on all nodes
two p-pGRE tunnels on Spokes
192.168.0.0/24 192.168.0.0/24
.2 .1 .2 .1

Physical: 172.17.0.5 Physical: 172.17.0.1 Physical: 172.17.0.5 Physical: 172.17.0.1

Tunnel0: 10.0.1.1 Tunnel0: 10.0.0.1 Tunnel0: 10.0.0.2 Tunnel0: 10.0.0.1

Physical: (dynamic)
Tunnel0: 10.0.0.12 Physical: (dynamic)
Tunnel1: 10.0.1.12 Tunnel0: 10.0.0.12

Spoke B Spoke B .1
Physical: (dynamic) .1
Tunnel0: 10.0.0.11 Physical: (dynamic)
Tunnel1: 10.0.1.11 192.168.2.0/24 Tunnel0: 10.0.0.11 192.168.2.0/24

Spoke A Spoke A
.1 .1 ...
192.168.1.0 /24 192.168.1.0/24
= Dynamic Spoke-to-spoke
Multiple DMVPNs versus Single DMVPN
• Multiple DMVPNs
• Best for Hub-and-spoke only
• Easier to manipulate RP metrics between DMVPNs for Load-sharing
• EIGRP – Route tags, Delay; iBGP – Communities, MED; OSPF – Cost
• Performance Routing (PfR) selects between interfaces
• Load-balancing over multiple ISPs (physical paths)
• Load-balance data flows over tunnels  Better statistical load-balancing
• Single DMVPN
• Best for spoke-spoke DMVPN
• Can only build spoke-spoke within a DMVPN not between DMVPNs*
• Slightly more difficult to manipulate RP metrics within DMVPN for Load-sharing
• EIGRP – Route tags, delay; iBGP – Communities, MED; OSPF – Can’t do
• Load-balancing over multiple ISPs (physical paths)
• Load-balance tunnel destinations over physical paths  Worse statistical load-balancing
DMVPN Combination Designs
Retail/Franchise Dual ISP

ISP ISP
1 2

Spoke-to-hub tunnels
Spoke-to-hub tunnels
Spoke-to-spoke tunnels
Spoke-to-spoke tunnels
Spoke-hub-hub-spoke tunnel
DMVPN Combination Designs (cont)
Hierarchical Server Load Balancing

Spoke-to-hub tunnels
Spoke-to-spoke tunnels
Spoke-to-hub tunnels
Spoke-to-spoke tunnels
Hub-to-hub tunnel
Network Virtualization
Separate DMVPN mGRE tunnel per VRF (VRF-lite)
• Hub routers handle all DMVPNs VRF-lite
• Multiple Hub routers for redundancy and load
• IGP used for routing protocol over
DMVPNs on Spokes and Hubs
• Address family per VRF
• Routing neighbor per spoke per VRF
• BGP used only on the hub
• Redistribute between IGP and BGP for
import/export of routes between VRFs
• “Internet” VRF for Internet access and routing
between VRFs
• Global routing table used for routing
DMVPN tunnel packets VRF-A tunnels
VRF-B tunnels
VRF-A to VRF-B Path (optional)
Network Virtualization
MPLS over DMVPN – 2547oDMVPN
• MPLS VPN over DMVPN 2547oDMVPN
• Single DMVPN/mGRE tunnel on all routers
• Multiple Hub routers for redundancy and load
• MPLS configuration – routers are PEs
• Spoke to spoke via hub and direct shortcut
• MPLS labels via NHRP, ‘mpls nhrp’ (15.4(1)S, 15.4(2)T)
• Replaces ‘mpls ip’; No LDP
• Routing
• Global for routing DMVPN tunnel packets
• IGP for routing outside of DMVPN
• MP-BGP for routing over DMVPN
• Redistribute between IGP and BGP for over DMVPN
• Import/export routes between VRFs and Global (or
Internet VRF) VRF-A tunnels
VRF-B tunnels
• One routing neighbor per spoke VRF-A/B Tunnels
DMVPN designs for IWAN
• Multiple DMVPNs • PfRv3 interoperability
• One per physical transport network • Dynamic path selection
• Path diversity • Per application
• Load Balancing
• Separate failure domains
• Brownout circumvention
• Each Phase 3 DMVPN • Communicates with NHRP via RIB
• Single layer hub-and-spoke; • Triggers secondary spoke-spoke tunnels
hierarchical not currently supported
• Single Overlay Routing Domain
• Physical WAN interface in f-VRF
• Simplified operations and support
• Single Hub; Multi-Hub
• Simple ECMP load-balancing and
• PfRv3 Multi-NH and Multi-DC features
• MTT (Multiple Tunnel Termination) feature primary path provisioning
• Spoke-Spoke dynamic tunnels • EIGRP or BGP
• PfRv3 gets secondary path directly from RP
• Per-Tunnel QOS
Basic DMVPN Design for IWAN
Dual (multi)DMVPN Dual (multi)Hub
Internet DMVPN
MPLS DMVPN 192.168.100.0/24
192.168.20.0/24
Dynamic Spoke-to-spoke 192.168.10.0/24
.2 .1
.2 .1 Physical: 172.16.0.5
Tunnel0: 10.0.0.2
Physical: 172.16.0.1 Physical: 172.17.0.5 Loop0: 172.18.1.1
Tunnel0: 10.0.0.1 Tunnel1: 10.0.1.1 Physical: 172.17.0.1
Loop0: 172.18.0.1 Loop0: 172.18.0.2 Tunnel1: 10.0.1.2
Loop0: 172.18.1.2

Supported in IWAN 2.2

April 2017 (MTT)
MPLS Internet

Physical: (dynamic)
Tunnel0: 10.0.0.13
Physical: (dynamic) Tunnel1: 10.0.1.13
Tunnel0: 10.0.0.11 Loop0: 172.18.0.13
Tunnel1: 10.0.1.11
Loop0: 172.18.0.11

Spoke C
.1
Spoke A 192.168.3.0/24
.1
Physical: (dynamic) Physical: (dynamic)
192.168.1.0 /24 Tunnel0: 10.0.0.12 Tunnel1: 10.0.1.12
Spoke B1 .1 .2 Spoke B2

192.168.2.0 /24
VPN Selection
Use Case/ DMVPN GETVPN FlexVPN SSLVPN Easy VPN IPsec VPN
(mGRE, (CM, sVTI,
Solution p-pGRE)
(Tunnel-less) (dVTI, IKEv2) (TLS) (IKEv1)
p-pGRE)

Remote Access N-R N-S R R N-R N-R

Hub-Spoke (HS) R N-S R N-R N-R N-R

Non-Cisco Spoke

HS + Spoke-Spoke R R N-R N-S N-S N-S

IoT R N-R R R N-R N-R

IWAN R N-S N-S N-S N-S N-S

MPLS over xVPN R R N-R N-S N-S N-R

MPLS-o-DMVPN MPLS-o-mGRE MPLS-o-Flex MPLS-o-GRE
N-R = Not Recommended
R = Recommended N-S = Not Supported
DMVPN Details
Agenda
• DMVPN Design Overview
• DMVPN Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• DMVPN Network Segmentation
• VRF-lite over DMVPN
• MPLSoDMVPN
NHRP Message Types
• Registration
• Build base hub-and-spoke network for control and data traffic
(Phase 1 and 2 – single layer, Phase 3 – hierarchical)
• Resolution – Phase 2 and 3
• Get mapping to build dynamic spoke-spoke tunnels
• Traffic Indication (Redirect) – Phase 3
• Trigger resolution requests at previous GRE tunnel hop
• Purge
• Clear out stale dynamic NHRP mappings
• Error
• Signal error conditions
NHRP Main Functionality
• NHRP Registrations – Phase 1, 2 and 3
• Static NHRP mappings on spokes for Hub (NHS)
• Spoke (NHC) dynamically registers its VPN to NBMA address mapping with hub (NHS)
• NHRP Resolutions – Phase 2 and 3
• Dynamically resolve spoke to spoke VPN to NBMA mapping for spoke-spoke tunnels
• Phase 2 – NHC self triggers to send NHRP Resolution request
• Phase 3 – NHC triggered by first hop NHS to send NHRP Resolution request
• NHRP Resolution requests sent via hub-and-spoke or direct spoke-spoke path
• NHRP Resolution replies sent via direct spoke-spoke path
• NHRP Redirects (Traffic Indication) – Phase 3
• Data packets forwarded via NHS, which “hairpins” data packets back onto DMVPN
• NHS sends redirect message to “trigger” NHC to resolve direct spoke-spoke path
• Check for redirect configuration on egress, send redirect out ingress interface
NHRP Message Extension Types
• Responder Address Extension
• Address mapping for Responding node (Reply messages)
• Forward Transit NHS Record Extension
• List of NHSs that NHRP request message traversed (loop detection) – copied to reply message
• Reverse Transit NHS Record Extension
• List of NHSs that NHRP reply message traversed (usually empty – reply over direct tunnel)
• Authentication Extension
• NHRP Authentication (clear-text)
• NAT Address Extension*
• Address mapping: For peer (Registration request/reply); For self (Resolution request/reply)
• Cisco Vendor Extension* * Added to NHRP by Cisco
• NHRP Group name; Smart-spoke attributes (name; value);
MPLS Transport Labels; CMD or NSH header negotiation (PfRv3, TrustSec, ...)
NHRP Mapping Entries
• Static • Temporary (/32) (12.4(22)T)
• Both host (/32, /128) and network (/<x>) • Same as “Incomplete” mapping except that
mappings NBMA is set to Hub
• Dynamic • Data packets CEF-switched via NHS while
building spoke-spoke tunnels. (Phase 2)
• Registered (/32, /128)
• From NHRP Registration • Local (/32, /128 or /<x>)
• NAT – record both inside and outside address • Mapping for local network sent in an NHRP
• Learned (/32, /128 or /<x>) Resolution Reply
• From NHRP Resolution • Record which nodes were sent this mapping
• NAT – record both inside and outside address
• (no socket)
• Incomplete (/32, /128) • Not used to forward data packets
• Rate-limit NHRP Resolution Requests • Do not trigger IPsec encryption
• Data packets process-switched via NHS • Set on Local entries
while building spoke-spoke tunnels. (Phase 2)
NHRP Mapping Entries
Static 10.0.0.1/32 via 10.0.0.1, Tunnel0 created 01:20:10, never expire
Type: static, Flags: used
NBMA address: 172.17.0.9
Registered 10.0.0.19/32 via 10.0.0.19, Tunnel0 created 01:20:08, expire 00:05:51
Type: dynamic, Flags: unique registered used
NBMA address: 172.16.3.1
10.0.0.18/32 via 10.0.0.18, Tunnel0 created 00:16:09, expire 00:05:50
Type: dynamic, Flags: unique registered used
NBMA address: 172.18.0.2
NAT (Claimed NBMA address: 172.16.2.1)

10.0.0.18/32 via 10.0.0.18, Tunnel0 created 00:09:04, expire 00:00:22

Type: dynamic, Flags: router implicit
NBMA address: 172.18.0.2
(Claimed NBMA address: 172.16.2.1)

Resolution 192.168.23.0/24 via 10.0.0.19, Tunnel0 created 00:00:11, expire 00:05:48

Type: dynamic, Flags: router used
NBMA address: 172.16.3.1
Incomplete 10.0.0.45/32, Tunnel0 created 00:00:21, expire 00:02:43
Type: incomplete, Flags: negative
Cache hits: 2
Temporary 10.0.0.17/32 via 10.0.2.17, Tunnel0 created 00:00:09, expire 00:02:55
Type: dynamic, Flags: used temporary
NBMA address: 172.17.0.9
Local (no-socket) 192.168.15.0/24 via 10.0.0.11, Tunnel0 created 00:05:39, expire 00:05:50
Type: dynamic, Flags: router unique local
NBMA address: 172.16.1.1
(no-socket)
NHRP Mapping Flags
unique Mapping entry is unique, don’t allow overwrite with new NBMA

registered Mapping entry from an NHRP registration

authoritative Mapping entry can be used to answer NHRP resolution requests

used Mapping entry was used in last 60 seconds to forward data traffic

router Mapping entry for remote router

implicit Mapping entry from source information in NHRP resolution request packet

local Mapping entry for a local network, record remote requester

nat Remote peer supports the NHRP NAT extension
(added 12.4(6)T, hidden 12.4(15)T)

rib Routing Table entry created

(12.2(33)XNE, 15.2(1)T)

nho Next-Hop-Override Routing Table entry created

(12.2(33)XNE, 15.2(1)T)

nhop Explicit Next-Hop route out tunnel interface added to RIB/FIB

(15.3(2)S, 15.3(2)T)

nf Non-forwarding Entry (No Socket)

Agenda
• DMVPN Design Overview
• DMVPN Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• DMVPN Network Segmentation
• VRF-lite over DMVPN
• MPLSoDMVPN
Phase 1: Hub-and-Spoke – Features
• GRE, NHRP and IPsec configuration
• p-pGRE or mGRE on spokes; mGRE on hubs
• ISAKMP/IKEv2 Authentication
• Certificate (PKI), (Pairwise/Wildcard) Pre-shared Key (PSK)
• NHRP Registration
• Spoke has static NHRP mapping for Hubs
• Hub dynamically learns Spoke’s NHRP mapping
• Handles dynamically addressed spokes (DHCP, NAT , …)
• NAT detection support
• Check source protocol address in NHRP registration message with source IP on GRE/IP header
• Same  No NAT; Different  NAT
• Each spoke must get unique outside NAT IP address
• Does not handle spokes using the same outside NAT IP address (no ports on GRE)
• Can switch to IPsec tunnel mode, but then lose spoke-spoke tunnel capability
NHRP Registration
• Builds base hub-and-spoke network
• Hub-and-spoke data traffic
• Control traffic; NHRP, Routing protocol, IP multicast
• Phase 2 – Single layer hub-and-spoke
• Phase 3 – Hierarchical hub-and-spoke (tree).
• Next Hop Client (NHC) has static mapping for Next Hop Servers (NHSs)
• NHC dynamically registers own mapping with NHS
• Supports spokes with dynamic NBMA addresses or NAT
• Reports outside address of Hub (if Hub behind NAT)
• NHRP-group for per-Tunnel QoS (HS)
• IPv6: Includes both Unicast-Global and Link-local spoke mappings
• NHS registration reply gives liveliness of NHS
• Supplies outside NAT address of spoke (if spoke behind NAT)
• NHRP-group for per-Tunnel QoS (HS)
• IPv6: Includes link-local address hub mapping (needed by EIGRP; OSPF)
NHRP Registration
Before Building Spoke-Hub Tunnels
NHRP Registration 192.168.0.1/24

NHRP mapping
Physical: 172.17.0.1
Routing Table Tunnel0: 10.0.0.1 192.168.0.0/24  Conn.

Physical: (dynamic)
Tunnel0: 10.0.0.12
Physical: (dynamic)
Tunnel0: 10.0.0.11

Spoke A Spoke B 192.168.2.1/24

192.168.1.1/24

10.0.0.1  172.17.0.1 10.0.0.1  172.17.0.1

192.168.1.0/24  Conn.
192.168.2.0/24  Conn.

= Dynamic permanent IPsec tunnels

NHRP Registration
Building Spoke-Hub Tunnels
NHRP Registration 192.168.0.1/24
10.0.0.11  172.16.1.1
NHRP mapping 10.0.0.12  172.16.2.1
Physical: 172.17.0.1
Routing Table Tunnel0: 10.0.0.1 192.168.0.0/24  Conn.

Physical: 172.16.2.1
(dynamic)
Tunnel0: 10.0.0.12
Physical: 172.16.1.1
(dynamic)
Tunnel0: 10.0.0.11

Spoke A Spoke B 192.168.2.1/24

192.168.1.1/24

10.0.0.1  172.17.0.1 10.0.0.1  172.17.0.1

192.168.1.0/24  Conn.
192.168.2.0/24  Conn.

= Dynamic permanent IPsec tunnels

NHRP Registration
Routing Adjacency
Routing packet 192.168.0.1/24 10.0.0.11  172.16.1.1
10.0.0.12  172.16.2.1
NHRP mapping
Physical: 172.17.0.1 192.168.0.0/24  Conn.
Routing Table Tunnel0: 10.0.0.1 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12
192.168.0.0/16  Summ.

Physical: 172.16.2.1
Tunnel0: 10.0.0.12
Physical: 172.16.1.1
Tunnel0: 10.0.0.11

Spoke B 192.168.2.1/24
Spoke A
192.168.1.1/24

10.0.0.1  172.17.0.1 10.0.0.1  172.17.0.1

192.168.0.0/16  10.0.0.1 192.168.0.0/16  10.0.0.1

192.168.1.0/24  Conn. 192.168.2.0/24  Conn.

= Dynamic permanent IPsec tunnels

Hub-and-Spoke
Data Packet Forwarding
• Process-switching
• Routing table selects outgoing interface and IP next-hop
• NHRP looks up packet IP destination to select IP next-hop,
overriding IP next-hop from routing table.
• Could attempt to trigger spoke-spoke tunnel
• ‘tunnel destination …’  Can only send to hub
• ‘ip nhrp server-only’  Don’t send NHRP resolution request
• If no matching NHRP mapping then send to NHS (hub)
• CEF switching
• IP Next-hop from FIB table (Routing table)
• IP Next-hop  Hub  data packets send to Hub
• Adjacency will be complete so CEF switch packet to hub
• NHRP not involved
Agenda
• DMVPN Design Overview
• DMVPN Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• DMVPN Network Segmentation
• VRF-lite over DMVPN
• MPLSoDMVPN
Phase 2: Spoke-Spoke Features
• mGRE tunnel interface per DMVPN cloud
• On Hubs and Spokes
• Hubs must be inter-connected in a “Daisy chain” over same mGRE tunnel
• IKE authentication information (Certificates, Wildcard Pre-shared Keys)
• Spoke-spoke data traffic direct
• Reduced load on hub
• Reduced latency
• Single IPsec encrypt/decrypt
• Routing Protocol
• Still hub-and-spoke
• Hub cannot summarize spoke routes
• Routes on spokes must have IP next-hop of remote spoke (preserve next-hop)
Phase 3: Spoke-Spoke Features
• Increase scale • Spokes don’t need full routing tables
• Hierarchical network Layout • Can summarize routes at the hub
• Increase total number of spokes; • Reduces RIB space and RP load
same spoke to hub ratio
• Reduce RP load on hub
• Distribution hubs off load central hub
• 1000 spokes, 1 route per spoke;
• Manage local spoke-spoke tunnels
• hub advertises 1 route to 1000 spokes
• IP multicast and routing protocol
 1000 advertisements
• No hub daisy-chain • Phase 2 to Phase 3 migration
• NHS still interconnected; any pattern
• Build separate Phase 3 DMVPN
• Use RIB to forward NHRP packets
(can be on same hub and spokes)
• Reduces RP complexity and load
• Migrate spokes one by one from
• OSPF not limited to 2 hubs Phase 2 to Phase 3 DMVPN
• Network point-multipoint mode • Remove Phase 2 DMVPN
• Single OSPF area; No summarization
Phase 3 – Building Spoke-spoke Tunnels
• Originating spoke
• IP Data packet is forwarded out tunnel interface to destination via Hub (NHS)
• Hub (NHS)
• Receives and forwards data packet on tunnel interfaces with same NHRP Network-id.
• Check if ‘ip nhrp redirect’ configured on outbound tunnel interface
• If yes, trigger to send NHRP Redirect message to originating spoke out inbound tunnel
• Originating spoke
• Receives NHRP redirect message
• Sends NHRP Resolution Request for Data IP packet destination
• Destination spoke
• Receives NHRP Resolution Request
• Builds spoke-spoke tunnel
• Sends NHRP Resolution Reply over spoke-spoke tunnel
Phase 3 – NHRP Redirects
Data packet 192.168.0.1/24 10.0.0.11  172.16.1.1
NHRP Redirect 10.0.0.12  172.16.2.1
NHRP Resolution
Physical: 172.17.0.1 192.168.0.0/24  Conn.
NHRP mapping Tunnel0: 10.0.0.1 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12
CEF FIB Table
10.0.0.11  172.16.1.1
CEF Adjacency 10.0.0.12  172.16.2.1
Physical: 172.16.2.1
(dynamic)
Physical: 172.16.1.1
(dynamic) Tunnel0: 10.0.0.12
Tunnel0: 10.0.0.11

Spoke A Spoke B 192.168.2.1/24

192.168.1.1/24
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1
192.168.2.1  ???
192.168.2.0/24  Conn.
192.168.1.0/24  Conn. 192.168.0.0/16  10.0.0.1
192.168.0.0/16  10.0.0.1
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1
Phase 3 – NHRP Redirect Processing
• Sender
• Insert (GRE IP header source, packet destination IP address) in NHRP redirect table –
used to rate-limit NHRP redirect messages ‘show ip nhrp redirect’
• Send NHRP redirect to GRE/IP header source (previous tunnel hop out inbound tunnel)
• Time out rate-limit entries from the NHRP redirect table
• Receiver
• Check data IP source address from data IP header in redirect
• If routing to the IP source is out:
• A GRE tunnel interface with the same NHRP Network-id
• then drop redirect
• Another interface, ‘ip nhrp shortcut’ is configured on inbound tunnel and
the IP destination is permitted by ‘ip nhrp interest ACL’ (if configured)
• then trigger an NHRP resolution request to data IP destination from data IP header in redirect
• otherwise drop redirect
Phase 3 – NHRP Resolution Request
Data packet 192.168.0.1/24 10.0.0.11  172.16.1.1
NHRP Redirect 10.0.0.12  172.16.2.1
NHRP Resolution
Physical: 172.17.0.1 192.168.0.0/24  Conn.
NHRP mapping Tunnel0: 10.0.0.1 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12
CEF FIB Table
10.0.0.11  172.16.1.1
CEF Adjacency 10.0.0.12  172.16.2.1
Physical: 172.16.2.1
(dynamic)
Physical: 172.16.1.1
(dynamic) Tunnel0: 10.0.0.12
Tunnel0: 10.0.0.11

Spoke A Spoke B 192.168.2.1/24

192.168.1.1/24
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1 10.0.0.11  172.16.1.1
192.168.2.1  ???
192.168.2.0/24  Conn.
192.168.1.0/24  Conn. 192.168.0.0/16  10.0.0.1
192.168.0.0/16  10.0.0.1
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1 10.0.0.11  172.16.1.1
Phase 3 – NHRP Resolution Processing
• Spoke (NHC) routing table has Hub (NHS) as IP next-hop for networks behind
remote Spoke
• If routing table has IP next-hop of remote spoke then process as in Phase 2
• Data packets are forwarded (CEF-switched) via routed path
• Redirect message sent by every tunnel hop on routed path
• Redirect for data packet triggers resolution request only on source spoke
• Send resolution request for IP destination from data packet header in redirect
• Resolution requests forwarded via routed path
• Resolution replies forwarded over direct tunnel
• Direct tunnel initiated from remote  local spoke
• Forward data packets over direct tunnel after receipt of resolution reply.
Phase 3 – NHRP Resolution Reply
Data packet 192.168.0.1/24 10.0.0.11  172.16.1.1
NHRP Redirect 10.0.0.12  172.16.2.1
NHRP Resolution
Physical: 172.17.0.1 192.168.0.0/24  Conn.
NHRP mapping Tunnel0: 10.0.0.1 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12
CEF FIB Table
10.0.0.11  172.16.1.1
CEF Adjacency 10.0.0.12  172.16.2.1
Physical: 172.16.2.1
(dynamic)
Physical: 172.16.1.1
(dynamic) Tunnel0: 10.0.0.12
Tunnel0: 10.0.0.11

Spoke A Spoke B 192.168.2.1/24

192.168.1.1/24

10.0.0.1  172.17.0.1 10.0.0.1  172.17.0.1

10.0.0.12  172.16.2.1 10.0.0.11  172.16.1.1
192.168.2.1  ???
192.168.2.0/24  172.16.2.1 192.168.1.0/24  172.16.1.1
192.168.1.0/24  Conn. 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12 192.168.2.0/24  Conn.
192.168.0.0/16  10.0.0.1 192.168.0.0/16  10.0.0.1
10.0.0.1  172.17.0.1 10.0.0.1  172.17.0.1
10.0.0.12  172.16.2.1 10.0.0.11  172.16.1.1
Phase 3 – Refresh or Remove Dynamic Mappings
• Dynamic NHRP mapping entries have finite lifetime
• Controlled by ‘ip nhrp holdtime …’ on source of mapping (remote spoke)
• Two types of mapping entries
• Master entry – Remote Spoke Tunnel IP address
• Child entries – Remote Network address(es) behind remote-spoke

• Background process checks mapping entries every 60 seconds

• Master entry: Timing out*  mark CEF adjacency stale * Expire timer < 120 seconds
• If CEF adjacency is then used
• Refresh Master entry and for each child entry that is also timing out*  queue for immediate refresh

• Refreshing entries
• Send another Resolution request and reply
• Resolution request/reply sent over direct tunnel
• If entry expires it is removed
• If using IPsec and last entry using this NBMA address
• Trigger IPsec to remove IPsec and ISAKMP/IKEv2 SAs
NHRP Purge Messages
• Used to clear invalid NHRP mapping information from the network
• NHRP “local”, “(no socket)” mapping entries
• Created when sending an NHRP resolution reply
• Copy of mapping information sent in reply
• Entry tied to corresponding entry in routing table
• Keeps list of nodes where resolution reply was sent – ‘show ip nhrp detail’
• If routing table changes so that local mapping entry is no longer valid
• Purge message is sent to each NHRP node in list
• NHRP nodes clear that mapping from their table
• Purge messages forwarded over direct tunnel if available, otherwise sent via routed path
Phase 3 – NHRP and Routing Table
Data Packet Forwarding
• When NHRP resolution is received
• Insert mapping information in mapping table replacing Incomplete/Temporary mapping
• Insert NHRP routing entry in Routing Table (RT)
• NHRP NET/Mask is longer (more specific) than RT Net/Mask
• Add new route owned by NHRP (Type = H)
• NHRP Net/Mask is equal to RT Net/Mask
• Add Override Alternate Next-hop (% flag)
• Route still owned by original owner
• NHRP Net/Mask is shorter (less specific) than RT Net/Mask
• Increase (make more specific) NHRP mask to = RT Mask
• Add Override Alternate Next-hop (% flag)
• Route still owned by original owner
• Insert connected route for tunnel next-hop of NHRP parent mapping (nhop flag)
Phase 3 – NHRP and RT
Routing Table
#show ip route
H 192.168.11.0/24 [250/1] via 10.0.1.11, 00:01:02
D % 192.168.128.0/24 [90/3200000] via 10.0.2.16, 00:50:56, Tunnel0
NHRP
Routes #show ip route next-hop-override | section H|%
H 192.168.11.0/24 [250/1] via 10.0.1.11, 00:01:02
D % 192.168.128.0/24 [90/3200000] via 10.0.2.16, 00:50:56, Tunnel0
[NHO][90/1] via 10.0.0.1, 00:00:40, Tunnel0

Routing entry for 192.168.11.0/24

Known via "nhrp", distance 250, metric 1
EIGRP Last update from 10.0.1.11 00:05:29 ago
Routing Descriptor Blocks:
Routes * 10.0.1.11, from 10.0.1.11, 00:05:29 ago
Route metric is 1, traffic share count is 1

Routing entry for 192.168.128.0/24

Known via "eigrp 1", distance 90, metric 3200000, type internal
Redistributing via eigrp 1
Next-Hop- Last update from 10.0.2.16 on Tunnel0, 00:43:44 ago
Routing Descriptor Blocks:
Override * 10.0.2.16, from 10.0.2.16, 00:43:44 ago, via Tunnel0
Entries Route metric is 3200000, traffic share count is 1
…
[NHO]10.0.0.1, from 10.0.0.1, 00:05:57 ago, via Tunnel0
Route metric is 1, traffic share count is 1
…
Phase 3 – NHRP and Routing Table
NHRP Parent Route Rules
• Insert NHRP routing entry in Routing Table (RIB)
• NHRP follows the rules outlined above for inserting RIB routes
BUT
• NHRP also makes sure to not contradict routing protocol routes
• Check for “parent” route
• Parent – next route with mask prefix less than or equal to NHRP route
• If Parent route via:
• same tunnel interface  add NHRP route
• another interface  do not add NHRP route
• After adding NHRP route  Watch Parent route
• If Parent route changed or removed (attach to next parent route)
• If new Parent route now via:
• same tunnel interface  leave NHRP route
• another interface  remove NHRP route
• Override with ‘no nhrp route-watch’ – can misroute or black-hole traffic
DMVPN MTT
(Multiple Tunnel Termination)

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Multiple Tunnel Termination (MTT)
• Issue
• Multiple DMVPN clouds (IWAN Transports) terminating on the same Hub
• Spoke-spoke tunnels don’t always get built
• Data packets CEF switched between DMVPNs
• No NHRP Redirect sent  No Spoke-spoke tunnel
• NHRP Resolution (NHRP) switched between DMVPNs
• Hub answers NHRP resolution  No Spoke-spoke tunnel
• Spoke-spoke traffic continues to traverse the hub
• Solution
• Forward NHRP and Data traffic out the same DMVPN on which it arrived
• Install regular and secondary routes into RIB
• Part 1: NHRP traffic; controlled by NHRP control plane
• Part 2: Data plane traffic forwarding; controlled by FIB/CEF (future)
• Spoke-spoke tunnel
DMVPN without MTT
Routing preferred via MPLS
192.168.1.0/24 (10)  10.0.0.11 Hub Router
MPLS
192.168.2.0/24 (10)  10.0.0.12 INET
Global 10.0.0.0/24 M
P
DMVPN Tunnel0
• CEF MPLSMPLS L
S
• Send NHRP Redirect MPLS
L
• Forward NHRP Resolution
A CEF
Request MPLSMPLS
N
• Spoke-spoke
I
N
DMVPN Tunnel1
10.0.1.0/24
E
T
NHRP
DMVPN without MTT
Routing preferred via MPLS (cont)
192.168.1.0/24 (10)  10.0.0.11 Hub Router
MPLS
192.168.2.0/24 (10)  10.0.0.12 INET
Global 10.0.0.0/24 M
P
DMVPN Tunnel0
• CEF MPLSMPLS L
S
• Send NHRP Redirect MPLS
L
• Forward NHRP Resolution
A CEF
Request MPLSMPLS
N
• Spoke-spoke
I
• CEF INETMPLS DMVPN Tunnel1
N
E
• Don’t send NHRP Redirect 10.0.1.0/24
T
• No spoke-spoke NHRP
DMVPN without MTT
ECMP routing via MPLS and INET
192.168.1.0/24 (10)  10.0.0.11 Hub Router
(10)  10.0.1.11 MPLS
192.168.2.0/24 (10)  10.0.0.12 INET
(10)  10.0.1.12 Global 10.0.0.0/24 M
P
DMVPN Tunnel0
• CEF inbound MPLS L
S
• Forward: 50% MPLS
• Send NHRP Redirect MPLS L
• NHRP Resolution Request A CEF
• 50% Forward MPLS (spoke-spoke) N
• 50% Hub answers (no spoke-spoke) I
N
DMVPN Tunnel1
10.0.1.0/24
E
T
NHRP
DMVPN without MTT
ECMP routing via MPLS and INET (cont)
192.168.1.0/24 (10)  10.0.0.11 Hub Router
(10)  10.0.1.11 MPLS
192.168.2.0/24 (10)  10.0.0.12 INET
(10)  10.0.1.12 Global 10.0.0.0/24 M
P
DMVPN Tunnel0
• CEF inbound MPLS L
S
• Forward: 50% MPLS
• Send NHRP Redirect MPLS L
• NHRP Resolution Request A CEF
• 50% Forward MPLS (spoke-spoke) N
• 50% Hub answers (no spoke-spoke) I
• Forward: 50% INET DMVPN Tunnel1
N
• Don’t send NHRP Redirect 10.0.1.0/24
E
T
• No spoke-spoke NHRP
• CEF inbound INET (similar)
Kinds of RIB Paths
• Regular next-hops/paths
• Most common kind of paths, often equal cost but could be unequal cost.
• Governed by 'maximum-paths <n>' (up to 32)
• Installed in the RIB and passed to FIB/CEF for immediate use
• Repair next-hop/path
• Special paths that are used for IP FRR, BGP PIC, etc.
• Only ONE repair path (per-prefix) for one or more regular paths.
• Installed in the RIB and passed to FIB/CEF, but NOT USED as long as
one or more regular next hops are active.
• Secondary next-hops/paths
• Special loop free paths that are typically inferior to regular and repair paths.
• Governed by 'maximum-secondary-paths <n>' (up to 32; default 0).
• Installed in RIB but not passed to FIB
 16.3.2, 16.4.1, 15.6(3)M2, 15.5(3)S5
DMVPN with MTT (Part 1)
Routing via MPLS and INET (ECMP or secondary)
192.168.1.0/24 (10)  10.0.0.11 Hub Router
(10)  10.0.1.11 MPLS
192.168.2.0/24 (10)  10.0.0.12 INET
(10)  10.0.1.12 Global 10.0.0.0/24 M
P
DMVPN Tunnel0
• CEF inbound MPLS L
S
• Forward: 50% MPLS
• Send NHRP Redirect MPLS L
• NHRP Resolution Request A CEF
• 100% Forward MPLS (spoke-spoke) N
• Forward: 50% INET I
• Don’t send NHRP Redirect N
DMVPN Tunnel1
E
• No spoke-spoke 10.0.1.0/24
T
NHRP
• CEF inbound INET (similar)
 On Roadmap
DMVPN with MTT (Part 2)
Routing via MPLS and INET (ECMP or secondary)
192.168.1.0/24 (10)  10.0.0.11 Hub Router
[SEC] (20)  10.0.1.11 MPLS
192.168.2.0/24 (10)  10.0.0.12 INET
[SEC] (20)  10.0.1.12 Global 10.0.0.0/24 M
P
DMVPN Tunnel0
• CEF inbound MPLS L
S
• CEF Forward: 100% MPLS
• Send NHRP Redirect MPLS L
• NHRP Resolution Request A CEF
• 100% Forward MPLS (spoke-spoke) N
I
N
DMVPN Tunnel1
10.0.1.0/24
E
T
NHRP
DMVPN with MTT (Part 2)
Routing via MPLS and INET (ECMP or secondary)
192.168.1.0/24 (10)  10.0.0.11 Hub Router
[SEC] (20)  10.0.1.11 MPLS
192.168.2.0/24 (10)  10.0.0.12 INET
[SEC] (20)  10.0.1.12 Global 10.0.0.0/24 M
P
DMVPN Tunnel0
• CEF inbound MPLS L
S
L
• CEF inbound INET A CEF
• CEF Forward 100% INET N
• Send NHRP Redirect INET I
• NHRP Resolution Request N
DMVPN Tunnel1
E
• 100% Forward INET (spoke-spoke) 10.0.1.0/24
T
NHRP
DMVPN with Multiple Tunnel Termination (MTT)
(16.3.2, 16.4.1)
• Configure Routing Protocol to insert secondary routes BGP:
maximum-secondary-paths [eibgp|ibgp] <x>
• Mandatory on Hub; Recommended on Spokes (IOS/XE*) EIGRP:
topology base
• Part 1: maximum-secondary-paths <x>

• DMVPN – ECMP routes over both tunnels

• Statistical per flow (src-IP, dst-IP) whether spoke-spoke tunnel is triggered
• All flows use spoke-spoke tunnel when built
• IWAN 2.2
• Data traffic over primary (preferred) route – spoke-spoke triggered by:
• Data packets for primary tunnel; PfR probes for secondary tunnel
• Data traffic over secondary (non-preferred) route – spoke-spoke triggered by:
• PfR probes for both tunnels; up to 10 second delay for secondary tunnel
• Part 2 (future)
• ECMP or Preferred route over one tunnel (all routes (regular, secondary) in RIB)
• Data traffic and/or PfR Probes trigger spoke-spoke tunnels over both tunnels
DMVPN Network Segmentation
Agenda
• DMVPN Design Overview
• DMVPN Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• DMVPN Network Segmentation
• VRF-lite over DMVPN
• MPLSoDMVPN
Network Segmentation over DMVPN
Two main techniques
• VRF-lite over DMVPN
• Separate DMVPN cloud (mGRE tunnel) per VRF
• Single IPsec session used by all GRE tunnels between same two peers
• Separate Routing Protocol neighbor per spoke and per VRF
• Tunnel Key used to separate packets over DMVPN and transport network
• Must manually match tunnel keys to VRFs the same way on all nodes
• 2547oDMVPN (MPLS over DMVPN)
• MPLS VPN running over single DMVPN cloud (mGRE tunnel)
• Single IPsec session and single GRE tunnel between same two peers
• Single MP-BGP routing neighbor per spoke regardless of number of VRFs
• MPLS VPN tag used to separate packets over the DMVPN and transport network
• MP-BGP automatically matches VPN tags to VRFs on all nodes
Network Segmentation over DMVPN
• On transport network both techniques look similar
• Single IPsec session  Cannot differentiate between style being used
• Encapsulated tunnel packets are similar (effectively MPLS VPN tag == Tunnel Key)

Encrypted

Outer ESP GRE Tunnel Inner ESP ESP

VRF-Lite IP Header
Header NextIP Key IP
Data Trailer Auth
NextESP NextESP

Outer ESP GRE MPLS Inner ESP ESP

MPLS IP Header
Header NextMPLS VPN * IP
Data Trailer Auth
NextESP NextESP

* Can also have MPLS transport tags

Network Segmentation over DMVPN
• On DMVPN nodes the two techniques present to the router differently
• VRF-lite
• Separate mGRE tunnel interface per VRF – ‘vrf forwarding <vrf>’
• Tunnels match with per VRF LAN interfaces – ‘vrf forwarding <vrf>’
• Separate Routing Protocol neighbor per spoke and per VRF
• Can use MP-BGP on hub to “leak” routes between VRFs for cross VRF forwarding

• MPLS
• Single mGRE tunnel interface for all VRFs – ‘mpls nhrp’
• MPLS maps per VRF LAN interfaces (‘vrf forwarding <vrf>’) to/from single tunnel
• Single MP-BGP routing neighbor per spoke regardless of number of VRFs
• MPLS just on the DMVPN or part of a larger MPLS network
• Hub is an MPLS P/PE
• Spokes can be an MPLS PE or P/PE
• VRF RD and RT tags must match on all DMVPN PE routers
Network Segmentation over DMVPN
VRFs on the DMVPN
• VRF definition on the DMVPN nodes
• Define VRFs used on that node
• VRF-lite:
• Defined VRFs will match the configured VRFs on mGRE tunnels and LAN interfaces
• MPLS:
• Hubs: Must define all VRFs, even if no local interface uses that VRF*
• Spokes: Must define all VRFs used at spoke site, even if no local interface uses that VRF*

• Add a new VRF to the network * Extending an existing

MPLS over DMVPN
• VRF-lite:
• On Hubs and Spokes – if already defined on Hub only do this on the Spoke
• Add new mGRE tunnel for VRF with matching Tunnel Key
• Add Routing Protocol address-family for new VRF
• MPLS:
• On Hubs and Spokes – if already defined on Hub only need this on the Spoke
• Redistribute LAN VRF routes in/out of MP-BGP VRF address family
Configuration
Crypto and Physical Interfaces
crypto ikev2 keyring DMVPN
vrf-name = Yellow,
peer DMVPN
address 0.0.0.0 0.0.0.0 Red,
pre-shared-key cisco123 Green
!
crypto ikev2 profile DMVPN
match fvrf Outside
match identity remote address 0.0.0.0 x = Hub (0), Spoke (1,2,3)
authentication local pre-share
authentication remote pre-share y = BU# (Yellow = 0,
keyring local DMVPN Red = 2,
! Green = 4)
crypto ipsec transform-set DMVPN esp-aes esp-sha-hmac
mode transport z = Hub (17.0), Spoke (16.(1,2,3))
!
crypto ipsec profile DMVPN
set transform-set DMVPN
set ikev2-profile DMVPN
!
interface Ethernet<y>/0 ! <inside LAN/VRF interface(s)>
vrf forwarding <vrf-name> Configuration Block
ip address 192.168.<x><y>.1 255.255.255.0 repeated per VRF
!
interface Ethernet3/0 ! <Internet access, on hub only>
vrf forwarding Internet Support Internet
ip address 192.168.254.1 255.255.255.0 access for all VRFs
!
interface Serial<#>/0 ! <outside (public) interface>
vrf forwarding Outside fVRF to separate out
ip address 172.<z>.1 255.255.255.252 transport routing
!
ip route vrf Outside 0.0.0.0 0.0.0.0 172.<z>.2
Configuration
Basic VRF and Tunnel
vrf definition <vrf-name> ! <spokes and hub> x = rd# (Yellow = 1,
rd <x>:<x> ! <repeated per VRF> Red = 2,
route-target export <x>:<x> Green = 3)
route-target import <x>:<x>
!
vrf definition Outside vrf-name = Yellow,
! Red,
vrf definition Internet ! <hub only> Green
rd 10:10
route-target export 10:10
route-target import 10:10
route-target import 1:1
route-target import 2:2 ! <import routes into Internet VRF>
route-target import 3:3
!
interface Tunnel<y> ! <spokes and hubs> y = BU# (Yellow = 0,
bandwidth 1000 ! <repeated per VRF for VRF-lite> Red = 2,
ip address 10.0.<y>.1 255.255.255.0 Green = 4)
ip mtu 1400 ! <VRF-lite solution only>
ip nhrp authentication <vrf-name> or MPLS ! <MPLS (single instance) for MPLS
ip nhrp map multicast dynamic ! <hub only>
ip nhrp network-id 10<y>
ip nhrp holdtime 600
ip nhrp redirect
ip tcp adjust-mss 1360 ! <VRF-lite solution only>
tunnel source Serial<#>/0
tunnel mode gre multipoint
tunnel key 10<y>
tunnel vrf Outside
tunnel protection ipsec profile DMVPN shared
Configuration
EIGRP and BGP
vrf-name = Yellow,
router eigrp 1 Red,
no auto-summary Green
!
address-family ipv4 vrf <vrf-name>
redistribute bgp 1
network <LAN-network> Configuration Block
default-metric 1000 100 255 1 1500 repeated per VRF
no auto-summary
autonomous-system 1
exit-address-family
!
...
!
router bgp 1 ! <VRF-lite solution on hub only>
no synchronization
bgp log-neighbor-changes
no auto-summary
!
address-family ipv4 vrf [<vrf-name>, Internet]
redistribute connected Configuration Block
redistribute eigrp 1 repeated per VRF
no synchronization
exit-address-family
!
VRF-lite over DMVPN
VRF-lite – Separate DMVPNs
n = number of spokes
v = average number of VRFs per spoke
m = number of hubs for spoke-hub tunnels
• Separate mGRE tunnel per VRF (BU)
• Dynamic spoke-spoke tunnels per DMVPN (VRF)
• Hub routers have all VRF (BU) DMVPNs; Spokes only those they need
• Multiple Hub routers for load and redundancy
• Load:
• (m) Hubs to support n×v hub-spoke tunnels (each hub limited by number of RP neighbors)
• Redundancy:
• Manually map (2×(m) Hubs) or dynamically map ((IOS SLB) (m)+1 hubs)
• Routing
• EIGRP, BGP or OSPF over DMVPN and LAN
• Forwarding between VRFs (optional)
• MP-BGP can be used on hub to leak routes between VRFs
• Use a router or firewall behind the hub
VRF-lite – Separate DMVPNs
Example uses:
• EIGRP for routing protocol outside of and over DMVPNs
• An ipv4 address-family per VRF
• MP-BGP only on the hub
• Import/export routes between VRFs and “Internet” VRF
• An ipv4 address-family per VRF
• Internet access via Hub router for all VRFs – no split-tunneling
• IP addresses spaces unique across all VRFs
• Routing used to forward return packets from Internet back to correct VRF
• “Outside” f-VRF used for forwarding tunnel packets on WAN (transport)
• Separation of Transport address space from Overlay (VRF) address space
Separate DMVPNs VRF-lite
Logical Topology
.254.x .1 .1 .0.x .100.1 192.168.x.y/24
.2 .2
.1 .2.x .102.1
Internet .2
Hub1
.1 .4.x .104.1
.2
.1,.1,.1
Interface Interface
Tunnel 0 Tunnel 2
Interface DMVPN
DMVPN Tunnel 1
10.0.0.0/24 10.0.4.0/24
Tunnel Key Tunnel Key
100 DMVPN 104
10.0.2.0/24
Tunnel Key
.11 102
.13,.13

Spoke1
192.168.x.y/24 Spoke3
192.168.x.y/24
.12,.12,.12
.10.x .20.x .120.1
.110.1 .1 .2 .32.x
.1 .2 .132.1
Spoke2 .22.x .122.1 .2 .1
192.168.x.y/24 .1 .2 .34.x
.24.x .134.1
.124.1 .2 .1
.1 .2
Separate DMVPNs – VRF-lite
Spoke-spoke
Bus. #1, #2, #3

WAN
LAN
Spoke 1 Bus. #1 Separate
mGRE tunnels

LAN Internet
Bus. #1 “global”

WAN Interface

WAN
LAN
Spoke 2 Bus. #2
Hub
LAN Server

Interface
Bus. #3 LANs

LAN WAN Int.

Bus. #2
Spoke 3
LAN Hub-spoke VRF-lite
Bus. #3 Bus. #1, #2, #3
Separate DMVPNs – VRF-lite
Hub Configuration – BU VRFs
vrf definition Yellow vrf definition Red vrf definition Green
Import default rd 1:1 rd 2:2 rd 3:3
route-target export 1:1 route-target export 2:2 route-target export 3:3
from Internet route-target import 1:1 route-target import 2:2 route-target import 3:3
route-target import 10:10 route-target import 10:10 route-target import 10:10
! ! !
interface Tunnel0 interface Tunnel2 interface Tunnel4
vrf forwarding Yellow vrf forwarding Red vrf forwarding Green
EIGRP routing ip address 10.0.0.1 255.255.255.0 ip address 10.0.2.1 255.255.255.0 ip address 10.0.4.1 255.255.255.0
ip nhrp network-id 100 ip nhrp network-id 102 ip nhrp network-id 104
no split-horizon ip nhrp authentication Yellow ip nhrp authentication Red ip nhrp authentication Green
no ip split-horizon eigrp 1 no ip split-horizon eigrp 1 no ip split-horizon eigrp 1
tunnel key 100 tunnel key 102 tunnel key 104
! ! !
interface Ethernet0/0 interface Ethernet1/0 interface Ethernet2/0
VRF vrf forwarding Yellow vrf forwarding Red vrf forwarding Green
ip address 192.168.0.1 255.255.255.0 ip address 192.168.2.1 255.255.255.0 ip address 192.168.4.1 255.255.255.0
! ! !
router eigrp 1 router eigrp 1 router eigrp 1
! ! !
address-family ipv4 vrf Yellow address-family ipv4 vrf Red address-family ipv4 vrf Green
EIGRP routing redistribute bgp 1 redistribute bgp 1 redistribute bgp 1
over tunnel, LAN network 10.0.0.0 0.0.0.255 network 10.0.2.0 0.0.0.255 network 10.0.4.0 0.0.0.255
network 192.168.0.0 network 192.168.2.0 network 192.168.4.0
exit-address-family exit-address-family exit-address-family
! ! !
router bgp 1 router bgp 1 router bgp 1
BGP routing ! ! !
for route address-family ipv4 vrf Yellow address-family ipv4 vrf Red address-family ipv4 vrf Green
import/export redistribute connected redistribute connected redistribute connected
redistribute eigrp 1 redistribute eigrp 1 redistribute eigrp 1
exit-address-family exit-address-family exit-address-family
Separate DMVPNs – VRF-lite
Hub Configuration – Internet VRF
vrf definition Internet
rd 10:10
route-target export 10:10
route-target import 10:10
route-target import 1:1
Import VRF routes route-target import 2:2
into Internet route-target import 3:3
!
interface Ethernet3/0
vrf forwarding Internet
ip address 192.168.254.1 255.255.255.0
!
router eigrp 1
!
EIGRP routing address-family ipv4 vrf Internet
to Internet redistribute bgp 1
network 192.168.254.0
autonomous-system 1
exit-address-family
!
router bgp 1
!
address-family ipv4 vrf Internet
Import Internet redistribute connected
routes (default) redistribute eigrp 1
to VRFs default-information originate
exit-address-family
Separate DMVPNs – VRF-lite
Spoke 2 – Configuration

VRF config vrf definition Yellow vrf definition Red vrf definition Green
rd 1:1 rd 2:2 rd 3:3
route-target export 1:1 route-target export 2:2 route-target export 3:3
EIGRP routing route-target import 1:1 route-target import 2:2 route-target import 3:3
over DMVPN ! ! !
interface Tunnel0 interface Tunnel1 interface Tunnel2
No BGP config vrf forwarding Yellow vrf forwarding Red vrf forwarding Green
ip address 10.0.0.12 255.255.255.0 ip address 10.0.2.12 255.255.255.0 ip address 10.0.4.12 255.255.255.0
ip nhrp authentication Yellow ip nhrp authentication Red ip nhrp authentication Green
ip nhrp network-id 100 ip nhrp network-id 102 ip nhrp network-id 104
ip nhrp nhs 10.0.0.1 nbma 172.17.0.1
ipmulticast
nhrp nhs 10.0.2.1 nbma 172.17.0.1
ipmulticast
nhrp nhs 10.0.4.1 nbma 172.17.0.1 multicast
tunnel key 100 tunnel key 102 tunnel key 104
! ! !
router eigrp 1 router eigrp 1 router eigrp 1
no auto-summary no auto-summary no auto-summary
! ! !
address-family ipv4 vrf Yellow address-family ipv4 vrf Red address-family ipv4 vrf Green
network 10.0.0.0 0.0.0.255 network 10.0.2.0 0.0.0.255 network 10.0.4.0 0.0.0.255
network 192.168.20.0 network 192.168.22.0 network 192.168.24.0
autonomous-system 1 autonomous-system 1 autonomous-system 1
exit-address-family exit-address-family exit-address-family
! ! !
interface Ethernet0/0 interface Ethernet1/0 interface Ethernet2/0
vrf forwarding Yellow vrf forwarding Red vrf forwarding Green
ip address 192.168.20.1 255.255.255.0
ip address 192.168.22.1 255.255.255.0
ip address 192.168.24.1 255.255.255.0
Separate DMVPNs – VRF-lite
Routing Tables – Hub

vrf Outside S*
C
0.0.0.0/0 [1/0] via 172.17.0.2
172.17.0.0/30 is directly connected, Serial4/0

D*EX 0.0.0.0/0 [170/281600] via 192.168.254.2, 3w6d, Ethernet3/0

vrf Internet B 10.0.0.0/24 is directly connected, 3w6d, Tunnel0
B 192.168.0.0/24 is directly connected, 3w6d, Ethernet0/0
B 192.168.10.0/24 [20/3865600] via 10.0.0.11 (Yellow), 3w6d, Tunnel0
B 192.168.20.0/24 [20/3865600] via 10.0.0.12 (Yellow), 3w6d, Tunnel0
B 192.168.100.0/24 [20/307200] via 192.168.0.2 (Yellow), 3w6d, Ethernet0/0
B 192.168.110.0/24 [20/3891200] via 10.0.0.11 (Yellow), 3w6d, Tunnel0
B 192.168.120.0/24 [20/3891200] via 10.0.0.12 (Yellow), 3w6d, Tunnel0
B 10.0.2.0/24 is directly connected, 00:01:22, Tunnel2
B 192.168.2.0/24 is directly connected, 3w6d, Ethernet1/0
B 192.168.22.0/24 [20/3865600] via 10.0.2.12 (Red), 00:01:25, Tunnel2
B 192.168.32.0/24 [20/3865600] via 10.0.2.13 (Red), 00:01:13, Tunnel2
B 192.168.102.0/24 [20/307200] via 192.168.2.2 (Red), 3w6d, Ethernet1/0
B 192.168.122.0/24 [20/3891200] via 10.0.2.12 (Red), 00:01:25, Tunnel2
B 192.168.132.0/24 [20/3891200] via 10.0.2.13 (Red), 00:01:13, Tunnel2
B 10.0.4.0/24 is directly connected, 00:02:05, Tunnel4
B 192.168.4.0/24 is directly connected, 3w6d, Ethernet2/0
B 192.168.24.0/24 [20/3865600] via 10.0.4.12 (Green), 00:01:57, Tunnel4
B 192.168.34.0/24 [20/3865600] via 10.0.4.13 (Green), 00:01:48, Tunnel4
B 192.168.104.0/24 [20/307200] via 192.168.4.2 (Green), 3w6d, Ethernet2/0
B 192.168.124.0/24 [20/3891200] via 10.0.4.12 (Green), 00:01:57, Tunnel4
B 192.168.134.0/24 [20/3891200] via 10.0.4.13 (Green), 00:01:48, Tunnel4
C 192.168.254.0/24 is directly connected, Ethernet3/0
Separate DMVPNs – VRF-lite
Routing Tables – Hub (VRFs)
C 10.0.0.0/24 is directly connected, Tunnel0
vrf Yellow C 192.168.0.0/24 is directly connected, Ethernet0/0
D 192.168.100.0/24 [90/307200] via 192.168.0.2, 4w0d, Ethernet0/0
D 192.168.10.0/24 [90/3865600] via 10.0.0.11, 4w0d, Tunnel0
D 192.168.20.0/24 [90/3865600] via 10.0.0.12, 4w0d, Tunnel0
D 192.168.110.0/24 [90/3891200] via 10.0.0.11, 4w0d, Tunnel0
D 192.168.120.0/24 [90/3891200] via 10.0.0.12, 4w0d, Tunnel0
B 192.168.254.0/24 is directly connected, 4w0d, Ethernet3/0
B* 0.0.0.0/0 [20/281600] via 192.168.254.2 (Internet), 4w0d, Ethernet3/0

C 10.0.2.0/24 is directly connected, Tunnel2

vrf Red C 192.168.2.0/24 is directly connected, Ethernet1/0
D 192.168.102.0/24 [90/307200] via 192.168.2.2, 00:12:10, Ethernet1/0
D 192.168.22.0/24 [90/3865600] via 10.0.2.12, 00:09:19, Tunnel2
D 192.168.32.0/24 [90/3865600] via 10.0.2.13, 00:09:07, Tunnel2
D 192.168.122.0/24 [90/3891200] via 10.0.2.12, 00:09:19, Tunnel2
D 192.168.132.0/24 [90/3891200] via 10.0.2.13, 00:09:07, Tunnel2
B 192.168.254.0/24 is directly connected, 4w0d, Ethernet3/0
B* 0.0.0.0/0 [20/281600] via 192.168.254.2 (Internet), 4w0d, Ethernet3/0

C 10.0.4.0/24 is directly connected, Tunnel4

vrf Green C 192.168.4.0/24 is directly connected, Ethernet2/0
D 192.168.104.0/24 [90/307200] via 192.168.4.2, 00:14:09, Ethernet2/0
D 192.168.24.0/24 [90/3865600] via 10.0.4.12, 00:13:51, Tunnel4
D 192.168.34.0/24 [90/3865600] via 10.0.4.13, 00:13:42, Tunnel4
D 192.168.124.0/24 [90/3891200] via 10.0.4.12, 00:13:51, Tunnel4
D 192.168.134.0/24 [90/3891200] via 10.0.4.13, 00:13:42, Tunnel4
B 192.168.254.0/24 is directly connected, 4w0d, Ethernet3/0
B* 0.0.0.0/0 [20/281600] via 192.168.254.2 (Internet), 4w0d, Ethernet3/0
Separate DMVPNs – VRF-lite
Routing Tables – Spoke2
C 10.0.0.0/24 is directly connected, Tunnel0
Spoke2: vrf Yellow C 192.168.20.0/24 is directly connected, Ethernet0/0
D 192.168.120.0/24 [90/307200] via 192.168.20.2, 4w0d, Ethernet0/0
D 192.168.0.0/24 [90/2841600] via 10.0.0.1, 4w0d, Tunnel0
D 192.168.10.0/24 [90/4121600] via 10.0.0.1, 4w0d, Tunnel0
D 192.168.100.0/24 [90/2867200] via 10.0.0.1, 4w0d, Tunnel0
D 192.168.110.0/24 [90/4147200] via 10.0.0.1, 4w0d, Tunnel0
D 192.168.254.0/24 [90/2841600] via 10.0.0.1, 4w0d, Tunnel0
D*EX 0.0.0.0/0 [170/2841600] via 10.0.0.1, 4w0d, Tunnel0

C 10.0.2.0/24 is directly connected, Tunnel2

vrf Red C 192.168.22.0/24 is directly connected, Ethernet1/0
D 192.168.122.0/24 [90/307200] via 192.168.22.2, 00:05:22, Ethernet1/0
D 192.168.2.0/24 [90/2841600] via 10.0.2.1, 00:05:22, Tunnel2
D 192.168.32.0/24 [90/4121600] via 10.0.2.1, 00:05:22, Tunnel2
D 192.168.102.0/24 [90/2867200] via 10.0.2.1, 00:05:22, Tunnel2
D 192.168.132.0/24 [90/4147200] via 10.0.2.1, 00:05:22, Tunnel2
D 192.168.254.0/24 [90/2841600] via 10.0.2.1, 00:05:22, Tunnel2
D*EX 0.0.0.0/0 [170/2841600] via 10.0.2.1, 00:05:22, Tunnel2

C 10.0.4.0/24 is directly connected, Tunnel4

vrf Green C 192.168.24.0/24 is directly connected, Ethernet2/0
D 192.168.124.0/24 [90/307200] via 192.168.24.2, 00:01:41, Ethernet2/0
D 192.168.4.0/24 [90/2841600] via 10.0.4.1, 00:01:41, Tunnel4
D 192.168.34.0/24 [90/4121600] via 10.0.4.1, 00:01:41, Tunnel4
D 192.168.104.0/24 [90/2867200] via 10.0.4.1, 00:01:41, Tunnel4
D 192.168.134.0/24 [90/4147200] via 10.0.4.1, 00:01:41, Tunnel4
D 192.168.254.0/24 [90/2841600] via 10.0.4.1, 00:01:41, Tunnel4
D*EX 0.0.0.0/0 [170/2841600] via 10.0.4.1, 00:01:41, Tunnel4
Separate DMVPNs – VRF-lite
Routing Tables – Spoke 1 and 3
C 10.0.0.0/24 is directly connected, Tunnel0
Spoke1: vrf Yellow D 192.168.0.0/24 [90/2841600] via 10.0.0.1, 4w0d, Tunnel0
C 192.168.10.0/24 is directly connected, Ethernet0/0
D 192.168.20.0/24 [90/4121600] via 10.0.0.1, 4w0d, Tunnel0
D 192.168.100.0/24 [90/2867200] via 10.0.0.1, 4w0d, Tunnel0
D 192.168.110.0/24 [90/307200] via 192.168.10.2, 4w0d, Ethernet0/0
D 192.168.120.0/24 [90/4147200] via 10.0.0.1, 4w0d, Tunnel0
D 192.168.254.0/24 [90/2841600] via 10.0.0.1, 4w0d, Tunnel0
D*EX 0.0.0.0/0 [170/2841600] via 10.0.0.1, 4w0d, Tunnel0

C 10.0.2.0/24 is directly connected, Tunnel2

Spoke3: vrf Red D 192.168.2.0/24 [90/2841600] via 10.0.2.1, 00:01:33, Tunnel2
D 192.168.22.0/24 [90/4121600] via 10.0.2.1, 00:01:33, Tunnel2
C 192.168.32.0/24 is directly connected, Ethernet1/0
D 192.168.102.0/24 [90/2867200] via 10.0.2.1, 00:01:33, Tunnel2
D 192.168.122.0/24 [90/4147200] via 10.0.2.1, 00:01:33, Tunnel2
D 192.168.132.0/24 [90/307200] via 192.168.32.2, 00:01:33, Ethernet1/0
D 192.168.254.0/24 [90/2841600] via 10.0.2.1, 00:01:33, Tunnel2
D*EX 0.0.0.0/0 [170/2841600] via 10.0.2.1, 00:01:33, Tunnel2

C 10.0.4.0/24 is directly connected, Tunnel4

vrf Green D 192.168.4.0/24 [90/2841600] via 10.0.4.1, 00:02:15, Tunnel4
D 192.168.24.0/24 [90/4121600] via 10.0.4.1, 00:02:15, Tunnel4
C 192.168.34.0/24 is directly connected, Ethernet2/0
D 192.168.104.0/24 [90/2867200] via 10.0.4.1, 00:02:15, Tunnel4
D 192.168.124.0/24 [90/4147200] via 10.0.4.1, 00:02:15, Tunnel4
D 192.168.134.0/24 [90/307200] via 192.168.34.2, 00:02:15, Ethernet2/0
D 192.168.254.0/24 [90/2841600] via 10.0.4.1, 00:02:15, Tunnel4
D*EX 0.0.0.0/0 [170/2841600] via 10.0.4.1, 00:02:15, Tunnel4
Separate DMVPNs – VRF-lite
Summary
• Separate DMVPN mGRE tunnel per BU VRF
• Hub routers handle all DMVPNs
• Multiple Hub routers for redundancy and load
• EIGRP used for routing protocol outside of and over DMVPNs on Spokes and
Hubs
• Address family per VRF
• BGP used only on the hub
• Redistribute between EIGRP and BGP for import/export of routes between VRFs
• “Internet” VRF for Internet access and routing between VRFs
• “Outside” VRF for routing DMVPN tunnel packets
MPLS over DMVPN
MPLS over DMVPN
n = number of spokes
m = number of hubs for spoke-hub tunnels
• Single DMVPN mGRE tunnel on all routers
• Dynamic spoke-spoke tunnels support all common VRFs between spokes
• Hub routers support all VRFs (BU); Spokes only those they need
• Multiple Hub routers for load and redundancy
• Load:
• (m) Hubs to support n hub-spoke tunnels (each hub limited by number of MP-BGP neighbors)
• Redundancy:
• Manually map (2×(m) Hubs) or dynamically ((IOS SLB) map (m)+1 hubs)
• Routing
• MP-BGP over DMVPN; EIGRP, BGP or OSPF on LANs
• Forwarding between VRFs (optional)
• MP-BGP can be used on hub to leak routes between VRFs
• Use a router or firewall behind the hub
MPLS over DMVPN
Example uses:
• EIGRP for routing protocol on LANs (outside of DMVPN)
• An ipv4 address-family per VRF
• MP-BGP over DMVPN
• Transport all VRF routes over a single MP-BGP neighborship
• Import/export routes between VRFs and “Internet” VRF
• An ipv4 address-family per VRF plus one vpnv4 address-family
• Internet access via Hub router for all VRFs – no split-tunneling
• IP address spaces are unique across all VRFs
• Routing used to forward return packets from Internet back to correct VRF
• “Outside” f-VRF used for forwarding tunnel packets on WAN (transport)
• Separation of Transport address space from Overlay (VRF) address space
MPLS over DMVPN
• DMVPN Phase 1 – hub-and-spoke only
• LDP (mpls ip) is used for MPLS tag distribution only on hub-spoke tunnels
• Hub is configured as MPLS P router
• Spoke to spoke packets are MPLS tag-switched via Hub
• DMVPN Phase 2 – spoke-spoke only after shortcut tunnel is up
• LDP (mpls ip) is used for MPLS tag distribution only on hub-spoke tunnels
• Hub is configured as MPLS PE router
• Spoke to spoke packets:
• Cannot be MPLS tag-switched via Hub  dropped
• Can be MPLS tag-switched through dynamic spoke-spoke tunnel once it is up
• DMVPN Phase 3 – full spoke-spoke support (15.4(1)S, 15.4(2)T)
• NHRP (mpls nhrp) is used for MPLS tag distribution on hub-spoke and spoke-spoke tunnels
• Hub is configured as MPLS P/PE router
• Spoke to spoke packets are MPLS tag-switched via Hub then through spoke-spoke tunnel once it is up
MPLS over DMVPN Phase 3
• New support in NHRP to
• Keep track of NHRP mapping table entries per VRF
• Transport MPLS forwarding labels interface Tunnel0
• MPLS LDP not used over DMVPN bandwidth 1000
• MP-BGP propagates VPN labels ip address 10.0.0.1 255.255.255.0
no ip redirects
• VRF RD and RT tags must be the same on all nodes. ip nhrp authentication test
• New CLI ip nhrp map multicast dynamic
ip nhrp network-id 100000
• ‘mpls nhrp’ replaces ‘mpls ip’ on the tunnel interface ip nhrp holdtime 360
• Tag switching on spoke-hub-spoke and spoke-spoke path ip nhrp redirect 15.5(3)M,S
mpls mtu 1400
• Hub router is MPLS P/PE mpls nhrp
• ‘mpls mtu ...’ applied before MPLS “encapsulation” tunnel source Serial2/0
• Apply ‘ip tcp adjust-mss ...’ on any IP interface in path tunnel mode gre multipoint
tunnel key 100000
• Per-tunnel QoS tunnel protection ipsec profile vpnprof
• MPLS experimental bits (15.5(3)M,S)
MPLS over DMVPN – 2547oDMVPN
Logical Topology
.254.x .1 .1 .0.x .100.1 192.168./24
.2 .2
.1 .2.x .102.1 MPLS CE
Internet .2
Hub1 .1 .4.x .104.1
MPLS P/PE .2
.1

DMVPN
10.0.0.0/24
MPLS VPN
(10,12,14)
.11 .13
Spoke1
MPLS PE
MPLS CE Spoke3
Spoke2 .12 MPLS PE
MPLS CE
192.168./24 MPLS PE .20.x .120.1
.10.x .1 .2
.110.1 192.168./24 .32.x
.2 .1 .22.x .132.1
192.168./24 .122.1 .1 .2
.1 .2
.34.x .134.1
MPLS CE .24.x .124.1 .1 .2
.1 .2
MPLS over DMVPN – 2547oDMVPN
Spoke-spoke: (Bus. #1)
Spoke-spoke: (Bus. #2, #3)
mGRE Tunnel

WAN
LAN
Spoke 1 Bus. #1 MPLS P/PE

Internet
LAN “global”
Bus. #1

WAN
WAN
LAN
Spoke 2 Bus. #2

LAN
Hub
Bus. #3 Server

Interface
LANs

LAN
Bus. #2 WAN
Spoke 3
LAN
Bus. #3
Hub-spoke: (Bus. #1);
mGRE Tunnel (Bus. #1, #2, #3); (Bus. #2, #3)
MPLS PE
MPLS over DMVPN – 2547oDMVPN
Hub Configuration – BU VRFs
No import vrf definition Yellow vrf definition Red vrf definition Green
from Internet rd 1:1 rd 2:2 rd 3:3
route-target export 1:1 route-target export 2:2 route-target export 3:3
No EIGRP routing route-target import 1:1 route-target import 2:2 route-target import 3:3
! ! !
over tunnel interface Ethernet0/0 interface Ethernet1/0 interface Ethernet2/0
vrf forwarding Yellow vrf forwarding Red vrf forwarding Green
ip address 192.168.0.1 255.255.255.0 ip address 192.168.2.1 255.255.255.0 ip address 192.168.4.1 255.255.255.0
VRF ip tcp adjust-mss 1360 ip tcp adjust-mss 1360 ip tcp adjust-mss 1360
! ! !
router eigrp 1 router eigrp 1 router eigrp 1
! ! !
EIGRP routing address-family ipv4 vrf Yellow address-family ipv4 vrf Red address-family ipv4 vrf Green
over LAN and default-metric 1000 100 255 1 1500 default-metric 1000 100 255 1 1500 default-metric 1000 100 255 1 1500
redistribute redistribute bgp 1 redistribute bgp 1 redistribute bgp 1
with BGP network 192.168.0.0 network 192.168.2.0 network 192.168.4.0
autonomous-system 1 autonomous-system 1 autonomous-system 1
exit-address-family exit-address-family exit-address-family
! ! !
router bgp 1 router bgp 1 router bgp 1
BGP routing over ! ! !
DMVPN and for address-family ipv4 vrf Yellow address-family ipv4 vrf Red address-family ipv4 vrf Green
route import/export redistribute connected redistribute connected redistribute connected
redistribute static redistribute static redistribute static
redistribute eigrp 1 redistribute eigrp 1 redistribute eigrp 1
default-information originate default-information originate default-information originate
exit-address-family exit-address-family exit-address-family
Static route ! ! !
ip route vrf Yellow 0.0.0.0 0.0.0.0 – ip route vrf Red 0.0.0.0 0.0.0.0 – ip route vrf Green 0.0.0.0 0.0.0.0 –
for default Ethernet3/0 192.168.254.2 Ethernet3/0 192.168.254.2 Ethernet3/0 192.168.254.2
MPLS over DMVPN – 2547oDMVPN
Hub Configuration – Internet VRF
vrf definition Internet
rd 10:10
route-target export 10:10
route-target import 10:10
Import VRF routes route-target import 1:1
route-target import 2:2
into Internet route-target import 3:3
!
address-family ipv4
import map No-Default
exit address-family
!
interface Ethernet3/0
vrf forwarding Internet
Don’t import ip address 192.168.254.1 255.255.255.0
!
Default router eigrp 1
!
address-family ipv4 vrf Internet
default-metric 1000 100 255 1 1500
redistribute bgp 1
network 192.168.254.0
EIGRP routing autonomous-system 1
to Internet exit-address-family
!
router bgp 1
!
address-family ipv4 vrf Internet
network 192.168.254.0
redistribute eigrp 1
exit-address-family
!
access-list 20 deny host 0.0.0.0
access-list 20 permit any
!
route-map No-Default permit 10
match ip address 20
MPLS over DMVPN – 2547oDMVPN
Hub Configuration – MP-BGP over DMVPN

interface Tunnel0
ip address 10.0.0.1 255.255.255.0
ip nhrp authentication MPLS
ip nhrp network-id 1000
NHRP distributes mpls nhrp
MPLS Labels mpls mtu 1400
tunnel key 1000
tunnel vrf Outside
!
router bgp 1
bgp router-id 10.0.0.1
Spokes are dynamic neighbors bgp listen range 10.0.0.0/24 peer-group Spokes
and route-reflector clients neighbor Spokes peer-group
neighbor Spokes remote-as 1
neighbor Spokes update-source Tunnel0
!
address-family vpnv4
neighbor Spokes activate
neighbor Spokes send-community extended
Hub is IP next-hop neighbor Spokes route-reflector-client
neighbor spokes next-hop-self all
(DMVPN Phase 3)
exit-address-family
!
MPLS over DMVPN – 2547oDMVPN
Spoke2 Configuration – BU VRFs

vrf definition Yellow vrf definition Red vrf definition Green

rd 1:1 rd 2:2 rd 3:3
route-target export 1:1 route-target export 2:2 route-target export 3:3
route-target import 1:1 route-target import 2:2 route-target import 3:3
! ! !
VRF interface Ethernet0/0 interface Ethernet1/0 interface Ethernet2/0
vrf forwarding Yellow vrf forwarding Red vrf forwarding Green
ip address 192.168.20.1 255.255.255.0 ip address 192.168.22.1 255.255.255.0 ip address 192.168.24.1 255.255.255.0
ip tcp adjust-mss 1360 ip tcp adjust-mss 1360 ip tcp adjust-mss 1360
EIGRP on LAN ! ! !
redistribute router eigrp 1 router eigrp 1 router eigrp 1
with BGP ! ! !
address-family ipv4 vrf Yellow address-family ipv4 vrf Red address-family ipv4 vrf Green
default-metric 1000 100 255 1 1500 default-metric 1000 100 255 1 1500 default-metric 1000 100 255 1 1500
redistribute bgp 1 redistribute bgp 1 redistribute bgp 1
network 192.168.20.0 network 192.168.22.0 network 192.168.24.0
autonomous-system 1 autonomous-system 1 autonomous-system 1
exit-address-family exit-address-family exit-address-family
! ! !
BGP over DMVPN router bgp 1 router bgp 1 router bgp 1
! ! !
address-family ipv4 vrf Yellow address-family ipv4 vrf Red address-family ipv4 vrf Green
redistribute eigrp 1 redistribute eigrp 1 redistribute eigrp 1
exit-address-family exit-address-family exit-address-family
MPLS over DMVPN – 2547oDMVPN
Spoke2 Configuration – MP-BGP over DMVPN
interface Tunnel0
ip address 10.0.0.12 255.255.255.0
ip nhrp authentication MPLS
ip nhrp network-id 1000
ip nhrp holdtime 600
ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
NHRP distributes ip nhrp shortcut
MPLS Labels mpls nhrp
mpls mtu 1400
tunnel source Serial4/0
tunnel mode gre multipoint
tunnel key 1000
tunnel vrf Outside
!
!
router bgp 1
bgp router-id 10.0.0.12
neighbor 10.0.0.1 remote-as 1
BGP routing neighbor 10.0.0.1 update-source Tunnel0
!
over DMVPN address-family vpnv4
neighbor 10.0.0.1 activate
neighbor 10.0.0.1 send-community extended
exit-address-family
!
MPLS over DMVPN
NHRP Redirects
10.0.0.11  172.16.1.1
10.0.0.12  172.16.2.1
Data packet 192.168.0.1/24
NHRP Redirect 192.168.0.0/16  Null0
NHRP Resolution 192.168.0.0/24  Conn.
Physical: 172.17.0.1 192.168.110.0/24  10.0.0.11 (L: 17)
NHRP mapping Tunnel0: 10.0.0.1 192.168.120.0/24  10.0.0.12 (L: 17)
36 17 192.168.110.0/24  10.0.0.11
CEF FIB Table 52 17 192.168.120.0/24  10.0.0.12
MPLS Forwarding
Physical: 172.16.2.1
(dynamic)
Physical: 172.16.1.1
(dynamic) Tunnel0: 10.0.0.12
Tunnel0: 10.0.0.11
Spoke B
Spoke A
VRF Yellow: 192.168.120.1/24
VRF Yellow: 192.168.110.1/24
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1
192.168.120.1  ???
192.168.120.0/24  192.168.20.2
192.168.110.0/24  192.168.10.2 192.168.110.0/24  10.0.0.1 (L:36)
192.168.120.0/24  10.0.0.1 (L:52)
N 36 192.168.110.0/24  10.0.0.1
17 NL 192.168.110.0/24  192.168.10.2 17 NL 192.168.120.0/24  192.168.20.2
N 52 192.168.120.0/24  10.0.0.1
MPLS over DMVPN
NHRP Resolution Request
10.0.0.11  172.16.1.1
10.0.0.12  172.16.2.1
Data packet 192.168.0.1/24
NHRP Redirect 192.168.0.0/16  Null0
NHRP Resolution 192.168.0.0/24  Conn.
Physical: 172.17.0.1 192.168.110.0/24  10.0.0.11 (L: 17)
NHRP mapping Tunnel0: 10.0.0.1 192.168.120.0/24  10.0.0.12 (L: 17)
36 17 192.168.110.0/24  10.0.0.11
CEF FIB Table 52 17 192.168.120.0/24  10.0.0.12
MPLS Forwarding
Physical: 172.16.2.1
(dynamic)
Physical: 172.16.1.1
(dynamic) Tunnel0: 10.0.0.12
Tunnel0: 10.0.0.11
Spoke B
Spoke A
VRF Yellow: 192.168.120.1/24
VRF Yellow: 192.168.110.1/24
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1 10.0.0.11  172.16.1.1
192.168.120.1  ???
192.168.120.0/24  192.168.20.2
192.168.110.0/24  192.168.10.2 192.168.110.0/24  10.0.0.1 (L:36)
192.168.120.0/24  10.0.0.1 (L:52)
N 36 192.168.110.0/24  10.0.0.1
17 NL 192.168.110.0/24  192.168.10.2 17 NL 192.168.120.0/24  192.168.20.2
N 52 192.168.120.0/24  10.0.0.1
MPLS over DMVPN
NHRP Resolution Reply
10.0.0.11  172.16.1.1
10.0.0.12  172.16.2.1
Data packet 192.168.0.1/24
NHRP Redirect 192.168.0.0/16  Null0
NHRP Resolution 192.168.0.0/24  Conn.
Physical: 172.17.0.1 192.168.110.0/24  10.0.0.11 (L: 17)
NHRP mapping Tunnel0: 10.0.0.1 192.168.120.0/24  10.0.0.12 (L: 17)
36 17 192.168.110.0/24  10.0.0.11
CEF FIB Table 52 17 192.168.120.0/24  10.0.0.12
MPLS Forwarding
Physical: 172.16.2.1
(dynamic)
Physical: 172.16.1.1
(dynamic) Tunnel0: 10.0.0.12
Tunnel0: 10.0.0.11

Spoke B VRF Yellow: 192.168.120.1/24

Spoke A
VRF Yellow: 192.168.110.1/24
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1 10.0.0.11  172.16.1.1
10.0.0.12  172.16.2.1 192.168.110.0/24  172.16.1.1 (L:17)
192.168.120.1/32  172.16.2.1
192.168.120.0/24 ??? (L:17)
192.168.120.0/24  192.168.20.2
192.168.110.0/24  192.168.10.2 192.168.110.0/24  10.0.0.12 (L:17)
10.0.0.1 (L:36)
192.168.120.0/24  10.0.0.1
10.0.0.12(L:52)
(L:17)
17 NL 192.168.120.0/24  192.168.20.2
17 NL 192.168.110.0/24  192.168.10.2 36 192.168.110.0/24  10.0.0.12
NN 17 10.0.0.1
52 192.168.120.0/24  10.0.0.12
N 17 10.0.0.1
MPLS over DMVPN – 2547oDMVPN
Routing Tables – Hub
Global C 10.0.0.0/24 is directly connected, Tunnel0

S* 0.0.0.0/0 [1/0] via 172.17.0.2

vrf Outside C 172.17.0.0/30 is directly connected, Serial4/0
S* 0.0.0.0/0 [1/0] via 192.168.254.2, Ethernet3/0
vrf Yellow C 192.168.0.0/24 is directly connected, Ethernet0/0
D 192.168.100.0/24 [90/307200] via 192.168.0.2, 03:27:24, Ethernet0/0
B 192.168.10.0/24 [200/0] via 10.0.0.11, 03:26:56
B 192.168.20.0/24 [200/0] via 10.0.0.12, 03:26:56
B 192.168.110.0/24 [200/307200] via 10.0.0.11, 03:26:56
B 192.168.120.0/24 [200/307200] via 10.0.0.12, 03:26:56
S* 0.0.0.0/0 [1/0] via 192.168.254.2, Ethernet3/0
vrf Red C 192.168.2.0/24 is directly connected, Ethernet1/0
D 192.168.102.0/24 [90/307200] via 192.168.1.2, 03:27:22, Ethernet1/0
B 192.168.22.0/24 [200/0] via 10.0.0.12, 03:26:54
B 192.168.32.0/24 [200/0] via 10.0.0.13, 03:26:54
B 192.168.122.0/24 [200/307200] via 10.0.0.12, 03:26:54
B 192.168.132.0/24 [200/307200] via 10.0.0.13, 03:26:54
S* 0.0.0.0/0 [1/0] via 192.168.254.2, Ethernet3/0
vrf Green C 192.168.4.0/24 is directly connected, Ethernet2/0
D 192.168.104.0/24 [90/307200] via 192.168.2.2, 03:27:18, Ethernet2/0
B 192.168.24.0/24 [200/0] via 10.0.0.12, 03:26:53
B 192.168.34.0/24 [200/0] via 10.0.0.13, 03:26:53
B 192.168.134.0/24 [200/307200] via 10.0.0.13, 03:26:53
B 192.168.124.0/24 [200/307200] via 10.0.0.12, 03:26:53
MPLS over DMVPN – 2547oDMVPN
MPLS Tables – Hub
Hub1#show mpls forwarding

Local Outgoing Prefix Bytes label Outgoing Next Hop

label label or Tunnel Id Switched interface
25 No Label 192.168.254.0/24[V] 0 aggregate/Internet
16 No Label 0.0.0.0/0[V] 0 Et3/0 192.168.254.2
17 No Label 192.168.0.0/24[V] 0 aggregate/Yellow
18 No Label 192.168.100.0/24[V] 0 Et0/0 192.168.0.2
36 17 192.168.110.0/24[V] 0 Tu0 10.0.0.11
50 16 192.168.10.0/24[V] 0 Tu0 10.0.0.11
51 16 192.168.20.0/24[V] 0 Tu0 10.0.0.12
52 17 192.168.120.0/24[V] 0 Tu0 10.0.0.12
19 No Label 0.0.0.0/0[V] 0 Et3/0 192.168.254.2
21 No Label 192.168.102.0/24[V] 0 Et1/0 192.168.2.2
30 24 192.168.122.0/24[V] 0 Tu0 10.0.0.12
35 19 192.168.32.0/24[V] 0 Tu0 10.0.0.13
38 No Label 192.168.2.0/24[V] 0 aggregate/Red
44 21 192.168.132.0/24[V] 0 Tu0 10.0.0.13
53 23 192.168.22.0/24[V] 0 Tu0 10.0.0.12
20 No Label 192.168.104.0/24[V] 0 Et2/0 192.168.4.2
22 No Label 0.0.0.0/0[V] 0 Et3/0 192.168.254.2
24 No Label 192.168.4.0/24[V] 0 aggregate/Green
31 25 192.168.124.0/24[V] 0 Tu0 10.0.0.12
32 20 192.168.134.0/24[V] 0 Tu0 10.0.0.13
34 22 192.168.24.0/24[V] 0 Tu0 10.0.0.12
45 17 192.168.34.0/24[V] 0 Tu0 10.0.0.13
MPLS over DMVPN – 2547oDMVPN
MPLS Tables – Spoke 2
Spoke2#show mpls forwarding

Local Outgoing Prefix Bytes label Outgoing Next Hop

label label or Tunnel Id Switched interface
16 No Label 192.168.20.0/24[V] 0 aggregate/Yellow
17 No Label 192.168.120.0/24[V] 23496 Et0/0 192.168.20.2
None 16 0.0.0.0/0[V] 0 Tu0 10.0.0.1
None 17 192.168.0.0/24[V] 0 Tu0 10.0.0.1
None 50 192.168.10.0/24[V] 0 Tu0 10.0.0.1
None 18 192.168.100.0/24[V] 0 Tu0 10.0.0.1
None 36 192.168.110.0/24[V] 0 Tu0 10.0.0.1
23 No Label 192.168.22.0/24[V] 0 aggregate/Red
24 No Label 192.168.122.0/24[V] 0 Et1/0 192.168.22.2
None 19 0.0.0.0/0[V] 0 Tu0 10.0.0.1
None 38 192.168.2.0/24[V] 0 Tu0 10.0.0.1
None 35 192.168.32.0/24[V] 0 Tu0 10.0.0.1
None 21 192.168.102.0/24[V] 0 Tu0 10.0.0.1
None 44 192.168.132.0/24[V] 0 Tu0 10.0.0.1
22 No Label 192.168.24.0/24[V] 0 aggregate/Green
25 No Label 192.168.124.0/24[V] 0 Et2/0 192.168.24.2
None 22 0.0.0.0/0[V] 0 Tu0 10.0.0.1
None 24 192.168.4.0/24[V] 0 Tu0 10.0.0.1
None 45 192.168.34.0/24[V] 0 Tu0 10.0.0.1
None 20 192.168.104.0/24[V] 0 Tu0 10.0.0.1
None 32 192.168.134.0/24[V] 0 Tu0 10.0.0.1
MPLS over DMVPN – 2547oDMVPN
Routing Tables – Spoke2

Spoke2: Yellow B* 0.0.0.0/0 [200/0] via 10.0.0.1, 00:39:20

B 192.168.0.0/24 [200/0] via 10.0.0.1, 00:39:20
B 192.168.10.0/24 [200/0] via 10.0.0.1, 00:39:20
C 192.168.20.0/24 is directly connected, Ethernet0/0
B 192.168.100.0/24 [200/307200] via 10.0.0.1, 00:39:20
B 192.168.110.0/24 [200/307200] via 10.0.0.1, 00:39:20
D 192.168.120.0/24 [90/307200] via 192.168.20.2, 5w0d, Ethernet0/0

Red B* 0.0.0.0/0 [200/0] via 10.0.0.1, 00:40:37

B 192.168.2.0/24 [200/0] via 10.0.0.1, 00:40:37
C 192.168.22.0/24 is directly connected, Ethernet1/0
B 192.168.32.0/24 [200/0] via 10.0.0.1, 00:40:33
B 192.168.102.0/24 [200/307200] via 10.0.0.1, 00:40:37
D 192.168.122.0/24 [90/307200] via 192.168.22.2, 4w1d, Ethernet1/0
B 192.168.132.0/24 [200/307200] via 10.0.0.1, 00:40:33

Green B* 0.0.0.0/0 [200/0] via 10.0.0.1, 00:41:07

B 192.168.4.0/24 [200/0] via 10.0.0.1, 00:41:07
C 192.168.24.0/24 is directly connected, Ethernet2/0
B 192.168.34.0/24 [200/0] via 10.0.0.1, 00:41:03
B 192.168.104.0/24 [200/307200] via 10.0.0.1, 00:41:07
D 192.168.124.0/24 [90/307200] via 192.168.24.2, 4w1d, Ethernet2/0
B 192.168.134.0/24 [200/307200] via 10.0.0.1, 00:41:03
MPLS over DMVPN – 2547oDMVPN
MPLS Tables – Spoke 1 and 3
Local Outgoing Prefix Bytes label Outgoing Next Hop
Spoke1: Yellow label label or Tunnel Id Switched interface
16 No Label 192.168.10.0/24[V] 0 aggregate/Yellow
17 No Label 192.168.110.0/24[V] 23496 Et0/0 192.168.20.2
None 16 0.0.0.0/0[V] 0 Tu0 10.0.0.1
None 17 192.168.0.0/24[V] 0 Tu0 10.0.0.1
None 51 192.168.20.0/24[V] 0 Tu0 10.0.0.1
None 18 192.168.100.0/24[V] 0 Tu0 10.0.0.1
None 52 192.168.120.0/24[V] 0 Tu0 10.0.0.1

Local Outgoing Prefix Bytes label Outgoing Next Hop

Spoke3: Red label label or Tunnel Id Switched interface
19 No Label 192.168.32.0/24[V] 0 aggregate/Red
21 No Label 192.168.132.0/24[V] 0 Et1/0 192.168.22.2
None 19 0.0.0.0/0[V] 0 Tu0 10.0.0.1
None 38 192.168.2.0/24[V] 0 Tu0 10.0.0.1
None 53 192.168.22.0/24[V] 0 Tu0 10.0.0.1
None 21 192.168.102.0/24[V] 0 Tu0 10.0.0.1
None 30 192.168.122.0/24[V] 0 Tu0 10.0.0.1
Green 17 No Label 192.168.34.0/24[V] 0 aggregate/Green
20 No Label 192.168.134.0/24[V] 0 Et2/0 192.168.24.2
None 22 0.0.0.0/0[V] 0 Tu0 10.0.0.1
None 24 192.168.4.0/24[V] 0 Tu0 10.0.0.1
None 34 192.168.24.0/24[V] 0 Tu0 10.0.0.1
None 20 192.168.104.0/24[V] 0 Tu0 10.0.0.1
None 31 192.168.124.0/24[V] 0 Tu0 10.0.0.1
MPLS over DMVPN – 2547oDMVPN
Routing Tables – Spoke 1 and 3

Spoke1: Yellow B* 0.0.0.0/0 [200/0] via 10.0.0.1, 00:43:03

B 192.168.0.0/24 [200/0] via 10.0.0.1, 00:43:03
C 192.168.10.0/24 is directly connected, Ethernet0/0
B 192.168.20.0/24 [200/0] via 10.0.0.1, 00:43:03
B 192.168.100.0/24 [200/307200] via 10.0.0.1, 00:43:03
D 192.168.110.0/24 [90/307200] via 192.168.10.2, 5w0d, Ethernet0/0
B 192.168.120.0/24 [200/307200] via 10.0.0.1, 00:43:03

Spoke3: Red B* 0.0.0.0/0 [200/0] via 10.0.0.1, 00:41:59

B 192.168.2.0/24 [200/0] via 10.0.0.1, 00:41:59
B 192.168.22.0/24 [200/0] via 10.0.0.1, 00:41:59
C 192.168.32.0/24 is directly connected, Ethernet1/0
B 192.168.102.0/24 [200/307200] via 10.0.0.1, 00:41:59
B 192.168.122.0/24 [200/307200] via 10.0.0.1, 00:41:59
D 192.168.132.0/24 [90/307200] via 192.168.32.2, 4w1d, Ethernet1/0

Green B* 0.0.0.0/0 [200/0] via 10.0.0.1, 00:42:32

B 192.168.4.0/24 [200/0] via 10.0.0.1, 00:42:32
B 192.168.24.0/24 [200/0] via 10.0.0.1, 00:42:32
C 192.168.34.0/24 is directly connected, Ethernet2/0
B 192.168.104.0/24 [200/307200] via 10.0.0.1, 00:42:32
B 192.168.124.0/24 [200/307200] via 10.0.0.1, 00:42:32
D 192.168.134.0/24 [90/307200] via 192.168.34.2, 4w1d, Ethernet2/0
MPLS over DMVPN – 2547oDMVPN
Summary
• Single DMVPN mGRE tunnel on all routers
• MPLS
• NHRP (mpls nhrp) is used for MPLS tag distribution on hub-spoke and spoke-spoke tunnels
• Hub is configured as MPLS P/PE and Spoke as MPLS PE
• Spoke to spoke packets are MPLS tag-switched via Hub then
through spoke-spoke tunnel once it is up
• Routing
• EIGRP, OSPF or BGP is used for routing outside of DMVPN (on “LAN”)
• MP-BGP used for routing over DMVPN
• Redistribute between RP on “LAN” and MP-BGP for transport over DMVPN
• “Outside” f-VRF used for forwarding tunnel packets on WAN (transport)
• Tunnel Interface is in “Global”
Extending
MPLS over DMVPN
Extending MPLS over DMVPN
• Mostly the same as above MPLS over DMVPN
• Single DMVPN mGRE tunnel on Hub and Spoke routers
• MPLS:
• NHRP (mpls nhrp) on DMVPN and LDP (mpls ip) on LAN, for MPLS label distribution
• Hubs and Spokes – Must be MPLS P/PE
• Must define VRFs, even if not used on any local interface
• Nodes behind Hubs and Spokes can be MPLS P, PE or P/PE
• Routing
• MP-BGP over DMVPN and LANs
• Hubs are BGP Route-reflectors for Spokes
• Hubs and Spokes may be BGP route-reflectors for MPLS PE nodes behind them
MPLS over DMVPN – Extending an MPLS
Logical Topology
MPLS PE .100.1
Hub1 .101.1 192.168.x.y/24
MPLS P/PE .102.1
.1

DMVPN
10.0.0.0/24

.11 .13
Spoke1
MPLS P/PE
Spoke3
Spoke2 .12 MPLS P/PE
MPLS P/PE
192.168.x.y/24
.120.1 .132.1
MPLS PE .110.1 MPLS PE
.122.1 .134.1
192.168.x.y/24 MPLS PE
.124.1 192.168.x.y/24
MPLS over DMVPN – Extending an MPLS
Spoke-spoke: (Bus. #1)
Spoke-spoke: (Bus. #2, #3)

Spoke 1 mGRE Tunnel

WAN
MPLS
LDP MPLS NHRP
P/PE
Hub
P/PE

WAN
Spoke 2

WAN
MPLS MPLS
LDP LDP
P/PE

Spoke 3 MPLS
WAN
P/PE LDP

Hub-spoke: (Bus. #1);

mGRE Tunnel (Bus. #1, #2, #3); (Bus. #2, #3)
MPLS NHRP
Extending MPLS over DMVPN
Current Issues with spoke-spoke tunnels
• NHRP redirects
• Issue:
• Hubs send redirects to IP data packet source  injected into MPLS on tunnel
• MPLS tag-switches NHRP redirect through to MPLS PE behind spoke  dropped
• Spoke doesn’t get NHRP redirect  no spoke-spoke tunnel
• Workaround:
• Configure Spoke to summarize IP subnets behind it
• NHRP redirect packet will be MPLS tag-switched to spoke  spoke-spoke tunnel
• Problem:
• May not be able to summarize IP subnets without covering subnets behind other spokes
• If subnets change, may need to manually modify summarization on spoke
• Solution:
• Use regular IP forwarding over hub-spoke tunnel for NHRP redirect
Extending MPLS over DMVPN
Current Issues with spoke-spoke tunnels
• NHRP spoke-spoke routes
• Issue:
• NHRP on spoke injects route to use spoke-spoke tunnel with MPLS tag into RIB/FIB
• This route is not redistributed into MP-BGP to be advertised to MPLS PE behind spoke
• MPLS PE still uses original tag that tag-switches the packet via the Hub
• Spoke-spoke tunnel is not used
• Workaround:
• Configure MP-BGP with network statements for data subnets behind other spokes
• MP-BGP will pick up the spoke-spoke route and tag when NHRP inserts it into the RIB
• Not Scalable: would need to add MP-BGP statement for all subnets behind other spokes
• Solution:
• Enable ‘redistribute nhrp’ under BGP configuration (also enable for EIGRP and OSPF)
• Then MP-BGP will automatically pick up NHRP routes
• May also want to use a route-map to filter which routes are picked up by MP-BGP
Recent and New Features
DMVPN Other Recent and Future Features
• Recently Available • Future (cont)
• Metadata (CMD, NSH) over DMVPN • DMVPN extended authentication
• PfR; TrustSec (SGT) • Strong NHRP authentication using HMAC
• Multiple Tunnel Termination (MTT) • NHRP spoke authentication using Radius
• NHRP fixes (done); CEF fixes (next) • Dynamic Tunnel Key on spokes
• Increased Multicast Support
• Coming Next
• Limited spoke-spoke multicast support
• Extending MPLS over DMVPN fixes • Large spoke to many small spokes
• MPLS on DMVPN part of larger MPLS • Native Multicast over DMVPN
• VXLAN-GPE encapsulation for DMVPN • Tunnel packets with multicast destination
• Support for multiple spokes behind NPAT • ISP network does replication
(**16.4.1) • ESON KS (Group and pair-wise keys)

• Future • NHRP route advertisement

• More MPLS over DMVPN • GRE tunnel sub-interfaces
• mVPN and VPLS support • EVN WAN using DMVPN
Thank you
 Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• LABIOT-2012 Implementing Dynamic Multipoint VPN
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Recommended Reading

• Explains all key • Copies are available at the

IWAN technologies CLUS Cisco Press bookstore
and components
Come Meet
The Authors
• VIRL labs are
available so that
you can practice
these concepts as • Anthony, Brad, David, and
you read them in Jean-Marc are signing books
the book at Cisco Press bookstore on
Weds. 1:30 – 2 PM
Thank you
Extras
Agenda
• DMVPN Design Overview
• DMVPN General
• IWAN Specific
• NHRP Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
Phase 2 – Features
• Single mGRE interface with ‘tunnel protection …’
• On Hubs and Spokes
• Hubs must be inter-connected in a “Daisy chain” over same mGRE tunnel
• IKE authentication information (Certificates, Wildcard Pre-shared Keys)
• Spoke-spoke data traffic direct
• Reduced load on hub
• Reduced latency
• Single IPsec encrypt/decrypt
• Routing Protocol
• Still hub-and-spoke
• Cannot summarize spoke routes on hub
• Routes on spokes must have IP next-hop of remote spoke (preserve next-hop)
Phase 2 – Process switching
• IP Data packet is forwarded out tunnel interface to IP next-hop from routing table
• NHRP looks in mapping table for IP destination
• If Entry Found
• Forward to NBMA from mapping table – overriding IP next-hop
• If No Entry Found
• Forward to IP next-hop (if in NHRP table) otherwise to NHS
• If arriving interface was not tunnel interface
• Initiate NHRP Resolution Request for IP next-hop and send via NHS path (first up NHS)
• If (no socket) Entry Found
• If arriving interface is not tunnel interface – convert entry to (socket)
• Trigger IPsec to bring up crypto socket
• Forward to IP next-hop (if in NHRP table) otherwise to NHS
Phase 2 – CEF Switching
• IP Data packet is forwarded out tunnel interface to IP next-hop from FIB table
• If adjacency is of type Valid
• Packet is encapsulated and forwarded by CEF out tunnel interface
• NHRP is not involved
• If adjacency is of type Glean or Incomplete
• Punt packet to process switching
• If original arriving interface was not this tunnel interface
• Initiate NHRP Resolution Request for IP next-hop
• Send resolution request for IP next-hop (tunnel IP address) of remote Spoke
• Resolution request forwarded via NHS path (first up NHS)
• Resolution reply is used to create NHRP mapping and to complete the Adjacency
Phase 2 – NHRP Resolution Request
Data packet 192.168.0.1/24 10.0.0.11  172.16.1.1
10.0.0.11  172.16.1.1
NHRP Resolution 10.0.0.12 
10.0.0.12  172.16.2.1
172.16.2.1

Physical: 172.17.0.1 192.168.0.0/24  Conn.

NHRP mapping 192.168.1.0/24  10.0.0.11
Tunnel0: 10.0.0.1
192.168.2.0/24  10.0.0.12
CEF FIB Table
10.0.0.11  172.16.1.1
CEF Adjacency 10.0.0.12  172.16.2.1
Physical: 172.16.2.1
(dynamic)
Tunnel0: 10.0.0.12
Physical: 172.16.1.1
(dynamic)
Tunnel0: 10.0.0.11

Spoke B 192.168.2.1/24
Spoke A
192.168.1.1/24
10.0.0.1  172.17.0.1 (*) 10.0.0.1  172.17.0.1 (*)
10.0.0.12  ??? 10.0.0.11  172.16.1.1

192.168.0.0/24  10.0.0.1 192.168.0.0/24  10.0.0.1

192.168.1.0/24  Conn. 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12 192.168.2.0/24  Conn.

10.0.0.1  172.17.0.1 10.0.0.1  172.17.0.1

10.0.0.12  incomplete 10.0.0.11  incomplete
Phase 2 – NHRP Resolution Reply
Data packet 192.168.0.1/24 10.0.0.11  172.16.1.1
NHRP Resolution 10.0.0.12  172.16.2.1

Physical: 172.17.0.1 192.168.0.0/24  Conn.

NHRP mapping 192.168.1.0/24  10.0.0.11
Tunnel0: 10.0.0.1
192.168.2.0/24  10.0.0.12
CEF FIB Table
10.0.0.11  172.16.1.1
CEF Adjacency 10.0.0.12  172.16.2.1
Physical: 172.16.2.1
(dynamic)
Tunnel0: 10.0.0.12
Physical: 172.16.1.1
(dynamic)
Tunnel0: 10.0.0.11

Spoke B 192.168.2.1/24
Spoke A
192.168.1.1/24
10.0.0.1  172.17.0.1 (*)
10.0.0.1  172.17.0.1 (*) 10.0.0.11  172.16.1.1
10.0.0.11  172.16.1.1 (l) 10.0.0.12  172.16.2.1 (l)
10.0.0.12  ???
172.16.2.1
192.168.0.0/24  10.0.0.1
192.168.0.0/24  10.0.0.1 192.168.1.0/24  10.0.0.11
192.168.1.0/24  Conn.
192.168.2.0/24  Conn.
192.168.2.0/24  10.0.0.12
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1 10.0.0.11  172.16.1.1
incomplete
10.0.0.12  172.16.2.1
incomplete
Phase 2 – NHRP Resolution Response Processing
• Receive NHRP Resolution reply
• If using IPsec (tunnel protection …) then
• Trigger IPsec to setup ISAKMP and IPsec SAs for tunnel
• Data packets still forwarded via spoke-hub-…-hub-spoke path
• IPsec triggers back to NHRP when done
• Install new mapping in NHRP mapping table
• Send trigger to CEF to complete corresponding CEF adjacency
• Data packets now forwarded via direct spoke-spoke tunnel by CEF
• NHRP no longer involved
Phase 2 – Refresh or Remove Dynamic mappings
• Dynamic NHRP mapping entries have finite lifetime
• Controlled by ‘ip nhrp holdtime …’ on source of mapping (spoke)
• Background process checks mapping entry every 60 seconds
• Process-switching
• Used flag set each time mapping entry is used
• If used flag is set and expire time < 120 seconds then refresh entry, otherwise clear used flag
• CEF-switching
• If expire time < 120 seconds, CEF Adjacency entry marked “stale”
• If “stale” CEF Adjacency entry is then used, signal to NHRP to refresh entry
• Another resolution request is sent to refresh entry
• Resolution request via NHS path; reply via direct tunnel
• If entry expires it is removed
• If using IPsec  Trigger IPsec to remove IPsec/ISAKMP SAs
Phase 3 – NHRP Resolution Reply (Prior to 15.2(1)T – ISR, 7200)

Data packet 192.168.0.1/24 10.0.0.11  172.16.1.1

NHRP Redirect 10.0.0.12  172.16.2.1
NHRP Resolution
Physical: 172.17.0.1 192.168.0.0/24  Conn.
NHRP mapping Tunnel0: 10.0.0.1 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12
CEF FIB Table
10.0.0.11  172.16.1.1
CEF Adjacency 10.0.0.12  172.16.2.1
Physical: 172.16.2.1
(dynamic)
Physical: 172.16.1.1
(dynamic) Tunnel0: 10.0.0.12
Tunnel0: 10.0.0.11

Spoke A Spoke B 192.168.2.1/24

192.168.1.1/24
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1 10.0.0.11  172.16.1.1
10.0.0.12  172.16.2.1 192.168.1.0/24  172.16.1.1
192.168.2.1  ???
192.168.2.0/24  172.16.2.1
192.168.2.0/24  Conn.
192.168.1.0/24  Conn. 192.168.0.0/16  10.0.0.1
192.168.0.0/16  10.0.0.1
10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1 10.0.0.11  172.16.1.1
10.0.0.12  172.16.2.1
Phase 3 – CEF Switching
Data Packet Forwarding (Prior to 15.2(1)T – ISR, 7200)

• IP Data packet is forwarded out tunnel interface

1. IP next-hop from CEF FIB mapped to Adjacency
If adjacency is:
• Glean or Incomplete  Punt to process switching
• Valid  Select adjacency for the packet
2. NHRP in Outbound CEF Feature path
Look up packet IP destination in NHRP mapping table
• Matching entry: Reselect adjacency  use direct spoke-spoke tunnel
• No matching entry: Leave CEF adjacency  packet goes to hub
• If packet arrived on and is forwarded out the same tunnel interface
• Forward data packet
• If ‘ip nhrp redirect’ is on inbound tunnel then send NHRP redirect
• Packet is encapsulated, encrypted and forwarded
Interaction with IWAN
Agenda
• DMVPN Design Overview
• General and IWAN Specific
• NHRP Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• Interaction with IWAN
• f-VRFs
• NHRP the RIB and PfR
• Recent and New Features
DMVPN with IWAN f-VRFs
ISAKMP and IKE packets
Router in respective f-VRF WAN
MPLS f-VRF
INTERNET f-VRF
Global W
A
DMVPN Tunnel0 N
0
L
Data packets
A encapsulated
N in f-VRF tunnel
W
A
Data Packets N
using Global
1
DMVPN Tunnel1
DMVPN with IWAN f-VRFs
• Create VRF for each transport WAN interface (Ex: INTERNET, MPLS)
• vrf definition <fvrf-name>
• “Outside” of tunnel is in front-door VRF (f-VRF)
• interface tunnel<x>; tunnel vrf <fvrf-name>
• WAN (transport) interface is in f-VRF
• interface <wan-interface>; vrf forwarding <fvrf-name>
• Crypto – ISAKMP/IKEv2 are also in f-VRFs
• ISAKMP – need keyring for each f-VRF
• IKEv2 – need keyring, IKEv2 profile and IPsec profile
• Separate one for each f-VRF
Or
• Single one for all fVRFs by using ‘match fvrf any’ in IKEv2 profile
DMVPN with IWAN f-VRFs
f-VRF Configuration
vrf definition INTERNET interface Tunnel0
... ip address 10.0.0.11 255.255.255.0
vrf definition MPLS ...
... tunnel source FastEthernet0
! tunnel key 100000
crypto ikev2 keyring DMVPN tunnel vrf INTERNET
peer ANY tunnel protection ipsec profile DMVPN
address 0.0.0.0 0.0.0.0 interface Tunnel1
pre-shared-key cisco123 ip address 10.0.1.11 255.255.255.0
! ...
crypto ikev2 profile DMVPN tunnel source FastEthernet1
match fvrf any tunnel key 100001
match identity remote address 0.0.0.0 tunnel vrf MPLS
authentication remote pre-share tunnel protection ipsec profile DMVPN
authentication local pre-share !
keyring local DMVPN interface FastEthernet0
dpd 20 5 on-demand ! Spokes only vrf forwarding INTERNET
! ip address 172.16.1.1 255.255.255.240
crypto ipsec transform-set DMVPN esp-aes 256 esp-sha256-hmac !
mode transport interface FastEthernet1
! vrf forwarding MPLS
crypto ipsec profile DMVPN ip address 172.17.1.1 255.255.255.240
set transform-set DMVPN !
set ikev2-profile DMVPN ip route vrf MPLS 0.0.0.0 0.0.0.0 172.17.1.2
ip route vrf INTERNET 0.0.0.0 0.0.0.0 172.16.1.2
DMVPN with IWAN f-VRFs
Routing Crypto
Spoke1#show ip route vrf * Spoke1#show crypto ikev2 session
D*EX 0.0.0.0/0 [170/2918400] via 10.0.1.2, 00:00:04, Tunnel1 Session-id:1845, Status:UP-ACTIVE, IKE count:1, CHILD count:1
10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
C 10.0.0.0/24 is directly connected, Tunnel0 T-id Local Remote fvrf/ivrf Status
C 10.0.1.0/24 is directly connected, Tunnel1 2 172.16.1.1/500 172.16.0.1/500 INTERNET/none READY
D 192.168.0.0/21 [90/2892800] via 10.0.1.2, 00:20:27, Tunnel1
192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512,
C 192.168.1.0/24 is directly connected, Ethernet0/0 DH Grp:5, Auth sign: PSK, Auth verify: PSK
D 192.168.10.0/24 [90/2918400] via 10.0.1.2, 00:32:39, Tunnel1 Life/Active Time: 86400/1263 sec
Child sa: local selector 172.16.1.1/0 - 172.16.1.1/65535
Routing Table: INTERNET remote selector 172.16.0.1/0 - 172.16.0.1/65535
Gateway of last resort is 172.16.1.2 to network 0.0.0.0 ESP spi in/out: 0x86D2651B/0x1B72FEB6

S* 0.0.0.0/0 [1/0] via 172.16.1.2 Session-id:1844, Status:UP-ACTIVE, IKE count:1, CHILD count:1
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 172.16.1.0/28 is directly connected, FastEthernet0 T-id Local Remote fvrf/ivrf Status
1 172.17.1.1/500 172.17.0.5/500 MPLS/none READY
Routing Table: MPLS
Gateway of last resort is 172.17.1.2 to network 0.0.0.0 Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512,
DH Grp:5, Auth sign: PSK, Auth verify: PSK
S* 0.0.0.0/0 [1/0] via 172.17.1.2 Life/Active Time: 86400/1290 sec
172.17.0.0/16 is variably subnetted, 2 subnets, 2 masks Child sa: local selector 172.17.1.1/0 - 172.17.1.1/65535
C 172.17.1.0/28 is directly connected, FastEthernet1 remote selector 172.17.0.5/0 - 172.17.0.5/65535
ESP spi in/out: 0xF8C63D42/0x66DEA87D
DMVPN with IWAN DIA
Router
MPLS f-VRF
INTERNET f-VRF
Global W
A
DMVPN Tunnel0 N
0
L
A DIA packets
N “route” between
Global and f-VRF W
A
N
1
DMVPN Tunnel1
DMVPN with IWAN DIA
• Outbound
• Block learning default through tunnel
• Access-list: deny default; match everything else
• Route-map: if match “learn” route
• Apply route-map in Routing Protocol
• EIGRP: use “distribute-list ... in <tunnel-interface>
• BGP: use “neighbor ... in”
• Static default route in global table forwarding out Internet WAN interface
• ip route 0.0.0.0 0.0.0.0 <Internet-WAN> <next-hop>|dhcp <admin-distance>
• Inbound
• Policy-based routing (PBR)
• access-list: match internal networks
• route-map: if match use global routing table
DMVPN with IWAN DIA
Inbound Outbound
interface FastEthernet0 router eigrp 1
description INTERNET distribute-list route-map BLOCK-DEFAULT in Tunnel0
vrf forwarding INTERNET [distribute-list route-map BLOCK-DEFAULT in Tunnel1]
ip address 172.16.1.1 255.255.255.240 network 10.0.0.0 0.0.1.255
ip policy route-map INET-INTERNAL network 192.168.1.0
! !
ip access-list extended INTERNAL-NETS ip access-list standard ALL-EXCEPT-DEFAULT
permit ip any 10.0.0.0 0.0.1.255 deny 0.0.0.0
permit ip any 192.168.0.0 0.0.255.255 permit any
permit ip any 172.20.0.0 0.0.255.255 !
route-map BLOCK-DEFAULT permit 10
route-map INET-INTERNAL permit 10 match ip address ALL-EXCEPT-DEFAULT
match ip address INTERNAL-NETS !
set global ip route 0.0.0.0 0.0.0.0 FastEthernet0 172.16.1.2 10
! !
DMVPN with IWAN DIA
Before After
Spoke1#show ip eigrp topology Spoke1#sho ip eigrp topology
P 192.168.10.0/24, 1 successors, FD is 2918400 P 192.168.10.0/24, 1 successors, FD is 2918400
via 10.0.1.2 (2918400/332800), Tunnel1 via 10.0.1.2 (2918400/332800), Tunnel1
via 10.0.0.1 (3020800/332800), Tunnel0 via 10.0.0.1 (3020800/332800), Tunnel0
P 172.20.1.0/24, 1 successors, FD is 409600 P 172.20.1.0/24, 1 successors, FD is 409600
via 192.168.1.2 (409600/128256), Ethernet0/0 via 192.168.1.2 (409600/128256), Ethernet0/0
P 192.168.0.0/21, 1 successors, FD is 2892800 P 192.168.0.0/21, 1 successors, FD is 2892800
via 10.0.1.2 (2892800/307200), Tunnel1 via 10.0.1.2 (2892800/307200), Tunnel1
via 10.0.0.1 (2995200/307200), Tunnel0 via 10.0.0.1 (2995200/307200), Tunnel0
P 192.168.1.0/24, 1 successors, FD is 281600 P 192.168.1.0/24, 1 successors, FD is 281600
via Connected, Ethernet0/0 via Connected, Ethernet0/0
P 0.0.0.0/0, 1 successors, FD is 2918400 P 0.0.0.0/0, 0 successors, FD is Infinity
via 10.0.1.2 (2918400/2636800), Tunnel1 via 10.0.1.2 (2918400/2636800), Tunnel1
via 10.0.0.1 (3020800/2636800), Tunnel0

Spoke1#show ip route Spoke1#show ip route

D*EX 0.0.0.0/0 [170/2918400] via 10.0.1.2, 00:00:04, Tunnel1 S* 0.0.0.0/0 [10/0] via 172.16.1.2, Fastethernet0
... ...
D 172.20.1.0 [90/409600] via 192.168.1.2, 01:47:00, Ethernet0/0 D 172.20.1.0 [90/409600] via 192.168.1.2, 01:47:00, Ethernet0/0
D 192.168.0.0/21 [90/2892800] via 10.0.1.2, 00:20:27, Tunnel1 D 192.168.0.0/21 [90/2892800] via 10.0.1.2, 01:46:28, Tunnel1
C 192.168.1.0/24 is directly connected, Ethernet0/0 C 192.168.1.0/24 is directly connected, Ethernet0/0
D 192.168.10.0/24 [90/2918400] via 10.0.1.2, 00:32:39, Tunnel1 D 192.168.10.0/24 [90/2918400] via 10.0.1.2, 01:46:28, Tunnel1
Agenda
• DMVPN Design Overview
• General and IWAN Specific
• NHRP Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• Interaction with IWAN
• f-VRFs
• NHRP the RIB and PfRv3
• Recent and New Features
Routing Protocol (RP), NHRP and PfRv3
• Routing protocol (RP) – destinations outside of the DMVPN
• Advertises reachability of these destinations over any/all DMVPNs
• Sets base forwarding within DMVPNs via the RIB
• PfRv3 – optimize forwarding of flows over different DMVPN paths
• PfR RIB used to control forwarding of flows
• Lookup alternate paths directly in RP database (except OSPF)
• Bring up alternate paths, with probe traffic
• NHRP – optimizes forwarding within a single DMVPN
• Shortcut (spoke-spoke) tunnels
• Triggered by data traffic, including PfRv3 probe traffic
• Changes forwarding by making changes in the RIB
• Tracks RIB RP entries to control adding/removing shortcut tunnel
Basic DMVPN Design for IWAN
Dual DMVPN MC
Internet DMVPN Physical: 192.168.10.3
192.168.10.0/24 Loop0: 172.18.0.10
MPLS DMVPN
.2 .1
Dynamic Spoke-to-spoke
Hub1 Hub2
Physical: 172.16.0.1 Physical: 172.17.0.5
Tunnel0: 10.0.0.1 Tunnel1: 10.0.1.1
Loop0: 172.18.0.1 Loop0: 172.18.0.2

MPLS Internet

Physical: (dynamic)
Tunnel0: 10.0.0.13
Physical: (dynamic) Tunnel1: 10.0.1.13
Tunnel0: 10.0.0.11 Loop0: 172.18.0.13
Tunnel1: 10.0.1.11
Loop0: 172.18.0.11

Spoke C
.1
Spoke A 192.168.3.0/24
.1
Physical: (dynamic) Physical: (dynamic) 192.168.13.0/14
192.168.1.0 /24 Tunnel0: 10.0.0.12 Tunnel1: 10.0.1.12
192.168.11.0/24 Spoke B1 .1 .2 Spoke B2

192.168.2.0 /24
192.168.12.0/24
DMVPN with Routing Protocol
Routing Protocol – Both paths
SpokeA# show ip eigrp topology In RIB MPLS
Default over MPLS P 0.0.0.0/0, 0 successors, FD is Infinity Not in RIB INET
via 10.0.1.2 (1769472000/1048576000), Tunnel1
P 10.0.1.0/24, 1 successors, FD is 1376256000
Tunnel subnets via Connected, Tunnel1
P 10.0.0.0/24, 1 successors, FD is 1638400000
via Connected, Tunnel0
Data Summary Route P 192.168.0.0/21, 1 successors, FD is 1703936000
via 10.0.1.2 (1703936000/393216000), Tunnel1
via 10.0.0.1 (1966080000/393216000), Tunnel0
Local Subnet P 192.168.1.0/24, 1 successors, FD is 131072000
via Connected, Ethernet0/0
P 192.168.10.0/24, 1 successors, FD is 1769472000
Data Specific Routes via 10.0.1.2 (1769472000/458752000), Tunnel1
via 10.0.0.1 (2031616000/458752000), Tunnel0
P 192.168.11.0/24, 1 successors, FD is 196608000
via 192.168.1.2 (196608000/131072000), Ethernet0/0
P 192.168.13.0/24, 1 successors, FD is 2228224000
via 10.0.1.2 (2228224000/1507328000), Tunnel1
Not including MC/BR
Loopback Routes via 10.0.0.1 (2752512000/1769472000), Tunnel0
DMVPN with Routing Protocol
RIB – Path via MPLS
SpokeA# show ip route MPLS
INET
Static Default for DIA Gateway of last resort is 172.16.1.2 to network 0.0.0.0

S* 0.0.0.0/0 [10/0] via 172.16.1.2, Serial1/0

Tunnel Interface Subnets 10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
C 10.0.0.0/24 is directly connected, Tunnel0
C 10.0.1.0/24 is directly connected, Tunnel1
MC/BR Loopback Subnet 172.18.0.0/32 is subnetted, 8 subnets
D 172.18.0.1 [90/12800640] via 10.0.0.1, 01:10:55, Tunnel0
D 172.18.0.2 [90/10752640] via 10.0.1.2, 01:10:55, Tunnel1
D 172.18.0.10 [90/13312640] via 10.0.1.2, 01:10:55, Tunnel1
C 172.18.0.11 is directly connected, Loopback0
D 172.18.0.13 [90/16384640] via 10.0.1.2, 01:10:55, Tunnel1
Data Summary Route D 192.168.0.0/21 [90/13312000] via 10.0.1.2, 01:10:55, Tunnel1
192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.1.0/24 is directly connected, Ethernet0/0
D 192.168.10.0/24 [90/13824000] via 10.0.1.2, 01:10:55, Tunnel1
Data Specific Routes D 192.168.11.0/24 [90/1536000] via 192.168.1.2, 01:10:55, Ethernet0/0
D 192.168.13.0/24 [90/17408000] via 10.0.1.2, 01:10:55, Tunnel1
Forwarding over Primary DMVPN
Dual DMVPN MC
Physical: 192.168.10.3
Internet DMVPN
192.168.10.0/24 Loop0: 172.18.0.10
MPLS DMVPN
.2 .1
Primary path
Hub1 Hub2
nhrp route-watch Physical: 172.16.0.1 Physical: 172.17.0.5
no nhrp route-watch Tunnel0: 10.0.0.1 Tunnel1: 10.0.1.1
Loop0: 172.18.0.1 Loop0: 172.18.0.2

MPLS Internet

Physical: (dynamic)
Tunnel0: 10.0.0.13
Physical: (dynamic) Tunnel1: 10.0.1.13
Tunnel0: 10.0.0.11 Loop0: 172.18.0.13
Tunnel1: 10.0.1.11
Loop0: 172.18.0.11

Spoke C
.1
Spoke A 192.168.3.0/24
.1
192.168.13.0/14
192.168.1.0 /24
192.168.11.0/24
Forwarding over Primary DMVPN
NHRP RIB
SpokeA# show ip nhrp SpokeA# show ip route Parent Routes
10.0.1.13/32 via 10.0.1.13 10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
Tunnel1 created 00:04:23, expire 00:04:19 C 10.0.0.0/24 is directly connected, Tunnel0
Type: dynamic, Flags: router nhop rib L 10.0.0.11/32 is directly connected, Tunnel0
NBMA address: 172.17.3.1 C 10.0.1.0/24 is directly connected, Tunnel1
192.168.1.0/24 via 10.0.1.11 L 10.0.1.11/32 is directly connected, Tunnel1
Tunnel1 created 00:04:25, expire 00:01:36 H 10.0.1.13/32 is directly connected, 00:05:28, Tunnel1
Type: dynamic, Flags: router unique local D 192.168.0.0/21 [90/13312000] via 10.0.1.2, 00:11:02, Tunnel1
NBMA address: 172.17.1.1 192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
(no-socket) C 192.168.1.0/24 is directly connected, Ethernet0/0
192.168.3.0/24 via 10.0.1.13 L 192.168.1.1/32 is directly connected, Ethernet0/0
Tunnel1 created 00:01:40, expire 00:04:19 H 192.168.3.0/24 [250/1] via 10.0.1.13, 00:03:06, Tunnel1
Type: dynamic, Flags: router rib D 192.168.10.0/24 [90/13824000] via 10.0.1.2, 00:11:02, Tunnel1
NBMA address: 172.17.3.1 D 192.168.11.0/24 [90/1536000] via 192.168.1.2, 00:11:02, Ethernet0/0
192.168.11.0/24 via 10.0.1.11 D % 192.168.13.0/24 [90/17408000] via 10.0.1.2, 00:11:02, Tunnel1
Tunnel1 created 00:04:02, expire 00:01:57 [NHO][90/1] via 10.0.1.13, 00:05:28, Tunnel1
Type: dynamic, Flags: router unique local
NBMA address: 172.17.1.1
(no-socket)
192.168.13.0/24 via 10.0.1.13
Tunnel1 created 00:04:02, expire 00:01:57
Type: dynamic, Flags: router rib nho
NBMA address: 172.17.3.1
Forwarding over Secondary DMVPN (nhrp route-watch)

Dual DMVPN MC
Physical: 192.168.10.3
Internet DMVPN
192.168.10.0/24 Loop0: 172.18.0.10
MPLS DMVPN
.2 .1
Primary path
Hub1 Hub2
nhrp route-watch Physical: 172.16.0.1 Physical: 172.17.0.5
no nhrp route-watch Tunnel0: 10.0.0.1 Tunnel1: 10.0.1.1
Loop0: 172.18.0.1 Loop0: 172.18.0.2

MPLS Internet

Physical: (dynamic)
Tunnel0: 10.0.0.13
Physical: (dynamic) Tunnel1: 10.0.1.13
Tunnel0: 10.0.0.11 Loop0: 172.18.0.13
Tunnel1: 10.0.1.11
Loop0: 172.18.0.11

Spoke C
.1
Spoke A 192.168.3.0/24
.1
192.168.13.0/14
192.168.1.0 /24
192.168.11.0/24
Forwarding over Secondary DMVPN (nhrp route-watch)
NHRP RIB
SpokeA# show ip nhrp SpokeA# show ip route
10.0.0.13/32 via 10.0.0.13 10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
Tunnel0 created 00:01:01, expire 00:05:07 C 10.0.0.0/24 is directly connected, Tunnel0
Type: dynamic, Flags: router nhop L 10.0.0.11/32 is directly connected, Tunnel0
NBMA address: 172.16.3.1 C 10.0.1.0/24 is directly connected, Tunnel1
192.168.1.0/24 via 10.0.0.11 L 10.0.1.11/32 is directly connected, Tunnel1
Tunnel0 created 00:01:01, expire 00:04:58 D 192.168.0.0/21 [90/13312000] via 10.0.1.2, 00:04:38, Tunnel1
Type: dynamic, Flags: router unique local 192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
NBMA address: 172.16.1.1 C 192.168.1.0/24 is directly connected, Ethernet0/0
(no-socket) L 192.168.1.1/32 is directly connected, Ethernet0/0
192.168.3.0/24 via 10.0.0.13 D 192.168.10.0/24 [90/13824000] via 10.0.1.2, 00:04:38, Tunnel1
Tunnel0 created 00:01:00, expire 00:04:59 D 192.168.11.0/24 [90/1536000] via 192.168.1.2, 00:04:38, Ethernet0/0
Type: dynamic, Flags: router D 192.168.13.0/24 [90/17408000] via 10.0.1.2, 00:04:38, Tunnel1
NBMA address: 172.16.3.1
192.168.11.0/24 via 10.0.0.11 NHRP mapping entries not in RIB
Tunnel0 created 00:00:52, expire 00:05:07 No matching Parent Route
Type: dynamic, Flags: router unique local
NBMA address: 172.16.1.1
(no-socket)
192.168.13.0/24 via 10.0.0.13
Tunnel0 created 00:00:52, expire 00:05:07
Type: dynamic, Flags: router
NBMA address: 172.16.3.1
Forwarding over Secondary DMVPN (no nhrp route-watch)

Dual DMVPN MC
Physical: 192.168.10.3
Internet DMVPN
192.168.10.0/24 Loop0: 172.18.0.10
MPLS DMVPN
.2 .1
Primary path
Hub1 Hub2
nhrp route-watch Physical: 172.16.0.1 Physical: 172.17.0.5
no nhrp route-watch Tunnel0: 10.0.0.1 Tunnel1: 10.0.1.1
Loop0: 172.18.0.1 Loop0: 172.18.0.2

MPLS Internet

Physical: (dynamic)
Tunnel0: 10.0.0.13
Physical: (dynamic) Tunnel1: 10.0.1.13
Tunnel0: 10.0.0.11 Loop0: 172.18.0.13
Tunnel1: 10.0.1.11
Loop0: 172.18.0.11

Spoke C
.1
Spoke A 192.168.3.0/24
.1
192.168.13.0/14
192.168.1.0 /24
192.168.11.0/24
Forwarding over Secondary DMVPN (no nhrp route-watch)
NHRP RIB
SpokeA# show ip nhrp SpokeA# show ip route
10.0.0.13/32 via 10.0.0.13 10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
Tunnel0 created 00:00:36, expire 00:05:25 C 10.0.0.0/24 is directly connected, Tunnel0
Type: dynamic, Flags: router nhop rib L 10.0.0.11/32 is directly connected, Tunnel0
NBMA address: 172.16.3.1 H 10.0.0.13/32 is directly connected, 00:00:34, Tunnel0
192.168.1.0/24 via 10.0.0.11 C 10.0.1.0/24 is directly connected, Tunnel1
Tunnel0 created 00:00:35, expire 00:05:24 L 10.0.1.11/32 is directly connected, Tunnel1
Type: dynamic, Flags: router unique local D 192.168.0.0/21 [90/13312000] via 10.0.1.2, 00:11:02, Tunnel1
NBMA address: 172.16.1.1 192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
(no-socket) C 192.168.1.0/24 is directly connected, Ethernet0/0
192.168.3.0/24 via 10.0.0.13 L 192.168.1.1/32 is directly connected, Ethernet0/0
Tunnel0 created 00:00:34, expire 00:05:25 H 192.168.3.0/24 [250/1] via 10.0.0.13, 00:00:34, Tunnel0
Type: dynamic, Flags: router rib D 192.168.10.0/24 [90/13824000] via 10.0.1.2, 00:11:02, Tunnel1
NBMA address: 172.16.3.1 D 192.168.11.0/24 [90/1536000] via 192.168.1.2, 00:11:02, Ethernet0/0
192.168.11.0/24 via 10.0.0.11 D % 192.168.13.0/24 [90/17408000] via 10.0.1.2, 00:11:02, Tunnel1
Tunnel0 created 00:00:24, expire 00:05:35 [NHO][90/1] via 10.0.0.13, 00:00:28, Tunnel0
Type: dynamic, Flags: router unique local
NBMA address: 172.16.1.1 No Check for Parent Routes
(no-socket)
192.168.13.0/24 via 10.0.0.13
Tunnel0 created 00:00:24, expire 00:05:35
Type: dynamic, Flags: router rib nho
NBMA address: 172.16.3.1
Building spoke-spoke tunnels with NHRP and PfRv3
• PfRv3 Controlled Data flows
• Forwards data flows over both primary and secondary DMVPN
• PfR controls any load-balancing
• Uses PfR Loopback as next-hop (Ex: 172.18.0.x)
• NHRP triggered to build spoke-spoke tunnel over both DMVPNs
• NHRP mapping entries to Loopback (Ex: 172.18.0.x)
• NHRP modifies RIB for Loopback next-hop
• If routing changes  PfR controlled flows quickly rerouted

• PfRv3 Uncontrolled Data flows

• Data flows forwarded via the RIB
• Uses primary DMVPN
• Need ECMP routes to load-balancing over both DMVPNs
Building spoke-spoke tunnels with NHRP and PfRv3
Dual DMVPN MC
Physical: 192.168.10.3
Internet DMVPN
192.168.10.0/24 Loop0: 172.18.0.10
MPLS DMVPN
.2 .1
Dynamic Spoke-to-spoke
Hub1 Hub2
Physical: 172.16.0.1 Physical: 172.17.0.5
Tunnel0: 10.0.0.1 Tunnel1: 10.0.1.1
Loop0: 172.18.0.1 Loop0: 172.18.0.2

MPLS Internet

Physical: (dynamic)
Tunnel0: 10.0.0.13
Physical: (dynamic) Tunnel1: 10.0.1.13
Tunnel0: 10.0.0.11 Loop0: 172.18.0.13
Tunnel1: 10.0.1.11
Loop0: 172.18.0.11

Spoke C
.1
Spoke A 192.168.3.0/24
.1
192.168.13.0/14
192.168.1.0 /24
192.168.11.0/24
Forwarding over Primary and Secondary DMVPN
NHRP RIB
SpokeA# show ip nhrp brief SpokeA# show ip route next-hop-override
Target Via NBMA Mode Intfc 10.0.0.0/8 is variably subnetted, 6 subnets, 2 masks
10.0.0.1/32 10.0.0.1 172.16.0.1 static Tu0 C 10.0.0.0/24 is directly connected, Tunnel0
10.0.0.11/32 10.0.0.11 172.16.1.1 dyn,loc Tu0 L 10.0.0.11/32 is directly connected, Tunnel0
10.0.0.13/32 10.0.0.13 172.16.3.1 dyn,rib Tu0 H 10.0.0.13/32 is directly connected, 00:08:40, Tunnel0
172.18.0.11/32 10.0.0.11 172.16.1.1 dyn,loc Tu0 C 10.0.1.0/24 is directly connected, Tunnel1
172.18.0.13/32 10.0.0.13 172.16.3.1 dyn,nho Tu0 L 10.0.1.11/32 is directly connected, Tunnel1
10.0.1.2/32 10.0.1.2 172.17.0.5 static Tu1 H 10.0.1.13/32 is directly connected, 00:09:05, Tunnel1
10.0.1.11/32 10.0.1.11 172.17.1.1 dyn,loc Tu1 172.18.0.0/32 is subnetted, 8 subnets
10.0.1.13/32 10.0.1.13 172.17.3.1 dyn,rib Tu1 D 172.18.0.1 [90/12800640] via 10.0.0.1, 02:07:25, Tunnel0
172.18.0.11/32 10.0.1.11 172.17.1.1 dyn,loc Tu1 D 172.18.0.2 [90/10752640] via 10.0.1.2, 02:07:25, Tunnel1
172.18.0.13/32 10.0.1.13 172.17.3.1 dyn,nho Tu1 D 172.18.0.10 [90/13312640] via 10.0.1.2, 02:07:25, Tunnel1
192.168.1.0/24 10.0.1.11 172.17.1.1 dyn,loc Tu1 C 172.18.0.11 is directly connected, Loopback0
192.168.3.0/24 10.0.1.13 172.17.3.1 dyn,rib Tu1 D % 172.18.0.13 [90/16384640] via 10.0.1.2, 02:04:46, Tunnel1
192.168.11.0/24 10.0.1.11 172.17.1.1 dyn,loc Tu1 [NHO][90/1] via 10.0.0.13, 00:02:19, Tunnel0
192.168.13.0/24 10.0.1.13 172.17.3.1 dyn,nho Tu1 [NHO][90/1] via 10.0.1.13, 00:08:40, Tunnel1
D 192.168.0.0/21 [90/13312000] via 10.0.1.2, 02:07:25, Tunnel1
192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.1.0/24 is directly connected, Ethernet0/0
L 192.168.1.1/32 is directly connected, Ethernet0/0
H 192.168.3.0/24 [250/1] via 10.0.1.13, 00:09:05, Tunnel1
D 192.168.10.0/24 [90/13824000] via 10.0.1.2, 02:04:46, Tunnel1
D 192.168.11.0/24 [90/1536000] via 192.168.1.2, 02:07:25, Ethernet0/0
D % 192.168.13.0/24 [90/17408000] via 10.0.1.2, 02:04:46, Tunnel1
[NHO][90/1] via 10.0.1.13, 00:08:59, Tunnel1
Summary
Routing Protocol (RP), NHRP and PfRv3
• Routing protocol (RP) – destinations outside of the DMVPN
• Sets base forwarding for IWAN
• Set preference for one DMVPN or can setup up ECMP routes
• PfRv3 – optimize forwarding of flows over different DMVPN paths
• Find paths directly in RP database (except OSPF)
• PfR RIB forwards flows over paths to MC/BR Loopback next-hop
• Probe traffic over alternate paths
• NHRP – optimizes forwarding within a single DMVPN
• Shortcut (spoke-spoke) tunnels
• Triggered by data traffic and/or PfRv3 probe traffic
• Use ‘no nhrp route-watch’ to enable shortcut tunnels over alternate paths
• NHRP mapping/routes to MC/BR Loopback addresses
Agenda
• DMVPN Design Overview
• DMVPN General
• IWAN Specific
• NHRP Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• Recent and New Features
• Configuration, Resiliency,
Routing and Forwarding,
Centralized Control
Configuration Reduction
• Issue
• CLI commands need to be configured to recommended values because defaults are not
very useful.
• Solution
• Change CLI command defaults to recommended values
• Set other CLI commands as default so that they don’t have to be configured at all
• Derive CLI command values from other parts of the configuration so they don’t have to
be configured.
Configuration
• New defaults (IOS/XE 16.3) • Future Defaults & Auto-config.
• NHRP • NHRP
• Spoke: (ip/ipv6) • ip/ipv6 nhrp network-id #
• nhrp holdtime 600 • 1st: tunnel key #
• nhrp shortcut • 2nd: Interface tunnel#
• nhrp registration no-unique • Tunnel Defaults
• Hub: (ip/ipv6) • tunnel vrf <tunnel-source-vrf>
• nhrp holdtime 600
• Miscellaneous Defaults
• nhrp map multicast dynamic
• ip mtu
• nhrp max-send 10000 every 10 (15.5(3)[S,M]2)
• ip tcp adjust-mss
• bandwidth (inherit)
NHRP Original Configuration
interface Tunnel0
bandwidth 1000
ip address 10.0.0.1 255.255.255.0
Hub interface Tunnel0
bandwidth 1000
ip address 10.0.0.11 255.255.255.0
Spoke
no ip redirects no ip redirects
ip mtu 1400 ip mtu 1400
ip nhrp authentication test ip nhrp authentication test
ip nhrp map multicast dynamic ip nhrp map multicast 172.17.0.1
ip nhrp map multicast 172.17.0.1 ip nhrp map 10.0.0.1 172.17.0.1
ip nhrp map 10.0.0.1 172.17.0.1 ip nhrp map multicast 172.17.0.5
ip nhrp network-id 100000 ip nhrp map 10.0.0.2 172.17.0.5
ip nhrp holdtime 600 ip nhrp network-id 100000
ip nhrp nhs 10.0.0.1 ip nhrp holdtime 600
ip nhrp redirect ip nhrp nhs 10.0.0.1
ip tcp adjust-mss 1360 ip nhrp nhs 10.0.0.2
delay 1000 ip nhrp registration no-unique
tunnel source Serial2/0 ip nhrp shortcut
tunnel mode gre multipoint ip tcp adjust-mss 1360
tunnel key 100000 delay 1000
tunnel vrf Outside tunnel source Serial1/0
tunnel protection ipsec profile DMVPN tunnel mode gre multipoint
! tunnel key 100000
tunnel vrf Outside
tunnel protection ipsec profile DMVPN
!
NHRP NHS Configuration Reduction – IOS 12.4(20)
interface Tunnel0
• Main use of NHRP mapping is to ...
ip nhrp map multicast 172.17.0.1
Hub
create static mapping for NHS. ip nhrp map 10.0.0.1 172.17.0.1
...
ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
• Combine associated NHRP ...
!
mapping and NHS commands
into a single line.
• Can still configure separate NHRP
mappings for other purposes. interface Tunnel0
...
ip nhrp map multicast 172.17.0.1
ip nhrp map 10.0.0.1 172.17.0.1
Spoke
ip nhrp map multicast 172.17.0.5
ip nhrp map 10.0.0.2 172.17.0.5
...
ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
ip nhrp nhs 10.0.0.2 nbma 172.17.0.5 multicast
...
!
NHRP Configuration New Defaults – IOS/XE 16.3
interface Tunnel0
• Spoke: (ip/ipv6) ...
ip nhrp authentication test Hub
• nhrp holdtime 600 ip nhrp map multicast dynamic
ip nhrp network-id 100000
• nhrp shortcut ip nhrp holdtime 600
ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
ip nhrp redirect
• nhrp registration no-unique ...
!
• Hub: (ip/ipv6)
• nhrp holdtime 600
• nhrp map multicast dynamic interface Tunnel0
...
ip nhrp authentication test
ip nhrp network-id 100000
Spoke
ip nhrp holdtime 600
ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
ip nhrp nhs 10.0.0.2 nbma 172.17.0.5 multicast
ip nhrp registration no-unique
ip nhrp shortcut
...
!
Tunnel Configuration Automatic Settings – Future
interface Tunnel0
• NHRP network-id bandwidth 1000
ip address 10.0.0.11 255.255.255.0
• Set to tunnel key <value> if configured no ip redirects
ip mtu 1400
• Otherwise, set to tunnel interface <#> ip nhrp authentication test
ip nhrp network-id 100000
• Tunnel VRF ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
ip nhrp nhs 10.0.0.2 nbma 172.17.0.5 multicast
• Set to VRF of tunnel source interface ip tcp adjust-mss 1360
delay 1000
• MTU tunnel source Serial1/0
tunnel mode gre multipoint
• Set to 1400 bytes tunnel key 100000
tunnel vrf Outside
• Use tunnel source <interface> tunnel protection ipsec profile DMVPN
(IPv4/IPv6) MTU – (100/120) bytes !
interface Serial1/0
• MSS ip mtu 1500
vrf forwarding Outside
• Set to (IPv4/IPv6) MTU – (40/60) bytes ip address 172.16.1.1 255.255.255.252
serial restart-delay 0
end
NHRP Final Configuration
interface Tunnel0
bandwidth 1000
ip address 10.0.0.1 255.255.255.0
Hub interface Tunnel0
bandwidth 1000
ip address 10.0.0.11 255.255.255.0
Spoke
ip nhrp authentication test ip nhrp authentication test
ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
ip nhrp redirect ip nhrp nhs 10.0.0.2 nbma 172.17.0.5 multicast
delay 1000 delay 1000
tunnel source Serial2/0 tunnel source Serial1/0
tunnel mode gre multipoint tunnel mode gre multipoint
tunnel key 100000 tunnel key 100000
tunnel protection ipsec profile DMVPN tunnel protection ipsec profile DMVPN
! !
Agenda
• DMVPN Design Overview
• DMVPN General
• IWAN Specific
• NHRP Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• Recent and New Features
• Configuration, Resiliency,
Routing and Forwarding,
Centralized Control
Resiliency
• Issues
• Many backup NHSs configured, but don’t want them all up
• Quickly failover all spokes to alternate hubs when a hub fails
• Quickly failover a spoke to alternate hub when spoke-hub tunnel fails

• Solutions
• Backup and FQDN NHS
• Fast Hub Failover using BGP (BFD between hubs)
• BFD over DMVPN (BFD on spoke-hub and spoke-spoke tunnels)
Tunnel Health Monitoring
Interface State – 15.0(1)M
• Issue
• mGRE tunnel Interface is always “up”
• Can’t use standard backup/recovery mechanisms
• backup interface, static interface routes, …

interface Tunnel0
• Solution ip address 10.0.0.11 255.255.255.0
• New Command ‘if-state nhrp’ …
ip nhrp map multicast 172.17.0.1
• Monitor NHRP registration replies ip nhrp map 10.0.0.1 172.17.0.1
ip nhrp map multicast 172.17.0.5
• If all NHSs are “down” then set tunnel interface up/down ip nhrp map 10.0.0.2 172.17.0.5
• Continue to send NHRP registration requests …
ip nhrp nhs 10.0.0.1
• If a single NHS is “up” then set tunnel interface up/up ip nhrp nhs 10.0.0.2
…
• Combine with ‘backup interface ...’ if-state nhrp
• Backup (tunnel) interface only up when main interface is down. …
Tunnel Health Monitoring – Interface State (cont.)
#show ip nhrp nhs detail
10.0.0.1 RE req-sent 100 req-failed 0 repl-recv 90 (00:01:38 ago)
10.0.0.2 RE req-sent 125 req-failed 0 repl-recv 79 (00:01:38 ago)
#show interface tunnel0
Tunnel0 is up, line protocol is up
*Apr 19 21:32:52 NHRP: NHS-DOWN: 10.0.0.1
*Apr 19 21:32:52 NHRP: NHS 10.0.0.1 Tunnel0 vrf 0 Cluster 0 Priority 0 Transitioned to 'E' from 'RE'
*Apr 19 21:32:53 NHRP: NHS-DOWN: 10.0.0.2
*Apr 19 21:32:53 NHRP: NHS 10.0.0.2 Tunnel0 vrf 0 Cluster 0 Priority 0 Transitioned to 'E' from 'RE'
*Apr 19 21:33:02 %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
*Apr 19 21:33:02 NHRP: if_down: Tunnel0 proto IPv4
#show ip nhrp nhs detail
10.0.0.1 E req-sent 105 req-failed 0 repl-recv 90 (00:02:12 ago)
10.0.0.2 E req-sent 130 req-failed 0 repl-recv 79 (00:02:12 ago)

#show interface tunnel0

Tunnel0 is up, line protocol is down
*Apr 19 21:33:12 NHRP: Send Registration Request via Tunnel0 vrf 0, packet size: 92
*Apr 19 21:33:13 NHRP: Send Registration Request via Tunnel0 vrf 0, packet size: 92
…
*Apr 19 21:34:36 NHRP: NHS 10.0.0.1 Tunnel0 vrf 0 Cluster 0 Priority 0 Transitioned to 'RE' from 'E'
*Apr 19 21:34:36 NHRP: NHS-UP: 10.0.0.1
*Apr 19 21:34:42 %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
*Apr 19 21:34:42 NHRP: if_up: Tunnel0 proto 0
#show ip nhrp nhs detail
10.0.0.1 RE req-sent 110 req-failed 0 repl-recv 96 (00:00:19 ago)
10.0.0.2 E req-sent 135 req-failed 0 repl-recv 79 (00:04:09 ago)
#show interface tunnel0
Tunnel0 is up, line protocol is up
Backup and FQDN NHS – 15.1(2)T
• Issue
• Backup NHSs only needed when primary NHSs are down
• Backup NHSs can be over subscribed
• Solution
• Set NHS ‘max-connections’
• Can set NHS priority (default=0 (best)) – Can have multiple hubs at the same priority
• Can group NHSs into clusters (default=0) – Separate max-connection value per cluster
• Configuration reduction – Single line NHS configuration and FQDN NHS
• Functionality
• NHSs are brought up in priority order, until cluster max-connections
• Down NHS at same priority is probed if not at max-connections
• Down NHS at a lower priority than an active NHS is probed even when max-connections is
reached
• FQDN resolved when bringing up NHS
Backup and FQDN NHS (cont.)
interface Tunnel0 #show ip nhrp
… 10.0.0.1/32 via 10.0.0.1 Tunnel0 Type: static, Flags: used
ip nhrp map 10.0.0.1 172.17.0.1 NBMA address: 172.17.0.1
ip nhrp map multicast 172.17.0.1 10.0.0.2/32 via 10.0.0.2 Tunnel0 Type: static, Flags: used
ip nhrp map 10.0.0.2 172.17.0.5 NBMA address: 172.17.0.5
ip nhrp map multicast 172.17.0.5 10.0.0.3/32 via 10.0.0.3 Tunnel0 Type: static, Flags: used
ip nhrp map 10.0.0.3 172.17.0.9 NBMA address: 172.17.0.9 (no-socket)
ip nhrp map multicast 172.17.0.9 10.0.0.4/32 via 10.0.0.4 Tunnel0 Type: static, Flags: used
ip nhrp map 10.0.0.4 172.17.0.13 NBMA address: 172.17.0.13 (no-socket)
ip nhrp map multicast 172.17.0.13
… #show ip nhrp nhs
ip nhrp nhs 10.0.0.1 Legend: E=Expecting replies, R=Responding, W=Waiting
ip nhrp nhs 10.0.0.2 Tunnel0:
ip nhrp nhs 10.0.0.3 10.0.0.1 RE priority = 0 cluster = 0
ip nhrp nhs 10.0.0.4 10.0.0.2 RE priority = 0 cluster = 0
ip nhrp nhs cluster 0 max-connections 2 10.0.0.3 W priority = 0 cluster = 0
… 10.0.0.4 W priority = 0 cluster = 0

interface Tunnel0
…
ip nhrp nhs 10.0.0.1 nbma Hub1.cisco.com multicast priority 10 cluster 1
ip nhrp nhs 10.0.0.2 nbma 172.17.0.5 multicast priority 20 cluster 1
ip nhrp nhs 10.0.0.3 nbma 172.17.0.9 multicast priority 10 cluster 2
ip nhrp nhs 10.0.0.4 nbma 172.17.0.13 multicast priority 10 cluster 2
ip nhrp nhs cluster 1 max-connections 1
ip nhrp nhs cluster 2 max-connections 1
#show ip nhrp nhs
Legend: E=Expecting replies, R=Responding, W=Waiting
Tunnel0:
10.0.0.1 RE NBMA Address: 172.17.0.1 (Hub1.Cisco.com) priority = 10 cluster = 1
10.0.0.2 W NBMA Address: 172.17.0.5 priority = 20 cluster = 1
10.0.0.3 RE NBMA Address: 172.17.0.9 priority = 10 cluster = 2
10.0.0.4 W NBMA Address: 172.17.0.13 priority = 10 cluster = 2
Fast Hub Failover using BGP
• Normal forwarding • Hubs “watch” each other (cont.)
• Few summary routes advertised to • Track loss of trigger route
spokes • When lost
• Covering all spoke site networks • Install static null0 route with special
• May have separate summary for Hub site tag for the summary routes
networks • Use BGP route-map to increase the
• Use MED to load balance or prefer one Local-Pref on tagged routes
• Spokes use Local-Pref over MED
hub over the other
• Recovery
• Hubs “watch” each other • Remove static null0 route with special tag
• Use BFD on physical link or tunnel link • Local-Pref reverts back to normal
between hubs • Spokes go back to using MED

• Special trigger route advertised only

between hubs over BFD link
• Example: 1.0.0.[1,2]/32 on Hub[1,2]
Fast Hub Failover using BGP
BGP Configuration
router bgp 1 Hub1 <x>=1
bgp listen range 10.0.0.0/24 peer-group spokes
neighbor spokes peer-group Hub2 <x>=2
neighbor spokes remote-as 1
neighbor spokes timers 20 60
neighbor 10.0.0.<x> remote-as 1
neighbor 10.0.0.<x> timers 20 60
Enable BFD for BGP neighbor 192.168.0.<x> remote-as 1
neighbor 192.168.0.<x> timers 20 60
neighbor 192.168.0.<x> fall-over bfd
!
address-family ipv4
Trigger Route network 1.0.0.<x> mask 255.255.255.255
network 192.168.0.0
network 192.168.0.0 mask 255.255.0.0 route-map Local-Pref
network 192.168.252.0 mask 255.255.252.0 route-map Local-Pref
aggregate-address 192.168.0.0 255.255.0.0 summary-only suppress-map BGP-LEAK
Modify Local-Pref when neighbor spokes activate
adding routes to BGP neighbor spokes route-reflector-client
neighbor spokes next-hop-self all
neighbor spokes route-map Block-Special out
neighbor 10.0.0.<x> activate
neighbor 10.0.0.<x> next-hop-self all
Block Trigger route to Spokes neighbor 10.0.0.<x> route-map Add-Metric-Hub in
and to other Hub over tunnel neighbor 10.0.0.<x> route-map Block-Special out
neighbor 192.168.0.<x> activate
neighbor 192.168.0.<x> next-hop-self all
neighbor 192.168.0.<x> route-map Add-Metric-Hub in
distance bgp 20 150 150
exit-address-family
Fast Hub Failover using BGP
Tracking, Route-maps and Routes Configuration
track timer ip route msec 500 Hub1 <x>=1,<y>=2
Track trigger route track 1 ip route 1.0.0.<y> 255.255.255.255 reachability
Hub2 <x>=2,<y>=1
track 2 list boolean and
object 1 not
!
Turn on BFD on Ethernet interface Ethernet0/0
ip address 192.168.0.<x> 255.255.255.0
(250 ms × 4 = 1 sec) delay 1000
bfd interval 250 min_rx 250 multiplier 4
!
router eigrp 1
Enable BFD for EIGRP bfd interface Ethernet0/0
(used on LAN) default-metric 1000 1000 255 1 1500
network 192.168.0.0
redistribute bgp 1 route-map BGP-EIGRP
!
Static routes with tag 200 ip route 192.168.0.0 255.255.0.0 Null0 tag 200 track 2
ip route 192.168.252.0 255.255.252.0 Null0 tag 200 track 2
and tracking object 2 ip route 1.0.0.<x> 255.255.255.255 Null0
ip route 192.168.0.0 255.255.0.0 Null0
ip route 192.168.252.0 255.255.252.0 Null0
!
Block Trigger route to Spokes access-list 1 permit 1.0.0.0 0.0.0.3
and to other Hub over tunnel route-map Block-Special deny 10
match ip address 1
route-map Block-Special permit 20
!
route-map Local-Pref permit 10
Change Local-Pref match tag 200
when route tag is 200 set local-preference 200
route-map Local-Pref permit 20
 Fast Hub Failover using BGP (normal)
RIB 192.168.253.0/24 .1
192.168.254.0/24 R2
BGP 192.168.255.0/24
.3 192.168.0.0/24
.1 .2
S 1.0.0.1  Null0 Hub1 Hub2 B 1.0.0.1 [150/51200]  192.168.0.1
B 1.0.0.2 [150/51200]  192.168.0.2 S 1.0.0.2  Null0
C 10.0.0.0/24  Tunnel0 C 10.0.0.0/24  Tunnel0
B 192.168.0.0/16  Null0 B 192.168.0.0/16  Null0
C 192.168.0.0/24  Ethernet0/0 .2 C 192.168.0.0/24  Ethernet0/0
.1
B 192.168.1.0/24 [150/0]  10.0.0.11 B 192.168.1.0/24 [150/25600]  10.0.0.11
B 192.168.2.0/24 [150/0]  10.0.0.12 B 192.168.2.0/24 [150/25600]  10.0.0.12
... ...
B 194.168.11.0/24 [150/307200]  10.0.0.11 B 194.168.11.0/24 [150/332800]  10.0.0.11
B 194.168.12.0/24 [150/307200]  10.0.0.12 B 194.168.12.0/24 [150/332800]  10.0.0.12
... DMVPN ...
S 192.168.252.0/22  Null0 S 192.168.252.0/22  Null0
D 192.168.253.0/24 [90/537600]  192.168.0.3 10.0.0.0/24 D 192.168.253.0/24 [90/537600]  192.168.0.3
... ...

.11
.12
C 10.0.0.0/24  Tunnel0 C 10.0.0.0/24  Tunnel0
B 192.168.0.0/16 [200/0]  10.0.0.1 B 192.168.0.0/16 [200/0]  10.0.0.1
Spoke1 B 192.168.0.0/24 [200/0]  10.0.0.1 Spoke2 B 192.168.0.0/24 [200/0]  10.0.0.1
.1 C 192.168.1.0/24  Ethernet0/0 .1 C 192.168.2.0/24  Ethernet0/0
D 192.168.11.0/24 [90/307200]  192.168.1.2 D 192.168.12.0/24 [90/307200]  192.168.2.2
192.168.1.0/24 .2 B 192.168.252.0/22 [200/0]  10.0.0.1 192.168.2.0/24 .2 B 192.168.252.0/22 [200/0]  10.0.0.1

*>i 192.168.0.0 10.0.0.1 0 100 0 i *>i 192.168.0.0 10.0.0.1 0 100 0 i

RS1 * i 10.0.0.2 25600 100 0 i RS2 * i 10.0.0.2 25600 100 0 i
*>i 192.168.0.0/16 10.0.0.1 0 100 0 i *>i 192.168.0.0/16 10.0.0.1 0 100 0 i
192.168.11.0/24 .1 * i 10.0.0.2 51200 100 0 i 192.168.12.0/24 .1 * i 10.0.0.2 51200 100 0 i
*>i 192.168.252.0/22 10.0.0.1 0 100 0 i *>i 192.168.252.0/22 10.0.0.1 0 100 0 i
* i 10.0.0.2 51200 100 0 i * i 10.0.0.2 51200 100 0 i
Fast Hub Failover using BGP
Hub2 Debugs

00:47:08.732: BFD-DEBUG Event: V1 FSM ld:17 handle:1 event:ECHO FAILURE state:UP (0)
00:47:08.732: BFD-DEBUG Event: notify client(BGP) IP:192.168.0.1, ld:17, handle:1, event:DOWN, cp independent failure (0)

00:47:08.744: %BGP-5-NBR_RESET: Neighbor 192.168.0.1 reset (BFD adjacency down)

00:47:08.756: %BGP-5-ADJCHANGE: neighbor 192.168.0.1 Down BFD adjacency down
00:47:08.756: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.0.1 IPv4 Unicast topology base removed from session
BFD adjacency down

00:47:08.756: BFD-DEBUG EVENT: bfd_session_destroyed, proc:BGP, handle:1 act

1 sec 00:47:08.756: BFD-DEBUG Event: V1 FSM ld:17 handle:1 event:Session delete state:DOWN (0)

00:47:08.756: RT: del 1.0.0.1 via 192.168.0.1, bgp metric [150/51200]

00:47:08.756: RT: delete subnet route to 1.0.0.1/32

00:47:09.104: %TRACK-6-STATE: 1 ip route 1.0.0.1/32 reachability Up -> Down

00:47:09.884: %TRACK-6-STATE: 2 list boolean and Down -> Up

00:47:09.888: RT: updating static 192.168.0.0/16 (0x0) : via 0.0.0.0 Nu0 1048578
00:47:09.888: RT: updating static 192.168.252.0/22 (0x0) : via 0.0.0.0 Nu0 104878
Fast Hub Failover using BGP
Spoke1 Debugs

00:47:10.025: RT: updating bgp 192.168.0.0/16 (0x0) : via 10.0.0.2 1048577

00:47:10.025: RT: closer admin distance for 192.168.0.0, flushing 1 routes
00:47:10.025: RT: add 192.168.0.0/16 via 10.0.0.2, bgp metric [200/51200]
Switch routing to Hub2 (~1.5 secs)
00:47:10.025: RT: updating bgp 192.168.252.0/22 (0x0) : via 10.0.0.2 1048577
00:47:10.025: RT: closer admin distance for 192.168.252.0, flushing 1 routes
00:47:10.025: RT: add 192.168.252.0/22 via 10.0.0.2, bgp metric [200/51200]

00:48:00.725: %BGP-3-NOTIFICATION: sent to neighbor 10.0.0.1 4/0 (hold time expired) 0 bytes
00:48:00.725: %BGP-5-NBR_RESET: Neighbor 10.0.0.1 reset (BGP Notification sent)
00:48:00.725: %BGP-5-ADJCHANGE: neighbor 10.0.0.1 Down BGP Notification sent
00:48:00.725: %BGP_SESSION-5-ADJCHANGE: neighbor 10.0.0.1 IPv4 Unicast topology base removed from session
BGP Notification sent

00:48:00.725: RT: updating bgp 192.168.0.0/24 (0x0) : via 10.0.0.2 1048577 BGP Hub1 neighbor down (60 secs)
00:48:00.725: RT: closer admin distance for 192.168.0.0, flushing 1 routes
00:48:00.725: RT: add 192.168.0.0/24 via 10.0.0.2, bgp metric [200/25600]
Fast Hub Failover using BGP (failover) (after 1-2
60 secs)

RIB 192.168.253.0/24 .1
192.168.254.0/24 R2
BGP 192.168.255.0/24
.3 192.168.0.0/24
.1 .2
Hub1 Hub2
S 1.0.0.2  Null0
C 10.0.0.0/24  Tunnel0
B 192.168.0.0/16  Null0, Tag 200
.2 C 192.168.0.0/24  Ethernet0/0
.1
B 192.168.1.0/24 [150/25600]  10.0.0.11
B 192.168.2.0/24 [150/25600]  10.0.0.12
...
B 194.168.11.0/24 [150/332800]  10.0.0.11
B 194.168.12.0/24 [150/332800]  10.0.0.12
DMVPN ...
S 192.168.252.0/22  Null0, Tag 200
10.0.0.0/24 D 192.168.253.0/24 [90/537600]  192.168.0.3
...

.11
.12
C 10.0.0.0/24  Tunnel0 C 10.0.0.0/24  Tunnel0
B 192.168.0.0/16 [200/51200]  10.0.0.2 B 192.168.0.0/16 [200/51200]  10.0.0.2
Spoke1 B [200/0]  10.0.0.1
192.168.0.0/24 [200/25600]  10.0.0.2 Spoke2 B 192.168.0.0/24 [200/0]  10.0.0.1
[200/25600]  10.0.0.2
.1 C 192.168.1.0/24  Ethernet0/0 .1 C 192.168.1.0/24  Ethernet0/0
192.168.2.0/24
D 192.168.11.0/24 [90/307200]  192.168.1.2 D 192.168.11.0/24 [90/307200]  192.168.2.2
192.168.12.0/24 192.168.1.2
192.168.1.0/24 .2 B 192.168.252.0/22 [200/51200]  10.0.0.2 192.168.2.0/24 .2 B 192.168.252.0/22 [200/51200]  10.0.0.2

*>i 192.168.0.0 10.0.0.2

10.0.0.1 25600
0 100 0i *>i 192.168.0.0 10.0.0.2
10.0.0.1 25600
0 100 0i
RS1 **>ii 10.0.0.2 51200
25600 200
100 0i RS2 **>ii 10.0.0.2 51200
25600 200
100 0i
**>ii 192.168.0.0/16 10.0.0.1
10.0.0.2 51200
0 200
100 0i **>ii 192.168.0.0/16 10.0.0.2
10.0.0.1 51200
0 200
100 0i
192.168.11.0/24 .1 *>i 10.0.0.2 51200 200 0i 192.168.12.0/24 .1 *>i 10.0.0.2 51200 200 0i
* i 192.168.252.0/22 10.0.0.1 0 100 0i * i 192.168.252.0/22 10.0.0.1 0 100 0i
*>i 10.0.0.2 51200 200 0i *>i 10.0.0.2 51200 200 0i
 16.3.1,
BFD over DMVPN 15.7(3)M, 15.6(2)S

• BFD configured on mGRE tunnel interface

• Use Echo mode
• BFD maximum probe interval increased to 10 seconds (9999 msec)
• Spoke-hub tunnel  Only Spoke sends/receives BFD probes*
* Currently both Hub and
• Spoke-spoke tunnel  Both spokes send/receive BFD probes Spoke will send/receive
• NHRP is a BFD client separate BFD probe sets
• BFD notifies NHRP when tunnel endpoint is down
• NHRP provides a registry for other applications (RP, PfR, IPsec, ...)
• Applications register with NHRP for a tunnel endpoint (peer, neighbor) address
• NHRP notifies application when tunnel endpoint is down

bfd-template single-hop DMVPN

Echo mode BFD
interval min-tx 2000 min-rx 2000 multiplier 3
2/6 second keepalive/hold
echo
!
interface Tunnel0
···
bfd template DMVPN Apply on Tunnel interface
···
BFD over DMVPN
BFD session for NHRP static peer (hub)
• If the BFD session is reporting the static peer as down NHRP will:
• Notify upper layer applications (RP, PFR, …).
• Initiate NHRP registration requests, if the peer is an NHS
• If NHRP registration reply is received, peer is up
• BFD should reflect this state
• Upper layers should have reset and re-attached to the peer
• No change in lower layer (IKE/IPsec) crypto session stayed up
• If NHRP registration reply is not received after 3 retransmissions (~15 seconds)
• Notify Lower layers (IKE/IPsec) to tear down the crypto session
• NHRP continues to send registration requests – trigger (IKE/IPsec) crypto session back up
• Eventually an NHRP registration reply is received
• The upper layer application sessions (RP and PFR) come back up
• Note, BFD session is not cleared
BFD over DMVPN
BFD session for NHRP dynamic peer (spoke)
• If the BFD session is reporting the dynamic peer as down NHRP will:
• Notify upper layer applications (RP, PFR, …)
• Notify lower layer applications (IPsec) to clear the crypto session
• Clear the BFD session, NHRP mapping and associated RIB routes
• Routing will revert back to spoke-hub-spoke
• A new spoke-spoke tunnel will be attempted if there is more data traffic
• Detect when spoke-spoke tunnel is no longer used for data packets
• Use packet count estimates to detect when only BFD probes are using tunnel
• NHRP mapping times out normally
• Clear NHRP mapping, BFD session, RIB routes and IKE/IPsec session
BFD over DMVPN
Spoke-Hub tunnel
18:13:56.096: BFD-DEBUG Event: V1 FSM ld:1 handle:2 event:DETECT TIMER EXPIRED state:UP (0)
18:13:56.096: BFD-DEBUG Event: notify client(NHRP) IP:10.0.0.1, ld:1, handle:2, event:DOWN, (0)
18:13:56.096: BFD-DEBUG Event: notify client(EIGRP) IP:10.0.0.1, ld:1, handle:2, event:DOWN, (0)
Switch routing
18:13:56.097: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.0.0.1 (Tunnel0) is down: BFD peer down notified to Hub2
18:13:56.097: RT: delete route to 192.168.0.0 via 10.0.0.1, eigrp metric [90/15360000]
18:13:56.097: RT: add 192.168.0.0/16 via 10.0.0.2, eigrp metric [90/15360015]
18:13:57.073: NHRP: Setting retrans delay to 2 for nhs dst 10.0.0.1
18:13:57.073: NHRP: Send Registration Request via Tunnel0 vrf global(0x0), packet size: 104 src: 10.0.0.11, dst: 10.0.0.1
15 sec 18:13:59.059: NHRP: Setting retrans delay to 4 for nhs dst 10.0.0.1
18:13:59.060: NHRP: Send Registration Request via Tunnel0 vrf global(0x0), packet size: 104 src: 10.0.0.11, dst: 10.0.0.1 Trigger NHRP
18:14:02.771: NHRP: Setting retrans delay to 8 for nhs dst 10.0.0.1
Registrations
18:14:02.771: NHRP: Send Registration Request via Tunnel0 vrf global(0x0), packet size: 104 src: 10.0.0.11, dst: 10.0.0.1
18:14:10.092: NHRP: Setting cache expiry for 172.17.0.1 to 1 milliseconds in cache
18:14:10.092: NHRP: Setting retrans delay to 16 for nhs dst 10.0.0.1

18:14:10.103: IKEv2:(SESSION ID = 1,SA ID = 2):Sending DELETE INFO message for IPsec SA [SPI: 0xAC54C857]
18:14:10.103: IKEv2:(SESSION ID = 1,SA ID = 2):Sending Packet [To 172.17.0.1:500/From 172.16.1.1:500/VRF i0:f0]
18:14:10.104: IKEv2:(SESSION ID = 1,SA ID = 2):Check for existing active SA
18:14:10.104: IKEv2:Searching Policy with fvrf 0, local address 172.16.1.1 Reset Crypto
18:14:10.105: IKEv2:(SESSION ID = 1,SA ID = 1):Generating IKE_SA_INIT message
18:14:10.105: IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 172.17.0.1:500/From 172.16.1.1:500/VRF i0:f0]
18:14:12.010: IKEv2:(SESSION ID = 1,SA ID = 2):Retransmitting packet
18:14:12.010: IKEv2:(SESSION ID = 1,SA ID = 2):Sending Packet [To 172.17.0.1:500/From 172.16.1.1:500/VRF i0:f0]
BFD over DMVPN
Spoke-Spoke tunnel
18:46:52.695: NHRP: Receive Traffic Indication via Tunnel0 vrf global(0x0), packet size: 96
18:46:52.705: NHRP: Send Resolution Request for dest: 192.168.12.1 to nexthop: 192.168.12.1 src: 10.0.0.11
18:46:52.784: NHRP: Receive Resolution Request via Tunnel0 vrf global(0x0), packet size: 104
18:46:52.839: %BFD-6-BFD_SESS_CREATED: bfd_session_created, neigh 10.0.0.12 proc:NHRP, idb:Tunnel0 handle:7 act
18:46:52.839: NHRP: Send Resolution Reply via Tunnel0 vrf global(0x0), packet size: 132
18:46:52.875: %BFDFSM-6-BFD_SESS_UP: BFD session ld:2 handle:7 is going UP
18:46:52.875: NHRP: Receive Resolution Reply via Tunnel0 vrf global(0x0), packet size: 132 Normal tunnel down
(no data traffic) (10 min)
18:56:52.875: %BFD-6-BFD_SESS_DESTROYED: bfd_session_destroyed, ld:2 neigh proc:NHRP, handle:7 act

19:19:04.622: NHRP: Receive Traffic Indication via Tunnel0 vrf global(0x0), packet size: 96
19:19:04.632: NHRP: Send Resolution Request for dest: 192.168.12.1 to nexthop: 192.168.12.1 using our src: 10.0.0.11
19:19:04.703: NHRP: Receive Resolution Request via Tunnel0 vrf global(0x0), packet size: 104
19:19:04.734: %BFD-6-BFD_SESS_CREATED: bfd_session_created, neigh 10.0.0.12 proc:NHRP, idb:Tunnel0 handle:7 act
19:19:04.734: NHRP: Send Resolution Reply via Tunnel0 vrf global(0x0), packet size: 132
19:19:04.771: NHRP: Receive Resolution Reply via Tunnel0 vrf global(0x0), packet size: 132
19:19:04.782: %BFDFSM-6-BFD_SESS_UP: BFD session ld:10 handle:7 is going UP

19:19:24.209: %BFDFSM-6-BFD_SESS_DOWN: BFD session ld:10 handle:7,is going Down Reason: DETECT TIMER EXPIRED
19:19:24.209: BFD-DEBUG Event: notify client(NHRP) IP:10.0.0.12, ld:10, handle:7, event:DOWN, (0)
19:19:24.211: NHRP: Calling for delete of Tunnel Endpoints (VPN: 10.0.0.12, NBMA: 172.16.2.1) Abnormal tunnel down
19:19:24.211: %BFD-6-BFD_SESS_DESTROYED: bfd_session_destroyed, ld:10 neigh proc:NHRP, handle:7 act (BFD triggered) (20 sec)
19:19:24.800: NHRP: Receive Traffic Indication via Tunnel0 vrf global(0x0), packet size: 96
 Agenda

• DMVPN Design Overview

• DMVPN General
• IWAN Specific
• NHRP Details
• NHRP Overview
• NHRP
Registrations/Resolutions/Redirects
• Recent and New Features
• Configuration, Resiliency, Routing and
Forwarding, Centralized Control
NHRP routes and routing
• Issues
• Can’t control NHRP short-cut routes on spokes
• Can’t prefer/order routes using multiple short-cut tunnels
• Can’t summarize NHRP short-cut routes like you can with RP routes
• Routing protocol limits scale of DMVPN on hubs (IoT)
• Need separate DMVPN hub router per Cloud (Transport)
• Solutions
• NHRP Route Metric control
• NHRP Route Summarization
• NHRP Route Advertisement
• Multiple Tunnel Termination (MTT) on Hub routers
Controlling NHRP routes 15.5(3)S&M,15.6(2)T

NHRP route metric control per tunnel interface

• Egress Load-balancing or Ingress traffic engineering interface Tunnel0
ip address 10.0.0.11 ...
• Peer NHS path preference used to calculate NHRP route metric ···
ip nhrp path preference 16
• Preference value is (1-255); best = 1; default = 255
···
• NHRP route metric = (2552 = 65025)/preference
• Examples: (preference = 16  metric = 4064); (preference = 255  metric = 255)
• Strict preference: (p1 > 16×p2)
• Example: (p1=32, p2=1)  2032/65025 = 1/32
• Unequal load-balancing: (p1 ≤ 16×p2)
• Example: (p1=16, p2=4)  4064/16256 = 1/4
#show ip nhrp
# show ip route nhrp ...
... 192.168.11.0/24 via 10.0.0.11
192.168.11.0/27 is subnetted, 1 subnets Tunnel0 created 00:01:46, expire 00:08:13
H 192.168.11.32 [250/4064] via 10.0.0.11, 00:00:09, Tunnel0 Type: dynamic, Flags: router rib
NBMA address: 172.16.1.1
Controlling NHRP routes 15.5(3)S, 15.5(3)M

NHRP summarization
• Current Behavior
• NHRP answers resolution request with most specific RIB network/mask
• Ability to summarize NHRP mappings and routes like RP routes
• ip nhrp summary-map { network/mask-length | network mask }
• Used in resolution responses instead of matching RIB network/mask* * NHRP summary network/mask
• Similar to a summary route for a Routing Protocol used even if more specific
than RIB network/mask
• Use Cases
• Summary of spoke subnets for NHRP resolution replies
• Fixes 1st subnet of summary route use at spoke, spoke-spoke refresh issue
• Default (0/0)  NHRP /32 resolution replies mitigation rather than static routes

#show ip nhrp 192.168.11.0

interface Tunnel0 192.168.11.0/24 via 10.0.0.11
ip nhrp summary-map 192.168.11.0/24 Tunnel0 created 00:00:07, never expire
Type: static, Flags: local
NBMA address: 172.16.1.1 (no-socket)
 NHRP Summary Map
Original Setup C 10.0.0.0/24  Tunnel0
B 192.168.0.0/16  Null0
NHRP mapping C 192.168.2.0/24  Ethernet0/0
10.0.0.11/32  172.16.1.1
10.0.0.12/32  172.16.2.2 192.168.0.1/24 C 192.168.0.0/24  Ethernet0/0
RIB Table B 192.168.1.0/24  10.0.0.11
B 192.168.2.0/24  10.0.0.12
Physical: 172.17.0.1 B 192.168.11.0/24  10.0.0.11
Tunnel0: 10.0.0.1 B 192.168.12.0/24  10.0.0.12

Physical: 172.16.2.1
Tunnel0: 10.0.0.12
Physical: 172.16.1.1
10.0.0.1/32  172.17.0.1 Tunnel0: 10.0.0.11 Spoke B
Spoke A
C 10.0.0.0/24  Tunnel0 10.0.0.1/32  172.17.0.1
B 192.168.0.0/16  10.0.0.1
C 192.168.1.0/24  Ethernet0/0 192.168.2.1/24 C 10.0.0.0/24  Tunnel0
B 192.168.11.0/24  Null0 B 192.168.0.0/16  10.0.0.1
192.168.1.1/24
D 192.168.11.0/27  192.168.1.2 C 192.168.2.0/24  Ethernet0/0
D 192.168.11.32/27  192.168.1.2 D 192.168.12.0/24  192.168.2.2
D 192.168.11.64/27  192.168.1.2
D 192.168.11.96/27  192.168.1.2
D 192.168.11.128/27  192.168.1.2
D 192.168.11.160/27  192.168.1.2
D 192.168.11.192/27  192.168.1.2
D 192.168.11.224/27  192.168.1.2
 NHRP Summary Map
Without Summary-Map C 10.0.0.0/24  Tunnel0
B 192.168.0.0/16  Null0
NHRP mapping C 192.168.2.0/24  Ethernet0/0
10.0.0.11/32  172.16.1.1
10.0.0.12/32  172.16.2.2 192.168.0.1/24 C 192.168.0.0/24  Ethernet0/0
RIB Table B 192.168.1.0/24  10.0.0.11
B 192.168.2.0/24  10.0.0.12
10.0.0.1/32  172.17.0.1 Physical: 172.17.0.1 B 192.168.11.0/24  10.0.0.11
192.168.11.0/27  172.16.1.1 (l) Tunnel0: 10.0.0.1 B 192.168.12.0/24  10.0.0.12
10.0.0.1/32  172.17.0.1
192.168.11.32/27  172.16.1.1 (l) 10.0.0.11/32  172.16.1.1
192.168.11.64/27  172.16.1.1 (l) 192.168.11.0/27  172.16.1.1
192.168.11.96/27  172.16.1.1 (l) 192.168.11.32/27  172.16.1.1
192.168.11.128/27  172.16.1.1 (l) Physical: 172.16.2.1 192.168.11.64/27  172.16.1.1
192.168.11.160/27  172.16.1.1 (l) Tunnel0: 10.0.0.12 192.168.11.96/27  172.16.1.1
Physical: 172.16.1.1
192.168.11.192/27  172.16.1.1 (l) 192.168.11.128/27  172.16.1.1
Tunnel0: 10.0.0.11 Spoke B
192.168.11.224/27  172.16.1.1 (l) 192.168.11.160/27  172.16.1.1
Spoke A 192.168.11.192/27  172.16.1.1
C 10.0.0.0/24  Tunnel0 192.168.11.224/27  172.16.1.1
B 192.168.0.0/16  10.0.0.1
C 192.168.1.0/24  Ethernet0/0 192.168.2.1/24 C 10.0.0.0/24  Tunnel0
B 192.168.11.0/24  Null0 B 192.168.0.0/16  10.0.0.1
192.168.1.1/24  Ethernet0/0
D 192.168.11.0/27  192.168.1.2 C 192.168.2.0/24
D 192.168.11.32/27  192.168.1.2 H 192.168.11.0/27  10.0.0.11
D 192.168.11.64/27  192.168.1.2 H 192.168.11.32/27  10.0.0.11
D 192.168.11.96/27  192.168.1.2 H 192.168.11.64/27  10.0.0.11
D 192.168.11.128/27  192.168.1.2 H 192.168.11.96/27  10.0.0.11
D 192.168.11.160/27  192.168.1.2 H 192.168.11.128/27  10.0.0.11
D 192.168.11.192/27  192.168.1.2 H 192.168.11.160/27  10.0.0.11
D 192.168.11.224/27  192.168.1.2 H 192.168.11.192/27  10.0.0.11
H 192.168.11.224/27  10.0.0.11
D 192.168.12.0/24  192.168.2.2
 NHRP Summary Map
With Summary-Map C 10.0.0.0/24  Tunnel0
B 192.168.0.0/16  Null0
NHRP mapping C 192.168.2.0/24  Ethernet0/0
10.0.0.11/32  172.16.1.1
10.0.0.12/32  172.16.2.2 192.168.0.1/24 C 192.168.0.0/24  Ethernet0/0
RIB Table B 192.168.1.0/24  10.0.0.11
B 192.168.2.0/24  10.0.0.12
Physical: 172.17.0.1 B 192.168.11.0/24  10.0.0.11
Tunnel0: 10.0.0.1 B 192.168.12.0/24  10.0.0.12

Physical: 172.16.2.1
Tunnel0: 10.0.0.12
10.0.0.1/32  172.17.0.1 Physical: 172.16.1.1
Tunnel0: 10.0.0.11 Spoke B  172.17.0.1
192.168.11.0/24  172.16.1.1 (s,l) 10.0.0.1/32
Spoke A 10.0.0.11/32  172.16.1.1
C 10.0.0.0/24  Tunnel0 192.168.11.0/24  172.16.1.1
B 192.168.0.0/16  10.0.0.1
C 192.168.1.0/24  Ethernet0/0 192.168.2.1/24 C 10.0.0.0/24  Tunnel0
B 192.168.11.0/24  Null0 B 192.168.0.0/16  10.0.0.1
192.168.1.1/24
D 192.168.11.0/27  192.168.1.2 C 192.168.2.0/24  Ethernet0/0
D 192.168.11.32/27  192.168.1.2 H 192.168.11.0/24  10.0.0.11
D 192.168.11.64/27  192.168.1.2 D 192.168.12.0/24  192.168.2.2
D 192.168.11.96/27  192.168.1.2
D 192.168.11.128/27  192.168.1.2
D 192.168.11.160/27  192.168.1.2
D 192.168.11.192/27  192.168.1.2
D 192.168.11.224/27  192.168.1.2
NHRP Route Advertisement
• Route advertisement between hub and spoke in NHRP registration message
• Ability to redistribute routes between NHRP and other routing protocols
• redistribute nhrp ...
• Control NHRP routing using standard ‘router nhrp ...’ CLI construct
• Not a replacement for regular routing protocols (EIGRP, BGP, ...)
• RPs handle much more complex networks
• For simple hub-spoke and spoke-spoke DMVPNs (IoT)
• 10,000s small spoke sites with one or few subnets
• 100,000s of spokes sites in hub-spoke IoT networks
• Preliminary scaling to 10,000  30,000+ spokes per hub (CSR)
Routing Protocol Features – BGP
• iBGP Local-AS (15.2(2)T, 15.1(3)S (CSCtj48063))
• Run iBGP over DMVPN
• Tunnel end-point routers may have different native BGP ASs
• Allows ‘neighbor ... local-as #’ and ‘neighbor ... remote-as #’ to be the same (iBGP)
• ’neighbor ... local-as #’ is different from local native BGP AS, ‘router bgp #’
• Almost like eBGP within the router between the native AS and the AS over DMVPN
• BGP Dynamic Neighbors to reduce configuration on hub
• Added IPv6 Dynamic Neighbor support in 16.3, 15.6(3)M
router bgp 65000
bgp listen range 10.0.0.0/24 peer-group spokes BGP Dynamic Neighbors
...
neighbor spokes peer-group
neighbor spokes remote-as 65001
neighbor spokes local-as 65001 iBGP Local-AS
...
Routing Protocol Features – EIGRP
• Equal Cost MultiPath (15.2(3)T, 15.2(1)S (CSCsj31328))
• Destination network is reachable via more than one DMVPN (mGRE tunnel)
and the ip next-hop needs to be preserved (Phase 2).
no ip next-hop-self eigrp <as> [no-ecmp-mode]

• Add-path (15.3(1)S (CSCtw86791))

• Spoke site has multiple DMVPN spoke routers
and want to be able to load-balance spoke-spoke tunnels (Phase 2).
• Requires new “named” EIGRP router configuration

router eigrp <name>

address-family ipv4 unicast autonomous-system 1
af-interface Tunnel0
no next-hop-self
add-path <paths> (<paths> = number of extra paths)
no split-horizon
...
Agenda
• DMVPN Design Overview
• DMVPN General
• IWAN Specific
• NHRP Details
• NHRP Overview
• NHRP Registrations
• NHRP Resolutions/Redirects
• Recent and New Features
• Configuration, Resiliency,
Routing and Forwarding,
Centralized Control
Centralized Control
Separating Control and Data Planes
• Issues – Converged control and data planes on hubs
• Routing Protocol scaling limits scale of DMVPN hubs
• ISP managed DMVPN – want Hubs in ISP network
• Data traffic traverses DMVPN hubs while short-cut tunnel is built
• Multicast traffic (replication*) goes through DMVPN hubs
• Solution – Separate control plane from data plane
• Separate DMVPN Control Plane Hub (CPH) and Data Plane Hub (DPH)
• Scale CPH and DPH independently of each other
• Data traffic only ever goes through DPHs never CPH
• Other central control services at CPH
• Routing Protocol (BGP, EIGRP*); Key Management (ESON*)
• ISP managed DMVPN  CPH in ISP network; DPHs in customer network
Centralized Routing and NHS
Separating the Control and Data planes
• Control Plane Hub (CPH)
• Routing – peer with DPHs and spokes
• iBGP (route-reflector), in future EIGRP (OTP route-reflector)
• NHS  NHRP registrations and resolution request processing
• Future:
• Smart data plane hub selection pushed to spokes
• Optional Centralized Key Server (ESON)
• Data Plane Hub (DPH)
• Provide data path for spoke-hub-spoke
• Routing – peer with CPH
• Advertise network and/or regional summaries to CPH
• NHS  NHRP redirect; NHRP registrations (for no-drop); Backup CPH
Centralized Routing and NHS
192.168.0.1/24
CPH
DPH

Phy: 172.31.0.1 Phy: 172.17.0.1

Tu0: 10.0.0.254 Tu0: 10.0.0.1

Spoke B

Spoke A Phy: 172.16.2.1

Phy: 172.16.1.1
Tu0: 10.0.0.11 Tu0: 10.0.0.12 192.168.2.1/24
192.168.1.1/24

= Control Plane tunnels

= Data Plane tunnels
Centralized Routing and NHS
CPH Configuration
interface Tunnel0
ip address 10.0.0.254 255.255.255.0
ip nhrp authentication test
‘no ip nhrp redirect’ ip nhrp network-id 100000
tunnel source Serial2/0
tunnel mode gre multipoint
tunnel key 100000
DPH is a regular neighbor tunnel protection ipsec profile DMVPN
end
!
router bgp 1
Spokes are dynamic and bgp listen range 10.0.0.0/24 peer-group spokes
neighbor spokes peer-group
route-reflector-clients neighbor spokes remote-as 1
neighbor spokes timers 20 60 ip community-list 1 permit 1:255
neighbor 10.0.0.1 remote-as 1 ip bgp-community new-format
Only send summary to Spokes neighbor 10.0.0.1 timers 20 60 !
! route-map SUMMARY-ONLY permit 10
Send everything to DPH address-family ipv4 match community-list 1
neighbor spokes route-reflector-client
neighbor spokes route-map SUMMARY-ONLY out
exit-address-family
Don‘t reset next-hop !
Centralized Routing and NHS
DPH Configuration
interface Tunnel0
ip address 10.0.0.1 255.255.255.0
ip nhrp authentication test
CPH is NHS and force ip nhrp network-id 100000
NHRP resolution requests ip nhrp nhs 10.0.0.254 nbma 172.31.0.1 multicast
no ip nhrp send-routed
ip nhrp redirect
Send NHRP redirects to spokes tunnel source Serial2/0
tunnel mode gre multipoint
to trigger spoke-spoke tunnels tunnel key 100000
tunnel protection ipsec profile DMVPN
end
!
router bgp 1
bgp log-neighbor-changes
Neighbor only with CPH neighbor 10.0.0.254 remote-as 1
neighbor 10.0.0.254 timers 20 60
! ip bgp-community new-format
address-family ipv4 route-map SUMMARY-CMNTY permit 10
Create Summary; bgp redistribute-internal set community 1:255
Set Community; network 192.168.0.0
aggregate-address 192.168.0.0 255.255.0.0 attribute-map SUMMARY-CMNTY
Send to CPH neighbor 10.0.0.254 activate
neighbor 10.0.0.254 send-community
Set next-hop to Self neighbor 10.0.0.254 next-hop-self all
exit-address-family
!
Centralized Routing and NHS
Spoke Configuration
interface Tunnel0
ip address 10.0.0.11 255.255.255.0
ip nhrp authentication test
ip nhrp network-id 100000
CPH is main NHS ip nhrp nhs 10.0.0.254 nbma 172.31.0.1 multicast
DPH is secondary NHS ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast priority 16
NHRP Res. Req. via CPH no ip nhrp send-routed
tunnel source Serial1/0
tunnel mode gre multipoint
tunnel key 100000
tunnel protection ipsec profile DMVPN
!
router bgp 1
bgp log-neighbor-changes
Neighbor only with CPH neighbor 10.0.0.254 remote-as 1
neighbor 10.0.0.254 timers 20 60
!
address-family ipv4
bgp redistribute-internal
network 192.168.1.0
neighbor 10.0.0.254 activate
Set next-hop to Self neighbor 10.0.0.254 next-hop-self all
exit-address-family
!
 Centralized Routing and NHS NHRP mapping

NHRP Registration and Routing Routing Table

192.168.0.1/24
10.0.0.1  172.17.0.1 CPH 10.0.0.11  172.16.1.1
10.0.0.11  172.16.1.1 DPH 10.0.0.12  172.16.2.1
10.0.0.12  172.16.2.1 10.0.0.254  172.31.0.1
Phy: 172.17.0.1
192.168.0.0/16  10.0.0.1 192.168.0.0/16  Null0

Tu0: 10.0.0.254
Phy: 172.31.0.1
Tu0: 10.0.0.1
192.168.0.0/24  10.0.0.1 192.168.0.0/24  Ethernet0/0
192.168.1.0/24  10.0.0.11 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12 192.168.2.0/24  10.0.0.12

Spoke B

Spoke A Phy: 172.16.2.1

Phy: 172.16.1.1
Tu0: 10.0.0.11 Tu0: 10.0.0.12 192.168.2.1/24
192.168.1.1/24

10.0.0.1  172.17.0.1
10.0.0.1  172.17.0.1 10.0.0.254  172.31.0.1
10.0.0.254  172.31.0.1 = Control Plane tunnels
= Data Plane tunnels 192.168.0.0/16  10.0.0.1
192.168.0.0/16  10.0.0.1 192.168.2.0/24  Ethernet0/0
192.168.1.0/24  Ethernet0/0
 Centralized Routing and NHS NHRP mapping

Data packets via DPH Routing Table

10.0.0.1  172.17.0.1 CPH 192.168.0.1/24 DPH 10.0.0.11  172.16.1.1

10.0.0.11  172.16.1.1 10.0.0.12  172.16.2.1
10.0.0.12  172.16.2.1 10.0.0.254  172.31.0.1
Phy: 172.17.0.1
192.168.0.0/16  10.0.0.1 192.168.0.0/16  Null0

Tu0: 10.0.0.254
Phy: 172.31.0.1
Tu0: 10.0.0.1
192.168.0.0/24  10.0.0.1 192.168.0.0/24  Ethernet0/0
192.168.1.0/24  10.0.0.11 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12 192.168.2.0/24  10.0.0.12

Spoke B
Spoke A
Phy: 172.16.2.1 192.168.2.1/24
192.168.1.1/24 Phy: 172.16.1.1
Tu0: 10.0.0.12
10.0.0.1  172.17.0.1 (16)
Tu0: 10.0.0.11 10.0.0.1  172.17.0.1 (16)

10.0.0.254  172.31.0.1 10.0.0.254  172.31.0.1

= Control Plane tunnels
192.168.2.1  ???
= Data Plane tunnels
192.168.0.0/16  10.0.0.1 192.168.0.0/16  10.0.0.1
192.168.1.0/24  Ethernet0/0
192.168.2.0/24  Ethernet0/0
 Centralized Routing and NHS NHRP mapping

Control packets via CPH Routing Table

10.0.0.1  172.17.0.1 CPH 192.168.0.1/24 DPH 10.0.0.11  172.16.1.1

10.0.0.11  172.16.1.1 10.0.0.12  172.16.2.1
10.0.0.12  172.16.2.1 10.0.0.254  172.31.0.1
Phy: 172.17.0.1
192.168.0.0/16  10.0.0.1 192.168.0.0/16  Null0

Tu0: 10.0.0.254
Phy: 172.31.0.1
Tu0: 10.0.0.1
192.168.0.0/24  10.0.0.1 192.168.0.0/24  Ethernet0/0
192.168.1.0/24  10.0.0.11 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12 192.168.2.0/24  10.0.0.12

Spoke B
Spoke A
Phy: 172.16.2.1 192.168.2.1/24
192.168.1.1/24 Phy: 172.16.1.1
Tu0: 10.0.0.12
10.0.0.1/32  172.17.0.1 (16)
Tu0: 10.0.0.11 10.0.0.1/32  172.17.0.1 (16)
10.0.0.12/32  172.16.2.1 10.0.0.11/32  172.16.1.1
10.0.0.254/32  172.31.0.1 10.0.0.254/32  172.31.0.1
192.168.2.0/24  ???
192.168.2.1/32 172.16.2.1
= Control Plane tunnels 192.168.1.0/24  172.16.1.1
192.168.1.1/32
= Data Plane tunnels
192.168.0.0/16  10.0.0.1 = Dynamic spoke-spoke tunnel
192.168.0.0/16  10.0.0.1
192.168.1.0/24  Ethernet0/0 192.168.1.0/24  10.0.0.11
192.168.2.0/24  10.0.0.12 192.168.2.0/24  Ethernet0/0
Centralized Routing and NHS
• Summary
• Separation of Control and Data Planes
• ISP Managed DMVPN Service (CPH in ISP network, DPHs in customer network)
• Separate scaling for CPH (RP peers) and DPH (Encryption throughput)
• Uses the same DMVPN/mGRE infrastructure
• Main NHS at CPH, Natural backup NHS at DPH
• Future
• Download from CPH to spokes, NHS summary-map configuration for DPH
{ip | ipv6} nhrp summary-map {all-routed [nbma] | prefix [[nbma [preference pref]]} [multicast] [resolve]
[match {group group_name | geo-location geo-location | topo-location topo-location | attribute attr_type
attr_value}]
• All-routed: RP advertises summary  temporary map to use NBMA as DPH
• Prefix: Default/summary prefix passed to spokes
• Resolve: Prefix is specified, but not NBMA  forces resolution for all packets; hub-less model
• Match: Push different summary maps depending on attributes from spoke registration to CPH
Centralized Control
Extensible Security for Overlay Network (ESON)
• A Centralized Key Server Solution with pairwise key capability.
• Centralized management of policy & pairwise and group keys for IPsec overlay VPNs
• Leverages GetVPN control plane (GDOI/G-IKEv2) as underlying infrastructure
• GM-KS: G-IKEv2 Registrations for initial pull of policy & keying material
• KS-GM: KS pushes periodic rekeys (unicast/multicast)
• KS-KS: Multiple KSs for redundancy using COOP over IKEv2

• IKEv2 is not used between GMs (no Diffee Hellman (DH))

• Peer Introduction Protocol (PIP) is lightweight control plane (2 messages)
between GMs
• GM-GM: Exchanges cryptographic identities and nonces for pair-wise key generation
and detects NAT between Peers
DMVPN with ESON
G-IKEv2 based centralized management of pairwise and group IPsec session keys
KS1(DC) COOP *(IKE) KS2(DR) Control Plane
TEK: Traffic Encryption Key
KEK: Key Encryption Key
GM – KS
• G-IKEv2 (KEK, TEK and key material
from KS)
GM – GM
• PIP*(TEK): (Encrypted with TEK key)

Control Plane Redundancy

Internet/WAN
KS – KS
• COOP over IKEv2

Data Plane
PIP *(TEK)
GM – GM
• IPsec *(GM1-GM2 Pairwise key)
IPsec
GM1 / DMVPN *(GM1-GM2 pairwise key) GM2 / DMVPN
Hub/Spoke Hub/Spoke Data Plane Redundancy
Group Keys: TEK, KEK Group Keys: TEK, KEK GM – GM
GM1 Key Material GM2 Key Material • Redundant Hubs
GM1 Identity GM2 Identity
GM1-GM2 Pairwise key GM1-GM2 Pairwise key
DMVPN with ESON - Value Proposition
• Centralized key server and management
• Centralized authentication & authorization of GMs (DMVPN Hub/spoke)
• Centralized management of crypto policy and keys
• Crypto Control-plane/Data-plane separation, no IKEv2 or DH between GMs
• Easier to manage
• Elasticity of scale; Reduced setup latency; Virtualized Key Server
• Faster & more effective removal of compromised GMs
• Better enforcement of enterprise security policy & centralized trust management
• Allows varying key management schemes
• Group keys: Control Plane (PIP); Data Plane (Native Multicast)
• Pairwise keys for better security – Data Plane (Unicast)
• Various rekey policies/schemes are possible
IKEv2 with DMPVN
• DMVPN works with ISAKMP (IKEv1) and/or IKEv2
• Transparent to DMVPN
• Node can be responder for both ISAKMP and IKEv2
• Both ISAKMP and IKEv2 are configured.
• Node can be Initiator for either ISAKMP or IKEv2 not both
• Configure under the ‘crypto ipsec profile ...’
crypto isakmp policy 2 crypto isakmp key cisco123 address 0.0.0.0 0.0.0.0
encr aes
crypto ipsec transform-set DMVPN esp-aes esp-sha-hmac
authentication pre-share
mode transport [require]
group 2
crypto ikev2 keyring DMVPN crypto ipsec profile DMVPN
peer DMVPN set transform-set DMVPN With  initiate IKEv2
address 0.0.0.0 0.0.0.0 set ikev2-profile DMVPN Without  initiate IKEv1
pre-shared-key cisco123
crypto ikev2 profile DMVPN interface Tunnel0
match identity remote address 0.0.0.0 ...
authentication local pre-share tunnel protection ipsec profile DMVPN
authentication remote pre-share
keyring DMVPN
Per-tunnel QoS
(hubspoke) 12.4(22)T; (spokehub, spokespoke) 15.5(1)S,T
interface Tunnel0
• QoS per tunnel on hub and spokes nhrp group name
…
• Dynamically select Hierarchical (parent/child) QoS Policy nhrp map group name1 service-policy output qos-template1
nhrp map group name2 service-policy output qos-template2
• Receiving Node: Configure NHRP group name on tunnel …
• Sending Node: Configure QoS template policies; Map NHRP group name to QoS template policy
• Nodes with same NHRP group name are mapped to separate instances of QoS policy
• Same policy used for both IPv4 and IPv6
• QoS policy applied at outbound physical interface
• Classification done before GRE encapsulation by tunnel
• ACL matches against Data IP packet
• Don’t configure ‘qos pre-classify’ on tunnel interface
• Shaping/policing done on physical after IPsec encryption
• On physical may have separate aggregate QoS policy
• With only a class-default shaper (15.2(2)T,S)
• CPU intensive; can reduce hub scaling by about 50% on software forwarding platforms
Per-tunnel QoS – Configurations
interface Tunnel0
class-map match-all typeA_voice
Hub and ip address 10.0.0.1 255.255.255.0 Hub
…
match access-group 100 Spokes ip nhrp map multicast dynamic
class-map match-all typeB_voice
nhrp group typeB
match access-group 100
class-map match-all typeA_Routing …
nhrp map group typeA service-policy output typeA_parent
match ip precedence 6
class-map match-all typeB_Routing nhrp map group typeB service-policy output typeB_parent
…
match ip precedence 6
ip nhrp redirect
…
policy-map typeA
class typeA_voice interface Tunnel0
priority 1000 ip address 10.0.0.[11,13] 255.255.255.0 Spoke1,3
class typeA_Routing …
bandwidth percent 20 nhrp group typeA
…
policy-map typeB nhrp map group typeA service-policy output typeA_parent
class typeB_voice nhrp map group typeB service-policy output typeB_parent
priority percent 20 …
class typeB_Routing ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
bandwidth percent 10 …
policy-map typeA_parent interface Tunnel0
class class-default ip address 10.0.0.12 255.255.255.0 Spoke2
shape average 3000000 …
service-policy typeA nhrp group typeB
…
policy-map typeB_parent nhrp map group typeA service-policy output typeA_parent
class class-default nhrp map group typeB service-policy output typeB_parent
shape average 2000000 …
service-policy typeB ip nhrp nhs 10.0.0.1 nbma 172.17.0.1 multicast
…
Per-tunnel QoS – QoS Output on Hub
Hub#show ip nhrp Hub#show policy-map multipoint tunnel 0 <spoke> output
10.0.0.11/32 via 10.0.0.11 Interface Tunnel0  172.16.1.1
Tunnel0 created 21:24:03, expire 00:04:01 Service-policy output: typeA_parent
Type: dynamic, Flags: unique registered Class-map: class-default (match-any)
NBMA address: 172.16.1.1 19734 packets, 6667163 bytes
Group: typeA shape (average) cir 3000000, bc 12000, be 12000
10.0.0.12/32 via 10.0.0.12
Tunnel0 created 21:22:33, expire 00:05:30 Service-policy : typeA
Type: dynamic, Flags: unique registered Class-map: typeA_voice (match-all) 3737 packets, 4274636 bytes
NBMA address: 172.16.2.1 Class-map: typeA_Routing (match-all) 14424 packets, 1269312 bytes
Group: typeB Class-map: class-default (match-any) 1573 packets, 1123215 bytes
10.0.0.13/32 via 10.0.0.13 Interface Tunnel0  172.16.2.1
Tunnel0 created 00:09:04, expire 00:04:05 Service-policy output: typeB_parent
Type: dynamic, Flags: unique registered Class-map: class-default (match-any)
NBMA address: 172.16.3.1 11420 packets, 1076898 bytes
Group: typeA shape (average) cir 2000000, bc 8000, be 8000
Hub#show ip nhrp group-map Service-policy : typeB
Class-map: typeB_voice (match-all) 1005 packets, 128640 bytes
Interface: Tunnel0 Class-map: typeB_Routing (match-all) 10001 packets, 880088 bytes
NHRP group: typeA Class-map: class-default (match-any) 414 packets, 68170 bytes
QoS policy: typeA_parent
Tunnels using the QoS policy: Interface Tunnel0  172.16.3.1
Tunnel destination overlay/transport address Service-policy output: typeA_parent
10.0.0.11/172.16.1.1 Class-map: class-default (match-any)
10.0.0.13/172.16.3.1 5458 packets, 4783903 bytes
NHRP group: typeB shape (average) cir 3000000, bc 12000, be 12000
QoS policy: typeB_parent Service-policy : typeA
Tunnels using the QoS policy: Class-map: typeA_voice (match-all) 4914 packets, 4734392 bytes
Tunnel destination overlay/transport address Class-map: typeA_Routing (match-all) 523 packets, 46004 bytes
10.0.0.12/172.16.2.1 Class-map: class-default (match-any) 21 packets, 14995 bytes
Thank you