Implementing Next

Generation Performance
Routing – PfRv3
Jean-Marc Barozet – Technical Leader, IWAN
BRKRST-2362
Agenda
• IWAN Introduction
• PfR Principles
• Domain Discovery
• Performance Monitoring
• Enterprise Deployment
• IWAN Management
• Key Takeaways
Cisco Intelligent WAN
Solution Components
Unified
Private
Branch Cloud
MPLS
Virtual
Private
Cloud
3G/4G-LTE

Internet Public
Cloud

Management Automation

Transport Intelligent Application Secure

Independent Path Control Optimization Connectivity

Simplified Application Enhanced Application Comprehensive

Hybrid WAN Aware Routing Visibility and Performance Threat Defense
 IWAN: Intelligent Path Control
Leveraging the Internet

Voice/Video/Critical take
the best delay, jitter, and/or
loss path

MPLS
Private Cloud

Branch Virtual Private

Cloud
Other traffic is load Internet
balanced to maximize
bandwidth
Voice/Video/Critical will be
• PfR monitors network performance and routes applications rerouted if the current path
based on application performance policies degrades below policy thresholds
• PfR load balances traffic based upon link utilization levels
to efficiently utilize all available WAN bandwidth
Cisco Intelligent WAN
Solution Components for SPs

Hybrid WAN Intelligent Application Secure

Path Control Optimization Connectivity
Application-centric Design Load Balancing Application Visibility Scalable, Strong Encryption
Common Operational Model Policy-Based Path Selection App Acceleration App-Aware Threat Defense
Deployment Flexibility Network Availability Intelligent Caching Cloud Web Security
SP-IWAN: Intelligent Path Control
Leveraging Multiple Paths with Different SLA

INET SP Cloud
MC1
Services
SG
PfR BR1
PfR
path green path-id 1
MC/BR PE
PE
path blue path-id 2
MPLS
Branch PE BR2
PE PfR

Transit
• Developing IWAN Within the SP’s
• IWAN As a Service
• Delivering Value Within the Service Provider Network
 Can the Internet Deliver Enterprise Apps?
Verizon Booth - Insights from Client Designs and Lab Testing
Network Professional Services
Hybrid WAN Designs with Cisco
IWAN Opportunities Considerations

• Application-level path selection • Some applications (esp “as a

• MPLS + Internet service” comms apps) may
improves utilization and
resiliency of hybrid WAN require split tunneling /
centralized provisioning
• MPLS + MPLS • Beyond MPLS + Internet, IWAN
also enables other architecture • Backhaul or breakout? Internet at
designs such as dual-MPLS branches changes security
• Internet + Internet • Economic justification for
requirements
increased Internet bandwidth at • Centralized policy orchestration
branch offices enables new requires unified global QoS
• MPLS + Wireless or Satellite options standard
• Always-on transport
• MPLS Cloud Gateway + requirements and wireless
Internet for Cloud Diversity network design parameters

Getting Started with IWAN? www.verizonciscocollaboration.com/HybridWANflightplan

IWAN Layers
Intelligent Path
AVC PfR QoS Selection

Overlay routing
Overlay Routing Protocol (BGP, EIGRP) over tunnels

Transport Independent Design (DMVPN) Transport Overlay

ZBFW
MPLS Routing Internet Routing CWS
Infrastructure Routing
Why a Transport Independent Design?
Hybrid Internet MSP

Central Central Central

DMVPN DMVPN

MPLS
MPLS INET INET INET
ezVPN

eBGP EIGRP EIGRP EIGRP eBGP INET

Branch Branch Branch

Multiple routing domains, Multiple access technologies, multiple paths

Transport Independent Design – Overlay Model
IWAN

• Simplified configuration Central

iBGP iBGP
• Active/Active WAN Paths
• One Overlay – DMVPN
DMVPN1 DMVPN2
• One WAN Routing Domain – BGP, EIGRP
iBGP iBGP

Branch

Consistent VPN Overlay Enables Security Across Transition

Intelligent Path Control
Performance Routing Evolution
Today

IWAN 2.0

PfRv3
PfRv2 • Centralized provisioning PfRv3
PfR/OER • Policy simplification • AVC Infrastructure
• Multiple Data Centers
• App Path Selection • VRF Awareness
• Internet Edge • Multiple Next Hop per
• Blackout ~6s • Blackout ~ 1s DMVPN cloud
• Basic WAN
• Brownout ~9s • Brownout ~ 2s
• Provisioning per
site per policy • Scale 500 sites • Scale 2000 sites
• 1000s of lines of • 10s of lines of config • Hub config only
config
Supporting Advanced Topology
DC1 DCI DC2
IWAN POP1 WAN Core IWAN POP2

MC MC

BR1 BR2 BR3 BR4 BR5 BR6 BR7 BR8

10.8.0.0/16
10.9.0.0/16 10.8.0.0/16
10.9.0.0/16

• Support for multiple BRs per cloud DMVPN1 DMVPN2

MPLS INET
• Horizontal scaling
• Support for Multiple POPs
• Different Prefix
• Common Prefix R10 R11 R12 R13
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Horizontal Scaling Architecture
• Requirements
IWAN POP
• Multiple DMVPN Hubs per cloud for
redundancy and scaling MC

• HA
- If the current exit/channel to a remote site fails,
BR1 BR2
converge over to an alternate exit/channel on
the same (DMVPN1) network. Else, converge
over to the alternate (DMVPN2) network.
• Scale • Multiple path to
- Distribute traffic across multiple BRs/exits on a the same DMVPN
single (DMVPN) to utilize all WAN and router • Multiple next hops
capacity. in the same
- Convergence across hubs/pops should only DMVPN DMVPN1 DMVPN1
occur when all exits/channels
in a hub/pop fail or reach max-bw limits.
Multiple POPs – Common Prefixes
• Requirements: DC1 DC2
DCI
– 2 (or more) Transit Sites advertise the very WAN Core
IWAN POP1 IWAN POP2
same set of prefixes
– Datacenter may not be collocated with the MC MC
Transit Sites 10.8.0.0/16 10.8.0.0/16

– DCs/DMZs are reachable across the WAN

Core for each Transit Site BR1 BR3 BR5 BR7

– Branches can access any DC or DMZ across

either POP(hub). And, DC/DMZs can reach
any branch across multiple Transit Sites
(hubs).
– Multiple BRs per DMVPN per site may be
required for crypto and bandwidth horizontal
scaling DMVPN1 DMVPN2 DMVPN1 DMVPN2
PfRv3 Principles
Define IWAN Traffic Policies
Admin Policies Performance Policies
 Preferred Path  Delay threshold
DSCP
Voice Application
 Fallback Path  Loss threshold
 Jitter threshold

 Preferred Path  Delay threshold

DSCP
Interactive Video Application
 Fallback Path  Loss threshold
 Jitter threshold

 Preferred Path  Delay threshold

DSCP
Critical Data Application
 Fallback Path  Loss threshold

 N/A  Load Balancing

Default Default class
 DC1 DCn
Performance Routing
DCI
IWAN POP1 WAN Core IWAN POP2
Define your Traffic Policy
MC1 MC2
 Centralized on a Domain Controller
 Application or DSCP based Policy
 Preferred Path
 Load Balancing BR1 BR2 BR3 BR4
 Performance thresholds (loss, delay and Jitter)

Performance Measurement
 Learn the flows and monitor performance
DMVPN DMVPN
 Passive Monitoring with Unified Performance MPLS INET
Monitor on the CPE, aka Border Routers
 Smart Probing
 Reports to a local Master Controller

Path Control
MC/BR MC/BR MC/BR BR
 Decision on the local Master Controller
 Path enforcement on the Border Routers Branch Branch Branch
 Routing tables unchanged
Performance Monitoring
Passive Monitoring
CPE12 SITE2
Dual CPE
MPLS

CPE1 CPE11
SITE1

CPE2 SITE3
Single CPE
INET
CPE10

Bandwidth on egress
Per Traffic Class Performance Monitor
(dest-prefix, DSCP, AppName) • Collect Performance Metrics
• Per Channel
- Per DSCP
- Per Source and Destination Site
- Per Interface
Performance Monitoring
Smart Probing
CPE12 SITE2
Dual CPE
MPLS

CPE1 CPE11
SITE1

CPE2 SITE3
Single CPE
INET
CPE10

Integrated Smart Probes Performance Monitor

• Traffic driven – intelligent on/off • Collect Performance Metrics
• Site to site and per DSCP • Per Channel
- Per DSCP
- Per Source and Destination Site
- Per Interface
Performance Violation

CPE12
SITE2
MPLS Dual CPE

CPE1 CPE11
SITE1

CPE2
SITE3
INET
CPE10 Single CPE

ALERT
• From Destination site
• Sent to source site
• Loss, delay, jitter, unreachable
Policy Decision

CPE12
SITE2
MPLS Dual CPE

CPE1 CPE11
SITE1

CPE2
SITE3
INET
CPE10 Single CPE

• Reroute Traffic to a Secondary Path

Domain Discovery
 DC1 DCn
IWAN Domain
DCI
IWAN POP1 WAN Core IWAN POP2
• A collection of sites sharing the
same set of policies MC1
Transit Transit
MC2

• Each site runs Performance

Routing components BR1 BR2 BR3 BR4

• They exchange services through

the Enterprise Domain Peering
framework IWAN
Domain
DMVPN DMVPN
• Centralized configuration from a
Controller
MPLS INET
Domain Controller
• Overlay network per Transport
for flexibility and simplification
MC/BR MC/BR MC/BR BR

Branch Branch Branch

 PfR Components
MC1 MC2

The Decision Maker: Master Controller (MC)

 Apply policy, verification, reporting BR1 BR2 BR3 BR4
 No packet forwarding/ inspection required
 Standalone of combined with a BR
 VRF Aware

DMVPN DMVPN
MPLS INET

The Forwarding Path: Border Router (BR)

 Gain network visibility in forwarding path (Learn, measure)
 Enforce MC’s decision (path enforcement)
 VRF aware MC/BR MC/BR MC/BR BR
PfR Sites POP1 - TRANSIT
Site ID = 10.8.3.3
POP2 - TRANSIT
Site ID = 10.9.3.3

Transit Sites MC1 MC2

 Enterprise POPs or Hubs

 Transit to DC or spoke to spoke
BR1 BR2 BR3 BR4

Branch Sites
 Stub
DMVPN DMVPN
MPLS INET

• Site Definition:
– Controlled by a local Master Controller (MC)
– Site ID – the IP address of the MC loopback
– One/Multiple BRs MC/BR MC/BR MC/BR BR

– Each BR one/multiple links BRANCH SITE 10.1.10.0/24 10.1.11.0/24

10.1.12.0/24
Site10 10.1.13.0/24
Site ID = 10.2.10.10
Hub Master Controller
HUB SITE TRANSIT SITE
Site ID = 10.8.3.3 Site ID = 10.9.3.3

• One of the MC is assigned the Domain DC/MC1

Hub MC
MC2
Transit MC

Controller (DC) role POP-ID 0 POP-ID 1

– DC + MC = Hub Master Controller

BR1 BR2 BR3 BR4
• Central point of provisioning for Domain
policies
• Each POP is allocated an unique POP-ID
in the entire domain. DMVPN
MPLS
DMVPN
INET
– Hub MC POP-ID = 0

MC/BR MC/BR MC/BR BR

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24
BRANCH SITE 10.1.13.0/24
Site10
Site ID = 10.2.10.10
 IOS-XE 3.15
IOS 15.5(2)T

Transit Master Controller

HUB SITE TRANSIT SITE
Site ID = 10.8.3.3 Site ID = 10.9.3.3
• Introduce “Transit Master Controller" concept Hub MC Transit MC
for the 2nd Transit site MC1 MC2
POP-ID 0 POP-ID 1
• Behaves like a Hub MC without provisioning
BR1 BR2 BR3 BR4
• Each POP is allocated an unique POP-ID in
the entire domain

DMVPN DMVPN
MPLS INET

MC/BR MC/BR MC/BR BR

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Branch Sites HUB SITE
Site ID = 10.8.3.3
TRANSIT SITE
Site ID = 10.9.3.3

Hub MC Transit MC
• Hub MC listening for incoming
Policies MC1 MC2
Monitors

requests
• Branch MC connects to Hub MC BR1 BR1 BR3 BR4

• Service Exchange MC Peering

– Timers
DMVPN DMVPN
– Policies and Monitor configurations MPLS INET

– Site Prefixes

MC/BR MC/BR MC/BR BR

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
WAN Interface Discovery HUB SITE
Site ID = 10.8.3.3
TRANSIT SITE
Site ID = 10.9.3.3

• Transit BRs have path names manually MC1

Hub MC
MC2
Transit MC

defined, ie MPLS and INET POP-ID 0 POP-ID 1

• Transit BRs send Discovery Packet with path BR1 BR1 BR3 BR4
names from to all discovered sites Path MPLS Path INET Path MPLS Path INET
• Path Discovery from the Hub Border Routers Path-id 1 Path-id 2 Path-id 1 Path-id 2

DMVPN DMVPN
MPLS INET

WAN Path is detected on the branch

- Path Name
- POP-ID
MC/BR MC/BR MC/BR BR
- Path-Id
- DSCP 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
WAN Interface – Performance Monitors
• Apply 3 Performance Monitors instances (PMI)
over external interfaces
Monitor1 – Site Prefix Learning (egress direction)
•
• Monitor2 – Aggregate Bandwidth per Traffic Class
(egress direction)
1 2 3 1 2 3
• Monitor3 – Performance measurements (ingress
direction) BR
• Creates a Channel (see later)
Site Prefix Discovery
TRANSIT SITE

• Every MC in the domain owns a Site Hub MC

10.8.3.3/32
MC1
Prefix database
• Gives the mapping between site and BR1 BR2
prefixes
• 2 options:
– Static
MPLS INET
– Automatic Learning

R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Site Prefix – Automatic Learning
IWAN POP1

Hub MC
10.8.3.3/32
• Source Prefix and Mask collected from MC1

Performance Monitor SAF - Site 10

10.1.10.0/24
BR1 BR2
• Monitor interval is 30 sec
• BR send to its local MC
• MC send information to all peers via Peering MPLS INET

SAF - Site 10
10.1.10.0/24 SAF - Site 10 SAF - Site 10
Source Destination DSCP App
1 10.1.10.0/24 10.1.10.0/24
10.1.10.200 10.8.1.200 AF41 AppXY
R10 R11 R12 R13

10.1.12.0/24
R10 Site-Pfx Mask 10.1.10.0/24 10.1.11.0/24
MC 10.1.13.0/24
10.1.10.0 /24
Site Prefix Discovery
TRANSIT SITE

Hub MC
10.8.3.3/32
MC1

BR1 BR2

Site Prefix List

Hub 10.8.0.0/16
R10 10.1.10.0/24
MPLS INET
R11 10.1.11.0/24
R12 10.1.12.0/24
R12 10.1.13.0/24
R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
 IOS-XE 3.15
IOS 15.5(2)T

Shared Prefixes (M)

HUB SITE TRANSIT SITE
Site ID = 10.8.3.3 Site ID = 10.9.3.3
• Prefix (10.8.0.0/16 in this example) can
belong to multiple Sites. Hub MC Transit MC
MC1 MC2
• Prefix associated with a list of site-ids POP ID 0 POP ID 1

• Marked with 'M' flag now in the Site Prefix

BR1 BR2 BR3 BR4
Database
• A TC may be associated with more than 1 10.8.0.0/16 10.8.0.0/16
site
DMVPN DMVPN
MPLS INET

SITE-ID PREFIXES FLAGS

10.8.3.3 10.8.0.0/16 S,C,M
R10
10.9.3.3 10.8.0.0/16 S,C,M
MC/BR

10.1.10.0/24
Performance Monitoring
What is a Traffic Class? IWAN POP

DSCP Based Policies

MC1 Traffic with EF, AF41, AF31 and 0
Next-
Prefix DSCP AppID Dest Site
Hop
10.1.11.0/24 EF N/A Site 11 ? BR1 BR2
10.1.11.0/24 AF41 N/A Site 11 ?
10.1.11.0/24 AF31 N/A Site 11 ?
10.1.11.0/24 0 N/A Site 11 ?
10.1.10.0/24 EF N/A Site 10 ?
10.1.10.0/24 AF41 N/A Site 10 ?
10.1.10.0/24 AF31 N/A Site 10 ? MPLS INET
10.1.10.0/24 0 N/A Site 10 ?
10.1.12.0/24 EF N/A Site 10 ?
10.1.12.0/24 AF41 N/A Site 10 ?
10.1.12.0/24 AF31 N/A Site 10 ?
10.1.12.0/24 0 N/A Site 10 ?
R10 R11 R12 R13

Traffic Class 10.1.10.0/24 10.1.11.0/24 10.1.12.0/24

 Destination Prefix
 DSCP Value
 Application (N/A when DSCP policies used)
What is a Traffic Class? IWAN POP

Application based Policies

MC1 Traffic with EF, AF41, AF31 and 0
Prefix DSCP AppID Dest Site Next-Hop App1, App2, etc
10.1.11.0/24 EF N/A Site 11 ?
10.1.11.0/24 AF41 App1 Site 11 ? BR1 BR2
10.1.11.0/24 AF41 App2 Site 11 ?
10.1.11.0/24 AF31 N/A Site 11 ?
10.1.11.0/24 0 N/A Site 11 ?
10.1.10.0/24 EF N/A Site 10 ?
10.1.10.0/24 AF41 N/A Site 10 ?
10.1.10.0/24 AF31 N/A Site 10 ? MPLS INET
10.1.10.0/24 0 N/A Site 10 ?
10.1.12.0/24 EF N/A Site 10 ?
10.1.12.0/24 AF41 N/A Site 10 ?
10.1.12.0/24 AF31 N/A Site 10 ?
10.1.12.0/24 0 N/A Site 10 ?
R10 R11 R12 R13

Traffic Class 10.1.10.0/24 10.1.11.0/24 10.1.12.0/24

 Destination Prefix
 DSCP Value
 Application (N/A when DSCP policies used)
 Source Site: Collecting TC Bandwidth
IWAN POP

Hub MC
10.8.3.3/32
MC1 Dst-Site-Pfx App DSCP Dst-Site- State BW BR Exit MC1
Id
10.1.10.0 APP1 AF41 10.2.10.1 CN 24 BR1 Tu10
0
BR1 BR2
Source Destination DSCP App 2

10.8.1.200 10.1.10.200 AF41 APP1

• Traffic going outside a source site DMVPN DMVPN

MPLS INET
• Initially based on Routing Information
• Captured by a Performance Monitor on the
external interface on egress
• BR reports to its local MC R10 R11 R12 R13

• Monitor interval is 30 sec (fixed) 10.1.10.0/24 10.1.11.0/24

10.1.12.0/24
10.1.13.0/24
 Destination Site: Collecting Performance Metrics
Actual User Traffic IWAN POP

Hub MC
• Traffic flow captured on the destination MC1
10.8.3.3/32

site
• Performance Monitor collects BR1 BR2

Performance Metrics
• Per Channel
• Default Monitor interval is 30 sec DMVPN DMVPN
MPLS INET
(configurable)
Source Destination DSCP App
10.8.1.200 10.1.10.200 AF41 APP1
3

R10 R11 R12 R13

Channel Dst-Site-id Path DSCP BW Delay Jitter Loss
R10 MC
5 Hub Tu1 AF41 24 51 2 1 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Destination Site: Collecting Performance Metrics
Smart Probes
IWAN POP

• Without actual traffic Hub MC

10.8.3.3/32
MC1
• 20 pps for channel without traffic
• IOS-XE: BR sends 10 probes spaced 20ms apart
in the first 500ms and another similar 10 probes in BR1 BR2

the next 500ms Smart

Probes
• IOS: BR sends one packet every 50ms
• With actual traffic
MPLS INET
• Lower frequency when real traffic is observed over
the channel
• Probes sent every 1/3 of [Monitor Interval], ie
every 10 sec by default 3

• Measured by Performance Monitor just like R10 R11 R12 R13

other data traffic 10.1.10.0/24 10.1.11.0/24

10.1.12.0/24
10.1.13.0/24
What is a Channel? IWAN POP

Hub MC
10.8.3.3/32
• A Channel is a unique combination MC1

of Interface, sites, next-hop and

path BR1 BR2

• Created based on real traffic Present Channel 3

• Site 10
observed on the BRs • DSCP AF41
• MPLS
• Added every time there is a new • Path 1 MPLS INET
DSCP or a new interface or a new
site added to the prefix database. Backup Channel 4
• Site 10
• Smart Probe is received • DSCP AF41
• INET
• On all exits • Path 2 R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
What is a Channel?
IWAN POP

Hub MC Between Any Pair

10.8.3.3/32
MC1
of Sites that has
traffic!
BR1 BR2

Present Channel 13 MPLS INET

Backup Channel 14
• Site 11
• Site 11
• DSCP EF
• DSCP EF
• MPLS
• INET

R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
 IOS-XE 3.15
IOS 15.5(2)T

Monitoring Channel
10.8.0.0/16
10.9.0.0/16
• Monitoring performance per channel
Hub MC Transit MC
• Channel per destination prefix, DSCP and MC1 MC2
Path Id POP ID 0 POP ID 1

• Include all sites advertising that prefix

BR1 BR2 BR3 BR4
• Load balance maybe done between POPs if
Path MPLS Path INET Path MPLS Path INET
prefix is shared between multiple transit sites Id 1 Id 2 Id 1 Id 2

• Track individual BR performance on the 10.8.0.0/16

10.9.0.0/16
10.8.0.0/16
10.9.0.0/16
hub DMVPN DMVPN
MPLS INET
• A PfR-label uniquely identify a path
between sites across clouds

MC/BR
POP-ID PATH-ID POP-ID PATH-ID
10.1.10.0/24
 Performance Violation
Enterprise HQ
• Performance notification exported ONLY Hub MC
when there is a violation on a specific MC1
10.8.3.3/32

channel
• Generated from ingress monitor attached on BR1 BR2
BRs to the source site MC
• Based on Monitor interval (30 sec default, R10
configurable) TCA Delay
• Via all available external interfaces. DSCP AF41
MPLS INET
Path MPLS

3
Channel Dst-Site-id DSCP Path BW Delay Jitter Loss
R10 R10 R11 R12 R13
5 Hub AF41 Tu1 24 250 2 1
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Performance Violation – Detected on Dst Site
Enterprise HQ

Hub MC
10.8.3.3/32
Dst-Site-Pfx Dst-Site-id App DSCP State BR Exit MC1

10.1.10.0 R10 APP1 AF41 CN BR1 Tu1

10.1.10.0 R10 N/A AF41 CN BR1 Tu1

BR1 BR2
10.1.10.0 R10 N/A AF31 CN BR1 Tu1

10.1.10.0 R10 N/A 0 CN BR2 Tu2

10.1.11.0 R11 N/A EF CN BR1 Tu1

10.1.11.0 R11 N/A AF31 CN BR1 Tu1

10.1.11.0 R11 N/A 0 CN BR2 Tu2 MPLS INET

10.1.12.0 R12 N/A 0 CN BR2 Tu2

3
R10 R10 R11 R12 R13
TCA Delay
10.1.12.0/24
DSCP AF41 10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Path MPLS
Policy Decision – Reroute TC
Enterprise HQ
• MC computes a new path for each Hub MC
10.8.3.3/32
impacted TC MC1

• MC tells the BRs to enforce the new BR1 BR2

path

MPLS INET

R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Reroute TC – Path Enforcement
Enterprise HQ
• Dataplane forwarding Hub MC
10.8.3.3/32
MC1
• Activated on all but external interfaces
• Lookup per packet - output-if/next hop BR1 BR2

retrieved
• Packet Forwarded
• If no entry – Uses FIB entry MPLS INET

• TC flows redirected to the new path over

the auto mGRE tunnels between the BRs
3
• No change in the routing table R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
 Deploying IWAN
Intelligent Path Control
 Performance Routing – Platform Support

Cisco CSR-1000

MC
Cisco ASR-1000 BR(1)

Cisco ISR 4000 MC

BR
4400
Cisco ISR G2 family 4300
3900-AX
2900-AX MC
1900-AX BR
890
MC
BR
(1) Future
IWAN Deployment – DMVPN http://docwiki.cisco.com/wiki/PfR3:Solutions:IWA
N
DCI
IWAN POP1 WAN Core IWAN POP2
• Transport Independent Design based on DMVPN
• Branch spoke sites establish an IPsec tunnel to and R83 R93
register with the hub site
• Data traffic flows over the DMVPN tunnels
• WAN interface IP address used for the tunnel source R84 R85 R94 R95
address (in a Front-door VRF)
• One tunnel per user inside VRF 10.8.0.0/16 10.8.0.0/16
10.9.0.0/16 10.9.0.0/16
• Overlay Routing
MPLS INET
• BGP or EIGRP are typically used for scalability DMVPN DMVPN
• IP routing exchanges prefix information for each site
• Per-tunnel QOS is applied to prevent hub site
oversubscription to spoke sites
• Optional: Performance Monitoring (Advanced IWAN)
R10 R11 R12 R13
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Front-door VRF
Internet default
Internet Edge

EIGRP
INSIDE
R84 R85
INSIDE
VPN-DMZ
DMVPN
Hub
INTERNET VRF
default OUTSIDE default
MPLS VRF
OUTSIDE default

Internet
MPLS MPLS VRF
default
INTERNET
VRF default

R10

Branch
 For Your
Reference

DMVPN Configuration – FVRF

TRANSIT SITE
R83

R84 R85
172.16.84. 4 172.16.85.5
vrf definition IWAN-TRANSPORT-1
!
address-family ipv4
exit-address-family Front-door VRF definition for
MPLS Transport
! MPLS INTERNET

vrf definition IWAN-TRANSPORT-2

!
172.16.101.10 172.16.102.10
address-family ipv4
exit-address-family R10
!
10.1.10.0/24
 For Your
Reference

DMVPN Configuration – IPSec

crypto ikev2 keyring DMVPN-KEYRING-1 TRANSIT SITE
peer ANY
address 0.0.0.0 0.0.0.0 R83
pre-shared-key c1sco123
!
!
R84 R85
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-1
match fvrf IWAN-TRANSPORT-1 10.0.100.84 10.0.200.85
match identity remote address 0.0.0.0
authentication remote pre-share Maximise window size to
authentication local pre-share eliminate future anti-replay issue
keyring local DMVPN-KEYRING-1
! DMVPN DMVPN
crypto ipsec security-association replay window-size 512 MPLS INTERNET
!
crypto ipsec transform-set GCM256/SHA/TRANSPORT esp-gcm 256
mode transport
!
crypto ipsec profile DMVPN-PROFILE-1
set transform-set AES256/SHA/TRANSPORT 10.0.100.10 10.0.200.10
set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-1
R10

10.1.10.0/24

crypto ikev2 dpd 40 5 on-demand ! Set DPD timers for Branch

Configs ONLY!
 For Your
Reference

DMVPN Hub Configuration – Interfaces & Routing

MPLS TRANSPORT – R84
interface GigabitEthernet0/0/3 TRANSIT SITE
description MPLS-TRANSPORT Put Transport Interface into
R83
vrf forwarding IWAN-TRANSPORT-1 MPLS Front-door VRF
ip address 172.16.84.4 255.255.255.0
!
interface Tunnel100 R84 R85
bandwidth 100000 10.0.100.84 10.0.200.85
ip address 10.0.100.84 255.255.255.0 Define the bandwidth
no ip redirects
ip mtu 1400
ip pim nbma-mode
ip pim sparse-mode
ip nhrp authentication cisco123 DMVPN DMVPN
ip nhrp map multicast dynamic MPLS INTERNET
ip nhrp network-id 100
DMVPN Network ID: MPLS
ip nhrp holdtime 600
ip nhrp redirect Set DMVPN Ph3
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/3 Map to Physical Interface
tunnel mode gre multipoint
tunnel key 101
tunnel vrf IWAN-TRANSPORT-1
tunnel protection ipsec profile DMVPN-PROFILE-1 Tunnel endpoint is in Front-door VRF
!
ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 172.16.84.8
Default route for Tunnel endpoints
 For Your
Reference

DMVPN Spoke Configuration – Interfaces & Routing

interface GigabitEthernet0/1
Put Transport Interface into
vrf forwarding IWAN-TRANSPORT-1
ip address 172.16.101.10 255.255.255.0 MPLS Front-door VRF
!
interface Tunnel100 Note: ip nhrp registration no-unique
bandwidth 1000
Instantiate DMVPN Tunnel
Should be use only on dynamically
ip address 10.0.100.10 255.255.255.0 addressed spokes (usually over the
no ip redirects Internet
ip mtu 1400
ip pim dr-priority 0
ip pim sparse-mode
ip nhrp authentication cisco123
ip nhrp network-id 100 DMVPN Network ID: MPLS
ip nhrp holdtime 600
ip nhrp nhs 10.0.100.84 nbma 172.16.84.4 multicast Multiple DMVPN Hub for
ip nhrp nhs 10.0.100.94 nbma 172.16.94.4 multicast
Resiliency
ip nhrp registration no-unique
ip nhrp shortcut
ip tcp adjust-mss 1360 Set DMVPN Ph3
if-state nhrp 10.0.100.10 10.0.200.10
tunnel source GigabitEthernet0/1 Tunnel state based on NHRP
tunnel mode gre multipoint R10
tunnel key 101 Tunnel endpoint is in Front-
tunnel vrf IWAN-TRANSPORT-1 door VRF 10.1.10.0/24
tunnel protection ipsec profile DMVPN-PROFILE-1
!
ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 172.16.101.8
Default route for Tunnel endpoints
Overlay Routing Protocols
Which protocol should I use?
• IWAN Profiles are based upon BGP and EIGRP for scalability and optimal Intelligent Path
Control
• Intelligent Path Control:
• PfR can be used with any routing protocols by relying on the routing table (RIB).
• Requires all valid WAN paths be ECMP so that each valid path is in the RIB.
• For BGP and EIGRP, PfR can look into protocol’s topology information to determine both best paths
and secondary paths thus, ECMP is not required.

• PfRv3 always checks for a parent route before being able to control a Traffic Class. Parent
route check is done as follows:
• Check to see if there is an NHRP shortcut route
• If not – Check in the order of BGP, EIGRP, Static and RIB
• Make sure that all Border Routers have a route over each external path to the destination sites PfR
will NOT be able to effectively control traffic otherwise.
Routing Deployment – EIGRP IWAN POP2 10.8.0.0/16
10.9.0.0/16

• Single EIGRP process for Branch, WAN IWAN POP1 10.8.0.0/16

10.9.0.0/16
and POP/hub sites R83

• Extend Hello/Hold timers for WAN

R84 R85
• Adjust tunnel interface “delay” to ensure Delay TAG Delay TAG
WAN path preference 1000 DMVPN-1 2000 DMVPN-2

• MPLS primary, INET secondary

Set Interface
• Hubs Delay to MPLS INET
influence best 10.0.100.0/24 10.0.200.0/24
• Route tag filtering to prevent routing loops path
across DMVPNs
• Branch prefix summary route for Delay 1000
spoke-to-spoke tunnels EIGRP
Stub R10 R11 R12 R13
• Spokes 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24
• EIGRP Stub for scalability 10.1.13.0/24
Routing Deployment – BGP IWAN POP2 10.5.0.0/16
10.6.33.0/24

• A single iBGP routing domain is used IWAN POP1 10.4.0.0/16

10.6.32.0/24
• Set BGP Hello/Hold timers for IWAN (20, 60) IGP IGP

• POP (Hub) Sites BR1 BR2

Community 6:34 Community 6:36
• DMVPN hub are also BGP route-reflectors for the iBGP
Local Pref 1000 Local Pref 200
spokes
10.4.0.0/16 10.4.0.0/16
• BGP dynamic peering configured on route-reflectors 10.0.0.0/8 10.0.0.0/8
0.0.0.0 0.0.0.0
• Default and Internal Summary routes to spokes
MPLS INET
• Set Community and Local Pref for all prefixes 10.6.34.0/23 10.6.36.0/23
• Redistribute BGP into local IGP
10.1.12.0/24
10.1.13.0/24
• Branch (Spokes) Sites: 10.1.12.254/32
• peer to each DMVPN/BGP route reflector in each POP IGP
• Local Pref set at POPs to prefer MPLS over INET Paths R10 R11 R12 R13

• Branches are Stubs and only advertise local prefixes 10.7.12.0/24

10.7.10.0/24 10.7.11.0/24 10.7.13.0/24
• Redistribute - connected, IGP routes
 PfR Deployment – Hub
HUB SITE
domain IWAN Site ID = 10.8.3.3
vrf default
master hub
Hub MC
source-interface Loopback0 R83 R93
R83 enterprise-prefix prefix-list ENTERPRISE_PREFIX
site-prefixes prefix-list DC_PREFIX
POP ID 0

domain IWAN domain IWAN

R84 R85 R94 R95
vrf default vrf default Path MPLS Path INET
border border Id 1 Id 2
master 10.8.3.3 master 10.8.3.3
R84 source-interface Loopback0 source-interface Loopback0
R85 ! !
interface Tunnel100 interface Tunnel200
description -- Primary Path -- description – Secondary Path -- DMVPN DMVPN
domain IWAN path MPLS path-id 1 domain IWAN path INET path-id 2 MPLS INET

• Enterprise Prefix: summary prefix for the entire domain

• Site Prefix: no automatic learning – Mandatory
• POP Id unique per domain
R10 R11 R12 R13
• Path ID unique per Site 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Enterprise Prefix List
• The main use of the enterprise prefix list is to determine the
enterprise boundary.
• With enterprise-prefix
• If a prefix doesn't match any site-prefix but matches enterprise-prefix then
HMC the prefix belongs to a site that is not participating in PfRv3 but it does
belong to the enterprise.
HUB • PfR will not influence traffic towards sites that has NOT enabled PFR.
Master MC
• Without enterprise-prefix
• All the traffic that would be going towards a spoke that is NOT PfR enabled
will be learnt as internet traffic class and therefore subjected to load
balancing.

domain IWAN
vrf default
master hub
source-interface Loopback0
enterprise-prefix prefix-list ENTERPRISE_PREFIX
!
ip prefix-list ENTERPRISE_PREFIX seq 10 permit 10.0.0.0/8
Site Prefixes – Static Configuration
TRANSIT SITE
• This allows configuring site-prefix manually instead of
learning.
MC1
• This configuration should be used at the site if the
site is used for transit. BR1 BR2
• For example, Site A reaches Site B via Hub-Site, where
Hub-Site is transit site. The configuration is used to
prevent learning of Site A prefix as Hub-Site prefix when it
is transiting from Hub.
domain IWAN
Source Destination DSCP App
vrf default
10.1.10.200 10.1.11.200 AF41 AppXY
master hub
source-interface Loopback0
site-prefixes prefix-list DC1_PREFIX
!
ip prefix-list DC1_PREFIX seq 10 permit 10.8.0.0/16
!
IWAN 2.0 – HUB-MC Scaling
ASR 1002-X
2000 sites

ASR 1001-X
1000 sites

ISR 4451
200 sites
ISR 4431
50 sites
CSR1000v
2 vCPU
CSR1000v 500 sites
1 vCPU
200 sites
 IWAN Policies – DSCP or App Based
domain IWAN
vrf default • Policies:
master hub – DSCP or Application Based Policies
load-balance (NBAR2)
class MEDIA sequence 10 – DSCP marking can be used with
match application <APP-NAME1> policy real-time-video NBAR2 on the LAN interface
match application <APP-NAME2> policy custom (ingress on BR)
priority 1 one-way-delay threshold 200 • Default Class is load balanced
priority 2 loss threshold 1
path-preference MPLS fallback INET
class VOICE sequence 20
match dscp <DSCP-VALUE> policy voice • Custom thresholds
path-preference MPLS fallback INET
class CRITICAL sequence 30
match dscp af31 policy low-latency-data • Pre-defined thresholds

R83
 For Your
Reference

Built-in Policy Templates

Pre-defined Threshold Definition
Template
Voice priority 1 one-way-delay threshold 150 threshold 150 (msec)
priority 2 packet-loss-rate threshold 1 (%)
priority 2 byte-loss-rate threshold 1 (%)
priority 3 jitter 30 (msec) Pre- Threshold Definition
Real-time-video priority 1 packet-loss-rate threshold 1 (%) defined
priority 1 byte-loss-rate threshold 1 (%) Template
priority 2 one-way-delay threshold 150 (msec) Bulk-data priority 1 one-way-delay threshold 300 (msec)
priority 3 jitter 20 (msec) priority 2 byte-loss-rate threshold 5 (%)
Low-latency-data priority 1 one-way-delay threshold 100 (msec) priority 2 packet-loss-rate threshold 5 (%)
priority 2 byte-loss-rate threshold 5 (%) Best-effort priority 1 one-way-delay threshold 500 (msec)
priority 2 packet-loss-rate threshold 5 (%) priority 2 byte-loss-rate threshold 10 (%)
priority 2 packet-loss-rate threshold 10 (%)
scavenger priority 1 one-way-delay threshold 500 (msec)
priority 2 byte-loss-rate threshold 50 (%)
priority 2 packet-loss-rate threshold 50 (%)
 IWAN Deployment – Recommended Policies
CVD IWAN 2.-0 – DSCP Based Policies recommended
domain IWAN
vrf default • When load balancing is enabled, PfRv3 adds a
master hub “default class for match all DSCP (lowest priority
load-balance compared to all the other classes)” and PfRv3
class VOICE sequence 10
controls this traffic.
match dscp ef policy voice • When load balancing is disabled, PfRv3 deletes this
path-preference MPLS fallback INET “default class” and as a part of that frees up the TCs
class INTERACTIVE-VIDEO sequence 20 that was learnt as a part of LB – they follow the
match dscp cs4 policy real-time-video routing table
match dscp af41 policy real-time-video
match dscp af42 policy real-time-video
path-preference MPLS fallback INET
• Pre-defined thresholds
class CRITICAL-DATA sequence 30
match dscp af21 policy low-latency-data • Custom thresholds can also be used
path-preference MPLS fallback INET

R83
Redundant MC – Anycast IP HUB SITE

Hub MC Backup Hub MC

• What happens when a MC fails? R83 10.8.3.3/32 R83b 10.8.3.3/31
• Traffic forwarded based on routing information –
ie no drop
R84 R85
• What happens when the Hub MC fails?
• Branch MCs keep their configuration and
policies
• Continue to optimize traffic MPLS INET
• A backup MC can be defined on the hub.
• Using the same IP address as the primary
• Routing Protocol is used to make sure BRs and
branch MC connect to the primary
R10 R11 R12 R13
• Stateless redundancy
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24
• Backup MC will re-learn the traffic 10.1.13.0/24
 PfR Deployment – Transit Site
TRANSIT SITE
domain IWAN Site ID = 10.9.3.3
vrf default
master transit 1
Transit MC
source-interface Loopback0 R83 R93
R93 site-prefixes prefix-list DC_PREFIX
hub 10.8.3.3
POP ID 1

domain IWAN domain IWAN

R84 R85 R94 R95
vrf default vrf default Path MPLS Path INET
border border Id 1 Id 2
master 10.9.3.3 master 10.9.3.3
R94 source-interface Loopback0 source-interface Loopback0
R95 ! !
interface Tunnel100 interface Tunnel200
description -- Primary Path -- description – Secondary Path -- DMVPN DMVPN
domain IWAN path MPLS path-id 1 domain IWAN path INET path-id 2 MPLS INET

• Site Prefix: no automatic learning – Mandatory

• POP Id unique per domain
• Path ID unique per Site
R10 R11 R12 R13
• Peering with Hub MC 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
PfR Deployment – Single CPE Branch
HUB SITE TRANSIT SITE
Site ID = 10.8.3.3 Site ID = 10.9.3.3

R83 R93
domain IWAN
vrf default
master branch
source-interface Loopback0 R84 R85 R94 R95
hub 10.8.3.3
R10 border
master local
source-interface Loopback0
DMVPN DMVPN
MPLS INET

• Single CPE Branch Sites

• Branch MCs connect to the Hub R10 R11 R12 R13
R10 R11 R12 R13
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
PfR Deployment – Dual CPE Branch
HUB SITE TRANSIT SITE
Site ID = 10.8.3.3 Site ID = 10.9.3.3

domain IWAN
vrf default R83 R93
master branch
source-interface Loopback0
hub 10.8.3.3
R12 border
R84 R85 R94 R95

master local
source-interface Loopback0

DMVPN DMVPN
domain IWAN
MPLS INET
vrf default
R13 border
master 10.2.12.12
source-interface Loopback0
R10 R11 R12 R13
• Dual CPE Branch Sites R10 R11 R12 R13
10.1.12.0/24
• Branch MCs connect to the Hub 10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
 IWAN Peering HUB SITE
Site ID = 10.8.3.3
TRANSIT SITE
Site ID = 10.9.3.3

Hub MC Transit MC
R83 R93

R84 R85 R94 R95

R83#sh eigrp service-family ipv4 neighbors
EIGRP-SFv4 VR(#AUTOCFG#) Service-Family Neighbors for AS(59501)
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
5 10.2.10.10 Lo0 513 01:17:12 65 390 0 39
4 10.2.11.11 Lo0 582 01:17:01 59 354 0 40
3 10.2.12.12 Lo0 510 01:17:04 61 366 0 78
2 10.8.4.4 Lo0 538 01:17:04 1 100 0 15 MPLS INET
1 10.8.5.5 Lo0 562 01:17:05 4 100 0 18
0 10.9.3.3 Lo0 546 01:17:15 5 100 0 40
MC1#

10.2.10.10
• Hub MC gets all remote MC IP
Addresses => Site-Ids R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
 Branch Sites
R10#show domain IWAN master status
Master VRF: Global
Instance Type: Branch
Instance id: 0 R10
Operational status: Up
Configured status: Up 10.1.10.0/24
[SNIP] R10#show domain IWAN master status
Minimum Requirement: Met Borders:
[SNIP] IP address: 10.2.10.10
Version: 2
Connection status: CONNECTED (Last Updated 00:47:53 ago )
Interfaces configured:
• External interfaces Name: Tunnel100 | type: external | Service Provider: MPLS | Status: UP | Zero-SLA: NO
discovered Number of default Channels: 2
Path-id list: 1:1 0:0 0:1
• Branch MC receives
and applies the Name: Tunnel200 | type: external | Service Provider: INET | Status: UP | Zero-SLA: NO
monitors Number of default Channels: 2
Path-id list: 1:2 0:0 0:2

Tunnel if: Tunnel0

 Traffic Classes Summary
R83#shmtcs

APP - APPLICATION, TC-ID - TRAFFIC-CLASS-ID, APP-ID - APPLICATION-ID

SP - SERVICE PROVIDER, PC = PRIMARY CHANNEL ID,
BC - BACKUP CHANNEL ID, BR - BORDER, EXIT - WAN INTERFACE
UC - UNCONTROLLED, PE - PICK-EXIT, CN - CONTROLLED, UK - UNKNOWN

Dst-Site-Pfx Dst-Site-Id APP DSCP TC-ID APP-ID State SP PC/BC BR/EXIT

10.1.12.0/24 10.2.12.12 N/A default 3 N/A CN MPLS 4/NA 10.8.4.4/Tunnel100

10.1.12.0/24 10.2.12.12 N/A ef 2 N/A CN INET 8/5 10.8.5.5/Tunnel200

10.1.12.0/24 10.2.12.12 N/A af31 4 N/A CN MPLS 6/7 10.8.4.4/Tunnel100

10.1.11.0/24 10.2.11.11 N/A default 8 N/A CN MPLS 10/NA 10.8.4.4/Tunnel100
10.1.11.0/24 10.2.11.11 N/A ef 6 N/A CN INET 13/14 10.8.5.5/Tunnel200
10.1.11.0/24 10.2.11.11 N/A af31 10 N/A CN MPLS 16/15 10.8.4.4/Tunnel100
10.1.10.0/24 10.2.10.10 N/A default 7 N/A CN INET 11/12 10.8.5.5/Tunnel200
10.1.10.0/24 10.2.10.10 N/A ef 5 N/A CN INET 19/17 10.8.5.5/Tunnel200
10.1.10.0/24 10.2.10.10 N/A af31 9 N/A CN MPLS 18/20 10.8.4.4/Tunnel100
Total Traffic Classes: 9 Site: 9 Internet: 0
R83#

Traffic Class – Site11 - Critical TC Id Controlled Path Information - Channels

Check Traffic Classes Details
Dst-Site-Prefix: 10.1.12.0/24 DSCP: ef [46] Traffic class id:2
Clock Time: 20:49:31 (CET) 06/01/2015 Check Traffic Class
TC Learned: 01:03:18 ago Voice for site 10
Present State: CONTROLLED
Current Performance Status: in-policy
Current Service Provider: MPLS path-id:1 since 00:01:31
Previous Service Provider: INET pfr-label: 0:0 | 0:2 [0x2] for 3439 sec Active Path used
BW Used: 14 Kbps
Present WAN interface: Tunnel100 in Border 10.8.4.4
Present Channel (primary): 5 MPLS pfr-label:0:0 | 0:1 [0x1] Check Channels
Backup Channel: 8 INET pfr-label:0:0 | 0:2 [0x2] used (Primary and
Destination Site ID bitmap: 0
Destination Site ID: 10.2.12.12
Backup)
Class-Sequence in use: 10
Class Name: VOICE using policy User-defined
priority 2 packet-loss-rate threshold 5.0 percent
priority 1 one-way-delay threshold 150 msec Class name and
priority 2 byte-loss-rate threshold 5.0 percent Policies
BW Updated: 00:00:00 ago
Check Traffic Classes Details (Cont’d)
Reason for Latest Route Change: Backup to Primary path preference transition
Route Change History:
Date and Time
Previous Exit
Current Exit
Reason

1: 20:48:00 (CET) 06/01/2015

INET/10.8.5.5/Tu200 (Ch:8)
MPLS/10.8.4.4/Tu100 (Ch:5) Last 5 reasons for
Backup to Primary path preference transition change
2: 19:50:41 (CET) 06/01/2015
MPLS/10.8.4.4/Tu100 (Ch:5)
INET/10.8.5.5/Tu200 (Ch:8)
One Way Delay : 252 msec*

3: 19:46:44 (CET) 06/01/2015

None/0.0.0.0/None (Ch:0)
MPLS/10.8.4.4/Tu100 (Ch:5)
Uncontrolled to Controlled Transition

Not the actual output – reformatted for better reading

 On Demand Export
Check Channel after TCA (ODE)

Channel Id: 5 Dst Site-Id: 10.2.12.12 Link Name: MPLS DSCP: ef [46] pfr-label: 0:0 | 0:1 [0x1] TCs: 1
Channel Created: 01:13:07 ago
Provisional State: Initiated and open
Operational state: Available
Channel to hub: FALSE Threshold Crossing
Interface Id: 11 Alert (TCA)
Supports Zero-SLA: Yes
Muted by Zero-SLA: No
Estimated Channel Egress Bandwidth: 15 Kbps
Immitigable Events Summary:
Total Performance Count: 0, Total BW Count: 0 TCA Statistics:
ODE Stats Bucket Number: 1 Received:801 ; Processed:801 ;
Last Updated : 00:14:40 ago Unreach_rcvd:0
Packet Count : 40 Latest TCA Bucket
Byte Count : 3360 Last Updated : 00:14:42 ago
One Way Delay : 221 msec* One Way Delay : 252 msec*
Loss Rate Pkts: 0.0 % Loss Rate Pkts: NA
Loss Rate Byte: 0.0 % Loss Rate Byte: NA
Jitter Mean : 43000 usec Jitter Mean : NA
Unreachable : FALSE Unreachability: FALSE
Load Balancing
• Current Situation
- Load balancing works on physical links only Hub MC Transit MC
R83 R93
- both R84 and R94 share the same physical link(both POP ID 0 POP ID 1
the MPLS NH's converge into the same physical link
on the spoke) and so PfR will think of both of them R84 R85 R94 R95
having the same Physical link BW Path MPLS Path INET Path MPLS Path INET
Id 1 Id 2 Id 1 Id 2
• Default Classes TCs
- Load balancing at any time (not only at creation
time).
MPLS INET
- TC will be moved to ensure bandwidth on all links is
within the defined range

• Performance TCs
- Initial load-balancing while placing the TCs, on a per R10
TC basis. PfR does not account for the TCs getting
fatter.
Failover Time HUB SITE
Site ID = 10.8.3.3
TRANSIT SITE
Site ID = 10.9.3.3

Hub MC Transit MC
• Channel Unreachable: ~1 sec MC1 MC2
POP ID 0 POP ID 1
• PfRv3 considers a channel reachable as long as
the site receives a PACKET on that channel BR1 BR2 BR3 BR4

• A channel is declared unreachable in both Path MPLS Path INET Path MPLS Path INET
Id 1 Id 2 Id 1 Id 2
direction if
• There is NO traffic on the Channel, probes are our only way
of detecting unreachability. So if no probe is received within 1 DMVPN DMVPN
sec, we detect unreachability. MPLS INET
• When there IS traffic on the channel, if we don’t see any
packet for more than a second on a channel we detect
unreachability.
domain IWAN
• Ingress Performance Violation detected vrf default
MC/BR master hub
• Delay, loss or jitter thresholds monitor-interval 2 dscp ef
10.1.10.0/24
• Based on Monitor-interval monitor-interval 2 dscp af41
monitor-interval 2 dscp cs4
monitor-interval 2 dscp af31
Direction from POPs to Spokes
HUB SITE TRANSIT SITE
Site ID = 10.8.3.3 Site ID = 10.9.3.3
• Each POP is a unique site by itself and so it
will only control traffic towards the spoke on MC1 MC2
the WAN’s that belong to that POP.
• PfRv3 will NOT be redirecting traffic between BR1 BR2 BR3 BR4
POP across the DCI or WAN Core. If it is Path MPLS Path INET Path MPLS Path INET
required that all the links are considered from Id 1 Id 2 Id 1 Id 2

POP to spoke, then the customer will need to

use a single MC. DMVPN DMVPN
MPLS INET

MC/BR MC/BR MC/BR BR

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Direction from Spoke to HUB
• The spoke considers all the paths (multiple NH’s) towards the POPs and will be
maintaining a list of Active/Standby candidate next hops per prefix and interface,
that will be derived based on the routing configuration.
• Active next hop: A next hop is considered active for a given prefix if it has the best
metric.
• Standby next hop: A next hop is considered standby for a given prefix if it advertises a
route for the prefix but does not have the best metric.
• Unreachable next hop: A next hop is considered unreachable for a given prefix if it does
not advertise any route for the prefix.
Direction from Spoke to HUB
• With Path Preference
• PfR will be giving Path Preference more priority and so our candidate channels for a
particular traffic-class will first consider all the links belonging to the preferred path
preference (i.e it will include the active and then the standby links belonging to the
preferred path) and will then go to the fallback provider links.
• Without Path Preference
• PfR will give preference to the active channels and then the standby channels
(active/standby will be per prefix) with respect to the performance and policy decisions.
• Note that the Active and Standby channels per prefix will span across the POP’s.
• Spoke will randomly (hash) choose the active channel
Example
HUB SITE TRANSIT SITE
Site ID = 10.8.3.3 Site ID = 10.9.3.3
• PfR Policies:
– Voice, Video and Business App with Path R83 R93
Preference DMVPN-MPLS 10.8.0.0/16 10.8.0.0/16
– Default Class Load Balanced
R84 R85 R94 R95
• Routing Configuration will determine the use 10.0.100.84 10.0.200.85 10.0.100.94 10.0.200.95

of possible next-hops
BGP BGP

10.8.0.0/16 DMVPN DMVPN 10.8.0.0/16

10.0.0.0/8 MPLS INET 10.0.0.0/8

R10 R11

10.1.10.0/24 10.1.11.0/24
 Path Selection – Transit Site Affinity
DC1 DC2
Site ID = 10.8.3.3 Site ID = 10.9.3.3

NEXT-
PREFIX INTERFACE BGP LP Status R83 R93
HOPS 10.8.0.0/16 10.8.0.0/16
R84 201 Preferred – Active
With PfR Path R94 200 Preferred – Standby
Preference R85 151 Fallback - Active - R84 R85 R94 R95
R95 150 Fallback – Standby
10.8.0.0/16 LP201 LP151 LP200 LP150
R84 201 Active
Without PfR R85 151 Active
Path Preference R94 200 Standby
R95 150 Standby DMVPN DMVPN
MPLS INET
• Routing policies
– MPLS DMVPN best path
– In each DMVPN cloud, DC1 preferred (R84 and R85)
• Important Note:
• PfR Policies: R10 – Single NH used in each tunnel – failover to
– MPLS DMVPN preferred the standby NH if OOP
10.1.10.0/24
– INET DMVPN fallback – Load balancing is only between external
interfaces, not within a DMVPN cloud.
 Path Selection – No Transit Site Affinity
DC1 DC2
Site ID = 10.8.3.3 Site ID = 10.9.3.3

NEXT-
PREFIX INTERFACE BGP LP Status R83 R93
HOPS 10.8.0.0/16 10.8.0.0/16
R84 200 Preferred – Active
With PfR Path R94 200 Preferred – Active
Preference R85 150 Fallback - Active - R84 R85 R94 R95
R95 150 Fallback – Active
10.8.0.0/16 LP200 LP150 LP200 LP150
R84 200 Active
Without PfR R85 150 Active
Path Preference R94 200 Active
R95 150 Active DMVPN DMVPN
MPLS INET
• Routing policies
– MPLS DMVPN best path
– In each DMVPN cloud, equal cost between all next-hops
• Important Note:
• PfR Policies: R10 – Multiple NH active on a tunnel but still only
– MPLS DMVPN preferred one is used for traffic forwarding
10.1.10.0/24
– INET DMVPN fallback – Load balancing is only between external
interfaces, not within a DMVPN cloud.
Direct Internet Access Routing with F-VRF
• Direct Internet Access managed outside of PfRv3 with a combination of route-maps and
default route
• Internet Traffic forwarded to the external interface pointing toward Internet
• No option to forward a subset of the traffic except the use of PBR and IP addresses
• Path Control:
Forward Traffic to
• PBR or Default Route get to the Internet
• NAT directly to the

DMVPN
DMVPN
MPLS VPN Internet physical interface
• Return Traffic: leak from F-VRF to Global

G0/0
Policy Route for
IOS NAT/FW 10.0.0.0/8 traffic
Set next-hop VRF to
VRF INET-PUBLIC2 CPE VRF INET-PUBLIC1 Global Table

IOS NAT/FW Global Table

Deploying with Multiple
VRFs
 Deploying with user VRFs TRANSIT SITE

vrf definition TEST1

! MC1
address-family ipv4 1 2 1 2
exit-address-family
! R84 R85
interface Tunnel 101
vrf definition TEST2
vrf forwarding TEST1
!
tunnel key 101
address-family ipv4
tunnel vrf IWAN-TRANSPORT-1
exit-address-family
!
! MPLS INET
interface Tunnel 102
vrf forwarding TEST2
tunnel key 102
tunnel vrf IWAN-TRANSPORT-1

• DMVPN Tunnel per VRF 1

R10
2 1
R11
2 1
R12
2 1
R13
2
10.1.12.0/24
• Over the top routing per VRF 10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
• SAF Peering per VRF
Enterprise Branch Sites
Deploying with VRF – Hub MC TRANSIT SITE
GLOBAL: 10.8.3.3
VRF TEST1: 11.8.3.3
interface Loopback1 MC1 VRF TEST2: 12.8.3.3
vrf forwarding TEST1
!
interface Loopback2 R84 R85
vrf forwarding TEST2

domain IWAN
vrf TEST1 MPLS INET
master hub
source-interface Loopback1
!
vrf TEST2
master hub
R10 R11 R12 R13
source-interface Loopback2
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

Enterprise Branch Sites

Deploying with VRF – Hub MC Policies
domain IWAN [Cont’d]
vrf TEST1 vrf TEST2
master hub master hub
load-balance load-balance
class VOICE sequence 10 class VOICE sequence 10
match dscp ef policy voice match dscp ef policy voice
path-preference MPLS fallback INET path-preference MPLS fallback INET
class VIDEO sequence 20 class CRITICAL sequence 30
match dscp af41 policy voice match dscp af31 policy low-latency-
path-preference MPLS fallback INET data
class CRITICAL sequence 30
match dscp af31 policy low-latency-
data
Deploying with VRF – Hub BR TRANSIT SITE

domain IWAN GLOBAL: 10.8.3.3

vrf TEST1 VRF TEST1: 11.8.3.3
MC1 VRF TEST2: 12.8.3.3
border
master 11.8.3.3
source-interface Loopback1 R84 R85
! Tu101
vrf TEST2 Tu102
border
master 12.8.3.3
source-interface Loopback2
! MPLS INET
interface Tunnel101
description -- Primary Path –
vrf forwarding TEST1
domain IWAN path MPLS path-id 1
!
interface Tunnel102
description -- Primary Path – R10 R11 R12 R13

vrf forwarding TEST2 10.1.12.0/24

10.1.10.0/24 10.1.11.0/24
domain IWAN path MPLS path-id 2 10.1.13.0/24

Enterprise Branch Sites

 Deploying with VRF – Branch MC/BR TRANSIT SITE
GLOBAL: 10.8.3.3
VRF TEST1: 11.8.3.3
MC1 VRF TEST2: 12.8.3.3
domain IWAN
vrf TEST1
master branch R84 R85
source-interface Loopback1
hub 11.8.3.3
border
master local
source-interface Loopback1 MPLS INET
!
vrf TEST2
R10 master branch
source-interface Loopback2 Tu101
hub 12.8.3.3 Tu102
border
R10 R11 R12 R13
master local
source-interface Loopback2 10.1.10.0/24
10.1.12.0/24
10.1.11.0/24 10.1.13.0/24

Enterprise Branch Sites

PfRv3 Management
 PfRv3 Exporter Configuration IWAN POP

Hub MC
domain IWAN 10.8.3.3/32
MC1
vrf default
master hub
collector 10.151.1.95 port 2055 BR1 BR2

MC1

• Enable exporter on the Hub MC

• Distributed through SAF to all MCs and BRs in MPLS INET
the domain
• Cisco Prime Infrastructure 3.0
• LiveAction 4.3
R10 R11 R12 R13
• All records available at:
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24
• http://docwiki.cisco.com/wiki/PfRv3:Reporting 10.1.13.0/24
 IOS-XE 3.16
IOS 15.5(3)M

PfRv3 Syslogs
• Syslog messages for all major PfRv3 events
• Use cisco standard format (Facility-Severity-Mnemonic) for all syslogs with common Facility name
'DOMAIN’
• Add TCA-ID to all syslog to allow correlation of TCA syslog to PFR reaction syslog. If PFR action is not
related to TCA then TCA-ID will be 0
• Command '[no] logging' in domain submode default is syslog on
• Distributed through SAF to all MCs and BRs in the domain
• http://docwiki.cisco.com/wiki/PfRv3:Syslogs
• DOMAIN-2-IME
• DOMAIN-2-IME_DETAILS *Jun 1 18:50:41.104: %DOMAIN-5-TC_PATH_CHG: Traffic class Path
• DOMAIN-4-MC_SHUTDOWN Changed. Details: Instance=0: VRF=default: Source Site ID=10.8.3.3:
• DOMAIN-5-TCA Destination Site ID=10.2.11.11: Reason=Delay: TCA-ID=4: Policy
Violated=VOICE: TC=[Site id=10.2.11.11, TC ID=6, Site
• DOMAIN-6-TC_CTRL prefix=10.1.11.0/24, DSCP=ef(46), App ID=0]: Original Exit=[CHAN-
• DOMAIN-5-TC_PATH_CHG ID=14, BR-IP=10.8.4.4, DSCP=ef[46], Interface=Tunnel100,
• DOMAIN-3-PLR_INT_CFG Path=MPLS[label=0:0 | 0:1 [0x1]]]: New Exit=[CHAN-ID=13, BR-
• DOMAIN-5-MC_STATUS IP=10.8.5.5, DSCP=ef[46], Interface=Tunnel200, Path=INET[label=0:0 |
0:2 [0x2]]]
 Management Solutions for Cisco Infrastructure
On-Prem Cloud-Based

Prime IWAN App

Infrastructure

Enterprise Network Prescriptive Application Aware Advanced

Mgmt and Monitoring Policy Automation Performance Mgmt Orchestration

• Customer needs feature • Customer wants massive • Customer looking for • Customer wants advanced
configurable enterprise simplicity and operational advanced monitoring and provisioning, life cycle
network management and automation visualization management, and
end-to-end monitoring customized policies
• Highly consistent network • Network troubleshooting and
• One Assurance across requirement with prescriptive QoS/ PfR/ AVC configuration • Multi-tenant
Cisco portfolio from Branch Cisco Validated Designs
to Datacenter • Real-time analytics and • System-wide network
• Lean IT flow/device scalability consistency assurance
• IT Network team
• IT Network team • Lean IT OR IT Network team
Two Deployment Modes for SDN led Provisioning
Control and Customization Business Driven
FEATURE CONFIGURABLE NMS with APIC-EM POLICY PRESCRIPTIVE APPS on APIC-EM
Prime Infra NMS integrated with APIC-EM Custom apps utilizing Cisco developed modular, policy Custom apps utilizing
providing full GUI based configuration and feature programmability automated management apps with policy programmability
FCAPS management orchestrated by the via Prime NB APIs for common UI/UX framework with and via APIC-EM NB
System of Automation configuration and data
embedded service automation REST APIs

Customer, Threat . Customer,

Segme
Partner or 3rd Collab Partner or 3rd
ntation Defense .
Cisco Prime Infrastructure party party
developed . developed
Automation IWAN Access Wireless Apps
.

Application Policy Infrastructure Controller (APIC) – Enterprise Module (EM)

PKI Service PnP Service

IWAN App
IWAN App
IWAN App
Key Takeaways
Performance Routing Phases – Summary
PfR/OER version 1 PfR version 2 PfR version 3 PfR version 3 PfR version 3
IOS 12.3(8)T, XE 2.6.1 IOS 15.2(3)T, IOS-XE IOS 15.4(3)M IOS 15.5(1)T IOS 15.5(2)T
3.6 IOS-XE 3.13 IOS-XE 3.14 IOS-XE 3.15

Per Device provisioning Per Device provisioning • PfR Domain • Zero SLA • Transit Sites
Passive monitoring with Target Discovery (TD) • One touch provisioning • WCCP Support • Multiple Next Hop per
Traditional NetFlow (TNF) Automatic provisioning of • Auto Discovery of sites DMVPN
Active monitoring with IP jitter probes • NBAR2 support • Multiple POPs
SLA Passive monitoring with • Passive Monitoring • Show last 5 TCA
Manual provisioning jitter Traditional NetFlow (TNF) (performance monitor)
probes Active monitoring with IP • Smart Probing
1000’s lines of SLA • VRF Awareness
configuration (pfr-map per 10’s lines of configuration • IPv4/IPv6 (Future)
site) • <10 lines of
configuration and
centralized

Blackout 6 seconds Blackout 6 seconds • Blackout ~ sub second

Brownout 9 seconds Brownout 9 seconds • Brownout ~ 2 sec
Limited scalability due to Scale 500 sites • Scale 2000 sites
provisioning (~ tens of
sites)
Key Takeaways
• IWAN Intelligent Path Control pillar is based upon Performance
Routing (PfR)
• Maximizes WAN bandwidth utilization
• Protects applications from performance degradation
• Enables the Internet as a viable WAN transport
• Provides multisite coordination to simplify network wide provisioning.
• Application-based policy driven framework and is tightly integrated with
existing AVC components.
• Smart and Scalable multi-sites solution to enforce application SLAs while
optimizing network resources utilization.

• PfRv3 is the 3rd generation Multi-Site aware Bandwidth and Path

Control/Optimization solution for WAN/Cloud based applications.
• Available now on ISR-G2, ISR-4000, CSR1000v, ASR1k
More Information
• Cisco.com IWAN and PfRv3 Page:
• http://www.cisco.com/go/iwan
• http://www.cisco.com/go/pfr
• DocWiki
• http://docwiki.cisco.com/wiki/PfRv3:Home
• CVD IWAN 2.0
• IWAN Technical Design Guide:
http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Jan2015/CVD-IWANDesignGuide-JAN15.pdf
• Intelligent WAN Config Files:
http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Jan2015/CVD-IWANConfigurationFilesGuide-JAN15.pdf
• IWAN Security for Remote Site DIA and Guest
Wirelesshttp://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-DIADesignGuide-Mar15.pdf
• IWAN Application Optimization using Cisco WAAS and Akamai
Connecthttp://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-WAASDesignGuide-
Mar15.pdf
Participate in the “My Favorite Speaker” Contest
Promote Your Favorite Speaker and You Could Be a Winner
• Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)
• Send a tweet and include
• Your favorite speaker’s Twitter handle @jbarozet
• Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers
• Don’t forget to follow @CiscoLive and @CiscoPress
• View the official rules at http://bit.ly/CLUSwin
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
• Complete your session surveys
though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you