Deploying MPLS Traffic

Engineering
James Moffat
Consulting Systems Engineer
[email protected]
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 1
 Agenda

• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 2
 Prerequisites

• Must know how to configure a router!

…Preferably Cisco…
• Basic knowledge of MPLS forwarding
push, pop, swap, etc.
• Some exposure to MPLS-TE helps
Not much time spent on basic configs

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 3
 Agenda

• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 4
 MPLS Is Key technology for delivery of L2
& L3 services
Optical IP
ATM IP
Services Services
Services Services

IP
IP
O-UNI MPLS
PNNI MPLS
IP+Optical Switch
IP+ATM Switch
Traffic Engineering: Optimization for IP+Optical Integration
IP+ATM Integration
Additional traffic =>$$
Frame
Frame
Relay
Relay

MPLS VPNs: Scalable

Protection solution ATM
Network based VPNs
Reduction in CAPEX & OPEX
Layer 2 Integration for
A single converged
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved.
Network Infrastructure5
 Introduction
• MPLS-TE was designed to move traffic along a
path other than the IGP shortest path
Bring ATM/FR traffic engineering abilities to an IP network
Avoid full IGP mesh and O(N2) flooding
Bandwidth-aware connection setup

• Fast ReRoute (FRR) is emerging as another

application of MPLS-TE
O(msec) of packet loss when a link goes down
Replace expensive SONET gear with routers
Can be used in conjunction with MPLS-TE for primary paths, can
also be used standalone

• Diffserv Aware Traffic Engineering

Delivering strict QOS guarantees through the integration of MPLS-
TE and advanced QOS techniques
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 6
 The Problem with Shortest-Path
• Some links are DS3, some
are OC-3
Node Next-Hop Cost
B B 10 • Router A has 40Mb of traffic for
C C 10 Route F, 40Mb of traffic for
D C 20
E B 20 Router G
F B 30
G B 30
• Massive (44%) packet loss at
Router B->Router E!
• Changing to A->C->D->E
Router B
won’t help
Router F
35M
OC-3 bD OC-3
Router A rop Router E
s!
ic DS3
Traff Router G
b
80M
OC-3
OC-3 DS3
DS3
Router C Router D
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 7
 What MPLS-TE Address
• Router A sees all links
Node Next-Hop Cost
B B 10 • Router A computes paths
C C 10
D C 20 on properties other than
E
F
B
Tunnel 0
20
30
just shortest cost
G Tunnel 1 30
• No link oversubscribed!

Router B
Router F

OC-3 OC-3
Router A Router E
DS3 Router G
0Mb
4
OC-3
OC-3 40Mb DS3
DS3
Router C Router D
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 8
 Fast ReRoute
• FRR: A mechanism to minimize packet loss during
a failure
• Pre-provision protection tunnels that carry traffic
when a protected resource (link/node) goes down
• Use MPLS-TE to signal the FRR protection
tunnels, taking advantage of the fact that MPLS-TE
traffic doesn’t have to follow the IGP shortest path
• Can protect MPLS traffic or IP traffic, depends on
the type of protection
• See later slides on FRR
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 9
 Agenda

• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 10
 How MPLS-TE Works

• Information distribution
• Path calculation
• Path setup
• Forwarding traffic down tunnels

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 11
 Information Distribution

• Need to tell the network about per-link

resources (mostly available bandwidth)
• This is done using extensions to IGP
(OSPF, ISIS)
• EIGRP, RIP not supported for MPLS-TE
EIGRP, RIP will work for other MPLS
applications (like VPNs!), just not for TE.

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 12
 Information Distribution

• OSPF
Uses type 10 (opaque area—local) lSAs
See draft-katz-yeung-ospf-traffic

router ospf 1
mpls traffic-eng area <x>
mpls traffic-eng router-id Loopback0

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 13
 Information Distribution

• IS-IS
Uses Type 22 TLVs
See draft-ietf-isis-traffic

router isis foo

mpls traffic-eng level-1|level-2
mpls traffic-eng router-id Loopback0
metric-style wide

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 14
 Path Calculation

• Modified Dijkstra at tunnel head-end

• Often referred to as CSPF
Constrained SPF
• …or PCALC (path calculation)

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 15
 Path Calculation
• What if there’s more than one path that
meets the minimum requirements
(bandwidth, etc.)?
• PCALC algorithm:
Find all paths with the lowest IGP cost
Then pick the path with the highest minimum
bandwidth along the path
Then pick the path with the lowest hop count
(not IGP cost, but hop count)
Then just pick one path at random
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 16
 Path Calculation
• Normal SPF—Find
shortest path across
“What’s the all links
Shortest Path to
All Routers?” • See Perlman (2nd ed),
Moy, etc. for
explanation of SPF
RtrB
RtrF

RtrA RtrE

RtrG

RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 17
 Path Calculation
• Normal SPF—Find
shortest path across
“What’s the all links
Shortest Path to
All Routers?” • See Perlman (2nd ed),
Moy, etc. for
explanation of SPF

RtrA

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 18
 Path Calculation
• Normal SPF—Find
shortest path across
“What’s the all links
Shortest Path to
All Routers?” • See Perlman (2nd ed),
Moy, etc. for
explanation of SPF
RtrB

RtrA

RtrC
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 19
 Path Calculation
• Normal SPF—Find
shortest path across
“What’s the all links
Shortest Path to
All Routers?” • See Perlman (2nd ed),
Moy, etc. for
explanation of SPF
RtrB

RtrA

RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 20
 Path Calculation
• Normal SPF—Find
shortest path across
“What’s the all links
Shortest Path to
All Routers?” • See Perlman (2nd ed),
Moy, etc. for
explanation of SPF
RtrB

RtrA RtrE

RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 21
 Path Calculation
• Normal SPF—Find
shortest path across
“What’s the all links
Shortest Path to
All Routers?” • See Perlman (2nd ed),
Moy, etc. for
explanation of SPF
RtrB
RtrF

RtrA RtrE

RtrG

RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 22
 Path Calculation
• Normal SPF—Find
shortest path across
“What’s the all links
Shortest Path to
All Routers?” • See Perlman (2nd ed),
Moy, etc. for
explanation of SPF
RtrB
RtrF

RtrA RtrE

RtrG

RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 23
 Path Calculation
• Normal SPF—Find
shortest path across
“What’s the all links
Shortest Path to
All Routers?” • See Perlman (2nd ed),
Moy, etc. for
explanation of SPF
RtrB
RtrF

RtrA RtrE

RtrG

RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 24
 Path Calculation
• Normal SPF—Find
shortest path across
“What’s the all links
Shortest Path to
All Routers?” • See Perlman (2nd ed),
Moy, etc. for
explanation of SPF
RtrB
RtrF

RtrA RtrE

RtrG

RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 25
 Path Calculation

• Constrained SPF—
“What’s the
Find shortest path to a
Shortest Path to
specific node
Router F with
40Mb Available?” • Consider more than
just link cost!
RtrB
RtrF
OC3
RtrA RtrE OC3
DS3
RtrG
OC3
OC3 DS3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 26
 Path Calculation

• Constrained SPF—
“What’s the
Find shortest path to a
Shortest Path to
specific node
Router F with
40Mb Available?” • Consider more than
just link cost!

RtrA

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 27
 Path Calculation

• Constrained SPF—
“What’s the
Find shortest path to a
Shortest Path to
specific node
Router F with
40Mb Available?” • Consider more than
just link cost!
RtrB

OC3
RtrA

OC3

RtrC
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 28
 Path Calculation

• Constrained SPF—
“What’s the
Find shortest path to a
Shortest Path to
specific node
Router F with
40Mb Available?” • Consider more than
just link cost!
RtrB

OC3
RtrA

OC3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 29
 Path Calculation

• Constrained SPF—
“What’s the
Find shortest path to a
Shortest Path to
specific node
Router F with
40Mb Available?” • Consider more than
just link cost!
RtrB

OC3
RtrA RtrE
DS3

OC3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 30
 Path Calculation

• Constrained SPF—
“What’s the
Find shortest path to a
Shortest Path to
specific node
Router F with
40Mb Available?” • Consider more than
just link cost!
RtrB
RtrF
OC3
RtrA RtrE OC3
DS3
RtrG
OC3
OC3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 31
 Path Calculation

• Constrained SPF—
“What’s the
Find shortest path to a
Shortest Path to
specific node
Router F with
40Mb Available?” • Consider more than
just link cost!
RtrB
RtrF
OC3
RtrA RtrE OC3
DS3
RtrG
OC3
OC3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 32
 Path Calculation

• Constrained SPF—
“What’s the
Find shortest path to a
Shortest Path to
specific node
Router F with
40Mb Available?” • Consider more than
just link cost!
RtrB
RtrF
OC3
RtrA RtrE OC3
DS3

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 33
 Path Calculation

• “But wait! There’s nothing different

between the two SPF results!”
• …but…

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 34
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrB
RtrF
115MB
RtrA RtrE115MB
5MB
RtrG
OC3
OC3 DS3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 35
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrA

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 36
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrB

115MB
RtrA

OC3

RtrC
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 37
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrB

115MB
RtrA

OC3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 38
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrB

115MB
RtrA RtrE
5MB

OC3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 39
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrB

115MB
RtrA RtrE
5MB

OC3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 40
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrB

115MB
RtrA

OC3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 41
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrA

OC3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 42
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrA RtrE

OC3 DS3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 43
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrF

RtrA RtrE115MB

RtrG
OC3
OC3 DS3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 44
 Path Calculation

“What’s the • What about the

Shortest Path to 2nd path?
Router G with
• Available bandwidth
40Mb Available?”
has changed!

RtrA RtrE

RtrG
OC3
OC3 DS3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 45
 Path Calculation

Node Next-Hop Cost

• End result:
B B 10
C C 10 Bandwidth used
D C 20
E B 20
efficiently!
F Tunnel0 30
G Tunnel1 30

RtrB
RtrF
OC3
RtrA RtrE OC3
DS3
RtrG
OC3
OC3 DS3

DS3
RtrC RtrD
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 46
 Path Calculation
• What if there’s more than one path that
meets the minimum requirements
(bandwidth, etc.)?
• PCALC algorithm:
Find all paths with the lowest IGP cost
Then pick the path with the highest minimum
bandwidth along the path
Then pick the path with the lowest hop count
(not IGP cost, but hop count)
Then just pick one path at “random” (take the
top path on the TENT list)
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 47
 Path Calculation

Path Has Cost of

What’s the Best {cost,available BW} 25, Not the
Lowest Cost!
Path from A to Z {10,100M}
with BW of 20M?

{8,80M}
RtrA RtrZ
{4,90M}

{8,90M}

All Left-side Links {8,90M} All Right-side Links

Are {10,100M} Are {5,150M}
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 48
 Path Calculation

Path Min BW Is
What’s the Best {cost,available BW} Lower than the
Other Paths!
Path from A to Z
with BW of 20M?

{8,80M}
RtrA RtrZ
{4,90M}

{8,90M}

All Left-side Links {8,90M} All Right-side Links

Are {10,100M} Are {5,150M}
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 49
 Path Calculation

Hop Count Is 5,
What’s the Best {cost,available BW} Other Paths
Are 4!
Path from A to Z
with BW of 20M?

RtrA RtrZ
{4,90M}

{8,90M}

All Left-side Links {8,90M} All Right-side Links

Are {10,100M} Are {5,150M}
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 50
 Path Calculation

Pick a Path at
What’s the Best {cost,available BW} Random!
Path from A to Z
with BW of 20M?

RtrA RtrZ

{8,90M}

All Left-side Links {8,90M} All Right-side Links

Are {10,100M} Are {5,150M}
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 51
 Path Calculation

What’s the Best {cost,available BW}

Path from A to Z
with BW of 20M?

RtrA RtrZ

{8,90M}

All Left-side Links All Right-side Links

Are {10,100M} Are {5,150M}
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 52
 Path Setup

• Cisco MPLS-TE uses RSVP

• RFC2205 (base RSVP), RFC 3209 (TE extensions
for RSVP)
• No CR-LDP; no plans for CR-LDP

• Once the path is calculated, it is handed to RSVP

• RSVP uses PATH and RESV messages to request
an LSP along the calculated path

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 53
 Path Setup

• PATH message: “Can I have 40Mb along this path?”

• RESV message: “Yes, and here’s the label to use”
• LFIB is set up along each hop
= PATH Messages
= RESV Messages
Router B
Router F

Router A Router E

Router G

Router C Router D
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 54
 Path Setup

• PATH message: “Can I have 40Mb along this path?”

• RESV message: “Yes, and here’s the label to use”
• LFIB is set up along each hop
= PATH Messages
= RESV Messages
Router B
Router F

Router A Router E

Router G

IMP
LI
23 NU CIT
15 LL

Router C 39 Router D
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 55
 Path Setup

• Errors along the way will trigger

RSVP errors
• May also trigger re-flooding of TE
information if appropriate

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 56
 Forwarding Traffic Down a Tunnel

• There are four ways traffic can be

forwarded down a TE tunnel
Static routes
Policy routing
Auto-route
Forwarding-adjacency
• With all but PBR, MPLS-TE gets you
unequal cost load balancing

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 57
 Static Routing

RtrA(config)#ip route H.H.H.H

255.255.255.255 Tunnel1

Router B
Router F
Router H
Router A Router E

Router G
Tunnel1

Router C Router D Router 1

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 58
 Static Routing
• Router H is known via
Node Next-Hop Cost the tunnel
B B 10
C C 10
D C 20 • Router G is not routed
E B 20
F
G
B
B
30
30
to over the tunnel,
H
I
Tunnel 1
B
40
40
even though it’s the
tunnel tail!
Router B
Router F
Router H
Router A Router E

Router G
Tunnel1

Router C Router D Router 1

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 59
 Policy Routing

RtrA(config-if)#ip policy route-map set-tunnel

RtrA(config)#route-map set-tunnel
RtrA(config-route-map)#match ip address 101
RtrA(config-route-map)#set interface Tunnel1

Router B
Router F
Router H
Router A Router E

Router G
Tunnel1

Router C Router D Router 1

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 60
 Policy Routing

Node Next-Hop Cost

• Routing table isn’t affected by
B B 10 policy routing
C C 10
D C 20
E B 20 • Need (12.0(16)ST or 12.2T) or
F B 30
G B 30 higher for ‘set interface tunnel’
H B 40
I B 40 to work (CSCdp54178)
Router B
Router F
Router H
Router A Router E

Router G
Tunnel1

Router C Router D Router 1

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 61
 Auto-Route

• Auto-route = “Use the tunnel as a directly

connected link for SPF purposes”
• This is not the CSPF (for path
determination), but the regular IGP SPF
(route determination)
• Behavior is intuitive, operation can be
confusing

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 62
 Auto-Route

This Is the Physical Topology

Router B
Router F
Router H
Router A Router E

Router G

Router C Router D Router I

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 63
 Auto-Route

• This is Router A’s logical topology

• By default, other routers don’t see
the tunnel!
Router B
Router F
Router H
Router A Router E

Tunnel1 Router G

Router C Router D Router I

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 64
 Auto-Route

• Router A’s routing table,

Node Next-Hop Cost
B B 10 built via auto-route
C C 10
D C 20
E B
B
20 • Everything “behind” the
F 30
G
H
Tunnel 1
Tunnel 1
30
40
tunnel is routed via the
I Tunnel 1 40
tunnel
Router B
Router F
Router H
Router A Router E

Tunnel1 Router G

Router C Router D Router I

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 65
 Auto-Route

• How does autoroute avoid loops?

Process SPF with physical links, replace
OIF with tunnel for tail and all subsequent
neighbors

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 66
 Forwarding Adjacency

• Autoroute metric change is purely local to

the headend
• This makes MPLS TE different from TE
with ATM
In ATM TE, the TE link (PVC) has its cost and
neighbor advertised into the network
In MPLS TE, no such thing is done—Until FA

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 67
 ATM Model

E F G
H

A I

C
B
D

• Cost of ATM links (blue) is unknown to routers

• A sees two links in IGP—E->H and B->D
• A can load-share between B and E
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 68
 Before FA

F G
E
H

A I

B C D

• All links have cost of 10

• A’s shortest path to I is A->B->C->D->I
• A doesn’t see TE tunnels on {E,B}, alternate path never gets used!
• Changing link costs is undesirable, can have strange
adverse effects
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 69
 F-A Advertises TE Tunnels in the IGP

F G
E
H

A I

B C D

• With forwarding-adjacency, A can see the TE tunnels as links

• A can then send traffic across both paths
• This is desirable in some topologies (looks just like ATM did,
same methodologies can be applied)
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 70
 F-A Issues

• In order for A to use F-A links, they need

to be the best cost IGP path
Otherwise the physical topo gets used
• F-A configured with
tunnel mpls traffic-eng forwarding-adjacency
isis metric <x> level-<y>

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 71
 F-A Issues

• Only ISIS supports F-A

OSPF support coming RSN
• F-A must be bidirectional
• IGP adjacency still not run over TE tunnel
• F-A cost should probably be lower than
lowest possible IGP path from head to tail,
otherwise it might not always get used

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 72
 Unequal Cost Load Balancing

• IP routing has equal-cost load balancing,

but not unequal cost*
• Unequal cost load balancing difficult to do
while guaranteeing a loop-free topology

*EIGRP Has ‘Variance’, but That’s Not as Flexible

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 73
 Unequal Cost Load Balancing

• Since MPLS doesn’t forward based on IP

header, permanent IGP routing loops
don’t happen with unequal cost
• 16 hash buckets for next-hop, shared in
rough proportion to configured tunnel
bandwidth or load-share value

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 74
 Unequal Cost: Example 1
Router F

Router A 40MB
Router E

Router G
20MB

gsr1#show ip route 192.168.1.8

Routing entry for 192.168.1.8/32
Known via "isis", distance 115, metric 83, type level-2
Redistributing via isis
Last update from 192.168.1.8 on Tunnel0, 00:00:21 ago
Routing Descriptor Blocks:
* 192.168.1.8, from 192.168.1.8, via Tunnel0
Route metric is 83, traffic share count is 2
192.168.1.8, from 192.168.1.8, via Tunnel1
Route metric is 83, traffic share count is 1

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 75
 Unequal Cost: Example 1
Router F

Router A 40MB
Router E

Router G
20MB

gsr1#sh ip cef 192.168.1.8 internal

………
Load distribution: 0 1 0 1 0 1 0 1 0 1 0 0 0 0 0 0 (refcount 1)
Hash OK Interface Address Packets Tags imposed
1 Y Tunnel0 point2point 0 {23}
2 Y Tunnel1 point2point 0 {34}
………

Note That the Load Distribution

Is 11:5—Very Close to 2:1, but Not Quite!

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 76
 Unequal Cost: Example 2
Router F

Router A Router E
100MB
10MB
1MB
Router G

gsr1#sh ip rou 192.168.1.8

Routing entry for 192.168.1.8/32
Known via "isis", distance 115, metric 83, type level-2
Redistributing via isis
Last update from 192.168.1.8 on Tunnel2, 00:00:08 ago
Routing Descriptor Blocks:
* 192.168.1.8, from 192.168.1.8, via Tunnel0
Route metric is 83, traffic share count is 100
192.168.1.8, from 192.168.1.8, via Tunnel1
Route metric is 83, traffic share count is 10
192.168.1.8, from 192.168.1.8, via Tunnel2
Route metric is 83, traffic share count is 1

Q: How Does 100:10:1 Fit into a 16-Deep Hash?

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 77
 Unequal Cost: Example 2
Router F

Router A Router E
100MB
10MB
1MB
Router G

gsr1#sh ip cef 192.168.1.8 internal

………
Load distribution: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (refcount 1)

Hash OK Interface Address Packets Tags imposed

1 Y Tunnel0 point2point 0 {36}
2 Y Tunnel1 point2point 0 {37}
………
A: Any Way It Wants to! 15:1, 14:2, 13:2:1, It Depends
on the Order the Tunnels Come Up
Deployment Guideline: Don’t Use Tunnel Metrics
That Don’t Reduce to 16 Buckets!
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 78
 Forwarding Traffic down a Tunnel

• You can use any combination of auto-

route, forwarding-adjacency, static routes,
or PBR
• …but simple is better unless you have a
good reason
• Recommendation: autoroute, forwarding-
adjacency, or statics to BGP next-hops,
depending on your needs

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 79
 Agenda

• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 80
 Fast ReRoute
• FRR: A mechanism to minimize packet loss
during a failure
• Pre-provision protection tunnels that carry
traffic when a protected resource (link/node)
goes down
• Use MPLS-TE to signal the FRR protection
tunnels, taking advantage of the fact that
MPLS-TE traffic doesn’t have to follow the IGP
shortest path
• Can protect MPLS traffic or IP traffic, depends on
the type of protection
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 81
 Fast Reroute

• In an IP network, a link failure causes several

seconds of outage

Thing Dependency Time

Media- and
Link Failure Detection Platform-specific ~µsecs (POS + APS)

IGP Timers, Network

Information ~5–30 sec
Size, Collective
Propagation
Router Load
Route Recalculation LSDB Size, CPU Load ~1–3 sec

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 82
 Fast Reroute

• In an MPLS network, there’s more work to be

done, so a (slightly) longer outage happens

Thing Dependency Time

Media- and
Link Failure Detection Platform-specific ~Usecs (POS + APS)

IGP Timers, Network

Information ~5–30 sec
Size, Collective
Propagation
Router Load
Route Recalculation LSDB Size, CPU Load ~1–3 sec
Network Size,
New LSP Setup ~5–10 sec
CPU Load

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 83
 Three Kinds of Fast Reroute

• Link protection
Implemented today
• Node protection
Implemented today
• Path protection
On development radar

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 84
 Link Protection

• TE Tunnel A -> B -> D -> E

10 34 POP
Router A Router B Router D Router E

Router C

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 85
 Link Protection

• B has a pre-provisioned backup tunnel to the

other end of the protected link (Router D)
• B relies on the fact that D is using global
label space
10 34 POP
Router A Router B Router D Router E

Protected
PLR Link MP
NHOP
27 POP

NHOP
Backup
Tunnel

Router C

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 86
 Link Protection

• B -> D link fails, A -> E tunnel is encapsulated in

B -> D tunnel
• Backup tunnel is used until A can re-compute tunnel
path as A -> B -> C -> D -> E (10–30 seconds or so)
10 POP
Router A Router B Router D Router E

27, 34 34

Router C

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 87
 Link Protection

• On tunnel head-end:
tunnel mpls traffic-eng fast-reroute

Router A Router B Router D Router E

• On protected link:
mpls traffic-eng backup-path <backup-tunnel>
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 88
 Node Protection

• Router A has a tunnel A -> B -> D -> E -> F

• Router B has a protect tunnel B -> C -> E -> D

Router A Router B Router D Router E Router F

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 89
 Node Protection

• Link protection is OK if the B -> D link goes down

• What if Router D goes away?

Router A Router B Router D Router E Router F

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 90
 Node Protection

• Solution: Protect tunnel to the hop past the

protected link
Protected
10 34 Node 22 POP
Router A Router B Router D Router E Router F

PLR NHOP MP
NNHOP
27 POP

NNHOP
Backup
Tunnel

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 91
 Node Protection

• Node protection still has the same convergence

properties as link protection
• Deciding where to place your backup tunnels is a
much harder to problem to solve large-scale
• For small-scale protection, link may be better
• Cisco has developed tools to solve these hard
problems for you (Tunnel Builder Pro)

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 92
 Path Protection

• Path protection: Multiple tunnels from TE head to

tail, across diverse paths

Router A Router B Router D Router E Router F

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 93
 Path Protection

• Path protection: Least scalable, most

resource-consuming, slowest
convergence of all 3 protection schemes
• Path protection is useful in two places:
1. When you have more links than tunnels
2. When you need to protect links not using
global label space

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 94
 Path vs. Local Protection

Local (Link/Node) Protection

Thing Dependency Time
Media- and
Link Failure Detection ~Usecs (POS + APS)
Platform-specific
Local Switch-over to RP->
~Few msec or Less
Protect Tunnel Communication Time

Path Protection
Thing Dependency Time
Media- and
Link Failure Detection ~Usecs (POS + APS)
Platform-specific
IGP Timers, Network
Information
Size, Collective ~5–30+ sec
Propagation
Router Load
Head-end Switch-over Network Size, ~Msec
to Protect LSP CPU Load
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 95
 Local Protection vs. APS Protection

Local (Link/Node) Protection

Thing Dependency Time
Media- and
Link Failure Detection ~Usecs (POS + APS)
Platform-specific
Local Switch-over to RP->
~Few msec or Less
Protect Tunnel Communication Time

APS Protection
Thing Dependency Time
Media-and
Link Failure Detection ~Usecs (POS + APS)
Platform-specific

Generally a
APS/MSP Cutover <50ms, per spec
Fixed Time

IGP Reconverges IGP Timers, Seconds or Less

on New Link IGP Size, CPU Load, Etc.
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 96
 Agenda

• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 97
 Design

• Two ways to deploy MPLS-TE

As needed to clear up congestion—Tactical
Full mesh between a set of routers—Strategic
• Strategic can be online or offline path
calculation
• Both methods are valid, both have their
pros and cons

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 98
 Tactical
Case Study: A Large US ISP

Router A

Router B Router C

• All links are OC12

• A has consistent ±700MB to
send to C
• ~100MB constantly dropped!

Router D Router E

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 99
 Tactical

• Solution: Multiple tunnels, unequal cost

load sharing!
Router A

Router B Router C
• Tunnels with bandwidth in 3:1
(12:4) ratio
• 25% of traffic sent the long way
• 75% sent the short way
• No out-of-order packet issues—
CEF’s normal per-flow hashing
is used!
Router D Router E

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 100
 Tactical

• From Router A’s perspective,

topology is:
Router A

Router B Router C

Router D Router E

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 101
 Tactical

• As needed—Easy, quick, but hard to track

over time
• Easy to forget why a tunnel is in place
• Inter-node BW requirements may change,
tunnels may be working around issues
that no longer exist
• Link protection pretty straightforward,
node protection harder to track

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 102
 Strategic

• Put a full mesh of TE tunnels

between routers
• Initially deploy tunnels with 0 bandwidth
• Watch tunnel interface statistics, see how
much bandwidth you are using between
router pairs
Tunnels are interfaces—Use IF-MIB!
Make sure that Σtunnel <= Σnetwork BW

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 103
 Strategic

• Some folks deploy full mesh just to get

router-to-router (pop-to-pop) traffic matrix
• Largest TE network ~80 routers full mesh
(~6400 tunnels)
• As tunnel bandwidth is changed, tunnels
will find the best path across your network

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 104
 Strategic

• Physical topology is:

Router A

Router B Router C

Router D Router E

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 105
 Strategic
• Logical topology is*
*Each link is actually 2 unidirectional tunnels
• Total of 20 tunnels in this network
Router A

Router B Router C

Router D Router E

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 106
 Strategic

• Things to remember with full mesh

N routers, N*(N-1) tunnels
Routing protocols not run over TE tunnels—
Unlike an ATM/FR full mesh!
Tunnels are unidirectional—This is a
good thing
…Can have different bandwidth reservations
in two different directions

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 107
 Strategic

• Best practices for full mesh:

Periodically re-optimize tunnels based on need
(just like an ATM network)
TE was designed to be a combination of online
(router-based) and offline (NMS) calculation
Node protection more practical in a full-mesh,
offline-generate TE topography

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 108
 Recommended Reading

• Traffic Engineering
with MPLS
ISBN: 1-58705-031-5

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 109
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 110
 Scalability
How Many Tunnels on a Router?

Number Number Number of

Code of Head-End of Mid-Points Tunnel Tails
Tunnels

12.0ST 600 10,000 5,000

• Tests were done on a GSR

• RSP4, RSP8, VXR300, VXR400 will be similar
• 10,000 tunnels come up in 3-5 minutes
RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 111
 Scalability

• Largest TE network today = 80 routers,

~6400 tunnels full mesh
• 12.0ST—600 head-ends, 360,000 tunnels
full mesh with 10,000 tunnels per midpoint
• 600 = 80*7.50
Or (360,000=6400*56) if you’re in marketing
• Bottom line: MPLS-TE is not a gating
factor in scaling most networks!

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 112
 Scalability

http://www.cisco.com/univercd/cc/td/doc/pro
duct/software/ios120/120newft/120limit/120st
/120st14/scalable.htm

• Or just search CCO for “Scalability

Enhancements for MPLS Traffic
Engineering”

RST-251
5465_05_2002_c1 © 2002, Cisco Systems, Inc. All rights reserved. 113