Tutorial BGP Multihoming Techniques SANOG24

BGP Multihoming Tutorial
SANOG 22
4 August 2014
th
Srinath Beldona
Senior Technical Consultant
[email protected]
APNIC Training and Development

Agenda
p Simple Multihoming
p Service Provider Multihoming
p Conclusion
Simple Multihoming
SANOG 24
Last updated 25 July 2013 3

Why Multihome?
p Redundancy
n One connection to internet means the network
is dependent on:
p Local router (configuration, software, hardware)
p WAN media (physical failure, carrier failure)
p Upstream Service Provider (configuration, software,

hardware)
4
Why Multihome?
p Reliability
n Business critical applications demand
continuous availability
n Lack of redundancy implies lack of reliability
implies loss of revenue
5
Why Multihome?
p Supplier Diversity
n Many businesses demand supplier diversity as
a matter of course
n Internet connection from two or more suppliers
p With two or more diverse WAN paths
p With two or more exit points
p With two or more international connections
p Two of everything
6
Why Multihome?
p Not really a reason, but often quoted
p Leverage:
n Playing one ISP off against the other for:
p Service Quality
p Service Offerings
p Availability
7
Why Multihome?
p Summary:
n Multihoming is easy to demand as requirement
of any operation
n But what does it really mean:
p In real life?
p For the network?
p For the Internet?
n And how do we do it?
8
Multihoming Definition
p More
than one link external to the local
network
n two or more links to the same ISP
n two or more links to different ISPs
p Usually two external facing routers

n one router gives link and provider redundancy
only
9
Multihoming
p The scenarios described here apply equally
well to end sites being customers of ISPs
and ISPs being customers of other ISPs
p Implementation detail may be different
n end site ISP ISP controls config
n ISP1 ISP2 ISPs share config
10
Autonomous System Number
(ASN)
p Two ranges
0-65535 (original 16-bit range)
65536-4294967295 (32-bit range RFC6793)
p Usage:
0 and 65535 (reserved)
1-64495 (public Internet)
64496-64511 (documentation RFC5398)
64512-65534 (private use only)
23456 (represent 32-bit range in 16-bit world)
65536-65551 (documentation RFC5398)
65552-4199999999 (public Internet)
4200000000-4294967295 (private use only)
p 32-bit range representation specified in RFC5396
n Defines asplain (traditional format) as standard notation
11
Autonomous System Number
(ASN)
p ASNs are distributed by the Regional Internet
Registries
n They are also available from upstream ISPs who are
members of one of the RIRs
p Current 16-bit ASN allocations up to 63487 have
been made to the RIRs
n Around 44500 are visible on the Internet
n Around 1500 left unassigned
p Each RIR has also received a block of 32-bit ASNs
n Out of 4800 assignments, around 3700 are visible on
the Internet
p See www.iana.org/assignments/as-numbers
12
Private-AS Application
65001
p Applications 193.0.32.0/24 65002
n An ISP with customers 193.0.33.0/24
multihomed on their
backbone (RFC2270)
-or- C
1880
n A corporate network 193.0.34.0/24 65003
with several regions B 193.0.35.0/24
but connections to the
A
Internet only in the
core
-or- 193.0.32.0/22 1880
n Within a BGP
Confederation
13
Private-AS Removal
p Private
ASNs MUST be removed from all
prefixes announced to the public Internet
n Include configuration to remove private ASNs
in the eBGP template
p As
with RFC1918 address space, private
ASNs are intended for internal use
n They should not be leaked to the public
Internet
p Cisco IOS
neighbor x.x.x.x remove-private-AS
14
Transit/Peering/Default
p Transit
n Carrying traffic across a network
n Usually for a fee
p Peering
n Exchanging locally sourced routing information
and traffic
n Usually for no fee
n Sometimes called settlement free peering
p Default
n Where to send traffic when there is no explicit
match in the routing table
Configuring Policy
p Assumptions:
n prefix-lists are used throughout
n easier/better/faster than access-lists
p Three BASIC Principles

n prefix-lists to filter prefixes
n filter-lists to filter ASNs
n route-maps to apply policy
p Route-maps can be used for filtering, but

this is more advanced configuration
16
Policy Tools
p Local preference
n outbound traffic flows
p Metric (MED)
n inbound traffic flows (local scope)
p AS-PATH prepend
n inbound traffic flows (Internet scope)
p Communities
n specific inter-provider peering
17
Originating Prefixes: Assumptions
p MUST announce assigned address block to
Internet
p MAY also announce subprefixes
reachability is not guaranteed
p Current minimum allocation is from /20
to /24 depending on the RIR
n Several ISPs filter RIR blocks on this boundary
n Several ISPs filter the rest of address space
according to the IANA assignments
n This activity is called Net Police by some
18
Originating Prefixes
p The RIRs publish their minimum allocation sizes per /8 address block
n AfriNIC: www.afrinic.net/docs/policies/afpol-v4200407-000.htm
n APNIC: www.apnic.net/db/min-alloc.html
n ARIN: www.arin.net/reference/ip_blocks.html
n LACNIC: lacnic.net/en/registro/index.html
n RIPE NCC: www.ripe.net/ripe/docs/smallest-alloc-sizes.html
n Note that AfriNIC only publishes its current minimum allocation size, not
the allocation size for its address blocks
p IANA publishes the address space it has assigned to end-sites and
allocated to the RIRs:
n www.iana.org/assignments/ipv4-address-space
p Several ISPs use this published information to filter prefixes on:
n What should be routed (from IANA)
n The minimum allocation size from the RIRs
Net Police prefix list issues
p Meant to punish ISPs who pollute the routing table with
specifics rather than announcing aggregates
p Impacts legitimate multihoming especially at the Internets
edge
p Impacts regions where domestic backbone is unavailable or
costs $$$ compared with international bandwidth
p Hard to maintain requires updating when RIRs start
allocating from new address blocks
p Dont do it unless consequences understood and you are
prepared to keep the list current
n Consider using the Team Cymru or other reputable bogon BGP
feed:
n www.team-cymru.org/Services/Bogons/routeserver.html
20
How to Multihome
Some choices
21
Transits
p Transit provider is another autonomous system which
is used to provide the local network with access to
other networks
n Might be local or regional only
n But more usually the whole Internet
p Transit providers need to be chosen wisely:
n Only one
p no redundancy
n Too many
p more difficult to load balance
p no economy of scale (costs more per Mbps)
p hard to provide service quality
p Recommendation: at least two, no more than
three
Common Mistakes
p ISPs sign up with too many transit providers
n Lots of small circuits (cost more per Mbps than larger
ones)
n Transit rates per Mbps reduce with increasing transit
bandwidth purchased
n Hard to implement reliable traffic engineering that
doesnt need daily fine tuning depending on customer
activities
p No diversity
n Chosen transit providers all reached over same satellite
or same submarine cable
n Chosen transit providers have poor onward transit and
peering
Peers
p A peer is another autonomous system with which
the local network has agreed to exchange locally
sourced routes and traffic
p Private peer
n Private link between two providers for the purpose of
interconnecting
p Public peer
n Internet Exchange Point, where providers meet and
freely decide who they will interconnect with
p Recommendation: peer as much as possible!
Common Mistakes
p Mistaking a transit providers Exchange
business for a no-cost public peering point
p Not working hard to get as much peering
as possible
n Physically near a peering point (IXP) but not
present at it
n (Transit sometimes is cheaper than peering!!)
p Ignoring/avoiding
competitors because
they are competition
n Even though potentially valuable peering
partner to give customers a better experience
Multihoming Scenarios
p Stub network
p Multi-homed stub network
p Multi-homed network
p Multiple Sessions to another AS
26
Stub Network
AS101
AS100
p No need for BGP

p Point static default to upstream ISP
p Upstream ISP advertises stub network
p Policy confined within upstream ISPs policy
Multi-homed Stub Network
AS65530
AS100
p Use BGP (not IGP or static) to loadshare

p Use private AS (ASN > 64511)
p Upstream ISP advertises stub network
p Policy confined within upstream ISPs policy
Multi-homed Network
Global Internet
AS300 AS200
AS100
p Many situations possible

n multiple sessions to same ISP
n secondary for backup only
n load-share between primary and secondary
n selectively use different ISPs
Multiple Sessions to an ISP
p Several options
n ebgp multihop
ISP
n bgp multipath
n cef loadsharing
n bgp attribute manipulation
AS 201
30
Multiple Sessions to an AS
ebgp multihop
p Use ebgp-multihop
n Run eBGP between loopback addresses
n eBGP prefixes learned with loopback address as
next hop
AS 200
p Cisco IOS 1.1.1.1
router bgp 100
neighbor 1.1.1.1 remote-as 200 B
neighbor 1.1.1.1 ebgp-multihop 2
!
ip route 1.1.1.1 255.255.255.255 serial 1/0
ip route 1.1.1.1 255.255.255.255 serial 1/1
ip route 1.1.1.1 255.255.255.255 serial 1/2
p Common error made is to point remote A
loopback route at IP address rather than
specific link
AS 100
ebgp multihop
p One serious eBGP-multihop
caveat:
n R1 and R3 are eBGP peers
that are loopback peering
R1 R3
n Configured with:
neighbor x.x.x.x ebgp-multihop 2 AS 100 AS 200
n If the R1 to R3 link goes
down the session could
establish via R2 R2
p Usually happens when
routing to remote loopback
is dynamic, rather than
static pointing at a link Desired Path
Used Path
Multiple Sessions to an ISP
ebgp multihop
p Tryand avoid use of ebgp-multihop
unless:
n Its absolutely necessary or
n Loadsharing across multiple links
p Many
ISPs discourage its use, for
example:
We will run eBGP multihop, but do not support it as a standard offering
because customers generally have a hard time managing it due to:
routing loops
failure to realise that BGP session stability problems are usually due
connectivity problems between their CPE and their BGP speaker
33
bgp multi path
p Three BGP sessions required
p Platform limit on number of paths
(could be as little as 6) AS 200
p Full BGP feed makes this unwieldy
n 3 copies of Internet Routing Table
goes into the FIB
router bgp 100

neighbor 1.1.2.1 remote-as 200
maximum-paths 3
AS 100
bgp attributes & filters
p Simplest scheme is to use
defaults
p Learn/advertise prefixes for AS 200
better control
p Planning and some work
required to achieve loadsharing C D
n Point default towards one ISP
n Learn selected prefixes from
second ISP
n Modify the number of prefixes
learnt to achieve acceptable load
sharing A B
No magic solution
p
AS 201
Basic Principles of
Multihoming
Lets learn to walk before we try
running
36
The Basic Principles
p Announcing address space attracts traffic
n (Unless policy in upstream providers
interferes)
p Announcing the ISP aggregate out a link
will result in traffic for that aggregate
coming in that link
p Announcing a subprefix of an aggregate
out a link means that all traffic for that
subprefix will come in that link, even if the
aggregate is announced somewhere else
n The most specific announcement wins!
37
p To split traffic between two links:
n Announce the aggregate on both links - ensures
redundancy
n Announce one half of the address space on each link
n (This is the first step, all things being equal)
p Results in:
n Traffic for first half of address space comes in first link
n Traffic for second half of address space comes in second
link
n If either link fails, the fact that the aggregate is
announced ensures there is a backup path
38
p The
keys to successful multihoming
configuration:
n Keeping traffic engineering prefix
announcements independent of customer iBGP
n Understanding how to announce aggregates
n Understanding the purpose of announcing
subprefixes of aggregates
n Understanding how to manipulate BGP
attributes
n Too many upstreams/external paths makes
multihoming harder (2 or 3 is enough!)
39
IP Addressing &
Multihoming
How Good IP Address Plans
assist with Multihoming
40
IP Addressing & Multihoming
p IP Address planning is an important part of
Multihoming
p Previously have discussed separating:
n Customer address space
n Customer p-t-p link address space
n Infrastructure p-t-p link address space
n Loopback address space
101.10.0.0/21
101.10.0.1 101.10.5.255 101.10.6.255 /24
Customer Address & p-t-p links Infrastructure Loopbacks

41
IP Addressing & Multihoming
p ISP Router loopbacks and backbone point to point
links make up a small part of total address space
n And they dont attract traffic, unlike customer address
space
p Links from ISP Aggregation edge to customer
router needs one /30
n Small requirements compared with total address space
n Some ISPs use IP unnumbered
p Planning customer assignments is a very
important part of multihoming
n Traffic engineering involves subdividing aggregate into
pieces until load balancing works
42
Unplanned IP addressing
p ISP fills up customer IP addressing from one end
of the range:
101.10.0.0/21
12345
Customer Addresses ISP
p Customers generate traffic

n Dividing the range into two pieces will result in one /22
with all the customers, and one /22 with just the ISP
infrastructure the addresses
n No loadbalancing as all traffic will come in the first /22
n Means further subdivision of the first /22 = harder work
43
Planned IP addressing
p If ISP fills up customer addressing from both
ends of the range:
101.10.0.0/21
13579 2 4 6 810
Customer Addresses Customer Addresses ISP

p Scheme then is:
n First customer from first /22, second customer from
second /22, third from first /22, etc
p This works also for residential versus commercial
customers:
n Residential from first /22
n Commercial from second /22
44
Planned IP Addressing
p This works fine for multihoming between
two upstream links (same or different
providers)
p Can also subdivide address space to suit
more than two upstreams
n Follow a similar scheme for populating each
portion of the address space
p Dont
forget to always announce an
aggregate out of each link
45
Basic Multihoming
Lets try some simple worked
examples
46
Basic Multihoming
p No frills multihoming
p Will look at two cases:
n Multihoming with the same ISP
n Multihoming to different ISPs
p Will keep the examples easy

n Understanding easy concepts will make the
more complex scenarios easier to comprehend
n All assume that the site multihoming has a /19
address block
47
Basic Multihoming
p This
type is most commonplace at the
edge of the Internet
n Networks here are usually concerned with
inbound traffic flows
n Outbound traffic flows being nearest exit is
usually sufficient
p Can
apply to the leaf ISP as well as
Enterprise networks
48
Two links to the same ISP
One link primary, the other link
backup only
49
(one as backup only)
p Applies
when end-site has bought a large
primary WAN link to their upstream a
small secondary WAN link as the backup
n For example, primary path might be an E1,
backup might be 64kbps
50
primary
C
A
AS 100 AS 65534
E B
D
backup
p AS100
removes private AS and any
customer subprefixes from Internet
announcement
51
p Announce /19 aggregate on each link
n primary link:
p Outbound announce /19 unaltered
p Inbound receive default route
n backup link:
p Outbound announce /19 with increased metric
p Inbound received default, and reduce local
preference
p Whenone link fails, the announcement of
the /19 aggregate via the other link
ensures continued connectivity
52
p Router A Configuration
router bgp 65534
network 121.10.0.0 mask 255.255.224.0
neighbor 122.102.10.2 description RouterC
neighbor 122.102.10.2 prefix-list aggregate out
neighbor 122.102.10.2 prefix-list default in
!
ip prefix-list aggregate permit 121.10.0.0/19
ip prefix-list default permit 0.0.0.0/0
!
ip route 121.10.0.0 255.255.224.0 null0
53
p Router B Configuration
router bgp 65534
network 121.10.0.0 mask 255.255.224.0
neighbor 122.102.10.6 description RouterD
neighbor 122.102.10.6 route-map routerD-out out
neighbor 122.102.10.6 route-map routerD-in in
!
..next slide
54
!
ip route 121.10.0.0 255.255.224.0 null0
!
route-map routerD-out permit 10
set metric 10
!
route-map routerD-in permit 10
set local-preference 90
!
55
p Router C Configuration (main link)
router bgp 100
neighbor 122.102.10.1 default-originate
neighbor 122.102.10.1 prefix-list Customer in
neighbor 122.102.10.1 prefix-list default out
!
ip prefix-list Customer permit 121.10.0.0/19
56
p Router D Configuration (backup link)
router bgp 100
!
57
p Router E Configuration
router bgp 100
neighbor 122.102.10.17 remove-private-AS
neighbor 122.102.10.17 prefix-list Customer out
!
p Router E removes the private AS and customers
subprefixes from external announcements
p Private AS still visible inside AS100
58
With Loadsharing
59
Loadsharing to the same ISP
p More common case
p End sites tend not to buy circuits and
leave them idle, only used for backup as
in previous example
p This example assumes equal capacity
circuits
n Unequal capacity circuits requires more
refinement see later
60
Link one
C
A
AS 100 AS 65534
E B
D
Link two
p Border router E in AS100 removes private AS and any

customer subprefixes from Internet announcement
61
(with redundancy)
p Split /19 and announce as two /20s, one on each
link
n basic inbound loadsharing
n assumes equal circuit capacity and even spread of traffic
across address block
p Vary the split until perfect loadsharing achieved
p Accept the default from upstream
n basic outbound loadsharing by nearest exit
n okay in first approx as most ISP and end-site traffic is
inbound
62
(with redundancy)
router bgp 65534
network 121.10.0.0 mask 255.255.224.0
network 121.10.0.0 mask 255.255.240.0
neighbor 122.102.10.2 prefix-list routerC out
!
ip prefix-list routerC permit 121.10.0.0/20
!
ip route 121.10.0.0 255.255.240.0 null0
ip route 121.10.0.0 255.255.224.0 null0
63
(with redundancy)
router bgp 65534
network 121.10.0.0 mask 255.255.224.0
network 121.10.16.0 mask 255.255.240.0
neighbor 122.102.10.6 prefix-list routerD out
!
ip prefix-list routerD permit 121.10.16.0/20
!
ip route 121.10.16.0 255.255.240.0 null0
ip route 121.10.0.0 255.255.224.0 null0
64
(with redundancy)
p Router C Configuration
router bgp 100
!
ip prefix-list Customer permit 121.10.0.0/19 le 20
p Router C only allows in /19 and /20 prefixes from
customer block
p Router D configuration is identical
65
(with redundancy)
router bgp 100
neighbor 122.102.10.17 prefix-list Customer out
!
p Private AS still visible inside AS100
66
(with redundancy)
p Default route for outbound traffic?
n Use default-information originate for the IGP
and rely on IGP metrics for nearest exit
n e.g. on router A:
router ospf 65534

default-information originate metric 2 metric-type 1
Or
router isis as65534

default-information originate
67
(with redundancy)
p Loadsharing configuration is only on
customer router
p Upstream ISP has to
n remove customer subprefixes from external
announcements
n remove private AS from external
announcements
p Could also use BGP communities
68
Multiple Dualhomed Customers
(RFC2270)
69
(RFC2270)
p Unusual
for an ISP just to have one
dualhomed customer
n Valid/valuable service offering for an ISP with
multiple PoPs
n Better for ISP than having customer multihome
with another provider!
p Look at scaling the configuration
n Simplifying the configuration
n Using templates, peer-groups, etc
n Every customer has the same configuration
(basically)
70
(RFC2270)
C
A1
AS 65534
AS 100 B1
E D A2
AS 65534
B2
A3
AS 65534
p Border router E in AS100 removes
private AS and any customer B3
subprefixes from Internet
announcement 71
(RFC2270)
p Customer announcements as per previous
example
p Use the same private AS for each
customer
n documented in RFC2270
n address space is not overlapping
n each customer hears default only
p Router
An and Bn configuration same as
Router A and B previously
72
(RFC2270)
p Router A1 Configuration
router bgp 65534
network 121.10.0.0 mask 255.255.224.0
network 121.10.0.0 mask 255.255.240.0
neighbor 122.102.10.2 prefix-list routerC out
!
!
ip route 121.10.0.0 255.255.240.0 null0
ip route 121.10.0.0 255.255.224.0 null0
73
(RFC2270)
p Router B1 Configuration
router bgp 65534
network 121.10.0.0 mask 255.255.224.0
network 121.10.16.0 mask 255.255.240.0
neighbor 122.102.10.6 prefix-list routerD out
!
!
ip route 121.10.0.0 255.255.224.0 null0
ip route 121.10.16.0 255.255.240.0 null0
74
(RFC2270)
router bgp 100
neighbor bgp-customers peer-group
neighbor bgp-customers remote-as 65534
neighbor bgp-customers default-originate
neighbor bgp-customers prefix-list default out
neighbor 122.102.10.1 peer-group bgp-customers
neighbor 122.102.10.1 description Customer One
neighbor 122.102.10.1 prefix-list Customer1 in
neighbor 122.102.10.9 description Customer Two
75
(RFC2270)
neighbor 122.102.10.17 description Customer Three
!
ip prefix-list Customer1 permit 121.10.0.0/19 le 20
p Router C only allows in /19 and /20 prefixes from

customer block
76
(RFC2270)
p Router D Configuration
router bgp 100
neighbor bgp-customers peer-group
neighbor bgp-customers remote-as 65534
neighbor bgp-customers default-originate
neighbor bgp-customers prefix-list default out
neighbor 122.102.10.5 description Customer One
neighbor 122.102.10.13 description Customer Two
77
(RFC2270)
neighbor 122.102.10.21 description Customer Three
!
p Router D only allows in /19 and /20 prefixes from

customer block
78
(RFC2270)
nassumes customer address space is not part of
upstreams address block
router bgp 100
neighbor 122.102.10.17 prefix-list Customers out
!
ip prefix-list Customers permit 121.10.0.0/19
p Private AS still visible inside AS100 79

(RFC2270)
p If customers prefixes come from ISPs address
block
n do NOT announce them to the Internet
n announce ISP aggregate only
p Router E configuration:
router bgp 100

neighbor 122.102.10.17 prefix-list my-aggregate out
!
ip prefix-list my-aggregate permit 121.8.0.0/13
80
Multihoming Summary
p Use private AS for multihoming to the
same upstream
p Leak subprefixes to upstream only to aid
loadsharing
p Upstream router E configuration is
identical across all situations
81
Basic Multihoming
Multihoming to Different ISPs
82
Two links to different ISPs
p Use a Public AS
n Or use private AS if agreed with the other ISP
n But some people dont like the inconsistent-
AS which results from use of a private-AS
p Address space comes from
n both upstreams or
n Regional Internet Registry
p Configuration concepts very similar
83
Inconsistent-AS?
AS 65534
p Viewing the prefixes
originated by AS65534 in
the Internet shows they
appear to be originated
by both AS210 and
AS200 AS 200
n This is NOT bad
n Nor is it illegal
AS 210
p IOS command is
show ip bgp inconsistent-as
Internet
84
Two links to different
ISPs
One link primary, the other link
backup only
85
Internet
AS 100 AS 120
C D
Announce /19 block

Announce /19 block with longer AS PATH
A B
AS 130
86
n primary link makes standard announcement
n backup link lengthens the AS PATH by using
AS PATH prepend
the /19 aggregate via the other link
87
router bgp 130
network 121.10.0.0 mask 255.255.224.0
!
!
ip route 121.10.0.0 255.255.224.0 null0
88
router bgp 130
network 121.10.0.0 mask 255.255.224.0
neighbor 120.1.5.1 route-map routerD-out out
neighbor 120.1.5.1 route-map routerD-in in
!
!
route-map routerD-out permit 10
set as-path prepend 130 130 130
!
route-map routerD-in permit 10
89
p Not
a common situation as most sites tend
to prefer using whatever capacity they
have
n (Useful when two competing ISPs agree to
provide mutual backup to each other)
p But
it shows the basic concepts of using
local-prefs and AS-path prepends for
engineering traffic in the chosen direction
90
ISPs
With Loadsharing
91
(with loadsharing)
Internet
AS 100 AS 120
C D
Announce first Announce second

/20 and /19 block /20 and /19 block
A B
AS 130
92
(with loadsharing)
p Split /19 and announce as two /20s, one
on each link
n basic inbound loadsharing
the /19 aggregate via the other ISP
93
(with loadsharing)
router bgp 130
network 121.10.0.0 mask 255.255.224.0
network 121.10.0.0 mask 255.255.240.0
neighbor 122.102.10.1 prefix-list firstblock out
!
!
ip prefix-list firstblock permit 121.10.0.0/20
ip prefix-list firstblock permit 121.10.0.0/19
94
(with loadsharing)
router bgp 130
network 121.10.0.0 mask 255.255.224.0
network 121.10.16.0 mask 255.255.240.0
neighbor 120.1.5.1 prefix-list secondblock out
!
!
ip prefix-list secondblock permit 121.10.16.0/20
ip prefix-list secondblock permit 121.10.0.0/19
95
(with loadsharing)
p Loadsharing in this case is very basic
p But shows the first steps in designing a
load sharing solution
n Start with a simple concept
n And build on it!
96
ISPs
More Controlled Loadsharing
97
Loadsharing with different ISPs
Internet
AS 100 AS 120
C D
Announce /20 subprefix,

Announce /19 block and /19 block with
A B
longer AS path
AS 130
98
n On first link, announce /19 as normal
n On second link, announce /19 with longer AS
PATH, and announce one /20 subprefix
p controls loadsharing between upstreams and the
Internet
p Vary the subprefix size and AS PATH
length until perfect loadsharing achieved
p Still require redundancy!
99
router bgp 130
network 121.10.0.0 mask 255.255.224.0
!
!
ip route 121.10.0.0 255.255.224.0 null0
100
router bgp 130
network 121.10.0.0 mask 255.255.224.0
network 121.10.16.0 mask 255.255.240.0
neighbor 120.1.5.1 prefix-list subblocks out
neighbor 120.1.5.1 route-map routerD out
!
route-map routerD permit 10
match ip address prefix-list aggregate
set as-path prepend 130 130
route-map routerD permit 20
!
ip prefix-list subblocks permit 121.10.0.0/19 le 20
101
p This example is more commonplace
p Shows how ISPs and end-sites subdivide
address space frugally, as well as use the
AS-PATH prepend concept to optimise the
load sharing between different ISPs
p Notice that the /19 aggregate block is
ALWAYS announced
102
Summary
103
Summary
p Previous examples dealt with simple case
p Load balancing inbound traffic flow
n Achieved by modifying outbound routing
announcements
n Aggregate is always announced
p Wehave not looked at outbound traffic

flow
n For now this is left as nearest exit
104
Simple Multihoming
ISP Workshops
105
Service Provider
Multihoming
ISP Workshops
Last updated 25 July 2013 106

Service Provider Multihoming
p Previous
examples dealt with loadsharing
inbound traffic
n Of primary concern at Internet edge
n What about outbound traffic?
p Transit
ISPs strive to balance traffic flows
in both directions
n Balance link utilisation
n Try and keep most traffic flows symmetric
n Some edge ISPs try and do this too
p The original Traffic Engineering

107
p Balancing
outbound traffic requires
inbound routing information
n Common solution is full routing table
n Rarely necessary
p Why use the routing mallet to try solve loadsharing
problems?
n Keep It Simple is often easier (and $$$
cheaper) than carrying N-copies of the full
routing table
108
MYTHS!!
Common MYTHS
1. You need the full routing table to multihome
n People who sell router memory would like you to believe this
n Only true if you are a transit provider
n Full routing table can be a significant hindrance to
multihoming
2. You need a BIG router to multihome
n Router size is related to data rates, not running BGP
n In reality, to multihome, your router needs to:
p Have two interfaces,
p Be able to talk BGP to at least two peers,
p Be able to handle BGP attributes,
p Handle at least one prefix
3. BGP is complex
n In the wrong hands, yes it can be! Keep it Simple!
109
Service Provider Multihoming:
Some Strategies
p Take
the prefixes you need to aid traffic
engineering
n Look at NetFlow data for popular sites
p Prefixes
originated by your immediate
neighbours and their neighbours will do
more to aid load balancing than prefixes
from ASNs many hops away
n Concentrate on local destinations
p Use default routing as much as possible
n Or use the full routing table with care
110
p Examples
n One upstream, one local peer
n One upstream, local exchange point
n Two upstreams, one local peer
n Three upstreams, unequal link bandwidths
p Require BGP and a public ASN

p Examples assume that the local network
has their own /19 address block
111
Service Provider
Multihoming
One upstream, one local peer
112
One Upstream, One Local Peer
p Very common situation in many regions of
the Internet
p Connect to upstream transit provider to
see the Internet
p Connect to the local competition so that
local traffic stays local
n Saves spending valuable $ on upstream transit
costs for local traffic
113
Upstream ISP
AS130
C
Local Peer
AS120
A
AS 110
114
p Accept default route only from upstream
n Either 0.0.0.0/0 or a network which can be
used as default
p Accept all routes the local peer originates
115
p Router A Configuration Prefix filters
router bgp 110 inbound
network 121.10.0.0 mask 255.255.224.0
neighbor 122.102.10.2 prefix-list my-block out
neighbor 122.102.10.2 prefix-list AS120-peer in
!
ip prefix-list AS120-peer permit 122.5.16.0/19
ip prefix-list AS120-peer permit 121.240.0.0/20
ip prefix-list my-block permit 121.10.0.0/19
!
ip route 121.10.0.0 255.255.224.0 null0 250
116
p Router A Alternative Configuration
router bgp 110
network 121.10.0.0 mask 255.255.224.0 AS Path filters
more trusting
neighbor 122.102.10.2 filter-list 10 in
!
ip as-path access-list 10 permit ^(120_)+$
!
!
ip route 121.10.0.0 255.255.224.0 null0
117
router bgp 110
network 121.10.0.0 mask 255.255.224.0
!
!
ip route 121.10.0.0 255.255.224.0 null0
118
p Two configurations possible for Router A
n Filter-lists assume peer knows what they are
doing
n Prefix-list higher maintenance, but safer
n Some ISPs use both
p Local
traffic goes to and from local peer,
everything else goes to upstream
119
Aside:
Configuration Recommendations
p Private Peers
n The peering ISPs exchange prefixes they
originate
n Sometimes they exchange prefixes from
neighbouring ASNs too
p Be
aware that the private peer eBGP
router should carry only the prefixes you
want the private peer to receive
n Otherwise they could point a default route to
you and unintentionally transit your backbone
120
Service Provider
Multihoming
One upstream, Local Exchange
Point
121
One Upstream, Local Exchange
Point
p Very common situation in many regions of
the Internet
p Connect to upstream transit provider to
see the Internet
p Connect to the local Internet Exchange
Point so that local traffic stays local
p This
example is a scaled up version of the
previous one
122
Point
Upstream ISP
AS130
IXP
A
AS 110
123
Point
p Announce /19 aggregate to every
neighbouring AS
p Accept default route only from upstream
used as default
p Accept all routes originated by IXP peers
124
Point
interface fastethernet 0/0
description Exchange Point LAN
ip address 120.5.10.1 mask 255.255.255.224
!
router bgp 110
neighbor ixp-peers peer-group
neighbor ixp-peers prefix-list my-block out
neighbor ixp-peers remove-private-AS
neighbor ixp-peers send-community
neighbor ixp-peers route-map set-local-pref in
next slide
125
Point
neighbor 120.5.10.2 peer-group ixp-peers
neighbor 120.5.10.2 prefix-list peer100 in
...next slide
126
Point
!
ip prefix-list peer100 permit 122.0.0.0/19
!
route-map set-local-pref permit 10
!
127
p Note that Router A does not generate the
aggregate for AS110
n If Router A becomes disconnected from backbone, then
the aggregate is no longer announced to the IX
n BGP failover works as expected
p Note the inbound route-map which sets the local
preference higher than the default
n This is a visual reminder that BGP Best Path for local
traffic will be across the IXP
128
Point
router bgp 110
network 121.10.0.0 mask 255.255.224.0
!
!
ip route 121.10.0.0 255.255.224.0 null0
129
Point
p Note Router A configuration
n Prefix-list higher maintenance, but safer
n No generation of AS110 aggregate
p IXP
traffic goes to and from local IXP,
everything else goes to upstream
130
Aside:
IXP Configuration Recommendations
p IXP peers
n The peering ISPs at the IXP exchange prefixes they originate
n Sometimes they exchange prefixes from neighbouring ASNs too
p Be aware that the IXP border router should carry only the
prefixes you want the IXP peers to receive and the
destinations you want them to be able to reach
n Otherwise they could point a default route to you and
unintentionally transit your backbone
p If IXP router is at IX, and distant from your backbone
n Dont originate your address block at your IXP router
131
Service Provider
Multihoming
Two upstreams, one local peer
132
Two Upstreams, One Local Peer
p Connectto both upstream transit
providers to see the Internet
n Provides external redundancy and diversity
the reason to multihome
p Connect to the local peer so that local
traffic stays local
133
Upstream ISP
AS130 Upstream ISP
AS140
C
Local Peer
AS120 D
A
AS 110
134
p Accept default route only from upstreams
used as default
p Accept all routes originated by local peer
p Note separation of Router C and D
n Single edge router means no redundancy
p Router A
n Same routing configuration as in example with
one upstream and one local peer
135
router bgp 110
network 121.10.0.0 mask 255.255.224.0
!
!
ip route 121.10.0.0 255.255.224.0 null0
136
router bgp 110
network 121.10.0.0 mask 255.255.224.0
!
!
ip route 121.10.0.0 255.255.224.0 null0
137
p This is the simple configuration for
Router C and D
p Traffic out to the two upstreams will take
nearest exit
n Inexpensive routers required
n This is not useful in practice especially for
international links
n Loadsharing needs to be better
138
p Better configuration options:
n Accept full routing from both upstreams
p Expensive & unnecessary!
n Accept default from one upstream and some
routes from the other upstream
p The way to go!
139
Full Routes
Allow all prefixes in
p Router C Configuration apart from RFC1918
router bgp 110 and friends
network 121.10.0.0 mask 255.255.224.0
neighbor 122.102.10.1 prefix-list rfc1918-deny in
neighbor 122.102.10.1 route-map AS130-loadshare in
!
! See www.cymru.com/Documents/bogon-list.html
! ...for RFC1918 and friends list
...next slide
140
Full Routes
ip route 121.10.0.0 255.255.224.0 null0
!
ip as-path access-list 10 permit ^(130_)+_[0-9]+$
!
route-map AS130-loadshare permit 10
match ip as-path 10
!
route-map AS130-loadshare permit 20
!
141
Full Routes
Allow all prefixes in
p Router D Configuration apart from RFC1918
router bgp 110 and friends
network 121.10.0.0 mask 255.255.224.0
neighbor 122.102.10.5 prefix-list rfc1918-deny in
!
! See www.cymru.com/Documents/bogon-list.html
! ...for RFC1918 and friends list
142
Full Routes
p Router C configuration:
n Accept full routes from AS130
n Tag prefixes originated by AS130 and AS130s
neighbouring ASes with local preference 120
p Traffic to those ASes will go over AS130 link
n Remaining prefixes tagged with local
preference of 80
p Traffic to other all other ASes will go over the link to
AS140
p Router
D configuration same as Router C
without the route-map
143
Full Routes
p Full routes from upstreams
n Expensive needs lots of memory and CPU
n Need to play preference games
n Previous example is only an example real life
will need improved fine-tuning!
n Previous example doesnt consider inbound
traffic see earlier in presentation for
examples
144
Partial Routes: Strategy
p Ask one upstream for a default route
n Easy to originate default towards a BGP
neighbour
p Ask other upstream for a full routing table
n Then filter this routing table based on
neighbouring ASN
n E.g. want traffic to their neighbours to go over
the link to that ASN
n Most of what upstream sends is thrown away
n Easier than asking the upstream to set up
custom BGP filters for you
145
Partial Routes
Allow all prefixes and
p Router C Configuration default in; deny
router bgp 110 RFC1918 and friends
network 121.10.0.0 mask 255.255.224.0
neighbor 122.102.10.1 prefix-list rfc1918-nodef-deny in
neighbor 122.102.10.1 route-map tag-default-low in
!
...next slide
AS filter list filters
prefixes based on
origin ASN
146
Partial Routes
!
ip route 121.10.0.0 255.255.224.0 null0
!
ip as-path access-list 10 permit ^(130_)+_[0-9]+$
!
route-map tag-default-low permit 10
match ip address prefix-list default
!
route-map tag-default-low permit 20
!
147
Partial Routes
router bgp 110
network 121.10.0.0 mask 255.255.224.0
!
!
ip route 121.10.0.0 255.255.224.0 null0
148
Partial Routes
p Router C configuration:
n Accept full routes from AS130
p (or get them to send less)
n Filter ASNs so only AS130 and AS130s
neighbouring ASes are accepted
n Allow default, and set it to local preference 80
n Traffic to those ASes will go over AS130 link
n Traffic to other all other ASes will go over the
link to AS140
n If AS140 link fails, backup via AS130 and
vice-versa
149
Partial Routes
p Router C IGP Configuration
router ospf 110
default-information originate metric 30
passive-interface Serial 0/0
!
ip route 0.0.0.0 0.0.0.0 serial 0/0 254
p Router D IGP Configuration
router ospf 110
default-information originate metric 10
passive-interface Serial 0/0
!
ip route 0.0.0.0 0.0.0.0 serial 0/0 254
Partial Routes
p Partial routes from upstreams
n Use OSPF to determine outbound path
n Router D default has metric 10 primary
outbound path
n Router C default has metric 30 backup
outbound path
n Serial interface goes down, static default is
removed from routing table, OSPF default
withdrawn
Partial Routes
p Partial routes from upstreams
n Not expensive only carry the routes
necessary for loadsharing
n Need to filter on AS paths
n Previous example is only an example real life
will need improved fine-tuning!
n Previous example doesnt consider inbound
traffic see earlier in presentation for
examples
152
Aside:
Configuration Recommendation
p Whendistributing internal default by iBGP
or OSPF/ISIS
n Make sure that routers connecting to private
peers or to IXPs do NOT carry the default
route
n Otherwise they could point a default route to
you and unintentionally transit your backbone
n Simple fix for Private Peer/IXP routers:
ip route 0.0.0.0 0.0.0.0 null0
153
Service Provider
Multihoming
Three upstreams, unequal
bandwidths
154
Three upstreams, unequal
bandwidths
p Autonomous System has three upstreams
n 16Mbps to ISP A
n 8Mbps to ISP B
n 4Mbps to ISP C
p What is the strategy here?

n One option is full table from each
p 3x 450k prefixes 1350k paths
n Other option is partial table and defaults from
each
p How??
155
Strategy
p Two external routers (gives router redundancy)
n Do NOT need three routers for this
p Connect biggest bandwidth to one router
n Most of inbound and outbound traffic will go here
p Connect the other two links to the second router
n Provides maximum backup capacity if primary link fails
p Use the biggest link as default
n Most of the inbound and outbound traffic will go here
p Do the traffic engineering on the two smaller links
n Focus on regional traffic needs
156
Diagram
ISP B
AS120
ISP A
AS110
ISP C
AS130
B A
AS 100
p Router A has 16Mbps circuit to ISP A

p Router B has 8Mbps and 4Mbps circuits to ISPs
B&C 157
Outbound load-balancing strategy
p Available
BGP feeds from Transit
providers:
n Full table
n Customer prefixes and default
n Default Route
p These are the common options on Internet

today
n Very rare for any provider to offer anything
different
n Very rare for any provider to customise BGP
feed for a customer
158
p Accept only a default route from the provider with
the largest connectivity, ISP A
n Because most of the traffic is going to use this link
p If ISP A wont provide a default:
n Still run BGP with them, but discard all prefixes
n Point static default route to the upstream link
n Distribute the default in the IGP
p Request the full table from ISP B & C
n Most of this will be thrown away
n (Default plus customers is not enough)
159
p How to decide what to keep and what to discard
from ISPs B & C?
n Most traffic will use ISP A link so we need to find a
good/useful subset
p Discard prefixes transiting the global transit ISPs
n Global transit ISPs generally appear in most non-local or
regional AS-PATHs
p Discard prefixes with ISP As ASN in the path
n Makes more sense for traffic to those destinations to go
via the link to ISP A
160
p Global Transit ISPs include:
209 CenturyLink 3549 Level 3
701 VerizonBusiness 3356 Level 3
1239 Sprint 3561 Savvis
1668 AOL TDN 7018 AT&T
2914 NTT America
161
ISP B peering Inbound AS-PATH
filter
ip as-path access-list 1 deny _209_
!
ip as-path access-list 1 deny _ISPA_ Dont need ISPA and
ip as-path access-list 1 deny _ISPC_ ISPC prefixes via ISPB
!
ip as-path access-list 1 permit _ISPB$
ip as-path access-list 1 permit _ISPB_[0-9]+$
ip as-path access-list 1 permit _ISPB_[0-9]+_[0-9]+$
ip as-path access-list 1 permit _ISPB_[0-9]+_[0-9]+_[0-9]+$
ip as-path access-list 1 deny .*
162
Outbound load-balancing strategy:
ISP B peering configuration
p Part 1: Dropping Global Transit ISP prefixes
n This can be fine-tuned if traffic volume is not sufficient
n (More prefixes in = more traffic out)
p Part 2: Dropping prefixes transiting ISP A & C
network
p Part 3: Permitting prefixes from ISP B, their BGP
neighbours, and their neighbours, and their
neighbours
n More AS_PATH permit clauses, the more prefixes
allowed in, the more egress traffic
n Too many prefixes in will mean more outbound traffic
than the link to ISP B can handle
163
p Similar AS-PATH filter can be built for the
ISP C BGP peering
p If the same prefixes are heard from both
ISP B and C, then establish proximity of
their origin ASN to ISP B or C
n e.g. ISP B might be in Japan, with the
neighbouring ASN in Europe, yet ISP C might
be in Europe
n Transit to the ASN via ISP C makes more
sense in this case
164
Inbound load-balancing strategy
p The largest outbound link should announce just
the aggregate
p The other links should announce:
a) The aggregate with AS-PATH prepend
b) Subprefixes of the aggregate, chosen according to
traffic volumes to those subprefixes, and according to
the services on those subprefixes
p Example:
n Link to ISP B could be used just for Broadband/Dial
customers so number all such customers out of one
contiguous subprefix
n Link to ISP C could be used just for commercial leased
line customers so number all such customers out of
one contiguous subprefix
165
Router A: eBGP Configuration
Example
router bgp 100
network 100.10.0.0 mask 255.255.224.0
neighbor 122.102.10.1 remote 110
!
!
166
Router B: eBGP Configuration
Example
router bgp 100
network 100.10.0.0 mask 255.255.224.0
neighbor 120.103.1.1 prefix-list ISP-B out
neighbor 120.103.1.1 route-map to-ISP-B out
neighbor 121.105.2.1 prefix-list ISP-C out
neighbor 121.105.2.1 route-map to-ISP-C out
!
!
...next slide
167
Router B: eBGP Configuration
Example
ip prefix-list ISP-B permit 100.10.0.0/19
ip prefix-list ISP-B permit 100.10.0.0/21 /21 to ISP B
! dial customers
ip prefix-list ISP-C permit 100.10.0.0/19
ip prefix-list ISP-C permit 100.10.28.0/22
!
/22 to ISP C
route-map to-ISP-B permit 10
biz customers
set as-path prepend 100
! e.g. Single
route-map to-ISP-B permit 20 prepend on ISP B
! link
route-map to-ISP-C permit 10
set as-path prepend 100 100
e.g. Dual prepend
!
on ISP C link
route-map to-ISP-C permit 20
168
What about outbound backup?
p We have:
n Default route from ISP A by eBGP
n Mostly discarded full table from ISPs B&C
p Strategy:
n Originate default route by OSPF on Router A (with
metric 10) link to ISP A
n Originate default route by OSPF on Router B (with
metric 30) links to ISPs B & C
n Plus on Router B:
p Static default route to ISP B with distance 240
p Static default route to ISP C with distance 245
n When link goes down, static route is withdrawn
169
Outbound backup: steady state
p Steady state (all links up and active):
n Default route is to Router A OSPF metric 10
n (Because default learned by eBGP default is
in RIB OSPF will originate default)
n Backup default is to Router B OSPF metric
20
n eBGP prefixes learned from upstreams
distributed by iBGP throughout backbone
n (Default can be filtered in iBGP to avoid RIB
failure error)
170
Outbound backup: failure examples
p Link to ISP A down, to ISPs B&C up:
n Default route is to Router B OSPF metric 20
n (eBGP default gone from RIB, so OSPF on
Router A withdraws the default)
p Above is true if link to B or C is down as
well
p Link to ISPs B & C down, link to ISP A is
up:
n Default route is to Router A OSPF metric 10
n (static defaults on Router B removed from RIB,
so OSPF on Router B withdraws the default)
171
Other considerations
p Default route should not be propagated to
devices terminating non-transit peers and
customers
p Rarely any need to carry default in iBGP
n Best to filter out default in iBGP mesh peerings
p Still
carry other eBGP prefixes across iBGP
mesh
n Otherwise routers will follow default route rules
resulting in suboptimal traffic flow
n Not a big issue because not carrying full table
172
Router A: iBGP Configuration
Example
router bgp 100
network 100.10.0.0 mask 255.255.224.0
neighbor ibgp-peers peer-group
neighbor ibgp-peers remote-as 100
neighbor ibgp-peers prefix-list ibgp-filter out
neighbor 100.10.0.2 peer-group ibgp-peers
neighbor 100.10.0.3 peer-group ibgp-peers
!
ip prefix-list ibgp-filter deny 0.0.0.0/0
ip prefix-list ibgp-filter permit 0.0.0.0/0 le 32
!
173
Three upstreams, unequal bandwidths:
Summary
p Example based on many deployed working
multihoming/loadbalancing topologies
p Many variations possible this one is:
n Easy to tune
n Light on border router resources
n Light on backbone router infrastructure
n Sparse BGP table faster convergence
174
Service Provider
Multihoming
ISP Workshops
175
Thank you!
End of Session

Tutorial BGP Multihoming Techniques SANOG24

Uploaded by

Copyright:

Available Formats

Tutorial BGP Multihoming Techniques SANOG24

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Tutorial BGP Multihoming Techniques SANOG24

Uploaded by

Copyright:

Available Formats

BGP Multihoming Tutorial

APNIC Training and Development

Last updated 25 July 2013 3

p Upstream Service Provider (configuration, software,

p With two or more international connections

p For the Internet?

n And how do we do it?

p Usually two external facing routers

p Three BASIC Principles

p Route-maps can be used for filtering, but

p No need for BGP

p Use BGP (not IGP or static) to loadshare

p Many situations possible

router bgp 100

101.10.0.1 101.10.5.255 101.10.6.255 /24

Customer Address & p-t-p links Infrastructure Loopbacks

Customer Addresses ISP

p Customers generate traffic

Customer Addresses Customer Addresses ISP

p Will keep the examples easy

p Border router E in AS100 removes private AS and any

router ospf 65534

router isis as65534

p Router C only allows in /19 and /20 prefixes from

p Router D only allows in /19 and /20 prefixes from

p Private AS still visible inside AS100 79

router bgp 100

p Configuration concepts very similar

Announce /19 block

Announce first Announce second

Announce /20 subprefix,

p Wehave not looked at outbound traffic

Last updated 25 July 2013 106

p The original Traffic Engineering

p Require BGP and a public ASN

ip route 0.0.0.0 0.0.0.0 null0

p What is the strategy here?

p Router A has 16Mbps circuit to ISP A

p These are the common options on Internet

You might also like