ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

GUIDE - SEPTEMBER 2018

ROUTING DESIGN GUIDE

Underlay / Overlay Routing

Guide | 1
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

This document describes the routing design options when integrating VMware SD-WAN by
VeloCloud into an existing network and the steps necessary to achieve optimized routing. It also
covers the various techniques VMware SD-WAN by VeloCloud uses to easily insert the VMware
SD-WAN by VeloCloud Edge (VCE) in hybrid networks with MPLS. The content is relevant as of
Release 3.0.

Overview 3
1.1 Understanding Overlay/Underlay ................................................................ 3
1.2 Configuring Redistribution ........................................................................... 3
1.3 Significance of Static Route’s “Advertise” Flag ........................................... 4
1.4 Insertion Topologies .................................................................................... 5

Routing Challenges 6
2.1 Challenges with SD-WAN Overlay and MPLS ............................................ 6
2.2 Solutions with SD-WAN Overlay and MPLS ............................................... 6
2.2.1 Routing to MPLS Sites with Hub as Transit ......................................... 7
2.2.2 Routing to Legacy MPLS Sites Directly Through Underlay ................. 8

Redistribution Control 10
3.1 BGP Specific Features .............................................................................. 10
3.1.1 BGP Uplink Neighbor ......................................................................... 10
3.1.2 BGP Uplink Community ..................................................................... 12

Best Practice for Various Insertions 17

4.1 Data Center Insertions .............................................................................. 17
4.1.1 Data Center Deployment Overview ................................................... 17
4.1.2 Configuring DC Hub as Backup for Traffic Between SD-WAN Branches and MPLS 18
4.1.3 Blocking Default Route from BGP Peer(s) ......................................... 20
4.1.4 Configure Route Map on Core Switch for BGP Peering to CE Router21
4.1.5 Inserting Hub VCE with Direct BGP Peering to PE Router ................ 21
4.2 Internet-Only Branch Insertions................................................................. 23
4.3 Hybrid Branch Insertion with Hub as Transit ............................................. 23
4.4 In-Path Hybrid Branch Insertion with BGP to MPLS ................................. 24
4.4.1 Configuring BGP for In-Path Hybrid Branch VCE .............................. 25
4.4.2 Caveat with Same BGP AS Number Used on SD-WAN Branch VCE’s27
4.5 Off-Path Hybrid Branch Insertion with BGP to MPLS................................ 28
4.5.1 Configuring BGP for Off-Path Hybrid Branch VCE ............................ 29
4.5.2 Configure Route Map on Core Switch for BGP Peering to CE Router31
4.6 Off-Path Hybrid Branch Insertion with OSPF to MPLS ............................. 33
4.6.1 Configure CE Router to Deny Local Prefix from MPLS ..................... 34
4.6.2 Configuring OSPF for Off-Path Hybrid Branch VCE .......................... 34
4.6.3 Route Redistribution Control for Off-Path Hybrid Branch VCE .......... 37

Guide | 2
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Overview
1.1 Understanding Overlay/Underlay
Within VMware SD-WAN by VeloCloud (VeloCloud SD-WAN) solution, the VeloCloud
Gateway (VCG) is responsible for distributing the routes to all the VeloCloud Edges (VCE).
The routes learned from other VCEs are referred to as the Overlay routes and traffic
destined to the Overlay are encapsulated within VeloCloud Multi-Path (VCMP) tunnels. The
Underlay routes refer to the routes learned from routing protocols such as OSPF or BGP,
and locally configured static routes where traffic destined to the Underlay are simply routed
or switched without any encapsulation.
Per Figure below, the VeloCloud SD-WAN solution supports mutual redistribution between
the Underlay to Overlay routes. In this example, if the VCE is running BGP with the L3
Switch and a route behind the L3 Switch is learned by the VCE via BGP from the Underlay,
then the route would be redistributed to the Overlay and seen by remote VCE’s as an
Overlay route. Similarly, routes from remote VCE’s learned through the Overlay can also be
redistributed into the Underlay as a BGP or OSPF route to be advertised to the local peers.
Note that within the VeloCloud solution, route redistribution in either direction can be
selectively disabled. Under certain topologies, it may be necessary to disable redistribution
to prevent sub-optimal routing or even routing loops and they will be covered in subsequent
sections.

Figure 1 - Underlay vs. Overlay

1.2 Configuring Redistribution

As discussed in section 1.1, redistribution occurs in two directions: Overlay to Underlay and
Underlay to Overlay. The option to configure Overlay to Underlay redistribution is available
in the Routing Protocol configuration on the VCO. For example, enabling SD-WAN to BGP
redistribution is under the “Advanced” section of a Profile or Edge’s BGP configuration per
Figure below; OSPF configuration has a similar option to redistribute SD-WAN routes to
OSPF as external routes.

Guide | 3
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 2 - Overlay to Underlay redistribution for BGP

The options to configure Underlay to Overlay redistribution is available on the Overlay Flow
Control (OFC) page which applies to all Edges under the Enterprise. Note that the options
are available separately for the Hubs and Branches due to Hubs often needing to be a
transit site or a backup route in case of connectivity failures at the branches.

Figure 3 - Underlay to Overlay redistribution in OFC

1.3 Significance of Static Route’s “Advertise” Flag

When static routes are configured on a VeloCloud VCE per Figure below, the “Advertise”
flag checkbox enables redistribution of the static routes to both the Overlay and the routing

Guide | 4
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

protocols if they are configured. This implies that if a static route is configured with the
“Advertise” flag, it will be advertised to other VCE’s via the Overlay and there’s no need to
configure the same route at the other sites.

Figure 4 - Static Routes with Advertise Flag

1.4 Insertion Topologies

Figure below demonstrates a network topology with the VeloCloud Edge (VCE) inserted in
various setups. Note that in several of these insertions, the VCE is deployed at hybrid sites
where WAN connectivity include both MPLS and Internet. Per section 1.1 this implies if bi-
directional route redistribution is enabled between the Underlay and Overlay, the VCE will
redistribute routes learned from MPLS to the Overlay, as well as mutually redistribute
remote branches’ routes learned from the Overlay to MPLS.

Figure 5 - Network Insertion Topologies

Branch 1, 2, and 3 are Hybrid sites with both a MPLS and Internet WAN connection while
Branch 4 is an Internet only site. Assuming the Branch 4 VCE uses the DC VCE’s as hubs,
the only way for Branch 4 to reach the Legacy MPLS Site is through the Overlay via the
DC’s. It is not necessarily the case with Branch 1, 2, and 3 - if the VCE’s run a routing

Guide | 5
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

protocol with the adjacent router/switch and learns the routes from MPLS, the Legacy MPLS
Site will be reachable from both the Overlay and Underlay behind those Branches.

Routing Challenges
2.1 Challenges with SD-WAN Overlay and MPLS
When a site deployed with SD-WAN such as Branch 1, 2, and 3 exchanges routes with the
MPLS network, several challenges arise due to redistribution between the Underlay and the
Overlay, or SD-WAN domain. Take Branch 1 for example, a subnet B1 behind Branch 1
may be advertised to MPLS through the Underlay by the CE router in Branch 1. Yet, since
other branch VCE’s learns subnet B1 through the Overlay from the SD-WAN controller, B1
may also be advertised by other hybrid site (i.e. Branch 3) to MPLS due to Overlay to
Underlay redistribution. This suggests for hybrid sites, Underlay and Overlay mutual
redistribution should only be enabled for transit sites such as the Data Centers - so only the
DC VCE’s will announce non-local routes to MPLS for the purpose of them becoming
backup in the case that MPLS connectivity at the hybrid branch fails. The problem can be
illustrated by the Figure below which compares the route advertisement of B1 to MPLS
before and after introduction of SD-WAN.

Figure 6 - Route Advertisement to MPLS before and after SD-WAN

Before SD-WAN, routing to branch subnets with MPLS is simple and deterministic as each
CE router at the branch would only advertise its own local subnet. However, when a VCE is
inserted at the branch and redistributing routes to the CE router, the CE may receive routes
from SD-WAN that belong to other branches and start advertising non-local subnets to
MPLS.
2.2 Solutions with SD-WAN Overlay and MPLS
In general, there are two solutions to eliminate the aforementioned challenges - 1. Do not
allow routing to the Underlay at the hybrid SD-WAN sites and strictly use the Hubs as transit
point for routing to legacy MPLS sites, and 2. Control redistribution at all the SD-WAN sites
so they only advertise their local prefixes, with exception being the hub locations which

Guide | 6
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

advertise all the other branch prefixes for the purpose of being a backup path. Figure below
demonstrates the difference between the two concepts.

Figure 7 - Routing to Legacy MPLS sites using Hub as transit vs Directly through the Underlay

2.2.1 Routing to MPLS Sites with Hub as Transit

For solution 1 where the Hubs are used as transit point for traffic from SD-WAN sites to
Legacy MPLS sites, only the SD-WAN Hubs have to exchange routes with MPLS and all
traffic from hybrid SD-WAN branches will go through the Hubs to get to the Legacy MPLS
sites. Using the figure below as an example, the Hybrid SD-WAN site is using the VCE to
connect and establish Overlays over both the Internet and MPLS. However, the VeloCloud
Edge (VCE) at the branch is not running BGP or any routing protocol with the MPLS PE
router and the MPLS is simply used as a network on which the overlay is built to the other
VCEs including the Hubs. The MPLS provider will only see encapsulated Overlay packets
and not the actual end user packets. The only network the PE has to advertise in this case
is the subnet connecting the VCE and the PE so the Overlay tunnels to the Hubs can be
established.
The Hub device in the DC will run routing protocols (BGP) to learn the Legacy MPLS
prefixes such as L1 and redistribute those prefixes to the Overlay so the Hybrid SD-WAN
branch can learn L1 through the Overlay. Similarly, the Hybrid Branch subnet B1 will be
advertised to MPLS by the Hub using Overlay to Underlay redistribution. In this design the
Hubs become the transit points for all traffic between SD-WAN and Legacy MPLS Sites.

Guide | 7
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 8 - Routing from Hybrid SD-WAN site to MPLS using SD-WAN Hub as transit

2.2.2 Routing to Legacy MPLS Sites Directly Through Underlay

The second solution which is more preferred but complex is to exchange routes with MPLS
(can be through CE or branch router) and route traffic from Hybrid SD-WAN branches to
Legacy MPLS sites directly through the Underlay. Per the Figure below, the goal is to
ensure each Hybrid branch only advertises its local networks to MPLS - even though it
learns the other SD-WAN prefixes from the Overlay, and prefers routing to the Legacy
MPLS prefixes through the local CE/PE directly to MPLS. The Hubs can still be used as
transit to the Legacy MPLS sites if the Hybrid Branch VCE stops learning the routes MPLS
routes locally.
In the example below, Legacy prefix L1 will be learned by both the SD-WAN branches and
the Hubs. The Hybrid SD-WAN branch should prefer MPLS Underlay to reach L1 rather than
the Overlay via Hubs since the AS-Path length through the Hubs will be longer. Mutual
redistribution between Overlay and MPLS Underlay should be disabled on all Hybrid SD-
WAN branch sites to prevent the Branch VCE from redistributing L1 from BGP to the

Guide | 8
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Overlay, and also to prevent the Branch VCE from redistributing other SD-WAN Branches’
prefixes from Overlay to BGP.
VeloCloud has a built in feature “Uplink Neighbor” that will be discussed later that can be
configured on a BGP neighbor to prevent redistribution from both Overlay -> BGP and BGP
-> Overlay for the peering to MPLS. In this example, Uplink Neighbor should be configured
on the BGP peering interface on the VCE toward MPLS.

Figure 9 - Routing from Hybrid SD-WAN site to MPLS using Underlay

Guide | 9
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Redistribution Control
3.1 BGP Specific Features
As previously mentioned, the flags on the Overlay Flow Control page controls routing
redistribution from the Underlay to Overlay, and there are options under the routing protocol
configuration that controls redistribution from Overlay to the routing protocol. However,
VeloCloud has introduced two features that make route redistribution even more seamless -
“Uplink Neighbor” and “Uplink Community”.
The “Uplink Neighbor” feature is configured directly on the BGP peering to identify the peer
as a WAN side neighbor with connectivity to MPLS. There will be no mutual redistribution in
either direction to the BGP peer when it’s marked as the uplink neighbor.
Similarly, the “Uplink Community” feature is also used for route redistribution control when
the VCE is deployed out of path. The idea is if the MPLS routes learned by the CE can be
tagged with a community before advertised to the local router/switch the VCE is connected
to, then the VCE can identify those routes based on the community value and deny those
routes from being redistributed to the Overlay. Details to the two features are described
below.
3.1.1 BGP Uplink Neighbor
The Uplink Neighbor feature is configured on a per-peering basis. Routes received from the
neighbor marked as Uplink will be refrained from redistribution to the Overlay and routes
learned from the Overlay will not be redistributed and advertised to that BGP neighbor.

Figure 10 - BGP Uplink Neighbor

Per Figure above, this feature is typically used on a branch VCE when it is either a CE
replacement or directly connected to the CE - in which mutual redistribution should be

Guide | 10
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

avoided because this branch should only advertise the local LAN subnet and learn the
MPLS routes without becoming the transit.
As an example, we will look at the following topology where the Legacy MPLS site has a
prefix of 172.28.0.0/24 and the site of interest is Branch2 where the VCE functions as a CE.

Figure 11 - Explaining BGP Uplink Neighbor

Without the Uplink Neighbor feature configured on the Branch2 VCE’s BGP peering to
MPLS, the prefix L1 would be learned by the Branch2 VCE and redistributed to the Overlay.
The VeloCloud SD-WAN controller will mark Branch2 VCE as the exit point for L1 prefix
172.28.0.0/24. With the Branch2 VCE nicknamed “Hybrid CPE”, the OFC entry would look
like the following where Hybrid CPE is the preferred exit over Router - where Router means
the preferred exit is the Underlay if it’s available.

Guide | 11
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 12 - OFC for a Route without Uplink Neighbor

Now, if Branch2 VCE configures the BGP peering to MPLS with the Uplink Neighbor flag,
we can see that the VCE will no longer be a preferred VPN Exit for the 172.28.0.0/24 prefix
as it is no longer redistributed to the Overlay. If we click on “Edit”, we can see that the prefix
was learned by multiple VCE’s but they are not selected as VPN exit. These prefixes are not
redistributed to the Overlay because they are learn from BGP peers that are marked as
Uplink - which displays as (BGP-E, U), or it was an OSPF External route not redistributed
due to a flag on the OFC page - which displays as (OSPF-OE2).

Figure 13 - OFC for a Route with Uplink Neighbor

3.1.2 BGP Uplink Community

The BGP Uplink Community feature was introduced for insertion topologies where the VCE
is placed off path to prevent routes previously tagged with a matching community from being
redistributed into the Overlay.

Guide | 12
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 14 - Uplink Community Feature

Per Figure above, this feature is typically used in off-path insertions or data center
deployments where the VCE is not directly connected to the CE or PE and using BGP to
attract traffic to remote branches. The idea is if routes learned by the CE from MPLS is
tagged with a certain community, then when the VCE receives the routes from the core
router/switch which may contain other local prefixes, it can distinguish the routes from MPLS
and refrain from redistributing these prefixes into the Overlay to prevent this site from
becoming the transit for Overlay to MPLS traffic.
As an example, we will look at the following topology where the Legacy MPLS site has a
prefix of 172.28.0.0/24 and the site of interest is Branch1 where the VCE is inserted off path
and peering BGP with the core switch. The proper use of BGP Uplink community involves
the CE or the core switch at Branch1 tagging the routes learned from MPLS with a
community string such as 12345:777, and then configuring Uplink Community on the VCE’s
matching that 12345:777 to prevent only the MPLS routes from redistributed into the
Overlay. Note that the community value can be configured both as decimal value or the
new-format.

Guide | 13
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 15 - Explaining BGP Uplink Community

In the above example, the previously discussed Uplink Neighbor feature cannot be applied
on the Branch1 VCE’s BGP peering session to the core switch because the redistribution of
the prefix B1 which is only local to Branch1 is still required. Without utilizing Uplink
Community, the prefixes L1 and B3 would be redistributed into the Overlay - which is
undesirable because the other hybrid SD-WAN branches should already be learning L1
through their local BGP peering and learn prefix B3 through the Overlay directly from
Branch3 VCE. Note that in this example, the Branch1 VCE should already have a route for
B3 prefix of 192.168.129.0/24 from the Overlay and it will prefer the Overlay route over the
same prefix learned from BGP. The default behavior for the VCE’s is to prefer Overlay route
over Underlay route except for local static routes. Also, note that for insertions such as
Branch1 where it’s off path, redistribution from Overlay to BGP should be enabled by
checking the flag “Overlay Prefix” under “Route Redistribution” to redistribute other SD-WAN
prefixes to the peering core switch. This is demonstrated in the Figure below.

Guide | 14
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 16 - Configuring Uplink Community on a Off-Path Branch VCE

Similar to the Uplink Neighbor feature, the routes marked with Uplink Community by a VCE
will cause the VCE to be removed from the Preferred VPN Exits for that prefix. In the
example below, Branch1 is nicknamed “Silver3 VCE” and you can see that with the prefix
172.28.0.0/24 tagged with the Uplink Community, Silver3 VCE (and other VCE’s) have been
prevented from being a transit for 172.28.0.0/24 and that prefix will only be reachable
through the Underlay through BGP if it’s available.

Figure 17 - OFC for a Route with Uplink Community

As a summary to compare Uplink Neighbor and Uplink Community, Uplink Neighbor should
be configured on a single BGP peer to prevent bi-directional redistribution to that BGP
neighbor; whereas Uplink Community requires the routes learned from BGP to have been
tagged with a community previously and only prevents redistribution from Underlay to

Guide | 15
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Overlay. BGP Uplink Neighbor generally should only be applied to the link directly
connected to the CE or the PE. The table below provide a quick overview of their
comparison.

Figure 18 - Uplink Neighbor Vs Uplink Community

Guide | 16
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Best Practice for Various Insertions

4.1 Data Center Insertions
4.1.1 Data Center Deployment Overview
VeloCloud Edge (VCE) inserted in the data center is typically configured as a hub in the
VeloCloud SD-WAN deployment with various transport types available for Overlays. Per the
Figure below, the pair of HA Hubs are not configured as a cluster and therefore seen as a
single VCE with connectivity to both Internet and MPLS. The best practice is to connect the
WAN interface used for Internet Overlay in the DMZ behind the firewall and a second routed
interface connected to the core switches for the MPLS Overlay and also Underlay routing to
the local DC prefixes. The VCE is inserted in an off-path topology in this example and it’s
often the best method to alleviate changes in the data center. The second WAN interface for
MPLS Overlay can also be connected to the firewall in the trusted zone if site-to-site VPN
traffic also requires firewall filtering.

Figure 19 - Data Center Insertion as a Hub VCE

For Data Center Insertions, it’s required that the VCE has unique public IP addresses for its
Internet Overlays as Hubs do not initiate tunnels outbound and the static public IP
addresses also maintain stability of the tunnels to branch VCE’s in a large deployment.
In this insertion topology where the VCE is placed as an off-path deployment, the VCE
should mutually redistribute the routes between the learned prefixes from CE and its
Overlay routes - although it’s still desirable for Hybrid SD-WAN branches to reach the

Guide | 17
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Legacy MPLS prefixes directly through the Underlay. Redistribution control is needed to
make the Legacy MPLS prefixes redistributed by the Hub into the Overlay to be less
preferred than the Underlay routes the Hybrid SD-WAN branches learn from MPLS. The DC
Hub should only be a backup transit site for traffic between Hybrid SD-WAN branches and
MPLS if the Hybrid branch loses its MPLS connectivity - which means from a Hybrid SD-
WAN Branch, the MPLS prefixes redistributed by the Hubs to the Overlay must be
secondary to the concurrent BGP routes learned from MPLS. This is achieved through the
use of Uplink Community feature and the OFC Preferences and that is discussed in section
4.1.2.
4.1.2 Configuring DC Hub as Backup for Traffic Between SD-WAN Branches and MPLS
Using Figure below as an example, the goal is to configure the Hub such that traffic from
Hybrid SD-WAN prefix B2 to MPLS prefix L1 will prefer Underlay to reach L1 via BGP, but
still uses the Hub as a transit when the Hybrid branch loses its MPLS connectivity.

Figure 20 - Configuring DC Hub as a Backup for Traffic between SD-WAN Branches and MPLS

Guide | 18
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

VeloCloud has automated the mechanism to make the L1 prefix redistributed by the Hub
less preferred than the same Underlay L1 prefix from MPLS from the Hybrid SD-WAN
branch. Configuring Uplink Community on the Hub to mark L1 as a MPLS prefix and
enabling “Advertise Uplink Routes” for the Hub in OFC will achieve the desired result. The
use of Uplink Community will allow the Hub to mark the prefixes from MPLS so they can be
treated differently than the local data center prefixes. Since configuring Uplink Community
on the Hub prevents the Hub from redistributing L1 to the Overlay, checking the option to
enable “Advertise Uplink Routes” in OFC per Figure below will enable the redistribution but
cause the Hub to redistribute L1 to the Hybrid branch with a high cost automatically. This will
allow the Hybrid branch to retain both the routes from BGP as well as the Overlay routes
redistributed from the Hub - yet prefer the Underlay route from MPLS via BGP. This is also
required so the Hubs can become transit sites for broadband-only SD-WAN sites or when a
Hybrid SD-WAN site loses its MPLS connectivity.

Figure 21 - OFC Flag to Advertise Uplink Routes

A sample of BGP configuration on the Hub is shown below. The inbound filter in this
example denies the default route received from the BGP peer and the tunnel source prefixes
that are marked by other VCE’s with a BGP community of 12345:999. An explicit “Allow All”
is needed as the last filter to allow the other prefixes unless no filter is applied. For the
outbound filter, a route filter is placed to mark the tunnel source prefixes with a community of
12345:999 for the routes to be denied by other VCE’s. The last outbound filter tags the
routes belonging to the SD-WAN site with a community of 12345:888 so they can be easily
identified in the routing table.
Note that when multiple filters are applied to the inbound and outbound BGP filter, they are
concatenated and routes are matched in order like a router ACL; the matching stops and
exits when a route matches a rule in the list.

Guide | 19
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 22 - Configuring BGP on Data Center Hubs

If Uplink Community is correctly implemented on the Hubs and the Hybrid SD-WAN
Branches are also configured with Uplink/Uplink Community for their BGP peers toward
MPLS, then the OFC table should show the output for the Legacy MPLS prefixes similar to
the following assuming prefix L1 is 172.28.0.0. Notice “Router” or the Underlay is the
preferred exit while the Hubs are also showing as available VPN Exits with the Route Type
marked as “BGP-E, U” where U indicates the routes were marked with the Uplink or Uplink
Community flag.

Figure 23 - OFC Table Output for a Legacy MPLS Prefix

4.1.3 Blocking Default Route from BGP Peer(s)

In the VeloCloud SD-WAN architecture, a route learned by a VCE is advertised to the
VeloCloud Gateway (VCG) through the control plane and then propagated to the other
VCE’s in the SD-WAN domain. If a default route was learned via BGP and redistributed to
the Overlay, it would cause the VCE to become the default route and therefore transit site

Guide | 20
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

for all traffic that do not match a more specific route. Per Figure above, it’s highly
recommended to configure a BGP route filter to block the default route from all BGP peers
as a precaution.
4.1.4 Configure Route Map on Core Switch for BGP Peering to CE Router
Per the configuration above, several BGP communities are used to identify and tag routes
for filtering to and from remote branches. Community 12345:777 is used to identify prefixes
received from MPLS, 12345:888 is used to tag local SD-WAN prefixes advertised outbound,
and 12345:999 is used to tag and filter VCE’s MPLS tunnel source prefixes.
For data center insertions with “Uplink Community”, it’s recommended to configure route
maps on the BGP session from the core switch to the CE router. For outbound
advertisement from the core switch to the CE router, no restriction is needed since the Hub
is a transit site and all Overlay prefixes should be advertised to MPLS. On the other hand,
the inbound route map serves two purposes. The first purpose is to tag the prefixes
incoming from MPLS with the Uplink Community (i.e. 12345:777) matching the configuration
on the VCE. The Uplink Community has to be applied on the core switch to distinguish the
MPLS prefixes from the local data center prefixes so when these MPLS prefixes are
redistributed to the SD-WAN Overlay by the DC Hub, these prefixes previously marked with
Uplink Community of 12345:777 can be treated as less preferred than the Underlay routes
by remote SD-WAN branches.
The desired end result is that the core switch in the data center will learn the MPLS prefixes
from only the local CE router. It will learn the SD-WAN prefixes from both the VCE and the
CE router via BGP while it prefers the VCE due to AS-Path Prepend applied on the inbound
route map. The output below demonstrates the route map configured on the core switch for
its BGP session to the CE router. The example below is provided assuming the BGP ASN of
the CE router is AS 100.
ip community-list 1 permit 12345:888

route-map BGP_INBOUND permit 10

description AS prepend routes with community 12345:888 from other SD-WAN sites and mark uplink community
match community 88
set as-path prepend 100 100
set community 12345:777 additive
!
route-map BGP_INBOUND permit 20
description mark any other routes from MPLS with community of 12345:777 to be used for BGP Uplink Community
set community 12345:777 additive
4.1.5 Inserting Hub VCE with Direct BGP Peering to PE Router
In insertion topologies where the Hub VCE has a BGP peering session to the PE router
directly, Uplink Community is no longer needed because the Hub can clearly identify
prefixes learned from the PE router as non-local routes. We can accomplish the same
results as discussed in sections 4.1.1-4.1.4 by configuring the BGP session to the PE router
as an Uplink. As a reminder, marking a BGP neighbor as Uplink neighbor disables mutual

Guide | 21
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

redistribution between BGP and the Overlay in both directions. Since the Hub has to be a
transit site in almost all design scenarios, we must override the mutual redistribution in both
directions but in the process make the MPLS prefixes learned from the PE router to be less
preferred when advertising them to the Hybrid SD-WAN spoke sites.

Guide | 22
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

4.2 Internet-Only Branch Insertions

Figure 24 - Network Insertion Topologies

SD-WAN routing from Internet-Only sites are much simpler compare to Hybrid SD-WAN
branches. Per Figure above, with topologies such as Branch4 where the site only has
Internet connectivity and not MPLS, the only way for the prefix at Branch4 to reach the
Legacy MPLS site is to use the DC Hubs as a transit. Traffic from Branch4 will first go to one
of the Hubs depending on route preference and then follow BGP routes learned by the Hubs
to reach the MPLS site using the Underlay. The Hubs in this topology are the only locations
where the Internet-Only branch prefixes are being advertised to MPLS via BGP.
4.3 Hybrid Branch Insertion with Hub as Transit
In network architectures where the data center is used as the transit point for all branch-to-
branch traffic, the insertion of a VCE at a hybrid branch can be done without running routing
protocol on the VCE’s MPLS facing interface. This was discussed previously in section
2.2.1. Per Figure below, there’s no route exchange between the branch VCE and the MPLS
network and MPLS is only used as a transit to build the Overlay tunnels. All traffic from the
branch prefix to the Legacy MPLS sites will go through the Hubs. The only network the PE
has to advertise in this case is the subnet connecting the VCE and the PE so the Overlay
tunnels to the Hubs can be established.
Note that similar to the Internet-Only SD-WAN site discussed in section 4.2, if the DC Hub is
configured with “Uplink Community” - which prevents the redistribution of MPLS prefixes

Guide | 23
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

tagged with a community to the Overlay, then this hybrid branch VCE will not have the
specific Legacy MPLS prefixes. Summary routes must be configured on the Hub and
advertised to this hybrid branch VCE to draw traffic from the branch prefix to the MPLS
prefixes. If all hybrid SD-WAN branches are configured this way and there is not a
requirement for a branch prefix to route to the MPLS prefixes directly through the MPLS
Underlay, then “Uplink Community” is not necessary on the Hubs and mutual redistribution
can be allowed to redistribute the MPLS prefixes to the VCE’s through the Overlay.

Figure 25 - Routing from Hybrid SD-WAN site to MPLS using SD-WAN Hub as transit

4.4 In-Path Hybrid Branch Insertion with BGP to MPLS

Guide | 24
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

In this section we will discuss best practice for insertion topologies where the VCE is in path
and running BGP directly with the upstream CE or PE. Using Figure below as an example,
the VCE is in path for traffic from prefix B3 and its MPLS facing WAN interface is BGP
peered to the CE. The VCE will learn the MPLS prefixes via the Underlay from the CE and
learn the other SD-WAN prefixes from the Overlay.

Figure 26 - In-Path Hybrid Branch VCE with Direct BGP Peering to CE or PE

4.4.1 Configuring BGP for In-Path Hybrid Branch VCE

As discussed previously, to prevent the branch VCE from redistributing other SD-WAN
branch prefixes to MPLS through BGP, it’s best practice to configure the BGP peer to the
CE router as an “Uplink” to stop mutual redistribution. If the MPLS interface or peering were

Guide | 25
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

to go down, Branch 3 would be treated as an Internet-Only site and follow the Overlay
summary routes as discussed in section 4.1.2. A sample BGP configuration for the VCE is
shown below.

Figure 27 - BGP Configuration for In-Path Hybrid Branch VCE

Note that this configuration should always be paired with the option “Advertise Uplink
Routes” unchecked for the branches in Overlay Flow Control page; it is disabled by default.
The “Advertise Uplink Routes” flag is meant to override the Uplink feature for redistribution
from BGP to Overlay, but not in the other direction from Overlay to BGP - which is controlled
by the option “Overlay Prefixes Over Uplink” under BGP settings.

Guide | 26
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 28 - “Advertise Uplink Routes” Flag in Overlay Flow Control

4.4.2 Caveat with Same BGP AS Number Used on SD-WAN Branch VCE’s
With scale and simplification in mind, it’s generally acceptable to configure the same ASN
on multiple VCE’s at the profile level to reduce the number of manual changes at the Edge
Override level. However, since eBGP by default reject routes with its own ASN in the AS-
path, the option “Allow-AS” should be configured on upstream BGP peer to override the
behavior for received routes. The flag is configured on a per-peer basis per the Figure
below.

Figure 29 - “Allow AS” Flag for BGP Neighbor

Note that similar to traditional routing setups, this behavior also applies to router or core
switch downstream from the VCE that will be learning remote branches’ prefixes from the
VCE. If the downstream device does not have the “Allow AS” option, you can also disable
AS-Path Carryover on the VCE for routes redistributed from the Overlay - although this is
not recommended due to troubleshooting purposes as origin AS of the BGP route will not be
easily determined. The placement of the flag is in the Advanced Settings section in BGP
Settings per Figure below.

Guide | 27
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 30 - “Disable AS-Path Carry Over” for Routes Redistributed from Overlay to BGP

4.5 Off-Path Hybrid Branch Insertion with BGP to MPLS

This section will discuss insertion topologies where the VCE is off-path and connected to a
router or core switch via BGP for connectivity and routing to both MPLS and the local
prefixes. Similar to the DC insertion discussed earlier, the Uplink Community feature should
be applied to distinguish the MPLS prefixes from the local subnets and prevent redistribution
of the MPLS prefixes into the Overlay. This is shown in the Figure below.

Guide | 28
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 31 - Off-Path Insertion with BGP Peering to the Core Switch

4.5.1 Configuring BGP for Off-Path Hybrid Branch VCE

The corresponding BGP configuration is shown below. The VCE is configured with “Uplink
Community” of 12345:777. This means that we’re expecting all prefixes learned from MPLS
to be tagged by the core switch or the CE with a community of 12345:777, so consequently
the VCE can use the community to prevent those routes from being redistributed to the
Overlay.
Take note the uniform tagging of routes with communities similar to other configurations
shown previously. Community of 12345:888 is used to mark the SD-WAN prefixes
advertised by the VCE and community of 12345:999 is used to tag the Overlay tunnel

Guide | 29
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

sources. The idea is to always deny prefixes tagged with 12345:999 on other VCE’s
because those tunnel source prefixes only need to be routable within MPLS since they don’t
carry end user subnets. The community of 12345:888 is used to tag the routes advertised
from the VCE to the BGP neighbor so the router or core switch can identify the prefixes from
the VCE and block them from being advertised to the CE and MPLS. This is discussed
further in section 4.5.2.

Figure 32 - Off-Path Hybrid Branch BGP configuration

Per configurations highlighted in red above, the main difference between the off-path and in-
path insertion at a hybrid branch is the requirement for the off-path VCE to redistribute the
Overlay Prefixes to its uplink BGP peer, which is not desired for in-path insertions. This is
accomplished by configuring Uplink Community rather than Uplink neighbor under
Advanced Settings shown above. It’s needed to redistribute SD-WAN prefixes to the router
or core switch to attract traffic from the local branch prefixes (B1) so SD-WAN can be
preferred over MPLS if those routes are also learned from MPLS. Using default route to
point at the VCE is not a good practice.

Guide | 30
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Using Uplink Community and allowing redistribution in the direction from Overlay to BGP
does present a challenge where the off-path Hybrid Branch VCE may become a transit. The
Figure below demonstrate the issues in hand. The main caveat here is that if the branch
VCE is redistributing Overlay routes from other SD-WAN branches to the core switch
through BGP, then these branch prefixes such as B2 which do not long to Branch1 may get
advertised to MPLS upstream and cause Branch1 to become a transit site. Though this may
not occur 100% of the time, it can potentially happen if the MPLS provider sees a shorter
AS-Path for Prefix B2 that is being advertised by the CE router in Branch 1. This can be
avoided by applying the methods discussed in section 4.5.2.

Figure 33 - Challenges with Off-Path Insertion with BGP Peering

4.5.2 Configure Route Map on Core Switch for BGP Peering to CE Router
Similar to Hub insertion discussed in section 4.1.4, to proactively prevent Branch 1 from
becoming a transit site, it’s highly recommended to configure route maps on the core switch
for BGP session to the CE router. The purpose of the outbound route map is to block the

Guide | 31
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Overlay prefixes advertised by the local VCE from being advertised to MPLS. As discussed
previously, the inbound route-map is used for two purposes. One is to tag the prefixes
incoming from MPLS with the Uplink Community (i.e. 12345:777) matching the configuration
on the VCE, and the other is to apply AS-Path prepend to the SD-WAN prefixes previously
tagged by other SD-WAN sites with a designated community. The community for the 2nd
purpose is 12345:888 in this example.
The desired end result is that the core switch at Branch 1 will learn the MPLS prefixes from
only the local CE router. It will learn the SD-WAN prefixes from both the VCE and the CE
router via BGP while it prefers the VCE due to AS-Path Prepend applied on the inbound
route map. The core switch will advertise only prefix B1 and block all other prefixes such as
B2 from being advertised to the CE router due to the outbound route-map
“BG_OUTBOUND”. In the failure scenario where the VCE at Branch 1 goes down, the traffic
from B1 to the other SD-WAN prefixes simply follows the routes with a longer AS-Path. The
output below demonstrates the route map configured on the core switch for its BGP session
to the CE router.
ip community-list 1 permit 12345:888
route-map BGP_OUTBOUND deny 10
description drop routes with community 12345:888 that were advertised from local VCE
match community 1
!
route-map BGP_OUTBOUND permit 20
description mark local routes from behind Switch (not from VCE) with community of 12345:888
set community 12345:888 additive
!
route-map BGP_INBOUND permit 10
description AS prepend routes with community 12345:888 from other SD-WAN sites and mark uplink community
match community 1
set as-path prepend 65901 65901
set community 12345:777 additive
!
route-map BGP_INBOUND permit 20
set community 12345:777 additive
Here’s a snippet from the BGP routing table of the core switch showing the same SD-WAN
prefixes being learned from the VCE (10.12.5.1) and from the CE (10.12.4.1) with the longer
AS-Path. Since all these prefixes belong to other SD-WAN branches, the VCE with next hop
IP of 10.12.5.1 is seeing as the better path on the core switch for the shown prefixes.
*> 192.168.128.0 10.12.5.1 10 0 65001 ?
* 10.12.4.1 0 65901 100 100 ?
*> 192.168.129.0 10.12.5.1 10 0 65001 65001 ?
* 10.12.4.1 0 65901 65901 65901 100 100 65001 ?
*> 10.12.5.1 10 0 65001 65001 ?
* 192.168.130.0 10.12.4.1 0 65901 65901 65901 65503 i

Guide | 32
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

*> 0.0.0.0 0 32768 i

*> 192.168.131.0 10.12.5.1 10 0 65001 65010 ?
* 10.12.4.1 0 65901 65901 65901 100 65010 ?
*> 192.168.132.0 10.12.5.1 10 0 65001 ?
* 10.12.4.1 0 65901 65901 65901 100 101 11 ?
4.6 Off-Path Hybrid Branch Insertion with OSPF to MPLS
The last off-path insertion topology we want to discuss is where the VCE is off-path and
connected to a router or core switch via OSPF for connectivity and routing to both MPLS
and the local prefixes. This is the least preferred deployment option due to the following:
1. Route manipulation and optimization with BGP is much easier than OSPF due to the number of
attributes such as AS-Path and Local Preference
2. Routing with BGP is more deterministic and easier to troubleshoot with AS-Path and other
attributes
3. Route tagging with BGP communities enables a simple loop prevention mechanism when
multiple Data Center Hubs exist
4. Currently VeloCloud does not have the OSPF equivalent of “Uplink” and “Uplink Community” for
automatic redistribution control
The insertion topology is shown in the Figure below where the VCE has a OSPF neighbor
with the core switch which also runs OSPF with the CE router. MPLS routes are
redistributed from BGP to OSPF by the CE router as E2 routes to the core switch. On the
VCE, if the “Overlay Prefixes” flag is checked in the Profile, then the VCE will redistribute its
Overlay prefixes to OSPF as E1 routes to make itself more preferred over the prefixes
redistributed from MPLS.

Guide | 33
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 34 - Off-Path VCE Insertion with OSPF Neighbor to the Core Switch

4.6.1 Configure CE Router to Deny Local Prefix from MPLS

Note the race condition mentioned in the Figure above. This is due to the possibility of prefix
B1 being learned from MPLS that was originally redistributed to MPLS from the Hub VCE. If
B1 is not directly connected on the core switch, this may result in race conditions where a
route flap on the core switch may cause the CE router to redistribute the B1 prefix learned
from the Hub into OSPF. Therefore, it’s recommended to apply a route map or prefix-list on
the CE router to prevent it from learning B1 from MPLS.
4.6.2 Configuring OSPF for Off-Path Hybrid Branch VCE

Guide | 34
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

On the Orchestrator, OSPF must be enabled in the profile for it to be applied to the VCE’s
assigned to that profile. Per the Figure below the option to redistribute prefixes from the
Overlay to OSPF is enabled by the “Overlay Prefixes” Flag. As mentioned previously, this
will cause the Overlay prefixes to be redistributed as E1 routes.

Figure 35 - Enable OSPF and Redistribution from Overlay in the Profile

After OSPF is enabled in the profile, the VCE’s interface forming the OSPF neighbor
relationship must be enabled with OSPF per Figure below. Note that currently as of Release
3.0 OSPF neighbor cannot be configured on the Vlan interface for peering. The Vlan
interface can only be configured as a passive interface.

Guide | 35
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 36 - Enabling OSPF on VCE’s interface

If you click on “toggle advance OSPF settings”, you will see that the default OSPF settings is
to enable “Inbound Route Learning” and disable “Route Advertisement”. It is configured this
way to prevent inadvertent advertisement of routes to OSPF. The Figure below shows
configuration settings where the Default Action for route advertisement is switched to
“Advertise”, and there are four routes that are being blocked. In this case, the 4 routes are
data center Hubs’ tunnel source prefixes that are being blocked from being advertised into
OSPF for precaution. The Overlay tunnel over the MPLS network may go down if the core
switch learns the Hub’s tunnel source route from the VCE instead of pointing to MPLS.

Guide | 36
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

Figure 37 - Configure OSPF Route Learning and Route Advertisement Settings

4.6.3 Route Redistribution Control for Off-Path Hybrid Branch VCE

Similar to the Hybrid SD-WAN branches with BGP, redistribution control is also needed in
this topology to prevent the branch from becoming a transit. To prevent the branch VCE in
this example from redistributing routes from MPLS to the Overlay, it’s recommended to
disable “Advertise External” flag for branch Edges in Overlay Flow Control page so the VCE
does not redistribute OSPF External routes to the Overlay. This will restrict the VCE to
redistribute only the IntraArea or InterArea routes that are local to the branch.

Figure 38 - Overlay Flow Control Flag to Disable Redistribution of OSPF External Routes to Overlay

Redistribution control from the Overlay to the MPLS through OSPF is not as easily
controlled compare to BGP (discussed in section 4.5.2) as there is no community attribute in
OSPF and VeloCloud currently does not support route tagging to OSPF through other
means. Therefore, the simple but crude method is to exclude redistribution of OSPF

Guide | 37
 ROUTING DESIGN GUIDE – UNDERLAY / OVERLAY ROUTING

External routes to BGP on the CE Router. This should not be configured in deployments
where the VCE is inserted in the data center since the Hub location should behave as a
transit.
Here’s a snippet of a BGP configuration on a CE router where the redistribution from OSPF
to BGP only redistributes OSPF inter- and intra-area routes.
address-family ipv4 vrf SIL1_MPLS
bgp router-id 192.168.100.3
redistribute ospf 1 metric 100
neighbor 10.99.128.1 remote-as 100
neighbor 10.99.128.1 activate
neighbor 10.99.128.1 send-community both
neighbor 10.99.128.1 soft-reconfiguration inbound
exit-address-family

Guide | 38