Sprint IP Backbone Network and MPLS: Contents
Sprint IP Backbone Network and MPLS: Contents
Multiprotocol Label Switching or MPLS1 is an The Sprint services described in this paper run
encapsulation-based control and data plane over Sprint’s 100% native IP backbone and have
designed to provide services that are claimed received the Cisco Powered Network designation
to be superior to those provided by conventional from Cisco Systems. The designation and logo,
IP networks. shown below, identify network services that are
business-ready and offer the performance, security
While some carriers have implemented MPLS,
and scalability of networking solutions built with
Sprint has arrived at the conclusion that there are
Cisco technologies and products. A service offering
no compelling technical, market or financial reasons
with the Cisco Powered Network mark offers users
to deploy MPLS in the Sprint IP backbone network.
confidence that they can easily extend their LAN or
This technical report will examine the capabilities campus networks and obtain optimal, end-to-end
of MPLS, its strengths and its weaknesses. We will service quality and compatibility with their existing
also describe the current architecture and design networking equipment. Sprint services that have
philosophy behind the Sprint Tier 1 IP backbone received the Cisco Powered Network designation
network and explain why our approach results in from Cisco may be found at www.cisco.com/cpn.
a network that is easier to operate and scale,
while simultaneously providing equivalent or
exceptional functionality.
This whitepaper is provided for informational purposes only. Please consult your company’s policies and procedures before
implementing any solutions presented in this document.
2
MPLS overview
MPLS was originally designed to overcome IP MPLS found its first real application as a traffic
forwarding hardware performance limitations by engineering (TE) tool with the introduction of
switching on fixed-length tags or labels (as opposed signaling protocols such as reservation/traffic
to IP’s longest-match lookup), an issue that has engineering (RSVP-TE)5 and CR-LDP.6 These
since been resolved through hardware develop- signaling protocols are used in conjunction with
ment. In addition, MPLS was developed to address constraint-based routing to form explicit-routed
scaling limitations with ATM’s segmentation and (ER) paths. Explicit routing (also known as source
reassembly function (SAR). This is supported in routing) is central to MPLS’s traffic engineering
earlier proposals, such as Ipsilon’s IP Switching,2 capabilities. Forwarding information, such as
IBM’s Aggregate Route-based IP Switching,3 source and destination IP address, can be sum-
John Moy’s IP Navigator and Cisco’s Tag Switching.4 marized into forwarding equivalence classes
MPLS switches packets using a 4-octet label. Like (FECs). If two or more FECs with a common
an ATM VPI/VCI or frame relay DLCI, the label destination are forwarded on different paths,
values have only local significance. In particular, the they are said to be explicitly routed. Because
labels pertain only to hops between label-switching packets with a common destination might not
routers (LSRs). Each label is part of a complete follow the same path, explicitly routed traffic does
path through the network, referred to as a label- not necessarily follow ordinary IP routing rules,
switched path (LSP). The creation of LSPs is the allowing some forms of TE that standard IP load
responsibility of the signaling protocol. Although balancing cannot achieve. In particular, MPLS-TE
router vendors provided mechanisms for manual is accomplished by explicitly routing FECs on
LSP configuration (static LSPs), the lack of adequate different possible paths, in contrast to standard
signaling protocols was an early limitation for MPLS. IP-based load balancing, in which an FEC is
evenly divided across multiple paths (i.e., equal-
cost multi-path). In practice, MPLS-TE is most
frequently used to direct traffic around
underprovisioned segments of a network.
3
MPLS capabilities
4
MPLS capabilities
5
Sprint’s IP backbone design
6
Sprint’s IP backbone design
cost associated with building capacity early is supporting work includes industry-wide operator
measured in months, not years. Also, Sprint’s experience and recent work by Mikkel Thourup of
minimum unit of investment is at the OC-48 level, AT&T labs that shows that metrics for a link-state
moving toward OC-192. This significantly lowers protocol such as IS-IS can be computed to dis-
the cost basis of excess capacity. Given Moore’s law, tribute traffic efficiently.18 Finally, effective traffic
Sprint has found it more cost-effective to scale distribution using shortest-path-based routing is
hardware than to scale the operational overhead also supported in Bernard Fortz and Mikkel
traffic engineering requires. Thourup’s INFOCOMM paper on traffic engineering.19
7
Sprint’s IP backbone design
Quality of Service (QoS) The delay assertions previously made are based
Providing ample network capacity is a simple on both queuing theory and empirical data. In
and effective technique to ensure high-quality particular, studies of the Sprint IP backbone core
service to all packets on the Sprint IP backbone. network show that its service is characterized by
If the network is properly provisioned, data travels low loss and minimal jitter.22
through the network routers on a first in first out
basis without queuing or delay. Since all known VoIP requirements and Sprint IP
backbone network performance
QoS mechanisms operate by reordering and
The subjective quality of voice transmission is
prioritizing traffic in the router queues, deploying
predicted by the E-model standard.23 E-model
these mechanisms on the Sprint IP backbone will
ratings of interest are values below 60, which
provide little value and may add queuing delay. In
indicate unacceptable quality, and values of 70
addition, as the capacity of network links increases,
and above, which correspond to public switched
end-to-end delay associated with queuing
telephone network (PSTN) toll quality. In addition
decreases.21 The component of delay due to
to the low loss and jitter results reported by a
propagation of light in fiber will remain constant
recent analysis show that the average E-model
and probably dominate the queuing delay. The end
rating for the Sprint IP backbone network was
result of this evolution will be a decreased impact
90.27 and only one out of approximately 3,600
of complex queuing disciplines.
calls was rated below 70.24 These results indicate
Although the network core has ample capacity and that the Sprint IP backbone network can support
is expected to have minimal queuing, the same a high-quality voice service based on voice over IP
cannot be said for the edges of the network. (VoIP) without additional QoS mechanisms.
Customer access links represent a prime location
for implementing queuing mechanisms, as these Layer 2 transport
links are frequently of low bandwidth, overutilized Sprint’s position is that IP encapsulation has the
and experience bursty use patterns. The application best potential to provide a flexible and scalable
of Class of Service (CoS) features on the customer layer 2 transport mechanism for the Sprint IP
access links has the best potential to offer meaningful backbone network. This can be achieved using
service differentiation. The mechanisms deployed various encapsulations, including GRE25 and
on the customer access links to implement end-to- L2TPv3.26 These encapsulations attach a tag to
end QoS are independent of MPLS. Edge CoS packets being transmitted in a method similar to
mechanisms supported by Sprint include Class MPLS’s encapsulation. However, in these cases, the
Based Weighted Fair Queuing (CBWFQ) and Low tag is simply another IP header, which allows the
Latency Queuing (LLQ). downstream routers to forward the encapsulated
packet without network storage of additional
8
Sprint’s IP backbone design
control plane state. Another way to look at this is Finally, while BGP/MPLS VPN offers potential
that the additional complexity involved in deploying configuration scaling advantages over other
MPLS is unwarranted if L2 VPN is the primary approaches,27 no in-depth analysis has been offered
application.17 of its effect on the stability of the existing platform
or its associated OPEX. Industry experience shows
IP VPNs that carriers that have deployed BGP/MPLS VPNs
Sprint provides a layer 3 VPN solution based on have not done so on their IP backbones; rather,
secure tunneling through the Sprint IP backbone they have migrated their ATM networks to MPLS
using dedicated equipment to connect customers and implemented BGP/MPLS VPNs on that
to the edges. This solution offers the best scaling infrastructure. That is, best industry practice is not
properties from a core network perspective, as the to deploy BGP/MPLS VPNs on a Tier 1 IP backbone.
core network knows nothing of the VPN. However, As such, Sprint has chosen to develop solutions,
customers often want to outsource more of their such as CPE-based VPNs, network-based VPNs and
VPN requirement to a service provider, so carriers IP Intelligent Frame Relay services, that offer
seek to provide network-based solutions. The customers a wide-range of service flexibility.
Sprint IP backbone network approach is to support
IPSec-based VPN solutions, which will use standard Restoration
IP datagrams for transport. This approach allows In the past, Sprint addressed the restoration
the Sprint IP backbone network to provide a problem with SONET protection, which detects net-
secure, robust and flexible VPN. Because these work link failures and restores connectivity within
services are managed at the edges, no complex 50 milliseconds (ms). However, as link capacity
modifications — such as adding new address grew beyond SONET’s capabilities, a new solution
families to BGP — are required in the core. had to be identified. Circuits that are not protected
by SONET are restored at layer 3 using IS-IS.
As a point of comparison, the BGP/MPLS approach
to VPNs is not really limited to the MPLS encap- Several mitigating factors minimize the impact of
sulation or control plane. BGP/MPLS VPNs could the larger convergence time associated with layer 3
just as easily use an IP encapsulation and nested restoration. Convergence time is the combination
tunnels. (Label stacking provides no additional of the time required to distribute link-state
power over nested IP tunnels; in fact, a label stack information throughout the network and the time
is just another way to represent tunnel nesting.) required to recompute the shortest-path topology
Rather, the real scaling power of BGP/MPLS VPN (SPF runtime). The latter is a function of the
comes from the RD address family extension and number of adjacencies and the network topology.
the use of BGP to carry VPN routes.17 The fact that Sprint carries a minimal number of
9
Sprint’s IP backbone design
adjacencies in IS-IS, combined with its relatively As MPLS has not been provably shown to provide
simple network topology, results in SPF runtimes in significantly better service over what is achievable
the sub-100 ms range. The link-state distribution with IS-IS routing in the area of convergence and
time is gated by several factors, including fault restoration times, Sprint will continue to follow the
detection delay, hold-down timer delay and the Simplicity Principle for network restoration.
delay associated with propagating information to
the outer diameter of the network. Alaettinoglu and Provisioning
colleagues describe techniques to minimize these Given that the Sprint IP backbone currently takes
delay components and improve SPF run times.28, 29 advantage of Sprint’s large embedded fiber
infrastructure, the value associated with dynamic
MPLS-based restoration may or may not offer a
reprovisioning appears to be minimal. However,
significant improvement in convergence time. In
there is potential value in the underlying switching
particular, if alternative paths are not precomputed,
capability offered by programmable OXCs. Sprint is
the convergence time would be similar to non-MPLS
currently investigating the potential benefits of an
systems. Precomputing protection paths might
optical control plane for its optical network.
improve the convergence time, but it might also
create new problems. If one wishes to precompute
a protection path for a given circuit, every link must
be distinct from those comprising the primary path.
This may significantly limit the efficiency of the
protection path, which runs counter to the MPLS-TE
goals of efficient traffic distribution. In addition, the
delays associated with detecting the link failure and
propagating that information along the LSP may
dominate the convergence times in both the MPLS
and non-MPLS cases.
10
Conclusion
Sprint operates a native IP backbone. After Sprint is committed to providing the highest quality
extensive and ongoing study, Sprint finds no network in the industry, and as such, we will
compelling technical or financial reason to deploy continue to evaluate new technologies. Sprint’s
MPLS. Sprint’s view is that MPLS, at its current state guiding philosophy is “simplicity.” It is based on
of maturity, provides few functions for a native IP the observation that simple networks are easier to
backbone that cannot be delivered through simpler operate and scale, resulting in better service to
and more scalable means. customers and greater cost effectiveness.
11
References
1. “Multiprotocol Label Switching Architecture,” E. Rosen 13. “Multiprotocol Extensions for BGP4,” T. Bates, R. Chandra,
et al., RFC 3031, January, 2001. D. Katz, and Y. Rekhter, RFC 2283, February, 1998.
2. “IP Switching: ATM Under IP,” Peter Newman, Greg Minshall, 14. “Nonlinear Dynamics and Chaos,” J.M.T. Thompson and
and Tom Lyon, IEEE ACM Transactions on Networking, Vol. H.B. Stewart, John Wiley & Sons, 1994, ISBN 0471909602.
6, No. 2, April, 1998, pp. 117–129.
15. “The Synchronization of Periodic Routing Messages,”
3. “Aggregate Route-based IP Switching (ARIS),” IBM. Sally Floyd and Van Jacobson, IEEE ACM Transactions
on Networking, 1994.
4. www.cisco.com/warp/public/732/tag/tagsw_ov.htm
16. “Congestion Avoidance and Control,” Van Jacobson,
5. “A Provider Architecture for Differentiated Services and
Proceedings of ACM Sigcomm 1988, pp. 273-288.
Traffic Engineering (PASTE),” T. Li and Y. Rekhter, RFC
3209, October, 1998. 17. “Comments on MPLS,” Bruce Davie, invited talk, Sprint
Advanced Technology Laboratory, January 15, 2002.
6. “Constraint-Based LSP Setup Using LDP,” B. Jamoussi, Editor,
L. Andersson, R. Callon, R. Dantu, L. Wu, P. Doolan, T. 18. “Optimizing OSPF/IS-IS Weights in a Changing World,”
Worster, N. Feldman, A. Fredette, M. Girish, E. Gray, J. Bernard Fortz and Mikkel Thourup, to appear in IEEE
Heinanen, T. Kilty, A. Malis, RFC 3212, January, 2002. JSAC Special Issue on Advances in Fundementals of
Network Management, Sprint, 2002.
7. “MPLS-based layer 2 VPNs,” K. Kompella, et al., draft-kom-
pella-mpls-l2vpn-02.txt, Work in Progress. 19. “Fortifying OSPF/IS-IS against link failure,” Mikkel
Thourup, AT&T Labs, September 7, 2001.
8. “Encapsulation Methods for Transport of layer 2 Frames
Over IP and MPLS Networks,” Luca Martini, Nasser El- 20. “IS-IS Link Weight Assignment for Transient Link Failures,”
Aawar, Steve Vogelsang, Daniel Tappan, John Shirron, Eric Bhattacharyya, Supratik, et al. [list all authors], Sprint
C. Rosen, Toby Smith, Alex Hamilton, Jayakumar Advanced Technology Laboratory, February, 2002.
Jayakumar, Vasile Radoaca, Dimitri Stratton Vlachos,
21. “Models for a Self-Managed Internet,” F. P. Kelly,
Andrew G. Malis, Chris Liljenstolpe, Vinai Sirkay, Giles
Philosophical Transactions of the Royal Society, 2000.
Heron, Dave Cooper, Kireeti Kompella, draft-martini-l2cir-
cuit-encap-mpls-04.txt, Work in Progress. 22. www.sprintlabs.com/Department/IP-Interworking/Monitor
9. “Encapsulation Methods for Transport of layer 2 Frames Over 23. “The E-Model — A Computational Model for Use in
MPLS,” Luca Martini, Nasser El-Aawar, Steve Vogelsang, Transmission Planning,” ITU-T Recommendation G.107,
Daniel Tappan, John Shirron, Eric C. Rosen, Toby Smith, Alex May, 2000.
Hamilton, Jayakumar Jayakumar, Vasile Radoaca, Dimitri 24. “Impact of Link Failures on VoIP Performance,” Catherine
Stratton Vlachos, Andrew G. Malis, Chris Liljenstolpe, Vinai Boutremans, Gianluca Iannaccone, and Christophe Diot,
Sirkay, Giles Heron, Dave Cooper, Kireeti Kompella, draft- submitted to the NOSSDAV Workshop, January, 2002.
martini-l2circuit-trans-mpls-09.txt, Work in Progress.
25. “Generic Routing Encapsulation,” Dino Farinacci, Tony Li, Stan
10. “BGP/MPLS VPNs,” E. Rosen and Y. Rekhter, RFC 2547, Hanks, David Meyer, Paul Traina, RFC 2784, March, 2000.
March, 1999.
26. “Layer Two Tunneling Protocol (Version 3) ‘L2TPv3,’” J.
11. “A Method for Setting an Alternative Label Switched Path to Lau, M. Townsley, A. Valencia, G. Zorn, I. Goyret, G. Pall,
Handle Fast Reroute,” Dimitry Haskin and Ram Krishnan, A. Rubens, B. Palter, draft-ietf-l2tpext-l2tp-base-02.txt,
draft-haskin-mpls-fast-reroute-05.txt, Work in Progress. Work in Progress.
12. “Generalized Multiprotocol Label Switching (GMPLS) 27. “Network-based VPN Testing,” Sprint Advanced Technology
Architecture,” Peter Ashwood-Smith, Daniel Awduche, Laboratory, December, 2001.
Ayan Banerjee, Debashis Basak, Lou Berger, Greg
Bernstein, Sudheer Dharanikota, John Drake, Yanhe Fan, 28. “Toward Milli-Second IGP Convergence,” C. Alaettinoglu, V.
Don Fedyk, Gert Fortz, Bernard Grammel, Dan Guo, Jacobson, and H. Yu, draft-alaettinoglu-isis-convergence-
Kireeti Kompella, Alan Kullberg, Jonathan P. Lang, Fong 00.txt, Work in Progress.
Liaw, Thomas D. Nadeau, Lyndon Ong, Dimitri 29. “ISIS Routing on the Qwest Backbone,” C. Alaettinoglu and
Papadimitriou, Dimitrios Pendarakis, Bala Rajagopalan, Steve Casner, www.nanog.org/mtg-0202/cengiz.html,
Yakov Rekhter, Debanjan Saha, Hal Sandick, Vishal NANOG24, February 10-12, 2002.
Sharma, George Swallow, Z. Bo Tang, Jennifer Thourup,
Mikkel Yates, George R. Young, John Yu, Alex Zinin, draft-
ietf-ccamp-gmpls-architecture-02.txt, Work in Progress.
12