Architecture and Design of Metro

Ethernet over Optical Transport

Networks

OPT-4040
8060_05_2003_c2 © 2003, Cisco Systems, Inc. All rights reserved. 1
 Architecture and Design of Optical
Transport Networks

• Metro Ethernet Services

• Metro Ethernet options
• Mapping Data over SONET
• Optimizing SONET for Data Transport
• Topologies and Architectures

© 2003, Cisco Systems, Inc. All rights reserved.

2
Metro Ethernet Services Market

• Ethernet Private Line (EPL) Pt-Pt CAGR:58.6%

• Ethernet Internet Access (EIA) CAGR: 102.4%
• Ethernet Any-to-Any CAGR: 134%
Yankee Group Report August 19,2003

© 2003, Cisco Systems, Inc. All rights reserved.

3
What is Metro Ethernet?
SONET/SDH
RPR Remote
Regional Office 1
Headquarters DWDM/CWDM
Ethernet
MPLS/IP
Ethernet-
Connected
10 Mbps Branch
Ethernet

SP Remote
100 Mbps Metro Ethernet Office 2
Ethernet Network
10 Mbps
Ethernet

• Delivers an Ethernet UNI to

Remote
Enterprises/SMB for MAN/WAN Office 3
connectivity Ethernet-
Connected
Branch
• SP has multiple transport options
© 2003, Cisco Systems, Inc. All rights reserved.
4
What Does Ethernet as a
LAN/MAN/WAN Transport Offer?

• Ethernet becomes the ubiquitous interface:

single technology for LAN, MAN and WAN
• Efficient packet-based infrastructure: IP friendly
• Cost effective interface with flexible bandwidth
offerings: 10/100/1000/10000 Mbps
• Geographical independence: Ethernet over
Optical, IP or MPLS

© 2003, Cisco Systems, Inc. All rights reserved.

5
Enterprise Applications Drive
Metro Ethernet
Transparent
• LAN interconnect Services
• Service aggregation High Availability
• Interconnect
data centers High Bandwidth

• Backup and Low Latency

disaster recovery
Cost Effective
• Connect to
hosting services
Manageable
• Value-added services and Secure

How SPs Deliver This Is Largely Irrelevant…

Metro Ethernet Is Simply a Tool in the Tool Box
© 2003, Cisco Systems, Inc. All rights reserved.
6
Summary of Ethernet-Based Services

Ethernet-Based Services

Layer 1 Layer 2 Layer 3

Point-to-Point Multipoint

Ethernet
Ethernet Ethernet Ethernet Ethernet
Relay MPLS
Private Relay Wire Multipoint
Multipoint VPN
Line Service Service Service
Service

Transparent LAN Service

Similar to Leased Line

Analogous to Frame Relay

Analogous to Private Line

© 2003, Cisco Systems, Inc. All rights reserved.

7
Offering a Gigabit Ethernet Private
Line Service

Cisco Cisco
ONS 15454 ONS 15454

CE CE

• Dedicated, transparent private line between sites

• Guaranteed SLAs per connection
• Built with dedicated bandwidth/wavelength
• ML-series card integrates switching functionality
within the ONS-15454
• Can be built with SONET, CWDM, DWDM
© 2003, Cisco Systems, Inc. All rights reserved.
8
 The Ethernet Relay Service (ERS)
Non-Multiplexed
Interface
s 1
bp T
M
Service Multiplexed 20

Interface Remote Office 1

T3 20 Mbps
T 1

Remote Office 2

• Service multiplexing allows single port to

10
M
T1 A
provide service to multiple end-points (referred

bp
cce

s
ss
to as “point-to-multipoint)
• EVC identified by the VLAN
Remote Office 3
• CE-VLAN must be the same as SP-VLAN
• Service analogous to Frame Relay, functionally,
it is the same – encourages a router as CE edge
device, not a switch (except for remote site)
• ERS allows interworking with FR/ATM (future)
© 2003, Cisco Systems, Inc. All rights reserved.
9
 The Ethernet Wire Service (EWS)

int faste 0/1 int faste 4/0

T3

VLANs 101-110 Remote Office 2

VLANs 101-110
• Analogous to a private line in that all data
transverses, unaltered across the EVC
• Port-to-Port mapping, no service
multiplexing allowed. Therefore all services
must exist on one port (All-to-One Bundling)
• Switches or routers can be deployed as CE
edge devices © 2003, Cisco Systems, Inc. All rights reserved.

DMAC 10
The Ethernet Multipoint Service (EMS)

• Often referred to as a Transparent LAN

Service (TLS)
• Service Provider cloud appears to be a
switch, with UNI supporting VLAN
transparency and All-to-One Bundling
• While multipoint by definition, it can also
be point-to-point (such as EWS)
© 2003, Cisco Systems, Inc. All rights reserved.
11
Ethernet-based MPLS VPNs
Connection-Oriented Connectionless
VPN Topology VPN Topology
VPN A VPN A
VPN B VPN B
VPN C VPN C
VPN C VPN B VPN C VPN B

VPN A
VPN A
VPN A
VPN A

VPN B
VPN B VPN C VPN C
VPN B
VPN C VPN C VPN A
VPN B
VPN A

• Managed service opportunity for the provider (more

money for the SP)
• Opportunity for provider to offer “value-added” services
(such as content hosting, IP Centrex)
• Ethernet access provides low-cost, “customer friendly”
interface
• Any Layer 2 access mechanism at the UNI (interworking
comes “for free”) © 2003, Cisco Systems, Inc. All rights reserved.
12
 Metro Ethernet Options

OPT-4040
8060_05_2003_c2 © 2003, Cisco Systems, Inc. All rights reserved. 13
Transport Options – Ethernet over Sonet
vs.Ethernet over Dark Fiber/xWDM
Ethernet Relay Ethernet Relay
Service Service

Cisco 7600 Cisco 7600

Catalyst 3550 Cisco 12000 Cisco 12000
Catalyst 3550
End-to-End Dark Fiber SONET
STM-16/
Or CWDM/DWDM STM-64
ONS 15454
Ethernet Relay ONS 15454
Service
• Transparent LAN-optimized • Ethernet service over an existing
structured transport network
• Greenfield and overbuild
architectures • Single CPE
• Effective for a small number of • Larger implementations
customers • Build-in resiliency scheme (UPSR,
• Redundancy has to be carefully BLSR, SRP) can eliminate
planned due to Spanning Tree Spanning Tree
and MPLS © 2003, Cisco Systems, Inc. All rights reserved.
14
 Logical Hub-and-Spoke on a Physical
Fibre Ring

MUX-AD-1510
IP/MPLS

MUX-8
MUX-AD-1530

MUX-8 MUX-AD-1550

East Facing GBIC

MUX-AD-1570
West Facing GBIC

• Many providers will look to deploying a physical ring,

but using a logical point-to-point service with
guaranteed SLAs on top
• Technologies such as SONET/SDH, CWDM, and
DWDM provide this capability
© 2003, Cisco Systems, Inc. All rights reserved.
15
Different Methods Solve Different Problems

SONET/SDH Switched Ethernet using

Spanning Tree Protocol

DWDM and SDH

Backbone
Network

OC-N

STM1/4/16/64 STM1/4/16/64
Business Metropolitan
Network Network

STM-N STM-N
Metro
POP

DPT/RPR DWDM/CWDM

© 2003, Cisco Systems, Inc. All rights reserved.

16
 Comparison of Ring Technologies
SONET/SDH Switched Ethernet using Spanning Tree
• Installed base in service providers • Low cost solution over dark fiber
• Evolutionary approach for the service • Perceived simplicity of Ethernet
providers vs. revolutionary switching
• Best choice for large scale deployments • Fairness, bandwidth, delay/jitter
• Best use of fiber infrastructure dependant on location on the ring
• Hierarchical bandwidth • 10-12 node limit
• Best fiber utilization for dual homing • LAN switch as edge device (no MPLS,
• Next-gen SONET, with VCAT, LCAS, Traffic shaping, BGP, etc)
GFP, will help optimize SONET for data • 1-50 seconds convergence (standard
• 50 ms convergence 802.1d or 802.1w)
DPT/RPR CWDM
• Shared packet ring scales bandwidth • Guaranteed bandwidth per lambda up
up to 5 Gbps to 8 nodes
• SONET framing provides insertion • Logical star over a physical ring
point for many providers • EtherChannel or Layer 3 load balancing
• Spatial reuse provides good bandwidth for redundancy
utilization • 200 millisecond failover
• Optimized for Layer 3 (currently) • More consistent delay/jitter and better
• Large number of nodes (128) per ring access onto the ring
• 50 ms convergence •Still a low-cost Ethernet switch at edge
© 2003, Cisco Systems, Inc. All rights reserved.
17
Cisco COMET Multiservice over
SONET/SDH (MSOS) Strategy
• Deploying advanced services is
key to profitability Advanced Services
• Ethernet, Video, VoIP, and SAN
Interconnect key service needs

• QoS is necessary for advanced

services and providing SLAs
• Cisco IOS provides advanced per- QoS Engine
packet, per port QoS
• Packet Multiplexing is crucial for
network efficiency and scale
• Cisco technology innovations Packet Multiplexing to the
(MPLS, DPT, CDL, EoS) enable this Transport Layer
• SONET/SDH will need to become
more efficient
• GFP, CCAT, VCAT, and LCAS all Data-Optimized SONET/SDH
offer incremental improvements
© 2003, Cisco Systems, Inc. All rights reserved.
18
ONS 15454 ML-Series Ethernet Cards
• ML-Series enables private line
services and Layer 2/3 Packet
Multiplexing into SONET/SDH
• Packet processing capabilities
allow creation of multipoint
services
• Common QoS feature set and
code base with existing
enterprise networks
• Common management with
Cisco Transport Manager, as
well as SNMP and TL1 support

© 2003, Cisco Systems, Inc. All rights reserved.

19
ONS 15454 Ethernet Capability
Customer Driven – Continued Traction

Shipping Ports Services Deployed

Since Deployed

E-Series 1999 Tens of Layer 1 Private Line and

Thousands basic switched services

G-Series Thousands Line Rate Gigabit Ethernet

2002
Private Line

Integrated Layer 2/3

ML-Series 2003 Trials capabilities to provide
advanced switched
NEW!!! services and SLAs

© 2003, Cisco Systems, Inc. All rights reserved.

20
 Packet Multiplexing – Flexible
Architectures
Internet/LH/Data BB
Point to Point, Hub & Spoke,
and Resilient Packet Ring
architectures
Cisco
7600
Seamless transition to
Enterprise A pure packet network
ONS with Cisco IOS
15454

Multimillion
ONS
packets per 15454
second ONS Access Ring X Mix & match:
performance at 15454 Catalyst, G-
L1/L2/L3 Series & ML-
Series solutions
ONS
Enterprise C Enterprise C
15454

Frame Relay-like SONET/SDH and/or per VLAN

Ethernet Services Rapid Spanning Tree Protected
With QoS Enterprise B Enterprise C
© 2003, Cisco Systems, Inc. All rights reserved.
21
 Advanced QoS and Layer 3
Capabilities

QoS Actions QoS Actions Layer 3

at Ingress at Egress Functionality

Classification Policing & Marking Queue & Schedule

Tokens

— Burst Siz e
Token Ar rival Rate
Overflow
Tokens

Packets
Packets deQueued
Arr iving

Exceed or Violat e

• Classification • Mark or • Fairly • Static IP

by: Discard non- allocate routing:
conformant unused
Interface OSPF, ISIS,
packets bandwidth
EIGRP, BGP,
Bridge Group • Multiple RIP, HSRP,
802.1p queues/port VRF-Lite and
ACL
IP Precedence
• MPLS-capable
DSCP

© 2003, Cisco Systems, Inc. All rights reserved.

22
 Provisioning of ML-Series based
Profitable Ethernet Services
SONET/SDH Circuit provisioning via CTM, CTC, and TL1 ONS
A to Z Point-Click Auto Provisioning … 15454

Internet Enterprise B
Access Ring
OC-12
Enterprise A InterOffice Ring
OC-48
Enterprise A

Access Ring
OC-12
STEP 3: Select Destination (Z)
– node, slot, port
Enterprise C
ML-Series and IOS: STEP 1: Set Circuit Attributes –
CTM or CLI for name, type, size
Configuration, SNMP Enterprise B Enterprise C
for Monitoring STEP 2: Select Source (A) –
• Alarms node, slot, port
• Performance
• All L2/L3 Features © 2003, Cisco Systems, Inc. All rights reserved.
23
 Mapping Data over
SONET/SDH
(GFP—Generic Framing Procedure)

OPT-4040
8060_05_2003_c2 © 2003, Cisco Systems, Inc. All rights reserved. 24
ITU Recommendation G.7041 (GFP)

• Generic Framing Procedure expands on

functionality of X.86 in that it provides for:
Encapsulation of L2/L3 PDU client signals (GFP-F)
Encapsulation/mapping of block coded client
signals (GFP-T)
Multiplexing of multiple client signals into a
single payload

© 2003, Cisco Systems, Inc. All rights reserved.

25
 GFP Network View
Escon/Ficon Escon/Ficon

10/100/GbE 10/100/GbE
L2/L3 MSPP MSPP L2/L3
Switch Switch

Data Layer
Bridge/Switch/Router in TNE Network Bridge/Switch/Router in TNE

PHY-T GFP-T GFP-F GFP-F GFP-F GFP-F GFP-T PHY-T

PHY-X PHY-X
10Base
10Base Transport Layer Network 100Base
100Base ESCON ESCON 1000Base
1000Base FICON FICON 10GBase
10GBase Fiber Channel Fiber Channel
1000BaseLX 1000BaseLX
1000BaseCX 1000BaseCX
© 2003, Cisco Systems, Inc. All rights reserved.
26
Pros and Cons GFP

• Pro:
Supports Ethernet, PPP and SAN interfaces
ITU Standard

• Con:
Added flexibility adds complexity (not really that bad
though and can be overcome by a good provisioning
scheme)

© 2003, Cisco Systems, Inc. All rights reserved.

27
Pros and Cons BCP

• Pro:
IETF endorsed
It is simple
Can be deployed today!
• Con:
Most customers want GFP

© 2003, Cisco Systems, Inc. All rights reserved.

28
Points To Be Heard

• In terms of overhead, GFP has more overhead;

however, BCP uses padding and GFP uses idle
patterns for synchronization; these can all add
considerable overhead, thus making the two
roughly equal
• BCP transports Ethernet and some useless LAN
protocols, while GFP transports Ethernet, PPP
and SAN protocols
• GFP is an ITU standard which carries more
weight with IXCs, RBOCs and PTTs

© 2003, Cisco Systems, Inc. All rights reserved.

29
 Optimizing SONET/SDH for
Data Transport
(STS BW Scaling, Virtual Concatenation)

OPT-4040
8060_05_2003_c2 © 2003, Cisco Systems, Inc. All rights reserved. 30
STS Bandwidth Scaling
10/100 EoS Card STS-12c GigE EoS Card STS-12c GigE EoS Card
STS-6c STS-6c
Side View STS-3c Side View STS-3c Side View

Gigabit Ethernet (GBIC) Ports

Gigabit Ethernet (GBIC) Ports
STS-1 STS-1
Port 1 STS 1 STS 1
10/100 RJ 45 Ethernet Ports

Port 2 STS 1 STS 1 STS-1

Port 1

ETHERNET CARD
ETHERNET CARD
STS-3c
ETHERNET CARD

Port 3 STS 1 STS 1

Port 1 STS-6c
Port 4 STS 1 STS 1
STS-9c
Port 2
Port 5 STS 1 STS 1 STS-12c
Port 6 Port 2 STS-24c
STS 1 STS 1
Port 3
Port 7 STS 1 STS 1
Port 8 STS 1 STS 1 Port 4
Port 9 STS 1 STS 1
Port 10 STS 1 STS 1
Port 11 STS 1 STS 1
Port 12 STS 1 STS 1

Example:
• 100 mbps service required for a CMTS (residential cable access) and a high
school LAN
• Assume STS bandwidth scaling onto a shared STS-3c
• Service provider preserves 155 mbps of transport bandwidth (enough to deliver
84 DS-1 services for over $20K/month or 3 DS-3 services for over $10K/month)

Note: Same Operation for SDH, Mapping Changes to Fit VC-n Structure
© 2003, Cisco Systems, Inc. All rights reserved.
31
SONET/SDH Virtual Concatenation (VC)

• Virtual Concatenation is a method of creating a

payload made up of 2 or more associated SPEs
transported through a network completely
independently
• Channels are “administrated” together; common
processing of channels is limited
to end points
• Channels not constrained to same path
(end point delay equalization required)
• Channels not necessarily constrained to same
transport channel (e.g., same STS-12)
© 2003, Cisco Systems, Inc. All rights reserved.
32
SONET:
Virtual Concatenation Efficiencies
SONET Paths (SPEs)
Service Bit Rate Without VC With VC

Ethernet 10 Mbit/s STS-1 (20%) VT1.5–7v (89%)

Fast Ethernet 100 Mbit/s STS-3c (67%) STS-1–2v (Approx. 100%)

Gigabit Ethernet 1000 Mbit/s STS-48c (42%) STS-3c–7v (95%)

Low Speed ATM 25 Mbit/s STS-1 (50%) VT1.5–16v (98%)

Fibre Channel 200 Mbit/s STS-12c (33%) STS-1–4v (100%)

Fibre Channel 1000 Mbit/s STS-48c (42%) STS-3c–7v (95%)

ESCON 200 Mbit/s STS-12c (33%) STS-1–4v (100%)

© 2003, Cisco Systems, Inc. All rights reserved.

33
Link Capacity Adjustment Scheme

• A mechanism for dynamically adjusting the size of a

virtually concatenated channel
Allows TDM services more flexibility for handling dynamic
bandwidth demands
Relies on the NMS/EMS to provision the bandwidth change
Allows channel size adjustment to be hitless

• Currently defined for SONET/SDH

Proposed for inclusion into G.709 OTN with ODUk VC

• Bit-oriented protocol encapsulated in control packets

• Applies to high order (STS) and low order (VT)

© 2003, Cisco Systems, Inc. All rights reserved.

34
 LCAS Signaling Protocol

• For high order VCs, it is communicated in a

control packet carried in bits 1–4 of the H4 Path
Overhead byte
Carried across a 16 frame multiframe

• For low order VCs, it is communicated in a

control packet in bit 2 of the Z7* Path Overhead
byte
Carried across a 32 frame multiframe

* aka K4 and originally V8

© 2003, Cisco Systems, Inc. All rights reserved.
35
 Topologies and Architectures

OPT-4040
8060_05_2003_c2 © 2003, Cisco Systems, Inc. All rights reserved. 36
Architecture Scenarios

• Dedicated Network

DWDM and
SONET
Backbone

• High Capacity Network

OC-N

Regional Network OC48/192

Business
Network
OC48/12
Metropolitan
Network

• High Volume Metro Network

Metro
POP © 2003, Cisco Systems, Inc. All rights reserved.
37
 Packet Multiplexing – Flexible
Architectures
Internet/LH/Data BB
Point to Point, Hub & Spoke,
and Resilient Packet Ring
architectures
Cisco
7600
Seamless transition to
Enterprise A pure packet network
ONS with Cisco IOS
15454

Multimillion
ONS
packets per 15454
second ONS Access Ring X Mix & match:
performance at 15454 Catalyst, G-
L1/L2/L3 Series & ML-
Series solutions
ONS
Enterprise C Enterprise C
15454

Frame Relay-like SONET/SDH and/or per VLAN

Ethernet Services Rapid Spanning Tree Protected
With QoS Enterprise B Enterprise C
© 2003, Cisco Systems, Inc. All rights reserved.
38
 ML-Series Layer 2 Features
For Service Providers
ML-Series L2 Advantages: Customer A L2VPN
L2 Internet Access/L2VPNs
L2 Mappings to MPLS E/FE/GE

L2 and SONET Protection

Stat Muxing Creates B/W Efficiencies Central Office
VLANs
Access Ring IP/MPLS
E/FE/GE SONET/SDH
Customer Switches/Routers

EoMPLS

15454 15454 ML/G-Series

VLANs
Cisco 7600
Access Ring IOF Ring
SONET/SDH GE/POS
E/FE/GE SONET/SDH
Customer Switches/Routers
15454 15454
Cisco 12000

E/FE/GE VLANs E/FE/GE

Internet

Customer A L2VPN
Customer Switches/Routers

© 2003, Cisco Systems, Inc. All rights reserved.

39
 ML-Series Layer 3 Features
For Service Providers
ML-Series L3 Advantages:
L3 Internet Access for Customers
L3 and SONET Protection
L3 Routing Protocol Support 12xxx
Rate Limiting Creates B/W Efficiencies IP / MPLS BGP
Internet
TDM Customer Network
PE Router OSPF- Area 0
Static Route DS-1/3
Cisco 7600 IS-IS – Level 2
OC-n Customer Router
VRF-Lite EIGRP - ASN

PoS FE/GE
GigE
15454
15454
SONET/SDH SONET/SDH

15454
15454 15454 15454
15454
FE/GE

Customer Routers FE/GE

FE/GE OSPF- Area Y
OSPF- Area X
IS-IS – Level 1
IS-IS – Level 1
EIGRP – ASN
EIGRP – ASN
BGP- ASN
BGP - ASN
Customer Routers Customer Routers
© 2003, Cisco Systems, Inc. All rights reserved.
40
 ML-Series Managed Services
For Service Providers
ML-Series Advantages:
Provide L2/L3 Services
Internet Access/Private Line
Single Customer Ring
Central Office
Cisco 7600
TDM Services
Internet
Voice
FE/GE
Services
DS-1/3
OC-n

Site 1 Site 4
15454 15454
E/FE/GE E/FE/GE

SP Owned
SDH/SONET Ring
15454 15454 15454
PBX PBX
E/FE/GE E/FE/GE

Site 2 Site 3
PBX PBX
© 2003, Cisco Systems, Inc. All rights reserved.
41
ML-Series Layer 2 Features
For Municipalities and Large Enterprises
ML-Series L2 Advantages:
LAN Interconnection
IP Video Distribution
VoIP – QoS Guarantees
Enterprise
Headquarters
VLAN A -Data
Voice Gateway
VLAN B To PSTN

VLAN C
IP Video Server

G/ML-Series
VLAN A -Data VLAN A -Data
Site 2 15454 Site 3
Enterprise Owned/
VLAN B – IP Voice VLAN B – IP Voice
Leased Fiber Ring

15454 15454
VLAN C -Video VLAN C -Video
ML-Series ML-Series

© 2003, Cisco Systems, Inc. All rights reserved.

42
ML-Series Layer 3 Features
For Municipalities and Large Enterprises
ML-Series L3 Advantages:
Internet Access/Intranet Web Access
IP Routing Protocol Support
Works with L2 Features
Intranet Web Servers

Enterprise
Headquarters
Internet Voice Gateway
Firewall
To PSTN

IP Video Server

G/ML-Series
IP Data IP -Data
Site 2 15454 Site 3

IP Voice OSPF, EIGRP, RIP, BGP IP Voice

15454 15454
IP Video IP Video
ML-Series ML-Series

© 2003, Cisco Systems, Inc. All rights reserved.

43
Summary
• Cisco’s Multiservice over SONET/SDH (MSOS)
Strategy enables profitability through:
– Support for Advanced Services
– Enhanced QoS
– Integrated Packet Multiplexing
– Data Optimized SONET/SDH Transport

• ML-Series for the ONS 15454 Multiservice

Provisioning Platform (MSPP) delivers private
line and switched Ethernet services
• Only Cisco Systems has the expertise in both
packets and circuits

© 2003, Cisco Systems, Inc. All rights reserved.

44
Presentation_ID © 2002, Cisco Systems, Inc. All rights reserved. 45