VERSION 1.

0 1

Software-Defined Networking:
A Comprehensive Survey
Diego Kreutz, Member, IEEE, Fernando M. V. Ramos, Member, IEEE, Paulo Verissimo, Fellow, IEEE,
Christian Esteve Rothenberg, Member, IEEE, Siamak Azodolmolky, Senior Member, IEEE,
and Steve Uhlig, Member, IEEE

Abstract—The Internet has led to the creation of a digital tems, network hypervisor, programming languages, flow-based
society, where (almost) everything is connected and is accessible network control, survey, scalability and dependability, software-
from anywhere. However, despite their widespread adoption, defined environments.
arXiv:1406.0440v1 [cs.NI] 2 Jun 2014

traditional IP networks are complex and very hard to manage.

It is both difficult to configure the network according to pre-
defined policies, and to reconfigure it to respond to faults, load
and changes. To make matters even more difficult, current I. I NTRODUCTION
networks are also vertically integrated: the control and data
planes are bundled together. Software-Defined Networking (SDN) The distributed control and transport network protocols run-
is an emerging paradigm that promises to change this state of ning inside the routers and switches are the key technologies
affairs, by breaking vertical integration, separating the network’s
control logic from the underlying routers and switches, promoting that allow information, in the form of digital packets, to
(logical) centralization of network control, and introducing the travel around the world. Despite their widespread adoption,
ability to program the network. The separation of concerns traditional IP networks are complex and hard to manage [1].
introduced between the definition of network policies, their To express the desired high-level network policies, network
implementation in switching hardware, and the forwarding of operators need to configure each individual network device
traffic, is key to the desired flexibility: by breaking the network
control problem into tractable pieces, SDN makes it easier to separately using low-level and often vendor-specific com-
create and introduce new abstractions in networking, simplifying mands. In addition to the configuration complexity, network
network management and facilitating network evolution. environments have to endure the dynamics of faults and
Today, SDN is both a hot research topic and a concept gaining adapt to load changes. Automatic reconfiguration and response
wide acceptance in industry, which justifies the comprehensive mechanisms are virtually non-existent in current IP networks.
survey presented in this paper. We start by introducing the Enforcing the required policies in such a dynamic environment
motivation for SDN, explain its main concepts and how it differs
from traditional networking. Next, we present the key building is therefore highly challenging.
blocks of an SDN infrastructure using a bottom-up, layered To make it even more complicated, current networks are
approach. We provide an in-depth analysis of the hardware also vertically integrated. The control plane (that decides how
infrastructure, southbound and northbounds APIs, network vir- to handle network traffic) and the data plane (that forwards
tualization layers, network operating systems (SDN controllers),
network programming languages, and management applications. traffic according to the decisions made by the control plane)
We also look at cross-layer problems such as debugging and are bundled inside the networking devices, reducing flexibility
troubleshooting. In an effort to anticipate the future evolution of and hindering innovation and evolution of the networking
this new paradigm, we discuss the main ongoing research efforts infrastructure. The transition from IPv4 to IPv6, started more
and challenges of SDN. In particular, we address the design of than a decade ago and still largely incomplete, bears witness
switches and control platforms – with a focus on aspects such as
resiliency, scalability, performance, security and dependability – to this challenge, while in fact IPv6 represented merely a
as well as new opportunities for carrier transport networks and protocol update. Due to the inertia of current IP networks,
cloud providers. Last but not least, we analyze the position of a new routing protocol can take 5 to 10 years to be fully
SDN as a key enabler of a software-defined environment. designed, evaluated and deployed. Likewise, a clean-slate
Index Terms—Software-defined networking, decoupled control approach to change the Internet architecture (e.g., replacing
and data plane, network virtualization, network operating sys- IP), is regarded as a tantalizing task – simply not feasible
in practice [2], [3]. Ultimately, this situation has inflated the
D. Kreutz, F. Ramos and P. Verı́ssimo are with the Department of Informat- capital and operational expenses of running an IP network.
ics of Faculty of Sciences, University of Lisbon, Lisbon 1749-016 Portugal Software-Defined Networking (SDN) [4], [5] is an emerging
e-mail: [email protected], [email protected], [email protected].
C. Esteve Rothenberg is with the School of Electrical and Com- networking paradigm that gives hope to change the lim-
puter Engineering (FEEC, University of Campinas, Brazil. e-mail: itations of current network infrastructures. First, it breaks
[email protected]. the vertical integration by separating the network’s control
S. Azodolmolky is with Gesellschaft für Wissenschaftliche Datenverar-
beitung mbH Göttingen (GWDG), Am Faßberg 11, 37077 Göttigen, Germany logic (the control plane) from the underlying routers and
e-mail: [email protected]. switches that forward the traffic (the data plane). Second,
S. Uhlig is with Queen Mary University of London. is with Queen Mary, with the separation of the control and data planes, network
University of London, Mile End Road, London E1 4NS, United Kingdom
e-mail [email protected]. switches become simple forwarding devices and the control
Manuscript received May 31, 2014. logic is implemented in a logically centralized controller (or
VERSION 1.0 2

Network  Applica4on(s)   Facebook, Yahoo, Microsoft, Verizon, and Deutsche Telekom

fund Open Networking Foundation (ONF) [10] with the main
Open northbound API
goal of promotion and adoption of SDN through open stan-
Controller  Pla+orm   dards development driven by the users (i.e., equipment buyers)
Open southbound API rather than the vendors (i.e., equipment manufacturers). As
the initial concerns with SDN scalability were addressed [11]
– in particular the myth that logical centralization implied
a physically centralized controller, an issue we will return
to later on – SDN ideas have matured and evolved from
nts
le me es) an academic exercise to a commercial success. Google, for
e itch
ding sw example, has deployed a software-defined network to inter-
r w
wa lo connect its data centers across the globe. This production
for nF
a ta Ope
D g., network has been in deployment for 3 years, helping the
(e.
company to improve operational efficiency and significantly
Network Infrastructure reduce costs [8]. VMware’s network virtualization platform,
NSX [12], is another example. NSX is a commercial so-
Fig. 1. Simplified view of an SDN architecture.
lution that delivers a fully functional network in software,
provisioned independent of the underlying networking devices,
network operating system1 ), simplifying policy enforcement entirely based around SDN principles. As a final example, the
and network (re)configuration and evolution [6]. A simplified world’s largest IT companies (from carriers and equipment
view of this architecture is shown in Figure 1. It is important manufacturers to cloud providers and financial-services com-
to emphasize that a logically centralized programmatic model panies) have recently joined SDN consortia such as the ONF
does not postulate a physically centralized system [7]. In fact, and the OpenDaylight initiative [13], another indication of the
the need to guarantee adequate levels of performance, scala- importance of SDN from an industrial perspective.
bility, and reliability would preclude such a solution. Instead, In this paper, we present a comprehensive literature survey
production-level SDN network designs resort to physically on SDN organized as depicted in Figure 2. We start, in the
distributed control planes [8], [7]. next two sections, by explaining the context, introducing the
The separation of the control plane and the data plane motivation for SDN and explaining the main concepts of this
can be realized by means of a well-defined programming new paradigm and how it differs from traditional networking.
interface between the switches and the SDN controller. The Our aim in the early part of the survey is also to explain
controller exercises direct control over the state in the data- that SDN is not as novel as a technological advance. Indeed,
plane elements via this well-defined application programming its existence is rooted at the intersection of a series of “old”
interface (API), as depicted in Figure 1. The most notable ideas, technology drivers, and current and future needs. The
example of such an API is OpenFlow [9], [10]. An OpenFlow concepts underlying SDN – the separation of the control
switch has one or more tables of packet-handling rules (flow and data planes, the flow abstraction upon which forwarding
table). Each rule matches a subset of the traffic and performs decisions are made, the (logical) centralization of network
certain actions (dropping, forwarding, modifying, etc.) on control, and the ability to program the network – are not
the traffic. Depending on the rules installed by a controller novel by themselves [14]. However, the integration of already
application, an OpenFlow switch can – instructed by the tested concepts with recent trends in networking – namely the
controller – behave like a router, switch, firewall, or perform availability of merchant switch silicon and the huge interest in
other roles (e.g., load balancer, traffic shaper, and in general feasible forms of network virtualization – are leading to this
those of a middlebox). paradigm shift in networking.
An important consequence of the software-defined net- Section IV comes next and is the core of this survey,
working principles is the separation of concerns introduced presenting an extensive and comprehensive analysis of the
between the definition of network policies, their implemen- building blocks of an SDN infrastructure using a bottom-
tation in switching hardware, and the forwarding of traffic. up, layered approach. The option for a layered approach is
This separation is key to the desired flexibility, breaking the grounded on the fact that SDN allows thinking of networking
network control problem into tractable pieces, and making it along two fundamental concepts, which are common in other
easier to create and introduce new abstractions in networking, disciplines of computer science: a) separation of concerns
simplifying network management and facilitating network (leveraging the concept of abstraction) and b) recursion. Our
evolution and innovation. layered, bottom-up approach divides the networking problem
Although SDN and OpenFlow started as academic exper- into eight parts: 1) hardware infrastructure, 2) southbound
iments [9], they gained significant traction in the industry interfaces, 3) network virtualization (hypervisor layer between
over the past few years. Most vendors of commercial switches the forwarding devices and the network operating systems), 4)
now include support of the OpenFlow API in their equipment. network operating systems (SDN controllers and control plat-
The SDN momentum was strong enough to make Google, forms), 5) northbound interfaces (to offer a common program-
ming abstraction to the upper layers, mainly the network ap-
1 We will use these two terms interchangeably. plications), 6) virtualization using slicing techniques provided
VERSION 1.0 3

systems, and control programs (network applications). Net-

work operators have to acquire and maintain different man-
agement solutions and the corresponding specialized teams.
The capital and operational cost of building and maintaining
a networking infrastructure is significant, with long return on
investment cycles, which hamper innovation and addition of
new features and services (for instance access control, load
balancing, energy efficiency, traffic engineering). To alleviate
the lack of in-path functionalities within the network, a myriad
of specialized components and middleboxes, such as firewalls,
intrusion detection systems and deep packet inspection en-
gines, proliferate in current networks. A recent survey of 57
Fig. 3. Layered view of networking functionality. enterprise networks shows that the number of middleboxes
is already on par with the number of routers in current
networks [19]. Despite helping in-path functionalities, the
by special purpose libraries and/or programming languages net effect of middleboxes has been increased complexity of
and compilers, 7) network programming languages, and finally network design and its operation.
8) management applications. In addition, we also look at
cross-layer problems such as debugging and troubleshooting III. W HAT IS S OFTWARE -D EFINED N ETWORKING ?
mechanisms. The discussion in Section V on ongoing research
efforts, challenges, future work and opportunities concludes The term SDN (Software-Defined Networking) was origi-
this paper. nally coined to represent the ideas and work around OpenFlow
at Stanford University [20]. As originally defined, SDN refers
II. S TATE OF Q UO IN N ETWORKING to a network architecture where the forwarding state in the data
Computer networks can be divided in three planes of plane is managed by a remote control plane decoupled from
functionality: the data, control and management planes (see the former. The networking industry has on many occasions
Figure 3). The data plane corresponds to the networking de- shifted from this original view of SDN, by referring to
vices, which are responsible for (efficiently) forwarding data. anything that involves software as being SDN. We therefore
The control plane represents the protocols used to populate the attempt, in this section, to provide a much less ambiguous
forwarding tables of the data plane elements. The management definition of software-defined networking.
plane includes the software services, such as SNMP-based We define an SDN as a network architecture with four
tools [15], used to remotely monitor and configure the control pillars:
functionality. Network policy is defined in the management
plane, the control plane enforces the policy, and the data plane 1) The control and data planes are decoupled. Control
executes it by forwarding data accordingly. functionality is removed from network devices that will
In traditional IP networks, the control and data planes are become simple (packet) forwarding elements.
tightly coupled, and embedded in the same networking de- 2) Forwarding decisions are flow-based, instead of destina-
vices, and the whole structure is highly decentralized. This was tion-based. A flow is broadly defined by a set of packet
considered important for the design of the Internet in the early field values acting as a match (filter) criterion and a set
days: it seemed the best way to guarantee network resilience, of actions (instructions). The flow abstraction allows uni-
which was a crucial design goal. In fact, this approach has fying the behavior of different types of network devices,
been quite effective in terms of network performance, with a including routers, switches, firewalls, and middleboxes.
rapid increase of line rate and port densities. Flow programming enables unprecedented flexibility,
However, the outcome is a very complex and relatively static limited only to the capabilities of the implemented flow
architecture. It is also the fundamental reason why traditional tables [9].
networks are rigid, and complex to manage and control. These 3) Control logic is moved to an external entity, the so-
two characteristics are largely responsible for a vertically- called SDN controller or Network Operating System
integrated industry where innovation is difficult. (NOS). The NOS is a software platform that runs on
Network misconfigurations and related errors are extremely commodity server technology and provides the essential
common in today’s networks. For instance, more than 1000 resources and abstractions to facilitate the programming
configuration errors have been observed in BGP routers [16]. of forwarding devices based on a logically centralized,
A single misconfigured device can be a big headache for net- abstract network view. Its purpose is therefore similar to
work operators and may have extremely severe consequences. that of a traditional operating system.
Indeed, while rare, a single misconfigured router is able to 4) The network is programmable through software appli-
compromise the correct operation of the whole Internet for cations running on top of the NOS that interacts with
hours [17], [18]. the underlying data plane devices. This is a fundamental
To support network management, a small number of vendors characteristic of SDN, considered as its main value
offer proprietary solutions of specialized hardware, operating proposition.
VERSION 1.0 4

Section V: Ongoing research efforts and challenges

SDN  for  
Controller   Resilience  and   Performance   Security  and   Migra$on     SDN  completes  
Switch  designs   telecom  and  
pla<orms   Scalability   evalua$on   dependability   to  SDN   the  SDE  puzzle  
cloud  

Section IV: Comprehensive survey: Bottom-up approach

Cross-­‐layer  issues  (debugging,  tes$ng  &  simula$on)  

Applica$ons  
Measurement  &   Security  &   Data  Center  
Traffic  engineering   Mobility    &  Wireless  
Monitoring   Dependability   Networking  

Programming  languages  

Language-­‐based  virtualiza$on  

Northbound  API  

Network  Opera$ng  System  (SDN  Controllers)  

Hypervisor-­‐based  virtualiza$on  

Southbound  interfaces  

Infrastructure  (data  plane  –  forwarding  devices)  

Sec$on  III:  What  is  SoFware-­‐Defined  Networking?  

Sec$on  II:  State  of  quo  in  “computer  networking”  and  mo$va$on  for  SDN  

Sec$on  I:  Introduc$on  

Fig. 2. Condensed overview of this survey on SDN.

Note that the logical centralization of the control logic, in requires a common distribution layer, which in SDN resides
particular, offers several additional benefits. First, it is simpler in the NOS. This layer has two essential functions. First,
and less error-prone to modify network policies through high- it is responsible for installing the control commands on the
level languages and software components, compared with low- forwarding devices. Second, it collects status information
level device specific configurations. Second, a control program about the forwarding layer (network devices and links), to offer
can automatically react to spurious changes of the network a global network view to network applications.
state and thus maintain the high-level policies intact. Third, the The last abstraction is specification, which should allow a
centralization of the control logic in a controller with global network application to express the desired network behavior
knowledge of the network state simplifies the development of without being responsible for implementing that behavior
more sophisticated networking functions, services and appli- itself. This can be achieved through virtualization solutions,
cations. as well as network programming languages. These approaches
Following the SDN concept [5], an SDN can be defined by map the abstract configurations that the applications express
three fundamental abstractions: (i) forwarding, (ii) distribution, based on a simplified, abstract model of the network, into a
and (iii) specification. In fact, abstractions are essential tools physical configuration for the global network view exposed
of research in computer science and information technology, by the SDN controller. Figure 4 depicts the SDN architecture,
being already an ubiquitous feature of many computer archi- concepts and building blocks.
tectures and systems [21]. As previously mentioned, the strong coupling between
Ideally, the forwarding abstraction should allow any for- control and data planes has made it difficult to add new
warding behavior desired by the network application (the con- functionality to traditional networks. The introduction of new
trol program) while hiding details of the underlying hardware. features requires the inclusion of expensive and hard-to-
OpenFlow is a practical realization of one such abstraction, configure equipment in the network – load balancers, intrusion
which can be seen as the equivalent to a “device driver” in an detection systems and firewalls are common examples. These
operating system. middleboxes need to be placed strategically in the network,
The distribution abstraction should shield SDN applications making it even harder to later change the network topology,
from the vagaries of distributed state, making the distributed configuration and functionality. This can be observed in Fig-
control problem a logically centralized one. Its realization ure 5. For instance, an intrusion detection system might need
VERSION 1.0 5

Net  App  1   Net  App  2   Net  App  n   re-using control plane software modules.
• These applications can take actions (i.e., reconfigure
Abstract network views forwarding devices) from any part of the network. There
Open northbound API is therefore no need to devise a precise strategy about the
Control plane

Network  Abstrac5ons  (e.g.,  topology  abstrac5on)  

location of the new functionality.
• The integration of different applications becomes more
Global network view
straightforward. For instance, load balancing and routing
Network  OS  (SDN  controllers)   applications can be combined sequentially, with load bal-
Open southbound API ancing decisions having precedence over routing policies.
Data Plane

A. Terminology
To identify the different elements of an SDN as unequiv-
ocally as possible, we now present the essential terminology
es
vic used throughout this work.
g De
in
ard Fowarding Devices (FD): Hardware- or software-based data
rw
Fo
plane devices that perform a set of elementary operations. The
Network Infrastructure forwarding devices have well-defined instruction sets (e.g.,
flow rules) used to take actions on the incoming packets
Fig. 4. SDN architecture and its fundamental abstractions.
(e.g., forward to specific ports, drop, forward to the controller,
rewrite some header). These instructions are defined by south-
Conventional Networking

bound interfaces (e.g., OpenFlow [9], ForCES [22], Protocol-

Oblivious Forwarding (POF) [23]) and are installed in the
forwarding devices by the SDN controllers implementing the
southbound protocols.
Data Plane (DP): Forwarding devices are interconnected
through wireless radio channels or wired cables. The net-
work infrastructure comprises the interconnected forwarding
devices, which represent the data plane.
Southbound Interface (SI): The instruction set of the forward-
Network  applica1ons   ing devices is defined by the southbound API, which is part
Software-Defined Networking

Intrusion  
of the southbound interface. Furthermore, the SI also defines
Rou1ng   Firewalling     Load  
Detec1on   (e.g.,  Packet  
balancer  
the communication protocol between forwarding devices and
Algorithms  
System   Filters)   control plane elements. This protocol formalizes the way the
SDN  controller  
control and data plane elements interact.
Control Plane (CP): Forwarding devices are programmed by
control plane elements through well-defined SI embodiments.
The control plane can therefore be seen as the “network brain”.
All control logic rests in the applications and controllers,
which form the control plane.
Northbound Interface (NI): The network operating system can
offer an API to application developers. This API represents a
northbound interface, i.e., a common interface for developing
Fig. 5. Traditional networks versus Software-Defined Networks (SDNs).
With SDN, management becomes simpler and traditional middleboxes can be applications. Typically, a northbound interface abstracts the
removed. low level instruction sets used by southbound interfaces to
program forwarding devices.
Management Plane (MP): The management plane is the set
to receive a cloned copy of the traffic of all switching devices of applications that leverage the functions offered by the
of the network through specific physical and or logical links. NI to implement network control and operation logic. This
In contrast, introducing new functionality in SDN is made includes applications such as routing, firewalls, load balancers,
simply by adding a new software application to run on top of monitoring, and so on. Essentially, a management applica-
the NOS. This approach has several advantages: tion defines the policies, which are ultimately translated to
southbound-specific instructions that program the behavior of
• It becomes easier to program these applications since the
the forwarding devices.
abstractions provided by the control platform and/or the
network programming languages can be shared.
• All applications can take advantage of the same network B. Alternative/Broadening SDN Definitions
information (the global network view), leading (arguably) Since its inception in 2010 [20], the original OpenFlow-
to more consistent and effective policy decisions while centered SDN term has seen its scope broadened beyond
VERSION 1.0 6

architectures with a cleanly decoupled control plane interface. we defined, the second and third columns of the table mention
The definition of SDN will likely continue to broaden, driven past initiatives (pre-SDN, i.e., before the OpenFlow-based
by the industry business-oriented views on SDN – irrespective initiatives that sprung into the SDN concept), and recent
of the decoupling of the control plane. In this survey, we developments that led to the definition of SDN.
focus on the original, “canonical” SDN definition based on Data plane programmability has a long history. Active
the aforementioned key pillars and the concept of layered ab- networks [35] represent one of the early attempts on building
stractions. However, for the sake of completeness and clarity, new network architectures based on this concept. The main
we acknowledge alternative SDN definitions [24], including: idea behind active networks is for each node to have the
Control Plane / Broker SDN: A networking approach that capability to perform computations on, or modify the content
retains existing distributed control planes but offers new APIs of, packets. To this end, active networks propose two distinct
that allow applications to interact (bidirectionally) with the approaches: programmable switches and capsules. The former
network. An SDN controller – often called orchestration does not imply changes in the existing packet or cell format.
platform – acts as a broker between the applications and the It assumes that switching devices support the downloading of
network elements. This approach effectively presents control programs with specific instructions on how to process packets.
plane data to the application and allows a certain degree of The second approach, on the other hand, suggests that packets
network programmability by means of “plug-ins” between the should be replaced by tiny programs, which are encapsulated
orchestrator function and network protocols. This API-driven in transmission frames and executed at each node along their
approach corresponds to a hybrid model of SDN, since it path.
enables the broker to manipulate and directly interact with ForCES [22], OpenFlow [9] and POF [23] represent recent
the control planes of devices such as routers and switches. approaches for designing and deploying programmable data
Examples of this view on SDN include recent IETF efforts plane devices. In a manner different from active networks,
(e.g., ALTO [25], I2RS [26], ABNO [27]) and the design these new proposals rely essentially on modifying forwarding
philosophy behind the OpenDaylight project [13] that goes devices to support flow tables, which can be dynamically
beyond the OpenFlow split control mode. configured by remote entities through simple operations such
Overlay SDN: A networking approach where the (software- or as adding, removing or updating flow rules, i.e., entries on the
hardware-based) network edge is dynamically programmed to flow tables.
manage tunnels between hypervisors and/or network switches, The earliest initiatives on separating data and control sig-
introducing an overlay network. In this hybrid networking nalling date back to the 80s and 90s. The network control point
approach, the distributed control plane providing the underlay (NCP) [36] is probably the first attempt to separate control
remains untouched. The centralized control plane provides and data plane signalling. NCPs were introduced by AT&T to
a logical overlay that utilizes the underlay as a transport improve the management and control of its telephone network.
network. This flavor of SDN follows a proactive model This change promoted a faster pace of innovation of the
to install the overlay tunnels. The overlay tunnels usually network and provided new means for improving its efficiency,
terminate inside virtual switches within hypervisors or in by taking advantage of the global view of the network provided
physical devices acting as gateways to the existing network. by NCPs. Similarly, other initiatives such as Tempest [46],
This approach is very popular in recent data center network ForCES [22], RCP [37], and PCE [48] proposed the separation
virtualization [28], and are based on a variety of tunneling of the control and data planes for improved management in
technologies (e.g., STT, NVGRE, VXLAN, LISP) [29]. ATM, Ethernet, BGP, and MPLS networks, respectively.
In addition, the term SDN is often used to define extensible More recently, initiatives such as SANE [51], Ethane [52],
network management planes (e.g., OpenStack [30]), whitebox OpenFlow [9], NOX [53] and POF [23] proposed the decou-
switches and open-source dataplanes (e.g., Pica8 Xorplus [31], pling of the control and data planes for Ethernet networks.
Quagga [32]), specialized programmable hardware devices Interestingly, these recent solutions do not require signifi-
(e.g., NetFPGA [33]), virtualized software-based appliances cant modifications on the forwarding devices, making them
(e.g., Network Function Virtualization - NFV [34]), in spite attractive not only for the networking research community,
of lacking a decoupled control and data plane or common but even more to the networking industry. OpenFlow-based
interface along its API. Hybrid SDN models will be further devices [9], for instance, can easily co-exist with traditional
discussed in Section V-G. Ethernet devices, enabling a progressive adoption (i.e., not
requiring a disruptive change to existing networks).
Network virtualization has gained a new traction with the
C. History of Software-Defined Networking advent of SDN. Nevertheless, network virtualization also has
Albeit a fairly recent concept, SDN leverages on network- its roots back in the 90s. The Tempest project [46] is one
ing ideas with a longer history [14]. In particular, it builds of the first initiatives to introduce network virtualization, by
on work made on programmable networks, such as active introducing the concept of switchlets in ATM networks. The
networks [35], and on proposals for control and data plane core idea was to allow multiple switchlets on top of a single
separation, such as NCP [36] and RCP [37]. ATM switch, enabling multiple independent ATM networks to
In order to present an historical perspective, we summarize share the same physical resources. Similarly, MBone [54] was
in Table I different instances of SDN-related work prior to one of the early initiatives that targeted the creation of virtual
SDN, splitting it into five categories. Along with the categories network topologies on top of legacy networks, or overlay
VERSION 1.0 7

TABLE I
S UMMARIZED OVERVIEW OF THE HISTORY OF PROGRAMABLE NETWORKS

Category Pre-SDN initiatives More recent SDN developments

Tennenhouse Wetherall [38], smart packets [39], ANTS [40], SwitchWare [41],
Data plane programmability ForCES [22], OpenFlow [9], POF [23]
Calvert [42], high performance router [43], NetScript [44], IEEE P1520 [45]

Control and data plane NCP [36], Tempest [46], ForCES [22], RCP [37], SoftRouter [47], PCE [48], SANE [51], Ethane [52], OpenFlow [9],
decoupling 4D [49], IRSCP [50] NOX [53], POF [23]

Tempest [46], MBone [54], 6Bone [55], RON [56], Planet Lab [57], Impasse [58], Open vSwitch [61], Mininet [62],
Network virtualization
GENI [59], VINI [60] FlowVisor [63], NVP [64]
Network operating systems Cisco IOS [65], JUNOS [66]. ExtremeXOS [67], SR OS [68] NOX [53], Onix [7], ONOS [69]
Technology pull initiatives Open Signaling [70] ONF [10]

networks. This work was followed by several other projects own specific functions. While some of them are always present
such as Planet Lab [57], GENI [59] and VINI [60]. It is also in an SDN deployment, such as the southbound API, network
worth mentioning FlowVisor [71] as one of the first recent ini- operating systems, northbound API and management applica-
tiatives to promote a hypervisor-like virtualization architecture tions, others may be present only in particular deployments,
for network infrastructures, resembling the hypervisor model such as hypervisor- or language-based virtualization.
common for compute and storage. More recently, Koponen et Figure 6 presents a tri-fold perspective of SDNs. The SDN
al. proposed a Network Virtualization Platform (NVP [64]) for layers are represented in the center (b) of the figure, as
multi-tenant datacenters using SDN as a base technology. explained above. Figures 6 (a) and 6 (c) depict a plane-
The concept of a network operating system was reborn oriented view and a system design perspective, respectively.
with the introduction of OpenFlow-based network operating The following sections introduce each layer, following a
systems, such as NOX [53], Onix [7] and ONOS [69]. Indeed, bottom-up approach. For each layer, the core properties and
network operating systems have been in existence for decades. concepts are explained based on the different technologies
One of the most widely known and deployed is the Cisco and solutions. Additionally, debugging and troubleshooting
IOS [65], which was originally conceived back in the early techniques and tools are discussed.
90s. Other network operating systems worth mentioning are
JUNOS [66], ExtremeXOS [67] and SR OS [68]. Despite A. Layer I: Infrastructure
being more specialized network operating systems, targeting
An SDN infrastructure, similarly to a traditional network, is
network devices such as high-performance core routers, these
composed of a set of networking equipment (switches, routers
NOSs abstract the underlying hardware to the network oper-
and middlebox appliances). The main difference resides in the
ator, making it easier to control the network infrastructure as
fact that those traditional physical devices are now simple
well as simplifying the development and deployment of new
forwarding elements without embedded control or software
protocols and management applications.
to take autonomous decisions. The network intelligence is
Finally, it is also worth recalling initiatives that can be seen
removed from the data plane devices to a logically-centralized
as “technology pull” drivers. Back in the 90s, a movement
control system, i.e., the network operating system and ap-
towards open signalling [70] started to happen. The main
plications, as shown in Figure 6 (c). More importantly,
motivation was to promote the wider adoption of the ideas
these new networks are built (conceptually) on top of open
proposed by projects such as NCP [36] and Tempest [46].
and standard interfaces (e.g., OpenFlow), a crucial approach
The open signalling movement worked towards separating
for ensuring configuration and communication compatibility
the control and data signalling, by proposing open and pro-
and interoperability among different data and control plane
grammable interfaces. Curiously, a rather similar movement
devices. In other words, these open interfaces enable controller
can be observed with the recent advent of OpenFlow and SDN,
entities to dynamically program heterogeneous forwarding
with the lead of the Open Networking Foundation (ONF) [10].
devices, something difficult in traditional networks, due to
This type of movement is crucial to promote open technologies
the large variety of proprietary and closed interfaces and the
into the market, hopefully leading equipment manufacturers
distributed nature of the control plane.
to support open standards and thus fostering interoperability, In an SDN/OpenFlow architecture, there are two main
competition, and innovation. elements, the controllers and the forwarding devices, as shown
For a more extensive intellectual history of programmable in Figure 7. A data plane device is a hardware or software
networks and SDN we forward the reader to the recent paper element specialized in packet forwarding, while a controller
by Feamster et al. [14]. is a software stack (the “network brain”) running on a com-
modity hardware platform. An OpenFlow-enabled forwarding
IV. S OFTWARE -D EFINED N ETWORKS : B OTTOM - UP device is based on a pipeline of flow tables where each entry
An SDN architecture can be depicted as a composition of of a flow table has three parts: (1) a matching rule, (2)
different layers, as shown in Figure 6 (b). Each layer has its actions to be executed on matching packets, and (3) counters
VERSION 1.0 8

Management plane Network  Applica8ons   Network  Applica8ons  

Debugging,  Tes8ng  &  Simula8on  

Programming  languages  

balancer  
Rou8ng  

Control  
Access  

Load  
Net  App   Net  App  
Net  App   Net  App  
Net  App   Net  App  
Language-­‐based  Virtualiza8on  

Control plane Northbound  Interface  

Network  Opera8ng  
Network  Opera8ng  System   System  and  
Hypervisors  
Network  Hypervisor  
Data plane
Southbound  Interface  
Network  infrastructure  

(a)   (b)   (c)  

Fig. 6. Software-Defined Networks in (a) planes, (b) layers, and (c) system design architecture

that keep statistics of matching packets. This high-level and of earlier protocol versions.
simplified model derived from OpenFlow is currently the most
widespread design of SDN data plane devices. Nevertheless, Overview of available OpenFlow devices
other specifications of SDN-enabled forwarding devices are Several OpenFlow-enabled forwarding devices are available
being pursued, including POF [23], [72] and the Negotiable on the market, both as commercial and open source products
Datapath Models (NDMs) from the ONF Forwarding Abstrac- (see Table III). There are many off-the-shelf, ready to deploy,
tions Working Group (FAWG) [73]. OpenFlow switches and routers, among other appliances. Most
Inside an OpenFlow device, a path through a sequence of of the switches available on the market have relatively small
flow tables defines how packets should be handled. When a Ternary Content-Addressable Memory (TCAMs), with up to
new packet arrives, the lookup process starts in the first table 8K entries. Nonetheless, this is changing at a fast pace.
and ends either with a match in one of the tables of the pipeline Some of the latest devices released in the market go far
or with a miss (when no rule is found for that packet). A flow beyond that figure. Gigabit Ethernet switches for common
rule can be defined by combining different matching fields, as business purposes are already supporting up to 32K L2+L3
illustrated in Figure 7. If there is no default rule, the packet or 64K L2/L3 exact match flows [74]. Enterprise class 10GE
will be discarded. However, the common case is to install switches are being delivered with more than 80K Layer 2 flow
a default rule which tells the switch to send the packet to entries [75]. Other switching devices using high performance
the controller (or to the normal non-OpenFlow pipeline of the chips such as the EZchip NP-4, provide optimized TCAM
switch). The priority of the rules follows the natural sequence memory that already supports from 125K up to 1000K flow
number of the tables and the row order in a flow table. Possible table entries [76]. This is a clear sign that the size of the flow
actions include (1) forward the packet to outgoing port(s), (2) tables is growing at a pace aiming to meet the needs of future
encapsulate it and forward it to the controller, (3) drop it, (4) SDN deployments.
send it to the normal processing pipeline, (5) send it to the Networking hardware manufacturers have produced various
next flow table or to special tables, such as group or metering kinds of OpenFlow-enabled devices, as is shown in Table III.
tables introduced in the latest OpenFlow protocol. These devices range from equipment for small businesses (e.g.,
As detailed in Table II, each version of the OpenFlow Gigabit Ethernet switches) to high-class data center equip-
specification introduced new match fields including Ethernet, ment (e.g., high-density switch chassis with up to 100GbE
IPv4/v6, MPLS, TCP/UDP, etc. However, only a subset of connectivity for edge-to-core applications, with tens of Tbps
those matching fields are mandatory to be compliant to a given of switching capacity).
protocol version. Similarly, many actions and port types are Software switches are emerging as one of the most promis-
optional features. Flow match rules can be based on almost ing solutions for data centers and virtualized network in-
arbitrary combinations of bits of the different packet headers frastructures [99], [100], [101]. Examples of software-based
using bit masks for each field. Adding new matching fields OpenFlow switch implementations include Switch Light [97],
has been eased with the extensibility capabilities introduced ofsoftswitch13 [93], Open vSwitch [94], OpenFlow Refer-
in OpenFlow version 1.2 through an OpenFlow Extensible ence [95], Pica8 [102], Pantou [98], and XorPlus [31]. Recent
Match (OXM) based on type-length-value (TLV) structures. reports show that the number of virtual access ports is already
To improve the overall protocol extensibility, with OpenFlow larger than physical access ports on data centers [101]. Net-
version 1.4 TLV structures have been also added to ports, ta- work virtualization has been one of the drivers behind this
bles, and queues in replacement of the hard-coded counterparts trend. Software switches such as Open vSwitch have been
VERSION 1.0 9

FLOW  TABLE  
SDN  CONTROLLER   RULE   ACTION   STATS  
Net  App   Net  App  
Net  App   Net  App  
Net  App   Net  App  
SDN  DEVICE   Packet  +  counters  

Communica5ons  

Communica5ons  
FLOW  TABLES   1.  Forward  packet  to  port(s)  
Network     2.  Encapsulate  and  forward  to  controller  

Control  

Control  
Opera5ng     3.  Drop  packet  
System   4.  Send  to  normal  processing  pipeline  

Switch   MAC   MAC   Eth   VLAN   IP     IP     TCP   TCP  

port   src   dst   type   ID   src   dst   psrc   pdst  

Fig. 7. OpenFlow-enabled SDN devices

TABLE II
D IFFERENT MATCH FIELDS , STATISTICS AND CAPABILITIES HAVE BEEN ADDED ON EACH O PEN F LOW PROTOCOL REVISION . T HE NUMBER OF REQUIRED
(R EQ ) AND OPTIONAL (O PT ) CAPABILITIES HAS GROWN CONSIDERABLY.

# Matches # Instructions # Actions # Ports

OpenFlow Version Match fields Statistics
Req Opt Req Opt Req Opt Req Opt
Ingress Port Per table statistics
Ethernet: src, dst, type, VLAN Per flow statistics
v 1.0 18 2 1 0 2 11 6 2
IPv4: src, dst, proto, ToS Per port statistics
TCP/UDP: src port, dst port Per queue statistics
Metadata, SCTP, VLAN tagging Group statistics
v 1.1 23 2 0 0 3 28 5 3
MPLS: label, traffic class Action bucket statistics
OpenFlow Extensible Match (OXM)
v 1.2 14 18 2 3 2 49 5 3
IPv6: src, dst, flow label, ICMPv6
Per-flow meter
v 1.3 PBB, IPv6 Extension Headers 14 26 2 4 2 56 5 3
Per-flow meter band
—
v 1.4 — 14 27 2 4 2 57 5 3
Optical port properties

used for moving network functions to the edge (with the core that can take up to nine months. The software development for
performing traditional IP forwarding), thus enabling network a new product can take from six months to one year [104]. The
virtualization [64]. initial investment is high and risky. As a central component
An interesting observation is the number of small, start- of its design the southbound APIs represent one of the
up enterprises devoted to SDN, such as Big Switch, Pica8, major barriers for the introduction and acceptance of any new
Cyan, Plexxi, and NoviFlow. This seems to imply that SDN is networking technology. In this light, the emergence of SDN
springing a more competitive and open networking market, one southbound API proposals such as OpenFlow [9] is seen as
of its original goals. Other effects of this openness triggered by welcome by many in the industry. These standards promote
SDN include the emergence of so-called “bare metal switches” interoperability, allowing the deployment of vendor-agnostic
or “whitebox switches”, where the software and hardware are network devices. This has already been demonstrated by the
sold separately and the end-user is free to load an operating interoperability between OpenFlow-enabled equipments from
system of its choice [103]. different vendors.
As of this writing, OpenFlow is the most widely accepted
B. Layer II: Southbound Interfaces and deployed open southbound standard for SDN. It provides
Southbound interfaces (or southbound APIs) are the con- a common specification to implement OpenFlow-enabled for-
necting bridges between control and forwarding elements, thus warding devices, and for the communication channel between
being the crucial instrument for clearly separating control and data and control plane devices (e.g., switches and controllers).
data plane functionality. However, these APIs are still tightly The OpenFlow protocol provides three information sources for
tied to the forwarding elements of the underlying physical or network operating systems. First, event-based messages are
virtual infrastructure. sent by forwarding devices to the controller when a link or
Typically, a new switch can take two years to be ready for port change is triggered. Second, flow statistics are generated
commercialization if built from scratch, with upgrade cycles by the forwarding devices and collected by the controller.
VERSION 1.0 10

TABLE III
O PEN F LOW ENABLED HARDWARE AND SOFTWARE DEVICES

Group Product Type Maker/Developer Version Short description

8200zl and 5400zl [77] chassis Hewlett-Packard v1.0 Data center class chassis (switch modules).
Arista 7150 Series [78] switch Arista Networks v1.0 Data centers hybrid Ethernet/OpenFlow switches.
BlackDiamond X8 [79] switch Extreme Networks v1.0 Cloud-scale hybrid Ethernet/OpenFlow switches.
CX600 Series [80] router Huawei v1.0 Carrier class MAN routers.
EX9200 Ethernet [81] chassis Juniper v1.0 Chassis based switches for cloud data centers.
EZchip NP-4 [82] chip EZchip Technologies v1.1 High performance 100-Gigabit network processors.
MLX Series [83] router Brocade v1.0 Service providers and enterprise class routers.
NoviSwitch 1248 [76] switch NoviFlow v1.3 High performance OpenFlow switch.
Hardware
NetFPGA [33] card NetFPGA v1.0 1G and 10G OpenFlow implementations.
RackSwitch G8264 [84] switch IBM v1.0 Data center switches supporting Virtual Fabric and OpenFlow.
PF5240 and PF5820 [85] switch NEC v1.0 Enterprise class hybrid Ethernet/OpenFlow switches.
Pica8 3920 [86] switch Pica8 v1.0 Hybrid Ethernet/OpenFlow switches.
Plexxi Switch 1 [87] switch Plexxi v1.0 Optical multiplexing interconnect for data centers.
V330 Series [88] switch Centec Networks v1.0 Hybrid Ethernet/OpenFlow switches.
Z-Series [89] switch Cyan v1.0 Family of packet-optical transport platforms.
contrail-vrouter [90] vrouter Juniper Networks v1.0 Data-plane function to interface with a VRF.
LINC [91], [92] switch FlowForwarding v1.3 Erlang-based soft switch with OF-Config 1.1 support.
ofsoftswitch13 [93] switch Ericsson, CPqD v1.3 OF 1.3 compatible user-space software switch implementation.
Open vSwitch [94], [61] switch Open Community v1.0 Switch platform designed for virtualized server environments.
Software OpenFlow Reference [95] switch Stanford v1.0 OF Switching capability to a Linux PC with multiple NICs.
OpenFlowClick [96] vrouter Yogesh Mundada v1.0 OpenFlow switching element for Click software routers.
Switch Light [97] switch Big Switch v1.0 Thin switching software platform for physical/virtual switches.
Pantou/OpenWRT [98] switch Stanford v1.0 Turns a wireless router into an OF-enabled switch.
XorPlus [31] switch Pica8 v1.0 Switching software for high performance ASICs.

Third, packet-in messages are sent by forwarding devices to One of the first direct competitors of OpenFlow is POF [23],
the controller when they do not known what to do with a [72]. One of the main goals of POF is to enhance the current
new incoming flow or because there is an explicit “send to SDN forwarding plane. With OpenFlow, switches have to
controller” action in the matched entry of the flow table. These understand the protocol headers to extract the required bits to
information channels are the essential means to provide flow- be matched with the flow tables entries. This parsing represents
level information to the network operating system. a significant burden for data plane devices, in particular if we
Albeit the most visible, OpenFlow is not the only available consider that OpenFlow version 1.3 already contains more than
southbound interface for SDN. There are other API proposals 40 header fields. Besides this inherent complexity, backward
such as ForCES [22], OVSDB [105], POF [23], [72], and compatibility issues may arise every time new header fields
OpFlex [106]. ForCES proposes a more flexible approach to are included in or removed from the protocol. To achieve its
traditional network management without changing the cur- goal, POF proposes a generic flow instruction set (FIS) that
rent architecture of the network, i.e., without the need of a makes the forwarding plane protocol-oblivious. A forwarding
logically-centralized external controller. The control and data element does not need to know, by itself, anything about the
planes are separated but can potentially be kept in the same packet format in advance. Forwarding devices are seen as
network element. However, the control part of the network white boxes with only processing and forwarding capabilities.
element can be upgraded on-the-fly with third-party firmware. In POF, packet parsing is a controller task that results in a
sequence of generic keys and table lookup instructions that
OVSDB [105] is another type of southbound API, de- are installed in the forwarding elements. The behavior of data
signed to provide advanced management capabilities for Open plane devices is therefore completely under the control of the
vSwitches. Beyond OpenFlow’s capabilities to configure the SDN controller. Similar to a CPU in a computer sytem, a POF
behavior of flows in a forwarding device, an Open vSwitch switch is application- and protocol-agnostic.
offers other networking functions. For instance, it allows the
control elements to create multiple virtual switch instances, set A very recent soutbound interface proposal is OpFlex [106].
QoS policies on interfaces, attach interfaces to the switches, Contrary to OpenFlow (and similar to ForCES), one of the
configure tunnel interfaces on OpenFlow data paths, manage ideas behind OpFlex is to distribute part of the complexity of
queues, and collect statistics. Therefore, the OVSDB is a managing the network back to the forwarding devices, with the
complementary protocol to OpenFlow for Open vSwitch. aim of improving scalability. Similar to OpenFlow, policies are
VERSION 1.0 11

logically centralized and abstracted from the underlying im- basis configuration, i.e., there is no single unifying abstraction
plementation. The differences between OpenFlow and OpFlex that can be leveraged to configure (or reconfigure) the network
are a clear illustration of one of the important questions to in a global manner. As a consequence, current network provi-
be answered when devising a southbound interface: where to sioning can take months, while computing provisioning takes
place each piece of the overall functionality. only minutes [64], [108], [109], [110].
There is hope that this situation will change with SDN
and the availability of new tunneling techniques (e.g.,
C. Layer III: Network Hypervisors
VXLAN [111], NVGRE [112]). For instance, solutions such
Virtualization is already a consolidated technology in mod- as FlowVisor [113], [63], [114], FlowN [115], NVP [64],
ern computers. The fast developments of the past decade have OpenVirteX [116] and IBM SDN VE [117], [118] have been
made virtualization of computing platforms mainstream. Based recently proposed, evaluated and deployed in real scenarios
on recent reports, the number of virtual servers has already for on-demand provisioning of virtual networks.
exceeded the number of physical servers [107], [64].
Hypervisors enable distinct virtual machines to share the Slicing the network
same hardware resources. In a cloud infrastructure-as-a-service FlowVisor is one of the early technologies to virtualize a
(IaaS), each user can have its own virtual resources, from Software-Defined Network. Its basic idea is to allow multiple
computing to storage. This enabled new revenue and business logical networks share the same OpenFlow networking infras-
models where users allocate resources on-demand, from a tructure. For this purpose, it provides an abstraction layer that
shared physical infrastructures, at a relatively low cost. At the makes it easier to slice a data plane based on off-the-shelf
same time, providers make better use of the capacity of their OpenFlow-enabled switches, allowing multiple and diverse
installed physical infrastructures, creating new revenue streams networks to co-exist.
without significantly increasing their CAPEX and OPEX costs. Five slicing dimensions are considered in FlowVisor: band-
One of the interesting features of virtualization technologies width, topology, traffic, device CPU and forwarding tables.
today is the fact that virtual machines can be easily migrated Moreover, each network slice supports a controller, i.e., mul-
from one physical server to another and can be created and/or tiple controllers can co-exist on top of the same physical
destroyed on-demand, enabling the provisioning of elastic network infrastructure. Each controller is allowed to act only
services with flexible and easy management. Unfortunately, on its own network slice. In general terms, a slice is defined
virtualization has been only partially realized in practice. De- as a particular set of flows on the data plane. From a system
spite the great advances in virtualizing computing and storage design perspective, FlowVisor is a transparent proxy that inter-
elements, the network is still mostly statically configured in a cepts OpenFlow messages between switches and controllers.
box-by-box manner [28]. It partitions the link bandwidth and flow tables of each switch.
The main network requirements can be captured along two Each slice receives a minimum data rate and each guest
dimensions: network topology and address space. Different controller gets its own virtual flow table in the switches.
workloads require different network topologies and services, Similarly to FlowVisor, OpenVirteX [116] acts as a proxy
such as flat L2 or L3 services, or even more complex L4- between the network operating system and the forwarding
L7 services for advanced functionality. Currently, it is very devices. However, its main goal is to provide virtual SDNs
difficult for a single physical topology to support the diverse through both topology, address, and control function virtu-
demands of applications and services. Similarly, address space alization. All these properties are necessary in multi-tenant
is hard to change in current networks. Nowadays, virtualized environments where virtual networks need to be managed
workloads have to operate in the same address of the physical and migrated according to the computing and storage virtual
infrastructure. Therefore, it is hard to keep the original network resources. Virtual network topologies have to be mapped
configuration for a tenant, virtual machines can not migrate to onto the underlying forwarding devices, with virtual addresses
arbitrary locations, and the addressing scheme is fixed and hard allowing tenants to completely manage their address space
to change. For example, IPv6 cannot be used by the VMs of without depending on the underlying network elements ad-
a tenant if the underlying physical forwarding devices support dressing schemes.
only IPv4. AutoSlice [119] is another SDN-based virtualization pro-
To provide complete virtualization the network should pro- posal. Similar to FlowVisor, the idea is to allow multiple
vide similar properties to the computing layer [28]. The net- controllers to manage their respective virtual SDN. The main
work infrastructure should be able to support arbitrary network difference is that AutoSlice intends to develop a transparent
topologies and addressing schemes. Each tenant should have virtualization layer, or SDN hypervisor, to automate the de-
the ability to configure both the computing nodes and the ployment of virtual SDNs in a less cumbersome manner than
network simultaneously. Host migration should automatically FlowVisor.
trigger the migration of the corresponding virtual network FlowN [115], [120] is based on a slightly different concept.
ports. One might think that long standing virtualization primi- Whereas FlowVisor can be compared to a full virtualization
tives such as VLANs (virtualized L2 domain), NAT (Virtu- technology, FlowN is analogous to a container-based virtu-
alized IP address space), and MPLS (virtualized path) are alization, i.e., a lightweight virtualization approach. FlowN
enough to provide full and automated network virtualization. was also primarily conceived to address multi-tenancy in the
However, these technologies are anchored on a box-by-box context of cloud platforms. It is designed to be scalable and
VERSION 1.0 12

allows a unique shared controller platform to be used for key role in future virtualized environments, similarly to the
managing multiple domains in a cloud environment. Each expansion we have been witnessing in virtualized computing.
tenant has full control over its virtual networks and is free
to deploy any network abstraction and application on top of
D. Layer IV: Network Operating Systems / Controllers
the controller platform.
Traditional operating systems provide abstractions (e.g.,
Commercial multi-tenant network hypervisors
high-level programming APIs) for accessing lower-level de-
None of the aforementioned approaches is designed to ad-
vices, manage the concurrent access to the underlying re-
dress all challenges of multi-tenant data centers. For instance,
sources (e.g., hard drive, network adapter, CPU, memory), and
tenants want to be able to migrate their enterprise solutions
provide security protection mechanisms. These functionalities
to cloud providers without the need to modify the network
and resources are key enablers for increased productivity, mak-
configuration of their home network. Existing networking
ing the life of system and application developers easier. Their
technologies and migration strategies have mostly failed to
widespread use has significantly contributed to the evolution
meet both the tenant and the service provider requirements.
of various ecosystems (e.g., programming languages) and the
A multi-tenant environment should be anchored in a network
development of a myriad of applications.
hypervisor capable of abstracting the underlaying forwarding
In contrast, networks have so far been managed and con-
devices and physical network topology from the tenants.
figured using lower level, device-specific instruction sets and
Moreover, each tenant should have access to control abstrac-
mostly closed proprietary network operating systems (e.g.,
tions and manage its own virtual networks independently and
Cisco IOS and Juniper JunOS). Moreover, the idea of op-
isolated from other tenants.
erating systems abstracting device-specific characteristics and
With the market demand for network virtualization and the
providing, in a transparent way, common functionalities is still
recent research on SDN showing promise as an enabling tech-
almost absent in networks. For instance, nowadays designers
nology, different commercial virtualization platforms based on
of routing protocols need to deal with complicated distributed
SDN concepts have started to appear. VMWare has proposed a
algorithms when solving networking problems. Network prac-
network virtualization platform (NVP) [64] that provides the
titioners have therefore been solving the same problems over
necessary abstractions to allow the creation of independent
and over again.
virtual networks for large-scale multi-tenant environments.
SDN is promised to facilitate network management and ease
NVP is a complete network virtualization solution that allows
the burden of solving networking problems by means of the
the creation of virtual networks, each with independent service
logically-centralized control offered by a network operating
model, topologies, and addressing architectures over the same
system (NOS) [53]. As with traditional operating systems,
physical network. With NVP, tenants do not need to know
the crucial value of a NOS is to provide abstractions, es-
anything about the underlying network topology, configuration
sential services, and common application programming inter-
or other specific aspects of the forwarding devices. NVP’s
faces (APIs) to developers. Generic functionality as network
network hypervisor translates the tenants configurations and
state and network topology information, device discovery,
requirements into low level instruction sets to be installed on
and distribution of network configuration can be provided as
the forwarding devices. For this purpose, the platform uses a
services of the NOS. With NOSs, to define network policies a
cluster of SDN controllers to manipulate the forwarding tables
developer no longer needs to care about the low-level details
of the Open vSwitches in the host’s hypervisor. Forwarding
of data distribution among routing elements, for instance.
decisions are therefore made exclusively on the network edge.
Such systems can arguably create a new environment capable
After the decision is made, the packet is tunneled over the
of fostering innovation at a faster pace by reducing the
physical network to the receiving host hypervisor (the physical
inherent complexity of creating new network protocols and
network sees nothing but ordinary IP packets).
management applications.
IBM has also recently proposed SDN VE [117], [118],
A NOS (or controller) is a critical element in an SDN
another commercial and enterprise-class network virtualization
architecture as it is the key supporting piece for the control
platform. SDN VE uses OpenDaylight as one of its build-
logic (applications) to generate the network configuration
ing blocks for Software-Defined Environments (SDEs)2 . This
based on the policies defined by the network operator. Similar
solutions also offers a complete implementation framework
to a traditional operating system, the control platform abstracts
for network virtualization. Like NVP, it uses a host-based
the lower-level details of connecting and interacting with
overlay approach, achieving advanced network abstraction that
forwarding devices (i.e., of materializing the network policies).
enables application-level network services in large-scale multi-
tenant environments. Interestingly, SDN VE 1.0 is capable Architecture and design axes
of supporting in one single instantiation up to 16,000 virtual
networks and 128,000 virtual machines [117], [118]. There are very diverse controllers and control platforms
To summarize, currently there are only a few network with different design and architectural choices [7], [13], [121],
hypervisor proposals leveraging the advances of SDN. We [122], [123], [124]. Existing controllers can be categorized
antecipate, however, this ecosystem to expand in the near based on many aspects. From an architectural point of view,
future since network virtualization will most likely play a one of the most relevant is if they are centralized or distributed.
This is one of the key design axes of SDN control platforms,
2 We will return to OpenDaylight and SDE later. so we start by discussing this aspect next.
VERSION 1.0 13

Centralized vs. Distributed more devices. Ultimately, a distributed controller can improve
A centralized controller is a single entity that manages all the control plane resilience, scalability and reduce the impact
forwarding devices of the network. Naturally, it represents a of problems caused by network partition, for instance. SDN
single point of failure and may have scaling limitations. A resiliency as a whole is an open challenge that will be further
single controller may not be enough to manage a network with discussed in Section V-C.
a large number of data plane elements. Centralized controllers
Dissecting SDN Controller Platforms
such as NOX-MT [125], Maestro [126], Beacon [127], and
Floodlight [128] have been designed as highly concurrent To provide a better architectural overview and understanding
systems, to achieve the throughput required by enterprise class the design a network operating system, Table IV summarizes
networks and data centers. These controllers are based on some of the most relevant architectural and design prop-
multi-threaded designs to explore the parallelism of multi-core erties of SDN controllers and control platforms. We have
computer architectures. As an example, Beacon can deal with focused on the elements, services and interfaces of a selection
more than 12 million flows per second by using large size com- of production-level, well-documented controllers and control
puting nodes of cloud providers such as Amazon [127]. Other platforms. Each line in the table represent a component we
centralized controllers such as Trema [129], Ryu NOS [130], consider important in a modular and scalable control platform.
Meridian [131], and ProgrammableFlow [132], [85] target We observe a highly diversified environment, with different
more or less specific environments such as data centers, cloud properties and components being used by distinct control
infrastructures, and carrier grade networks. platforms. This is not surprising, given an environment with
Contrary to a centralized design, a distributed network many competitors willing to be at the forefront of SDN
operating system can be scaled up to meet the requirements development. Note also that not all components are available
of potentially any environment, from small to large-scale on all platforms. For instance, east/westbound APIs are not
networks. A distributed controller can be a centralized cluster required in centralized controllers such as Beacon. In fact,
of nodes or a physically distributed set of elements. While some platforms have very specific niche markets, such as
the first alternative can offer high throughput for very dense telecom companies and cloud providers, so the requirements
data centers, the latter can be more resilient to different kinds will be different.
of logical and physical failures. A cloud provider that spans Based on the analysis of the different SDN controllers
multiple data centers interconnected by a wide area network proposed to date (both those presented in Table IV and
may require a hybrid approach, with clusters of controllers others, such as NOX [53], Meridian [131], ForCES [22], and
inside each data center and distributed controller nodes in the FortNOX [135]), we extract several common elements and
different sites [8]. provide a first attempt to clearly and systematically dissect
Onix [7], HyperFlow [133], HP VAN SDN [122], an SDN control platform in Figure 8.
ONOS [69], DISCO [123], and yanc [134] are examples There are at least three relatively well-defined layers in most
of distributed controllers. Most distributed controllers offer of the existing control platforms: (i) the application, orches-
weak consistency semantics, which means that data updates tration and services; (ii) the core controller functions, and (iii)
on distinct nodes will eventually be updated on all controller the elements for southbound communications. The connection
nodes. This implies that there is a period of time in which at the upper-level layers is based on northbound interfaces
distinct nodes may read different values (old value or new such as REST APIs [136] and programming languages such
value) for a same property. Strong consistency, on the other as FML [137], Frenetic [138] and NetCore [139]. On the
hand, ensures that all controller nodes will read the most lower-level part of a control platform, southbound APIs and
updated property value after a write operation. Despite its protocol plugins interface the forwarding elements. The core
impact on system performance, strong consistency offers a of a controller platform can be characterized as a combination
simpler interface to application developers. To date, only Onix its base network service functions and the various interfaces.
and ONOS provide different data consistency models (both
weak and strong). Core controller functions
Another common property of distributed controllers is fault The base network service functions are what we consider
tolerance. When one node fails, another neighbor node should the essential functionality all controllers should provide. As
take over the duties and devices of the failed node. So far, an analogy, these functions are like base services of op-
despite some controllers tolerating crash failures, they do not erating systems, such as program execution, I/O operations
tolerate arbitrary failures, which means that any node with an control, communications, protection, and so on. These services
abnormal behavior will not be replaced by a potentially well are used by other operating system level services and user
behaved one. applications. In a similar way, functions such as topology,
A single controller may be enough to manage a small statistics, notifications and device management, together with
network, however it represents a single point of failure. shortest path forwarding and security mechanisms are essential
Similarly, independent controllers can be spread across the network control functionalities that network applications may
network, each of them managing a network segment, reducing use in building its logic. For instance, the notification manager
the impact of a single controller failure. Yet, if the control should be able to receive, process, and forward events (e.g.,
plane availability is critical, a cluster of controllers can be used alarm notifications, security alarms, state changes) [140].
to achieve a higher degree of availability and/or for supporting Security mechanisms are another example, as they are critical
VERSION 1.0 14

TABLE IV
A RCHITECTURE AND DESIGN ELEMENTS OF CONTROL PLATFORMS

Component OpenDaylight OpenContrail HP VAN SDN Onix Beacon

Base network Topology/Stats/Switch Routing, Tenant Isola- Audit Log, Alerts, Discovery, Multi- Topology, device man-
services Manager, Host tion Topology, Discovery consistency Storage, ager, and routing
Tracker, Shortest Read State, Register
Path Forwarding for updates
East/Westbound — Control Node (XMPP- Sync API Distribution I/O module Not present
APIs like control channel)
Integration OpenStack Neutron CloudStack, OpenStack OpenStack — —
Plug-ins
Management GUI/CLI, REST API GUI/CLI REST API Shell / GUI — Web
Interfaces Shell
Northbound REST, RESTCONF, REST APIs (configu- REST API, GUI Shell Onix API (general pur- API (based on Open-
APIs Java APIs ration, operational, and pose) Flow events)
analytic)
Service Service Abstraction — Device Abstraction API Network Information —
abstraction Layer (SAL) Base (NIB) Graph
layers with Import/Export
Functions
Southbound OpenFlow, OVSDB, — OpenFlow, L3 Agent, OpenFlow, OVSDB OpenFlow
APIs or SNMP, PCEP, BGP, L2 Agent
connectors NETCONF

Routing Load Security ! Network Network Attack

Management Applications!
Protocols! Balancers! ACLs! Virtualization! Monitoring! Detection!

REST! Programming Languages! Northbound Interfaces!

Shortest Path Forwarding! Notification Manager! Security Mechanisms!

East/Westbound !
Mechanisms & Controller Platform!
Protocols!
Topology Manager! Stats Manager! Device Manager!

East/Westbound 

Southbound Abstraction Layer! Common Interfaces!
Abstraction Layer!

ForCES Southbound Interfaces!

SDNi! OpenFlow! OVSDB! ForCES! POF!
CE-CE!

Hardware-based Forwarding Software-based Forwarding

Data Plane Elements !
Devices! Devices!

Fig. 8. SDN control platforms: elements, services and interfaces

components to provide basic isolation and security enforce- of physical devices, virtual devices (e.g., Open vSwitch [94],
ment between services and applications. For instance, rules [61], vRouter [141]) and a variety of device interfaces (e.g.,
generated by high priority services should not be overwritten OpenFlow, OVSDB, of-config [142], NetConf, and SNMP)
with rules created by applications with a lower priority. can co-exist.
Most controllers support only OpenFlow as a southbound
Southbound API. Still, a few of them, such as OpenDaylight, Onix and HP
On the lower-level of control platforms, the southbound VAN SDN Controller, offer a wider range of southbound APIs
APIs can be seen as a layer of device drivers. They provide and/or protocol plugins. Onix supports both the OpenFlow
a common interface for the upper layers, while allowing and OVSDB protocols. The HP VAN SDN Controller has
a control platform to use different southbound APIs (e.g., other southbound connectors such as L2 and L3 agents.
OpenFlow, OVSDB, ForCES) and protocol plugins to manage OpenDaylight goes a step beyond by providing a Service
existing or new physical or virtual devices (e.g., SNMP, BGP, Layer Abstraction (SLA) that allows several southbound APIs
NetConf). This is essential both for backward compatibility and protocols to co-exist in the control platform. For instance,
and heterogeneity, i.e., to allow multiple protocols and device its original architecture was designed to support at least seven
management connectors. Therefore, on the data plane, a mix different protocols and plugins: OpenFlow, OVSDB [105],
VERSION 1.0 15

NETCONF [143], PCEP [48], SNMP [144], BGP [145] and bound APIs, such as ad-hoc APIs, RESTful APIs [136],
LISP Flow Mapping [13]. Hence, OpenDaylight is one of the multi-level programming interfaces, file systems, among other
few control platforms being conceived to support a broader more specialized APIs such as NVP NBAPI [7], [64] and
integration of technologies in a single control platform. SDMN API [154]. Section IV-E is devoted to a more detailed
discussion on the evolving layer of northbound APIs. A
Eastbound and Westbound
second kind of northbound interfaces are those stemming
East/westbound APIs, as illustrated in Figure 9, are a special
out of SDN programming languages such as Frenetic [138],
case of interfaces required by distributed controllers. Currently,
Nettle [155], NetCore [139], Procera [156], and Pyretic [157].
each controller implements its own east/westbound API. The
Section IV-G gives a more detailed overview of the several
functions of these interfaces include import/export data be-
existing programming languages for SDN.
tween controllers, algorithms for data consistency models, and
monitoring/notification capabilities (e.g., check if a controller Wrapping up remarks and platforms comparison
is up or notify a take over on a set of forwarding devices). Table V shows a summary of some of the existing con-
trollers with their respective architectures and characteristics.
SDN  Controller  Node   SDN  Controller  Node   As can be observed, most controllers are centralized and multi-
Onix   Westbound/   Trema   threaded. Curiously, the northbound API is very diverse. In
ONOS   Eastbound  APIs   ODL  
yanc   Floodlight   particular, five controllers (Onix, Floodlight, MuL, Meridian
and SDN Unified Controller) pay a bit more attention to this
interface, as a statement of its importance. Consistency models
and fault tolerance are only present in Onix, HyperFlow, HP
VAN SDN, and ONOS. Lastly, when it comes to the OpenFlow
0.75 standard as southbound API, only Ryu supports its three major
versions (v1.0, v1.2 and v1.3).
Fig. 9. Distributed controllers: east/westbound APIs. To conclude, it is important to emphasize that the control
platform is one of the critical points for the success of
Similarly to southbound and northbound interfaces, SDN [163]. One of the main issues that needs to be address
east/westbound APIs are essential components of distributed in this respect is interoperability. This is rather interesting,
controllers. To identify and provide common compatibility as it was the very first problem that southbound APIs (such
and interoperability between different controllers, it is neces- as OpenFlow) tried to solve. For instance, while WiFi and
sary to have standard east/westbound interfaces. For instance, LTE networks [164] need specialized control platforms such
SDNi [146] defines common requirements to coordinate flow as MobileFlow [154] or SoftRAN [165], data center networks
setup and exchange reachability information across multiple have different requirements that can be met with platforms
domains. In essence, such protocols can be used in an or- such as Onix [7] or OpenDaylight [13]. For this reason, in
chestrated and interoperable way to create more scalable and environments where diversity of networking infrastructures is
dependable distributed control platforms. Interoperability can a reality, coordination and cooperation between different con-
be leveraged to increase the diversity of the control platform trollers is crucial. Standardized APIs for multi-controller and
element. Indeed, diversity increases the system robustness by multi-domain deployments are therefore seen as an important
reducing the probability of common faults, such as software step to achieve this goal.
faults [147].
Other proposals that try to define interfaces between con- E. Layer V: Northbound Interfaces
trollers include Onix data import/export functions [7], ForCES
The North- and Southbound interfaces are two key ab-
CE-CE interface [22], [148], ForCES Intra-NE cold-standby
stractions of the SDN ecosystem. The southbound interface
mechanisms for high availability [149], and distributed data
has already a widely accepted proposal (OpenFlow), but a
stores [150]. An east/westbound API requires advanced data
common northbound interface is still an open issue. At this
distribution mechanisms such as the Advanced Message Queu-
moment it may still be a bit too early to define a standard
ing Protocol (AMQP) [151] used by DISCO [123], transac-
northbound interface, as use-cases are still being worked
tional databases and DHTs [152] ( as used in Onix [7]), or
out [166]. Anyway, it is to be expected a common (or
advanced algorithms for strong consistency and fault toler-
a de facto) northbound interface to arise as SDN evolves.
ance [150].
An abstraction that would allow network applications not to
In a multi-domain setup, east/westbound APIs may require
depend on specific implementations is important to explore the
also more specific communication protocols between SDN
full potential of SDN
domain controllers [153]. Some of the essential functions of
The northbound interface is mostly a software ecosystem,
such protocols are to coordinate flow setup originated by
not a hardware one as is the case of the southbound APIs.
applications, exchange reachability information to facilitate
In these ecosystems, the implementation is commonly the
inter-SDN routing, reachability update to keep the network
forefront driver, while standards emerge later and are essen-
state consistent, among others.
tially driven by wide adoption [167]. Nevertheless, an initial
Northbound and minimal standard for northbound interfaces can still play
Current controllers offer a quite broad variety of north- an important role for the future of SDN. Discussions about
VERSION 1.0 16

TABLE V
C ONTROLLERS CLASSIFICATION

Name Architecture Northbound API Consistency Faults License Prog. language Version
Beacon [127] centralized multi-threaded ad-hoc API no no GPLv2 Java v1.0
DISCO [123] distributed REST — yes — Java v1.1
Floodlight [128] centralized multi-threaded RESTful API no no Apache Java v1.1
HP VAN SDN [122] distributed RESTful API weak yes — Java v1.0
HyperFlow [133] distributed — weak yes — C++ v1.0
Kandoo [158] hierarchically distributed — no no — C, C++, Python v1.0
Onix [7] distributed NVP NBAPI weak, strong yes commercial Python, C v1.0
Maestro [126] centralized multi-threaded ad-hoc API no no LGPLv2.1 Java v1.0
Meridian [131] centralized multi-threaded extensible API layer no no — Java v1.0
MobileFlow [154] — SDMN API — — — — v1.2
MuL [159] centralized multi-threaded multi-level interface no no GPLv2 C v1.0
NOX [53] centralized ad-hoc API no no GPLv3 C++ v1.0
NOX-MT [125] centralized multi-threaded ad-hoc API no no GPLv3 C++ v1.0
NVP Controller [64] distributed — — — commercial — —
OpenContrail [121] — REST API no no Apache 2.0 Python, C++, Java v1.0
OpenDaylight [13] distributed REST, RESTCONF weak no EPL v1.0 Java v1.{0,3}
ONOS [69] distributed RESTful API weak, strong yes — Java v1.0
POX [160] centralized ad-hoc API no no GPLv3 Python v1.0
ProgrammableFlow [161] centralized — — — — C v1.3
Ryu NOS [130] centralized multi-threaded ad-hoc API no no Apache 2.0 Python v1.{0,2,3}
SNAC [162] centralized ad-hoc API no no GPL C++ v1.0
Trema [129] centralized multi-threaded ad-hoc API no no GPLv2 C, Ruby v1.0
Unified Controller [117] — REST API — — commercial — v1.0
yanc [134] distributed file system — — — — —

this issue have already begun [166], [167], [168], [169], also abstract the inner details of the controller functions and
[170], [171], [172], [173], and a common consensus is that data plane behavior from the application developers. Moreover,
northbound APIs are indeed important but that it is indeed too as we explain in Section IV-G, programming languages can
early to define a single standard right now. The experience provide a wide range of powerful abstractions and mechanisms
from the development of different controllers will certainly such as application composition, transparent data plane fault
be the basis for coming up with a common application level tolerance, and a variety of basic building blocks to ease
interface. software module and application development.
Open and standard northbound interfaces are crucial to SFNet [177] is another example of a northbound interface.
promote application portability and interoperability among It is a high-level API that translates application requirements
the different the control platforms. A northbound API can into lower level service requests. However, SFNet has a limited
be compared to the POSIX standard in operating systems, scope, targeting queries to request the congestion state of
representing an abstraction that guarantees programming lan- the network and services such as bandwidth reservation and
guage and controller independence. NOSIX [174] is one of the multicast.
first examples of an effort in this direction. It tries to define Other proposals use different approaches to allow appli-
portable low-level (e.g., flow model) application interfaces, cations to interact with controllers. The yanc control plat-
making southbound APIs such as OpenFlow look like “device form [134] explores this idea by proposing a general control
drivers”. However, NOSIX is not exactly a general purpose platform based on Linux and abstractions such as the virtual
northbound interface, but rather a higher-level abstraction for file system (VFS). This approach simplifies the development of
southbound interfaces. Indeed, it could be part of the common SDN applications as programmers are able to use a traditional
abstraction layer in a control platform as the one described in concept (files) to communicate with lower level devices and
Section IV-D. sub-systems.
Existing controllers such as Floodlight, Trema, NOX, Onix, Eventually, it is unlikely that a single northbound interface
and OpenDaylight propose and define their own northbound emerges as the winner, as the requirements for different
APIs [168], [175]. However, each of them has its own specific network applications are quite different. APIs for security
definitions. Programming languages such as Frenetic [138], applications are likely to be different from those for routing
Nettle [155], NetCore [139], Procera [156] and Pyretic [176] or financial applications. One possible path of evolution for
VERSION 1.0 17

northbound APIs are vertically-oriented proposals, before any libNetVirt is a library designed to provide a flexible way to
type of standardization occurs, a challenge the ONF has create and manage virtual networks in different computing
started to undertake in addition to open-source SDN controller environments. Its main idea is similar to the OpenStack Quan-
platform developments (e.g., OpenDaylight, Floodlight, Open- tum project [180]. While Quantum is designed for OpenStack
Stack). (cloud environments), libNetVirt is a more general purpose
library which can be used in different environments. Addition-
ally, it goes one step beyond OpenStack Quantum by enabling
F. Layer VI: Language-based Virtualization
QoS capabilities in virtual networks [179]. The libNetVirt
Two essential characteristics of virtualization solutions are library has two layers: (1) a generic network interface; and (2)
the capability of expressing modularity and of allowing dif- technology specific device drivers (e.g., VPN, MPLS, Open-
ferent levels of abstractions while still guaranteeing desired Flow). On top of the layers are the management applications
properties such as protection. For instance, virtualization tech- and virtual network descriptions. The OpenFlow driver uses a
niques can allow different views of a single physical infrastruc- NOX controller to manage the underlying infrastructure, using
ture. As an example, one virtual “big switch” could represent OpenFlow rule-based flow tables to create isolated virtual
a combination of several underlying forwarding devices. This networks. By supporting different technologies, it can be used
intrinsically simplifies the task of application developers as as a bridging component in heterogeneous networks.
they do not need to think about the sequence of switches Table VI summarizes the hypervisor and non-hypervisor
where forwarding rules have to be installed, but rather see based virtualization technologies. As can be observed, only
the network as a simple “big switch”. Such kind of abstrac- libNetVirt supports heterogeneous technologies, not restricting
tion significantly simplify the development and deployment its application to OpenFlow-enabled networks. FlowVisor,
of complex network applications, such as advanced security AutoSlice and OpenVirteX allow multiple controllers, one per
related services. network slice. FlowN provides a container-based approach
Pyretic [176] is an interesting example of a programming where multiple applications from different users can co-exist
language that offers this type of high-level abstraction of on a single controller. FlowVisor allows QoS provisioning
network topology. It incorporates this concept of abstraction guarantees by using VLAN PCP bits for priority queues. SDN
by introducing network objects. These objects consist of an VE and NVP also provide their own provisioning methods for
abstract network topology and the sets of policies applied to guaranteeing QoS.
it. Network objects simultaneously hide information and offer
the required services. G. Layer VII: Programming languages
Another form of language-based virtualization is static Programming languages have been proliferating for decades.
slicing. This a scheme where the network is sliced by a Both academia and industry have evolved from low-level
compiler, based on application layer definitions. The output hardware-specific machine languages, such as assembly for
of the compiler is a monolithic control program that has x86 architectures, to high-level and powerful programming
already slicing definitions and configuration commands for the languages such as Java and Python. The advancements towards
network. In such a case, there is no need for a hypervisor more portable and reusable code has driven a significant shift
to dynamically manage the network slices. Static slicing can on the computer industry [181], [182].
be valuable for deployments with specific requirements, in Similarly, programmability in networks is starting to move
particular those where higher performance and simple isolation from low level machine languages such as OpenFlow (“assem-
guarantees are preferrable to dynamic slicing. bly”) to high-level programming languages [138], [137], [155],
One example of static slicing approach it the Splendid [139], [156], [157], [64]. Assembly-like machine languages,
isolation [178]. In this solution the network slices are made such as OpenFlow [9] and POF [23], [72], essentially mimic
of 3 components: (a) topology, consisting of switches, ports, the behavior of forwarding devices, forcing developers to
and links; (b) mapping of slice-level switches, ports and spend too much time on low-level details rather than on
links on the network infrastructure; (c) predicates on packets, the problem solve. Raw OpenFlow programs have to deal
where each port of the slice’s edge switches has an associated with hardware behavior details such as overlapping rules, the
predicate. The topology is a simple graph of the sliced nodes, priority ordering of rules, and data-plane inconsistencies that
ports and links. Mapping will translate the abstract topology arise from in-flight packets whose flow rules are under installa-
elements into the corresponding physical ones. The predicates tion [138], [139], [183]. The use of these low-level languages
are used to indicate whether a packet is permitted or not to makes it difficult to reuse software, to create modular and
enter a specific slice. Different applications can be associated extensive code, and leads to a more error-prone development
to each slice. The compiler takes the combination of slices process [157], [184], [185].
(topology, mapping, and predicates) and respective programs Abstractions provided by high level programming languages
to generate a global configuration for the entire network. It also can significantly help address many of the challenges of these
ensures that properties such as isolation are enforced among lower-level instruction sets [138], [137], [155], [139], [156],
slices, i.e., no packets of a slice A can traverse to a slice B [157]. In SDNs, high-level programming languages can be
unless explicitly allowed. designed and used to:
Other solutions, such as libNetVirt [179], try to integrate 1) create higher level abstractions for simplifying the task
heterogeneous technologies for creating static network slices. of programming forwarding devices;
VERSION 1.0 18

TABLE VI
V IRTUALIZATION SOLUTIONS

Solution Multiple controllers Slicing QoS “guarantees” Multi-technology

AutoSlice [119] yes, one per slice VLAN tags no no, OF only
FlowVisor [113], [114] yes, one per slice virtual flow tables per slice yes (VLAN PCP bits) no, OF only
FlowN [115], [120] no (contained applications) VLAN tags unknown no, OF only
IBM SDN VE [117] yes, a cluster of controllers logical datapaths yes (priority-based) yes (VXLAN, OVS, OpenFlow)
libNetVirt [179] no, one single controller VLAN tags no yes (e.g., VPN, MPLS, OpenFlow)
NVP’s Hypervisor [64] yes, a cluster of controller logical datapaths yes no, OVS only
OpenVirteX [116] yes, one per slice virtual flow tables per slice unknown no, OF only
Pyretic [176] no, one single controller compiler time OF rules no no, OF only
Splendid Isolation [178] no, one single controller compiler time VLANs no no, OF only

2) enable more productive and problem-focused environ- installing the lower-level instructions required at each forward-
ments for network software programmers, speeding up ing device to enforce the user policy across the network. With
development and innovation; such kind of abstractions, developing a routing application
3) promote software modularization and code reusability in becomes a straightforward process. Similarly, a single physical
the network control plane; switch could be represented as a set of virtual switches, each
4) foster the development of network virtualization. of them belonging to a different virtual network. These two
Several challenges can be better addressed by programming examples of abstract network topologies would be much harder
languages in SDNs. For instance, in pure OpenFlow-based to implement with low-level instruction sets. In contrast, a
SDNs, it is hard to ensure that multiple tasks of a single programming language or runtime system can more easily
application (e.g., routing, monitoring, access control) do not provide abstractions for virtual network topologies, as has
interfere with each other. For example, rules generated for one already been demonstrated by languages such as Pyretic [176].
task should not override the functionality of another task [138], High-level SDN programming languages
[183]. Another example is when multiple applications run on Low-level instruction sets suffer from several problems. To
a single controller [157], [183], [135], [186], [187]. Typically, address some of these challenges, higher-level programming
each application generates rules based on its own needs and languages have been proposed, with diverse goals, such as:
policies without further knowledge about the rules generated
by other applications. As a consequence, conflicting rules • Avoiding low-level and device-specific configurations and
can be generated and installed in forwarding devices, which dependencies spread across the network, as happens in
can create problems for network operation. Programming lan- traditional network configuration approaches;
guages and runtime systems can help to solve these problems • Providing abstractions that allow different management
that would be otherwise hard to prevent. tasks to be accomplished through easy to understand and
Important software design techniques such as code mod- maintain network policies;
ularity and reusability are very hard to achieve using low- • Decoupling of multiple tasks (e.g., routing, access con-
level programming models [157]. Applications thus built are trol, traffic engineering);
monolithic and consist of building blocks that can not be • Implementing higher-level programming interfaces to
reused in other applications. The end result is a very time avoid low-level instruction sets;
consuming and error prone development process. • Solving forwarding rules problems, e.g., conflicting or
Another interesting feature that programming language ab- incomplete rules that can prevent a switch event to be
stractions provide is the capability of creating and writing triggered, in an automated way;
programs for virtual network topologies [176], [178]. This • Addressing different race condition issues which are
concept is similar to object-oriented programming, where inherent to distributed systems;
objects abstract both data and specific functions for application • Enhancing conflict-resolution techniques on environ-
developers, making it easier to focus on solving a particular ments with distributed decision makers;
problem without worrying about data structures and their • Provide native fault-tolerance capabilities on data plane
management. For instance, in an SDN context, instead of path setup;
generating and installing rules in each forwarding device, one • Reducing the latency in the processing of new flows;
can think of creating simplified virtual network topologies that • Easing the creation of stateful applications (e.g., stateful
represent the entire network, or a subset of it. For example, the firewall).
application developer should be able to abstract the network Programming languages can also provide specialized ab-
as an atomic big switch, rather than a combination of several stractions to cope with other management requirements, such
underlying physical devices. The programming languages or as monitoring [156], [138], [188]. For instance, the runtime
runtime systems should be responsible for generating and system of a programming language can do all the “laundry
VERSION 1.0 19

work” of installing rules, polling the counters, receiving the runtime system.
responses, combining the results as needed, and composing Advanced operators for parallel and sequential composition
monitoring queries in conjunction with other policies. Con- help bind (through internal workflow operators) the key char-
sequently, application developers can take advantage of the acteristics of programming languages such as Pyretic [157].
simplicity and power of higher level query instructions to Parallel composition makes it possible to operate multiple poli-
easily implement monitoring modules or applications. cies on the same set of packets, while sequential composition
Another aspect of paramount importance is the portability facilitates the definition of a sequential workflow of policies to
of the programming language, necessary so that developers be processed on a set of packets. Sequential policy processing
do not need to re-implement applications for different control allows multiple modules (e.g., access control and routing) to
platforms. The portability of a programming language can operate in a cooperative way. By using sequential composition
be considered as a significant added value to the control complex applications can be built out of a combination of
plane ecosystem. Mechanisms such as decoupled back-ends different modules (in a similar way as pipes can be used to
could be key architectural ingredients to enable platform build sophisticated Unix applications).
portability. Similarly to the Java virtual machine, a portable Further advanced features are provided by other SDN
northbound interface will easily allow applications to run on programming languages. FatTire [189] is an example of a
different controllers without requiring any modification. As an declarative language that heavily relies on regular expressions
example, the Pyretic language requires only a standard socket to allow programmers to describe network paths with fault-
interface and a simple OpenFlow client on the target controller tolerance requirements. For instance, each flow can have its
platform [157]. own alternative paths for dealing with failure of the pri-
Several programming languages have been proposed for mary paths. Interestingly, this feature is provided in a very
SDNs, as summarized in Table VII. The great majority propose programmer-friendly way, with the application programmer
abstractions for OpenFlow-enabled networks. The predomi- having only to use regular expressions with special characters,
nant programming paradigm is the declarative one, with a such as an asterisk. In the particular case of FatTire, an
single exception, Pyretic, which is an imperative language. asterisk will produce the same behavior as a traditional regular
Most declarative languages are functional, while but there are expression, but translated into alternative traversing paths.
instances of the logic and reactive types. The purpose – i.e., the Programming languages such as FlowLog [184] and
specific problems they intend to solve – and the expressiveness Flog [185] bring different features, such as model checking,
power vary from language to language, while the end goal is dynamic verification and stateful middleboxes. For instance,
almost always the same: to provide higher-level abstractions using a programming language such as Flog, it is possible
to facilitate the development of network control logic. to build a stateful firewall application with only five lines of
Programming languages such as FML [137], Nettle [155], code [185].
and Procera [156] are functional and reactive. Policies and Merlin [191] is one of the first examples of unified frame-
applications written in these languages are based on reactive work for controlling different network components, such as
actions triggered by events (e.g., a new host connected to forwarding devices, middleboxes, and end-hosts. An important
the network, or the current network load). Such languages advantage is backward-compatibility with existing systems.
allow users to declaratively express different network configu- To achieve this goal, Merlin generates specific code for
ration rules such as access control lists (ACLs), virtual LANs each type of component. Taking a policy definition as input,
(VLANs), and many others. Rules are essentially expressed Merlin’s compiler determines forwarding paths, transformation
as allow-or-deny policies, which are applied to the forwarding placement, and bandwidth allocation. The compiled outputs
elements to ensure the desired network behavior. are sets of component-specific low-level instructions to be
Other SDN programming languages such as Frenetic [138], installed in the devices. Merlin’s policy language also allows
Hierarchical Flow Tables (HFT) [183], NetCore [139], and operators to delegate the control of a sub-network to tenants,
Pyretic [157], were designed with the simultaneous goal of while ensuring isolation. This delegated control is expressed
efficiently expressing packet-forwarding policies and dealing by means of policies that can be further refined by each tenant
with overlapping rules of different applications, offering ad- owner, allowing them to customize policies for their particular
vanced operators for parallel and sequential composition of needs.
software modules. To avoid overlapping conflicts, Frenetic Other recent initiatives (e.g., systems programming lan-
disambiguates rules with overlapping patterns by assigning dif- guages [192]) target problems such as detecting anomalies
ferent integer priorities, while HFT uses hierarchical policies to improve the security of network protocols (e.g., Open-
with enhanced conflict-resolution operators. Flow), and optimizing horizontal scalability for achieving
See-every-packet abstractions and race-free semantics also high throughput in applications running on multicore archi-
represent interesting features provided by programming lan- tectures [190].
guages (such as Frenetic [138]). The former ensures that Most of the value of SDN will come from the network
all control packets will be available for analysis, sooner or managements applications built on top of the infrastructure.
later, while the latter provides the mechanisms for suppressing Advances in high-level programming languages are a funda-
unimportant packets. As an example, packets that arise from mental component to the success of a prolific SDN application
a network race condition, such as due to a concurrent flow development ecosystem. To this end, efforts are undergoing to
rule installation on switches, can be simply discarded by the shape forthcoming standard interfaces (cf. [193]) and towards
VERSION 1.0 20

TABLE VII
P ROGRAMMING LANGUAGES

Name Programming paradigm Short description/purpose

FatTire [189] declarative (functional) Uses regular expressions to allow programmers to describe network paths and respective fault-
tolerance requirements.
Flog [185] declarative (logic), event-driven Combines ideas of FML and Frenetic, providing an event-driven and forward-chaining logic
programming language.
FlowLog [184] declarative (functional) Provides a finite-state language to allow different analysis, such as model-checking.
FML [137] declarative (dataflow, reactive) High level policy description language (e.g., access control).
Frenetic [138] declarative (functional) Language designed to avoid race conditions through well defined high level programming
abstractions.
HFT [183] declarative (logic, functional) Enables hierarchical policies description with conflict-resolution operators, well suited for
decentralized decision makers.
Maple [190] declarative (functional) Provides a highly-efficient multi-core scheduler that can scale efficiently to controllers with
40+ cores.
Merlin [191] declarative (logic) Provides mechanisms for delegating management of sub-policies to tenants without violating
global constraints.
nlog [64] declarative (functional) Provides mechanisms for data log queries over a number of tables. Produces immutable tuples
for reliable detection and propagation of updates.
Nettle [155] declarative (functional, reactive) Based on functional reactive programming principles in order to allow programmers to deal
with streams instead of events.
NetCore [139] declarative (functional) High level programming language that provides means for expressing packet-forwarding policies
in a high level.
Procera [156] declarative (functional, reactive) Incorporates a set of high level abstractions to make it easier to describe reactive and temporal
behaviors.
Pyretic [157] imperative Specifies network policies at a high level of abstraction, offering transparent composition and
topology mapping.

the realization of integrated development environments (e.g., Despite the wide variety of use cases, most SDN ap-
NetIDE [194]) with the goal of fostering the development of plications can be grouped in one of five categories: traffic
a myriad of SDN applications. We discuss these next. engineering, mobility and wireless, measurement and moni-
toring, security and dependability and data center networking.
Table VIII summarizes several applications categorized as
H. Layer VIII: Management applications such, stating their main purpose, controller where it was
implemented/evaluated, and southbound API used.
Management applications can be seen as the “network
brains”. They implement the control-logic that will be trans- Traffic engineering
lated into commands to be installed in the data plane, dictating Several traffic engineering applications have been pro-
the behavior of the forwarding devices. Taking a simple appli- posed, including ElasticTree [198], Hedera [199], OpenFlow-
cation as routing as an example. The logic of this application based server load balancing [239], Plug-n-Serve [202] and
is to define the path through which packets will flow from a Aster*x [197], In-packet Bloom filter [200], SIMPLE [206],
point A to a point B. To achieve this goal a routing application QNOX [203], QoS framework [204], ALTO [195], and Vi-
has to, based on the topology input, decide on the path to use Aggre SDN [207]. The main goals of most applications is to
and instruct the controller to install the respective forwarding engineer traffic with the aim of minimizing power consump-
rules in all forwarding devices on the chosen path, from A to tion, maximizing aggregate network utilization, providing op-
B. timized load balancing, and other generic traffic optimization
Software-defined networks can be deployed on any tradi- techniques.
tional network environment, from home and enterprise net- Load balancing was one of the first applications envisioned
works to data centers and Internet exchange points. Such for SDN/OpenFlow. Different algorithms and techniques have
variety of environments has led to a wide array of man- been proposed for this purpose [239], [197], [202]. One partic-
agement applications. Existing network management applica- ular concern is the scalability of these solutions. A technique to
tions perform traditional functionality such as routing, load allow this type of applications to scale is to use wildcard-based
balancing, and security policy enforcement, but also explore rules to perform proactive load balancing [239]. Wildcards
novel approaches, such as reducing power consumption . Other can be utilized for aggregating clients requests based on the
examples include fail-over and reliability functionalities to the ranges of IP prefixes, for instance, allowing the distribution
data plane, end-to-end QoS enforcement, network virtualiza- and directing of large groups of client requests without re-
tion, mobility management in wireless networks, among many quiring controller intervention for every new flow In tandem,
others. operation in reactive mode may still be used when traffic
VERSION 1.0 21

TABLE VIII
M ANAGEMENT APPLICATIONS AND SERVICES

Group Solution/Application Main purpose Controller Southbound API

ALTO VPN [195] on-demand VPNs NMS [196], [25] SNMP
Aster*x [197] load balancing NOX OpenFlow
ElasticTree [198] energy aware routing NOX OpenFlow
Hedera [199] scheduling / optimization — OpenFlow
In-packet Bloom filter [200] load balancing NOX OpenFlow
Traffic OpenQoS [201] dynamic QoS routing for multimedia apps Floodlight OpenFlow
engineering Plug-n-Serve [202] load balancing NOX OpenFlow
QNOX [203] QoS enforcement NOX Generalized OpenFlow
QoS framework [204] QoS enforcement NOX OF with QoS extensions
QoSFlow [205] multiple packet schedulers to improve QoS — OpenFlow
SIMPLE [206] middlebox-specific “traffic steering” Extended POX OpenFlow
ViAggre SDN [207] divide and spread forwarding tables NOX OpenFlow
CROWD [208] overlapping of LTE and WLAN cells — OpenFlow
CloudMAC [209] outsourced processing of WLAN MACs — OpenFlow
FAMS [210] flexible VLAN system based on OpenFlow ProgrammableFlow OpenFlow

Mobility MobileFlow [154] flow-based model for mobile networks MobileFlow SDMN API
& Odin [211] smooth hand-off and load balancing Floodlight OpenFlow
Wireless OpenRAN [212] vertical programmability and virtualization — —
OpenRoads [213] control of the data path using OpenFlow FlowVisor OpenFlow
SoftRAN [165] load balancing and interference management — Femto API [214], [215]
BISmark [6] active and passive measurements Procera framework OpenFlow
FleXam [216] flexible sampling extension for OpenFlow — —
Measurement FlowSense [217] measure link utilization in OF networks — OpenFlow
& measurement model [218] model for OF switch measurement tasks — OpenFlow
Monitoring
OpenSketch [219] separated measurement data plane OpenSketch “OpenSketch sketches”
OpenTM [188] traffic matrix estimation tool NOX OpenFlow
PaFloMon [220] passive monitoring tools defined by users FlowVisor OpenFlow
Active security [221] integrated security using network feedback Floodlight OpenFlow
control
AVANT-GUARD [222] DoS security specific extensions to OF POX OpenFlow
CloudWatcher [223] framework for monitoring clouds NOX OpenFlow
DDoS detection [224] attacks detection and mitigation NOX OpenFlow
Elastic IP and Security an SDN based implementation of Amazon’s NOX OpenFlow
Group [225] Elastic IP and Security Groups
Security Ethane [52] flow-rule enforcement (match/action) Ethane controller first instance of OpenFlow
&
Dependability FortNOX [135] security flow rules prioritization NOX OpenFlow
FRESCO [186] framework for security services composition NOX OpenFlow
LiveSec [226] security policy enforcement NOX OpenFlow
NetFuse [227] protection against OF traffic overload — OpenFlow
OF-RHM [228] random host mutation (defense) NOX OpenFlow
OpenSAFE [229] direct spanned net traffic in arbitrary ways NOX OpenFlow
Reliable multicasting [230] reduce packet loss when failures occur Trema OpenFlow
SANE [51] security policy enforcement SANE controller SANE header (pre-OpenFlow)
VAVE [231] source address validation with a global view NOX OpenFlow
Big Data Apps [232] optimize network utilization — OpenFlow
CloudNaaS [233] networking primitives for cloud applications NOX OpenFlow
FlowComb [234] predicts application workloads NOX OpenFlow
Data Center
Networking FlowDiff [235] detects operational problems FlowVisor OpenFlow
LIME [236] live network migration Floodlight OpenFlow
NetGraph [237] graph queries for network management — OpenFlow, SNMP
OpenTCP [238] dynamic and programmable TCP adaptation — —
VERSION 1.0 22

bursts are detected. The controller application needs to monitor nity for making it easier to deploy and manage different
the network traffic and use some sort of threshold in the types of wireless networks, such as WLANs and cellular
flow counters to redistribute clients among the servers when networks [211], [213], [208], [165], [244], [245]. Traditionally
bottlenecks are likely to happen. hard-to-implement but desired features are indeed becoming
SDN load-balancing also simplifies the placement of net- a reality with the SDN-based wireless networks. These in-
work services in the network [202]. Every time a new server clude seamless mobility through efficient hand-overs [211],
is installed, the load-balancing service can take the appropriate [246], [244], load balancing [211], [165], creation of on-
actions to seamlessly distribute the traffic among the available demand virtual access points (VAPs) [211], [209], downlink
servers, taking into consideration both the network load and scheduling (e.g., an OpenFlow switch can do a rate shaping
the available computing capacity of the respective servers. This or time division) [209], dynamic spectrum usage [209],
simplifies network management and provides more flexibility enhanced inter-cell interference coordination [209], [244],
to network operators. device to device offloading (i.e., decide in when and how
Existing southbound interfaces can be used for actively LTE transmissions should be offloaded to users adopting the
monitoring the data plane load. This information can be lever- D2D paradigm [247]) [208], per client and/or base station
aged to optimize the energy consumption of the network [198]. resource block allocations (i.e., time and frequency slots
By using specialized optimization algorithms and diversified in LTE/OFDMA networks, which are known as resource
configuration options, it is possible to meet the infrastruc- blocks) [165], [208], [245], control and assign transmission
ture goals of latency, performance, and fault tolerance, for and power parameters in devices or in a group basis (e.g.,
instance, while reducing power consumption. With the use of algorithms to optimize the transmission and power parameters
simple techniques, such as shutting down links and devices of transmission and power parameters of WLAN devices,
intelligently in response to traffic load dynamics, data center define and assign transmission power values to each resource
operators can save up to 50% of the network energy in normal block, at each base station, in LTE/OFDMA networks) [208],
traffic conditions [198]. [165], simplified administration [211], [213], [165], easy man-
One of the important goals of data center networks is to agement of heterogenous network technologies [213], [165],
avoid or mitigate the effect of network bottlenecks on the [248], interoperability between different networks [248], [245],
operation of the computing services offered. Linear bisection shared wireless infrastructures [248], seamless subscriber mo-
bandwidth is a technique that can be adopted for traffic bility and cellular networks [244], QoS and access control
patterns that stress the network by exploring path diversity policies made feasible and easier [244], [245], and easy
in a data center topology. Such technique has been proposed deployment of new applications [211], [165], [248].
in an SDN setting, allowing the maximization of aggregated One of the first steps towards realizing these features in
network utilization with minimal scheduling overhead [199]. wireless networks is to provide programmable and flexible
SDN can also be used to provide a fully automated system stack layers for wireless networks [249], [165]. One of the
for controlling the configuration of routers. This can be partic- first examples is OpenRadio [249], which proposes a soft-
ularly useful in scenarios that apply virtual aggregation [240] ware abstraction layer for decoupling the wireless protocol
. This technique allows network operators to reduce the data definition from the hardware, allowing shared MAC layers
replicated on routing tables, which is one of the causes of across different protocols using commodity multi-core plat-
routing tables’ growth [241]. A specialized routing applica- forms. OpenRadio can be seen as the “OpenFlow for wireless
tion [207] can calculate, divide and configure the routing tables networks”. Similarly, SoftRAN [165] proposes to rethink the
of the different routing devices through a southbound API such radio access layer of current LTE infrastructures. Its main
as OpenFlow. goal is to allow operators to improve and optimize algorithms
Traffic optimization is another interesting application for for better hand-overs, fine-grained control of transmit powers,
large scale service providers, where dynamic scale-out is resource block allocation, among other management tasks.
required. For instance, the dynamic and scalable provisioning Light virtual access points (LVAPs) is another interesting
of VPNs in cloud infrastructures, using protocolols such as way of improving the management capabilities of wireless
ALTO [25], can be simplified through an SDN-based ap- networks, as proposed by Odin [211]. Differently from Open-
proach [195]. Radio, it works with existing wireless hardware and does
Other applications that perform routing and traffic engineer- not impose any change on IEEE 802.11 standards. An LVAP
ing include application-aware networking for video stream- is implemented as a unique BSSID associated with a spe-
ing [242] and improved QoS by employing multiple packet cific client, which means that there is a one-to-one mapping
schedulers [205] and other techniques [204], [203], [201], between LVAPs and clients. This empowers infrastructure
[243]. operators to provide seamless mobility, load balancing and
hidden terminal mitigation. For instance, when a user moves
Mobility & wireless from one access point (AP) to another, the network mobility
The current distributed control plane of wireless networks management application can automatically and proactively act
is suboptimal for managing the limited spectrum, allocating and move the client LVAP from AP to the other. In this way,
radio resources, implementing handover mechanisms, man- a wireless client will not even notice that it started to use a
aging interference, and performing efficient load-balancing different AP because there is no perceptive hand-off delay, as
between cells. SDN-based approaches represent an opportu- it would be the case in traditional wireless networks.
VERSION 1.0 23

Very dense heterogeneous wireless networks have also been with three stages (hashing, classification, and counting). Input
a target for SDN. These DenseNets have limitations due to packets first pass through a hashing function. Then, they are
constraints such as radio access network bottlenecks, control classified according to a matching rule. Finally, the match
overhead, and high operational costs [208]. A dynamic two-tier rule identifies a counting index, which is used to calculate the
SDN controller hierarchy can be adapted to address some of counter location in the counting stage. While a TCAM with
these constraints [208]. Local controllers can be used to take few entries is enough for the classification stage, the flexible
fast and fine-grained decisions, while regional (or “global”) counters are stored in SRAM. This makes the OpenSketch’s
controllers can have a broader, coarser-grained scope, i.e., operation efficient (fast matching) and cost-effective (cheaper
that take slower but more global decisions. In such a way, SRAMs to store counters).
designing a single integrated architecture that encompasses
Security & Dependability
LTE (macro/pico/femto) and WiFi cells, while challenging,
seems feasible. An already diverse set of security and dependability propos-
als is emerging in the context of SDNs. Most take advantage of
Measurement & monitoring SDN for improving services required to secure systems and
Measurement and monitoring solutions can be divided in networks, such as policy enforcement (e.g., access control,
two classes. First, applications that provide new functionality firewalling) [51], [226], [231], [225], DoS attacks detection
for other networking services. Second, proposals that target to and mitigation [224], random host mutation (stabler2012) (i.e.,
improve features of OpenFlow-based SDNs, such as to reduce randomly and frequently mutate the IP addresses of end-hosts
control plane overload due to the collection of statistics. to break the attackers’ assumption about static IPs, which is
An example of the first class of applications is improving the the common case) [228], monitoring of cloud infrastructures
visibility of broadband performance [250], [6]. An SDN-based for fine-grained security inspections (i.e., automatically ana-
broadband home connection can simplify the addition of new lyze and detour suspected traffic to be further inspected by
functions in measurement systems such as BISmark [250], al- specialized network security appliances, such as deep packet
lowing the system to react to changing conditions in the home inspection systems) [223], traffic anomaly detection [251],
network [6]. As an example, a home gateway can perform [224], and so forth [51], [226], [224], [228], [223], [225],
reactive traffic shaping considering the current measurement [231], [251]. Others address OpenFlow-based networks issues,
results of the home network. such as flow rule prioritization, security services composition,
The second class of solutions typically involve different and protection against traffic overload [135], [186], [222],
kinds of sampling and estimation techniques to be applied, [227].
in order to reduce the burden of the control plane with There are essentially two approaches, one involves using
respect to the collection of data plane statistics. Different SDNs to improve network security, and another for improving
techniques have been applied to achieve this goal, such as the security of the SDN itself. The focus has been, thus far,
stochastic and deterministic packet sampling techniques [251], in the latter.
traffic matrix estimation [188], and fine-grained monitoring of Using SDN to improve the security of current networks. Prob-
wildcard rules [252]. Point-to-point traffic matrix estimation, ably the first instance of SDN was an application for security
in particular, can help in network design and operational tasks policies enforcement [51]. An SDN allows the enforcement to
such as load balancing, anomaly detection, capacity planning be done on the first entry point to the network (e.g., the Eth-
and network provisioning. With information on the set of ernet switch to which the user is connected to). Alternatively,
active flows in the network, routing information (e.g., from in a hybrid environment, security policy enforcement can be
the routing application), flow paths, and flow counters in the made on a wider network perimeter through programmable
switches it is possible to construct a traffic matrix using diverse devices (without the need to migrate the entire infrastructure
aggregation levels for sources and destinations [188]. to OpenFlow) [226]. With either application, malicious actions
Other initiatives of this second class propose a stronger de- are blocked before entering the critical regions of the network.
coupling between basic primitives (e.g., matching and count- SDN has been successfully applied for other purposes,
ing) and heavier traffic analysis functions such as the detection namely for the detection (and reaction) against DDoS flooding
of anomaly conditions attacks [253]. A stronger separation attacks [224], and active security [221]. OpenFlow forwarding
favors portability and flexibility. For instance, a functionality devices make it easier to collect a variety of information from
to detect abnormal flows should not be constrained by the the network, in a timely manner, which is very handy for
basic primitives or the specific hardware implementation. Put algorithms specialized in detecting DDoS flooding attacks
another way, developers should be empowered with streaming The capabilities offered by software-defined networks in
abstractions and higher level programming capabilities. increasing the ability to collect statistics data from the network
In that vein, some data and control plane abstractions and of allowing applications to actively program the forward-
have been specifically designed for measurement purposes. ing devices, are powerful for proactive and smart security pol-
OpenSketch [219] is a special-purpose southbound API de- icy enforcement techniques such as Active security [221]. This
signed to provide flexibility for network measurements. For novel security methodology proposes a novel feedback loop
instance, by allowing multiple measurement tasks to execute to improve the control of defense mechanisms of a networked
concurrently without impairing accuracy. The internal design infrastructure, and is centered around five core capabilities:
of an OpenSketch switch can be thought of as a pipeline protect, sense, adjust, collect, counter. In this perspective,
VERSION 1.0 24

active security provides a centralized programming interface clouds, an essential features is virtual network migration.
that simplifies the integration of mechanisms for detecting Similarly to traditional virtual machine migration, a virtual
attacks, by a) collecting data from different sources (to identify network may need to be migrated when its virtual machines
attacks), b) converging to a consistent configuration for the move from one place to another. Integrating live migration of
security appliances, and c) enforcing countermeasures to block virtual machines and virtual networks is one of the forefront
or minimize the effect of attacks. challenges [236]. To achieve this goal it is necessary to dynam-
Improving the security of SDN itself. There are already some ically reconfigure all affected networking devices (physical or
research efforts on identifying the critical security threats of virtual). This as shown to be possible with SDN platforms,
SDNs and in augmenting its security and dependability [135], such as NVP [64].
[186], [254]. Early approaches try to apply simple techniques, Another potential application of SDN in data centers is in
such as classifying applications and using rule prioritization, detecting abnormal behaviors of the network operation [235].
to ensure that rules generated by security applications will By using different behavioral models and collecting the nec-
not be overwritten by lower priority applications [135]. Other essary information from elements involved in the operation
proposals try to go a step further by providing a framework of a data center (infrastructure, operators, applications), it is
for developing security-related applications in SDNs [186]. possible to continuously build signatures for applications by
However, there is still a long way to go in the development of passively capturing control traffic. Then, the signature history
secure and dependable SDN infrastructures [254]. An in-deep can be used to identify differences in behavior. Every time a
overview of SDN security issues and challenges can be found difference is detected, operators can reactively or proactively
in Section V-F. take corrective measures. This can help to isolate abnormal
components and avoid further damage to the infrastructure.
Data Center Networking
From small enterprises to large scale cloud providers, Towards SDN App Stores
most of the existing IT systems and services are strongly As can be observed in Table VIII, most SDN applications
dependent on highly scalable and efficient data centers. Yet, rely on NOX and OpenFlow. NOX was the first controller
these infrastructures still pose significant challenges regarding available for general use, making it a natural choice for most
computing, storage and networking. Concerning the latter, data use-cases so far. As indicated by the sheer number of security-
centers should be designed and deployed in such a way as related applications, security is probably one of the killer
to offer high and flexible cross-section bandwidth and low- applications for SDNs. Curiously, while most use cases rely
latency, QoS based on the application requirements, high on OpenFlow, new solutions such as SoftRAN are considering
levels of resilience, intelligent resource utilization to reduce different APIs, as is the case of the Femto API [214], [215].
energy consumption and improve overall efficiency, agility This diversity of applications and APIs will most probably
in provisioning network resources, for example by means of keep growing in SDN.
network virtualization and orchestration with computing and There are other kinds of management applications that
storage, and so forth [255], [256], [257]. Not surprisingly, do not easily fit in our taxonomy, such as Avior [259],
many of these issues remain open due to the complexity and OESS [260], and SDN App Store [261], [262]. Avior and
inflexibility of traditional network architectures. OESS are graphical interfaces and sets of software tools that
The emergence of software-defined networks has been ex- make it easier to configure and manage controllers (e.g.,
pected to change the current state of affairs. Early research Floodlight) and OpenFlow-enabled switches, respectively. By
efforts have indeed showed that data center networking can leveraging their graphical functions it is possible to program
significantly benefit from SDN in solving different problems OpenFlow enabled devices without coding in a particular
such as live network migration [236], improved network programming language.
management [236], [235], eminent failure avoidance [236], The SDN App Store [261], [262], owned by HP, is probably
[235], rapid deployment from development to production the first SDN application market store. Customers using HP’s
networks [236], troubleshooting [236], [237], and optimization OpenFlow controller have access to the online SDN App
of network utilization [237], [232], [234], [235]. SDN can also Store and are able to select applications to be dynamically
offer networking primitives for cloud applications, solutions downloaded and installed in the controller. The idea is similar
to predict network transfers of applications [232], [234], to the Android Market or the Apple Store, making is easier
mechanisms for fast reaction to operation problems, network- for developers to provide new applications and for customers
aware VM placement [237], [233], QoS support [237], [233], to obtain them.
realtime network monitoring and problem detection [237],
[234], [235], security policy enforcement services and mech- I. Cross-layer issues
anisms [237], [233], and enable programmatic adaptation of
In this section we look at cross-layer problems such as de-
transport protocols [232], [238].
bugging and troubleshooting, testing, verification, simulation
SDN can help infrastructure providers to expose more
and emulation.
networking primitives to their customers, by allowing virtual
network isolation, custom addressing, and the placement of Debugging and troubleshooting
middleboxes and virtual desktop cloud applications [233], Debugging and troubleshooting have been important sub-
[258]. To fully explore the potential of virtual networks in jects in computing infrastructures, parallel and distributed
VERSION 1.0 25

systems, embedded systems, and desktop applications [263], network will be recorded and, afterwards, select specific parts
[264], [265], [266], [267], [268], [269]. The two predomi- of the traces to be replayed. These replays provide valuable
nant strategies applied to debug and troubleshoot are runtime information to find the root cause of the network misbehavior.
debugging (e.g., gdb-like tools) and post-mortem analysis Despite the availability of these debugging and verification
(e.g., tracing, replay and visualization). Despite the constant tools, it is still difficult to answer questions such as: What is
evolution and the emergence of new techniques to improve happening to my packets that are flowing from point A to point
debugging and troubleshooting, there are still several open B? What path do they follow? What header modifications do
avenues and research questions [264]. they undergo on the way? To answer some of these questions
Debugging and troubleshooting in networking is at a very one could recur to the history of the packets. A packet’s history
primitive stage. In traditional networks, engineers and de- corresponds to the paths it uses to traverse the network, and the
velopers have to use tools such as ping, traceroute, header modifications in each hop of the path. NetSight [280]
tcpdump, nmap, netflow, and SNMP statistics for de- is a platform whose primary goal is to allow applications
bugging and troubleshooting. Debugging a complex network that use the history of the packets to be built, in order to
with such primitive tools is very hard. Even when one con- find out problems in a network. This platform is composed
siders frameworks such as XTrace [268], Netreplay [270] of three essential elements: (1) NetSight, with its dedicate
and NetCheck [271], which improve debugging capabilities servers that receive and process the postcards for building
in networks, it is still difficult to troubleshoot networking the packet history, (2) the NetSigh-SwitchAssist, which can
infrastructures. For instance, these frameworks require a huge be used in switches to reduce the processing burden on the
effort in terms of network instrumentation. The additional dedicated servers, and (3) the NetSight-HostAssist to generate
complexity introduced by different types of devices, tech- and process postcards on end hosts (e.g., in the hypervisor on
nologies and vendor specific components and features make a virtualized infrastructure).
matters worse. As a consequence, these solutions may find netwatch [280], netshark [280] and nprof [280] are
it hard to be widely implemented and deployed in current three examples of tools built over NetSight. The first is a
networks. live network invariant monitor. For instance, an alarm can be
Software-defined networks’ capability of programming the trigged every time a packet violates any invariant (e.g., no
network offers some hope in this respect. Its software-based loops). The second, netshark, enables users to define and
control and the use of open standards for control communica- execute filters on the entire history of packets. With this tool,
tion can potentially make debug and troubleshoot easier. The a network operator can view a complete list of properties of
flexibility and programmability introduced by SDN is indeed packets at each hop, such as input port, output port, and packet
opening new avenues for developing better tools to debug, header values. Finally, nprof can be used to profile sets of
troubleshoot, verify and test networks [272], [273], [274], network links to provide data for analyzing traffic patterns and
[275], [276], [277], [278], [279], [278]. routing decisions that might be contributing to link load.
Early debugging tools for OpenFlow-enabled networks,
such as ndb [272], OFRewind [273] and NetSight [280], make Testing and verification
it easier to discover the source of network problems such as Verification and testing tools can complement debugging
faulty device firmware [272], inconsistent or non-existing flow and troubleshooting. Recent research [277], [279], [276],
rules [272], [273], lack of reachability [272], [273], and faulty [274], [278], [281], [282] has shown that verification tech-
routing [272], [273]. Similarly to the well-known gdb soft- niques can be applied to detect and avoid problems in SDN,
ware debugger, ndb provides basic debugging actions such as such as forwarding loops and black holes. Verification can
breakpoint, watch, backtrace, single-step, and continue. These be done at different layers (at the controllers, management
primitives help application developers to debug networks in a applications, or network devices).
similar way to traditional software. By using ndb’s postcards Tools such as NICE [274] generate sets of diverse streams
(i.e., a unique packet identifier composed of a truncated copy of packets to test as many as possible events, exposing corner
of the packet’s header, the matching flow entry, the switch, and cases such as race conditions. Similarly, OFLOPS [275] pro-
the output port), for instance, a programmer is able to quickly vides a set of features and functions that allow the development
identify and isolate a buggy OpenFlow switch with hardware and execution of a rich set of tests on OpenFlow-enabled
or software problems. If the switch is presenting abnormal devices. Its ultimate goal is to measure the processing ca-
behavior such as corrupting parts of the packet header, by pacity and bottlenecks of control applications and forwarding
analyzing the problematic flow sequences with a debugging devices. With this tool, users are able to observe and evaluate
tool one can find (in a matter of few seconds) where the forwarding table consistency, flow setup latency, flow space
packets of a flow are being corrupted, and take the necessary granularity, packet modification types, and traffic monitoring
actions to solve the problem. capabilities (e.g., counters).
The OFRewind [273] tool works differently. The idea is FlowChecker [276], OFTEN [278], and VeriFlow [277]
to record and replay network events, in particular control are three examples of tools to verify correctness properties
messages. These usually account for less than 1% of the data violations on the system. While the former two do offline
traffic and are responsible for 95%-99% of the bugs [279]. analysis, the latter is capable of online checking of network
This tool allows operators to perform fine-grained tracing of invariants. Verification constraints include security and reach-
network behavior, being able to decide which subsets of the ability issues, configuration updates on the network, loops,
VERSION 1.0 26

black holes, etc. switching components with OpenFlow support. Its main goal
Other formal modeling techniques, such as Alloy, can be is to provide a more realistic and scalable simulation platform
applied to SDNs to identify unexpected behavior [281]. For as compared to Mininet. STS [291] is a simulator designed to
instance, a protocol specification can be weak when it under- allow developers to specify and apply a variety of test cases,
specifies some aspects of the protocol or due to a very specific while allowing them to interactively examine the state of the
sequence of events. In such situations, model checking tech- network.
niques such as Alloy can help to find and correct unexpected
behaviors. V. O NGOING R ESEARCH E FFORTS AND C HALLENGES
One of the challenges in testing and verification is to verify
The research efforts we have surveyed so far seek to
forwarding tables in very large networks to find routing errors,
overcome the challenges of realizing the vision and fulfilling
which can cause traffic losses and security breaches, as quickly
the potential of SDN. While Section IV provided a perspective
as possible. In large scale networks, it is not possible to assume
structured across the layers of the “SDN stack”, this section
that the network snapshot, at any point, is consistent, due
highlights research we consider of particular importance for
to the frequent changes in routing state. Therefore, solutions
unleashing the full potential of SDN, and that therefore
such as HSA [283], Anteater [284], NetPlumber [285] and
deserves a specific coverage in this survey.
VeriFlow [277] are not suited for this kind of environment.
Another important issue is related on how fast the verification
process is done, especially in modern data centers that have A. Switch Designs
very tight timing requirements. Libra [282] represents one of Currently available OpenFlow switches are very diverse and
the first attempts to address these particular challenges of large exhibit notable differences in terms of feature set (e.g., flow
scale networks. This tool provides the means for capturing table size, optional actions), performance (e.g., fast vs. slow
stable and consistent snapshots of large scale network deploy- path, control channel latency/throughput), interpretation and
ments, while also applying long prefix matching techniques to adherence to the protocol specification (e.g., BARRIER com-
increase the scalability of the system. By using MapReduce mand), and architecture (e.g., hardware vs. software designs).
computations, Libra is capable of verifying the correctness of
a network with up to 10k nodes within one minute. Heterogenous Implementations
Implementation choices have a fundamental impact on the
Simulation and emulation behavior, accuracy, and performance of switches, ranging from
Simulation and emulation software is of particular impor- differences in flow counter behavior [295] to a number of
tance for fast prototyping and testing without the need for other performance metrics [275]. One approach to accommo-
expensive physical devices. Mininet [62] is the first system date such heterogeneity is through NOSIX, a portable API
that provides a quick and easy way to prototype and evaluate that separates the application expectations from the switch
SDN protocols and applications. One of the key properties heterogeneity [174]. To do so, NOSIX provides a pipeline of
of Mininet is its use of software-based OpenFlow switches multiple virtual flow tables and switch drivers. Virtual flow
in virtualized containers, providing the exact same semantics tables are intended to meet the expectations of applications
of hardware-based OpenFlow switches. This means that con- and are ultimately translated by the drivers into actual switch
trollers or applications developed and tested in the emulated flow tables. Towards taming the complexity of multiple Open-
environment can be (in theory) deployed in an OpenFlow- Flow protocol versions with different sets of required and
enabled network without any modification. Users can easily optional capabilities – a roadblock for SDN practitioners –,
emulate an OpenFlow network with hundreds of nodes and tinyNBI [296] has been proposed as a simple API providing a
dozens of switches by using a single personal computer. unifying set of core abstractions of five OpenFlow protocol
Mininet-HiFi [286] is an evolution of Mininet that enhances versions (from 1.0 to 1.4). Ongoing efforts to introduce a
the container-based (lightweight) virtualization with mecha- new Hardware Abstraction Layer (HAL) for non-OpenFlow
nisms to enforce performance isolation, resource provisioning, capable devices [297] include the development of open source
and accurate monitoring for performance fidelity. One of the artifacts like ROFL (Revised OpenFlow Library) and the xDPd
main goals of Mininet-HiFi is to enable repeatable and realistic (eXtensible DataPath daemon), a framework for creating new
network experiments. OpenFlow datapath implementations based on a diverse set
Mininet CE [287] and SDN Cloud DC [288] are extensions of hardware and software platforms. A related open source
to Mininet for enabling large scale simulations. Mininet CE effort to develop a common library to implement OpenFlow
combines groups of Mininet instances into one cluster of simu- 1.0 and 1.3 protocol endpoints (switch agents and controllers)
lator instances to model global scale networks. SDN Cloud DC is libfluid [298], winner of the OpenFlow driver competition
enhances Mininet and POX to emulate an SDN-based intra- organized by the ONF.
DC network by implementing new software modules such as Within the ONF, the Forwarding Abstraction Working
data center topology discovery and network traffic generation. Group (FAWG) is pursuing another solution to the heterogen-
The capability of simulating OpenFlow devices has been ity problem, through Table Type Patterns (TTPs) [73]. A TTP
added to the popular ns-3 simulator [289]. Another example is a standards-based and negotiated switch-level behavioral
of large scale simulation is fs-sdn, which extends the fs abstraction. It consists of the relationships between tables
simulation engine [290] by incorporating a controller and forming a graph structure, the types of tables in the graph,
VERSION 1.0 27

TABLE IX
D EBUGGING , VERIFICATION AND SIMULATION

Group Solution Main purpose Short description

ndb [272] gdb alike debugging for net- It provides basic debugging primitives that help developers to debug their
works networks.
Debugging NetSight [280] multi purpose packet history Allows operators to build flexible debugging, monitoring and profiling applica-
tions.
OFRewind [273] tracing and replay OFRewind allows operators to do a fine-grained tracing of the network behavior.
Operators can decide which subsets of the network will be recorded.
Cbench [292] evaluate OpenFlow controllers The Cbench framework can be used to emulate OpenFlow switches which are
configured to generate workload to the controller.
FLOVER [187] model checking for security FLOVER provides a provably correct and automatic method for verifying security
policies properties with respect to a set of flow rules committed by an OF controller.
FlowChecker [276] flow table configuration verifi- A tool used to verify generic properties of global behaviors based on flow tables.
cation
NetPlumber [285] real time policy checking NetPlumber uses a set of policies and invariants to do real time checking. It
leverages header space analysis and keeps a dependency graph between rules.
Verification NICE [274] remove bugs in controllers Its main goal is to test controller programs without requiring any type of
modification or extra work for application programmers.
OFCBenchmark [293] evaluate OpenFlow controllers creates independent virtual switches, making is possible to emulate different
scenarios. Each switch has its how configuration and statistics.
OFTEN [278] catch correctness property vio- A framework designed to check SDN systems, analyzing controller and switch
lations interaction, looking for correctness condition violation.
OFLOPS [275] evaluate OpenFlow switches A framework with a rich set of tests for OpenFlow protocol, enabling to measure
capabilities of both switch and applications.
VeriFlow [277] online invariant verification It provides real time verification capabilities, while the network state is still
evolving.
fs-sdn [294] fast simulation Like Mininet, it provides a simulation environment, but with speed and scalability
advantages.
Mininet [62] fast prototyping It emulates and OpenFlow network using Open vSwitches to provide the exact
same semantics of hardware devices.
Mininet CE [287] global network modeling It is a combination of tools to create a Mininet cluster for large scale simulation
of network topologies and architectures.

Simulation Mininet-HiFi [286] fast prototyping for repro- An evolution of Mininet to enable repeatable and high fidelity network experi-
ducibility ments.
ns-3 [289] network simulation The latest version of ns-3 simulator provides support to OpenFlow, enabling to
create programmable network devices.
SDN Cloud DC [288] cloud data center emulation The SDN Cloud DC solution allows users to evaluate the performance of their
controllers at scale.
STS [291] troubleshooting It simulates a network, allows to generate tricky test cases, and allows interac-
tively examine the state of the network.

a set of the parameterized table properties for each table in Flow devices have TCAMs with roughly 8K entries, where
the graph, the legal flow-mod and table-mod commands the actual capacity in terms of OpenFlow table size has
for each flow table, and the metadata mask that can be passed a non-trivial relationship to the type of flow entries being
between each table pair in the graph. used [303], [304]. OpenFlow version 1.1 introduced multiple
tables, thereby adding extra flexibility and scalability. Indeed,
Flow Table Capacity
OpenFlow 1.0 implied state explosion due to its flat table
Flow matching rules are stored in flow tables inside network model [73]. However, supporting multiple tables in hardware
devices. One practical challenge is to provide switches with is challenging and limited – yet another motivation for the
large and efficient flow tables to store the rules [299]. TCAMs ongoing ONF FAWG work on TTPs [73].
are a common choice to hold flow tables. While flexible and
efficient in terms of matching capabilities, TCAMs are costly Performance
and usually small (from 4K to 32K entries). Some TCAM Commercial OpenFlow switches today support only around
chips today integrate 18 M-bit (configured as 500k entries ∗ 200 control events (e.g., packet-in, flow-mod) per sec-
36 bit per entry) into a single chip working at 133 Mhz [300], ond [305]. This is clearly a limiting factor that shall be
i.e., capable of 133M lookups per second. However, these addressed in the switch design process – support of OpenFlow
chips are expensive and have a high-power consumption [301], in existing product lines has been more a retrofitting activity
representing a major power drain in a switching device [302]. than a clean feature planning and implementation activity.
These are some of the reasons why currently available Open- Deployment experiences [306] have pointed to a series of
VERSION 1.0 28

challenges stemming from the limited embedded CPU power proposes to augment switches with FPGA to (remotely) define
of current commercial OpenFlow switches. One approach to the queue management and scheduling behaviour of packet
handle the problem consists of adding more powerful CPUs switches [323].
into the switches, as proposed in [307]. Others have proposed
Native SDN Switch Designs
to rethink the distribution of control actions between external
Most of the SDN switch (re)design efforts so far follow
controllers and the OpenFlow agent inside the switch [295].
an evolutionary approach, to retrofit OpenFlow-specific pro-
Our current understanding indicates that an effective way
grammable features into existing hardware layouts, following
forward is a native design of SDN switches consistent with
common wisdom on switch/router designs and consolidated
the evolution of the southbound API standardization activi-
technologies (e.g., SRAM, TCAM). One departure from this
ties [308], [73].
approach is the ongoing work on forwarding metamorpho-
Evolving Switch Designs sis [308], a reconfigurable match table model inspired from
New SDN switch designs are expected to appear in a RISC-like pipelined architecture applied to switching chips.
myriad of hardware combinations to efficiently work together This work illustrates the feasibility of realizing a minimal set
with TCAMs, such as SRAM, RLDRAM, DRAM, GPU, of action primitives for flexible header processing in hardware,
FPGA, NPs, CPUs, among other specialized network proces- at almost no additional cost or power. Also in line with the core
sors [309], [310], [311], [312], [313], [314]. These early works SDN goals of highly flexible and programmable (hardware-
suggest the need for additional efforts into new hardware based) dataplanes, Protocol-Oblivious Forwarding (POF) [72]
architectures for future SDN switching devices. For instance, aims at overcoming some of the limitations of OpenFlow (e.g.,
some proposals target technologies such as GPUs that have expressiveness, support of user-defined protocols, memory
demonstrated 20 Gbps with flow tables of up to 1M exact efficiency), through generic flow instruction sets. Open-source
match entries and up to 1K wildcard entries [311]. Alternatives prototypes are available [23] as well as evaluation results
to TCAM-based designs include new hardware architectures showing the line-speed capabilities using a network processing
and components, as well as new and more scalable for- unit (NPU)-based [324] proof of concept implementation.
warding planes, such as the one proposed by the Rain Man Pretty much as TTPs allow controllers to compile the
firmware [315]. Other design solutions, such as parallel lookup right set of low-lever instructions known to be supported
models [316], can also be applied to SDN to reduce costs in by the switches, a new breed of switch referred to as P4
switching and routing devices. Recent proposals on cache- (programmable, protocol-independent packet processor) [325]
like OpenFlow switch arrangements [317] shed some light suggests an evolution path for OpenFlow, based on a high-level
on overcoming the practical limitations of flow table sizes compiler. The proposed flexibility would allow the function-
with clever switching designs. Additionally, counters represent ality of programmable switches (i.e., pipeline, header parsing,
another practical challenge in SDN hardware implementations. field matching) to be not only specified by the controller but
Many counters already exists, and they could lead to sig- also changed in the field. In this model, programmers are able
nificant control plane monitoring overhead [295]. Software- to decide how the forwarding plane processes packets without
defined counters (SDC) [307] have been proposed to provide caring about implementation details due to a compiler that
both scalability and flexibility. transforms an imperative program into a control flow graph
that can be mapped to different specific target switches.
Hardware Enhancements & Support
As in any software/hardware innovation cycle, a number of
advancements can be expected from the hardware perspective B. Controller Platforms
to improve SDN capabilities and performance [318], [319], In the SDN model, the controller platform is a critical pillar
[299], [320], [310], [321]. Microchip companies such as Intel of the architecture, and, as such, efforts are being devoted
are already shipping processors with flexible SDN capabili- to turn SDN controllers into high-performance, scalable, dis-
ties to the market [318]. Recent advances in Intel general- tributed, modular, and high-available pieces of programming-
purpose CPU technology include a data-plane development friendly software.
kit (DPDK) [322] that allows high-level programming of how
Performance
data packets shall be processed directly within network inter-
As the SDN community learns from the development and
face cards. Prototype implementations of an Intel DPDK accel-
operational experiences with OpenFlow controllers (e.g., Bea-
erated switch shows the potential to deliver high-performance
con [124]), further advancements are expected in terms of raw
SDN software switches without giving up the flexibility of
performance of controller implementations [326], including
programmable data planes [314]. This trend is likely to con-
the exploitation of hierarchical designs and optimized buffer
tinue since high-speed and specialized hardware is needed
sizing [326]. A more detailed discussion on performance
to boost SDN performance and scalability for large, real-
evaluation will be presented in Section V-E.
world networks. Hardware-programmable technologies such as
FPGA are widely used to reduce time and costs of hardware- Modularity
based feature implementations. NetFPGA, for instance, has As in software engineering in general, lack of modularity
been a pioneering technology used to implement OpenFlow results in controller implementations that are hard to build,
1.0 switches [310], providing a commodity cost-effective maintain, and extend – and ultimately become resistant to
prototyping solution. Another line of work on SDN data planes further innovations, resembling traditional “hardware-defined”
VERSION 1.0 29

networks. As surveyed in Section IV-G, SDN programming in Section IV-D are examples of approaches towards resilient
abstractions (e.g., Pyretic [157]) introduce modularity in SDN SDN controller platforms with different tradeoffs in terms of
applications and simplify their development altogether. Further consistency, durability and scalability.
research efforts (e.g., Corybantic [327]) try to achieve modu- On a detailed discussion on whether the CAP theorem [337]
larity in SDN control programs. Other contributions towards applies to networks, by Panda et al. [332], the authors ar-
achieving modular controllers can be expected from other gue that the trade-offs in building consistent, available and
areas of computer science (e.g., principles from Operating Sys- partition-tolerant distributed databases (i.e., CAP theorem)
tem [134]) and best practices of modern cloud-scale software may apply to SDN. The CAP theorem demonstrated that it
applications. is impossible for datastore systems to simultaneously achieve
strong consistency, availability and partition tolerance. While
High Availability
availability and partition tolerance problems are similar in both
In production, SDN controllers need to sustain healthy
distributed databases and networks, the problem of consistency
operation under the pressure of different objectives from
in SDN relates to the consistent application of policies.
the applications they host. Many advances are called for in
Taking the example of an OpenFlow network, when a switch
order to deal with potential risk vectors of controller-based
detects that a link failure (port-down event), a notification
solutions [254]. Certainly, many solutions will leverage on
is sent to the controller, which then takes the required actions
results from the distributed systems and security communities
(re-route computation, pre-computed back-up path lookups)
made over the last decade. Recent efforts are evolving towards
and installs updated flow entries in the required switches to
consistent, fault-tolerant data stores [150].
redirect the affected traffic. Such reactive strategies imply (1)
Interoperability and application portability high restoration time due to the necessary interaction with the
Similarly to forwarding device vendor agnosticism that controller; and (2) additional load on the control channel. One
stems from standard southbound interfaces, it is impor- experimental work on OpenFlow for carrier-grade networks
tant to foster interoperability between controllers. Early ini- investigated the restoration process and measured a restoration
tiatives towards more interoperable control platforms in- times in the order of 100 ms [331]. The delay introduced by
clude portable programming languages such as Pyretic [157] the controller may, in some cases, be prohibitive. In order to
and east/westbound interfaces among controllers, such as meet carrier grade requirements (i.e., 50 ms recovery time),
SDNi [146], ForCES CE-CE interface [22], [148], and ForCES protection schemes are required to mitigate the effects of a
Intra-NE mechanisms [149]. However, these efforts are yet far separated control plane. Suitable protection mechanisms (e.g.,
from fully realizing controller interoperability and application installation of pre-established backup paths in the switches)
portability. are possible in the most recent versions of the OpenFlow
protocol, by means of OpenFlow group table entries using
“fast-failover” actions.
C. Resilience An OpenFlow fault management approach [330] similar
Achieving resilient communication is a top purpose of net- to MPLS global path protection could be a viable solution,
working. As such, SDNs are expected to yield the same levels provided that OpenFlow switches are extended with end-to-
of availability as legacy and new alternative technologies. Split end path monitoring capabilities in the spirit of Bidirectional
control architectures as SDN are commonly questioned [328] Forwarding Detection (BFD). Such protection schemes are a
about their actual capability of being resilient to faults that may critical design choice for larger scale networks and may also
compromise the control-to-data plane communications and required considerable additional flow space.
thus result in “brainless” networks. Indeed, the malfunctioning Another related line of work is SlickFlow [335], leveraging
of particular SDN elements should not result in the loss of the idea of using packet header space to carry alternative path
availability. The relocation of SDN control plane functionality, information to implement resilient source routing in OpenFlow
from inside the boxes to remote, logically centralized loci, networks. Under the presence of failures along a primary path,
becomes a challenge when considering critical control plane packets can be rerouted to alternative paths by the switches
functions such as those related to link failure detection or fast themselves without involving the controller. Another recent
reaction decisions. The resilience of an OpenFlow network proposal that uses in-packet information is INFLEX [336],
depends on fault-tolerance in the data plane (as in traditional an SDN-based architecture for cross-layer network resilience
networks) but also on the high availability of the (logically) which provides on-demand path fail-over by having end-points
centralized control plane functions. Hence, the resilience of tag packets with virtual routing plane information that can
SDN is challenging due to the multiple possible failures of be used by egress routers to re-route by changing tags upon
the different pieces of the architecture. failure detection.
As noted in [329], there is a lack of sufficient research Language-based solutions to the data plane fault-tolerance
and experience in building and operating fault-tolerant SDNs. problem have also been proposed [189]. In this work the
Google B4 [8] may be one of the few examples that have authors propose a language that compiles regular expressions
proven that SDN can be resilient at scale. A number of related into OpenFlow rules to express what network paths packets
efforts [330], [331], [332], [189], [333], [334], [335], [336] may take and what degree of (link level) fault tolerance
have started to tackle the concerns around control plane split is required. Such abstractions around fault tolerance allow
architectures. The distributed controller architectures surveyed developers to build fault recovery capabilities into applications
VERSION 1.0 30

without huge coding efforts. the data plane, targeting a more scalable and efficient control
plane. Authoritative switches are responsible for installing
rules on the remaining switches, while the controller is still
D. Scalability
responsible for generating all the rules required by the logic
Scalability has been one of the major concerns of SDNs of applications. By dividing the controller work with these
from the outset. This is a problem that needs to addressed in special switches, the overall system scales better.
any system – e.g., in traditional networks – and is obviously Table X provides a non-exhaustive list of proposals address-
also a matter of much discussion in the context of SDN [11]. ing scalability issues of SDN. We characterize these issues by
Most of the scalability concerns in SDNs are related to application domain (control or data plane), their purpose, the
the decoupling of the control and data planes. Of particu- throughput in terms of number of flows per second (when
lar relevance are reactive network configurations where the the results of the experiments are reported), and the strategies
first packet of a new flow is sent by the first forwarding used. As can be observed, the vast majority are control plane
element to the controller. The additional control plane traffic solutions that try to increase scalability by using distributed
increases network load and makes the control plane a potential and multi-core architectures.
bottleneck. Additionally, as the flow tables of switches are Some figures are relatively impressive, with some solutions
configured in real-time by an outside entity, there is also the achieving up to 20M flows/s. However, we should caution
extra latency introduced by the flow setup process. In large- the reader that current evaluations consider only simple ap-
scale networks controllers will need to be able to process plications and count basically the number of packet-in
millions of flows per second [338] without compromising the and packet-out messages to measure throughput. The
quality of its service. Therefore, these overheads on the control actual performance of controllers will be affected by other
plane and on flow setup latency are (arguably) two of the major factors, such as the number and complexity of the applications
scaling concerns in SDN. running on the controller and security mechanisms imple-
As a result, several efforts have been devoted to tackle the mented. For example, a routing algorithm consumes more
SDN scaling concerns, including DevoFlow [295], Software- computing resources and needs more time to execute than a
Defined Counters (SDCs) [307], DIFANE [339], Onix [7], Hy- simple learning switch application Also, current evaluations
perFlow [133], Kandoo [158], Maestro [126], NOX-MT [125], are done using plain TCP connections. The performance is
and Maple [190]. Also related to scalability, the notion of very likely to change when basic security mechanisms are
elasticity in SDN controllers is also being pursued [334]. Elas- put in place, such as TLS, or more advanced mechanisms to
tic approaches include dynamically changing the number of avoid eavesdropping, man-in-the-middle and DoS attacks on
controllers and their locations under different conditions [340]. the control plane.
Most of the research efforts addressing scaling limitations Another important issue concerning scalability is data dis-
of SDN can be classified in three categories: data plane, tribution among controller replicas in distributed architectures.
control plane, and hybrid. While targeting the data plane, Distributed control platforms rely on data distribution mech-
proposals such as DevoFlow [295] and Software-Defined anisms to achieve their goals. For instance, controllers such
Counters (SDC) [307] actually reduce the overhead of the as Onix, HyperFlow, and ONOS need mechanisms to keep a
control plane by delegating some work to the forwarding consistent state in the distributed control platform. Recently,
devices. For instance, instead of requesting a decision from experimental evaluations have shown that high performance
the controller for every flow, switches can selectively iden- distributed and fault-tolerant data stores can be used to tackle
tify the flows (e.g., elephant flows) that may need higher- such challenges [150]. Nevertheless, further work is necessary
level decisions from the control plane applications. Another to properly understand state distribution trade-offs [342].
example is to introduce more powerful general purpose CPUs
in the forwarding devices to enable SDCs. A general purpose
CPU and software-defined counters offer new possibilities for E. Performance evaluation
reducing the control plane overhead by allowing software- As introduced in Section IV-A, there are already several
based implementations of functions for data aggregation and OpenFlow implementations from hardware and software ven-
compression, for instance. dors being deployed in different types of networks, from
Maestro [126], NOX-MT [125], Kandoo [158], Bea- small enterprise to large-scale data centers. Therefore, a grow-
con [124], and Maple [190] are examples of the effort on ing number of experiments over SDN-enabled networks is
designing and deploying high performance controllers, i.e., expected in the near future. This will naturally create new
trying to increase the performance of the control plane. challenges, as questions regarding SDN performance and scal-
These controllers mainly explore well-known techniques from ability have not yet been properly investigated. Understanding
networking, computer architectures and high performance the performance and limitation of the SDN concept is a
computing, such as buffering, pipelining and parallelism, to requirement for its implementation in production networks.
increase the throughput of the control platform. There are very few performance evaluation studies of Open-
The hybrid category is comprised of solutions that try to Flow and SDN architecture. Although simulation studies and
split the control logic functions between specialized data plane experimentation are among the most widely used performance
devices and controllers. In this category, DIFANE [339] pro- evaluation techniques, analytical modeling has its own benefits
poses authoritative (intermediate) switches to keep all traffic in too. A closed-form description of a networking architecture
VERSION 1.0 31

TABLE X
S UMMARY AND CHARACTERIZATION OF SCALABILITY PROPOSALS FOR SDN S .

Solution Domain Proposes Main purpose Flows/s Resorts to

DevoFlow [295] data plane thresholds for counters, reduce the control — Reduce the control traffic generated by counters
type of flow detection plane overhead statistics monitoring.
HyperFlow [133] control plane a distributed controller distribute the control — Application on top of NOX to provide control
plane message distribution among controllers.
Kandoo [158] control plane a hierarchical controller distribute the control — Use two levels of controller (local and root) to
plane hierarchically reduce control traffic.
Onix [7] control plane a distributed control robust and scalable — Provide a programmable and flexible distributed
platform control platform NIB for application programmers.
SDCs [307] data plane Software-Defined Coun- reduce the control — Remove counters from the ASIC to a general
ters plane overhead purpose CPU, improving programmability.
DIFANE [339] control and authoritative specialized improve data plane 500K Maintain flows in the data plane reducing controller
data plane switches performance work.
Floodlight [127] control plane a multi-threaded control- Improve controller 1.2M High performance flow processing capabilities.
ler performance
NOX-MT [125] control plane a multi-threaded control- improve controller 1.8M High performance flow processing capabilities.
ler performance
Maestro control plane coordination framework create clusters of con- 1.8M A coordination framework to create high-
cluster [341] trollers performance clusters of controllers.
NOX control plane coordination framework create clusters of con- 3.2M A coordination framework to create high-
cluster [341] trollers performance clusters of controllers.
Maestro [126] control plane a multi-threaded control- improve controller 4.8M High performance flow processing capabilities.
ler performance
NOX [127] control plane a multi-threaded control- improve controller 5.3M High performance flow processing capabilities.
ler performance
Beacon control plane coordination framework create clusters of con- 6.2M A coordination framework to create high-
cluster [341] trollers performance clusters of controllers.
Beacon [127] control plane a multi-threaded control- improve controller 12.8M High performance flow processing capabilities us-
ler performance ing pipeline threads and shared queues.
Maple [190] control plane programming language scaling algorithmic 20M Algorithmic policies and user- and OS-level
policies threads on multicore systems (e.g., 40+ cores).

paves the way for network designers to have a quick (and By utilizing Intel’s DPDK library [322], it has been shown
approximate) estimate of the performance of their design, that is possible to provide flexible traffic steering capability
without the need to spend considerable time for simulation at the hypervisor level (e.g., KVM) without the performance
studies or expensive experimental setup [306]. limitations imposed by traditional hardware switching tech-
Some work has investigated ways to improve the perfor- niques [345], such as SR-IOV [346]. This is particularly
mance of switching capabilities in SDN. These mainly consist relevant since most of the current enterprise deployments
of observing the performance of OpenFlow-enabled networks of SDN are in virtualized data center infrastructures, as in
regarding different aspects, such as lookup performance [343], VMware’s NVP solution [64].
hardware acceleration [312], the influence of types of rules and Current OpenFlow switch implementations can lead to per-
packet sizes [321], performance bottlenecks of current Open- formance bottlenecks with respect to the CPU load [295]. Yet,
Flow implementations [295], how reactive settings impact the modifications on the protocol specification can help reduce the
performance on data center networks [344], and the impact of occurrence of these bottlenecks. Further investigations provide
configuration on OpenFlow switches [292]. measurements regarding the performance of the OpenFlow
Design choices can have a significant impact on the lookup switch for different types of rules and packet sizes [321].
performance of OpenFlow switching in Linux operating sys- In data centers, a reactive setting of flow rules can lead
tem using standard commodity network interface cards [343]. to an unacceptable performance when only eight switches are
Just by using commodity network hardware the packet switch- handled by one OpenFlow controller [344]. This means that
ing throughput can be improved by up to 25% when compared large-scale SDN deployments should probably not rely on a
to one based on soft OpenFlow switching [343]. Similarly, purely reactive “modus operandi”, but rather on a combination
hardware acceleration based on network processors can also be of proactive and reactive flow setup.
applied to perform OpenFlow switching. In such cases, early To foster the evaluation of different performance aspects
reports indicate that performance, in terms of packet delay, of OpenFlow devices, frameworks such as OFlops [275],
can be improved by 20% when compared to conventional Cbench [125], and OFCBenchmark [293] have been proposed.
designs [312]. They provide a set of tools to analyze the performance of
VERSION 1.0 32

OpenFlow switches. Cbench [125], [292] is a benchmark tions and bottlenecks of SDN architectures can take a lot
tool developed to evaluate the performance of OpenFlow of time and effort to produce consistent outputs due to the
controllers. By taking advantage of the Cbench, it is possible to practical development and experimentation requirements. As
identify performance improvements for OpenFlow controllers mentioned before, analytic models can quickly provide per-
based on different environment and system configurations, formance indicators and potential scalability bottlenecks for
such as the number of forwarding devices, network topology, an OpenFlow switch-controller system before detailed data is
overall network workload, type of equipments, forwarding available. While simulation can provide detailed insight into
complexity, and overhead of the applications being executed a certain configuration, the analytical model greatly simplifies
on top of controllers [125]. Therefore, such tools can help sys- a conceptual deployment decision. For instance, a Network
tem designers make better decisions regarding the performance calculus-based model can be used to evaluate the performance
of devices and the network, while also allowing end-users to of an SDN switch and the interaction of SDN switches and
measure the device performance and better decide which one controllers [348]. The proposed SDN switch model captured
is best suited for the target network infrastructure. the closed form of the packet delay and buffer length inside
Surprisingly, despite being designed to evaluate the perfor- the SDN switch according to the parameters of a cumulative
mance of controllers, Cbench is currently a single-threaded arrival process. Using recent measurements, the authors have
tool. Therefore, multiple instances have to be started to utilize reproduced the packet processing delay of two variants of
multiple CPUs. It also only establishes one controller con- OpenFlow switches and computed the buffer requirements of
nection for all emulated switches. Unfortunately, this means an OpenFlow controller. Analytic models based on queuing
little can be derived from the results in terms of controller theory for the forwarding speed and blocking probability of
performance and behavior or estimation of different bounds current OpenFlow switches can also be used to estimate the
at the moment. For instance, aggregated statistics are gathered performance of the network [343].
for all switches but not for each individual switch. As a result,
it is not possible to identify whether all responses of the
F. Security and dependability
controller are for a single switch, or whether the capacity of
the controller is actually shared among the switches. Flex- Cyber-attacks against financial institutions, energy facilities,
ible OpenFlow controller benchmarks are available though. government units and research institutions are becoming one
OFCBenchmark [293] is one of the recent developments. It of the top concerns of governments and agencies around
creates a set of message-generating virtual switches, which the globe [349], [350], [351], [352], [353], [354]. Different
can be configured independently from each other to emulate incidents, such as Stuxnet [353], have already shown the per-
a specific scenario and to maintain their own statistics. sistence of threat vectors [355]. Put another way, these attacks
Another interesting question to pose when evaluating the are capable of damaging a nation’s wide infrastructure, which
performance of SDN architectures is what is the required represent a significant and concerning issue. As expected,
number of controllers for a given network topology and where one of the most common means of executing those attacks
to place the controllers [347]. By analyzing the performance is through the network, either the Internet or the local area
of controllers in different network topologies, it is possible network. It can be used as a simple transport infrastructure
to conclude that one controller is often enough to keep the for the attack or as a potentialized weapon to amplify the
latency at a reasonable rate [347]. Moreover, as observed in the impact of the attack. For instance, high capacity networks can
same experiments, in the general case adding k controllers to be used to launch large-scale attacks, even though the attacker
the network can reduce the latency by a factor of k. However, has only a low capacity network connection at his premises.
there are cases, such as large scale networks and WANs, where Due to the danger of cyber-attacks and the current landscape
more controllers should be deployed to achieve high reliability of digital threats, security and dependability are top priorities
and low control plane latency. in SDN. While research and experimentation on software-
Recent studies also show that the SDN control plane cannot defined networks is being conducted by some commercial
be fully physically centralized due to responsiveness, reli- players (e.g., Google, Yahoo!, Rackspace, Microsoft), com-
ability and scalability metrics [342]. Therefore, distributed mercial adoption is still in its early stage. Industry experts
controllers are the natural choice for creating a logically believe that security and dependability are issues that need to
centralized control plane, while being capable of coping with be addressed and further investigated in SDN [254], [356],
the demands of large scale networks. However, distributed [357].
controllers bring additional challenges, such as the consistency Additionally, from the dependability perspective, availabil-
of the global network view, which can significantly affect the ity of Internet routers is nowadays a major concern with the
performance of the network if not carefully engineered. Taking widespread of clouds and their strong expectations about the
two applications as examples, one that ignores inconsistencies network [358]. It is therefore crucial to achieve high levels of
and another that takes inconsistency into consideration, it is availability on SDN control platforms once they become pillars
possible to observe that optimality is significantly affected of networked applications. Accordingly, the dependability of
when inconsistencies are not considered and that the robust- software-defined networks cannot be overlooked when we
ness of an application is increased when the controller is aware think about enterprise class deployments.
of the network state distribution [342]. Different threat vectors have already been identified in
Most of these initiatives towards identifying the limita- SDN architectures [254], as well as several security issues
VERSION 1.0 33

TABLE XI
4   5   6   SDN SPECIFIC VS . NON - SPECIFIC THREATS

Threat Specific Consequences in software-defined net-

vectors to SDN? works
Vector 1 no Open door for DDoS attacks.

3  
Vector 2 no Potential attack inflation.
7   Vector 3 yes Exploiting logically centralized controllers.
Vector 4 yes Compromised controller may compromise the
2   entire network.
Vector 5 yes Development and deployment of malicious
applications on controllers.
Vector 6 no Potential attack inflation.
1   Vector 7 no Negative impact on fast recovery and fault
diagnosis.
Fig. 10. Main threat vectors of SDN architectures

ing [359], repudiation [359], information disclosure [359],

and weaknesses in OpenFlow-based networks [359], [360], denial of service [359], [361], [362], and elevation of priv-
[361], [135], [362]. While some threat vectors are common ileges [359]. The lack of isolation, protection, access control
to existing networks, others are more specific to SDN, such and stronger security recommendations [360], [361], [135],
as attacks on control plane communication and logically- [362] are some of the reasons for these vulnerabilities. We
centralized controllers. It is worth mentioning that most threats will explore these next.
vectors are independent of the technology or the protocol (e.g.,
OpenFlow security assessment
OpenFlow, POF, ForCES), because they represent threats on
By applying the STRIDE methodology [363], it is possible
conceptual and architectural layers of SDN itself.
to identify different attacks to OpenFlow-enabled networks.
As shown in Figure 10 and Table XI, there are at least Table XII summarizes these attacks (based on [359]). For
seven identified threats vector in SDN architectures. The first instance, information disclosure can be achieved through side
threat vector consists of forged or faked traffic flows in the channel attacks targeting the flow rule setup process. When
data plane, which can be used to attack forwarding devices reactive flow setup is in place, obtaining information about
and controllers. The second allows an attacker to exploit network operation is relatively easy. An attacker that measures
vulnerabilities of forwarding devices and consequently wreak the delay experienced by the first packet of a flow and the
havoc with the network. Threat vectors three, four and five subsequent can easily infer that the target network is a reactive
are the most dangerous ones, since they can compromise the SDN, and proceed with a specialized attack. This attack –
network operation. Attacks on the control plane, controllers known as fingerprinting [361] – may be the first step to launch
and applications can easily grant an attacker the control of a DoS attack intended to exhaust the resources of the network,
the network. For instance, a faulty or malicious controller or for example. If the SDN is proactive, guessing its forwarding
application could be used to reprogram the entire network for rule policies is harder, but still feasible [359]. Interestingly, all
data theft purposes, e.g., in a data center. The sixth threat reported threats and attacks affect all versions (1.0 to 1.3.1)
vector is linked to attacks on and vulnerabilities in admin- of the OpenFlow specification. It is also worth emphasizing
istrative stations. A compromised critical computer, directly that some attacks, such as spoofing, are not specific to SDN.
connected to the control network, will empower the attacker However, these attacks can have a larger impact in SDNs. For
with resources to launch more easily an attack to the controller, instance, by spoofing the address of the network controller, the
for instance. Last, threat vector number seven represents the attacker (using a fake controller) could take over the control of
lack of trusted resources for forensics and remediation, which the entire network. A smart attack could persist for only a few
can compromise investigations (e.g., forensics analysis) and seconds, i.e., just the time needed to install special rules on
preclude fast and secure recovery modes for bringing the all forwarding devices for its malicious purposes (e.g., traffic
network back into a safe operation condition. cloning). Such attack could be very hard to detect.
As can be observed in Table XI, threat vectors 3 to 5 are Taking counters falsification as another example, an attacker
specific to SDN as they stem from the separation of the control can try to guess installed flow rules and, subsequently, forge
and data planes and the consequent introduction of a new packets to artificially increase the counter. Such attack would
entity in these networks — the logically centralized controller. be specially critical for billing and load balancing systems, for
The other vectors were already present in traditional networks. instance. A customer could be charged for more traffic than
However, the impact of these threats could be larger than today she in fact used, while a load balancing algorithm may take
— or at least it may be expressed differently — and as a non-optimal decisions due to forged counters.
consequence it may need to be dealt with differently. Other conceptual and technical security concerns in Open-
OpenFlow networks are subject to a variety of security Flow networks include the lack of strong security recommen-
and dependability problems such as spoofing [359], tamper- dations for developers, the lack of TLS and access control
VERSION 1.0 34

TABLE XII
ATTACKS TO O PEN F LOW NETWORKS .

Attack Security Property Examples

Spoofing Authentication MAC and IP address spoofing, forged ARP and IPv6 router advertisement
Tampering Integrity Counter falsification, rule installation, modification affecting data plane.
Repudiation Non-repudiation Rule installation, modification for source address forgery.
Information disclosure Confidentiality Side channel attacks to figure out flow rule setup.
Denial of service Availability Flow requests overload of the controller.
Elevation of privilege Authorization Controller take-over exploiting implementation flaws.

TABLE XIII
support on most switch and controller implementations [360], C OUNTERMEASURES FOR SECURITY THREATS IN O PEN F LOW NETWORKS .
the belief that TCP is enough because links are “physically
secure” [362], [360], the fact that many switches have listener
Measure Short description
mode activated by default (allowing the establishment of
Access control Provide strong authentication and authorization
malicious TCP connections, for instance) [362] or that flow mechanisms on devices.
table verification capabilities are harder to implement when Attack detection Implement techniques for detecting different types
TLS is not in use [360], [187]. In addition, it is worth of attacks.
mentioning the high denial of service risk posed to centralized Event filtering Allow (or block) certain types of events to be
controllers [361], [187], the vulnerabilities in the controllers handled by special devices.
themselves [187], [254], and the risk of resource depletion Firewall and IPS Tools for filtering traffic, which can help to prevent
attacks [361], [362]. For instance, it has been shown that an different types of attacks.
attacker can easily compromise control plane communications Flow aggregation Coarse-grained rules to match multiple flows to
prevent information disclosure and DoS attacks.
through DoS attacks and launch a resource depletion attack on
control platforms by exploiting a single application such as a Forensics support Allow reliable storage of traces of network activities
to find the root causes of different problems.
learning switch [362], [361].
Intrusion tolerance Enable control platforms to maintain correct oper-
Countermeasures for OpenFlow based SDNs ation despite intrusions.
Several countermeasures can be put in place to mitigate the Packet dropping Allow devices to drop packets based on security
policy rules or current system load.
security threats in SDNs. Table XIII summarizes a number of
countermeasures that can be applied to different elements of Rate limiting Support rate limit control to avoid DoS attacks on
the control plane.
an SDN/OpenFlow-enabled network. Some of these measures,
Shorter timeouts Useful to reduce the impact of an attack that diverts
namely rate limiting, event filtering, packet dropping, shorter traffic.
timeouts, and flow aggregation, are already recommended in
more recent versions of the OpenFlow specification (version
1.3.1 and later). However, most of them are not yet supported
or implemented in SDN deployments. network to make the controller install rules that divert traffic
Traditional techniques such as access control, attack detec- to a malicious machine. With reduced timeouts, the attacker
tion mechanisms, event filtering (e.g., controller decides which would be forced to constantly generate a number of forged
asynchronous messages he is not going to accept), firewalls, packets to avoid timeout expiration, making the attack more
and intrusion detection systems, can be used to mitigated likely to be detected. Rate limiting and packet dropping can be
the impact of or avoid attacks. They can be implemented in applied to avoid DoS attacks on the control plane or stop on-
different devices, such as controllers, forwarding devices, mid- going attacks directly on the data plane by installing specific
dleboxes, and so forth. Middleboxes can be a good option for rules on the devices where the attacks is being originated.
enforcing security policies in an enterprise because they are (in Forensics and remediation encompass mechanisms such as
general) more robust and special purpose (high performance) secure logging, event correlation and consistent reporting. If
devices. Such a strategy also reduces the potential overhead anything wrong happens with the network, operators should
cause by implementing these countermeasures directly on be able to safely figure out the root cause of the problem
controllers or forwarding devices. However, middleboxes can and put the network to work on a secure operation mode as
add extra complexity to the network management, i.e., increase fast as possible. Additionally, techniques to tolerate faults and
the OPEX at the cost of a better performance. intrusions, such as state machine replication [364], proactive-
Rate limiting, packet dropping, shorter timeouts and flow reactive recovery [365], and diversity [147], can be added to
aggregations are techniques that can be applied on controlled control platforms for increasing the robustness and security
and forwarding devices to mitigate different types of attacks, properties by automatically masking and removing faults. Put
such as denial-of-service and information disclosure. For in- differently, SDN controllers should be able to resist against
stance, reduced timeouts can be used to mitigate the effect different types of events (e.g., power outages, network dis-
of an attack exploring the reactive operation mode of the ruption, communication failures, network partitioning) and
VERSION 1.0 35

attacks (e.g., DDoS, resource exhaustion) [254], [150]. One the transport and IP network divisions of service providers,
of the most traditional ways of achieving high availability or the system administrator, storage, networking, and security
is through replication. Yet, proactive-reactive recover and teams of enterprise organizations. Such a challenge is observ-
diversity are two examples of crucial techniques that add value able on today’s virtualized data centers, through the shift in
to the system for resisting against different kinds of attacks role and decision power between the networking and server
and failures (e.g., those exploring common vulnerabilities or people. Similarly, the development and operations (DevOps)
caused by software aging problems). movement has caused a shift in the locus of influence, not only
Other countermeasures to address different threats and is- on the network architecture but also on purchasing, and this
sues of SDN include enhancing the security and dependability is an effect that SDN may exacerbate. These changes in role
of controllers, protection and isolation of applications [356], and power causes a second order effect on the sales division
[254], [135], trust management between controllers and for- of vendors that are required to adapt accordingly.
warding devices [254], integrity checks of controllers and Pioneering SDN operational deployments have been mainly
applications [254], forensics and remediation [356], [254], greenfield scenarios and/or tightly controlled single admin-
verification frameworks [366], [135], [367], and resilient con- istrative domains. Initial roll-out strategies are mainly based
trol planes [368], [367], [254], [356]. Protection and isolation on virtual switch overlay models or OpenFlow-only network-
mechanisms should be part of any controller. Applications wide controls. However, a broader adoption of SDN beyond
should be isolated from each other and from the controller. data center silos – and between themselves – requires con-
Different techniques such as security domains (e.g., kernel, sidering the interaction and integration with legacy control
security, and user level) and data access protection mecha- planes providing traditional switching; routing; and operation,
nisms should be put in place in order to avoid security threats administration, and management (OAM) functions. Certainly,
from management applications. rip-and-replace is not a viable strategy for the broad adoption
Implementing trust between controllers and forwarding is of new networking technologies.
another requirement for insuring that malicious elements can- Hybrid networking in SDN should allow deploying Open-
not harm the network without being detected. An attacker Flow for a subset of all flows only, enable OpenFlow on a
can try to spoof the IP address of the controller and make subset of devices and/or ports only, and provide options to
switches connect to its own controller. This is currently interact with existing OAM protocols, legacy devices, and
the case since most controllers and switches only establish neighboring domains. As in any technology transition period
insecure TCP connections. Complementarly, integrity checks where fork-lift upgrades may not be a choice for many,
on controller and application software can help to ensure that migration paths are critical for adoption.
safe code is being bootstrapped, which eliminates harmful Hybrid networking in SDN spans several levels. The Mi-
software from being started once the system restarts. Besides gration Working Group of the ONF is tackling the scenario
integrity checks, other things such as highly specialized mal- where hybrid switch architectures and hybrid (OpenFlow
ware detection systems should be developed for SDN. Third- and non-OpenFlow) devices co-exist. Hybrid switches can be
party management applications should always be scanned for configured to behave as a legacy switch or as an OpenFlow
bad code and vulnerabilities because a malicious application switch and, in some cases, as both simultaneously. This can
represents a significant security threat to the network. be achieved, for example, by partitioning the set of ports of a
It is worth mentioning that there are also other approaches switch, where one subset is devoted to OpenFlow-controlled
for mitigating security threats in SDN, such as declarative networks, and the other subset to legacy networks. For these
languages to eliminate network protocol vulnerabilities [192]. subsets to be active at the same time, each one having its
This kind of descriptive languages can specify semantic con- own data plane, multi-table support at the forwarding engine
straints, structural constraints and safe access properties of (e.g., via TCAM partitioning) is required. Besides port-based
OpenFlow messages. Then, a compiler can use these inputs partitioning, it is also possible to rely on VLAN-based (prior
to find programmers’ implementation mistakes on message to entering the OpenFlow pipeline) or flow-based partitioning
operations. In other words, such languages can help find using OpenFlow matching and the LOCAL and/or NORMAL
and eliminate implementation vulnerabilities of southbound actions to redirect packets to the legacy pipeline or the switch’s
specifications. local networking stack and its management stack. Flow-based
partitioning is the most flexible option, as it allows each
packet entering a switch to be classified by an OpenFlow
G. Migration to SDN flow description and treated by the appropriate data plane
A prime SDN adoption challenge relates to organizational (OpenFlow or legacy).
barriers that may arise due to the first (and second) order The promises by SDN to deliver easier design, operation and
effects of SDN automation capabilities and “layer/domain management of computer networks are endangered by chal-
blurring”. Some level of human resistance is to be expected lenges regarding incremental deployability, robustness, and
and may affect the decision and deployment processes of SDN, scalability. Full SDN deployments are difficult and straightfor-
especially by those that may regard the control refactorization ward only in some green field deployments such as data center
of SDN as a risk to the current chain of control and command, networks or by means of an overlay model approach. Hybrid
or even to their job security. This complex social challenge SDN approaches represent however a very likely deployment
is similar (and potentially larger) to known issues between model that can be pursued by different means, including [369]:
VERSION 1.0 36

• Topology-based hybrid SDN: Based on a topological networks through network orchestration under strict budget
separation of the nodes controlled by traditional and SDN constraints. The proposed architecture includes policy config-
paradigms. The network is partitioned in different zones urations, troubleshooting and maintenance tasks establishing
and each node belongs to only one zone. transitional networks (SDN and legacy) in structures called
• Service-based hybrid SDN: Conventional networks and Solitary Confinement Trees (SCTs), where VLAN IDs are
SDN provide different services, where overlapping nodes, efficiently used by orchestration algorithms to build paths in
controlling a different portion of the FIB (or generalized order to steer traffic through SDN switches. Defying the partial
flow table) of each node. Examples include network-wide SDN implementation concept, they confirm that this could
services like forwarding that can be based on legacy be a long-term operational strategy solution for enterprise
distributed control, while SDN provides edge-to-edge networks.
services such as enforcement of traffic engineering and HybNET [373] presents a network management framework
access policies, or services requiring full traffic visibility for hybrid OpenFlow-legacy networks. It provides a common
(e.g., monitoring). centralized configuration interface to build virtual networks us-
• Class-based hybrid SDN: Based on the partition of traffic ing VLANs. An abstraction of the physical network topology
in classes, some controlled by SDN and the remaining is taken into account by a centralized controller that applies a
by legacy protocols. While each paradigm controls a path finder mechanism, in order to calculate network paths and
disjoint set of node forwarding entries, each paradigm program the OpenFlow switches via REST interfaces [136]
is responsible for all network services for the assigned and legacy devices using NETCONF [143].
traffic classes.
• Integrated hybrid SDN: A model where SDN is respon- H. SDN for telecom and cloud providers
sible for all the network services, and uses traditional A number of carrier-grade infrastructure providers (e.g.,
protocols (e.g., BGP) as an interface to node FIBs. For NTT, AT&T, Verizon, Deutsch Telekom) have already joined
example, it can control forwarding paths by injecting the SDN community and its activities with the ultimate goal of
carefully selected routes into a routing system or adjust- solving their long standing networking problems. One of the
ing protocol settings (e.g., IGP weights). Past efforts on forefront runners (and early SDN adopter) was NTT, already
RCPs [37] and the ongoing efforts within ODL [13] can taking advantage o this new paradigm to provide new on-
be considered examples of this hybrid model. demand network provisioning models. In 2013, NTT launched
In general, benefits of hybrid approaches include enabling an SDN-based, on-demand elastic provisioning platform of
flexibility (e.g., easy match on packet fields for middlebox- network resources (e.g., bandwidth) for HD video broadcast-
ing) and SDN-specific features (e.g., declarative management ers [374]. Similarly, as a global cloud provider with data
interface) while partially keeping the inherited characteristics centers spread across the globe [375], the same company
of conventional networking such as robustness, scalability, launched a similar service for its cloud customers, who are
technology maturity, and low deployment costs. On the neg- now capable of taking advantage of dynamic networking pro-
ative side, the drawbacks of hybridization include the need visioning intra- and inter-data centers [376]. AT&T is another
for ensuring profitable interactions between the networking telecom company that is investing heavily in new services,
paradigms (SDN and traditional) while dealing with the het- such as user-defined network clouds, that take advantage of
erogeneity that largely depends on the model. recent developments in NFV and SDN [377]. These are some
Initial trade-off analyses suggest that the combination of of the early examples of the opportunities SDNs seem to bring
centralized and distributed paradigms may provide mutual to telecom and cloud providers.
benefits. However, future work is required to devise techniques Carrier networks are using the SDN paradigm as the
and interaction mechanisms that maximize such benefits while technology means for solving a number of long standing
limiting the added complexity of the paradigm coexistence. problems. Some of these efforts include new architectures for
Some efforts have been already devoted to the challenges a smooth migration from the current mobile core infrastructure
of migration and hybrid SDNs. RouteFlow [370] implements to SDN [154], and techno-economic models for virtualization
an IP level control plane on top of an OpenFlow network, of these networks [378], [379]; carrier-grade OpenFlow virtu-
allowing the underlying devices to act as IP routers under alization schemes [380], [64], including virtualized broadband
different possible arrangements. LegacyFlow [371] extends access infrastructures [381], techniques that are allowing the
the OpenFlow-based controlled network to embrace non- offer of network-as-a-service [382]; flexible control of network
OpenFlow nodes. The common grounds of these pieces of resources [383], including offering MPLS services using an
work are (1) considering hybrid as the coexistence of tradi- SDN approach [384]; and the investigation of novel network
tional environments of closed vendor’s routers and switches architectures, from proposals to separate the network edge
with new OpenFlow-enabled devices; (2) targeting the inter- from the core [385], [386], with the latter forming the fabric
connection of both control and data planes of legacy and new that transports packets as defined by an intelligent edge, to
network elements; and (3) taking a controller-centric approach, software-defined Internet exchange points [387], [388].
drawing the hybrid line outside of any device itself, but into SDN technology also brings new possibilities for cloud
the controller application space. providers. By taking advantage of the logically centralized
Panopticon [372] defines an architecture and methodol- control of network resources [389], [8] it is possible to sim-
ogy to consistently implement SDN inside enterprise legacy plify and optimize network management of data centers and
VERSION 1.0 37

achieve: (i) efficient intra-datacenter networking, including fast • Software-Defined Compute (SDC) [117], and
recovery mechanisms for the data and control planes [331], • Software-Defined Management (SDM) [404].
[390], [391], simplified fault-tolerant routing [392], perfor- In the last decade the advances in virtualization of compute
mance isolation [393], and easy and efficient resource migra- and storage, together with the availability of sophisticated
tion (e.g., of VMs and virtual networks) [331]; (ii) improved cloud orchestration tools have enabled SDS, SDC and SDM.
inter-datacenter communication, including the ability to fully These architectural components have been widely used by
utilize the expensive high-bandwidth links without impairing cloud providers and for building IT infrastructures in different
quality of service [8], [394]; (iii) higher levels of reliability enterprise environments. However, the lack of programmable
(with novel fault management mechanisms, etc.) [392], [331], network control has so far hindered the realization of a
[390]; and (iv) cost reduction by replacing complex, expensive complete Software-Defined Environment. SDN is seen as the
hardware by simple and cheaper forwarding devices [395], [8]. technology that may fill this gap, as attested by the emergence
Table XIV summarizes some of the carrier-grade network of cloud-scale network virtualization platforms based on this
and cloud infrastructure providers’ requirements. In this table new paradigm [64].
we show the current challenges and what is to be expected
with SDN. As we saw before, some of the expectations are Business  Need  
already becoming a reality, but many are still open issues.
What seems to be clear is that SDN represents an opportunity
for telecom and cloud providers, in providing flexibility, cost- So<ware  Defined   Workload  Defini2on,  
Management   Orchestra2on,  and  Op2miza2on  
effectiveness, and easier management of their networks.
Service  Delivery  
Opera2onal  
I. SDN: the missing piece towards Software-Defined Environ- So<ware  Defined  Environments  
Level  
ments Agreement  

The convergence of different technologies is enabling the So<ware  Defined   So<ware  Defined   So<ware  Defined  
emergence of fully programmable IT infrastructures. It is Network   Compu2ng   Storage  

already possible to dynamically and automatically configure

or reconfigure the entire IT stack, from the network infras- Fig. 11. Overview of an IT infrastructure based on a SDE.
tructure up to the applications, to better respond to workload
changes. Recent advances makes on-demand provisioning of The IBM SmartCloud Orchestrator is one of the first
resources possible, at nearly all infrastructural layers. The fully examples of an SDE [118], [117]. It integrates compute,
automated provisioning and orchestration of IT infrastruc- storage, management and networking in a structured way
tures as been recently named Software-Defined Environments Figure 11 gives a simplified overview of an SDE, by taking
(SDEs) [117], [118], by IBM. This is a novel approach that the approach developed by IBM as its basis. The main idea
is expected to have significant potential in simplifying IT of an SDE-based infrastructure is that the business needs
management, optimizing the use of the infrastructure, reduce that define the workloads trigger the reconfiguration of the
costs, and reduce the time to market of new ideas and products. global IT infrastructure (compute, storage, network). This is an
In an SDE, workloads can be easily and automatically assigned important step towards a more customizable IT infrastructure
to the appropriate IT resources based on application character- that focuses on the business requirements rather than on the
istics, security and service level policies, and the best-available limitations of the infrastructure itself.
resources to deliver continuous, dynamic optimization and
reconfiguration to address infrastructure issues in a rapid VI. C ONCLUSION
and responsive manner. Table XV summarizes the traditional Traditional networks have always been complex and hard
approaches and some of the key features being enabled by to manage. One of the reasons is that the control and data
SDEs [400], [401]. planes are vertically integrated and vendor specific. Another,
In an SDE the workloads are managed independently of concurring reason, is that typical networking devices are also
the systems and underlying infrastructure, i.e., are not tied to tightly tied to line products and versions. In other words, each
a specific technology or vendor [118], [117]. Another charac- line of product may have its own particular configuration and
teristic of this new approach is to offer a programmatic access management interfaces, implying long cycles for producing
to the environment as a whole, selecting the best available product updates (e.g., new firmware) or upgrades (e.g., new
resources based on the current status of the infrastructure, versions of the devices). All this has given rise to vendor
and enforcing the policies defined. In this sense, it shares lock-in problems for network infrastructure owners, as well
much of the philosophy of SDN. Interestingly, one of the as posing severe restrictions to change and innovation.
missing key pieces of an SDE was, until now, Software- Software-Defined Networking (SDN) created an opportunity
Defined Networking. for solving these long-standing problems. Some of the key
The four essential building blocks of an SDE [118], [117], ideas of SDN are the introduction of dynamic programmability
[401] are: in forwarding devices through open southbound interfaces,
• Software-Defined Networks (SDN) [402], [403], the decoupling of the control and data plane, and the global
• Software-Defined Storage (SDS) [400], view of the network by logical centralization of the “network
VERSION 1.0 38

TABLE XIV
C ARRIER - GRADE AND CLOUD PROVIDER EXPECTATIONS & CHALLENGES

What Currently Expected with SDN

Complex load balancing configuration. Automatic load balancing reconfiguration. [396], [8]
Low virtualization capabilities across hardware plat- NFV for virtualizing network functionality across hardware appli-
forms ances. [395], [377]
Hard and costly to provide new services. Create and deploy new network service quickly. [395], [377]
Resource
Provisioning No bandwidth on demand. Automatic bandwidth on demand. [379]
Per network element scaling. Better incremental scaling. [396], [390]
Resources statically pre-provisioned. Dynamic resource provisioning in response to load. [396], [8], [395],
[378], [389]
All traffic is filtered. Only targeted traffic is filtered. [396]
Fixed only. Fixed and mobile. [396]
Traffic Steering
Per network element scaling. Better incremental scaling. [378], [390]
Statically configured on a per-device basis. Dynamically configurable. [8], [379], [397]
All traffic from all probes collected. Only targeted traffic from targeted probes is collected.
Massive bandwidth required. Efficient use of bandwidth. [8], [379]
Ad Hoc Topologies
Per network element scaling. Better incremental scaling. [396], [379]
Statically configured. Dynamically configured. [396], [398], [374]
Complex configuration, management and upgrade. Simplified management and upgrade. [8], [396], [395], [379], [390]
Different kinds of routers, such as CO. No need for CO routers, reducing aggregation costs. [396], [395], [378]
Manual provisioning. Automated provisioning. [396], [379], [397]
Managed Router
On-premises router deployment. Virtual routers (either on-site or not). [379], [396], [378]
Services
Operational burden to support different equipments. Reduced technology obsolescence. [378]
Router change-out as technology or needs change. Pay-as-you grow CAPEX model. [378]
Systems complex and hard to integrate. Facilitates simplified system integrations. [396], [395], [398]
Fixed long term contracts. More flexible and on-demand contracts. [379], [383]
Revenue Models
Traffic consumption. QoS metrics per-application. [379], [390], [390], [399]
Composition of services is hard to implement. Easily expand functionality to meet the infrastructure needs. [395]
Middleboxes Determine where to place middleboxes a priori (e.g., Dynamic placement using shortest or least congested path. [206], [399],
Deployment & large path inflation problems). [398]
Management
Excessive over-provisioning to anticipate demands. Scale up to meet demands, and scale down to conserve resources (elastic
middleboxes). [396], [378]
Energy saving strategies are hard to implement. Flexible and easy to deploy energy saving strategies. [390]
Other Issues Complex and static control and data plane restoration Automated and flexible restoration techniques for both control and data
techniques. plane. [390]

TABLE XV
SDE PUSHING IT TO THE NEXT FRONTIER

Traditionally Expected with SDEs

IT operations manually map the resources for apps for software deployment. Software maps resources to the workload and deploys the workload.
Networks are mostly statically configured and hard to change. Networks are virtualized and dynamically configured on-demand.
Optimization and reconfiguration to reactively address issues are manual. Analytics-based optimization and reconfiguration of infrastructure issues.
Workloads are typically manually assigned to resources. Workloads are dynamically assigned.

brain”. While data plane elements became dumb, but highly paradigm shift in the development and evolution of networks,
efficient and programmable packet forwarding devices, the introducing a whole new world of possibilities and a new pace
control plane elements are now represented by a single en- of innovation in networking infrastructures.
tity, the controller or network operating system. Applications
implementing the network logic run on top of the controller In spite of recent and interesting attempts to survey this
and are much easier to develop and deploy when compared new chapter in the history of networks [405], [406], [407],
to traditional networks. Given the global view, consistency of the literature was still lacking, to the best of our knowledge, a
policies is straightforward to enforce. SDN represented a major single extensive and comprehensive overview of the building
blocks, concepts and challenges of SDNs. Trying to address
VERSION 1.0 39

this gap, the present paper used a layered approach to method- [6] H. Kim and N. Feamster, “Improving network management with soft-
ically dissect the state of the art in terms of concepts, ideas ware defined networking,” Communications Magazine, IEEE, vol. 51,
no. 2, pp. 114–119, 2013.
and components of software-defined networking, covering a [7] T. Koponen, M. Casado, N. Gude, J. Stribling, L. Poutievski, M. Zhu,
broad range of existing solutions, as well as future directions. R. Ramanathan, Y. Iwata, H. Inoue, T. Hama, and S. Shenker, “Onix:
We started by comparing this new paradigm with traditional a distributed control platform for large-scale production networks,”
in Proceedings of the 9th USENIX conference on Operating systems
networks and discussing how academy and industry helped design and implementation, ser. OSDI’10. Berkeley, CA, USA:
shape software-defined networking. Following a bottom-up USENIX Association, 2010, pp. 1–6.
approach, we provided an in-depth overview of what we [8] S. Jain, A. Kumar, S. Mandal, J. Ong, L. Poutievski, A. Singh,
S. Venkata, J. Wanderer, J. Zhou, M. Zhu, J. Zolla, U. Hölzle, S. Stuart,
consider the eight fundamental facets of the SDN problem: and A. Vahdat, “B4: experience with a globally-deployed software
1) hardware infrastructure, 2) southbound interfaces, 3) net- defined wan,” in Proceedings of the ACM SIGCOMM 2013 conference
work virtualization (hypervisor layer between the forward- on SIGCOMM, ser. SIGCOMM ’13. New York, NY, USA: ACM,
2013, pp. 3–14.
ing devices and the network operating systems), 4) network [9] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson,
operating systems (SDN controllers and control platforms), J. Rexford, S. Shenker, and J. Turner, “Openflow: enabling innovation
5) northbound interfaces (common programming abstractions in campus networks,” SIGCOMM Comput. Commun. Rev., vol. 38,
no. 2, pp. 69–74, Mar. 2008.
offered to network applications), 6) virtualization using slicing [10] ONF, “Open networking foundation,” 2014. [Online]. Available:
techniques provided by special purpose libraries and/or pro- https://www.opennetworking.org/
gramming languages and compilers, 7) network programming [11] S. Yeganeh, A. Tootoonchian, and Y. Ganjali, “On scalability
of software-defined networking,” Communications Magazine, IEEE,
languages, and finally, 8) management applications. vol. 51, no. 2, pp. 136–141, 2013.
SDN has successfully managed to pave the way towards [12] VMware, Inc., “VMware NSX Virtualization Platform,” 2013.
next generation networking, spawning an innovative research [Online]. Available: https://www.vmware.com/products/nsx/
[13] OpenDaylight, “OpenDaylight: A Linux Foundation Collaborative
and development environment, promoting advances in sev- Project,” 2013. [Online]. Available: http://www.opendaylight.org
eral areas: switch and controller platform design, evolution [14] N. Feamster, J. Rexford, and E. Zegura, “The road to SDN,” Queue,
of scalability and performance of devices and architectures, vol. 11, no. 12, pp. 20:20–20:40, Dec. 2013.
[15] R. Presuhn, “Version 2 of the Protocol Operations for the Simple
promotion of security and dependability. Network Management Protocol (SNMP),” RFC 3416 (INTERNET
We will continue to witness extensive activity around SDN STANDARD), Internet Engineering Task Force, Dec. 2002. [Online].
in the near future. Emerging topics requiring further research Available: http://www.ietf.org/rfc/rfc3416.txt
[16] N. Feamster and H. Balakrishnan, “Detecting BGP configuration faults
are, for example: the migration path to SDN, extending SDN with static analysis,” in Proceedings of the 2nd conference on Sympo-
towards carrier transport networks, realization of the network- sium on Networked Systems Design & Implementation - Volume 2, ser.
as-a-service cloud computing paradigm, or software-defined NSDI’05. Berkeley, CA, USA: USENIX Association, 2005, pp. 43–
56.
environments (SDE). [17] R. Barrett, S. Haar, and R. Whitestone, “Routing snafu causes internet
outage,” Interactive Week, 1997.
[18] K. Butler, T. Farley, P. McDaniel, and J. Rexford, “A survey of bgp
ACKNOWLEDGMENT security issues and solutions,” Proceedings of the IEEE, vol. 98, no. 1,
pp. 100–122, Jan 2010.
The authors would like to thank Jennifer Rexford for her [19] J. Sherry and S. Ratnasamy, “A survey of enterprise middlebox
feedback on an early version of this work and encouragement deployments,” EECS Department, University of California, Berkeley,
to get it finished, Srini Seetharaman for reviewing the draft Tech. Rep. UCB/EECS-2012-24, Feb 2012.
[20] K. Greene, “MIT Tech Review 10 Breakthrough Technologies:
and providing inputs to alternative SDN views, David Meyer Software-defined Networking,” http://www2.technologyreview.com/
for his inspiration on the organizational challenges that are article/412194/tr10-software-defined-networking/, 2009.
part of the migration path towards SDN, Thomas Nadeau [21] H. Alkhatib, P. Faraboschi, E. Frachtenberg, H. Kasahara, D. Lange,
P. Laplante, A. Merchant, D. Milojicic, and K. Schwan, “IEEE CS 2022
for his inputs on the OpenDaylight initiative, Raphael Rosa report (draft),” IEEE Computer Society, Tech. Rep., February 2014.
and Regivaldo Costa for their various contributions, and the [22] A. Doria, J. H. Salim, R. Haas, H. Khosravi, W. Wang, L. Dong,
anonymous reviewers. R. Gopal, and J. Halpern, “Forwarding and Control Element Separation
(ForCES) Protocol Specification,” Internet Engineering Task Force,
Mar. 2010. [Online]. Available: http://www.ietf.org/rfc/rfc5810.txt
R EFERENCES [23] H. Song, “Protocol-oblivious Forwarding: Unleash the power of SDN
through a future-proof forwarding plane,” in Proceedings of the Sec-
[1] T. Benson, A. Akella, and D. Maltz, “Unraveling the complexity of ond ACM SIGCOMM Workshop on Hot Topics in Software Defined
network management,” in Proceedings of the 6th USENIX Symposium Networking, ser. HotSDN ’13. New York, NY, USA: ACM, 2013, pp.
on Networked Systems Design and Implementation, ser. NSDI’09, 127–132.
Berkeley, CA, USA, 2009, pp. 335–348. [24] T. D. Nadeau and K. Gray, SDN : software defined networks.
[2] B. Raghavan, M. Casado, T. Koponen, S. Ratnasamy, A. Ghodsi, Sebastopol: O’Reilly, 2013. [Online]. Available: http://opac.inria.fr/
and S. Shenker, “Software-defined internet architecture: Decoupling record=b1135288
architecture from infrastructure,” in Proceedings of the 11th ACM [25] R. Alimi, R. Penno, and Y. Yang, “ALTO Protocol,” Internet Draft,
Workshop on Hot Topics in Networks, ser. HotNets-XI. New York, Internet Engineering Task Force, March 2014. [Online]. Available:
NY, USA: ACM, 2012, pp. 43–48. https://datatracker.ietf.org/doc/draft-ietf-alto-protocol/
[3] A. Ghodsi, S. Shenker, T. Koponen, A. Singla, B. Raghavan, and [26] IETF I2RS Working Group, “Interface to the routing system
J. Wilcox, “Intelligent design enables architectural evolution,” in Pro- (I2RS),” Internet Engineering Task Force, 2014. [Online]. Available:
ceedings of the 10th ACM Workshop on Hot Topics in Networks, ser. http://datatracker.ietf.org/wg/i2rs/charter/
HotNets-X. New York, NY, USA: ACM, 2011, pp. 3:1–3:6. [27] D. King and A. Farrel, “A PCE-based architecture for application-
[4] N. Mckeown, “How SDN will Shape Networking,” October 2011. based network operations,” Internet Engineering Task Force, Feb 2014.
[Online]. Available: http://www.youtube.com/watch?v=c9-K5O qYgA [Online]. Available: http://datatracker.ietf.org/doc/draft-farrkingel-pce-
[5] S. Schenker, “The Future of Networking, and the Past of Protocols,” abno-architecture/
October 2011. [Online]. Available: http://www.youtube.com/watch?v= [28] N. M. K. Chowdhury and R. Boutaba, “A survey of network virtual-
YHeyuD89n1Y ization,” Computer Networks, vol. 54, no. 5, pp. 862 – 876, 2010.
VERSION 1.0 40

[29] R. Jain and S. Paul, “Network virtualization and software defined [51] M. Casado, T. Garfinkel, A. Akella, M. J. Freedman, D. Boneh,
networking for cloud computing: a survey,” Communications Magazine, N. McKeown, and S. Shenker, “SANE: a protection architecture for
IEEE, vol. 51, no. 11, pp. 24–31, 2013. enterprise networks,” in Proceedings of the 15th conference on USENIX
[30] A. Corradi, M. Fanelli, and L. Foschini, “VM consolidation: A real Security Symposium - Volume 15, ser. USENIX-SS’06, Berkeley, CA,
case based on openstack cloud,” Future Generation Computer Systems, USA, 2006.
vol. 32, no. 0, pp. 118 – 127, 2014. [52] M. Casado, M. J. Freedman, J. Pettit, J. Luo, N. McKeown, and
[31] A. Shang, J. Liao, and L. Du, “Pica8 Xorplus,” 2014. [Online]. S. Shenker, “Ethane: taking control of the enterprise,” in Proceedings of
Available: http://sourceforge.net/projects/xorplus/ the 2007 conference on Applications, technologies, architectures, and
[32] P. Jakma and D. Lamparter, “Introduction to the quagga routing suite,” protocols for computer communications, ser. SIGCOMM ’07. New
Network, IEEE, vol. 28, no. 2, pp. 42–48, March 2014. York, NY, USA: ACM, 2007, pp. 1–12.
[33] “NetFPGA,” 2014. [Online]. Available: http://netfpga.org/ [53] N. Gude, T. Koponen, J. Pettit, B. Pfaff, M. Casado, N. McKeown, and
[34] J. Martins, M. Ahmed, C. Raiciu, V. Olteanu, M. Honda, R. Bifulco, S. Shenker, “NOX: towards an operating system for networks,” Comp.
and F. Huici, “Clickos and the art of network function virtualization,” Comm. Rev., 2008.
in 11th USENIX Symposium on Networked Systems Design and Imple- [54] M. Macedonia and D. Brutzman, “Mbone provides audio and video
mentation (NSDI 14). Seattle, WA: USENIX Association, Apr. 2014, across the internet,” Computer, vol. 27, no. 4, pp. 30–36, 1994.
pp. 459–473. [55] R. Fink and R. Hinden, “6bone (IPv6 Testing Address Allocation)
[35] D. Tennenhouse, J. Smith, W. Sincoskie, D. Wetherall, and G. Minden, Phaseout,” RFC 3701 (Informational), Internet Engineering Task Force,
“A survey of active network research,” Communications Magazine, Mar. 2004. [Online]. Available: http://www.ietf.org/rfc/rfc3701.txt
IEEE, vol. 35, no. 1, pp. 80–86, 1997. [56] D. Andersen, H. Balakrishnan, F. Kaashoek, and R. Morris, “Resilient
[36] D. Sheinbein and R. P. Weber, “800 service using SPC network overlay networks,” SIGOPS Oper. Syst. Rev., vol. 35, no. 5, pp. 131–
capability,” The Bell System Technical Journal, vol. 61, no. 7, Sep. 145, Oct. 2001.
1982. [57] B. Chun, D. Culler, T. Roscoe, A. Bavier, L. Peterson, M. Wawrzoniak,
[37] M. Caesar, D. Caldwell, N. Feamster, J. Rexford, A. Shaikh, and and M. Bowman, “Planetlab: An overlay testbed for broad-coverage
J. van der Merwe, “Design and implementation of a routing control services,” SIGCOMM Comput. Commun. Rev., vol. 33, no. 3, pp. 3–
platform,” in Proceedings of the 2nd conference on Symposium on Net- 12, Jul. 2003.
worked Systems Design & Implementation - Volume 2, ser. NSDI’05. [58] T. Anderson, L. Peterson, S. Shenker, and J. Turner, “Overcoming the
Berkeley, CA, USA: USENIX Association, 2005, pp. 15–28. internet impasse through virtualization,” Computer, vol. 38, no. 4, pp.
[38] D. L. Tennenhouse and D. J. Wetherall, “Towards an active network 34–41, April 2005.
architecture,” SIGCOMM Comput. Commun. Rev., vol. 37, no. 5, pp. [59] L. Peterson, T. Anderson, D. Blumenthal, D. Casey, D. Clark, D. Estrin,
81–94, Oct. 2007. J. Evans, D. Raychaudhuri, M. Reiter, J. Rexford, S. Shenker, and
J. Wroclawski, “Geni design principles,” Computer, vol. 39, no. 9, pp.
[39] B. Schwartz, A. Jackson, W. Strayer, W. Zhou, R. Rockwell, and C. Par-
102–105, Sept 2006.
tridge, “Smart packets for active networks,” in Open Architectures and
[60] A. Bavier, N. Feamster, M. Huang, L. Peterson, and J. Rexford,
Network Programming Proceedings, 1999. OPENARCH’99. 1999 IEEE
“In VINI veritas: realistic and controlled network experimentation,”
Second Conference on, Mar 1999, pp. 90–97.
SIGCOMM Comput. Commun. Rev., vol. 36, no. 4, pp. 3–14, Aug.
[40] D. Wetherall, J. V. Guttag, and D. Tennenhouse, “Ants: a toolkit
2006.
for building and dynamically deploying network protocols,” in Open
[61] B. Pfaff, J. Pettit, T. Koponen, K. Amidon, M. Casado, and S. Shenker,
Architectures and Network Programming, 1998 IEEE, Apr 1998, pp.
“Extending networking into the virtualization layer,” in Proc. of work-
117–129.
shop on Hot Topics in Networks (HotNets-VIII), 2009.
[41] D. Alexander, W. Arbaugh, M. Hicks, P. Kakkar, A. Keromytis, [62] B. Lantz, B. Heller, and N. McKeown, “A network in a laptop: rapid
J. Moore, C. Gunter, S. Nettles, and J. Smith, “The switchware active prototyping for software-defined networks,” in Proceedings of the 9th
network architecture,” Network, IEEE, vol. 12, no. 3, pp. 29–36, May ACM SIGCOMM Workshop on Hot Topics in Networks, ser. Hotnets-
1998. IX. New York, NY, USA: ACM, 2010, pp. 19:1–19:6.
[42] K. Calvert, S. Bhattacharjee, E. Zegura, and J. Sterbenz, “Directions [63] R. Sherwood, G. Gibb, K.-K. Yap, G. Appenzeller, M. Casado,
in active networks,” Communications Magazine, IEEE, vol. 36, no. 10, N. McKeown, and G. Parulkar, “Can the production network be the
pp. 72–78, Oct 1998. testbed?” in Proceedings of the 9th USENIX conference on Operating
[43] T. Wolf and J. Turner, “Design issues for high performance active systems design and implementation, ser. OSDI’10, Berkeley, CA, USA,
routers,” in Broadband Communications, 2000. Proceedings. 2000 2010, pp. 1–6.
International Zurich Seminar on, 2000, pp. 199–205. [64] T. Koponen, K. Amidon, P. Balland, M. Casado, A. Chanda, B. Fulton,
[44] S. da Silva, Y. Yemini, and D. Florissi, “The NetScript active network I. Ganichev, J. Gross, P. Ingram, E. Jackson, A. Lambeth, R. Lenglet,
system,” IEEE J.Sel. A. Commun., vol. 19, no. 3, pp. 538–551, Mar. S.-H. Li, A. Padmanabhan, J. Pettit, B. Pfaff, R. Ramanathan,
2001. S. Shenker, A. Shieh, J. Stribling, P. Thakkar, D. Wendlandt, A. Yip,
[45] J. Biswas, A. A. Lazar, J. F. Huard, K. Lim, S. Mahjoub, L. F. and R. Zhang, “Network virtualization in multi-tenant datacenters,” in
Pau, M. Suzuki, S. Torstensson, W. Wang, and S. Weinstein, “The 11th USENIX Symposium on Networked Systems Design and Imple-
IEEE P1520 standards initiative for programmable network interfaces,” mentation (NSDI 14), Seattle, WA, Apr. 2014, pp. 203–216.
Comm. Mag., vol. 36, no. 10, pp. 64–70, Oct. 1998. [65] V. Bollapragada, C. Murphy, and R. White, Inside Cisco IOS software
[46] J. Van der Merwe, S. Rooney, I. Leslie, and S. Crosby, “The tempest- architecture, 1st ed. Cisco Press, Jul 2000.
a practical framework for network programmability,” Network, IEEE, [66] Juniper Networks, “Junos OS Architecture Overview,” 2012.
vol. 12, no. 3, pp. 20–28, May 1998. [Online]. Available: http://www.juniper.net/techpubs/en US/junos12.
[47] T. Lakshman, T. Nandagopal, R. Ramjee, K. Sabnani, and T. Woo, 1/topics/concept/junos-software-architecture.html
“The SoftRouter Architecture,” in Third ACM Workshop on Hot Topics [67] Extreme Networks, “ExtremeXOS Operating System, Version
in Networks (HotNets-III), San Diego, CA, November 2004. 15.4,” 2014. [Online]. Available: http://learn.extremenetworks.com/
[48] J. Vasseur and J. L. Roux, “Path Computation Element (PCE) rs/extreme/images/EXOS-DS.pdf
Communication Protocol (PCEP),” RFC 5440 (Proposed Standard), [68] Alcatel-Lucent, “SR OS,” 2014. [Online]. Available: http://www3.
Internet Engineering Task Force, Mar. 2009. [Online]. Available: alcatel-lucent.com/products/sros/
http://www.ietf.org/rfc/rfc5440.txt [69] U. Krishnaswamy, P. Berde, J. Hart, M. Kobayashi, P. Radoslavov,
[49] A. Greenberg, G. Hjalmtysson, D. A. Maltz, A. Myers, J. Rexford, T. Lindberg, R. Sverdlov, S. Zhang, W. Snow, and G. Parulkar,
G. Xie, H. Yan, J. Zhan, and H. Zhang, “A clean slate 4D approach “ONOS: An Open Source Distributed SDN OS,” 2013.
to network control and management,” SIGCOMM Comput. Commun. [Online]. Available: http://www.slideshare.net/umeshkrishnaswamy/
Rev., vol. 35, no. 5, pp. 41–54, Oct. 2005. open-network-operating-system
[50] J. Van der Merwe, A. Cepleanu, K. D’Souza, B. Freeman, A. Green- [70] A. T. Campbell, I. Katzela, K. Miki, and J. Vicente, “Open signaling for
berg, D. Knight, R. McMillan, D. Moloney, J. Mulligan, H. Nguyen, atm, internet and mobile networks (opensig’98),” SIGCOMM Comput.
M. Nguyen, A. Ramarajan, S. Saad, M. Satterlee, T. Spencer, D. Toll, Commun. Rev., vol. 29, no. 1, pp. 97–108, Jan. 1999.
and S. Zelingher, “Dynamic connectivity management with an intel- [71] R. Sherwood, M. Chan, A. Covington, G. Gibb, M. Flajslik, N. Hand-
ligent route service control point,” in Proceedings of the SIGCOMM igol, T.-Y. Huang, P. Kazemian, M. Kobayashi, J. Naous, S. Seethara-
workshop on Internet network management, ser. INM ’06. New York, man, D. Underhill, T. Yabe, K.-K. Yap, Y. Yiakoumis, H. Zeng,
NY, USA: ACM, 2006, pp. 29–34. G. Appenzeller, R. Johari, N. McKeown, and G. Parulkar, “Carving
VERSION 1.0 41

research slices out of your production networks with OpenFlow,” [100] S. Shenker, “Stanford Seminar - Software-Defined Networking at the
SIGCOMM Comput. Commun. Rev., vol. 40, no. 1, pp. 129–130, Jan. Crossroads,” June 2013. [Online]. Available: http://www.youtube.com/
2010. watch?v=WabdXYzCAOU
[72] H. Song, J. Gong, J. Song, and J. Yu, “Protocol Oblivious Forwarding [101] M. Casado, “OpenStack and Network Virtualization,” April
(POF),” 2013. [Online]. Available: http://www.poforwarding.org/ 2013. [Online]. Available: http://blogs.vmware.com/vmware/2013/
[73] ONF, “Charter: Forwarding Abstractions Working Group,” April 04/openstack-and-network-virtualization.html
2014. [Online]. Available: https://www.opennetworking.org/images/ [102] Pica8 Open Networking, “Pica8’s os for open switches,”
stories/downloads/working-groups/charter-forwarding-abstractions.pdf 2013. [Online]. Available: http://www.pica8.org/open-switching/open-
[74] Centec Networks, “V350 - centec open SDN platform,” 2013. switching-overview.php
[Online]. Available: http://www.valleytalk.org/wp-content/uploads/ [103] ONIE, “Open Network Install Environment,” 2013. [Online]. Available:
2013/04/Centec-Open-SDN-Platform.pdf http://onie.org/
[75] NEC, “Nec ProgrammableFlow UNIVERGE PF5820,” 2013. [104] T. Kato, M. Kawakami, T. Myojin, H. Ogawa, K. Hirono, and
[Online]. Available: http://www.nec.com/en/global/prod/pflow/images T. Hasegawa, “Case study of applying SPLE to development of network
documents/ProgrammableFlow Switch PF5820.pdf switch products,” in Proceedings of the 17th International Software
[76] NoviFlow, “NoviSwitch 1248 High Performance OpenFlow Product Line Conference, ser. SPLC ’13. New York, NY, USA: ACM,
Switch,” 2013. [Online]. Available: http://205.236.122.20/gestion/ 2013, pp. 198–207.
NoviSwitch1248Datasheet.pdf [105] B. Pfaff and B. Davie, “The Open vSwitch Database Management
[77] HP, “HP 8200 ZL switch series,” 2013. [Online]. Avail- Protocol,” Internet Draft, Internet Engineering Task Force, September
able: http://h17007.www1.hp.com/us/en/networking/products/switches/ 2013. [Online]. Available: http://tools.ietf.org/id/draft-pfaff-ovsdb-
HP 8200 zl Switch Series/ proto-03.txt
[78] Arista Networks, “7150 series,” 2013. [Online]. Available: http://www. [106] M. Smith, M. Dvorkin, Y. Laribi, V. Pandey, P. Garg, and
aristanetworks.com/media/system/pdf/Datasheets/7150S Datasheet.pdf N. Weidenbacher, “OpFlex Control Protocol,” Internet Draft, Internet
[79] Extreme Networks, “Blackdiamond x8,” 2013. [Online]. Available: Engineering Task Force, April 2014. [Online]. Available: http:
http://www.extremenetworks.com/libraries/products/DSBDX 1832.pdf //tools.ietf.org/html/draft-smith-opflex-00
[80] Huawei Technologies Co., Ltd., “Cx600 metro services platform,” [107] T. J. Bittman, G. J. Weiss, M. A. Margevicius, and P. Dawson, “Magic
2013. [Online]. Available: http://www.huawei.com/ucmf/groups/public/ Quadrant for x86 Server Virtualization Infrastructure,” Gartner, Tech.
documents/attachments/hw 132369.pdf Rep., June 2013.
[81] Juniper Networks, “Ex9200 ethernet switch,” 2013. [Online]. Available: [108] D. W. Cearley, D. Scott, J. Skorupa, and T. J. Bittman,
http://www.juniper.net/us/en/local/pdf/datasheets/1000432-en.pdf “Top 10 Technology Trends, 2013: Cloud Computing and
[82] I. Yokneam, “EZchip Announces OpenFlow 1.1 Implementations on Hybrid IT Drive Future IT Models,” February 2013. [Online].
its NP-4 100-Gigabit Network Processor,” 2011. [Online]. Available: Available: http://www.gartnersummit.com/Gartnertop 10 technology
http://www.ezchip.com/pr 110713.htm trends 201 237716.pdf
[83] BROCADE, “Brocade MLX series,” 2013. [On- [109] C. Peng, M. Kim, Z. Zhang, and H. Lei, “VDN: virtual machine
line]. Available: http://www.brocade.com/products/all/routers/product- image distribution network for cloud data centers,” in INFOCOM, 2012
details/netiron-mlx-series/system-options.page Proceedings IEEE, March 2012, pp. 181–189.
[84] IBM, “IBM System Networking RackSwitch G8264,” 2013. [Online]. [110] Z. Zhang, Z. Li, K. Wu, D. Li, H. Li, Y. Peng, and X. Lu, “VMThunder:
Available: http://www-03.ibm.com/systems/networking/switches/rack/ fast provisioning of large-scale virtual machine clusters,” Parallel and
g8264/ Distributed Systems, IEEE Transactions on, vol. PP, no. 99, pp. 1–1,
2014.
[85] NEC, “Nec ProgrammableFlow family of products,” 2013. [Online].
Available: http://www.necam.com/SDN/ [111] M. Mahalingam, D. Dutt, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar,
M. Bursell, and C. Wright, “VXLAN: A Framework for Overlaying
[86] Pica8, “Pica8 3920,” 2013. [Online]. Available: http://www.pica8.org/
Virtualized Layer 2 Networks over Layer 3 Networks,” Internet Draft,
documents/pica8-datasheet-64x10gbe-p3780-p3920.pdf
Internet Engineering Task Force, November 2013. [Online]. Available:
[87] Plexxi, “Plexxi Switch 1,” 2013. [Online]. Avail- http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-06.txt
able: http://www.plexxi.com/wp-content/themes/plexxi/assets/pdf/
[112] M. Sridharan, A. Greenberg, Y. Wang, P. Garg, N. Venkataramiah,
Plexxi Switch 1 Datasheet Dec 2012.pdf
K. Duda, I. Ganga, G. Lin, M. Pearson, P. Thaler, and
[88] Centec Networks, “Centec v330 OpenFlow switch reference C. Tumuluri, “NVGRE: Network Virtualization using Generic
design,” 2013. [Online]. Available: http://www.centecnetworks.com/ Routing Encapsulation,” Internet Draft, Internet Engineering Task
en/SolutionList.asp?ID=42 Force, August 2013. [Online]. Available: http://tools.ietf.org/id/draft-
[89] Cyan, Inc., “Z-series,” 2013. [Online]. Available: http://www.cyaninc. sridharan-virtualization-nvgre-03.txt
com/en/our-solutions/z-series/ [113] R. Sherwood, G. Gibb, K.-K. Yap, G. Appenzeller, M. Casado,
[90] Juniper Networks, Inc., “Contrail virtual router,” 2013. [Online]. N. McKeown, and G. Parulkar, “FlowVisor: A Network Virtualization
Available: https://github.com/Juniper/contrail-vrouter Layer,” Deutsche Telekom Inc. R&D Lab, Stanford, Nicira Networks,
[91] FlowForwarding, “LINC-Switch,” 2013. [Online]. Available: http: Tech. Rep., 2009.
//www.flowforwarding.org/ [114] S. Azodolmolky, R. Nejabati, S. Peng, A. Hammad, M. P. Chan-
[92] K. Rutka, K. Kaplita, S. Narayan, and S. Bailey, “LINC Switch,” negowda, N. Efstathiou, A. Autenrieth, P. Kaczmarek, and D. Sime-
2013. [Online]. Available: http://www.opennetsummit.org/pdf/2013/ onidou, “Optical FlowVisor: An OpenFlow-based Optical Network Vir-
research track/poster papers/ons2013-final36.pdf tualization Approach,” in National Fiber Optic Engineers Conference,
[93] CPqD, “ofsoftswitch13,” 2013. [Online]. Available: https://github.com/ ser. OSA Technical Digest. Optical Society of America, Mar. 2012.
CPqD/ofsoftswitch13 [115] D. A. Drutskoy, “Software-Defined Network Virtualization with
[94] “Open vSwitch,” 2013. [Online]. Available: http://vswitch.org/ FlowN,” Ph.D. dissertation, Department of Computer Science of
[95] OpenFlow Community, “OpenFlow switching reference system,” Princeton University, Jun 2012.
2009. [Online]. Available: http://www.openflow.org/wp/downloads/ [116] A. Al-Shabibi, M. D. Leenheer, M. Gerolay, A. Koshibe, W. Snow, and
[96] Y. Mundada, R. Sherwood, and N. Feamster, “An OpenFlow G. Parulkar, “OpenVirteX: A Network Hypervisor,” 2014. [Online].
switch element for click,” in in Symposium on Click Modular Router, Available: http://ovx.onlab.us/wp-content/uploads/2014/04/ovx-ons14.
2009. [Online]. Available: http://www.cc.gatech.edu/∼yogeshm3/click pdf
symposium2009.pdf [117] S. Racherla, D. Cain, S. Irwin, P. Ljungstrom, P. Patil, and A. M.
[97] Big Switch Networks, “Project Floodlight,” 2013. [Online]. Available: Tarenzio, Implementing IBM Software Defined Network for Virtual
http://www.projectfloodlight.org/ Environments. IBM RedBooks, May 2014.
[98] Y. Yiakoumis, J. Schulz-Zander, and J. Zhu, “Pantou : OpenFlow 1.0 [118] C. Li, B. Brech, S. Crowder, D. Dias, H. Franke, M. Hogstrom,
for OpenWRT,” 2011. [Online]. Available: http://www.openflow.org/ D. Lindquist, G. Pacifici, S. Pappe, B. Rajaraman, J. Rao, R. Ratna-
wk/index.php/OpenFlow 1.0 for OpenWRT parkhi, R. Smith, and M. Williams, “Software defined environments:
[99] A. Weissberger, “VMware’s Network Virtualization Poses Huge An introduction,” IBM Journal of Research and Development, vol. 58,
Threat to Data Center Switch Fabric Vendors,” 2013. no. 2, pp. 1–11, March 2014.
[Online]. Available: http://viodi.com/2013/05/06/vmwares-network- [119] Z. Bozakov and P. Papadimitriou, “Autoslice: automated and scalable
virtualization-poses-huge-threat-to-data-center-switch-fabric-vendors/ slicing for software-defined networks,” in Proceedings of the 2012 ACM
VERSION 1.0 42

conference on CoNEXT student workshop, ser. CoNEXT Student ’12. [143] R. Enns, M. Bjorklund, J. Schoenwaelder, and A. Bierman, “Network
New York, NY, USA: ACM, 2012, pp. 3–4. Configuration Protocol (NETCONF),” RFC 6241 (Proposed Standard),
[120] D. Drutskoy, E. Keller, and J. Rexford, “Scalable network virtualization Internet Engineering Task Force, Jun. 2011. [Online]. Available:
in software-defined networks,” Internet Computing, IEEE, vol. 17, http://www.ietf.org/rfc/rfc6241.txt
no. 2, pp. 20–27, 2013. [144] D. Harrington, R. Presuhn, and B. Wijnen, “An Architecture
[121] Juniper Networks, “Opencontrail,” 2013. [Online]. Available: http: for Describing Simple Network Management Protocol (SNMP)
//opencontrail.org/ Management Frameworks,” Internet Engineering Task Force, dec
[122] HP, “Hp SDN controller architecture,” Hewlett-Packard Development 2002. [Online]. Available: http://www.ietf.org/rfc/rfc3411.txt
Company, L.P., Tech. Rep., September 2013. [145] Y. Rekhter, T. Li, and S. Hares, “A Border Gateway Protocol 4 (BGP-
[123] K. Phemius, M. Bouet, and J. Leguay, “DISCO: Distributed Multi- 4),” RFC 4271 (Draft Standard), Internet Engineering Task Force, Jan.
domain SDN Controllers,” ArXiv e-prints, Aug. 2013. 2006. [Online]. Available: http://www.ietf.org/rfc/rfc4271.txt
[124] D. Erickson, “The beacon OpenFlow controller,” in Proceedings of the [146] H. Yin, H. Xie, T. Tsou, D. Lopez, P. Aranda, and R. Sidi, “SDNi: A
second ACM SIGCOMM workshop on Hot topics in software defined Message Exchange Protocol for Software Defined Networks (SDNS)
networking, ser. HotSDN ’13. New York, NY, USA: ACM, 2013, pp. across Multiple Domains,” Internet Draft, Internet Engineering Task
13–18. Force, June 2012. [Online]. Available: http://tools.ietf.org/id/draft-yin-
[125] A. Tootoonchian, S. Gorbunov, Y. Ganjali, M. Casado, and R. Sher- sdn-sdni-00.txt
wood, “On controller performance in software-defined networks,” in [147] M. Garcia, A. Bessani, I. Gashi, N. Neves, and R. Obelheiro, “Anal-
Proceedings of the 2nd USENIX conference on Hot Topics in Manage- ysis of operating system diversity for intrusion tolerance,” Software:
ment of Internet, Cloud, and Enterprise Networks and Services, ser. Practice and Experience, vol. 44, no. 6, pp. 735–770, 2014.
Hot-ICE’12. Berkeley, CA, USA: USENIX Association, 2012, pp. [148] Z. Wang, T. Tsou, J. Huang, X. Shi, and X. Yin, “Analysis of
10–10. Comparisons between OpenFlow and ForCES,” Internet Draft, Internet
[126] Z. Cai, A. L. Cox, and T. S. E. Ng, “Maestro: A System for Scalable Engineering Task Force, December 2011. [Online]. Available: http:
OpenFlow Control,” Rice University, Tech. Rep., 2011. //tools.ietf.org/id/draft-wang-forces-compare-openflow-forces-00.txt
[127] D. Erickson, “The beacon OpenFlow controller,” in Proceedings of [149] K. Ogawa, W. M. Wang, E. Haleplidis, and J. H. Salim, “ForCES
the second workshop on Hot topics in software defined networks, ser. Intra-NE High Availability,” Internet Draft, Internet Engineering Task
HotSDN ’13. New York, NY, USA: ACM, 2013. Force, October 2013. [Online]. Available: http://tools.ietf.org/id/draft-
[128] “Floodlight Is A Java-Based OpenFlow Controller,” 2012. [Online]. ietf-forces-ceha-08.txt
Available: http://floodlight.openflowhub.org/ [150] F. A. Botelho, F. M. V. Ramos, D. Kreutz, and A. N. Bessani, “On
[129] Y. Takamiya and N. Karanatsios, “Trema OpenFlow controller the feasibility of a consistent and fault-tolerant data store for SDNs,”
framework,” 2012. [Online]. Available: https://github.com/trema/trema in Proceedings of the 2013 Second European Workshop on Software
[130] Nippon Telegraph and Telephone Corporation, “Ryu Network Defined Networks, ser. EWSDN ’13. Washington, DC, USA: IEEE
Operating System,” 2012. [Online]. Available: http://osrg.github.com/ Computer Society, 2013, pp. 38–43.
ryu/ [151] S. Vinoski, “Advanced message queuing protocol,” IEEE Internet
[131] M. Banikazemi, D. Olshefski, A. Shaikh, J. Tracey, and G. Wang, Computing, vol. 10, no. 6, pp. 87–89, Nov. 2006.
“Meridian: an SDN platform for cloud network services,” Communi- [152] A. Ghodsi, “Distributed k-ary system: Algorithms for distributed hash
cations Magazine, IEEE, vol. 51, no. 2, pp. 120–127, 2013. tables,” Ph.D. dissertation, KTH-Royal Institute of Technology, 2006.
[132] NEC, “Award-winning Software-defined Networking NEC
[153] W. Stallings, “Software-Defined Networks and OpenFlow,” The Inter-
ProgrammableFlow Networking Suite,” September 2013. [On-
net Protocol Journal, vol. 16, no. 1, 2013.
line]. Available: http://www.necam.com/docs/?id=67c33426-0a2b-
[154] K. Pentikousis, Y. Wang, and W. Hu, “MobileFlow: Toward software-
4b87-9a7a-d3cecc14d26a
defined mobile networks,” Communications Magazine, IEEE, vol. 51,
[133] A. Tootoonchian and Y. Ganjali, “HyperFlow: a distributed control
no. 7, pp. 44–53, 2013.
plane for OpenFlow,” in Proceedings of the 2010 internet network
[155] A. Voellmy and P. Hudak, “Nettle: taking the sting out of programming
management conference on Research on enterprise networking, ser.
network routers,” in Proceedings of the 13th international conference
INM/WREN’10. Berkeley, CA, USA: USENIX Association, 2010,
on Practical aspects of declarative languages, ser. PADL’11. Berlin,
pp. 3–3.
Heidelberg: Springer-Verlag, 2011, pp. 235–249.
[134] M. Monaco, O. Michel, and E. Keller, “Applying Operating System
Principles to SDN Controller Design,” in Twelfth ACM Workshop on [156] A. Voellmy, H. Kim, and N. Feamster, “Procera: a language for high-
Hot Topics in Networks (HotNets-XII), College Park, MD, November level reactive network control,” in Proceedings of the first workshop
2013. on Hot topics in software defined networks, ser. HotSDN ’12. New
[135] P. Porras, S. Shin, V. Yegneswaran, M. Fong, M. Tyson, and G. Gu, “A York, NY, USA: ACM, 2012, pp. 43–48.
security enforcement kernel for OpenFlow networks,” in Proceedings [157] C. Monsanto, J. Reich, N. Foster, J. Rexford, and D. Walker, “Com-
of the First Workshop on Hot Topics in Software Defined Networks, posing software-defined networks,” in Proceedings of the 10th USENIX
ser. HotSDN ’12. New York, NY, USA: ACM, 2012, pp. 121–126. conference on Networked Systems Design and Implementation, ser.
[Online]. Available: http://doi.acm.org/10.1145/2342441.2342466 nsdi’13. Berkeley, CA, USA: USENIX Association, 2013, pp. 1–14.
[136] L. Richardson and S. Ruby, RESTful web services. O’Reilly Media, [158] S. Hassas Yeganeh and Y. Ganjali, “Kandoo: A framework for efficient
Inc., 2008. and scalable offloading of control applications,” in Proceedings of
[137] T. L. Hinrichs, N. S. Gude, M. Casado, J. C. Mitchell, and S. Shenker, the First Workshop on Hot Topics in Software Defined Networks, ser.
“Practical declarative network management,” in Proceedings of the 1st HotSDN ’12. New York, NY, USA: ACM, 2012, pp. 19–24.
ACM workshop on Research on enterprise networking, ser. WREN ’09. [159] D. Saikia, “MuL OpenFlow controller,” 2013. [Online]. Available:
New York, NY, USA: ACM, 2009, pp. 1–10. http://sourceforge.net/projects/mul/
[138] N. Foster, R. Harrison, M. J. Freedman, C. Monsanto, J. Rexford, [160] M. McCauley, “POX,” 2012. [Online]. Available: http://www.noxrepo.
A. Story, and D. Walker, “Frenetic: a network programming language,” org/
SIGPLAN Not., 2011. [161] H. Shimonishi and S. Ishii, “Virtualized network infrastructure using
[139] C. Monsanto, N. Foster, R. Harrison, and D. Walker, “A compiler and OpenFlow,” in Network Operations and Management Symposium Work-
run-time system for network programming languages,” SIGPLAN Not., shops (NOMS Wksps), 2010 IEEE/IFIP, 2010, pp. 74–79.
vol. 47, no. 1, pp. 217–230, Jan. 2012. [162] G. Appenzeller, “SNAC,” 2011. [Online]. Available: http://www.
[140] ONF, “OpenFlow Notifications Framework OpenFlow openflowhub.org/display/Snac
Management,” October 2014. [Online]. Available: [163] B. Casemore, “SDN controller ecosystems critical to market success,”
https://www.opennetworking.org/images/stories/downloads/sdn- 2012. [Online]. Available: http://nerdtwilight.wordpress.com/2012/06/
resources/onf-specifications/openflow-config/of-notifications- 05/sdn-controller-ecosystems-critical-to-market-success/
framework-1.0.pdf [164] R. Kwan and C. Leung, “A survey of scheduling and interference
[141] A. Singla and B. Rijsman, “Contrail Architecture,” Juniper Networks, mitigation in lte,” JECE, vol. 2010, pp. 1:1–1:10, Jan. 2010. [Online].
Tech. Rep., 2013. Available: http://dx.doi.org/10.1155/2010/273486
[142] ONF, “OpenFlow Management and Configuration Proto- [165] A. Gudipati, D. Perry, L. E. Li, and S. Katti, “SoftRAN: Software
col (OF-Config 1.1.1),” March 2014. [Online]. Avail- defined radio access network,” in Proceedings of the second workshop
able: https://www.opennetworking.org/images/stories/downloads/sdn- on Hot topics in software defined networks, ser. HotSDN ’13. New
resources/onf-specifications/openflow-config/of-config-1-1-1.pdf York, NY, USA: ACM, 2013.
VERSION 1.0 43

[166] J. Dix, “Clarifying the role of software-defined networking northbound [190] A. Voellmy, J. Wang, Y. R. Yang, B. Ford, and P. Hudak, “Maple:
APIs,” May 2013. [Online]. Available: http://www.networkworld.com/ simplifying SDN programming using algorithmic policies,” in Pro-
news/2013/050213-sherwood-269366.html ceedings of the ACM SIGCOMM 2013 conference on SIGCOMM, ser.
[167] I. GUIS, “The SDN Gold Rush To The Northbound API,” November SIGCOMM ’13. New York, NY, USA: ACM, 2013, pp. 87–98.
2012. [Online]. Available: http://www.sdncentral.com/technology/the- [191] R. Soule, S. Basu, R. Kleinberg, E. G. Sirer, and N. Foster, “Managing
sdn-gold-rush-to-the-northbound-api/2012/11/ the Network with Merlin,” in Twelfth ACM Workshop on Hot Topics
[168] B. Salisbury, “The northbound API- a big little problem,” 2012. in Networks (HotNets-XII), College Park, MD, November 2013.
[169] G. Ferro, “Northbound API, southbound api, east/north lan navigation [192] C. Jasson Casey, A. Sutton, G. Dos Reis, and A. Sprintson, “Eliminat-
in an OpenFlow world and an SDN compass,” Aug. 2012. ing Network Protocol Vulnerabilities Through Abstraction and Systems
[170] B. Casemore, “Northbound API: The standardization debate,” Sept. Language Design,” ArXiv e-prints, Nov. 2013.
2012. [Online]. Available: http://nerdtwilight.wordpress.com/2012/09/ [193] P. Pereini, M. Kuzniar, and D. Kostic, “OpenFlow needs you! a call
18/northbound-api-the-standardization-debate/ for a discussion about a cleaner OpenFlow API,” in Software Defined
[171] I. Pepelnjak, “SDN controller northbound API is the crucial missing Networks (EWSDN), 2013 Second European Workshop on, Oct 2013,
piece,” Sept. 2012. [Online]. Available: http://blog.ioshints.info/2012/ pp. 44–49.
09/sdn-controller-northbound-api-is.html [194] F. Facca, E. Salvadori, H. Karl, D. Lopez, P. Aranda Gutierrez,
[172] S. Johnson, “A primer on northbound APIs: Their role in D. Kostic, and R. Riggio, “NetIDE: First steps towards an integrated
a software-defined network,” December 2012. [Online]. Avail- development environment for portable network apps,” in Software
able: http://searchsdn.techtarget.com/feature/A-primer-on-northbound- Defined Networks (EWSDN), 2013 Second European Workshop on, Oct
APIs-Their-role-in-a-software-defined-network 2013, pp. 105–110.
[173] R. G. Little, “ONF to standardize northbound [195] M. Scharf, V. Gurbani, T. Voith, M. Stein, W. Roome, G. Soprovich,
API for SDN applications?” October 2013. [Online]. and V. Hilt, “Dynamic VPN optimization by ALTO guidance,” in Soft-
Available: http://searchsdn.techtarget.com/news/2240206604/ONF-to- ware Defined Networks (EWSDN), 2013 Second European Workshop
standardize-northbound-API-for-SDN-applications on, Oct 2013, pp. 13–18.
[174] M. Yu, A. Wundsam, and M. Raju, “NOSIX: A lightweight portability [196] M. Stiemerling, S. Kiesel, S. Previdi, and M. Scharf, “ALTO
layer for the sdn os,” SIGCOMM Comput. Commun. Rev., vol. 44, Deployment Considerations,” Internet Draft, Internet Engineering Task
no. 2, pp. 28–35, Apr. 2014. Force, February 2014. [Online]. Available: http://tools.ietf.org/html/
[175] R. Chua, “OpenFlow northbound API: A new olympic sport,” 2012. draft-ietf-alto-deployments-09
[Online]. Available: http://www.sdncentral.com/sdn-blog/openflow- [197] N. Handigol, S. Seetharaman, M. Flajslik, A. Gember, N. McKeown,
northbound-api-olympics/2012/07/ G. Parulkar, A. Akella, N. Feamster, R. Clark, A. Krishnamurthy,
[176] J. Reich, C. Monsanto, N. Foster, J. Rexford, and D. Walker, “Modular V. Brajkovic, and T. A. and, “Aster*x: Load-Balancing Web Traffic
SDN Programming with Pyretic,” USENIX ;login, vol. 38, no. 5, over Wide-Area Networks,” 2009.
October 2013. [198] B. Heller, S. Seetharaman, P. Mahadevan, Y. Yiakoumis, P. Sharma,
S. Banerjee, and N. McKeown, “ElasticTree: saving energy in data
[177] K.-K. Yap, T.-Y. Huang, B. Dodson, M. S. Lam, and N. McKeown,
center networks,” in Proceedings of the 7th USENIX conference on
“Towards software-friendly networks,” in Proceedings of the first ACM
Networked systems design and implementation, ser. NSDI’10. Berke-
asia-pacific workshop on Workshop on systems, ser. APSys ’10. New
ley, CA, USA: USENIX Association, 2010, pp. 17–17.
York, NY, USA: ACM, 2010, pp. 49–54.
[199] M. Al-Fares, S. Radhakrishnan, B. Raghavan, N. Huang, and A. Vahdat,
[178] S. Gutz, A. Story, C. Schlesinger, and N. Foster, “Splendid isolation:
“Hedera: dynamic flow scheduling for data center networks,” in Pro-
A slice abstraction for software-defined networks,” in Proceedings of
ceedings of the 7th USENIX conference on Networked systems design
the First Workshop on Hot Topics in Software Defined Networks, ser.
and implementation, ser. NSDI’10. Berkeley, CA, USA: USENIX
HotSDN ’12. New York, NY, USA: ACM, 2012, pp. 79–84.
Association, 2010, pp. 19–19.
[179] D. Turull, M. Hidell, and P. Sjödin, “Evaluating OpenFlow in lib- [200] C. Macapuna, C. Rothenberg, and M. Magalhaes, “In-packet bloom
netvirt,” in The 8th Swedish National Computer Networking Workshop filter based data center networking with distributed OpenFlow con-
2012 (SNCNW 2012), Oct 2012. trollers,” in GLOBECOM Workshops (GC Wkshps), 2010 IEEE, 2010,
[180] Quantum Communicty, “OpenStack Networking (”Quantum”),” 2012. pp. 584–588.
[181] M. Guzdial, “Education: Paving the way for computational thinking,” [201] H. Egilmez, S. Dane, K. Bagci, and A. Tekalp, “Openqos: An Open-
Commun. ACM, vol. 51, no. 8, pp. 25–27, Aug. 2008. Flow controller design for multimedia delivery with end-to-end quality
[182] M. S. Farooq, S. A. Khan, F. Ahmad, S. Islam, and A. Abid, “An of service over software-defined networks,” in Signal Information
evaluation framework and comparative analysis of the widely used first Processing Association Annual Summit and Conference (APSIPA ASC),
programming languages,” PLoS ONE, vol. 9, no. 2, 02 2014. 2012 Asia-Pacific, 2012, pp. 1–8.
[183] A. D. Ferguson, A. Guha, C. Liang, R. Fonseca, and S. Krishnamurthi, [202] N. Handigol, S. Seetharaman, M. Flajslik, N. McKeown, and R. Johari,
“Hierarchical policies for software defined networks,” in Proceedings “Plug-n-serve: Load-balancing web traffic using OpenFlow,” 2009.
of the first workshop on Hot topics in software defined networks, ser. [203] K. Jeong, J. Kim, and Y.-T. Kim, “QoS-aware Network Operating
HotSDN ’12. New York, NY, USA: ACM, 2012, pp. 37–42. System for software defined networking with Generalized OpenFlows,”
[184] T. Nelson, A. Guha, D. J. Dougherty, K. Fisler, and S. Krishnamurthi, in Network Operations and Management Symposium (NOMS), 2012
“A balance of power: expressive, analyzable controller programming,” IEEE, april 2012, pp. 1167 –1174.
in Proceedings of the second ACM SIGCOMM workshop on Hot topics [204] W. Kim, P. Sharma, J. Lee, S. Banerjee, J. Tourrilhes, S.-J. Lee, and
in software defined networking, ser. HotSDN ’13. New York, NY, P. Yalagandula, “Automated and scalable QoS control for network
USA: ACM, 2013, pp. 79–84. convergence,” in Proceedings of the 2010 internet network management
[185] N. P. Katta, J. Rexford, and D. Walker, “Logic programming for conference on Research on enterprise networking, ser. INM/WREN’10.
software-dedefine networks,” in ACM SIGPLAN Workshop on Cross- Berkeley, CA, USA: USENIX Association, 2010, pp. 1–1.
Model Language Design and Implementation, ser. XLDI, 2012. [205] A. Ishimori, F. Farias, E. Cerqueira, and A. Abelem, “Control of
[186] S. Shin, P. Porras, V. Yegneswaran, M. Fong, G. Gu, and M. Tyson, multiple packet schedulers for improving qos on OpenFlow/SDN
“FRESCO: Modular composable security services for software-defined networking,” in Software Defined Networks (EWSDN), 2013 Second
networks,” in Internet Society NDSS., Feb. 2013. European Workshop on, Oct 2013, pp. 81–86.
[187] S. Son, S. Shin, V. Yegneswaran, P. Porras, and G. Gu, “Model check- [206] Z. A. Qazi, C.-C. Tu, L. Chiang, R. Miao, V. Sekar, and M. Yu,
ing invariant security properties in OpenFlow,” in Communications “SIMPLE-fying middlebox policy enforcement using SDN,” in Pro-
(ICC), 2013 IEEE International Conference on, June 2013, pp. 1974– ceedings of the Conference on Applications, technologies, architectures,
1979. and protocols for computer communications, ser. SIGCOMM ’13.
[188] A. Tootoonchian, M. Ghobadi, and Y. Ganjali, “OpenTM: traffic matrix New York, NY, USA: ACM, 2013.
estimator for OpenFlow networks,” in Proceedings of the 11th inter- [207] P. Skoldstrom and B. C. Sanchez, “Virtual Aggregation using SDN,”
national conference on Passive and active measurement, ser. PAM’10. in 2013 Second European Workshop on Software Defined Networks,
Berlin, Heidelberg: Springer-Verlag, 2010, pp. 201–210. 2013, pp. –.
[189] M. Reitblatt, M. Canini, A. Guha, and N. Foster, “Fattire: Declarative [208] H. Ali-Ahmad, C. Cicconetti, A. de la Oliva, M. Draxler, R. Gupta,
fault tolerance for software defined networks,” in Proceedings of the V. Mancuso, L. Roullet, and V. Sciancalepore, “CROWD: An SDN
second workshop on Hot topics in software defined networks, ser. approach for densenets,” in Software Defined Networks (EWSDN), 2013
HotSDN ’13. New York, NY, USA: ACM, 2013. Second European Workshop on, Oct 2013, pp. 25–31.
VERSION 1.0 44

[209] J. Vestin, P. Dely, A. Kassler, N. Bayer, H. Einsiedler, and C. Peylo, [228] J. H. Jafarian, E. Al-Shaer, and Q. Duan, “OpenFlow random host
“CloudMAC: towards software defined WLANs,” SIGMOBILE Mob. mutation: transparent moving target defense using software defined
Comput. Commun. Rev., vol. 16, no. 4, pp. 42–45, Feb. 2013. networking,” in Proceedings of the first workshop on Hot topics in
[210] Y. Yamasaki, Y. Miyamoto, J. Yamato, H. Goto, and H. Sone, “Flexible software defined networks, ser. HotSDN ’12. New York, NY, USA:
access management system for campus VLAN based on OpenFlow,” ACM, 2012, pp. 127–132.
in Applications and the Internet (SAINT), 2011 IEEE/IPSJ 11th Inter- [229] J. R. Ballard, I. Rae, and A. Akella, “Extensible and scalable network
national Symposium on, 2011, pp. 347–351. monitoring using OpenSAFE,” in Proceedings of the 2010 internet net-
[211] L. Suresh, J. Schulz-Zander, R. Merz, A. Feldmann, and T. Vazao, “To- work management conference on Research on enterprise networking,
wards programmable enterprise WLANS with Odin,” in Proceedings ser. INM/WREN’10. Berkeley, CA, USA: USENIX Association, 2010,
of the first workshop on Hot topics in software defined networks, ser. pp. 8–8.
HotSDN ’12. New York, NY, USA: ACM, 2012, pp. 115–120. [230] D. Kotani, K. Suzuki, and H. Shimonishi, “A design and implemen-
[212] M. Yang, Y. Li, D. Jin, L. Su, S. Ma, and L. Zeng, “OpenRAN: a tation of OpenFlow controller handling ip multicast with fast tree
software-defined ran architecture via virtualization,” in Proceedings of switching,” in Applications and the Internet (SAINT), 2012 IEEE/IPSJ
the ACM SIGCOMM 2013 conference on SIGCOMM, ser. SIGCOMM 12th International Symposium on, 2012, pp. 60–67.
’13. New York, NY, USA: ACM, 2013, pp. 549–550. [231] G. Yao, J. Bi, and P. Xiao, “Source address validation solution with
[213] K.-K. Yap, M. Kobayashi, R. Sherwood, T.-Y. Huang, M. Chan, OpenFlow/NOX architecture,” in Network Protocols (ICNP), 2011 19th
N. Handigol, and N. McKeown, “OpenRoads: empowering research in IEEE International Conference on, 2011, pp. 7–12.
mobile networks,” SIGCOMM Comput. Commun. Rev., vol. 40, no. 1, [232] G. Wang, T. E. Ng, and A. Shaikh, “Programming your network at
pp. 125–126, Jan. 2010. run-time for big data applications,” in HotSDN. ACM, 2012.
[214] Small Cell Forum, “Femto APIs,” 2013. [Online]. Available: [233] T. Benson, A. Akella, A. Shaikh, and S. Sahu, “Cloudnaas: a cloud
http://www.smallcellforum.org/developers/ networking platform for enterprise applications,” in Proceedings of the
[215] V. Chandrasekhar, J. Andrews, and A. Gatherer, “Femtocell networks: 2nd ACM Symposium on Cloud Computing, ser. SOCC ’11. New
a survey,” Communications Magazine, IEEE, vol. 46, no. 9, pp. 59–67, York, NY, USA: ACM, 2011, pp. 8:1–8:13.
September 2008. [234] A. Das, C. Lumezanu, Y. Zhang, V. Singh, G. Jiang, and C. Yu,
[216] S. Shirali-Shahreza and Y. Ganjali, “FleXam: flexible sampling ex- “Transparent and flexible network management for big data processing
tension for monitoring and security applications in OpenFlow,” in in the cloud,” in Proceedings of the 5th USENIX conference on Hot
Proceedings of the second ACM SIGCOMM workshop on Hot topics Topics in Cloud Ccomputing, ser. HotCloud’13. Berkeley, CA, USA:
in software defined networking, ser. HotSDN ’13. New York, NY, USENIX Association, 2013.
USA: ACM, 2013, pp. 167–168. [235] A. Arefin, V. K. Singh, G. Jiang, Y. Zhang, and C. Lumezanu, “Diag-
[217] C. Yu, C. Lumezanu, Y. Zhang, V. Singh, G. Jiang, and H. V. nosing data center behavior flow by flow,” in IEEE 33rd International
Madhyastha, “Flowsense: monitoring network utilization with zero Conference on Distributed Computing Systems. Philadelphia, USA:
measurement cost,” in Proceedings of the 14th international conference IEEE, July 2013.
on Passive and Active Measurement, ser. PAM’13. Berlin, Heidelberg: [236] E. Keller, S. Ghorbani, M. Caesar, and J. Rexford, “Live migration of
Springer-Verlag, 2013, pp. 31–41. an entire network (and its hosts),” in Proceedings of the 11th ACM
[218] L. Jose, M. Yu, and J. Rexford, “Online measurement of large traf- Workshop on Hot Topics in Networks, ser. HotNets-XI. New York,
fic aggregates on commodity switches,” in Proceedings of the 11th NY, USA: ACM, 2012, pp. 109–114.
USENIX conference on Hot topics in management of internet, cloud, [237] R. Raghavendra, J. Lobo, and K.-W. Lee, “Dynamic graph query
and enterprise networks and services, ser. Hot-ICE’11. Berkeley, CA, primitives for sdn-based cloudnetwork management,” in Proceedings
USA: USENIX Association, 2011, pp. 13–13. of the first workshop on Hot topics in software defined networks, ser.
[219] M. Yu, L. Jose, and R. Miao, “Software defined traffic measurement HotSDN ’12. New York, NY, USA: ACM, 2012, pp. 97–102.
with OpenSketch,” in Proceedings of the 10th USENIX conference on [238] M. Ghobadi, “TCP Adaptation Framework in Data Centers,” Ph.D.
Networked Systems Design and Implementation, ser. nsdi’13. Berke- dissertation, Graduate Department of Computer Science of University
ley, CA, USA: USENIX Association, 2013, pp. 29–42. of Toronto, 2013.
[220] C. Argyropoulos, D. Kalogeras, G. Androulidakis, and V. Maglaris, [239] R. Wang, D. Butnariu, and J. Rexford, “OpenFlow-based server load
“PaFloMon – a slice aware passive flow monitoring framework for balancing gone wild,” in Proceedings of the 11th USENIX conference
OpenFlow enabled experimental facilities,” in Software Defined Net- on Hot topics in management of internet, cloud, and enterprise net-
working (EWSDN), 2012 European Workshop on, 2012, pp. 97–102. works and services, ser. Hot-ICE’11. Berkeley, CA, USA: USENIX
[221] R. Hand, M. Ton, and E. Keller, “Active Security,” in Twelfth ACM Association, 2011, pp. 12–12.
Workshop on Hot Topics in Networks (HotNets-XII), College Park, MD, [240] H. Ballani, P. Francis, T. Cao, and J. Wang, “Making routers last
November 2013. longer with viaggre,” in Proceedings of the 6th USENIX symposium
[222] S. Shin, V. Yegneswaran, P. Porras, and G. Gu, “AVANT-GUARD: on Networked systems design and implementation, ser. NSDI’09.
Scalable and Vigilant Switch Flow Management in Software-De?ned Berkeley, CA, USA: USENIX Association, 2009, pp. 453–466.
Networks,” in Proceedings of the 2013 ACM conference on Computer [241] D. Meyer, L. Zhang, and K. Fall, “Report from the IAB
and communications security, ser. CCS ’13. New York, NY, USA: Workshop on Routing and Addressing,” RFC 4984 (Informational),
ACM, 2013. Internet Engineering Task Force, Sep. 2007. [Online]. Available:
[223] S. Shin and G. Gu, “Cloudwatcher: Network security monitoring using http://www.ietf.org/rfc/rfc4984.txt
OpenFlow in dynamic cloud networks (or: How to provide security [242] M. Jarschel, F. Wamser, T. Hohn, T. Zinner, and P. Tran-Gia, “SDN-
monitoring as a service in clouds?),” in Proceedings of the 2012 20th based application-aware networking on the example of youtube video
IEEE International Conference on Network Protocols (ICNP), ser. streaming,” in Software Defined Networks (EWSDN), 2013 Second
ICNP ’12. Washington, DC, USA: IEEE Computer Society, 2012, European Workshop on, Oct 2013, pp. 87–92.
pp. 1–6. [243] H. Kumar, H. H. Gharakheili, and V. Sivaraman, “User control of
[224] R. Braga, E. Mota, and A. Passito, “Lightweight DDoS flooding attack quality of experience in home networks using SDN,” in Advanced
detection using NOX/OpenFlow,” in Local Computer Networks (LCN), Networks and Telecommuncations Systems (ANTS), 2013 IEEE Inter-
2010 IEEE 35th Conference on, oct. 2010, pp. 408 –415. national Conference on, 2013.
[225] G. Stabler, A. Rosen, S. Goasguen, and K.-C. Wang, “Elastic ip [244] L. Li, Z. Mao, and J. Rexford, “Toward software-defined cellular
and security groups implementation using OpenFlow,” in Proceedings networks,” in Software Defined Networking (EWSDN), 2012 European
of the 6th international workshop on Virtualization Technologies in Workshop on, 2012, pp. 7–12.
Distributed Computing Date, ser. VTDC ’12. New York, NY, USA: [245] X. Jin, L. Erran Li, L. Vanbever, and J. Rexford, “SoftCell: Scalable
ACM, 2012, pp. 53–60. and Flexible Cellular Core Network Architecture,” in Proceedings of
[226] K. Wang, Y. Qi, B. Yang, Y. Xue, and J. Li, “LiveSec: Towards the 9th international conference on Emerging networking experiments
Effective Security Management in Large-Scale Production Networks,” and technologies, ser. CoNEXT ’13. New York, NY, USA: ACM,
in Distributed Computing Systems Workshops (ICDCSW), 2012 32nd 2013.
International Conference on, june 2012, pp. 451 –460. [246] P. Dely, A. Kassler, and N. Bayer, “OpenFlow for wireless mesh
[227] Y. Wang, Y. Zhang, V. Singh, C. Lumezanu, and G. Jiang, “NetFuse: networks,” in Computer Communications and Networks (ICCCN), 2011
Short-Circuiting Traffic Surges in the Cloud,” in IEEE International Proceedings of 20th International Conference on, 31 2011-aug. 4 2011,
Conference on Communications, 2013. pp. 1 –6.
VERSION 1.0 45

[247] M. J. Yang, S. Y. Lim, H. J. Park, and N. H. Park, “Solving the data [267] J. Tan, S. Kavulya, R. Gandhi, and P. Narasimhan, “Visual, log-based
overload: Device-to-device bearer control architecture for cellular data causal tracing for performance debugging of mapreduce systems,” in
offloading,” Vehicular Technology Magazine, IEEE, vol. 8, no. 1, pp. Distributed Computing Systems (ICDCS), 2010 IEEE 30th Interna-
31–39, March 2013. tional Conference on, June 2010, pp. 795–806.
[248] K.-K. Yap, R. Sherwood, M. Kobayashi, T.-Y. Huang, M. Chan, [268] R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Stoica, “X-trace:
N. Handigol, N. McKeown, and G. Parulkar, “Blueprint for introducing a pervasive network tracing framework,” in Proceedings of the 4th
innovation into wireless mobile networks,” in Proceedings of the second USENIX conference on Networked systems design & implementation,
ACM SIGCOMM workshop on Virtualized infrastructure systems and ser. NSDI’07. Berkeley, CA, USA: USENIX Association, 2007, pp.
architectures, ser. VISA ’10. New York, NY, USA: ACM, 2010, pp. 20–20.
25–32. [269] V. Trivedi, “Software development: Debugging and testing,” in How to
[249] M. Bansal, J. Mehlman, S. Katti, and P. Levis, “Openradio: a pro- Speak Tech. Apress, 2014, pp. 89–95.
grammable wireless dataplane,” in Proceedings of the first workshop [270] A. Anand and A. Akella, “Netreplay: a new network primitive,”
on Hot topics in software defined networks, ser. HotSDN ’12. New SIGMETRICS Perform. Eval. Rev., vol. 37, no. 3, pp. 14–19, Jan. 2010.
York, NY, USA: ACM, 2012, pp. 109–114. [271] Y. Zhuang, E. Gessiou, S. Portzer, F. Fund, M. Muhammad, I. Beschast-
[250] S. Sundaresan, W. de Donato, N. Feamster, R. Teixeira, S. Crawford, nikh, and J. Cappos, “Netcheck: Network diagnoses from blackbox
and A. Pescapè, “Broadband internet performance: a view from the traces,” in 11th USENIX Symposium on Networked Systems Design
gateway,” SIGCOMM Comput. Commun. Rev., vol. 41, no. 4, pp. 134– and Implementation (NSDI 14). Seattle, WA: USENIX Association,
145, Aug. 2011. Apr. 2014, pp. 115–128.
[251] S. A. Mehdi, J. Khalid, and S. A. Khayam, “Revisiting traffic anomaly [272] N. Handigol, B. Heller, V. Jeyakumar, D. Maziéres, and N. McKe-
detection using software defined networking,” in Proceedings of the own, “Where is the debugger for my software-defined network?” in
14th international conference on Recent Advances in Intrusion Detec- Proceedings of the First Workshop on Hot Topics in Software Defined
tion, ser. RAID’11. Berlin, Heidelberg: Springer-Verlag, 2011, pp. Networks, ser. HotSDN ’12. New York, NY, USA: ACM, 2012, pp.
161–180. 55–60.
[252] P. Wette and H. Karl, “Which flows are hiding behind my wildcard [273] A. Wundsam, D. Levin, S. Seetharaman, and A. Feldmann,
rule?: adding packet sampling to OpenFlow,” in Proceedings of the “OFRewind: enabling record and replay troubleshooting for networks,”
ACM SIGCOMM 2013 conference on SIGCOMM, ser. SIGCOMM ’13. in Proceedings of the 2011 USENIX conference on USENIX annual
New York, NY, USA: ACM, 2013, pp. 541–542. technical conference, ser. USENIXATC’11. Berkeley, CA, USA:
[253] G. Bianchi, M. Bonola, G. Picierro, S. Pontarelli, and M. Monaci, USENIX Association, 2011, pp. 29–29.
“StreaMon: a data-plane programming abstraction for Software-defined [274] M. Canini, D. Venzano, P. Perešı́ni, D. Kostić, and J. Rexford, “A NICE
Stream Monitoring,” ArXiv e-prints, Nov. 2013. way to test OpenFlow applications,” in Proceedings of the 9th USENIX
[254] D. Kreutz, F. M. Ramos, and P. Verissimo, “Towards secure and conference on Networked Systems Design and Implementation, ser.
dependable software-defined networks,” in Proceedings of the second NSDI’12. Berkeley, CA, USA: USENIX Association, 2012, pp. 10–
ACM SIGCOMM workshop on Hot topics in software defined network- 10.
ing, ser. HotSDN ’13. New York, NY, USA: ACM, 2013, pp. 55–60. [275] C. Rotsos, N. Sarrar, S. Uhlig, R. Sherwood, and A. W. Moore,
[255] K. Kant, “Data center evolution: A tutorial on state of the art, issues, “OFLOPS: an open framework for OpenFlow switch evaluation,” in
and challenges,” Computer Networks, vol. 53, no. 17, pp. 2939 – 2965, Proceedings of the 13th international conference on Passive and Active
2009, virtualized Data Centers. Measurement, ser. PAM’12. Berlin, Heidelberg: Springer-Verlag,
2012, pp. 85–95.
[256] A. Greenberg, J. Hamilton, D. A. Maltz, and P. Patel, “The cost
[276] E. Al-Shaer and S. Al-Haj, “FlowChecker: configuration analysis and
of a cloud: Research problems in data center networks,” SIGCOMM
verification of federated OpenFlow infrastructures,” in Proceedings of
Comput. Commun. Rev., vol. 39, no. 1, pp. 68–73, Dec. 2008.
the 3rd ACM workshop on Assurable and usable security configuration,
[Online]. Available: http://doi.acm.org/10.1145/1496091.1496103
ser. SafeConfig ’10. New York, NY, USA: ACM, 2010, pp. 37–44.
[257] M. Bari, R. Boutaba, R. Esteves, L. Granville, M. Podlesny, M. Rab-
[277] A. Khurshid, W. Zhou, M. Caesar, and P. B. Godfrey, “VeriFlow:
bani, Q. Zhang, and M. Zhani, “Data center network virtualization: A
verifying network-wide invariants in real time,” in HotSDN, 2012.
survey,” Communications Surveys Tutorials, IEEE, vol. 15, no. 2, pp.
[278] M. Kuzniar, M. Canini, and D. Kostic, “OFTEN Testing OpenFlow
909–928, 2013.
Networks,” in Proceedings of the 1st European Workshop on Software
[258] P. Calyam, S. Rajagopalan, A. Selvadhurai, S. Mohan, A. Venkatara- Defined Networks (EWSDN), 2012.
man, A. Berryman, and R. Ramnath, “Leveraging OpenFlow for [279] G. Altekar and I. Stoica, “Focus Replay Debugging Effort On the Con-
resource placement of virtual desktop cloud applications,” in Integrated trol Plane,” Electrical Engineering and Computer Sciences University
Network Management (IM 2013), 2013 IFIP/IEEE International Sym- of California at Berkeley, Tech. Rep., May 2010.
posium on, 2013, pp. 311–319. [280] N. Handigol, B. Heller, V. Jeyakumar, D. Mazières, and N. McKeown,
[259] J. Parraga, “Avior,” 2013. [Online]. Available: http://openflow.marist. “I know what your packet did last hop: Using packet histories to
edu/avior troubleshoot networks,” in 11th USENIX Symposium on Networked
[260] GlobalNOC, “OESS - Open Exchange Software Suite,” 2013. [Online]. Systems Design and Implementation (NSDI 14). Seattle, WA: USENIX
Available: http://globalnoc.iu.edu/sdn/oess.html Association, Apr. 2014, pp. 71–85.
[261] C. Duckett, “Software Defined Networking: HP has an App Store [281] N. Ruchansky and D. Proserpio, “A (not) nice way to verify the
for that,” 2013. [Online]. Available: http://www.zdnet.com/software- OpenFlow switch specification: formal modelling of the OpenFlow
defined-networking-hp-has-an-app-store-for-that-7000021365/ switch using alloy,” in Proceedings of the ACM SIGCOMM 2013
[262] HP, “SDN App Store,” 2013. [Online]. conference on SIGCOMM, ser. SIGCOMM ’13. New York, NY, USA:
Available: http://h17007.www1.hp.com/us/en/networking/solutions/ ACM, 2013, pp. 527–528.
technology/sdn/devcenter/#sdnAppstore [282] H. Zeng, S. Zhang, F. Ye, V. Jeyakumar, M. Ju, J. Liu, N. McKeown,
[263] B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, and A. Vahdat, “Libra: Divide and conquer to verify forwarding
D. Beaver, S. Jaspan, and C. Shanbhag, “Dapper, a large-scale dis- tables in huge networks,” in 11th USENIX Symposium on Networked
tributed systems tracing infrastructure,” Google, Inc., Tech. Rep., 2010. Systems Design and Implementation (NSDI 14). Seattle, WA: USENIX
[264] L. Layman, M. Diep, M. Nagappan, J. Singer, R. Deline, and G. Veno- Association, Apr. 2014, pp. 87–99.
lia, “Debugging revisited: Toward understanding the debugging needs [283] P. Kazemian, G. Varghese, and N. McKeown, “Header space analysis:
of contemporary software developers,” in Empirical Software Engineer- Static checking for networks,” in Proceedings of the 9th USENIX
ing and Measurement, 2013 ACM / IEEE International Symposium on, Conference on Networked Systems Design and Implementation, ser.
Oct 2013, pp. 383–392. NSDI’12. Berkeley, CA, USA: USENIX Association, 2012, pp. 9–9.
[265] U. Erlingsson, M. Peinado, S. Peter, M. Budiu, and G. Mainar-Ruiz, [284] H. Mai, A. Khurshid, R. Agarwal, M. Caesar, P. B. Godfrey, and S. T.
“Fay: Extensible distributed tracing from kernels to clusters,” ACM King, “Debugging the data plane with anteater,” SIGCOMM Comput.
Trans. Comput. Syst., vol. 30, no. 4, pp. 13:1–13:35, Nov. 2012. Commun. Rev., vol. 41, no. 4, pp. 290–301, Aug. 2011.
[266] S. Tomaselli and O. Landsiedel, “Towards Lightweight Logging and [285] P. Kazemian, M. Chang, H. Zeng, G. Varghese, N. McKeown, and
Replay of Embedded, Distributed Systems,” in Proceedings of Work- S. Whyte, “Real time network policy checking using header space
shop ASCoMS (Architecting Safety in Collaborative Mobile Systems) analysis,” in Proceedings of the 10th USENIX conference on Networked
of the 32nd International Conference on Computer Safety, Reliability Systems Design and Implementation, ser. NSDI’13. Berkeley, CA,
and Security, M. ROY, Ed., Toulouse, France, Sep. 2013. USA: USENIX Association, 2013, pp. 99–112.
VERSION 1.0 46

[286] N. Handigol, B. Heller, V. Jeyakumar, B. Lantz, and N. McKeown, [308] P. Bosshart, G. Gibb, H.-S. Kim, G. Varghese, N. McKeown, M. Iz-
“Reproducible network experiments using container-based emulation,” zard, F. Mujica, and M. Horowitz, “Forwarding metamorphosis: fast
in Proceedings of the 8th international conference on Emerging net- programmable match-action processing in hardware for SDN,” in
working experiments and technologies, ser. CoNEXT ’12. New York, Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM,
NY, USA: ACM, 2012, pp. 253–264. ser. SIGCOMM ’13. New York, NY, USA: ACM, 2013, pp. 99–110.
[287] V. Antonenko and R. Smelyanskiy, “Global network modelling based [309] O. Ferkouss, I. Snaiki, O. Mounaouar, H. Dahmouni, R. Ben Ali,
on Mininet approach.” in Proceedings of the second ACM SIGCOMM Y. Lemieux, and C. Omar, “A 100gig network processor platform for
workshop on Hot topics in software defined networking, ser. HotSDN openflow,” in Network and Service Management (CNSM), 2011 7th
’13. New York, NY, USA: ACM, 2013, pp. 145–146. International Conference on, 2011, pp. 1–4.
[288] J. Teixeira, G. Antichi, D. Adami, A. Del Chiaro, S. Giordano, and [310] J. Naous, D. Erickson, G. A. Covington, G. Appenzeller, and N. McK-
A. Santos, “Datacenter in a box: Test your sdn cloud-datacenter eown, “Implementing an OpenFlow switch on the netfpga platform,”
controller at home,” in Software Defined Networks (EWSDN), 2013 in Proceedings of the 4th ACM/IEEE Symposium on Architectures for
Second European Workshop on, Oct 2013, pp. 99–104. Networking and Communications Systems, ser. ANCS ’08. New York,
[289] ns-3 project, “ns-3: OpenFlow switch support,” 2013. [Online]. Avail- NY, USA: ACM, 2008, pp. 1–9.
able: http://www.nsnam.org/docs/release/3.13/models/html/openflow- [311] G. Memon, M. Varvello, R. Laufer, T. Lakshman, J. Li, and M. Zhang,
switch.html “FlashFlow: a GPU-based Fully Programmable OpenFlow Switch,”
[290] J. Sommers, R. Bowden, B. Eriksson, P. Barford, M. Roughan, and University of Oregon, Tech. Rep., 2013.
N. Duffield, “Efficient network-wide flow record generation,” in IN- [312] Y. Luo, P. Cascon, E. Murray, and J. Ortega, “Accelerating Open-
FOCOM, 2011 Proceedings IEEE, 2011, pp. 2363–2371. Flow switching with network processors,” in Proceedings of the 5th
[291] ucb-sts, “STS - SDN troubleshooting simulator,” 2013. [Online]. ACM/IEEE Symposium on Architectures for Networking and Commu-
Available: http://ucb-sts.github.io/sts/ nications Systems, ser. ANCS ’09. New York, NY, USA: ACM, 2009,
[292] R. Sherwood and K.-K. Yap, “Cbench controller benchmarker,” 2011. pp. 70–71.
[Online]. Available: http://www.openflow.org/wk/index.php/Oflops [313] A. Rostami, T. Jungel, A. Koepsel, H. Woesner, and A. Wolisz,
[293] M. Jarschel, F. Lehrieder, Z. Magyari, and R. Pries, “A flexible “Oran: OpenFlow routers for academic networks,” in High Performance
OpenFlow-controller benchmark,” in Proceedings of the 2012 Switching and Routing (HPSR), 2012 IEEE 13th International Confer-
European Workshop on Software Defined Networking, ser. EWSDN ence on, 2012, pp. 216–222.
’12. Washington, DC, USA: IEEE Computer Society, 2012, pp. [314] G. Pongracz, L. Molnar, and Z. Kis, “Removing roadblocks from SDN:
48–53. [Online]. Available: http://dx.doi.org/10.1109/EWSDN.2012.15 OpenFlow software switch performance on intel DPDK,” in Software
[294] M. Gupta, J. Sommers, and P. Barford, “Fast, accurate simulation Defined Networks (EWSDN), 2013 Second European Workshop on, Oct
for sdn prototyping,” in Proceedings of the Second ACM SIGCOMM 2013, pp. 62–67.
Workshop on Hot Topics in Software Defined Networking, ser. HotSDN [315] B. Stephens, “Designing Scalable Networks for Future Large Datacen-
’13. New York, NY, USA: ACM, 2013, pp. 31–36. ters,” Ph.D. dissertation, Rice University, May 2012.
[295] A. R. Curtis, J. C. Mogul, J. Tourrilhes, P. Yalagandula, P. Sharma, [316] Y. Li, D. Zhang, K. Huang, D. He, and W. Long, “A memory-efficient
and S. Banerjee, “DevoFlow: scaling flow management for high- parallel routing lookup model with fast updates,” Comput. Commun.,
performance networks,” Comput. Commun. Rev., vol. 41, no. 4, pp. vol. 38, pp. 60–71, Feb. 2014.
254–265, Aug. 2011. [317] N. Katta, J. Rexford, and D. Walker, “Infinite CacheFlow in Software-
Defined Networks,” Princeton School of Engineering and Applied
[296] C. J. Casey, A. Sutton, and A. Sprintson, “tinyNBI: Distilling an
Science, Tech. Rep., October 2013.
API from essential OpenFlow abstractions,” CoRR, vol. abs/1403.6644,
[318] Intel Processors, “Software Defined Networking and Softwarebased
2014.
Services with Intel Processors,” Intel Corporation, 2012.
[297] L. Ogrodowczyk et al., “Hardware abstraction layer for non-OpenFlow
[Online]. Available: http://www.intel.com/content/dam/doc/white-
capable devices,” in The 30th Trans European Research and Education
paper/communications-ia-software-defined-networking-paper.pdf
Networking Conference (TNC). TERENA, 2014.
[319] G. Brebner, “Softly defined networking,” in Proceedings of the eighth
[298] A. Vidal, C. E. Rothenberg, and F. L. Verdi, “The libfluid OpenFlow ACM/IEEE symposium on Architectures for networking and communi-
driver implementation,” in SBRC, 2014. cations systems, ser. ANCS ’12. New York, NY, USA: ACM, 2012,
[299] M. APPELMAN and M. D. BOER, “Performance Analysis of Open- pp. 1–2.
Flow Hardware,” University of Amsterdam, Tech. Rep., Feb 2012. [320] C. Matsumoto, “Arista’s new hardware packs SDN,” 2012. [Online].
[300] K. Kannan and S. Banerjee, “Compact tcam: Flow entry compaction in Available: http://www.lightreading.com/document.asp?doc id=225028
tcam for power aware sdn,” in Distributed Computing and Networking, [321] A. Bianco, R. Birke, L. Giraudo, and M. Palacin, “OpenFlow Switch-
ser. Lecture Notes in Computer Science, D. Frey, M. Raynal, S. Sarkar, ing: Data Plane Performance,” in Communications (ICC), 2010 IEEE
R. Shyamasundar, and P. Sinha, Eds. Springer Berlin Heidelberg, International Conference on, may 2010, pp. 1 –5.
2013, vol. 7730, pp. 439–444. [322] Intel Corporation, “Intel data plane development kit,” 2014.
[301] J. Liao, “SDN System Performance,” June 2012. [Online]. Available: [Online]. Available: http://www.intel.com/content/dam/www/public/us/
http://pica8.org/blogs/?p=201 en/documents/guides/intel-dpdk-getting-started-guide.pdf
[302] B. Agrawal and T. Sherwood, “Modeling tcam power for next gen- [323] A. Sivaraman, K. Winstein, S. Subramanian, and H. Balakrishnan, “No
eration network devices,” in Performance Analysis of Systems and Silver Bullet: Extending SDN to the Data Plane,” in Twelfth ACM
Software, 2006 IEEE International Symposium on, 2006, pp. 120–129. Workshop on Hot Topics in Networks (HotNets-XII), College Park, MD,
[303] B. Owens, “OpenFlow Switching Performance: Not All TCAM Is Cre- November 2013.
ated Equal,” February 2013. [Online]. Available: http://packetpushers. [324] S. Hauger, T. Wild, A. Mutter, A. Kirstaedter, K. Karras, R. Ohlendorf,
net/openflow-switching-performance-not-all-tcam-is-created-equal/ F. Feller, and J. Scharf, “Packet processing at 100 gbps and beyond -
[304] B. Salisbury, “TCAMs and OpenFlow - What Every SDN Practitioner challenges and perspectives,” in Photonic Networks, 2009 ITG Sympo-
Must Know,” Jul. 2012. [Online]. Available: http://www.sdncentral. sium on, May 2009, pp. 1–10.
com/technology/sdn-openflow-tcam-need-to-know/2012/07/ [325] P. Bosshart, D. Daly, M. Izzard, N. McKeown, J. Rexford, D. Talayco,
[305] B. Stephens, A. Cox, W. Felter, C. Dixon, and J. Carter, “Past: scalable A. Vahdat, G. Varghese, and D. Walker, “Programming protocol-
ethernet for data centers,” in Proceedings of the 8th international independent packet processors,” CoRR, vol. abs/1312.1719, 2013.
conference on Emerging networking experiments and technologies, ser. [326] S. Azodolmolky, P. Wieder, and R. Yahyapour, “Performance evaluation
CoNEXT ’12. New York, NY, USA: ACM, 2012, pp. 49–60. of a scalable software-defined networking deployment,” in Software
[306] M. Kobayashi, S. Seetharaman, G. Parulkar, G. Appenzeller, J. Little, Defined Networks (EWSDN), 2013 Second European Workshop on, Oct
J. van Reijendam, P. Weissmann, and N. McKeown, “Maturing of 2013, pp. 68–74.
OpenFlow and software-defined networking through deployments,” [327] A. AuYoung, S. Banerjee, J. Lee, J. C. Mogul, J. Mudigonda, L. Popa,
Computer Networks, vol. 61, no. 0, pp. 151 – 175, 2014, special P. Sharma, and Y. Turner, “Corybantic: Towards the Modular Compo-
issue on Future Internet Testbeds Part I. [Online]. Available: sition of SDN Control Programs,” in Twelfth ACM Workshop on Hot
http://www.sciencedirect.com/science/article/pii/S138912861300371X Topics in Networks (HotNets-XII), College Park, MD, November 2013.
[307] J. C. Mogul and P. Congdon, “Hey, you darned counters!: Get off my [328] M. Desai and T. Nandagopal, “Coping with link failures in centralized
asic!” in Proceedings of the First Workshop on Hot Topics in Software control plane architectures,” in Communication Systems and Networks
Defined Networks, ser. HotSDN ’12. New York, NY, USA: ACM, (COMSNETS), 2010 Second International Conference on. IEEE, 2010,
2012, pp. 25–30. pp. 1–10.
VERSION 1.0 47

[329] H. Kim, J. Santos, Y. Turner, M. Schlansker, J. Tourrilhes, and A network calculus-based approach,” in IEEE GlobeCom 2013, Oct.
N. Feamster, “Coronet: Fault tolerance for software defined networks,” 2013.
in Network Protocols (ICNP), 2012 20th IEEE International Confer- [349] M. Marchetti, M. Colajanni, M. Messori, L. Aniello, and Y. Vigfusson,
ence on, Oct 2012, pp. 1–2. “Cyber attacks on financial critical infrastructures,” in Collaborative
[330] J. Kempf, E. Bellagamba, A. Kern, D. Jocha, A. Takacs, and P. Skold- Financial Infrastructure Protection, R. Baldoni and G. Chockler, Eds.
strom, “Scalable fault management for OpenFlow,” in Communications Springer Berlin Heidelberg, 2012, pp. 53–82.
(ICC), 2012 IEEE International Conference on, June 2012, pp. 6606– [350] S. Amin and A. Giacomoni, “Smart grid, safe grid,” Power and Energy
6610. Magazine, IEEE, vol. 10, no. 1, pp. 33–40, 2012.
[331] S. Sharma, D. Staessens, D. Colle, M. Pickavet, and P. Demeester, [351] A. Nicholson, S. Webber, S. Dyer, T. Patel, and H. Janicke, “SCADA
“Openflow: Meeting carrier-grade recovery requirements,” Comput. security in the light of cyber-warfare,” Computers & Security, vol. 31,
Commun., vol. 36, no. 6, pp. 656–665, Mar. 2013. no. 4, pp. 418 – 436, 2012.
[332] A. Panda, C. Scott, A. Ghodsi, T. Koponen, and S. Shenker, “Cap for [352] K.-K. R. Choo, “The cyber threat landscape: Challenges and future
networks,” in Proceedings of the second ACM SIGCOMM workshop research directions,” Computers & Security, vol. 30, no. 8, pp. 719 –
on Hot topics in software defined networking, ser. HotSDN ’13. New 731, 2011.
York, NY, USA: ACM, 2013, pp. 91–96. [353] D. Kushner, “The Real Story of Stuxnet,” IEEE Spectrum, Mar 2013.
[333] M. Kuźniar, P. Perešı́ni, N. Vasić, M. Canini, and D. Kostić, “Automatic [Online]. Available: http://spectrum.ieee.org/telecom/security/the-real-
failure recovery for software-defined networks,” in Proceedings of the story-of-stuxnet
Second ACM SIGCOMM Workshop on Hot Topics in Software Defined [354] R. Perez-Pena, “Universities face a rising barrage of cyberattacks,”
Networking, ser. HotSDN ’13. New York, NY, USA: ACM, 2013, pp. Jul. 2013. [Online]. Available: http://www.nytimes.com/2013/07/17/
159–160. education/barrage-of-cyberattacks-challenges-campus-culture.html
[334] A. Dixit, F. Hao, S. Mukherjee, T. Lakshman, and R. Kompella, [355] C. Tankard, “Advanced persistent threats and how to monitor and deter
“Towards an elastic distributed SDN controller,” in Proceedings of the them,” Network Security, vol. 2011, no. 8, pp. 16 – 19, 2011.
second ACM SIGCOMM workshop on Hot topics in software defined
[356] S. Sorensen, “Security implications of software-defined networks,”
networking, ser. HotSDN ’13. New York, NY, USA: ACM, 2013, pp.
2012. [Online]. Available: http://www.fiercetelecom.com/story/
7–12.
security-implications-software-defined-networks/2012-05-14
[335] R. Ramos, M. Martinello, and C. Esteve Rothenberg, “SlickFlow: Re-
[357] S. M. Kerner, “Is SDN Secure?” Mar 2013. [Online]. Available: http:
silient source routing in data center networks unlocked by OpenFlow,”
//www.enterprisenetworkingplanet.com/netsecur/is-sdn-secure.html
in Local Computer Networks (LCN), 2013 IEEE 38th Conference on,
Oct 2013, pp. 606–613. [358] A. Agapi, K. Birman, R. Broberg, C. Cotton, T. Kielmann, M. Millnert,
R. Payne, R. Surton, and R. van Renesse, “Routers for the cloud: Can
[336] J. T. Araújo, R. Landa, R. G. Clegg, and G. Pavlou, “Software-defined
the internet achieve 5-nines availability?” Internet Computing, IEEE,
network support for transport resilience,” in IEEE NOMS, 2014.
vol. 15, no. 5, pp. 72–77, 2011.
[337] E. Brewer, “Pushing the cap: Strategies for consistency and availabil-
ity,” Computer, vol. 45, no. 2, pp. 23–29, Feb. 2012. [359] R. Kloti, “Openflow: A security analysis,” Master’s thesis, Swiss
Federal Institute of Technology Zurich (ETH), Zurich, Swiss, 2013.
[338] T. Benson, A. Akella, and D. A. Maltz, “Network traffic characteristics
of data centers in the wild,” in Proceedings of the 10th ACM SIGCOMM [360] M. Wasserman and S. Hartman, “Security analysis of the open
Conference on Internet Measurement, ser. IMC ’10. New York, NY, networking foundation (onf) OpenFlow switch specification,” Internet
USA: ACM, 2010, pp. 267–280. Engineering Task Force, Apr 2013. [Online]. Available: https:
[339] M. Yu, J. Rexford, M. J. Freedman, and J. Wang, “Scalable flow-based //datatracker.ietf.org/doc/draft-mrw-sdnsec-openflow-analysis/
networking with difane,” SIGCOMM Comput. Commun. Rev., vol. 41, [361] S. Shin and G. Gu, “Attacking software-defined networks: A first
no. 4, pp. –, Aug. 2010. feasibility study,” in Proceedings of the second workshop on Hot topics
[340] M. F. Bari, A. R. Roy, S. R. Chowdhury, Q. Zhang, M. F. Zhani, in software defined networks, ser. HotSDN ’13. New York, NY, USA:
R. Ahmed, and R. Boutaba, “Dynamic controller provisioning in soft- ACM, 2013, pp. 1–2.
ware defined networks,” in 9th International Conference on Network [362] K. Benton, L. J. Camp, and C. Small, “OpenFlow vulnerability
and Service Management, ser. CNSM’13, 2013. assessment,” in Proceedings of the second ACM SIGCOMM workshop
[341] V. Yazici, M. O. Sunay, and A. O. Ercan, “Controlling a software- on Hot topics in software defined networking, ser. HotSDN ’13. New
defined network via distributed controllers,” in Proceedings of the York, NY, USA: ACM, 2013, pp. 151–152.
Conference on Implementing Future Media Internet Towards New [363] S. Hernan, S. Lambert, T. Ostwald, and A. Shostack, “Uncover security
Horizons, ser. 2012 NEM SUMMIT. Heidelberg, Germany: Eurescom design flaws using the STRIDE approach,” MSDN Magazine, Nov.
GmbH, Oct. 2012, pp. 16–22. 2006.
[342] D. Levin, A. Wundsam, B. Heller, N. Handigol, and A. Feldmann, [364] W. J. Bolosky, D. Bradshaw, R. B. Haagens, N. P. Kusters, and P. Li,
“Logically centralized? state distribution trade-offs in software defined “Paxos replicated state machines as the basis of a high-performance
networks,” in Proceedings of the first workshop on Hot topics in data store,” in Symposium on Networked Systems Design and Imple-
software defined networks, ser. HotSDN ’12. New York, NY, USA: mentation (NSDI), 2011, pp. 141–154.
ACM, 2012, pp. 1–6. [365] P. Sousa, A. Bessani, M. Correia, N. Neves, and P. Verissimo, “Highly
[343] M. Jarschel, S. Oechsner, D. Schlosser, R. Pries, S. Goll, and P. Tran- available intrusion-tolerant services with proactive-reactive recovery,”
Gia, “Modeling and performance evaluation of an OpenFlow archi- Parallel and Distributed Systems, IEEE Transactions on, vol. 21, no. 4,
tecture,” in Teletraffic Congress (ITC), 2011 23rd International, Sept pp. 452–465, April 2010.
2011, pp. 1–7. [366] R. Chua, “SDN security: Oxymoron? new interview with phil porras of
[344] R. Pries, M. Jarschel, and S. Goll, “On the usability of OpenFlow SRI international,” 2013. [Online]. Available: http://www.sdncentral.
in data center environments,” in Communications (ICC), 2012 IEEE com/technology/sdn-security-oxymoron-phil-porras-sri/2013/02/
International Conference on, June 2012, pp. 5533–5537. [367] J. Korniak, “The GMPLS controlled optical networks as industry
[345] J. Hwang, K. K. Ramakrishnan, and T. Wood, “Netvm: High per- communication platform,” Industrial Informatics, IEEE Transactions
formance and flexible networking using virtualization on commodity on, vol. 7, no. 4, pp. 671–678, Nov 2011.
platforms,” in 11th USENIX Symposium on Networked Systems Design [368] P. Fonseca, R. Bennesby, E. Mota, and A. Passito, “A replication
and Implementation (NSDI 14). Seattle, WA: USENIX Association, component for resilient OpenFlow-based networking,” in Network
Apr. 2014, pp. 445–458. Operations and Management Symposium (NOMS), 2012 IEEE, april
[346] Y. Dong, Z. Yu, and G. Rose, “Sr-iov networking in xen: Architecture, 2012, pp. 933 –939.
design and implementation,” in Proceedings of the First Conference [369] S. Vissicchio, L. Vanbever, and O. Bonaventure, “Opportunities and
on I/O Virtualization, ser. WIOV’08. Berkeley, CA, USA: USENIX research challenges of hybrid software defined networks,” SIGCOMM
Association, 2008, pp. 10–10. Comput. Commun. Rev., vol. 44, no. 2, pp. 70–75, Apr. 2014.
[347] B. Heller, R. Sherwood, and N. McKeown, “The controller placement [370] C. E. Rothenberg, M. R. Nascimento, M. R. Salvador, C. N. A.
problem,” in Proceedings of the first workshop on Hot topics in Corrêa, S. Cunha de Lucena, and R. Raszuk, “Revisiting routing control
software defined networks, ser. HotSDN ’12. New York, NY, USA: platforms with the eyes and muscles of software-defined networking,”
ACM, 2012, pp. 7–12. in Proceedings of the First Workshop on Hot Topics in Software Defined
[348] S. Azodolmolky, R. Nejabati, M. Pazouki, P. Wieder, R. Yahyapour, and Networks, ser. HotSDN ’12. New York, NY, USA: ACM, 2012, pp.
D. Simeonidou, “An analytical model for software defined networking: 13–18.
VERSION 1.0 48

[371] C. E. Rothenberg, A. Vidal, M. R. Salvador, C. Correa, S. Lucena, [390] D. Staessens, S. Sharma, D. Colle, M. Pickavet, and P. Demeester,
F. Farias, E. Cerqueira, and A. Abelem, “Hybrid networking towards “Software Defined Networking: Meeting Carrier Grade Requirements,”
a software defined era,” in Network Innovation through OpenFlow and in Local Metropolitan Area Networks (LANMAN), 2011 18th IEEE
SDN: Principles and Design book, Taylor & Francis LLC, CRC Press., Workshop on, oct. 2011, pp. 1 –6.
2014. [391] S. Sharma, D. Staessens, D. Colle, M. Pickavet, and P. Demeester,
[372] D. Levin, M. Canini, S. Schmid, and A. Feldmann, “Incremental “A demonstration of automatic bootstrapping of resilient OpenFlow
SDN deployment in enterprise networks,” in Proceedings of the ACM networks,” in Integrated Network Management (IM 2013), 2013
SIGCOMM 2013 conference on SIGCOMM, ser. SIGCOMM ’13. New IFIP/IEEE International Symposium on, 2013, pp. 1066–1067.
York, NY, USA: ACM, 2013, pp. 473–474. [392] R. Niranjan Mysore, A. Pamboris, N. Farrington, N. Huang, P. Miri,
[373] H. Lu, N. Arora, H. Zhang, C. Lumezanu, J. Rhee, and G. Jiang, S. Radhakrishnan, V. Subramanya, and A. Vahdat, “PortLand: A
“Hybnet: Network manager for a hybrid network infrastructure,” in scalable fault-tolerant layer 2 data center network fabric,” SIGCOMM
Proceedings of the Industrial Track of the 13th ACM/IFIP/USENIX Comput. Commun. Rev., vol. 39, no. 4, pp. 39–50, Aug. 2009.
International Middleware Conference, ser. Middleware Industry ’13. [393] A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri,
New York, NY, USA: ACM, 2013, pp. 6:1–6:6. D. A. Maltz, P. Patel, and S. Sengupta, “VL2: a scalable and flexible
[374] P. Bernier, “NTT Recognized with IBC Award for data center network,” in Proceedings of the ACM SIGCOMM 2009
SDN-based HDTV Service,” September 2013. [Online]. conference on Data communication, ser. SIGCOMM ’09. New York,
Available: http://www.sdnzone.com/topics/software-defined-network/ NY, USA: ACM, 2009, pp. 51–62.
articles/353466-ntt-recognized-with-ibc-award-sdn-based-hdtv.htm [394] A. Sadasivarao, S. Syed, P. Pan, C. Liou, I. Monga, C. Guok, and
[375] NTT DATA, “Infrastructure Services,” 2014. [Online]. Available: A. Lake, “Bursting data between data centers: Case for transport SDN,”
http://www.nttdata.com/global/en/services/infrastructure/solution.html in High-Performance Interconnects (HOTI), 2013 IEEE 21st Annual
[376] M. Wagner, “NTT Taps SDN to Enhance Cloud Flexibility,” March Symposium on, 2013, pp. 87–90.
2014. [Online]. Available: http://www.lightreading.com/ntt-taps-sdn- [395] J. C. Tanner, “Taking SDN to transport and beyond,” 2013.
to-enhance-cloud-flexibility/d/d-id/708133 [Online]. Available: http://www.telecomasia.net/content/taking-sdn-
[377] AT&T Inc., “AT&T Introduces the ”User-Defined Network transport-and-beyond
Cloud”: A Vision for the Network of the Future,” [396] S. Elby, “Carrier Vision of SDN,” 2012. [Online]. Available:
February 2014. [Online]. Available: http://www.att.com/gen/press- http://www.brighttalk.com/webcast/6985/58527
room?pid=25274&cdvn=news&newsarticleid=37439&mapcode= [397] B. Anwer, T. Benson, N. Feamster, D. Levin, and J. Rexford, “A
[378] B. Naudts, M. Kind, F. Westphal, S. Verbrugge, D. Colle, and M. Pick- slick control plane for network middleboxes,” in Proceedings of the
avet, “Techno-economic analysis of software defined networking as second ACM SIGCOMM workshop on Hot topics in software defined
architecture for the virtualization of a mobile network,” in Software networking, ser. HotSDN ’13. New York, NY, USA: ACM, 2013, pp.
Defined Networking (EWSDN), 2012 European Workshop on, Oct 2012, 147–148.
pp. 67–72. [398] C. Gerlach and H.-M. Foisel, “OIF carrier WG requirements on trans-
[379] ONF, “Operator network monetization through OpenFlow- port networks in SDN architectures,” Optical Internetworking Forum,
enabled SDN,” Apr. 2013. [Online]. Avail- The Optical Internetworking Forum, 48377 Fremont Blvd., Suite 117,
able: https://www.opennetworking.org/images/stories/downloads/sdn- Fremont, CA 94538, Tech. Rep., September 2013.
resources/solution-briefs/sb-network-monetization.pdf [399] L. Velasco, A. Asensio, J. Berral, A. Castro, and V. Lopez, “Towards
[380] P. Skoldstrom and W. John, “Implementation and evaluation of a a carrier SDN: An example for elastic inter-datacenter connectivity,”
carrier-grade OpenFlow virtualization scheme,” in Software Defined in Optical Communication (ECOC 2013), 39th European Conference
Networks (EWSDN), 2013 Second European Workshop on, Oct 2013, and Exhibition on, 2013, pp. 1–3.
pp. 75–80. [400] A. Alba, G. Alatorre, C. Bolik, A. Corrao, T. Clark, S. Gopisetty,
[381] H. H. Gharakheili and V. Sivaraman, “Virtualizing National Broad- R. Haas, R. Kat, B. Langston, N. Mandagere, D. Noll, S. Padbidri,
band Access Infrastructure,” in Proceedings of the 9th international R. Routray, Y. Song, C. Tan, and A. Traeger, “Efficient and agile
conference on Emerging networking experiments and technologies, ser. storage management in software defined environments,” IBM Journal
CoNEXT ’13. New York, NY, USA: ACM, 2013. of Research and Development, vol. 58, no. 2, pp. 1–12, March 2014.
[382] Pacnet, “Pacnet Offers First Pan-Asia Network- [401] W. Arnold, D. Arroyo, W. Segmuller, M. Spreitzer, M. Steinder, and
as-a-Service Architecture,” 2013. [Online]. Avail- A. Tantawi, “Workload orchestration and optimization for software
able: http://www.cmo.com.au/mediareleases/17701/pacnet-offers-first- defined environments,” IBM Journal of Research and Development,
pan-asia-network-as-a-service/ vol. 58, no. 2, pp. 1–12, March 2014.
[383] N. D. Corporation, “NTT DATA Advance in SDN Business [402] C. Dixon, D. Olshefski, V. Jain, C. DeCusatis, W. Felter, J. Carter,
Provides Highly-Flexible Control of Network by Software,” June M. Banikazemi, V. Mann, J. Tracey, and R. Recio, “Software defined
2012. [Online]. Available: http://www.nttdata.com/global/en/news- networking to support the software defined environment,” IBM Journal
center/pressrelease/2012/060801.html of Research and Development, vol. 58, no. 2, pp. 1–14, March 2014.
[384] S. Das, A. Sharafat, G. Parulkar, and N. McKeown, “MPLS with [403] IBM Systems and Technology Group, “IBM software defined network
a simple OPEN control plane,” in Optical Fiber Communication for virtual environments,” IBM Corporation, Tech. Rep., January 2014.
Conference and Exposition (OFC/NFOEC), 2011 and the National [404] IBM Systems, “Manage all workloads with an efficient, scalable
Fiber Optic Engineers Conference, 2011, pp. 1–3. software defined environment (SDE),” 2014. [Online]. Avail-
[385] M. Casado, T. Koponen, S. Shenker, and A. Tootoonchian, “Fabric: a able: http://www-03.ibm.com/systems/infrastructure/us/en/software-
retrospective on evolving SDN,” in Proceedings of the first workshop defined-environment/
on Hot topics in software defined networks, ser. HotSDN ’12. New [405] A. Lara, A. Kolasani, and B. Ramamurthy, “Network innovation
York, NY, USA: ACM, 2012, pp. 85–90. using OpenFlow: A survey,” Communications Surveys Tutorials, IEEE,
[386] M. Martinello, M. Ribeiro, R. de Oliveira, and R. de Angelis Vitoi, vol. 16, no. 1, pp. 493–512, First 2014.
“Keyflow: a prototype for evolving SDN toward core network fabrics,” [406] Y. Jarraya, T. Madi, and M. Debbabi, “A survey and a layered taxonomy
Network, IEEE, vol. 28, no. 2, pp. 12–19, March 2014. of software-defined networking,” Communications Surveys Tutorials,
[387] N. Feamster, J. Rexford, S. Shenker, R. Clark, R. Hutchins, D. Levin, IEEE, vol. PP, no. 99, pp. 1–1, 2014.
and J. Bailey, “SDX: A software-defined internet exchange,” IETF [407] B. Nunes, M. Mendonca, X. Nguyen, K. Obraczka, and T. Turletti,
86 Proceedings, Orlando, US, March 2013. [Online]. Available: “A survey of software-defined networking: Past, present, and future of
http://www.ietf.org/proceedings/86/slides/slides-86-sdnrg-6 programmable networks,” Communications Surveys Tutorials, IEEE,
[388] J. P. Stringer, Q. Fu, C. Lorier, R. Nelson, and C. E. Rothenberg, vol. PP, no. 99, pp. 1–18, 2014.
“Cardigan: deploying a distributed routing fabric,” in Proceedings of
the second ACM SIGCOMM workshop on Hot topics in software
defined networking, ser. HotSDN ’13. New York, NY, USA: ACM,
2013, pp. 169–170.
[389] C.-Y. Hong, S. Kandula, R. Mahajan, M. Zhang, V. Gill, M. Nanduri,
and R. Wattenhofer, “Achieving high utilization with software-driven Diego Kreutz received his Computer Science degree, MSc degree in Infor-
WAN,” in Proceedings of the ACM SIGCOMM 2013 conference on matics, and MSc degree in Production Engineering from Federal University
SIGCOMM, ser. SIGCOMM ’13. New York, NY, USA: ACM, 2013, of Santa Maria. Over the past 11 years he has worked as an Assistant
pp. 15–26. Professor in the Lutheran University of Brazil and in the Federal University
VERSION 1.0 49

of Pampa, and as a researcher member of the Software/Hardware Integration Siamak Azodolmolky received his Computer Engineering degree from
Lab (LISHA) at Federal University of Santa Catarina. Out of the academia, Tehran University and his first MSc. degree in Computer Architecture from
he has also experience as an independent technical consultant on network Azad University in 1994 and 1998 respectively. He was employed by Data
operations and management for small and medium enterprises and government Processing Iran Co. (IBM in Iran) as a Software Developer, Systems Engineer,
institutions. Currently, he is a PhD student at Faculty of Sciences of Univer- and as a Senior R& D Engineer during 1992-2001. He received his second
sity of Lisbon, Portugal, involved in research projects related to intrusion MSc. degree with distinction from Carnegie Mellon University in 2006. He
tolerance, security, and future networks including the TRONE, and SecFuNet joined Athens Information Technology (AIT) as a Research Scientist and
international projects. His main research interests are in network control Software Developer in 2007, while pursuing his PhD degree. In August 2010,
platforms, software-defined networks, intrusion tolerance, system security and he joined the High Performance Networks research group of the School of
dependability, high performance computing, and cloud computing. Computer Science and Electronic Engineering (CSEE) of the University of
Essex as a Senior Research Officer. He received his PhD from Universitat
Politécnica de Catalunya (UPC) in 2011. He has been the technical investigator
of various national and EU funded projects. Software Defined Networking
(SDN) has been one of his research interests since 2010, in which he has
been investigating the extension of OpenFlow towards its application in
core transport (optical) networks. He has published more than 50 scientific
papers in international conferences, journals, and books. Software Defined
Networking with OpenFlow is one of his recent books. Currently, he is
with Gesellschaft für Wissenschaftliche Datenverarbeitung mbH Göttingen
Fernando M. V. Ramos Fernando M. V. Ramos is an Assistant Professor in (GWDG) as a Senior Researcher and has lead SDN related activities since
the University of Lisbon. Previous academic positions include those of Teach- September 2012. He is a professional member of ACM and a senior member
ing Assistant (supervisor) in the University of Cambridge, in the ISEL and in of IEEE.
the University of Aveiro. Over the past 12 years he has taught over a dozen
courses: from physics (Electromagnetism) to EE (digital electronics, electric
circuits, telecommunication systems and foundations) to CS (operating and
distributed systems, computer networks, algorithms, programming languages).
Periods outside academia include working as a researcher in Portugal Telecom
and in Telefonica Research. He holds a PhD degree from the University
of Cambridge where he worked on IPTV networks. His current research
interests are: software-defined networking, network virtualization, and cloud
computing, with security and dependability as an orthogonal concern.

Paulo Verı́ssimo Paulo Verı́ssimo is a Professor of the Department of

Computer Science and Engineering, U. of Lisbon Faculty of Sciences (FCUL-
http://www.di.fc.ul.pt/ pjv), elected member of the Board of the U. of Lisbon
and Director of LaSIGE (http://lasige.di.fc.ul.pt). He is currently Chair of
the IFIP WG 10.4 on Dependable Computing and Fault-Tolerance and vice-
Chair of the Steering Committee of the IEEE/IFIP DSN conference. PJV is Steve Uhlig Steve Uhlig obtained a Ph.D. degree in Applied Sciences from
Fellow of the IEEE and Fellow of the ACM. He is associate editor of the the University of Louvain, Belgium, in 2004. From 2004 to 2006, he was a
Elsevier Int’l Journal on Critical Infrastructure Protection. Verı́ssimo leads Postdoctoral Fellow of the Belgian National Fund for Scientific Research
the Navigators group of LaSIGE, and is currently interested in distributed (F.N.R.S.). His thesis won the annual IBM Belgium/F.N.R.S. Computer
architectures, middleware and algorithms for: adaptability and safety of real- Science Prize 2005. Between 2004 and 2006, he was a visiting scientist at Intel
time networked embedded systems; and resilience of secure and dependable Research Cambridge, UK, and at the Applied Mathematics Department of
large-scale systems. He is author of over 170 peer-refereed publications and University of Adelaide, Australia. Between 2006 and 2008, he was with Delft
co-author of 5 books. University of Technology, the Netherlands. Prior to joining Queen Mary, he
was a Senior Research Scientist with Technische Universität Berlin/Deutsche
Telekom Laboratories, Berlin, Germany. Starting in January 2012, he is the
Professor of Networks and Head of the Networks Research group at Queen
Mary, University of London.

Christian Esteve Rothenberg Christian Esteve Rothenberg is an Assistant

Professor in the Faculty of Electrical and Computer Engineering at University
of Campinas (UNICAMP), where he received his Ph.D. in 2010. From 2010
to 2013, he worked as Senior Research Scientist in the areas of IP systems
and networking at CPqD Research and Development Center in Telecommuni-
cations (Campinas, Brazil), where he was technical lead of R&D acitivities in
the field of OpenFlow/SDN such as the RouteFlow project, the OpenFlow 1.3
Ericsson/CPqD softswitch, or the ONF Driver competition. Christian holds
the Telecommunication Engineering degree from Universidad Politécnica de
Madrid (ETSIT - UPM), Spain, and the M.Sc. (Dipl. Ing.) degree in Electrical
Engineering and Information Technology from the Darmstadt University
of Technology (TUD), Germany, 2006. Christian holds two international
patents and has over 50 publications including scientific journals and top-
tier networking conferences such as SIGCOMM and INFOCOM. Since April
2013, Christian is an ONF Research Associate.