IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO.

2, APRIL 2018 895

Toward Unified Control of Networks of

Switches and Sensors Through a
Network Operating System
Angelos-Christos G. Anadiotis, Member, IEEE, Sebastiano Milardo , Giacomo Morabito ,
and Sergio Palazzo, Senior Member, IEEE

Abstract—Wireless sensor networks (WSNs) are an integral platforms. The academic research community is following
part of an Internet of Things (IoT) ecosystem. In fact, a typi- (and in certain cases riding) this wave and today IoT is one
cal IoT service deployment considers an integrated network of of the hottest research topic in computer science, whereas it
switches and sensors (as well as actuators). In order to manage
this heterogeneous network infrastructure, the use of software is also spreading to other domains, such as transportation,
defined networking (SDN) technologies is anticipated. However, production and manufacturing, as well as environmental and
existing SDN-enabling technologies consider only isolated net- social sciences.
works and there has been no concrete solution that treats the Fig. 1 depicts a standard IoT ecosystem, where cloud ser-
whole network in a unified way. In this paper, a network oper- vices, such as SVC-1, SVC-2, and SVC-N, receive data from
ating system is proposed as a way to address the above issue.
Starting from the open network operating system, which cur- networks of sensors through a transport network of switches.
rently supports only OpenFlow networks, the proposed solution More specifically, typical IoT deployments are either per-
leverages SDN-WISE to integrate WSNs and achieves unified sys- formed vertically, i.e., along with the supporting infrastructure,
tem representation and treatment. The proposed system has been or leverage data originally generated by wireless sensor net-
prototyped and is publicly available in open source. Experiments works (WSNs) and finally stored in a data center. In the first
have been executed demonstrating that, by exploiting the pro-
posed solution, it is also possible to improve efficiency of the case, the service provider has full control over the devices,
overall network. but with increased cost and operational complexity. The cost
is significantly lower in the second case which, furthermore,
Index Terms—Internet of Things (IoT), software defined
networking (SDN), wireless sensor networks (WSNs). perfectly fits the most widely used WSN protocol stack, such
as 6LoWPAN and ZigBee. In fact, the above protocol stacks
are perfectly designed for the case in which sensor nodes for-
I. I NTRODUCTION ward data toward their sink(s). However, processing in the
ARKET forecasts released by different institutions cloud causes loss of context, as it cannot be unambiguously
M anticipate the deployment of larger and larger num-
bers of Internet of Things (IoT) devices in the near future.
described in metadata, and real-time processing cannot be
efficiently supported.
For example, the most recent study conducted by IDC for The fragmentation of the IoT landscape, due to the
Intel and the United Nations forecasts that there will be deployment of many Intranets of Things, which cannot
around 200 billion IoT devices by 2020.1 As a result of cooperate effectively and efficiently, instead of a unique
these high expectations industry is investing huge resources IoT [1], sets interoperability constraints despite the integration
in the development and deployment of IoT technologies and efforts [2], [3].
Software defined networking (SDN) solutions for WSNs
Manuscript received January 31, 2017; revised August 31, 2017; accepted
January 26, 2018. Date of publication February 12, 2018; date of cur-
come to question this way of operation, though. Indeed, by
rent version April 10, 2018. This work was supported by the European leveraging SDN technologies, it has now become feasible to
Commission in the framework of the In-Network Programmability for Next- control the infrastructure without having to deploy specific
Generation Personal Cloud Service Support Project through the Horizon
2020 Program (Call H2020-ICT-2014-1) under Grant 644672. (Corresponding
solutions.
author: Giacomo Morabito.) Even though the SDN-driven approach is definitely a step
A.-C. G. Anadiotis was with CNIT UdR Catania, 95125 Catania, Italy. He toward the vision of an easily (re-)configurable IoT environ-
is now with the Data-Intensive Applications and Systems Laboratory, École
Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland (e-mail:
ment, as also outlined in [4] and [5], there is still a major issue
[email protected]). that must be addressed. More specifically, traditional SDN
S. Milardo is with the Dipartimento Energia, Ingegneria dell’Informazione solutions focus on device-level protocols, thereby providing
e Modelli Matematici, University of Palermo, 90133 Palermo, Italy (e-mail:
[email protected]).
isolated views of the individual network segments consisting
G. Morabito and S. Palazzo are with the Dipartimento di Ingegneria an IoT ecosystem. In fact, considering for example that the net-
Elettrica Elettronica e Informatica, University of Catania, 95125 Catania, Italy work of switches of Fig. 1 supports OpenFlow [6] and that the
(e-mail: [email protected]; [email protected]).
Digital Object Identifier 10.1109/JIOT.2018.2805191
WSN supports some specialized SDN solution such as SDN-
1 [Online]. Available: http://www.intel.com/content/www/us/en/internet-of- WISE [7] or Sensor OpenFlow [8], there will be two different
things/infographics/guide-to-iot.html SDN controllers managing each network segment. Bearing in
2327-4662 c 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
896 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018

exploiting both the extensible nature of the NOS, which

allows the implementation of any device-level proto-
col, and the common abstractions for sensors and rules,
which are enforced, irrespectively of the underlying
protocol.
Note that in this paper we are not proposing a new proto-
col (or set of protocols) which should replace existing ones,
like ZigBee and 6LoWPAN. Instead, we are proposing a gen-
eral approach that leverages the existence of SDN solutions
in both the infrastructured and infrastructureless segments of
the IoT to provide application developers with abstractions of
the whole network and the related services that can be easily
utilized in efficient and flexible manner.
In this context, we have focused on SDN-WISE for the
infrastructureless segment because it is stateful and, as a con-
sequence, Turing complete. Therefore, it can implement any
protocol if correctly instructed by the corresponding network
application. For example, we have implemented a network
Fig. 1. Typical IoT ecosystem. application that programs the sensor nodes so that they exe-
cute geographical routing [9] and we are implementing another
mind the large number of network segments involved in a network application that enables nodes to interpret and gen-
standard IoT ecosystem, it is obvious that, compared to the erate typical 6LoWPAN signaling messages, such as router
existing application-level integration solutions, this approach solicitation, router advertisement, node registration, and node
introduces very high complexity and therefore, does not scale. confirmation so that interaction with standard 6LoWPAN
Actually, a similar problem arose in the first days of SDN nodes is possible.
even for fixed networks, where several controllers existed, each As a whole the proposed solution has several advantages for
one optimized for different scenarios. In order to tackle with all actors involved in the IoT ecosystem. In fact, by exploit-
this issue, the traditional SDN controllers evolved to become ing the overall view of the network topology it is possible
network operating systems (NOSs), which provide high-level to establish the optimal communication path according to the
abstractions for several heterogeneous devices and protocols application requirements. Considering for example the net-
and can be used to manage even large networks. The open work of Fig. 1, it is typically more efficient for nodes n2 and n6
NOS (ONOS), depicted in Fig. 1, is an example of such a sys- to communicate through the switches than through the WSN,
tem. Unfortunately, NOSs have not been sufficiently exploited which would be the standard solution. This is valuable for final
in the context of WSNs, furthermore, no attention has been users (they can obtain higher levels of QoS) and network oper-
paid to the network-level integration of IoT devices. ators (they can increase resource efficiency). Furthermore, by
In this paper, we leverage the developments in NOSs and leveraging the virtualization/abstractions of physical resources
we specify a unified system, which controls an IoT ecosystem it is possible for application developers to focus on the imple-
at the network layer in a holistic way. More specifically, our mentation of the application, ignoring the specific low level
contributions are as follows. details of the physical objects and their interconnections. This
1) We provide generic abstractions for sensor nodes as well would involve significant reduction in the development cost
as encapsulations of non-IP packets, in order to enable and time to market. Finally, by leveraging the possibility to
IoT service providers to deploy their network configu- program the behavior of nodes through network applications
ration over any allocated infrastructure, transparently. running on top of the NOS it becomes possible to have IoT
2) We propose an architecture enabling the IoT service nodes that, in principle, can execute any existing or novel IoT
providers to create networks of sensors and switches protocol. It is even possible to have IoT nodes that handle
on the fly, while supporting packet forwarding through packets belonging to different applications according to dif-
different network segments. ferent protocols. This very high level of flexibility can be
3) We develop a framework which supports a shift in the exploited.
IoT communication paradigm from device-to-cloud to 1) Integrate IoT platforms running different IoT solutions,
device-to-device, allowing the direct communication of so contributing to solve the IoT fragmentation problem,
sensor nodes over networks of switches. In this way, and making it possible to implement and deploy appli-
clouds of sensors can be created in an ad-hoc manner, cations that exploit the services offered by a larger pool
exploiting subsets of the already deployed WSN devices, of IoT resources.
thereby fostering granularity and reusability. 2) Improve the quality of service offered to applications
4) We introduce the software sensor nodes (SSNs) con- and increase efficiency. This is possible because it is
cept which acts as an integration element for different well known fact that there is no one-fits-all solution in
IoT protocol stacks so enabling the communication of the IoT context, and by exploiting the proposed approach
devices adopting different standards. This is achieved by any application can rely on the most appropriate set
ANADIOTIS et al.: TOWARD UNIFIED CONTROL OF NETWORKS OF SWITCHES AND SENSORS THROUGH NOS 897

of protocols (which should be implemented as network NOSs. Recently, two open source NOSs have been pro-
application on top of the NOS). posed: 1) OpenDaylight [15] and 2) ONOS [16]. OpenDaylight
The individual components of the proposed architecture provides a project, IoTDM, which focuses on the IoT, but
have been previously described in [10]. The main goal of this it follows the data-level integration approach, rather than a
paper is to propose an architecture which accommodates the network-oriented solution, which would actually complement
above components and explain how they interact in order to existing solutions for networks of switches.2 More specif-
achieve the aforementioned contributions. Also, in this paper, ically, IoTDM does not enable the direct communication
the overall system is assessed in an application scenario involv- among devices, whereas it relies on application-level pro-
ing the use of MapReduce to process the data produced by IoT tocols, such as CoAP,3 in order to integrate heterogeneous
devices. devices. Apart from being a hybrid, rather than a pure network-
In order to maximize compatibility with relevant standards, level approach, IoTDM considers only one of the several
major requirement we considered in this paper has been existing standards and as has been well documented [2], [4],
to minimize the number of new abstractions needed in the much effort is still required in order to settle to a particular
ONOS framework and avoid any need for modifications of standard for the IoT. Even though the architectural princi-
the OpenFlow (and other) standard(s). ples between ONOS and OpenDaylight are similar, in this
The rest of this paper is organized as follows. Section II out- paper, we build our prototype leveraging ONOS, currently sup-
lines related work in the area of SDN protocols and existing porting only OpenFlow networks, by extending it to support
NOS-based solutions for the IoT. Then, Section III provides SDN-WISE enabled WSNs.
a general view of the overall system architecture and sum-
marizes the motivation for the proposed system. Section IV III. M OTIVATION AND A RCHITECTURE
introduces the concept of the SSN, which is used to integrate
The key motivation of this paper is the realization of a
WSNs and networks of switches. The fundamental opera-
unified system which enables the communication among het-
tions offered by the proposed system, namely sensor node
erogeneous devices in order to create IoT ecosystems on the
registration and integrated packet forwarding are explained in
fly, exploiting the infrastructure already in place. Following
Sections V and VI, respectively. A MapReduce-based use case
a pure network-level approach, the proposed system does not
for the system is presented in Section VII and it is used to
depend on any particular IoT architecture. This is achieved by
extract performance metrics, which highlight measurable ben-
the following.
efits of the proposed approach. Finally, the conclusions are
1) Employing SDN technologies in both the transport and
drawn in Section VIII.
the sensor networks.
2) Specifying generic, protocol-agnostic abstractions for
II. BACKGROUND AND R ELATED W ORK sensor nodes and network packets.
A. SDN Protocols 3) Introducing the concept of the SSN, an SDN-enabled
SDN has been envisioned as a way to reduce complexity sensor node, which can be easily integrated with any
and increase flexibility of network configuration and manage- existing WSN deployment, by implementing the corre-
ment [6]. In SDN networks management operations (control sponding protocol procedures inside the NOS.
plane) are logically centralized and physically decoupled from Fig. 2 depicts the extended ONOS architecture, where
forwarding operations (data plane), so allowing to easily the dark-colored components are required to integrate soft-
update the behavior of the network. ware defined sensor nodes [10]; whereas the light colored
The most popular and widely used SDN solution for wired components have been kept back in their original ONOS
networks is OpenFlow [6]. OpenFlow success mainly stems implementation.
from its standardized interfaces, which can work on networks The proposed architecture follows the logical separation
made of heterogeneous switches. of ONOS into subsystems. The SensorNode Subsystem pro-
Despite the vast adoption of OpenFlow, and SDN in gen- vides the abstractions as well as an up-to-date registry of the
eral, in the wired domain, there has been no such widely sensor nodes connected to the system. More specifically, the
accepted solution in wireless networks and, especially, WSNs. SensorNode API is a generic, extensible, API, which specifies
Early solutions such as [11] and [12] have been outperformed a generic sensor node, so that it can be efficiently accessed
by recent ones, either based on TinyOS [13] or Contiki [7]. from the other layers of the NOS. The SensorNode API is used
In the context of this paper, we consider SDN-WISE [7], for both sending information, such as forwarding rules, to the
since it is open source and provides a software prototype sensor nodes as well as receiving information, like the battery
of an SDN-enabled sensor node, which has been extensively level and their neighbors, from them. The API is extensible,
used in practice [14], which can be easily integrated into any so that application-specific sensors can also be easily inte-
emulation framework. grated. The SensorNode Manager implements the SensorNode
API functionality. Essentially, it is a registry for all the sensor
B. SDN Controllers nodes managed by the NOS by maintaining up-to-date infor-
mation either for logging reasons (e.g., forwarding rules) or
At the control plane, research has evolved from sim-
ple services, called controllers, to more sophisticated solu- 2 [Online]. Available: https://wiki.opendaylight.org/view/IoTDM_Overview
tions that can efficiently manage large scale systems, the 3 [Online]. Available: https://tools.ietf.org/html/rfc7252
898 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018

Fig. 2. ONOS integrated architecture.

for status maintenance (e.g., battery level). The SensorNode the proposed architecture includes the server-side SDN-WISE
Provider translates protocol-specific data to data that can be implementation, which is responsible for communicating with
accessed through the SensorNode API and vice-versa. More the sensor nodes and provides access to low-level proto-
specifically, incoming data are decoded into ONOS-specific col semantics. This is achieved through a protocol-specific
APIs, so that they can be effectively managed by the higher API, with respect to the already existing one in ONOS
layers. Outgoing data are encoded into protocol-specific APIs, for OpenFlow, which is able to encode/decode information
by leveraging the Protocols layer. coming from the WSN/NOS.
The SensorFlowRule Subsystem is used to create and install Observe that in the APP layer, ONOS supports the imple-
flow rules, by leveraging the protocol supported by the tar- mentation of network applications. In fact, these applications
get device. The SensorFlowRule API specifies the high-level are responsible for creating the rules, which are then installed
semantics for creating rules that are sensor-node specific and in the nodes by leveraging the corresponding subsystems.
can look at any portion of the packets to classify them into When a sensor node, which is connected to ONOS, receives a
flows. This is supported by SDN-WISE and enables a high packet that it cannot handle, it attaches this packet to a request
degree of flexibility which goes well beyond what OpenFlow and sends it to ONOS. ONOS decapsulates the various packet
can obtain. The SensorFlowRule Provider is used to encode layers as described previously in the context of the subsys-
flow rules, by leveraging the corresponding protocol, so that tems and, in case it cannot directly be handled by them (e.g.,
they can be sent to the sensor nodes. Let us note here that the it is not a trivial beacon packet), it is delivered to the net-
FlowRule Manager and the FlowRuleProvider API have been work application associated with this node. Then, the network
kept unchanged with respect to their original ONOS imple- application decides how to handle this packet. Note here that
mentation. This has been achieved by exploiting the high-level a network application essentially implements a network pro-
semantics of ONOS combined with the generic design of the tocol. Therefore, even existing protocols, such as 6LoWPAN
FlowRule API. or ZigBee can, in principle, be implemented in ONOS. For
The SensorPacket Subsystem handles incoming and out- instance, if an SDN-WISE node receives a 6LoWPAN packet,
going packets from/to the sensor nodes, by using a non- it will encapsulate it in an SDN-WISE packet by inserting
IP, generic packet encapsulation API. In particular, the it in the payload. Then, the network application will decode
SensorPacket API provides access to incoming and outgoing that payload, it will understand that it is a 6LoWPAN packet
packets from/to sensor nodes. Even though the network-level and will send the corresponding reply packet to the sensor
packet format depends on the corresponding underlying net- node.
work protocol, the SensorPacket Provider encapsulates all Given the heterogeneity of the supported devices (sensor
this information into ONOS high-level abstractions. The cur- nodes and switches), ONOS has to associate each device type
rent API has been designed by considering requirements of with a set of subsystems in order to handle them accordingly.
generic sensor nodes and provides access to information such This is achieved by adopting system-wide naming schemes
as host and destination address, protocol version and other in the form of uniform resource identifiers for every com-
low-level details. However, this API can be extended to sup- ponent. For example, an OpenFlow switch is represented as
port application-specific protocols by building on top of the of:000000001 and an SDN-WISE node as sdnwise:00000101.
existing system. In this way, every device connected to the system as well
In the protocols layer, the specific semantics of the under- as every device-specific software module are registered using
lying network protocol are implemented. More specifically, the same prefix. As a result, ONOS can deliver packets and
ANADIOTIS et al.: TOWARD UNIFIED CONTROL OF NETWORKS OF SWITCHES AND SENSORS THROUGH NOS 899

Then, the SSN, can send the appropriate neighborhood dis-

covery packets to n1 and n4 , through the respective sinks n2
and n3 , and become neighbor with both them. Furthermore, it
can also exchange neighborhood information with both these
nodes, thereby creating a symmetric link n1 −SSN−n4 , which
in fact hinders the path of the intermediate sinks and switches,
giving the impression to the sensor nodes that they have two
hops distance.
This way, every SSN actually represents the set of sensor
nodes of a particular WSN that are connected to it. Therefore,
the overall network topology can be abstracted as network of
SDN-enabled switches and sensors. Moreover, dynamic ad-
hoc networks can be created by connecting individual nodes
over an heterogeneous infrastructure, consisting of switches
and SDN-WISE sensor nodes, either physical or virtual (soft-
ware) ones. As a result, transferable network configurations
can be enforced on the fly to already existing WSNs, with
Fig. 3. Heterogeneous network integration with SSN.
respect to the IoT vision presented in [4], by controlling the
whole infrastructure in a holistic way.
The proposed NOS performs holistic management of the
events coming from the devices to the respective subsystems networks of switches and sensors by introducing generic
and viceversa, transparently. abstractions for sensor nodes and packets and by performing
Despite the complexity and the variety of operations pro- integrated packet forwarding transparently across these net-
vided by the NOS, in this paper, we focus on a fundamental works. Note that the implementation of 6LoWPAN and other
subset, in order to illustrate how the proposed solution achieves WSN protocols in the NOS has not been prototyped in the
the contributions outlined at the beginning of this section are context of this paper, since it mostly involves development
accomplished. More specifically, in the following, we first work, whereas it does not add any complexity in the overall
introduce the concept of the SSN and then, we explain the architecture; eventually, the challenge is for SDN-WISE nodes
procedure implemented for registering and representing a sen- to communicate over OpenFlow switches.
sor node in the system. Then, we illustrate how different sensor
network applications can register their packet types to the sys-
tem, so that they can efficiently make use of the corresponding V. S ENSOR N ODE R EGISTRATION AND R EPRESENTATION
flow rules capabilities. Then, we present the integrated for- A key requirement of the proposed system is the unified
warding network application, which essentially bridges the gap representation of the whole network, consisting of both sensor
between networks of switches and sensors by leveraging both nodes and switches. This is achieved by introducing new and
existing and newly introduced NOS components. extending existing core ONOS abstractions, which provide fine
grained access to the network components, namely the devices
and the links. More specifically, two access layers are consid-
IV. S OFTWARE S ENSOR N ODE
ered for devices and links, the base and the enhanced layer.
The fundamental component introduced at the device level The base layer is used to create an internal graph representa-
for network-layer integration is the SSN. SSNs are essentially tion of the network topology, where every node is represented
SDN-WISE-enabled nodes, which do not necessarily corre- by a Device and every edge by a Link. The enhanced layer
spond to a physical device, with respect to the concept of for the sensor nodes, SensorNode, extends the Device
OVSwitch for OpenFlow networks. In fact, such nodes may and it is used to access implementation-specific information,
as well be deployed as services in the network, e.g., using such as the energy level of a sensor, its geographic coordi-
network function virtualization, by leveraging the SDN-WISE nates and its neighborhood. Whereas, the enhanced layer for
node implementation,4 which has already been used for WSNs the sensor links, SensorLink, extends the Link to provide
emulation. information on the received signal strength indicator (RSSI)
Due to their programmability stemming from their SDN of a wireless link, which is typically used as a metric for the
support, SSNs can be easily integrated with any IoT proto- weight of an edge representing a wireless link.
col, by just implementing the corresponding APIs as an NOS Information provided at the enhanced layer can be further
application. This way, an SSN can be virtually attached to extended through annotations, implemented as key-value pairs.
any WSN by just sending the corresponding signaling mes- Such annotations are typically useful for passing application-
sages to the nodes it has to connect to. For example, in Fig. 3, specific parameters to ONOS, which are required to make
suppose that nodes n1 and n2 belong to a 6LoWPAN net- network-layer decisions. For example, consider a WSN that
work, whereas nodes n3 and n4 belong to a ZigBee network. contains two types of sensor nodes, temperature and pressure
and assigns a different aggregator to each type. In order to
4 [Online]. Available: https://github.com/sdnwiselab forward the packets to the corresponding aggregator, ONOS
900 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018

(a)

(b)

Fig. 4. Representation of the (a) sink node in ONOS and (b) its protocol
stack.
Fig. 5. Sensor node registration flowchart.

needs to be aware of the sensor type each node carries. This

information cannot be contained in any network packet header, then, use the corresponding response to identify the link, in the
though. Therefore, through the annotations, such information same respect as with links between two SDN-WISE nodes. For
can be passed to the system from an external service, rather example, SDN-WISE nodes can generate and interpret typi-
than the network, so that it can be later used for taking cal 6LoWPAN signaling messages, such as router solicitation,
networking decisions, such as packet forwarding. router advertisement, node registration, and node confirma-
The registration of SDN-WISE nodes is quite straight- tion. In this way it is possible for SDN-WISE nodes to build
forward and is achieved through the SDN-WISE protocol a network topology overlayed at the 6LoWPAN nodes.
mechanisms by leveraging the SensorLink abstraction pro- This is a fundamental feature of the proposed solu-
vided by the proposed system. Links between sensors are tion as there is already a large number of WSN devices
periodically reported to ONOS through the REPORT packets already deployed worldwide (e.g., according to an ABI market
issued by every sensor and containing the list of its neighbors research 850 million IEEE 802.15.4 devices have been shipped
and the associated RSSI. Links between sensors (sinks) and in 2016).
switches are reported through the DP_CONNECTION packets, On the ONOS side, there are two subsystems involved
which contain the MAC address of the sink nodes. in the sensor node registration process: 1) the SensorPacket
In fact, as shown in Fig. 4(a) our extension of ONOS repre- Subsystem and 2) the SensorNode Subsystem. The flowchart
sents sinks both as sensors (for the wireless link) and as edge given in Fig. 5 depicts the role of the SensorPacket Subsystem,
hosts (for the wired link), because they typically carry two for example. When a REPORT packet arrives, a direct
different types of network interfaces. This design approach is SensorLink is created between that node and all its neighbors.
the consequence of the two major requirements: 1) to avoid In the special case of the DP_CONNECTION, a standard edge
any change in the standard OpenFlow and 2) minimize the Link is created, so that ONOS treats the sink as the edge of the
number of novel abstractions to be introduced in ONOS. To network of switches. Links between SDN-WISE and standard
this purpose sink nodes will run the protocol stack represented sensor nodes are created in the context of the ONOS appli-
in Fig. 4(b). cation implementing the corresponding protocol, based on the
At the physical and link layers two sets of protocols will be SensorLink abstraction. Note that these are also edge Links, as
implemented that are specific of the infrastructured and WSN the standard nodes cannot accept SDN rules and, thus, they are
segments, such as IEEE 802.3 and IEEE 802.15.4, respec- treated as regular hosts. After the links have been established,
tively. SDN-WISE is executed on top of the IEEE 802.15.4 the network topology is updated accordingly.
interface; whereas a relay layer is also executed which is ONOS keeps an updated view of the WSN using the
responsible of the operations needed to bridge the two different SensorNode Subsystem. More specifically, the SensorNode
network segments; one of its major roles is packet adaptation, Provider checks for sensor nodes connectivity, based on the
for example. reports received by them, whereas it creates a new device
Finally, links between SDN-WISE nodes and others imple- upon a sensor node arrival to the system. At the high level,
menting any other stack (e.g., 6LoWPAN), are established by the SensorNodeManager introduces functionality to store and
leveraging the communication protocol of the standard nodes. access fundamental information on the nodes, such as their
More specifically, SDN-WISE node can be instructed by the address, their battery level, the sink they are connected to
NOS to send the corresponding protocol-specific packets and or, in case the node is a sink itself, the switch that it is
ANADIOTIS et al.: TOWARD UNIFIED CONTROL OF NETWORKS OF SWITCHES AND SENSORS THROUGH NOS 901

attached to. Note that this information is either directly com-

ing or implicitly derived from the REPORT packets. Sensor
nodes are abstracted by the SensorNode data structure, which
extends the device, thus directly enabling its integration with
the topology, while providing the abstractions described ear-
lier in this section. All information regarding sensor nodes are
currently stored in hash tables. However, in case the networks
grow larger, more sophisticated and highly efficient in-memory
databases can be leveraged (e.g., Hekaton, VoltDB, etc.), in
order to address any scalability issue, without any changes to
the existing API.

VI. N ETWORK -W IDE PACKET R EPRESENTATION

AND F ORWARDING
In our solution, the packets sent by sensor nodes to ONOS,
in order to obtain a corresponding rule, are delivered to
the Integrated Forwarding application. The latter extracts the
source and the destination and calculates the shortest path con-
sidering the whole topology.5 Wireless link cost is calculated
based on the link RSSI and is dynamically passed to the path
extraction algorithm. Accordingly, in the example of Fig. 1, a
packet from node n1 to node n5 will follow the path includ-
ing switches, since it contains fewer wireless links, which are Fig. 6. Integrated forwarding flowchart.
most costly. Whereas, a standard controller would suggest the
path of sensor nodes only, since it would not have a full view
of the topology.
Unfortunately, packets coming from sensor nodes have dif- Fig. 6 describes the steps followed by the Integrated
ferent format than the standard IP packets, sent by hosts Forwarding application when it receives a packet (typically
and are currently supported by ONOS. Accordingly, we have from the Packet Provider). After the whole path from source
introduced a new API, the SensorPacket API, on top of the to destination has been calculated, it is broken into individual
existing PacketManager, as shown in Fig. 2, in order to enable segments, which separate the sequence of switches and sensor
ONOS to deal with the new packet formats. SensorPacket API nodes, as described in the flowchart of Fig. 6. Each segment
includes the PacketType, which represents sensor packet types is represented as a set of links connecting only devices of the
similarly to the type of service of IP packets. The main novelty same type. Accordingly, every time a link connecting devices
of this API is that it allows different network applications to of different type is met, the current segment is closed and a
register their own packet types, without interfering with each new one is opened.
other, thereby fostering reuse of existing packet headers with For example, considering that node n1 of Fig. 1 sends a
application decoupling. packet with destination node n5 and that ONOS calculates as
Once ONOS can deal with packets generated by sensors, shortest path the one passing through the OpenFlow switches,
then, in order to achieve integrated forwarding we need to Integrated Forwarding will split the path into two different
define how ONOS will instruct the network nodes in such a segments. More specifically, when ONOS receives the packet,
way that the traffic flow can traverse sequences of switches it triggers the process described in the flowchart of Fig. 6.
and sensors, transparently. A major problem in this context is After the path is calculated, the first link is considered and
the passage between one segment of switches and one segment switch s1 is added to the segment of switches, by entering
of sensors (or viceversa), since they use different instruction the flowchart parts 1, 2, 5, and 6. Then, the switches s2 and
sets. In our solution, when a packet arrives to the system, s3 are also added to the same segment by entering parts 1,
the SensorPacket Provider checks its type and, if it is not 2, and 5. When the link s1 → n7 is visited, the segment of
used for signaling, it encapsulates it into an ONOS-specific switches is closed, by executing the operations shown in parts
packet format that can be accessed by the SensorPacket API. 1, 2, and 4. Observe that ONOS understands n7 as host h2 in
Accordingly, incoming packets are transformed to Ethernet this case, since it is connected to switch s3 . Finally, for each
frames, using ONOS core APIs. Then, they are delivered to one of the two remaining sensor links from n7 to n5 , parts
the PacketProcessors, which are typically implemented in the 1, 2, and 3 are considered. After that step, there are no more
application layer. links in the path and hence, the active sensor node segment is
closed as well and the process ends.
5 Note that, even though not explicitly studied, multicast can also be sup- For each generated segment, the appropriate flow rule sub-
ported, since it can be approached as a set of multiple source, destination system is used in order to specify the forwarding rules.
pairs. More specifically, if the request is made by a sensor, the
902 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018

We consider a WSN of 23 nodes connected with a transport

network of six switches, as depicted in Fig. 7. Each sen-
sor node implements the SDN-WISE protocol, whereas each
switch implements OpenFlow. Switches and sensor nodes are
deployed in Mininet, which has been extended to support the
SDN-WISE SSNs.
Sensor nodes are equipped with sensors of four different
types: 1) temperature; 2) humidity; 3) pressure; and 4) noise.
Values generated by each of these types are associated with
the corresponding key in the context of the map function.
According to MapReduce, there has to be exactly one node,
namely the reducer, which collects all values associated
with a single key. The selection of the reducer for each key
is made as explained in [18] and is outside the scope of this
paper.
In order to select the appropriate reducer for each key, it
is essential to know the sensor types that every node car-
Fig. 7. Network topology with different reducers for each case. ries, which is not included in the signaling messages. This
is achieved in the application layer by leveraging the exten-
sibility of the SensorNode API, based on the inherent ONOS
SensorFlowRule Subsystem is used, whereas if it is made by Annotations API. This enables the update of the nodes descrip-
a switch, the OpenFlowRule API is leveraged. tion by inserting the corresponding sensor types as key-value
pairs of the form sensor, sensor type.
In our experiments, every node carries a random number of
VII. C ASE S TUDY—I N -N ETWORK P ROCESSING sensor types. Furthermore, a map function is loaded in each
W ITH M AP R EDUCE node by ONOS, which periodically generates key-value pairs
As we have already discussed, the proposed solution has for each sensor type and pushes them to the network layer.
several advantages ranging from very high flexibility to the At the first execution, the node does not know how to treat
possibility for developers to implement IoT services and appli- the packet generated by the map function and therefore, it
cations ignoring the specific low level details of the physical makes a request to the NOS. The latter selects the reducer
objects and their interconnections. In this section, however, and, through Integrated Forwarding application, installs the
we will focus on the performance improvement that can appropriate rules to every device, as described in the previous
be achieved by exploiting the overall view of the network section.
topology, including both infrastructured and infrastructureless We have implemented a prototype of the proposed solution
segments. for TI CC2530 devices as well as the software modules that
To this purpose we will compare the performance obtained extend Mininet to consider sensor nodes also. Our prototype
by the proposed approach to the performance obtained when has been released in open source and can be downloaded at
the infrastructured segments of the network implement a the following URLs.
shortest path routing whereas the infrastructureless segments 1) https://github.com/sdnwiselab/onos (source code of
implement RPL,6 which is the most widely deployed routing ONOS).
protocol for 6LoWPAN networks [17]. 2) https://github.com/sdnwiselab/sdn-wise-cc2530 (source
More specifically, the effectiveness of the proposed system code for the TI CC2530 devices).
is assessed by considering the case, where MapReduce opera- 3) https://github.com/sdnwiselab/onos-sdn-wise-app-
tions are executed inside the WSN network and the results are samples/tree/master/mapreduce (source code of the
sent to a service deployed in a host outside the WSN, as sug- MapReduce application and the Mininet scripts).
gested in [18]. For example, considering the system of Fig. 1, Since we have a small number of sensors, the results shown
suppose that the sensor nodes are executing MapReduce oper- in this paper have been obtained by using the Mininet sim-
ations and then, the results are sent to service SVC-1, deployed ulator. In our experiments energy related metrics have been
in the cloud, by sending the data over the OpenFlow network. calculated by elaborating the logs of the simulation consid-
Let us explicitly observe that in-network processing for big ering the information about battery and energy consumption
data applications has already been successfully applied in data provided in the datasheets of the TI CC2530.
centers [19], [20]. In the use case addressed here, we shift this The use of Integrated Forwarding first affects the selection
paradigm to in-network processing for sensor networks. of the reducers, which is made considering the cost of commu-
nication among the nodes, derived from the forwarding path.
6 Although a large number of RPL improvements have been proposed in the In particular, as shown in Fig. 7, in the standard WSN case, the
recent past, we have chosen the standard RPL implementation because we are node in the center of the network is selected to be a reducer.
focusing on the advantages obtained by exploiting the complete and integral
view of the overall network rather than the possibility to select shorter paths On the other hand, when Integrated Forwarding is enabled, the
inside the infrastructureless segment. optimal reducer is one of the sinks (node 1 in this case). In
ANADIOTIS et al.: TOWARD UNIFIED CONTROL OF NETWORKS OF SWITCHES AND SENSORS THROUGH NOS 903

(a) (b)

(c) (d)

Fig. 8. Nodes resource usage. (a) Remaining battery CDF. (b) delay. (c) nodes traffic CDF. (d) Rules CDF.

order to have a common point of reference, the evaluation will sensor nodes in the plain WSN and the integrated forwarding
be made considering that the reducer will be node 1 in both approach. Overall, the integrated forwarding approach results
the standard WSN and the integrated forwarding approach, in less communication between the nodes, which is essential
since the goal is to study the communication aspects of the for WSNs and is the key factor for the reduction on the energy
proposed architecture and not the MapReduce implementation consumption shown in Fig. 8(a).
as a whole. Finally, the proposed integrated forwarding approach results
The use of sensor nodes resources after executing the experi- in less flow rules in the nodes, as depicted in Fig. 8(d). This
ment for 10 min is shown in Fig. 8. More specifically, Fig. 8(a) means less signaling messages and lower memory occupancy
shows the CDF of the remaining energy level of the sensor for the nodes, which is very important given the strict limi-
nodes after the experiment is over. tations on both communication and hardware resources. The
As shown in the figure, in the integrated forwarding case, reduction of flow rules in the integrated forwarding approach
there is always a larger fraction of nodes with more residual is the result of the use of the core network. Indeed, when
battery than in the typical one. Moreover, since the difference using the WSN alone to forward packets, each intermediate
increases as the energy levels become lower, the network life- node requires 1 rule per destination. Therefore, for each path
time is expected to be significantly higher when using the of N hops, which traverses the core network, a reduction of N
proposed approach. rules at most (e.g., in sink-to-sink communication) is achieved
Furthermore, as shown in Fig. 8(b), in case the network of for the WSN.
switches is used, i.e., when nodes 18, 19, 21, 22, and 23 are
sending data to node 1 (reducer), the delivery times are sig- VIII. C ONCLUSION
nificantly smaller than in the typical WSN-specific approach. This paper presents a network operating system which
The reason is that the core network has much larger capacity, is used to integrate networks of switches with networks
and therefore packets are forwarded much faster than in the of sensors, in view of the IoT. Starting from ONOS, an
context of the WSN. existing solution for OpenFlow networks, we have intro-
The effectiveness of the use of the core network also duced new components, which efficiently abstract both the
becomes clear by considering Fig. 8(c), which shows the CDF network elements and their operations. Moreover, we have
of the overall number of packets sent and received by the explained how the proposed extensions can work together with
904 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018

existing modules, eventually enabling ONOS to perform holis- Angelos-Christos G. Anadiotis (GS’13–M’14) received the Dipl.-Ing. degree
tic network management and integrated packet forwarding in from the Department of Electrical and Computer Engineering, University of
Thessaly, Volos, Greece, in 2008, the M.Sc. degree in advanced information
software defined transport and WSNs. systems from the Department of Informatics, University of Piraeus, Piraeus,
The proposed system has been prototyped, whereas its eval- Greece, in 2011, and the Dr.-Ing. degree from the School of Electrical and
uation is performed in the context of a novel in-network Computer Engineering, NTUA, Athens, Greece, in 2014.
From 2014 to 2015, he was with the University of Catania Research Unit,
packet processing approach, which enables the execution of Catania, Italy, as a Researcher. Since 2016, he has been a Post-Doctoral
MapReduce operations in WSNs. Researcher with the École Polytechnique Fédérale de Lausanne, Lausanne,
Switzerland. His current research interests include large scale distributed
systems, software-defined networks, and data management systems. He has
R EFERENCES authored or co-authored several publications and input documents to ISO
[1] M. Zorzi, A. Gluhak, S. Lange, and A. Bassi, “From today’s INTRAnet standards in the above areas.
of things to a future INTERnet of Things: A wireless-and mobility- Dr. Anadiotis has served as a Project Editor of the second version of
related view,” IEEE Wireless Commun., vol. 17, no. 6, pp. 44–51, ISO/IEC 23006-3 and a Co-Guest Editor of the Transactions on Emerging
Dec. 2010. Telecommunication Technologies (Wiley).
[2] A. Al-Fuqaha, A. Khreishah, M. Guizani, A. Rayes, and
M. Mohammadi, “Toward better horizontal integration among IoT
services,” IEEE Commun. Mag., vol. 53, no. 9, pp. 72–79, Sep. 2015.
[3] M. A. Razzaque, M. Milojevic-Jevric, A. Palade, and S. Clarke,
“Middleware for Internet of Things: A survey,” IEEE Internet Things
J., vol. 3, no. 1, pp. 70–95, Feb. 2016. Sebastiano Milardo received the bachelor’s and master’s degrees in computer
[4] V. Cerf and M. Senges, “Taking the Internet to the next physical level,” engineering from the University of Catania, Catania, Italy, in 2011 and 2013,
Computer, vol. 49, no. 2, pp. 80–86, Feb. 2016. respectively. He is currently pursuing the Ph.D. degree in information and
[5] P. D. Dio et al., “Exploiting state information to support QoS in
software-defined WSNs,” in Proc. Mediterr. Ad Hoc Netw. Workshop communication technologies at the University of Palermo, Palermo, Italy.
(Med-Hoc-Net), Jun. 2016, pp. 1–7. From 2014 to 2015, he was with the Italian National Consortium of
[6] N. McKeown et al., “OpenFlow: Enabling innovation in campus net- Telecommunications, Catania, as a Researcher within the NEWCOM# and
works,” SIGCOMM Comput. Commun. Rev., vol. 38, no. 2, pp. 69–74, SIGMA Projects. From 2016 to 2017, he was with the Senseable City
Mar. 2008. Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA,
[7] L. Galluccio, S. Milardo, G. Morabito, and S. Palazzo, “SDN-WISE: as a visiting student. His current research interests include network operating
Design, prototyping and experimentation of a stateful SDN solution systems, software defined networks, sensor networks, and protocols for the
for wireless sensor networks,” in Proc. IEEE Conf. Comput. Commun. Internet of Things and Big Data.
(INFOCOM), Apr. 2015, pp. 513–521.
[8] T. Luo, H.-P. Tan, and T. Q. S. Quek, “Sensor OpenFlow: Enabling
software-defined wireless sensor networks,” IEEE Commun. Lett.,
vol. 16, no. 11, pp. 1896–1899, Nov. 2012.
[9] A. C. Anadiotis, L. Galluccio, S. Milardo, G. Morabito, and S. Palazzo,
“Offloading software defined WSNs through distributed geographic for- Giacomo Morabito received the Laurea degree in electrical engineering and
warding,” Dipartimento di Ingegneria Elettrica Elettronica e Informatica,
Univ. Catania, Catania, Italy, Rep. 1, Jan. 2017. [Online]. Available: Ph.D. degree in electrical, computer, and telecommunications engineering
http://bit.ly/2EpwlKp from the Istituto di Informatica e Telecomunicazioni, University of Catania,
[10] A.-C. G. Anadiotis, L. Galluccio, S. Milardo, G. Morabito, Catania, Italy, in 1996 and 2000, respectively.
and S. Palazzo, “Towards a software-defined network operat- From 1999 to 2001, he was with the Broadband and Wireless Networking
ing system for the IoT,” in Proc. IEEE 2nd World Forum Laboratory, Georgia Institute of Technology, Atlanta, GA, USA, as a Research
Internet Things (WF-IoT), Milan, Italy, Dec. 2015, pp. 579–584, Engineer. Since 2001, he has been with the Dipartimento di Ingegneria
doi: 10.1109/WF-IoT.2015.7389118. Elettrica Elettronica e Informatica, University of Catania, where he is cur-
[11] A. Mahmud and R. Rahmani, “Exploitation of OpenFlow in wire- rently an Associate Professor. His current research interest includes analysis
less sensor networks,” in Proc. Int. Conf. Comput. Sci. Netw. Technol. and solutions for wireless networks.
(ICCSNT), vol. 1. Harbin, China, Dec. 2011, pp. 594–600. Dr. Morabito serves on the Editorial Boards of IEEE Wireless
[12] S. Costanzo, L. Galluccio, G. Morabito, and S. Palazzo, “Software
defined wireless networks: Unbridling SDNs,” in Proc. Eur. Workshop Communication Magazine, Computer Networks (Elsevier), and Wireless
Softw. Defined Netw. (EWSDN), Darmstadt, Germany, Oct. 2012, Networks (ACM/Baltzer). He has been Co-Guest Editor for the IEEE
pp. 1–6. T RANSACTIONS ON M ULTIMEDIA, Computer Networks (Elsevier), and
[13] B. Trevizan de Oliveira, C. B. Margi, and L. B. Gabriel, “TinySDN: Mobile Networks and Applications (ACM/Baltzer).
Enabling multiple controllers for software-defined wireless sensor net-
works,” in Proc. IEEE Latin America Conf. Commun. (LATINCOM),
Nov. 2014, pp. 1–6.
[14] C. Buratti et al., “Testing protocols for the Internet of Things on the
EuWIn platform,” IEEE Internet Things J., vol. 3, no. 1, pp. 124–133,
Feb. 2016. Sergio Palazzo (M’92–SM’99) received the degree in electrical engineering
[15] J. Medved, R. Varga, A. Tkacik, and K. Gray, “OpenDaylight: Towards from the University of Catania, Catania, Italy, in 1977.
a model-driven SDN controller architecture,” in Proc. IEEE 15th Int. Since 1987, he has been with the University of Catania, where he is cur-
Symp. World Wireless Mobile Multimedia Netw. (WoWMoM), Sydney, rently a Professor of telecommunications networks. In 1994, he was with the
NSW, Australia, Jun. 2014, pp. 1–6. International Computer Science Institute, Berkeley, CA, USA, as a Senior
[16] “ONOS—A SDN network operating system for service providers,”
Menlo Park, CA, USA, ONF and Open Netw. Lab, White Paper, 2014. Visitor. In 2003, he was with the University of Canterbury, Christchurch,
[Online]. Available: http://bit.ly/2HE8WpZ New Zealand, as a recipient of the Visiting Erskine Fellowship. His current
[17] T. Winter et al., “Rpl: Ipv6 routing protocol for low-power and lossy research interests include modeling, optimization, and control of wireless net-
networks,” IETF, Fremont, CA, USA, RFC 6550, 2013. works, with applications to cognitive and cooperative networking, SDN, and
[18] A.-C. G. Anadiotis, G. Morabito, and S. Palazzo, “An SDN-assisted sensor networks.
framework for optimal deployment of mapreduce functions in WSNs,” Prof. Palazzo has been serving on the Technical Program Committee of
IEEE Trans. Mobile Comput., vol. 15, no. 9, pp. 2165–2178, Sep. 2016, INFOCOM, IEEE Conference on Computer Communications, since 1992.
doi: 10.1109/TMC.2015.2496582. He has been the General Chair of some ACM conferences (MobiHoc 2006
[19] L. Mai et al., “Supporting application-specific in-network processing and MobiOpp 2010), and currently is a member of the MobiHoc Steering
in data centres,” in Proc. ACM SIGCOMM Conf. (SIGCOMM), Committee. He has also been the TPC Co-Chair of some other conferences,
2013, pp. 519–520. [Online]. Available: http://doi.acm.org/
including IFIP Networking 2011, IWCMC 2013, and European Wireless 2014.
10.1145/2486001.2491733
[20] P. Costa, A. Donnelly, A. Rowstron, and G. O’Shea, “Camdoop: He also served on the Editorial Board of several journals, including the
Exploiting in-network aggregation for big data applications,” in IEEE/ACM T RANSACTIONS ON N ETWORKING, the IEEE T RANSACTIONS
Proc. 9th USENIX Conf. Netw. Syst. Design Implement. (NSDI), San ON M OBILE C OMPUTING , IEEE Wireless Communications Magazine,
Jose, CA, USA, 2012, p. 3. [Online]. Available: http://dl.acm.org/ Computer Networks, Ad Hoc Networks, and Wireless Communications and
citation.cfm?id=2228298.2228302 Mobile Computing.