Cloud, the soul of any IOT system

• The IoT revolution comes with trillions of connected devices; however, the real value
of IoT is in the advanced processing of the collected data.

• By nature, IoT data is more dynamic, heterogeneous and unstructured than typical
business data.

• It demands more sophisticated, IoT specific analytics to make it meaningful.

• The exploitation in the Cloud of data obtained in real time from sensors is therefore
very much a necessity.

• This data processing leads to advanced proactive and intelligent applications and
services.

• The connection of IoT and BigData can offer:

i. Deep understanding of the context and situation;
ii. Real-time actionable insight;
iii. Performance optimization;
iv. Proactive and predictive knowledge.

• Cloud technologies offer decentralized and scalable information processing and

analytics, and data management capabilities.
Cloud-based IoT Platform

• According to the National Institute of Standards and Technology (NIST) definition,

Cloud computing is a model for enabling convenient, on-demand network access to a
shared pool of configurable computing resources that can be rapidly provisioned and
released with minimal management effort or service provider interaction.

• The Cloud paradigm can be delivered using essentially three different service models
namely, infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software
as a Service (SaaS).

• A Cloud-based IoT platform is then a dynamic and flexible resource sharing platform
delivering IoT services and offers scalable resources and services management.

• The exploitation of IoT data depends on massive resources, which should be available
when needed and scaled back when not needed.
IaaS, PaaS and SaaS Paradigms

A Cloud based IoT platform needs usually to select one from the three different service
models: IaaS, PaaS or SaaS.

• IaaS allows delivering computer infrastructure on an outsourced basis in order to

support enterprise operations.

• This service model is based on the paradigm of virtualization of resources.

• The initial success of the Cloud is due to the possibility to embed practically any
legacy applications within Virtual Machines (VMs), which are managed by an external
stakeholder.

• This permits to relieve the application owner from managing physical infrastructures.

• PaaS, on the other hand, provides a platform allowing customers to develop, run, and
manage applications.

• It removes the complexity of building and maintaining the infrastructure typically

associated with developing and deploying an application.
• Typically, a PaaS framework will compile an application from its source code, and
then deploy it inside lightweight VMs, or containers.

• Furthermore, PaaS environments offer an interface to scale up or down applications, or

to schedule various tasks within the applications.

• Finally, SaaS is a software licensing and delivery model in which software is licensed
on a subscription basis and is centrally hosted.

• It is sometimes referred to as “on-demand software”.

• SaaS is typically accessed by users using a thin client via a web browser.

• Cloud-based IoT platforms are usually based on the SaaS paradigm and provide IoT-
related services using a web interface on a pay-per-use basis.

• For example, a service such as Xively1 provides a web service with a database able to
store sensors data points. This data is then processed and displayed in various
graphics.
Requirements of IoT BigData Analytics Platform

• An IoT BigData analytic platform should be able to dynamically

manage IoT data and provide connectivity with the diverse
heterogeneous objects, considering the interoperability issues.

• It is able to derive useful information and knowledge from large

volume of IoT data.

• The platform shall offer ubiquitous accessibility and connectivity of

the diverse objects, services and users, in a mobile context.

• It shall allow dynamic management and orchestration of users, a

huge amount of connected devices as well as massive amount of
data produced by those devices.

• Finally it shall allow personalization of users and services, providing

services based on users' preference and requirements including real-
world context.

Intelligent and Dynamic

• The platform should include intelligent and autonomic features in

• The platform should also be capable to make proactive decisions,
dynamic deployment, and intelligent decisions based on the
understanding of the context of the environments, users and
applications requirements.

• The platform provides dynamic resources management for IoT,

considering performance targets and constraints.

• This includes offloading workload from clients/hosts to the Cloud and

dynamic resources and service migration

Distributed

• The platform includes distributed information processing and

computing capabilities, distributed storage, distributed intelligence, and
distributed data management capabilities.
• These capabilities should be distributed across smart devices,
gateway/server and multiple cloud environments. The processing
capability needs to be migrated closer to users, to save bandwidth.

Scalable
• The platform needs to be scalable in order to address the needs of a
variable number of the devices, services and users.
Real-Time
• The platform needs to be able to process data in real-time, i.e.
providing a fast analysis and responses for situations of urgency.
• Areal-time data analysis platform needs to be able to prioritize urgent
traffic and processing from nonurgent ones.
Programmable
• The platform shall support programmable capabilities of IoT business
and service logics, data warehouse scheme, template of data and
service model.

Interoperable
• The platform provides interoperability between the different IoT
services and infrastructure.
• The APIs need to follow the existing standards. The components are
published and maintained as Open Source software.
• The target is to deliver a common data model able to exploit both
structured and unstructured data.
• In order to create multimodal and cross-domain smart applications, it
is necessary to move from raw data to linked data and adopt
unambiguous description of relevant information.
Secure

• The platform shall include security and privacy by design.

Big Data Functional Architecture

The functional overview of the BigData IoT platform:

• The topmost block represents the Cloud platform, the middle one is
the network connectivity while the bottom one is the local
deployment, including gateway and sensors.

The following functional domains have been identified:

• The “Smart Applications” domain is the IoT application itself.

• The “Users Management” allows the management of the

identification, roles and connections of users.

• The “Interoperable Service and Dynamic Workflow” domain allows

application writing, deploying, hosting and execution.

• The “Processing and Analytic Engine”, provides services of stream

processing and data analytics.
• The “Network communication” domain provides the IoT connectivity.
• The “Embedded software” and Hardware domains represent the IoT
gateway and sensors themselves.
Data Analytics for the IoT

The amount of IoT data coming from real-world smart objects with sensing, actuating,
computing and communication capabilities is exploding. The sensors and devices are more
and more deployed, within more applications and across industries.

Characteristics of IoT Generated Data

• The volume and quality of the data generated by IoT devices is very different from the
traditional transaction-oriented business data.

• Coming from millions of sensors and sensor-enabled devices, IoT data is more
dynamic, heterogeneous, imperfect, unprocessed, unstructured and real-time than
typical business data.

• It demands more sophisticated, IoT-specific analytics to make it meaningful.

• Figure 2.2, the BigData is defined by 4 “Vs”, which are Volume, Velocity, Variety and
Veracity.

• The first V is for a large volume of data, not gigabytes but rather thousands of terabytes.
The second V is referencing data streams and real-time processing.
• The third V is referencing the heterogeneity of the data: structure and unstructured,
diverse data models, query language, and data sources.

• The fourth V is defining the data uncertainty, heterogeneous, imperfect, unprocessed,

unstructured and real-time than typical business data.
Data Analytic Techniques and Technologies

• A cloud-based IoT analytics platform provides IoT-specific analytics that reduce the
time, cost and required expertise to develop analytics-rich, vertical IoT applications.

• Platform’s IoT-specific analytics uncover insights, create new information, monitor

complex environments, make accurate predictions, and optimize business processes
and operations.

• The applications of the IoT BigData Platform can be classified into four main categories
i. Deep understanding and insight knowledge
ii. Real time actionable insight
iii. Performance optimization
iv. Proactive and predictive applications.

Technologies for building an IoT analytics platform

Batch Processing

• Batch processing supposes that the data to be treated is present in a database.

• The most widely used tool for the case is Hadoop MapReduce.
• MapReduce is a programming model and Hadoop an implementation, allowing
processing large data sets with a parallel, distributed algorithm on a cluster.

• It can run on inexpensive hardware, lowering the cost of a computing cluster.

• The latest version of MapReduce is YARN, called also MapReduce 2.0.

• Pig provides a higher level of programming, on top of MapReduce.

• It has its own language, PigLatin, similar to SQL. Pig Engine parses, optimizes and
automatically executes PigLatin scripts as a series of MapReduce jobs on a Hadoop
cluster.

• Apache Spark is a fast and general-purpose cluster computing system.

• It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that
supports general execution graphs.

• It can be up to a hundred times faster than MapReduce with its capacity to work in-
memory, allowing keeping large working datasets in memory between jobs, reducing
considerably the latency.

• It supports batch and stream processing.

Stream Processing

• Stream processing is a computer programming paradigm, equivalent to dataflow

programming and reactive programming, which allows some applications to more
easily exploit a limited form of parallel processing.

• Flink is a streaming dataflow engine that provides data distribution, communication

and fault tolerance.

• It has almost no latency as the data are streamed in real-time (row by row).

• It runs on YARN and works with its own extended version of MapReduce.

Machine Learning

• Machine learning is especially useful in the context of IoT when some properties of the
data collected need to be discovered automatically.

• Apache Spark comes with its own machine learning library, called MLib. It consists of
common learning algorithms and utilities, including classification, regression,
clustering, collaborative filtering, dimensionality reduction.
• Algorithms can be grouped in 3 domains of actions: Classification, association and
clustering.

• To choose an algorithm, different parameters must be considered: scalability,

robustness, transparency and proportionality.

• KNIME is an analytic platform that allows the user to process the data in a user-friendly
graphical interface.

• It allows training of models and evaluation of different machine learning algorithms

rapidly.

• If the workflow is already deployed on Hadoop, Mahout, a machine learning library can
be used.

• H20 is a software dedicated to machine-learning, which can be deployed on Hadoop

and Spark.

• It has an easy-to-use Web interface, which makes possible to combine BigData

analytics easily with machine learning algorithm to train models.
Data Visualisation

• Freeboard offers simple dashboards, which are readily useable sets of widgets able to
display data.

• Freeboard offers a REST API allowing controlling of the displays.

• Tableau Public is a free service that lets anyone publish interactive data to the web.

• Once on the web, anyone can interact with the data, download it, or create their own
visualizations of it.

• No programming skills are required. Tableau allows the upload of analysed data
from .csv format, for instance.

• The visualization tool is very powerful and allows a deep exploration the data.

• Kibana is an open source analytics and visualization platform designed to work with
Elasticsearch.
• Kibana allows searching, viewing, and interacting with data stored in Elasticsearch
indices.

• It can perform advanced data analysis and visualize data in a variety of charts, tables,
and maps.

• Elasticsearch is a highly scalable open-source full-text search and analytics engine.

• It allows to store, search, and analyze big volumes of data quickly and in near real
time.

• It is generally used as the underlying engine/technology that powers applications that

have complex search features and requirements.

• It provides a distributed, multitenant-capable full-text search engine with an HTTP web

interface and schema-free JSON documents.

• It is really designed for real-time analytics, most commonly used with Flink or Spark
streaming.
Data Collection Using Low-power, Long-range Radios

• Regarding the deployment of IoT devices in a large scale, it is still held back by technical
challenges such as short communication distances.

• Using the traditional mobile telecommunications infrastructure is still very expensive

(e.g., GSM/GPRS, 3G/4G) and not energy efficient for autonomous devices that must
run on battery for months.

• During the last decade, low-power but short-range radio such as IEEE 802.15.4 radio
have been considered by the WSN community with multi-hop routing to overcome the
limited transmission range.

• While such short-range communications can eventually be realized on smart cities

infrastructures where high node density with powering facility can be achieved, it can
hardly be generalized for the large majority of surveillance applications that need to be
deployed in isolated or rural environments.

• Future 5G/LTE standards do have the IoT orientation, but these technologies and
standards are not ready yet while the demand is already high.
• Recent so-called Low-Power Wide Area Networks (LPWAN) such as those based on
SigfoxTM or Semtech’s LoRaTM [1] technology provide a better connectivity answer for
IoT as several kilometers can be achieved without relay nodes to reach a central
gateway or base station.

• Most of long range technologies can achieve 20 km or higher range in LOS condition
and about 2 km in urban NLOS [2].

• With cost and network availability constraints, LoRa technology, which can be privately
deployed in a given area without any operator, has a clear advantage over Sigfox which
coverage is entirely operator-managed.

• These low-power, long-range radio technologies will definitely allow a huge number of
sensors to be installed in remote area, thus augmenting the amount of data to be
treated in the IoT Cloud platform.

Architecture and Deployment

• The deployment of LPWAN (both operator-based and privately-owned scenarios) is

centred on gateways that usually have Internet connectivity as shown in Figure (next
slide).
 • Although direct communications between devices are possible, most of IoT
applications follow the gateway-centric approach with mainly uplink traffic patterns.

• In this typical architecture data captured by end-devices are sent to a gateway which will
push data to well identified network servers.

• Then application servers managed by end-users could retrieve data from the network
server.
• If encryption is used for confidentiality, the application server can be the place where data could
be decrypted and presented to end-users.

• The LoRa Alliance has issued the LoRaWAN specification [3] in a tentative for standardization of
public, large-scale LoRa LPWAN infrastructures featuring multi-gateways and full
network/application servers’ architecture.

• This specification also defines the set of common channels for communications, the packet format,
Medium Access Control (MAC) commands.

• This architecture can however be greatly simplified for small, ad-hoc deployment scenarios where
the gateway can directly push data to some servers or IoT-specific cloud platforms if properly
configured.

Low-cost LoRa Implementation

• The implementation of the full LoRaWAN specification requires gateways to be able to listen on
several channels and LoRa settings simultaneously.

• Commercial gateways therefore use advanced concentrators chips capable of scanning up to 8

different channels: the SX1301 concentrator is typically used instead of the SX127x chip which is
designed for end-devices. Commercial gateways cost several hundredth euros with the cost of the
SX1301-capable board alone to be more than a hundred euro.
• Cloud forms the virtual world in an IOT ecosystem, on the other hand the embedded
devices all together constitute the physical world of the IOT system. These two lies at
the extreme ends of IOT interconnected by the internet.

• Cloud is a remote server that host applications, services and platforms to provide
services and solutions within an IOT environment.

• The raging cloud idea is an amalgamation of a number of established technologies

such as on-demand, utility, autonomic, cluster and grid computing models,
virtualization and containerization, service-oriented architecture (SOA) and
microservices architecture (MSA), multitenancy, and others.

• Network solutions such as firewall, intrusion detection and prevention, application

delivery controllers (ADCs), switches, routers, load balancers, and WAN optimization
solutions also play a solid role in diligently propping up the cloud paradigm.
• Responsible as a server ,cloud needs to be more robust and responsive.

• Within an IOT ecosystem, there could be billions of devices which could be

generating gigabytes or terabytes of data at any time. The data from these billions
of devices could have great deal of diversity and volatility. A cloud must be efficient
enough to handle such scalability and diversity of data and be able to produce
useful insights from that data in real time.

• The IOT devices producing this data may be located anywhere around the globe. So,
a cloud must be able to connect, communicate and direct data and information
on an “Any Time, Any Place & Any Thing " basis.

• The storage, processing and analysis of data on a cloud to derive useful insights are
collectively known as 'Cloud Computing’.
• The major activities in cloud Computing involve delivering data, applications,
photos, videos and much more things over the Internet to the cloud.

• Cloud computing acts as a paradigm for big data storage and analytics. The real
evolution comes from combining IoT with cloud computing. For example, sensory
data can be uploaded and stored with cloud computing, later to be used
intelligently for smart monitoring and actuation with other smart devices.

• Cloud computing in IoT features rapid elasticity which means that users can
readily scale the services to their needs. They can easily and quickly edit the
software setup, add or remove users, increase storage space, etc. These
characteristics further empower IoT by providing elastic computing power,
storage, and networking.
IoT cloud Challenges
 Any cloud (server) must have the following
capabilities -
1. It must be capable to access, store, analyze, process and
monitor big data.
IOT services may be accessed by billions of devices which could
have lot of data to share and exchange over the network and
across the cloud. There can be many other real-life examples of
cloud-based services and applications. For example, even Gmail is
a cloud service that is utilized by computers, laptops and mobile
devices. A Gmail user can register an account, access his/her
account, send or receive emails from anywhere and any device.
There are billions of Gmail accounts and each account could have
hundreds and thousands of sent and received mails.
2. Data must remain secure and protected on a cloud.
The data stored on a cloud could be private or even confidential. It
must be securely stored over the cloud so that only an authentic
user or device could access it. Also, while communicating over the
network, it must remain secured so that it could not be hacked by
an attacker.
3. The cloud must be capable of providing inter-service and
inter-device communication.

It is the cloud that is responsible to manage devices and manage

data sharing and exchange among the devices. It is also
responsible to synchronize data communication between the
devices so that an IOT application could be successfully executed.
It may also need to connect and communicate with other clouds,
cloud services and applications to execute an application.

4. The cloud despite all the complexities must be cost effective

The big data and big data analytics require abundant IT resources
and efficient softwares. A cloud must be able to optimize the
requirements of IT infrastructure, so that the purpose of an IOT
system is fulfilled in the most cost-effective manner.
Architecture of IOT cloud
The architecture underlines all the background processes that
usually remain hidden and abstracted. A typical cloud architecture
can be represented by the following block diagram

IOT Cloud Architecture and Processes

 Any cloud is designed to perform the following basic processes

Data Ingestion: The first thing that a cloud needs to do is to collect data from
the network. The process of importing data from IOT devices to the cloud is
called Data Ingestion Process.

It aggregates the data from IOT devices and distributes the data for further
processing. It also works in reverse direction by distributing data over to the
IoT devices. Clouds use certain protocols when receiving the data from the
network as well as when transmitting the data to the devices.

Pipelining:
After the data has been collected, it is the responsibility of IOT cloud to
manage the data in a proper way. The process of pipelining includes several
tasks as follow
Device Registry:
To manage the data onto the cloud, there is a section for registration
of devices with their unique IDs so that both users and cloud can
uniquely identify the devices and access those devices.
Transforming Data:
When the data is arrived to cloud, it may need to be converted to other
formats. For example, the signal voltages acquired from a temperature and
humidity sensor may need to be converted to a calibrated unit of temperature
and humidity.

Aggregating Data and Computing:

Data aggregation is responsible for increasing the network lifetime and reducing
the energy consumption eliminating redundancy and decreasing the amount of
transferred data . By combining data you can add checks, such as averaging data
across multiple devices to avoid acting on a single, spurious, device. You can
ensure that you have actionable data if a single device goes offline. By adding
computation to your pipeline, you can apply streaming analytics to data while it is
still in the processing pipeline.

Metadata contains information about a device. Most metadata is immutable or

rarely changes. Examples of metadata fields include:
Identifier (ID) - An identifier that uniquely identifies a device. The device ID
should never change for the lifespan of a deployed device.
Class or type, Model, Revision, Date manufactured , Hardware serial number
 Enriching Data: The data generated by a device can be combined with
the metadata about the device or with other datasets. For example, the
temperature data can be used for further analysis to derive useful
insights.

Moving Data: The processed data can be passed on to be stored in one

or more storage locations or caches.

Data storage:

The Data from the physical world comes in various forms. Cloud Platforms
provide enough storage to store the data in a structured manner. The Cloud
platforms provide reliable, high-performance and secured storage space. The
bulk amount of data can be stored on a cloud and can be accessed remotely
at anytime from anywhere in the world.
Real Time Data Analytics:

After the data storage, cloud platforms can perform some kind of analytics on that
data and convert it into feasible actions for human use. Cloud Platforms analyze
information and process the information with some kind of mathematics or logic i.e.
converting the data into information in order to perform automated tasks.

Application and Presentation:

The cloud after processing the data, can send the data to other IOT devices to
perform some tasks based on that or can provide the data to other computing
engines.
IOT Clouds as Service Providers

The clouds are the service providers in an IOT environment. There are many as-
a-Service roles of cloud computing. All these as-a-Service roles are cloud
services rendered by the cloud platforms.

IOT Cloud Services

 Infrastructure-as-a-Service (IaaS):

• IaaS comes at the bottom of the cloud computing services stack in which IaaS
provider provides fundamental building blocks of computing resources to the
clients over the Internet such as networking resources, hardware, software,
storage and many other IT infrastructure components.

• IaaS provider grants access to all these resources to the clients and is responsible
for housing, operating and maintaining all these equipments.
• IaaS users pay on a per-use basis, typically by the hour, week or month. Some ISP
providers also charge users according to the storage space used by the users.
• Vendor/Provider is responsible for configuring and maintaining the
infrastructure. So, users or enterprises do not need to worry about infrastructure
upgrades and they can focus on the task or application instead.

• Some of the well-known IaaS providers are Amazon Web services, Windows Azure
and Google Compute Engine.
 Platform as a service (PaaS):

• PaaS provides a platform to the users to develop, run and manage applications
without the complexity of building and dealing with the infrastructure of the
platform
• PaaS is typically associated with developing and launching an app in a constrained
environment.

• Development tools provided by the IaaS provider are customized according to

the need of the user. PaaS allows for higher-level programming with extremely
reduced complexity at the user end as it has built-in infrastructure stack which
makes the software end more manageable and easily accessible.

• Some examples of PaaS solutions are the “Google App Engine” system, “Heroku”
which operates on top of the Amazon Web Services and “Force.com” built as part of
the SalesForce.com Software as a Service offering.
 Software as a Service (SaaS):

• The top layer of the cloud computing is SaaS in which the end user
operates the software applications only like email, word
processing etc. Instead of installing and maintaining software, the
users simply access these softwares via Internet.
• They do not need to worry about complex software
development and hardware management.

• These applications run on the SaaS provider’s servers. The users

only request to access them.

• The provider manages the access of the application and its

security, reliability and performance.
• Some of the most well-known SaaS providers are “Google Apps”,
Microsoft’s business productivity online suite etc.
Google Cloud IOT (paas)-

Google IOT cloud is a set of fully managed and integrated services that allow to easily
and securely connect, manage and ingest IOT data from globally dispersed devices,
process and analyze the data in real-time, implement the operational changes and take
actions as needed.

The Google Cloud IOT has the following features -

a) End-to-End Security -
It enables end-to-end security with certificate based authentication and TLS
1.2 transportation security.
b) Integrated Services –
It seamlessly move IoT data across Google Cloud services.

c) Advanced Data Analytics –

It performs ad hoc analysis using Google BigQuery, visualize data using Cloud
Data Studio, and derive intelligence using Cloud Machine Learning.
Amazon AWS IOT (paas)-

• Amazon AWS is a managed cloud platform that provides access to the IoT devices to
securely connect and interact with the cloud applications and as well as other
devices.

• AWS IOT can handle up to billions of devices and trillions of messages securely with
low latency and low over head.

• The AWS IoT supports communication protocol and standards like HTTP, MQTT and
websockets.

• Communication is secured using TLS/SSL (Transport Layer security/Secure Sockets

Layer) security.

Note: Some of the other PaaS Cloud platforms like Microsoft azure IoT suite, IBM
Watson IoT, Salesforce IoT cloud, Carriots are also worth mentioning.
Thingspeak (saas)-

• Thingspeak is a software as a service open source platform for IoT solutions.

• It provides API to store and retrieve data from IoT devices over HTTP protocol.

• ThingSpeak has integrated support of numeric computing software MATLAB from

MathWorks.

• It allows users to upload and visualize the real time data from IoT devices. For
example, a developer can publish the surrounding temperature and humidity data
as well as time logging on thingspeak.