UNIT 3

IOT WITH CLOUD COMPUTING

PRUDHVI KIRAN P
Assistant Professor, CSE - IOT Dept.
R. V. R. & J. C. College of Engineering
UNIT - 3 [ 13 Periods ]
loT Analytics for Cloud
Review of loT analytics for the cloud, IoT Analytics challenges, Microsoft Azure Analytics
overview, Designing data processing for analytics, Designing visual analytics for loT data,
Data Science for loT, Feature engineering with loT data.

SUBJECT CODE - CO 314 (EL01)

CLASS DETAILS - IOT V SEM - AUGUST 2023
FACULTY DETAILS - PRUDHVI KIRAN P, Asst. Prof., CSE (IOT), RVR&JC College of Engineering
QUESTION BANK - UNIT 3 [Assignment Questions - 4, 5,6]
1. Mention ten detailed points justifying the positive role of IoT data analytics in cloud
environment.
2. Discuss in detail about various challenges in the area of data analytics in IoT.
3. Discuss in detail about suite of solutions offered by Microsoft azure analytics.
4. Explain and visualize the various visual forms of data analytics for IoT data.
5. Discuss the role of data science in IoT ecosystem
6. Define feature engineering and discuss its association with IoT data.
END
3.1. Review of IoT Analytics for the Cloud
IoT cloud platforms bring together capabilities of IoT devices and cloud computing delivered as a
end-to-end (edge-to-cloud) service. They are also referred by other terms such as Cloud Service
IoT Platform. In this age, where billions of IoT devices (IoT devices are devices with multiple sensors
connected to the cloud, typically via gateways.) are connected to the Internet, with this, there is
rapid growth in applications such as Smart Cities, Agriculture, Wearable Tech, Vehicle Tracking,
Retail Shops, and many more. These IoT devices generate very large amounts of data, which is
transported to the cloud to be analyzed/monitored.

CLICK THE BELOW LINK

https://www.orientsoftware.com/blog/iot-data-analytics/
3.2. IoT Analytics Challenges

CLICK THE BELOW LINK

https://www.tutorialspoint.com/what-are-the-challenges-to-iot-analytics-success
3.3. Microsoft Azure Analytics Overview
 Microsoft Azure Analytics offers a comprehensive suite of solutions designed to store and
analyse your data sets with very good accuracy and performance, delivering vital insights.
These suite of solutions can operate individually or can also operate together as a full analytics
stack. Azure Analytics stands out from other cloud providers by being 14 times faster and
reducing costs by up to 94%.
Data Analytics options on Microsoft Azure;
1. Azure Synapse Analytics
2. Azure Databricks
3. HDInsight
4. Azure Data Factory
5. Azure Machine Learning
6. Azure Stream Analytics
7. Data Lake Analytics
8. Azure Analysis Services
9. Event Hubs
10. Azure Data Explorer
 11. Azure Time Series Insights
12. Azure Data Share
13. Microsoft Graph Data Connect
14. Azure Chaos Studio
1. Azure Synapse Analytics
Azure Synapse Analytics is the next generation of Azure SQL Data Warehouse. It lets you load any
number of data sources - both relational and non-relational databases, whether on-premise or in the
Azure cloud. It unifies all the data and lets you process and analyse it using the SQL language. In
addition, it provides the Azure Synapse Studio that offers a workspace for big data analysis and AI tasks
and creates engaging visualizations of your data.
2. Azure Databricks
Databricks is an analytics service based on Apache Spark. Apache Spark is a veteran tool used to
process huge amounts of unstructured data at high speed. Databricks supports languages like Python,
Scala, Java, SQL, and R, as well as AI/ML libraries like TensorFlow and PyTorch, allowing you to work
with Spark data using any of these languages and frameworks. In addition, Databricks integrates with
Azure Machine Learning (see below), giving you access to a large number of pre-trained machine
learning algorithms. Databricks lets you set up managed Apache Spark clusters with auto-scaling and
auto-termination, eliminating the complexity of setting up Spark in your local data center.
3. HDInsight
Azure HDInsight is a managed cloud service that allows users to process large amounts of data using
open-source frameworks such as; Apache Spark, Apache Hive, Apache Kafka, Hadoop, LLAP, Apache
Storm, R, HBase, Microsoft ML Server. HDInsight can be used for;
1. Big data analytics
2. Streaming
3. Analytics
4. Warehousing
4. Azure Data Factory
• Azure Data Factory is an Extract Transform Load (ETL) service. ETL is a term from the old days of
large-scale processing of structured data. An ETL process takes a structured database, cleans it,
and converts the data into a format that is suitable for analysis.
• Azure Data Factory helps you build ETL and also Extract Load Transform (ELT) strategies with no
code or configuration using a visual editor.
• Data Factory provides built-in connectors with over 90 data sources including Amazon S3, Google
BigQuery, and many on-premise data sources. You can also copy the data from Data Factory to
Azure File Storage.
• Azure Data Factory (ADF) is a fully managed, serverless data integration solution for ingesting,
preparing, and transforming all your data at scale.
5. Azure Machine Learning
• This is a huge library of pre-packaged, pre-trained machine learning algorithms. It also provides
an environment for consuming these algorithms and applying them to real-world tasks. Azure ML
speeds up model creation with a convenient machine learning UI that allows you to build machine
learning pipelines combining multiple algorithms, with steps like model training, testing, and
evaluation.
• In addition, Azure ML provides solutions for interpretable AI. It includes visualization and other
data that can help understand model behavior, apply fairness metrics, and make comparisons
between algorithms to understand the best variant to choose.
6. Azure Stream Analytics
• Azure Stream Analytics is a real-time analytics service that allows users to analyze and process fast-
moving streams of data from sources such as: Sensors, Web data sources, Social media,
Applications, Devices, Infrastructure systems.
Azure Stream Analytics can be used to:
1. Get insights
2. Build reports
3. Trigger alerts and actions.
• This service is fully managed and serverless (Backend as a Service” (BaaS), Functions as a Service”
(FaaS)). Stream Analytics guarantees at least 99.9 percent availability.
7. Data Lake Analytics
A data lake is a centralized repository that allows you to store all your structured and unstructured
data at any scale. You can store your data as-is, without having to first structure the data. Data lake
allows us to run different types of analytics on the stored data using dashboards and visualizations to
big data processing, real-time analytics, and machine learning to guide better decisions.
8. Azure Analysis Services
Azure Analysis Services is a fully managed platform as a service (PaaS) that provides enterprise-grade
data models in the cloud. Use advanced mashup and modeling features to combine data from
multiple data sources, define metrics, and secure your data in a single, trusted tabular semantic data
model. The data model provides an easier and faster way for users to perform ad hoc data analysis
using tools like Power BI and Excel. It allows users to:
1. Combine data from multiple sources
2. Secure data in a single, trusted tabular semantic data model
3. Query massive amounts of data
4. Scale up or down based on demand
9. Event Hubs
• Azure Event Hubs is a cloud native data streaming service that can stream millions of events per
second, with low latency, from any source to any destination. Event Hubs is compatible with
Apache Kafka, and it enables you to run existing Kafka workloads without any code changes.
• Using Event Hubs to ingest and store streaming data, businesses can harness the power of
streaming data to gain valuable insights, drive real-time analytics, and respond to events as
they happen, enhancing overall efficiency and customer experience.
10. Azure Data Explorer
Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large
volumes of data streaming from applications, websites, IoT devices, and more. Ask questions and
iteratively explore data on the fly to improve products, enhance customer experiences, monitor
devices, and boost operations. Quickly identify patterns, anomalies, and trends in your data. Explore
new questions and get answers in minutes. Run as many queries as you need.
11. Azure Time Series Insights
Azure Time Series Insights is one of the cloud services that users can use to integrate with their data
that is constantly changing with time such as data from various sensors or machines, data from
satellites, airlines etc. Any data that can be generated on a high scale and needs to be analyzed, can be
used through Azure Time Series Insights.
12. Azure Data Share
• Azure Data Share is a simple and safe service for sharing big data securely and effortlessly with
other organizations in Azure. Share data, in any format and any size, from multiple sources with
other organizations and easily control what you share, who receives your data, and the terms of
use.
• Data Share provides full visibility into your data sharing relationships with a user-friendly
interface. Share data in just a few clicks, or build your own application using the REST API.
13. Microsoft Graph Data Connect
• Microsoft Graph Data Connect (Data Connect) is a secure and scalable solution that allows
developers to copy Microsoft 365 datasets into Azure data stores. Data Connect is ideal for big
data applications and machine learning. It allows developers to:
Access rich data at scale
1. Develop applications for analytics, intelligence, and business process optimization
2. Copy relevant Microsoft 365 datasets into their Azure tenant
3. Enable advanced analytics and insights
• Microsoft charges for using Data Connect based on how many “objects” are extracted. An object is
an individual instance, such as an email, a file, or a message. The extraction cost is $0.375 per 1,000
objects.
14. Azure Chaos Studio
Azure Chaos Studio is a Microsoft service that uses chaos engineering to improve application
resilience. Chaos engineering is the practice of intentionally introducing controlled faults into a
system to test its resilience. The goal is to identify potential failure points and correct them before
they cause an actual outage or other disruption. This helps to measure, understand, and improve your
cloud application and service.
3.4. Designing Data Processing for Analytics
 One of the greatest challenges facing the IoT industry is the management and interpretation of
its vast streams of data. IoT visualizations of data, taking the form of real-time, interactive graphs
and charts, are crucial for processing and deriving actionable insights from big data analytics in
IoT.
 IoT data visualization is all about taking the data generated by IoT devices and transforming it
into captivating visual formats. Think colorful charts, dynamic graphs, and interactive
dashboards. It’s like turning raw numbers and data points into a visual masterpiece.
IoT data visualizations are created using a variety of approaches and technologies. The types of
data visualization are listed below however they are not exhaustive:
1. Graphs
2. Histograms
3. Pictogram
4. Graph with bars
5. Temperature Maps
6. Gauges
7. Statistics
8. Line charts
9. Maps of locations
10. Infographics
11. Tables
12. Cloud of bubbles
13. Graphical forms
14. Timeline
15. Scatter graph
The following parameters can be displayed using data analytics:
16. Climate conditions
17. Temperature
18. pH value of moisture level
19. Humidity
20. Proximity
21. Location
7. Molecular-level
8. Voltage or current
9. Different element levels
10. Pressure
11. The quality of various environments
12. Air or water toxicity
13. Motion

CLICK THE BELOW LINK

https://bytebeam.io/blog/guide-to-iot-data-visualization/#iot-data-visualization
3.6. Data Science for loT
 The simplest definition of data science is the study of processes that help us derive value from
data. In the case of IoT, data refers to information created by sensors, devices, applications, and
other smart gadgets. At the same time, value means predicting future trends and outcomes
based on that data.
For example, let’s say you use a fitness tracker that measures the number of steps you take per
day. With this information, data science can tell you:
• How many calories you burn
• How much weight you lose
• When the best time for your workout is
 But that’s just a simple example of data science in action. IoT is different because it’s a high-
volume data producer. According to the report, the amount of data that IoT devices will
generate by 2025 is expected to reach 73.1 zettabytes. The standard data science can’t handle
it, so it has to evolve. IoT helps data science go to the next level.
Data science is a multidisciplinary field that involves using scientific methods, algorithms, and systems to extract insights
and knowledge from data. This combines techniques from mathematics, statistics, computer science, and domain-specific
knowledge to process, analyze, and make predictions based on data.
Usually, the IoT devices are in charge of generating data; then data science enters the scene to do the
following:
1. Data collection
2. Cleaning and preprocessing
3. Exploration and visualization
4. Modelling
5. Evaluation
1. The data collection not only comes from IoT devices, but also from sensors, databases, and online
platforms.
2. The cleaning and preprocessing stage involves transforming and preparing the data to make it
suitable for analysis.
3. Exploration and visualization allows data scientists to identify the best information from all
gathered data.
4. Modelling allows to analyse the data objects and their relationship to the other objects; results in
higher analysis for predictive models, here is necessary a deep understanding of statistics and
mathematics, as well as experience in working with large-scale datasets. It also involves the use of
specialized software tools and platforms, such as Python, R, and SQL, to perform the analysis and
interpretation of data.
5. Finally, evaluation aims to define how well the model performs its task.
Data science plays a pivotal role in various applications within the IoT ecosystem, enabling businesses
and organizations to derive insights from IoT data, make data-driven decisions, and optimize processes
for improved efficiency and effectiveness. Data science became a critical component in leveraging the
full potential of IoT technologies. Below are the few applications of data science in IoT;
Predictive Maintenance
Data science is used to analyze data from sensors and devices to predict when equipment or
machinery might fail. This helps in scheduling maintenance activities proactively, reducing downtime,
and saving on repair costs.
Anomaly Detection
Data science models are applied to IoT data to identify anomalies or unusual patterns in real-time.
This is critical for detecting security breaches or operational issues.
Environmental Monitoring
IoT sensors are used to collect data on environmental conditions such as air quality, temperature,
humidity, and pollution. Data science can analyze this data to provide insights into climate trends and
pollution levels.
Smart Agriculture
IoT devices in agriculture monitor soil conditions, weather, and crop health. Data science helps
farmers make data-driven decisions on irrigation, fertilization, and pest control, increasing crop yield
and sustainability.
Smart Cities
Data science is applied to IoT data to make cities more efficient. This includes traffic management,
waste management, energy consumption optimization, and public safety enhancements.
Healthcare and Remote Monitoring
IoT devices like wearables and medical sensors collect health data. Data science helps in monitoring
patient health, early disease detection, and providing personalized healthcare recommendations.
Supply Chain Optimization
IoT sensors in the supply chain monitor the location and condition of goods during transit. Data
science optimizes routing, reduces theft and spoilage, and ensures timely delivery.
Energy Management
IoT devices track energy consumption in buildings and industrial facilities. Data science helps in
identifying energy-saving opportunities and optimizing power usage.
Smart Homes
Data science is used in home automation systems to learn user preferences and adapt to them, such
as adjusting lighting and temperature settings.
Wearable Technology
Data science is central to the functionality of wearable devices that track physical activity, sleep, and
health indicators, providing insights to users for personal wellness.
Industrial IoT (IIoT)
In manufacturing, IIoT collects data on machine performance, production lines, and quality control.
Data science helps in optimizing production processes and improving product quality.
Smart Energy Grids
IoT devices in energy grids monitor power distribution and consumption. Data science assists in load
forecasting, grid stability, and energy efficiency.
Fleet Management
IoT-enabled vehicles are equipped with sensors and GPS for tracking and managing fleets. Data science
optimizes routing, maintenance schedules, and fuel efficiency.
Water Quality Monitoring
IoT sensors in water treatment facilities and natural water sources collect data on water quality. Data
science helps in ensuring clean and safe water supplies.
Security and Surveillance
IoT devices for security, such as cameras and motion sensors, generate vast amounts of data. Data
science helps in real-time threat detection and post-event analysis.
Environmental Conservation
IoT devices monitor wildlife and ecosystems. Data science can be used to analyze data for conservation
efforts, such as protecting endangered species.
3.7. Feature Engineering with loT Data
Feature engineering is the process of selecting, extracting, and changing raw data into
features that a machine learning (ML) algorithm can use to make predictions or
classifications. The quality and relevance of the features can have a significant impact on
the performance of the ML model.

IOT SENSOR 1

ML
IOT SENSOR 2

IOT SENSOR n
 Feature engineering can involve the creation of new features from existing data, combining existing
features, or selecting the most relevant features.
 Feature engineering is part of the data pre-processing stage in a machine learning development
lifecycle, as mentioned in the above figure. It typically consists of four different techniques;
1. Feature Creation
2. Feature Transformation
3. Features Extraction
4. Feature Selection
1. Feature Creation
• Feature creation is one of the most common techniques in feature engineering. It is also known as
feature construction or feature synthesis. It involves the creation of new features from existing
data that is performed by combining or transforming the existing features. For example, if you have
a dataset with the birth date of individuals, you can create a new feature age by subtracting it from
the current date.
• This technique is advantageous when the existing features do not provide enough information to
train and develop a robust machine-learning model or when the existing features are not in a
usable format for an ML model development. But it can also be time-consuming and require a
deep understanding of the data and the problem at hand.
2. Feature Transformation
• Transformations are a set of mathematical operations that can be applied to existing features to
create new ones. Some of the most common transformation techniques include data aggregation
(summary), scaling (up & down), normalization (tabulation), binning (clustering), etc.
3. Feature Extraction
• Feature extraction is a feature engineering technique that creates new variables by extracting
them from the raw data. I.e. transforming the original data into a new feature space. The main
objective of this process is to reduce the volume and dimensionality of the input data.
• A few of the most common techniques used for this method include - Principal Component Analysis
(PCA), Cluster Analysis, etc., which are proved efficient in dimensionality reduction of data.
4. Feature Selection
• The feature selection technique involves the selection of a subset of the most relevant features
from the data and dropping the remaining features. It is performed using criteria such as the
importance of the features, the correlation between the features and target variables, etc.
• It can greatly reduce the complexity of the ML model by removing redundant and irrelevant
features from the training process.
Feature engineering is a crucial step in the machine learning development lifecycle; it results in, Better
data representation, Improvement in model performance, Reduced complexity of the model.
 PROGRAM ELEMENTS-STRUCTURE FEATURE ENGINEERING - EXAMPLE
For example, if we would like to predict the price of a car, the target variable would be the Market
Value. The predictor variables start as a long list of attributes that, through feature engineering, is
slimmed down and manipulated to produce a set of effective predictor variables.
The process of feature engineering would involve more explanatory questions like;
• Is number of seats a good predictor?
• Should horsepower and torque be separate predictor variables, or do they provide similar
information and only one of them is needed?

As the above example conveys, feature engineering is a process that is highly dependent on the
dataset and the target variables. As a result, there is no single correct method of conducting feature
engineering. Feature engineering is a process that is heavily dependent on the efficiency of the
automated tools used and experience and expertise of the data scientists conducting the analysis.
 FEATURE ENGINEERING WITH IOT DATA
 Feature engineering in the context of Internet of Things (IoT) data
involves extracting, creating, and transforming relevant features
from raw sensor data collected by IoT devices.
 Feature engineering is a crucial step in the process of preparing data
for machine learning, and it's equally important when working with
IoT (Internet of Things) data.
 IoT data typically involves a wide range of sensor readings,
timestamps, and potentially other contextual information. The goal
of feature engineering is to extract meaningful and informative
features from this raw data that can be used to build effective
machine learning models in IoT space.
Steps in Feature Engineering
A. Data Preparation
This data pre-processing step consists of collecting, manipulating, and
combining raw data from various sources into a standardized format.
B. Data Cleaning
This step requires handling missing values, removing erroneous and
inconsistent values, etc.
C. Exploratory Data Analysis (EDA)
• EDA involves analysing and investigating datasets through various statistical and visualization
methods.
• It can help data scientists to understand data better, its characteristics, how to manipulate it,
choose the right features for the models, etc.
D. Benchmark/Evaluation
This step involves evaluating the performance of the ML model using the engineered features and
comparing it to the performance of the ML model without feature engineering.

Feature engineering is a critical step in the data preprocessing process when working with IoT
(Internet of Things) data; because IoT or sensor data consists of a very large continuous stream of
data where the time interval between successive updates of the data is very small; usually minutes,
seconds or even milli-seconds.
UNIT - 3 [ 13 Periods ]

END OF UNIT 3

SUBJECT CODE - CO 314 (EL01)

CLASS DETAILS - IOT V SEM - AUGUST 2023
FACULTY DETAILS - PRUDHVI KIRAN P, Asst. Prof., CSE (IOT), RVR&JC College of Engineering