EL

Data Handling and Analytics – Part II

Data is Precious
PT Dr. Sudip Misra
Associate Professor
Department of Computer Science and Engineering
N
IIT KHARAGPUR
Email: [email protected]
Website: http://cse.iitkgp.ac.in/~smisra/

Introduction to Internet of Things 1

What is Data Analytics
 “Data analytics (DA) is the process of examining data sets in order to draw

EL
conclusions about the information they contain, increasingly with the aid of
specialized systems and software. Data analytics technologies and techniques are
widely used in commercial industries to enable organizations to make more‐
informed business decisions and by scientists and researchers to verify or disprove

PT
scientific models, theories and hypotheses.”
[An admin's guide to AWS data management]
N

Introduction to Internet of Things 2

Types of Data Analysis
 Two types of analysis

EL
 Qualitative Analysis
 Deals with the analysis of data that is categorical in nature

 Quantitative Analysis

PT
 Quantitative analysis refers to the process by which numerical data is analyzed
N

Introduction to Internet of Things 3

Qualitative Analysis
 Data is not described through numerical values

EL
 Described by some sort of descriptive context such as text
 Data can be gathered by many methods such as interviews, videos and audio
recordings, field notes
 Data needs to be interpreted
PT
 The grouping of data into identifiable themes
 Qualitative analysis can be summarized by three basic principles (Seidel, 1998):
 Notice things
N
 Collect things
 Think about things

Introduction to Internet of Things 4

Quantitative Analysis
 Quantitative analysis refers to the process by which numerical data is analyzed

EL
 Involves descriptive statistics such as mean, media, standard deviation
 The following are often involved with quantitative analysis:





Statistical models

Data dispersion
PT
Analysis of variables

Analysis of relationships between variables





Regression analysis
Statistical significance
Precision
Error limits
N
 Contingence and correlation

Introduction to Internet of Things 5

Comparison
Qualitative Data Quantitative Data

EL
Data is observed Data is measured

Involves descriptions Involves numbers

Emphasis is on quality
PT
Examples are color, smell, taste, etc.
Emphasis is on quantity

Examples are volume, weight, etc.

N

Introduction to Internet of Things 6

Advantages
 Allows for the identification of important (and often mission‐critical) trends

EL
 Helps businesses identify performance problems that require some sort of action
 Can be viewed in a visual manner, which leads to faster and better decisions
 Better awareness regarding the habits of potential customers

PT
It can provide a company with an edge over their competitors
N

Introduction to Internet of Things 7

Statistical models
 The statistical model is defined as the mathematical equation that are formulated

EL
in the form of relationships between variables.
 A statistical model illustrates how a set of random variables is related to another
set of random variables.
 A statistical model is represented as the ordered pair (X , P)

PT
 X denotes the set of all possible observations
 P refers to the set of probability distributions on X
N

Introduction to Internet of Things 8

Statistical models (Contd.)
 Statistical models are broadly categorized as

EL
 Complete models
 Incomplete models

 Complete model does have the number of variables equal to the number of
equations
PT
 An incomplete model does not have the same number of variables as the number
of equations
N

Introduction to Internet of Things 9

Statistical models (Contd.)
 In order to build a statistical model

EL
 Data Gathering
 Descriptive Methods
 Thinking about Predictors
 Building of model

PT
Interpreting the Results
N

Introduction to Internet of Things 10

Analysis of variance
 Analysis of Variance (ANOVA) is a parametric statistical technique used to compare

EL
datasets.
 ANOVA is best applied where more than 2 populations or samples are meant to be
compared.
 To perform an ANOVA, we must have a continuous response variable and at least one

PT
categorical factor (e.g. age, gender) with two or more levels (e.g. Locations 1, 2)
 ANOVAs require data from approximately normally distributed populations
N

Introduction to Internet of Things 11

Analysis of variance (Contd.)
 Properties to perform ANOVA –

EL
 Independence of case
 The sample should be selected randomly
 There should not be any pattern in the selection of the sample
 Normality

 Homogeneity PT
 Distribution of each group should be normal

 Variance between the groups should be the same (e.g. should not compare data from
cities with those from slums)
N

Introduction to Internet of Things 12

Analysis of variance (Contd.)
 Analysis of variance (ANOVA) has three types:

EL
 One way analysis
 One fixed factor (levels set by investigator). Factors: age, gender, etc.
 Two way analysis
 Factor variables are more than two
 K‐way analysis
 Factor variables are k
PT
N

Introduction to Internet of Things 13

Analysis of variance (Contd.)
 Total Sum of square
 In statistical data analysis, the total sum of squares (TSS or SST) is a quantity that
appears as part of a standard way of presenting results of such analyses. It is defined

EL
as being the sum, over all observations, of the squared differences of each
observation from the overall mean.
 F –ratio
 Helps to understand the ratio of variance between two data sets

PT
 The F ratio is approximately 1.0 when the null hypothesis is true and is greater than
1.0 when the null hypothesis is false.
 Degree of freedom
N
 Factors which have no effect on the variance
 The number of degrees of freedom is the number of values in the final calculation of a
statistic that are free to vary.

Introduction to Internet of Things 14

Data dispersion
 A measure of statistical dispersion is a nonnegative real number that is zero if all

EL
the data are the same and increases as the data becomes more diverse.

 Examples of dispersion measures:

 Range

PT
 Average absolute deviation
 Variance and Standard deviation
N

Introduction to Internet of Things 15

Data dispersion (Contd.)
 Range

EL
 The range is calculated by simply taking the difference between the maximum and
minimum values in the data set.
 Average absolute deviation
 The average absolute deviation (or mean absolute deviation) of a data set is the average of the


PT
absolute deviations from the mean.
 Variance
Variance is the expectation of the squared deviation of a random variable from its mean
 Standard deviation
N
 Standard deviation (SD) is a measure that is used to quantify the amount of variation
or dispersion of a set of data values

Introduction to Internet of Things 16

Contingence and correlation
 In statistics, a contingency table (also known as a cross tabulation or crosstab) is a

EL
type of table in a matrix format that displays the (multivariate) frequency
distribution of the variables.

 Provides a basic picture of the interrelation between two variables

PT
 A crucial problem of multivariate statistics is finding (direct‐)dependence structure
underlying the variables contained in high‐dimensional contingency tables
N

Introduction to Internet of Things 17

Contingence and correlation (Contd.)
 Correlation is a technique for investigating the relationship between two

EL
quantitative, continuous variables

 Pearson's correlation coefficient (r) is a measure of the strength of the association

between the two variables.

PT
 Correlations are useful because they can indicate a predictive relationship that can
be exploited in practice
N

Introduction to Internet of Things 18

Regression analysis
 In statistical modeling, regression analysis is a statistical process for estimating the

EL
relationships among variables

 Focuses on the relationship between a dependent variable and one or more

independent variables

PT
 Regression analysis estimates the conditional expectation of the dependent
variable given the independent variables
N

Introduction to Internet of Things 19

Regression analysis (Contd.)
 The estimation target is a function of the independent variables called the

EL
regression function
 Characterize the variation of the dependent variable around the regression
function which can be described by a probability distribution
 Regression analysis is widely used for prediction and forecasting, where its use has

PT
substantial overlap with the field of machine learning
 Regression analysis is also used to understand which among the independent
variables are related to the dependent variable
N

Introduction to Internet of Things 20

Statistical significance
 Statistical significance is the likelihood that the difference in conversion rates

EL
between a given variation and the baseline is not due to random chance

 Statistical significance level reflects the risk tolerance and confidence level

 Sample size
 Effect size
PT
 There are two key variables that go into determining statistical significance:
N

Introduction to Internet of Things 21

Statistical significance (Contd.)
 Sample size refers to the sample size of the experiment

EL
 The larger your sample size, the more confident you can be in the result of the
experiment (assuming that it is a randomized sample)

PT
 The effect size is just the standardized mean difference between the two groups

 If a particular experiment replicated, the different effect size estimates from each
N
study can easily be combined to give an overall best estimate of the effect size

Introduction to Internet of Things 22

Precision and Error limits
 Precision refers to how close estimates from different samples are to each other

EL
 The standard error is a measure of precision

value and vice versa

PT
 When the standard error is small, estimates from different samples will be close in

 Precision is inversely related to standard error

N

Introduction to Internet of Things 23

Precision and Error limits (Contd.)
 The limits of error are the maximum overestimate and the maximum

EL
underestimate from the combination of the sampling and the non‐sampling errors

 The margin of error is defined as –

 Limit of error = Critical value x Standard deviation of the statistic

PT
 Critical value: Determines the tolerance level of error.
N

Introduction to Internet of Things 24

References
 Agrawal, R., Mannila, H., Srikant, R., Toivonen, H. and Verkamo, A. I. (1995). Fast discovery of association rules, Advances in
Knowledge Discovery and Data Mining , AAAI/MIT Press, Cambridge, MA.
 Agresti, A. (1996). An Introduction to Categorical Data Analysis , Wiley, New York.

EL
 Agresti, A. (2002). Categorical Data Analysis (2nd Ed.), Wiley, New York
 Anderson, T. (2003). An Introduction to Multivariate Statistical Analysis, 3rd ed., Wiley, New York.
 Bair, E., Hastie, T., Paul, D. and Tibshirani, R. (2006). Prediction by supervised principal components, Journal of the American
Statistical Association, 101: 119–137.
 Barron, A. (1993). Universal approximation bounds for superpositions of a sigmoid function, IEEE Transactions on
Information Theory, 39: 930–945.

PT
 Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple
testing, Journal of the Royal Statistical Society Series B. 85: 289–300.
 Copas, J. B. (1983). Regression, prediction and shrinkage (with discussion), Journal of the Royal Statistical Society, Series B,
Methodo logical, 45: 311–354.
N

CSE, IIT Kharagpur Introduction to Internet of Things 25

 EL
PT
N

Introduction to Internet of Things 26

 EL
Case Study: Agriculture

PT Dr. Sudip Misra

Associate Professor
Department of Computer Science and Engineering
IIT KHARAGPUR
N
Email: [email protected]
Website: http://www.cse.iitkgp.ac.in/~smisra/

1
 Future of IoT application in agriculture
 Soil moisture and
water level

EL
monitoring
 Automated
irrigation system
 Automation in
Recycling of

PT Organic Waste
and
Vermicomposting
 Automated
sowing and
N
weeding system

Image template source: https://pixabay.com/p‐747175/?no_redirect

2
 EL
Case study on
Smart Water Management Using IoT
PT
N
3
 AgriSens: Smart Water Management using IoT

 Objectives

EL
 More yields with less water
 Save limited water resource in a country
 Automatic irrigation

PT
Dynamic irrigation treatments in the different phases of a crop’s life
cycle
 Remote monitoring and controlling
N
Source: Project name: Development of a Sensor based Networking System for Improved Water Management for Irrigated Crops, funded by MHRD, Govt. of India

4
 AgriSens: Smart Water Management using IoT
(Contd.)
 Proposed architecture

EL
 Sensing and actuating layer
 Processing, storage, and service
layer
 Application layer
PT
N
Fig 1: The proposed architecture of AgriSens
Source: Project name: Development of a Sensor based Networking System for Improved Water Management for Irrigated Crops, funded by MHRD, Govt. of India

5
 AgriSens: Smart Water Management using IoT
(Contd.)
 Design

EL
 Integrated design for sensors
 Integrated design for sensor node
 Integrated design for remote server

PT
N
Source: Project name: Development of a Sensor based Networking System for Improved Water Management for Irrigated Crops, funded by MHRD, Govt. of India

6
 AgriSens: Smart Water Management using IoT
(Contd.)
 Integrated design for sensors

EL
PT
Fig 4: Designed water‐level sensor
N
Fig 5: EC‐05 soil moisture sensor
Source: Project name: Development of a Sensor based Networking System for Improved Water Management for Irrigated Crops, funded by MHRD, Govt. of India

7
AgriSens: Smart Water Management using IoT
(Contd.)
 Integrated design for sensor node

EL
PT
N
Fig 2: The block diagram of a sensor node

8
 AgriSens: Smart Water Management using IoT
(Contd.)
 Integrated design for sensor node

EL
PT
N
Fig 3: Designed sensor node
Source: Project name: Development of a Sensor based Networking System for Improved Water Management for Irrigated Crops, funded by MHRD, Govt. of India

9
 AgriSens: Smart Water Management using IoT
(Contd.)
 Integrated design for remote server

EL
 Repository data server: Communicates with the deployed IoT gateway
in the field by using GPRS technology
 Web server: To access field data remotely

PT
 Multi users server: Sends field information to farmer’s cell using SMS
technology and also executes farmer’s query and controlling messages
N
Source: Project name: Development of a Sensor based Networking System for Improved Water Management for Irrigated Crops, funded by MHRD, Govt. of India

10
AgriSens: Smart Water Management using IoT
(Contd.)
 Implementation

EL
 Field demo
 Website demo
 Project details from website

PT
N
11
 AgriSens: Smart Water Management using IoT
(Contd.)
 Results Reproductive phase

EL
Vegetative phase Maturity phase

PT
N
Fig. 6: Average soil moisture
Source: Project name: Development of a Sensor based Networking System for Improved Water Management for Irrigated Crops, funded by MHRD, Govt. of India

12
 AgriSens: Smart Water Management using IoT
(Contd.)
 Results

EL
Vegetative phase Reproductive phase Maturity phase

PT
N
Fig. 7: Average water level

Source: Project name: Development of a Sensor based Networking System for Improved Water Management for Irrigated Crops, funded by MHRD, Govt. of India

13
 AgriSens: Smart Water Management using IoT
(Contd.)
 Results

EL
Avg. PDR:
98.75 – 89.75%

Noises:

PT Air flow,
Temperature,
Solar radiation,
Rain
N
Fig. 8: Average packet delivery ratio
Source: Project name: Development of a Sensor based Networking System for Improved Water Management for Irrigated Crops, funded by MHRD, Govt. of India

14
 N
PT
15
EL
 EL
Case study: Healthcare

PT Dr. Sudip Misra

Associate Professor
Department of Computer Science and Technology
IIT KHARAGPUR
N
Email: [email protected]
Website: http://www.cse.iitkgp.ac.in/~smisra/

1
Emergence of IoT Healthcare

 Advances in sensor and connectivity

EL
 Collect patient data over time
 Enable preventive care
 Understanding of effects of therapy on a patient
 Ability of devices to collect data on their own

PT
 Automatically obtain data when and
where needed by doctors
 Automation reduces risk of error
 Lower error implies increased efficiency
N
and reduced cost

2
Components of IoT Healthcare
 Components of IoT is organized in 4 layers

EL
 Sensing layer: Consists of all sensor, RFIDs and wireless sensor
networks (WSN). E.g: Google glass, Fitbit tracker
 Aggregated layer: Consists of different types of aggregators based on

PT
the sensors of sensing layer. E.g: Smartphones, Tablets
 Processing layer: It consists of servers for processing information
coming from aggregated layer.
 Cloud platform: All processed data are uploaded in cloud platform,
N
which can be accessed by large no. of users

3
 EL
PT
N
Sensing & Measurement Data Aggregation Cloud storage & Analytics

4
 EL
IoT in Healthcare : Directions

PT
N
5
 IoT Healthcare : Remote Healthcare
 Many people without ready access to

EL
effective healthcare
 Wireless IoT driven solutions bring
healthcare to patients rather than bring
patients to healthcare
 Securely capture a variety of medical data
with smart algorithms PT
through IoT based sensors, analyze data

 Wirelessly share data with health

professionals for appropriate health
Withings BP Monitor*
N
recommendations
*http://www.withings.com/ ^http://www.shimmersensing.com/ Shimmer Temperature Monitor^

6
IoT Healthcare : Real-time Monitoring
 IoT‐driven non‐invasive monitoring

EL
 Sensors to collect comprehensive
physiological information
 Gateways and cloud‐based

PT
analytics and storage of data
 Wirelessly send data to caregivers
 Lowers cost of healthcare
N
7
IoT Healthcare : Preventive care

 Fall detection for seniors

EL
 Emergency situation detection
and alert to family members
 Machine learning for health
PT
trend tracking and early
anomaly detection
N
8
 EL
AmbuSens: Use-case of Healthcare system using IoT

PT
N
9
Problem Definition & its Scope
 Telemedicine and Remote Healthcare:

EL
 Problem ‐ Physical presence
necessary
 Solution ‐ Wireless sensors
 Emergency Response Time:

 Solution
PT
 Problem – Not equipped to deal
with complications.

 Instant remote monitoring

 Feedback by the skilled medical
N
professionals

10
Problem Definition & its Scope (cont.)
 Real Time Patient Status Monitoring:

EL
 Problem – Lack of collaboration.
 Solution ‐ Real‐time monitoring.
 Digitized Medical History:
 Problem
 Inconsistent
PT
 Physical records vulnerable to wear
and tear and loss.
 Solution ‐ Consistent cloud‐based
N
digital record‐keeping system

11
AmbuSens: Physiological Parameters

EL
PT
Electrocardiogram (ECG)
Temperature
N
Heart Rate
Galvanic Skin Response (GSR)

12
AmbuSens: Development of WBAN
 Single hop wireless body

EL
area network (WBAN)
 Communication protocol
used is Bluetooth i.e. IEEE
802.15.1
PT
 Power management and
data‐rate tuning
 Calibration of data
N
 Filtering and noise removal

13
AmbuSens: Development of Cloud Framework
 Health‐cloud framework

EL
 The developed system is
strictly privacy‐aware
 Patient‐identity masking

PT
involves hashing and
reverse hashing of patient
ID
 Scalable architecture
N
14
AmbuSens: Web Interface
 URL: ambusens.iitkgp.ac.in

EL
 Paramedic and Doctor portals
for ease of use.
 Provision for recording medical
history and sending feedback.

data streaming. PT
 Allows sensor initialization and

 Includes data visualization

tools for better understanding.
N
15
AmbuSens: System Architecture

EL
PT
N
16
AmbuSens: Implementation

 AmbuSens Implementation demo

EL
 Field demo animation
 Part 1
 AmbuSens in the Hospital

 Part 2 PT
 Brief description of the sensors

 Ambulatory Healthcare
N
17
AmbuSens: System Trials

EL
PT
Figure 1: Hospital system trials Figure 2: Ambulatory system trials
N
18
AmbuSens: Results (Comparison of ECG tracing)

EL
PT
N
ECG tracing from manual system Real‐time ECG tracing from AmbuSens

19
 EL
Thank You

PT
N
20
 EL
Activity Monitoring - Part 1
PT Dr. Sudip Misra
Associate Professor
Department of Computer Science and Engineering
IIT KHARAGPUR
N
Email: [email protected]
Website: http://cse.iitkgp.ac.in/~smisra/

Introduction to Internet of Things 1

Introduction
 Wearable sensors have become very popular for different purposes

EL
such as:
 Medical
 Child‐care



Elderly‐care
Entertainment
Security
PT
N
 These sensors help in monitoring the physical activities of humans

Introduction to Internet of Things 2

Introduction (Contd.)

 Particularly in IoT scenarios, activity monitoring plays an

EL
important role for providing better quality of life and safe
guarding humans.
 Provides information accurately in a reliable manner
PT
 Provides continuous monitoring support.
N
Introduction to Internet of Things 3
Traditional Architecture

EL
Analyzer

PT
Continuous
monitoring
N
Introduction to Internet of Things 4
Advantages
 Continuous monitoring of activity results in daily observation of

EL
human behavior and repetitive patterns in their activities.
 Easy integration and fast equipping
 Long term monitoring




Accelerometer
Gyroscope
GPS
PT
 Utilization of sensors of handheld devices
N
 Others

Introduction to Internet of Things 5

Important Human Activities

EL
• Running
Actions • Jumping

PT
Gesture
• Folding legs
• Moving hand
N
Introduction to Internet of Things 6
Types of Sensors

EL
Camera
PT Smart Phone Activity Tracker Band
N
Introduction to Internet of Things 7
Data Analysis Tools
 Statistical

EL
 Sensor data
 Machine Learning Based
 Sensor data

 Sensor data
 Images
PT
 Deep Learning Based
N
 Videos

Introduction to Internet of Things 8

Approaches
 In‐place

EL
 On the device
 Power intensive
 No network connection required
 Network Based



PT
Larger and processing intensive methods can be applied
Group based analytics possible
Low power consumption
N
 Average to good network connection

Introduction to Internet of Things 9

 EL
PT
N
Introduction to Internet of Things 10