Scribd
Upload
381 views27 pages

Analyzing IoT Data in Python Chapter1

Uploaded by

Fgpeqw
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
381 views27 pages

Analyzing IoT Data in Python Chapter1

Uploaded by

Fgpeqw
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 27

Introduction to IoT

data
A N A LY Z I N G I OT D ATA I N P Y T H O N

Matthias Voppichler
IT Developer
Course overview
Collect and analyze IoT data

Gather data
API Endpoints

Data Streams

Visualize data

Combine datasets

Detect patterns

ML Model based alerts

ANALYZING IOT DATA IN PYTHON


What is IoT?
IoT == Internet of Things

Network of connected devices

Measure and collect data

Interact with environment

ANALYZING IOT DATA IN PYTHON


IoT Devices
Connected devices Industrial connected devices

Smart locks Connected machines

Connected thermostats Robots / Cobots

Temperature sensors Package tracking

ANALYZING IOT DATA IN PYTHON


IoT Data formats
http / json

plain text

binary data

XML

Proprietary protocols

ANALYZING IOT DATA IN PYTHON


Data aquisition
Data streams

Gathered from a device

API endpoints

ANALYZING IOT DATA IN PYTHON


Data aquisition - requests
import requests
url = "https://demo.datacamp.com/api/temp?count=3"
r = requests.get(url)
print(r.json())

[{'timestamp': 1536924000000, 'value': 22.3},


{'timestamp': 1536924600000, 'value': 22.8},
{'timestamp': 1536925200000, 'value': 23.3}]

print(pd.DataFrame(r.json()).head())

timestamp value
0 1536924000000 22.3
1 1536924600000 22.8
2 1536925200000 23.3

ANALYZING IOT DATA IN PYTHON


Data aquisition - pandas
import pandas as pd
df_env = pd.read_json("https://demo.datacamp.com/api/temp?count=3")

print(df_env.head())

timestamp value
0 2018-09-14 11:20:00 22.3
1 2018-09-14 11:30:00 22.8
2 2018-09-14 11:40:00 23.3

print(df_env.dtypes)

timestamp datetime64[ns]
value float64
dtype: object

ANALYZING IOT DATA IN PYTHON


Let's Practice
A N A LY Z I N G I OT D ATA I N P Y T H O N
Understand the data
A N A LY Z I N G I OT D ATA I N P Y T H O N

Matthias Voppichler
IT Developer
Store data to disk
Reasons to store IoT Data

Limited historical data availability

Reproducible results

Training ML Models

ANALYZING IOT DATA IN PYTHON


Store data using pandas
df_env.to_json("data.json", orient="records")

!cat data.json
[{'timestamp': 1536924000000, 'value': 22.3},
{'timestamp': 1536924600000, 'value': 22.8},
{'timestamp': 1536925200000, 'value': 23.3},
{'timestamp': 1536925800000, 'value': 23.6},
{'timestamp': 1536926400000, 'value': 23.5}]

ANALYZING IOT DATA IN PYTHON


Reading stored data
From JSON les

import pandas as pd
df_env = pd.read_json("data.json")

From CSV le

import pandas as pd
df_env = pd.read_csv("data.csv")

ANALYZING IOT DATA IN PYTHON


Validate data load
Correct column headers

Check Data formats

df_env.head()

timestamp humidity pressure sunshine temperature


0 2018-09-01 00:00:00 95.6 1016.3 599.2 16.1
2 2018-09-01 00:10:00 95.5 1016.4 600.0 16.1
4 2018-09-01 00:20:00 95.2 1016.5 598.9 16.1
6 2018-09-01 00:30:00 95.1 1016.4 600.0 16.1
8 2018-09-01 00:40:00 95.3 1016.3 600.0 16.1

ANALYZING IOT DATA IN PYTHON


dataframe.info()
df_env.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 13085 entries, 0 to 13085
Data columns (total 5 columns):
pressure 13085 non-null float64
humidity 13085 non-null float64
sunshine 13083 non-null float64
temperature 13059 non-null float64
timestamp 13085 non-null datetime64[ns]
dtypes: datetime64[ns](1), float64(6)
memory usage: 1.4 MB

ANALYZING IOT DATA IN PYTHON


pandas describe()
df_env.describe()

humidity pressure sunshine temperature


count 13057.000000 13057.000000 13057.000000 13057.00000
mean 73.748350 1019.173003 187.794746 14.06647
std 20.233558 6.708031 274.094951 6.61272
min 8.900000 989.500000 0.000000 -1.80000
25% 57.500000 1016.000000 0.000000 9.80000
50% 78.800000 1019.700000 0.000000 13.40000
75% 91.300000 1023.300000 598.900000 18.90000
max 100.100000 1039.800000 600.000000 30.40000

ANALYZING IOT DATA IN PYTHON


Time for Practice!
A N A LY Z I N G I OT D ATA I N P Y T H O N
Introduction to Data
streams
A N A LY Z I N G I OT D ATA I N P Y T H O N

Matthias Voppichler
IT Developer
What is a Data Stream
Constant stream of Data

Examples
Twitter messages

Online News Articles

Video streams

Sensor data (IoT)

Market orders ( nancial)

ANALYZING IOT DATA IN PYTHON


What is a Data Stream
Constant stream of Data

Examples
Twitter messages

Online News Articles

Video streams

Sensor data (IoT)

Market orders ( nancial)

ANALYZING IOT DATA IN PYTHON


MQTT
Message protocol Message Queuing Telemetry Transport

Publish / subscribe

Small footprint

Server -> Acts as a message Broker

Client:

Connects to a Broker

Publishes data

Subscribes to topics

ANALYZING IOT DATA IN PYTHON


Python library
Eclipse Paho™ MQTT Python Client

# Import MQTT library


import paho.mqtt

More information and the documentation available at GitHub


https://github.com/eclipse/paho.mqtt.python

ANALYZING IOT DATA IN PYTHON


Single message
import paho.mqtt.subscribe as subscribe
msg = subscribe.simple("paho/test/simple",
hostname="test.mosquitto.org")

print(f"{msg.topic}, {msg.payload}")

Output:

paho/test/simple, {"time": 1549481572, "humidity": 77, "temp": 21}

ANALYZING IOT DATA IN PYTHON


Callback
def on_message(client, userdata, message):

print(f"{message.topic} : {message.payload}")

Arguments

client - client instance

userdata - private user data

message - instance of MQTTMessage

ANALYZING IOT DATA IN PYTHON


Callback
import paho.mqtt.subscribe as subscribe

subscribe.callback(on_message,
topics="datacamp/roomtemp",
hostname="test.mosquitto.org")

ANALYZING IOT DATA IN PYTHON


MQTT Subscribe
import paho.mqtt.subscribe as subscribe

def on_message(client, userdata, message):


print("{} : {}".format(message.topic, message.payload))

subscribe.callback(on_message,
topics="datacamp/roomtemp",
hostname="test.mosquitto.org")

datacamp/roomtemp : b'{"time": 1543344857, "hum": 34, "temp": 24}'


datacamp/roomtemp : b'{"time": 1543344858, "hum": 35, "temp": 23}'
datacamp/roomtemp : b'{"time": 1543344860, "hum": 36, "temp": 22}'
datacamp/roomtemp : b'{"time": 1543344946, "hum": 37, "temp": 22}'
datacamp/roomtemp : b'{"time": 1543345010, "hum": 36, "temp": 13}'

ANALYZING IOT DATA IN PYTHON


Let's practice!
A N A LY Z I N G I OT D ATA I N P Y T H O N

You might also like