IoT Messages and

Storage

Troy Merritt 2025

Unit Co-ordinator Week 6
1
Recap: IoT Architecture

2
Webservers from and IoT Perspective

3
Messaging

4
What are Middleware/Carrier Languages?

Messages between IoT devices that carry data.

These messages are not instructions that a
compiler will execute.
Used to represent states or conditions.
Used for all levels of IoT communication.
Things to middleware.
Middleware to cloud.
Cloud to user output.
Cloud back to things.

5
Components of carrier languages

Identity of devices:
1000s of devices in a IoT system
Each one needs to have a different identification
mechanism
Sensor/Actuator data:
Sensing information
Decision information
Metadata:
Time
Geo Locations
Application Details

6
Best Practices for Carrier Languages

Keep it small.
Especially for the Things to Middleware.
Keep it with descriptions
Especially for the Middleware to Cloud and Cloud to
User related outputs
Keep a mapping between the things’ components
with the storage all throughout the IoT system
Easy to identify things
recordtemp.php?t=%d&h=%d&ts=%d
Easy to scale up
Easy for maintenance

7
Data Storage in IoT Systems

IoT End node devices have very limited storage

(often only a few KBs) and that data is lost when
the device resets.
Cloud services are used to store large volumes of
data from many devices for big data analysis.
The cloud application has to maintain the data.
i.e. establish the relationship between the data
and devices and the applications.

8
IoT Data exchange

9
IoT Data exchange

10
Storage

11
On Device Storage

On device storage could be short term (data being

stored before being packaged and deleted) or long term
(logging)
EEPROM
SD Card
Reads and writes cost power.
But so does data transmission, so this has to be
balanced.

12
Middleware Storage

Middleware devices are all about communication.

Communication settings
Routing information
Managing device identities
Storage on these devices tends to be minimal
Storage is also fairly volatile
Data is quickly deleted once it’s been forwarded.

13
Cloud Server

Data can be stored in databases

Relational (MySQL)
Non-relational (MongoDB)
Duplicate data across multiple instances.
Implement cloud and database related security issues
Encryption
Timestamp everything
SQL injections

14
Text based
communication

15
Data structure

When data is being exchanged between devices, the

data must be structured.
The sender needs to send everything the receiver
needs.
The receiver needs to understand what data is being
received so that it knows what to do with it.
Common structures include CSV, XML, JSON.
For end node devices, the data aims to be as small as
possible, sometimes packaged down to the byte.

16
Comma / Tab Separated values

Minimal data structure.

More data, less metadata.
Easy for IoT devices to parse.
Difficult for humans to parse.
Structure needs a key to explain.
Example message:
100,25.57,on,HIGH

17
XML
Highly structured data
Lots of metadata relative to data.
larger files
Easy for devices to parse, but requires more
memory/storage.
Easy for humans to parse.
Example message:
<iot id=e2f9a344>
<ldr>
<value>24 </value>
</ldr>
<distance>
<value> 10 </value>
<direction> 270 </direction>
</distance>
</iot>

18
JSON
Moderately data structure.
Less metadata than XML.
Smaller files than XML
Easy for devices to parse, but requires more
memory/storage.
Easy for humans to parse.
Example message:
{
“IoT":
[
{“type": “id", “value": “e2f9a344"},
{“type": “ldr", “value": “24"},
{“type": “distance", “value": “10", “direction": “270"}
]
}

19
Applications
and Data

20
Application Logic (for IoT devices)

The logic of what you want to achieve with the IoT

device and devices in group.
A program that tells the IoT device what to do OR
accepts inputs from device(s) to decide what to do.
e.g. authentication­
Send username, password or any ID details
e.g. send records or control outputs
send temperature etc …
e.g. receive records or setting
retrieve any settings
e.g. turn on the fan and at what speed

21
IoT Data Generation

Requirements for data exchange will differ for each

IoT system will depending on what the end node
device:
wants from the internet
and can give to the internet
Request/response methods need to be agreed upon
so that they are predictable.
Communication with end node devices can not
always be guaranteed.
Often end devices sleep in between requests.
But the cloud never sleeps...

22
IoT Data Generation
Authentication requests
The device sends:
Authentication inputs ie:
Username
Password
or any ID details such as RFID card numbers,
finger prints etc…
What the device gets:
True or false response
Authentication mechanism:
PHP matches credential details and then responds
Server will log the authentication request.

23
IoT Data Generation
Log data request
The device sends:
Raw data ie:
Current temperature (float)
Time stamp
Device ID (and authentication if required)
What the device gets:
Success / Failure
How it works:
PHP matches the device ID.
The temp data and time stamp is logged to the
database.

24
IoT Data Generation
Automation request (Thermostat)
The device sends:
Raw data ie:
Current temperature (float)
Time stamp
Device ID (and authentication if required)
What the device gets:
Current fan mode and setting (on/off and percentage)
How it works:
PHP matches the device ID.
The temp data and time stamp is logged to the
database, and compared to a threshold.
The server returns the automation data for the fan.

25
MQTT

26
Traditional IoT communication

Traditional pull methods have several

downsides.
Requests are sent when the
information might not have changed.
More data is exchanged.
This uses more power.

27
P2P Data exchange for IoT

IoT devices can communicate using P2P methods

Mesh networks exchange data in this way.
IoT devices can implement TCP/IP protocols
which allow them to serve content (data) to other
devices.
This is very difficult to implement in Class 0/1
devices due to the lack of memory and storage
space.

28
What do IoT devices need for communication?

29
Message Queue Telemetry Transport

MQTT is an attempt to deliver a protocol for IoT

communications that matches its data needs.
MQTT implements a publisher/subscriber model to
address these needs.
MQTT organises data into topics.
MQTT uses a broker to manage communications and
connections.
MQTT uses a binary message format for data transfer
More compact that text-based protocols.
Messages are only sent went data is updated.

30
The Publish and Subcribe Model

31
Publish and Subcribe (Broker)

The broker is responsible for:

Authentications: checking the details of the
clients (publisher and subscribers).
Message passing: pass messages to any
subscriber.
Message Queuing: Hold on to messages for
limited period of time (but not infinitely).

32
Publish and Subcribe (Publisher)

Source of data publishes sensor and actuator

information
Example:
Temperature: 24
FanSpeed: 3
Should publish in time (as promised): They
should publish in definite time intervals.
A publisher can publish to multiple topics.

33
Publish and Subcribe (Subscriber)

A subscriber:
Can ask for any data at any time (subscribe
to a feed/channel/topic)
Expect the data in time
Can ask for multiple feeds at the same time
The broker maintains a list of subscribers and
pushes their subscribed topics to them as they
are updated.

34
Topics

Topics are pieces of data that can be published

to or subscribed to.
Topics can have paths.
example:
temperature/24
room1/temperature/27
room1/humidity/65
A subscriber can subscribe to multiple topics.
A publisher can publish to multiple topics.

35
How it works

Client 1 and 4 are publishers to the temperature

topic.
Client 2 and 3 are subscribers to the temperature
topic.
The broker accepts the most recent data to the
temperature topic.
When it’s updated, this is published to
subscribers.

36
MQTT Advantages
Can be Asynchronous as in HTTP:
Device can send the output and don’t stay
blocked up to when it gets the response
Persistent connection
Not stateless like in HTTP
Real Time
Low energy as it is more light weight (lower
overhead)
Packet headers are very small
Flexible
No need to know IP address
Only the feed name is sufficient

37
MQTT Disadvantages

Difficult to use in non-real-time environments

Devices cannot go to sleep on their own.
Need to follow a definite pattern as the data
in the channel/topic/feed is not held forever
Less Security of data
Useful for reporting “non-critical” data only.

38
Coming up...
Next week:
IoT Data Visualisatons

Thanks for attending!

39