BCSE401L: Internet of Things

Application Layer Protocols

Avuthu Avinash Ph.D.,

Assistant Professor, SCOPE, VIT Chennai

1
What is MQTT?
MQTT : Message Queueing Telemetry Transport protocol.
• Reliable, Lightweight, Cost-effective protocol that transports messages between
devices.
• Suited for the transport of telemetry data (sensor and actuator data)

• Invented by Andy Stanford Clark of IBM

and Arlen Nipper of Arcom (now
Eurotech) in 1999

• Used by real-life IoT frameworks:

• Amazon Web Services (AWS),
• IBM WebSphere MQ,
• Microsoft Azure IoT,
• Facebook Messenger,
• etc.

M
Q
T
T

A
 MQTT Characteristics

• In 2013-14, MQTT was

adopted and published as
an official standard

• MQTT was mainly designed

for M2M communication


Publish / Subscribe
(PubSub) model

Decoupling of data
producer (publisher)
and data consumer
(subscriber) through
topics (message
queues)


Asynchronous
communication model
with messages (events)

Publisher and
Subscriber need not be
Advantages of MQTT

Source: https://www.u-
blox.com/en/blogs/insights/mqtt-beginners-guide
 Publish & Subscribe Messaging

A producer sends (publishes) a message (publication) on a topic (subject)

A consumer subscribes (makes a subscription) for messages on a topic (subject)

A message server / broker matches publications to subscriptions

Who will get the message ?


If no matches, the message is discarded

If one / more matches, the message is delivered to each matching subscriber/consumer

Topic

A topic forms the namespace in hierarchical with each “sub topic” separated by /
(forward slash)
 An example topic space :

A house publishes information about itself on:
<country>/<region>/<town>/<postalcode>/<house no>/energyConsumption
<country>/<region>/<town>/<postalcode>/<house no>/solarEnergy
• It subscribes for control commands:
<country>/<region>/<town>/<postalcode>/<house no>/thermostat/setTemp
 Cont…
Wildcards

A subscriber can subscribe to an absolute topic OR can use wildcards:

Single-level wildcards “+” can appear anywhere in the topic string
For example:
Let the Topic:
<country>/<region>/<town>/<postalcode>/<house
no>/energyConsumption

 Energy consumption for 1 house in Chennai

 India/Tamilnadu/Chennai/600127/+/
energyConsumption

India/Tamilnadu/Chennai/600127/AB1/
energyConsumption
India/Tamilnadu/Chennai/600127/AB2/
energyConsumption
India/Tamilnadu/Chennai/600128/AB3/
energyConsumption
India/Tamilnadu/Chennai/600127/AB4/smartMeter

 Energy consumption for all houses in Chennai

 India/Tamilnadu/Chennai/600127/+/+/
energyConsumption
 Cont…
Wildcards

Multi-level wildcards “#” must appear at the end of the string For example:
 Details of energy consumption, solar and alarm for all houses in 600127
 India/Tamilnadu/Chennai/600127/#
India/Tamilnadu/Chennai/600127/101/energyConsumption
India/Tamilnadu/Chennai/600128/BlockA/101/energyConsumption
India/Tamilnadu/Chennai/600130/BlockB/201/energyConsumption

NOTE :

Wildcards must be next to a separator

Wildcards cannot be used when publishing
 Cont…

A subscription can be durable or non-durable

Durable:

Once a subscription is in place,

A broker will forward matching messages to the subscriber immediately if the subscriber is
connected.

If the subscriber is not connected, messages are stored on the server/broker until the next
time the subscriber connects.


Non-durable / Transient (i.e. subscription ends with client session):
• The subscription lifetime is the same as the time the subscriber is connected to the server /
broker
MQTT Message Format

• Message Type: identifies the kind of MQTT packet within a message

• DUP: Duplicate flag – indicates whether the packet has been sent previously or not

• QoS Level: it allows to select different QoS level

• Retain: this flag notifies the server to hold onto the last received PUBLISH message
data
• Remaining Length: specifies the size of optional fields


MQTT is lightweight

because each packets consists of a 2-byte fixed header with optional variable header
and payload fields
Cont…
Message header field Description / Values
Message Type (4 bits) 0: Reserved 8: SUBSCRIBE
1: CONNECT 9: SUBACK
2: CONNACK 10: UNSUBSCRIBE
14 message types, 3: PUBLISH 11: UNSUBACK
2 are reserved
4: PUBACK (Publish ACK) 12: PINGREQ
5: PUBREC (Publish Received) 13: PINGRESP
6: PUBREL (Publish Release) 14: DISCONNECT
7: PUBCOMP (Publish Complete) 15: Reserved
DUP (1 bit) Duplicate message flag. Indicates to the receiver that this message may have already been received.
1: Client or server (broker) re-delivers a PUBLISH, PUBREL, SUBSCRIBE or UNSUBSCRIBE message
(duplicate message).

QoS Level (2 bits) Indicates the level of delivery assurance of a PUBLISH message.
0: At-most-once delivery, no guarantees, «Fire and Forget».
1: At-least-once delivery, acknowledged delivery.
2: Exactly-once delivery.

RETAIN (1 bit) 1: Instructs the server to retain the last received PUBLISH message and deliver it as a first message
to new subscriptions.

Remaining Length (1-4 bytes) Indicates the number of remaining bytes in the message, i.e. the length of the (optional) variable
length header and (optional) payload.
Message Types
 RETAIN
RETAIN=1 in a PUBLISH message instructs the server to keep the message for this topic.
When a new client subscribes to the topic, the server sends the retained message quickly.

 Typical application scenarios:


Clients publish only changes in data, so subscribers receive the last known good value.

 Example:

Subscribers receive last known temperature value from the temperature data topic.

RETAIN=1 indicates to subscriber B that the message may have been published some time ago.
 Remaining Length (RL)
• The remaining length field encodes the sum of the lengths of:
1. (Optional) variable length header
2. (Optional) variable length payload

• To save bits, RL is a variable length field with 1 to 4 bytes.

• The most significant bit (msb) of a length field byte has the meaning continuation bit
(CB).
• If more bytes follow, it is set to 1.

RL is encoded as:: a * 1280 + b * 1281 + c * 1282 + d * 1283

and placed into the RL field bytes as follows:
7 bits
msb
MQTT QoS

Even though TCP/IP provides guaranteed data delivery, data loss can still occur if a TCP connection
breaks down and messages in transit are lost.


Therefore, MQTT adds 3 quality of service (QoS) levels on top of TCP

PUBREC (REC:
received) Packet is the
response to a PUBLISH
Packet
at least
once PUBREL (REL: release)
Packet is the response to a
PUBREC Packet

PUBCOMP (COMP:
complete) Packet is the
response to a PUBREL
Packet.
 Cont…
QoS level 0:

At-most-once delivery («best effort»).

Messages are delivered according to the delivery guarantees of the underlying network (TCP/IP).

Example application: Temperature sensor data which is regularly published. Loss of an individual
value is not critical since applications (i.e. consumers of the data) will anyway integrate the values
over time

QoS level 1:
•
At-least-once delivery.

Messages are guaranteed to arrive, but there may be duplicates.

Example application: A door sensor senses the door state. It is important that door state
changes (closed->open, open->closed) are published losslessly to subscribers (e.g. alarming
function). Applications simply discard duplicate messages by comparing the message ID field.

QoS level 2:

Exactly-once delivery.

This is the highest level that also incurs most overhead in terms of control messages and the need
for locally storing the messages.

Exactly-once is a combination of at-least-once and at-most-once delivery guarantee.

Example application: Applications where duplicate events could lead to incorrect actions, e.g.
sounding an alarm as a reaction to an event received by a message. So, it avoids duplicate.
 PUBLISH msg flow
QoS level 0:
 With QoS level 0, a message is delivered with at-most-once delivery semantics («fire-and-forget»).

QoS level 1:
 QoS level 1 affords at-least-once delivery semantics. If the client does not receive the PUBACK in
time, it re-sends the message.
Cont…
QoS level 2:
 QoS level 2 affords the highest quality delivery semantics exactly-once, but comes with the cost of
additional control messages.
 Example: CONNECT & SUBSCRIBE msg flow
Case 1:
• Session/connection and subscription setup with clean session flag = 1 (non-durable subscription)
Cont…
Case 2:
• Session/connection and subscription setup with clean session flag = 0 (durable subscription)
What is MQTT-SN?
MQTT-SN (MQTT for sensor networks) is a variant of MQTT that has been optimized for
use in low power environments such as sensor networks, as the name suggests.

SN adds extra functionality for use cases where lower power is

required.

• QoS mode -1: allows for fire-and-forget messaging

• Topic aliases: allows for simplified publishing and reduced data overheads

• Sleep mode (disconnected sessions): allows messages to be queued on the broker while the
remote Thing or device is powered off
 Send Sensor values to Cloud using MQTT
Protocol
• The Ultrasonic Sensor
reads the distance
between itself & the
object

• Sends the measured

distance to the cloud
server using MQTT
protocol.

• Here, the cloud is

ThingSpeak.com
Requirements
Hardware
• ESP8266 NodeMCU
• Breadboard
• Jumper Wires
• HC-SR04 Ultrasonic
Sensor
• Raspberry Pi

Software
•Arduino IDE
•Fritzing

Cloud Platform
•ThingSpeak Account

ThingSpeak is an IoT analytics PaaS

that allows you to aggregate,
visualize, and analyze live data
streams
 ESP8266 NodeMCU: Libraries
The ESP8266 NodeMCU requires certain libraries for optimal use of
the board.

• PubSubClient – It is required for the MQTT Messaging.

• ThingSpeak – It is a communication library for supporting to send data to
the cloud.
• ESP8266WiFi – It is required to use the Wi-Fi capabilities of the NodeMCU.

Step to install above libraries for the

programming part:
• Open Arduino IDE
• Move on to Tools
• Manage Libraries…
• Search and install the above mentioned libraries
Setup Connections
Subscribe to Cloud
TP-LINK_ED34 Topic
Serve
“distance”
r
Publish on MQTT Client
Topic
“distance” (Subscriber)

MQTT Client
(Publisher) MQTT Server
(Broker)
"192.168.0.104"
Programming ESP8266
#include <ESP8266WiFi.h> void setup_wifi () {
#include <PubSubClient.h> delay(10);
Serial.println();
#define TRIGGER D6 Serial.print("Connecting to ");
#define ECHO D8 Serial.println(ssid);
long duration, cm;
WiFi.begin(ssid, password);
const char* ssid = while (WiFi.status() != WL_CONNECTED)
"TP-LINK_ED34"; { delay(500);
const char* Serial.print(".");
password = }
"48193580"; Serial.println("");
Serial.print("WiFi connected - ESP IP
//Raspberry Pi IP address: ");
Address
const char* Serial.println(WiFi.localIP());
}
mqtt_server =
"192.168.0.104";

WiFiClient
espClient;

PubSubClient
 Cont…
void callback (String topic, byte* message, void reconnect ()
unsigned int length) {
{ // Loop until we're reconnected
Serial.print("Message arrived on topic: "); while (! client.connected ()) {
Serial.print(topic); Serial.print("Attempting
Serial.print("Message: "); MQTT connection...");

String messageTemp; if
for (int i = 0; i < length; i++) (client.connect("ESP8266Clie
{ Serial.print((char)message[i]); nt"))
messageTemp += (char)message[i]; {
} Serial.println("connected"
Serial.println(); );
} client.subscribe("esp8266
/4");
}
else {
A callback is a function that is
} Serial.print("failed, rc=");
passed as an argument to } Serial.print(client.state());
another function. Serial.println(" try again in 5 seconds");
delay(15000);
}
Cont…
void setup () digitalWrite(TRIGGER, LOW);
{ Serial.begin(115200); delayMicroseconds(2);
pinMode(TRIGGER,OUTPUT); digitalWrite(TRIGGER, HIGH);
pinMode(ECHO,INPUT); delayMicroseconds(10);
digitalWrite(TRIGGER, LOW);
setup_wifi ();
duration = pulseIn(ECHO, HIGH);
client.setServer (mqtt_server, 1883);
cm = duration / 29 / 2;
client.setCallback (callback); static char distanceincm[7];
}
dtostrf(cm, 6, 2, distanceincm);

client.publish("/esp8266/distance", distanceincm);

void loop () { Serial.print("Distance: ");

if (! client.connected ()) { Serial.print(cm);
reconnect(); Serial.print("cm");
} Serial.println();
if(! client.loop() ) delay(5000);
client.connect("ESP8266Client"); }
 Programming R. Pi
import paho.mqtt.client as mqtt
import urllib3 # powerful HTTP client for Python if val != '' :
conn = http.request('GET', baseURL +'&field1=%s'%(val))
myAPI = 'SMLI56456456RHUB' print(conn.status)
baseURL = 'https://api.thingspeak.com/update?api_key=%s' conn.read()
% myAPI conn.close()
val='' val = ''
http = urllib3.PoolManager()
def main():
def on_connect (client, userdata, flags, rc): mqtt_client = mqtt.Client()
print("Connected with result code "+ str(rc)) mqtt_client.on_connect = on_connect
client.subscribe("/esp8266/distance") mqtt_client.on_message = on_message

def on_message (client, userdata, message): mqtt_client.connect ('localhost', 1883,

print("Received message '" + str(message.payload) + 60)
"' on topic '" + message.topic) # Connect to the MQTT server and
distance = 0 process messages in a
if message.topic == "/esp8266/distance": ba
print("Distance updated") ck
distance = str(message.payload, 'UTF-8') gr
distance = distance.strip() ou
print(distance) nd
global val thr
val = distance ea
d.
mqtt_client.loop_start()
ThingSpeak Cloud Dashboard
 What is CoAP
• CoAP - Constrained
Application Protocol.

• Specialized web transfer

protocol.

• Devised for constrained and

low power networks. CoAP is
an application layer protocol
(similar as HTTP).

• Inspired from HTTP, it

follows the request- response
pattern

• CoAP has a transparent

mapping to HTTP.

• CoAP protocol spec is

specified in RFC 7252.
 Characteristics of CoAP
• It is very efficient RESTful protocol.
• It is Embedded web transfer protocol What is REST?
• Representational State Transfer (REST) is
a software architecture that imposes
• Low parsing complexity. conditions on how an API should work.
• Proxied to/from HTTP.
 GET, POST, PUT and DELETE methods are used. • API developers can design APIs using
 Uses subset HTTP response codes. several different architectures. APIs that
follow the REST architectural style are
 URI is supported.
called REST APIs or RESTful web APIs.

• Low header overhead • Some of the constraints of the REST

architectural style:
• It uses small and simple 4 byte header. • Uniform interface
• Statelessness
• Supports binding to UDP, and TCP. • Cacheability
• Code on demand
• Layered system
• DTLS based certificate security is used. • Etc.
 CoAP Structure
• There are two different layers that make CoAP Protocol:
 Message
 Request/Response.

The Messages layer

 deals with UDP
 deals with asynchronous messages.
 is meant for Re-transmitting lost packets.

• Message layer supports four types of messages:

 Confirmable (CON)
For Request
 Non-confirmable (NON)
 Acknowledgment (ACK)
 Reset (RST) For Response

 The Request/Response layer

 manages request/response interaction based on request/response messages.

• Request/Response layer uses different methods

 Request Methods: GET, POST, PUT, and DELETE.
 Response Methods: 2.xx (success), 4.xx (client error), 5.xx (server error).
CoAP Status Codes in Response

Source: https://www.slideshare.net/aniruddha.chakrabarti/coap-web-protocol-for-iot
CoAP Message Format

• Ver (2 bit): indicating the CoAP version.

• T (2 bit): indicating the message type
• Confirmable (CON), Non-confirmable (NON), Acknowledgement (ACK), Reset (RST)

• TKL (4 bit): Specifies the size (0-8 bytes) of the Token field
• Token (0-8 byte): correlates requests and responses

• Code (8 bit): indicates

• request method for a request message, e.g: GET is the request method
• response code for a response message. e.g. 2.05 is the response code

• Message ID (16 bit):

• Detects message duplication
• Used to match ACK and RST message types to CON and NON message types.
 CoAP Messaging Model
• CoAP deals with UDP for exchanging messages between endpoints.
• Each CoAP message has a unique ID
• Unique ID is useful to detect message duplicates.

• Reliable messaging is obtained using a Confirmable message (CON).

 A CON message is sent again and again until the other party sends an acknowledge (ACK) message or
negative acknowledgement through reset message (RST).
 The ACK message contains the same ID of the CON message
 If the server has troubles managing the incoming request, send back a Reset message (RST) instead of the ACK.
 Retransmissions are made until all attempts are exhausted
 Cont…
• Non-confirmable (NON) messages don’t require an ACK by the server.

• They are unreliable messages or in other words messages that do not contain critical information that
must be delivered to the server.
o Example : Messages that contain sensed values from sensors.
o Even if these messages are unreliable, they have a unique ID.
 CoAP Request/Response Model
The CoAP Request/Response is the second layer in the CoAP Abstraction layer.
• The request is sent using a Confirmable (CON) or Non-confirmable (NON) message.
• There are several scenarios (e.g. Piggy-Backed, Separate Response) depending on if the server can
answer immediately to the client request or if not available.

CoAP
URI:
 Cont…
If the server can answer immediately to the client request (Piggy-Backed response)
• If the request is carried using a Confirmable message (CON).
 the server sends back an ACK message to client containing the response or the error code.

• The Token is different from the Message-ID and it is used to match the request and the response.
 Cont…
If the server can’t answer immediately to the request coming from the client (Separate Response).
• It sends an ACK message with an empty response.
• When response is available, then the server sends a new CON message to the client containing the
response.
• At this point, the client sends back an ACK message.
 CoAP Security Aspects
• CoAP uses UDP to transport information.
• CoAP relies on UDP security aspects to protect the information.

• As HTTP uses TLS over TCP,

CoAP uses Datagram TLS over
UDP. DTLS supports RSA, AES.

• In some constrained devices,

some of
DTLS cipher suits may not be
available.

• Some cipher suites introduces

more complexity and constrained
devices may not have resources
enough to manage it.
 CoAP Vs. MQTT
• MQTT uses a publisher -subscriber. • CoAP uses a request-response
paradigm
• MQTT uses a central broker to dispatch • CoAP is essentially a one-to-one protocol
messages coming from the very similar to the HTTP protocol.
publisher to the clients.

• MQTT is an event-oriented protocol. • While CoAP is more suitable for state

transfer.

• MQTT uses Asynchronous messaging. • CoAP uses both Asynchronous &

Synchronous messaging
CoAP vs. HTTP