ESB Communication Guide

API & Rabbit MQ Usage

Authors
Lucas Pons Bayarri, Mario González Carabayo, Moisés Antón Garcı́a
(ETRA I+D)
 API & Rabbit MQ Usage

Contents
1 Introduction 2

2 Rabbit MQ basics 3
2.1 Constructing the route . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Managing subscriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 List of all available queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Perform the connection to the Rabbit MQ 5

3.1 Example in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2 Encrypted MQTT conection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4 API Usage 9
4.1 API basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2 Common endpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.2.1 Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.2.2 Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.3 Last Value Data API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.3.1 Access all elements from a collection . . . . . . . . . . . . . . . . . . . . 13
4.3.2 Access one element of the collection . . . . . . . . . . . . . . . . . . . . 14
4.3.3 Get one specific field from every element of the collection . . . . . . . . 15
4.3.4 Get one specific field from one element of the collection . . . . . . . . . 15
4.4 Historical Data API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.5 Aggregate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.6 Find . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

A Data models for the ESB communications 19

A.1 Measurement data model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
A.1.1 Example of valid message . . . . . . . . . . . . . . . . . . . . . . . . . . 20
A.2 Status data model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
A.2.1 Example of valid message . . . . . . . . . . . . . . . . . . . . . . . . . . 21
A.3 cluster data model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
A.3.1 Example of valid message . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Page 1
 API & Rabbit MQ Usage

1 Introduction
The Enterprise Service Bus (ESB) is a software architecture model that facilitates commu-
nication between various web services. In this context, our ESB is composed of two distinct
parts, each serving a different function: a message broker (RabbitMQ) and an API.

• Message Broker: This element allows users to send messages to a specified queue or topic,
and also receive all messages dispatched to those locations. Assets will be publishing
messages to these queues or topics, which means that anyone with the appropriate
permissions can view these messages by subscribing to the topic directly. This provides
a system for low-latency communication.

• API: All messages published through the message broker are also pushed to a database.
They can then be accessed via API calls. The topic of the message determines the
collection in which it will be stored. However, this is only the case if the message
satisfies a predefined data structure. Each message type has a unique schema.

This document will explain both, the rabbit MQ and the API usage.

Figure 1: Diagram of the ESB

Page 2
 API & Rabbit MQ Usage

2 Rabbit MQ basics

2.1 Constructing the route

Rabbit MQ is a messaging middleware that implements the AMQP and MQTT protocols,
this means it is a server that allows clients to connect to messaging queues to publish and/or
receive messages. There are a couple key concepts to understand how the connections are
performed.
The RabbitMQ server is hosted at the URL ”mqtt.k8s.etra-id.com”, and all connections
are made to this endpoint.. Given that, extra routing is needed to specify the queue where we
want to establish communications. The routing in AMQP/MQTT works by concatenating
words, separated with /, in our case, will be “/virtual host/island/queue/asset id” Each of
thees words are called topic levels, and are separated with an topic separator ‘/’. The topic
levels used in this case are:

• Virtual Host: This identifies a virtual separation on groups of topics, allowing to give
permissions to the users only to some of thees virtual hosts. In this case, will be “ianos”

• Island: As there are 2 islands that will need the same message types, all the queues are
duplicated so there are the same number of queues for Ameland and for Terceira.

• Message type: For each island, there are a bunch of queues where to send the messages
depending on the payload. Theese are:

– measurement
– status
– cluster
– message

• Location ID: This routing key allows to group assets based on publisher defined loca-
tions, such as house 001 or any identifier.

• Asset ID: This last routing key will act as the ID of the message, it will be stored at
the DataBase as it, so be sure to place it correctly. While the asset id of the message
in the database will be ”/island/location/ id

In this specfic case, the queue name is the combination of the island and the message type,
so the queues are: ameland/measurement, ameland/cluster, terceira/status,...
For example, “/ianos/ameland/measurement/house 001/asset 002” would be a correct
route for publishing a message.

Page 3
 API & Rabbit MQ Usage

2.2 Managing subscriptions

All the discussed above is mandatory in order to publish a message correctly to one queue,
but working with subscriptions is more flexible. You can subscribe to anything under ianos
virtual host, that meaning, you can subscribe to every single queue at once or just to one
specific Asset ID for example. The queues are just meant to receive the messages and manage
how the are consumed by the clients, but AMQP/MQTT allows the clients to subscribe to
any topics that match their preferences (filtering only by the routing, not the content of the
messages). To facilitate this, there are two routing operators ‘*’ and ‘#’, which work in a
manner similar to regular expressions.

• The * operator matches any single topic level. For example, /ianos/ameland/*/*/device 003
would receive every message sent to ianos virtual host, any of the queue names from
ameland and that are sent with device 003 asset id. You can concatenate the ‘*’ like
/ianos/ameland/*/*/* for example

• The # operator matches any number of topic levels.. Its not recommended to avoid
assets from sending messages to a wrong route and still getting received. An example
could be /ianos/ameland/#. Anyway, the example with the * operator will be always
preferred.

One key concern is that the complete route must always be specified, with routing keys or
without them, but for example if you subscribe to “ianos/ameland/cluster”, won’t receive any
message that has extra routing keys, as it is only receiving the messages that match exactly
this route (and they should be sent adding the asset id as an extra routing key. The correct
way to subscribe to one queue, would be adding a /+ after the queue name

2.3 List of all available queues

The current list of topics is the following:

• ianos/ameland/measurement/<location>/asset id: used to send specific measurements

of the asset

• ianos/ameland/status/<location>/asset id: used to send the current status of the asset

• ianos/ameland/cluster/<location>/asset id: used to send the cluster information of the

asset

• ianos/ameland/message/*: used by services to listen for specific messages that may

come from other services (inter-process communication).

Page 4
 API & Rabbit MQ Usage

• ianos/terceira/measurement/<location>/asset id: used to send specific measurements

of the asset

• ianos/terceira/status/<location>/asset id: used to send the current status of the asset

• ianos/terceira/cluster/<location>/asset id: used to send the cluster information of the

asset

• ianos/terceira/message/*: used by services to listen for specific messages that may come
from other services (inter-process communication).

3 Perform the connection to the Rabbit MQ

With the user and password provided you will be able to publish and subscribe to queues in
the virtual-host ianos using the mqtt protocol. The configuration you should use is as follows:

• url: mqtt://mqtt.k8s.etra-id.com In some cases, the prefix mqtt:// may not be

necessary. Please check whether your modules automatically add this prefix
or not

• ports:

– 1883 Don’t use unless certificate is not working

– 8883 Encrypted communications (needs the provided ca certificate)

• virtual-host: ianos

• queue: you can see the list at 2.3, keep in mind that ianos is the virtual host, don’t
duplicate the key.

• credentials: with the fields ’username‘ and ’password‘ as provided

• payload: should include the field id (matching the id of the topic’s routing key) and
the desired data

• tls: with the provided certificate (only to the 8883 port). Connections can be performed
with or without the certificate, as it only identifies the server before negotiating any
encryption key.

Page 5
 API & Rabbit MQ Usage

3.1 Example in Python

Here you can see examples for publisher and subscriber using the open source library paho-
mqtt.
Publish
1 import paho . mqtt . publish as publish
2 import string
3 import random
4
5 # Example of a autogenerated ID
6 def ge ne ra te _ra nd om _I D ( length ):
7 return ( ' '. join ( random . choice ( string . ascii_letters +
8 string . digits ) for _ in range ( length )))
9
10 def publisher ( route , host , port , auth , data ):
11 publish . single ( route , json . dumps ( data ) ,
12 hostname = host , port = port , auth = auth )
13
14 if __name__ == " __main__ ":
15 auth = {
16 " username ": " ianos : username " ,# IMPORTANT set the ianos :
17 " password ": " yourPassword "
18 }
19 port = 1883
20 host = " mqtt . k8s . etra - id . com "
21 route = " ianos / ameland / message /+"
22 data = {
23 " _id ": ge ne ra te _r an do m_ ID (17) ,
24 " data ": " test " ,
25 " more_data ": " more_test "
26 }
27 publisher ( auth , data )

Page 6
 API & Rabbit MQ Usage

Subscribe
1 import paho . mqtt . subscribe as subscribe
2
3 def on_message_print ( client , userdata , message ):
4 print ("% s % s " % ( message . topic , message . payload ))
5
6 def subscriber ( route , host , port , auth ):
7 subscribe . callback ( on_message_print , route ,
8 hostname = host , port = port ,
9 client_id =" PythonJJ " , auth = auth )
10
11 if __name__ == __main__ :
12 host = " mqtt . k8s . etra - id . com "
13 auth = {
14 " username ": " ianos : username " , # IMPORTANT set the ianos :
15 " password ": " yourPassword "
16 }
17 port = 1883
18 route = " ianos / ameland / message /+"
19 subscriber ( route , host , port , auth );

3.2 Encrypted MQTT conection

If you want to use the encrypted communication to the same mqtt service, you can use the
implemented TLS protocol by connecting to port 8883 and including the Certificate Authority
certificate that should have been provided alongside the credentials. This way, both examples
above should look like the next example.
1 import ssl
2 import paho . mqtt . subscribe as subscribe
3
4 def on_message_print ( client , userdata , message ):
5 print ("% s % s " % ( message . topic , message . payload ))
6
7 def subscriber ( route , host , port , auth ):
8 # Creates a ssl context
9 context = ssl . c r e a t e _ d e f a u l t _ c o n t e x t ( cafile =" ca_certificate . pem ")
10
11 # Disable hostname verify mode ( If no certificate is provided )
12 context . check_hostname = False
13 context . verify_mode = ssl . CERT_NONE
14
15 subscribe . callback ( on_message_print , route ,
16 hostname = host , port = port ,
17 client_id =" PythonJJ " , auth = auth , # Unique client_id , or messages will be div
18 tls = context )
19 print (" Subscribed ")
20
21 if __name__ == " __main__ ":

Page 7
 API & Rabbit MQ Usage

22 host = " mqtt . k8s . etra - id . com "

23 auth = {
24 " username ": " ianos : username " , # IMPORTANT set the ianos :
25 " password ": " password "
26 }
27 port = 8883
28 route = " ianos /#" #
29 subscriber ( route , host , port , auth )

Page 8
 API & Rabbit MQ Usage

4 API Usage

4.1 API basics

In addition to the Rabbit MQ (the current ESB), Etra offers a way to access data sent to
Rabbit MQ queues, facilitating both the retrieval of current values and historical data. To
accomplish this, two distinct APIs are provided, one for each function.

• Current value: Contains only the most recent message sent to the corresponding queue
(since there is one collection per queue) for each asset.

• Historical value: Each collection is linked to another collection that replaces the asset
id with a different key, generating a unique id for every message intended for storage
in the corresponding collection. This design ensures that the database maintains every
historical value of the collection, accessible at any time.

The two API deployed are:

• https://citric-api.k8s.etra-id.com/api/v2 for last value

• https://ianos-api.k8s.etra-id.com/api/v2 for historical data

When a message is published to a queue, a microservice also consumes it, checks the
routing keys, and stores the message in the appropriate database, but only if the message
adheres to the specified schema. If a client is subscribed to the same queue, they will also
receive the message. This is because AMQP, the protocol used, allows for multiple consumers
and does not restrict a message to be exclusively consumed by one client, enabling lower
latency message communication. However, the client is responsible for verifying that the
message adheres to the valid schema, as AMQP’s scope is strictly message delivery and does
not include data validation. If a client wishes to receive only validated messages, they should
use the provided API.
The messages stored in the database are organized into collections, with the name based
on a combination of a layer and a collection. The layer is ’ianos’, and the collection name
is derived from the queue to which the message was sent, converted to camel case. This
means, for example, that a message sent to the queue ’/ianos/ameland/measurement/+’ will
be referenced in the API as ’ianos/amelandMeasurement’. Now that we have established the
basics, we will delve into the specifics of using the APIs.
First the common endpoints in both API’s, and then the specific ones.

Page 9
 API & Rabbit MQ Usage

!Important: Use JSON format for the communications with the API, otherwise there
could be unexpected issues.
All the methods below will be under url/api/v2

4.2 Common endpoints

4.2.1 Health

You can always call to /health to check if the API is working (or check if the route is correct),
there is no credentials needed and accepts any method.

Figure 2: Health request with response using Postman

Page 10
 API & Rabbit MQ Usage

4.2.2 Login

All the requests that retrieve data from any collection, will require the user to log in to get
his userId and an authorization token that will be added to the communications with the
API. To get them, you have to call /login with a POST message, and set your username and
password in the body as in the figure 3.
You will receive a userID and a token that can be used as headers, being X-User-ID and
X-Auth-Token or as the query parameters appId and keyId.

Figure 3: Login request with response using Postman

!Important: the JSON keys are user and password, won’t work with any other.

Page 11
 API & Rabbit MQ Usage

4.3 Last Value Data API

This API allows access to the current state of the collections, where the last message received
for each id overrides the previous one. You can access any of the collections associated with
the queues to get, add, edit or delete, as it is a REST API. If this was a problem, the write
permissions could be removed for all the users.
To see all the method and use them interactively, you can check the swagger Clicking
Here.
Or searching the following url: https://citric-api.k8s.etra-id.com/api/v2/help

Figure 4: Swagger of the last value data API

Page 12
 API & Rabbit MQ Usage

4.3.1 Access all elements from a collection

/{layer}/{collection} → for example: /ianos/amelandMeasurement

Access point to get all the elements from the collection. Use the method GET to receive
all the elements

Figure 5: Get all the elements of the colletion

The other methods available are:

• POST: For insert

• PATCH: For update

• DELETE: For removal

All of them, will affect to the elements included in the body, check the swagger to see
further examples.

Page 13
 API & Rabbit MQ Usage

4.3.2 Access one element of the collection

/{layer}/{collection}{id} → for example: /ianos/amelandMeasurement/device 002

Figure 6: Get selected item

The other methods available are:

• POST: For insert

• PATCH: For update

• DELETE: For removal

All of them, will affect to the element with the id in the url.

Page 14
 API & Rabbit MQ Usage

4.3.3 Get one specific field from every element of the collection

/{layer}/{collection}/field/{field} → for example: /ianos/amelandMeasurement/field-

/asset id

Figure 7: Get specific field of every item

4.3.4 Get one specific field from one element of the collection

/{layer}/{collection}/{id}/field/{field} → for example: /ianos/amelandMeasuremen-

t/device 002/field/asset id

Figure 8: Get specific field of selected item

Page 15
 API & Rabbit MQ Usage

4.4 Historical Data API

This API allows access to the historical data of the collections. Every time a message with the
same id is received by the corresponding queue and written to the database, is also written
in other collection, that changes the id, saving the previous id in other field, changes the id
of the element to a random one, so elements with same “real” id can coexist. This way, you
can access every historical value at any time over API calls.
To see all the method and use them interactively, you can check the swagger Clicking
Here.
Or searching the following url: https://ianos-api.k8s.etra-id.com/api/v2/help

Figure 9: Swagger of the historical data API

Page 16
 API & Rabbit MQ Usage

4.5 Aggregate

/{aggregate}/{layer}/{collection} → for example: /aggregate/ianos/amelandMeasure-

ment
With this route, you can get the aggregation of all the elements in the historical collection,
receiven an array with one element per id, and the message data separated by measurement
name. This route allows to filter by some fields, using query parameters, as requested in the
meetings. As it was already implemented, the POST method is still allowed if someone wants
to use it, anyway, the allowed parameters are:

• start date: ISO string formatted. For example ”2020-11-08T12:45:40.035Z”

• end date: ISO string formatted. For example ”2020-11-08T12:45:40.035Z”
• asset id
• measurement name

Figure 10: Aggregate historical data

Page 17
 API & Rabbit MQ Usage

4.6 Find

/{find}/{layer}/{collection} → for example: /find/ianos/amelandMeasurement

This route works similar to the aggregate, but instead doesn’t aggregate the received data,
just returns all the elements on the historical database, filtered by the same query params
as the aggregation. None of them being mandatory, if no param is provided, will just return
every element in the collection.

• start date: ISO string formatted. For example ”2020-11-08T12:45:40.035Z”

• end date: ISO string formatted. For example ”2020-11-08T12:45:40.035Z”

• asset id

• measurement name

Figure 11: Find historical data

Page 18
 API & Rabbit MQ Usage

A Data models for the ESB communications

As we discussed, for each island we will have different topics, and depending on it, the message
will have to follow or not some specific data model so the database can guarantee the data is
in the right format.
In this case, as discussed with the WP4 partners, we will have 3 different data models,
that differ in some specific fields but almost identical overall. This being:

• Measurement

• Status

• Cluster

Each of them linked to their corresponding topic (which has on of theese names in the
mqtt topic path). All the fields that will be specified are mandatory, extra fields are not
allowed and some fields must be a fixed value type. Some examples will be provided, one per
model, for clarification purposes.

Page 19
 API & Rabbit MQ Usage

A.1 Measurement data model

• asset id: with /{island}/{locationId}/{assetId} (string, assetId being the same as id)

• id: This field will be automatically filled with the corresponding routing key of the
mqtt topic (string, not required)

• measurements message: Array with all the different measurements, only 1 value per
measurement allowed at a time (array of objects)

– measurement name: Type of measurement (string)

– measurement unit: Unit of the measurement (string)
– measurements: Array with ONE measurement value (array of 1 object)
∗ value: The corresponding value of the measurement (float)
∗ timestamp: Date of the measurement (ISO format string)

A.1.1 Example of valid message

1 {
2 "asset_id": "/ameland/house03/device_002",
3 "_id": "device_002",
4 "measurements_message":[{
5 "measurement_name": "activePower",
6 "measurement_unit": "kW",
7 "measurements": [{
8 "value": 87.643,
9 "timestamp": "2019-01-01T12:00:00.000"
10 }]
11 },{
12 "measurement_name": "reactivePower",
13 "measurement_unit": "kW",
14 "measurements": [{
15 "value": 22.413,
16 "timestamp": "2019-01-01T12:00:00.000"
17 }]
18 }]
19 }

Page 20
 API & Rabbit MQ Usage

A.2 Status data model

• asset id: with /{island}/{locationId}/{assetId} (string, assetId being the same as id)

• id: This field will be automatically filled with the corresponding routing key of the
mqtt topic (string, not required)

• status message: Array with all the different status, only 1 value per measurement al-
lowed at a time (array of objects)

– measurement name: Type of measurement (string, should be ”measurement”)

– status: Array with ONE status value (array of 1 object)
∗ value: The corresponding value of the status (string)
∗ timestamp: Date of the measurement (ISO format string)

A.2.1 Example of valid message

1 {
2 "asset_id": "/ameland/house03/device_002",
3 "_id": "device_002",
4 "status_message":[
5 {
6 "measurement_name": "status",
7 "status": [
8 {
9 "value": "ok",
10 "timestamp": "2019-01-01T12:00:00.000"
11 }
12 ]
13 }
14 ]
15 }

Page 21
 API & Rabbit MQ Usage

A.3 cluster data model

• asset id: with /{island}/{locationId}/{assetId} (string, assetId being the same as id)

• id: This field will be automatically filled with the corresponding routing key of the
mqtt topic (string, not required)

• cluster message: Array with all the different measurements, only 1 value per measure-
ment allowed at a time (array of objects)

– measurement name: Type of measurement (string, should be cluster)

– clusters: Array with ONE element (array of 1 object)
∗ category: Category of the cluster (string)
∗ timestamp: Date of the measurement (ISO format string)

A.3.1 Example of valid message

1 {
2 "asset_id": "/ameland/house03/device_002",
3 "_id": "device_002",
4 "cluster_message":[
5 {
6 "measurement_name": "cluster",
7 "clusters": [
8 {
9 "category": "3",
10 "timestamp": "2019-01-01T12:00:00.000"
11 }
12 ]
13 }
14 ]
15 }

Page 22