Datalogging with MQTT, Node-

RED, InfluxDB, and Grafana

Note: This page is still a draft! The instructions are still
incomplete.

Resources
 Raspberry Pi Imager
 Pi My Life guide to installing InfluxDB
 Official guide to installing Node-RED on Raspberry Pi
 Official guide to install Grafana on Raspberry Pi

The plan
Whatever data source you want to record, there are a few
common building blocks that you will need. Each of these
building blocks has multiple alternatives. There are also
totally different architectures that do things in a different
way.

However, this particular combination of software is ideal for

home automation, because these software building blocks
can also be used for other purposes within your overall
system.
You can install the various software elements on different
computers, or in virtual machines, or using Docker
containers, or on a NAS, or in many other ways.

To keep it simple and because many people already have

one spare, we’re going to use a Raspberry Pi.

Prepare Raspberry Pi
Begin by installing Raspberry Pi OS (formerly known as
“Raspbian”) on your Raspberry Pi. The easiest way to do it
these days is to use the Raspberry Pi Imager, which runs on
Windows, MacOS, or Linux and can be downloaded
from www.raspberrypi.org/downloads/.

Use the Imager to download and install Raspberry Pi OS

onto a micro SD card, insert it into your Pi, and start it up.

TODO: logging in first time.

Make sure your Pi has the latest packages by applying all

updates:
sudo apt update
sudo apt upgradeCopy

You now have a Raspberry Pi with a basic Raspbian OS

installation, ready for customisation.

Step 0: Get the IP address

We’ll need to know the IP address of the Raspberry Pi later.
If you’ve connected using the hostname or you’re using a
screen and keyboard, you can get the

Command: ipconfig

Step 1: Install the Mosquitto

MQTT broker and clients
There are many different MQTT brokers available, but
Mosquitto is the most popular and it has served me well for
many years. Install both the “mosquitto” package and the
“mosquitto-clients” package, so that you have both the
broker and some handy command line clients that you can
use for testing:

sudo apt install mosquitto mosquitto-clients

Command to uninstall Mosquitto

sudo apt autoremove mosquitto mosquitto-clients

The Mosquitto broker will be set up with a default

configuration and will work fine out of the box, but we’re
going to change the configuration to make it more secure.
You can leave your MQTT broker unprotected, but it’s a good
idea to set a username and password on it.

First, create a text file that contains the username and

password with a colon to separate them. You can do this on
the command line with a single command:

echo "mqtt_username:mqtt_password" > pwfile

This will create a text file called “pwfile” with the details in
them.

Use the passwd utility provided as part of the Mosquitto

package to encrypt the file:

mosquitto_passwd -U pwfile

Display the contents of the file to verify that it has now been
encrypted:

cat pwfile

You will see your username in plaintext, followed by a string

of gibberish which is the encrypted form of the password. If
the password is still plaintext, it means the encryption
command above didn’t work.

Move the encrypted file into the Mosquitto configuration

directory:

sudo mv pwfile /etc/mosquitto/

Mosquitto needs to be told where the new file is located.

Use an editor such as Nano to open the Mosquitto
configuration file:

sudo nano /etc/mosquitto/mosquitto.conf

Near the bottom of the file, just above the “include_dir” line,
add these lines:

allow_anonymous false

password_file /etc/mosquitto/pwfile

The first line tells Mosquitto to reject all anonymous

connections, and require a password.

The second line tells Mosquitto where to find the list of

allowed usernames and passwords.

To save your changes and exit, type CTRL+X, then Y,

then ENTER.

Restart Mosquitto so that it uses the new configuration:

sudo /etc/init.d/mosquitto restart

From now on, all connections to your MQTT broker will need
to supply the username and password that you configured.

Step 2: Install the InfluxDB

time-series database
To install InfluxDB we can use its official repository,
because the developers have provided a packages
specifically for different operating systems on the Raspberry
Pi.

Start by fetching the official repository key and adding it to

the local keyring:

 wget -q https://repos.influxdata.com/influxdata-archive_compat.key
 echo
393e8779c89ac8d958f81f942f9ad7fb82a25e133faddaf92e15b16e6ac9ce4c
influxdata-archive_compat.key' | sha256sum -c && cat influxdata-
 archive_compat.key | gpg --dearmor | sudo tee
/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg > /dev/null
 echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg]
https://repos.influxdata.com/debian stable main' | sudo tee
/etc/apt/sources.list.d/influxdata.list
 wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add

Now you can add the repository. There are a few different
versions available, so you need to copy and paste the
command that matches your operating system.

To find out what version you’re running, you can type the
following command:

lsb_release -a

This is will report the operating system type and version,

usually with a codename like “Stretch” or “Buster”. Copy
and paste the command that matches your system.

For Raspbian Bullseye:

 echo "deb https://repos.influxdata.com/debian bullseye stable" | sudo tee

/etc/apt/sources.list.d/influxdb.list

For Raspbian Stretch:

 echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee

/etc/apt/sources.list.d/influxdb.list

For Raspbian Buster:

 echo "deb https://repos.influxdata.com/debian buster stable" | sudo tee

/etc/apt/sources.list.d/influxdb.list

For Ubuntu 20.04LTS:

 echo "deb https://repos.influxdata.com/ubuntu focal stable" | sudo tee

/etc/apt/sources.list.d/influxdb.list
For Ubuntu 18.04LTS:

 echo "deb https://repos.influxdata.com/ubuntu bionic stable" | sudo tee

/etc/apt/sources.list.d/influxdb.list

Now that the repository has been added, we need to update

the list of packages that are available:

 sudo apt update

Install the database package:

 sudo apt install influxdb

Tell the systemctl service manager to enable InfluxDB at

startup:

 sudo systemctl unmask influxdb

 sudo systemctl enable influxdb

Start InfluxDB manually this time. In future, it will be started

automatically whenever your Raspberry Pi boots up:

 sudo systemctl start influxdb

Let’s set up access control for InfluxDB before we do

anything else. The default installation of InfluxDB leaves the
system wide open, so we’ll start by creating an admin user
and setting a password.

Connect to InfluxDB by running the client. We don’t need to

use a username or password this time, because nothing has
been set yet:

 influx

Create a user called “admin”, and put in the password you

want to use for it:
  CREATE USER admin WITH PASSWORD 'adminpassword' WITH ALL
PRIVILEGES

Now you can exit out of InfluxDB. Simply type “exit” and
press ENTER.

 exit

The InfluxDB configuration needs to be edited so that it will

use authentication. Otherwise the admin user that we just
created will be ignored. Use a text editor to open the
InfluxDB config file:

sudo nano /etc/influxdb/influxdb.conf

Press CTRL+W to search for the section called [HTTP].

auth-enabled = true

pprof-enabled = true

pprof-auth-enabled = true

ping-auth-enabled = true

To save your changes and exit, type CTRL+X, then Y,

then ENTER.

The config change won’t be applied until InfluxDB has been

restarted, so restart it manually:

sudo systemctl restart influxdb

From now on, any time you want to connect to the InfluxDB
command line you will need to supply the username and
password.

We need to do that now, so substitute the password you set:

influx -username admin -password <adminpassword>

If the previous changes worked, you should now be

connected to InfluxDB again and authenticated as the admin
user that you just created.

Next we need to tell InfluxDB to create a database where

we can store sensor data. In this example I’ve simply called
the database “sensors”:

CREATE DATABASE sensors

That was easy! Because of the way InfluxDB works, there’s

no need to create a schema with tables and columns like
you would be a relational database such as MariaDB,
MySQL, Postgres, or SQL Server.

All you need to do is create an empty database, and it will

be automatically populated when you start sending data to
it.

Leave the InfluxDB client by typing exit, as before:

exit

Step 3: Install Node-RED

There are several different ways to install Node-RED, and it’s
readily available in the Raspbery OS packaging system.
However, the Node-RED team recommend that you do NOT
use the packaged version.

Instead, they provide a handy script that installs the latest

version of Node-RED from the official release, and helps you
keep it updated. The installation script goes far beyond that,
however. It also:
  makes sure there’s no existing installation of Node-RED
and removes any that it finds in order to prevent
conflicts
 installs the latest Node.js
 gives you the option of installing a set of useful nodes
specifically designed to run on a Pi
 sets up Node-RED to run as a service and installs tools
to help you manage it

Before running the Node-RED installation script, install the

tools needed by NPM to build binary modules:

sudo apt install build-essential git

Now you can run the official Node-RED installation script:

bash <(curl -sL

https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-
nodejs-and-nodered)

The script will ask you a couple of questions about whether

you are sure you want to proceed, and whether to install Pi-
specific nodes. Say “y” (yes) to both questions.

Node-RED can be extended by installing modules to give it

extra features. We need to do that now so that it can
connect to your InfluxDB database. Use NPM to install the
InfluxDB nodes:

npm install node-red-contrib-influxdb

At this point Node-RED is be installed, but just like with

InfluxDB you need to configure it to be automatically started
on boot:

sudo systemctl enable nodered.service

You can start the service manually this time, but in future it
will happen automatically when your Pi starts up:
sudo systemctl start nodered.service

Step 4: Install Grafana

Just like with InfluxDB, we can install Grafana by adding the
official repository and installing the package. Start by
fetching the public key for the repository and adding it to the
local keyring:

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Now add the repository itself:

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a

/etc/apt/sources.list.d/grafana.list

Update the package list (again!) and install the Grafana

package:

sudo apt update

sudo apt install grafana

Just like the other packages we’ve installed, we need to

enable the service so that it will start automatically:

sudo systemctl enable grafana-server

That takes care of starting Grafana at boot. For now, let’s

start it manually:

sudo systemctl start grafana-server

Step 5: Configure your

sensors to send data to MQTT
For each sensor that you want to log and report, you will
need to go through a series of steps to ensure that the data
is received by your Raspberry Pi, processed by Node-RED,
stored in InfluxDB, and then charted using Grafana. This is a
process that you will repeat multiple times as your build up
your home automation system.

Let’s break it down into a series of simple steps.

Begin watching MQTT for messages. There won’t be any

messages yet, but this is a handy technique for discovering
new devices when they begin reporting to MQTT. On your
Raspberry Pi, open a terminal and run the Mosquitto client
command that we ran earlier:

mosquitto_sub -u mqtt_username -P mqtt_password -v -t "#"

This launches the Mosquitto client in “verbose” mode (the “-

v” flag) and subscribes to all topics using the wildcard
character. What this means is the client will display every
message that is published by any client to any topic, and
report not just the message but also the topic that it
appeared in.

On a busy MQTT broker this is like drinking from a fire-hose!

However, we don’t have any sensors using our MQTT broker

yet so you shouldn’t see anything happen: the terminal will
just sit there waiting to display messages. Leave the
terminal open so that you can see any messages that are
published.

Configure sensor to report using MQTT. The exact process

for this will depend on your device. Typically you will need
to configure three things in your sensor:

1. The IP address (or hostname) of your MQTT broker. In

this case, it’s the IP address of the Raspberry Pi.
2. The username and password for MQTT. We configured
this way back at the start of the process. If you don’t
 have a username and password on your broker, you can
skip this.
3. The MQTT topics for reporting. In many devices this
should default to something sensible, and you may not
need to change it. For our Air Quality Sensor project,
the topic will be generated automatically based on the
chip ID of the ESP chip.

If you are using the example Arduino sketch for the Air
Quality Sensor, open it in the Arduino IDE and go to the tab
called “config.h”. Edit the broker IP address and the MQTT
username / password to match your own settings:

Compile the sketch and flash it to the Air Quality Sensor,

just like in the previous episodes.

After the sensor starts up with the new settings, you should
see some action in the Mosquitto client! You’ll see a startup
message from the sensor, and then it will periodically begin
publishing its readings.
Step 6: Receiving and storing
single-value sensor readings
aoeu

Step 7: Receiving and storing

JSON-formatted sensor
readings
aoeu

Step 8: Create a dashboard

and widgets in Grafana
You can access the Grafana user interface using a web
browser, by connecting to port 3000 on your Raspberry Pi.
The URL will look something like:

http://192.168.1.248:3000

Substitute the IP address or hostname for your Raspberry Pi.

To log in, use the username and password “admin” and

“admin”. Grafana will ask you to change the password on
first login, so set it to something secure.
IP Address of Raspberry PI
1. Using the Terminal:

If your Raspberry Pi is connected to a monitor, keyboard,

and mouse:

1. Open a terminal.

2. Enter the following command:

hostname -I (Worked in our case)

3. The displayed result is the IP address of your Raspberry
Pi on the local network. If you have both IPv4 and IPv6
addresses configured, both may be displayed.

2. Using the Raspberry Pi GUI:

If you are using the Raspberry Pi Desktop (GUI):

1. Click on the network icon on the top right of the screen.

2. Hover over the network to which you're connected.

Your IP address will be displayed in the tooltip.

3. Using your Router's Admin Page:

1. Access your router's web interface (often at

192.168.1.1 or 192.168.0.1, but this can vary).

2. Look for a section called "Connected Devices", "Device

List", or something similar.

3. Find your Raspberry Pi in the list (often listed by its

MAC address or hostname) to see its IP address.

4. Using a Network Scanning Tool:

You can use tools like nmap on another computer:

1. First, install nmap (e.g., sudo apt install nmap on

Debian/Ubuntu).

2. Scan your network. If your network is 192.168.1.x,

you'd use:
nmap -sn 192.168.1.0/24
3. Look for your Raspberry Pi in the results. It might
appear with its hostname.

5. Using the ifconfig command (deprecated method):

While ifconfig is older and deprecated, it might still be

available on some systems:

1. Open a terminal.

2. Enter the command:

ifconfig
3. Look under the eth0 entry (for wired connection) or
wlan0 (for wireless). The inet address is your IP.

6. Using the ip command:

This is the modern replacement for ifconfig:

1. Open a terminal.

2. For wired connections:

ip addr show eth0

For wireless connections:

ip addr show wlan0

3. Look for the inet entry which shows your IP.

Remember, if your Raspberry Pi is connected to a network

via both Ethernet and WiFi, it might have two different IP
addresses. Choose the appropriate one based on how you
want to connect.
curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -