Streaming Analytics using Node-RED

Introduction
In this first project the students will learn how to use a platform to create a simple
analysis application for an IoT use case. The platform to be used is Node-RED, which
provides a series of services to develop in a simple, effective and fast way a data
analytics application.

Specifically, students will create a streaming analytics service provided by the

application to gain insights in a series of data collected to make decisions.

Through this internship, students will learn how to perform data transmissions and
develop services that will lay the foundations for a professional activity and that can be
reproduced free of charge for their own purposes.

Using Public REST APIs

When a student learns how to use endpoints of a public REST API, they gain several valuable
skills and knowledge:

1. API integration: By learning how to use endpoints of a public REST API, the student gains
experience in working with APIs and integrating different software systems to achieve a
common goal. This is an important skill for developers and IT professionals, as it allows
them to create and integrate systems that work together seamlessly.

2. Data capture: By using endpoints of a public REST API, the student gains experience in
capturing data from external sources and processing it for further use. This skill is
particularly useful in data analysis and machine learning applications, where large
volumes of data are needed to train models and make predictions.

3. HTTP requests: By working with REST APIs and endpoints, the student gains experience
in making HTTP requests, including GET, POST, PUT, DELETE, and other methods. This
skill is important for web development and other areas where data is exchanged over
the internet.

4. Authentication and security: By using endpoints of a public REST API, the student gains
experience in working with authentication and security mechanisms, such as API keys,
OAuth, and other forms of access control. This skill is important for ensuring that data
is accessed only by authorized users and that sensitive data is protected from
unauthorized access.

5. Troubleshooting and problem-solving: Working with APIs and external data sources can
be complex and prone to errors. By learning how to use endpoints of a public REST API,
the student gains experience in troubleshooting errors, identifying and resolving issues,
and thinking critically about how to improve the overall system.

In summary, by learning how to use endpoints of a public REST API, a student gains valuable
skills and knowledge in API integration, data capture, HTTP requests, authentication and
security, and troubleshooting. These skills are highly sought after by employers and are essential
for anyone interested in a career in software development or IT.

Using MQTT with Streaming Data

IoT devices often use "fire-and-forget" solutions to minimize bandwidth and power
consumption. For example, a Raspberry Pi can be installed as a monitoring station in a
cornfield to provide data on air and soil temperature, moisture, hydration and pH levels.
In its simplest form, the farmer's data dashboard is just another client on the network.
Each of the sensors publishes data, and the dashboard, acting as just another client,
subscribes to topics from all the sensors.

The beauty of this system is quite obvious. No one has to deal with a massive server-
client infrastructure. The farmer can easily install clients on a cell phone, tablet, in-
vehicle display and laptop. Information is available everywhere and at all times, and all
this is achieved with little power consumption, a lightweight central intermediary and
minimal bandwidth consumption. This represents a very agile approach to data
transmission.

The MQTT technology allows for a simple data stream using an open-source protocol.

Node-Red: A visual tool for wiring everything

Node-RED is a programming tool created by two IBM Engineers, as a visual tool for wiring
together hardware devices, APIs, and online services without writing code.

The project has recently become popular as it is the quickest way to prototype web services
inside a business during a hackathon or an experiment. Developers who attach it to IoT devices
enjoy it very well.

Node-RED runs on local workstations, the cloud, and edge devices. It has become an ideal tool
for the Raspberry Pi and other low-cost hardware.

The Node-RED runtime is lightweight and built on top of Node.js. It takes full advantage of
Node.js’ event-driven, non-blocking I/O model.

With all the Node-RED community resources and vast NPM ecosystem, you can create IoT flows
in a user-friendly manner.

Node-RED allows us to build services such as streaming data analytics. Using this technology the
student will be able to create an ecosystem where data is analyzed in real time and can be used
for decision making or generation of subsequent models.

Amazon database insertion

The insertion of data into an AWS database offers several benefits, including:
 1. Scalability: AWS databases can scale up or down easily, depending on the amount of
data being inserted, without the need for additional hardware or infrastructure. This
allows for efficient data storage and retrieval, and ensures that the database can handle
large amounts of data over time.

2. Reliability: AWS databases are designed to be highly available and reliable, with built-in
redundancy and failover mechanisms. This ensures that the data is always accessible
and that there is minimal risk of data loss or downtime.

3. Security: AWS databases provide robust security features, such as encryption at rest and
in transit, to protect the data from unauthorized access or breaches. This helps ensure
that the data remains secure and confidential.

4. Cost-effectiveness: AWS databases offer a pay-as-you-go model, which means that

users only pay for the storage and processing resources they use. This makes it a cost-
effective option for businesses or individuals who need to store and analyze large
amounts of data.

5. Integration with other AWS services: AWS databases can be easily integrated with other
AWS services, such as AWS Lambda, Amazon Kinesis, and Amazon S3, to provide a
complete and efficient data storage and analysis solution.

In summary, the insertion of data into an AWS database offers benefits in terms of scalability,
reliability, security, cost-effectiveness, and integration with other AWS services. These benefits
make it an ideal choice for businesses or individuals who need to store and analyze large
amounts of data in a secure and efficient manner.

When a student learns how to insert data into an AWS database using an endpoint, they gain
several valuable skills and knowledge:

1. API integration: By learning how to use an endpoint to insert data into an AWS database,
the student gains experience in working with APIs and integrating different software
systems to achieve a common goal. This is an important skill for developers and IT
professionals, as it allows them to create and integrate systems that work together
seamlessly.
2. Troubleshooting and problem-solving: Working with APIs and cloud services can be
complex and prone to errors. By learning how to insert data into an AWS database using
an endpoint, the student gains experience in troubleshooting errors, identifying and
resolving issues, and thinking critically about how to improve the overall system.

In summary, by learning how to insert data into an AWS database using an endpoint, a student
gains valuable skills and knowledge in cloud computing, API integration, database management,
REST API design, and troubleshooting. These skills are highly sought after by employers and are
essential for anyone interested in a career in software development or IT.

Objectives
1. Develop the data flow design of the solution, identifying the different sensors and
devices that will be used and how the information will be captured and processed.
2. Capture real-time data from a public REST API.
3. Create a data stream using MQTT protocol to send the captured data to a centralized
server. Implementation should use HiveMQ public broker: http://www.mqtt-
dashboard.com.
4. Implement data analysis techniques to extract useful information from the captured
data. This includes developing basic analytics such as averages and deviations, as well
as implementing prediction or classification algorithms.
5. Integrate the solution with AWS to store the captured data. This will include
implementing data insertion into an AWS database.

This practice focuses on the design and implementation of an IoT solution for real-time data
monitoring and analysis. The solution will allow the capture of information from different
sensors and devices, and the performance of basic and advanced analysis to extract useful
information from the captured data. Additionally, the AWS service will be used to store and
visualize the captured information.

To achieve these objectives, the practice is divided into five main steps:

1. Data flow design: the different sensors and devices that will be used to capture
information will be identified, as well as how the information will be processed and
stored. Scalability of the solution should be considered.
2. Capture real-time data using a public REST API: students will use a public REST API to
capture real-time data from a selected source.
3. Data stream creation: MQTT protocol will be used to create a data stream and send the
captured data to a centralized server. This step will enable the transmission of data
efficiently and in real-time.
4. Implementation of data analysis techniques: data analysis techniques will be used to
extract useful information from the captured data. This includes implementing
prediction or classification algorithms to obtain more advanced information.
5. Integration with AWS: the AWS service will be used to store the captured data. Data
insertion into an AWS database will be implemented.

With this practice, students will be able to develop skills in the design and implementation of
IoT solutions, as well as in the use of public REST APIs, data analysis techniques, and cloud
storage and visualization services.

Project practical sessions

The realization of the project includes three practical sessions in the classroom with the
guidance of the teacher and the possibility of discussing doubts and possible solutions with
classmates.