BRINGING INTELLIGENCE

TO THE EDGE

Designing Industrial IoT Applications with TinyML

 Foreword
MO DOGAR | VICE PRESIDENT RENESAS

Contents ChatGPT has shown us the We all know the potential of the Internet of Things (IoT)
tremendous potential of large and the benefits that can be derived from a connected
language models when trained network. While IoT enables devices to be connected,
AI gives them a brain. Imagine the potential that can be
properly, and we are just
3 Foreword getting started. What is equally
unlocked by enabling these IoT devices to think!
A staggering 73.1 zettabytes (ZB) of data is expected to
fascinating is the other extreme,
4 AI in IoT Endpoints where artificial intelligence
be generated by IoT devices in 2025 according to IDC
research. We are entering a disruptive era of business
(AI) and machine learning (ML) transformation built around the convergence of
7 Revolution of Endpoint AI in Embedded Vision are built to run within highly AI and IoT—or AIoT, the Artificial Intelligence of
Applications constrained environments, a Things. AI is also fundamentally transforming the
field now known as TinyML. design mindset and influencing system-level
11 Vision AI Gateway Solution approaches.
We hope to get you started in the right
12 Implementing Machine Learning at the Edge direction with this collection of articles
C ONNEC T WITH MO USE R and product highlights from Renesas.
AI and ML have tremendous potential
14 Four Metrics You Must Consider When Developing to solve problems in innovative ways
TinyML Systems and provide value for end users. We
Terms Of Use: ©2023 Mouser Electronics, Inc., a TTI Berkshire-
Hathaway company. Mouser and Mouser Electronics are hope you share our excitement
16 Data Science and AI-Driven Real-Time Analytics registered trademarks of Mouser Electronics, Inc., Renesas
Electronics Corporation, logo, product names, service marks,
and make us part of your
are trademarks, registered trademarks of Renesas Electronics innovation journey. ■
Corporation. All other trademarks are the property of their
21 Endpoint MCU Implementation of Voice User Interface respective owners. Names, products, specifications, links,
resources, and similar were verified at the time of publication.
Article content, reference designs, datasheets, conceptual
illustrations, and other images provided herein are for
25 Reality AI: A Game Changer in Sensor Data Analysis informational purposes only.

2 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 3

 Foreword
MO DOGAR | VICE PRESIDENT RENESAS

Contents ChatGPT has shown us the We all know the potential of the Internet of Things (IoT)
tremendous potential of large and the benefits that can be derived from a connected
language models when trained network. While IoT enables devices to be connected,
AI gives them a brain. Imagine the potential that can be
properly, and we are just
3 Foreword getting started. What is equally
unlocked by enabling these IoT devices to think!
A staggering 73.1 zettabytes (ZB) of data is expected to
fascinating is the other extreme,
4 AI in IoT Endpoints where artificial intelligence
be generated by IoT devices in 2025 according to IDC
research. We are entering a disruptive era of business
(AI) and machine learning (ML) transformation built around the convergence of
7 Revolution of Endpoint AI in Embedded Vision are built to run within highly AI and IoT—or AIoT, the Artificial Intelligence of
Applications constrained environments, a Things. AI is also fundamentally transforming the
field now known as TinyML. design mindset and influencing system-level
11 Vision AI Gateway Solution approaches.
We hope to get you started in the right
12 Implementing Machine Learning at the Edge direction with this collection of articles
C ONNEC T WITH MO USE R and product highlights from Renesas.
AI and ML have tremendous potential
14 Four Metrics You Must Consider When Developing to solve problems in innovative ways
TinyML Systems and provide value for end users. We
Terms Of Use: ©2023 Mouser Electronics, Inc., a TTI Berkshire-
Hathaway company. Mouser and Mouser Electronics are hope you share our excitement
16 Data Science and AI-Driven Real-Time Analytics registered trademarks of Mouser Electronics, Inc., Renesas
Electronics Corporation, logo, product names, service marks,
and make us part of your
are trademarks, registered trademarks of Renesas Electronics innovation journey. ■
Corporation. All other trademarks are the property of their
21 Endpoint MCU Implementation of Voice User Interface respective owners. Names, products, specifications, links,
resources, and similar were verified at the time of publication.
Article content, reference designs, datasheets, conceptual
illustrations, and other images provided herein are for
25 Reality AI: A Game Changer in Sensor Data Analysis informational purposes only.

2 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 3

AI in IoT With the introduction of AI, IoT
devices can become more intelligent Making Smart Devices Smarter— “ AI models can be slower
to run and consume more

Endpoints
and less reliant on external systems— but Not without Trade-Offs
but not without trade-offs in AI is revolutionizing the way that smart devices operate, power depending on the

”
performance and cost. Understanding enabling them to make decisions without relying on model and size.
external systems for input. This is beneficial in many
how to make that decision is key. ways—the data can stay on the device, which is safer
and more private. The devices are also capable of However, conducting research on prevailing methods in the
BECKS SIMPSON | FOR MOUSER ELECTRONICS making decisions based on the data without the need for field and applications that continue to elude satisfactory
immediate connectivity, which is helpful for endpoints solutions from ML could reveal potential issues before
that may have only sporadic access to a connection. starting the process of adding AI to IoT endpoints.
However, this comes with trade-offs: AI models can be
At this point, design engineers should also determine
slower to run and consume more power depending on the
if the AI capabilities can be achieved without ML and if
model and size. This can be a major limiting factor for AI
the use case is better suited for an AI-enabled device or
on edge devices because for machine learning (ML) to be
a simpler alternative solution. Typically, the first place
effective in those applications, it must be small and fast
to start in this process is to review the literature and
enough to run on an edge device. Therefore, although
identify which methods have been used successfully
AI-enabled devices can be more powerful, they come
before. Then, using that information as a guide,
with trade-offs that engineers must consider.
build a baseline proof of concept using simple, rule-
based approaches or less sophisticated algorithmic
Balancing Improved Intelligence implementations. This baseline can be used to assess
with Impact on Performance how well a non-ML solution might work; if it does not
Design engineers should consider the ROI when deciding meet success criteria for performance, then progressing
whether to add AI to an IoT endpoint. Before embarking to an ML-based solution may be in order.
on the decision to embed ML in these endpoints, the
expected value should be defined and quantified as Assessing Feasibility among
much as possible. Doing so helps not only to understand All the Constraints
how the costs stack up compared to expected gains
Assuming that the value expected by introducing ML
but also to inform the criteria for success of the project
outweighs the costs and potential performance impacts,
(i.e., accuracy of the model or some other metric). This
the next step in the decision-making process is to assess

A
will make it easier to determine if the value of the AI
whether the ML lifecycle is possible within the current
rtificial intelligence (AI) is transforming the outweighs the impact on the device’s performance.
technical or physical constraints of the devices. IoT
ever-expanding Internet of Things (IoT) quickly In most cases, ML algorithms require more computing endpoints that are not normally Wi-Fi enabled and that
as manufacturers rush to embed the technology power and more energy, which can reduce the device’s produce vast quantities of unlabeled data stored for only
into their devices, from smart watches and home battery life or noticeably degrade its performance in short periods of time at the edge will score much lower
monitors to self-driving cars and manufacturing robots. terms of latency. If the device is already running at on the feasibility scale than those with cloud access,
Using AI in IoT endpoints enables powerful devices that can maximum capacity, the ML algorithms may not be able where data can be pooled and annotated more easily
make decisions without the need for outside assistance, to run at a satisfactory level. Thus, design engineers and where compute power is more readily available.
but this comes with certain trade-offs. To determine should examine the use case of the device carefully to In particular, since AI generally requires some training
when AI should be introduced to IoT endpoints, design determine if AI would be beneficial and not too much that involves the model having access to enough data
engineers should carefully assess the return on investment of an additional strain on the device’s resources. For to learn from, how this training will be done is an aspect
(ROI), evaluate the feasibility of the project within existing devices that have constant access to a power source, the of feasibility. For example, putting ML in an endpoint,
constraints, and consider data requirements and what data additional cost of computation at the source becomes though not impossible, is significantly less feasible if the
engineering may be needed. Embracing the right strategy more of an issue than battery life. data are distributed and cannot be pooled for training
when considering the costs and benefits of introducing In addition to evaluating ROI versus device performance, a model owing to privacy, storage, or connectivity
AI to an IoT device can help to ensure the success of the design engineers should examine whether the task can be requirements and constraints. In these cases, training
project and maximize the impact of smart devices in our achieved with ML. Given the hype around AI, applying it to could occur in a federated fashion, but the cost and
increasingly digitized world. all use cases without thorough investigation is tempting. complexity of doing so could outweigh the benefits.

4 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 5

AI in IoT With the introduction of AI, IoT
devices can become more intelligent Making Smart Devices Smarter— “ AI models can be slower
to run and consume more

Endpoints
and less reliant on external systems— but Not without Trade-Offs
but not without trade-offs in AI is revolutionizing the way that smart devices operate, power depending on the

”
performance and cost. Understanding enabling them to make decisions without relying on model and size.
external systems for input. This is beneficial in many
how to make that decision is key. ways—the data can stay on the device, which is safer
and more private. The devices are also capable of However, conducting research on prevailing methods in the
BECKS SIMPSON | FOR MOUSER ELECTRONICS making decisions based on the data without the need for field and applications that continue to elude satisfactory
immediate connectivity, which is helpful for endpoints solutions from ML could reveal potential issues before
that may have only sporadic access to a connection. starting the process of adding AI to IoT endpoints.
However, this comes with trade-offs: AI models can be
At this point, design engineers should also determine
slower to run and consume more power depending on the
if the AI capabilities can be achieved without ML and if
model and size. This can be a major limiting factor for AI
the use case is better suited for an AI-enabled device or
on edge devices because for machine learning (ML) to be
a simpler alternative solution. Typically, the first place
effective in those applications, it must be small and fast
to start in this process is to review the literature and
enough to run on an edge device. Therefore, although
identify which methods have been used successfully
AI-enabled devices can be more powerful, they come
before. Then, using that information as a guide,
with trade-offs that engineers must consider.
build a baseline proof of concept using simple, rule-
based approaches or less sophisticated algorithmic
Balancing Improved Intelligence implementations. This baseline can be used to assess
with Impact on Performance how well a non-ML solution might work; if it does not
Design engineers should consider the ROI when deciding meet success criteria for performance, then progressing
whether to add AI to an IoT endpoint. Before embarking to an ML-based solution may be in order.
on the decision to embed ML in these endpoints, the
expected value should be defined and quantified as Assessing Feasibility among
much as possible. Doing so helps not only to understand All the Constraints
how the costs stack up compared to expected gains
Assuming that the value expected by introducing ML
but also to inform the criteria for success of the project
outweighs the costs and potential performance impacts,
(i.e., accuracy of the model or some other metric). This
the next step in the decision-making process is to assess

A
will make it easier to determine if the value of the AI
whether the ML lifecycle is possible within the current
rtificial intelligence (AI) is transforming the outweighs the impact on the device’s performance.
technical or physical constraints of the devices. IoT
ever-expanding Internet of Things (IoT) quickly In most cases, ML algorithms require more computing endpoints that are not normally Wi-Fi enabled and that
as manufacturers rush to embed the technology power and more energy, which can reduce the device’s produce vast quantities of unlabeled data stored for only
into their devices, from smart watches and home battery life or noticeably degrade its performance in short periods of time at the edge will score much lower
monitors to self-driving cars and manufacturing robots. terms of latency. If the device is already running at on the feasibility scale than those with cloud access,
Using AI in IoT endpoints enables powerful devices that can maximum capacity, the ML algorithms may not be able where data can be pooled and annotated more easily
make decisions without the need for outside assistance, to run at a satisfactory level. Thus, design engineers and where compute power is more readily available.
but this comes with certain trade-offs. To determine should examine the use case of the device carefully to In particular, since AI generally requires some training
when AI should be introduced to IoT endpoints, design determine if AI would be beneficial and not too much that involves the model having access to enough data
engineers should carefully assess the return on investment of an additional strain on the device’s resources. For to learn from, how this training will be done is an aspect
(ROI), evaluate the feasibility of the project within existing devices that have constant access to a power source, the of feasibility. For example, putting ML in an endpoint,
constraints, and consider data requirements and what data additional cost of computation at the source becomes though not impossible, is significantly less feasible if the
engineering may be needed. Embracing the right strategy more of an issue than battery life. data are distributed and cannot be pooled for training
when considering the costs and benefits of introducing In addition to evaluating ROI versus device performance, a model owing to privacy, storage, or connectivity
AI to an IoT device can help to ensure the success of the design engineers should examine whether the task can be requirements and constraints. In these cases, training
project and maximize the impact of smart devices in our achieved with ML. Given the hype around AI, applying it to could occur in a federated fashion, but the cost and
increasingly digitized world. all use cases without thorough investigation is tempting. complexity of doing so could outweigh the benefits.

4 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 5

 Revolution of Endpoint
AI in Embedded Vision
Applications
The impact of these
constraints often extends
beyond the initial data
annotation and model training
phases of the ML lifecycle. Model
performance monitoring and updates
are key, requiring retraining on new SUAD JUSUF | SENIOR MANAGER RENESAS ELECTRONICS

E
data if performance has declined as well as
connectivity to push new deployments to devices. ndpoint AI is a new frontier in the space
Engineers should consider monitoring in terms of of artificial intelligence (AI) that takes
the model and the data: Assessing feasibility includes the processing power of AI to the edge.
determining whether data drift between the training It is a revolutionary way of managing
data and new points being measured can be captured. information, accumulating relevant data, and
For example, monitoring is more difficult for endpoints Data engineering depends on the device configuration making decisions locally on a device. Endpoint
where data are stored only for a short period of time, and how the data are stored. To ensure the success and AI employs intelligent functionality at the edge
which may impact the final decision about whether to accuracy of an ML project, proper data engineering is of the network; in other words, it transforms
introduce ML to the process. essential and IoT applications are no exception. This the Internet of Things (IoT) devices that
includes understanding the data format, quality, and are used to compute data into smarter
Addressing the Age-Old Question of Data type, as well as how the data might be cleaned (e.g., tools embedded with AI features. This in
granting access for initial data exploration and deploying turn improves real-time decision-making
Regardless of the final application, when deciding processes that transform the data to be usable). Data
whether to embark on an ML project, one of the most capabilities and functionalities. The goal
engineering is typically performed by data engineers is to bring machine-learning-based,
common questions is whether enough data exist. This who require access to the data they need to work with;
goes hand in hand with labeling requirements—for intelligent decision-making physically
in cases where access is not permissible due to device closer to the source of the data. In
example, whether the task requires annotated data location, lack of connectivity, or privacy concerns, the
for a supervised training approach and whether the this context, embedded vision shifts
feasibility of project success may decrease. to the endpoint. Embedded vision
data are currently in the required state. Within the
IoT context, if the data cannot be pooled in the cloud, incorporates more than breaking
design engineers must determine whether enough data
Conclusion down images or videos into pixels—it
exist per device for the task at hand and if they need to is the means to understand pixels,
When deciding whether to introduce AI to IoT endpoints, In precise terms, computer vision is one of the AI fields
be labeled. Data quality is essential for successful ML make sense of what is inside, and support making a
design engineers must take a variety of considerations that enables machines to extract meaningful information
projects, so engineers should also evaluate the ability to smart decision based on specific events that transpire.
into account. From assessing the expected value of the from digital multimedia sources to take actions or make
manage noise and the heterogeneity of sensor data to an There have been massive endeavors at the research and
AI versus the cost and impact on device performance, recommendations based on the information that has
acceptable standard. industry levels to develop and improve AI technologies
to evaluating the feasibility of the ML lifecycle within been obtained. Computer vision is, to some extent, akin
and algorithms.
the existing constraints, and finally to understanding to the human sense of sight. However, the two differ on
the data requirements and what data engineering several grounds. Human sight has the exceptional ability
might be needed, engineers should carefully consider What Is Embedded Vision?
to understand many and varied things from what it sees.
each element of the decision. With the right approach Embedded computer vision is a technology that imparts On the other hand, computer vision recognizes only
and research, design engineers can make an informed machines with the sense of sight, which enables what it has been trained on and what it is designed to do
decision that incorporates the costs and benefits of them to explore the environment with the support of
Renesas MCU Family exactly, and that too with an error rate. AI in embedded
introducing AI to an IoT device, helping to ensure the machine-learning and deep-learning algorithms. There vision processes trains the machines to perform
success of the project. ■ are numerous applications across several industries supposed functions with the least processing time
whose functionality relies on computer vision, thus and has an upper edge over human sight in analyzing
Learn More
becoming an integral part of technological procedures. hundreds of thousands of images in a lesser timeframe.

6 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 7

 Revolution of Endpoint
AI in Embedded Vision
Applications
The impact of these
constraints often extends
beyond the initial data
annotation and model training
phases of the ML lifecycle. Model
performance monitoring and updates
are key, requiring retraining on new SUAD JUSUF | SENIOR MANAGER RENESAS ELECTRONICS

E
data if performance has declined as well as
connectivity to push new deployments to devices. ndpoint AI is a new frontier in the space
Engineers should consider monitoring in terms of of artificial intelligence (AI) that takes
the model and the data: Assessing feasibility includes the processing power of AI to the edge.
determining whether data drift between the training It is a revolutionary way of managing
data and new points being measured can be captured. information, accumulating relevant data, and
For example, monitoring is more difficult for endpoints Data engineering depends on the device configuration making decisions locally on a device. Endpoint
where data are stored only for a short period of time, and how the data are stored. To ensure the success and AI employs intelligent functionality at the edge
which may impact the final decision about whether to accuracy of an ML project, proper data engineering is of the network; in other words, it transforms
introduce ML to the process. essential and IoT applications are no exception. This the Internet of Things (IoT) devices that
includes understanding the data format, quality, and are used to compute data into smarter
Addressing the Age-Old Question of Data type, as well as how the data might be cleaned (e.g., tools embedded with AI features. This in
granting access for initial data exploration and deploying turn improves real-time decision-making
Regardless of the final application, when deciding processes that transform the data to be usable). Data
whether to embark on an ML project, one of the most capabilities and functionalities. The goal
engineering is typically performed by data engineers is to bring machine-learning-based,
common questions is whether enough data exist. This who require access to the data they need to work with;
goes hand in hand with labeling requirements—for intelligent decision-making physically
in cases where access is not permissible due to device closer to the source of the data. In
example, whether the task requires annotated data location, lack of connectivity, or privacy concerns, the
for a supervised training approach and whether the this context, embedded vision shifts
feasibility of project success may decrease. to the endpoint. Embedded vision
data are currently in the required state. Within the
IoT context, if the data cannot be pooled in the cloud, incorporates more than breaking
design engineers must determine whether enough data
Conclusion down images or videos into pixels—it
exist per device for the task at hand and if they need to is the means to understand pixels,
When deciding whether to introduce AI to IoT endpoints, In precise terms, computer vision is one of the AI fields
be labeled. Data quality is essential for successful ML make sense of what is inside, and support making a
design engineers must take a variety of considerations that enables machines to extract meaningful information
projects, so engineers should also evaluate the ability to smart decision based on specific events that transpire.
into account. From assessing the expected value of the from digital multimedia sources to take actions or make
manage noise and the heterogeneity of sensor data to an There have been massive endeavors at the research and
AI versus the cost and impact on device performance, recommendations based on the information that has
acceptable standard. industry levels to develop and improve AI technologies
to evaluating the feasibility of the ML lifecycle within been obtained. Computer vision is, to some extent, akin
and algorithms.
the existing constraints, and finally to understanding to the human sense of sight. However, the two differ on
the data requirements and what data engineering several grounds. Human sight has the exceptional ability
might be needed, engineers should carefully consider What Is Embedded Vision?
to understand many and varied things from what it sees.
each element of the decision. With the right approach Embedded computer vision is a technology that imparts On the other hand, computer vision recognizes only
and research, design engineers can make an informed machines with the sense of sight, which enables what it has been trained on and what it is designed to do
decision that incorporates the costs and benefits of them to explore the environment with the support of
Renesas MCU Family exactly, and that too with an error rate. AI in embedded
introducing AI to an IoT device, helping to ensure the machine-learning and deep-learning algorithms. There vision processes trains the machines to perform
success of the project. ■ are numerous applications across several industries supposed functions with the least processing time
whose functionality relies on computer vision, thus and has an upper edge over human sight in analyzing
Learn More
becoming an integral part of technological procedures. hundreds of thousands of images in a lesser timeframe.

6 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 7

 Embedded vision is one of the leading In applications where power consumption at the
technologies, with embedded AI used in endpoint is limited, microcontrollers (MCUs) or
smart endpoint applications in a wide range microprocessors (MPUs) need to be more efficient to
of consumer and industrial applications. take on the high volumes of multiply-accumulate (MAC)
There are a number of value-added use cases operations that are required for AI processing.
examples, such as counting/analyzing the
quality of products on a factory line, keeping a Deployment of AI Vision Applications
tally of people in a crowd, identifying objects,
and analyzing the contents of a specific area in There are unlimited use cases for the deployment of
the environment. AI in vision applications in the real world. Here are
some of the examples where Renesas Electronics can
While considering the processing of embedded provide comprehensive MCU- and MPU-based solutions,
vision applications at the endpoint, the including all the necessary software and tools to enable
performance of such an operation may face Figure 1: Computer vision is used to detect and classify objects on the road.
quick development. (Source: Renesas Electronics)
some challenges. The data flow from the vision-
sensing device to the cloud for the purposes of from the environment using cameras and sensors,
Smart Access Control
analyzing and processing could be very large and which then interpret and analyze the data to make the
Security access control systems are becoming more
may exceed the network available bandwidth. For most suitable response by using vision techniques such
valuable with the addition of voice and facial recognition
instance, a 1920×1080px camera operating with as pattern recognition, feature extraction, and object
features. Real-time recognition requires embedded
30 frames per second (FPS) may generate about tracking (Figure 1).
systems with very high computational capabilities and
190MB/s of data. In addition to privacy concerns,
on-chip hardware acceleration. To meet this challenge, In general, embedded vision can serve many
this substantial amount of data contributes to
Renesas provides a choice of MCU or MPU that offers purposes, and these functionalities can be used after
latency during the round trip of data from the edge
very high computational power that also integrates customization and the needed training on different
to the cloud, then back again to the endpoint. These
many key features that are critical to high-performance types of datasets from many areas. Functionalities
limitations could negatively impact the employment
facial and voice recognition systems, such as built-in include monitoring physical area, recognizing intrusion,
of embedded vision technologies in real-time
H.265 hardware decoding, 2D/3D graphic acceleration, detecting crowd density, and counting humans or objects
applications.
and error correction code (ECC) on internal and external or animals. They also include identifying people or finding
IoT security is also a concern in the adoption and memory to eliminate soft errors and allow for high-speed cars based on license plate numbers, detecting motion,
growth of embedded vision applications across any video processing. and analyzing human behavior in different cases.
segment. In general, all IoT devices must be secured.

“
A critical issue and concern in the use of smart vision Industrial Control
devices is the possible misuse of sensitive images and
Embedded vision has a huge impact as it enhances many
videos. Unauthorized access to smart cameras, for
applications, including product safety, automation, and Embedded vision is one of
example, is not only a breach of privacy, but it could also
pave a way for a more harmful outcome.
product sorting. AI techniques can perform multiple the leading technologies,
operations in the production process (such as packaging
and distribution), which can ensure quality and safety with embedded AI
Vision AI at the Endpoint during production in all stages. Safety is needed in areas
such as critical infrastructure, warehouses, production
used in smart endpoint
• E
 ndpoint AI can enable image processing to infer
a complex insight from a huge number of captured plants, and buildings that require a high level of human applications in a wide
images. resources.
range of consumer and

”
• A
 I uses machine-learning and deep-learning
capabilities within smart imaging devices to check a
Transportation industrial applications.
huge amount of previously well-known use cases. Computer vision presents a large scale of ways to
improve transportation services. For example, in self-
• F
 or optimum performance, embedded vision requires driving cars computer vision is used to detect and
AI algorithms to run on the endpoint devices and not classify objects on the road. It is also used to create
transmit data to the cloud. The data here is captured 3D maps and estimate movement around. By using
by the imaging recognition device, then processed and computer vision, self-driving cars gather information
analyzed in the same device.

8 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 9

 Embedded vision is one of the leading In applications where power consumption at the
technologies, with embedded AI used in endpoint is limited, microcontrollers (MCUs) or
smart endpoint applications in a wide range microprocessors (MPUs) need to be more efficient to
of consumer and industrial applications. take on the high volumes of multiply-accumulate (MAC)
There are a number of value-added use cases operations that are required for AI processing.
examples, such as counting/analyzing the
quality of products on a factory line, keeping a Deployment of AI Vision Applications
tally of people in a crowd, identifying objects,
and analyzing the contents of a specific area in There are unlimited use cases for the deployment of
the environment. AI in vision applications in the real world. Here are
some of the examples where Renesas Electronics can
While considering the processing of embedded provide comprehensive MCU- and MPU-based solutions,
vision applications at the endpoint, the including all the necessary software and tools to enable
performance of such an operation may face Figure 1: Computer vision is used to detect and classify objects on the road.
quick development. (Source: Renesas Electronics)
some challenges. The data flow from the vision-
sensing device to the cloud for the purposes of from the environment using cameras and sensors,
Smart Access Control
analyzing and processing could be very large and which then interpret and analyze the data to make the
Security access control systems are becoming more
may exceed the network available bandwidth. For most suitable response by using vision techniques such
valuable with the addition of voice and facial recognition
instance, a 1920×1080px camera operating with as pattern recognition, feature extraction, and object
features. Real-time recognition requires embedded
30 frames per second (FPS) may generate about tracking (Figure 1).
systems with very high computational capabilities and
190MB/s of data. In addition to privacy concerns,
on-chip hardware acceleration. To meet this challenge, In general, embedded vision can serve many
this substantial amount of data contributes to
Renesas provides a choice of MCU or MPU that offers purposes, and these functionalities can be used after
latency during the round trip of data from the edge
very high computational power that also integrates customization and the needed training on different
to the cloud, then back again to the endpoint. These
many key features that are critical to high-performance types of datasets from many areas. Functionalities
limitations could negatively impact the employment
facial and voice recognition systems, such as built-in include monitoring physical area, recognizing intrusion,
of embedded vision technologies in real-time
H.265 hardware decoding, 2D/3D graphic acceleration, detecting crowd density, and counting humans or objects
applications.
and error correction code (ECC) on internal and external or animals. They also include identifying people or finding
IoT security is also a concern in the adoption and memory to eliminate soft errors and allow for high-speed cars based on license plate numbers, detecting motion,
growth of embedded vision applications across any video processing. and analyzing human behavior in different cases.
segment. In general, all IoT devices must be secured.

“
A critical issue and concern in the use of smart vision Industrial Control
devices is the possible misuse of sensitive images and
Embedded vision has a huge impact as it enhances many
videos. Unauthorized access to smart cameras, for
applications, including product safety, automation, and Embedded vision is one of
example, is not only a breach of privacy, but it could also
pave a way for a more harmful outcome.
product sorting. AI techniques can perform multiple the leading technologies,
operations in the production process (such as packaging
and distribution), which can ensure quality and safety with embedded AI
Vision AI at the Endpoint during production in all stages. Safety is needed in areas
such as critical infrastructure, warehouses, production
used in smart endpoint
• E
 ndpoint AI can enable image processing to infer
a complex insight from a huge number of captured plants, and buildings that require a high level of human applications in a wide
images. resources.
range of consumer and

”
• A
 I uses machine-learning and deep-learning
capabilities within smart imaging devices to check a
Transportation industrial applications.
huge amount of previously well-known use cases. Computer vision presents a large scale of ways to
improve transportation services. For example, in self-
• F
 or optimum performance, embedded vision requires driving cars computer vision is used to detect and
AI algorithms to run on the endpoint devices and not classify objects on the road. It is also used to create
transmit data to the cloud. The data here is captured 3D maps and estimate movement around. By using
by the imaging recognition device, then processed and computer vision, self-driving cars gather information
analyzed in the same device.

8 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 9

 Offline training
and testing models
Applying the model
to a real-world sample
Display results and
analysis, in addition
Vision AI Gateway Solution
R
image to detect its case to predictions and
use a huge image
based on the tested suggested solutions.
type data set enesas Electronics’ Vision AI Gateway solution
model from phase 1. efficiently handles vision information from multiple
cameras and performs high-speed artificial
Figure 2: Three main steps for using deep learning in computer/machine vision. (Source: Renesas Electronics)
intelligence (AI) processing. It includes a Renesas
RZ/V2M AI-MPU with rich peripheral functions and an
optimized power supply system. The solution realizes high
Case Study: Agricultural Plant Disease Conclusion performance and low power consumption at the same time. It
Detection We are experiencing a revolution in high-performance delivers better value for gateway applications by integrating
Vision AI and deep learning may be employed to detect smart vision applications across a number of segments. edge AI capability.
various anomalies—for example, detecting plant disease. The trend is well supported by the growing computational
Deep learning algorithms—one of the AI techniques—are power of microcontrollers and microprocessors at the System Benefits​:
used widely for this purpose. According to research,
computer vision gives better, more accurate, faster, and
endpoints, opening up great opportunities for exciting
new vision applications. Renesas Vision AI solutions
• F
 ast vision AI processing via decoding of video streams
(H.264 or H.265) and handling multiple AI inferences with
AI EDGE GATEWAY DEMONSTRATIONS

lower-cost results compared to the costly, slow, and can help you to enhance overall system capability by minimum switching overhead
labor-intensive results of previous methods. The process delivering embedded AI technology with intelligent data
• S
 upport for high-speed vision gateway function with
that is used in this case study can be applied to any other processing at the endpoint. Renesas’s advanced image
Ethernet protocol: 100/1000Mbps and fast USB connection:
detection. There are three main steps for using deep processing solutions at the edge are provided through
USB 3.1 Gen 1 (5Gbps)
learning in computer/machine vision (Figure 2). a unique combination of low-power, multimodal, and
multi-feature AI inference capabilities. Take the chance • P
 eripheral extension capabilities with PCI express 2.0 (Gen
Step one is performed on normal computers in the lab,
now and start developing your vision AI application with 2/2 Lane) RZ/V2L High Precision
whereas step two is deployed on a microcontroller at the
endpoint, which can be on the farm. Results in step three
Renesas Electronics. ■ • B
 uilt-in power sequencing control function of the RZ/V2MA, Entry-Level AI MPUs
are displayed on the screen on the user side. Figure 3 making power supply design easier and improving reliability
shows the process in general. • I ntegration of various communication devices, Learn More
high-performance Wi-Fi module, compact and high-
performance Bluetooth® Low Energy module, and LTE Cat-
M1 cellular IoT module ■

Figure 3: General process of deep learning in computer/machine vision. (Source: Renesas Electronics) Figure 1: Vision AI Gateway solution. (Source: Renesas Electronics)

10 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 11

 Offline training
and testing models
Applying the model
to a real-world sample
Display results and
analysis, in addition
Vision AI Gateway Solution
R
image to detect its case to predictions and
use a huge image
based on the tested suggested solutions.
type data set enesas Electronics’ Vision AI Gateway solution
model from phase 1. efficiently handles vision information from multiple
cameras and performs high-speed artificial
Figure 2: Three main steps for using deep learning in computer/machine vision. (Source: Renesas Electronics)
intelligence (AI) processing. It includes a Renesas
RZ/V2M AI-MPU with rich peripheral functions and an
optimized power supply system. The solution realizes high
Case Study: Agricultural Plant Disease Conclusion performance and low power consumption at the same time. It
Detection We are experiencing a revolution in high-performance delivers better value for gateway applications by integrating
Vision AI and deep learning may be employed to detect smart vision applications across a number of segments. edge AI capability.
various anomalies—for example, detecting plant disease. The trend is well supported by the growing computational
Deep learning algorithms—one of the AI techniques—are power of microcontrollers and microprocessors at the System Benefits​:
used widely for this purpose. According to research,
computer vision gives better, more accurate, faster, and
endpoints, opening up great opportunities for exciting
new vision applications. Renesas Vision AI solutions
• F
 ast vision AI processing via decoding of video streams
(H.264 or H.265) and handling multiple AI inferences with
AI EDGE GATEWAY DEMONSTRATIONS

lower-cost results compared to the costly, slow, and can help you to enhance overall system capability by minimum switching overhead
labor-intensive results of previous methods. The process delivering embedded AI technology with intelligent data
• S
 upport for high-speed vision gateway function with
that is used in this case study can be applied to any other processing at the endpoint. Renesas’s advanced image
Ethernet protocol: 100/1000Mbps and fast USB connection:
detection. There are three main steps for using deep processing solutions at the edge are provided through
USB 3.1 Gen 1 (5Gbps)
learning in computer/machine vision (Figure 2). a unique combination of low-power, multimodal, and
multi-feature AI inference capabilities. Take the chance • P
 eripheral extension capabilities with PCI express 2.0 (Gen
Step one is performed on normal computers in the lab,
now and start developing your vision AI application with 2/2 Lane) RZ/V2L High Precision
whereas step two is deployed on a microcontroller at the
endpoint, which can be on the farm. Results in step three
Renesas Electronics. ■ • B
 uilt-in power sequencing control function of the RZ/V2MA, Entry-Level AI MPUs
are displayed on the screen on the user side. Figure 3 making power supply design easier and improving reliability
shows the process in general. • I ntegration of various communication devices, Learn More
high-performance Wi-Fi module, compact and high-
performance Bluetooth® Low Energy module, and LTE Cat-
M1 cellular IoT module ■

Figure 3: General process of deep learning in computer/machine vision. (Source: Renesas Electronics) Figure 1: Vision AI Gateway solution. (Source: Renesas Electronics)

10 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 11

 Implementing Machine
Learning at the Edge
Implementing Machine
HOW TO USE IT?
Automatically generates

Learning at the Edge

machine learning models,
explanatory visualizations, and
hardware design analytics.

Software tool
2 Complete frameworks ORGANIZE
DATA
1 3 EXPLORE
for developing for specific use cases, GATHER
intelligent, embedded including hardware,
AI applications for firmware, software and DATA
edge devices. ML reference designs.

⊲ Create potential
⊲ Assign training labels. feature sets and ML
⊲ Collect data from models for classification.
WHAT IS different sources. ⊲ Segment files into consistent
sample sizes for AI ⊲ Select the best-fit model
REALITY AI? ⊲ Upload or link exploration, training, based on accuracy
source files. or testing. and computational
complexity.

Automotive Sound
Recognition (SWS)
⊲ Train the new base tool
against a subset of data. 4
⊲ Balance is important. TRAIN

Industrial Anomaly Detection

⊲ Reality AI software ⊲ Test the trained tool
comes with integration ⊲ Optimize for sensitivity against a subset of data.
to Renesas e² studio. and tolerances.
⊲ Validate performance using
⊲ Compile for a wide range ⊲ Retrain as needed. training samples and
of Renesas processors. k-fold validation.
Self-diagnosing HVAC and
refrigerant systems

7 DEPLOY
5 TEST
WHERE TO USE IT?
YOUR MODEL
6 OPTIMIZE
12 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 13
 Implementing Machine
Learning at the Edge
Implementing Machine
HOW TO USE IT?
Automatically generates

Learning at the Edge

machine learning models,
explanatory visualizations, and
hardware design analytics.

Software tool
2 Complete frameworks ORGANIZE
DATA
1 3 EXPLORE
for developing for specific use cases, GATHER
intelligent, embedded including hardware,
AI applications for firmware, software and DATA
edge devices. ML reference designs.

⊲ Create potential
⊲ Assign training labels. feature sets and ML
⊲ Collect data from models for classification.
WHAT IS different sources. ⊲ Segment files into consistent
sample sizes for AI ⊲ Select the best-fit model
REALITY AI? ⊲ Upload or link exploration, training, based on accuracy
source files. or testing. and computational
complexity.

Automotive Sound
Recognition (SWS)
⊲ Train the new base tool
against a subset of data. 4
⊲ Balance is important. TRAIN

Industrial Anomaly Detection

⊲ Reality AI software ⊲ Test the trained tool
comes with integration ⊲ Optimize for sensitivity against a subset of data.
to Renesas e² studio. and tolerances.
⊲ Validate performance using
⊲ Compile for a wide range ⊲ Retrain as needed. training samples and
of Renesas processors. k-fold validation.
Self-diagnosing HVAC and
refrigerant systems

7 DEPLOY
5 TEST
WHERE TO USE IT?
YOUR MODEL
6 OPTIMIZE
12 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 13
Four Metrics You As the four metrics are correlated—there tends to be an Benchmarking TinyML Models

Must Consider
inverse correlation between accuracy and memory but a
Benchmarks are necessary tools to set a reproducible
positive correlation between memory, latency, and power
standard to compare different technologies,
consumption—improving one could affect the others.
architectures, software, etc. In AI/ML, accuracy is the
So when developing a TinyML system, it is important to

When Developing
key metric to benchmark different models. In embedded
carefully consider this. A general rule would be to define
systems, common benchmarks include EEMBC’s
the necessary model accuracy required as per the use
CoreMark and ULPMark, measuring performance and
case and then compare a variety of developed models
power consumption, respectively. In the case of TinyML,

TinyML Systems
against the three other metrics (Figure 1), given a dummy
MLCommons has been gaining traction as the industry
example of a variety of models that have been trained.
standard where the four metrics discussed previously are
The marker shapes represent different model measured. Due to the heterogeneity of TinyML systems,
architectures with different hyperparameters, that tend to ensure fairness, four AI use cases with four different
to improve accuracy with an increase in architecture size AI models are used and have to achieve a certain level
at the expense of the other three metrics. Depending on of accuracy to qualify for the benchmark. Renesas
ELDAR SIDO | PRODUCT MARKETING SPECIALIST RENESAS ELECTRONICS the system-defined use case, a typical region of interest benchmarked two of its microcontrollers, RA6M4 and
is shown, from that, only one model has 90% accuracy, if RX65N, using TensorFlow Lite for microcontrollers as an

R
higher accuracy is required, the entire system should be inference engine, and the results can be viewed here. ■
ecently, with advancements in machine learning Power consumption is a critical consideration, as TinyML reconsidered to accommodate the increase in the other
(ML) there has been a split into two scales: systems are expected to operate for prolonged periods metrics.
traditional large ML (cloud ML), with models on batteries (typically in the order of milliwatts). The
getting larger to achieve the best performance power consumption of the TinyML model would depend
in terms of accuracy, and the nascent field of tiny on the hardware instruction sets available. For example,
machine learning (TinyML), where models are shrunk an Arm® Cortex®-M85 processor is significantly more
to fit into constrained devices to perform at ultra-low energy efficient than an Arm Cortex-M7 processor,
power. As TinyML is a nascent field, this blog will discuss thanks to the Helium instruction set. It would also
the metrics to consider when developing systems depend on the underlying software used to run the
incorporating TinyML and current industry standards into models (i.e., the inference engine); for example, using the
benchmarking TinyML devices. CMSIS-NN library improves the performance drastically
The four metrics that will be discussed are accuracy, as compared to reference kernels.
power consumption, latency, and memory requirements. Latency is important, as TinyML systems operate at
The system metric requirement will vary greatly the endpoint and do not require cloud connectivity, the
depending on the use case being developed. inference speeds of such systems are significantly better
Accuracy has been used as the main metric for the than cloud-based systems. Furthermore, in some use
performance of ML models for the last decade, with cases, having ultra-high inference speed (in milliseconds)
larger models tending to outperform their smaller is critical to be production ready. Similar to the power
predecessors. In TinyML systems, accuracy is also a consumption metric, it depends on the underlying
critical metric, but a balance with the other metrics is hardware and software.
more necessary, compared to cloud ML. Memory is a big hurdle in TinyML, as designers

“
squeeze down ML models to fit into size-constrained
Figure 1: Example of metrics to consider when developing systems incorporating TinyML. (Source: Renesas Electronics)
microcontrollers (less than 1MB). Thus, reducing memory
The system metric requirements has been a challenge, and during model
development, many techniques, such as pruning and
requirement will vary quantization, are used. Furthermore, the underlying
RA4E1 32-Bit RA6M3 32-Bit
greatly depending on the software plays a large role as better inference engines
optimize the models more effectively (better memory Microcontroller Group Microcontroller Group
use case being developed.

”
management and libraries to execute layers).
Learn More Learn More

14 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 15

Four Metrics You As the four metrics are correlated—there tends to be an Benchmarking TinyML Models

Must Consider
inverse correlation between accuracy and memory but a
Benchmarks are necessary tools to set a reproducible
positive correlation between memory, latency, and power
standard to compare different technologies,
consumption—improving one could affect the others.
architectures, software, etc. In AI/ML, accuracy is the
So when developing a TinyML system, it is important to

When Developing
key metric to benchmark different models. In embedded
carefully consider this. A general rule would be to define
systems, common benchmarks include EEMBC’s
the necessary model accuracy required as per the use
CoreMark and ULPMark, measuring performance and
case and then compare a variety of developed models
power consumption, respectively. In the case of TinyML,

TinyML Systems
against the three other metrics (Figure 1), given a dummy
MLCommons has been gaining traction as the industry
example of a variety of models that have been trained.
standard where the four metrics discussed previously are
The marker shapes represent different model measured. Due to the heterogeneity of TinyML systems,
architectures with different hyperparameters, that tend to ensure fairness, four AI use cases with four different
to improve accuracy with an increase in architecture size AI models are used and have to achieve a certain level
at the expense of the other three metrics. Depending on of accuracy to qualify for the benchmark. Renesas
ELDAR SIDO | PRODUCT MARKETING SPECIALIST RENESAS ELECTRONICS the system-defined use case, a typical region of interest benchmarked two of its microcontrollers, RA6M4 and
is shown, from that, only one model has 90% accuracy, if RX65N, using TensorFlow Lite for microcontrollers as an

R
higher accuracy is required, the entire system should be inference engine, and the results can be viewed here. ■
ecently, with advancements in machine learning Power consumption is a critical consideration, as TinyML reconsidered to accommodate the increase in the other
(ML) there has been a split into two scales: systems are expected to operate for prolonged periods metrics.
traditional large ML (cloud ML), with models on batteries (typically in the order of milliwatts). The
getting larger to achieve the best performance power consumption of the TinyML model would depend
in terms of accuracy, and the nascent field of tiny on the hardware instruction sets available. For example,
machine learning (TinyML), where models are shrunk an Arm® Cortex®-M85 processor is significantly more
to fit into constrained devices to perform at ultra-low energy efficient than an Arm Cortex-M7 processor,
power. As TinyML is a nascent field, this blog will discuss thanks to the Helium instruction set. It would also
the metrics to consider when developing systems depend on the underlying software used to run the
incorporating TinyML and current industry standards into models (i.e., the inference engine); for example, using the
benchmarking TinyML devices. CMSIS-NN library improves the performance drastically
The four metrics that will be discussed are accuracy, as compared to reference kernels.
power consumption, latency, and memory requirements. Latency is important, as TinyML systems operate at
The system metric requirement will vary greatly the endpoint and do not require cloud connectivity, the
depending on the use case being developed. inference speeds of such systems are significantly better
Accuracy has been used as the main metric for the than cloud-based systems. Furthermore, in some use
performance of ML models for the last decade, with cases, having ultra-high inference speed (in milliseconds)
larger models tending to outperform their smaller is critical to be production ready. Similar to the power
predecessors. In TinyML systems, accuracy is also a consumption metric, it depends on the underlying
critical metric, but a balance with the other metrics is hardware and software.
more necessary, compared to cloud ML. Memory is a big hurdle in TinyML, as designers

“
squeeze down ML models to fit into size-constrained
Figure 1: Example of metrics to consider when developing systems incorporating TinyML. (Source: Renesas Electronics)
microcontrollers (less than 1MB). Thus, reducing memory
The system metric requirements has been a challenge, and during model
development, many techniques, such as pruning and
requirement will vary quantization, are used. Furthermore, the underlying
RA4E1 32-Bit RA6M3 32-Bit
greatly depending on the software plays a large role as better inference engines
optimize the models more effectively (better memory Microcontroller Group Microcontroller Group
use case being developed.

”
management and libraries to execute layers).
Learn More Learn More

14 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 15

Data Science and
AI-Driven Real-Time
Analytics
KAUSHAL VORA | SENIOR DIRECTOR RENESAS ELECTRONICS
SUAD JUSUF | SENIOR MANAGER RENESAS ELECTRONICS

AI is proving to be a more
precise and time-efficient tool in
processing the big data crunch by
recognizing patterns and noticing
inconsistencies in real time.

M
ost modern consumer-driven industries and even preventing potential issues before they become
accumulate large volumes of data, but without problems. AI can restructure data collection and analysis
a filtering mechanism that maps, charts, and operations in real time and reduce unanticipated network
trends data models, the raw data has little interruptions. The marriage of AI and data science can
to no utility. Traditional analytical mechanisms are no improve data set structuring, help reach conclusions and
longer up to the task—the sheer amount of data and significantly enhance IoT operations.
the speed at which it’s being accumulated has made it
economically and operationally necessary to scale up Data Science and Machine Learning Figure 1: CRISP-DM steps. (Source: Renesas Electronics)
analytical capacity using data science.
Data science is the integrated approach of extracting
The Internet of Things (IoT) is constantly collecting data, insights from growing volumes of data using various
and the organization, distribution, and analysis of all scientific methods, algorithms, and processes. Data
this data is the new frontier of artificial Intelligence (AI). science is driven by software that recognizes hidden
While conventional data analysis methods are still being Machine learning is a branch of AI dependent on models When critical thinking encounters machine-learning
patterns within unprocessed data and draws conclusions to perform autonomous tasks without required extensive algorithms, data science can help achieve better
implemented, AI is proving to be a more precise and time- from those patterns. These valuable insights help facilitate
efficient tool in processing the big data crunch. AI can programming. It relies on statistical techniques or insights, guide efficiency efforts, and inform predictions.
corporate decision-making by interpreting problems algorithms that allow users to make predictions and/ The goal is for businesses to benefit from data science to
recognize patterns and notice inconsistencies in real time. analytically and generating viable solutions.
AI algorithms can save significant time and energy by or decisions based on historical data. Data scientists make progressive decisions and create more innovative
compiling data from multiple sources and presenting it with Cross-Industry Standard Process for Data Mining utilize technologies like machine learning and AI to products and services.
a uniform, consistent approach. AI and machine learning (CRISP-DM) is a process model that provides an overview of examine data and make predictions about imminent
can substantively improve secure and predictive analytics the data science life cycle. It’s essentially a framework that developments and potential issues, significantly
operations in the data center and beyond, anticipating assists in planning, organizing, and implementing a data reducing the need for human intervention.
science project. CRISP-DM consists of the steps in Figure 1:

16 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 17

Data Science and
AI-Driven Real-Time
Analytics
KAUSHAL VORA | SENIOR DIRECTOR RENESAS ELECTRONICS
SUAD JUSUF | SENIOR MANAGER RENESAS ELECTRONICS

AI is proving to be a more
precise and time-efficient tool in
processing the big data crunch by
recognizing patterns and noticing
inconsistencies in real time.

M
ost modern consumer-driven industries and even preventing potential issues before they become
accumulate large volumes of data, but without problems. AI can restructure data collection and analysis
a filtering mechanism that maps, charts, and operations in real time and reduce unanticipated network
trends data models, the raw data has little interruptions. The marriage of AI and data science can
to no utility. Traditional analytical mechanisms are no improve data set structuring, help reach conclusions and
longer up to the task—the sheer amount of data and significantly enhance IoT operations.
the speed at which it’s being accumulated has made it
economically and operationally necessary to scale up Data Science and Machine Learning Figure 1: CRISP-DM steps. (Source: Renesas Electronics)
analytical capacity using data science.
Data science is the integrated approach of extracting
The Internet of Things (IoT) is constantly collecting data, insights from growing volumes of data using various
and the organization, distribution, and analysis of all scientific methods, algorithms, and processes. Data
this data is the new frontier of artificial Intelligence (AI). science is driven by software that recognizes hidden
While conventional data analysis methods are still being Machine learning is a branch of AI dependent on models When critical thinking encounters machine-learning
patterns within unprocessed data and draws conclusions to perform autonomous tasks without required extensive algorithms, data science can help achieve better
implemented, AI is proving to be a more precise and time- from those patterns. These valuable insights help facilitate
efficient tool in processing the big data crunch. AI can programming. It relies on statistical techniques or insights, guide efficiency efforts, and inform predictions.
corporate decision-making by interpreting problems algorithms that allow users to make predictions and/ The goal is for businesses to benefit from data science to
recognize patterns and notice inconsistencies in real time. analytically and generating viable solutions.
AI algorithms can save significant time and energy by or decisions based on historical data. Data scientists make progressive decisions and create more innovative
compiling data from multiple sources and presenting it with Cross-Industry Standard Process for Data Mining utilize technologies like machine learning and AI to products and services.
a uniform, consistent approach. AI and machine learning (CRISP-DM) is a process model that provides an overview of examine data and make predictions about imminent
can substantively improve secure and predictive analytics the data science life cycle. It’s essentially a framework that developments and potential issues, significantly
operations in the data center and beyond, anticipating assists in planning, organizing, and implementing a data reducing the need for human intervention.
science project. CRISP-DM consists of the steps in Figure 1:

16 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 17

What Is Data Analytics? Real-time analytics is the process of preparing, bandwidth and power to support large and potentially Distributing intelligence across the end of the
assessing, and analyzing data as it becomes available. lengthy data transmissions. Potential latency issues are network provides an opportunity for more efficient
Data analytics is the science of scrutinizing raw or
Coupling real-time analytics with AI has provided also a factor; as the time expended during transmission data analytics and real-time decision-making at the
unprocessed data to derive meaning so it can be applied
businesses with fresh and revealing insights into the increases, so does the need for higher server capacities endpoints with little to no latency. Performing these
to decision-making. Data analytics employs several
consumer experience. It has provided support personnel to handle all incoming data. Situations may also arise functions directly on endpoint devices with low-
modern tools and techniques that help to structure data
and IT the opportunity to act preemptively rather than where IoT devices lack a stable internet connection, and processing capabilities is referred to as endpoint data
and make it understandable.
react, resolving potential issues before endpoint users decisions must be made at the endpoint. Solutions that analytics (Figure 3).
Several preliminary steps need to be applied in the even realize there is an issue to report. The combination
process before data analytics can begin: of AI and real-time analytics has completely transformed
• Data needs to be collected from a variety of sources. the consumer and consumer support experiences.

• D
 ata needs to be interpreted correctly so it may be
accurately organized and grouped. Endpoint Analytics of Things
• D
 ata needs to be examined to eliminate any According to the analytics firm International Data
replications or transcription errors. Corporation (IDC), 45% of data collected from IoT devices
needs to be analyzed nearer to the endpoint (i.e., closer
• Data may be managed by tables and/or spreadsheets.
to the device itself) to optimize overall efficiency rather
Data analytics can be subdivided into four main than transmitted to the cloud. Shifting away from the
categories (Figure 2): descriptive, diagnostic, predictive, cloud has several potential advantages: Transferring
and prescriptive. data to the cloud can be costly, as it requires significant

This is the process where data is employed to examine,

Descriptive Analytics
understand, and describe an event that’s already transpired.

This goes deeper than descriptive analytics as it pursues

Diagnostic Analytics
to comprehend the why behind what happened.
Figure 3: Endpoint data analytics. (Source: Renesas Electronics)

It is dependent on historical data, past trends, and assumptions

Predictive Analytics can support the consumer experience using analytics at The Data Science Support System
to answer projective questions about the future
the endpoint are often required. Various technologies and sensors facilitate the
Analytics of Things (AoT) is a term used to describe the processing of data analytics at the endpoint. Sensors
This process aims to identify specific actions that an individual analysis of data generated by IoT devices. AoT makes are required not only to gather and accumulate data
Prescriptive Analytics but also to assist in assessing and understanding the
or organization should take to reach future targets or goals it possible to operate business intelligence within
an application rather than a data warehouse. It also surrounding environment. Commonly implemented
Figure 2: Four main categories of data analytics. (Source: Renesas Electronics)
assists in understanding patterns, detecting anomalies, types of sensors include GPS for determining location
predicting potential issues, setting maintenance and thermometers and barometers for measuring
intervals, and optimizing processes. temperature and pressure. Cameras, lasers, and radar/
sonar can also be used to detect people and/or objects
Over the years, organizations have relied on data analytics
nearby.
based on a centralized architecture for their data-driven
planning and visions of the future. The data is increasing Communication technologies such as Wi-Fi and 5G
every second, and there is a requirement for a revolutionary assist in gathering and transmitting data. The data
approach to overcoming data transfer latency, improving is then fed into data science algorithms that play the
privacy, and meeting customer expectations. The real- central role in assessing the data and recommending
time response has especially become necessary after the potential courses of action. In certain use cases,
convergence of AI, 5G, and IoT. the outcome of the analytics performed at the

18 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 19

What Is Data Analytics? Real-time analytics is the process of preparing, bandwidth and power to support large and potentially Distributing intelligence across the end of the
assessing, and analyzing data as it becomes available. lengthy data transmissions. Potential latency issues are network provides an opportunity for more efficient
Data analytics is the science of scrutinizing raw or
Coupling real-time analytics with AI has provided also a factor; as the time expended during transmission data analytics and real-time decision-making at the
unprocessed data to derive meaning so it can be applied
businesses with fresh and revealing insights into the increases, so does the need for higher server capacities endpoints with little to no latency. Performing these
to decision-making. Data analytics employs several
consumer experience. It has provided support personnel to handle all incoming data. Situations may also arise functions directly on endpoint devices with low-
modern tools and techniques that help to structure data
and IT the opportunity to act preemptively rather than where IoT devices lack a stable internet connection, and processing capabilities is referred to as endpoint data
and make it understandable.
react, resolving potential issues before endpoint users decisions must be made at the endpoint. Solutions that analytics (Figure 3).
Several preliminary steps need to be applied in the even realize there is an issue to report. The combination
process before data analytics can begin: of AI and real-time analytics has completely transformed
• Data needs to be collected from a variety of sources. the consumer and consumer support experiences.

• D
 ata needs to be interpreted correctly so it may be
accurately organized and grouped. Endpoint Analytics of Things
• D
 ata needs to be examined to eliminate any According to the analytics firm International Data
replications or transcription errors. Corporation (IDC), 45% of data collected from IoT devices
needs to be analyzed nearer to the endpoint (i.e., closer
• Data may be managed by tables and/or spreadsheets.
to the device itself) to optimize overall efficiency rather
Data analytics can be subdivided into four main than transmitted to the cloud. Shifting away from the
categories (Figure 2): descriptive, diagnostic, predictive, cloud has several potential advantages: Transferring
and prescriptive. data to the cloud can be costly, as it requires significant

This is the process where data is employed to examine,

Descriptive Analytics
understand, and describe an event that’s already transpired.

This goes deeper than descriptive analytics as it pursues

Diagnostic Analytics
to comprehend the why behind what happened.
Figure 3: Endpoint data analytics. (Source: Renesas Electronics)

It is dependent on historical data, past trends, and assumptions

Predictive Analytics can support the consumer experience using analytics at The Data Science Support System
to answer projective questions about the future
the endpoint are often required. Various technologies and sensors facilitate the
Analytics of Things (AoT) is a term used to describe the processing of data analytics at the endpoint. Sensors
This process aims to identify specific actions that an individual analysis of data generated by IoT devices. AoT makes are required not only to gather and accumulate data
Prescriptive Analytics but also to assist in assessing and understanding the
or organization should take to reach future targets or goals it possible to operate business intelligence within
an application rather than a data warehouse. It also surrounding environment. Commonly implemented
Figure 2: Four main categories of data analytics. (Source: Renesas Electronics)
assists in understanding patterns, detecting anomalies, types of sensors include GPS for determining location
predicting potential issues, setting maintenance and thermometers and barometers for measuring
intervals, and optimizing processes. temperature and pressure. Cameras, lasers, and radar/
sonar can also be used to detect people and/or objects
Over the years, organizations have relied on data analytics
nearby.
based on a centralized architecture for their data-driven
planning and visions of the future. The data is increasing Communication technologies such as Wi-Fi and 5G
every second, and there is a requirement for a revolutionary assist in gathering and transmitting data. The data
approach to overcoming data transfer latency, improving is then fed into data science algorithms that play the
privacy, and meeting customer expectations. The real- central role in assessing the data and recommending
time response has especially become necessary after the potential courses of action. In certain use cases,
convergence of AI, 5G, and IoT. the outcome of the analytics performed at the

18 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 19

 Endpoint MCU
Implementation of
endpoint can even be
used to make automatic
Voice User Interface
adjustments to a device’s
plan of action. An autonomous GEORGIOS FLAMIS | SENIOR STAFF SYSTEM DESIGN ENGINEER RENESAS ELECTRONICS

R
vehicle, for example, may change
routes depending on traffic data apid consumer adoption of connected smart
collected from other vehicles in the same speakers and voice assistants suggests that
area or on weather and/or road conditions many people prefer to control products with
being monitored by onboard sensors. Data speech rather than counterintuitive button
analytics can help categorize potential actions pushes. Consumers also seem to appreciate devices and
and adjustments into classes to improve the overall home appliances that are smart without being connected
decision-making process; for example, the autonomous to the cloud via the internet. The Renesas Electronics
vehicle can classify potential actions as operational voice user interface (VUI) solution, a third-generation
(e.g., optimizing the route for better road and weather location, and behavioral patterns can assist in creating reference platform built on the scalable Arm®-based
conditions) or tactical (e.g., optimizing the route to highly targeted advertising campaigns. These ads RA2, RA4, and RA6 series of MCUs, is especially
reduce travel time). provide more precise recommendations by relying targeted to applications in home automation, white
Leveraging the power of predictive maintenance, on browsing and purchase history to evaluate the goods, and small appliances. The solution gives
intelligent sensors embedded in factory equipment can customer’s inclination toward buying a specific product. access to a variety of developer tools and training
collect data on optimal operations to help manufacturers In banking, monitoring unusual customer behavior can resources that will take product designs to the
reduce downtime and avoid potential revenue loss. aid in detecting and exposing fraud. By scrutinizing and next level, including design and training methods
Anticipating and acting on potential out-of-service assessing a patient’s medical records, data science may of the integrated machine learning model and
issues means that plants can run at peak efficiency. help predict and prevent potential medical problems. full evaluation with online and offline methods.
Developments in data analytics and AI have transformed Introduction
Where AI and Data Science Go from Here the consumer support experience. Real-time analytics
respond to the data collected immediately (when its VUI is speech recognition technology that enables users
Traditional business intelligence tools are not structured to interact with a computer, smartphone, or other daily-
value is at its peak), and data analytics unlock the almost
to process vast amounts of unstructured data. Data use device with voice commands (Figure 1). The unique
unlimited potential of that data, increasing overall
science utilizes more advanced tools to assist in feature of VUI is the use of voice as primary mode of
business value and potentially enabling enterprises to
sifting, grouping, and analyzing large volumes of data interaction, in contrast with traditional keyboard, mouse,
secure a high position in the marketplace. Businesses
from different sources across multiple interconnected display, or touch screen.
in all sectors rely on machine learning models to make
fields. Take marketing, for example: Exploring essential
more accurate predictions and effective decisions, The new, easy-to-use Renesas hardware platform for
demographic factors like a customer’s age, orientation,
thereby improving production efficiency and assisting in VUI solutions is based on the high-performance Renesas
overcoming potential security issues in communications Advanced (RA) family of 32-bit microcontroller units (MCUs).
between devices and the cloud. Endpoint AI has The RA family delivers key advantages compared to
enhanced the entire process by speeding up the work, competitive Arm Cortex®-M MCUs by providing stronger
providing better results, and improving business embedded security, superior CoreMark® performance,
RA4M3 32-Bit
decision-making. ■ and ultra-low-power operation. PSA Certified provides
Microcontroller Group
customers the confidence and assurance to quickly
deploy secure IoT endpoint and edge devices, and smart
Learn More Figure 1: Daily-use devices with VUI. (Source: Renesas Electronics)
factory equipment for Industry 4.0.

20 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 21

 Endpoint MCU
Implementation of
endpoint can even be
used to make automatic
Voice User Interface
adjustments to a device’s
plan of action. An autonomous GEORGIOS FLAMIS | SENIOR STAFF SYSTEM DESIGN ENGINEER RENESAS ELECTRONICS

R
vehicle, for example, may change
routes depending on traffic data apid consumer adoption of connected smart
collected from other vehicles in the same speakers and voice assistants suggests that
area or on weather and/or road conditions many people prefer to control products with
being monitored by onboard sensors. Data speech rather than counterintuitive button
analytics can help categorize potential actions pushes. Consumers also seem to appreciate devices and
and adjustments into classes to improve the overall home appliances that are smart without being connected
decision-making process; for example, the autonomous to the cloud via the internet. The Renesas Electronics
vehicle can classify potential actions as operational voice user interface (VUI) solution, a third-generation
(e.g., optimizing the route for better road and weather location, and behavioral patterns can assist in creating reference platform built on the scalable Arm®-based
conditions) or tactical (e.g., optimizing the route to highly targeted advertising campaigns. These ads RA2, RA4, and RA6 series of MCUs, is especially
reduce travel time). provide more precise recommendations by relying targeted to applications in home automation, white
Leveraging the power of predictive maintenance, on browsing and purchase history to evaluate the goods, and small appliances. The solution gives
intelligent sensors embedded in factory equipment can customer’s inclination toward buying a specific product. access to a variety of developer tools and training
collect data on optimal operations to help manufacturers In banking, monitoring unusual customer behavior can resources that will take product designs to the
reduce downtime and avoid potential revenue loss. aid in detecting and exposing fraud. By scrutinizing and next level, including design and training methods
Anticipating and acting on potential out-of-service assessing a patient’s medical records, data science may of the integrated machine learning model and
issues means that plants can run at peak efficiency. help predict and prevent potential medical problems. full evaluation with online and offline methods.
Developments in data analytics and AI have transformed Introduction
Where AI and Data Science Go from Here the consumer support experience. Real-time analytics
respond to the data collected immediately (when its VUI is speech recognition technology that enables users
Traditional business intelligence tools are not structured to interact with a computer, smartphone, or other daily-
value is at its peak), and data analytics unlock the almost
to process vast amounts of unstructured data. Data use device with voice commands (Figure 1). The unique
unlimited potential of that data, increasing overall
science utilizes more advanced tools to assist in feature of VUI is the use of voice as primary mode of
business value and potentially enabling enterprises to
sifting, grouping, and analyzing large volumes of data interaction, in contrast with traditional keyboard, mouse,
secure a high position in the marketplace. Businesses
from different sources across multiple interconnected display, or touch screen.
in all sectors rely on machine learning models to make
fields. Take marketing, for example: Exploring essential
more accurate predictions and effective decisions, The new, easy-to-use Renesas hardware platform for
demographic factors like a customer’s age, orientation,
thereby improving production efficiency and assisting in VUI solutions is based on the high-performance Renesas
overcoming potential security issues in communications Advanced (RA) family of 32-bit microcontroller units (MCUs).
between devices and the cloud. Endpoint AI has The RA family delivers key advantages compared to
enhanced the entire process by speeding up the work, competitive Arm Cortex®-M MCUs by providing stronger
providing better results, and improving business embedded security, superior CoreMark® performance,
RA4M3 32-Bit
decision-making. ■ and ultra-low-power operation. PSA Certified provides
Microcontroller Group
customers the confidence and assurance to quickly
deploy secure IoT endpoint and edge devices, and smart
Learn More Figure 1: Daily-use devices with VUI. (Source: Renesas Electronics)
factory equipment for Industry 4.0.

20 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 21

The RA family currently includes three product The RA2 Series are ideal for designs where the low power third-party solutions as part of the Arm ecosystem
series: RA6, RA4, and RA2. Each of these series has requirements of an application matter most. To achieve increases the range of choice for application
a unique feature set, making them ideal for various the best performance, special power-down modes are development. This means that developers can
applications and market needs. The RA6 Series offers provided, making these devices well suited for battery- choose the software model that best suits their needs
the widest integration of communication interfaces, powered applications. The RA2 Series provides memory while utilizing Renesas’s excellent Arm-based silicon
with integrated Ethernet and TFT display drivers. Flash densities of up to 256KB of embedded flash and a wide solutions as well as speed up the implementation time
memory densities range from 256KB to 2MB. The RA6 single voltage supply range of 1.6 to 5.5V. These devices of complex areas like connectivity and security.
Series offers up to 240MHz performance running on the use the Cortex-M23 core running at up to 48MHz.
Cortex-M4 or Cortex-M33 core with TrustZone®. The RA6 The Renesas Flexible Software Package (FSP) is an Voice Recognition Engine
Series supports full security integration, making these enhanced software package designed to provide
devices widely desired for security applications. Based on the Renesas ecosystem, Cyberon DSpotter
easy-to-use, scalable, high-quality software for (Figure 3) is a local voice trigger and command
The RA4 Series balances the requirements for low power embedded system designs using Renesas RA Family recognition solution with robust noise reduction that
consumption with the demand for connectivity. It offers microcontrollers (Figure 2). It uses an open software consumes very low resources and provides high-
up to 1MB of flash and a wide range of communication ecosystem and provides flexibility in using bare-metal accuracy performance. It supports multiple languages as
interfaces. The utilized core is the Cortex-M4 or Cortex-M33 programming, including Azure RTOS, FreeRTOS, well as many connectivity function and security capable
with TrustZone and additional security IP integration. other preferred RTOS, legacy code, and third-party depending on the selected MCU. The major features are
Memory densities range from 256KB to 1MB of flash. These ecosystem solutions. The combination of the flexible listed below:
devices provide a CPU frequency of up to 100MHz. open architecture of the FSP plus the wide choice of
• L
 ocal voice recognition algorithm, no network • S
 mall footprint and cost effective (single DMIC + RA6E1
connectivity needed or RA4E1)
• Phoneme-based modeling • D
 SMT Tool: wake-word and commands customization,
• Q
 uick command customization—removes the need to performance tuning, testing, no prior neural network
collect speech data in advance knowledge needed.

• O
 ptimization by model adaptation just with small • S
 eparation of recognition engine and command model,
amount speech data switching commands dynamically

• Global language support: 44+ languages worldwide • L

 ow-power, high-efficiency RA MCU with strong
security function

Figure 2: Renesas Flexible Software Package. (Source: Renesas Electronics)

Figure 3: Cyberon DSpotter voice recognition engine. (Source: Renesas Electronics)

22 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 23

The RA family currently includes three product The RA2 Series are ideal for designs where the low power third-party solutions as part of the Arm ecosystem
series: RA6, RA4, and RA2. Each of these series has requirements of an application matter most. To achieve increases the range of choice for application
a unique feature set, making them ideal for various the best performance, special power-down modes are development. This means that developers can
applications and market needs. The RA6 Series offers provided, making these devices well suited for battery- choose the software model that best suits their needs
the widest integration of communication interfaces, powered applications. The RA2 Series provides memory while utilizing Renesas’s excellent Arm-based silicon
with integrated Ethernet and TFT display drivers. Flash densities of up to 256KB of embedded flash and a wide solutions as well as speed up the implementation time
memory densities range from 256KB to 2MB. The RA6 single voltage supply range of 1.6 to 5.5V. These devices of complex areas like connectivity and security.
Series offers up to 240MHz performance running on the use the Cortex-M23 core running at up to 48MHz.
Cortex-M4 or Cortex-M33 core with TrustZone®. The RA6 The Renesas Flexible Software Package (FSP) is an Voice Recognition Engine
Series supports full security integration, making these enhanced software package designed to provide
devices widely desired for security applications. Based on the Renesas ecosystem, Cyberon DSpotter
easy-to-use, scalable, high-quality software for (Figure 3) is a local voice trigger and command
The RA4 Series balances the requirements for low power embedded system designs using Renesas RA Family recognition solution with robust noise reduction that
consumption with the demand for connectivity. It offers microcontrollers (Figure 2). It uses an open software consumes very low resources and provides high-
up to 1MB of flash and a wide range of communication ecosystem and provides flexibility in using bare-metal accuracy performance. It supports multiple languages as
interfaces. The utilized core is the Cortex-M4 or Cortex-M33 programming, including Azure RTOS, FreeRTOS, well as many connectivity function and security capable
with TrustZone and additional security IP integration. other preferred RTOS, legacy code, and third-party depending on the selected MCU. The major features are
Memory densities range from 256KB to 1MB of flash. These ecosystem solutions. The combination of the flexible listed below:
devices provide a CPU frequency of up to 100MHz. open architecture of the FSP plus the wide choice of
• L
 ocal voice recognition algorithm, no network • S
 mall footprint and cost effective (single DMIC + RA6E1
connectivity needed or RA4E1)
• Phoneme-based modeling • D
 SMT Tool: wake-word and commands customization,
• Q
 uick command customization—removes the need to performance tuning, testing, no prior neural network
collect speech data in advance knowledge needed.

• O
 ptimization by model adaptation just with small • S
 eparation of recognition engine and command model,
amount speech data switching commands dynamically

• Global language support: 44+ languages worldwide • L

 ow-power, high-efficiency RA MCU with strong
security function

Figure 2: Renesas Flexible Software Package. (Source: Renesas Electronics)

Figure 3: Cyberon DSpotter voice recognition engine. (Source: Renesas Electronics)

22 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 23

Table 1: Results of hit rate

SNR Background noise Distance Hit-Rate Alexa Requirements Reality AI:

A Game Changer in
(Clean) none 1.5m 100.00% 90%

(Clean) none 3m 100.00% 90%

Sensor Data Analysis

10dB Babble 1.5m 98.55% 80%

10dB Babble 3m 98.84% 80%

10dB Music 1.5m 98.26% 80%

10dB Music 3m 98.55% 80%

JOSEPH DOWNING | MOUSER ELECTRONICS
10dB TV 1.5m 98.84% 80%

10dB TV 3m 98.55% 80%

U “
5dB Babble 1.5m 98.84% 80%
nscheduled equipment downtime can be
costly for companies—both financially and in
5dB Babble 3m 96.24% 80%
time spent on repairs. By having systems to Reality AI has become a
5dB Music 1.5m 98.84% 80% monitor and report on potential equipment game changer in the field
issues, companies can catch potential issues before
of sensor data analysis.

”
5dB Music 3m 97.08% 80% they become major problems. This helps them take
5dB TV 1.5m 93.37% 80% corrective action quickly and avoid losing resources due
to unscheduled or extended equipment downtime. It also
5dB TV 3m 90.72% 80% helps them identify issues that can be quickly and easily
Renesas Reality AI supports a wide range of use cases,
fixed, saving time and money in the long run.
including predictive maintenance, anomaly detection,
Results To mitigate downtime, predictive maintenance using and voice. Using its robust ML capabilities, developers
machine learning (ML) algorithms can be trained can easily train and optimize models using their own
Hit rate has been captured with voice commands
and deployed based on data gathered from sensor data and quickly deploy those models to their target
mixed with different types of noise in levels suitable to
information, providing early detection in critical systems. devices. The tool also includes comprehensive support
create distinguished signal-to-noise ratios. The test
Renesas Reality AI has revolutionized the way businesses for a wide range of Renesas hardware platforms and
bench shown in Figure 4 is utilized and the results are
analyze sensor data. The platform uses ML algorithms microcontrollers (MCUs), making it easy to integrate with
summarized in Table 1.
and advanced analytics to extract meaningful insights existing hardware and software systems.
from data streams generated by equipment sensors.
Conclusions With its powerful tools, Reality AI has become a game
Renesas Reality AI supplies a powerful and efficient
solution for developers looking to build intelligent,
The complete implementation of an endpoint voice changer in the field of sensor data analysis.
embedded AI applications for edge devices.
commands recognition system has been presented
that is capable to execute on a simple MCU device. The What Is Reality AI?
reference design enables local voice recognition without
Features of Reality AI
Renesas Reality AI is a software tool that allows
a network connection and allows to quickly start building The Reality AI platform offers several key features that
developers to develop intelligent, embedded artificial
an enhanced VUI in minutes that recognizes voice make it a valuable resource for businesses looking to
intelligence (AI) applications quickly and easily for edge
commands to trigger the corresponding operation. ■ analyze sensor data:
devices. The suite of tools includes an intuitive drag-and-
drop interface for data upload, pre-trained ML models, • A
 nomaly detection: By identifying anomalies in the
and automatic code generation for Renesas’s e² studio data, the platform can automatically generate ML
integrated development environment (IDE), allowing models, explanatory visualizations, and hardware
developers to build and deploy applications faster and design analytics. This can be especially useful for
with less effort. predictive maintenance applications, where identifying
Figure 4: Evaluation test bench. (Source: Renesas Electronics) issues early can prevent costly downtime and repairs.

24 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 25

Table 1: Results of hit rate

SNR Background noise Distance Hit-Rate Alexa Requirements Reality AI:

A Game Changer in
(Clean) none 1.5m 100.00% 90%

(Clean) none 3m 100.00% 90%

Sensor Data Analysis

10dB Babble 1.5m 98.55% 80%

10dB Babble 3m 98.84% 80%

10dB Music 1.5m 98.26% 80%

10dB Music 3m 98.55% 80%

JOSEPH DOWNING | MOUSER ELECTRONICS
10dB TV 1.5m 98.84% 80%

10dB TV 3m 98.55% 80%

U “
5dB Babble 1.5m 98.84% 80%
nscheduled equipment downtime can be
costly for companies—both financially and in
5dB Babble 3m 96.24% 80%
time spent on repairs. By having systems to Reality AI has become a
5dB Music 1.5m 98.84% 80% monitor and report on potential equipment game changer in the field
issues, companies can catch potential issues before
of sensor data analysis.

”
5dB Music 3m 97.08% 80% they become major problems. This helps them take
5dB TV 1.5m 93.37% 80% corrective action quickly and avoid losing resources due
to unscheduled or extended equipment downtime. It also
5dB TV 3m 90.72% 80% helps them identify issues that can be quickly and easily
Renesas Reality AI supports a wide range of use cases,
fixed, saving time and money in the long run.
including predictive maintenance, anomaly detection,
Results To mitigate downtime, predictive maintenance using and voice. Using its robust ML capabilities, developers
machine learning (ML) algorithms can be trained can easily train and optimize models using their own
Hit rate has been captured with voice commands
and deployed based on data gathered from sensor data and quickly deploy those models to their target
mixed with different types of noise in levels suitable to
information, providing early detection in critical systems. devices. The tool also includes comprehensive support
create distinguished signal-to-noise ratios. The test
Renesas Reality AI has revolutionized the way businesses for a wide range of Renesas hardware platforms and
bench shown in Figure 4 is utilized and the results are
analyze sensor data. The platform uses ML algorithms microcontrollers (MCUs), making it easy to integrate with
summarized in Table 1.
and advanced analytics to extract meaningful insights existing hardware and software systems.
from data streams generated by equipment sensors.
Conclusions With its powerful tools, Reality AI has become a game
Renesas Reality AI supplies a powerful and efficient
solution for developers looking to build intelligent,
The complete implementation of an endpoint voice changer in the field of sensor data analysis.
embedded AI applications for edge devices.
commands recognition system has been presented
that is capable to execute on a simple MCU device. The What Is Reality AI?
reference design enables local voice recognition without
Features of Reality AI
Renesas Reality AI is a software tool that allows
a network connection and allows to quickly start building The Reality AI platform offers several key features that
developers to develop intelligent, embedded artificial
an enhanced VUI in minutes that recognizes voice make it a valuable resource for businesses looking to
intelligence (AI) applications quickly and easily for edge
commands to trigger the corresponding operation. ■ analyze sensor data:
devices. The suite of tools includes an intuitive drag-and-
drop interface for data upload, pre-trained ML models, • A
 nomaly detection: By identifying anomalies in the
and automatic code generation for Renesas’s e² studio data, the platform can automatically generate ML
integrated development environment (IDE), allowing models, explanatory visualizations, and hardware
developers to build and deploy applications faster and design analytics. This can be especially useful for
with less effort. predictive maintenance applications, where identifying
Figure 4: Evaluation test bench. (Source: Renesas Electronics) issues early can prevent costly downtime and repairs.

24 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 25

• A
 dvanced signal processing: Reality AI automatically • I ndustrial anomaly detection: Through a hardware
searches a wide range of signal-processing transforms and software solution for equipment monitoring and
to create a custom, optimized feature transform. end-of-line testing for quality control, users can deploy
• M
 CU/MPU edge nodes: Reality AI runs on almost every an edge node with one or more included sensors and
MCU and microprocessor (MPU) core available from begin monitoring production equipment for unusual or
Renesas, with new ones added constantly. Reality AI anomalous functions.
also supports Renesas Motor Control boards. • H
 VAC and refrigeration systems: By combining ML
and edge processing and using data such as vibration
Sensorless
• D
 ata visualization: The platform offers a user-friendly Motor Demo
interface that allows businesses to visualize their and electrical signals, this application can potentially
data in real time. This can be useful for monitoring predict a wide array of anomalies, faults, and operating With support provided for
operations and identifying trends or issues as they conditions. the Renesas Motor Control
occur. • A
 utomotive sound recognition: By analyzing audio Kits, one way to evaluate the
data, the solution can detect and localize potential features available through Reality
AI Tools is by using an existing
Applications of Reality AI hazards (e.g., emergency vehicles, motorcycles) that
may be hidden from view from line-of-sight sensors. demo, such as the Reality AI demo on
The Reality AI platform has a wide range of applications mouser.com. The demo incorporates
in numerous industries, including industrial automation; • P
 redictive maintenance: By analyzing sensor data from
the Renesas RA6T2 Motor Control Kit and
heating, ventilation, and air conditioning (HVAC); and industrial machines, the platform can identify potential
provides insight and instructions for creating
automotive. The following are just a few examples of how issues before they occur. This can prevent costly
a predictive sensorless ML model that detects
the platform could be used: downtime and repairs and increase the lifespan of the
unbalanced motor loads using current, voltage,
machines.
and speed. The RA6T2 Motor Control Kit features an
enhanced central processing unit (CPU) and hardware
accelerator for home appliance and industrial automation
application deployments. The kit comes complete with
all the basics, including a CPU board, inverter board, of interest, the Reality AI toolset extracts meaningful
communication board, and brushless DC motor. insights and offers potential BOM optimization. These
The project article outlines the necessary resources and capabilities can also be incorporated via firmware
required hardware, with applicable links as well as steps updates into existing systems without additional BOM
to train and deploy your ML model. If you are looking to costs. The models generated are extremely resource
jump right in with Reality AI without the hardware, pre- efficient, consuming insignificant CPU load so the
built data files are also available to help you get started. MCU is practically free to perform all the usual tasks
The steps in the demo take you through everything while incorporating these features. With its complete
from uploading and curating your data to training and integration into the Renesas toolchain and full support
deployment. Of course, this only scratches the surface for the MCU portfolio and development boards, the
of the entire suite of available features, such as using Reality AI toolset provides the speed and accuracy
new data to further test and optimize as well as ways to required to deliver the precision businesses expect to
potentially decrease the complexity of the model without make sense of vast amounts of data, helping to minimize
sacrificing accuracy. costly equipment downtime.■
An example of a simple application for this type of
demo is detecting issues with load balance in a washing
machine during a spin cycle and notifying the end user.

Conclusion
Renesas Reality AI is a powerful software toolset that can
RA6M4 Evaluation Kit
revolutionize the way businesses analyze sensor data.
Through automated analysis and selection of the most
relevant sensor data for the specific prediction model Learn More

26 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 27

• A
 dvanced signal processing: Reality AI automatically • I ndustrial anomaly detection: Through a hardware
searches a wide range of signal-processing transforms and software solution for equipment monitoring and
to create a custom, optimized feature transform. end-of-line testing for quality control, users can deploy
• M
 CU/MPU edge nodes: Reality AI runs on almost every an edge node with one or more included sensors and
MCU and microprocessor (MPU) core available from begin monitoring production equipment for unusual or
Renesas, with new ones added constantly. Reality AI anomalous functions.
also supports Renesas Motor Control boards. • H
 VAC and refrigeration systems: By combining ML
and edge processing and using data such as vibration
Sensorless
• D
 ata visualization: The platform offers a user-friendly Motor Demo
interface that allows businesses to visualize their and electrical signals, this application can potentially
data in real time. This can be useful for monitoring predict a wide array of anomalies, faults, and operating With support provided for
operations and identifying trends or issues as they conditions. the Renesas Motor Control
occur. • A
 utomotive sound recognition: By analyzing audio Kits, one way to evaluate the
data, the solution can detect and localize potential features available through Reality
AI Tools is by using an existing
Applications of Reality AI hazards (e.g., emergency vehicles, motorcycles) that
may be hidden from view from line-of-sight sensors. demo, such as the Reality AI demo on
The Reality AI platform has a wide range of applications mouser.com. The demo incorporates
in numerous industries, including industrial automation; • P
 redictive maintenance: By analyzing sensor data from
the Renesas RA6T2 Motor Control Kit and
heating, ventilation, and air conditioning (HVAC); and industrial machines, the platform can identify potential
provides insight and instructions for creating
automotive. The following are just a few examples of how issues before they occur. This can prevent costly
a predictive sensorless ML model that detects
the platform could be used: downtime and repairs and increase the lifespan of the
unbalanced motor loads using current, voltage,
machines.
and speed. The RA6T2 Motor Control Kit features an
enhanced central processing unit (CPU) and hardware
accelerator for home appliance and industrial automation
application deployments. The kit comes complete with
all the basics, including a CPU board, inverter board, of interest, the Reality AI toolset extracts meaningful
communication board, and brushless DC motor. insights and offers potential BOM optimization. These
The project article outlines the necessary resources and capabilities can also be incorporated via firmware
required hardware, with applicable links as well as steps updates into existing systems without additional BOM
to train and deploy your ML model. If you are looking to costs. The models generated are extremely resource
jump right in with Reality AI without the hardware, pre- efficient, consuming insignificant CPU load so the
built data files are also available to help you get started. MCU is practically free to perform all the usual tasks
The steps in the demo take you through everything while incorporating these features. With its complete
from uploading and curating your data to training and integration into the Renesas toolchain and full support
deployment. Of course, this only scratches the surface for the MCU portfolio and development boards, the
of the entire suite of available features, such as using Reality AI toolset provides the speed and accuracy
new data to further test and optimize as well as ways to required to deliver the precision businesses expect to
potentially decrease the complexity of the model without make sense of vast amounts of data, helping to minimize
sacrificing accuracy. costly equipment downtime.■
An example of a simple application for this type of
demo is detecting issues with load balance in a washing
machine during a spin cycle and notifying the end user.

Conclusion
Renesas Reality AI is a powerful software toolset that can
RA6M4 Evaluation Kit
revolutionize the way businesses analyze sensor data.
Through automated analysis and selection of the most
relevant sensor data for the specific prediction model Learn More

26 BRINGING INTELLIGENCE TO THE EDGE Renesas 2023 27

Mouser stocks the widest
selection of the newest products

mouser.com/renesas
Worldwide leading authorized distributor of
semiconductors and electronic components

Mouser and Mouser Electronics are registered trademarks of Mouser Electronics, Inc. Other products, logos, and company names mentioned herein, may be
trademarks of their respective owners.