Technologies for Artificial Intelligence

Prof. Manuel Roveri – [email protected]

Lecture 3 – Algorithms for Embedded and Edge AI

Course Topics

1. Introduction to technological platforms for AI

2. Embedded and Edge AI
a. The technology
b. The Algorithms
c. Machine Learning for Embedded and Edge AI
d. Deep Learning for Embedded and Edge AI
3. Cloud computing and AI
a. Cloud computing and the ”as-a-service” approach
b. Machine and Deep Learning as a service
c. Time-series: analysis and prediction
d. Generative AI

2
An overview of an embedded and edge AI system
Software

Preprocessing
Sensors
Feature Extraction

Wake-word detection
Machine and Deep
Learning Algorithm

Postprocessing
Actuators Decision making
Person detection

Hardware
Gesture recognition

3
An overview of an embedded and edge AI system
Software

Preprocessing
Previous lecture!
Sensors
Feature Extraction

Wake-word detection
Machine and Deep
Learning Algorithm

Postprocessing
Actuators Decision making
Person detection

Hardware
Gesture recognition

4
An overview of an embedded and edge AI system
Software

Preprocessing
This lecture!
Sensors
Feature Extraction

Wake-word detection
Machine and Deep
Learning Algorithm

Postprocessing
Actuators Decision making
Person detection

Hardware
Gesture recognition

5
Algorithms for embedded and edge AI

Software

• Receive in input the stream(s) of data coming from the

Sensors

Preprocessing
Feature Extraction sensors, extract windows of data, remove noise and
extract relevant features
Machine and Deep • Apply machine/deep learning algorithms on the
Learning Algorithm
preprocessed data to generate and output (e.g., for
classification, regression or prediction)
Actuators

Postprocessing • The output of the algorithm is used to make decision or

Decision making
activate a reaction (e.g., open a door, raise an alarm or
transmit a signal)

6
1) Data preprocessing and feature extraction

• Goal: transform raw signals into data to be processed by machine and

deep learning algorithm
• How to achieve this goal?
a) Chopping of raw signals into chunks of data
b) Apply digital signal processing algorithms on chunks of data to remove noise and highlight
relevant ”portions” of the signal present in the chunks
c) Extract features from the pre-processed chunks of data

7
a) Chopping streams of data

Sensors
produces
streams of data

Windows of data are extracted from the stream (a chunk of data)

The chunk of data is

processed by the
algorithm generating
a result

The amount of time required to process a single

chunk of data is called latency

8
 1_ acquire the data

Latency and window size

2_preprocessing
3_machine learning
4_results

• The latency defines how fast the embedded system can process the chunk of data
• The lower the latency, the higher the amount of chunks of data that can be processed in a
given amount of time (e.g., a second or a minute)
latency in preprocessing, machine learning, and the results.

• the larger the window, the higher

the latency
• the larger the window, the larger
the amount of information that is
present (hence better accuracies in
the processing :-)

• The design choice of the algorithm has a strong effect on the computational
requirement (the latency) as well as the memory demand (see next lectures)

Image taken from [1]

9
Latency and frame rate

• The rate the system is able to acquire and process the data is called frame rate
(the same holds also for images and videos streaming)

Non-overlapping windows Overlapping windows

for example some sentences has to be read together so we have overlapping

Image taken from [1]

10
 Latency of
the algorithm

Latency of
the algorithm

Latency of
the algorithm
The latency has a strong effect on the effectiveness of the system

Latency of
the algorithm
11
Image taken from [1]
1) Data preprocessing and feature extraction (recap)

• Goal: transform raw signals into data to be processed by machine and

deep learning algorithm
• How to achieve this goal?
a) Chopping of raw signals into chunks of data
b) Apply digital signal processing algorithms on chunks of data to remove noise and highlight
relevant ”portions” of the signal present in the chunks
c) Extract features from the pre-processed chunks of data

12
Digital processing algorithms for data preprocessing

Reconstruction
Resampling Filtering
of missing data

13
Digital processing algorithms for data preprocessing

• “Global” filling methods: When we fill in missing data based on

observations about the entire data set.

• “Local” filling methods: When we use neighboring data points to

estimate the missing value (Forward Fill, Moving average, local
Reconstruction Interpolation)
of missing data
• Deletion of affected time periods: When we choose not to use
time periods that have missing data at all.

14
Digital processing algorithms for data preprocessing

• Dealing with time series having • Dealing with images requiring to

different sampling frequencies change the spatial resolution (pixel
per image)
• Change the sampling frequency:
• Downsample to reduce the
• Upsampling: increasing the resolution
timestamp frequency
(replicating, interpolating) • Interpolation to increase the
Resampling resolution
• Downsampling: decreasing the
• Dealing with images requiring to
timestamp frequency
change the shape:
(subsampling)
• Cropping
• We do not change the actual rate
• Resizing
• Mind the aliasing (see Nyquist)

15
Digital processing algorithms for data preprocessing

• “Low pass filter” to keep low frequencies (mind the cutoff

frequency and frequency response)
• “High pass filter” to keep high frequencies (mind the cutoff
frequency and frequency response)
• “Band pass filter” to keep low frequencies (mind the band
frequency and frequency response)
Filtering
E.g., for a speech recognition application the
frequencies of interest range from 100Hz to
4KHz (i.e., the band of human speech).

We could band-pass the frequencies of

interest and discard the rest

Many embedded and edge processors provide hardware support for

the fast and energy-efficient computation of filtering operations
16
1) Data preprocessing and feature extraction (recap)

• Goal: transform raw signals into data to be processed by machine and

deep learning algorithm
• How to achieve this goal?
a) Chopping of raw signals into chunks of data
b) Apply digital signal processing algorithms on chunks of data to remove noise and highlight
relevant ”portions” of the signal present in the chunks
c) Extract features from the pre-processed chunks of data

17
Feature Extraction in the time domain

• Mean (before PCA) • SNR

• PCA eigenvalue • Peak decay
• Max amplitude • Energy

18
Feature Extraction in the frequency domain

• Max amplitude
• Main frequency
• Variance of peaks

19
Feature Extraction in the time-frequency domain: the spectrogram

Image taken from [1]

20
Feature Extraction in the radar domain

Distance
Time

21
Image feature detection

• Edge detection: highlight boundaries

• Corner detection: find corners in shape
• Blob detection: highlight areas with similar content
• Ridge detection: highlight curves

OpenCV for feature detection (and

OpenMV for feature detection
other image processing tasks)
on High-End MCUs
on SoC

22
Combining features and sensors: sensor fusion

Ferature
Sensor 1
extraction 1

Ferature
Sensor 2 Machine and Deep
extraction 2 Learning Algorithm

Postprocessing
Decision making
Ferature
Sensor 3
extraction 3

23
Combining features and sensors: normalization and standardization

• Features with different scales might underfit the training of machine learning
• Min-Max Normalization in the range [0,1]:

𝑥𝑖 − MIN(𝑥𝑖 )
𝑖
𝑥𝑖𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 =
MAX( 𝑥𝑖 ) − MIN( 𝑥𝑖 )
𝑖 𝑖

• Standardization (removing mean and dividing by the std):

𝑥𝑖 − MEAN(𝑥𝑖 )
𝑥𝑖𝑠𝑡𝑎𝑛𝑑𝑎𝑟𝑑𝑖𝑧𝑒𝑑 =
𝑆𝑇𝐷(𝑥𝑖 )

24
Algorithms for embedded and edge AI

Software

• Receive in input the stream(s) of data coming from the

Sensors

Preprocessing
Feature Extraction sensors, extract windows of data, remove noise and
extract relevant features
Machine and Deep • Apply machine/deep learning algorithms on the
Learning Algorithm
preprocessed data to generate and output (e.g., for
classification, regression or prediction)
Actuators

Postprocessing • The output of the algorithm is used to make decision or

Decision making
activate a reaction (e.g., open a door, raise an alarm or
transmit a signal)

25
Algorithms for embedded and edge AI

Machine and deep learning Machine and deep learning

algorithms by functionality algorithms by implementation

26
Machine and deep learning algorithms by functionality

• Classification
• Regression
• Object detection
• Segmentation
• Anomaly detection
• Prediction
• Feature reduction

27
Machine and deep learning algorithms by functionality

• Classification • Classify micro-acoustic

• Regression emissions of a rock face
• Object detection • Recognize bird vocalizations
• Classify the behavior of a
• Segmentation patient in a retirement house by
• Anomaly detection analyzing radar data
• Prediction
• Feature reduction

Image taken from [1]

28
Machine and deep learning algorithms by functionality

• Classification • Measure the level of a liquid in

• Regression cup
• Object detection • Estimate the number of people
in the room by analyzing radar
• Segmentation data
• Anomaly detection
• Prediction Person 1

• Feature reduction
Person 2

29
Machine and deep learning algorithms by functionality

• Classification
• Regression
• Object detection
• Segmentation
• Anomaly detection
• Prediction
• Feature reduction

Image taken from [1]

30
Machine and deep learning algorithms by functionality

• Classification
• Regression
• Object detection
• Segmentation
• Anomaly detection
• Prediction
• Feature reduction

Image taken from [1]

31
Machine and deep learning algorithms by functionality

• Classification • Detect persons in a conveyer belt

• Regression • Detect micro-acoustic emissions
• Object detection • Detect enlargement of fractures
• Segmentation Pluviometers

• Anomaly detection High precision

inclinometers
• Prediction
Feature reduction
Strain gauges
• Mid precision
inclinometers

MEMS
accelerometer
Flow meters

32
Machine and deep learning algorithms by functionality

• Classification • Predict the next value of a time

• Regression series (e.g., temperature, humidity,
• Object detection power consumption, etc.)
• Segmentation
• Anomaly detection
• Prediction
• Feature reduction

33
Machine and deep learning algorithms by functionality

• Classification
• Regression
• Object detection
• Segmentation
• Anomaly detection
• Prediction • Compress or extract features
from audio chunks to reduce
• Feature reduction
the size

34
Algorithms for embedded and edge AI

Machine and deep learning Machine and deep learning

algorithms by functionality algorithms by implementation

35
Machine and deep learning algorithms by implementation

• Conditional logic
• Machine Learning
• Deep Learning

36
Conditional logic

• Deterministic
• Efficient
• Do not require training data
Image taken from [1]

37
Machine Learning and Deep Learning (and AI)

Artificial Intelligence

Machine Learning

Deep Learning

38
Machine Learning and Deep Learning (this time w/o AI)

39
 Software

Embedded
SW Cloud
Edge

Machine/Deep
IoT Devices Learning

40
Heterogeneous hardware platforms for ML/DL

CPU RAM

… ...
Cloud

… …
PC

Edge Computing 1GHz 1Gb

Internet of Things 16MHz 1Mb

41
How does TinyML work?

Image from Medium.com “Introducing cAInvas for TinyML devices”, July 2020

42
To train or not to train…. this is the problem with TinyML

Image taken from [1]

43
In the next two lectures we will see how to design
machine and deep learning solutions for TinyML

44