Deep Learning Application for Communication

Engineering
Subject Code: ECE7419

By

Dr. RAM SEWAK SINGH

Associate Professor

Electronics and Communication Engineering Department

School of Electrical Engineering and Computing
Adama Science and Technology University,
Ethiopia, Po. Box: 1888 1
ASTU
Chapter 5: : 2
Sequence Models

ASTU
 Lecture Overview 3

Problem With ML,ANN and CNN

 Recurrent Neural Networks (RNN) for sequences

Backpropagation Through Time

Vanishing and Exploding Gradients and Remedies

RNNs using Long Short-Term Memory (LSTM)

ASTU
 4

ASTU
 5

ANN

ASTU
 Recurrent Neural Network 6

A recurrent neural network (RNN) is a deep learning structure that uses past
information to improve the performance of the network on current and future
inputs. What makes an RNN unique is that the network contains a hidden state and
loops.

The looping structure allows the network to store past information in the hidden
state and operate on sequences.

A recurrent neural network (RNN) is a network architecture for deep learning

that predicts on time-series or sequential data.

ASTU
RNNs are particularly effective for working with sequential data that varies in
7
length and solving problems such as natural signal classification, language
processing, and video analysis.

What a sequence really is?

Data inside a sequence are non i.i.d. ◦Identically, independently

distributed.

The next “word” depends on the previous “words”

Ideally on all of them.

We need context, and we need memory!

How to model context and memory ?

ASTU
 8

ASTU
 9

ASTU
 10

ASTU
 11

ASTU
 Mathematical Model
12

Example: Predict Sequence of Characters

ASTU
 13

ASTU
 14

ASTU
 15

ASTU
 16

ASTU
 Problem With RNN
17

ASTU
 18

ASTU
 19

ASTU
LSTM Networks (Long Short Term Memory) networks
20
Long Short Term Memory networks: usually just called “LSTMs” – are a special
kind of RNN, capable of learning long-term dependencies.

 LSTMs are explicitly designed to avoid the long-term dependency problem.

Remembering information for long periods of time is practically their default
behavior, not something they struggle to learn!

All recurrent neural networks have the form of a chain of repeating modules of
neural network. In standard RNNs, this repeating module will have a very simple
structure, such as a single tanh layer.

ASTU
LSTMs also have this chain like structure, but the repeating module has a different
21
structure. Instead of having a single neural network layer, there are four, interacting
in a very special way.

ASTU
(1) Memory Cell
22
(2) Forget Cell (1)
(3) I/P Gate (4)
(4) O/P Gate

(2) (3)

ASTU
In the above diagram, each line carries an entire vector, from the output of one 23
node
to the inputs of others.

The pink circles represent pointwise operations, like vector addition, while the
yellow boxes are learned neural network layers.

Lines merging denote concatenation, while a line forking denote its content being
copied and the copies going to different locations.

The Core Idea Behind LSTMs

The key to LSTMs is the cell state, the horizontal line running through the top of the
diagram. The cell state is kind of like a conveyor belt. It runs straight down the entire
chain, with only some minor linear interactions. It’s very easy for information to just
flow along it unchanged.

ASTU
 24

The LSTM does have the ability to remove or add information to the cell state,
carefully regulated by structures called gates.

Gates are a way to optionally let information through. They are composed out of a
sigmoid neural net layer and a pointwise multiplication operation.

ASTU
Step-by-Step LSTM Walk Through 25

ASTU
 26

ASTU
 27

ASTU
 28

ASTU
Variants on Long Short Term Memory
29

ASTU
 Autoencoders 30

Autoencoders are a specific type of feed forward neural networks where the input is
the same as the output. They compress the input into a lower-dimensional code and
then reconstruct the output from this representation.

The code is a compact “summary” or “compression” of the input, also called

the latent-space representation.

An autoencoder consists of 3 components: Encoder, Code and Decoder. The encoder

compresses the input and produces the code, the decoder then reconstructs the input
only using this code.

ASTU
Autoencoders are mainly a dimensionality reduction (or compression) algorithm with
31
a couple of important properties:

Data-specific: Autoencoders are only able to meaningfully compress data similar to

what they have been trained on. Since they learn features specific for the given training
data, they are different than a standard data compression algorithm like Gzip.

Lossy: The output of the autoencoder will not be exactly the same as the input, it will
be a close but degraded representation. If you want lossless compression they are not
the way to go.

Unsupervised: To train an autoencoder we don’t need to do anything fancy, just throw

the raw input data at it. Autoencoders are considered an unsupervised learning
technique since they don’t need explicit labels to train on. But to be more precise they
are self-supervised because they generate their own labels from the training data.

ASTU
Architecture
32

ASTU
 33

ASTU
 34

ASTU
 Applications, including anomaly detection, text generation, image generation, image
denoising, and digital communications. 35
 Autoencoders will naturally ignore any input noise as the encoder is trained. This
feature is ideal for removing noise or detecting anomalies when the inputs and
outputs are compared (see Figures 2 and 3).

ASTU
 The latent representation can also be used to generate synthetic data. For example,
36
you can automatically create realistic looking handwriting or phrases of text (Figure
4).

Time series-based auto encoders can also be used to detect anomalies in signal data.
For example, in predictive maintenance, an auto encoder can be trained on normal
operating data from an industrial machine (Figure 5).

Figure 5: Training on normal operating data for predictive maintenance.

 The trained auto encoder is then tested on new incoming data. A large variation from
the autoencoder’s output indicates an abnormal operation, which could require 37
investigation (Figure 6).

ASTU
 38

Thank you for your attention!!

ASTU