Deep Learning

Course Code: WID 3011

By

Amarachukwu Felix. E (PhD)

Email: [email protected]

Amara Felix
 Introduction to Deep Learning
Topics and Activities
Week 1: Introduction to Deep Learning
Week 2: Vanilla Neural Networks I
Week 3: Vanilla Neural Networks II
Week 4: Multi-layer Perceptron I
Week 5: Multi-layer Perceptron II
Week 6: Convolutional Neural Networks I
Week 7: Convolutional Neural Networks II
Week 8: Recurrent Neural Network I
Week 9: Recurrent Neural Network II
Week 10: Generative Adversarial Network I
Week 11: Generative Adversarial Network II
Week 12: Deep Reinforcement Learning
Week 13: Good Practise in Deep Learning
Week 14: Deep Learning Project Presentation
 Introduction to Deep Learning

Learning Outcome
1. What is Deep Learning?
2. Why is Deep Learning Important?
3. Differentiate between a simple Neural Network and Deep Neural Network
4. Deep Learning techniques
-CNN
-RNN
-Transformer architecture
5. Conclusion
 Introduction to Deep Learning
Key elements where the magic of deep learning takes place:

 Neural Networks Architecture

The architecture of deep neural networks, characterized by multiple layers of interconnected

neurons, allows for the modeling of intricate hierarchical patterns. Deeper layers capture
increasingly abstract and complex representations.

 Activation Functions

These introduce non-linearities into the network. Without them, no matter how deep the
network, it would only be able to model linear relationships. Activation functions enable the
network to capture non-linear patterns and relationships in the data.
 Introduction to Deep Learning
Key elements where the magic of deep learning takes place:

 Backpropagation

This is the cornerstone algorithm for training deep networks. By calculating the gradient of the
loss with respect to each weight and iteratively adjusting these weights, networks learn to make
better predictions over time.

 Large Datasets

One of the significant reasons deep learning models have become so powerful is the availability of
massive datasets. These datasets provide the necessary examples for networks to learn intricate
patterns.
 Introduction to Deep Learning
Key elements where the magic of deep learning takes place:

 Computational Power

The rise of GPU computing has played a pivotal role in the current success of deep learning. GPUs
can perform parallel computations efficiently, making it feasible to train large and deep networks
in a reasonable amount of time

 Regularization Techniques

Techniques like dropout, weight decay, and batch normalization prevent overfitting and help
models generalize better, especially when the networks are deep.
 Introduction to Deep Learning
Key elements where the magic of deep learning takes place:

 Transfer Learning

This involves using pre-trained models on large datasets and fine-tuning them for specific tasks. It
allows for achieving high accuracy even with smaller datasets by leveraging knowledge captured in
previously trained networks.

(For instance, one of the most common uses of transfer learning is in image classification tasks
using pre-trained neural networks. These networks have been trained on large datasets like
ImageNet and have learned useful features from these datasets. When we employ transfer
learning, we can leverage these learned features without starting the training process from
scratch.)
 Introduction to Deep Learning
Key elements where the magic of deep learning takes place:

 Representation Learning

Unlike traditional methods that rely heavily on feature engineering, deep learning models learn
representations from the data automatically. This self-learned feature extraction often surpasses
manually designed features.

(For instance, Deep Learning, especially with Convolutional Neural Networks (CNNs), is a classic
example of representation learning. In image classification tasks using CNNs, the initial layers learn
low-level features (like edges and textures), and as you go deeper into the network, the layers
capture higher-level, abstract features specific to the classes in the dataset.
 Introduction to Deep Learning
Key elements where the magic of deep learning takes place:

 End-to-End Learning

Deep learning models can learn to map raw input data to desired outputs without the need for
explicit intermediate steps or processing, further simplifying model design and improving
performance.

(For instance, One of the most famous examples of end-to-end learning is the application of Deep
Learning to the problem of speech recognition. Traditional speech recognition systems involve
multiple steps: feature extraction (like MFCC- Mel-Frequency Cepstral Coefficients), acoustic
modeling, and language modeling. In an end-to-end system, you'd feed the raw audio waveform
(or maybe some minimal
 Introduction to Deep Learning
Key elements where the magic of deep learning takes place:

 End-to-End Learning

Deep learning models can learn to map raw input data to desired outputs without the need for
explicit intermediate steps or processing, further simplifying model design and improving
performance.

(For instance, One of the most famous examples of end-to-end learning is the application of Deep
Learning to the problem of speech recognition. Traditional speech recognition systems involve
multiple steps: feature extraction (like MFCC- Mel-Frequency Cepstral Coefficients), acoustic
modeling, and language modeling. In an end-to-end system, you feed the raw audio waveform (or
maybe some minimal preprocessing like spectrograms) directly to the neural network, which then
produces transcriptions without the traditional intermediate steps.
 Introduction to Deep Learning
Key elements where the magic of deep learning takes place:

 Innovative Architectures

Beyond standard feedforward networks, architectures like Convolutional Neural Networks (CNNs)
for image tasks and Recurrent Neural Networks (RNNs) for sequential data have paved the way for
state-of-the-art results in various domains.
 Introduction to Deep Learning

NNs contain two phases;

1.Forward Propagation
2.Back Propagation

 During the training of a NN you perform both of these phases. When you would like the NN to make a
prediction for an unlabeled example, you would simply perform forward propagation.

 Back propagation is really the magic that allows NN to perform so well for tasks that are traditionally very
difficult for computers to carry out.
Imagine that we are tasked to write a computer program that can identify handwritten digits. Each
image we receive will be 28 x 28 pixels and we will also have access to the correct label for that
image. The first thing we do is set up our neural network as in the figure below.
 Introduction to Deep Learning

AI is a branch of computer science that aims to create machines/systems that have

Connection between fields the ability to imitate intelligent human behavior. E.g. problem-solving, understanding natural
language, recognizing patterns, and making decisions

-ML is a subset of AI that focuses on the development of algorithms and statistical

models.
-Rely on patterns and inference. specifically concentrates on the use of data and
algorithms to simulate the learning process
DL is a subset of ML that focuses on algorithms inspired by the structure and function
of the brain called artificial neural networks. While neural networks have been around
for a while, the term "deep" refers to the number of layers in the network. Modern
networks can be deep, meaning they can have tens or even hundreds of layers.
 Deep Learning techniques

-Convolutional Neural Networks (CNNs) for image tasks.

•Key Components:
• Convolutional Layers: Extract features by sliding a small window (or filter) over the
input data.
• Pooling Layers: Reduce the spatial size, making the network faster and more robust.
• Fully Connected Layers: Classify the extracted features into various classes.
•Applications: Image classification, object detection, facial recognition, etc.

•Purpose: Primarily designed for

image processing.

•Specialty: They automatically and

adaptively learn spatial hierarchies
of features from input images.

Source: Convolutional Neural Network: An Overview (analyticsvidhya.com)

 The Deep-dive in Deep Learning

-Recurrent Neural Networks (RNNs) for sequential data

Purpose: Designed to handle sequential data.
Specialty: They have a "memory" which captures information about previous steps in the sequence. This makes
them suitable for tasks where context from earlier in the sequence is needed to understand the current input.

Challenge: They can struggle with long sequences due to the vanishing (or exploding) gradient problem

To overcome the above challenge:

•Types
• LSTM (Long Short-Term Memory): A type of RNN designed to remember long sequences without losing
track of the context.
• GRU (Gated Recurrent Units): A simplified version of LSTM with fewer parameters.

They incorporate mechanisms called gates to control the flow of information, making them more capable of
learning from long-term dependencies.

•Applications: Time series forecasting, machine translation, speech recognition, etc.

 Transformer Architectures

•Aim: Revolutionized Natural Language Processing (NLP) tasks.

•Specialty: Unlike RNNs, transformers can pay selective attention to different parts of the input data, allowing
them to handle long sequences more effectively.
•Key Components:
• Attention Mechanisms: Allows the model to focus on different parts of the input for different tasks. The
self-attention mechanism lets it consider other words in the input sentence when encoding a particular
word.
• Positional Encodings: Since transformers do not process data in order (like RNNs), they need
information about the position of words in a sequence.

•Popular Models:
• BERT (Bidirectional Encoder Representations from Transformers): Designed to understand the
context of words in search queries or other text by looking at the words before and after it.
• GPT (Generative Pre-trained Transformer): A model for generating human-like text by predicting the
next word in a sequence.

•Applications: Text classification, machine translation, question-answering systems, and even chatbots.
 Conclusion

 Deep learning, a subset of machine learning, employs multi-layered neural networks to analyze various types of data.

 Offers unprecedented accuracy and automation capabilities.

 NNs contain two phases; Forward Propagation and Back Propagation

 Its architectures, particularly Convolutional Neural Networks (CNNs) for image processing and Recurrent Neural
Networks (RNNs) for sequential data, have revolutionized fields ranging from computer vision to natural language
processing.

 Leveraging vast datasets and computational power, deep learning techniques have enabled breakthroughs in
applications such as image and speech recognition, medical diagnosis, financial forecasting, and autonomous vehicles,
solidifying its transformative role in today's technology landscape.
 References

1. Nielsen MA. Neural networks and deep learning. San Francisco, CA, USA: Determination
press; 2015 Sep 25.
2. Neural networks and deep learning