AIS302: ANN (Artificial Neural Networks)

Lecture 1: Introduction
Spring 2025

Dr. Ensaf Hussein

Associate Professor, Artificial Intelligence,
School of Information Technology and Computer Science,
Nile University.
Course learning outcomes
• Understand the principles behind deep learning
• Have practical experience applying deep learning to novel situations

• By the end of the semester, you should be able to

• Look at a problem
• Identify if DL could be a solution
• Apply the appropriate DL model

2
 Prerequisites
AIS301: Machine
Learning

AIS302: Artificial Neural AIS411: Natural Language AIS462: Computational

Networks Processing Intelligence

AIS412: Deep Learning AIS421: NLP Applications

Course tentative schedule
• Deep Neural Networks and Hyperparameter tuning
• Regularization and Optimization
• Sequence Modeling: Recurrent and Recursive Nets
• Sequence models & attention mechanism
• Autoencoders / Applications
• Sequence-to-sequence models
• Attention / Multi-head attention
• Transformers
• Vision Transformers (ViT).

4
 Grading Policy

This course is Project-Based Learning.

• Grading items:
❑ Classwork
❑10% Lab Tasks
❑10% Programming Assignments
❑10% Lecture Quizzes
❑ 20% Midterm Exam
❑ 10 % Project (Follow-Up)
❑Final assessment
❑ 10 % Final Project
❑ 30% Final Exam
•students with less than 30% in final exam will get an F in the course
• Students should attend 75% of lectures and labs to enter the final exam 5
 Resources:

Refernece : https://www.deeplearningbook.org/
 Textbook
• A great book!

• Very recent (and up-to-date!)

• Easy to read and follow
• Very nice figures!
• Neither terribly theoretical nor
extremely about coding!

• Available freely online by author

https://udlbook.github.io/udlbook/
• Many slides are adopted from
content by Simon.

8
 Topics
Deep neural networks
How to train them
How to measure their performance
How to make that performance better

Networks specialized to images

Image classification
Image segmentation

Networks specialized to text

Text generation
ChatGPT
Selective

Generative learning (unsupervised)

based on

Generating random cats!

time

RL using Deep Learning

9
 Warm-Up

https://play.blooket.com/play?id=3123283
2
 What is AI?
•AI aims to simulate intelligent behavior.
3
 What is Machine Learning?

•ML is a subset of AI that learns from data to make decisions.

4
5
 What is Deep Learning?
•Deep Learning (DL) is a type of ML based on deep neural
networks.

What is Deep Neural

Network?
A Deep Neural Network (DNN) is a machine learning model with multiple layers of artificial neurons,
enabling it to learn complex patterns from data. It consists of an input layer, multiple hidden layers,
and an output layer, using backpropagation and gradient descent for training. DNNs are widely used
in image recognition, speech processing, and natural language understanding, powering applications
like self-driving cars, chatbots, and medical diagnostics. 7
 8
Figures from http://udlbook.com
17
 AI is all about deep learning
➢ Yes
➢ No

…...... is fitting mathematical models to observed data

➢ AI
➢ Machine Learning
➢ Deep Learning
18
 Deep learning == Machine learning?
➢ Yes
➢ No

…...... is a type of machine learning

➢ Deep Neural Networks
➢ Supervised Learning
➢ Deep Learning
19
 6
Figures from http://udlbook.com
12
 Supervised learning
• Define a mapping from input to output
• Learn this mapping from paired input/output (labeled) data examples

Model Model
Real world input Real world output
input output

22
 Simple example …
• Predict the height of a child given his/her age.

Model Model
Age of child Height of child
input output

23
 What is a supervised learning model?

Regression

• An equation relating input (age) to output (height)

• Search through family of possible equations to find one that fits training data well

24
Figures from http://udlbook.com
 What is a supervised learning model?

Deep neural networks are just a very flexible family of equations

Fitting deep neural networks = “Deep Learning”
25
Figures from http://udlbook.com
Regression

Deep learning
model

Univariate regression problem (one output, real value)

Fully connected network
26
Figures from http://udlbook.com
 Graph regression

Deep learning
model

Multivariate regression problem (>1 output, real value)

Graph neural network
27
Figures from http://udlbook.com
 Text classification

Deep learning
model

Binary classification problem (two discrete classes)

Transformer network
28
Figures from http://udlbook.com
 Music genre classification

Deep learning
model

Multiclass classification problem (discrete classes, >2 possible values)

Recurrent neural network (RNN)
29
Figures from http://udlbook.com
 Image classification

Deep learning
model

Multiclass classification problem (discrete classes, >2 possible values)

Convolutional network
21
Figures from http://udlbook.com
 Image segmentation

• Multivariate binary classification problem (many outputs, two discrete classes)

• Convolutional encoder-decoder network
 Depth estimation

• Multivariate regression problem (many outputs, continuous)

• Convolutional encoder-decoder network
 Pose estimation

• Multivariate regression problem (many outputs, continuous)

• Convolutional encoder-decoder network
Terms
• Regression = continuous numbers as output
• Classification = discrete classes as output
• Two class and multiclass classification treated differently
• Univariate = one output
• Multivariate = more than one output

34
35
“Given a news article, I want to predict if it is political,
sports, or economical”.
What type of a problem is that?
➢ multivariate classification
➢ univariate regression
➢ binary classification
➢ multivariate regression
➢ multiclass classification
36
Other type of examples …

37
Translation

Deep learning
model

38
Figures from http://udlbook.com
Image captioning

Deep learning
model

39
Figures from http://udlbook.com
Image generation from text

Deep learning
model

40
Figures from http://udlbook.com
What do these examples
have in common?

41
 What do these examples have in common?
• Very complex relationship between input and output
• Sometimes may be many possible valid answers
• But outputs (and sometimes inputs) obey rules

Language obeys grammatical rules Natural images also have “rules”

Learn the “grammar” of the data

from LOTS of unlabeled examples
42
Figures from http://udlbook.com
33
 Unsupervised Learning
• Learning from data without labels
• Clustering
• Finding outliers
• Generating new examples
• Filling in missing data

34
DeepCluster: Deep Clustering for Unsupervised Learning of Visual Features (Caron et al., 2018) 35
Figures from http://udlbook.com
 Unsupervised Learning
• Learning about a dataset without labels
• e.g., clustering
• Generative models can create examples
• e.g., generative adversarial networks

36
 Unsupervised Learning
• Learning about a dataset without labels
• e.g., clustering
• Generative models can create examples
• e.g., generative adversarial networks
• PGMs learn distribution over data
• e.g., variational autoencoders,
• e.g., normalizing flows,
• e.g., diffusion models

47
Figures from http://udlbook.com
Generative models

48
Figures from http://udlbook.com
Generative models

49
Figures from http://udlbook.com
Generative models

50
Figures from http://udlbook.com
 Conditional synthesis

Original image Removed regions Generated regions

51
Figures from http://udlbook.com
Figures from http://udlbook.com 42
ChatGPT
53
54
45
 Reinforcement learning

• Build an agent which lives in a world and can perform certain actions
at each time step.
• Goal: take actions to change the state so that you receive rewards
• You don’t receive any data – you have to explore the environment
yourself to gather data as you go.

46
 Example: chess
• States are valid states of the chess board
• Actions at a given time are valid possible moves
• Positive rewards for taking pieces, negative rewards for losing
them

57
Figures from http://udlbook.com
 Example: chess
• States are valid states of the chess board
• Actions at a given time are valid possible moves
• Positive rewards for taking pieces, negative rewards for losing
them

58
Figures from http://udlbook.com
 Why is this difficult?
• Stochastic
• Make the same move twice, the opponent might not do the same thing
• Rewards also stochastic (opponent does or doesn’t take your piece)
• Temporal credit assignment problem
• Did we get the reward because of this move? Or because we made good
tactical decisions somewhere in the past?
• Exploration-exploitation trade-off
• If we found a good opening, should we use this?
• Or should we try other things, hoping for something better?

59
 Landmarks in (Machine) Deep Learning
• 1958 Perceptron (Simple `neural’ model)
• 1986 Backpropagation (Practical Deep Neural networks)
• 1989 Convolutional networks (Supervised learning)
• 2012 AlexNet Image classification (Supervised learning)
• 2014 Generative adversarial networks (Unsupervised learning)
• 2014 Deep Q-Learning -- Atari games (Reinforcement learning)
• 2016 AlphaGo (Reinforcement learning)
• 2017 Machine translation (Supervised learning)
• 2019 Language models ((Un)supervised learning)
• 2022 Dall-E2 Image synthesis from text prompts ((Un)supervised learning)
• 2022 ChatGPT ((Un)supervised learning)
• 2023 GPT4 Multimodal model ((Un)supervised learning)
60
2018 Turing award winners

61
 Where are we going?
• Supervised learning (overview with regression example)
• Shallow neural networks (a more flexible model)
• Deep neural networks (an even more flexible model)
• Loss functions (guiding the training)
• How to train neural networks (gradient descent and variants)
• How to measure performance of neural networks (generalization)