Neural Networks

Intro to AI and Data Science

NGN 112 – Fall 2024

Amer S. Zakaria
Department of Electrical Engineering
College of Engineering

American University of Sharjah

Prepared by Dr. Usman Tariq, ELE

Some slides adapted from Ethem Alpeydin, Geoffrey Hinton, with Nitish
Srivastava and Kevin Swersky

Last Updated on: 18th of Nov. 2024

 Table of Content
2

Brain vs. Computer

Neurons

Neural Network

Forward and Backpropagation Operations

Implementation in Python
 Brain versus Computer
3

 The brain is evolved to do pattern recognition

 Computers are designed to solve logic and
arithmetic problems
 Can we model Neural Networks in computers to
make them artificially intelligent?
 Neuron: Basic Brain Processor
4

 Neurons are nerve cells

 transmit
signals to and from the brain
 speed of transmission: around 200 mph (322 kph)

 Each neuron cell

 communicates to anywhere from 1000 to 10,000 neurons,
muscle cells, glands, so on
 Neuron: Basic Brain Processor
5

 Number of neurons in our brain

 around 86 billion neurons
 Most neurons a person is ever going to have, are
already present at birth
 they make and break connections while learning
 Structure of a Neuron
6

 Main components of a neuron

 Cell body: holds DNA information in the nucleus
 Dendrites: branches to connect a neuron to other neurons. Each neuron
may have thousands of dendrites. Dendrites are usually short.
 Axon: a long structure that splits into possibly thousands of branches at
the end. Each neuron typically has one axon. It may be up to 1 m long.
 Neuron in Action (simplified)
7

 Input : neuron collects signals from other neurons through

dendrites, may have thousands of dendrites
 Processor: Signals are accumulated and processed by the cell
body
 Output: If the strength of incoming signals is large enough, the
cell body sends a signal (a spike of electrical activity) to the
axon
 Neural Network
8
 Modeling a Neuron
9

 Input and output layers

 y = 𝑓(𝑤0 + σ𝑖=1 𝑥𝑖 𝑤𝑖 ) (Between the brackets is similar to the
regression model)
 Some books use 𝑏 in place of 𝑤0.
 Depending on function 𝑓, we can have different kinds of “neurons”.
 Different kinds of Neurons
10

 What are the different kinds of neurons?

Linear Neuron Sigmoid Neuron

Binary Threshold Neuron Rectified Linear Unit (ReLU) Neuron

 Linear Neurons
11

 These are simple but computationally limited, let

bias ith input

𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
𝑖=1 weight on
weighted
sum ith input
index over
input connections

 Suppose 𝑧 = 𝑤0 + σ𝑖=1 𝑥𝑖 𝑤𝑖 , then in the case

of linear neuron
𝑦=𝑓 𝑧 =𝑧
 Linear Neurons
12

 These are simple but computationally limited

𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
𝑖 𝑦
0
𝑦=𝑓 𝑧 =𝑧
0
𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
𝑖
 Linear Neurons: Example

13
 Linear Neurons: Example

14
 Linear Neurons: Example

15
 Linear Neurons: Example

16
 Linear Neurons: Example

17
 Sigmoid Neurons: Used for Classification
18

 These give a real-valued

(float number) output that is
a smooth and bounded 𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
function of their total input. 𝑖
1
 The output is numbers in the 𝑦=
range [0,1] 1 + 𝑒 −𝑧
1

y 0.5

0
0 z
 Sigmoid Neurons: Example

19
 Sigmoid Neurons: Example

20
 Sigmoid Neurons: Example

21
 Sigmoid Neurons: Example

22
 Sigmoid Neurons: Example

23
 Using Sigmoid Neurons for Classification
24

 One way to use sigmoid neurons for classification is to provide

a feature vector of a sample at the input of the neuron.
 Then you have a look at the output of the neuron; if the output
is greater than or equal to 0.5, you say the Class is 1;
otherwise, you say Class is 0.
 This is a very simple case for binary classification (when you
have two classes).
 You can encode one of the class labels as ‘0’.
26
27
28
29
30
31
32
33
34
35
36
37
 Neural Network
38
 Modeling a Neural Network
39

Bias in a neural network is

like adjusting the vertical
position (y-intercept) of
the curve

Output layer

Input layer

Hidden layer
 Operation of Neural Networks
40

In the neural networks, there are two


modes of operation
Forwardoperation (for inference)
Backward operation (for training)
 Forward Operation
41

 In the forward operation, you provide the feature vector representing an instance at the
input of the network.
 The numbers in the feature vector are then multiplied by the weights of the network
(input to hidden layer weights), and bias is added for each of the nodes in the hidden
layer.
 The results are then the “activation” (weighted sum) of each of the neurons in the
hidden layer. These activations are passed through a function (activation function, e.g.,
sigmoid, ReLU, etc.), and then the output of the neurons in the hidden layer becomes the
input to the neuron(s) in the proceeding layers.
 These outputs are then multiplied by the hidden-to-output layer weights, biases are
added, and this makes up the activation(s) of the output neuron(s). These activations(s) are
then passed through the activation function of the output layer (e.g., sigmoid for
classification or linear for regression (enough at your level)).
 For classification, if the output value is above or equal to 0.5, you say that you detected
Class 1. Otherwise, you say that you detected Class 0.

Note: You also compute the cost function value to see if your network is doing a good job.
One example cost function can be the mean squared error computed over the training,
validation, or testing examples.
If e.g., one input instance is from Class 1 (in a binary classification problem) and the output
of the network is 0.8. Then the squared error is (1-0.8)2 for this instance
 Forward Operation
42

https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6
43

https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6
 Backward Operation (Backpropagation)
44

 This is used for updating the weights (and biases) of the

network, to improve the performance on the training set.
This is also known as the Backpropagation Algorithm.
 You compute the cost function value to see if your network
performs well and is doing a good job.
 One example cost function can be the mean squared
error computed over the training examples. If, e.g., one
input instance is from Class 1 (in a binary classification
problem) and the network output of the network is 0.8.
Then, the squared error is (1-0.8)2 for this instance.
 You can compute the average of this error over all
training examples (or part of them) and then update the
weights (and biases) of the network to reduce the cost
function (error function) value.
 Backpropagation
45
46
NNs Classification in Python: Example
47

# Generating some data for classification example

from sklearn.datasets import make_classification
X, y = make_classification(n_samples=100, random_state=1)

# Splitting data into training and testing

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

# Initializing a neural network (NN) classifier

from sklearn.neural_network import MLPClassifier
clf = MLPClassifier(random_state=1, max_iter=300)
# clf = MLPClassifier(hidden_layer_sizes=(5,20), random_state=1, max_iter=10000, activation = 'logistic')
# hidden_layer_sizes=(5, 20): This means 2 hidden layers with 5 and 20 neurons, respectively
# hidden_layer_sizes=(5,) means that there is one hidden layer and 5 neurons in that layer
# hidden_layer_sizes=(5,3) means that there are 5 neurons in the first hidden layer and 3 in the next
# hidden_layer_sizes=(10,7,5): 3 hidden layers, 10 neurons in 1st, 7 in 2nd layer, and 5 in 3rd layer
# The default is 1 hidden layer with 100 neurons.
# The default activation function is 'relu'. To change it to sigmoid, option: activation = 'logistic'
NNs Classification in Python: Example (cont.)
48

# Training/fitting the model

clf.fit(X_train, y_train)

# Using the model for prediction

y_predict = clf.predict(X_test)

# Evaluating the model performance

from sklearn import metrics
acc = metrics.accuracy_score(y_test, y_predict)
print("Accuracy score is ", acc)
# Output: Accuracy score is 0.88

# More insight about prediction probability of the NN classifier

clf.predict_proba(X_test[:1])
# Output: array([[0.03838405, 0.96161595]])#probability of 1st sample: class 0 or 1
NNs Regression in Python : Example
49

# Generating some data for regression example

from sklearn.datasets import make_regression
X, y = make_regression(n_samples=200, random_state=1)

# Splitting data into training and testing

from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=1)

# Initializing a regressor
from sklearn.neural_network import MLPRegressor
regr = MLPRegressor(max_iter=10000, random_state=1)
NNs Regression in Python : Example (cont.)
50

# Training/fitting the model

regr.fit(X_train, y_train)

# Using the model for predictions

y_pred = regr.predict(X_test)

# Evalulating the performance

from sklearn.metrics import mean_squared_error, r2_score
mse_test = mean_squared_error(y_test, y_pred)
print("The MSE is : ", mse_test)
r2_test = r2_score(y_test, y_pred)
print("The R^2 is : ", r2_test)
 Case Study: Iris Database
51

Solve the Iris Classification Example using the Neural

Network classifier (MLPClassifier) and compare results
accuracy to the use of conventional classifiers.

Input
Sepal Length, Sepal Output
NN Classifier
Width, Petal Length, Iris Species
Petal Width
 TensorFlow Playground
52

 TensorFlow Playground is an interactive, web-based

visualization tool developed by the TensorFlow team at Google.
 It allows users to explore and understand neural networks in

real time. You can manipulate network architectures, adjust

hyperparameters, and visualize how changes impact the
model’s behavior.
https://playground.tensorflow.org/
 Learning Outcomes
53

Upon completion of the course, students will be able to:

1. Identify the importance of AI and Data Science for society
2. Perform data loading, preprocessing, summarization and
visualization
3. Apply machine learning methods to solve basic regression
and classification problems
4. Apply artificial neural networks to solve simple
engineering problems
5. Implement basic data science and machine learning tasks
using programming tools
 Extra: Binary Threshold Neurons
54

 McCulloch-Pitts (1943):
 Firstcompute a weighted sum of the inputs. 1
 Then send out a fixed size spike of activity if

output
the weighted sum exceeds a threshold.
 McCulloch and Pitts thought that each spike is
like the truth value of a proposition and each 0
threshold
neuron combines truth values to compute the
truth value of another proposition! weighted input
 Extra: Binary Threshold Neurons
55

𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖 1
𝑖
𝑦
1 if 𝑧 ≥ 0
𝑦=ቊ
0 otherwise 0
0 𝑧
 Extra: Binary Threshold Neurons: Example

56
 Extra: Binary Threshold Neurons: Example

57
 Extra: Binary Threshold Neurons: Example

58
 Extra: Binary Threshold Neurons: Example

59
 Extra: Binary Threshold Neurons: Example

60
 Extra: Rectified Linear (ReLU) Neurons
(sometimes called Linear Threshold Neurons)
61

They compute a linear weighted sum of their inputs.

The output is a non-linear function of the total input.
It is as if the binary threshold neuron was multiplied
by the linear neuron.

𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
𝑖

𝑧 if 𝑧 ≥ 0
𝑦
𝑦=ቊ 0 𝑧
0 otherwise
 Extra: ReLU Neurons: Example

62
 Extra: ReLU Neurons: Example

63
 Extra: ReLU Neurons: Example

64
 Extra: ReLU Neurons: Example

65
 Extra: ReLU Neurons: Example

66