Neural Networks

Intro to AI and Data Science

NGN 112 – Fall 2024

Ammar Hasan
Department of Electrical Engineering
College of Engineering

American University of Sharjah

Prepared by Dr. Usman Tariq, ELE

Some slides adapted from Ethem Alpeydin, Geoffrey Hinton, with Nitish
Srivastava and Kevin Swersky

Last Updated on: 22nd of August 2024

 Table of Content
2

Brain vs. Computer

Artificial Neurons

Artificial Neural Network

Forward and Backpropagation Operations

Implementation in Python
 Brain versus Computer
3

 The brain is evolved to do pattern recognition

 Computers are designed to solve logic and
arithmetic problems
 Can we model Neural Networks in computers to
make them artificially intelligent?
 Biological Neuron: Basic Brain Processor
4

 Neurons are nerve cells

 transmit
signals to and from brain
 speed of transmission: around 200mph

 Each neuron cell

 communicates to anywhere from 1000 to10,000 neurons,
muscle cells, glands, so on
 Biological Neuron: Basic Brain Processor
5

 Number of neurons in our brain

 around 86 billion neurons
 Most neurons a person is ever going to have, are
already present at birth
 they make and break connections while learning
 Structure of a Biological Neuron
6

 Main components of a neuron

 Cellbody: holds DNA information in nucleus
 Dendrites: may have thousands of dendrites, usually short
 Axon: a long structure, which splits in possibly thousands
branches at the end. May be up to 1 meter long
 Biological Neuron in Action (simplified)
7

 Input : neuron collects signals from other neurons through

dendrites, may have thousands of dendrites
 Processor: Signals are accumulated and processed by the cell
body
 Output: If the strength of incoming signals is large enough, the
cell body sends a signal (a spike of electrical activity) to the
axon
8

Artificial Neurons
 Modeling a Neuron
9

 Input and output layers

 y = f( σ𝑖 𝑥𝑖 𝑤𝑖 + 𝑤0 ) (Between the brackets is similar to the
regression model)
 Some books use b in place of w0.
 Depending on function f, we can have different kinds of “neurons”.
 Different kinds of Neurons
10

 What are the different kinds of neurons?

Linear Neuron Sigmoid Neuron

Binary Threshold Neuron Rectified Linear Unit (ReLU) Neuron

 Linear Neurons
11

 These are simple but computationally limited, let

bias ith input

𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
𝑖 weight on
weighted
sum ith input
index over
input connections

 Suppose 𝑧 = σ𝑖 𝑥𝑖 𝑤𝑖 + 𝑤0 , then in the case of

linear neuron
𝑦=𝑓 𝑧 =𝑧
 Linear Neurons
12

 These are simple but computationally limited

𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
𝑖 𝑦
0
𝑦=𝑓 𝑧 =𝑧
0
𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
𝑖
 Linear Neurons: Example

13
 Linear Neurons: Example

14
 Linear Neurons: Example

15
 Linear Neurons: Example

16
 Linear Neurons: Example

17
 Sigmoid Neurons: Used for Classification
18

 These give a real-valued

(float number) output that is
a smooth and bounded 𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
function of their total input. 𝑖
1
 The output is numbers in the 𝑦=
range [0,1] 1 + 𝑒 −𝑧
1

y 0.5

0
0 z
 Sigmoid Neurons: Example

19
 Sigmoid Neurons: Example

20
 Sigmoid Neurons: Example

21
 Sigmoid Neurons: Example

22
 Sigmoid Neurons: Example

23
 Using Sigmoid Neurons for Classification
24

 One way to use sigmoid neurons for classification is to provide

a feature vector of a sample at the input of the neuron.
 Then you have a look at the output of the neuron; if the output
is greater than or equal to 0.5, you say the Class is 1;
otherwise, you say Class is 0.
 This is a very simple case for binary classification (when you
have two classes).
 You can encode one of the class labels as ‘0’.
26
27
28
29
30
31
32
33
34
35
36
37
38

Neural Network
 Neural Network
39
 Modeling a Neural Network
40

Bias in a neural network is

like adjusting the vertical
position (y-intercept) of
the curve

Output layer

Input layer

Hidden layer
 Operation of Neural Networks
41

In the neural networks, there are two


modes of operation
Forwardoperation (for inference)
Backward operation (for training)
42

Forward and Backpropagation Operations

 Forward operation
43

 In the forward operation, you provide the feature vector

representing an instance at the input of the network.

 The numbers in the feature vector are then multiplied by the

weights of the network (input to hidden layer weights) and bias
is added for each of the node in the hidden layer.

 The results is then the “activation” (weighted sum) of each of

the neurons in the hidden layer. These activations are passed
through a function (activation function; e.g., sigmoid, relu, etc.)
and then the output of the neurons in the hidden layer becomes
the input to the neuron(s) in the proceeding layers.

 ….
 Forward operation (continued)
44

 These outputs are then multiplied by the hidden-to-output layer

weights, biases are added, and this makes up the activation(s) of the
output neuron(s). These activations(s) are then passed through the
activation function of the output layer (e.g., sigmoid for classification
or linear for regression (enough at your level)).

 For classification, if the output value is above or equal to 0.5, you

say that you detected Class 1, otherwise you say that you detected
Class 0.

Note: You also compute the cost function value to see if your network
is doing a good job. One example cost function can be the mean
squared error computed over the training, validation or testing
examples.
If e.g., one input instance is from Class 1 (in a binary classification
problem) and the output of the network is 0.8. Then the squared error
is (1-0.8)2 for this instance
 Forward Operation
45

https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6
46

https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6
 Backward Operation (Backpropagation)
47

 This is used for updating the weights (and biases) of the

network, to improve the performance on the training set.
This is also known as the Backpropagation Algorithm.
 You compute the cost function value to see if your network
performs well and is doing a good job.
 One example cost function can be the mean squared
error computed over the training examples. If, e.g., one
input instance is from Class 1 (in a binary classification
problem) and the network output of the network is 0.8.
Then, the squared error is (1-0.8)2 for this instance.
 You can compute the average of this error over all
training examples (or part of them) and then update the
weights (and biases) of the network to reduce the cost
function (error function) value.
 Backpropagation
48
49
50

Implementation in Python
NNs Classification in Python: Example
51

# Step 1: Generating some data for classification example

from sklearn.datasets import make_classification
X, y = make_classification(n_samples=100, random_state=1)

# Step 2: Splitting data into training and testing

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

# Step 3: Initializing a neural network (NN) classifier

from sklearn.neural_network import MLPClassifier
clf = MLPClassifier(random_state=1, max_iter=300)
# clf = MLPClassifier(hidden_layer_sizes=(5,20), random_state=1, max_iter=10000, activation = 'logistic')
# hidden_layer_sizes=(5, 20): This means 2 hidden layers with 5 and 20 neurons, respectively
# hidden_layer_sizes=(5,) means that there is one hidden layer and 5 neurons in that layer
# hidden_layer_sizes=(5,3) means that there are 5 neurons in the first hidden layer and 3 in the next
# hidden_layer_sizes=(10,7,5): 3 hidden layers, 10 neurons in 1st, 7 in 2nd layer, and 5 in 3rd layer
# The default is 1 hidden layer with 100 neurons.
# The default activation function is 'relu'. To change it to sigmoid, option: activation = 'logistic'
NNs Classification in Python: Example (cont.)
52

# Step 4: Training/fitting the model

clf.fit(X_train, y_train)

# Step 5: Using the model for prediction

y_predict = clf.predict(X_test)

# Step 6: Evaluating the model performance

from sklearn import metrics
acc = metrics.accuracy_score(y_test, y_predict)
print("Accuracy score is ", acc)
# Output: Accuracy score is 0.88

# More insight about prediction probability of the NN classifier

clf.predict_proba(X_test[:1])
# Output: array([[0.03838405, 0.96161595]])
# probability of 1st sample: class 0 or 1
NNs Regression in Python : Example
53

# Step 1: Generating some data for regression example

from sklearn.datasets import make_regression
X, y = make_regression(n_samples=200, random_state=1)
# Step 2: Splitting data into training and testing
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=1)
# Step 3: Initializing a regressor
from sklearn.neural_network import MLPRegressor
regr = MLPRegressor(max_iter=10000, random_state=1)
# Step 4: Training/fitting the model
regr.fit(X_train, y_train)
# Step 5: Using the model for predictions
y_pred = regr.predict(X_test)
# Step 6: Evalulating the performance
from sklearn.metrics import mean_squared_error, r2_score
mse_test = mean_squared_error(y_test, y_pred)
print("The MSE is : ", mse_test)
r2_test = r2_score(y_test, y_pred)
print("The R^2 is : ", r2_test)
 Case Study: Iris Database
54

Solve the Iris Classification Example using the Neural

Network classifier (MLPClassifier) and compare results
accuracy to the use of conventional classifiers.

Input
Sepal Length, Sepal Output
NN Classifier
Width, Petal Length, Iris Species
Petal Width
 TensorFlow Playground
55

 TensorFlow Playground is an interactive, web-based

visualization tool developed by the TensorFlow team at Google.
 It allows users to explore and understand neural networks in

real time. You can manipulate network architectures, adjust

hyperparameters, and visualize how changes impact the
model’s behavior.
https://playground.tensorflow.org/
 Learning Outcomes
56

Upon completion of the course, students will be able to:

1. Identify the importance of AI and Data Science for society
2. Perform data loading, preprocessing, summarization and
visualization
3. Apply machine learning methods to solve basic regression
and classification problems
4. Apply artificial neural networks to solve simple
engineering problems
5. Implement basic data science and machine learning tasks
using programming tools
57

Extra Slides. Not included in syllabus.

Not included in syllabus
Extra: Binary Threshold Neurons
58

 McCulloch-Pitts (1943):
 Firstcompute a weighted sum of the inputs. 1
 Then send out a fixed size spike of activity if

output
the weighted sum exceeds a threshold.
 McCulloch and Pitts thought that each spike is
like the truth value of a proposition and each 0
threshold
neuron combines truth values to compute the
truth value of another proposition! weighted input
Not included in syllabus
Extra: Binary Threshold Neurons
59

𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖 1
𝑖
𝑦
1 if 𝑧 ≥ 0
𝑦=ቊ
0 otherwise 0
0 𝑧
Not included in syllabus
Extra: Binary Threshold Neurons: Example

60
Not included in syllabus
Extra: Binary Threshold Neurons: Example

61
Not included in syllabus
Extra: Binary Threshold Neurons: Example

62
Not included in syllabus
Extra: Binary Threshold Neurons: Example

63
Not included in syllabus
Extra: Binary Threshold Neurons: Example

64
 Extra: inRectified
Not included Linear (ReLU) Neurons
syllabus

(sometimes called Linear Threshold Neurons)

65

They compute a linear weighted sum of their inputs.

The output is a non-linear function of the total input.
It is as if the binary threshold neuron was multiplied
by the linear neuron.

𝑧 = 𝑤0 + ෍ 𝑥𝑖 𝑤𝑖
𝑖

𝑧 if 𝑧 ≥ 0
𝑦
𝑦=ቊ 0 𝑧
0 otherwise
Not included in syllabus
Extra: ReLU Neurons: Example

66
Not included in syllabus
Extra: ReLU Neurons: Example

67
Not included in syllabus
Extra: ReLU Neurons: Example

68
Not included in syllabus
Extra: ReLU Neurons: Example

69
Not included in syllabus
Extra: ReLU Neurons: Example

70