Neural Network
Output
Input
Module 5 Neural Networks: Gradient-Based Operations
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then Neurons
passes the sum through the activation function to the other neurons.
Drawing by Santiago Ramón y Cajal (1899) of neurons in the pigeon cerebellum
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Neuron or node
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Input Signal
Neuron or node
2
Input Signal
Input Signal
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Input Signal
Neuron or node
2 2 Output
Input Signal
Input Signal
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Input Signal
Neuron or node
2 2 Output Signal or Value
Input Signal
Input Signal
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Input Signal
Neuron or node
2 2 Output Signal or Value
Input Signal
Input Signal
Synapse
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Input Signal 1
These are independent Neuron or node
variable
2 2 Output Signal or Value
Input Signal 2
These are independent
variable
Input Signal 3
These are independent Synapse
variable
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Input Signal 1
These are independent Neuron or node
variable
2 2 Output Signal or Value
Input Signal 2
These are independent
variable
Input Signal 3
These are independent
variable
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Input Signal 1
These are independent Neuron or node Output Signal or Value
variable
Categorical
2 Continues
2
Input Signal 2 Binary
These are independent
variable
Input Signal 3
These are independent
variable
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Output 1
2
Input Signal 1
These are independent Neuron or node
variable
Categorical
2 Continues Output Signal or Value
2
Input Signal 2 Binary
These are independent Output 2
variable
Input Signal 3 2 Output 3
These are independent
variable
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Output 1
2
Input Signal 1
These are independent Neuron or node
variable
Categorical
2 Continues Output Signal or Value
2
Input Signal 2 Binary
These are independent Output 2
variable
Input Signal 3 2 Output 3
These are independent
variable
Same Obervations
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Input Signal 1
w1
These are independent Neuron or node
variable
w2 2 2 Output Signal or Value
Input Signal 2
These are independent W
variable …
Input Signal 3 Weight
These are independent
variable
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Neuron or node
Input Signal 1
These are independent w1
variable
1st Step
w2
Input Signal 2 2 Output Signal or Value
These are independent W
variable …
Input Signal 3 Weight
These are independent
variable
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Activation Function
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Neuron or node
Input Signal 1
These are independent w1
variable 1st Step
2nd Step
w2
Input Signal 2 1rd Step 2 Output Signal or Value
These are independent W
variable …
Input Signal 3 Weight
These are independent
variable
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Activation Function
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.
Neuron or node
Input Signal 1
These are independent w1
variable 1st Step
2nd Step
w2
Input Signal 2 1rd Step 2 Output Signal or Value
These are independent W
variable …
Input Signal 3 Weight
These are independent
variable
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Threshold Activation Function
A binary step function is a threshold-based activation function. If the input value
is above or below a certain threshold, the neuron is activated and sends exactly
the same signal to the next layer.
0 x
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Sigmoid Activation Function
Smooth gradient, preventing “jumps” in output values.
Output values bound between 0 and 1, normalizing the output of each neuron.
Clear predictions—For X above 2 or below -2, tends to bring the Y value (the
prediction) to the edge of the curve, very close to 1 or 0. This enables clear
predictions.
0 x
[Link]
Module 5 Neural Networks: Gradient-Based Operations
ReLU Activation Function
•Computationally efficient—allows the network to converge very
quickly
•Non-linear—although it looks like a linear function, ReLU has a
derivative function and allows for back propagation
0 x
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Hyperbolic Tangent Activation Function
•Zero centered—making it easier to model inputs that have
strongly negative, neutral, and strongly positive values.
•Otherwise like the Sigmoid function.
0 x
[Link]
Module 5 Neural Networks: Gradient-Based Operations
Activation Function
Input 1 x1
Input 2 x1
Output
layer
Input n x
Input
layer Hidden
layer
Module 5 Neural Networks: Gradient-Based Operations
Working of Neural Network
Input 1
w1 Y / output
w2
Input 2
Wn
Input 3 Output Layer
Input Layer
Module 5 Neural Networks: Gradient-Based Operations
Working of Neural Network
Area x1
y
Bedroom x1
Distance x3
Output Layer
Age x4
Hidden Layer
Input Layer
Module 5 Neural Networks: Gradient-Based Operations
Working of Neural Network
Area x1
y
Bedroom x1
Distance x3
Output Layer
Age x4
Hidden Layer
Input Layer
