Introduction to Neural Networks

Artificial neural networks (ANNs)

Other terms/names
• Connectionist
• Parallel distributed processing
• Neural computation
• Adaptive networks..
 History of ANNs
• 1943-McCulloch & Pitts are generally
recognised as the designers of the first
neural network
• 1949-First learning rule
• 1969-Minsky & Papert - perceptron limitation
- Death of ANN
• 1980’s - Re-emergence of ANN - multi-layer
networks
 ANN
• Artificial neural networks (ANNs) are
computing systems vaguely inspired by
the biological neural networks
• An ANN is based on a collection of connected
units or nodes called artificial neurons which
loosely model the neurons in a biological brain.
 ANN
• Each connection, like the synapses in a
biological brain, can transmit a signal
from one artificial neuron to another.

• An artificial neuron that receives a

signal can process it and then signal
additional artificial neurons connected
to it.
 Brain and Machine

• The Brain
– Pattern Recognition
– Association
– Complexity
– Noise Tolerance

• The Machine
– Calculation
– Precision
– Logic
 The contrast in architecture

• The Von Neumann architecture uses a

single processing unit;
– Tens of millions of operations per second
– Absolute arithmetic precision

• The brain uses many slow unreliable

processors acting in parallel
 Computers vs. Neural Networks
“Standard” Computers Neural Networks

 One CPU Highly parallel

processing

 Fast processing units Slow processing units

 Reliable units Unreliable units

 Static infrastructure Dynamic

 Why Artificial Neural Networks?
 There are two basic reasons why we are interested in building

artificial neural networks (ANNs):

• Technical viewpoint: Some problems such as character

recognition or the prediction of future states of a system

require massively parallel and adaptive processing.

• Biological viewpoint: ANNs can be used to replicate and

simulate components of the human (or animal) brain, thereby

giving us insight into natural information processing.

 Biological Neural Network Vs Artificial Neural Network

Artificial Neural Network

Biological Neural Network

Biological Neural Network Artificial Neural Network

Dendrites Inputs
Cell nucleus Nodes
Synapse Weights

Axon Output
 ANN
• The “building blocks” of neural networks are the
neurons.
• In technical systems, we also refer to them as units or
nodes.
• Basically, each neuron
 receives input from many other neurons.
 changes its internal state (activation) based on the
current input.
 sends one output signal to many other neurons,
possibly including its input neurons (recurrent
network).
The Architecture of ANN
 ANN

• In common ANN implementations, the

signal at a connection between
artificial neurons is a real number, and
the output of each artificial neuron is
computed by some non-linear function
of the sum of its inputs.
• The connections between artificial
neurons are called 'edges'.
 ANN
• Artificial neurons and edges typically
have a weight that adjusts as learning
proceeds.
• The weight increases or decreases the
strength of the signal at a connection.
• Typically, artificial neurons are
aggregated into layers.
 ANN

• Different layers may perform different

kinds of transformations on their inputs.
• Signals travel from the first layer (the
input layer), to the last layer (the output
layer), possibly after traversing the layers
multiple times.
ANN
 Advantages of ANN
• Parallel processing
• Storing data on the entire network
• Capability to work with incomplete
knowledge
• Memory distribution
• Fault tolerance
 Disadvantages of ANN
• Hardware Dependence
• Understanding the network’s operation
• Assured network structure
• Difficulty in presenting the issue to the
network
• The network’s lifetime is unknown
 Applications ANN
• Image Processing and Character recognition
• Forecasting
• Stock Market Prediction.
• Social Media.
• Aerospace.
• Defence.
• Healthcare.
• Signature Verification and Handwriting Analysis
 Example
A four pixel image
Categorize images solid

vertical

diagonal

horizontal
Categorize images solid

vertical

diagonal

horizontal
Categorize images solid

vertical

diagonal

horizontal
Categorize images solid

vertical

diagonal

horizontal
 solid

Categorize images
vertical

diagonal

horizontal
Input neurons
 Pixel brightness

-1.0 -.75 -.50 -.25 0.0 +.25 +.50 +.75 +1.0

 Input vector
.50

0.0

-.75

.75
 A neuron

+
 Sum all the inputs
.50 .50
0.00
-.75
+ .75
0.0 .50
+
.50

-.75

.75
 Weights
.50 .50 x 1.0
1.0 0.00 x 1.0
-.75 x 1.0
1.0
+ .75 x 1.0
0.0 .50
+
1.0
.50

-.75
1.0

.75
 Weights
.50 .50 x -.2
-.2 0.00 x 0.0
-.75 x .8
0.0
+ .75 x -.5
0.0 -1.075
+
.8
-1.075

-.75
-.5

.75
 Weights
.50 .50 x -.2
-.2 0.00 x 0.0
-.75 x .8
0.0
+ .75 x -.5
0.0 -1.075
+
.8
-1.075

-.75
-.5

.75
 Squash the result
.50

0.0
+
1.075 .746

-.75

.75
Sigmoid squashing function
1.0

.5
-2.0 -1.5 -1.0 -.5

.5 1.0 1.5 2.0

-.5

-1.0
Sigmoid squashing function
1.0

.5
-2.0 -1.5 -1.0 -.5

.5 1.0 1.5 2.0

-.5

-1.0

Your number goes in here

Sigmoid squashing function
1.0

.5
-2.0 -1.5 -1.0 -.5

.5 1.0 1.5 2.0

-.5

-1.0
Sigmoid squashing function
The squashed version
comes out here 1.0

.5
-2.0 -1.5 -1.0 -.5

.5 1.0 1.5 2.0

-.5

-1.0
Sigmoid squashing function
1.0

.5
-2.0 -1.5 -1.0 -.5

.5 1.0 1.5 2.0

-.5

-1.0
Sigmoid squashing function
1.0

.5
-2.0 -1.5 -1.0 -.5

.5 1.0 1.5 2.0

-.5

-1.0
the answer stays between -1 and 1.
1.0

.5
-2.0 -1.5 -1.0 -.5

.5 1.0 1.5 2.0

-.5

-1.0
 Squash the result
.50

0.0
+
1.075 .746

-.75

.75
Weighted sum-and-squash neuron
.50

0.0

.746

-.75

.75
Make lots of neurons, identical except for
weights
To keep our picture
clear, weights will either
be
1.0 (white)
-1.0 (black) or
0.0 (missing)
Receptive fields get more complex
Repeat for additional layers
Receptive fields get still more complex
Repeat with a variation
 Rectified linear units (ReLUs)
1.0

.5
-2.0 -1.5 -1.0 -.5

.5 1.0 1.5 2.0

-.5
If your number is positive, keep it.
Otherwise you get a zero. -1.0
solid

vertical

diagonal

horizontal
solid

vertical

diagonal

horizontal
solid

vertical

diagonal

horizontal
solid

vertical

diagonal

horizontal
solid

vertical

diagonal

horizontal
solid

vertical

diagonal

horizontal
solid

vertical

diagonal

horizontal
Errors truth
0.
solid

vertical
0.

diagonal
0.

horizontal
1.
Errors truth
0.
answer
.5
solid

vertical
0. .75

diagonal
0. -.25

horizontal
1. -.75
Errors error
.5
truth
0.
answer
.5
solid

vertical
.75 0. .75

diagonal
.25 0. -.25

horizontal
1.75 1. -.75
Errors error
.5
truth
0.
answer
.5
solid

vertical
.75 0. .75

diagonal
.25 0. -.25

horizontal
1.75 1. -.75

total 3.25
 Learn all the weights: Gradient
descent
Error at:

original weight

Weight
 Learn all the weights: Gradient
descent
Error at:

lower weight

original weight

Weight
 Learn all the weights: Gradient
descent
Error at:

lower weight

original weight

higher weight

Weight
 Numerically calculating the gradient is expensive
Error at:

lower weight

original weight

higher weight

Weight
 Calculate the gradient (slope)
directly
Error at:

original weight

Weight
 Slope
Error at:

change in
weight = +1
original weight

Weight
 Slope
Error at:

change in
weight = +1
original weight

move along
the curve

Weight
 Slope
Error at:

change in
weight = +1
original weight
change in
error = -2

Weight
 Slope slope = change in error
change in weight

Error at:

change in
weight = +1
original weight
change in
error = -2

Weight