PALESTINE

TECHNICAL COLLEGE

Neural
Neural Networks
Networks

Eng. Akram Abu Garad

www.ptcdb.edu.ps
 Neural Networks

Chapter 1 – Introduction
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

The Brain as an Information

Processing System
 The human brain contains about 10 billion nerve cells, or
neurons. On average, each neuron is connected to other
neurons through about 10 000 synapses. The brain's network of
neurons forms a massively parallel information processing
system. This contrasts with conventional computers, in which a
single processor executes a single series of instructions.

 Against this, consider the time taken for each elementary

operation: neurons typically operate at a maximum rate of about
100 Hz, while a conventional CPU carries out several hundred
million machine level operations per second.
3
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Despite of being built with very slow hardware, the brain has
quite remarkable capabilities:
Its performance tends to degrade gracefully under partial
damage.
In contrast, most programs and engineered systems are
brittle: if you remove some arbitrary parts, very likely the whole
will cease to function.
It can learn (reorganize itself) from experience.
This means that partial recovery from damage is
possible if healthy units can learn to take over the functions
previously carried out by the damaged areas.
It performs massively parallel computations extremely
efficiently. For example, complex visual perception occurs
within less than 100 ms, that is, 10 processing steps!
It supports our intelligence and self-awareness. (Nobody
knows yet how this occurs.) 4
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

processing element energy processing style of fault intelligent,

learns
elements size use speed computation tolerant conscious

1014 parallel,
10-6 m 30 W 100 Hz yes yes usually
synapses distributed

108 30 W serial,
10-6 m 109 Hz no a little not (yet)
transistors (CPU) centralized

As a discipline of Artificial Intelligence, Neural Networks attempt to

bring computers a little closer to the brain's capabilities by imitating
certain aspects of information processing in the brain, in a highly
simplified way.

5
CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

What is a neural Network (NN) ?

 A network is a system with many elements connected
together.

 A neural network is a network with neurons as the basic

elements.

 A neuron is a brain cell in bio-neural networks.

 A neuron is a simple processing unit in artificial neural

networks (ANN).

 An artificial neural network (ANN) is a system composed of many

simple processing elements operating in parallel whose function
is determined by network structure, connection strengths, and
the processing performed at computing element or nodes.

6
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Why Neural Network ?

 Fast system with slow elements.

 Intelligent system with non-intelligent elements.

 Neural network is a different kind of computer analog to human brain.

 NN is also a universal computer, which is as powerful as the

conventional computer.

 NN can learn from experience, and can solve different kinds of

problems after learning.

 NN can learn in real-time, and can adapt to changing environments

flexibility.
A neural network can perform tasks that a linear program can not.
When an element of the neural network fails, it can continue without any
problem by their parallel nature.
A neural network learns and does not need to be reprogrammed.
7
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Disadvantages of Neural Networks

 Not good for fast, precise, and repeated arithmetic computations.
 The neural network needs training to operate.
 Requires high processing time for large neural networks.
 NN should be combined with existing computing
theory/technology to be more practically useful.
Application of Neural Networks
Recognizing patterns:
Facial identities or facial expressions
Face detection
Handwritten or spoken words

Prediction:
Future stock prices
Future currency exchange rates
Function Approximation 8
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Biological Neuron
 Each neuron has a body, an axon, and many dendrites:
- Can be in one of the two states: firing and rest.
- Neuron fires if the total incoming stimulus exceeds the threshold.
 Synapse: thin gap between axon of one neuron and dendrite of another:
Dendrites:
The dendrites are receptors for signals generated
by other neurons.
Receiving information.
Don't generate action potentials.
Cell body: axon
Sum and threshold the incoming information. body
Axon:
dendritic
Generate and relay action potential. tree
Relays information to next neuron in the
pathway. 9
CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Biological Neuron (cont.)

A biological neuron does nothing unless the collective influence of

all its inputs reaches a threshold level. Whenever that threshold
level is reached, the neuron produces a full-strength output in the
form of a narrow pulse that proceeds from the cell body down the
axon.

10
CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Artificial Neural Networks (ANN)

 A set of nodes (units, neurons, processing elements):

 Each node has input and output.
 Each node performs a simple computation by its node
function.
 Weighted connections between nodes:
 Connectivity gives the structure/architecture of the
net.
 What can be computed by a NN is primarily
determined by the connections and their weights.

11
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

ANN Neuron Models

General Neuron

Single Input Neuron: input weight net

f
Σ output

 Each node has one or more bias

Transfer
inputs from other nodes, and function
one output to other nodes 1

 Input/output values can be General Neuron Model

 Binary {0, 1}
 Bipolar {-1, 1}
 Continuous
 All inputs to one node come in at
the same time and remain
activated until the output is
produced
 Weights associated with links. Weighted Input Summation
12
CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Some Terminologies

 The parameters used to scale the inputs are

called the weights.
 The effective input is the inner product of the
input vector and the weight vector.
 The function for producing the final output is
called the transfer function (activation function).
 In general, there are many different kinds of
Transfer functions or activation functions.

13
CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Transfer Function

General Neuron
The transfer function may be linear or
nonlinear function of n. w n a
p Σ f

b
a  f ( n)  f ( wp  b)
1

14
CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Transfer Function (cont.)

15
CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Transfer Function (cont.)

16
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

ANN Neuron Models (cont.)

Multiple Input Neuron:
Typically, a neuron has more than one inputs.

n = Wp + b a = f(Wp + b)
17
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Neural Network Architectures

Commonly one neuron, even with
many inputs, may not be
sufficient. We might need more
than one neuron operating in
parallel which called a layer.

 A single Layer Networks:

p1 b1 a1
p = p2 b = b2 a = a2
pR bS aS 18
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Neural Network Architectures (cont.)

 A multi Layer Networks:
Hidden layer Hidden layer Output layer

19
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Neural Network Architectures (cont.)

 A multi Layer Networks:

Hidden layer Hidden layer Output layer

20
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Neural Network Architectures (cont.)

 Recurrent Networks:

Some Simple Blocks:

Delay Block Integrator Block

21
 CHAPTER
CHAPTER11 Introduction Neural
NeuralNetworks
Networks

Neural Network Architectures (cont.)

 Recurrent Networks:
A recurrent network is a network with feedback, some of its
output are connected to its inputs.

In this particular network the vector p supplies the initial conditions (a(0) = p).

22