Inspiration from Neurobiology

Human Biological Neuron

Neuron
Analogy between Brain & Computer
Input: Inputs are stimuli, data and objects that act as the
basis for action.
•In the machine world, inputs come through input devices
like keyboards, mouse, scanner, camera, microphone etc.
They come in different formats like images, text and
structured data.

•For the human brain, input is provided through internal

and external stimuli. Input ‘devices’ for brain would
include sensory cells (those associated with seeing,
hearing, touching, tasting), motor/muscle cells, and even
some cells within the brain. These cells pick up the
stimulus provided and take it in for further processing.
Storage: This is the place where things like input
information, output information and programs are held.

•Computers have two types of storage. A short-term

storage, which we commonly refer to as RAM, that stores
limited amounts of information (data or programs)
required by the computer to perform an immediate task.

• And a long-term storage with much greater capacity in

the form of Hard Disk Drives, Optical Disks, Tapes etc.

•
•The brain’s short-term storage is called ‘Working
Memory’ which uses the brain’s memory and attention
capabilities in short bursts of time to perform the immediate
task at hand (e.g. remembering what the teacher is saying
when taking notes in class).

• Long term storage for humans is through their ‘Long Term

Memory’ or LTM function which enables them to store
almost unlimited amounts of information for an indefinite
period of time. It includes memory (or storage) of things like
specific events, factual information and experiences.
Processor: This is probably the simplest parallel to
grasp. It refers to the central processing circuitry that
carries out instructions provided.
In a computer, this is managed by its CPU, whereas for
humans it is the Central Nervous System comprised of
the brain and spinal cord and their internal circuitry.

Program: Or ‘code’, in computer language.

This refers to the intricate set of instructions that lay
out what requires to be done with the input in order to
arrive at an output, bringing together a set of functions
that have to be performed, in a defined order, within
defined parameters.
•In humans, this code is executed via the neurons present in
the brain. The neurons make connections with each other
through which they pass on signals which are processed to
achieve desired action, the entire process controlled by the
central nervous system. The signal or information exchange
happens through very critical nervous system structures
called Synapses.

•Computer programs run via machine code, which are

patterns of bits (i.e. binary units of information like ‘1’ and
‘0’). Similarly, a human brain runs its ‘code’ via patterns of
chemical or electrical signals which are passed from one
neuron to the next forming a neural network.
Output: Output is the information or action that comes as a
result of processing the input.

•Output actions in the case of machines could be things like

printing something, adding numbers, projecting an image,
solving a complex problem based on an algorithm,
identifying patterns etc.

•The output from processing that occurs in a human brain is

obviously more sophisticated and includes activities like
controlling internal and external muscles, sensory
perception, managing internal bodily functions (like blood
pressure, heart rate, temperature), decision making,
problem solving, emotions, social behavior, etc.
Artificial Neural Networks (or ANNs) work exactly like the
biological ones. ANNs are a web of artificial neurons that
send signals to each other.

These networks function just like the brain, learning

through seeing, without needing to be specifically
programmed.

Consider how a baby recognizes a dog for the first time — 

you show them a dog, you say the word ‘dog’, you do this several
times, and now the baby’s brain has learned to identify the
pattern between the visual of a dog and the term used for it. A
machine learns in exactly the same manner.
Human Intelligence, just like Artificial Intelligence, needs
to be developed.

During the early years of brain development (typically

from conception to 4–5 years of age), there is a very rapid
creation of synapses that connect neurons.

In data science terms, this phase is somewhat akin to

‘Model Training’ — the more things a child is exposed to
the more the synapses associated with it get built.

A good example is language — the more a child is

exposed to linguistics in early years, the more evolved the
area of her brain that is associated with languages would
get.
Are all brain functions computational, i.e. can they
all be boiled down to the ‘input-processing-output’
equation which in theory can be replicated in the
physical world, however complex the ‘processing’
aspect of it.

Or are there parts of a human brain’s roles (like

consciousness, free will, memory classification
etc.) that are just not ‘programmable’?
Introduction
• What are Neural Networks?
• Neural networks are a new method of
programming computers.
• They are exceptionally good at performing pattern
recognition and other tasks that are very difficult
to program using conventional techniques.
• Programs that employ neural nets are also capable
of learning on their own and adapting to changing
conditions.
Neural networks are parallel computing devices which is
basically an attempt to make a computer model of the brain

Artificial neural network ANN is an efficient computing

system whose central theme is borrowed from Biological
neural networks

ANN ‘s are also termed as Artificial neural systems, Parallel

distributed processing systems.

ANN’s has been developed to perform computational tasks

faster than traditional systems.

These tasks include pattern recognition and classification,

approximation, optimisation and data clustering
 History
• 1943 McCulloch-Pitts neurons (modelled a simple neural
network using electrical circuits to describe how neurons
work)
• 1949 Hebb’s law (repeated activation of one neuron by
another increases its strength each time they are used)
• 1958 Perceptron : learning method for neuron model
(Rosenblatt)
• 1960 Adaline, better learning rule (Widrow, Huff)
• 1969 multilayer perceptron (Minsky, Papert)
• 1972 Kohonen nets, associative memory
 How the Human Brain learns

 In the human brain, a typical neuron collects signals from others

through a host of fine structures called dendrites.
 The neuron sends out spikes of electrical activity through a long,
thin strand known as an axon, which splits into thousands of
branches.
 At the end of each branch, a structure called a synapse converts
the activity from the axon into electrical effects that inhibit or
excite activity in the connected neurons.
 Artificial Neurons
ANN is an information processing system that has
certain performance characteristics in common with
biological nets.
Key features of the processing elements of ANN

1. The processing element receives many signals.

2. Signals may be modified by a weight at the
receiving synapse.
3. The processing element sums the weighted
inputs.
4. Under appropriate circumstances (sufficient
input), the neuron transmits a single output.
5. The output from a particular neuron may go to
many other neurons.
A Simple Neuron

• An artificial neuron is a device with many inputs and one

output.
• The neuron has two modes of operation;
• the training mode and
• the using mode.
A Simple Neuron (Cont.)
• In the training mode, the neuron can be trained to fire
(or not), for particular input patterns.
• In the using mode, when a taught input pattern is
detected at the input, its associated output becomes
the current output. If the input pattern does not
belong in the taught list of input patterns, the firing
rule is used to determine whether to fire or not.
• The firing rule is an important concept in neural
networks and accounts for their high flexibility. A firing
rule determines how one calculates whether a neuron
should fire for any input pattern. It relates to all the
input patterns, not only the ones on which the node
was trained on previously.
 Artificial Neurons
• From experience:
A physical neuron
examples / training
data
• Strength of connection
between the neurons
is stored as a weight-
value for the specific
connection.
• Learning the solution
to a problem =
changing the
connection weights

An artificial neuron
 Artificial Neural Network
Background
An Artificial Neural Network (ANN) is an information processing
paradigm that is inspired by the biological nervous systems, such
as the human brain’s information processing mechanism.

The key element of this paradigm is the novel structure of the

information processing system. It is composed of a large number
of highly interconnected processing elements (neurons) working
in unison to solve specific problems. NNs, like people, learn by
example.

An NN is configured for a specific application, such as pattern

recognition or data classification, through a learning process.
Learning in biological systems involves adjustments to the
synaptic connections that exist between the neurons. This is true
of NNs as well.
 Artificial Neuron

Four basic components of a human biological The components of a basic artificial neuron
neuron
 Model Of A Neuron
Wa
X1

Wb Y
X2  f()

Wc
X3

Input units Connection Summing

computation
weights function

(dendrite) (synapse) (axon)

(soma)
• A neural net consists of a large number of
simple processing elements called neurons,
units, cells or nodes.

• Each neuron is connected to other neurons by

means of directed communication links, each
with associated weight.

• The weight represent information being used by

the net to solve a problem.
• Each neuron has an internal state, called
its activation or activity level, which is a
function of the inputs it has received.
Typically, a neuron sends its activation as
a signal to several other neurons.

• It is important to note that a neuron can

send only one signal at a time, although
that signal is broadcast to several other
neurons.
• Neural networks are configured for a specific
application, such as pattern recognition or
data classification, through a learning
process
• In a biological system, learning involves
adjustments to the synaptic connections
between neurons
 same for artificial neural networks (ANNs)
 Artificial Neural Network
Synapse Nucleus

x1 w1
 
y
Axon
x2 w2 Activation Function:
yin = x1w1 + x2w2

Dendrite

-A neuron receives input, determines the strength or the weight of the input,
calculates the total weighted input, and compares the total weighted with a
value (threshold)
-The value is in the range of 0 and 1
- If the total weighted input greater than or equal the threshold value, the neuron
will produce the output, and if the total weighted input less than the threshold
value, no output will be produced
Pattern Recognition
• An important application of neural networks is pattern
recognition which can be implemented by using a feed-
forward neural network that has been trained accordingly.
• During training, the network is trained to associate outputs
with input patterns.
• When the network is used, it identifies the input pattern
and tries to output the associated output pattern.
• The power of neural networks comes to life when a pattern
that has no output associated with it, is given as an input.
• In this case, the network gives the output that corresponds
to a taught input pattern that is least different from the
given pattern.
Pattern Recognition (cont.)

• Suppose a network is trained to recognize the

patterns T and H. The associated patterns are all
black and all white respectively as shown above.
Pattern Recognition (cont.)

Since the input pattern looks more like a ‘T’, when

the network classifies it, it sees the input closely
resembling ‘T’ and outputs the pattern that
represents a ‘T’.
Pattern Recognition (cont.)

The input pattern here closely resembles ‘H’ with a

slight difference. The network in this case classifies
it as an ‘H’ and outputs the pattern representing an
‘H’.
Pattern Recognition (cont.)

• Here the top row is 2 errors away from a ‘T’ and 3 errors away from an H. So the top
output is a black.
• The middle row is 1 error away from both T and H, so the output is random.
• The bottom row is 1 error away from T and 2 away from H. Therefore the output is
black.
• Since the input resembles a ‘T’ more than an ‘H’ the output of the network is in favor
of a ‘T’.
A Complicated Perceptron

• A more sophisticated Neuron is know as the McCulloch and Pitts model

(MCP).
• The difference is that in the MCP model, the inputs are weighted and the
effect that each input has at decision making, is dependent on the weight
of the particular input.
• The weight of the input is a number which is multiplied with the input to
give the weighted input.
A Complicated Perceptron (cont.)
• The weighted inputs are then added together
and if they exceed a pre-set threshold value,
the perceptron / neuron fires.
• Otherwise it will not fire and the inputs tied
to that perceptron will not have any effect on
the decision making.
• In mathematical terms, the neuron fires if and
only if;
X1W1 + X2W2 + X3W3 + ... > T
 Characterization
• Architecture
– a pattern of connections between neurons
• Single Layer Feedforward
• Multilayer Feedforward
• Recurrent
• Strategy / Learning Algorithm
– a method of determining the connection weights
• Supervised
• Unsupervised
• Reinforcement
• Activation Function
– Function to compute output signal from input
signal
 Network Layers
 Input Layer - The activity of the input units
represents the raw information that is fed into
the network.
 Hidden Layer - The activity of each hidden unit is
determined by the activities of the input units
and the weights on the connections between
the input and the hidden units.
 Output Layer - The behavior of the output units
depends on the activity of the hidden units and
the weights between the hidden and output
units.
Diagram of an NN

A simple Neural Network

Continued
 This simple type of network is interesting
because the hidden units are free to
construct their own representations of the
input.
 The weights between the input and hidden
units determine when each hidden unit is
active, and so by modifying these weights,
a hidden unit can choose what it
represents.
 Single Layer Feedforward NN

x1 w11

w12 ym

w21

yn
x2

w22
output layer
Input layer
 Multilayer Neural Network
z1
V11
x1
w12
  w11
V1n
w12 y1

x2 z2

   y2


 zn

xm Vmn  
Input layer Output layer
Hidden layer

It contains more than one layer of artificial neurons or nodes

 Recurrent NN
Input Outputs

Hidden nodes
Output from the previous step are fed as input to the current step. It uses the
same parameters for each input as it performs the same task on all the inputs or
hidden layers to produce the output
Neural Network in Use
Since neural networks are best at identifying patterns or
trends
in data, they are well suited for prediction or forecasting
needs including:
• sales forecasting
• industrial process control
• customer research
• data validation
• risk management

ANN are also used in the following specific paradigms:

recognition of speakers in communications; diagnosis of
hepatitis; undersea mine detection; texture analysis;
three-dimensional object recognition; hand-written word
recognition; and facial recognition.
Neural networks in Medicine
• Artificial Neural Networks (ANN) are currently
a 'hot' research area in medicine and it is
believed that they will receive extensive
application to biomedical systems in the next
few years.
• At the moment, the research is mostly on
modeling parts of the human body and
recognizing diseases from various scans (e.g.
cardiograms, CAT scans, ultrasonic scans, etc.).
 Neural networks are ideal in recognizing diseases
using scans since there is no need to provide a
specific algorithm on how to identify the disease.

 Neural networks learn by example so the details of

how to recognize the disease are not needed.

 What is needed is a set of examples that are

representative of all the variations of the disease.

 The quantity of examples is not as important as the

'quality'. The examples need to be selected very
carefully if the system is to perform reliably and
efficiently.
Modeling and Diagnosing the
Cardiovascular System
 Neural Networks are used experimentally to model
the human cardiovascular system.

 Diagnosis can be achieved by building a model of

the cardiovascular system of an individual and
comparing it with the real time physiological
measurements taken from the patient.

 If this routine is carried out regularly, potential

harmful medical conditions can be detected at an
early stage and thus make the process of
combating the disease much easier.
 A model of an individual's cardiovascular
system must mimic the relationship among
physiological variables (i.e., heart rate,
systolic and diastolic blood pressures, and
breathing rate) at different physical activity
levels.

 If a model is adapted to an individual, then

it becomes a model of the physical
condition of that individual. The simulator
will have to be able to adapt to the features
of any individual without the supervision of
an expert. This calls for a neural network .
 Another reason that justifies the use of NN
technology, is the ability of NNs to provide sensor
fusion which is the combining of values from
several different sensors.
 Sensor fusion enables the NNs to learn complex
relationships among the individual sensor values,
which would otherwise be lost if the values were
individually analyzed.
 In medical modeling and diagnosis, this implies
that even though each sensor in a set may be
sensitive only to a specific physiological variable,
NNs are capable of detecting complex medical
conditions by fusing the data from the individual
biomedical sensors.
 Characteristics of Biological Neural
Networks
1) Massive connectivity
2) Nonlinear, Parallel, Robust and Fault Tolerant
3) Capability to adapt to surroundings
4) Ability to learn and generalize from known
examples
5) Collective behavior is different from individual
behavior
Artificial Neural Networks mimics some of the properties of
the biological neural networks