Subject Name: MACHINE LEARNING

Unit No: 2 INTRODUCTION TO NEURAL

NETWORK

Faculty Name : Dr. Bharti Joshi

Ms.Rajashree Shedge
Mr. Tushar Ghorpade
Index

Lecture 9 – Network Architecture

Lecture 10 - Activation Functions

Lecture 11 – MP Model

Lecture 12 – Linear Separability

2
Unit No: 2 Neural Networks

Lecture No: 9
Network Architecture
Network Architecture

• deals with how neurons form layers and how they are interconnected.
• Two types of NN: single-layer or multilayer
• A layer is formed by combining processing elements
• A layer is a stage that links input stage and output stage.

• 1.Single Layer Feed-forward Network

• 2. Multilayer Feed-forward Network
• 3. Feedback Network
• 4. Recurrent Networks
• 5. Multilayer Recurrent Network
 1.Single Layer Feed-forward Network

• When a layer is formed, inputs are connected to output nodes with various
weights.
• This results in a single-layer feed forward network.
2. Multilayer Feed-forward Network

• It is formed by the interconnection of several layers.

• Input layer receives input and it only buffers input signal
• Output layer generates output
• Any layer between input and output is a hidden layer
• Hidden layer is internal to network and has no direct contact with the external
environment.
• No. of hidden layers : zero or more
• More hidden layer 🡺 more complexity but may provide an efficient output
response
• A network is said to be a feed-forward network if output layer is not connected
to the same layer or preceding layers.
2. Multilayer Feed-forward Network
3. Feedback Network

• When outputs are connected back as inputs to same or preceding layer then it is
a feedback network.
• If the feedback of the output is directed back to the same layer then it is called
lateral feedback.
4. Recurrent Networks

• Recurrent networks are feedback networks with closed loop.

• It can be single layer or multilayer.
• Single layer network: Output processing element can be directed back to itself or
other processing elements or both.
5. Multilayer Recurrent Networks

• Multilayer Recurrent Network : Output processing element is directed back to

the nodes in a preceding layer or same layer.
Learning

• Learning or training is a process by which a neural network adapts itself to a

stimulus by making proper parameter adjustments resulting in the desired
response.
• Learning can be classified into three categories as:
– supervised learning
– unsupervised learning
– reinforcement learning.
Supervised Learning

• Learning with the help of a teacher.

• Like learning process of a small child : learns based on output he/she has to
produce.
• Each input vector requires a corresponding target vector which represents the
desired output.
• Input vector with target vector is training pair.
• The network is informed precisely about what should be the output.
Supervised Learning
Supervised Learning

• During training, the training pair is presented.

• Actual output vector is compared with target output to generate an error signal
• This error signal is used for adjustment of weights until the actual output
matches the desired (target) output.
Unsupervised Learning

• Without the help of a teacher.

• E.g. a child fish learns to swim by itself, it is not taught by its mother.
• Input vectors of the similar type are grouped together
• In the training process, the network receives input and organizes them to
form clusters.
• When a new input pattern is applied, the neural network gives an output
response indicating which class it belongs. If a class cannot be found then a
new class is generated.
Unsupervised Learning

• The network itself discover patterns, regularities, features or categories.

• While discovering these features, network undergoes change in its parameters.
This process is called self-organizing.
Reinforcement Learning

• similar to supervised learning

• In some cases, less information might be available. For e.g. , the network might
be told that its actual output is only “50%correct”.
• Here only critic information is available and not the exact information
• The feedback sent is the reinforcement signal.
Reinforcement learning
Unit No: 2 Neural Networks

Lecture No: 10
Activation Functions
A general neuron symbol

• The function f (wtk) is often referred to as an activation function.

• Its domain is the set of activation values called net.

• The variable net is defined as a scalar product of the weight and input vector.
Bipolar activation functions

• They are of two types: Bipolar Binary and Bipolar Continuous.

• Bipolar binary function is defined as
Bipolar continuous functions

• Notice that as λ🡺 ∞, the continuous function becomes the sgn (net) function.

• "bipolar" implies both positive and negative responses of neurons are produced
for this activation function.
Unipolar activation functions

• Obtained by shifting and scaling bipolar activation functions

• They are defined as

• Unipolar binary function is the limit of f(net) when λ🡺 ∞.

A general neuron symbol

• The neuron as a processing node obtain net and performs the nonlinear
operation f (net) through its activation function.

• Typical activation functions used are

– Bipolar activation functions
– Unipolar activation functions
Identity function

• It is a linear function and can be defined as

f (x) = x for all x
• The output here remains the same as input.
• The input layer uses the identity activation function.
Binary step function

• This function can be defined as

• where ϴ represents the threshold value.

• This function is most widely used in single-layer nets to convert the net
input to an output that is a binary (1 or 0).
Bipolar step function

• This function can be defined as

• where ϴ represents the threshold value.

• This function is also used in single-layer nets to convert the net input to an
output that is bipolar (+1 or -1).
Ramp function

• The ramp function is defined as

Unipolar Sigmoid activation functions

• Used in Back Propagation nets.

• Obtained by shifting and scaling bipolar activation functions
• They are defined as

• Unipolar binary function is the limit of f(net) when λ🡺 ∞.

Bipolar Sigmoid continuous functions

• The bipolar continuous function is defined as follows :

• where λ > 0 is proportional to the neuron gain determining the steepness of the
continuous function f(net) near net = 0.
Continuous activation function for various λ are :
Hard-limiting (Binary) activation functions describe the discrete neuron
model as given below:
Soft-limiting activation functions describe the continuous neuron model as
given below:
Unit No: 2 Neural Networks

Lecture No: 11
MP Model
McCulloch-Pitts Neuron Model

• The first formal definition was formulated by McCulloch and Pitts (1943).
• The McCulloch-Pitts model of the neuron is as shown below:
McCulloch-Pitts Neuron Model

• Inputs xi are 0 or 1, depending on the absence or presence of the input impulse

at instant k.
• The neuron's output signal is denoted as o.
• The firing rule for this model is
McCulloch-Pitts Neuron Model

• k = 0, 1, 2, . . . denotes discrete-time instant

• wi is the multiplicative weight

• wi = + 1 for excitatory synapses,

• wi = - 1 for inhibitory synapses

• T is the threshold value which needs to be exceeded by the weighted sum of

signals.
McCulloch-Pitts Neuron Model

• Although this neuron model is simple it has substantial computing potential.

• Perform basic logic operations NOT, OR, and AND
 McCulloch-Pitts Neuron Model: Limitations

• It allows binary 0, 1 states only,

• It operates under a discrete-time assumption

• Weights and thresholds are fixed in the model

• ANN employ a variety of neuron models that have more diversified features than
this model
 McCulloch-Pitts Neuron Model: Limitations

• It allows binary 0, 1 states only,

• It operates under a discrete-time assumption

• Weights and thresholds are fixed in the model

• ANN employ a variety of neuron models that have more diversified features than
this model
Logic Gates with MP Neurons

• We can use McCulloch-Pitts neurons to implement the basic logic gates.

• All we need to do is find the appropriate connection weights and neuron

thresholds to produce the right outputs for each set of inputs.

• We shall see explicitly how one can construct simple networks that perform NOT,
AND, and OR.

• It is then a well known result from logic that we can construct any logical function
from these three operations.

• The resulting networks, however, will usually have a much more complex
architecture than a simple Perceptron.

• We generally want to avoid decomposing complex problems into simple logic

gates, by finding the weights and thresholds that work directly in a Perceptron
architecture.

41
Implementation of Logical NOT, AND, and OR

• Logical OR

x1 x2 y
0 0 0 x θ=2
1 2
0 1 1
1 0 1
1 1 1 y

x 2
2

42
Implementation of Logical NOT, AND, and OR

• Logical AND

x1 x2 y
0 0 0 x θ=2
1 1
0 1 0
1 0 0
1 1 1 y

x 1
2

43
Implementation of Logical NOT, AND, and OR

• Logical NOT

X1 y
0 1 x θ=2
1 -1
1 0

1
2

bias

44
Implementation of Logical NOT, AND, and OR

• Logical AND NOT

x1 x2 y
0 0 0 x θ=2
1 2
0 1 0
1 0 1
1 1 0 y

x -1
2

45
Logical XOR

• Logical XOR

x1 x2 y
0 0 0 x
1 ?
0 1 1
1 0 1
1 1 0 y

x ?
2

46
Logical XOR

• How long do we keep looking for a solution? We need to be able to calculate

appropriate parameters rather than looking for solutions by trial and error.

• Each training pattern produces a linear inequality for the output in terms of the
inputs and the network parameters. These can be used to compute the weights
and thresholds.

47
Finding the Weights Analytically

• We have two weights w1 and w2 and the threshold q, and for each training
pattern we need to satisfy

48
Finding the Weights Analytically

• For the XOR network

– Clearly the second and third inequalities are incompatible with the fourth,
so there is in fact no solution. We need more complex networks, e.g. that
combine together many simple networks, or use different
activation/thresholding/transfer functions.

49
Unit No: 2 Neural Networks

Lecture No: 12
Linear Separability
Geometric Interpretation of MP Model

The M-P neuron just learnt a linear decision boundary! The M-P neuron is splitting
the input sets into two classes — positive and negative. Positive ones (which
output 1) are those that lie ON or ABOVE the decision boundary and negative ones
(which output 0) are those that lie BELOW the decision boundary.
AND Function

In this case, the decision boundary equation is x_1 + x_2 =2. Here, all the input
points that lie ON or ABOVE, just (1,1), output 1 when passed through the AND
function M-P neuron. It fits! The decision boundary works!

52 Week 2
Linear Separability

• Consider patterns (X1,X2) being

classified into two classes as shown in
figure.
• Each point with symbol x or o
represents a pattern with a set of
values (X1,X2) .
• Each pattern is classified into one of
two classes with a single line L.
• They are known as linearly
separable patterns.
Linear Separability

• Linear separability refers to the fact that

classes of patterns with n-dimensional
vector x = (x1,x2,…, xn) can be separated
with a single decision surface.
Linear Separability

• The processing unit of a single-layer perceptron network can categorize a set of

patterns into two classes as the linear threshold function.
• This is the main limitation of a single-layer perceptron network.
LINEAR SEPARABILITY

⮚ Linear separability is the concept wherein the separation of the input space
into regions is based on whether the network response is positive or negative.

⮚ Consider a network having positive

response in the first quadrant and
negative response in all other
quadrants (AND function) with either
binary or bipolar data, then the
decision line is drawn separating the
positive response region from the
negative response region.
Linearly Inseparable Pattern

• The classic example is XOR function

• Figure illustrates how XOR cannot be separated with a single line.
• The solution involves patterns classified with two lines L1 and L2.
Thank You