NPTEL

Video Course on Machine Learning

Professor Carl Gustaf Jansson, KTH

Topic: Perceptrons and ANNs

Solution and Explanations for Assignment Q 2 and 4 - Week 6

Assignment tasks - Week 6b 2021

Problem # 2 Correct Marks: 2 Theme: Neurons and Perceptrons Training set

Assume a perceptron: Input j Target J
• with 3 inputs (x1,x2,x3) plus a bias (x0) statically set to 1 1 0 0 -> 1
0 1 1 -> 1
• with weighted input= x0*w0+x1*w1+x2*w2+x3*w3 1 1 0 -> 0
• that outputs 1 if weighted input > 0, else 0 0 0 1 -> 0
• with initial weights are all set to 0
• with weight updating as follows: Wi j+1= Wi j+ a * (Target j- Output j) * X i and a learning rate a=1

Instance Target Weight Vector Weighted Output New Weight Vector

input
100 1 1 0000 0 0 (0 0 0 0) + (1 0 0 1) =1 0 0 1
011 1 1 1001 1 1 (1 0 0 1) + (0 0 0 0) = 1 0 0 1
110 1 0 1001
001 1 0 ?
How will the final weight vector look like when all data-items are processed?

A. 1011 B. 0 0 -1 0 C. 0 -1 0 0 D. 1 0 1 0 E. None of the above

Answer: C
Assignment tasks - Week 6 2021
Activation function
Problem # 4 Correct Marks: 2 Theme: Feed Forward Multiple Layer ANN

Assume a single neuron ANN: Training set 1

• with 3 inputs (x1,x2,x3) plus a bias (x0) statically set to 1
Input j Target J
• with weighted input= x0*w0+x1*w1+x2*w2+x3*w3 1 0 1 -> 1 2
0 1 0 -> 0
• that outputs Activation function(weighted input) if weighted input > 0, else 0 1 0 1 -> 0
• with initial weights are all set to 0 and a linear Activation function 0 1 0 -> 1
• with weight updating as follows: Wi j+1= Wi j+ a * (Target j- Output j) * X i and a learning rate a=1

Instance Target Weight Vector Weighted Output New Weight Vector

input
101 1 1 0000 0 0 (0 0 0 0) + (1 0 1 1) =1 0 1 1
010 1 0 1011 1 0.5 (1 0 1 1) + (0 -0.5 0 -0.5) = 1 -0.5 1 0.5
101 1 0 1 -0.5 1 0.5
010 1 1

How will the final weight vector look like when all data-items are processed?

A. -0.25 0 0 0.25 B. 0 -0.25 0.25 0 C. 0.5 0 0.75 -0.35 D. -0.25 0.5 -0.25 0.25 E. None of the above

Answer: D
Perceptron
The perceptron is an algorithm for learning a linear binary
classifier in the form of a threshold function that maps its
input X, a real-valued input vector, to a single output
binary value Y.

In the context of neural networks, a perceptron is a

single artificial neuron using the Heaviside step
function as the activation function.

As a linear classifier, the single-layer perceptron is the

simplest feedforward neural network.

Y j= if W * X = Sum Wi j * Xi j >𝜃 then 1 otherwise= 0.

i=1 .. N
W is a real valued vector, j = # of iteration

The binary value of Y is used to classify a data-item as either a

positive or a negative instance.
Perceptron
The perceptron is an algorithm for learning a linear binary
classifier in the form of a threshold function that maps its
input X, a real-valued input vector, to a single output
binary value Y.

In the context of neural networks, a perceptron is a

single artificial neuron using the Heaviside step
function as the activation function.

As a linear classifier, the single-layer perceptron is the

simplest feedforward neural network.

Y j= if W * X = Sum Wi j * Xi j >𝜃 then 1 otherwise= 0.

i=1 .. N
W is a real valued vector, j = # of iteration

The binary value of Y is used to classify a data-item as either a

positive or a negative instance.
 Perceptron Learning Rule
A simpler version of the delta rule for ONE neuron with specific properties.

, Wi j+1= Wi j+ a * (Tj-Yj)* Xi j, a = learning rate parameter.

delta weight
T
Yj = Xi j Wi j, i = input, j =iteration, T = target

The treshold can also be learned by transforming it to a bias = - threshold and completing the
dataitems with an extra input x0j=1 and a corresponding weight w0j=bias.

Steps in a Learning Procedure

Initialize the weights and the learning rate.
Weights may be initialized to 0 or to a small random value.

For each example j in the dataset, perform the following steps until total training set error ceases to improve:
• calculate the output
• calculate the new weights
Perceptron

x1 w1 Y j =1 if W * X+B = (Sum wi j * xi j) > 𝜃

i=1 .. n
x2 w2 𝜃θ Y otherwise Y= 0.
Wi j+1= Wi j+ a * (Tj-Yj)* Xi j a=0.2
delta weight
xn wn
€

X11=0.8 x21=0.3 T1= 0 W11=0.4 W21=-0.2

0.8 0.4

0.1 Y1=1

0.3 -0.2
0.4*0.8 -0.2*0.3 =0 .26 > 0.1 -> Y1 = 1

W12=0.4+0.2*(0-1)*0.8 = 0.24 W22= -0.2 +0.2*(0-1) *0.3=-0.26

7
Perceptron with threshold handled as a bias that can be learned

x1 w1 Y j =1 if W * X+B = (Sum wi j * xi j) > 0

i= .. n
x2 w2 θ Y otherwise Y= 0.
Wi j+1= Wi j+ a * (Tj-Yj)* Xi j a=0.2
delta weight
xn wn
€
X0=1 w0=−𝜃
X11=0.8 x21=0.3 T1= 0 W11=0.4 W21=-0.2 W01=-0.1
0.8 0.4

Y1=1

0.3 -0.2
0.4*0.8 + -0.2*0.3 - 0.1 =0 .16 > 0 -> Y1 = 1

1 -0.1 W12=0.4+0.2*(0-1)*0.8 = 0.24 W22= -0.2 +0.2*(0-1) *0.3=-0.26

W02=-0.1 + 0.2*(0-1)*1=-0.3 8
Solution to Q 2
Training set
Assume a perceptron: Input j Target J
• with 3 inputs (x1,x2,x3) plus a bias (x0) statically set to 1 1 0 0 -> 1
0 1 1 -> 1
• with weighted input= x0*w0+x1*w1+x2*w2+x3*w3 1 1 0 -> 0
• that outputs 1 if weighted input > 0, else 0 0 0 1 -> 0
• with initial weights are all set to 0
• with weight updating as follows: Wi j+1= Wi j+ a * (Target j- Output j) * X i and a learning rate a=1

How will the final weight vector look like when all data-items are processed?

Instance Target Weight Vector Weighted Output New Weight Vector

input
100 1 1 0000 0 0 (0 0 0 0) + (1 0 0 1) =1 0 0 1
011 1 1 1001 1 1 (1 0 0 1) + (0 0 0 0) = 1 0 0 1
110 1 0 1001 2 1 (1 0 0 1) + (-1 -1 0 -1) = 0 -1 0 0
001 1 0 0 -10 0 0 0 0 -1 0 ´0 + (0 0 0 0) = 0 -1 0 0
Abstract model of an ANN Neuron

Xn
Wn Y = if Sum>0 then f (Sum) else 0

X2 W2 f = activation function
Sum = Sum (Wi*Xi) Target value = T
i=1..n
for the output
W1
X1 The differences between the T
W0= and Y values is the basis for an
X0=1 - Treshold error estimate = E
The core computation of an ANN unit
1. We identify the studied Neuron by j. All parameters have Real values.

2. The Neuron j has a number of inputs i= 1...n.

3. Each connection is assigned a Weight wij.

4. Each node has an activation Threshold T

5. Typically the Threshold is remodelled as its negation and named Bias. An extra input x0 is added with
constant value = 1. The weight x0j is set to the Bias or –Threshold. This move enables the Bias to be
adapted in the same fashion as weights.

6. The weighted inputs are summed: Sum = Sum wij * ai

i=0..n

7. If Sum > 0 the output of unit j is calculated as

Y = f ( Sum ) where f is a so called activation or transfer function.
Y = 0 otherwise.
Selection of Activation or Transfer Functions

Step function Sigmoid

Identity function ReLU = Rectified Linear Unit

Delta Learning Rule
The Delta Learning Rule can be used when we have a Target T as a reference for the output Y.

The error measure is E= ½ * ( T-Y) ^2

The Delta Learning Rule can be inferred by applying a
gradient descent algorithm calculating the derivatives of
the error function E with respect to the weights W.

d Ej / d W ji = d (1/2* (T-Y)^2)/ d W ji - > Delta rule

The Delta rule becomes:

W j i =Wj i+ a *(Tj-Yj) G´(SUMj)*Xi

a = learning rate parameter.

Yj = output from neuron j, T j = target for neuron j
W j i =weights between input i and neuron j,
X i = input # i
SUMj = weighted sum of inputs to neuron j,
G´ = derivative of transfer function for j

If the activation function is linear the Delta rule is simplified.

Wj i =W j i+ a *(Tj-Yj) *X i
Single Neuron Excitation The activation function f is ReLU

If Sum wij * xi > 0 the output of unit j is calculated as

Y = f ( Sum (wij * xi) ) otherwise Y=0.

W j i =Wj i+ a *(Tj-Yj) G´(SUMj)*Xi

Wj i =W j i+ a *(Tj-Yj) *X i if f is linear.

X1=0.8 X2=0.3 T1= 0.26 W11=0.4 W21=-0.2 W01=-0.1, The segments are differentiable
a=1 , Treshold = 0.1 -> Bias =-1

0.8 0.4

Y1 0.4*0.8 + -0.2*0.3 – 1* 0.1 =0 .16 > 0 -> Y1 = f(0.16)=0.16

0.3 -0.2
W11=0.4+1*(0.26-0.16)*0.8 = 0.48 W12= -0.2 +1*(0.26-0.16) *0.3= - 0.17
1 -0.1
W00=-0.1 + 1*(0.26-0.16)*1= 0
14
Solution to Q 4
Activation function

Assume a single neuron ANN: Training set 1

• with 3 inputs (x1,x2,x3) plus a bias (x0) statically set to 1
Input j Target J
• with weighted input= x0*w0+x1*w1+x2*w2+x3*w3 1 0 1 -> 1 2
0 1 0 -> 0
• that outputs Activation function(weighted input) if weighted input > 0, else 0 1 0 1 -> 0
• with initial weights are all set to 0 and a linear Activation function 0 1 0 -> 1
• with weight updating as follows: Wi j+1= Wi j+ a * (Target j- Output j) * X i and a learning rate a=1
How will the final weight vector look like when all data-items are processed?

Instance Target Weight Vector Weighted Output New Weight Vector

input
101 1 1 0000 0 0 (0 0 0 0) + (1 0 1 1) =1 0 1 1
010 1 0 1011 1 0.5 (1 0 1 1) + (0 -0.5 0 -0.5) = 1 -0.5 1 0.5
101 1 0 1 -0.5 1 0.5 2.5 1.25 (1 -0.5 1 0.5) + (-1.25 0 -1.25 -1.25)
010 1 1 -0.25 -0.5 -0.25 -0.75 -1.25 0 (-0.25 -0.5 -0.25 -0.75) + (0 1 0 1) = -0.25 0.5 -0.25 0.25