In19-MA4043

1 Introduction
1.1 What is Neural Net?
An Artificial Neural Network (ANN) or Neural Network have been developed as generalizations
of mathematical models of human cognition or neural biology, based on following assumptions:

1. Information processing occurs at many simple elements called neurons

2. Signals are passed between neurons over connection links
3. Each connection link has an associated weight, which, in a typical neural net,
multiplies the signal transmitted
4. Each neuron applies an activation function to its net input to determine its output
signal

A neural network is characterized by

(a) Its pattern of connection between the neurons (called architecture)
(b) Its method of determining the weights on the connections (called its training, or
learning, algorithm)
(c) Its activation function

• A neural net consists of a large number of simple processing elements called neurons, units,
cells, or nodes.
• Each neuron is connected to other neurons by means of directed communication links, each
with an associated weight.
• Each neuron has an internal state, called its activation or activity level, which is a function of
the inputs it has received.

• Figure 1.1 shows a simple artificial neuron.

Figure 1.1: A simple artificial neuron

• A neuron 𝑌 that receives inputs from neurons 𝑋! , 𝑋" , and 𝑋# .

• The activations (output signals) of these neurons are 𝑥! , 𝑥" , 𝑥# respectively.
• The weights on the connections from 𝑋! , 𝑋" , and 𝑋# to neuron 𝑌 are 𝑤! , 𝑤" , 𝑤# respectively.
• The net input, 𝑦$% , to neuron 𝑌 is
𝑦$% = 𝑤! 𝑥! + 𝑤" 𝑥" + 𝑤# 𝑥#
(1. 1)

• The activation 𝑦 of neuron 𝑌 is given by some function of its net input, 𝑦 = 𝑓(𝑦$% ).
• The several common activation functions are given in the following table.

Dep. of Mathematics, UoM

1
 In19-MA4043

Table 1.1

• Now suppose further that neuron 𝑌 is connected to neuron 𝑍! and 𝑍" , with weights 𝑣! and
𝑣" respectively as shown in Figure below.

Figure 1.2: A Simple Neural Network

• Neuron 𝑌 sends its signal 𝑦 to each of these units. In general, the values received by neurons
𝑍! and 𝑍" will be different.
• The arrangement of the units (the architecture of the net) and the method of training the net
(find optimal value for the weight) are discussed further in the next section.

Dep. of Mathematics, UoM

2
 In19-MA4043

1.2 How Neural Networks Used?

Let us now consider some of the fundamental features of how neural networks operate.

1.2.1 Network Architectures

• It is convenient to visualize neurons as arranged in layers.
• Typically, neurons in the same layer behave in the same manner.
• Key factors in determining the behaviour of neuron are its activation function and the pattern
of weighted connection over which it sends and receive signals.
• Within each layer, neurons usually have the same activation function and the same pattern
of connection to other neurons.
• The arrangement of neurons into layers and the connection patterns within and between
layers is called the net architecture.
• The number of layers in the net can be defined to be the number of layers of weighted
interconnect links between the slabs of neurons.

A Single-Layer Net

• A single-layer net has one layer of connection weights.

• The weights for one output unit do not influence the weights for other output units.

Figure 1.2: A single layer neural net

A Multilayer Net

• A multilayer is a net with one or more layers of nodes between the input units and the output
units.
• Multilayer networks are more powerful than single-layer networks. For instance, a two-layer
network having a sigmoid first layer and a linear second layer can be trained to approximate
most functions arbitrarily well.

Dep. of Mathematics, UoM

3
 In19-MA4043

Figure 1.3: A multilayer neural net

1.2.2 Setting the Weights

• The method of setting the values of the weights (training) is an important distinguishing
characteristic of different neural nets.
• There are two major types of training: supervised training and unsupervised training

Supervised Training

• Training is accomplished by presenting a sequence of training vectors, or patterns, each with

an associated target output vector.
• The learning algorithm is then used to adjust the weights of the network to move the network
outputs closer to the targets.
• This process is known as supervised training.

Unsupervised Training

• A sequence of input vectors is provided but no target vectors are specified.

• The weights are modified in response to network inputs only.

1.2.3 Activation Functions

• The activation function may be linear or nonlinear.
• A particular transfer function is chosen to satisfy some specification of the problem that the
neuron is attempting to solve.

1.3 Summary of Notation

The following notations will be used throughout the class.

Dep. of Mathematics, UoM

4
 In19-MA4043

Table 1.2

𝑥$ , 𝑦& Activations of units 𝑋$ , 𝑌& , respectively:

For input units 𝑋$ , 𝑥$ =input signal;
For other units 𝑌& , 𝑦& = 𝑓(𝑦$%! )
𝑤$& Weight on connection from unit 𝑋$ to unit 𝑌&
𝑏& Bias on unit 𝑌& :
A bias acts like a weight on connection from a unit
with a constant activation of 1
𝑦$%! Net input to unit 𝑌&
𝑾 Weight matrix
𝒘.𝒋 Vector of weights: the jth column of the weight matrix
𝒘.𝒋 = (𝑤!& , 𝑤"& , … … … … … , 𝑤%& )
𝒔 Training input vector
𝒔 = (𝑠! , 𝑠" , … … … , 𝑠$ , … … … , 𝑠% )
𝒕 Training (or target) output vector
𝒕 = (𝑡! , 𝑡" , … … … , 𝑡$ , … … … , 𝑡% )
𝒙 Input vector
𝒙 = (𝑥! , 𝑥" , … … … , 𝑥$ , … … … , 𝑥% )
𝛼 Learning rate:
The learning rate is used to control the amount of
weight adjustment at each step training
𝜃 Threshold

Matrix multiplication method for calculating net input

If the connection weights for a neural net are stored in a matrix 𝑾 where
𝑤!! ⋯ 𝑤!"
𝑾=# ⋮ ⋱ ⋮ (,
𝑤#! ⋯ 𝑤#"

(1.2)

the net input to 𝑌& (with no bias on unit 𝑗) is simply the dot product of the vectors 𝒙 =
(𝑥! , 𝑥" , … … … , 𝑥$ , … … … , 𝑥% ) and 𝒘.𝒋 (the jth column of the weight matrix):
𝑦$%! = 𝒙 ∙ 𝒘.𝒋
= ∑%$)! 𝑥$ 𝑤$&
(1.3)

Dep. of Mathematics, UoM

5
 In19-MA4043

Bias

A bias can be included by adding a component 𝑥* = 1 to the vector 𝒙, i.e. 𝒙 =

(1, 𝑥! , 𝑥" , … … … , 𝑥$ , … … … , 𝑥% ). The bias is treated exactly like any other weight, i.e. 𝑤*& = 𝑏& .
The net input to unit 𝑌& is given by
𝑦$%! = 𝒙 ∙ 𝒘.𝒋
= ∑%$)* 𝑥$ 𝑤$&
= 𝑤*& + ∑%$)! 𝑥$ 𝑤$&
= 𝑏& + ∑%$)! 𝑥$ 𝑤$&
(1.4)

Figure 1.4: A Neuron with a Bias

Examples 1.1

(a) Given a two-input neuron with the following parameters: 𝑏 = 1.2, 𝑾 = [3 2] and 𝒙 =
[−5 6]+ ,calculate the neuron output for the following transfer functions:
(i) A symmetrical hard limit transfer function
(ii) A saturating linear transfer function
(iii) A hyperbolic tangent sigmoid (tansig) transfer function

(b) Given a two-input neuron with the following weight matrix and input vector: 𝑾 =
[3 2] and 𝒙 = [−5 7]+ , we would like to have an output of 0.5. Do you suppose that
there is a combination of bias and transfer function that might allow this?
(i) Is there a transfer function from Table 1.1 that will do the job if the bias is zero?
(ii) Is there a bias that will do the job if the linear transfer function is used? If yes, what
is it?
(iii) Is there a bias that will do the job if a log-sigmoid transfer function is used? Again,
if yes, what is it?
(iv) Is there a bias that will do the job if a symmetrical hard limit transfer function is
used? Again, if yes, what is it?

Dep. of Mathematics, UoM

6