L3: Networks of Artificial Neurons, Single Layer Perceptrons

1. Networks of McCulloch-Pitts Neurons

2. Single Layer Feed-Forward Neural Networks: The Perceptron

3. Implementing Logic Gates with McCulloch-Pitts Neurons

4. Finding Weights Analytically

5. Limitations of Simple Perceptrons

6. Introduction to More Complex Neural Networks

7. General Procedure for Building Neural Networks

 Networks of McCulloch-Pitts Neurons
A single neuron can't do much by its own. Usually we will have many neurons,
labeled by indices 1, 2, i, j, k, and activation flows between them via synapses
with strengths wki , wij

I1
I2

Oi
Iij
∑ wij

In 

neuron i neuron j
synapse ij

𝑛
𝐼𝑘𝑖 = 𝑂𝑘 ∙ 𝑤𝑘𝑖 𝑂𝑖 = 𝑠𝑔𝑛( 𝑘=1 𝐽𝑘𝑖 − 𝜃𝑖 ) 𝐼𝑖𝑗 = 𝑂𝑖 ∙ 𝑤𝑖𝑗
 The Perceptron
We can connect any number of McCulloch-Pitts neurons together in any way
we like.
An arrangement of one input layer McCulloch-Pitts neurons feeding forward to
one output layer of McCulloch-
Pitts neurons is known as a
1 1 1
Perceptron.

2 2 2
𝑂 𝑠𝑔𝑛 ∑ 𝑂 𝑤 − 
. .
. .
. .

n m
m
Already this is a powerful
computational device. Later we shall see variations that make it even more
powerful.
 Implementing Logic Gates with M-P Neurons

We can use McCulloch-Pitts neurons to implement the basic logic gates.

All we need to do is find the appropriate connection weights and neuron
thresholds to produce the right outputs for each set of the inputs.
We shall see explicitly how one can construct simple networks that perform
NOT, AND, and OR.
It is then a well known result from logic that we can construct any logical
function from these three operations.
The resulting networks, however, will usually have a much more complex
architecture than a simple Perceptron.
We generally want to avoid decomposing complex problems into simple logic
gates, by finding the weights and thresholds that work directly in a perceptron
architecture.
 Implementation of Logical NOT, AND, and OR

In each case, we have inputs Ii and outputs O, and need to determine the weights
and thresholds. It is easy to find solutions by inspection:

NOT AND OR
I O i1 i2 O i1 i2 O
0 1 0 0 0 0 0 0
1 0 0 1 0 0 1 1
1 0 0 1 0 1
1 1 1 1 1 1
-0.5 1.5 0.5

-1 1 1 1 1
 The Need to Find Weights Analytically
Constructing simple networks by hand is one thing. But what about harder
problems? For example, what about:

OR
i1 i2 O
?
0 0 0
0 1 1 ? ?

1 0 1
1 1 1

How long do we keep looking for a solution? We need to be able to calculate

appropriate parameters rather than looking for solutions by trial-and-error.
Each training pattern produces a linear inequality for the output in terms of the
inputs and the network parameters. These can be used to compute
 Finding Weights Analytically for the AND Network
We have two weights w and w and the threshold , and for each training pattern
we need to satisfy
−

So the training data lead to 4 inequalities:

I1 I2 O

0 0 0 −

0 1 0 −

1 0 0 −

1 1 1 − 

It is easy to see that are an infinite number of solutions. Similarly, there are an
infinite number of solutions for the NOT and OR networks.
 Limitations of Simple Perceptrons
We can follow the same procedure for the XOR network:

I1 I2 O

0 0 0 −

0 1 1 −

1 0 1 −

1 1 0 − 

Clearly, the second and third inequalities are incompatible with the fourth, so
there is in fact no solution. We need more complex networks, e.g. that combine
together many simple networks, or use different activation/thresholding/transfer
functions.
It then becomes much more difficult to determine all the weights and thresholds
by hand. Next lecture we shall see how a neural network can learn these
parameters.
First, we need to consider what these more complex networks might involve.
 ANN Architectures/ Structures/ Topologies
Mathematically, ANNs can be represented as weighted discrete graphs. For our
purposes, we can simply think in terms of activation flowing between processing
units via one-way connections. Three common ANN architectures are:
Single-layer Feed-Forward NNs: One input layer and one output layer of
processing units. No feedback connections. (For example, a simple Perceptron)
Multi-layer Feed-Forward NNs : One input layer, one output layer, and one or
more hidden layers of processing units. No feedback connections. The hidden
layers sit in between input and output layers, and thus are hidden from the outside
world. (For example, a Multi-Layer Perceptron.)
Recurrent NNs: Any network with at least one feed-back connection. It may, or
may not, have hidden units. (For example, a Simple Recurrent Network.)
Further interesting variations include: short-cut connections, partial connectivity,
time-delayed connections, Elman networks, Jordan Networks, moving windows,
…
 Examples of Network Architectures

Single-layer Multi-layer Recurrent

Feed-Forward Feed-Forward Network

Multi-Layer
Single-Layer Simple Recurrent
Perceptron
Perceptron Network
 Other Types of Activation/Transfer Function
Sigmoid Functions: These are smooth (differentiable) and monotonically
increasing.

The logistic function

𝑖𝑔𝑚𝑜𝑖

Hyperbolic tangent

Piecewise-Linear Functions: Approximations of a sigmoid functions.

𝑖
𝑖 −
𝑖
 The Threshold as a special Kind of Weight
It would simplify the mathematics if we could treat the neuron threshold as if it
were just another connection weight. The crucial thing we need to compute for
the each unit j is:
𝑂 𝑤 − 𝜃 𝑂 𝑤 𝑂 𝑤 𝑂 𝑤 − 𝜃
It is easy to see that if we define 𝑤 −𝜃 and 𝑂 , then this becomes
𝑂 𝑤 − 𝜃 𝑂 𝑤 𝑂 𝑤 𝑂 𝑤 𝑂 𝑤 𝑂 𝑤
This simplifies the basic Perceptron equation so that:
𝑂 𝑠𝑔𝑛( 𝑂 𝑤 − 𝜃) 𝑠𝑔𝑛 𝑂 𝑤
We just have to include an extra input unit with activation 𝑂 and then we
only need to compute "weights", and no explicit thresholds.
 Example: A Classification Task
A typical neural network application is classification. Consider the simple example
of classifying aeroplanes given their masses and speeds:

Mass Speed Class

1.0 0.1 Bomber
2.0 0.2 Bomber
0.1 0.3 Fighter
2.0 0.3 Bomber
0.2 0.4 Fighter
3.0 0.4 Bomber
0.1 0.5 Fighter
1.5 0.5 Bomber
0.5 0.6 Fighter
1.6 0.7 Fighter

How do we construct a neural network that can classify any Bomber and Fighter?
 General Procedure for Building Neural Networks
Formulating NN solutions for particular problems in a multi-stage process:
1. Understand and specify your problem in terms of inputs and required outputs,
e.g. for classification the outputs are the classes usually represented as binary
vectors.
2. Take the simplest form of network you think might be able to solve your
problem, e.g. a simple Perceptron.
3. Try to find appropriate connection weights (including neuron thresholds) so
that the network produces the right outputs for each input in its training data.
4. Make sure that the NW works on its training data, and test its generalization by
checking its performance on new testing data.
5. If the NW doesn't perform well enough, go back to stage 3 and try harder.
6. If the NW still doesn't perform well enough, go back to stage 2 and try harder.
7. If the NW still doesn't perform well enough, go back to stage 1 and try harder.
8. Problem solved – move on to next problem.
 Building a Neural Network for our Example
For our aeroplane classifier example, our inputs can be direct encodings of the
masses and speeds. Generally we would have one output unit for each class,
with activation 1 for 'yes' and '0' for 'no'. with just two classes here, we can have
just one output unit, with activation 1 for 'Fighter' and 0
for 'Bomber' (or vice versa). The simplest network to try
Mass
first is a simple Perceptron. We can further simplify
matters by replacing the threshold by an extra weight
as discussed above. This gives us:
𝑙 𝑠𝑠 𝑠𝑔𝑛 𝑤 𝑠𝑠 𝑤 𝑝

W0 W1 W2

1 Mass Speed

That's stages 1 and 2 done. Next lecture we begin a systematic look at how to
proceed with stage 3, first for the Perceptron, and then for more complex types
of networks.
 Conclusion

1. Biological neurons, consisting of a cell body, axons, dendrites and synapses,

are able to process and transmit neural activation.

2. The McCulloch-Pitts neuron model (Threshold Logic Function) is a crude

approximation to real neurons that perform a simple summation and
thresholding function on activation levels.

3. Appropriate mathematical notation facilitates the specification and

programming of artificial neurons and networks of artificial neurons.

References:

1. Haykin: sections 1.4 and 1.6.

2. Callan section 1.1 and 1.2
3. Gurney: sections 3.1 and 3.2
𝐼 𝑂 ∙𝑤 𝑂 𝑠𝑔𝑛 𝐽 −𝜃 𝐼 𝑂 ∙𝑤

𝑖
𝑠𝑔𝑛
𝑖

𝑖𝑔𝑚𝑜𝑖
In1

In2

Out
∑
Inn