Intro to Neural Networks

Part 1: NetworkBasics

1
Image Classification

input classifie output

r

“cat
”

“5” 2
Semantic
Segmentation
“a label
for each
pixel”

3
Neural Networks
 Machine learning technique
 Often used for classification,
semantic segmentation, and related
tasks
 First ideas discussed in the
1950/60ies
 Theory work on NNs in the 1990ies
 Increase in attention from 2000 on
 Deep learning took off around 2010
 CNNs for image tasks from 2012 on 4
 Part 1
Neural Networks Basics

5
 Neural Network

What is a neuron? What is a network?

fundamental unit connected

(of the brain) elements

neural networks are

connected elementary
(computing) units
6
Biological Neurons
Biological neurons are the
fundamental units of the brain
that
 Receive sensory input from the
external world or from other
neurons
 Transform and relay signals
 Send signals to other neurons and
also motor commands to the
muscles
7
Artificial Neurons
Artificial neurons are the
fundamental units of artificial neural
networks that
 Receive inputs
 Transform information
 Create an output

8
Neurons
 Receive inputs / activations from
sensors or other neurons
 Combine / transform information
 Create an output / activation

9
Neurons as Functions
We can see a neuron as a function
 Input given by
 Transformation of the input data
can be described by a function
 Output

10
Neural Network
 NN is a network/graph of neurons
 Nodes are neurons
 Edges represent input-output
connections of the data
flow

input computations output

11
Neural Network as a
Function
 The whole network is again a
function
 Multi-layer perceptron or MLP is
often seen as the “vanilla”
neural network

input layer hidden layers output

layer

12
Neural Networks are Functions
 Neural networks are functions
 Consist of connected artificial
neurons
 Input layer takes (sensor) data
 Output layer provides the
function result (informationor
command)
 Hidden layers do some
computations
input layer hidden layers output layer
13
Different Types of NNs
 Perceptron
 MLP – Multilayer perceptron
 Autoencoder
 CNN – Convolutional NN
 RNN – Recurrent NN
 LSTM – Long/short term memory NN
 GANs – Generative adversarial network
 Graph NN
 Transformer
 ...
14
 [Image courtesy: van Veen]
6
1
 [Image courtesy: van Veen]
7
1
Multi-layer Perceptron (MLP)

17
Multi-layer
Perceptron Seen as a
Function
input layer hidden layers output layer

18
Image Classification Example

“cat
”

input function that maps labe

images to labels l
imag
e 19
What is the Network’s Input?

An image consists
of individual
pixels.

imag 20
e
What is the Network’s Input?

An image consists
of individual
pixel intensities
pixels.
Each pixel stores
an intensity
value.

imag 21
e
What is the Network’s Input?
pixel intensities

An image consists
of individual
pixels.
Each pixel stores
an intensity
value.

imag 22
e
What is the Network’s Input?

An image consists
of individual
pixels.
Each pixel stores
an intensity
value.
We have N+1 such
intensity values.
23
What is the Network’s Input?

Arrange all the

intensity
values
in a N+1 dim vector.
24
What is the Network’s Input?

Arrange all the

intensity
values
in a N+1 dim vector.
25
What is the Network’s Input?

Arrange all the

intensity
values
in a N+1 dim vector.
26
Input Layer of the
Network

This vector is
the input layer
of our network!
27
What is the Network’s
Output?

“cat
”

28
What is the Network’s
Output?

Is it a...
cat or a
dog or a
human
or a
...?

29
What is the Network’s
Output?

Is it a...
cat or a
dog or a
human
or a
...?

indicato
r
30
vector
What is the Network’s
Output?

Is it a...
cat or a
dog or a
human
or a
...?

indicato
r
31
vector
What is the Network’s
Output?

Is it a...
cat or a
dog or a
human
or a
...?

we are
never
certain.. 32

.
Output of the
Network

the output layer is

vector indicating an
activation/ likelihood
33
for each label
Image Classification

“cat”

largest
value

pixels intensities output layer is a

are the values vector of likelihoods
of the input for the possible labels
layer 34
Multi-layer Perceptron
Let’s Look at a Single
Neuron

35
Multi-layer Perceptron
Let’s Look at a Single
Neuron

36
Perceptron (Single Neuron)

output

How does this

inputs function look like?
37
Perceptron (Single Neuron)

output activation
for the next layer

activations from weights

previous layer
38
Function Behind a
Neuron

(input) activations

weights

bias

activation function

output activation

39
Function Behind a
Neuron
A neuron gets activated ( ) through
 A weighted sum of input activations
 A bias activation
 An activation function

40
Similarity to Convolutions?
 A neuron is similar to a
convolution
 Remember linear shift-invariant
kernels used as local
operators
This part looks like the
convolutions used for defining
local operators

Additionally: activation function and

41
bias
 Activation
Function
Biological neurons are either
active or not active
 We can see this as a step
function:

“activated”

“no activation”

 Bias tells us where the

activation happens 42
Activation
Function
“activated”

“no activation”

 We can model this behavior

through

 Non-smooth functions (eg, steps) have

disadvantages later down the line...
43
 Sigmoid Activation
Function
Common activation function is a
sigmoid (also called logistic function)
 Smooth function
 Squeezes values to [0,1]

44
 ReLU Activation
Function
Most commonly used one is the
so- called “rectified linear unit” or
ReLU

 Often advantages for deep
networks

45
 Neuron
Activation
A neuron is only activated if

“no activation” “activated”

(identity)

 If
 the weighted activations are larger than the
46
negative bias
Common Activation
Functions
There are different activation functions
 sigmoid()
 ReLU()
 tanh()
 atan()
 softplus()
 identity()
 step-function()
 …
ReLU is often used
47
 Illustration

[Courtesy of S. Sharma]
9
4
 Function Behind a
Neuron
Neuron gets activated if the
weighted sum of input activations is
large enough (larger thanthe
negative bias)

 This is the case for all neurons in

the neural network
50
For All
Neurons...

These are a lot of 51

values!
Let’s Use a Matrix
Notation

51
Each Layer Can Be Expressed
Through Matrix
Multiplications
layer 1 layer 0

52
Do It Layer by Layer...

53
Do It Layer by Layer...

input = layer 0

layer 1

layer 2

layer k = output

That not much more than linear

algebra... 54
Feedforward Networks
 MLPs are feedforward networks
 The information flows form left to
right
 There are no loops

 Such networks are called

feedforward networks
 There exist other variants (eg,
RNNs) 55
 Example:
Handwritten Digit
Recognition

56
Handwritten Digit
Recognition

=5
28x28 pixel image
[Image courtesy: Nielsen] 58
Handwritten Digit
Recognition

[Image courtesy: Nielsen/Lecun] 59

A Basic MLP Recognizing
Digits

[Image courtesy: Nielsen] 60

Images to Digits - A
Mapping from 784 to 10
Dimensions

28x28
pixel
input
images outpu
t
(784 vector
dim) (10 dim)
[Partial image courtesy: Nielsen] 61
What Happens in the Layers?

61
What Happens in the 1st Layer?

pixel
value
s

784 input activations = pixel intensities

784 weights = weights for pixel intensities
62
What Happens in the 1st Layer?
784 input activations = pixel intensities
784 weights = weights for pixel intensities
treat activations and weights as
images

-1 0 +1
white

black
pixel values weights effect on (rest
the doesn’t
weighted sum matter)
63
What Happens in the 1st Layer?

-1 0 +1

pixel
value weights tell
s what
us matters for
activating the
neuron!

individual “weight images” for a

neuron support individual patterns in 64

the image
Link to Local Operators
Defines Through Convolutions
 Direct link to defining
image operators through
-1 0 +1
convolutions

Here:
 Global (not local) operators
 Weight matrix does not
weights tell (yet) “slide over image”
us what
matters

65
 Weights & Bias =
Patterns
Weights define the patterns to
look for in the image
 Bias tells us how well the image
must
match the pattern
 Activation functions “switches the
neuron on” if it matches the
pattern

66
What Happens in the 2nd Layer?
 The weights in layer 2 tell us which
1st layer patterns should be
combined
 The deeper we go, the more
patterns get arranged and combined

 The last layer decides, which

final patterns make up a digit
67
What Happens in the Layers?

raw simple combine pattern

pixels pattern d to
s
s patterns digits
[Image courtesy: Nielsen] 68
No Manual
Features

Compared to several other

classifiers, this network also
includes the feature computations –
it operates directly on the input
data, no manual features!

raw simple combine pattern

pixels pattern d to
s
s patterns digits
[Image courtesy: Nielsen] 69
Classification
Performance

Such a simple MLP achieves a correct

classification for ~96% of the
examples
[Partial image courtesy: Nielsen] 71
 Classification
Performance
A simple MLP achieves a
classification accuracy of ~96%
 Note that there are tricky cases

 That is a good performance for a

simple model!
 Improved networks achieve
~99% 71
How to
Design a Neural
Network?

72
 How to Make the
Network Compute What
We Want?
So far, the network is a recipe for
sequentially performing
computations
 Structure and parameters are
the
design choices
 How to set them?

Learning! 73
Summary – Part 1
 What are neurons and neural
networks
 Lots of different networks exists
 Focus: multi-layer perceptrons (MLP)
 Activations, weights, bias
 Networks have many parameters
 “It’s just a bunch of matrices and
vectors”
 MLP for simple image
classification 74

 Part 2: Learning the

parameters