Deep

 Learning

Kairit  Sirts
Lecture  in  TUT  19.12.2016
Outline

• What can be done with deep learning?

• Deep learning demystified
• How can you get started with deep learning?

2
Why deep learning?
Deep learning Gradient boosting

Random Forest Linear model

3
http://www.infoworld.com/article/3003315/big-data/deep-learning-a-brief-guide-for-practical-problem-solvers.html
What can be done with deep learning?
Handwritten digit recognition

MNIST benchmark dataset

The best reported error rate is 0.21%

5
Street view number recognition

• Obtained from house numbers in

Google Street View images
• Best error rate is 1.69%

6
Image classification

7
Image classification
10 objects
6000 labeled instances for each object
Best accuracy so far 96.53%

8
Image classification

9
Image classification

20 superclasses
100 finegrained classes
600 labeled images per class
Best classification accuracy 75.72%

10
Detecting doodles

https://quickdraw.withgoogle.com
There are other simple and fun AI
experiments launched by Google
https://aiexperiments.withgoogle.com

11
Image captioning

12
Image captioning – not so great results

13
Automatic colorization of images

14
http://richzhang.github.io/colorization/resources/images/teaser3.jpg
Automatic colorization of images - failed

15
DeepDream

https://deepdreamgenerator.com
16
DeepDream

17
DeepDream

18
DeepDream

19
Word embeddings

20
http://metaoptimize.s3.amazonaws.com/cw-embeddings-ACL2010/embeddings-mostcommon.EMBEDDING_SIZE=50.png
Word embeddings
months

weekdays

numbers

21
Word embeddings

• 𝑊 man − 𝑊 woman ≈ 𝑊 king − 𝑊(queen)

• 𝑊 walking − 𝑊 walked ≈ 𝑊 swimming − 𝑊(swam)
22
Automatic text generation – pseudo Shakespeare

23
http://karpathy.github.io/2015/05/21/rnn-effectiveness
Machine translation

• Google Translate app

24
Learning to play Atari Arcade games

25
https://www.youtube.com/watch?v=cjpEIotvwFY
AlphaGo

26
https://www.youtube.com/watch?v=PQCrX1sQSzY
Other tasks tackled with deep neural networks

• Speech recognition
• Various tasks in robotics
• Log analysis/risk detection
• Recommendation systems
• Motion detection from videos
• Business and Economics analytics
• Etc …

27
Deep learning demystified
How does deep learning work?
• Biological neuron • Artificial neuron

http://www.theprojectspot.com/tutorial-post/introduction-to-artificial-neural-networks-part-1/7
29
 • Biological neural network • Artificial neural network

30
https://www.eeweb.com/blog/rob_riemen/deep-machine-learning-and-the-google-brain http://www.theprojectspot.com/tutorial-post/introduction-to-artificial-neural-networks-part-1/7
What happens inside a neuron?

<

ℎ = 𝑥7 𝑤7 + 𝑥: 𝑤: + ⋯ + 𝑥< 𝑤< = = 𝑥> 𝑤>

>?7

Output: ℎ = 𝑓(𝑧)

31
Activation function

1  if  𝑧 ≥ th 1 𝑒 E − 𝑒 DE
𝑓 𝑧 =J 𝑓 𝑧 = 𝑓 𝑧 = E 𝑓 𝑧 = max  (0, 𝑧)
0  if  𝑧 < th 1 + 𝑒 DE 𝑒 + 𝑒 DE

32
https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/neural_networks.html
Single neuron logic gates

• Threshold activation function

33
https://blog.abhranil.net/2015/03/03/training-neural-networks-with-genetic-algorithms/
XOR gate
• Cannot be done with a single neuron
• A hidden layer is necessary

𝒙𝟏 𝒙𝟐 OR NOT AND AND

0 0 𝕀 0 ∙ 1 + 0 ∙ 1 > 0.5 = 0 𝕀 0 ∙ −1 + 0 ∙ −1 > −1.5 = 1 𝕀 0 ∙ 1 + 1 ∙ 1 > 1.5 = 0
0 1 𝕀 0 ∙ 1 + 1 ∙ 1 > 0.5 = 1 𝕀 0 ∙ −1 + 1 ∙ −1 > −1.5 = 1 𝕀 1 ∙ 1 + 1 ∙ 1 > 1.5 = 1
1 0 𝕀 1 ∙ 1 + 0 ∙ 1 > 0.5 = 1 𝕀 1 ∙ −1 + 0 ∙ −1 > −1.5 = 1 𝕀 1 ∙ 1 + 1 ∙ 1 > 1.5 = 1
1 1 𝕀 1 ∙ 1 + 1 ∙ 1 > 0.5 = 1 𝕀 1 ∙ −1 + 1 ∙ −1 > −1.5 = 0 𝕀 1 ∙ 1 + 0 ∙ 1 > 1.5 = 0
34
https://blog.abhranil.net/2015/03/03/training-neural-networks-with-genetic-algorithms/
How to assign weights?

8Y9+9Y9+9Y9+9Y4=
= 270 weights

35
http://neuralnetworksanddeeplearning.com/
Backpropagation

• Standard and efficient method for training neural networks

• The general idea:
• Compute the error with a forward pass
• Propagate the error back to change the weights such that the error would become smaller

ERROR à ERROR’
ERROR’ < ERROR

36
Diversion to calculus - derivative

• 𝑦_ = 𝑓 _ 𝑥
• Derivative is the slope of the tangent
line
• It is the rate of change when going in
the direction of steepest ascent

37
Derivatives

• When 𝑓 _ 𝑥 = 0 then it is the local or

global maximum or minimum or a
saddle point
• When 𝑓 _ 𝑥 > 0 then the function is
increasing
• When 𝑓 _ 𝑥 < 0 then the function is
decreasing

38
Gradients
• Generalization of derivatives to
multivariate functions
• Derivative is a vector pointing to the
direction of steepest ascent
ab ab
• ∇𝑓(𝑥, 𝑦) = ,
ac ad
ab ab
• , - partial derivatives – take
ac ad
derivative wrt one variable while
treating all others as constant

39
Gradients and backpropagation

• Backpropagation is used to compute the gradients with respect to all parameters in a

neural network.
• The gradients are then used in a general method of gradient descent for minimizing
functions.
• We want to minimize the cost function that measures the error made by the neural
network.
• In order to do that we need to move to the direction of deepest descent given by the
gradients.

40
Gradient descent
• An iterative algorithm
• Start with initial parameter values 𝜃 f
• Update parameters iteratively until
convergence:
𝜃 gh7 =:  𝜃 g − 𝛼∇𝑓 𝜃
• 𝛼 - learning rate, controls the step size

41
Deep learning demystified
How does backpropagation work?
Backpropagation explained

• Example from:
https://mattmazur.com/2015/03/17/

• 2 inputs
• 1 hidden layer with 2 neurons
• Bias terms in both the hidden and
output layer
• 2 outputs

43
Initial configuration

• Training values

• Initial weights: 𝑤7 , … , 𝑤l

• Initial biases: 𝑏7 , 𝑏:

44
Forward pass – first hidden unit

45
Forward pass – first hidden unit

46
Forward pass – second hidden unit

47
Forward pass – first output unit

48
Forward pass – second output unit

49
Forward pass – error of the first output

50
Forward pass – output error

51
Forward pass – output error

52
Backwards pass
• Consider 𝑤n
• How much a change in 𝑤n affects the
total error?
• Apply the chain rule:

53
Chain rule
• Formula for computing derivative of the composition of two or more functions
• 𝐹 𝑥 ≡ 𝑓(𝑔 𝑥 ) ≡ (𝑓 ∘ 𝑔)(𝑥) – composition of functions 𝑓 and 𝑔
• 𝐹 _ 𝑥 = 𝑓 _ 𝑔 𝑥 𝑔_ 𝑥

• 𝐹 𝑥 =   𝑒 sc 𝑔 𝑥 = 3𝑥 𝑓 𝑔 𝑥 = 𝑒 u(c) = 𝑒 sc

• 𝐹 _ 𝑥 = 𝑓 _ 𝑔 𝑥 𝑔_ 𝑥 = (𝑒 u(c) )′𝑔′(𝑥) = 𝑒 u c (3𝑥)′ = 𝑒 sc Y 3 = 3𝑒 sc

54
Backwards pass
• Consider 𝑤n
• How much a change in 𝑤n affects the
total error?
• Apply the chain rule:

55
How much does error change wrt the output?

56
How much does output change wrt its net input?

57
Derivative of the sigmoid function

1
𝑓 𝑧 =
1 + 𝑒 DE

𝑓 _ 𝑧 = 𝑓(𝑧)(1 − 𝑓 𝑧 )

58
How much does output change wrt its net input?

59
How much does net input change wrt 𝑤n ?

60
Putting it all together

61
This is known as the delta rule

• Delta rule is the gradient descent rule for updating the weights of the inputs to
neurons in a single-layer neural network

62
Apply delta rule to outer layer weights

63
Update the weights with gradient descent
• set learning rate 𝛼 = 0.5 𝜽𝒕h𝟏 =:  𝜽𝒕 − 𝜶𝜵𝒇 𝜽

64
Backpropagation to hidden layer

• Continue backwards pass to

calculate new values for 𝑤7 , 𝑤: , 𝑤s
and 𝑤|

65
BP through hidden layer

• 𝑜𝑢𝑡€7 affects both 𝑜7 and 𝑜: and thus

needs to take into account both:

66
BP through hidden layer
• Consider one of those:

• First term can be calculated using values

computed before:

• Second term is just 𝑤n

67
BP through hidden layer
• Plug the values in:

• Compute the same value for 𝑜: :

• Compute the total:

68
BP through hidden layer
a•‚gƒ„ a<…gƒ„
• Next we need and for each
a<…gƒ„ a†
weight 𝑤

• Compute the partial derivative wrt a weight

69
BP through hidden layer
• Putting it together

• We can now update 𝑤7

70
BP through hidden layer
• Compute the partial derivatives in the same
way for 𝑤: , 𝑤s and 𝑤|
• Update 𝑤: , 𝑤s and 𝑤|

71
After first update with backpropagation

72
Did the error decrease?

• Old error was: 0.298371109

• Improvement: 0.007343335

• After 10000 updates the error will be

ca 0.000035085
• The generated outputs will be
0.015912196 for 0.01 target and
0.984065734 for 0.99 target

73
In conclusion
• Neural networks consist of artificial neurons organized into layers and connected
to each other with learnable weights.
• Backpropagation with gradient descent is the standard method for training neural
networks.
• Backpropagation can be used to compute the gradients of a neural network,
regardless of the depth of the network.
• Of course, there are other important tricks and tips but this is the basis of
understanding neural networks and deep learning.

74
Common neural network architectures
Feed-forward network

• Simplest type of neural network

• Connections between units do not
form cycles
• Information always moves in one
direction
• It never goes backwards

76
https://upload.wikimedia.org/wikipedia/en/5/54/Feed_forward_neural_net.gif
Recurrent neural network

• Connections between units form cycles

• They possess internal memory – they “remember” the past inputs
• Suitable for modeling sequential/temporal data, such as for instance text and
language data

77
Convolutional neural networks

• Convolutional layers have neurons

arranged in 3 dimensions
• Especially suitable for processing
image data

78
http://parse.ele.tue.nl/education/cluster2
Autoencoders
• Output layer attempts to reconstruct
the input
• Used for unsupervised feature learning
• The hidden layer has typically less
neurons, thus performing data
compression

79
Getting started with neural networks
Courses and tutorials
• https://www.coursera.org/learn/machine-learning -
• Introductory course on machine learning, provides necessary background

• https://www.coursera.org/learn/neural-networks
• Course on neural networks – assumes knowledge about machine learning

• http://ufldl.stanford.edu/tutorial/
• Tutorial on deep learning but covers also some simpler machine learning

• http://cs231n.stanford.edu/
• Course on convolutional neural networks

• https://www.udacity.com/course/deep-learning--ud730
• Course on deep learning

• There are many others … just google …

81
Books

• http://www.deeplearningbook.org/
• Deep Learning: A Practitioner’s approach – not released yet
• Fundamentals of deep learning – not released yet

• See more from:

• http://machinelearningmastery.com/deep-learning-books/

82
Low level libraries
• Theano - http://deeplearning.net/software/theano/
• Tensorflow - https://www.tensorflow.org/get_started/
• Python-based
• Automatic differentiation
• Can use cuda for computing on GPU

• Torch – http://torch.ch/
• Based on Lua
• Modular pieces that are easy to combine
• Lots of pretrained models

• See more: https://deeplearning4j.org/compare-dl4j-torch7-pylearn

83
Higher level libraries
• Keras - https://keras.io/
• On top of theano and tensorflow
• Based on python
• Modular
• Supports both convolutional and recurrent networks
• Supports arbitrary connectivity
• Runs on both CPU and GPU

84
Keras – example code

85
What else?
• Take the Machine Learning course in spring semester
• Use neural networks for your thesis work
• Potential supervisors in UT:
• Kairit Sirts (problems involving natural language)
• Mark Fishel (machine translation)
• Raul Vicente (computational neuroscience)
• Ilya Kuzovkin (computational neuroscience)

• Potential supervisors in TUT

• Juhan Ernits
• Tanel Alumäe (speech data)
• There are possibly others

86
In conclusion - Deep learning
• Can be used to solve very complex problems
• Based on artificial neural networks with many hidden layers
• Each artificial neuron is a simple computational unit
• Neural networks are trained with gradient descent algorithm
• Backpropagations algorithm is used to compute the gradients with respect to
tunable parameters
• There are many tutorials and online courses about deep learning
• There are various software libraries that enable to get started with deep learning
relatively easily
87