DeepLearning 01 Print1-Unlocked
DeepLearning 01 Print1-Unlocked
Learning Objectives
Historical Overview
Source:
https://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_pa
Auto.-Sys: Deep Learning rt1.html, 21.09.2017 Prof. Dr. N. Stache 8
Historical Overview 1943
ݕ
X Only binary inputs, inhibitory inputs ݔ ݑ
binary output
X No trainable weights ݔାଵ
inhibitory ݔା
inputs
Source:
https://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_pa
Auto.-Sys: Deep Learning rt1.html, 21.09.2017 Prof. Dr. N. Stache 9
Source:
https://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_pa
rt1.html, 21.09.2017;
Auto.-Sys: Deep Learning https://upload.wikimedia.org/wikipedia/en/5/52/Mark_I_perceptron.jpeg, Prof. Dr. N. Stache 10
03.10.2018
Historical Overview 1986 - 1995
But:
X Failed to train deep structures
X High computational effort
X Alternative Approach: Support Vector Machine
Æ 2nd AI Winter
Source:
http://deeplearning.net/tutorial/mlp.html, 03.10.2018
https://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_pa
Auto.-Sys: Deep Learning rt1.html, 21.09.2017
Prof. Dr. N. Stache 11
Source:
https://cdn.edureka.co/blog/wp-content/uploads/2017/05/Deep-Neural-
Network-What-is-Deep-Learning-Edureka.png, 03.10.2018
https://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_pa
Auto.-Sys: Deep Learning rt1.html, 21.09.2017 Prof. Dr. N. Stache 12
What is AI?
Artificial Intelligence
Minimax-Tree:
X Player x wins at maximized score, Player y wins at minimized
score
Auto.-Sys: Deep Learning Source: Russel, Norvig: Künstliche Intelligenz, Pearson Prof. Dr. N. Stache 14
What is Machine Learning?
Auto.-Sys: Deep Learning Source: Goodfellow et. al.: Deep Learning, MIT Press, 2016. Prof. Dr. N. Stache 15
X Machine learning:
“A field of study that gives computers the ability to learn without being
explicitly programmed” (Arthur Samuel)
X More specific:
X Data used to build models
X Models used to make predictions on new data
Features Result
X Example data:
Hours of Hours of Exam
study sleep passed
1 8 0
Entities, 10 6 1
Observations
Instances 20 4 0
11 10 1
X Examplary task: try to predict if exam is passed, given hours of study and
hours of sleep Æ so-called supervised learning
Auto.-Sys: Deep Learning Prof. Dr. N. Stache 16
Overview on Machine Learning
input compare
output
model
Auto.-Sys: Deep Learning Source: Trask: Grokking Deep Learning, Manning Publications, 2017. Prof. Dr. N. Stache 17
Auto.-Sys: Deep Learning Source: Trask: Grokking Deep Learning, Manning Publications, 2017. Prof. Dr. N. Stache 18
Two Main Applications of
Supervised Learning passed
Hours of study
failed
X Classification
X find the class boundaries
X output is categorical
(+ confidence),
e.g. exam passed, exam failed
Hours of sleep
Body size
Auto.-Sys: Deep Learning Prof. Dr. N. Stache 19
Parametric ML-Algorithms
passed
Hours of study
failed
X Parametric
X Model with fixed number of
parameters, number is
independent from data
X e.g.: model of a line
Hours of sleep
X Pro:
X Simple
X Fast to determine parameters (= train)
X Only few data is required to fit the model
X Con:
X Model is constrained to specific form
X Complexity is limited by number of parameters Æ could lead to
poor fit
Overfitting
X Goal:
Find model, using given data Æ make predictions on new data
X Example: KNN-Classification
Bildqquelle: https://kevinzakka.github.io/2016/07/13/k-nearest-neighbor/ (28.01.2018)
X Goal:
Find model, using given data Æ make predictions on new data
X Example: Regression
Training data
Real data
Model capacity: 2 Model capacity: 6
ݕ ݕ
Training error: 0.5 Training error: 0.1
Test error: 0.5 Test error: 2.3
Î “Overfitting”
ݔ ݔ
Underfitting
Training data
Model capacity: 3 Real data Model capacity: 2
ݕ Training error: 0.1 ݕ Training error: 3.8
Test error: 0.1 Test error: 3.9
Î “Underfitting”
ݔ ݔ
Auto.-Sys: Deep Learning Source: Goodfellow et. al.: Deep Learning, MIT Press, 2016. Prof. Dr. N. Stache 25
9
Auto.-Sys: Deep Learning Prof. Dr. N. Stache 26
Unsupervised Learning
Hours of study
X Input:
X ݇ … number of clusters we
want to find
X Input data points (h_sleep, h_study)
X Approach:
X Place ݇ centroids at random locations Hours of sleep
X Repeat until convergence:
X Assign each data point to the nearest centroid Æ this forms a cluster
X Recompute centroid positions with the points of each cluster
X Stop when none of the cluster assignments change
9 9
Auto.-Sys: Deep Learning Prof. Dr. N. Stache 29
Reinforcement Learning
Q-Learning
Go left: +100
Update Q
Auto.-Sys: Deep Learning Prof. Dr. N. Stache 30
Reinforcement Learning
Q-Learning
9 9 9
Auto.-Sys: Deep Learning Prof. Dr. N. Stache 32
Summary and Overview on Learning
Auto.-Sys: Deep Learning Source: Goodfellow et. al.: Deep Learning, MIT Press, 2016. Prof. Dr. N. Stache 34
Deep Learning vs. Other AI Disciplines
Legend:
Developed
Hand- by human
Input designed Output Learned by
program machine
Hand- Mapping
Input designed from Output
features features
Representation learning
Mapping
Input Features from Output
Machine Learning
features
More Mapping
Deep l.
Auto.-Sys: Deep Learning Source: Goodfellow et. al.: Deep Learning, MIT Press, 2016. Prof. Dr. N. Stache 35
What is a Feature?
X Definition:
Individual measurable property of a phenomenon being observed
[Source: Bishop, Christopher (2006). Pattern recognition and machine learning. Berlin: Springer]
Mapping
Features Features Features
Input from Output
level 1 level 2 level 3
features
Source: https://www.nature.com/news/computer-science-the-
Auto.-Sys: Deep Learning learning-machines-1.14481, Images Andrew Ng, 27.08.2017 Prof. Dr. N. Stache 37
30 28,2 300
25,8 269
25 250
20 200
Number of layers
16,4
Error (%)
15 152 150
10 100
7,3 6,7
Human 5 3,6 3
50
19 22
8
0 0
2010 2011 2012 2013 2014 2015 2016
error (%) layers
X Speech recognition
X Translation of spoken language into
nto text
X Computer vision
X Handwriting recognition
X Face & People Detection
X (Low-resolution) Object recognition
X Semantic labelling
X Applied mathematics
X Detection of objects
X Detection of lane markings
X Detection of free space +
classification of the boundary
Automotive-Related Applications
X Image Segmentation
Visualization of
relevant image
regions for automated
driving:
Fun Projects
X Style transfer
X Under Linux - open terminal, enter:
X conda create -n style-transfer1 python=3.5
X source activate style-transfer1
X pip install tensorflow
X conda install scipy pillow
X conda install -c conda-forge moviepy
X Under Windows – open terminal, enter:
X conda create -n style-transfer python=3.5
X activate style-transfer
X pip install tensorflow
X conda install scipy pillow
X conda install -c conda-forge moviepy
X Style transfer
X Download the zip archive of fast style transfer and extract it
https://github.com/lengstrom/fast-style-transfer
X Download the Rain Princess checkpoint (model with already
tuned parameters), put it into the style transfer folder
https://goo.gl/niFGRc
X Copy your image into the fast style transfer folder
X Enter in your terminal:
X python evaluate.py --checkpoint ./rain-
princess.ckpt --in-path <path_to_input_file> --
out-path ./output_image.jpg
Fun Projects
X Style transfer
X Other checkpoints:
X Rain-Princess: https://goo.gl/niFGRc
X La-muse: https://goo.gl/AqzB6P
X Udnie: https://goo.gl/B8gvAH
X Scream: https://goo.gl/dkA2K9
X Wave: https://goo.gl/UDTr3e
X Wreck: https://goo.gl/wRJh2v
X Deep Traffic
X Open: https://selfdrivingcars.mit.edu/deeptraffic/
X Modify the parameters
X lanesSide
X patchesAhead
X patchesBehind
X Number of hidden layers
X num_neurons
X Try to get it faster than 65 mph
Next week: