![The Perceptron (1958)
“A machine which senses, recognizes, remembers, and responds like the human mind”
“Remarkable machine… [was] capable of what amounts to thought” - The New Yorker](https://image.slidesharecdn.com/neuralnetworksanddeeplearning-140117060742-phpapp02/85/Neural-networks-and-deep-learning-14-320.jpg)
Neural networks and deep learning
- 1. Neural Networks and Deep Learning Tijmen Blankevoort
- 2. Scyfer Prof dr. Max Welling Drs. Jorgen Sandig Msc. Taco Cohen
- 3. Deep Learning All purpose machine learning Using Neural Networks: - Using large amounts of data - Learning very complex problems - Automatically learning features A new era of machine learning
- 4. Deep learning wins all competitions - IJCNN 2011 Traffic Sign Recognition Competition - ISBI 2012 Segmentation of neuronal structures in EM stacks challenge - ICDAR 2011 Chinese handwriting recognition
- 5. Applications A lot of state of the art systems use deep learning to some extent: - IBMs Watson: Jeopardy contest 2011 - Google’s self-driving car - Google Glasses - Facebook face recognition - Facebook user modelling Mostly image and sound recognition tasks (difficult)
- 6. Google Brain (2011) - 10 million youtube/imagenet images - 1 billion parameters - 16.000 processors - Largely unsupervised! - 20.000 categories - 15.8% accuracy
- 7. Bigger, better Deep Learning: - The scope of what computers can learn has greatly been increased - Interaction with the real world
- 10. Activation function Sigmoid activation function
- 11. Neuron computer model Perceptron - 1957 Rosenblatt
- 12. Easy functions with a neuron
- 13. Linking neurons and training - Initialize randomly - Sequentially give it data. - See what the difference is between network output and actual output. - Update the weights according to this error. - End result: give a model input, and it produces a proper output. Quest for the weights. The weights are the model!
- 14. The Perceptron (1958) “A machine which senses, recognizes, remembers, and responds like the human mind” “Remarkable machine… [was] capable of what amounts to thought” - The New Yorker
- 15. Criticism and downfall (1969) - Perceptrons are painfully limited. They can not even learn a simple XOR function! - No feasible way of learning networks with multiple layers - Interest in neural networks close to fully disappeared
- 16. Renewed interest (90’s) - Learning multiple layers - “Back propagation” - Can theoretically learn any function! But… Very slow and inefficient - Machine learning attention towards SVMs, random forests etc.
- 17. Deep learing (2006) - Quest: Mimic human brain representations - Large networks - Lots of data Problem: Simple back propagation fails on large networks.
- 18. Deep learning (2006) - Exactly same networks as before, just BIGGER - Combination of three factors: - (Big data) - Better algorithms - Parallel computing (GPU)
- 19. Better algorithms Restricted Boltzmann machine Pre-training: Learn the representation by parts! Very strong unsupervised learning After pre-training, use back propagation
- 20. Parallel (GPU) power - Every set of weights can be stored as a matrix (w_ij) - GPUs are made to do common parallel problems fast! - All similar calculations done at the same time, huge performance boost. - CPU parallelizing
- 21. Future of Deep Learning - Currently an explosion of developments - Hessian-Free networks (2010) - Long Short Term Memory (2011) - Large Convolutional nets, max-pooling (2011) - Nesterov’s Gradient Descent (2013) - Currently state of the art but... - No way of doing logical inference (extrapolation) - No easy integration of abstract knowledge - Hypothetic space bias might not conform with reality
- 22. When to apply Deep Learning - Generally, vision and sound recognition, but... - Works great for any other problem too! - A lot of data / features - Don’t want to make your own features - State of the art results
- 23. How to apply Deep Learning Deep learning is very difficult! - No easy plug and play software - Far too many different networks/options/additions - Mathematics and programming very challenging - Research is fast paced - Learning a network is both an art and a science My advice: Cooperation university <=> business
- 24. How to apply Deep Learning - For most current business problems, no need for expensive hardware. e.g. we use a laptop