Scribd
Upload
19 views

ResNet Overview

The document summarizes ResNet, a residual neural network architecture that won 1st place in the ILSVRC 2015 image classification competition. It discusses how ResNet addresses the problem of accuracy saturation and degradation in very deep networks through the use of "shortcut connections" that skip layers, allowing gradients to backpropagate more easily. The document then outlines the ResNet architecture and implementation using Keras, showing residual blocks, the overall structure, and results demonstrating ResNet can efficiently train networks with over 100 layers.

Uploaded by

TKNgu
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
19 views

ResNet Overview

The document summarizes ResNet, a residual neural network architecture that won 1st place in the ILSVRC 2015 image classification competition. It discusses how ResNet addresses the problem of accuracy saturation and degradation in very deep networks through the use of "shortcut connections" that skip layers, allowing gradients to backpropagate more easily. The document then outlines the ResNet architecture and implementation using Keras, showing residual blocks, the overall structure, and results demonstrating ResNet can efficiently train networks with over 100 layers.

Uploaded by

TKNgu
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 17

ResNet Overview*

Implementation using Keras

* Slides are modified from the original paper Deep Residual Learning for Image Recognition by Kaiming He, Xiangyu 1
Zhang , Shaoqing Ren and Jian Sun
Winning Model
 Won 1st place in the ILSVRC 2015
classification competition with top-5
error rate of 3.57%
 Won the 1st place in ILSVRC and
COCO 2015 competition in
ImageNet Detection, ImageNet
localization, Coco detection and
Coco segmentation.
 Replacing VGG-16 layers in Faster
R-CNN with ResNet-101. They
observed a relative improvements
of 28%
 Efficiently trained networks with 100
and 1000 layers.

2
3
4
5
Problem
• With the network depth
increasing, accuracy
gets saturated and then
degrades rapidly.
• A huge barrier to
training NN is vanishing
gradients: very deep
networks often have a
gradient signal that goes
to zero quickly, thus
making gradient descent
unbearably slow
6
Sequential Block

z[l+1] = W[l+1] * a[l] + b[l+1]


a[l+1] = g(z[l+1])
a[l+1]
a[l] a[l+2] z[l+2] = W[l+2] * a[l+1] + b[l+2]
a[l+2] = g(z[l+2])

a[l] -> Linear -> ReLu -> Linear -> ReLu


7
Residual Block (“shortcut" or "skip connection”)

z[l+1] = W[l+1] * a[l] + b[l+1]


a[l+1] = g(z[l+1])
a[l+1]
a[l] a[l+2] z[l+2] = W[l+2] * a[l+1] + b[l+2]
a[l+2] = g(z[l+2] + a[l])

a[l] -> Linear -> ReLu -> Linear -> ReLu


8
Building block

weight layer
relu

weight layer
H(x)

Plain block Residual block

9
Minimizing high-dimensional non-convex loss function

10
ResNet Structure

11
Implementation
The residual blocks
are based on the
improved scheme
proposed in “Identity
Mappings in Deep
Residual Networks”
by Kaiming He,
Xiangyu Zhang,
Shaoqing Ren, and
Jian Sun.

12
Architecture

13
14
Results

15
What’s next?

16
References
• Li, Hao, Zheng Xu, Gavin Taylor, and Tom Goldstein. "Visualizing the loss landscape of neural
nets." arXiv preprint arXiv:1712.09913 (2017).

• Huang, Gao, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. "Densely
Connected Convolutional Networks." In CVPR, vol. 1, no. 2, p. 3. 2017.

• He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Identity mappings in deep residual
networks." In European conference on computer vision, pp. 630-645. Springer, Cham, 2016.

• He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image
recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 770-778. 2016.

• Andrew Ng “Deep Learning” course at Coursera

• Article @ Medium “Understanding and Implementing Architectures of ResNet and ResNeXt for
state-of-the-art Image Classification: From Microsoft to Facebook”

17

You might also like