research highlights

DOI:10.1145/ 30 6 5 3 8 6

ImageNet Classification with Deep

Convolutional Neural Networks
By Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton

Abstract that were widely investigated in the 1980s. These networks

We trained a large, deep convolutional neural network to used multiple layers of feature detectors that were all learned
classify the 1.2 million high-resolution images in the from the training data. Neuroscientists and psychologists had
ImageNet LSVRC-2010 contest into the 1000 different hypothesized that a hierarchy of such feature detectors would
classes. On the test data, we achieved top-1 and top-5 error provide a robust way to recognize objects but they had no idea
rates of 37.5% and 17.0%, respectively, which is considerably how such a hierarchy could be learned. There was great excite-
better than the previous state-of-the-art. The neural network, ment in the 1980s because several different research groups
which has 60 million parameters and 650,000 neurons, con- discovered that multiple layers of feature detectors could be
sists of five convolutional layers, some of which are followed trained efficiently using a relatively straight-forward algorithm
by max-pooling layers, and three fully connected layers with a called backpropagation18, 22, 27, 33 to compute, for each image,
final 1000-way softmax. To make training faster, we used non- how the classification performance of the whole network
saturating neurons and a very efficient GPU implementation depended on the value of the weight on each connection.
of the convolution operation. To reduce overfitting in the Backpropagation worked well for a variety of tasks, but in
fully connected layers we employed a recently developed reg- the 1980s it did not live up to the very high expectations of its
ularization method called “dropout” that proved to be very advocates. In particular, it proved to be very difficult to learn
effective. We also entered a variant of this model in the networks with many layers and these were precisely the net-
ILSVRC-2012 competition and achieved a winning top-5 test works that should have given the most impressive results.
error rate of 15.3%, compared to 26.2% achieved by the sec- Many researchers concluded, incorrectly, that learning a
ond-best entry. deep neural network from random initial weights was just too
difficult. Twenty years later, we know what went wrong: for
deep neural networks to shine, they needed far more labeled
1. PROLOGUE data and hugely more computation.
Four years ago, a paper by Yann LeCun and his collaborators
was rejected by the leading computer vision conference on 2. INTRODUCTION
the grounds that it used neural networks and therefore pro- Current approaches to object recognition make essential
vided no insight into how to design a vision system. At the use of machine learning methods. To improve their perfor-
time, most computer vision researchers believed that a vision mance, we can collect larger datasets, learn more powerful
system needed to be carefully hand-designed using a detailed models, and use better techniques for preventing overfit-
understanding of the nature of the task. They assumed that ting. Until recently, datasets of labeled images were rela-
the task of classifying objects in natural images would never tively small—on the order of tens of thousands of images
be solved by simply presenting examples of images and the (e.g., NORB, 19 Caltech-101/256, 8, 10 and CIFAR-10/100 14).
names of the objects they contained to a neural network that Simple recognition tasks can be solved quite well with datas-
acquired all of its knowledge from this training data. ets of this size, especially if they are augmented with label-
What many in the vision research community failed to preserving transformations. For example, the current-best
appreciate was that methods that require careful hand-engi- error rate on the MNIST digit-recognition task (<0.3%)
neering by a programmer who understands the domain do approaches human performance.5 But objects in realistic
not scale as well as methods that replace the programmer settings exhibit considerable variability, so to learn to recog-
with a powerful general-purpose learning procedure. With nize them it is necessary to use much larger training sets.
enough computation and enough data, learning beats pro- And indeed, the shortcomings of small image datasets have
gramming for complicated tasks that require the integration been widely recognized (e.g., Ref.25), but it has only recently
of many different, noisy cues. become possible to collect labeled datasets with millions of
Four years ago, while we were at the University of Toronto,
our deep neural network called SuperVision almost halved
The original version of this paper was published in
the error rate for recognizing objects in natural images and
the Proceedings of the 25th International Conference on Neu-
triggered an overdue paradigm shift in computer vision.
ral Information Processing Systems (Lake Tahoe, NV, Dec.
Figure 4 shows some examples of what SuperVision can do.
2012), 1097–1105.
SuperVision evolved from the multilayer neural networks

84 COMMUNICATIO NS O F TH E AC M | J U NE 201 7 | VO L . 60 | NO. 6

images. The new larger datasets include LabelMe,28 which 3. THE DATASET
consists of hundreds of thousands of fully segmented ImageNet is a dataset of over 15 million labeled high-­resolution
images, and ImageNet,7 which consists of over 15 million images belonging to roughly 22,000 categories. The images
labeled high-resolution images in over 22,000 categories. were collected from the web and labeled by human labelers
To learn about thousands of objects from millions of using Amazon’s Mechanical Turk crowd-sourcing tool. Starting
images, we need a model with a large learning capacity. in 2010, as part of the Pascal Visual Object Challenge, an annual
However, the immense complexity of the object recogni- competition called the ImageNet Large-­S cale Visual
tion task means that this problem cannot be specified even Recognition Challenge (ILSVRC) has been held. ILSVRC uses a
by a dataset as large as ImageNet, so our model should also subset of ImageNet with roughly 1000 images in each of 1000
have lots of prior knowledge to compensate for all the data categories. In all, there are roughly 1.2 million training images,
we do not have. Convolutional neural networks (CNNs) con- 50,000 validation images, and 150,000 testing images.
stitute one such class of models.9, 15, 17, 19, 21, 26, 32 Their capacity ILSVRC-2010 is the only version of ILSVRC for which the
can be controlled by varying their depth and breadth, and test set labels are available, so this is the version on which we
they also make strong and mostly correct assumptions performed most of our experiments. Since we also entered
about the nature of images (namely, stationarity of statis- our model in the ILSVRC-2012 competition, in Section 7 we
tics and locality of pixel dependencies). Thus, compared to report our results on this version of the dataset as well, for
standard feedforward neural networks with similarly sized which test set labels are unavailable. On ImageNet, it is cus-
layers, CNNs have much fewer connections and parameters tomary to report two error rates: top-1 and top-5, where the
and so they are easier to train, while their theoretically best top-5 error rate is the fraction of test images for which the
performance is likely to be only slightly worse. correct label is not among the five labels considered most
Despite the attractive qualities of CNNs, and despite the probable by the model.
relative efficiency of their local architecture, they have still ImageNet consists of variable-resolution images, while
been prohibitively expensive to apply in large scale to high-­ our system requires a constant input dimensionality.
resolution images. Luckily, current GPUs, paired with a Therefore, we down-sampled the images to a fixed resolution
highly optimized implementation of 2D convolution, are of 256 × 256. Given a rectangular image, we first rescaled the
powerful enough to facilitate the training of interestingly- image such that the shorter side was of length 256, and then
large CNNs, and recent datasets such as ImageNet contain cropped out the central 256 × 256 patch from the resulting
enough labeled examples to train such models without severe image. We did not pre process the images in any other way,
overfitting. except for subtracting the mean activity over the training set
The specific contributions of this paper are as follows: we from each pixel. So we trained our network on the (centered)
trained one of the largest CNNs to date on the subsets of raw RGB values of the pixels.
ImageNet used in the ImageNet Large-Scale Visual
Recognition Challenge (ILSVRC)-2010 and ILSVRC-2012 4. THE ARCHITECTURE
competitions 2 and achieved by far the best results ever The architecture of our network is summarized in Figure 2. It
reported on these datasets. We wrote a highly optimized GPU contains eight learned layers—five convolutional and three
implementation of 2D convolution and all the other opera- fully connected. Below, we describe some of the novel or
tions inherent in training CNNs, which we make available unusual features of our network’s architecture. Sections 4.1–
publicly. a Our network contains a number of new and 4.4 are sorted according to our estimation of their impor-
unusual features which improve its performance and reduce tance, with the most important first.
its training time, which are detailed in Section 4. The size of
our network made overfitting a significant problem, even 4.1. Rectified Linear Unit nonlinearity
with 1.2 million labeled training examples, so we used several The standard way to model a neuron’s output f as a function
effective techniques for preventing overfitting, which are of its input x is with f(x) = tanh(x) or f(x) = (1 + e−x)−1. In terms
described in Section 5. Our final network contains five convo- of training time with gradient descent, these saturating
lutional and three fully connected layers, and this depth nonlinearities are much slower than the non-saturating
seems to be important: we found that removing any convolu- nonlinearity f(x) = max(0, x). Following Nair and Hinton,24
tional layer (each of which contains no more than 1% of the we refer to neurons with this non linearity as Rectified
model’s parameters) resulted in inferior performance. Linear Units (ReLUs). Deep CNNs with ReLUs train several
In the end, the network’s size is limited mainly by the times faster than their equivalents with tanh units. This is
amount of memory available on current GPUs and by the demonstrated in Figure 1, which shows the number of itera-
amount of training time that we are willing to tolerate. Our tions required to reach 25% training error on the CIFAR-10
network takes between 5 and 6 days to train on two GTX 580 dataset for a particular four-layer convolutional network.
3GB GPUs. All of our experiments suggest that our results can This plot shows that we would not have been able to experi-
be improved simply by waiting for faster GPUs and bigger ment with such large neural networks for this work if we
datasets to become available. had used traditional saturating neuron models.
We are not the first to consider alternatives to traditional
neuron models in CNNs. For example, Jarrett et al.13 claim
that the nonlinearity f(x) = |tanh(x)| works particularly well
a
 http://code.google.com/p/cuda-convnet/. with their type of contrast normalization followed by local

JU N E 2 0 1 7 | VO L. 6 0 | N O. 6 | C OM M U N IC AT ION S OF T HE ACM 85
research highlights

Figure 1. A four-layer convolutional neural network with ReLUs

net takes slightly less time to train than the one-GPU net.b
(solid line) reaches a 25% training error rate on CIFAR-10 six times
faster than an equivalent network with tanh neurons (dashed line). 4.3. Local response normalization
The learning rates for each network were chosen independently ReLUs have the desirable property that they do not require
to make training as fast as possible. No regularization of any input normalization to prevent them from saturating. If at
kind was employed. The magnitude of the effect demonstrated
here varies with network architecture, but networks with ReLUs
least some training examples produce a positive input to a
consistently learn several times faster than equivalents with ReLU, learning will happen in that neuron. However, we still
saturating neurons. find that the following local normalization scheme aids gen-
0.75
eralization. Denoting by aix, y the activity of a neuron com-
puted by applying kernel i at position (x, y) and then applying
the ReLU nonlinearity, the response-normalized activity bix, y
is given by the expression
Training error rate

0.5

where the sum runs over n “adjacent” kernel maps at the

0.25 same spatial position, and N is the total number of kernels in
the layer. The ordering of the kernel maps is of course arbi-
trary and determined before training begins. This sort of
response normalization implements a form of lateral inhibi-
tion inspired by the type found in real neurons, creating com-
0
0 5 10 15 20 25 30 35 40
petition for big activities among neuron outputs computed
Epochs using different kernels. The constants k, n, α, and β are hyper-
parameters whose values are determined using a validation
set; we used k = 2, n = 5, α = 10−4, and β = 0.75. We applied this
normalization after applying the ReLU nonlinearity in cer-
average pooling on the Caltech-101 dataset. However, on this tain layers (see Section 4.5).
dataset the primary concern is preventing overfitting, so the This scheme bears some resemblance to the local contrast
effect they are observing is different from the accelerated normalization scheme of Jarrett et al.,13 but ours would be
ability to fit the training set which we report when using more correctly termed “brightness normalization,” since we
ReLUs. Faster learning has a great influence on the perfor- do not subtract the mean activity. Response normalization
mance of large models trained on large datasets. reduces our top-1 and top-5 error rates by 1.4% and 1.2%,
respectively. We also verified the effectiveness of this scheme
4.2. Training on multiple GPUs on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test
A single GTX 580 GPU has only 3GB of memory, which limits error rate without normalization and 11% with
the maximum size of the networks that can be trained on it. normalization.c
It turns out that 1.2 million training examples are enough
to train networks which are too big to fit on one GPU. 4.4. Overlapping pooling
Therefore we spread the net across two GPUs. Current GPUs Pooling layers in CNNs summarize the outputs of neighbor-
are particularly well-suited to cross-GPU parallelization, as ing groups of neurons in the same kernel map. Traditionally,
they are able to read from and write to one another’s mem- the neighborhoods summarized by adjacent pooling units do
ory directly, without going through host machine memory. not overlap (e.g., Refs.5, 13, 20). To be more precise, a pooling
The parallelization scheme that we employ essentially puts layer can be thought of as consisting of a grid of pooling units
half of the kernels (or neurons) on each GPU, with one addi- spaced s pixels apart, each summarizing a neighborhood of
tional trick: the GPUs communicate only in certain layers. size z × z centered at the location of the pooling unit. If we set
This means that, for example, the kernels of layer 3 take s = z, we obtain traditional local pooling as commonly
input from all kernel maps in layer 2. However, kernels in employed in CNNs. If we set s < z, we obtain overlapping
layer 4 take input only from those kernel maps in layer 3
which reside on the same GPU. Choosing the pattern of con-
nectivity is a problem for cross-validation, but this allows us
b
  The one-GPU net actually has the same number of kernels as the two-GPU
net in the final convolutional layer. This is because most of the net’s param-
to precisely tune the amount of communication until it is eters are in the first fully connected layer, which takes the last convolutional
an acceptable fraction of the amount of computation. layer as input. So to make the two nets have approximately the same num-
The resultant architecture is somewhat similar to that of ber of parameters, we did not halve the size of the final convolutional layer
the “columnar” CNN employed by Cireşan et al.,4 except that (nor the fully connected layers which follow). Therefore this comparison is
biased in favor of the one-GPU net, since it is bigger than “half the size” of
our columns are not independent (see Figure 2). This scheme
the two-GPU net.
reduces our top-1 and top-5 error rates by 1.7% and 1.2%, c
  We cannot describe this network in detail due to space constraints, but it
respectively, as compared with a net with half as many kernels is specified precisely by the code and parameter files provided here: http://
in each convolutional layer trained on one GPU. The two-GPU code.google.com/p/cuda-convnet/.

86 COMMUNICATIO NS O F TH E AC M | J U NE 201 7 | VO L . 60 | NO. 6

Figure 2. An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs. One GPU
runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers.
The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 290,400–186,624–
64,896–64,896–43,264–4096–4096–1000.

3
3
3
5 3
3 3
5
11 3
2048 2048
dense
48 192 192 128
11 27 128
55
13 13 13
3 3
5
224 3 3
5 3
13
27 3 3 13 13 dense dense
11 3
55 1000
11 192 192 128 Max
pooling 2048 2048
224
Stride Max 128 Max
of 4 pooling pooling
3 48

pooling. This is what we use throughout our network, with s = of the second convolutional layer. The fourth convolu-
2 and z = 3. This scheme reduces the top-1 and top-5 error tional layer has 384 kernels of size 3 × 3 × 192, and the fifth
rates by 0.4% and 0.3%, respectively, as compared with the convolutional layer has 256 kernels of size 3 × 3 × 192. The
non overlapping scheme s = 2, z = 2, which produces output of fully connected layers have 4096 neurons each.
equivalent dimensions. We generally observe during training
that models with overlapping pooling find it slightly more dif- 5. REDUCING OVERFITTING
ficult to overfit. Our neural network architecture has 60 million parameters.
Although the 1000 classes of ILSVRC make each training
4.5. Overall architecture example impose 10 bits of constraint on the mapping from
Now we are ready to describe the overall architecture of image to label, this turns out to be insufficient to learn so
our CNN. As depicted in Figure 2, the net contains eight many parameters without considerable overfitting. Below,
layers with weights; the first five are convolutional and the we describe the two primary ways in which we combat
remaining three are fully connected. The output of the last overfitting.
fully connected layer is fed to a 1000-way softmax which
produces a distribution over the 1000 class labels. Our net- 5.1. Data augmentation
work maximizes the multinomial logistic regression The easiest and most common method to reduce overfitting
objective, which is equivalent to maximizing the average on image data is to artificially enlarge the dataset using label-
across training cases of the log-probability of the correct preserving transformations (e.g., Refs.4, 5, 30). We employ two
label under the prediction distribution. distinct forms of data augmentation, both of which allow
The kernels of the second, fourth, and fifth convolutional transformed images to be produced from the original images
layers are connected only to those kernel maps in the previous with very little computation, so the transformed images do
layer which reside on the same GPU (see Figure 2). The ker- not need to be stored on disk. In our implementation, the
nels of the third convolutional layer are connected to all ker- transformed images are generated in Python code on the CPU
nel maps in the second layer. The neurons in the while the GPU is training on the previous batch of images. So
fully-connected layers are connected to all neurons in the these data augmentation schemes are, in effect, computa-
previous layer. Response-normalization layers follow the first tionally free.
and second convolutional layers. Max-pooling layers, of the The first form of data augmentation consists of generating
kind described in Section 4.4, follow both response-normal- image translations and horizontal reflections. We do this by
ization layers as well as the fifth convolutional layer. The extracting random 224 × 224 patches (and their horizontal
ReLU non linearity is applied to the output of every convolu- reflections) from the 256 × 256 images and training our net-
tional and fully connected layer. work on these extracted patches.d This increases the size of
The first convolutional layer filters the 224 × 224 × 3 our training set by a factor of 2048, though the resulting train-
input image with 96 kernels of size 11 × 11 × 3 with a stride ing examples are, of course, highly inter dependent. Without
of 4 pixels (this is the distance between the receptive field this scheme, our network suffers from substantial overfit-
centers of neighboring neurons in a kernel map). The sec- ting, which would have forced us to use much smaller net-
ond convolutional layer takes as input the (response-nor- works. At test time, the network makes a prediction by
malized and pooled) output of the first convolutional layer extracting five 224 × 224 patches (the four corner patches and
and filters it with 256 kernels of size 5 × 5 × 48. The third, the center patch) as well as their horizontal reflections (hence
fourth, and fifth convolutional layers are connected to one 10 patches in all), and averaging the predictions made by the
another without any intervening pooling or normalization network’s softmax layer on the ten patches.
layers. The third convolutional layer has 384 kernels of size
3 × 3 × 256 connected to the (normalized, pooled) outputs   This is the reason why the input images in Figure 2 are 224 × 224 × 3 dimensional.
d

JU N E 2 0 1 7 | VO L. 6 0 | N O. 6 | C OM M U N IC AT ION S OF T HE ACM 87
research highlights

The second form of data augmentation consists of altering where i is the iteration index, u is the momentum variable, ε is
the intensities of the RGB channels in training images. the learning rate, and 〈 w 〉D is the average over the ith batch
i i
Specifically, we perform PCA on the set of RGB pixel values Di of the derivative of the objective with respect to w, evaluated
throughout the ImageNet training set. To each training at wi.
image, we add multiples of the found principal components, We initialized the weights in each layer from a zero-mean
with magnitudes proportional to the corresponding eigen Gaussian distribution with standard deviation 0.01. We ini-
values times a random variable drawn from a Gaussian with tialized the neuron biases in the second, fourth, and fifth
mean 0 and standard deviation 0.1. Therefore to each RGB convolutional layers, as well as in the fully connected hidden
image pixel Ixy = [IRxy, IGxy, IBxy]T we add the following quantity: layers, with the constant 1. This initialization accelerates the
early stages of learning by providing the ReLUs with positive
[p1, p2, p3] [α1λ1, α2λ2, α3λ3]T, inputs. We initialized the neuron biases in the remaining lay-
ers with the constant 0.
where pi and λi are ith eigenvector and eigenvalue of the 3 × 3 We used an equal learning rate for all layers, which we
covariance matrix of RGB pixel values, respectively, and αi is adjusted manually throughout training. The heuristic which
the aforementioned random variable. Each αi is drawn only we followed was to divide the learning rate by 10 when the vali-
once for all the pixels of a particular training image until that dation error rate stopped improving with the current learning
image is used for training again, at which point it is re drawn. rate. The learning rate was initialized at 0.01 and reduced
This scheme approximately captures an important property three times prior to termination. We trained the network for
of natural images, namely, that object identity is invariant to roughly 90 cycles through the training set of 1.2 million
changes in the intensity and color of the illumination. This images, which took 5–6 days on two NVIDIA GTX 580 3GB
scheme reduces the top-1 error rate by over 1%. GPUs.

5.2. Dropout 7. RESULTS

Combining the predictions of many different models is a very Our results on ILSVRC-2010 are summarized in Table 1. Our
successful way to reduce test errors,1, 3 but it appears to be too network achieves top-1 and top-5 test set error rates of 37.5%
expensive for big neural networks that already take several and 17.0%, respectively.e The best performance achieved dur-
days to train. There is, however, a very efficient version of ing the ILSVRC-2010 competition was 47.1% and 28.2% with
model combination that only costs about a factor of two dur- an approach that averages the predictions produced from six
ing training. The recently introduced technique, called sparse-coding models trained on different features,2 and
“dropout”,12 consists of setting to zero the output of each hid- since then the best published results are 45.7% and 25.7%
den neuron with probability 0.5. The neurons which are with an approach that averages the predictions of two classi-
“dropped out” in this way do not contribute to the forward fiers trained on Fisher Vectors (FVs) computed from two types
pass and do not participate in back propagation. So every of densely sampled features.29
time an input is presented, the neural network samples a dif- We also entered our model in the ILSVRC-2012 competi-
ferent architecture, but all these architectures share weights. tion and report our results in Table 2. Since the ILSVRC-2012
This technique reduces complex co adaptations of neurons, test set labels are not publicly available, we cannot report test
since a neuron cannot rely on the presence of particular other
neurons. It is, therefore, forced to learn more robust features e
  The error rates without averaging predictions over 10 patches as described
that are useful in conjunction with many different random in Section 5.1 are 39.0% and 18.3%.
subsets of the other neurons. At test time, we use all the neu-
rons but multiply their outputs by 0.5, which is a reasonable Table 1. Comparison of results on ILSVRC-2010 test set.
approximation to taking the geometric mean of the predic-
tive distributions produced by the exponentially-many drop- Model Top-1 (%) Top-5 (%)
out networks. Sparse coding 2
47.1 28.2
We use dropout in the first two fully connected layers of SIFT + FVs29 45.7 25.7
Figure 2. Without dropout, our network exhibits substantial CNN 37.5 17.0
overfitting. Dropout roughly doubles the number of itera- In italics are best results achieved by others.
tions required to converge.

6. DETAILS OF LEARNING Table 2. Comparison of error rates on ILSVRC-2012 validation and test sets.
We trained our models using stochastic gradient descent Model Top-1 (val, %) Top-5 (val, %) Top-5 (test, %)
with a batch size of 128 examples, momentum of 0.9, and
SIFT + FVs 6
– – 26.2
weight decay of 0.0005. We found that this small amount of 1 CNN 40.7 18.2 –
weight decay was important for the model to learn. In other 5 CNNs 38.1 16.4 16.4
words, weight decay here is not merely a regularizer: it 1 CNN* 39.0 16.6 –
reduces the model’s training error. The update rule for weight 7 CNNs* 36.7 15.4 15.3
w was In italics are best results achieved by others. Models with an “*” were “pre-trained” to classify
the entire ImageNet 2011 Fall release (see Section 7 for details).

88 COM MUNICATIO NS O F TH E ACM | J U NE 201 7 | VO L . 60 | NO. 6

error rates for all the models that we tried. In the remainder of color-specific. This kind of specialization occurs during every
this paragraph, we use validation and test error rates inter- run and is independent of any particular random weight ini-
changeably because in our experience they do not differ by tialization (modulo a renumbering of the GPUs).
more than 0.1% (see Table 2). The CNN described in this In the left panel of Figure 4 we qualitatively assess what the
paper achieves a top-5 error rate of 18.2%. Averaging the pre- network has learned by computing its top-5 predictions on
dictions of five similar CNNs gives an error rate of 16.4%. eight test images. Notice that even off-center objects, such as
Training one CNN, with an extra sixth convolutional layer the mite in the top-left, can be recognized by the net. Most of
over the last pooling layer, to classify the entire ImageNet Fall the top-5 labels appear reasonable. For example, only other
2011 release (15M images, 22K categories), and then “fine- types of cat are considered plausible labels for the leopard. In
tuning” it on ILSVRC-2012 gives an error rate of 16.6%. some cases (grille, cherry) there is genuine ambiguity about
Averaging the predictions of two CNNs that were pre-trained the intended focus of the photograph.
on the entire Fall 2011 release with the afore mentioned five Another way to probe the network’s visual knowledge is to
CNNs gives an error rate of 15.3%. The second-best contest consider the feature activations induced by an image at the
entry achieved an error rate of 26.2% with an approach that last, 4096-dimensional hidden layer. If two images produce
averages the predictions of several classifiers trained on FVs feature activation vectors with a small Euclidean separation,
computed from different types of densely sampled features.6 we can say that the higher levels of the neural network con-
Finally, we also report our error rates on the Fall 2009 ver- sider them to be similar. Figure 4 shows five images from the
sion of ImageNet with 10,184 categories and 8.9 million test set and the six images from the training set that are most
images. On this dataset we follow the convention in the litera- similar to each of them according to this measure. Notice that
ture of using half of the images for training and half for test- at the pixel level, the retrieved training images are generally
ing. Since there is no established test set, our split necessarily not close in L2 to the query images in the first column. For
differs from the splits used by previous authors, but this does example, the retrieved dogs and elephants appear in a variety
not affect the results appreciably. Our top-1 and top-5 error
rates on this dataset are 67.4% and 40.9%, attained by the net
Figure 3. Ninety-six convolutional kernels of size 11 × 11 × 3 learned
described above but with an additional, sixth convolutional by the first convolutional layer on the 224 × 224 × 3 input images.
layer over the last pooling layer. The best published results on The top 48 kernels were learned on GPU 1 while the bottom 48
this dataset are 78.1% and 60.9%.23 kernels were learned on GPU 2 (see Section 7.1 for details).

7.1. Qualitative evaluations

Figure 3 shows the convolutional kernels learned by the net-
work’s two data-connected layers. The network has learned a
variety of frequency- and orientation-selective kernels, as well
as various colored blobs. Notice the specialization exhibited
by the two GPUs, a result of the restricted connectivity
described in Section 4.5. The kernels on GPU 1 are largely
color-agnostic, while the kernels on on GPU 2 are largely

Figure 4. (Left) Eight ILSVRC-2010 test images and the five labels considered most probable by our model. The correct label is written
under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5). (Right) Five
ILSVRC-2010 test images in the first column. The remaining columns show the six training images that produce feature vectors in the last
hidden layer with the smallest Euclidean distance from the feature vector for the test image.

JU N E 2 0 1 7 | VO L. 6 0 | N O. 6 | C OM M U N IC AT ION S OF T HE ACM 89
research highlights

of poses. We present the results for many more test images in 8. Fei-Fei, L., Fergus, R., Perona, P. learning of hierarchical
Learning generative visual models representations. In Proceedings of
the supplementary material. from few training examples: the 26th Annual International
Computing similarity by using Euclidean distance An incremental Bayesian approach Conference on Machine Learning
tested on 101 object categories. (2009). ACM, 609–616.
between two 4096-dimensional, real-valued vectors is ineffi- Comput. Vision Image Understanding 22. Linnainmaa, S. Taylor expansion of the
cient, but it could be made efficient by training an auto 106, 1 (2007), 59–70. accumulated rounding error. BIT
9. Fukushima, K. Neocognitron: A Numer. Math. 16, 2 (1976), 146–160.
encoder to compress these vectors to short binary codes. This self-organizing neural network model 23. Mensink, T., Verbeek, J., Perronnin, F.,
should produce a much better image retrieval method than for a mechanism of pattern recognition Csurka, G. Metric learning for large
unaffected by shift in position. Biol. scale image classification:
applying auto encoders to the raw pixels,16 which does not Cybern. 36, 4 (1980), 193–202. Generalizing to new classes at
10. Griffin, G., Holub, A., Perona, P. near-zero cost. In ECCV – European
make use of image labels and hence has a tendency to retrieve Caltech-256 object category dataset. Conference on Computer Vision
images with similar patterns of edges, whether or not they are Technical Report 7694, California (Florence, Italy, Oct. 2012).
Institute of Technology, 2007. 24. Nair, V., Hinton, G.E. Rectified linear
semantically similar. 11. He, K., Zhang, X., Ren, S., Sun, J. Deep units improve restricted Boltzmann
residual learning for image recognition. machines. In Proceedings of the 27th
arXiv preprint arXiv:1512.03385, 2015. International Conference on Machine
8. DISCUSSION 12. Hinton, G., Srivastava, N., Learning (2010).
Our results show that a large, deep CNN is capable of achieving Krizhevsky, A., Sutskever, I., 25. Pinto, N., Cox, D., DiCarlo, J. Why is
Salakhutdinov, R. Improving neural real-world visual object recognition
record-breaking results on a highly challenging dataset using networks by preventing co-adaptation hard? PLoS Comput. Biol. 4, 1 (2008),
purely supervised learning. It is notable that our network’s per- of feature detectors. arXiv preprint e27.
arXiv:1207.0580 (2012). 26. Pinto, N., Doukhan, D., DiCarlo, J., Cox,
formance degrades if a single convolutional layer is removed. 13. Jarrett, K., Kavukcuoglu, K., D. A high-throughput screening
For example, removing any of the middle layers results in a loss Ranzato, M.A., LeCun, Y. What is the approach to discovering good forms of
best multi-stage architecture for biologically inspired visual
of about 2% for the top-1 performance of the network. So the object recognition? In International representation. PLoS Comput. Biol. 5,
depth really is important for achieving our results. Conference on Computer Vision 11 (2009), e1000579.
(2009). IEEE, 2146–2153. 27. Rumelhart, D.E., Hinton, G.E., Williams,
To simplify our experiments, we did not use any unsuper- 14. Krizhevsky, A. Learning multiple layers R.J. Learning internal representations
vised pre-training even though we expect that it will help, espe- of features from tiny images. Master’s by error propagation. Technical report,
thesis, Department of Computer DTIC Document, 1985.
cially if we obtain enough computational power to significantly Science, University of Toronto, 2009. 28. Russell, BC, Torralba, A., Murphy, K.,
15. Krizhevsky, A. Convolutional deep Freeman, W. Labelme: A database and
increase the size of the network without obtaining a corre- belief networks on cifar-10. web-based tool for image annotation.
sponding increase in the amount of labeled data. Thus far, our Unpublished manuscript, 2010. Int. J. Comput Vis. 77, 1 (2008),
16. Krizhevsky, A., Hinton, G. Using very 157–173.
results have improved as we have made our network larger and deep autoencoders for content-based 29. Sánchez, J., Perronnin, F. High-
trained it longer but we still have many orders of magnitude to image retrieval. In ESANN (2011). dimensional signature compression for
17. LeCun, Y., Boser, B., Denker, J., large-scale image classification. In
go in order to match the infero temporal pathway of the human Henderson, D., Howard, R., Hubbard, W., IEEE Conference on Computer Vision
visual system. Ultimately we would like to use very large and Jackel, L., et al. Handwritten digit and Pattern Recognition (CVPR), 2011
recognition with a back-propagation (2011). IEEE, 1665–1672.
deep convolutional nets on video sequences where the tempo- network. In Advances in Neural 30. Simard, P., Steinkraus, D., Platt, J. Best
ral structure provides very helpful information, that is, missing Information Processing Systems (1990). practices for convolutional neural
18. LeCun, Y. Une procedure networks applied to visual document
or far less obvious in static images. d’apprentissage pour reseau a seuil analysis. In Proceedings of the
asymmetrique (a learning scheme for Seventh International Conference on
asymmetric threshold networks). 1985. Document Analysis and Recognition.
9. EPILOGUE 19. LeCun, Y., Huang, F., Bottou, L. Volume 2 (2003), 958–962.
The response of the computer vision community to the suc- Learning methods for generic object 31. Szegedy, C., Liu, W., Jia, Y., Sermanet,
recognition with invariance to pose and P., Reed, S., Anguelov, D., Erhan, D.,
cess of SuperVision was impressive. Over the next year or two, lighting. In Proceedings of the 2004 Vanhoucke, V., Rabinovich, A.
IEEE Computer Society Conference on Going deeper with convolutions.
they switched to using deep neural networks and these are Computer Vision and Pattern In Proceedings of the IEEE Conference
now widely deployed by Google, Facebook, Microsoft, Baidu Recognition, 2004, CVPR 2004. on Computer Vision and Pattern
Volume 2 (2004). IEEE, II–97. Recognition (2015), 1–9.
and many other companies. By 2015, better hardware, more 20. LeCun, Y., Kavukcuoglu, K., Farabet, C. 32. Turaga, S., Murray, J., Jain, V., Roth, F.,
hidden layers, and a host of technical advances reduced the Convolutional networks and Helmstaedter, M., Briggman, K., Denk,
applications in vision. In Proceedings W., Seung, H. Convolutional networks
error rate of deep convolutional neural nets by a further factor of 2010 IEEE International can learn to generate affinity graphs
of three so that they are now quite close to human perfor- Symposium on Circuits and Systems for image segmentation. Neural
(ISCAS) (2010). IEEE, 253–256. Comput. 22, 2 (2010), 511–538.
mance for static images.11, 31 Much of the credit for this revolu- 21. Lee, H., Grosse, R., Ranganath, R., Ng, 33. Werbos, P. Beyond regression: New
tion should go to the pioneers who spent many years A. Convolutional deep belief tools for prediction and analysis in
networks for scalable unsupervised the behavioral sciences, 1974.
developing the technology of CNNs, but the essential missing
ingredient was supplied by FeiFei et al.7 who put a huge effort
into producing a labeled dataset that was finally large enough Alex Krizhevsky and Geoffrey E. Hinton Ilya Sutskever ([email protected]),
to show what neural networks could really do. ({akrizhevsky, geoffhinton}@google.com), OpenAI.
Google Inc.

References visual object classification. Arxiv

1. Bell, R., Koren, Y. Lessons from the preprint arXiv:1102.0183, 2011.
netflix prize challenge. ACM SIGKDD 5. Cireşan, D., Meier, U., Schmidhuber, J.
Explor. Newsl. 9, 2 (2007), 75–79. Multi-column deep neural networks
2. Berg, A., Deng, J., Fei-Fei, L. Large for image classification. Arxiv preprint
scale visual recognition challenge arXiv:1202.2745, 2012.
2010. www.image-net.org/challenges. 6. Deng, J., Berg, A., Satheesh, S., Su, H.,
2010. Khosla, A., Fei-Fei, L. In ILSVRC-2012
3. Breiman, L. Random forests. Mach. (2012).
Learn. 45, 1 (2001), 5–32. 7. Deng, J., Dong, W., Socher, R., Li, L.-J.,
4. Cireşan, D., Meier, U., Masci, J., Li, K., Fei-Fei, L. ImageNet: A
Gambardella, L., Schmidhuber, J. large-scale hierarchical image
High-performance neural networks for database. In CVPR09 (2009). Copyright held by Authors/Owners.

90 COMMUNICATIO NS O F TH E ACM | J U NE 201 7 | VO L . 60 | NO. 6