2023-10-26

ConvNets Match Vision Transformers at Scale

Samuel L Smith1 , Andrew Brock1 , Leonard Berrada1 and Soham De1
1 Google DeepMind

Many researchers believe that ConvNets perform well on small or moderately sized datasets, but are not
competitive with Vision Transformers when given access to datasets on the web-scale. We challenge this
belief by evaluating a performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset of
images often used for training foundation models. We consider pre-training compute budgets between
0.4k and 110k TPU-v4 core compute hours, and train a series of networks of increasing depth and width
from the NFNet model family. We observe a log-log scaling law between held out loss and compute
budget. After fine-tuning on ImageNet, NFNets match the reported performance of Vision Transformers
arXiv:2310.16764v1 [cs.CV] 25 Oct 2023

with comparable compute budgets. Our strongest fine-tuned model achieves a Top-1 accuracy of 90.4%.

Keywords: ConvNets, CNN, Convolution, Transformer, Vision, ViTs, NFNets, JFT, Scaling, Image

Introduction pute budgets beyond 500k TPU-v3 core hours

(Zhai et al., 2022), which significantly exceeds
Convolutional Neural Networks (ConvNets) were the compute used to pre-train ConvNets.
responsible for many of the early successes of
deep learning. Deep ConvNets were first de- We evaluate the scaling properties of the NFNet
ployed commercially over 20 years ago (Le- model family (Brock et al., 2021), a pure con-
Cun et al., 1998), while the success of AlexNet volutional architecture published concurrently
on the ImageNet challenge in 2012 re-ignited with the first ViT papers, and the last ConvNet
widespread interest in the field (Krizhevsky et al., to set a new SOTA on ImageNet. We do not
2017). For almost a decade ConvNets (typically make any changes to the model architecture or
ResNets (He et al., 2016a,b)) dominated com- the training procedure (beyond tuning simple
puter vision benchmarks. However in recent years hyper-parameters such as the learning rate or
they have increasingly been replaced by Vision epoch budget). We consider compute budgets
Transformers (ViTs) (Dosovitskiy et al., 2020). up to a maximum of 110k TPU-v4 core hours,1
and pre-train on the JFT-4B dataset which con-
Simultaneously, the computer vision commu- tains roughly 4 billion labelled images from 30k
nity has shifted from primarily evaluating the classes (Sun et al., 2017). We observe a log-log
performance of randomly initialized networks scaling law between validation loss and the com-
on specific datasets like ImageNet, to evaluat- pute budget used to pre-train the model. After
ing the performance of networks pre-trained on fine-tuning on ImageNet, our networks match the
large general purpose datasets collected from the performance of pre-trained ViTs with comparable
web. This raises an important question; do Vision compute budgets (Alabdulmohsin et al., 2023;
Transformers outperform ConvNet architectures Zhai et al., 2022), as shown in Figure 1.
pre-trained with similar computational budgets?
Although most researchers in the community Pre-trained NFNets obey scaling laws
believe Vision Transformers show better scaling
properties than ConvNets, there is surprisingly We train a range of NFNet models of varying
little evidence to support this claim. Many papers depth and width on JFT-4B. Each model is trained
studying ViTs compare to weak ConvNet base- for a range of epoch budgets between 0.25 and
lines (typically the original ResNet architecture 8, using a cosine decay learning rate schedule.
(He et al., 2016a)). Additionally, the strongest ViT
1 TPU-v4 cores have roughly double the theoretical flops
models have been pre-trained using large com-
of TPU-v3 cores, however both cores have similar memory.

Corresponding author(s): [email protected]

© 2023 Google DeepMind. All rights reserved
 ConvNets Match Vision Transformers at Scale

ImageNet Top-1 Error (%) 14 F1

13
F3
F3+
12 F7
F7+
11 F7+ with RA
ViT-g/14
10 ViT-G/14
SoViT-400m/14
103 104 105
TPU-v4 Core Hours
Figure 1 | ImageNet Top-1 error, after fine-tuning pre-trained NFNet models for 50 epochs. Both axes
are log-scaled. Performance improves consistently as the compute used during pre-training increases.
Our largest model (F7+) achieves comparable performance to that reported for pre-trained ViTs with
a similar compute budget (Alabdulmohsin et al., 2023; Zhai et al., 2022). The performance of this
model improved further when fine-tuned with repeated augmentation (RA) (Hoffer et al., 2019).

The base learning rate is tuned separately for forming language modelling with transformers
each epoch budget on a small logarithmic grid. (Brown et al., 2020; Hoffmann et al., 2022).
In Figure 2, we provide the validation loss at the
The optimal model size and the optimal epoch
end of training on a held out set of 130k images,
budget (which achieve the lowest validation loss)
plotted against the compute budget required to
both increase in size as the compute budget in-
train each model2 . We note that F7 has the same
creases. We found that a reliable rule of thumb is
width as F3, but is double the depth. Similarly
to scale the model size and the number of train-
F3 is double the depth of F1, and F1 is double
ing epochs at the same rate, as previously ob-
the depth of F0. F3+ and F7+ have the same
served for language modelling by Hoffmann et al.
depths as F3 and F7 but larger width. We train us-
(2022). We note that the optimal epoch budget
ing SGD with Momentum and Adaptive Gradient
was greater than 1 for overall compute budgets
Clipping (AGC) at batch size 4096, and we use an
greater than roughly 5k TPU-v4 core hours.
image resolution of 224×224 during training and
256 × 256 at evaluation. For additional details In Figure 3 we plot the observed optimal learn-
describing the NFNet architecture and training ing rate (which minimizes validation loss), for 3
pipeline we refer the reader to the original paper of our models, across a range of epoch budgets.3
(Brock et al., 2021), including the pre-training Note that we tuned the learning rate on a loga-
framework for JFT described in Section 6.2. Note rithmic grid spaced by factors of 2. We find that
that we removed near-duplicates of images in the all models in the NFNet family show a similar
training and validation sets of ImageNet from optimal learning rate 𝛼 ≈ 1.6 for small epoch
JFT-4B before training (Kolesnikov et al., 2020). budgets. However the optimal learning rate falls
as the epoch budget rises, and for large models
Figure 2 shows a clear linear trend, consistent
the optimal learning rate falls more quickly. In
with a log-log scaling law between validation loss
practice one can efficiently tune the learning rate
and pre-training compute. This matches the log-
within 2 trials by assuming that the optimal learn-
log scaling laws previously observed when per-
ing rate falls slowly but monotonically as both the
model size and the epoch budget increases.
2We estimate the compute required to train each model
by eye from the typical steps per second achieved by each 3 The optimal learning rate showed very similar trends
model during training (when not pre-empted). for all models. We select 3 models here for visual clarity.

2
 ConvNets Match Vision Transformers at Scale

3.0
1.6

Optimal Learning Rate

2.9
JFT-4B Validation Loss

2.8
2.7
F0
F1 0.8
2.6 F3
2.5 F3+ F0
F7 F3
2.4 F7+ 0.4
F7+
103 104 105 0.25 0.5 1 2 4 8
TPU-v4 Core Hours Training Epochs
Figure 2 | Held out loss of NFNets on JFT-4B, Figure 3 | The optimal learning rate behaves pre-
plotted against the compute used during training. dictably and is easy to tune. All models show
Both axes are log-scaled, and each curve denotes similar optimal learning rates 𝛼 ∼ 1.6 when the
a different model trained for a range of epoch epoch budget is small. The learning rate falls
budgets. We observe a linear trend, matching the slowly as model size and epoch budget increases.
scaling laws observed for language modelling.
mentation multiplicity 4.4 For comparison, the
Finally, we note that some pre-trained models best reported Top-1 accuracy of an NFNet on Im-
in Figure 2 perform less well than expected. For ageNet without extra data is 86.8% (Fort et al.,
example, the curve for NFNet-F7+ models at dif- 2021), achieved by an NFNet-F5 with repeated
ferent pre-training budgets is not smooth. We be- augmentation. This demonstrates that NFNets
lieve this arises because our data loading pipeline benefit substantially from large scale pre-training.
did not guarantee that each training example
Despite the substantial differences between
would be sampled once per epoch if the training
the two model architectures, the performance of
run was pre-empted/restarted, potentially caus-
pre-trained NFNets at scale is remarkably similar
ing some training examples to be under-sampled
to the performance of pre-trained Vision Trans-
if a training run was restarted multiple times.
formers. For example, Zhai et al. (2022) achieve
90.2% Top-1 on ImageNet with a ViT-g/14, af-
Fine-tuned NFNets are competitive ter pre-training on JFT-3B for 210k TPU-v3 core
with Vision Transformers on ImageNet hours, and 90.45% with a ViT-G/14 after pre-
training on JFT-3B for over 500k TPU-v3 core
In Figure 1, we fine-tune our pre-trained NFNets hours. In a recent work, Alabdulmohsin et al.
on ImageNet, and plot the Top-1 error against (2023) optimize the ViT architecture and achieve
the compute used during pre-training. We fine- 90.3% Top-1 with a SoViT-400m/14 after pre-
tune each model for 50 epochs using sharpness training on JFT-3B for 230k TPU-v3 hours.
aware minimization (SAM) (Foret et al., 2020) We evaluated the pre-training speed for these
with stochastic depth and dropout. We train at models on TPU-v4 (using the original authors’
resolution 384 × 384 and evaluate at 480 × 480. codebase), and estimate that ViT-g/14 would take
The ImageNet Top-1 accuracy consistently im- 120k TPU-v4 core hours to pre-train, while ViT-
proves as the compute budget increases. Our most G/14 would take 280k TPU-v4 core hours and
expensive pre-trained model, an NFNet-F7+ pre- SoViT-400m/14 would take 130k TPU-v4 core
trained for 8 epochs, achieves an ImageNet Top-1 hours. We use these estimates to compare the pre-
accuracy of 90.3% while requiring roughly 110k training efficiency of ViTs and NFNets in Figure 1.
TPU-v4 core hours to pre-train and 1.6k TPU-v4 We note however that NFNets were optimized for
core hours to fine-tune. Furthermore, we achieve TPU-v4, and perform less well when evaluated
90.4% Top-1 accuracy if we additionally intro- 4When using repeated augmentation, we reduce the num-
duce repeated augmentation during fine-tuning ber of passes through the data such that the total computa-
(Fort et al., 2021; Hoffer et al., 2019) with aug- tional cost of fine-tuning is constant.

3
 ConvNets Match Vision Transformers at Scale

on other devices. For example, we estimate that R. Bavishi, E. Elsen, C. Hawthorne, M. Nye,
NFNet-F7+ would require 250 TPU-v3 core hours A. Odena, A. Somani, and S. Taşırlar. Intro-
to pre-train for 8 epochs in our codebase. ducing our multimodal models, 2023. URL
https://www.adept.ai/blog/fuyu-8b.
Finally, we note that the pre-trained check-
points achieving the lowest validation loss on A. Brock, S. De, S. L. Smith, and K. Simonyan.
JFT-4B did not always achieve the highest Top-1 High-performance large-scale image recogni-
accuracy on ImageNet after fine-tuning. In par- tion without normalization. In International
ticular, we found that, under a fixed pre-training Conference on Machine Learning, pages 1059–
compute budget, the fine-tuning regime consis- 1071. PMLR, 2021.
tently favoured slightly larger models and slightly
smaller epoch budgets. Intuitively, larger models T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D.
have more capacity and are therefore better able Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
to adapt to the new task. In some cases, slightly G. Sastry, A. Askell, et al. Language models
larger learning rates (during pre-training) also are few-shot learners. Advances in neural in-
achieved better performance after fine-tuning. formation processing systems, 33:1877–1901,
2020.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis-

Discussion senborn, X. Zhai, T. Unterthiner, M. Dehghani,
M. Minderer, G. Heigold, S. Gelly, et al. An
Our work reinforces the bitter lesson. The most
image is worth 16x16 words: Transformers
important factors determining the performance
for image recognition at scale. arXiv preprint
of a sensibly designed model are the compute
arXiv:2010.11929, 2020.
and data available for training5 (Tolstikhin et al.,
2021). Although the success of ViTs in computer P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur.
vision is extremely impressive, in our view there Sharpness-aware minimization for efficiently
is no strong evidence to suggest that pre-trained improving generalization. arXiv preprint
ViTs outperform pre-trained ConvNets when eval- arXiv:2010.01412, 2020.
uated fairly. We note however that ViTs may have
practical advantages in specific contexts, such S. Fort, A. Brock, R. Pascanu, S. De, and
as the ability to use similar model components S. L. Smith. Drawing multiple augmenta-
across multiple modalities (Bavishi et al., 2023). tion samples per image during training effi-
ciently decreases test error. arXiv preprint
arXiv:2105.13343, 2021.
Acknowledgements K. He, X. Zhang, S. Ren, and J. Sun. Deep residual
learning for image recognition. In Proceedings
We thank Lucas Beyer and Olivier Henaff for feed-
of the IEEE conference on computer vision and
back on an earlier draft of this note. We also
pattern recognition, pages 770–778, 2016a.
thank Lucas Beyer for providing training speed
estimates for ViT models on TPU-v4 devices. K. He, X. Zhang, S. Ren, and J. Sun. Identity map-
pings in deep residual networks. In European
conference on computer vision, pages 630–645.
References Springer, 2016b.

I. Alabdulmohsin, X. Zhai, A. Kolesnikov, and E. Hoffer, T. Ben-Nun, I. Hubara, N. Giladi, T. Hoe-

L. Beyer. Getting vit in shape: Scaling laws for fler, and D. Soudry. Augment your batch: bet-
compute-optimal model design. arXiv preprint ter training with larger batches. arXiv preprint
arXiv:2305.13035, 2023. arXiv:1901.09335, 2019.
5 By sensibly designed, we mean models that are suffi- J. Hoffmann, S. Borgeaud, A. Mensch,
ciently expressive and have stable gradient propagation. E. Buchatskaya, T. Cai, E. Rutherford, D. d. L.

4
 ConvNets Match Vision Transformers at Scale

Casas, L. A. Hendricks, J. Welbl, A. Clark, et al.

Training compute-optimal large language
models. arXiv preprint arXiv:2203.15556,
2022.

A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver,

J. Yung, S. Gelly, and N. Houlsby. Big trans-
fer (bit): General visual representation learn-
ing. In Computer Vision–ECCV 2020: 16th Eu-
ropean Conference, Glasgow, UK, August 23–28,
2020, Proceedings, Part V 16, pages 491–507.
Springer, 2020.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Im-

agenet classification with deep convolutional
neural networks. Communications of the ACM,
60(6):84–90, 2017.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.

Gradient-based learning applied to document
recognition. Proceedings of the IEEE, 86(11):
2278–2324, 1998.

C. Sun, A. Shrivastava, S. Singh, and A. Gupta.

Revisiting unreasonable effectiveness of data in
deep learning era. In Proceedings of the IEEE in-
ternational conference on computer vision, pages
843–852, 2017.

I. O. Tolstikhin, N. Houlsby, A. Kolesnikov,

L. Beyer, X. Zhai, T. Unterthiner, J. Yung,
A. Steiner, D. Keysers, J. Uszkoreit, et al. Mlp-
mixer: An all-mlp architecture for vision. Ad-
vances in neural information processing systems,
34:24261–24272, 2021.

X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer.

Scaling vision transformers. In Proceedings of
the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 12104–12113,
2022.