Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Ze Liu†* Yutong Lin†* Yue Cao* Han Hu*‡ Yixuan Wei†

Zheng Zhang Stephen Lin Baining Guo
Microsoft Research Asia
{v-zeliu1,v-yutlin,yuecao,hanhu,v-yixwe,zhez,stevelin,bainguo}@microsoft.com
arXiv:2103.14030v1 [cs.CV] 25 Mar 2021

Abstract

This paper presents a new vision Transformer, called

Swin Transformer, that capably serves as a general-purpose
backbone for computer vision. Challenges in adapting
Transformer from language to vision arise from differences
between the two domains, such as large variations in the
scale of visual entities and the high resolution of pixels in
images compared to words in text. To address these differ-
ences, we propose a hierarchical Transformer whose rep-
resentation is computed with shifted windows. The shifted
windowing scheme brings greater efficiency by limiting self-
attention computation to non-overlapping local windows Figure 1. (a) The proposed Swin Transformer builds hierarchical
while also allowing for cross-window connection. This hi- feature maps by merging image patches (shown in gray) in deeper
erarchical architecture has the flexibility to model at var- layers and has linear computation complexity to input image size
due to computation of self-attention only within each local win-
ious scales and has linear computational complexity with
dow (shown in red). It can thus serve as a general-purpose back-
respect to image size. These qualities of Swin Transformer
bone for both image classification and dense recognition tasks.
make it compatible with a broad range of vision tasks, (b) In contrast, previous vision Transformers [19] produce fea-
including image classification (86.4 top-1 accuracy on ture maps of a single low resolution and have quadratic compu-
ImageNet-1K) and dense prediction tasks such as object de- tation complexity to input image size due to computation of self-
tection (58.7 box AP and 51.1 mask AP on COCO test-dev) attention globally.
and semantic segmentation (53.5 mIoU on ADE20K val).
Its performance surpasses the previous state-of-the-art by a CNNs serving as backbone networks for a variety of vision
large margin of +2.7 box AP and +2.6 mask AP on COCO, tasks, these architectural advances have led to performance
and +3.2 mIoU on ADE20K, demonstrating the potential of improvements that have broadly lifted the entire field.
Transformer-based models as vision backbones. The code On the other hand, the evolution of network architectures
and models will be made publicly available at https:// in natural language processing (NLP) has taken a different
github.com/microsoft/Swin-Transformer. path, where the prevalent architecture today is instead the
Transformer [61]. Designed for sequence modeling and
transduction tasks, the Transformer is notable for its use
1. Introduction of attention to model long-range dependencies in the data.
Its tremendous success in the language domain has led re-
Modeling in computer vision has long been dominated
searchers to investigate its adaptation to computer vision,
by convolutional neural networks (CNNs). Beginning with
where it has recently demonstrated promising results on cer-
AlexNet [38] and its revolutionary performance on the
tain tasks, specifically image classification [19] and joint
ImageNet image classification challenge, CNN architec-
vision-language modeling [46].
tures have evolved to become increasingly powerful through
greater scale [29, 73], more extensive connections [33], and In this paper, we seek to expand the applicability of
more sophisticated forms of convolution [67, 17, 81]. With Transformer such that it can serve as a general-purpose
backbone for computer vision, as it does for NLP and
* Equal contribution. † Interns at MSRA. ‡ Contact person. as CNNs do in vision. We observe that significant chal-
lenges in transferring its high performance in the language
domain to the visual domain can be explained by differ-
ences between the two modalities. One of these differ-
ences involves scale. Unlike the word tokens that serve
as the basic elements of processing in language Trans-
formers, visual elements can vary substantially in scale, a
problem that receives attention in tasks such as object de-
tection [41, 52, 53]. In existing Transformer-based mod-
els [61, 19], tokens are all of a fixed scale, a property un- Figure 2. An illustration of the shifted window approach for com-
suitable for these vision applications. Another difference puting self-attention in the proposed Swin Transformer architec-
ture. In layer l (left), a regular window partitioning scheme is
is the much higher resolution of pixels in images com-
adopted, and self-attention is computed within each window. In
pared to words in passages of text. There exist many vi- the next layer l + 1 (right), the window partitioning is shifted, re-
sion tasks such as semantic segmentation that require dense sulting in new windows. The self-attention computation in the new
prediction at the pixel level, and this would be intractable windows crosses the boundaries of the previous windows in layer
for Transformer on high-resolution images, as the compu- l, providing connections among them.
tational complexity of its self-attention is quadratic to im-
age size. To overcome these issues, we propose a general- sliding window method, yet is similar in modeling power
purpose Transformer backbone, called Swin Transformer, (see Tables 5 and 6).
which constructs hierarchical feature maps and has linear The proposed Swin Transformer achieves strong perfor-
computational complexity to image size. As illustrated in mance on the recognition tasks of image classification, ob-
Figure 1(a), Swin Transformer constructs a hierarchical rep- ject detection and semantic segmentation. It outperforms
resentation by starting from small-sized patches (outlined in the ViT / DeiT [19, 60] and ResNe(X)t models [29, 67] sig-
gray) and gradually merging neighboring patches in deeper nificantly with similar latency on the three tasks. Its 58.7
Transformer layers. With these hierarchical feature maps, box AP and 51.1 mask AP on the COCO test-dev set sur-
the Swin Transformer model can conveniently leverage ad- pass the previous state-of-the-art results by +2.7 box AP
vanced techniques for dense prediction such as feature pyra- (Copy-paste [25] without external data) and +2.6 mask AP
mid networks (FPN) [41] or U-Net [50]. The linear compu- (DetectoRS [45]). On ADE20K semantic segmentation, it
tational complexity is achieved by computing self-attention obtains 53.5 mIoU on the val set, an improvement of +3.2
locally within non-overlapping windows that partition an mIoU over the previous state-of-the-art (SETR [78]). It also
image (outlined in red). The number of patches in each achieves a top-1 accuracy of 86.4% on ImageNet-1K image
window is fixed, and thus the complexity becomes linear classification.
to image size. These merits make Swin Transformer suit- It is our belief that a unified architecture across com-
able as a general-purpose backbone for various vision tasks, puter vision and natural language processing could benefit
in contrast to previous Transformer based architectures [19] both fields, since it would facilitate joint modeling of vi-
which produce feature maps of a single resolution and have sual and textual signals and the modeling knowledge from
quadratic complexity. both domains can be more deeply shared. We hope that
A key design element of Swin Transformer is its shift Swin Transformer’s strong performance on various vision
of the window partition between consecutive self-attention problems can drive this belief deeper in the community and
layers, as illustrated in Figure 2. The shifted windows encourage unified modeling of vision and language signals.
bridge the windows of the preceding layer, providing con-
nections among them that significantly enhance modeling 2. Related Work
power (see Table 4). This strategy is also efficient in re-
gards to real-world latency: all query patches within a win- CNN and variants CNNs serve as the standard network
dow share the same key set1 , which facilitates memory ac- model throughout computer vision. While the CNN has ex-
cess in hardware. In contrast, earlier sliding window based isted for several decades [39], it was not until the introduc-
self-attention approaches [32, 49] suffer from low latency tion of AlexNet [38] that the CNN took off and became
on general hardware due to different key sets for different mainstream. Since then, deeper and more effective con-
query pixels2 . Our experiments show that the proposed volutional neural architectures have been proposed to fur-
shifted window approach has much lower latency than the ther propel the deep learning wave in computer vision, e.g.,
VGG [51], GoogleNet [56], ResNet [29], DenseNet [33],
1 The query and key are projection vectors in a self-attention layer.
2 While
HRNet [62], and EfficientNet [57]. In addition to these
there are efficient methods to implement a sliding-window
based convolution layer on general hardware, thanks to its shared kernel
architectural advances, there has also been much work on
weights across a feature map, it is difficult for a sliding-window based improving individual convolution layers, such as depth-
self-attention layer to have efficient memory access in practice. wise convolution [67] and deformable convolution [17, 81].

2
While the CNN and its variants are still the primary back- vision tasks of object detection and semantic segmenta-
bone architectures for computer vision applications, we tion by direct upsampling or deconvolution but with rela-
highlight the strong potential of Transformer-like architec- tively lower performance [2, 78]. Concurrent to our work
tures for unified modeling between vision and language. are some that modify the ViT architecture [69, 14, 27]
Our work achieves strong performance on several basic vi- for better image classification. Empirically, we find our
sual recognition tasks, and we hope it will contribute to a Swin Transformer architecture to achieve the best speed-
modeling shift. accuracy trade-off among these methods on image classi-
fication, even though our work focuses on general-purpose
performance rather than specifically on classification. An-
Self-attention based backbone architectures Also in-
other concurrent work [63] explores a similar line of think-
spired by the success of self-attention layers and Trans-
ing to build multi-resolution feature maps on Transform-
former architectures in the NLP field, some works employ
ers. Its complexity is still quadratic to image size, while
self-attention layers to replace some or all of the spatial con-
ours is linear and also operates locally which has proven
volution layers in the popular ResNet [32, 49, 77]. In these
beneficial in modeling the high correlation in visual sig-
works, the self-attention is computed within a local window
nals [35, 24, 40]. Our approach is both efficient and ef-
of each pixel to expedite optimization [32], and they achieve
fective, achieving state-of-the-art accuracy on both COCO
slightly better accuracy/FLOPs trade-offs than the counter-
object detection and ADE20K semantic segmentation.
part ResNet architecture. However, their costly memory
access causes their actual latency to be significantly larger
than that of the convolutional networks [32]. Instead of us- 3. Method
ing sliding windows, we propose to shift windows between 3.1. Overall Architecture
consecutive layers, which allows for a more efficient imple-
mentation in general hardware. An overview of the Swin Transformer architecture is pre-
sented in Figure 3, which illustrates the tiny version (Swin-
T). It first splits an input RGB image into non-overlapping
Self-attention/Transformers to complement CNNs An- patches by a patch splitting module, like ViT. Each patch is
other line of work is to augment a standard CNN architec- treated as a “token” and its feature is set as a concatenation
ture with self-attention layers or Transformers. The self- of the raw pixel RGB values. In our implementation, we use
attention layers can complement backbones [64, 6, 68, 22, a patch size of 4 × 4 and thus the feature dimension of each
71, 54] or head networks [31, 26] by providing the capa- patch is 4 × 4 × 3 = 48. A linear embedding layer is ap-
bility to encode distant dependencies or heterogeneous in- plied on this raw-valued feature to project it to an arbitrary
teractions. More recently, the encoder-decoder design in dimension (denoted as C).
Transformer has been applied for the object detection and
Several Transformer blocks with modified self-attention
instance segmentation tasks [7, 12, 82, 55]. Our work ex-
computation (Swin Transformer blocks) are applied on these
plores the adaptation of Transformers for basic visual fea-
patch tokens. The Transformer blocks maintain the number
ture extraction and is complementary to these works.
of tokens ( H W
4 × 4 ), and together with the linear embedding
are referred to as “Stage 1”.
Transformer based vision backbones Most related to To produce a hierarchical representation, the number of
our work is the Vision Transformer (ViT) [19] and its tokens is reduced by patch merging layers as the network
follow-ups [60, 69, 14, 27, 63]. The pioneering work of gets deeper. The first patch merging layer concatenates the
ViT directly applies a Transformer architecture on non- features of each group of 2 × 2 neighboring patches, and
overlapping medium-sized image patches for image clas- applies a linear layer on the 4C-dimensional concatenated
sification. It achieves an impressive speed-accuracy trade- features. This reduces the number of tokens by a multiple
off on image classification compared to convolutional net- of 2 × 2 = 4 (2× downsampling of resolution), and the out-
works. While ViT requires large-scale training datasets put dimension is set to 2C. Swin Transformer blocks are
(i.e., JFT-300M) to perform well, DeiT [60] introduces sev- applied afterwards for feature transformation, with the res-
eral training strategies that allow ViT to also be effective olution kept at H W
8 × 8 . This first block of patch merging
using the smaller ImageNet-1K dataset. The results of ViT and feature transformation is denoted as “Stage 2”. The pro-
on image classification are encouraging, but its architec- cedure is repeated twice, as “Stage 3” and “Stage 4”, with
H
ture is unsuitable for use as a general-purpose backbone output resolutions of 16 × W H W
16 and 32 × 32 , respectively.
network on dense vision tasks or when the input image These stages jointly produce a hierarchical representation,
resolution is high, due to its low-resolution feature maps with the same feature map resolutions as those of typical
and the quadratic increase in complexity with image size. convolutional networks, e.g., VGG [51] and ResNet [29].
There are a few works applying ViT models to the dense As a result, the proposed architecture can conveniently re-

3
 MLP MLP

Stage 1 Stage 2 Stage 3 Stage 4

LN LN
Linear Embedding
Patch Partition

Patch Merging

Patch Merging

Patch Merging
Swin Swin Swin Swin
Images Transformer Transformer Transformer Transformer W-MSA SW-MSA
Block Block Block Block
LN LN

2 2 6 2

(a) Architecture (b) Two Successive Swin Transformer Blocks

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with
Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

place the backbone networks in existing methods for vari- where the former is quadratic to patch number hw, and the
ous vision tasks. latter is linear when M is fixed (set to 7 by default). Global
self-attention computation is generally unaffordable for a
large hw, while the window based self-attention is scalable.
Swin Transformer block Swin Transformer is built by
replacing the standard multi-head self attention (MSA)
module in a Transformer block by a module based on Shifted window partitioning in successive blocks The
shifted windows (described in Section 3.2), with other lay- window-based self-attention module lacks connections
ers kept the same. As illustrated in Figure 3(b), a Swin across windows, which limits its modeling power. To intro-
Transformer block consists of a shifted window based MSA duce cross-window connections while maintaining the effi-
module, followed by a 2-layer MLP with GELU non- cient computation of non-overlapping windows, we propose
linearity in between. A LayerNorm (LN) layer is applied a shifted window partitioning approach which alternates be-
before each MSA module and each MLP, and a residual tween two partitioning configurations in consecutive Swin
connection is applied after each module. Transformer blocks.
As illustrated in Figure 2, the first module uses a regular
3.2. Shifted Window based Self-Attention window partitioning strategy which starts from the top-left
pixel, and the 8 × 8 feature map is evenly partitioned into
The standard Transformer architecture [61] and its adap-
2 × 2 windows of size 4 × 4 (M = 4). Then, the next mod-
tation for image classification [19] both conduct global self-
ule adopts a windowing configuration that is shifted from
attention, where the relationships between a token and all
that of the preceding layer, by displacing the windows by
other tokens are computed. The global computation leads to
(b M M
2 c, b 2 c) pixels from the regularly partitioned windows.
quadratic complexity with respect to the number of tokens,
making it unsuitable for many vision problems requiring an With the shifted window partitioning approach, consec-
immense set of tokens for dense prediction or to represent a utive Swin Transformer blocks are computed as
high-resolution image.
ẑl = W-MSA LN zl−1 + zl−1 ,

zl = MLP LN ẑl + ẑl ,

Self-attention in non-overlapped windows For efficient
ẑl+1 = SW-MSA LN zl + zl ,



modeling, we propose to compute self-attention within lo-
zl+1 = MLP LN ẑl+1 + ẑl+1 ,



cal windows. The windows are arranged to evenly partition (3)
the image in a non-overlapping manner. Supposing each
window contains M × M patches, the computational com- where ẑl and zl denote the output features of the (S)W-
plexity of a global MSA module and a window based one MSA module and the MLP module for block l, respectively;
on an image of h × w patches are3 : W-MSA and SW-MSA denote window based multi-head
self-attention using regular and shifted window partitioning
Ω(MSA) = 4hwC 2 + 2(hw)2 C, (1) configurations, respectively.
2 2 The shifted window partitioning approach introduces
Ω(W-MSA) = 4hwC + 2M hwC, (2)
connections between neighboring non-overlapping win-
3 We omit SoftMax computation in determining complexity. dows in the previous layer and is found to be effective in im-

4
 A C A C The learnt relative position bias in pre-training can be
masked
MSA also used to initialize a model for fine-tuning with a differ-
B B B B

...
ent window size through bi-cubic interpolation [19, 60].
masked
window partition C A C A
MSA 3.3. Architecture Variants
cyclic shift reverse cyclic shift
Figure 4. Illustration of an efficient batch computation approach We build our base model, called Swin-B, to have of
for self-attention in shifted window partitioning. model size and computation complexity similar to ViT-
B/DeiT-B. We also introduce Swin-T, Swin-S and Swin-L,
which are versions of about 0.25×, 0.5× and 2× the model
age classification, object detection, and semantic segmenta- size and computational complexity, respectively. Note that
tion, as shown in Table 4. the complexity of Swin-T and Swin-S are similar to those
of ResNet-50 (DeiT-S) and ResNet-101, respectively. The
Efficient batch computation for shifted configuration window size is set to M = 7 by default. The query dimen-
An issue with shifted window partitioning is that it will re- sion of each head is d = 32, and the expansion layer of
sult in more windows, from d M h
e × dM w h
e to (d M e + 1) × each MLP is α = 4, for all experiments. The architecture
w
(d M e+1) in the shifted configuration, and some of the win- hyper-parameters of these model variants are:
dows will be smaller than M × M 4 . A naive solution is to • Swin-T: C = 96, layer numbers = {2, 2, 6, 2}
pad the smaller windows to a size of M × M and mask
out the padded values when computing attention. When • Swin-S: C = 96, layer numbers ={2, 2, 18, 2}
the number of windows in regular partitioning is small, e.g. • Swin-B: C = 128, layer numbers ={2, 2, 18, 2}
2 × 2, the increased computation with this naive solution is
considerable (2 × 2 → 3 × 3, which is 2.25 times greater). • Swin-L: C = 192, layer numbers ={2, 2, 18, 2}
Here, we propose a more efficient batch computation ap- where C is the channel number of the hidden layers in the
proach by cyclic-shifting toward the top-left direction, as il- first stage. The model size, theoretical computational com-
lustrated in Figure 4. After this shift, a batched window may plexity (FLOPs), and throughput of the model variants for
be composed of several sub-windows that are not adjacent ImageNet image classification are listed in Table 1.
in the feature map, so a masking mechanism is employed to
limit self-attention computation to within each sub-window. 4. Experiments
With the cyclic-shift, the number of batched windows re-
mains the same as that of regular window partitioning, and We conduct experiments on ImageNet-1K image classi-
thus is also efficient. The low latency of this approach is fication [18], COCO object detection [42], and ADE20K
shown in Table 5. semantic segmentation [80]. In the following, we first com-
pare the proposed Swin Transformer architecture with the
previous state-of-the-arts on the three tasks. Then, we ab-
Relative position bias In computing self-attention, we late the important design elements of Swin Transformer.
follow [48, 1, 31, 32] by including a relative position bias
2 2
B ∈ RM ×M to each head in computing similarity: 4.1. Image Classification on ImageNet-1K
√ Settings For image classification, we benchmark the pro-
Attention(Q, K, V ) = SoftMax(QK T / d + B)V, (4) posed Swin Transformer on ImageNet-1K [18], which con-
2
tains 1.28M training images and 50K validation images
where Q, K, V ∈ RM ×d are the query, key and value ma- from 1,000 classes. The top-1 accuracy on a single crop
trices; d is the query/key dimension, and M 2 is the number is reported. We consider two training settings:
of patches in a window. Since the relative position along
each axis lies in the range [−M + 1, M − 1], we parameter- • Regular ImageNet-1K training. This setting mostly
ize a smaller-sized bias matrix B̂ ∈ R(2M −1)×(2M −1) , and follows [60]. We employ an AdamW [36] optimizer
values in B are taken from B̂. for 300 epochs using a cosine decay learning rate
scheduler and 20 epochs of linear warm-up. A batch
We observe significant improvements over counterparts
size of 1024, an initial learning rate of 0.001, and a
without this bias term or that use absolute position embed-
weight decay of 0.05 are used. We include most of
ding, as shown in Table 4. Further adding absolute posi-
the augmentation and regularization strategies of [60]
tion embedding to the input as in [19] drops performance
in training, except for repeated augmentation [30] and
slightly, thus it is not adopted in our implementation.
EMA [44], which do not enhance performance. Note
4 To make the window size (M, M ) divisible by the feature map size of that this is contrary to [60] where repeated augmenta-
(h, w), bottom-right padding is employed on the feature map if needed. tion is crucial to stabilize the training of ViT.

5
 • Pre-training on ImageNet-22K and fine-tuning on (a) Regular ImageNet-1K trained models
ImageNet-1K. We also pre-train on the larger image throughput ImageNet
method #param. FLOPs
ImageNet-22K dataset, which contains 14.2 million size (image / s) top-1 acc.
images and 22K classes. We employ an AdamW opti- RegNetY-4G [47] 2242 21M 4.0G 1156.7 80.0
RegNetY-8G [47] 2242 39M 8.0G 591.6 81.7
mizer for 60 epochs using a linear decay learning rate
RegNetY-16G [47] 2242 84M 16.0G 334.7 82.9
scheduler with a 5-epoch linear warm-up. A batch size
EffNet-B3 [57] 3002 12M 1.8G 732.1 81.6
of 4096, an initial learning rate of 0.001, and a weight EffNet-B4 [57] 3802 19M 4.2G 349.4 82.9
decay of 0.01 are used. In ImageNet-1K fine-tuning, EffNet-B5 [57] 4562 30M 9.9G 169.1 83.6
we train the models for 30 epochs with a batch size of EffNet-B6 [57] 5282 43M 19.0G 96.9 84.0
1024, a constant learning rate of 10−5 , and a weight EffNet-B7 [57] 6002 66M 37.0G 55.1 84.3
decay of 10−8 . ViT-B/16 [19] 3842 86M 55.4G 85.9 77.9
ViT-L/16 [19] 3842 307M 190.7G 27.3 76.5
Results with regular ImageNet-1K training Table 1(a) DeiT-S [60] 2242 22M 4.6G 940.4 79.8
presents comparisons to other backbones, including both DeiT-B [60] 2242 86M 17.5G 292.3 81.8
DeiT-B [60] 3842 86M 55.4G 85.9 83.1
Transformer-based and ConvNet-based, using regular
Swin-T 2242 29M 4.5G 755.2 81.3
ImageNet-1K training.
Swin-S 2242 50M 8.7G 436.9 83.0
Compared to the previous state-of-the-art Transformer- Swin-B 2242 88M 15.4G 278.1 83.3
based architecture, i.e. DeiT [60], Swin Transformers no- Swin-B 3842 88M 47.0G 84.7 84.2
ticeably surpass the counterpart DeiT architectures with (b) ImageNet-22K pre-trained models
similar complexities: +1.5% for Swin-T (81.3%) over image throughput ImageNet
DeiT-S (79.8%) using 2242 input, and +1.5%/1.1% for method #param. FLOPs
size (image / s) top-1 acc.
Swin-B (83.3%/84.2%) over DeiT-B (81.8%/83.1%) using R-101x3 [37] 3842 388M 204.6G - 84.4
2242 /3842 input, respectively. R-152x4 [37] 4802 937M 840.5G - 85.4
Compared with the state-of-the-art ConvNets, i.e. Reg- ViT-B/16 [19] 3842 86M 55.4G 85.9 84.0
Net [47] and EfficientNet [57], the Swin Transformer ViT-L/16 [19] 3842 307M 190.7G 27.3 85.2
achieves a slightly better speed-accuracy trade-off. Not- Swin-B 2242 88M 15.4G 278.1 85.2
ing that while RegNet [47] and EfficientNet [57] are ob- Swin-B 3842 88M 47.0G 84.7 86.0
Swin-L 3842 197M 103.9G 42.1 86.4
tained via a thorough architecture search, the proposed
Table 1. Comparison of different backbones on ImageNet-1K clas-
Swin Transformer is adapted from the standard Transformer
sification. Throughput is measured using the GitHub repository
and has strong potential for further improvement. of [65] and a V100 GPU, following [60].

Results with ImageNet-22K pre-training We also pre-

train the larger-capacity Swin-B and Swin-L on ImageNet- RepPoints v2 [11], and Sparse RCNN [55] in mmdetec-
22K. Results fine-tuned on ImageNet-1K image classifica- tion [9]. For these four frameworks, we utilize the same
tion are shown in Table 1(b). For Swin-B, the ImageNet- settings: multi-scale training [7, 55] (resizing the input such
22K pre-training brings 1.8%∼1.9% gains over training that the shorter side is between 480 and 800 while the longer
on ImageNet-1K from scratch. Compared with the previ- side is at most 1333), AdamW [43] optimizer (initial learn-
ous best results for ImageNet-22K pre-training, our mod- ing rate of 0.0001, weight decay of 0.05, and batch size of
els achieve significantly better speed-accuracy trade-offs: 16), and 3x schedule (36 epochs). For system-level compar-
Swin-B obtains 86.0% top-1 accuracy, which is 2.0% higher ison, we adopt an improved HTC [8] (denoted as HTC++)
than that of ViT with similar inference throughput (84.7 with instaboost [21], stronger multi-scale training [6], 6x
vs. 85.9 images/sec) and slightly lower FLOPs (47.0G vs. schedule (72 epochs), soft-NMS [4], and ImageNet-22K
55.4G). The larger Swin-L model achieves 86.4% top-1 ac- pre-trained model as initialization.
curacy, slightly better than that of the Swin-B model. We compare our Swin Transformer to standard Con-
vNets, i.e. ResNe(X)t, and previous Transformer networks,
4.2. Object Detection on COCO
e.g. DeiT. The comparisons are conducted by changing only
Settings Object detection and instance segmentation ex- the backbones with other settings unchanged. Note that
periments are conducted on COCO 2017, which contains while Swin Transformer and ResNe(X)t are directly appli-
118K training, 5K validation and 20K test-dev images. An cable to all the above frameworks because of their hierar-
ablation study is performed using the validation set, and a chical feature maps, DeiT only produces a single resolu-
system-level comparison is reported on test-dev. For the tion of feature maps and cannot be directly applied. For fair
ablation study, we consider four typical object detection comparison, we follow [78] to construct hierarchical feature
frameworks: Cascade Mask R-CNN [28, 5], ATSS [76], maps for DeiT using deconvolution layers.

6
 (a) Various frameworks ADE20K val test
#param. FLOPs FPS
Method Backbone APbox APbox box
50 AP75 #param. FLOPs FPS Method Backbone mIoU score
Cascade R-50 46.3 64.3 50.5 82M 739G 18.0 DANet [22] ResNet-101 45.2 - 69M 1119G 15.2
Mask R-CNN Swin-T 50.5 69.3 54.9 86M 745G 15.3 DLab.v3+ [10] ResNet-101 44.1 - 63M 1021G 16.0
R-50 43.5 61.9 47.0 32M 205G 28.3 ACNet [23] ResNet-101 45.9 38.5 -
ATSS
Swin-T 47.2 66.5 51.3 36M 215G 22.3 DNL [68] ResNet-101 46.0 56.2 69M 1249G 14.8
R-50 46.5 64.6 50.3 42M 274G 13.6 OCRNet [70] ResNet-101 45.3 56.0 56M 923G 19.3
RepPointsV2 UperNet [66] ResNet-101 44.9 - 86M 1029G 20.1
Swin-T 50.0 68.5 54.2 45M 283G 12.0
Sparse R-50 44.5 63.4 48.2 106M 166G 21.0 OCRNet [70] HRNet-w48 45.7 - 71M 664G 12.5
R-CNN Swin-T 47.9 67.3 52.3 110M 172G 18.4 DLab.v3+ [10] ResNeSt-101 46.9 55.1 66M 1051G 11.9
(b) Various backbones w. Cascade Mask R-CNN DLab.v3+ [10] ResNeSt-200 48.4 - 88M 1381G 8.1
APbox APbox box mask
APmask mask SETR [78] T-Large‡ 50.3 61.7 308M - -
50 AP75 AP 50 AP75 paramFLOPsFPS
†
DeiT-S 48.0 67.2 51.7 41.4 64.2 44.3 80M 889G 10.4 UperNet DeiT-S† 44.0 - 52M 1099G 16.2
R50 46.3 64.3 50.5 40.1 61.7 43.4 82M 739G 18.0 UperNet Swin-T 46.1 - 60M 945G 18.5
Swin-T 50.5 69.3 54.9 43.7 66.6 47.1 86M 745G 15.3 UperNet Swin-S 49.3 - 81M 1038G 15.2
X101-32 48.1 66.5 52.4 41.6 63.9 45.2 101M 819G 12.8 UperNet Swin-B‡ 51.6 - 121M 1841G 8.7
Swin-S 51.8 70.4 56.3 44.7 67.9 48.5 107M 838G 12.0 UperNet Swin-L‡ 53.5 62.8 234M 3230G 6.2
X101-64 48.3 66.4 52.3 41.7 64.0 45.1 140M 972G 10.4 Table 3. Results of semantic segmentation on the ADE20K val
Swin-B 51.9 70.9 56.5 45.0 68.4 48.7 145M 982G 11.6 and test set. † indicates additional deconvolution layers are used
(c) System-level Comparison to produce hierarchical feature maps. ‡ indicates that the model is
mini-val test-dev pre-trained on ImageNet-22K.
Method #param. FLOPs
APbox APmask APbox APmask
RepPointsV2* [11] - - 52.1 - - -
is implemented with built-in PyTorch functions that are not
GCNet* [6] 51.8 44.7 52.3 45.4 - 1041G
RelationNet++* [12] - - 52.7 - - - all well-optimized. A thorough kernel optimization is be-
SpineNet-190 [20] 52.6 - 52.8 - 164M 1885G yond the scope of this paper.
ResNeSt-200* [75] 52.5 - 53.3 47.1 - -
EfficientDet-D7 [58] 54.4 - 55.1 - 77M 410G Comparison to DeiT The performance of DeiT-S us-
DetectoRS* [45] - - 55.7 48.5 - - ing the Cascade Mask R-CNN framework is shown in Ta-
YOLOv4 P7* [3] - - 55.8 - - -
ble 2(b). The results of Swin-T are +2.5 box AP and +2.3
Copy-paste [25] 55.9 47.2 56.0 47.4 185M 1440G
mask AP higher than DeiT-S with similar model size (86M
X101-64 (HTC++) 52.3 46.0 - - 155M 1033G
Swin-B (HTC++) 56.4 49.1 - - 160M 1043G
vs. 80M) and significantly higher inference speed (15.3 FPS
Swin-L (HTC++) 57.1 49.5 57.7 50.2 284M 1470G vs. 10.4 FPS). The lower inference speed of DeiT is mainly
Swin-L (HTC++)* 58.0 50.4 58.7 51.1 284M - due to its quadratic complexity to input image size.
Table 2. Results on COCO object detection and instance segmen-
tation. † denotes that additional decovolution layers are used to Comparison to previous state-of-the-art Table 2(c)
produce hierarchical feature maps. * indicates multi-scale testing. compares our best results with those of previous state-of-
the-art models. Our best model achieves 58.7 box AP and
51.1 mask AP on COCO test-dev, surpassing the previous
Comparison to ResNe(X)t Table 2(a) lists the results of
best results by +2.7 box AP (Copy-paste [25] without exter-
Swin-T and ResNet-50 on the four object detection frame-
nal data) and +2.6 mask AP (DetectoRS [45]).
works. Our Swin-T architecture brings consistent +3.4∼4.2
box AP gains over ResNet-50, with slightly larger model 4.3. Semantic Segmentation on ADE20K
size, FLOPs and latency.
Table 2(b) compares Swin Transformer and ResNe(X)t Settings ADE20K [80] is a widely-used semantic seg-
under different model capacity using Cascade Mask R- mentation dataset, covering a broad range of 150 semantic
CNN. Swin Transformer achieves a high detection accuracy categories. It has 25K images in total, with 20K for training,
of 51.9 box AP and 45.0 mask AP, which are significant 2K for validation, and another 3K for testing. We utilize
gains of +3.6 box AP and +3.3 mask AP over ResNeXt101- UperNet [66] in mmseg [15] as our base framework for its
64x4d, which has similar model size, FLOPs and latency. high efficiency. More details are presented in the Appendix.
On a higher baseline of 52.3 box AP and 46.0 mask AP us-
ing an improved HTC framework, the gains by Swin Trans- Results Table 3 lists the mIoU, model size (#param),
former are also high, at +4.1 box AP and +3.1 mask AP (see FLOPs and FPS for different method/backbone pairs. From
Table 2(c)). Regarding inference speed, while ResNe(X)t is these results, it can be seen that Swin-S is +5.3 mIoU higher
built by highly optimized Cudnn functions, our architecture (49.3 vs. 44.0) than DeiT-S with similar computation cost.

7
 ImageNet COCO ADE20k MSA in a stage (ms) Arch. (FPS)
method
top-1 top-5 APbox APmask mIoU S1 S2 S3 S4 T S B
w/o shifting 80.2 95.1 47.7 41.5 43.3 sliding window (naive) 122.5 38.3 12.1 7.6 183 109 77
shifted windows 81.3 95.6 50.5 43.7 46.1 sliding window (kernel) 7.6 4.7 2.7 1.8 488 283 187
no pos. 80.1 94.9 49.2 42.6 43.8 Performer [13] 4.8 2.8 1.8 1.5 638 370 241
abs. pos. 80.5 95.2 49.0 42.4 43.2 window (w/o shifting) 2.8 1.7 1.2 0.9 770 444 280
abs.+rel. pos. 81.3 95.6 50.2 43.4 44.0 shifted window (padding) 3.3 2.3 1.9 2.2 670 371 236
rel. pos. w/o app. 79.3 94.7 48.2 41.9 44.1 shifted window (cyclic) 3.0 1.9 1.3 1.0 755 437 278
rel. pos. 81.3 95.6 50.5 43.7 46.1 Table 5. Real speed of different self-attention computation meth-
Table 4. Ablation study on the shifted windows approach and dif- ods and implementations on a V100 GPU.
ferent position embedding methods on three benchmarks, using
the Swin-T architecture. w/o shifting: all self-attention modules ImageNet COCO ADE20k
adopt regular window partitioning, without shifting; abs. pos.: ab- Backbone top-1 top-5 APbox APmask mIoU
solute position embedding term of ViT; rel. pos.: the default set- sliding window Swin-T 81.4 95.6 50.2 43.5 45.8
tings with an additional relative position bias term (see Eq. (4)); Performer [13] Swin-T 79.0 94.2 - - -
app.: the first scaled dot-product term in Eq. (4). shifted window Swin-T 81.3 95.6 50.5 43.7 46.1
Table 6. Accuracy of Swin Transformer using different methods
for self-attention computation on three benchmarks.
It is also +4.4 mIoU higher than ResNet-101, and +2.4
mIoU higher than ResNeSt-101 [75]. Our Swin-L model
with ImageNet-22K pre-training achieves 53.5 mIoU on the invariance in image classification even though it has long
val set, surpassing the previous best model by +3.2 mIoU been shown to be crucial for visual modeling, we find that
(50.3 mIoU by SETR [78] which has a larger model size). inductive bias that encourages certain translation invariance
is still preferable for general-purpose visual modeling, par-
4.4. Ablation Study ticularly for the dense prediction tasks of object detection
and semantic segmentation.
In this section, we ablate important design elements in
the proposed Swin Transformer, using ImageNet-1K image
classification, Cascade Mask R-CNN on COCO object de- Different self-attention methods The real speed of dif-
tection, and UperNet on ADE20K semantic segmentation. ferent self-attention computation methods and implementa-
tions are compared in Table 5. Our cyclic implementation
is more hardware efficient than naive padding, particularly
Shifted windows Ablations of the shifted window ap-
for deeper stages. Overall, it brings a 13%, 18% and 18%
proach on the three tasks are reported in Table 4. Swin-T
speed-up on Swin-T, Swin-S and Swin-B, respectively.
with the shifted window partitioning outperforms the coun-
terpart built on a single window partitioning at each stage by The self-attention modules built on the proposed
+1.1% top-1 accuracy on ImageNet-1K, +2.8 box AP/+2.2 shifted window approach are 40.8×/2.5×, 20.2×/2.5×,
mask AP on COCO, and +2.8 mIoU on ADE20K. The re- 9.3×/2.1×, and 7.6×/1.8× more efficient than those of slid-
sults indicate the effectiveness of using shifted windows to ing windows in naive/kernel implementations on four net-
build connections among windows in the preceding layers. work stages, respectively. Overall, the Swin Transformer
The latency overhead by shifted window is also small, as architectures built on shifted windows are 4.1/1.5, 4.0/1.5,
shown in Table 5. 3.6/1.5 times faster than variants built on sliding windows
for Swin-T, Swin-S, and Swin-B, respectively. Table 6 com-
pares their accuracy on the three tasks, showing that they are
Relative position bias Table 4 shows comparisons of dif- similarly accurate in visual modeling.
ferent position embedding approaches. Swin-T with rela- Compared to Performer [13], which is one of the fastest
tive position bias yields +1.2%/+0.8% top-1 accuracy on Transformer architectures (see [59]), the proposed shifted
ImageNet-1K, +1.3/+1.5 box AP and +1.1/+1.3 mask AP window based self-attention computation and the overall
on COCO, and +2.3/+2.9 mIoU on ADE20K in relation to Swin Transformer architectures are slightly faster (see Ta-
those without position encoding and with absolute position ble 5), while achieving +2.3% top-1 accuracy compared to
embedding, respectively, indicating the effectiveness of the Performer on ImageNet-1K using Swin-T (see Table 6).
relative position bias. Also note that while the inclusion of
absolute position embedding improves image classification 5. Conclusion
accuracy (+0.4%), it harms object detection and semantic
segmentation (-0.2 box/mask AP on COCO and -0.6 mIoU This paper presents Swin Transformer, a new vision
on ADE20K). Transformer which produces a hierarchical feature repre-
While the recent ViT/DeiT models abandon translation sentation and has linear computational complexity with re-

8
spect to input image size. Swin Transformer achieves the (EMA) [44] which do not enhance performance. Note that
state-of-the-art performance on COCO object detection and this is contrary to [60] where repeated augmentation is cru-
ADE20K semantic segmentation, significantly surpassing cial to stabilize the training of ViT. An increasing degree of
previous best methods. We hope that Swin Transformer’s stochastic depth augmentation is employed for larger mod-
strong performance on various vision problems will encour- els, i.e. 0.2, 0.3, 0.5 for Swin-T, Swin-S, and Swin-B, re-
age unified modeling of vision and language signals. spectively.
As a key element of Swin Transformer, the shifted win- For fine-tuning on input with larger resolution, we em-
dow based self-attention is shown to be effective and effi- ploy an adamW [36] optimizer for 30 epochs with a con-
cient on vision problems, and we look forward to investi- stant learning rate of 10−5 , weight decay of 10−8 , and
gating its use in natural language processing as well. the same data augmentation and regularizations as the first
stage except for setting the stochastic depth ratio to 0.1.
Acknowledgement
We thank many colleagues at Microsoft for their help, ImageNet-22K pre-training We also pre-train on the
in particular, Li Dong and Furu Wei for useful discussions; larger ImageNet-22K dataset, which contains 14.2 million
Bin Xiao, Lu Yuan and Lei Zhang for help on datasets. images and 22K classes. The training is done in two stages.
For the first stage with 2242 input, we employ an AdamW
A1. Detailed Architectures optimizer for 60 epochs using a linear decay learning rate
scheduler with a 5-epoch linear warm-up. A batch size of
The detailed architecture specifications are shown in Ta- 4096, an initial learning rate of 0.001, and a weight decay
ble 7, where an input image size of 224×224 is assumed for of 0.01 are used. In the second stage of ImageNet-1K fine-
all architectures. “Concat n × n” indicates a concatenation tuning with 2242 /3842 input, we train the models for 30
of n × n neighboring features in a patch. This operation epochs with a batch size of 1024, a constant learning rate of
results in a downsampling of the feature map by a rate of n. 10−5 , and a weight decay of 10−8 .
“96-d” denotes a linear layer with an output dimension of
96. “win. sz. 7 × 7” indicates a multi-head self-attention A2.2. Object detection on COCO
module with window size of 7 × 7. For an ablation study, we consider four typical ob-
ject detection frameworks: Cascade Mask R-CNN [28, 5],
A2. Detailed Experimental Settings ATSS [76], RepPoints v2 [11], and Sparse RCNN [55] in
A2.1. Image classification on ImageNet-1K mmdetection [9]. For these four frameworks, we utilize the
same settings: multi-scale training [7, 55] (resizing the in-
The image classification is performed by applying a put such that the shorter side is between 480 and 800 while
global average pooling layer on the output feature map of the longer side is at most 1333), AdamW [43] optimizer
the last stage, followed by a linear classifier. We find this (initial learning rate of 0.0001, weight decay of 0.05, and
strategy to be as accurate as using an additional class to- batch size of 16), and 3x schedule (36 epochs with the learn-
ken as in ViT [19] and DeiT [60]. In evaluation, the top-1 ing rate decayed by 10× at epochs 27 and 33).
accuracy using a single crop is reported. For system-level comparison, we adopt an improved
HTC [8] (denoted as HTC++) with instaboost [21], stronger
Regular ImageNet-1K training The training settings multi-scale training [6] (resizing the input such that the
mostly follow [60]. For all model variants, we adopt a de- shorter side is between 400 and 1400 while the longer side
fault input image resolution of 2242 . For other resolutions is at most 1600), 6x schedule (72 epochs with the learning
such as 3842 , we fine-tune the models trained at 2242 reso- rate decayed at epochs 63 and 69 by a factor of 0.1), soft-
lution, instead of training from scratch, to reduce GPU con- NMS [4], and an extra global self-attention layer appended
sumption. at the output of last stage and ImageNet-22K pre-trained
When training from scratch with a 2242 input, we em- model as initialization. We adopt stochastic depth with ra-
ploy an AdamW [36] optimizer for 300 epochs using a co- tio of 0.2 for all Swin Transformer models.
sine decay learning rate scheduler with 20 epochs of linear
A2.3. Semantic segmentation on ADE20K
warm-up. A batch size of 1024, an initial learning rate of
0.001, a weight decay of 0.05, and gradient clipping with ADE20K [80] is a widely-used semantic segmentation
a max norm of 1 are used. We include most of the aug- dataset, covering a broad range of 150 semantic categories.
mentation and regularization strategies of [60] in training, It has 25K images in total, with 20K for training, 2K for val-
including RandAugment [16], Mixup [74], Cutmix [72], idation, and another 3K for testing. We utilize UperNet [66]
random erasing [79] and stochastic depth [34], but not re- in mmsegmentation [15] as our base framework for its high
peated augmentation [30] and Exponential Moving Average efficiency.

9
 downsp. rate
Swin-T Swin-S Swin-B Swin-L
(output size)
concat 4×4, 96-d,
 LN concat 4×4, 96-d, LN concat 4×4, 128-d,
 LN concat 4×4, 192-d,
 LN
4×    
stage 1 win. sz. 7×7, win. sz. 7×7, win. sz. 7×7, win. sz. 7×7,
(56×56) ×2 ×2 ×2 ×2
dim 96, head 3 dim 96, head 3 dim 128, head 4 dim 192, head 6
concat 2×2, 192-d , LN concat 2×2, 192-d , LN concat 2×2, 256-d , LN concat 2×2, 384-d, LN
8×   
stage 2 win. sz. 7×7, win. sz. 7×7, win. sz. 7×7, win. sz. 7×7,
(28×28) ×2 ×2 ×2 ×2
dim 192, head 6 dim 192, head 6 dim 256, head 8 dim 384, head 12
16× concat 2×2, 384-d, LN concat 2×2, 384-d , LN concat 2×2, 512-d , LN concat 2×2, 768-d , LN
stage 3 win. sz. 7×7, win. sz. 7×7, win. sz. 7×7, win. sz. 7×7,
(14×14) ×6 × 18 × 18 × 18
dim 384, head 12 dim 384, head 12 dim 512, head 16 dim 768, head 24
32× concat 2×2, 768-d, LN concat 2×2, 768-d, LN concat 2×2, 1024-d, LN concat 2×2, 1536-d, LN
stage 4 win. sz. 7×7, win. sz. 7×7, win. sz. 7×7, win. sz. 7×7,
(7×7) ×2 ×2 ×2 ×2
dim 768, head 24 dim 768, head 24 dim 1024, head 32 dim 1536, head 48
Table 7. Detailed architecture specifications.

Swin-T Swin-S Swin-B Backbone Optimizer APbox APbox box

50 AP75 AP
mask
APmask
50 APmask
75
input top-1 throughput top-1 throughput top-1 throughput SGD 45.0 62.9 48.8 38.5 59.9 41.4
R50
size acc (image / s) acc (image / s) acc (image / s) AdamW 46.3 64.3 50.5 40.1 61.7 43.4
2242 81.3 755.2 83.0 436.9 83.3 278.1 SGD 47.8 65.9 51.9 40.4 62.9 43.5
X101-32x4d
2562 81.6 580.9 83.4 336.7 83.7 208.1 AdamW 48.1 66.5 52.4 41.6 63.9 45.2
3202 82.1 342.0 83.7 198.2 84.0 132.0 SGD 48.8 66.9 53.0 41.4 63.9 44.7
3842 82.2 219.5 83.9 127.6 84.2 84.7 X101-64x4d
AdamW 48.3 66.4 52.3 41.7 64.0 45.1
Table 8. Swin Transformers with different input image size on Table 9. Comparison of the SGD and AdamW optimizers for
ImageNet-1K classification. ResNe(X)t backbones on COCO object detection using the Cas-
cade Mask R-CNN framework.

In training, we employ the AdamW [43] optimizer with

an initial learning rate of 6 × 10−5 , a weight decay of 0.01, A3.2. Different Optimizers for ResNe(X)t on COCO
a scheduler that uses linear learning rate decay, and a lin-
Table 9 compares the AdamW and SGD optimizers of
ear warmup of 1,500 iterations. Models are trained on 8
the ResNe(X)t backbones on COCO object detection. The
GPUs with 2 images per GPU for 160K iterations. For aug-
Cascade Mask R-CNN framework is used in this compar-
mentations, we adopt the default setting in mmsegmentation
ison. While SGD is used as a default optimizer for Cas-
of random horizontal flipping, random re-scaling within
cade Mask R-CNN framework, we generally observe im-
ratio range [0.5, 2.0] and random photometric distortion.
proved accuracy by replacing it with an AdamW optimizer,
Stochastic depth with ratio of 0.2 is applied for all Swin
particularly for smaller backbones. We thus use AdamW
Transformer models. Swin-T, Swin-S are trained on the
for ResNe(X)t backbones when compared to the proposed
standard setting as the previous approaches with an input
Swin Transformer architectures.
of 512×512. Swin-B and Swin-L with ‡ indicate that these
two models are pre-trained on ImageNet-22K, and trained
with the input of 640×640. References
In inference, a multi-scale test using resolutions that are [1] Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang,
[0.5, 0.75, 1.0, 1.25, 1.5, 1.75]× of that in training is em- Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming
ployed. When reporting test scores, both the training im- Zhou, et al. Unilmv2: Pseudo-masked language models for
ages and validation images are used for training, following unified language model pre-training. In International Con-
common practice [68]. ference on Machine Learning, pages 642–652. PMLR, 2020.
5
[2] Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew
A3. More Experiments Zhai, and Dmitry Kislyuk. Toward transformer-based object
A3.1. Image classification with different input size detection. arXiv preprint arXiv:2012.09958, 2020. 3
[3] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-
Table 8 lists the performance of Swin Transformers with Yuan Mark Liao. Yolov4: Optimal speed and accuracy of
different input image sizes from 2242 to 3842 . In general, object detection. arXiv preprint arXiv:2004.10934, 2020. 7
a larger input resolution leads to better top-1 accuracy but [4] Navaneeth Bodla, Bharat Singh, Rama Chellappa, and
with slower inference speed. Larry S. Davis. Soft-nms – improving object detection with

10
 one line of code. In Proceedings of the IEEE International [17] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong
Conference on Computer Vision (ICCV), Oct 2017. 6, 9 Zhang, Han Hu, and Yichen Wei. Deformable convolutional
[5] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delv- networks. In Proceedings of the IEEE International Confer-
ing into high quality object detection. In Proceedings of the ence on Computer Vision, pages 764–773, 2017. 1, 2
IEEE Conference on Computer Vision and Pattern Recogni- [18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
tion, pages 6154–6162, 2018. 6, 9 and Li Fei-Fei. Imagenet: A large-scale hierarchical image
[6] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han database. In 2009 IEEE conference on computer vision and
Hu. Gcnet: Non-local networks meet squeeze-excitation net- pattern recognition, pages 248–255. Ieee, 2009. 5
works and beyond. In Proceedings of the IEEE/CVF Inter- [19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov,
national Conference on Computer Vision (ICCV) Workshops, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner,
Oct 2019. 3, 6, 7, 9 Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl-
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is
Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- worth 16x16 words: Transformers for image recognition at
end object detection with transformers. In European Confer- scale. In International Conference on Learning Representa-
ence on Computer Vision, pages 213–229. Springer, 2020. 3, tions, 2021. 1, 2, 3, 4, 5, 6, 9
6, 9 [20] Xianzhi Du, Tsung-Yi Lin, Pengchong Jin, Golnaz Ghiasi,
[8] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaox- Mingxing Tan, Yin Cui, Quoc V Le, and Xiaodan Song.
iao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Spinenet: Learning scale-permuted backbone for recogni-
Shi, Wanli Ouyang, et al. Hybrid task cascade for instance tion and localization. In Proceedings of the IEEE/CVF Con-
segmentation. In Proceedings of the IEEE/CVF Conference ference on Computer Vision and Pattern Recognition, pages
on Computer Vision and Pattern Recognition, pages 4974– 11592–11601, 2020. 7
4983, 2019. 6, 9 [21] Hao-Shu Fang, Jianhua Sun, Runzhong Wang, Minghao
[9] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Gou, Yong-Lu Li, and Cewu Lu. Instaboost: Boosting
Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, instance segmentation via probability map guided copy-
Jiarui Xu, et al. Mmdetection: Open mmlab detection tool- pasting. In Proceedings of the IEEE/CVF International Con-
box and benchmark. arXiv preprint arXiv:1906.07155, 2019. ference on Computer Vision, pages 682–691, 2019. 6, 9
6, 9 [22] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhi-
[10] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian wei Fang, and Hanqing Lu. Dual attention network for
Schroff, and Hartwig Adam. Encoder-decoder with atrous scene segmentation. In Proceedings of the IEEE Conference
separable convolution for semantic image segmentation. In on Computer Vision and Pattern Recognition, pages 3146–
Proceedings of the European conference on computer vision 3154, 2019. 3, 7
(ECCV), pages 801–818, 2018. 7 [23] Jun Fu, Jing Liu, Yuhang Wang, Yong Li, Yongjun Bao, Jin-
[11] Yihong Chen, Zheng Zhang, Yue Cao, Liwei Wang, Stephen hui Tang, and Hanqing Lu. Adaptive context network for
Lin, and Han Hu. Reppoints v2: Verification meets regres- scene parsing. In Proceedings of the IEEE/CVF Interna-
sion for object detection. In NeurIPS, 2020. 6, 7, 9 tional Conference on Computer Vision, pages 6748–6757,
[12] Cheng Chi, Fangyun Wei, and Han Hu. Relationnet++: 2019. 7
Bridging visual representations for object detection via trans- [24] Kunihiko Fukushima. Cognitron: A self-organizing multi-
former decoder. In NeurIPS, 2020. 3, 7 layered neural network. Biological cybernetics, 20(3):121–
[13] Krzysztof Marcin Choromanski, Valerii Likhosherstov, 136, 1975. 3
David Dohan, Xingyou Song, Andreea Gane, Tamas Sar- [25] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-
los, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple
Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, copy-paste is a strong data augmentation method for instance
and Adrian Weller. Rethinking attention with performers. segmentation. arXiv preprint arXiv:2012.07177, 2020. 2, 7
In International Conference on Learning Representations, [26] Jiayuan Gu, Han Hu, Liwei Wang, Yichen Wei, and Jifeng
2021. 8 Dai. Learning region features for object detection. In Pro-
[14] Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and ceedings of the European Conference on Computer Vision
Huaxia Xia. Do we really need explicit position encodings (ECCV), 2018. 3
for vision transformers? arXiv preprint arXiv:2102.10882, [27] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu,
2021. 3 and Yunhe Wang. Transformer in transformer. arXiv preprint
[15] MMSegmentation Contributors. MMSegmentation: arXiv:2103.00112, 2021. 3
Openmmlab semantic segmentation toolbox and bench- [28] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir-
mark. https://github.com/open-mmlab/ shick. Mask r-cnn. In Proceedings of the IEEE international
mmsegmentation, 2020. 7, 9 conference on computer vision, pages 2961–2969, 2017. 6,
[16] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V 9
Le. Randaugment: Practical automated data augmenta- [29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
tion with a reduced search space. In Proceedings of the Deep residual learning for image recognition. In Proceed-
IEEE/CVF Conference on Computer Vision and Pattern ings of the IEEE conference on computer vision and pattern
Recognition Workshops, pages 702–703, 2020. 9 recognition, pages 770–778, 2016. 1, 2, 3

11
[30] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten [44] Boris T Polyak and Anatoli B Juditsky. Acceleration of
Hoefler, and Daniel Soudry. Augment your batch: Improving stochastic approximation by averaging. SIAM journal on
generalization through instance repetition. In Proceedings of control and optimization, 30(4):838–855, 1992. 5, 9
the IEEE/CVF Conference on Computer Vision and Pattern [45] Siyuan Qiao, Liang-Chieh Chen, and Alan Yuille. Detectors:
Recognition, pages 8129–8138, 2020. 5, 9 Detecting objects with recursive feature pyramid and switch-
[31] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen able atrous convolution. arXiv preprint arXiv:2006.02334,
Wei. Relation networks for object detection. In Proceed- 2020. 2, 7
ings of the IEEE Conference on Computer Vision and Pattern [46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Recognition, pages 3588–3597, 2018. 3, 5 Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
[32] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen
relation networks for image recognition. In Proceedings of Krueger, and Ilya Sutskever. Learning transferable visual
the IEEE/CVF International Conference on Computer Vision models from natural language supervision, 2021. 1
(ICCV), pages 3464–3473, October 2019. 2, 3, 5 [47] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick,
[33] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kil- Kaiming He, and Piotr Dollár. Designing network design
ian Q Weinberger. Densely connected convolutional net- spaces. In Proceedings of the IEEE/CVF Conference on
works. In Proceedings of the IEEE conference on computer Computer Vision and Pattern Recognition, pages 10428–
vision and pattern recognition, pages 4700–4708, 2017. 1, 2 10436, 2020. 6
[34] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kil- [48] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee,
ian Q Weinberger. Deep networks with stochastic depth. In Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and
European conference on computer vision, pages 646–661. Peter J. Liu. Exploring the limits of transfer learning with a
Springer, 2016. 9 unified text-to-text transformer. Journal of Machine Learn-
[35] David H Hubel and Torsten N Wiesel. Receptive fields, ing Research, 21(140):1–67, 2020. 5
binocular interaction and functional architecture in the cat’s [49] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan
visual cortex. The Journal of physiology, 160(1):106–154, Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-
1962. 3 attention in vision models. In Advances in Neural Informa-
tion Processing Systems, volume 32. Curran Associates, Inc.,
[36] Diederik P Kingma and Jimmy Ba. Adam: A method for
2019. 2, 3
stochastic optimization. arXiv preprint arXiv:1412.6980,
[50] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-
2014. 5, 9
net: Convolutional networks for biomedical image segmen-
[37] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan
tation. In International Conference on Medical image com-
Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.
puting and computer-assisted intervention, pages 234–241.
Big transfer (bit): General visual representation learning.
Springer, 2015. 2
arXiv preprint arXiv:1912.11370, 6(2):8, 2019. 6
[51] K. Simonyan and A. Zisserman. Very deep convolutional
[38] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. networks for large-scale image recognition. In International
Imagenet classification with deep convolutional neural net- Conference on Learning Representations, May 2015. 2, 3
works. In Advances in neural information processing sys- [52] Bharat Singh and Larry S Davis. An analysis of scale in-
tems, pages 1097–1105, 2012. 1, 2 variance in object detection snip. In Proceedings of the
[39] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, IEEE conference on computer vision and pattern recogni-
et al. Gradient-based learning applied to document recog- tion, pages 3578–3587, 2018. 2
nition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [53] Bharat Singh, Mahyar Najibi, and Larry S Davis. Sniper:
2 Efficient multi-scale training. In Advances in Neural Infor-
[40] Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Ben- mation Processing Systems, volume 31. Curran Associates,
gio. Object recognition with gradient-based learning. In Inc., 2018. 2
Shape, contour and grouping in computer vision, pages 319– [54] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon
345. Springer, 1999. 3 Shlens, Pieter Abbeel, and Ashish Vaswani. Bottle-
[41] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, neck transformers for visual recognition. arXiv preprint
Bharath Hariharan, and Serge Belongie. Feature pyramid arXiv:2101.11605, 2021. 3
networks for object detection. In The IEEE Conference [55] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chen-
on Computer Vision and Pattern Recognition (CVPR), July feng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan
2017. 2 Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end
[42] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, object detection with learnable proposals. arXiv preprint
Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence arXiv:2011.12450, 2020. 3, 6, 9
Zitnick. Microsoft coco: Common objects in context. In [56] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,
European conference on computer vision, pages 740–755. Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent
Springer, 2014. 5 Vanhoucke, and Andrew Rabinovich. Going deeper with
[43] Ilya Loshchilov and Frank Hutter. Decoupled weight de- convolutions. In Proceedings of the IEEE conference on
cay regularization. In International Conference on Learning computer vision and pattern recognition, pages 1–9, 2015.
Representations, 2019. 6, 9, 10 2

12
[57] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model [70] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-
scaling for convolutional neural networks. In International contextual representations for semantic segmentation. In
Conference on Machine Learning, pages 6105–6114. PMLR, 16th European Conference Computer Vision (ECCV 2020),
2019. 2, 6 August 2020. 7
[58] Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: [71] Yuhui Yuan and Jingdong Wang. Ocnet: Object context net-
Scalable and efficient object detection. In Proceedings of work for scene parsing. arXiv preprint arXiv:1809.00916,
the IEEE/CVF conference on computer vision and pattern 2018. 3
recognition, pages 10781–10790, 2020. 7 [72] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk
[59] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regular-
Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian ization strategy to train strong classifiers with localizable fea-
Ruder, and Donald Metzler. Long range arena : A bench- tures. In Proceedings of the IEEE/CVF International Con-
mark for efficient transformers. In International Conference ference on Computer Vision, pages 6023–6032, 2019. 9
on Learning Representations, 2021. 8 [73] Sergey Zagoruyko and Nikos Komodakis. Wide residual net-
[60] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco works. In BMVC, 2016. 1
Massa, Alexandre Sablayrolles, and Hervé Jégou. Training [74] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and
data-efficient image transformers & distillation through at- David Lopez-Paz. mixup: Beyond empirical risk minimiza-
tention. arXiv preprint arXiv:2012.12877, 2020. 2, 3, 5, 6, tion. arXiv preprint arXiv:1710.09412, 2017. 9
9 [75] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Mueller, R
reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Manmatha, et al. Resnest: Split-attention networks. arXiv
Polosukhin. Attention is all you need. In Advances in Neural preprint arXiv:2004.08955, 2020. 7, 8
Information Processing Systems, pages 5998–6008, 2017. 1, [76] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and
2, 4 Stan Z Li. Bridging the gap between anchor-based and
[62] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, anchor-free detection via adaptive training sample selection.
Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui In Proceedings of the IEEE/CVF Conference on Computer
Tan, Xinggang Wang, et al. Deep high-resolution represen- Vision and Pattern Recognition, pages 9759–9768, 2020. 6,
tation learning for visual recognition. IEEE transactions on 9
pattern analysis and machine intelligence, 2020. 2 [77] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Explor-
[63] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao ing self-attention for image recognition. In Proceedings of
Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. the IEEE/CVF Conference on Computer Vision and Pattern
Pyramid vision transformer: A versatile backbone for Recognition, pages 10076–10085, 2020. 3
dense prediction without convolutions. arXiv preprint [78] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu,
arXiv:2102.12122, 2021. 3 Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao
[64] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim- Xiang, Philip HS Torr, et al. Rethinking semantic segmen-
ing He. Non-local neural networks. In IEEE Conference tation from a sequence-to-sequence perspective with trans-
on Computer Vision and Pattern Recognition, CVPR 2018, formers. arXiv preprint arXiv:2012.15840, 2020. 2, 3, 6, 7,
2018. 3 8
[65] Ross Wightman. Pytorch image mod- [79] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and
els. https://github.com/rwightman/ Yi Yang. Random erasing data augmentation. In Proceedings
pytorch-image-models, 2019. 6 of the AAAI Conference on Artificial Intelligence, volume 34,
[66] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and pages 13001–13008, 2020. 9
Jian Sun. Unified perceptual parsing for scene understand- [80] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi-
ing. In Proceedings of the European Conference on Com- dler, Adela Barriuso, and Antonio Torralba. Semantic under-
puter Vision (ECCV), pages 418–434, 2018. 7, 9 standing of scenes through the ade20k dataset. International
[67] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Journal on Computer Vision, 2018. 5, 7, 9
Kaiming He. Aggregated residual transformations for deep [81] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. De-
neural networks. In Proceedings of the IEEE Conference formable convnets v2: More deformable, better results. In
on Computer Vision and Pattern Recognition, pages 1492– Proceedings of the IEEE Conference on Computer Vision
1500, 2017. 1, 2 and Pattern Recognition, pages 9308–9316, 2019. 1, 2
[68] Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, [82] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang,
Stephen Lin, and Han Hu. Disentangled non-local neural and Jifeng Dai. Deformable {detr}: Deformable transform-
networks. In Proceedings of the European conference on ers for end-to-end object detection. In International Confer-
computer vision (ECCV), 2020. 3, 7, 10 ence on Learning Representations, 2021. 3
[69] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi,
Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-
to-token vit: Training vision transformers from scratch on
imagenet. arXiv preprint arXiv:2101.11986, 2021. 3

13