UNET 3+: A FULL-SCALE CONNECTED UNET

FOR MEDICAL IMAGE SEGMENTATION

Huimin Huang 1, *Lanfen Lin1, Ruofeng Tong1, *Hongjie Hu2, Qiaowei Zhang2, Yutaro Iwamoto3,
Xianhua Han3, *Yen-Wei Chen3,4,1, Jian Wu1
1
College of Computer Science and Technology, Zhejiang University, China
2
Department of Radiology, Sir Run Run Shaw Hospital, China
3
College of Information Science and Engineering, Ritsumeikan University, Japan
4
Research Center for Healthcare Data Science, Zhejiang Lab, Hangzhou, China
*Corresponding Authors: Lanfen Lin ([email protected]), Hongjie Hu ([email protected]), Yen-Wei Chen ([email protected])
ABSTRACT connections, aiming at reducing the semantic gap between the
Recently, a growing interest has been seen in deep learning- encoder and decoder. Despite achieving good performance,
based semantic segmentation. UNet, which is one of deep this type of approach is still incapable of exploring sufficient
learning networks with an encoder-decoder architecture, is information from full scales.
widely used in medical image segmentation. Combining As witnessed in many segmentation studies [1-7], feature
multi-scale features is one of important factors for accurate maps in different scale explore distinctive information. Low-
segmentation. UNet++ was developed as a modified Unet by level detailed feature maps capture rich spatial information,
designing an architecture with nested and dense skip connec- which highlight the boundaries of organs; while high-level
tions. However, it does not explore sufficient information semantic feature maps embody position information, which
from full scales and there is still a large room for improve- locate where the organs are. Nevertheless, these exquisite sig-
ment. In this paper, we propose a novel UNet 3+, which takes nals may be gradually diluted when progressively down- and
advantage of full-scale skip connections and deep supervi- up-sampling. To make full use of the multi-scale features, we
sions. The full-scale skip connections incorporate low-level propose a novel U-shape-based architecture, named UNet 3+,
details with high-level semantics from feature maps in differ- in which we re-design the inter-connection between the en-
ent scales; while the deep supervision learns hierarchical rep- coder and the decoder as well as the intra-connection between
resentations from the full-scale aggregated feature maps. The the decoders to capture fine-grained details and coarse-
proposed method is especially benefiting for organs that ap- grained semantics from full scales. To further learn hierar-
pear at varying scales. In addition to accuracy improvements, chical representations from the full-scale aggregated feature
the proposed UNet 3+ can reduce the network parameters to maps, each side output is connected with a hybrid loss func-
improve the computation efficiency. We further propose a tion, which contributes to accurate segmentation especially
hybrid loss function and devise a classification-guided mod- for organs that appear at varying scales in the medical image
ule to enhance the organ boundary and reduce the over-seg- volume. In addition to accuracy improvements, we also show
mentation in a non-organ image, yielding more accurate seg- that the proposed UNet 3+ can reduce the network parameters
mentation results. The effectiveness of the proposed method to improve the computation efficiency.
is demonstrated on two datasets. The code is available at: To address the demand for more accurate segmentation in
github.com/ZJUGiveLab/UNet-Version medical image, we further investigate how to effectively re-
Index Terms—Segmentation, Full-scale skip connection, duce the false positives in non-organ images. Existing meth-
Deep supervision, Hybrid loss function, Classification. ods solve the problem by introducing attention mechanisms
[8] or conducting a pre-defined refinement approach such as
1. INTRODUCTION CRF [4] at inference. Different from these methods, we ex-
Automatic organ segmentation in medical images is a critical tend a classification task to predict the input image whether
step in many clinical applications. Recently, convolutional has organ, providing a guidance to the segmentation task.
neural networks (CNNs) greatly promoted to developed a va- In summary, our main contributions are four-fold: (i) de-
riety of segmentation models, e.g. fully convolutional neural vising a novel UNet 3+ to make full use of the multi-scale
networks (FCNs) [1], UNet [2], PSPNet [3] and a series of features by introducing full-scale skip connections, which
DeepLab version [4-6]. Especially, UNet, which is based on incorporate low-level details with high-level semantics from
an encoder-decoder architecture, is widely used in medical feature maps in full scales, but with fewer parameters; (ii)
image segmentation. It uses skip connections to combine the developing a deep supervision to learn hierarchical represen-
high-level semantic feature maps from the decoder and cor- tations from the full-scale aggregated feature maps, which
responding low-level detailed feature maps from the encoder. optimizes a hybrid loss function to enhance the organ bound-
To recede the fusion of semantically dissimilar feature from ary; (iii) proposing a classification-guided module to reduce
plain skip connections in UNet, UNet++ [7] further strength- over-segmentation on none-organ image by jointly training
ened these connections by introducing nested and dense skip with an image-level classification; (iv) conducting extensive
Fig .1: Comparison of UNet (a), UNet++(b) and proposed UNet 3+ (c). The depth of each node is presented below the circle.
experiments on liver and spleen datasets, where UNet 3+ on the concatenated feature map from five scales, which con-
yields consistent improvements over a number of baselines. sists of 320 filters of size 3 × 3, a batch normalization and a
ReLU activation function. Formally, we formulate the skip
2. METHODS
connections as follows: let i indexes the down-sampling layer
Fig.1 gives simplified overviews of UNet, UNet++ and the along the encoder, 𝑁 refers to the total number of the encoder.
proposed UNet 3+. Compared with UNet and UNet++, UNet The stack of feature maps represented by 𝑋"# ,
is computed as:
3+ combines the multi-scale features by re-designing skip ,
𝑋%& , 𝑖 = 𝑁
connections as well as utilizing a full-scale deep supervision,
,
which provides fewer parameters but yields a more accurate 𝑋"# = 4 ,6' , 4 B (1)
ℋ ∁ 𝒟 𝑋%& 45'
, ∁ 𝑋%& , ∁ 𝒰 𝑋"# 45,A'
, 𝑖 = 1, ⋯ , 𝑁 − 1
position-aware and boundary-enhanced segmentation map. 789:#;:'=> ~, => 789:#;:(,A')=> ~B =>

2.1. Full-scale Skip Connections where function ∁(∙) denotes a convolution operation, ℋ(∙)
realizes the feature aggregation mechanism with a convolu-
The proposed full-scale skip connections convert the inter- tion followed by a batch normalization and a ReLU activation
connection between the encoder and decoder as well as intra- function. 𝒟(∙) and 𝒰(∙) indicate up- and down-sampling op-
connection between the decoder sub-networks. Both UNet eration respectively, and [∙] represents the concatenation.
with plain connections and UNet++ with nested and dense It is worth mentioning that our proposed UNet 3+ is more
connections are short of exploring sufficient information efficient with fewer parameters. In the encoder sub-network,
from full scales, failing to explicitly learn position and bound- UNet, UNet++, and UNet 3+ share the same structure, where
ary of an organ. To remedy the defect in UNet and UNet++, ,
𝑋%& has 32× 2, channels. As for the decoder, the depth of
each decoder layer in UNet 3+ incorporates both smaller- and feature map in UNet is symmetric to the encoder, and thus
same-scale feature maps from encoder and larger-scale fea- ,
𝑋"# also has 32 × 2, channels. The number of parameters in
ture maps from decoder, which capturing fine-grained details LM ,
𝑖 decoder stage of UNet (𝑃O6"# ) can be computed as:
and coarse-grained semantics in full scales.
(
As an example, Fig. 2 illustrates how to construct the ,
𝑃O6"# ,A'
= 𝐷Q ×𝐷Q × 𝑑 𝑋"# ,
×𝑑 𝑋"# ,
+ 𝑑 𝑋"#
$
feature map of 𝑋"# . Similar to the UNet, the feature map from ,
+ 𝑑 𝑋%& ,
+ 𝑋"# ,
×𝑑 𝑋"# (2)
$
the same-scale encoder layer 𝑋%& are directly received in the
where 𝐷Q is the convolution kernel size, 𝑑 ∙ denotes the
decoder. In contrast to the UNet, a set of inter encoder-decode
depth of the nodes. When it comes to UNet++, it makes use
skip connections delivers the low-level detailed information
' ( of a dense convolution block along each skip pathway, where
from the smaller-scale encoder layer 𝑋%& and 𝑋%& , by apply-
ing non-overlapping max pooling operation; while a chain of 𝑃O, TT6"# can be computed as:
(
intra decoder skip connections transmits the high-level se- 𝑃O, TT 6"# = 𝐷Q ×𝐷Q × 𝑑 𝑋"#
,A' ,
×𝑑 𝑋"# ,
+ 𝑑 𝑋"# +
)
mantic information from larger-scale decoder layer 𝑋"# and 𝑑 ,
𝑋%& + B6'6,
45'
,,4
𝑋U# ,
+ 𝑋"# ,
×𝑑 𝑋"# (3)
*
𝑋"# , by utilizing bilinear interpolation. With the five same- As can be seen, 𝑃O, TT6"# is larger than 𝑃O6"#
,
. While in
resolution feature maps in hand, we need to further unify the UNet 3+, each decoder feature map is derived from 𝑁 scales,
number of channels, as well as reduce the superfluous infor-
yielding 64 × 𝑁 channels. 𝑃O, XT6"# can be computed as:
mation. It occurred to us that the convolution with 64 filters (
𝑃O, XT 6"# = 𝐷Q ×𝐷Q × ,
45' 𝑑
4
𝑋%& + B
45,A' 𝑑
4
𝑋"# ,
×64 + 𝑑 𝑋"# (4)
of size 3 × 3 can be a satisfying choice. To seamlessly merge
the shallow exquisite information with deep semantic infor- For the sake of the channel reduction, the parameters in
mation, we further perform a feature aggregation mechanism UNet 3+ is fewer than those in UNet and UNet++.
Fig. 2: Illustration of how to construct the full-scale aggre-
$
gated feature map of third decoder layer 𝑋"# . Fig. 3: Illustration of classification-guided module (CGM).
tance. It is, in all probability, caused by noisy information
2.2. Full-scale Deep Supervision from background remaining in shallower layer, leading to the
In order to learn hierarchical representations from the full- phenomenon of over-segmentation. To achieve more accu-
scale aggregated feature maps, the full-scale deep supervision rate segmentation, we attempt to solve this problem by adding
is further adopted in the UNet 3+. Compared with the deep an extra classification task, which is designed for predicting
supervision performed on generated full-resolution feature the input image whether has organ or not.
map in UNet++, the proposed UNet 3+ yields a side output As depicted in Fig. 3, after passing a series of operations
from each decoder stage, which is supervised by the ground including dropout, convolution, maxpooling and sigmoid, a
*
truth. To realize deep supervision, the last layer of each de- 2-dimensional tensor is produced from the deepest-level 𝑋%& ,
coder stage is fed into a plain 3 × 3 convolution layer fol- each of which represents the probability of with/without or-
lowed by a bilinear up-sampling and a sigmoid function. gans. Benefiting from the richest semantic information, the
To further enhance the boundary of organs, we propose a classification result can further guide each segmentation side-
multi-scale structural similarity index (MS-SSIM) [9] loss output in two steps. First, with the help of the argmax func-
function to assign higher weights to the fuzzy boundary. Ben- tion, 2-dimensional tensor is transferred into a single output
efiting from it, the UNet 3+ will keep eye on fuzzy boundary of {0,1}, which denotes with/without organs. Subsequently,
as the greater the regional distribution difference, the higher we multiply the single classification output with the side seg-
the MS-SSIM value. Two corresponding N×N sized patches mentation output. Due to the simpleness of binary classifica-
are cropped from the segmentation result P and the ground tion task, the module effortlessly achieves accurate classifi-
truth mask G, which can be denoted as 𝑝 = {𝑝[ : 𝑗 = cation results under the optimization of Binary Cross Entropy
1, … , 𝑁 ( } and 𝑔 = 𝑔[ : 𝑗 = 1, … , 𝑁 ( , respectively. The MS- loss function [12], which realizes the guidance for remedying
SSIM loss function of p and g is defined as: the drawback of over-segmentation on none-organ image.
U fg ig
2𝜇c 𝜇d + 𝐶' 2𝜎cd + 𝐶( 3. EXPERIMENTS AND RESULTS
ℓa;6;;,a = 1 − (5)
𝜇c( + 𝜇d( + 𝐶' 𝜎c( + 𝜎d( + 𝐶(
a5' 3.1. Datasets and Implementation
where 𝑀 is the total number of the scales, 𝜇c , 𝜇d and 𝜎c , 𝜎c
The method was validated on two organs: the liver and spleen.
are the mean and standard deviations of p and g, 𝜎cd denotes
The dataset for liver segmentation is obtained from the ISBI
their covariance. 𝛽a and 𝛾a define the relative importance of LiTS 2017 Challenge. It contains 131 contrast-enhanced 3D
the two components in each scale, which are set according to abdominal CT scans, of which 103 and 28 volumes are used
[9]. Two small constants 𝐶' = 0.01( and 𝐶( = 0.03( are for training and testing, respectively. The spleen dataset from
added to avoid the unstable circumstance of dividing by zero. the hospital passed the ethic approvals, containing 40 and 9
In our experiment, we set the scale to 5 based on [9]. CT volumes for training and testing. In order to speed up
By combining focal loss ( ℓp: ) [10], MS-SSIM loss training, the input image had three channels, including the
(ℓa;6;;,a ) and IoU loss (ℓ,qr ) [11], we develop a hybrid loss slice to be segmented and the upper and lower slices, which
for segmentation in three-level hierarchy – pixel-, patch- and was cropped to 320×320. We utilized the stochastic gradient
map-level, which is able to capture both large-scale and fine descent to optimize our network and its hyper parameters
structures with clear boundaries. The hybrid segmentation were set to the default values. Dice coefficient was used as
loss (ℓ;#d )is defined as: the evaluation metric for each case.
ℓ;#d = ℓp: + ℓa;6;;,a + ℓ,qr (6) 3.2. Comparison with UNet and UNet++
2.3. Classification-guided Module (CGM) In this section, we first compared the proposed UNet 3+ with
In the most medical image segmentations, the appearance of UNet and UNet++. The loss function used in each method is
false-positives in a non-organ image is an inevitable circums- the focal loss.
Table 1: Comparison of UNet, UNet++, the proposed UNet 3+ without deep supervision (DS) and UNet 3+ on liver and spleen
datasets in terms of Dice metrics. The best results are highlighted in bold. The loss function used in each method is focal loss.

Vgg-16 ResNet-101
Architecture 𝐷𝑖𝑐𝑒9u#v9d#
Params 𝐷𝑖𝑐𝑒:,u#v 𝐷𝑖𝑐𝑒;c:##& Params 𝐷𝑖𝑐𝑒:,u#v 𝐷𝑖𝑐𝑒;c:##&
UNet 39.39M 0.9206 0.9023 55.90M 0.9387 0.9332 0.9237
UNet++ 47.18M 0.9278 0.9230 63.76M 0.9475 0.9423 0.9352
UNet 3+ w/o DS 26.97M 0.9489 0.9437 43.55M 0.9580 0.9539 0.9511
UNet 3+ 26.97M 0.9550 0.9496 43.55M 0.9601 0.9560 0.9552

Table 2: Comparison of UNet 3+ and other 5 state-of-the-art

methods. The best results are highlighted in bold.
Method 𝐷𝑖𝑐𝑒:,u#v 𝐷𝑖𝑐𝑒;c:##&
PSPNet [3] 0.9242 0.9240
DeepLabV2 [4] 0.9021 0.9097
DeepLabV3 [5] 0.9217 0.9217
DeepLabV3+ [6] 0.9186 0.9290
Attention UNet [8] 0.9341 0.9324
UNet 3+ (focal loss) 0.9601 0.9560
Fig. 4: Qualitative comparisons of ResNet-101-based UNet, UNet 3+ (Hybrid loss) 0.9643 0.9588
UNet++, and proposed UNet 3+ on liver dataset. Purple ar- UNet 3+ (Hybrid loss + CGM) 0.9675 0.9620
eas: true positive (TP); Yellow areas: false negative (FN);
Green areas: the false positive (FP). fuzzy boundary, yielding more enhance boundary-aware seg-
(i) Quantitative comparison: Based on the backbone of mentation map. Moreover, taking advantages of the classifi-
Vgg-16 and ResNet-101, Table 1 compares UNet, UNet++ cation-guidance module, UNet 3+ skillfully avoids the over-
and the proposed UNet 3+ architecture in terms of the number segmentation in complex background. As it can be seen, this
of parameters and segmentation accuracy on both liver and approach is outstanding compared to all other previous ap-
spleen datasets. As seen, UNet 3+ without deep supervision proaches. It is also worth noting that the proposed method
achieves a surpassing performance over UNet and UNet++, outperforms the second best result on the liver (0.9675
obtaining average improvement of 2.7 and 1.6 point between against 0.9341) and spleen (0.9620 against 0.9324).
two backbones performed on two datasets. Taking account of
4. CONCLUSIONS
both liver and spleen appearing at varying scales in CT slices,
UNet 3+ combined with full-scale deep supervision further In this paper, we proposed a full-scale connected UNet,
improved 0.4 point. named UNet 3+ with deep supervision in order to make max-
(ii) Qualitative comparison: Figure 2 exhibits the segmenta- imum use of feature maps in full scales for accurate segmen-
tion results of ResNet-101-based UNet, UNet++ and UNet 3+ tation and efficient network architecture with fewer parame-
with full-scale deep supervision on liver datasets. It can be ters. The classification-guided module and a hybrid loss func-
observed that our proposed method not only accurately local- tion is further introduced to yielding more accurate position-
izes organs but also produces coherent boundaries, even in aware and boundary-aware segmentation map. Experimental
small object circumstances. results on both liver and spleen datasets demonstrated that
UNet 3+ surpasses all previous state-of-the-art approaches,
3.3. Comparison with the State of the Art
highlighting the organs and producing coherent boundaries.
We quantitatively compare our ResNet-101-based UNet 3+
with several recent state-of-the-art methods: PSPNet [3], De- 5. ACKNOWLEDGMENTS
epLabV2 [4], DeepLabV3 [5], DeepLabV3+ [6] and Atten- This work was supported in part by Major Scientific Research
tion UNet [8]. It is worth mentioning that all results are di- Project of Zhejiang Lab under the Grant No.2018DG0ZX01,
rectly from single-model test without relying on any post-pro- in part by the Key Science and Technology Innovation Sup-
cessing tools. Moreover, all networks were optimized by the port Program of Hangzhou under the Grant
loss function proposed in their own paper. No.20172011A038, and in part by the Grant-in Aid for Sci-
Table 2 summarizes the quantitative comparison results. entific Research from the Japanese Ministry for Education,
As seen, the proposed hybrid loss function greatly improves Science, Culture and Sports (MEXT) under the Grant No.
the performance by taking pixel-, patch-, map-level optimi- 18H03267 and No.17H00754.
zation in to consideration. Especially, the patch-level MS-
SSIM loss function contributes to assign higher weights to the
 6. REFERENCES

[1] J. Long, E. Shelhamer and T. Darrell, “Fully Convolutional

Networks for Semantic Segmentation,” The IEEE Conference
on Computer Vision and Pattern Recognition, pp. 3431-3440,
2015.
[2] O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional
Networks for Biomedical Image Segmentation,” Medical Im-
age Computing and Computer-Assisted Intervention, pp.234-
241, 2015.
[3] H.S. Zhao, J.P. Shi, X.J. Qi, X.G. Wang and J.Y. Jia, “Pyramid
scene parsing network,” The IEEE Conference on Computer Vi-
sion and Pattern Recognition, pp. 2881-2890, 2017.
[4] L-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A.L.
Yuille, “Semantic image segmentation with deep convolutional
nets, atrous convolution, and fully connected crfs,” IEEE trans-
actions on pattern analysis and machine intelligence, vol.40,
no.4, pp: 834-848, 2018.
[5] L-C. Chen, G. Papandreou, F. Schroff and H. Adam, “Rethink-
ing atrous convolution for semantic image segmentation”,
arXiv preprint arXiv:1706.05587, 2017.
[6] L-C. Chen, Y.K. Zhu, G. Papandreou and H. Adam, “Encoder-
decoder with atrous separable convolution for semantic image
segmentation,” Proceedings of the European Conference on
Computer Vision, 2018. 

[7] Z.W. Zhou, M.M.R. Siddiquee, N. Tajbakhsh and J.M. Liang,
“UNet++: A Nested U-Net Architecture for Medical Image
Segmentation,” Deep Learning in Medical Image Anylysis and
Multimodal Learning for Clinical Decision Support, pp: 3-11,
2018.
[8] O.O et al., “Attention u-net: Learning where to look for the pan-
creas,” Medical Imaging with Deep Learning, 2018.
[9] Z. Wang, E.P. Simoncelli and A.C. Bovik, “Multiscale struc-
tural similarity for image quality assessment,” The Thrity-Sev-
enth Asilomar Conference on Signals, Systems & Computers,
2003.
[10] T.-Y. Lin, P. Goyal, R. Girshick, K.M. He and P. Dollar. “Focal
loss for dense object detection,” The IEEE international con-
ference on computer vision, pp. 2980-2988, 2017. 

[11] G. Mattyus, W.J. Luo, and R. Urtasun, “Deep-roadmapper: Ex-
tracting road topology from aerial images”, The IEEE interna-
tional conference on computer vision, 2017.
[12] P.-T. Boer et al., “A tutorial on the cross-entropy method,” An-
nals of Operations Research, vol.134, no.1, pp. 19–67, 2005.