An overview over Capsule Networks

Luca Alessandro Dombetzki

Advisor: Marton Kajo
Seminar Innovative Internet Technologies and Mobile Communications
Chair of Network Architectures and Services
Department of Informatics, Technical University of Munich
Email: [email protected]

ABSTRACT ture of a Capsule Network by comparing it to a general CNN.

Hinton et. al recently published the paper “Dynamic Routing Furthermore, we explain the idea and implementation of
Between Capsules” [20], proposing a novel neural network routing-by-agreement algorithm, as well as the loss functions
architecture. This Capsule Network (CapsNet) outperforms and the training of the network (Section 3.3-3.4). Section
state-of-the-art Convolutional Neural Networks on simple 3.5 shows the network’s performance on multiple challenges,
challenges like MNIST [13], MultiMNIST [20] or smallNORB including an overview over novel matrix capsules in Section
[6]. In this paper, we describe multiple aspects of the current 3.6. After summarizing the main advantages and disadvan-
research in Capsule Networks. This includes explaining the tages in Section 3.7 and 3.8, we look at improvements to
shortcomings of CNNs, the idea and architecture of Cap- the CapsNet (Section 3.9). Before drawing a conclusion in
sule Networks and the evaluation on multiple challenges. Section 4, we outline some real world use cases of capsule
Furthermore, we give an overview of current research, im- networks in Section 3.10.
provements and real world applications, as well as advantages
and disadvantages of the CapsNet. 2. CONVOLUTIONAL NEURAL NETWORKS
2.1 Basic CNN architecture
Keywords
capsule networks, overview, computer vision, convolutional
neural networks, pooling

1. INTRODUCTION
In 1986 Geoffrey E. Hinton revolutionized artificial neural
networks with the use of the backpropagation algorithm.
Since then artificial intelligence has made a big leap for-
wards, especially in computer vision. Deep Learning gained Figure 1: Basic architecture of a CNN; figure from [15]
in popularity, when deep convolutional networks performed
extraordinarily well on the ImageNet challenge in 2012 [10].
It has since been a very active field of research, with compa- Fig. 1 shows the basic architecture of a Convolutional Neural
nies like Google and Microsoft being dedicated to it. This Network. The network first extracts learned features, which
brought forth ideas like Inception [23] and Residual blocks are then fed through a fully connected neural network, that
[5], boosting the performance of CNNs. However, all of these produces a classification. The network can learn features by
advancements build upon the basic structure of a CNN. chaining together convolutional blocks. Such a block consists
of a convolutional layer, an activation function and a pooling
Based on his research in human vision, Geoffrey E. Hinton layer. The convolutional layer learns multiple simple fea-
stated that there is something fundamentally wrong with tures, also called kernels. To learn more complex, non-linear,
CNNs [7]. By trying to replicate the human visual cortex, problems, the output is fed through a non-linear activation
he came up with the idea of a capsule as a group of neurons. function (e.g. ReLU). To connect the blocks together, the
In “Dynamic routing between capsules” [20] Hinton et. al
developed a first working implementation, proving this theory.
In computer graphics, a scene is built by putting known
parts into relation, forming a more complex object. Inverse
graphics does the exact opposite, the scene is deconstructed
into parts and their relationships. The main goal of the
Capsule Network is to be capable of performing inverse
graphics [7]. To achieve this, Hinton proposed to encode the
idea of an entity inside a neural network, a capsule [7].
Figure 2: Max pooling example; figure from [2]
In the following, we first explain the basic structure of a
Convolutional Neural Network and its possible shortcomings outputs of the previous block need to be routed to the next
in Section 2. In Section 3.1 and 3.2, we describe the architec- block’s inputs. The most commonly used routing algorithm

Seminars FI / IITM SS 18, 89 doi: 10.2313/NET-2018-11-1_12

Network Architectures and Services, September 2018
is pooling. Max pooling can be seen in Fig. 2 To improve However changes in viewpoint should ideally lead to changes
the classification significantly, it discards unimportant acti- in neural activities [7] (spacial equivariance). This is espe-
vations and only propagates the most prominent ones. This cially important for segmentation and detection tasks, but is
allows the next convolutional layer to work only on “im- also needed in classification. Fig. 4 shows that CNNs can
portant” data and makes the classifier robust against small only detect features, but cannot relate them. This means
transformations in the input data. that, in an extreme case, both images are perfect representa-
tions of a face for a CNN.

2.2.3 Not using the linear structure of vision

A single transformation, such as rotation, can change a signif-
icant number of pixels in the image. Thereby the viewpoint
is the largest source of variance in images [7]. In computer
graphics, composing the scene of multiple parts is a linear
problem, since we know all the relations between the parts.
Without using this linear structure in computer vision, the
problem of detecting an object is not linear anymore, but
far more complex. As a result, a normal CNN has to be
exponentially increased in size and trained with a similarly
exponential amount of data [20].
Figure 3: Feature detections of a CNN from [14]
2.2.4 Dynamic instead of static routing
Adding more blocks allows later blocks to extract features Pooling collects the most prominent activations, but trans-
from results of the previous block. Thereby the network can ports them to the same neurons of the following layer. This is
learn more and more complex features. Like in Fig. 3 this like broadcasting an important information. Thereby, if the
means that the first block of a CNN might learn edges, the input image is translated, a completely different set of neu-
next block learns parts of an object parts and ultimately rons is responsible for handling different kinds of activations.
the last block learns complete objects like a dog, a plane, Our human vision however is smart enough to understand
etc. With enough pooling layers, the network can become that the image is translated and activates the same neurons.
location invariant. Hence a car in the top left corner of the This happens at runtime, hence is dynamic, and not statically
image is detected as well as one in the lower right. preconnected like pooling. This is like letting the neurons
from the next layer choose, what is most interesting to them.
In a nutshell, pooling chooses the best input, while dynamic
2.2 Problems with pooling for routing routing chooses the best worker neuron.
In his talk “What is wrong with convolutional nets?”[7],
Hinton stresses his belief in convolution, however he brought
Nonetheless, pooling still works, as long as the network is
forth four main arguments against using pooling for routing.
big enough and trained with enough data, that neurons
They are explained in the following.
of the following layers can handle every input type. As a
result, CNNs perform very well, when classifying images
2.2.1 Pooling is unnatural with only one kind of object. But especially in detection and
Hinton states that pooling is a “bad fit to the psychology of segmentation, they lack in performance, since the information
shape perception”. Human vision detects the object instanta- of multiple objects has to stay separated.
neously. Based on the information this object holds, we route
it to the area in the brain that best handles that information. 2.3 Solution
Max pooling on the other hand routes the most active in-
To solve these shortcomings, Hinton proposes to propagate
formation to all subsequent neurons. This is unnatural and
not only the probability of a feature’s existence, but also
prevents the network from learning small details.
the spacial relationships, i.e. the pose of the feature[7].
By adopting biological research he tries to tackle all of the
2.2.2 Invariance vs. equivariance problems stated above. His implementation of this solution
is known as Capsule Networks.

3. CAPSULE NETWORKS
Hinton’s basic idea was to create a neural network capable of
inverse graphics. In other words the network should be able
to deconstruct a scene into co-related parts. To achieve this,
the architecture of a neural network needs to be changed to
Figure 4: Both images are being classified as “face” by a reflect the idea of an entity. Every entity gets its own part of
CNN; figure from [9] the network, encapsulating a number of neurons. This entity
is called a capsule.
CNNs try to make the neural activities invariant to small
changes in the viewpoint, pooling them together [7]. This 3.1 The capsule
is helpful for classification tasks, since the label should be A normal layer of neurons will be divided into many cap-
the same, no matter where the object is (spacial invariance). sules, which in turn contain the neurons [20] (see Sec. 3.2).

Seminars FI / IITM SS 18, 90 doi: 10.2313/NET-2018-11-1_12

Network Architectures and Services, September 2018
 as their activation function. This means that the weights, i.e.
the kernel, are shared between all capsules in each 6 × 6 grid.

3.2.3 DigitCaps
Following is the DigitCaps layer, fully connected to the
primary capsules. These are now pure 16D capsules getting
Figure 5: An capsule and neuron in comparison [8] their inputs from the previous primary capsules. The weight
matrix Wij transforms the 8D output of primary capsule i
to a 16D vector as input for digit capsule j (ûj|i ) Eq. 2.
Therefore a capsule is a wrapper around a dedicated group X
of neurons. Fig. 5 shows a simplified comparison between sj = cij ûj|i , ûj|i = Wij ui (2)
a capsule and a neuron. A neuron computes a scalar value i
from a list of scalar values. Since a capsule essentially wraps Therefore each digit capsule has a weighted sum of 32 × 6 × 6
a group of neurons, it computes a vector from a list of in- 8D vectors as input (sj ). Instead of using pooling, the new
put vectors. It is now able to encode entity parameters like technique of routing-by-agreement is used to focus on the
location, skew, etc. [20]. However this also means, that most important inputs. This is discussed in section 3.3.
it does not represent the probability for the existence of
a feature anymore. Instead the length of the vector can
3.2.4 Class predictions
be used as the probability for feature existence, while not
The 10 16D vectors correspond to the numbers 0-9 (10
losing the important pose information. Furthermore this
classes). Because of the squashing function, the length of
also enables the network to learn the parameters by itself,
each of the vectors can be directly used as a probability for
removing the need for crafting them by hand. This means
each class. Hence there are no more fully connected layers
that a n-dimensional (nD) Capsule can learn n parameters
needed for classification (compare to Fig. 1).
and outputs a n-dimensional vector.

For the output vector to model a probability, it’s length has 3.3 Routing-by-agreement
to stay between 0 and 1. Normal activation functions like Routing-by-agreement is a novel dynamic routing technique.
ReLU only work on scalar values, hence a novel non-linear In contrast to pooling, the routing happens at runtime. The
squashing function Eq. 1 was introduced. goal of this technique is to redirect previous capsule outputs
to a following capsule where it agrees with other inputs. In
||sj ||2 sj the scope of inverse graphics this can be compared to routing
vj = (1)
1 + ||sj ||2 ||sj || a detected nose to the face-capsule and not the car capsule.
A detected nose, eye and mouth agree together in the face-
capsule, while the nose would not agree with a wheel and
To understand how the inputs of the capsule are combined
door in the car capsule.
to sj , we will now look into the architecture of the Capsule
Network presented in [20].
This works because of “coincidence filtering”. In a high
dimensional space - in this case the parameter dimension - it
3.2 Architecture is very unlikely for agreements to lie close to another. So a
cluster of agreements can not be, in a probabilistic way, a
coincidence.

As an implementation, Hinton et. al chose an iterative

clustering algorithm, see Alg. 1.

Algorithm 1 Routing algorithm. (from [20])

Figure 6: Capsule Network Architecture as described in [20] 1: procedure Routing(ûj|i , r, l)
2: for all capsule i in layer l and capsule j in layer (l + 1):
The architecture in Fig. 6 shows a Capsule Network for bij ← 0.
classifying images from the MNIST dataset [13]. An input 3: for r iterations do
image is transformed into 10 scalar values, representing the 4: for all capsule i in layer l: ci ← softmax(b
P i)
probability for each number 0-9. 5: for all capsule j in layer (l + 1): sj ← i cij ûj|i
6: for all capsule j in layer (l + 1): vj ← squash(sj )
3.2.1 Conv1 7: for all capsule i in layer l and capsule j in layer
The first layer applies a normal convolution with a 9 × 9 × 1 (l + 1): bij ← bij + ûj|i .vj
kernel to the 28 × 28 × 1 image over 256 channels. return vj

3.2.2 PrimaryCaps In simple terms, the algorithm finds the mean vector of the
The next layer consists of 32 channels, each channel a 6 × 6 cluster (sj ), and weighs all inputs based on their distance to
grid of so called primary capsules. They serve as a transition this mean (bij ) and normalizes the weights with the “routing
between the scalar values of the convolution to 8D vector softmax” (ci ) 3.
outputs. The primary capsules can be seen as another convo- exp(bij )
cij = P (3)
lutional layer with a 9 × 9 × 256 kernel, just with squashing k exp(bik )

Seminars FI / IITM SS 18, 91 doi: 10.2313/NET-2018-11-1_12

Network Architectures and Services, September 2018
The number of iterations r is a hyperparameter for the Table 1: CapsNet classification accuracy. The MNIST aver-
network and is empirically found to produce the best results age and standard deviation results are reported from 3 trials.
around 3 to 5 iterations. When the routing is finished the Methods used were B=Baseline=CNN and C=CapsNet. [20]
input vectors now have an associated weight cij [20].
Routing Rec. MNIST MultiMNIST
3.4 Training Method Iterations Loss (%) (%)
A Capsule Network produces vectors as outputs. Hinton et. B - - 0.39 8.1
al proposed multiple loss functions to be used at the same C 1 no 0.34±0.032 -
time for training [20]. Reconstruction loss is used to train C 1 yes 0.29±0.011 7.5
the capsule parameters, while margin loss is used to optimize C 3 no 0.35±0.036 -
digit classification. Both are explained below. C 3 yes 0.25±0.005 5.2

3.4.1 Reconstruction loss Figure 8: Resulting reconstructions if one of the 16D in a

This loss is normally used to train a neural network unsu- digit capsule is alterered marginally [20].
pervised. It is mostly used in autoencoders [7] to force the
network to find a different representation of the data. There- Scale and
fore it makes sense to train Capsule Networks the same way, thickness
to force the capsules to find adequate pose parameters. Localized
part
Stroke
thickness
Localized
skew
Width and
translation
Localized
part

Figure 7: Decoder network, building on top of the DigitCaps

layer, as described in [20] 3.4.3 Hyperparameters
As stated in Sec. 3.4.2, the margin loss parameters were
To implement this loss, the previous architecture of Fig. 6 defined in [20] as m+ = 0.9, m− = 0.1 and λ = 0.5.
is extended with a decoder network (Fig. 7) to form the
typical architecture of an autoencoder. The decoder network The total loss to be optimized is a weighted combination
consists of two fully connected layers with ReLU activation of both losses. To prevent the reconstruction loss from
and one sigmoid activated layer. The output of 784 values dominating the margin loss, the reconstruction loss is scaled
represents the pixel intensities of a 28 × 28 image, the same down by 0.0005 [20].
size as the images from the MNIST dataset.
Hinton et. al [20] experimented with the number of itera-
As the actual reconstruction loss Hinton et. al [20] used tions in the routing algorithm 1. They empirically found 3
the euclidean distance between the actual image and the iterations, combined with the reconstruction loss, to produce
sigmoid layer output. Additionally, the DigitCaps layer is the best results. This can be seen in table 1.
masked to exactly one capsule as input for the decoder. The
masked capsule corresponds to the ground thruth label, e.g. 3.5 Evaluation
the numbers 0 to 9 as in [20]. This forces the DigitCaps to For evaluation, Hinton et. al generated a new dataset called
encode the actual digits [20]. MultiMNIST. For each sample two digits from the MNIST
dataset were overlapped by 80% [20].
3.4.2 Margin loss
Lk = Tk max(0, m+ −||vk ||)2 +λ (1−Tk ) max(0, ||vk ||−m− )2 The proposed network has been tested on the MNIST and
(4) MultiMNIST dataset. The results are depicted in table 1.
Eq. 4 shows the mathematical definition of the margin loss. This shows that Capsule Networks are able to outperform
In [20], the constants were chosen as m+ = 0.9 and m− = 0.1. the baseline CNN in both challenges, achieving significantly
Tk acts as a switch between two cases. Tk = 1 if a digit of better results in the MultiMNIST dataset.
class k is present Tk = 0 if not. This loss ensures that the
output vectors of the digit capsules are at least m+ long, 3.5.1 Representation of the pose parameters
when the class is detected, and at most m− long when that To prove their goal of encoding transformation parameters
class is not detected. λ = 0.5 is used to prevent the loss from in the capsule, Hinton et. al fed a capsule prediction to the
shrinking all vectors in the initial learning phase. decoder network (see 3.4.1. Fig. 8 displays how small changes
to some of the 16D parameters affect the reconstructions.
This loss function is applied to each digit capsule individually. The results suggest that their goal has been reached.
The total loss is simply the sum of the losses of all digit When decoding the two predicted capsules, this leads to
capsules [20]. very accurate reconstructions, see Fig. 9.

Seminars FI / IITM SS 18, 92 doi: 10.2313/NET-2018-11-1_12

Network Architectures and Services, September 2018
Figure 9: Correct reconstructions (lower image) of the Caps- Figure 11: Example images from the smallNORB dataset [6].
Net on MultiMNIST test dataset (upper image). L:(l1 , l2 ) Multiple object classes in different viewports.
represents the label for the two digits in the image. R:(r1 , r2 )
represents the two digits used for reconstruction. [20]

R:(6, 0) R:(6, 8) R:(7, 1) R:(8, 7) R:(9, 4) R:(9, 5) R:(8, 4)

L:(6, 0) L:(6, 8) L:(7, 1) L:(8, 7) L:(9, 4) L:(9, 5) L:(8, 4)

Figure 10: Capsule Network forced to reconstruct on false

lables (marked with *) on the MultiMNIST dataset. [20].
3.7 Advantages of Capsule Networks
*R:(5, 7) *R:(2, 3) *R:(0, 8) *R:(1, 6) Capsule Networks show multiple advantages compared to
L:(5, 0) L:(4, 3) L:(1, 8) L:(7, 6) classic Convolutional Neural Networks. The following list
with explanations is adapted and extended from [4].

3.7.1 Viewpoint invariance

The use of parameter vectors, or pose matrices [6], allows
Capsule Networks to recognize objects regardless of the view-
point from which they are viewed. Furthermore Capsule
3.5.2 Smart reconstructions Networks are moderately robust to small affine transforma-
As another experiment they forced the decoder network to tions of the data [20].
reconstruct non-predicted capsules (Fig. 10). It can be
observed, that the CapsNet only reconstructed digits that it 3.7.2 Fewer parameters
also detected. Hinton et. al proposed [20], that the model The connections between layers require fewer parameters,
is not just finding the best fit for all the digits in the image. since only neuron groups are fully connected, not the neurons
Instead it also includes the ones that do not exist. Hinton themselves. The CNN trained for MultiMNIST consisted of
et. al suggest, that “in case of (8, 1) the loop of 8 has not 24.56M parameters, while the CapsNet only needed 11.36M
triggered 0 because it is already accounted for by 8. Therefore parameters [20]. This is close to half as many as before.
it will not assign one pixel to two digits if one of them does Matrix capsules with EM-routing required even less [6]. This
not have any other support”[20]. also means that the model can generalize better.

3.6 Matrix Capsules with EM routing 3.7.3 Better generalization to new viewpoints
Hinton et. al published another paper, currently under open CNNs memorize, that an object can be viewed from different
review, called “Matrix Capsules with EM routing” [6]. They viewpoints. This requires the network to “see” all different
propose to use a EM [6] algorithm instead of the current transformations possible. Capsule Networks however general-
routing algorithm 1 from [20]. Additionally they changed ize better to new viewpoints, because parameter information
the capsules to use a 4 × 4 pose matrix instead of a vector. of a capsule can capture these viewpoints as mere linear
Such a matrix is used in computer graphics to compute the transformations [7]. Therefore CapsNets are not as prone to
scene, like it would be seen through a virtual camera. This misclassification of unseen data, as shown in Sec. 3.6.
is called the viewport. Since the network is able to learn this
matrix, it is able to become viewport invariant [6].
3.7.4 Defense against white-box adversarial attacks
They tested this new network on smallNORB dataset (Fig. Common attacks on CNNs use the Fast Gradient Sign Method.
11) and outperformed the current state of the art CNN by It evaluates the gradient of each pixel against the loss of the
45%, reducing the error percentage from 2.56% to 1.4% [6]. network. The pixels are then changed marginally to maximize
Furthermore, they conducted an experiment, training the the loss without distorting the original image. This method
network only on specific viewpoints and testing it on unseen can drop the accuracy of CNNs to below 20%. Capsule
viewpoints [6]. Both networks were trained to the same error Networks however maintain an accuracy over 70% [4].
of 3.7% on seen viewpoints. While the baseline CNN’s error
increased to 20% on unseen viewpoints, the Capsule Network 3.7.5 Validatable
still achieved 13.5%. Based on these results, CapsNets seem A problem for industry usage of CNNs is their black box
to be able to generalize better than CNNs, being able to behaviour. It is neither predictable how a CNN will perform
adapt to 3D viewports in 2D images. on new data, nor can its performance be properly analyzed
and understood. Because Capsule Networks build upon the
concept of inverse graphics, the network’s reasoning can be
explained considerably better than CNNs. Shahroudnejad

Seminars FI / IITM SS 18, 93 doi: 10.2313/NET-2018-11-1_12

Network Architectures and Services, September 2018
et. al [21] proposed an explainability method building natu- 3.8.6 Unoptimized implementation
rally upon capsules and their structure. This suggests, that The original, but not sole, implementation of the Capsule
Capsule Networks are superior to CNNs in validatability. Network can be found at [22]. It shows, that Capsule Net-
works present multiple challenges for current deep learning
3.7.6 Less amount of training data frameworks. Neural networks are often represented as a
Through unsupervised learning and the dynamic routing graph. Therefore, the number of routing iterations must
procedure, Capsule Networks converge in fewer iterations be defined empirically [20], allowing for the f or loop to be
than CNNs. Furthermore, CNNs need exponentially more unfolded beforehand and chained r times together in the
training data to understand affine transformations [20]. graph. Another problem for training neural networks is that
the routing algorithm is dynamic and not easy to paral-
lelize, preventing GPUs from leveraging their full computing
3.8 Challenges for Capsule Networks power. Nevertheless it is very likely that these deep learning
The CapsNet’s early development stage, brings not only frameworks will adapt with time.
common problems with it, but also reveals unique challenges.
In the following both kinds are listed in more detail.
3.9 Further improvements
3.8.1 Scalability to complex data The performance and scalability of the original Capsule Net-
Hinton et. al [20] evaluated the network experimentally on work has been analyzed by Xi et. al in [26]. They came to
the CIFAR10 dataset, failing to perform as good as cur- the conclusion that, apart from minor improvements, the
rent CNNs. The results were comparable to the first CNNs Capsule Network does not work very well on complex data.
tackling the challenge. Matrix capsules and other discussed Capsule Networks initially showed a lot of problems similar
approaches in section 3.9 try to tackle this problem but are to CNNs before 2012. These were problems like unoptimized
still far from performing on the ImageNet challenge. algorithms, vanishing gradients, etc. This inspired further
research in this area, producing multiple different approaches
for a new routing algorithm, new losses, or even complete
3.8.2 Capsules need to model everything restructuring of the architecture. Some of these publications
As described in [20], capsules share this problem with gener- are presented below.
ative models. The network tries to account for everything in
the image. This also means that it performs better, if it can Phaye et. al [17] investigated using DenseNet-like skip-
model the clutter like background noise, instead of having connections to increase the performance of the Capsule Net-
an extra “not-classifiable” category. LaLonde et. al [12] try work. The resulting DCNet was furthermore restructured
to solve this issue by reconstructing not the whole image, in a hierarchical manner (DCNet++), outperforming the
but only the segmentation. This removes the need for the original CapsNet.
network to model the background and allows it to concen-
trate only on the active class. For solving their challenge of Rawlinson et. al [18] also proposed a change in the Cap-
segmenting medical images, this approach shows promising sule Net architecture. By removing the margin loss (sec-
results. tion 3.4.2) and introducing sparsity to the capsules, the
network achieved similar results and generalized very well.
3.8.3 Structure forcing representation of entities Furthermore the network could now be trained completely
The concept of entities was introduced to Capsule Networks unsupervised.
to aid in computer vision and perform inverse graphics. This
network architecture could therefore prevent the network Bahadori et. al [3] developed a novel routing procedure.
from being applied to non-vision areas. However this has not The algorithm is based on the eigen-decomposition of the
yet been investigated thoroughly. votes and converges faster than EM-routing, described in [6].
The connection between the proposed S-Capsules and EM-
3.8.4 Loss functions Capsules is analogous to the connection between Gaussian
Since the network produces vector or matrix outputs, existing Mixture Models and Principal Component Analysis. This
loss functions cannot be simply reused. However, they can analogy suggests why S-Capsules are more robust during the
often be adapted and sometimes leverage the additional data, training [3].
as can be seen in the reconstruction loss. Still, using the
CapsNet on a new dataset will often require a new loss Wang et. al [24] optimized the routing algorithm by leverag-
function as well. ing Kullback-Leibler divergence [11] in regularization. Their
proposed routing outperforms the original routing procedure
in accuracy 1.
3.8.5 Crowding
Human Vision suffers from the “crowding” problem [16]. We
cannot distinguish objects, when they are very close together. 3.10 Use in real world applications
This can also be observed in Capsule Networks [20], since Capsule Networks are currently evaluated on challenging
this concept was used to model the capsule in the first place tasks and in difficult environment, where typical CNNs fail
[20]. Capsules are based upon the idea, that in each location to produce acceptable results. In the following, three example
in the image is at most one instance of the type of entity cases are presented.
that the capsule represents [20]. While this enables capsules
to efficiently encode the representation of the entity [20], this Afshar et. al [1] use Capsule Networks for brain tumor type
could also present itself as a problem for specific use cases. classification. Capsule Networks require less training data

Seminars FI / IITM SS 18, 94 doi: 10.2313/NET-2018-11-1_12

Network Architectures and Services, September 2018
than CNNs. This is very important in the medical environ- Imagenet classification with deep convolutional neural
ment, resulting in CapsNets being the superior network for networks. In Advances in neural information processing
this task. systems, pages 1097–1105, 2012.
[11] S. Kullback and R. A. Leibler. On information and
Wang et. al [25] employ the capsule concept in a Recurrent sufficiency. The annals of mathematical statistics,
Neural Network. Thereby they achieved state-of-the-art 22(1):79–86, 1951.
performance in sentiment analysis tasks. [12] R. LaLonde and U. Bagci. Capsules for Object
Segmentation. ArXiv e-prints, Apr. 2018.
LaLonde et. al [12] designed a new network architecture [13] Y. LeCun, C. Cortes, and C. Burges. Mnist dataset,
similar to U-Nets [19]. Apart from outperforming the baseline 1998.
U-Net model on large 512 × 512 images, they reduced the [14] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng.
parameters needed by 95.4%. Convolutional deep belief networks for scalable
unsupervised learning of hierarchical representations. In
4. CONCLUSION Proceedings of the 26th annual international conference
Hinton et. al propose with Capsule Networks a completely on machine learning, pages 609–616. ACM, 2009.
new way for Convolutional Neural Networks to analyze data. [15] Mathworks. Convolutional neural network. https:
The routing-by-agreement algorithm 1 tackles the problems of //www.mathworks.com/solutions/deep-learning/
pooling (2.2) [7]. Together with the concept of capsules, this convolutional-neural-network.html; last accessed
enables networks to be viewpoint invariant and robust against on 2018/06/14.
affine transformations. This eliminates the main reason [16] D. G. Pelli. Crowding: A cortical constraint on object
for huge amounts of data being needed in CNN training. recognition. Current opinion in neurobiology,
Overall the new architecture holds many advantages over 18(4):445–451, 2008.
the typical CNN. Current research shows that, with some [17] S. S. R. Phaye, A. Sikka, A. Dhall, and D. Bathula.
alterations, Capsule Networks are able to perform even in Dense and diverse capsule networks: Making the
complex scenarios [1]. However, Capsule Networks have to capsules learn better. arXiv preprint arXiv:1805.04001,
be developed further to outperform, or even replace CNNs in 2018.
real world scenarios, especially when data is not a problem. [18] D. Rawlinson, A. Ahmed, and G. Kowadlo. Sparse
unsupervised capsules generalize better. CoRR,
5. REFERENCES abs/1804.06094, 2018.
[1] P. Afshar, A. Mohammadi, and K. N. Plataniotis. [19] O. Ronneberger, P. Fischer, and T. Brox. U-net:
Brain tumor type classification via capsule networks. Convolutional networks for biomedical image
CoRR, abs/1802.10200, 2018. segmentation. In International Conference on Medical
[2] Aphex34. Convolutional neural network - max pooling. image computing and computer-assisted intervention,
https://en.wikipedia.org/wiki/Convolutional_ pages 234–241. Springer, 2015.
neural_network#Max_pooling_shape; last accessed on [20] S. Sabour, N. Frosst, and G. E. Hinton. Dynamic
2018/06/14. routing between capsules. In Advances in Neural
[3] M. T. Bahadori. Spectral capsule networks. 2018. Information Processing Systems, pages 3859–3869,
[4] S. Garg. Demystifying “matrix capsules with em 2017.
routing.”. https://towardsdatascience.com/ [21] A. Shahroudnejad, A. Mohammadi, and K. N.
demystifying-matrix-capsules-with-em-routing Plataniotis. Improved explainability of capsule
-part-1-overview-2126133a8457; last accessed on networks: Relevance path by agreement. CoRR,
2018/06/14. abs/1802.10204, 2018.
[5] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual [22] soskek. Capsnets tensorflow implementation.
learning for image recognition. CoRR, abs/1512.03385, https://github.com/soskek/dynamic_routing_
2015. between_capsules; last accessed on 2018/06/14.
[6] G. Hinton, S. Sabour, and N. Frosst. Matrix capsules [23] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
with em routing. 2018. D. Anguelov, D. Erhan, V. Vanhoucke, and
[7] G. E. Hinton. What is wrong with convolutional neural A. Rabinovich. Going deeper with convolutions. In The
nets? Talk recorded on youtube, IEEE Conference on Computer Vision and Pattern
https://youtu.be/rTawFwUvnLE; last accessed on Recognition (CVPR), June 2015.
2018/06/14. [24] D. Wang and Q. Liu. An optimization view on dynamic
[8] K.-Y. Ho. Capsules: Alternative to pooling. routing between capsules. 2018.
https://datawarrior.wordpress.com/2017/11/14/ [25] Y. Wang, A. Sun, J. Han, Y. Liu, and X. Zhu.
capsules-alternative-to-pooling/; last accessed on Sentiment analysis by capsules. In Proceedings of the
2018/08/18. 2018 World Wide Web Conference on World Wide
[9] T. Kothari. Uncovering the intuition behind capsule Web, pages 1165–1174. International World Wide Web
networks and inverse graphics. Conferences Steering Committee, 2018.
https://hackernoon.com/ [26] E. Xi, S. Bing, and Y. Jin. Capsule Network
uncovering-the-intuition-behind-capsule-networks Performance on Complex Data. ArXiv e-prints, Dec.
-and-inverse-graphics-part-i-7412d121798d; last 2017.
accessed on 2018/06/14.
[10] A. Krizhevsky, I. Sutskever, and G. E. Hinton.

Seminars FI / IITM SS 18, 95 doi: 10.2313/NET-2018-11-1_12

Network Architectures and Services, September 2018