Histopathology Image Classification using Bag of

Features and Kernel Functions

Juan C. Caicedo, Angel Cruz and Fabio A. Gonzalez

Bioingenium Research Group

National University of Colombia
{jccaicedoru,aacruzr,fagonzalezo}@unal.edu.co
http://www.bioingenium.unal.edu.co

Abstract. Image representation is an important issue for medical im-

age analysis, classification and retrieval. Recently, the bag of features
approach has been proposed to classify natural scenes, using an analogy
in which visual features are to images as words are to text documents.
This process involves feature detection and description, construction of
a visual vocabulary and image representation building through visual-
word occurrence analysis. This paper presents an evaluation of different
representations obtained from the bag of features approach to classify
histopathology images. The obtained image descriptors are processed us-
ing appropriate kernel functions for Support Vector Machines classifiers.
This evaluation includes extensive experimentation of different strate-
gies, and analyses the impact of each configuration in the classification
result.

1 Introduction
Medical imaging applications are challenging because they require effective and
efficient content representations to manage large image collections. The first
stage for medical image analysis is modeling image contents by defining an ap-
propriate representation. This is a fundamental problem for all image analysis
tasks such as image classification, automatic image annotation, object recogni-
tion and image retrieval, which require discriminative representations according
to the application domain. During the last few years, the bag of features ap-
proach has attracted great attention from the computer vision community. This
approach is an evolution of texton-based representations and is also influenced
by the bag of words assumption in text classification. In text documents, a word
dictionary is defined and all documents are processed so that the frequency of
each word is quantified. This representation ignores word relationships in the
document, i.e., it does not take into account the document structure. An anal-
ogy is defined for images in which a feature dictionary is built to identify visual
patterns in the collection. This representation has shown to be effective in dif-
ferent image classification, categorization and retrieval tasks [1,2,3].
The bag of features representation is an adaptive approach to model image
structure in a robust way. In contrast to image segmentation, this approach does
not attempt to identify complete objects inside images, which may be a harder
task than the image classification itself. Instead, the bag of features approach
looks for small characteristic image regions allowing the representation of com-
plex image contents without explicitly modeling objects and their relationships,
a task that is tackled in another stage of image content analysis. In addition,
an important advantage of the bag of features approach is its adaptiveness to
the particular image collection to be processed. In the same way as text docu-
ments, in which the appropriate word-list to be included in the dictionary may
be identified earlier in the process, the bag of features approach allows to iden-
tify visual patterns that are relevant to the whole image collection. That is, the
patterns that are used in a single image representation come from the analysis
of patterns in the complete collection. Other important characteristics of this
approach are the robustness to occlusion and affine transformations as well as
its computational efficiency [2].
Some of these properties are particularly useful for medical image analysis
and, in fact, the bag of features representation has been successfully applied to
some problems in medical imaging [4,5]. Histopathology images have a particular
structure, with a few colors, particular edges and a wide variety of textures. Also,
objects such as glands or tissues may appear anywhere in the image, in different
proportions and at different zoom levels. All those properties make the bag of
features a potentially appropriate representation for that kind of visual contents.
Up to our knowledge, the bag of features representation has not been evaluated
yet on histopathology images and that is the main goal of this paper.
This paper presents a systematic evaluation of different representations ob-
tained from the bag of features approach to classify histopathology images. There
are different possibilities to design an image descriptor using the bag of features
framework and each one lead to different image representations that may be
more or less discriminative. In addition, the obtained image descriptors are pro-
cessed using two kernel functions for Support Vector Machine classifiers. The
performed experiments allow to analyze the impact of different strategies in the
final classification result. The paper is organized as follows: Section 2 presents
the previous work on histopathology image classification. Section 3 describes
the bag of features methodology and all applied techniques. Section 4 presents
experimental results, and finally the concluding remarks are in Section 5.

2 Histopathology Image Classification

2.1 Previous work
In different application contexts, medical images have been represented using a
wide variety of descriptors including colors, textures, regions and transformation-
based coefficients. Such descriptors are usually motivated by visual properties
that can be identified by domain experts in target images. For instance, Long et.
al [6] developed an algorithm based on active contours to segment vertebrae in
x-ray spine images, and then match nine semantic points to evaluate a particular
disease. Although the algorithm is very effective in such a task, it has two main
disadvantages: first, the computational effort required to process each image, and
second, the method is devised to work only on that particular kind of medical
images. Other more generic descriptors have been proposed for classification and
retrieval of medical images. Guld et. al [7] proposes to down-scale medical images
up to 32x32 pixels and use that as a feature vector for classification tasks. In
[8] the use of global histogram features are used to retrieve a wide variety of
medical images. Even though that descriptors are by nature simple and generic,
they lack of direct semantics and may lead to poor results in large-scale real
applications. Hence, there is a trade-off between the semantics and the generality
of the representation.
In histopathology images that tendency can also be observed. Most of the
previous works in the context of histology, pathology and tissue image classifi-
cation have approached the problem using segmentation techniques [9,10]. They
first define the target object to be segmented (e.g. cells, nuclei, tissues) and then
apply a computational strategy to identify it. Global features have also been used
to retrieve and classify histology images [11,12]. Those two global approaches are
in one extreme of the balance between explicit semantics and practical gener-
alization or adaptation. Other kind of works have oriented the image content
analysis by window-based features, under the observation that histology images
are “usually composed of different kinds of texture components” [13]. In [14] those
sub-images are classified individually and then a semantic analyzer is used to de-
cide the final image classification on the complete image. This approach is close
to the bag of features one since the unit of analysis is a small sub-image and a
learning algorithm is applied to evaluate the categorization of small sub-images.
However, the approach requires the annotation of example sub-images to train
first-stage classifiers, a process that is performed in an unsupervised fashion
under the bag of features framework.

2.2 Histopathology image dataset

The image dataset has been previously used in an unrelated clinical study to
diagnose a special skin cancer known as basal-cell carcinoma. Basal-cell carci-
noma is the most common skin disease in white populations and its incidence
is growing world wide [15]. It has different risk factors and its development is
mainly due to ultraviolet radiation exposure. Pathologists confirm whether or
not this disease is present after a biopsied tissue is evaluated under microscope.
The database is composed of 1,502 images annotated by experts into 18 cate-
gories. Each label corresponds to a histopathology concept which may be found
in a basal-cell carcinoma image. An image may have one or several labels, that
is to say, different concepts may be recognized within the same image and the
other way around.

3 The Bag of Features Framework

The bag of features framework is an adaptation of the bag of words scheme used
for text categorization and text retrieval. The key idea is the construction of a
 Fig. 1. Overview of the Bag of Features framework

codebook, that is, a visual vocabulary, in which the most representative patterns
are codified as codewords or visual words. Then, the image representation is
generated through a simple frequency analysis of each codeword inside the image.
Csurka et. al [2] describe four steps to classify images using a bag of features
representation: (1) Feature detection and description, (2) Codebook generation,
(3) the bag of features construction and finally (4) training of learning algorithms.
Figure 1 shows an overview of those steps. The bag of features approach is a
flexible and adaptable framework, since each step may be determined by different
techniques according to the application domain needs. The following subsections
present the particular methods and techniques that have been evaluated in this
work.

3.1 Feature detection and description

Feature detection is the process in which the relevant components of an image
must be identified. Usually, the goal of feature detection is set to identify a spa-
tially limited image region that is salient or prominent. Different strategies have
been proposed by the computer vision community to detect local features, that
are motivated by different visual properties such as corners, edges or saliency.
Once local features are detected, the next step is to describe or characterize the
content of such local regions. Ideally, two local features should have the same
descriptor values if they refer to the same visual concept. That motivates the
implementation of descriptors that are invariant to affine transformations and
illumination changes.
In this work two feature detection strategies with their corresponding feature
descriptor have been evaluated. The first strategy is dense random sampling. The
goal of this strategy is to select points in the image plane randomly and then,
define a block of pixels around that coordinate. The size of the block is set to 9×9
pixels, and the descriptor for these blocks is the vector with explicit pixel values
in gray scales. This descriptor will be called raw block, but it is also known as
texton or raw pixel descriptor. The advantage of this strategy is its simplicity and
computational efficiency. In addition, a large number of blocks may be extracted
from different image scales, and that sample is a good approximation of the
probability distribution of visual patterns in the image [16].
The second strategy is based on Scale-Invariant Feature Transform (SIFT)
points [17]. This strategy uses a keypoint detector based on the identification of
interesting points in the location-scale space. This is implemented efficiently by
processing a series of difference-of-Gaussian images. The final stage of this algo-
rithm calculates a rotation invariant descriptor using predefined orientations over
a set of blocks. We use SIFT points with the most common parameter configura-
tion: 8 orientations and 4 × 4 blocks, resulting in a descriptor of 128 dimensions.
The SIFT algorithm has demonstrated to be a robust keypoint descriptor in dif-
ferent image retrieval and matching applications, since it is invariant to common
image transformations, illumination changes and noise.

3.2 Codebook construction

The visual dictionary or codebook is built using a clustering or vector quantiza-

tion algorithm. In the previous stage of the bag of features framework, a set of
local features has been extracted. All local features, over a training image set, are
brougth together independently of the source image and are clustered to learn a
set of representative visual words from the whole collection. The k-means algo-
rithm is used in this work to find a set of centroids in the local features dataset.
An important decision in the construction of the codebook is the selection of
its size, that is, how many codeblocks are needed to represent image contents.
According to different works on natural image classification, the larger the code-
book size the better [16,2]. However, Tomassi et. al [4] found that the size of the
codebook is not a significant aspect in a medical image classification task. We
evaluated different codebook sizes, to analyze the impact of this parameter in the
classification of histopathology images.
The goal of the codebook construction is to identify a set of visual patterns
that reflects the image collection contents. Li et. al [18] have illustrated a code-
book for natural scene image categorization that contains several visual primi-
tives, such as orientations and edges. The result is consistent with the contents of
that image collection. In the same way, we illustrate a codebook extracted from
the collection of histopathology images. It is composed of 150 codeblocks as is
shown in Figure 2. In this case the codeblocks are also reflecting the contents of
the histopathology image collection. Visual primitives in this case may be rep-
resenting cells and nuclei of different sizes and shapes. That codebook has been
generated using the raw-block descriptor described in the previous Subsection.
Fig. 2. A codebook with 150 codeblocks for the histopathology image collection. Code-
blocks are sorted by frequency.

3.3 Image Representation

The final image representation is calculated by counting the occurrence of each
codeblock in the set of local features of an image. This is why the framework
receives its name, the bag of features, since the geometric relationships of local
features are not taken into account. This representation is known as term fre-
quencies (TF) in text applications and is also adopted in this work. Normalized
term frequencies can also be calculated by normalizing according to the number
of features extracted from the image. This may be specially useful for the SIFT
points strategy, in which the number of features from image to image may have
a large variation. Those are the standard image representations, commonly used
for image categorization. In addition, the inverse document frequency (IDF) has
also been used as weighting scheme to produce a new image representation.

3.4 Kernel Functions

Classifiers used in this work are Support Vector Machines (SVM), that receives
as input a data representation impliciltly defined by a kernel function [19]. Kernel
functions describe a similarity relationship between the objects to be classified.
The image representation that we are dealing with are histograms with term
frequencies. In that sense, a natural choice of a kernel function would be a simi-
larity measure between histogram structures. The histogram intersection kernel
is the first kernel function evaluated in this work:
X
M
D∩ (H, H ′ ) = ′
min(Hm , Hm )
m=1
′
Where H and H are the term frequency histograms of two images, calcu-
lated using a codebook with M codeblocks. In addition, a Radial Basis Function
composed with the histogram intersection kernel has been also implemented and
tested:

K(H, H ′ ) = exp(−γD∩ (H, H ′ ))

4 Experimental Evaluation
4.1 Experimental setup
The collection has 1,502 histopathology images with examples of 18 different
concepts. The collection is split into 2 datasets, the first one for training and
validation, and the second one for testing. The dataset partition is done us-
ing stratified sampling in order to preserve the original distribution of examples
in both datasets. This is particularly important due to the high imbalance of
classes. In the same way, the performance measures reported in this work are
precision, recall and F-measure to evaluate the detection rate of positive exam-
ples, since the class imbalance may produce trivial classifiers with high accuracy
that do not recognize any positive sample. In addition, since one image can be
classified in many classes simultaneously, the classification strategy is based on
binary classifiers following the one-against-all rule. Experiments to evaluate dif-
ferent configurations of the bag of features approach have been done. For each
experiment, the regularization parameter of the SVM is controlled using 10-fold
cross validation in the training dataset, to guarantee good generalization on the
test dataset. Reported results are calculated on the test dataset and averaged
over all 18 classes.

4.2 Results
The first evaluation focuses on the codebook size. We have tested six different
codebook sizes, starting with 50 codeblocks and following with 150, 250, 500,
750 and 1000. Figure 3 shows a plot for codebook size vs. F-measure using two
different bag of features configurations. The first strategy, is based on SIFT points
and the second is based on raw blocks. Perhaps surprisingly, the plot shows that
the classification performance decreases while the codebook size increases. This
behaviour is explained by the intrinsic structure of histopathology images: they
are usually composed of some kinds of textures, that is, the number of distinctive
patterns in the collection is limited. This fact can also be seen in the codebook
illustrated in Figure 2, which shows several repeated patterns, even with just
150 codeblocks. In the limit, it is reasonable that a large codebook size does not
have any discriminative power because each different pattern in an image has its
own codeblock.
The nature of the descriptor is also a determinant factor in this behaviour
since the performance of the SIFT points decreases faster than the performance
of raw blocks. This suggests that a SIFT-based codebook requires less codeblocks
to express all different patterns in the image collection, which is consistent with
the rotation and scale invariance properties of that descriptor. On the other
 Codebook-Size vs. F-Measure
0.25
SIFT
Blocks

0.2

0.15
F-Measure

0.1

0.05

0
0 200 400 600 800 1000
Codebook Size

Fig. 3. Codebook size vs. F-Measure for two bag of features representation using SIFT
points and Blocks.

hand, a block-based codebook requires a larger size because it is representing the

same visual patterns using different codeblocks.
The second factor to evaluate is the feature descriptor. As is shown in Fig-
ure 3, the raw-block descriptor has obtained a better performance in terms of
F-measure among all codebook sizes. Table 1 shows the performance summary
of the different configurations evaluated in this work. In bold are the best val-
ues of precision, recall and F-measure, showing that block-based strategies are
more effective in general. An important question here is why SIFT points, that
are provided with robust invariant properties, are not a good choice with this
type of histopathology images. First, there is some evidence that they were not
designed to find the most informative patches for image classification [16], and
second, it is possible that all the attempts to increase the invariance of features
in histopathology images, lead to a loss of discriminative information.
The next aspect in this evaluation is the image representation, i.e. the use
of absolute term frequencies (TF) or the use of the weighted scheme provided
by inverse document frequencies (IDF). According to the results presented in
Table 1, it is not clear when IDF improves the classification performance. In
the case of the SIFT descriptor, IDF produces a poorer performance in most
of the cases. In constrast, in the raw-block strategy, the IDF is increasing the
importance of discriminative codeblocks, resulting in an improvement of the
classification performance. Finally, the use of the RBF kernel in general shows
an improvement in precision, either for SIFT points or blocks. However, the
recall value is in general hurted by the use of the RBF kernel.
Table 1. Performance measures for the evaluated configurations of the bag of features.

Kernel function BOF SIFT Raw-Blocks

Precision Recall F-Measure Precision Recall F-Measure
Hist. Intersection TF 0.480 0.152 0.207 0.610 0.162 0.234
Hist. Intersection IDF 0.473 0.128 0.189 0.634 0.152 0.231
RBF Kernel TF 0.393 0.146 0.205 0.647 0.123 0.190
RBF Kernel IDF 0.506 0.136 0.165 0.673 0.155 0.237

5 Conclusions and Future Work

This paper presented an evaluation of the bag of features approach to classify
histopathology images. This is the first systematic evaluation of this representa-
tion scheme on this particular medical image domain. The developed evaluation
includes a comparative study of different methods and techniques in each stage
of the bag of features approach. The main advantage of the proposed approach
is its adaptiveness to the particular contents of the image collection. Previous
work in histology and pathology image analysis used global-generic features or
segmentation-based approaches that are not easily extended to other applica-
tions even in the same domain.
The adaptiveness property of this framework is obtained with an automated
codebook construction, which should have enough patterns to describe the image
collection contents. In this domain, we found that the codebook size is very small
compared with the codebook size required in other applications such as natural
scene classification or even in other kinds of medical images (i.e. mamography
and x-rays). The main reason of this smaller size is due to the structure of
histopathology images which exhibit homogeneous tissues and the representative
visual patterns among the whole collection tends to be uniform.
The bag of features representation is a flexible framework that may be adapted
in different ways, either in the visual feature descriptors and the codebook con-
struction. The future work includes a more extensive evaluation of codebook
construction methods and different strategies to include color information into
visual words as well as more robust texture descriptors. This evaluation will also
include a comparison against other commonly used representation schemes.

Acknowledgments
This work has been partially supported by Ministerio de Educación Nacional de
Colombia grant 1101393199, according to the COLCIENCIAS call 393-2006 to
support research projects using the RENATA network.

References
1. Bosch, A., Muñoz, X., Martı́, R.: Which is the best way to organize/classify images
by content? Image and Vision Computing 25(6) (June 2007) 778–791
 2. Csurka, G., Dance, C.R., Fan, L., Willamowski, J., Bray, C.: Visual categorization
with bags of keypoints. In: Workshop on Statistical Learning in Computer Vision.
(2004)
3. Sivic, J., Zisserman, A.: Video Google: a text retrieval approach to object matching
in videos. (2003) 1470–1477 vol.2
4. Tommasi, T., O.F.C.B.: CLEF2007 Image annotation task: An SVM-based cue
integration approach. In: Working Notes of the 2007 CLEF Workshop, Budapest,
Hungary (2007)
5. Iakovidis, D.K., Pelekis, N., Kotsifakos, E.E., Kopanakis, I., Karanikas, H.,
Theodoridis, Y.: A pattern similarity scheme for medical image retrieval. In-
formation Technology in Biomedicine, IEEE Transactions on (2008)
6. Long, L.R., Antani, S.K., Thoma, G.R.: Image informatics at a national research
center. Computerized Medical Imaging and Graphics 29 (2005) 171–193
7. Guld, M.O., Keysers, D., Deselaers, T., Leisten, M., Schubert, H., Ney, H.,
Lehmann, T.M.: Comparison of global features for categorization of medical im-
ages. Medical Imaging 5371 (2004) 211–222
8. Deselaers, T., Keysers, D., Ney, H.: FIRE - Flexible Image Retrieval Engine:
imageCLEF 2004 evaluation. In: CLEF 2004, LNCS 3491. (2004) 688–698
9. Datar, M., Padfield, D., Cline, H.: Color and texture based segmentation of molec-
ular pathology images using hsoms. In: Biomedical Imaging: From Nano to Macro,
2008. ISBI 2008. 5th IEEE International Symposium on. (2008) 292–295
10. Comaniciu, D., Meer, P., Foran, D.: Shape-based image indexing and retrieval
for diagnostic pathology. In: Pattern Recognition, 1998. Proceedings. Fourteenth
International Conference on. Volume 1. (1998) 902–904 vol.1
11. Caicedo, J.C., Gonzalez, F.A., Romero, E.: A semantic content-based retrieval
method for histopathology images. Information Retrieval Technology LNCS 4993
(2008) 51–60
12. Zheng, L., Wetzel, A.W., Gilbertson, J., Becich, M.J.: Design and analysis of
a content-based pathology image retrieval system. Information Technology in
Biomedicine, IEEE Transactions on 7(4) (2003) 249–255
13. Lam, R.W.K., Ip, H.H.S., Cheung, K.K.T., Tang, L.H.Y., Hanka, R.: A multi-
window approach to classify histological features. In: Pattern Recognition, 2000.
Proceedings. 15th International Conference on. Volume 2. (2000) 259–262 vol.2
14. Tang, H.L., Hanka, R., Ip, H.H.S.: Histological image retrieval based on semantic
content analysis. Information Technology in Biomedicine, IEEE Transactions on
7(1) (2003) 26–36
15. Fletcher, C.D.M.: Diagnostic Histopathology of tumors. Elsevier Science (2003)
16. Nowak, E., Jurie, F., Triggs, B.: Sampling strategies for bag-of-features image
classification. (2006) 490–503
17. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Interna-
tional Journal of Computer Vision 60(2) (November 2004) 91–110
18. Li, F.F., Perona, P.: A bayesian hierarchical model for learning natural scene cate-
gories. In: CVPR ’05: Proceedings of the 2005 IEEE Computer Society Conference
on Computer Vision and Pattern Recognition (CVPR’05) - Volume 2, Washington,
DC, USA, IEEE Computer Society (2005) 524–531
19. Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge
University Press (2004)