Flower Classification with Deep CNN and Machine

Learning Algorithms
Büşra Rümeysa Mete Tolga Ensari
Computer Engineering Department Computer Engineering Department
Istanbul Kultur University Istanbul University - Cerrahpasa
Istanbul, Turkey Istanbul, Turkey
[email protected] [email protected]

Abstract—Development of the recognition of rare plant species species, such as elecampane, verbascum thapsus whose life is
will be advantageous in the fields such as the pharmaceutical limited to a specific area and grown only under special climatic
industry, botany, agricultural, and trade activities. It was also conditions, will support the development of the pharmaceutical
very challenging that there is diversity of flower species and it
is very hard to classify them when they can be very similar industry. Thanks to the studies in this field, little-known plants
to each other indeed. Therefore, this subject has already become will be able to see the true value they deserve.
crucial. In this context, this paper presents a classification system
for flower images by using Deep CNN and Data Augmentation. II. R ELATED W ORK
Recently, Deep CNN techniques have become the latest technol- In order to classify flowers, researchers were paying more
ogy for such problems. However, the fact is that getting better
performance for the flower classification is stuck due to the lack of attention to segmenting images and selecting them in some
labeled data. In the study, there are three primary contributions: ways that can be called as artificial. This traditional method
First, we proposed a classification model to cultivate the perfor- is more primitive today when development of the computer
mance of classifying of flower images by using Deep CNN for science and the technology -coming with it- have been getting
extracting the features and various machine learning algorithms improved unceasingly, because it is not running in a fully
for classifying purposes. Second, we demonstrated the use of
image augmentation for achieving better performance results. automatic process since it is required human interposition.
Last, we compared the performances of the machine-learning Moreover, this method can’t achieve high accuracy enough.
classifiers such as SVM, Random Forest, KNN, and Multi-Layer In [1] a visual vocabulary was introduced by the authors
Perceptron(MLP). In the study, we evaluated our classification from Oxford University. This vocabulary includes many differ-
system using two datasets: Oxford-17 Flowers, and Oxford-102 ent features such as colour, shape and texture that discriminate
Flowers. We divided each dataset into the training and test sets
by 0.8 and 0.2, respectively. As a result, we obtained the best between flowers from different classes. They also presented a
accuracy for Oxford 102-Flowers Dataset as 98.5% using SVM study [2] combining four different features of flowers in order
Classifier. For Oxford 17-Flowers Dataset, we found the best to classify them by using a multiple kernel framework with
accuracy as 99.8% with MLP Classifier. These results are better Support Vector Machine (SVM) classifier. So, they reached
than others’ that classify the same datasets in the literature. 88.33% accuracy on the quite challenging datasets of Oxford
Index Terms—machine learning; cnn; feature extraction; data
augmentation; flower classification
102 Flowers which is also one of the dataset used in our study.
In the study [3], fine-grained classification was used by the
authors on the same datasets used in our study that are Oxford
I. I NTRODUCTION
17-Flowers and Oxford-102 Flowers. In this study [3] they
Flowers are the most important producers of the earth that obtained 93.14% and 79.1 accuracy for the datasets of Oxford-
can grow in a wide variety of climates in terms of their 17 Flowers and Oxford-102 Flowers, respectively.
habitats. They also keep on playing a very important role In 2017, the study [4] used Google’s pre-trained model
in the food chain by feeding almost all insect species in the named GoogLeNet with inception-v3 module for the purpose
world. In addition to this they play an important role in the of classifying flowers. The authors selected two datasets same
food chain, and many drugs can be produced by using their as the studies have mentioned above and also in our study:
healing properties. For such reasons, having a good knowledge Oxford 17-Flowers and Oxford-102 Flowers. Although both
of flowers and knowing their species is very important in datasets are not quite large, they have reached quite high
terms of recognizing a new or rare plant species. Otherwise, accuracy of 95% and 94%, respectively.
many plants may be damaged because they are considered In order to make a good comparison, we used the multi-
harmful to one’s farmland or may be sold at very cheap prices. class datasets prepared by Oxford Geometry Group, which
And all this occurs due to the inadequate recognition of the are also used in other studies and also the most challenging
plant species. However, it is a real phenomenon that many datasets in this field. The original side of the study is that
of the plants grown in nature can be cultivated. In addition, we use GoogleNet’s inception-v3 module to extract features.
increasing the recognition capacity of numerous endemic plant Moreover, due to the lack of labeled data, we also used

978-1-7281-3789-6/19/$31.00 ©IEEE, 2019.

Authorized licensed use limited to: Rajamangala Univ of Technology Isan provided by UniNet. Downloaded on June 09,2023 at 06:11:44 UTC from IEEE Xplore. Restrictions apply.
 a data augmentation technique before the feature extraction
phase. After dividing our data into training and testing, we
classified the features we extracted using inception-v3 with
various machine learning algorithms.
III. DATASET
In this study, we use 17-Flowers and 102-Flowers datasets
belonging to Oxford Visual Geometry Group. Both of these
datasets are composed of various flowers that are common
in England. The 17-Flowers dataset contains 80 images from Fig. 2: Proposed classification system
each flower group while 102-Flowers dataset contains at least
40 and up to 200 images for each category. Images have large
scale, exposure and light variations. In addition to this, there
are many classes that show great differences in the dataset as
well as many classes with similarities to other classes. These
well-known datasets are used for fine-grained recognition
explained in Section I. Sample images of 20 species of flowers
from Oxford-102 Flowers dataset can be seen in Fig. 1.

Fig. 3: Samples of vertically flipped images

technique. We have doubled the number of images in both

datasets by vertically flipping all the images.

B. Feature Extraction with Deep CNN

There are several CNN models has been introduced and
Fig. 1: Samples of 20 species of flowers in Oxford 102-Flower utilized in the literature. For instance, in 2013, Min Lin and
Dataset her colleagues proposed a novel deep network architecture [5].
The name of this new deep network architecture is ”Network
in Network” which is the name of the article as well. With this
IV. P ROPOSED C LASSIFICATION S YSTEM article, they have been the pioneers of the idea that linear filters
The proposed classification system in the study includes in convolutional neural networks can increase the learning ca-
four fundamental components: the modules of data, and data pacity with the usage of non-linear activation functions. They
augmentation, and feature extraction, and classifier. It is shown obtained the model by adding a multilayer sensor structure
in Fig. 2 (perceptron) into the convolutional neural network which is
in linear structure. This model was successful in data sets
A. Data Augmentation such as MNIST, CIFAR10, and CIFAR100. Furthermore, it is
We used one of the data augmentation techniques for the observed that the performance is increased for all datasets after
purpose of obtaining better performance on natural image data augmentation process and dropout process were applied.
classification to handle the disadvantages of the lack of labeled However, in 2015 with the article [6] named ”Going Deeper
data. This method is very common as well as easy to diminish with Convolutions”, Google introduced the idea of going
overfitting problem of scarceness of tagged data in flower clas- deeper with the architecture named GoogLeNet (inception-
sification. In the study, we used a particular data augmentation v1). With the article [6] published in 2015, a deeper and

Authorized licensed use limited to: Rajamangala Univ of Technology Isan provided by UniNet. Downloaded on June 09,2023 at 06:11:44 UTC from IEEE Xplore. Restrictions apply.
 wider model was designed. They obtained the concept of width D. Support Vector Machine
within the model by the module called as inception. Support Vector Machine (SVM) is one of the most effective
There are 4 modules for GoogLeNet model in the literature and simplest supervised learning algorithm used in classifica-
that are called as inception-v1 [6], inception-v2 [7], inception- tion. However, it is often used in classification problems.
v3 [7], and inception-v4 [8]. Inception-v3 includes two parts In Support Vector Machines, there is no prior knowledge
that are feature extraction part and classification part. Its fea- or assumption about the distribution of input data. Input
ture extraction part includes convolutional neural network. On data can be separated linearly or non-linearly. In addition,
the other hand its classification part includes fully connected there is no over-fitting problem in SVM. In artificial neural
and softmax layers. All the mentioned layers in inception-v3 networks, over-fitting may occur unless cross validation is
has shown in Fig. 41 . applied. Moreover, various kernel functions can be used to
In this study, we used the previous module of GoogLeNet make inseparable problems detachable and to map data in
which is inception-v3 as a feature extractor by removing better viewing space. Kernel-based algorithms are quite flex-
its classification part. In Fig. 4, feature extraction part and ible. Because, the algorithm is independent from the hyper
removed classification part of the module are seen. We men- parameters such as learning rate and fixing parameters. The
tioned the images in datasets in Section III and we fed them other reason for this is that it is sufficient to change the kernel
into the model that we used in the study after removing its function when the problem area changes [10].
classification layer to achieve the purpose of producing a group The most used kernel functions such as polynomial kernel
of tagged feature vectors. (1), linear kernel (2) and Gaussian/RBF (3) kernel were given
Inception-v3 is a well-known architecture. The network’s in the specified equations.
input must be an image with sizes of 299x299 pixels.
K(x, y) = (x × y + 1)d (1)
K(x, y) = x × y (2)
2
||x − y||
K(x, y) = exp(− ) (3)
2σ 2
In this study, we obtained the highest success using SVM
classifier. Having experienced different types of kernels such
as polynomial, linear, Gaussian/RBF, we observed that the
highest accuracy of 98,5% in 114,99 seconds on 102 Flowers
Dataset. We obtained these results by using the LinearSVC
Fig. 4: Feature extraction and classification parts of Inception- library of sklearn with default values of its parameters.
v3.
E. Random Forest
Random Forest is one of the popular models of machine
C. Dimension Reduction learning, because it can be applied to both regression and
classification problems. Furthermore, it gives good results
We experienced that it is very difficult to understand or without hyper parameter estimation. In Random Forest, it is
discover the relationships between features for a multidimen- aimed to increase the classification value by using more than
sional data set during the experiments. So we applied the one decision tree during the classification process. One of
dimension reduction process and showed its outputs. Note that the biggest problems of decision trees is over-learning that
it is just an informative step. can also called as memorizing data or more technically over-
While Principle Component Analysis (PCA) is a linear fitting. To solve this problem, the random forest model selects
feature extraction technique, t-SNE is a non-linear technique and trains tens or even hundreds of different sets of randomly
for dimensionality reduction and it is more suitable for the from both the data set and the feature set. With this method,
visualization of high-dimensional datasets [9]. hundreds of decision trees are created and each decision tree
In this study, after CNN features obtained and saved t- is individually estimated.
distributed stochastic neighbor embedding (t-SNE) technique It is obtained the accuracy of 86.4% in 140.586323 seconds
was applied for an only informative reason. Fig. 5 demon- with Oxford 102 Flowers Dataset.
strates the transforming high dimensional features into two
F. KNN
dimensional feature plane for the datasets Oxford-17 Flowers
and Oxford 102-Flowers, respectively. In KNN, there are many different distance formulas in find-
It is also seen the same color points are frequently clustered ing distance such as Euclidean Distance, Manhattan Distance
jointly in Fig. 5. In this way, it was foreboded that the features and Minkowski Distance. Their equations have given in the
to train our classifier with superior accuracy. following Formulas 4, 5, 6, respectively.
v
u k
uX
1 The figure has been taken from https://codelabs.developers.google.com in t (x − y )2 (4)
i i
06.09.2019 i=1

Authorized licensed use limited to: Rajamangala Univ of Technology Isan provided by UniNet. Downloaded on June 09,2023 at 06:11:44 UTC from IEEE Xplore. Restrictions apply.
 Fig. 5: Transforming of the datasets into two dimensional feature plane without applying data augmentation

k
X from KNN classifier by using manhattan distance as 99.1% in
|xi − yi | (5)
1.53 seconds. Finally, the lowest accuracy was obtained from
i=1
the Random Forest classifier as 95,4% in 5,03 seconds. The
k
X findings were demonstrated in Fig.6
( (|xi − yi |)q )1/q (6)
i=1

In the study, the best accuracy was obtained with manhattan

distance as 96.2% in 41.943569 seconds. On the other hand, it
was obtained the accuracy of 96.0% with minkowski distance
in 39.699353 seconds. These findings were obtained with
Oxford 102 Flowers Dataset.
G. Multi Layer Perceptron
Deep Learning Networks are feedforward neural networks
with multiple layers. It is also known as Multilayer Perceptrons
or MLPs. General feedforward deep learning networks do not
include feedback. In a multilayer perceptron, the neurons are
arranged into an input layer, an output layer and one or more
hidden layers. We used ReLU as activation function in the
study with other defaults of MLPClassifier library of sklearn. Fig. 6: Comparative results for Oxford-17 Flowers Dataset
We obtained the second highest accuracy of 96,5% in 101,58
seconds at Oxford-102 Flowers Dataset.
Similiarly, we obtained the most efficient result for Oxford-
V. C OMPARATIVE R ESULTS 102 Flowers Dataset with SVM classifier as 98.5% in 114,99
For the purpose of focusing on the improvements gained seconds. The second highest accuracy was gained with Multi-
with data augmentation, the study proposes the classification Layer Perceptron Classifier as 96,5% in 101,58 seconds. The
system which can be seen in Fig. 2. In this study, we used two following highest accuracy was obtained from KNN classifier
datasets containing flower images with 17 and 102 classes. We by using manhattan distance as 96,2% in 39,58 seconds.
conducted the all experiments on a computer with a NVIDIA Finally, the lowest accuracy was obtained from the Random
GEFORCE GTX 960M display card. Forest classifier as 87,6% in 150,06 seconds. The findings were
From the experiments, we observed the most efficient find- demonstrated in Fig.7
ing for Oxford 17-Flowers Dataset with Multi-Layer Percep- In figure 8, we demonstrated the comparative results for
tron Classifier (MLPC) as 99,8% in 21,89 seconds. The second Oxford-17 Flowers and Oxford-102 Flowers datasets. As seen
highest accuracy was gained with SVM Classifier as 99.6% in in the figure the highest accuracy rates belong to the classifiers
12,01 second. The following highest accuracy was obtained named SVM and Multilayer Perceptron.

Authorized licensed use limited to: Rajamangala Univ of Technology Isan provided by UniNet. Downloaded on June 09,2023 at 06:11:44 UTC from IEEE Xplore. Restrictions apply.
 Fig. 7: Comparative results for Oxford-102 Flowers Dataset
Fig. 8: Accuracy rates obtained by different classifiers for
Oxford 17 and 102-Flowers datasets
ACKNOWLEDGEMENT
This study derives from the author’s master thesis. It is
supported by Technology and Project Support Unit of Istanbul the consistency of our results in detail. So we are to expand
Kultur University. our findings.

VI. C ONCLUSION R EFERENCES

[1] M.-E. Nilsback and A. Zisserman, “A visual vocabulary for flower
In visual recognition area, fine-grained recognition is known classification,” in 2006 IEEE Computer Society Conference on Computer
as a kind of a challenge that has been getting more popular Vision and Pattern Recognition (CVPR’06), vol. 2. IEEE, 2006, pp.
especially in recent years. In this study, that kind of a challenge 1447–1454.
[2] Z. Nilsback, Maria-Elena and Andrew, “Automated flower classification
-which is required developing algorithms in order to selecting over a large number of classes,” in 2008 Sixth Indian Conference on
classes with quite small subtle visual distinctions- has been Computer Vision, Graphics & Image Processing. IEEE, 2008, pp.
carried out. For this, two benchmark datasets including differ- 722–729.
[3] A. Angelova and S. Zhu, “Efficient object detection and segmentation
ent types of flowers mostly common in England were used: for fine-grained recognition,” in Proceedings of the IEEE Conference on
Oxford 17-Flowers [1] and Oxford-102 Flowers [2]. Computer Vision and Pattern Recognition, 2013, pp. 811–818.
In this study, with the proposed classification system and [4] X. Xia, C. Xu, and B. Nan, “Inception-v3 for flower classification,” in
2017 2nd International Conference on Image, Vision and Computing
two flower datasets containing many classes (17 and 102), (ICIVC). IEEE, 2017, pp. 783–787.
we obtained the highest result in the literature. Thus, with [5] M. Lin, Q. Chen, and S. Yan, “Network in network,” arXiv preprint
the pre-trained model, we observed the effect of both feature arXiv:1312.4400, 2013.
[6] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
extraction and data augmentation on accuracy. In addition, we V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”
used t-sne [9] to visualize the images and how the features in Proceedings of the IEEE conference on computer vision and pattern
extracted from the images in the dataset are distributed. In recognition, 2015, pp. 1–9.
[7] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking
addition, the classification system that we propose in this study the inception architecture for computer vision,” in Proceedings of the
can be used for different problems in image classification. IEEE conference on computer vision and pattern recognition, 2016, pp.
In this respect, the study contributes not only to the flower 2818–2826.
[8] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4,
classification area but also to other areas where labeled data inception-resnet and the impact of residual connections on learning,” in
are scarce. Thirty-First AAAI Conference on Artificial Intelligence, 2017.
As seen in the figure 8, it can be said that, the performance [9] G. Hinton and Y. Bengio, “Visualizing data using t-sne,” 2017.
[10] J. Kremer, K. Steenstrup Pedersen, and C. Igel, “Active learning with
rates decrease in all classifiers when the dataset grows. In this support vector machines,” Wiley Interdisciplinary Reviews: Data Mining
context, we found that the performance rate decrease in SVM and Knowledge Discovery, vol. 4, no. 4, pp. 313–326, 2014.
is less than the decrease in the Multilayer Perceptron Classifier.
In all algorithms, it is seen that time consumption also
increases as the data set grows. When the two algorithms
with the highest performances are compared in terms of time
consumption. At this point, we found that the performance
decrease in the Multilayer Perceptron Classifier is less than
that of SVM.
VII. F UTURE W ORK
In the future, we will apply our approach in this study to
a dataset containing different categories of images, discussing

Authorized licensed use limited to: Rajamangala Univ of Technology Isan provided by UniNet. Downloaded on June 09,2023 at 06:11:44 UTC from IEEE Xplore. Restrictions apply.