2018 IEEE 3rd International Conference on Signal and Image Processing

Convolution Neural Network based Transfer Learning for Classification of Flowers

*
Yong Wu,Xiao Qin , Yonghua Pan, Changan Yuan
College of Computer and Information Engineering，Guangxi Teachers Education University
Nanning, China
e-mail: {601702620, 7670172, 283889077,68852917}@qq.com

Abstract—Flower plays an extremely important role in our life, support vector machine (SVM) as the classifier, the
which has high research value and application value. The recognition rate of flowers in the Oxford-102 dataset can
traditional methods of flower classification is mainly based on reached 88.33%[4]; Angelova and Zhu[5] proposed a
shape, color or texture features, and this methods needs people segmentation approach, followed by the extraction of
to select features for flower classification lead to the accuracy Histogram of Oriented Gradient (HOG) features and using
of classification is not very high. This paper aims to develop an Locality-constrained Linear Coding (LLC). In 2014, Xie
effective flower classification approach using convolution used the fine-grained flower classification to improve the
neural network and transfer learning. In this paper, based on accuracy of flowers, the recognition accuracy was 93.14%
VGG-16, VGG-19, Inception-v3 and ResNet50 models were
on Oxford-17 flower dataset and the accuracy was 79.1% on
used to compare the network initialization model with the
transfer learning model. The results show that transfer
Oxford-102 flower dataset[6]. Wei Liu et al.[7] proposed an
learning can effectively avoid deep convolution networks are effective classification method using fusion descriptor and
prone to local optimal problems and over-fitting problems. support vector machines (SVM), by conducting an
Compared with the traditional methods, the accuracy of flower experiment study in Oxford 17 dataset, the method provided
recognition on Oxford flowers dataset is obviously improved, a satisfactory result with 86.17% accuracy rate. Liu et al.[8]
and has better robustness and generalization ability. propose a novel framework based on convolution neural
network (CNN) to improve the accuracy of flowers ,unlike
Keywords-deep learning; convolution neural network; other methods using hand-crafted visual features, the method
transfer learning; flowers classification utilizes convolution neural network to automatically learn
good features for flower classification, can achieves 84.02%
I. INTRODUCTION classification accuracy in Oxford-102 flowers dataset.
However, the feature extraction of artificially selected
Flowers can be seen everywhere in people's daily life, features may not be able to characterize the target, resulting
and flowers have great cultural value, economic value and in poor recognition performance, the recognition of flowers
ecological value in our life. Although there are significant by deep convolution neural networks is not easy and the
differences in shape, structure and habits between flowers, network is easy to fall into the local optimum and over fitting
there is a great deal of trouble in understanding and leads to insufficient recognition accuracy [9].
identifying flowers. Therefore, it is necessary to use a flower Recent research shows that the convolution neural
identification method to identify flowers quickly and network has great advantages in feature extraction and has
correctly. With the rapid development of science and certain degree of invariance to the operation. Current neural
technology and the popularity of smart phones, people tend network models have computational requirements and high
to use more and more vivid and easy-to-understand pictures computing resources, and deep convolution neural network
instead of cumbersome words. However, the recognition rate models are prone to over-fitting or fall into local
of existing flower recognition experiments is relatively low, optimization problems, making transfer learning [10] to be
and the better methods are needed to support the more the ideal choice. In order to improve the overall recognition
perfect identification of flowers. performance, this paper joined the transfer learning method,
The classification of flower species has the following comparing the parameter initialization model with the
problems: Firstly, the traditional feature extraction mainly transfer learning model and compared with other traditional
use the characteristics of color, shape and texture. The experimental methods. The experimental results show that
artificial selection of these features is more complicated and this method can obviously improve the accuracy of flower
difficult. Secondly, the same flower at different times and in recognition, also can get better robustness and generalization
various environments showing different gestures, and the ability.
color and shape of flower species are very similar, which
brings great difficulty to identification. In recent years, the II. RELATED WORK
important research progresses on flowers identification
The convolution neural network [11] [12]is one of the
include: In 2000 Saitoh et al. [1] use the method requires the
most commonly used models in deep learning[13] and has
user to place a black cloth behind the flower to recognize
become a research hotspot in the field of computer vision
flowers and this method is not feasible and inconvenient.
such as image classification [14][15]. In recent years, the
Maria-Elena et al. [2] [3] used different characteristics of
CNN model has been widely applied in the field of image
flowers and multi-core frame combination features, using
processing, so the so-called image classification is to extract

978-1-5386-6396-7/18/$31.00 ©2018 IEEE 562

features from the known images, and to classify them into a cross layer and adds the convolution processing result, so
known category, or to determine whether the images contain that the output of the posterior layer is input and the
objects in known categories. superimposition of input mapping. The purpose is to
The convolution neural network is one of artificial neural optimize the residual function in the network.
network, which is composed of convolution calculation and
pool operation to extract and process feature. The traditional B. Transfer Learning Based on CNN Models
convolution neural network structure is shown in Figure For convolution neural networks, transfer learning [5]
1.The input layer is responsible for loading the image into refers to learning the training model on a specific data set
the neural network and the resulting vector output as a and using the parameters in the target dataset of the new
convolution layer composed of multiple feature maps. The domain. Transfer learning not only can learn colors on the
convolution layer functions to extract the target features by pre-training dataset , texture and other low-level features,
the convolution operation, the convolution layer extract low- but also to learn to help the classification of the target
level features, and then the higher-level convolution layer dataset advanced semantic features, so that the classification
extracts more abstract and complex semantic features. These performance of the model. The four models in this paper
features are passed to the pool layer, which can reduce pre-trained the model on an ImageNet dataset containing
network parameters to speed up operation and retain more than 1.2 million natural images and more than 1,000
effective data information. Finally, it is connected to one or different categories, and then transfer the model parameters
more fully connected layers. The main function of the fully
to Oxford flower [2][4] dataset for fine-tuning training. In
connected layer is to process the information in the
this experiment, we transfer the weights of network-specific
convolution layer or the pooling layer into distinguishable
class information. The last layer of the full connection layer layers in each model and fine-tune the model by training
is connected with the output layer behind, and the output of and learning the original data. The method of transfer
the neural network model is transformed into a probability learning based on CNN models is shown in figure 2.
distribution through the Softmax layer to obtain the
probability information of the target category. ImageNet Labels Flowers Labels
Transfer
The basic idea of transfer learning is to transfer learned
knowledge from existing models and data to goals to be Large
knowledge
Small
amount of ImageNet Models Flowers Models amount of
learned by making use of the correlation between learning Data/Labels Data/Labels
goals and existing knowledge. At present, transfer learning
ImageNet
has been widely used in many applications of machine Dataset
Flowers Dataset
learning.
Figure 2. Transfer learning based on CNN model

IV. EXPERIMENTAL RESULTS

All the model experiments are trained on Intel i7-7700
processor, 1T solid state drive, 32G memory, NVIDIA
GTX1080Ti GPU, using the Keras software framework of
Figure 1. Traditional convolution neural network structure
TensorFlow platform; stochastic gradient descent (SGD)
function as an optimization method, the learning rate is set to
III. METHODS 1e-4, the weight attenuation is set to 1e-6, and the
momentum factor is set to 0.9. The experimental datasets are
A. CNN Models the Oxford 17 flower dataset [2] and the Oxford 102 flower
The convolution neural network models used in this dataset[4]. In this section, the following part is as follows:
paper were VGG16[14], VGG19[14], Resnet50[15]and first, we make a simple introduction on the dataset; second,
Inception-v3[16]. The main idea of the VGG model of is to we introduce the strategy of the experiment in detail; then,
improve the overall performance of the network by we show the result of this experiment; finally, we prove the
increasing the network layer network depth, its strategy is to effectiveness of the method through the comparison
the larger convolution kernel layer convolution transform experiment.
into multiple volume small laminated convolution kernels, to A. Dataset
reduce the number of model parameters, and make the Oxford-17 flower dataset[2]: this dataset was created by
network more discriminative. GoogleNet[17] through the Maria and Andrew in Britain in 2006, which contains 17
basic module of Inception structure in cascade into depth and species of flowers, each of species has 80 flower images. In
width are changed, the main idea is to filter the Inception this dataset, the scale of the image is large, the posture and
module first use different size to handle the input matrix, so the lighting are also greatly changed, and there is similarity
as to extract the feature information of different sizes, the between the different species of flowers and differences
characteristics of information fusion splicing different and between the same species of flowers.
passed to the next layer. The residual network joined the Oxford-102 flower dataset[4]: this dataset was also
shortcut connection technology, which transmits the input created by Maria and Andrew in 2008, which contains 102

563
species of flowers, each of species has 40-258 flower images. TABLE I. COMPARISON OF EVALUATION INDEX ON DIFFERENT
METHODS OF OXFORD 102 DATASET
Compared with the Oxford 17 dataset, contain more species
of flowers and there are more similarities between different Method Accuracy of Loss of Loss of
types of flowers, so the flower classification will be more training set training set validation set
complex. The example of Oxford flower dataset shown in VGG16 68%-69% 1.50 1.70
VGG16-transfer 91%-92% 1.21 1.48
Figure 3.
VGG19 57%-58% 2.06 1.83
VGG19-transfer 92%-93% 0.72 1.20
Inception-v3 82%-83% 0.82 1.57
Inception-v3-transfer 99%-100% 0.02 0.14
ResNet50 93%-94% 0.21 1.91
ResNet50-transfer 99%-100% 0.008 0.13

(a) (b)

Figure3.The example of Oxford flower dataset (c) (d)

Figure 4. (a) (b) (c) (d) show the comparison of the accuracy of VGG16、
B. Experimental strategy VGG19、Inception-v3、ResNet50 initialization model and transfer model
The dataset in this paper is divided into three parts. The in the validation set of Oxford-102 flower dataset
training set is used for model training and learning as well as TABLE II. COMPARISON OF EVALUATION INDEX ON DIFFERENT
adjusting the parameters. The validation set is used to test the METHODS OF OXFORD-17 DATASET
model in the training, optimize the model and fine-tune the
Method Accuracy of Loss of Loss of
model parameters. Test sets are used to test the recognition training set training set validation set
and generalization of models. Oxford-102 dataset has 6149 VGG-16 73%-74% 0.67 1.91
pictures as training set, 1020 pictures as validation set, 1020 VGG-16-transfer 98%-99% 0.07 1.90
pictures as test set, and Oxford-17 dataset has 1020 pictures VGG-19 72%-73% 0.69 1.69
as training set, 170 pictures as validation set, 170 pictures as VGG-19-transfer 98%-99% 0.04 1.66
test set. To ensure that the trained model generalizes to Inception-v3 96%-97% 0.08 2.03
unknown data, there is no intersection between datasets. In Inception-v3-transfer 99%-100% 0.01 0.29
ResNet50 84%-85% 0.27 1.85
order to verify the effect of transfer learning, two methods of
ResNet50-transfer 99%-100% 0.002 0.24
parameter initialization training and transfer fine tuning
training are adopted in this paper.
C. The Result of Experiment
TableⅠ and TableⅡshow the comparison of evaluation
index on different methods of Oxford 102 dataset and
Oxford 17 dataset respectively. And Figure 4 and Figure 5
show the comparison of the accuracy of VGG-16、VGG-
19 、 Inception-v3 、 ResNet50 initialization model and
transfer model in the validation set of Oxford 102 flower
dataset and Oxford 17 flower dataset respectively.
(a) (b)

564
 TABLE IV. CLASSIFICATION PERFORMANCE COMPARISON
OF DIFFERENT METHODS ON OXFORD-17 FLOWER DATASET
Method Performance
[5] 85.5%
[7] 86.17%
[6] 93%
VGG-16-transfer 83.53%
VGG-19-transfer 84.71%
Inception-v3-transfer 94.58%
ResNet50-transfer 95.29%

(c) (d)
Figure 5. (a) (b) (c) (d) show the comparison of the accuracy of VGG-16、
V. CONCLUSION
VGG-19、Inception-v3、ResNet50 initialization model and transfer model The feature extraction method of deep convolution neural
in the validation set of Oxford-17 flower dataset network model used in this paper is more convenient and
practical than the manual extraction feature method. At the
From Table Ⅰ and Table Ⅱ, we can conclude that same time, the neural network model added with the transfer
whether the larger dataset Oxford-102 or the smaller dataset learning method converges fast, and both robustness and
Oxford-17, the training level of the network with the transfer generalization ability are stronger than random initialization
learning neural network model is obviously better than the model. Contrast with other traditional experimental methods
random initialization network model. It can be seen that the to identify better. Experiments show that this method
convergence rate of the Inception_v3 transfer model and the improves the accuracy of flower recognition and has a
ResNet50 transfer model in the four types of transfer significant improvement.
learning model is faster and the network loss is smaller than
the VGG-16 transfer model and the VGG-19 transfer model. ACKNOWLEDGMENT
Combined with Figure 4 and Figure 5, it can be concluded This work is partially supported by the National Natural
that the model with transfer learning neural network has Science Foundation of Guangxi under Grant Nos.
better robustness and stronger generalization ability than the 2016GXNSFAA380209,2014GXNSFDA118037;the
initialization network model, and the random initialization “BAGUI Scholar” Program of Guangxi Zhuang
network model is more likely to fall into the local optimal Autonomous Region of China; the project of scientific
problem and appears In combination, the robustness and research and technology development (AB16380272 ) in
generalization ability are relatively poor. The initial network Guangxi in 2016;the project of scientific research and
model is affected by its own deep structure and large training technology development (#20175177) in Guangxi Nanning
parameters. The network training speed is slow, and local 2017;Innovation Project of Guangxi Graduate
optimum and over fitting situations are easy to occur. The Education(YCSW2017187).
specific network layer weight parameters of the transfer
model are frozen, and the semantic level parameters of the REFERENCES
model are fully trained and adjusted, and the extracted target [1] Saitoh, T.; Kaneko, T., “Automatic recognition of wild Flowers”,
features are more distinguishable. Pattern Recognition, Proceedings. 15th International Conference on,
vol.2, no.,pp.507-510 vol.2, 2000.
[2] Maria-Elena Nilsback, Andrew Zisserman: A Visual Vocabulary for
D. Control Experiment Flower Classification. CVPR (2) 2006.pp.1447-1454,2006.
In order to prove the effectiveness of our method, [3] Maria-Elena Nilsback, Andrew Zisserman: Delving into the Whorl of
experiments are carried out to compare the flower Flower Segmentation. BMVC 2007.pp. 1-10,2007.
classification method based on [4][6][8] on Oxford 102 [4] MarMaria-Elena Nilsback, Andrew Zisserman: Automated Flower
flower dataset[4],and compare the flower classification Classification over a Large Number of Classes. ICVGlP ,pp.722-
method based on [5][6][7] on Oxford 17 flower 729,2008.
dataset[2].The results are shown in Table Ⅲ and Table Ⅳ. [5] Angelova A, Zhu S. Efficient object detection and segmentation for
fine-grained recognition. In Computer Vision and Pattern
We can see that the classification accuracy of Inception-v3- Recognition(CVPR), 2013 IEEE Conference on, pp.811-818, 2013.
transfer and ResNet50-transfer is higher than other method.
[6] Xiaodong Xie, Research on Fine-Grained Classification for Visual
TABLE III. CLASSIFICATION PERFORMANCE COMPARISON Flower Image. Xiamen University, 2014.
OF DIFFERENT METHODS ON OXFORD-102 FLOWER DATASET [7] Liu, W., Rao, Y., Fan, B., Song, J., & Wang, Q. Flower classification
using fusion descriptor and SVM. International Smart Cities
Method Performance Conference.pp.1-4. 2017
[6] 79.1%
[8] 84.02% [8] Liu, Y., Tang, F., Zhou, D., Meng, Y., & Dong, W. “Flower
classification via convolutionalal neural network.”,International
[4] 88.33%
Conference on Functional-Structural Plant Growth Modeling,
VGG-16-transfer 83.82% Simulation, Visualization and Applications IEEE,pp.110-116. 2017
VGG-19-transfer 85.25%
Inception-v3-transfer 95.88%
ResNet50-transfer 96.57%

565
[9] SHEN Ping, ZHAO Bei. “Automatic Classification of Flowers Based [14] Simonyan K, Zisserman A. “Very Deep Convolutional Networks for
on Deep Learning Model”, Bulletin of Science and Technology, Large-Scale Image Recognition”. Computer Science, 2014.
33(3):pp.115-119, 2017 [15] He K., Zhang X., Ren S., Sun J. “Deep residual learning for image
[10] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE recognition”. Proceedings of the IEEE Conference on Computer
Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. Vision and Pattern Recognition, Las Vegas, USA, pp. 770-778,2016.
1345–1359, 2010. [16] Szegedy C, Vanhoucke V, Ioffe S. Rethinking the Inception
[11] Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. Gradient-based Architecture for Computer Vision, pp.2818-2826. 2015.
learning applied to document recognition. Proceedings of the IEEE, [17] Szegedy C, Liu W, Jia Y. Going deeper with convolutions[C]//
86(11):pp.2278-2324,1998. Computer Vision and Pattern Recognition. IEEE, pp.1-9, 2015
[12] Lecun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., &
Hubbard, W. “Backpropagation applied to handwritten zip code
recognition”, Neural Comput., vol. 1, no. 4, pp. 541–551, 1989.
[13] Geoffrey Hinton,Salakhutdinov RR. Reducing the dimensionality of
data with neural networks. Science, 313(5786):pp.504-507, 2006.

566