Nonparametric Discriminant Analysis in Relevance Feedback

for Content-based Image Retrieval

Dacheng Tao and Xiaoou Tang

Department of Information Engineering
The Chinese University of Hong Kong
{dctao2, xtang}@ie.cuhk.edu.hk

Abstract method [3] loses information contained in negative

samples. Classification based method in [4] treats the
Relevance feedback (RF) has been wildely used to positive and negative samples equally.
improve the performance of content-based image Recently, biased discriminant analysis (BDA) [5,6,7]
retrieval (CBIR). How to select a subset of features from has been used as a feature selection method to improve
a large-scale feature pool and to construct a suitable RF, because BDA models the RF better than many other
dissimilarity measure are key steps in RF. Biased methods. However, BDA assumes all positive samples
discriminant analysis (BDA) has been proposed to select form a single Gaussian distribution, which means all
features during relevance feedback iterations. However, positive samples should be similar with similar view
BDA assumes all positive feedbacks form a single angle, similar illumination, etc. Clearly, this is not the
Gaussian distribution which may not be the case for case for CBIR. The kernel-based learning is used in BDA
CBIR. Although kernel BDA can overcome the drawback to overcome the problem. However, kernel-based
to some extent, the kernel parameter tuning makes the learning has to rely on parameter tuning, which makes the
online learning unfeasible. To avoid the parameter tuning online learning unfeasible.
problem and the single Gaussian distribution assumption To avoid the parameter tuning problem and the single
in BDA, we construct a new nonparametric discriminant Gaussian distribution assumption in BDA, we develop a
analysis (NDA). To address the small sample size new discriminant analysis using a nonparametric
problem in NDA, we introduce the regularization method approach. The proposed nonparametric discriminant
and the null-space method. Because the regularization analysis (NDA) has the following properties: 1. NDA
method may meet the ill-posed problem and the null- assumes all positive samples are alike and each negative
space method will lose some discriminant information, we sample is negative in its own way; 2. NDA does not
proposed here a full-space method. The proposed full- require all positive samples form a single Gaussian
space NDA is demonstrated to outperform BDA based RF distribution. 3. NDA, similar to BDA and KBDA, may
significantly based on a large number of experiments in meet the Small-Sample-Size (SSS) problem. In this paper,
Corel database with 17, 800 images. we will solve the SSS problem with three methods: 1. the
regularization method, which is used by Zhou in BDA [5];
1. Introduction 2. the null-space method [8], which is a popular method
to solve the SSS problem in linear discriminant analysis
Relevance feedback (RF) [1] is an important tool to for face recognition; 3. the full-space method, which is
improve the performance of content-based image retrieval proposed to preserve all discriminant information of NDA.
(CBIR) [2]. In a RF process, the user first labels a number
of relevant retrieval results as positive feedbacks and 2. Nonparametric discriminant analysis
some irrelevant retrieval results as negative feedbacks.
Then the system refines all retrieval results based on these To better understand the proposed NDA based RF
feedbacks. The two steps are carried out iteratively to schemes, we first give a brief review of BDA.
improve the performance of image retrieval system by
gradually learning the user’s perception. 2.1. BDA
Many RF methods have been developed in recent
years. One approach [1] adjusts the weights of various BDA [5] tries to find the subspace to discriminate the
features to adapt to the user’s perception. Another positive (the only class concerned by the user) and
approach [3] estimates the density of the positive negative samples (unknown number of classes). It is
feedback examples. Support vector machine (SVM) has spanned by a set of vectors W maximizing the ratio
also been used as a classification method for RF [4]. between the positive covariance matrix S x and the biased
These methods all have their own limitations. The method matrix S y ,
in [1] is only heuristic based. The density estimation

Proceedings of the 17th International Conference on Pattern Recognition (ICPR’04)

1051-4651/04 $ 20.00 IEEE
 W TS y W 3. Small sample size problem in NDA
W arg max T
(1)
W W SxW .
In RF, the size of the training set is much smaller than
Let the training set contains Nx positive and Ny the dimension of the feature vector, thus it may cause the
negative samples. Then S x and S y are defined as, SSS problem. In this Section, we will address the SSS
­S Nx problem using three methods, the regularization method,
¦ x i  m x x i  m x 
T

°° x i 1 the null-space method, and the new full-space method.

® Ny
(2)
°S y ¦ y i  m x y i  m x  ,
T
°¯ i 1 3.1. Regularization method
where x i denote the positive samples, y i denote the
1 Nx Regularization method, which is proposed by
negative samples, m x ¦ x i is the mean vector of the Friedman to deal with the singularity issue, is
Nx i 1
implemented by adding small quantities to the diagonal of
positive samples, and W can be computed from the  
the scatter matrices S x and S y . The regularized version
eigenvectors of S x1S y . Firstly, BDA minimize the  
of S x and S y , with the dimension of the original feature
variance of the positive samples. Then BDA maximize
the distance between the center of the positive feedbacks space n and the identity matrix I, are:
  
and all negative feedbacks. S rx 1  P S x  P tr >S x @I (5)
n
  
2.2. NDA S ry 1  J S x  J tr >S y @I (6)
n
Similar to BDA, NDA is also biased toward to the where P and J control the shrinkage toward a multiple
positive examples. The objective function of NDA is:

of the identity matrix. tr >@. is the trace operation.
W TS y W It is well known that regularization method may meet
Wopt arg max  (3) the ill-posed problem. Hence, we select the null-space to
W W TS x W .
overcome the ill-posed issue.
Let the training set contains Nx positive and Ny
 
negative samples. Then S x and S y are defined as, 3.2. Null-space method
­S Nx
 kx kx
 
T
¦ x i  m xi x i  m xi Null-space linear discriminant analysis (LDA) [8]
°° x i 1
® Ny Nx
(4) accepts high-dimensional data as the input, and optimizes
°S y
°¯ i 1

¦ yi  m
kx
yi y i m kx T
yi  
 ¦ xi  m
i 1
ky
xi x i m xi,
ky T
LDA in the null space of within class scatter matrix. Here,
we generalize the null-space idea for NDA. The null
where x i are positive samples, y i are negative samples, 
space of S x is first calculated as:
1 k 
m kxxi ¦ x l is the mean vector of the k positive nearest YT S x Y 0 (7)
kl1
neighbors of the ith positive feedback sample x i , where Y are eigenvectors with zero eigenvalues and
 
1 k Y T Y I . S y is projected onto the null space of S x :
m kxyi ¦ y l is the mean vector of the k negative nearest  
kl1 S ny YT S y Y . (8)
neighbors of the ith positive feedback sample x i , 
The eigenvectors U of S ny with largest eigenvalues are
1 k
m kyxi ¦ x l is the mean vector of the k positive nearest selected to form the transformation matrix as:
kl1
W YU . (9)
neighbors of the ith negative feedback sample y i , and
 
Wopt can be computed from the eigenvectors of S x1S y . 3.3. Full-space method
NDA finds the optimal feature set to maximize the margin
between all positive feedbacks and all negative feedbacks Null-space method loses the information in the
in the input feature space. Because the original feature principle space of the within class scatter matrix. In order
dimension is much larger than the number of the feedback to preserve all discriminant information, we compute
samples, we can always find the subset feature to features from both the null space and the principle space

discriminant the positive and negative samples. of S x , and then integrate the two parts with a suitable
weighting. A rational choice of the weighting is to select

Proceedings of the 17th International Conference on Pattern Recognition (ICPR’04)

1051-4651/04 $ 20.00 IEEE
 a small eigenvalue of S w . The algorithm first computes algorithm. In precision and SD curves, the total feedback
 times are 9, with 0 feedback referring to the retrieval
the eigenvalues of S x as,
 based on Euclidean distance measure without RF.
YT S x Y
Dx , (10)
where D x diag O1 ,..., Oi ,..., O m , O m 1 ,...,0  , O m 1 HO1 , and H 4.1. K nearest neighbor evaluation
is a user selected threshold value (such as 0.01).
For a given H , the eigenvalue matrix D x is replaced The experiment shows NDA is insensitive to the k
 value of the k-nearest-neighbor. Figure 1, 2, and 3 show
by D x diag O1 ,..., Oi ,..., O m , O m 1 ,..., O m 1  . All values, which
the top 30 retrieved results with 3, 6, and 9 feedback
are smaller than O m 1 , are substituted by O m 1 . After the iterations by the regularization method, null-space

substitution, S y is projected onto the space by: method, and full-space method, respectively. Because all
    curves are flat, we can draw the conclusion that NDA is
Sy D x1 / 2 Y T S y YD x 1 / 2 . (11) insensitive to the k value in k nearest neighbor.

Finally, the eigenvectors U of S y with largest 0.8 Retrieval precision in top 30 retrieved images 0.4
Retrieval standard deviation in top 30 retrieved images

eigenvalues are selected to form the transformation matrix, 0.7 0.35

 0.6 0.3
W YD x 1 / 2 UD y1 / 2 . (12)

Standard deviation
0.5 0.25

Precision
0.4 0.2

0.3 0.15

4. Experimental results 0.2

3rd feedback iteration
0.1
3rd feedback iteration
0.1 6th feedback iteration 0.05 6th feedback iteration
9th feedback iteration 9th feedback iteration
0 0
In our experiments, three main features, color, texture, 3 4 5 6 7
K Nearest Neighbor
8 9 3 4 5 6 7
K Nearest Neighbor
8 9

and shape are extracted and used to represent the Figure 1. Regularization method
corresponding image. For color feature, we use the color 0.8
Retrieval precision in top 30 retrieved images
0.4
Retrieval standard deviation in top 30 retrieved images

histogram [9] in HSV color space. Here, the color 0.7 0.35

0.6 0.3
histogram is quantized into 256 levels. Hue, Saturation,

Standard deviation
0.5 0.25
Precision

and Value are quantized into 8, 8, and 4 bins respectively. 0.4 0.2

Texture is extracted from Y component in YCrCb space 0.3 0.15

0.2 0.1
by pyramidal wavelet transform (PWT) with Haar 0.1
3rd feedback iteration
6th feedback iteration 0.05
3rd feedback iteration
6th feedback iteration

wavelet. The mean value and standard deviation are 0

3 4 5 6
9th feedback iteration
7 8 9
0
3 4 5 6
9th feedback iteration
7 8 9
K Nearest Neighbor K Nearest Neighbor
calculated for each sub-band at each decomposition level.
The feature length is 2 u 4 u 3 . For shape feature, edge Figure 2. Null-space method
Retrieval precision in top 30 retrieved images
0.8 0.4 Retrieval standard deviation in top 30 retrieved images
histogram [10] is calculated on Y component in YCrCb 0.7 0.35

color space. Edges are grouped into four categories, 0.6 0.3
Standard deviation

horizontal, 45 diagonal, vertical, and 135 diagonal. We 0.5 0.25

Precision

0.4 0.2
combine the color, texture, and shape features into a 0.3 0.15

feature vector, and then we normalize each feature to a 0.2

3rd feedback iteration
0.1
3rd feedback iteration
normal distribution. 0.1 6th feedback iteration
9th feedback iteration
0.05 6th feedback iteration
9th feedback iteration
0 0
In this part, a large number of statistical experiments 3 4 5 6 7
K Nearest Neighbor
8 9 3 4 5 6 7
K Nearest Neighbor
8 9

are performed based on a subset of the Corel Photo Figure 3. Full-space method
Galley, which includes 17, 800 images with 90 concepts
(relabeled by ourselves). The experiments are simulated 4.2. Small samples size problem
by the computer automatically. First, 300 queries are
randomly selected from the data, and then RF is done by Fig. 4 shows the performance of the full-space method,
computer as: top 5 query relevant and irrelevant images the null-space method, and the regularization method in
are marked as positive and negative feedbacks in the top NDA to solve the SSS problem. From the left subfigure in
48 images, respectively. Fig. 4, we can see the precision curve of full-space
In this paper, precision and standard deviation (SD) are method is higher than that of null-space method and
used to evaluate the performance of a RF algorithm. regularization method, meanwhile the SD curve of full-
Precision is the ratio of the number of relevant images space method is lower than that of null-space method and
retrieved to the top N retrieved images. Precision curve is regularization method. Hence we can draw the conclusion
the averaged precision values of the 300 queries, and SD the new full-space method can work better than the
curve is the SD values of 300 queries’ precision. The existing null-space method and regularization method.
precision curve evaluates the effectiveness of a given Meanwhile, the null-space method can outperform the
algorithm and SD curve evaluates the robustness of the regularization method.

Proceedings of the 17th International Conference on Pattern Recognition (ICPR’04)

1051-4651/04 $ 20.00 IEEE
 Retrieval precision in top 20 retrieved images Retrieval standard deviation in top 20 retrieved images
1

0.9
0.34

0.32
6. Acknowledgement
0.3
0.8

Standard deviation
0.28
0.7 The work described in this paper was fully supported
Precision

0.26
0.6

0.5
0.24 by a grant from the Research Grants Council of the Hong
0.22
0.4
0.2
Kong SAR. (Project no. AoE/E-01/99).
Regularization method Regularization method
0.3 Null-space method 0.18 Null-space method
Full-space method Full-space method
0.2 0.16
0 1 2 3 4 5 6
Number of feedback iterations
7 8 9 0 1 2 3 4 5 6
Number of feedback iterations
7 8 9
7. References
Figure 4. Comparing the full-space method with the null-space
method and regularization method. [1] Y. Rui, T. Huang, and S. Mehrotra, “Content-based image retrieval
with relevance feedback in MARS,” In Proc. IEEE ICIP, 1997.
4.3. Evaluation experiments [2] J. Wang, J. Li, and G. Wiederhold, “SIMPLIcity: semantics-sensitive
integrated matching for picture libraries,” IEEE Trans. on PAMI, vol. 23
no. 9, pp. 947-963, 2001.
We will compare the new full-space NDA with the [3] Y. Chen, X. Zhou, and T. S. Huang, “One-class SVM for learning in
existing state-of-the-art algorithms, which are BDA [5], image retrieval,” In Proc. IEEE ICIP, 2001.
SVM [4], and constrained SVM (CSM) [11]. Results in [4] L. Zhang, F. Lin, and B. Zhang, “Support vector machine learning
for image retrieval,” In Proc. IEEE ICIP, 2001.
Fig. 5 shows that the full-space NDA by 3-nearest- [5] X. Zhou and T. S. Huang, “Small sample learning during multimedia
neighbor can significantly improve the CBIR RF retrieval using biasmap,” In Proc. IEEE CVPR, 2001.
compared with all the other algorithms [4,5,11]. [6] D. Tao and X. Tang, “Kernel full-space biased discriminant
analysis,” In Proc. IEEE ICME, 2004.
[7] D. Tao and X. Tang, “A direct method to solve the biased
5. Conclusion discriminant analysis in kernel feature space for content based image
retrieval,” In Proc. IEEE ICASSP, 2004.
In this paper, we propose a new nonparametric [8] L. F. Chen, H.Y. Liao, M. T. Ko, J. C. Lin, and G. J. Yu, “A new
LDA-based face recognition system which can solve the small sample
discriminant analysis (NDA) for relevance feedback (RF) size problem,” IJPR, vol 33, pp. 1713-1726, 2000.
in content-based image retrieval. To address the small [9] M. Swain and D.H. Ballard, “Color indexing,” IJCV,7. 11-32, 1991.
sample size problem in NDA, we proposed a full-space [10] B.S Manjunath, J. Ohm, V. Vasudevan, and A. Yamada, “Color and
method. Based on a large number of experiments with 17, texture descriptors,” IEEE Trans. on CSVT, Vol. 11, 2001.
[11] G. Guo, A. K. Jain, W. Ma, and H. Zhang, “Learning similarity
800 images, we can draw the conclusion that the new full- measure for natural image retrieval with relevance feedback,” IEEE
space NDA based RF can work much better than the- Trans. on NN, vol. 12, no. 4, pp.811-820, July 2002.
state-of-the-art methods.
Retrieval precision in top 20 retrieved images Retrieval precision in top 40 retrieved images Retrieval precision in top 60 retrieved images
1 0.8 0.65
NDA
0.9 0.6 BDA
0.7
0.55 SVM
0.8 CSM
0.6 0.5
0.7
Precision

0.45
Precision

Precision

0.5
0.6 0.4
0.4
0.5 0.35

0.3 0.3
0.4 NDA NDA
BDA BDA 0.25
0.3 SVM 0.2 SVM 0.2
CSM CSM
0.2 0.1 0.15
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
Number of feedback iterations Number of feedback iterations Number of feedback iterations

(a) (b) (c)

Retrieval standard deviation in top 20 retrieved images Retrieval standard deviation in top 40 retrieved images Retrieval standard deviation in top 60 retrieved images
0.4 0.35 0.3

0.28
0.35 0.3 0.26
Standard deviation

Standard deviation
Standard deviation

0.24
0.3 0.25
0.22

0.2
0.25 0.2
0.18
NDA NDA 0.16 NDA
0.2 BDA 0.15 BDA BDA
SVM SVM 0.14 SVM
CSM CSM CSM
0.15 0.1 0.12
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
Number of feedback iterations Number of feedback iterations Number of feedback iterations

(d) (e) (f)

Figure 5. Evaluation experimental results based on the Corel database with 17, 800 images with 300 queries. (a), (b), and (c) display the retrieval
precision in top 20, 40, and 60 retrieved images respectively. (d), (e), and (f) display the corresponding standard deviation of the precision curve in top
20, 40, and 60 retrieved images respectively.

Proceedings of the 17th International Conference on Pattern Recognition (ICPR’04)

1051-4651/04 $ 20.00 IEEE