Embedding Fuzzy Logic in Content Based Image Retrieval

Constantin Vertan, Nozha Boujemaa

INRIA Rocquancourt – Projet IMEDIA
Domaine de Voluceau BP105 Rocquancourt, 78153 Le Chesnay Cedex, France
Constantin.Vertan, [email protected]

Abstract of fuzzy theory used for their definition. Four classes

of fuzzy processing algorithms were identified in [15]:
This paper focuses on the possible embedding of the un- crude fuzzy, fuzzy paradigm-based, fuzzy aggregational
certainty regarding the colors of an image into histogram- and fuzzy inferential. The crude fuzzy techniques (as sug-


type descriptors. The uncertainty naturally arises from gested by their name) are characterized by the use of num-
both the quantization of the color components and the bers in the   interval (typically being weights attached
human perception of colors. Fuzzy histograms measure to the objects of the problem universe), that are compat-
the typicality of each color within the image. We define ible with a semantical description. The fuzzy paradigm-
various fuzzy color histograms following a taxonomy that based techniques use (implicitly or not) a fuzzy model of
classifies fuzzy techniques as crude fuzzy, fuzzy paradigm the objects of the problem universe. The fuzzy aggrega-
based, fuzzy aggregational and fuzzy inferential. For tional techniques use a fuzzy model for the combination
these fuzzy sets we must develop appropriate similarity (aggregation) of the individual objects in order to derive
measures and distances. We propose a class of such dis- the outcome. Finally, the fuzzy inferential techniques are
tances, derived from the fuzzy set equality and which we based on the inference mechanism that is applied with re-
particularize according to various T-norms
(fuzzy logical spect to a set of fuzzy rules that describe both the object
”or” operators). We also prove that the metric nat- model and the object aggregation. This is the most com-
urally arises as a distance for fuzzy sets, considering the plex model, that can be viewed as an instance of a rule-
fuzzy set symmetric difference. based expert system.
In this contribution we will revisit the use of color his-
1 Introduction tograms from a fuzzy-logic perspective: the value of each
bin must represent the typicality of the color within the
Content-based image retrieval (CBIR) became a must in image rather than its probability. Thus, in section 2, we
the last decade. Powered by the explosive development will define new type of histograms as membership func-
of the Internet and the Web and the continuously cheaper tions of the colors within the image (and thus as fuzzy
digital imagining devices and technologies, applications sets) and some new resemblance measures and distances
such as digital libraries, image archives, video-on-demand for fuzzy sets will be introduced in section 3. Finally, sec-
and specific image databases emerge as a real-life fact. tion 4 will present some experiments and conclusions.
The basic idea of the CBIR process is to compactly de-
scribe an image by a digital signature and then match
query images to the most resemblant image within the
database according to the similarity of their signatures.
Traditionally, the content description is done (for either
2 Fuzzy image histograms
global or partial queries) according to the notions of color
and texture. Thus the signatures are color distributions As already mentioned in the introduction, the histogram
(histograms [13], color moments [9], color coherence vec- (probability density function) of the values within an im-
tors [10]), second-order, spatially constrained color distri- age (either color or gray-scale) is largely used for the
butions (color correlograms [4], edge correlograms [3]) content-based image retrieval. The retrieval means that
or classical textural descriptors (Fourier or wavelet coeffi- the images within the databases are selected according to
cients, Markov random field models, etc.). the resemblance of their histogram to the histogram of the
The search for new, fuzzy color distributions will fol- query image. Given a color image , of size
by 
low the taxonomy proposed in [15] for the classification pixels, characterized by the color  at location  , i.e.
of fuzzy processing techniques according to the amount    , the color distribution (histogram) of the color
set  is given by


set. That means that we will associate to any color a

#
! %
" $ ,
+ *
" $ Lukasiewicz function, M ?ON 9PQ R and for any color
  &' &




 
 
 1
 2
0 
 *
 4
 5
3 7
 8
6   D of the color universe, M ?  D  is the resemblance degree

 (*) - (*)/. of the color  D to the color  . Further assuming a fuzzy
(1)


model that is not machiavelic (as discussed in [1]), we

The  values are normalized in order to sum to (as re- must logically admit a relation between the color resem-
quired by the definition of a probability density function) blance degree and the distance that separates the colors 
and  is the Dirac impulse function. The value of each and  D , and, more particularly, that the resemblance degree
bin is. thus the number of image pixels having the color  , decreases as the inter-color distance increases.
or, after normalization, is the probability that the color  The natural choice (according to the image processing
appears in the image. traditions) is to impose a smooth decay of the resemblance
TSU color space was supposed to
function with respect to the inter-color distance. Still, we
may remember that the
2.1 Crude fuzzy histograms offer the equivalence between the perceptual inter-color
In order to construct such a function we must provide a distance and the Euclidean distance between their tristim-
semantical description of the significance of the numbers ulus representations. Even more, the notion of JND as-
attached to each color in the color space 9 and a method sures visual equivalence for colors that are closer than 2.3
that assures that the numbers are well within the  

 [12]. Practical considerations and the analytical simplifi-
range. The immediate approach is to slightly modify the cation of the computational expressions impose the use of
construction of the normal color histogram. We may de- an unified formula for the resemblance degree. A linear
fine the concept of typicality of a color within the given descent would require little computation but could con-
image as the importance of the given color with respect tradict the smooth descent principle. A Gaussian operator
to its relative presence. In fact, this description relates the (3) could be a more appropriate choice.

\
D
? 
= <
^ _ 0#` XE [*a\   b H
typicality of the colors to the area that they occupy, and
thus to their probability of occurrence. But the most typ- M  D V W XZY%[*\] (3)
ical color must have a typicality degree of 1, regardless
its occurrence probability and thus we define this fuzzy This fuzzy color model enables to enlarge the influence
histogram as (2): of a given color to its neighboring colors, according to

 $    the uncertainty principle (of not being certain that a quan-

:<;>=?A@BC  EDF *3576G9IH

tized color has not erroneous changed the original color)
(2)
and the perceptual similarity. This means that each time
a color  is found in the image, it will influence all the
This is in fact the usual color histogram (1) but with a dif- quantized colors from 9 (and thus all the histogram bins)
ferent normalization condition; instead of the probability  according to their resemblance to the color  . Numeri-
density function normalization condition  ( J ?A@BLK  D  ), cally, this could be expressed as:
we normalize by the mode ( :<;>= ? @ BC  D  ) of the color !d"*$ +"%$
distribution. The new normalization does not change  \ c & & & M ?  D    ghji102V
much the intrinsic properties of the histogram, since it ? @ BLK e (*) f (*) .
preserves its shape (including the “holes” or empty bins &  ?
that appeared mainly as quantization effect). We may use  ?A@BLK  D M  D H (4)
this approach as a transitional step in changing the prob-
abilistic description of the color description by a fuzzy The expression in (4) is the linear convolution between
description. Also, such a histogram can be useful if fuzzy the usual color histogram and the fuzzy color model (sup-
distances are to be embedded in the retrieval system, since posing that the model is color-independent, that is M ? 
the fuzzy distances are defined on fuzzy sets and the nor- M ?@ %35Ra D 6k9 ). Thus we can compactly write:
mal histogram cannot be viewed as one.  \  l M ? H (5)
The convolution expresses the histogram smoothing, pro-
2.2 Fuzzy paradigm-based histograms
vided that the color model is indeed a smoothing, low-
As ennounced in the description of the taxonomic cat- pass filtering kernel; such an approach was proposed in
egories, the fuzzy paradigm-based techniques are con- [7] for gray scale images, but the grays model was a trian-
structed according to a fuzzy model of the objects of the gular function. The use of the Gaussian shape from (3) as
universe. In the case of color images, the objects are color model produces the smoothed histogram, proposed
the colors within the image or the possible colors within by many authors (that can be traced back to [8]) as a mean
the set 9 . The model assumes that any color is a fuzzy for the reduction of quantization errors.
 fuzzy sets defined over the same problem universe Š ,
2.3 Fuzzy aggregational histogram
€
that means ˆ{A‰ N Š‹PŒ R . We will also denote by
The colors are already described by a fuzzy model. As
MV ˆ{A‰< the resemblance measure between sets ˆ and ‰
and by ˆ{A‰< their corresponding distance. As sug-
expressed in (4), the contribution of each individual pixel
of the image is added in order to produce the global con- `
gested in [2], if MV ˆ{a‰7 is a resemblance measure, than
tribution of the image at each possible color. Thus, the
interaction between the colors (as result of a specific im- `
ˆa‰7I
0MV ˆ{A‰< is a distance
between the fuzzy
sets ˆ and ‰ , with ˆa‰7Ž .
age appearance) is modeled as a ”stockpile” and described `
We propose to investigate two approaches, derived
by a sum operation.
from a fuzzy set equality formulation and the fuzzy sym-
In order to produce a fuzzy interaction model that
metric difference.
groups together the individual pixel contributions, we
must re-define the fuzzy histogram. We will again use the
typicality approach (already used for the crude fuzzy his- 3.1 Distances between fuzzy sets from fuzzy
togram), but we consider the image as the reunion of the equality
colors that are present within. Thus, the typicality of the
color  is the measure of resemblance of the given color
As known, the set equality relation can be expressed as
a double inclusion, which can be further translated into
with respect to the reunion of all the colors in the image.
fuzzy logic operators. We may note that the approach was
Since the [reunion (logical or operator) is modeled by a
briefly mentioned in [2] as an extension of a previous dis-
T-conorm [16] we have:
similarity indicator introduced in [5], [6]. Still the au-
Rm T n Msr o e/p fEq  [ o e/p fEq Msr o e/p fEq  (6) thors did not investigated neither if the expressions meet
o e/p fEq the mathematical conditions (reflexivity in particular) nor
the generation of a complete class of such fuzzy distances
The T-conorm models the aggregation of the individual (or resemblance measures).
entities.[ Several T-conorms have been [ proposed [16]:
S U S U
Zadeh  O :7[ ;Z=   , algebraic  t
S U Svu a double
The set equality relation can be classically expressed as
ˆ‘‰“’”ˆ“•‡‰–—˜™‰š• ˆ . For
U 0 SU , Lukasiewicz S  U w :<xzy
 S{u|U  , Hamacher fuzzy sets,inclusion:
[ S U " \ [
 V~}€%$‚ " } , Einstein S  U VQ$ }s .
the inclusion relation can be evaluated [11] by
the number M%›œ5 xzyžŸ>B  ¡¢ ˆ{ £5a¤R¥¦‰# £5a , where
} that the relaxation of%}the  fuzzy con- ¡ is a fuzzy negation operator (we will use the classical
We may notice

 S
0 S ). Thus, we can evaluate the
straint (that any number should be within   ) in the Zadeh negation, ¡* 
definitions of the fuzzy T-conorms that impose it in an equality relation of the fuzzy sets ˆ and ‰ by the number
explicit manner (such as the Lukasiewicz operator) will M§|M ›œ* –—˜{M ›œ* .
simply replace the reunion operator by a simple addition Since any fuzzy logic conjunction is a [ T-norm ¨ and
of the resemblance degrees. Thus we obtain the paradigm- any fuzzy logic disjunction is a T-conorm , dual to each
operator ¡ [11],
based fuzzy histogram from (4). In fact, we can view other with respect to the given negation
the fuzzy aggregational histogram as a thresholding of the [16], we can rewrite M%[ ›œ5
x©yžAŸEB  [ ¡I ˆ £5a*A‰# £5j
smoothed, paradigm-based, fuzzy histogram; more colors and MsVœ5›
x©yžŸ>B  ª¡I ‰# £5a5aˆ £*j . Thus, the gen-
that are actually present in the image (but are close to the eral expression of the fuzzy resemblance degree of the two
real colors) have good chances to receive a typicality of 1. sets can be thus expressed as (7).
Usingƒm the Zadeh operators, the fuzzy histograms be-
comes „ :7;Z= Msr o e/p fEq  , which implies that any
MV ˆ{A‰<«¨„ M%›œ5jMsœ¬›¬H (7)

ƒm
%3*#6† ) and the other colors (those within We expect that any useful resemblance measure ex-
color that appears in the image will exhibit a typicality of
1 ( … hibits some common-sense properties, such as the value
9‡0 ) will have typicalities depending on the distance it should take for the case of the comparison of two iden-
to the closest existing color. Since all the colors from  , tical objects (histograms or fuzzy sets), i.e the reflexivity
regardless their relative proportions, have the same typi- property. This value must be 1, according to the usual nor-
cality, this kind of fuzzy histogram is less discriminant. malizations and mathematical conditions [2]. That means
we should impose:
if ˆ{ £5­‰# £5®3¬£v68Š then MV ˆ{a‰7

H
3 Fuzzy-based resemblance mea- (8)
sures and distances Replacing the condition (8) in the general expression of
the resemblance measure (7) we obtain:
[ [
¨¯ Ÿ>zx yB ž 
h
0 ˆ £5‚aˆ £5as ŸExzyBž  ˆ{ £5
02ˆ{ £*jƒ°,
H
In this section we will focus on the introduction of some
new resemblance measures (and subsequently distances)
between fuzzy sets. Let us denote by ˆ and ‰ two (9)
For any T-norm ¨ , ¨*   if and only if
S S
S 
[16], In the case of crisp sets, the symmetrical difference is
and thus the condition from (9) becomes: defined as:
[

 ³] ²´ x¶µZ;>· ] y¸¸a¹
>Ÿ xzyBž  2 0 ˆ{ £5‚aˆ{ £5a± (10) Í  ˆOÎ#‰70t ˆtÏ»‰< or (19)
[

35£v6vŠwH Í  ˆ«0h‰7%Îv ‰Ð02ˆIH
02ˆ{ £5‚aˆ{ £5a± (11) [
(20)

[ Since any T-conorm can be used as a logical ”or”

In fact, the condition (11) states that the T-conorm that operator (and thus as a reunion) and any T-norm ¨ can be
is to be used in the expression (7) must necessarily respect used as a logical ”and” operator (and thus as a intersec-
the excluded middle principle [16], [11]. The only such T- tion) [16], we can express (19) and (20) as:
[ Lukasiewicz (bounded [

u º .sum) ͧ £5T|¨¯ ˆ £5‚a‰d £5a*
0†¨¢ ˆ{ £5A‰# £5jƒ°EH (21)
conorm is the ”or” operator,
defined as £saº» :<xzy j£ Thus, the general
expression of the fuzzy resemblance measure becomes: [

Ív £* ¯¨„ ˆ{ £* 02‰# £*j*j¨„ 02ˆ{ £5‚a‰# £5aR°EH
MV ˆa‰7c ¨* S  U ƒ¼ x¶¸A½ (12) (22)
S  :<x©y

02ˆ{ £5 u ‰# £5a
ŸEx©yBž  The dissimilarity set used in [5], [6] for the Restle indi-
cator is the fuzzy symmetrical[ difference (22) according
U  :<x©y

¾u ˆ{ £502‰# £5a*H to the Zadeh fuzzy operators (  :7;>= , ¨d :<x©y ). Other
ŸEx©yBž 
distances could be potentially obtained from (21) and (22)
Equation (12) represents in fact a class of resemblance by using other fuzzy operators. The following proposition
measures, provided that we use different T-norms ¨ . We
S U
[14] will, however, limit their choice. [
can use the following T-conorms [16]: Zadeh ¨*  7 Proposition: For any fuzzy operators and ¨ , the
:<x©y   , algebraic ¨*  … , Lukasiewicz ¨* S  U …
S U S U 
S U

distance.
fuzzy distance (18) constructed according to the symmet-
:7;Z= R Su2U 0
 , Hamacher S  U T SU€¿ Su2U 0 SU  ,
X S ¨* UVuÀ rical set difference (21) or (22) is the
Einstein ¨*  T
S U 
S €
U ¿ 0 0 SU  . Using the proper combination of T-norms and T-
Using the above mentioned T-norms and the general conorms [16] in the general expression from (22), after
expression from (12), after some simple arithmetics we some simple arithmetics we obtain that the distance
tween the fuzzy sets ˆ and ‰ is, in all cases, the

dis-
be-
obtain the following distances between fuzzy sets: (13)
tance between the two sets:
when using the Zadeh operator, (14) when using the alge-
&
braic operator, (15) when using the Lukasiewicz operator, ` ˆa‰<T ŸEB  Ç ˆ £5V0h‰d £* ÇH (23)
(16) when using the Hamacher operator and (17) when us-
ingÁ Ã the Einstein operator. In all the equations, we denote
 ´ ^ Ÿ>B  ¯ˆ{ £5…0|‰# £5®° and ÄÅ xzyž ŸEB  ¯ˆ £*…0 The complete proof can be found in [14].
‰# £5j° .

` Æ ˆa‰7c ŸE ´ B^ Ç ˆ{ £*02‰# £* Ç (13) 4 Experiments and conclusions

ˆa‰7c Á 0hÄ u Á†É Ä The experiments performed in order to establish the re-
`È Á 0hÄ
(14)

`Ê ˆa‰7c Á 0hÄ u X Á8É Ä (15)

trieval capabilities of the proposed fuzzy histograms were
conducted using the Surfimage software platform devel-

`Ë ˆa‰7c
¾u Á8É oped at IMEDIA. We investigated both the objective and
Á 0hÄ Ä
(16)
the subjective retrieval quality, on two different, heteroge-

`Ì ˆa‰7c
0 Á8É Ä H (17) neous, generalist image databases.

The subjective retrieval quality was studied by visual
inspection and similarity estimation of the images re-
3.2 Distances between fuzzy sets from sym- trieved from a large (1500) heterogeneous database of
“natural” images (Comstock). All fuzzy histograms per-
metric difference
form well, providing perceptually appealing results.
According to a more general definition [2], a fuzzy dis- The objective retrieval quality is measured by the pre-
tance between the fuzzy sets ˆ and ‰ (defined over the cision of retrieval for a small, labeled database, consisting
discrete set Š ) is the fuzzy cardinality of the set Í , called of 21 image classes (key frames from a television broad-
dissimilarity set.: cast). The images were described by their color distribu-
& tion, expressed as a fuzzy histogram. Table 1 presents the
` ˆ{A‰< ŸEB  ͧ £5‚H maximal  (at recall 1) and average precision of the retrieval
by the $ histogram and various fuzzy set equality-issued
(18)
distances. Apparently, the Hamacher-operators induces of uncertainty in content based image retrieval is natural
distance performs somehow better, but the difference with and desirable as long as human perception remains the key

same time, the


distance (obtained as fuzzy set sym-
respect to the other distances is not significant. In the factor in judging and using the results.

metrical difference distance) provides significantly better

performance. The same type of results is obtained for the References
other fuzzy histograms.
[1] J. C. Bezdek. Fuzzy models - what are they and why ?
IEEE Trans. on Fuzzy Systems, 1(1):1–5, Feb. 1993.
Table 1: Maximal and average precision rates for the re- [2] I. Bloch. On fuzzy distances and their use in im-
age processing under imprecision. Pattern Recognition,
trieval by the crude fuzzy normalized color histogram (2)
32(11):1873–1895, Nov. 1999.
with various proposed fuzzy set distances [3] J. Huang, S. R. Kumar, M. Mitra, and Z. W.-J. Spatial
Maximal Average color indexing and applications. In IEEE International
precision precision Conference on Computer Vision ICCV ‘98, Bombay, In-
Zadeh (13) 72.38 % 57.11 % dia, 4-7 Jan. 1998.
Algebraic (14) 74.76 % 58.07 % [4] J. Huang, S. R. Kumar, M. Mitra, Z. W.-J., and R. Zabih.
Image indexing using correlograms. In Computer Vi-
Lukasiewicz (15) 75.71 % 57.27 %
sion and Pattern Recognition CVPR ‘97, San Juan, Puerto
Hamacher (16) 75.71 % 57.92 % Rico, 17-19 Jun. 1997.
Einstein (17) 75.24 % 58.06 % [5] R. C. Jain, S. N. Murthy, L. Tran, and S. Chatterjee. Sim-
L1 23) 80.95 % 64.33 % ilarity measures for image databases. In W. Niblack and
R. C. Jain, editors, Storage and Retrieval for Still Image
and Video Databases III, volume SPIE 2420, pages 58–
Table 2 presents the average precision and the precision
67, 1995.
for 10 retrieved images, for the retrieval by the usual color [6] R. C. Jain, S. N. Murthy, L. Tran, and S. Chatterjee. Sim-
histogram and the crude-fuzzy (mode normalized) color ilarity measures for image databases. In Proc. of IEEE
histogram, according to various fuzzy distances. It is clear Conference on Fuzzy Systems, volume 3, pages 1247–
that the use of the fuzzy histogram increases the retrieval 1254, Yokohama, Japan, 1995.
performance. [7] C. V. Jawahar and A. K. Ray. Fuzzy statistics of digital
images. IEEE Signal Processing Letters, 3(8):225–227,
Aug. 1996.
Table 2: Average precision and precision at recall 10 for [8] J. Kautsky, N. K. Nichols, and D. L. B. Jupp. Smoothed
the retrieval by the usual
fuzzy color histogram $ (2)
 color histogram and the crude histogram modification for image processing. CVGIP,
26(3):271–291, Jun. 1984.
[9] B. M. Mehtre, M. S. Kankanhalli, A. D. Narasimhalu, and
Distance class and Average Recall 10
G. C. Man. Color matching for image retrieval. Pattern
histogram type precision precision
 Recognition Letters, 16:325–331, Mar. 1995.
Zadeh, with (1)
Zadeh, with
 $ (2)
55.67 %
57.11 %
45.74 %
47.42 %
[10] G. Pass and R. Zabih. Histogram refinement for content


based image retrieval. In IEEE Workshop on Applications
of Computer Vision, pages 96–102, 1996.
Lukasiewicz, with (1)

Lukasiewicz, with $ (2) 57.27 %
54.18 % 45.45 %
47.03 %
[11] B. Reusch. Mathematics of fuzzy logic. In H. J. Zim-
 mermann, D. Dascălu, and M. G. Negoiţă, editors, Real
Hamacher, with (1)
Hamacher, with $ (2)
 54.58 %
57.92 %
45.41 %
47.94 %
World Applications of Intelligent Technologies, pages 15–
52, Bucureşti, Romania, 1996. Romanian Academy Publ.
House.
[12] G. Sharma and H. J. Trusell. Digital color imaging. IEEE
The various proposed color histograms may be not all- Trans. on Image Processing, 6(7):901–932, Jul. 1997.
ways quantitatively better than the usual color histogram, [13] M. J. Swain and D. H. Ballard. Color indexing. Interna-
but the actual retrieval results are visually appealing and tional Journal of Computer Vision, 7(1):11–32, 1991.
correct. However, we could not expect to obtain more [14] C. Vertan and N. Boujemaa. Using fuzzy histograms and
rigorous results (and thus improved objective measured distances for color image retrieval. In Proc. of CIR’2000,
performance) by relaxing (through fuzzy modeling) the Brighton, United Kingdom, 4-5 May 2000.
histogram image description. [15] C. Vertan and V. Buzuloiu. Fuzzy nonlinear filtering of
color images: A survey. In E. Kerre and M. Nachtegael,
The new introduced fuzzy distances are not specifically editors, Fuzzy Techniques in Image Processing, Heidel-
designed for content based image retrieval, but are gen- berg, Germany, 2000. Physica Verlag.
eral. Their use may prove effective in other application [16] H. J. Zimmermann. Fuzzy Sets, Decision Making and Ex-
areas also. As a final conclusion, we suggest that the re- pert Systems. Kluwer Academic Publ., Boston MA, 1987.
trieval results are very encouraging and prove that the use