SEGMENTATION AND HISTOGRAM GENERATION USING THE HSV COLOR SPACE FOR

IMAGE RETRIEVAL
Shamik Sural, Gang Qian and Sakti Pramanik
Dept. of Computer Science and Engineering,
31 15 Engineering Building, Michigan State University, East Lansing, MI 48824, USA.
[email protected], {qiangang, pramanik}@cse.msu.edu

ABSTRACT the Saturation. For each pixel we, therefore, choose either
its Hue or the Intensity as the dominant feature based on
We have analyzed the properties of the HSV (Hue, its Saturation. We then segment the image by grouping
Saturation and Value) color space with emphasis on the pixels with similar features using the K-means clustering
visual perception of the variation in Hue, Saturation and algorithm [3].
Intensity values of an image pixel. We extract pixel A standard way of generating a color histogram of an
features by either choosing the Hue or the Intensity as the image is to concatenate ‘N’ higher order bits for the Red,
dominant property based on the Saturation value of a Green and Blue values in the RGB space Ell]. The
pixel. The feature extraction method has been applied for histogram then has 23Nbins, which accumulate the count
both image segmentation as well as histogram generation of pixels with similar color. It is also possible to generate
applications - two distinct approaches to content based three separate histograms, one for each channel, and
image retrieval (CBIR). Segmentation using this method concatenate them into one [2]. Smith and Chang [8] have
shows better identification of objects in an image. The used a color set approach to extract spatially localized
histogram retains a uniform color transition that enables color information. Ortega et a1 [6] have used the HS co-
us to do a window-based smoothing during retrieval. The ordinates to form a two-dimensional histogram where
results have been compared with those generated using the each bin contains the percentage of pixels in the image
RGB color space. that have corresponding H and S colors for that bin. We
generate a one-dimensional histogram from the HSV
space where a perceptually smooth transition of color is
1. INTRODUCTION obtained in the feature vector. This enables us to use a
window-based smoothing of histograms so that similar
We have done in-depth analysis of the visual properties of colors can be matched between a query and each of the
the HSV color space and its usefulness in content based database images.
image retrieval applications. In particular, we have We explain the HSV-based feature extraction and
developed image segmentation and histogram generation image segmentation method in the next section and the
applications using this color space - two important histogram generation technique in section 3. We then
methods in CBIR [5,7]. present experimental results in section 4 and draw
Segmentation is done to decompose an image into conclusions from our work in the last section of the paper.
meaningful parts for further analysis, resulting in a higher-
level representation of the image pixels like the 2. IMAGE SEGMENTATION USING FEATURES
foreground objects and the background. In region-based FROM THE HSV COLOR SPACE
CBIR applications, segmentation is essential for
identifying objects present in a query image and each of 2.1. Analysis of the HSV Color Space
the database images. Wang et a1 [12] have used the LUV A three dimensional representation of the HSV color
values of a group of 4 x 4 pixels along with three features space is a hexacone, where the central vertical axis
obtained by wavelet transform of the L component for represents the Intensity [9]. Hue is defined as an angle in
determining regions of interest. Region-based retrieval has the range [0,2x] relative to the Red axis with red at angle
also been used in the NeTra system [4] and the Blobworld 0, green at 2nl3, blue at 4x13 and red again at 2x.
system [l]. We segment color images using features Saturation is the depth or purity of the color and is
extracted from the HSV space as a step in the region- measured as a radial distance from the central axis with
based matching approach to CBIR. The HSV color space value between 0 at the center to 1 at the outer surface. For
is fundamentally different from the widely known RGB S=O, as one moves higher along the Intensity axis, one
color space since it separates out the Intensity (luminance) goes from Black to White through various shades of gray.
from the color information (chromaticity). Again, of the On the other hand, for a given Intensity and Hue, if the
two chromaticity axes, a difference in Hue of a pixel is Saturation is changed from 0 to 1, the perceived color
found to be visually more prominent compared to that of changes from a shade of gray to the most pure form of the

0-7803-7622-6/02/$17.00 02002 IEEE I1 - 589 IEEE ICIP 2002

color represented by its Hue. Looked from a different based approximation can determine the intensity and
angle, any color in the HSV space can be transformed to a shade variations near the edges of an object, thereby
shade of gray by sufficiently lowering the Saturation. The sharpening the boundaries and retaining the color
value of Intensity determines the particular gray shade to information of each pixel. This phenomenon is exhibited
which this transformation converges. When Saturation is in detail in figure 3. Figure 3(a) shows a number of solid
near 0, all pixels, even with different Hues, look alike and colors with varying intensities. Figure 3(b)-(c) shows the
as we increase the Saturation towards 1, they tend to get result of approximation using the RGB color space taking
separated and are visually perceived as the true colors the 2 higher order bits. It is seen that some of the colors
represented by their Hues as shown in figure 1. Thus, for with high intensities cannot be recognized, as they are
low values of Saturation, a color can be approximated by a inseparable from the background. Also, we see that the
gray value specified by the Intensity level while for higher background of white and gray are considered equivalent
Saturation, the color can be approximated by its Hue. The due to approximation. The HSV features used by us retain
Saturation threshold that determines this transition is once the identity of the colors even at these intensity levels as
again dependent on the Intensity. For low intensities, even seen in figure 3(d). This makes the HSV-based features
for a high Saturation, a color is close to the gray value and very useh1 in running segmentation algorithms like
vice versa. Saturation gives an idea about the depth of clustering on the approximated pixels.
color and human eye is less sensitive to its variation
compared to variation in Hue or Intensity. We, therefore, 2.3. Pixel Grouping by K-means Clustering Algorithm
use the Saturation value of a pixel to determine whether The RGB value of a pixel is first transformed to the HSV
the Hue or the Intensity is more pertinent to human visual value using a method suggested in [9]. The feature is next
perception of the color of that pixel and ignore the actual extracted from each image pixel. After extraction, the
value of the Saturation. It is observed that for higher pixel features are clustered using the K-Means clustering
values of intensity, a saturation of 0.2 differentiates algorithm to group them into regions of similar color.
between Hue and Intensity dominance. Assuming the Since the Hue and the Intensity values belong to the same
maximum Intensity value to be 255, we use the following number space, the two sets of data are clustered separately
threshold function to determine if a pixel should be so that the color and the gray value pixels are not
represented by its Hue or its Intensity as its dominant considered in the same cluster. In the K-means clustering
feature. algorithm, we start with K=2 and adaptively increase the
number of clusters till the improvement in error falls
below a threshold or a maximum number of clusters is
In the above equation, we see that for V=O, th(V) = reached. We set the maximum number of clusters to 12
1.O, meaning that all the colors are approximated as black and an error improvement threshold over number of
whatever be the Hue or the Saturation. On the other hand, clusters to 5 %.
with increasing values of the Intensity, Saturation
threshold that separates Hue dominance from Intensity 3. HISTOGRAM GENERATION
dominance goes down.
We also use the HSV color space for histogram generation
2.2. Feature Generation using the HSV Color Space where each pixel contributes either its Hue or its Intensity
We generate features by utilizing the above properties of as explained in the last section. We extract the color
the HSV color space for clustering pixels into segmented histogram as the feature vector having two parts: (i) A
regions. Figure 2(a) shows an image and figure 2(b) representation of the Hue between 0 and 2n quantized
shows the same image using the approximated pixels after after a transformation and (ii) A quantized set of gray
Saturation thresholding. Pixels with sub-threshold values as shown in figure 4(a). The number of
Saturation have been represented by their gray values components in the feature vector generated based on Hue
while the other pixels have been represented by their is given by:
Hues. The feature generation used by us makes an Nh = r 2 X MULT-FCTR1 +1 (2)
approximation of the color of each pixel in the form of Here MULT-FCTR determines the quantization level
thresholding. On the other hand, features generated from for the Hues. We typically choose a value of 8. The
the RGB color space approximate by considering a few number of components representing gray values is:
higher order bits. In figures 2(c) - (d) we show the same N,= Imax +1
image approximated with the six lower-order bits all set to (3)
[DIV-FCTR]
0 and all set to 1, respectively. It is seen that the
approximation done by the RGB features blurs the Here I,, is the maximum value of the Intensity,
distinction between two visually separable colors by usually 255, and DIVFCTR determines the number of
changing the brightness. On the other hand, the HSV- quantized gray levels. We, typically, choose DIV-FCTR =

I1 - 590
16. The quantized values of Hue may be considered variations and come up with clusters that put neighboring
circularly arranged since Hue varies between 0 to 2 x , both pixels with similar color but small difference in shade to
the end points being red. The feature vector is thus a different clusters. Often, two distinct colors are merged
combination of two independent vectors as shown in together. For the first image in figure 5, it is seen that
figure 4(b). RGB clustering could not detect the object boundary at all
It has been observed that, when color histograms are with the color of the man's body merged with the color of
extracted from two similar images, often two neighboring the river due to high brightness. In the second image, we
components in the histograms have high values. This is see that the sky has been identified as three regions and
due to the fact that two colors that appear close to the also the bush in front of the castle is interspersed with the
human eye may have a small difference in shade and map color of the brick wall. In the third image, the faces of the
to two neighboring components in the histogram. When a people could not be identified distinctly. In some cases, a
standard measure like the Euclidean distance is used to face was clustered along with the color of the dress of the
order such feature vectors, the result shows a high subject. In the HSV-based approach, better clustering was
distance value. To overcome this drawback, we compare achieved in all the cases with proper segmentation. The
two histograms through smoothing windows instead of clustered image pixels may be further processed to merge
comparing the vector components directly. All the small image regions into larger blocks for marking the
conventional color histograms fail to provide the requisite exact object boundaries.
perceptual gradation of colors in the feature vectors as
required by such a comparison. The histogram generated 4.2. Histogram based image Retrieval Results
by us retains this property in the feature vector. For the j" We have developed an interface using Java applet that
component (i E [0, Nh+N,-l]), with a smoothing window displays images similar to a query image from a database
size of 2N+1, the average value is calculated as follows: of about 14,500 images obtained from the web and IMSI
j+N master clips. Figures 6(a)-(b) show the recall and
Hist, fi] = w(i - j)Hist[i] where w(i-j) = 2-Ii-J1 (4) precision of image retrieval using a standard RGB
i=j-N histogram and the new histogram. For the new histogram,
Since we derive both the Hue and the gray level we show results for different widths of the smoothing
features using the HSV space, there are two independent window. From the figures, it is seen that the new
color continuums in the histogram, one from R e d 3 histogram performs much better than an RGB histogram
Green3Blue3Red and the other from based system. In most cases, recall and precision values
Black+Gray+White. The circular nature of the Hue are higher for the same number of nearest neighbors. It is
components and the discontinuity at the Hue-Intensity also observed that application of a window with small
component boundary are taken care of programmatically. width improves the result set. Again, for a very large
The use of the new histogram shows an improved window width, different distinct colors tend to get added
performance over conventional histograms generated from up and hence we do not see better results anymore. Such a
the RGB color space. The smoothing window further comparison cannot be done with the RGB features due to
tunes the result of retrieval. the lack of color continuity in the generated histogram as
explained in the last section. Some of the preliminary
4. EXPERIMENTAL RESULTS results in terms of actual retrieved images are available in
[lo].
4.1. Segmentation Results
We have tested the algorithm on a large number of natural 5. CONCLUSIONS
scene images. In this paper we demonstrate results that
represent our findings from these experiments. In figures We have studied some of the important properties of the
5(a)-(c), we show three images, their HSV-based HSV color space and have developed a framework for
segmentation results and RGB-based segmentation results. extracting features that can be used both .for image
For RGB, we consider the higher order 2 bits to generate segmentation and color histogram generation - two
the feature vectors. In the images, we have painted the important approaches to content based image retrieval.
different regions using the color represented by the Our approach makes use of the Saturation value of a pixel
centroid of the clusters to give an idea about the to determine if the Hue or the Intensity of the pixel is
differentiation capabilities of the two color spaces. more close to human perception of color that pixel
Although exact segmentation of unconstrained color represents. The K-means clustering of features combines
images is still a difficult problem, we see that the object pixels with similar color for segmentation of the image
boundaries can be identified in a way more similar to into objects. We are also able to generate a histogram that
human perception of the same. The RGB features, on the enables us to perform a window-based smoothing of the
other hand, fail to determine the color and Intensity vectors during retrieval of similar images. While it is well

I1 - 591
established that color itself cannot retain semantic
information beyond a certain degree, we have shown that
retrieval results can be considerably improved b y
choosing a better histogram.

iIIII.I.III mmnm 04 1

II.I.I.I.I.Inn 2 5 10 20 40
Figure 1. Variation of color perception with saturation Nearest Neighbors
(Decreasing from 1 to 0 left to right ) for a fixed value of
Intensity and Hue = 0 (Red), Hue = 2x/3 (Green), Hue = 47d3 6(a)
(Blue).

\ , ~, .,
Figure 2. (a) Original Image (b) HSV Approximation (c) RGB 2 5 10 20 40
approximation with all low order bits set to 0 and (d) RGB Nearest Neighbors
approximation with all low order bits set to 1.

Figure 6. (a) Recall and (b) Precision variation of the new

histogram and a standard RGB histogram.
3(a) 3(b) 3(c) 3(d)
Figure 3. (a) Original Colors (b) HSV Approximation (c) RGB References
approximation with all low order bits set to 0 and (d) RGB
approximation with all low order bits set to I . [I] C.Carson et al, “Blobworld: A System for Region-based
Image Indexing and Retrieval”, Proc. Third Int. Conf. on Visual
Information Systems, June 1999.
[2] A.Jain and A.Vailaya, “Image Retrieval using Color and
Shape, Pattern Recognition, vol. 29, no. 8, pp. 1233-1244, 1996.
[3] L.Kaufman and P.J.Rousseeuw, “Finding Groups in Data:
An Introduction to Cluster Analysis”, John Wiley & Sons, 1990.
[4] W.Y.Ma and B.Manjunath, “NeTra: A Toolbox for
Navigating Large Image Databases”, Proc. IEEE Int. Conf. on
Image Processing, pp. 568-571, 1997.
[5] W.Niblack et al, “The QBIC Project: Querying Images by
4(b) Content using Color Texture and Shape”, Proc. SPIE Int. Soc.
Figure 4. (a) Representation of colors in the histogram and (b) Opt. Eng., in Storage and Retrieval for Image and Video
Circular representation of hue and linear representation of gray Databases, vol. 1908, pp. 173-187, 1993.
values in the histogram. [6] M. Ortega et al, “Supporting Ranked Boolean Similarity
Queries in MARS”, IEEE Trans. on Knowledge and Data
Engineering, vol. 10, no. 6, pp. 905-925, 1998.
[7] A.W.M. Smeulders et al, “Content Based Image Retrieval at
the End of the Early Years”, IEEE Trans. on PAMI, vol. 22, no.
12, pp. 1-32, December, 2000.
[8] J.R.Smith and S.-F. Chang, “VisualSeek: A Fully Automated
Content based Image Query System”, Proc. ACM Multimedia
Conf., Boston, MA, 1996.
[9] GStockman and L.Shapiro, “Computer Vision”, Prentice
Hall, 2001.
[IO] S.Sural, G.Qian and S.Pramanik, “A Histogram with
Perceptually Smooth Color Transition for Image Retrieval”,
Proc. Fourth Int. Conf. on CVPRIP, Durham, 2002 (to appear).
[ 1 11 MSwain and D.Ballard, “Color Indexing”, Int. Journal of
Computer Vision, vo1.7, no. I, pp. 11-32, 1991.
[ 121 J.Z.Wang, Jia Li, Gio Wiederhold, “SIMPLIcity:
5(c) Semantics-sensitive Integrated Matching for Picture LIbraries,”
Figure 5. (a) Original Images (b) Segmentation using HSV IEEE Trans. on PAMI, vol. 23, no. 9, pp., 2001.
features and (c) Segmentation using RGB features.

I1 - 592