International Journal of Medical Informatics (2004) 73, 123

REVIEW

A review of content-based image retrieval systems in medical applicationsclinical benets and future directions
Henning Mller* , Nicolas Michoux, David Bandon, Antoine Geissbuhler
Service of Medical Informatics, University Hospital of Geneva, Rue Micheli-du-Crest 24, 1211 Geneva 14, Switzerland
Received 22 July 2003; accepted 13 November 2003

KEYWORDS
Medical image retrieval; Content-based search; Visual information retrieval; PACS

Summary Content-based visual information retrieval (CBVIR) or content-based image retrieval (CBIR) has been one on the most vivid research areas in the eld of computer vision over the last 10 years. The availability of large and steadily growing amounts of visual and multimedia data, and the development of the Internet underline the need to create thematic access methods that offer more than simple text-based queries or requests based on matching exact database elds. Many programs and tools have been developed to formulate and execute queries based on the visual or audio content and to help browsing large multimedia repositories. Still, no general breakthrough has been achieved with respect to large varied databases with documents of differing sorts and with varying characteristics. Answers to many questions with respect to speed, semantic descriptors or objective image interpretations are still unanswered. In the medical eld, images, and especially digital images, are produced in everincreasing quantities and used for diagnostics and therapy. The Radiology Department of the University Hospital of Geneva alone produced more than 12,000 images a day in 2002. The cardiology is currently the second largest producer of digital images, especially with videos of cardiac catheterization (1800 exams per year containing almost 2000 images each). The total amount of cardiologic image data produced in the Geneva University Hospital was around 1 TB in 2002. Endoscopic videos can equally produce enormous amounts of data. With digital imaging and communications in medicine (DICOM), a standard for image communication has been set and patient information can be stored with the actual image(s), although still a few problems prevail with respect to the standardization. In several articles, content-based access to medical images for supporting clinical decision-making has been proposed that would ease the management of clinical data and scenarios for the integration of content-based access methods into picture archiving and communication systems (PACS) have been created. This article gives an overview of available literature in the eld of content-based access to medical image data and on the technologies used in the eld. Section 1 gives an introduction into generic content-based image retrieval and the technologies used. Section 2 explains the propositions for the use of image retrieval in medical practice and the various approaches. Example systems and application areas are

*Corresponding author. Tel.: +41-22-372-6175; fax: +41-22-372-8680. E-mail address: [email protected] (H. Mller). 1386-5056/$ see front matter 2004 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2003.11.024

H. Mller et al.
described. Section 3 describes the techniques used in the implemented systems, their datasets and evaluations. Section 4 identies possible clinical benets of image retrieval systems in clinical practice as well as in research and education. New research directions are being dened that can prove to be useful. This article also identies explanations to some of the outlined problems in the eld as it looks like many propositions for systems are made from the medical domain and research prototypes are developed in computer science departments using medical datasets. Still, there are very few systems that seem to be used in clinical practice. It needs to be stated as well that the goal is not, in general, to replace text-based retrieval methods as they exist at the moment but to complement them with visual search tools. 2004 Elsevier Ireland Ltd. All rights reserved.

1. Introduction to image retrieval

This section gives an introduction to content-based image retrieval systems (CBIRSs) and the technologies used in them. Image retrieval has been an extremely active research area over the last 10 years, but rst review articles on access methods in image databases appeared already in the early 1980s [1]. The following review articles from various years explain the state-of-the-art of the corresponding years and contain references to a large number of systems and descriptions of the technologies implemented. Enser [2] gives an extensive description of image archives, various indexing methods and common searching tasks, using mostly text-based searches on annotated images. In [3], an overview of the research domain in 1997 is given and in [4], the past, present and future of image retrieval is highlighted. In [5] an almost exhaustive overview of published systems is given and an evaluation of a subset of the systems is attempted [6]. Unfortunately, the evaluation is very limited and only for very few systems. The most complete overview of technologies to date is given by Smeulders et al. [7]. This article describes common problems such as the semantic gap or the sensory gap and gives links to a large number of articles describing the various techniques used in the domain. For an even deeper introduction into the domain, several theses and books are available [811]. The only article reviewing several medical retrieval systems so far, is to our knowledge [12]. It explains using one paragraph per topic a number of medical image retrieval systems. No systematic comparison of the techniques employed and the data/evaluation used has been attempted. This review paper in contrast is the rst review that concentrates on image retrieval in the medical domain and that does a systematic overview of techniques used, visual features employed, images indexed and medical departments involved. It also offers future perspectives for image retrieval in the

medical domain and will be a good starting point for research projects on medical image retrieval as useful techniques for certain sorts of images can be isolated and past errors can be avoided.

1.1. Content-based image retrieval systems

Although early systems existed already in the beginning of the 1980s [13], the majority would recall systems such as IBMs Query By Image Content1 (QBIC) as the start of content-based image retrieval [14,15]. The commercial QBIC system is denitely the most well-known system. Another commercial system for image [16] and video [17] retrieval is Virage2 that has well known commercial customers such as CNN. Most of the available systems are, however from academia. It would be hard to name or compare them all but some well-known examples include Candid [18], Photobook3 [19] and Netra [20] that all use simple color and texture characteristics to describe the image content. Using higher level information, such as segmented parts of the image for queries, was introduced by the Blobworld4 system [21,22]. PicHunter [23] on the other hand is an image browser that helps the user to nd an exact image in the database by showing to the user images on screen that maximize the information gain in each feedback step. A system that is available free of charge is the GNU Image Finding Tool (GIFT5 ) [24,25]. Some systems are available as demonstration versions on the web such as Viper,6 WIPE7 or Compass.8
http://wwwqbic.almaden.ibm.com/. http://www.virage.com/. 3 http://www-white.media.mit.edu/vismod/demos/facerec/ basic.html. 4 http://elib.cs.berkeley.edu/photos/blobworld/. 5 http://www.gnu.org/software/gift/. 6 http://viper.unige.ch/demo/php/demo.php. 7 http://wang.ist.psu.edu/IMAGE/. 8 http://compass.itc.it/demos.html.
2 1

Content-based image retrieval systems

3 most all systems employ colors. Although most of the images are in the red, green, blue (RGB) color space, this space is only rarely used for indexing and querying as it does not correspond well to the human color perception. It only seems reasonable to be used for images taken under exactly the same conditions each time such as trademark images. Other spaces such as hue, saturation, value (HSV) [24,27,28] or the CIE Lab [15] and Luv [29] spaces are much better with respect to human perception and are more frequently used. This means that differences in the color space are similar to the differences between colors that humans perceive. Much effort has also been spent on creating color spaces that are optimal with respect to lighting conditions or that are invariant to shades and other inuences such as viewing position [30,31]. This allows to identify colors even under varying conditions but on the other hand information about the absolute colors is lost. In specialized elds, namely in the medical domain, absolute color or grey level features are often of very limited expressive power unless exact reference points exist as it is the case for computed tomography images. 1.2.2. Texture Partly due to the imprecise understanding and denition of what exactly visual texture actually is, texture measures have an even larger variety than color measures. Some of the most common measures for capturing the texture of images are wavelets [32,33] and Gabor lters [24,34,35] where the Gabor lters do seem to perform better and correspond well to the properties of the human visual cortex for edge detection [36,37]. These texture measures try to capture the characteristics of the image or image parts with respect to changes in certain directions and the scale of the changes. This is most useful for regions or images with homogeneous texture. Again, invariances with respect to rotations of the image, shifts or scale changes can be included into the feature space but information on the texture can get lost in this process [38]. Other popular texture descriptors contain features derived from co-occurrence matrices [3941], features based on the factors of the Fourier transform [38] and the so-called Wold features [42]. 1.2.3. Local and global features Both, color and texture features can be used on a global image level or on a local level on parts of the image. The easiest way to use regional features is to use blocks of xed size and location, so-called partitioning of the image [7,24] for local feature extraction. These blocks do not take into account

Fig. 1 The principal components of all content-based image retrieval systems.

Most of these systems have a very similar architecture for browsing and archiving/indexing images comprising tools for the extraction of visual features, for the storage and efcient retrieval of these features, for distance measurements or similarity calculation and a type of graphical user interface (GUI). This general system setup is shown in Fig. 1. All shown components are described in more detail further on.

1.2. Visual features used

Visual features were classied in [5] into primitive features such as color or shape, logical features such as identity of objects shown and abstract features such as signicance of scenes depicted. Still, all currently available systems only use primitive features unless manual annotation is coupled with the visual features as in [26]. Even systems using segments and local features such as Blobworld [21,22] are still far away from identifying objects reliably. No system offers interpretation of images or even medium level concepts as they can easily be captured with text. This loss of information from an image to a representation by features is called the semantic gap [7]. The situation is surely not satisfactory and the semantic gap denitely accounts for part of the rejection to use image retrieval applications, but the technology can still be valuable when advantages and problems are understood by the users. The more a retrieval application is specialized for a certain, limited domain, the smaller the gap can be made by using domain knowledge. Another gap is the sensory gap that describes the loss between the actual structure and the representation in a (digital) image. 1.2.1. Color In stock photography (large, varied databases for being used by artists, advertisers and journalists), color has been the most effective feature and al-

4 any semantics of the image itself. When allowing the user to choose image regions (regions of interest (ROI)) [43], to delineate objects in the image [44] or when segmenting the image into areas with similar properties [45], the locally extracted features contain more information about the image objects or underlying structures. 1.2.4. Segmentation and shape features Fully automated segmentation of images into objects itself is an unsolved problem. Even in fairly specialized domains, fully automated segmentation causes many problems and is often not easy to realize. In image retrieval, several systems attempt to perform an automatic segmentation of the images in the collection for feature extraction [21,46]. To have an effective segmentation of images using varied image databases the segmentation process has to be done based on the color and texture properties of the image regions [45]. Much has also been written on medical image segmentation with respect to browsing image repositories [47,48]. After segmentation, the resulting segments can be described by shape features that commonly exist, including those with invariances with respect to shifts, rotations and scaling [49,50]. 1.2.5. Semantics? All these visual features, and even features derived from segmented regions, are still fairly low-level compared to high level concepts that are contained in the images. They do not necessarily correspond to objects in the images or the semantic concepts or structures that a user is interested in. Several articles speak of semantic or cognitive image retrieval [5154] but in the end this has not yet been realized with visual features alone. It often comes down to connecting visual low-level features with textual high level features which has already been proposed in [55] as early as 1996. The annotation of image collections for retrieval or for the combination with visual features for retrieval is another very active research area [26,56]. Many problems such as the subjectiveness of annotations need to be addressed even when working with restricted vocabularies. The users annotations do not only vary between persons, they are also varying in time for the same person and they depend strongly on the users actual search tasks. However, in the medical domain, good annotated atlases of medical images do exist that contain objective knowledge, for example based on the images of the visible human.9 The denition of visual similarity or relevance with
http://www.nlm.nih.gov/research/visible/visible/human. html.
9

H. Mller et al. respect to visual similarity are also philosophical questions that have been discussed for a long time [57].

1.3. Comparison techniques used

Basically all systems use the assumption of equivalence of an image and its representation in feature space. These systems often use measurement systems such as the easily understandable Euclidean vector space model [15,58] for measuring distances between a query image (represented by its features) and possible results representing all images as feature vectors in an n-dimensional vector space. This is done, although metrics have been shown to not correspond well to human visual perception (Tversky [59]). Several other distance measures do exist for the vector space model such as the city-block distance, the Mahalanobis distance [15] or a simple histogram intersection [60]. Still, the use of high-dimensional feature spaces has shown to cause problems and great care needs to be taken with the choice of distance measurement to be chosen in order to retrieve meaningful results [61,62]. These problems with a similarity denition in high-dimensional feature spaces is also known as the curse of dimensionality and has also been discussed in the domain of medical imaging [63]. Another approach is a probabilistic framework to measure the probability that an image is relevant [64]. A relationship between probabilistic image retrieval and vector-space distance measures is given in [65]. This paper concludes that the vector space distance measurements described in the literature correspond, in principal, to probabilistic retrieval under certain assumptions of the feature distributions. Another probabilistic retrieval form is the use of support vector machines (SVMs) [66] for a classication of images into classes for relevant and non-relevant. Various systems use methods that are well known from the text retrieval eld and apply them to visual features where the visual features have to correspond roughly to words in text [24,67,68]. This is based on the two principles. A feature frequent in an image describes this image well. A feature frequent in the collection is a weak indicator to distinguish images from each other. Several weighting schemes for text retrieval that have also been used in image retrieval are described in [69]. A formal denition of vector-space, probabilistic and boolean models for information retrieval is attempted in [70]. A general overview

Content-based image retrieval systems of pattern recognition methods and various comparison techniques is given in a very good review article [187]. This article describes the feature extraction, selection, feature space reduction techniques that are equally important in the image retrieval domain.

5 three-dimensional representation of the similarity space as well as the El Nio system, where the user moves images together into clusters that (s)he thinks are similar. The correlation across various media (text, image, video, audio) should also not be forgotten if these are available. Whenever additional information is available such as annotations of the images, it should be used for the retrieval.

1.4. Storage and access methods

Although most systems do not talk in detail about the underlying storage and access methods [23,52] this is extremely important for interactive systems to keep response times at bay. Common storage methods used are relational databases [15,71], inverted les [24], self-made structures or simply to keep the entire index in the main memory which will inevitably cause problems when using large databases. These methods often need to use dimension reduction techniques or pruning methods [72] to allow for an efcient and quick access to the data. Some indexing techniques such as the KD-trees are described in [73]. Principal component analysis (PCA) for feature space reduction is used in [74]. This technique is also called KarhunenLoeve transform (KLT) [75]. Another feature space reduction technique is the independent component analysis (ICA) described in [76,187]. [187] also explains a variety of other techniques such as for feature selection.

2. Use of image retrieval in medical applications

The number of digitally produced medical images is rising strongly. In the radiology department of the University Hospital of Geneva (HUG) alone, the number of images produced per day in 2002 was 12,000, and it is still rising. Videos and images produced in cardiology are equally multiplying and endoscopic videos promise to be another very large data source that are planned to be integrated into the PACS. The management and the access to these large image repositories become increasingly complex. Most accesses to these systems are based on the patient identication or study characteristics (modality, study description) [85] as it is also dened in the DICOM standard [86]. Imaging systems and image archives have often been described as an important economic and clinical factor in the hospital environment [8789]. Several methods from the computer vision and image processing elds have already been proposed for the use in medicine more than ten years ago [90,91]. Several radiological teaching les exist [92,93] and radiology reports have also been proposed in a multimedia form in [94]. Web-interfaces to medical image databases are described in [95]. Medical images have often been used for retrieval systems and the medical domain is often cited as one of the principal application domains for content-based access technologies [7,18,9698] in terms of potential impact. Still, there has rarely been an evaluation of the performance and the description of the clinical use of systems is even rarer. Two exceptions seem to be the Assert10 system on the classication of high resolution CTs of the lung [40,99] and the IRMA11 system for the classication of images into anatomical areas, modalities and view points [100]. Content-based retrieval has also been proposed several times from the medical community for the
10 http://rvl2.ecn.purdue.edu/cbirdev/www/CBIRmain. html. 11 http://irma-project.org/.

1.5. Other important techniques

There is a large number of other important techniques to improve the performance of retrieval systems. One of the most prominent techniques is relevance feedback that is well known from text retrieval [77]. This technique has proven to be important for image retrieval as well [7880] because often unexpected or unwanted images show up in the result of a similarity query. The active selection of relevant and unrelevant images by the user represents an interactive method for controlling the pertinence of the results adequately. Often, the performance of a retrieval system with feedback is regarded as being even more important than without as only with feedback the users subjectivity can seriously be taken into account. An overview of interaction techniques in image retrieval is given in [81]. Other techniques from the articial intelligence community are also used for image retrieval such as long-term learning from user behavior based on data mining in usage log les [82] using the well-known market basket analysis. Some interesting and innovative user interfaces are described in [83,84]. This includes a

H. Mller et al. tained in the DICOM headers and many imaging devices are DICOM-compliant at this time, there are still some problems. DICOM headers have proven to contain a fairly high rate of errors, for example for the eld anatomical region, error rates of 16% have been reported [105]. This can hinder the correct retrieval of all wanted images. Clinical decision support techniques such as case-based reasoning [106] or evidence-based medicine [107,108] can even produce a stronger need to retrieve images that can be valuable for supporting certain diagnoses. It could even be imagined to have image-based reasoning (IBR) as a new discipline for diagnostic aid. Decision support systems in radiology [109] and computer-aided diagnostics for radiological practice as demonstrated at the Radiological Society of North America (RSNA) [110] are on the rise and create a need for powerful data and meta-data management and retrieval. The general clinical benet of imaging systems has also already been demonstrated in [111]. In [112] an initiative is described to identify important tasks for medical imaging based on their possible clinical benets. It needs to be stated that the purely visual image queries as they are executed in the computer vision domain will most likely not be able to ever replace text-based methods as there will always be queries for all images of a certain patient, but they have the potential to be a very good complement to text-based search based on their characteristics. Still, the problems and advantages of the technology have to be stressed to obtain acceptance and use of visual and text-based access methods up to their full potential. A scenario for hybrid, textual and visual queries is proposed in the CBIR2 system [113]. Besides diagnostics, teaching and research especially are expected to improve through the use of visual access methods as visually interesting images can be chosen and can actually be found in the existing large repositories. The inclusion of visual features into medical studies is another interesting point for several medical research domains. Visual features do not only allow the retrieval of cases with patients having similar diagnoses but also cases with visual similarity but different diagnoses. In teaching, it can help lecturers as well as students to browse educational image repositories and visually inspect the results found. This can be the case for navigating in image atlases.12 It can also be used to cross-correlate visual and textual features of the images.
12

Fig. 2 A screenshot of a typical image retrieval system showing retrieved images similar to an example in a web browser interface.

inclusion into various applications [101103], often without any implementation. Still, for a real medical application of content-based retrieval methods and the integration of these tools into medical practice a very close cooperation between the two elds is necessary for a longer period of time and not simply an exchange of data or a list of the necessary functionality. An interface of a typical content-based retrieval system is shown in Fig. 2. The interface shows the images retrieved with their similarity score to an example image. The user can then mark images as relevant, non-relevant or leave them as neutral, change the parameters for retrieval and start a new query for renement.

2.1. The need for content-based medical image retrieval

There are several reasons why there is a need for additional, alternative image retrieval methods apart from the steadily growing rate of image production. It is important to explain these needs and to discuss possible technical and methodological improvements and the resulting clinical benets. The goals of medical information systems have often been dened to deliver the needed information at the right time, the right place to the right persons in order to improve the quality and efciency of care processes [104]. Such a goal will most likely need more than a query by patient name, series ID or study ID for images. For the clinical decision-making process it can be benecial or even important to nd other images of the same modality, the same anatomic region of the same disease. Although part of this information is normally con-

http://www.loni.ucla.edu/MAP/index.html.

Content-based image retrieval systems

2.2. The use in PACS and other medical databases

There is a large number of propositions for the use of content-based image retrieval methods in the medical domain in general [101103]. Other articles describe the use of image retrieval with an image management framework [114119], sometimes without stating what has actually been implemented and what is still in the status of ideas. Also the integration into PACS systems [85,120123] or other medical image databases [92,124126] has been proposed often, but implementation details are generally rare. Most of the general articles such as [101] state that the medical domain is very specialized so that general systems cannot be used. This is true but it is the case for all specialized domains such as trademark retrieval or face recognition, and specialized solutions need to be found. The more specialized the features of a system are the smaller the range of application and compromises for each specic application area needs to be found. Domain knowledge needs to be integrated into specialized query engines. Another proposition of what is needed for an efcient use in the medical domain is given in [102], including some implementation details. Clinically relevant indexing and selective retrieval of biomedical images is explained in [103]. Some examples are given but no implementation details. It is proposed to change the DICOM headers which is in principal not allowed according to the standard for the storage of DICOM images, but would, however, be allowed in DICOM structured reporting. Most of these articles ask for semantic retrieval based on images that are segmented automatically into objects and where diagnoses can be derived easily from the objects visual features. This is still a dream, as it has been in the computer vision domain for general segmentation methods for a while. Steps into the direction of solutions have to be taken using machine learning techniques and by including specic domain knowledge. Implementations of image retrieval systems are a step-by-step process and rst systems will denitely not meet all the high requirements that are asked for. Several frameworks for distributed image management solutions have been developed such as I2 Cnet [98,115]. When reading articles on these frameworks it is often not clear what had and had not been implemented. Image retrieval based on visual features is often proposed but unfortunately nothing is said about the visual features used or the performance obtained. [117] describes a telemedicine and image management framework

Fig. 3 The basic position of a PACS within the information system environment in a hospital.

and [114] is another very early article on the architecture of a distributed multimedia database. [127] describes an active index for medical image data management, and in [116] a newer image management environment is described. In [118,119], two frameworks for image management and retrieval are described focusing on technical aspects and stating application areas. One of the few frameworks with at least a partial implementation is the image retrieval in medical applications (IRMA) framework [100,128] that allows for a relatively robust classication of incoming images into anatomical regions, modality and the taken orientation. This project also developed a classication code for medical images based on four axes (modality, body orientations, body region, biological system) to uniquely classify medical images and allow to test and measure the performance of classication [129]. The use of content-based techniques has been proposed several times in a PACS environment. PACS are the main software components to store and access the large amount of visual data used in medical departments. Often, several-layer architectures exist for quick short-term access and slow long-term storage. More information on PACS can be found in [130]. A web-based PACS architecture is proposed in [131]. The general schema of a PACS system within the hospital is shown in Fig. 3. The Integrating the Healthcare Enterprise (IHE)13 standard is aiming at data integration in healthcare including all the systems described in Fig. 3. An indexing of the entire PACS causes problems with respect to the sheer amount of data that needs to be processed to efciently allow access by content to all the images. This issue of the amount of data that needs to be indexed is not discussed in any of the articles. [122] proposes to use content-based image retrieval techniques in a PACS system as a search method but no implementation
13

http://www.rsna.org/IHE/index.shtml.

8 details are given. In [120] an integration into the PACS is described that uses the text attached to the images as content. More on this IDEM project can be found at14 [123] proposes an extension to the database management system for integrating content-based queries based on simple visual features into PACS systems. A classication of systems is given in [121] proposing an integration into the PACS, but no implementation details are stated in the text. A coupling of a PACS and an image classication system is given in [85]. Here, it is possible to search for certain anatomic regions, modalities or views of an image. A simple interface for coupling the PACS and the image retrieval system is stated as well. The identication is based on the DICOM unique identier (UIDs) of the images. Still, there is lack of publications describing the integration of image retrieval into the workow in a medical institution and visual knowledge management in a learning institution has not been the subject of publications either. Besides the use directly within a PACS system or very general image database environment, content-based image retrieval has also been used or proposed in a couple of specialized collections. In [92], CBIR is proposed in the context of a case database containing images and attached case descriptions. [124] describes the use in a medical reference database and [132] the use within a teaching le assistant. An object-oriented approach to store and access medical databases is given in [126]. But it remains unclear what kind of visual features are supposed to be used. In [133] an online pathology atlas uses the search-by-similarity paradigm. Decision support systems are another application of content-based medical image retrieval [134]. In [135] access-control models for content-based retrieval are discussed. It can be seen that the number and sort of applications is large and diverse, and the techniques used or proposed for an implementation contain a variety almost as large as for general image retrieval.

H. Mller et al. A categorization of images from various departments has been described in [54,100]. A classication of dermatologic images is explained in [75,136,137]. Cytological specimens have already been described very early (in 1986, [138]) and also later on [139] whereas the search for 3D cellular structures followed later on [96]. Pathology images have often been proposed for content-based access [43,140] as the color and texture properties can relatively easy be identied. The tasks of a pathologist when searching for reference cases also supports the use of an image retrieval system instead of only reference books. The use with tuberculosis smears is described in [141]. An application with histopathologic images is described in [142] and histologic images are analyzed in [134,143,144]. Within cardiology, CBIR has been used to discover stenosis images [97]. MRIs of the heart have been used in [145]. Within the radiology department, mammographies are one of the most frequent application areas with respect to classication and content-based search [146149]. The negative psychological effects of removing tissue for false positive patients have been described of one of the principal goals to be reduced. Ultrasound images of the breast are used in [41]. Varied ultrasound images are used in [150]. Another active area is the classication of high resolution computed tomography (HRCT) scans of the lung as done by the Assert project [151,152]. A study about the diagnostic quality with and without using the system showed a signicant improvement of the diagnostic quality with using a retrieval system for nding similar cases [99]. A less sophisticated project also using HRCT lung images is described in [125,132]. A justication of use in this area is the hard decision-making task and the strong dependence of the diagnoses from texture properties. Descriptions of HRCT lung images, their visual features and their pathologies are given in [153,154]. The use of thorax radiographies is proposed in [110]. This will be an even harder task as several layers are superposed and many factors other than the pathology can inuence the visual content strongly. Many other articles use medical images to demonstrate their algorithms but a clinical evaluation of their use has rarely been done. In [53,54,155], magnetic resonance images (MRIs) of the brain are used to demonstrate the image search algorithms but the articles do not talk about any medical integration. [115,156] also use MRIs of the head for testing their algorithms. CT brain scans to classify lesions are used in [157]. The search for medical tumors by their shape properties (after segmentation) have

2.3. The use in various medical departments

The same variety that exists with respect to proposed applications exists also with respect to the medical departments where the use of content-based access methods has been implemented or proposed. Obviously, most applications are centered around images produced in radiology departments, but there are also several other departments where CBIRSs have been implemented.
http://www.hbroussais.fr/Broussais/InforMed/IDEM/ InterrogerBase.html.
14

Content-based image retrieval systems

9 Machine learning in medical applications also gets increasingly more important and it is essential to research the various possibilities. Specialized workshops exist for this area [164].

are using these images Images used HRCTs of the lung Functional PET Spine X-rays Pathologic images CTs of the head Mammographies Images from biology Dermatology Breast cancer biopsies Varied images

Table 1 Various image types and the systems that

Names of the systems ASSERT FICBDS CBIR2, MIRS IDEM, I-Browse, PathFinder, PathMaster MIMS APKS BioImage, BIRN MELDOQ, MEDS BASS I2 C, IRMA, KMed, COBRA, MedGIFT, ImageEngine

3.1. Features used

This section describes the (visual) features that are used in the various applications. The section text is added to discuss whether this should be named content-based retrieval or rather not. As the formulation of similarity queries without text can be quite a problem, another subsection is added to describe the various possibilities to formulate queries without text. 3.1.1. Query formulation The query formulation with using exclusively visual features can be a big problem. Most systems in CBIR use the query by example(s) (QBE) paradigm which needs an appropriate starting image for querying. This problem of a sometimes missing starting image is known as the page zero problem. If text is attached to the images, which is normally the case in medical applications, then the text can be used as a starting point and once visually relevant images have been found, further queries can be entirely visual [115] to nd visually similar cases not able to be found by text or to sort the found cases by their visual similarity. In the medical decision-making process, there are often images produced and available for the current case. The starting point does thus not need to be further dened but the images of the case can be used directly [121]. In connection with the segmentation of the images the user can also restrict the query to a certain region of interest (ROI) in the image [121], which can lead to much more specic queries than if using an image in its entirety. The use of human sketches has already been proposed in generic image retrieval [33,165] and it has also been proposed for the use in medical applications [113,115,121,166]. Considering the difculty in exact drawing and the need for some artistic skills and time, this method will only be applicable for a very small subset of queries, such as tumor shapes or spine X-rays, where outlines are possible directly in the image. For general image retrieval, sketches are too time-consuming and the retrieved results often not exact enough. 3.1.2. Text Many systems propose to use text from the patient record [120] or studies [121] to search by content. Others dene a context-free grammar [97], a standardized vocabulary for image description [142] or

been described in [147]. Functional photon emission tomography (PET) images for retrieval are used in [158]. Spine X-rays are used in [113,159]. Table 1 shows an overview of several image types and the systems that are used to retrieve these images.

2.4. The use in elds close to medicine

There is a number of elds close to the medical domain where the use of content-based access methods to visual data have been proposed as well or are already implemented. In the USA, a biomedical research network is about to be set up, and the sharing of visual data and their management include the use of similarity queries [160]. Multidimensional biological images from various devices are handled in the BioImage project [161]. In [162] drug tablets are retrieved by their visual similarity which is mainly for the identication of ecstacy tablets. Another pharmaceutical use is described in [163] where powders are retrieved based on visual properties.

3. Techniques used in medical image retrieval

This section describes the various techniques that are currently-used or that have been proposed for the use in medical image retrieval applications. Many of the techniques are similar to those used for general content-based retrieval but also techniques that have not yet been used in medical applications are identied. A special focus is put on the data sets that are used to evaluate the image retrieval systems and on the measurements used for evaluation. Unfortunately, the performance evaluation of systems is currently strongly neglected.

10 an image denition language [126] for the querying of images in image repositories. [167,168] uses text from radiology reports to transform it into concepts in the UMLS metathesaurus to then retrieve the images. The use of text for queries is undeniable efcient but the question is whether this can really be called content-based queries as the text does not necessarily dene the image content. It rather puts the images into the context they have been taken in, so it should maybe called context-based queries as dened in [67]. The combination of textual with visual features or content and context of the images does have the most potential to lead to good results [113]. One can also be used to control the quality of the other or to obtain a better recall of the retrieval results. Besides the free text that is frequently used for retrieval, medical patient records also contain very valuable structured information such as age, sex and profession of the patient. This information is just as important as free text to put the images into a context. 3.1.3. Visual features Unfortunately, most articles that propose contentbased queries do not explain in detail which visual features have been used or are planned to be used. Sometimes, only a very vague description such as general texture and color or grey level features are given as in [54,127,169]. Basically all systems that do give details use color and grey level features, mostly in the form of a histogram [134,143,150,151]. Local and global grey level features are used in [170]. [100,128] use statistical distributions of grey levels for the classication of images and [122] proposes a brightness histogram. As many of the images in the medical domain do not contain colors or are taken under controlled conditions, the color properties are not at all in the center of research and the same holds for invariants to lighting conditions. This can change when using photographs such as in dermatology. Pathologic images will need to be normalized in some way as different staining methods can produce different colors [171]. Within radiology, the normalization of grey levels between different modalities or even for the same modality can cause problems when there is no exact reference point as is for the density of the CT, for example. [172] illustrates the dependency of intensity values of the brain from the used modalities. As color and grey level features are of less importance in medical images than in stock photography, the texture and shape features gain in importance. Basically all of the standard techniques for texture characterization are used from edge detection

H. Mller et al. using Canny operators [141] to Sobel descriptors [151]. [113,139,151] also use Fourier descriptors to characterize shapes, [113,123,139] use invariant moments and [113] also scale-space ltering. Features derived from co-occurrence matrices are also frequently used [96,115,150,151], as well as responses of Gabor lters [134,143,170], wavelets [140,150] and Markov texture characteristics [139]. In mammography, denseness is used for nding small nodules [148]. It would be interesting to have a comparison of several texture descriptors. Many of them model the same information and will most likely deliver very similar results. In connection with segmentation, the shape of the segments can be used as a powerful feature. Again, often the exact nature of the shape features is not described [115] which makes it impossible to dene what exactly had been used. In [145] no segmentation has been done for the acquisition of shape features but computer-assisted outlining. The segmentation of pathologic images is described in [140]. In [156] even shape descriptors for 3D structures using modal modeling are described. Most common shape descriptors are Fourier descriptors [43,132,141] that easily allow to obtain invariant descriptions. The pattern spectrum is proposed in [147] and morphological features are used in [147]. Using segments in the images also allows to use spatial relationships as visual descriptors of the images. This is often proposed [114,116,121,169,173] but rarely any detail is given on how to obtain the objects/segments in the images, which does not permit to judge whether an implementation is possible. Another article not taking into account the problems of automatic segmentation is [116]. The use of Eigenimages for the retrieval of medical images in analogy to Eigenfaces for face recognition is proposed in [74,124]. These features can be used for classication when a number of images for each class exist. Still, the features are purely statistical and it is hard to actually explain the similarity of two images based on these features which can more easily be done for a histogram intersection, for example. In [121], signatures of the manually segmented objects of the images are proposed to reduce the list of resulting images. It is hard to say whether these features can count as visual features as they are not extracted automatically but based on semiautomatic segmentations and marking of the segments. Tissue timeactivity curve (TTAC) curves for the retrieval of PET images are used in [158]. These are not really image features but rather 1D

Content-based image retrieval systems temporal signals that are compared. However, the results seem to be good. Similar to general CBIR, semantic features are proposed for visual similarity queries with medical images [143,144]. But again, it comes down to simple textual labels attached to the images and a mapping between the text and the low-level features. A project for automatically attaching semantic labels to images or regions is described in [134] and in ProjetImage.15

11 more than 25,000 images in [92] is the spectrum of the articles analyzed for this review. Often, the images are pre-processed into sometimes fairly small blocks (18 12, [166], 64 64, [134] and 256 256, [170]) before the visual features are extracted. In some cases, even a reduction to 32 32 pixels has proven not to inuence the quality of the results compared with using the original size [100]. Some systems use pre-processing to remove artifacts from the image or to improve image quality such as the removal of hairs from dermatologic images [75]. Unfortunately, most of the larger databases such as [124] containing 10,000 MRI images, [116,173] containing 13,500 CT and MRI images and [147] using 1,000 tumor shapes only use simulated images. Although these simulated images are easy and cheap to obtain, their use for any qualitative assessments is more than questionable. On the other hand, only [121] uses a very large database containing 22,000 images of a PACS but without any further assessment of image categories and qualities and without an evaluation. [159] uses 17,000 spinal X-ray images as the basis of their research [92] proposes even more images, but here as well, no content-based access mechanisms are implemented as of yet. An interesting approach to obtain a large database is taken in [54], where 2000 images from freely available medical image databases on the web are taken. A database containing more than 8000 varied medical images is available free of charge from the casimage webpage or can be ordered from the author of this article.16 [123] uses a varied set of 4247 medical images. The other, often specialized image collections for content-based retrieval are unfortunately sometimes too small for delivering any statistically signicant measurements: [158] uses 15 PET studies, [149] 41 biopsy slides, [157] 48 brain CTs, [167] 50 varied images with radiology reports, [141] 65 smears for tuberculosis identication, [145] 85 MRI images, [74] 100 axial brain images, [75] 100 dermatologic images, [43] 261 cell images, [41] 263 ultrasound breast images, [132] 266 CT images and [96] 300 cell images. 312 HRCTs of the lung are used in [151,152], 345 liver disorders in [175], 404 biopsy proven mammography masses in [148] and 749 dermatological images in [136]. Almost as interesting as the image database itself is the question of how to choose query topics and then how to assess relevance for the query topics. The subject of relevance alone can ll several books [176,177]. This is relatively easy for simulated images as there is a model plus some added
16

3.2. Comparison methods and feature space reductions

Most systems do not give many details on the distance measurements or comparison methods used which most likely implies an Euclidian vector space model using either a simple Euclidean distance (L2) or something close such as city-block distance or L1. To efciently work with these distances even in large databases, the dimensionality is often reduced. This can be done with methods such as principal component analysis (PCA) [74,124] or minimum description length (MDL) [151] that try to reduce the dimensionality while staying as discriminative as possible. In principle, redundant information is removed but this can also remove small but important changes from the feature space. Techniques such as KD-trees [145] and R-trees [173] are also used in medicine for efcient access to such a large feature spaces. On the other hand, statistical methods are used for the comparison of features that can be trained with existing data and that can then be used on new, incoming cases. These can be neural networks for the classication of mammography images [63,148] or on images extremely reduced in size (18 12 pixels) in [166]. Other statistical approaches use Bayesian networks [157] or Hidden Markov Models (HMMs) [96]. In [174], an associative computing approach is proposed for retrieval assuming that a query is performed with a local part of the images. A receiver operating characteristic (ROC) curve for the comparison of methods is used in [136]. This is well known in the medical domain and easily interpretable.

3.3. Image databases used for evaluation

The data used for demonstrating the capabilities of the visual access methods are extremely varied in size and quality. From 15 PET studies in [158] to
15

http://perso-iti.enst-bretagne.fr/brunet/Boulot/ ProjetImage/ProjetImage.html.

http://www.casimage.com/.

12 noise and the noise level basically determines the measured quality of retrieval. Simulated images are consequently only usable for showing efciency of an algorithm using large image repositories. Nothing can really be said about retrieval quality when using simulated images. For the future, it is extremely important that image databases are made available free of charge and/or copyright for the comparison and verication of algorithms. Only such reference databases allow to compare systems and to have a reference for the evaluation that is done based on the same images. Some medical image collections are freely available on the Internet.17 18 19 20 An important effort is underway by the European Federation of Medical Informatics (EFMI) in a working group on medical image processing21 to generate reference databases and identify important medical imaging tasks [112].

H. Mller et al. In medical statistics commonly used measurements are sensitivity and specicity dened as follows: sensitivity = specicity = positive items classied as pos. all positive items negative items classied as neg. all negative items (1) (2)

Systems that use sensitivity and specicity include [41,136,141]. These values can also be presented in the form of a ROC curve which contains much more information and is done in [136,157]. As many of the presented systems use classications of images, accuracy is very often used to evaluate the system [96,100,125,141,143,146]. This can be dened as follows: accuracy = items classied correctly all items classied (3)

3.4. System evaluations

Already in the general image retrieval domain it is difcult to compare any two retrieval systems. For medical image retrieval systems, the evaluation issue is almost non-existent in most of the papers [54,102,114,115,118120,126,127,174]. Still, there are several articles on the evaluation of imaging systems in medicine [111] or on general evaluation of clinical systems and the problems with it [178]. Those systems that do perform evaluation often only use screenshots of example results to queries [121124,145,149,169]. A single example result does not reveal a great deal about the real performance of the system and is not objective as the best possible query result can be chosen arbitrarily by the authors. This problematic in retrieval system evaluation is described in detail in [179]. Most other system evaluations show measures with a limited power for comparison. In [151], the precision of the four highest ranked images is used which does not reveal much about the number of actually relevant items and gives very limited information about the system. [74] measures the number of times a differently scaled or rotated image retrieves the original which is also not very close to medical image retrieval reality.

Still, it has to be kept in mind that content-based retrieval systems are not mainly being employed for classication of the images but for nding similar images or cases. This is often more helpful as the practitioner must still judge the retrieved cases and the reasons for retrieving the images are often clearer whereas classication results are sometimes hard to detail and need to be explained. Only rarely are measurements used that are common to the domains of information retrieval [180] or content-based image retrieval [179] such as precision and recall dened as follows: precision = recall = number of relevant items retrieved (4) number of items retrieved (5)

number of relevant items retrieved number of relevant items

17 http://marathon.csee.usf.edu/Mammography/Database. html. 18 http://cir.ncc.go.jp/pub/gmain.html. 19 http://www.meduniv.lviv.ua/links/index multimedia.html. 20 http://wwwihm.nlm.nih.gov/. 21 http://www.efmi-wg-mip.net/.

In [140], for example, the precision after 50 images are retrieved is measured to describe the system performance [123] mentions precision and recall for the evaluation but then, does not use it. [116] uses the precision at ve different cutoff points. These data are incomplete and hard to interpret as little is known about the number of relevant images and thus on the difculty of the query task. Much better is the use of a precision vs. recall graph that puts the two values on the axis of a graph as in [147]. Another rarely mentioned evaluation parameter is the speed of the system which is very important for an interactive system. In [123] it is only mentioned that the speed is reduced from hours to minutes for a set of 4000 images which is completely insufcient for an interactive system where response times should be around one second at a maximum.

Content-based image retrieval systems This list with few in depth evaluations shows that evaluation is very often neglected in medical image retrieval. It is extremely important and crucial for the success of this technology. Measurement parameters need to show the usefulness of an application and the possible impact that an application of the method can have. Such an evaluation does not only contain the validation of a technology which is commonly evaluated with measures such as specicity and sensitivity but also the inclusion of human factors into the process such as usability issues and acceptance of the technology [178], which can be obtained through real user tests. Finally, it will be interesting to evaluate the clinical impact of the application when it is used in real clinical practice. Are these technologies able to reduce the length of stay of patients or do they manage to reduce the use of human resources for the patient care? Studies on clinical effects of image retrieval technologies might still be a distance away but there are several necessities that can be done at the moment such as the denition of standard databases that are freely available, the denition of query topics for these databases including the creation of a gold standard or ground truth for these topics. This can, in the long run, make way for real clinical studies once the general retrieval performance is proven.

13 Another domain not discussed at all for medical images are the user interfaces. Sometimes web-based interfaces are proposed [170,182] but no comparison of interfaces is reported and no real usability studies have been published to the authors knowledge so far. As there are several creative solutions in image retrieval it will be interesting to study the effects of interfaces, ergonomics and usability issues on the acceptance and use of the technology in clinical practice. Performance comparisons for different feature sets have also never been performed and are important to identify well-performing visual features and the applications that they can successfully be used for. This would help a great deal to start new projects in the domain and also to optimize existing systems.

4. Potential clinical benets and future research

This section gives an overview of the potential application areas of medical image retrieval systems by the image content and the potential clinical benets of it. Some propositions for future research are made that can inuence the research outcome of content-based retrieval methods in the medical domain.

3.5. Techniques not yet used in the medical eld

The preceding subsections already showed the large variability in techniques that are used for the retrieval of images. Still, several very successful techniques from the image retrieval domain have not been used for medical images as of yet. The entire discussion on relevance feedback that rst improved the performance of text retrieval systems and then, 30 years later, of image retrieval systems has not at all been discussed for the medical domain. A few articles mention it but without any details on use and performance. Often the argument for omitting relevance feedback is that medical doctors do not have the time to look at cases and judge them. If the systems are interactive (response times below 1 s, [181]) this should not be a reason as an expert can mark a few images as positive and negative relevance feedback within less than a minute and the improved quality will more than compensate for a minute lost. Also the prospect of long-term learning from this marking of images should motivate people to use it. Long-term learning has shown to be an extremely effective tool for system improvements.

4.1. Application elds in medicine and clinical benets

Already in Section 2.3 it has been shown that content-based retrieval methods are used in a large variety of applications and departments. This section gives a more ordered view on what in medicine image retrieval can be used for and what the effects can be if proper applications are developed. Three large domains can instantly be stated for the use of content-based access methods: teaching, research and diagnostics. Other very important elds are the automatic annotation/codication of images and the classication of medical images. First to benet will most likely be the domain of teaching. Here, lecturers can use large image repositories to search for interesting cases to present to the students. These cases can be chosen not only based on diagnosis or anatomical region but also visually similar cases with different diagnoses can be presented which can augment the educational quality. Indeed, in multiplying the routes to access the right data, cross-correlation approaches between media and various data can be eased. On the other hand, anonymized image archives can be made

14 available for medical students for educational purposes. Content-based techniques allow browsing databases and then comparisons of diagnoses of visually similar cases. Especially for Internet-based teaching, this can offer new possibilities. As most of the systems are based on Internet technologies this does not cause any implementation problems. Research can also benet from visual retrieval methods. Researchers have more options for the choice of cases to include into research and studies by allowing text-based and visual access. It can also be imagined that by including visual features directly into medical studies, new correlations between the visual nature of a case and its diagnosis or textual description could be found. Visual data can also be mined to nd changes or interesting patterns which can lead to the discovery of new knowledge by combining the various knowledge sources. Finally, diagnostics will be the hardest but most important application for image retrieval. To be used as a diagnostic aid, the algorithms need to prove their performance and they need to be accepted by the clinicians as a useful tool. This also implies an integration of the systems into daily clinical practice which will not be an easy task. It is often hard to change the methods that people are used to, condence needs to be won. For domains such as evidence-based medicine or case-based reasoning it is essential to supply relevant, similar cases for comparison. Such retrieval will need special visual features that model the visual detection of an MD using as much domain knowledge as possible. Images are normally taken for a very specic reason and this needs to be modeled. There are two principal ideas for supporting the clinical decision-making process. The rst one is to supply the medical doctor with cases that offer a similar visual appearance. This can supply a second opinion for the MD and (s)he can perform the reasoning based on the various cases that are supplied by the system and the data that is available on the current patient. Another idea is the creation of databases containing normal (non-pathologic) cases and compare the distance of a new case with the existing cases doing thus dissimilarity retrieval as opposed to similarity retrieval (distance to normality). This is even more natural compared to the normal workow in medicine where the rst requirement is to nd out whether the case is pathologic or not. A tumor or fracture are such differences from normal cases, for example. A dissimilarity could be combined with highlighting regions in the image where the strongest dissimilarity occurred. Such a technique can help to nd cases that might otherwise be

H. Mller et al. missed. A combination of the two approaches is also possible where rstly, the requirement is whether the image contains abnormalities and if it does, a query to nd similar cases is done with another image database containing the pathologic cases. High quality annotation/codication is a problem not only in radiology but also in other medical departments. Good annotation and codication takes time and experience that is unfortunately sometimes not available in medical routine. Much research is done on natural language processing techniques to extract diagnoses from the patient record [183] and many tools exist to ease the coding, for example for the American College of Radiology (ACR) codes22 in radiology.23 When large databases of correctly coded images are available, image retrieval systems can be used for semi-automatic coding by retrieving visually similar cases and proposing the codes of the images from the database. Studies will need to prove the quality of the coding but time can be saved even when a medical doctor only has to control the codes that the system is proposing. Retrieval methods can also be used as simple tools to have a quality control on the DICOM headers. The combination of textual and visual attributes denitely promises the best results. In principle, all image-producing departments can prot from content-based technologies but there are some departments and some sorts of images that seem to stand out as textures and colors do play an important role for the diagnostics. Color and texture features are normally easy to index with current retrieval systems. This includes Pathology where microscopic images are analyzed and the clinical decision-making depends on the color changes and textures within the images. Many books with example images for typical or hard cases exist and it is relatively easy to provide these books in a digital form and search for them not only based on text or hierarchies but also based on the visual content. Care needs to be taken with respect to different staining methods. Images need to be normalized with respect to that [171]. Hematology already contains a large number of tools to automatically count blood cells but an interesting application would be the classication of abnormal white blood cells and the comparison of diagnoses between a new case and cases with similar abnormalities stored in the databases. Dermatology already has classication applications for potential melanoma cases that work fairly

22 23

http://www.acr.org/. http://www.casimage.com/ACR.html.

Content-based image retrieval systems well. Content-based access can help to make understand the decision of an expert system to the practitioner. Within the Radiology department there are a number of possible applications that can deliver good results. For HRCTs of the lung, computer-based tools have already been proven to help in the diagnostics process and diagnostics in this case are fairly difcult. Three-dimensional retrieval can also help to retrieve tumor forms and to classify observed tumors. As a tool for the use in PACS systems, a large number of people can prot from the methods to retrieve similar cases for a number of applications, often without realizing that the results come from a content-based retrieval engine.

15 images and the linked case information are stored to allow for retrieval methods based on structured information, free text and the visual image content. Of course, evaluation of the retrieval quality is an extremely important topic as well. Research will need to focus on the development of open test databases and query topics plus dened gold standards for the images to be retrieved. Retrieval systems need to be compared to identify good techniques. This can advance the eld much more than any single technique developed so far. But evaluation also needs to go one step further and prepare eld studies on the use and the inuence of retrieval techniques on the diagnostic process. So far, only one study on the impact of image retrieval system on the diagnostics of HRCT images of the lung has been published and shows a signicant improvement in diagnostic quality even for senior radiologists [99]. Practitioners need to give their opinion on the usability and applicability of the technologies and acceptance needs to be gained before they can be used in daily practice. Such communication with the system users can also improve the interface and retrieval quality signicantly when good feedback is delivered. User interaction and relevance feedback are two other techniques that need to be integrated more into retrieval systems as this can help to lead to much better results. Image retrieval needs to be interactive and all the interaction needs to be exploited for delivering the best possible results. Multimedia data mining will also be made possible once features of good quality are available to describe the images. This will help to nd new relationships among images and certain diseases or it will simply improve the retrieval quality of medical image search engines. Although rst applications will most likely be on large image archives for teaching and research, a specialization of the retrieval systems for promising domains such as dermatology or pathology will be necessary to include as much domain knowledge as possible into the retrieval. This will be necessary for decision-support systems such as systems for case-based reasoning. Such a specialization can be done in the easiest way with a modular retrieval system based on components where feature sets can be exchanged easily and modules for new retrieval techniques or efcient storage methods can be integrated easily. Fig. 4 shows such a component-based architecture where system parts can be changed and optimized easily. Easy plug-in mechanisms for the different components need to be dened. Besides the use of images, system developments also need to put a focus on higher-dimensional data. Already tomographic images contain three

4.2. Future research

When thinking about future research directions it becomes apparent that the goal needs to be a real clinical integration of the systems. This implies a number of changes in the ways that research is done at the moment. It will become more important to design applications in a way that they can be integrated easier into existing systems through open communication interfaces, for example based on extensible markup language (XML) as a description language of the data or Hyper-Text Transport Protocol (HTTP) as a transport protocol for the data [184]. Such a use of standard Internet technologies can help for the integration of retrieval methods into other applications. Such access methods are necessary to make the systems accessible to a larger group of people and applications and to gain experience that goes far beyond a validation of retrieval results. This can not only be seen as engineering but as research as the practical use of the integrated methods needs to be researched. The integration into PACS is an essential step for the clinical use of retrieval systems. PACS solutions currently allow search by patient and study characteristics and are mainly a storage place for images. A project to allow further search methods in medical image databases based on a standard communication interface is the Medical Image Resource Center (MIRC).24 Here, search by several characteristics, including free-text, is allowed based on a standard platform. The future of PACS or medical image storage systems might be in a separate architecture with a storage component just as PACS systems currently are and an automatic indexing system where important characteristics from the
24

http://mirc.rsna.org/.

16

H. Mller et al. structure and that use open standards, so data can be exchanged with other applications. It needs to become easy to integrate these new functionalities into other existing applications such as a hospital information system (HIS)/radiology information system (RIS)/PACS or other medical image management or viewing software. In this way, it will become much easier to have prototypes running for a sample of users and to get feedback on the clinical use of systems. To get acceptance, it is important to be integrated into the current applications and with interfaces that the users are familiar with. To win acceptance from the users it is also important to show the performance of the systems and to optimize the performance of systems for certain specialized tasks or people. The development of open toolboxes is another important factor for successful applications. Not only do interfaces for the communication with other applications need to be developed, also within the application it is important to stay modular, so parts and pieces can be exchanged easily. This will help to reduce the number of applications developed and will make it possible to spend more time on the important tasks of integration and development of new methods and system optimizations. It is clear that new tools and methods are needed to manage the increasing amount of visual information that is produced in medical institutions. Content-based access methods have an enormous potential when used in the correct way. It is now the time to create medical applications and use this potential for clinical decision-making, research and teaching.

Fig. 4 ment.

A modular schema for retrieval system develop-

dimensions as do video sequences of endoscopy or ultrasound. Tools for retrieval of videos for example by motion parameters do exist for general videos [185,186] but to our knowledge do not exist specialized for the medical domain. Fast scanners also allow for the registration of 4D-data streams such as tomographic images taken over time. Combinations of modalities such as PET/CT scanners or the use of image fusion techniques also create multi-dimensional data that needs to be analyzed and retrieved. Omitting these high-dimensional informations will result in a signicant lack of knowledge.

5. Conclusion
The large number of research publications in the eld of content-based medical image retrieval especially in recent years shows that it is very active and that it is starting to get more attention. This will hopefully advance the eld as new tools and technologies will be developed and performance will increase. Content-based visual information retrieval denitely has a large potential in the medical domain. The amount of visual data produced in medical departments shows the importance of developing new and alternative access methods to complement text. Content-based methods can be used on a large variety of images and in a wide area of applications. Still, much work needs to be done to produce running applications and not only research prototypes. When looking at most current systems, it becomes clear that few to none of them are actually in routine use. An important factor is to build prototypes that are integrated with a hospital-wide communication

6. Summary
This article gives an overview of the currently available literature on content-based image retrieval in the medical domain. It evaluates after a few years of developments the need for image retrieval and presents concrete scenarios for promising future research directions. The necessity for additional, alternative access methods to the currently-used, text-based methods in medical information retrieval is detailed. This need is mainly due to the large amount of visual data produced and the unused information that these data contain, which could be used for diagnostics, teaching and research. The systems described in the literature and published propositions for image retrieval in medicine are critically reviewed and sorted by medical departments, image categories and technologies used. A short overview of nonmedical image retrieval is given as well.

Content-based image retrieval systems The lack of evaluations of the retrieval quality of systems becomes apparent along with the unavailability of large image databases free of charge with dened query topics and gold standards. However, some databases are available, from the National Institutes of Health (NIH), for example. Ideas for creating such image databases and evaluation methods are proposed. Also, several research directions for improving the retrieval quality based on the experiences from other closely related research elds are given in the paper. Possible clinical benets from the use of content-based access methods are described as well as promising elds of applications.

17
[14] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, P. Yanker, Query by image and video content: the QBIC system, IEEE Comput. 28 (9) (1995) 2332. [15] W. Niblack, R. Barber, W. Equitz, M.D. Flickner, E.H. Glasman, D. Petkovic, P. Yanker, C. Faloutsos, G. Taubin, QBIC project: querying images by content, using color, texture, and shape, in: W. Niblack (Ed.), Proceedings of the of SPIE Conference on Storage and Retrieval for Image and Video Databases, vol. 1908, 1993, pp. 173187. [16] J.R. Bach, C. Fuller, A. Gupta, A. Hampapur, B. Horowitz, R. Humphrey, R. Jain, C.-F. Shu, The Virage image search engine: an open framework for image management, in: I.K. Sethi, R.C. Jain (Eds.), Proceedings of the SPIE Conference on Storage & Retrieval for Image and Video Databases IV, vol. 2670, San Jose, CA, USA, 1996, pp. 7687. [17] A. Hampapur, A. Gupta, B. Horowitz, C.-F. Shu, C. Fuller, J. Bach, M. Gorkani, R. Jain, Virage video engine, in: I.K. Sethi, R.C. Jain (Eds.), Proceedings of the of SPIE Conference on Storage and Retrieval for Image and Video Databases V, vol. 3022, 1997, pp. 352360. [18] P.M. Kelly, M. Cannon, D.R. Hush, Query by image example: the Candid approach, in: W. Niblack, R.C. Jain (Eds.), Proceedings of the of SPIE Conference on Storage and Retrieval for Image and Video Databases III, vol. 2420, 1995, pp. 238248. [19] A. Pentland, R.W. Picard, S. Sclaroff, Photobook: tools for content-based manipulation of image databases, Int. J. Comput. Vis. 18 (3) (1996) 233254. [20] W.Y. Ma, Y. Deng, B.S. Manjunath, Tools for texture- and color-based search of images, in: B.E. Rogowitz, T.N. Pappas (Eds.), Proceedings of the of SPIE Conference on Human Vision and Electronic Imaging II, vol. 3016, San Jose, CA, 1997, pp. 496507. [21] C. Carson, M. Thomas, S. Belongie, J.M. Hellerstein, J. Malik, Blobworld: a system for region-based image indexing and retrieval, in: D.P. Huijsmans, A.W.M. Smeulders (Eds.), Proceedings of the Third International Conference On Visual Information Systems (VISUAL99), no. 1614 in Lecture Notes in Computer Science, Springer-Verlag, Amsterdam, The Netherlands, 1999, pp. 509516. [22] S. Belongie, C. Carson, H. Greenspan, J. Malik, Color- and texture-based image segmentation using EM and its application to content-based image retrieval, in: Proceedings of the International Conference on Computer Vision (ICCV98), Bombay, India, 1998, pp. 675682. [23] I.J. Cox, M.L. Miller, S.M. Omohundro, P.N. Yianilos, Target testing and the PicHunter Bayesian multimedia retrieval system, in: Advances in Digital Libraries (ADL96), Library of Congress, Washington, D.C., 1996, pp. 6675. [24] D.M. Squire, W. Mller, H. Mller, T. Pun, Content-based query of image databases: in-spirations from text retrieval, Pattern Recognition Letters, vol. 21, 2000, pp. 11931198 (Selected papers from the 11th Scandinavian Conference on Image Analysis, SCIA99). [25] D.M. Squire, H. Mller, W. Mller, S. Marchand-Maillet, T. Pun, Design and evaluation of a content-based image retrieval system, in: Design & Management of Multimedia Information Systems: Opportunities & Challenges, IDEA Group Publishing, pp. 125151 (Chapter 7), 2001. [26] T. Pfund, S. Marchand-Maillet, Dynamic multimedia annotation tool, in: G. Beretta, R. Schettini (Eds.), Proceedings of the SPIE Photonics West Conference on Internet Imaging III, vol. 4672, San Jose, CA, USA, 2002, pp. 216224. [27] C. Carson, S. Belongie, H. Greenspan, J. Malik, Region-based image querying, in: Proceedings of the 1997

Acknowledgements
The authors would like to thank the reviewers for their comments that helped to improve the quality of this paper.

References
[1] S.-K. Chang, T. Kunii, Pictorial data-base applications, IEEE Comput. 14 (11) (1981) 1321. [2] P.G.B. Enser, Pictorial information retrieval, J. Document. 51 (2) (1995) 126170. [3] A. Gupta, R. Jain, Visual information retrieval, Commun. ACM 40 (5) (1997) 7079. [4] Y. Rui, T.S. Huang, S.-F. Chang, Image retrieval: past, present and future, in: M. Liao (Ed.), Proceedings of the International Symposium on Multimedia Information Processing, Taipei, Taiwan, 1997. [5] J.P. Eakins, M.E. Graham, Content-based image retrieval, Tech. Rep. JTAP-039, JISC Technology Application Program, Newcastle upon Tyne, 2000. [6] C.C. Venters, M. Cooper, Content-based image retrieval, Tech. Rep. JTAP-054, JISC Technology Application Program, 2000. [7] A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain, Content-based image retrieval at the end of the early years, IEEE Trans. Pattern Anal. Machine Intel. 22 (12) (2000) 13491380. [8] H. Mller, User interaction and performance evaluation in content-based visual information retrieval, Ph.D. thesis, Computer Vision and Multimedia Laboratory, University of Geneva, Geneva, Switzerland, June 2002. [9] J.R. Smith, Integrated spacial and feature image systems: retrieval, compression and analysis, Ph.D. thesis, Graduate School of Arts and Sciences, Columbia University, 2960 Broadway, New York, NY, USA, 1997. [10] A. del Bimbo, Visual Information Retrieval, Academic Press, New York, 1999. [11] S.M. Rahman, Design & Management of Multimedia Information Systems: Opportunities & Challenges, Idea Group Publishing, London, 2001. [12] L.H.Y. Tang, R. Hanka, H.H.S. Ip, A review of intelligent content-based indexing and browsing of medical images, Health Informatics J. 5 (1999) 4049. [13] N.-S. Chang, K.-S. Fu, Query-by-pictorial-example, IEEE Trans. Software Eng. SE 6 (6) (1980) 519524.

18
IEEE Conference on Computer Vision and Pattern Recognition (CVPR97), IEEE Computer Society, San Juan, Puerto Rico, 1997, pp. 4251. J.R. Smith, S.-F. Chang, Visualseek: a fully automated content-based image query system, in: Proceedings of the Fourth ACM International Multimedia Conference and Exhibition, Boston, MA, USA, 1996. S. Sclaroff, L. Taycher, M. La Cascia, ImageRover: a content-based browser for the world wide web, in: IEEE Workshop on Content-Based Access of Image and Video Libraries, San Juan, Puerto Rico, 1997, pp. 29. T. Gevers, A.W.M. Smeulders, A comparative study of several color models for color image invariants retrieval, in: Proceedings of the First International Workshop ID-MMS96, Amsterdam, The Netherlands, 1996, pp. 1726. J.-M. Geusebroek, R. van den Boogaard, A.W.M. Smeulders, H. Geerts, Color invariance, IEEE Trans. Pattern Anal. Machine Intel. 23 (12) (2001) 13381350. M. Ortega, Y. Rui, K. Chakrabarti, K. Porkaew, S. Mehrotra, T.S. Huang, Supporting ranked boolean similarity queries in MARS, IEEE Trans. Knowledge Data Eng. 10 (6) (1998) 905925. J. Ze Wang, G. Wiederhold, O. Firschein, S. Xin Wei, Wavelet-based image indexing techniques with partial sketch retrieval capability, in: Proceedings of the Fourth Forum on Research and Technology Advances in Digital Libraries, Washington D.C., 1997, pp. 1324. W. Ma, B. Manjunath, Texture features and learning similarity, in: Proceedings of the 1996 IEEE Conference on Computer Vision and Pattern Recognition (CVPR96), San Francisco, California, 1996, pp. 425430. S. Santini, R. Jain, Gabor space and the development of preattentive similarity, in: Proceedings of the 13th International Conference on Pattern Recognition (ICPR96), IEEE, Vienna, Austria, 1996, pp. 4044. J.G. Daugman, An information theoretic view of analog representation in striate cortex, Computat. Neurosci. 2 (1990) 918. J.G. Daugman, High condence visual recognition of persons by a test of statistical independence, IEEE Trans. Pattern Anal. Machine Intel. 15 (11) (1993) 11481161. R. Milanese, M. Cherbuliez, A rotation, translation and scale-invariant approach to content-based image retrieval, J. Visual Commun. Image Represent. 10 (1999) 186196. J.S. Weszka, C.R. Dyer, A. Rosenfeld, A comparative study of texture measures for terrain classication, IEEE Trans. Sys. Man Cybernetics 6 (4) (1976) 269285. C.-R. Shyu, C.E. Brodley, A.C. Kak, A. Kosaka, A.M. Aisen, L.S. Broderick, ASSERT: A physician-in-the-loop content-based retrieval system for HRCT image databases, Comput. Vis. Image Understand. 75 (12) (1999) 111132. W.-J. Kuo, R.-F. Chang, C.C. Lee, W.K. Moon, D.-R. Chen, Retrieval technique for the diagnosis of solid breast tumors on sonogram, Ultrasound Med. Biol. 28 (7) (2002) 903909. C.-S. Lu, P.-C. Chung, Wold features for unsupervised texture segmentation, in: Proceedings of the 14th International Conference on Pattern Recognition (ICPR98), IEEE, Brisbane, Australia, 1998, pp. 16891693. D. Comaniciu, P. Meer, D. Foran, A. Medl, Bimodal system for interactive indexing and 18 retrieval of pathology images, in: Proceedings of the Fourth IEEE Workshop on Applications of Computer Vision (WACV98), Princeton, NJ, USA, 1998, pp. 7681.

H. Mller et al.
[44] S.T. Perry, P.H. Lewis, A novel image viewer providing fast object delineation for content based retrieval and navigation, in: I.K. Sethi, R.C. Jain (Eds.), Proceedings of the of SPIE Conference on Storage and Retrieval for Image and Video Databases VI, vol. 3312, 1997. [45] A. Winter, C. Nastar, Differential feature distribution maps for image segmentation and region queries in image databases, in: IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL99), Fort Collins, Colorado, USA, 1999, pp. 917. [46] L. Lucchese, S.K. Mitra, Unsupervised segmentation of color images based on -means clustering in the chromaticity plane, in: IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL99), Fort Collins, CO, USA, 1999, pp. 7478. [47] S. Ghebreab, Strings and necklaces on learning and browsing medical image segmentations, Ph.D. thesis, Faculty of computer science, University of Amsterdam, Amsterdam, The Netherlands, July 2002. [48] R.J. Lapeer, A.C. Tan, R. Aldridge, A combined approach to 3D medical image segmentation using marker-based watersheds and active contours: The active watershed method, in: T. Dohi, R. Kikin (Eds.), International Conference on Medical Image Computing and Computer-Assisted Interventions (MICCAI 2002), no. 2488 in Lecture Notes in Computer Science, Springer-Verlag, Tokyo, Japan, 2002, pp. 596603. [49] S. Loncaric, A survey of shape analysis techniques, Pattern Recog. 31 (8) (1998) 9831001. [50] R.C. Veltkamp, M. Hagedoorn, State-of-the-art in shape matching, in: Principles of Visual Information Retrieval, Springer, Heidelberg, 2000, pp. 87119. [51] D. Dori, Cognitive image retrieval, in: A. Sanfeliu, J.J. Villanueva, M. Vanrell, R. Alczar, J.-O. Eklundh, Y. Aloimonos (Eds.), Proceedings of the 15th International Conference on Pattern Recognition (ICPR 2000), IEEE, Barcelona, Spain, 2000, pp. 4245. [52] J.Z. Wang, J. Li, G. Wiederhold, Simplicity: semanticssensitive integrated matching for picture libraries, IEEE Trans. Pattern Anal. Machine Intel. 23 (9) (2001) 117. [53] Y. Liu, A. Lazar, W.E. Rothfus, M. Buzoiano, T. Kanade, Classication-driven feature space reduction for semantic-based neuroimage retrieval, in: Proceedings of the International Symposium on Information Retrieval and Exploration in Large Medical Image Collections (VISIM 2001), Utrecht, The Netherlands, 2001. [54] A. Mojsilovis, J. Gomes, Semantic based image categorization, browsing and retrieval in medical image databases, in: IEEE International Conference on Image Processing (ICIP2000), Rochester, NY, USA, 2000. [55] T.P. Minka, R.W. Picard, Interactive learning using a society of models, in: Proceedings of the 1996 IEEE Conference on Computer Vision and Pattern Recognition (CVPR96), San Francisco, California, 1996, pp. 447 452. [56] C. Jrgensen, Retrieving the unretrievable in electronic imaging systems: emotions, themes and stories, in: B. Rogowitz, T.N. Pappas (Eds.), Proceedings of the of SPIE Photonics West Conference on Human Vision and Electronic Imaging IV, vol. 3644, San Jose, California, USA, 1999. [57] K. Koffka, Principles of Gestalt Psychology, Lund Humphries, London, 1935. [58] A.K. Jain, A. Vailaya, Image retrieval using color and shape, Pattern Recog. 29 (8) (1996) 12331244. [59] A. Tversky, Features of similarity, Psychol. Rev. 84 (4) (1977) 327352.

[28]

[29]

[30]

[31] [32]

[33]

[34]

[35]

[36] [37] [38]

[39] [40]

[41]

[42]

[43]

Content-based image retrieval systems

[60] M.J. Swain, D.H. Ballard, Color indexing, Int. J. Comput. Vis. 7 (1) (1991) 1132. [61] C.C. Aggarwal, A. Hinneburg, D.A. Keim, On the surprising behavior of distance metrics in high dimensional space, in: Proceedings of the International Conference on Database Theory (ICDT 2001), no. 1973 in Lecture Notes in Computer Science, Springer-Verlag, London, UK, 2001. [62] A. Hinneburg, C.C. Aggarwal, D.A. Keim, What is the nearest neighbor in high-dimensional spaces?, in: Proceedings of 26th International Conference on Very Large Databases (VLDB 2000), Cairo, Egypt, 2000, pp. 506516. [63] R. Hanka, T.P. Harte, Curse of dimensionality: classifying large multi-dimensional images with neural networks, in: Proceedings of the European Workshop on Computer-Intensive Methods in Control and Signal Processing (CIMCSP 1996), Prague, Czech Republic, 1996. [64] N. Vasconcelos, A. Lippman, A probabilistic architecture for content-based image retrieval, in: Proceedings of the 2000 IEEE Conference on Computer Vision and Pattern Recognition (CVPR2000), IEEE Computer Society, Hilton Head Island, South Carolina, USA, 2000, pp. 216221. [65] N. Vasconcelos, A. Lippmann, A unifying view of image similarity, in: A. Sanfeliu, J.J. Villanueva, M. Vanrell, R. Alczar, J.-O. Eklundh, Y. Aloimonos (Eds.), Proceedings of the 15th International Conference on Pattern Recognition (ICPR 2000), IEEE, Barcelona, Spain, 2000, pp. 14. [66] K.-S. Goh, E. Chang, K.-T. Cheng, Support vector machine pairwise classiers with error reduction for image classication, in: Proceedings of the ACM Multimedia Workshop on Multimedia Information Retrieval (ACM MIR 2001), The Association for Computing Machinery, Ottawa, Canada, 2001, pp. 3237. [67] T. Westerveld, Image retrieval: content versus context, in: Recherche dInformations Assiste par Ordinateur (RIAO2000) Computer-Assisted Information Retrieval, vol. 1, Paris, France, 2000, pp. 276284. [68] L. Zhu, C. Tang, A. Rao, A. Zhang, Using thesaurus to model keyblock-based image retrieval, in: Proceedings of the second International Conference on Multimedia and Exposition (ICME2001), IEEE Computer Society, IEEE Computer Society, Tokyo, Japan, 2001, pp. 237240. [69] G. Salton, C. Buckley, Term weighting approaches in automatic text retrieval, Inform. Proces. Manage. 24 (5) (1988) 513523. [70] S. Dominich, Formal foundation of information retrieval, in: Proceedings of the Workshop on Mathematical/Formal Methods in Information Retrieval at the International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece, 2000. [71] A.P. de Vries, M.G.L.M. van Doorn, H.M. Blanken, P.M.G. Apers, The Mirror MMDBMS architecture, in: Proceedings of 24th International Conference on Very Large Databases (VLDB98), New York, NY, USA, 1998, pp. 758761. [72] H. Mller, D.M. Squire, W. Mller, T. Pun, Efcient access methods for content-based image retrieval with inverted les, in: S. Panchanathan, S.-F. Chang, C.-C.J. Kuo (Eds.), Proceedings of the SPIE Conference on Multimedia Storage and Archiving Systems IV (VV02), vol. 3846, Boston, MA, USA, 1999, pp. 461472. [73] D.A. White, R. Jain, Algorithms and strategies for similarity retrieval, Tech. Rep. VCL-96-101, Visual Computing Laboratory, University of California, San Diego, 9500 Gilman Drive, Mail Code 0407, La Jolla, CA 92093-0407, July 1996. [74] U. Sinha, H. Kangarloo, Principal component analysis for content-based image retrieval, RadioGraphics 22 (5) (2002) 12711289.

19
[75] P. Schmidt-Saugeon, J. Guillod, J.-P. Thiran, Towards a computer-aided diagnosis system for pigmented skin lesions, Comput. Med. Imag. Graphics 27 (2003) 65 78. [76] T. Kmpfe, T.W. Nattkemper, H. Ritter, Combining independent component analysis and self-organizing maps for cell image classication, in: B. Radig, S. Florczyk (Eds.), Pattern Recognition: DAGM Symposium 2001 Lecture Notes in Computer Science 2191, Munich, Germanny, 2001, pp. 262268. [77] J.J. Rocchio, Relevance feedback in information retrieval, in: The SMART Retrieval System, Experiments in Automatic Document Processing, Prentice Hall, Englewood Cliffs, NJ, USA, 1971, pp. 313323. [78] H. Mller, W. Mller, D.M. Squire, S. Marchand-Maillet, T. Pun, Strategies for positive and negative relevance feedback in image retrieval, in: A. Sanfeliu, J.J. Villanueva, M. Vanrell, R. Alczar, J.-O. Eklundh, Y. Aloimonos (Eds.), Proceedings of the 15th International Conference on Pattern Recognition (ICPR 2000), IEEE, Barcelona, Spain, 2000, pp. 10431046. [79] Y. Rui, T.S. Huang, M. Ortega, S. Mehrotra, Relevance feedback: a power tool for interactive content-based image retrieval, IEEE Trans.Circuits Sys. Video Technol. 8 (5) (1998) 644655 (special issue on segmentation, description, and retrieval of video content).. [80] Y. Rui, T.S. Huang, S. Mehrotra, Relevance feedback techniques in interactive content-based image retrieval, in: I.K. Sethi, R.C. Jain (Eds.), Proceedings of the SPIE Conference on Storage and Retrieval for Image and Video Databases VI, vol. 3312, 1997, pp. 2536. [81] M. Worring, A.W.M. Smeulders, S. Santini, Interaction in content-based image retrieval: An evaluation of the state of the art, in: R. Laurini (Ed.), Proceedings of the Fourth International Conference On Visual Information Systems (VISUAL2000), no. 1929 in Lecture Notes in Computer Science, Springer-Verlag, Lyon, France, 2000, pp. 2636. [82] H. Mller, D.M. Squire, T. Pun, Learning from user behavior in image retrieval: application of the market basket analysis, Int. J. Comput. Vis. (special issue on content-based image retrieval) vol. 56 (12), pp. 6577, 2004. [83] M. Nakazato, T.S. Huang, 3D MARS: Immersive virtual reality for content-based image retrieval, in: Proceedings of the Second International Conference on Multimedia and Exposition (ICME2001), IEEE Computer Society, IEEE Computer Society, Tokyo, Japan, 2001, pp. 4548. [84] S. Santini, R. Jain, Direct manipulation of image databases, Tech. rep., Department of Computer Science, University of California San Diego, San Diego, CA, November 1998. [85] T.M. Lehmann, M.O. Gld, C. Thies, B. Fischer, M. Keysers, D. Kohnen, H. Schubert, B.B. Wein, Content-based image retrieval in medical applications for picture archiving and communication systems, in: Proceedings of the SPIE Conference on Medical Imaging, vol. 5033, San Diego, CA, USA, 2003. [86] B. Revet, DICOM Cook Book for Implementations in Madalities, Philips Medical Systems, Eindhoven, Netherlands, 1997. [87] R.A. Greenes, J.F. Brinkley, Imaging systems, in: Medical Informatics: Computer Applications in Healthcare, second ed., Springer, New York, 2000, pp. 485538 (Chapter 14). [88] C. Kulikowski, E. Ammenwerth, A. Bohne, K. Ganser, R. Haux, P. Knaup, C. Maier, A. Michel, R. Singer, A.C. Wolff, Medical imaging informatics and medical informatics: opportunities and constraints, Methods Inform. Med. 41 (2002) 183189.

20
[89] M.W. Vannier, E.V. Staab, L.C. Clarke, Medical image archives present and future, in: H.U. Lemke, M.W. Vannier, K. Inamura, A.G. Farman, J.H.C. Reiber (Eds.), Proceedings of the International Conference on Computer-Assited Radiology and Surgery (CARS 2002), Paris, France, 2002. [90] A.P. Sarvazyan, F.L. Lizzi, P.N.T. Wells, A new philosophy of medical imaging, Med. Hypotheses 36 (1991) 327335. [91] T. Pun, G. Gerig, O. Ratib, Image analysis and computer vision in medicine, Comput. Med. Imag. Graphics 18 (2) (1994) 8596. [92] A. Rosset, O. Ratib, A. Geissbuhler, J.-P. Valle, Integration of a multimedia teaching and reference database in a PACS environment, RadioGraphics 22 (6) (2002) 1567 1577. [93] E. Binet, J.H. Trueblood, K.J. Macura, R.T. Macura, B.D. Morstad, R.V. Finkbeiner, Computer-based radiology information system: from oppy disk to CD-ROM, RadioGraphics 15 (1995) 12031214. [94] K. Maloney, C.T. Hamlet, The clinical display of radiologic information as an interactive multimedia report, J. Digital Imag. 12 (2) (1999) 119121. [95] T. Frankewitsch, U. Prokosch, Navigation in medical internet image databases, Med. Informatics 26 (1) (2001) 115. [96] S. Beretti, A. Del Bimbo, P. Pala, content-based retrieval of 3D cellular structures, in: Proceedings of the Second International Conference on Multimedia and Exposition (ICME2001), IEEE Computer Society, IEEE Computer Society, Tokyo, Japan, 2001, pp. 10961099. [97] M.R. Ogiela, R. Tadeusiewicz, Semantic-oriented syntactic algorithms for content recognition and understanding of images in medical databases, in: Proceedings of the second International Conference on Multimedia and Exposition (ICME2001), IEEE Computer Society, IEEE Computer Society, Tokyo, Japan, 2001, pp. 621624. [98] S.C. Orphanoudakis, C.E. Chronaki, S. Kostomanolakis, I2 Cnet: a system for the indexing. storage and retrieval of medical images by content, Med. Informatics 19 (2) (1994) 109122. [99] A.M. Aisen, L.S. Broderick, H. Winer-Muram, C.E. Brodley, A.C. Kak, C. Pavlopoulou, J. Dy, C.-R. Shyu, A. Marchiori, Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment, Radiology 228 (2003) 265270. [100] D. Keysers, J. Dahmen, H. Ney, B.B. Wein, T.M. Lehmann, A statistical framework for model-based image retrieval in medical applications, J. Electronic Imag. 12 (1) (2003) 5968. [101] H.D. Tagare, C. Jaffe, J. Duncan, Medical image databases: a content-based retrieval approach, J. Am. Med. Informatics Assoc. 4 (3) (1997) 184198. [102] H.J. Lowe, I. Antipov, W. Hersh, C. Arnott Smith, Towards knowledge-based retrieval of medical images. the role of semantic indexing, image content representation and knowledge-based retrieval, in: Proceedings of the Annual Symposium of the American Society for Medical Informatics (AMIA), Nashville, TN, USA, 1998, pp. 882886. [103] W.D. Bidgood, B. Bray, N. Brown, A.R. Mori, K.A. Spackman, A. Golichowsky, R.H. Jones, L. Korman, B. Dove, L. Hildebrand, M. Berg, Image acquisition context: procedure description attributes for clinically relevant indexing and selective retrieval of biomedical images, J. Am. Med. Informatics Assoc. 6 (1) (1999) 6175. [104] A. Winter, R. Haux, A three-level graph-based model for the management of hospital information systems, Methods Information Med. 34 (1995) 378396.

H. Mller et al.
[105] M.O. Gld, M. Kohnen, D. Keysers, H. Schubert, B.B. Wein, J. Bredno, T.M. Lehmann, Quality of DICOM header information for image categorization, in: Proceedings of the International Symposium on Medical Imaging, vol. 4685, San Diego, CA, USA, 2002, pp. 280287. [106] C. LeBozec, M.-C. Jaulent, E. Zapletal, P. Degoulet, Unied modeling language and design of a case-based retrieval system in medical imaging, in: Proceedings of the Annual Symposium of the American Society for Medical Informatics (AMIA), Nashville, TN, USA, 1998. [107] A.A.T. Bui, R.K. Taira, J.D.N. Dionision, D.R. Aberle, S. El-Saden, H. Kangarloo, Evidence-based radiology, Acad. Radiol. 9 (6) (2002) 662669. [108] J.-P. Boissel, M. Cucherat, E. Amsallem, P. Nony, M. Fardeheb, W. Manzi, M.C. Haugh, Getting evidence to prescribers and patients or how to make EBM a reality, in: Proceedings of the Medical Informatics Europe Conference (MIE 2003), St. Malo, France, 2003. [109] C.E. Kahn, Articial intelligence in radiology: decision support systems, RadioGraphics 14 (1994) 849861. [110] H. Abe, H. MacMahon, R. Engelmann, Q. Li, J. Shiraishi, S. Katsuragawa, M. Aoyama, T. Ishida, K. Ashizawa, C.E. Metz, K. Doi, Computer-aided diagnosis in chest radiography: Results of large-scale observer tests at the 1996-2001 RSNA scientic assemblies, RadioGraphics 23 (1) (2003) 255265. [111] B. Kaplan, H.P. Lundsgaarde, Toward an evaluation of an integrated clinical imaging system: Identifying clinical benets, Methods Inform. Med. 35 (1996) 221229. [112] A. Horsch, R. Thurmayr, How to identify and assess tasks and challenges of medical image processing, in: Proceedings of the Medical Informatics Europe Conference (MIE 2003), St. Malo, France, 2003. [113] S. Antani, L.R. Long, G.R. Thoma, A biomedical information system for combined content-based retrieval of spine X-ray images and associated text information, in: Proceedings of the Third Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP 2002), Ahamdabad, India, 2002. [114] W.W. Chu, A.F. Crdenas, R.K. Taira, KMED: A knowledgebased multimedia distributed database system, Inform. Sys. 19 (4) (1994) 3354. [115] S.C. Orphanoudakis, C.E. Chronaki, D. Vamvaka, I2 Cnet: content-based similarity search in geographically distributed repositories of medical images, Comput. Med. Imag. Graphics 20 (4) (1996) 193207. [116] E.G.M. Petrakis, Content-based retrieval of medical images, Int. J. Comput. Res. 11 (2) (2002) 171182. [117] M. Tsiknakis, D. Katehakis, C. Orphanoudakis, Stelios, intelligent image management in a distributed PACS and telemedicine environment, IEEE Commun. Magazine 34 (7) (1996) 3645. [118] H.J. Lowe, B.G. Buchanan, G.F. Cooper, J.K. Vreis, Building a medical multimedia database system to integrate clinical information: an application of high-performance computing and communications technology, Bull. Med. Library Assoc. 83 (1) (1995) 5764. [119] S.T. Wong, H.K. Huang, Networked multimedia for medical imaging, IEEE Multimedia Magazine AprilJune (1997) 2435. [120] C. Le Bozec, E. Zapletal, M.-C. Jaulent, D. Heudes, P. Degoulet, Towards content-based image retrieval in HIS-integrated PACS, in: Proceedings of the Annual Symposium of the American Society for Medical Informatics (AMIA), Los Angeles, CA, USA, 2000, pp. 477 481.

Content-based image retrieval systems

[121] E. El-Kwae, H. Xu, M.R. Kabuka, Content-based retrieval in picture archiving and communication systems, J. Digital Imag. 13 (2) (2000) 7081. [122] H. Qi, W.E. Snyder, Content-based image retrieval in PACS, J. Digital Imag. 12 (2) (1999) 8183. [123] J.M. Bueno, F. Chino, A.J.M. Traina, C.J. Traina, P.M. Azevedo-Marques, How to add content-based image retrieval capacity into a PACS, in: Proceedings of the IEEE Symposium on Computer-Based Medical Systems (CBMS 2002), Maribor, Slovenia, 2002, pp. 321326. [124] G. Bucci, S. Cagnoni, R. De Domicinis, Integrating content-based retrieval in a medical image reference database, Comput. Med. Imag. Graphics 20 (4) (1996) 231241. [125] C.-T. Liu, P.-L. Tai, A.Y.-J. Chen, C.-H. Peng, J.-S. Wang, A content-based medical teaching le assistant for CT lung image retrieval, in: Proceedings of the IEEE International Conference on Electronics, Circuits, Systems (ICECS 2000), Jouneih-Kaslik, Lebanon, 2000. [126] C. Traina, Jr., Agma J.M. Traina, R.R. dos Santos, E.Y. Senzako, A support system for content-based medical image retrieval in object oriented databases, J. Med. Sys. 21(6) (1997) 339352. [127] S.-K. Chang, Active index for content-based medical image retrieval, Comput. Med. Imag. Graphics 20 (4) (1996) 219229. [128] M.O. Gld, B.B. Wein, D. Keysers, C. Thies, M. Kohnen, H. Schubert, T.M. Lehmann, A distributed architecture for content-based image retrieval in medical applications, in: Proceedings of the International Conference on Enterprise Information Systems (ICEIS 2001), Setbal, Portugal, 2001, pp. 299314. [129] T.M. Lehmann, H. Schubert, D. Keysers, M. Kohnen, B.B. Wein, The irma code for unique classication of medical images, in: Proceedings of the SPIE Conference on Medical Imaging, vol. 5033, San Diego, CA, USA, 2003. [130] H.U. Lemke, PACS developments in Europe, Comput. Med. Imag. Graphics 27 (2002) 111120. [131] J. Zhang, J. Sun, J.N. Stahl, PACS and web-based image distribution and display, Comput. Med. Imag. Graphics 27 (2/3) (2003) 197206. [132] C.-T. Liu, P.-L. Tai, A.Y.-J. Chen, C.-H. Peng, T. Lee, J.-S. Wang, A content-based CT lung retrieval system for assisting differential diagnosis images collection, in: Proceedings of the second International Conference on Multimedia and Exposition (ICME2001), IEEE Computer Society, IEEE Computer Society, Tokyo, Japan, 2001, pp. 241244. [133] L. Zheng, A.W. Wetzel, Y. Yagi, M.J. Becich, A graphical user interface for content-based image retrieval engine that allows remote server access through the Internet, in: Proceedings of the Annual Symposium of the American Society for Medical Informatics (AMIA), Nashville, TN, USA, 1998. [134] L.H. Tang, R. Hanka, R. Lan, H.H.S. Ip, Automatic semantic labelling of medical images for content-based retrieval, in: Proceedings of the International Conference on Articial Intelligence, Expert Systems and Applications (EXPERSYS 1998), Virginia Beach, VA, USA, 1998, pp. 7782. [135] S. Tzelepi, D.K. Koukopoulos, G. Pangalos, A exible content and context-based access control model for multimedia medical image database systems, in: Proceedings of the Ninth ACM International Conference on Multimedia (ACM MM 2001), The Association for Computing Machinery, Ottawa, Canada, 2001. [136] R. Pompl, W. Bunk, A. Horsch, W. Stolz, W. Abmayr, W. Brauer, A. Glssl, G. Morll, MELDOQ: Ein System zur Un-

21
tersttzung der Frherkennung des malignen Melanoms durch digitale Bildverarbeitung, in: Proceedings of the Workshop Bildverarbeitung fr die Medizin, Munich, Germany, 2000. A. Sbober, C. Eccher, E. Blanzieri, P. Bauer, M. Cristifolini, G. Zumiani, S. Forti, A multiple classier system for early melanoma diagnosis, Articial Intel. Med. 27 (2003) 29 44. F. Meyer, Automatic screening of cytological specimens, Comput. Vis. Graphics Image Proces. 35 (1986) 356369. M.E. Mattie, L. Staib, E. Stratmann, H.D. Tagare, J. Duncan, P.L. Miller, PathMaster: Content-based cell image retrieval using automated feature extraction, J. Am. Med Informatics Assoc. 7 (2000) 404415. Pathnder: Region-based searching of Pathology Images using IRM. K. Veropoulos, C. Campbell, G. Learnmonth, Image processing and neural computing used in the diagnosis of tuberculosis, in: Proceedings of the Colloquium on Intelligent Methods in Healthcare and Medical Applications (IMHMA), York, UK, 1998. M.-C. Jaulent, C. Le Bozec, Y. Cao, E. Zapletal, P. Degoulet, A property concept frame representation for exible image content retrieval in histopathology databases, in: Proceedings of the Annual Symposium of the American Society for Medical Informatics (AMIA), Los Angeles, CA, USA, 2000. L.H. Tang, R. Hanka, H.H.S. Ip, R. Lam, Extraction of semantic features of histological images for content-based retrieval of images, in: Proceedings of the IEEE Symposium on Computer-Based Medical Systems (CBMS 2000), Houston, TX, USA, 2000. H.Y. Tang, Lilian, R. Hanka, H.H.S. Ip, K.K.T. Cheung, R. Lam, Semantic query processing and annotation generation for content-based retrieval of histological images, in: International Symposium on Medical Imaging, vol. 3976, San Diego, CA, USA, 2000. G.P. Robinson, H.D. Targare, J.S. Duncan, C.C. Jaffe, Medical image collection indexing: shape-based retrieval using KD-trees, Comput. Vis. Graphics Image Proces. 20 (4) (1996) 209217. A.S. Constantinidis, M.C. Fairhurst, A.F.R. Rahman, A new multi-expert decision combination algorithm and its application to the detection of circumscribed masses in digital mammograms, Pattern Recog. 34 (2001) 1527 1537. P. Korn, N. Sidiropoulos, C. Faloutsos, E. Siegel, Z. Protopapas, Fast and effective retrieval of medical tumor shapes, IEEE Trans. Knowledge Data Eng. 10 (6) (1998) 889904. S. Baeg, N. Kehtarnavaz, Classication of breast mass abnormalities using denseness and architectural distorsion, Electronic Lett. Comput. Vis. Image Anal. 1 (1) (2002) 120. F. Schnorrenberg, C.S. Pattichis, C.N. Schizas, K. Kyriacou, Content-based retrieval of breast cancer biopsy slides, Technol. Health Care 8 (2000) 291297. D.-M. Kwak, B.-S. Kim, O.-K. Yoon, C.-H. Park, J.-U. Won, K.-H. Park, Content-based ultrasound image retrieval using a coarse to ne approach, Annals New York Acad. Sci. 980 (2002) 212224. C. Brodley, A. Kak, C. Shyu, J. Dy, L. Broderick, A.M. Aisen, Content-based retrieval from medical image databases: A synergy of human interaction, machine learning and computer vision, in: Proceedings of the 10th National Conference on Articial Intelligence, Orlando, FL, USA, 1999, pp. 760767.

[137]

[138] [139]

[140] [141]

[142]

[143]

[144]

[145]

[146]

[147]

[148]

[149] [150]

[151]

22
[152] C.-R. Shyu, A. Kak, C. Brodley, L.S. Broderick, Testing for human perceptual categories in a physician-in-the-loop CBIR system for medical imagery, in: Proceedings of the IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL99), Fort Collins, CO, USA, 1999, pp. 102108. [153] C. Schaefer-Prokop, M. Prokop, D. Fleischmann, C. Herold, High-resolution CT of diffuse interstitial lung disease: key ndings in common disorders, Europe. Radiol. 11 (2001) 373392. [154] D.M. Hansell, High-resolution CT of diffuse lung disease, Radiol. Clin. North Am. 39 (6) (2001) 10911113. [155] C.Y. Han, H. Chen, L. He, W.G. Wee, A web-based distributed image processing system, in: S. Santini, R. Schettini (Eds.), Proceedings of the SPIE Photonics West Conference on Internet Imaging IV, vol. 5018, San Jose, CA, USA, 2003, pp. 111122. [156] S. Sclaroff, A.P. Pentland, On modal modeling for medical images: Underconstrained shape description and data compression, in: Proceedings of the IEEE Workshop on Biomedical Image Analysis (BIA1994), Seattle, WA, USA, 1994, pp. 7079. [157] Y. Liu, F. Dellaert, Classication-driven medical image retrieval, in: Proceedings of the ARPA Image Understanding Workshop, 1997. [158] W. Cai, D.D. Feng, R. Fulton, Content-based retrieval of dynamic PET functional images, IEEE Trans. Information Technol. Biomed. 4 (2) (2000) 152158. [159] L.R. Long, G.R. Thoma, L.E. Berman, A prototype client/server application for biomedical text/image retrieval on the internet, in: I.K. Sethi, R.C. Jain (Eds.), Proceedings of the Conference on Storage and Retrieval for Image and Video Databases VI, vol. 3312, 1997, pp. 362372. [160] S. Santini, A. Gupta, The role of internet images in the biomedical informatics research network, in: S. Santini, R. Schettini (Eds.), Proceedings of the Conference on Internet Imaging IV, vol. 5018, San Jose, CA, USA, 2003, pp. 99110. [161] J.M. Carazo, H.K. Stelter, The BioImage database project: organizing multidemensional biological images in an object-relational database, J. Struct. Biol. 125 (1999) 97102. [162] Z. Geradts, H. Hardy, A. Poortmann, J. Bijhold, Evaluation of contents based image retrieval methods for a database of logos on drug tablets, in: E.M. Carapezza (Ed.), Proceedings of the Conference on Technologies for Law Enforcement, vol. 4232, Boston, MA, USA, 2000. [163] N. Laitinen, O. Antikainen, J.-P. Mannermaa, J. Yliruusi, Content-based image retrieval: a new promising technique in powder technology, Pharma. Dev. Technol. 5 (2) (2000) 171179. [164] G.D. Magoulas, A. Prentza, Machine learning in medical applications, in: G. Paliouras, V. Karkaletsis, C.D. Spyrpoulos (Eds.), Machine Learning and its Applications, Lecture Notes in Computer Science, Springer-Verlag, Berlin, 2001, pp. 300307. [165] M.J. Egenhofer, Spatial-query-by-sketch, in: Proceedings of the IEEE Symposium on Visual Languages (VL 1996), Boulder, CO, USA, 1996, pp. 667. [166] T. Ikeda, M. Hagiwara, Content-based image retrieval system using neural networks, Int. J. Neural Sys. 10 (5) (2000) 417424. [167] W. Hersh, M. Mailhot, C. Arnott-Smith, H. Lowe, Selective automated indexing of ndings and diagnoses in radiology reports, J. Biomed. Informatics 34 (2001) 262273.

H. Mller et al.
[168] M.M. Wagner, G.F. Cooper, Evaluation of a meta-1-based automatic indexing methods for medical documents, Comput. Biomed Res. 25 (1992) 336350. [169] R. Chbeir, Y. Amghar, A. Flory, MIMS: A prototype for medical image retrieval, in: Recherche dInformations Assiste par Ordinateur (RIAO2000) Computer-Assisted Information Retrieval, vol. 1, Paris, France, 2000. [170] H. Mller, A. Rosset, J.-P. Valle, A. Geissbuhler, Integrating content-based visual access methods into a medical case database, in: Proceedings of the Medical Informatics Europe Conference (MIE 2003), St. Malo, France, 2003, pp. 480485. [171] T. Wringer, J. Stockhausen, D. Meyer-Ebrecht, A. Bcking, Automatic coregistration, segmentation and classication for multimodal cytopathology, in: Proceedings of the Medical Informatics Europe Conference (MIE 2003), St. Malo, France, 2003. [172] A.W. Toga, P. Thompson, An introduction to brain warping, in: Brain Warping, Academic Press, New York, 1998, pp. 128 (Chapter 1). [173] E.G.M. Patrakis, C. Faloutsos, Similarity searching in medical image databases, IEEE Trans. Knowledge Data Eng. 9 (3) (1997) 435447. [174] J.I. Khan, D.Y.Y. Yun, Holographic image archive, Comput. Med. Imag. Graphics 20 (4) (1996) 243257. [175] L. Yu, Kaia nd Ji, X. Zhang, Kernel nearest-neighbor algorithm, Neural Proces. Lett. 15(2) (2002) 147156. [176] T. Saracevis, Relevance: a review of and a framework for the thinking on the notion in information science, J. Am. Soc. Inform. Sci., NovemberDecember (1975) 321343. [177] L. Schamber, M.B. Eisenberg, M.S. Nilan, A re-examination of relevance: toward a dynamic, situational denition, Inform. Proces. Manage. 26 (6) (1990) 755775. [178] T. Brkle, E. Ammenwerth, H.-U. Prokosch, J. Dudeck, Evaluation of clinical information systems. what can be evaluated and what cannot, J. Evaluat. Clin. Practice 7 (4) (2001) 373385. [179] H. Mller, W. Mller, D.M. Squire, S. Marchand-Maillet, T. Pun, Performance evaluation in content-based image retrieval: Overview and proposals, Pattern Recog. Lett. 22 (5) (2001) 593601. [180] G. Salton, The evaluation of computer-based information retrieval systems, in: Proceedings of the 1965 Congress International Federation for Documentation (IFD 1965), Spartan Books Washington, Washington DC, USA, 1965, pp. 125133. [181] J. Nielsen, Usability Engineering, Academic Press, Boston, MA, 1993. [182] T.M. Lehmann, B.B. Wein, H. Greenspan, Integration of content-based image retrieval to picture archiving and communication systems, in: Proceedings of the Medical Informatics Europe Conference (MIE 2003), St. Malo, France, 2003. [183] P. Franz, A. Zaiss, U. Schulz, Stefana nd Hahn, R. Klar, Automated coding of diagnoses three methods compared, in: Proceedings of the Annual Symposium of the American Society for Medical Informatics (AMIA), Los Angeles, CA, USA, 2000. [184] A. Geissbuhler, C. Lovis, A. Lamb, S. Spahni, Experience with an XML/HTTP-based federative approach to develop a hospital-wide clinical information system, in: R. Rogers, R. Haux, V. Patel (Eds.), Proceedings of the International Medical Informatics Conference (Medinfo 2001), London, UK, 2001, pp. 735739. [185] J.R. Smith, VideoZoom: A spatial-temporal video browser for the internet, in: C.-C.J. Kuo, S.-F. Chang, S. Panchanathan (Eds.), Proceedings of the Conference on Mul-

Content-based image retrieval systems

timedia Storage and Archiving Systems III (VV02), vol. 3527, Boston, MA, USA, 1998, pp. 212222. [186] D. Schonfeld, D. Lelescu, VORTEX: Video retrieval and tracking from compressed multimedia databases template matching from MPEG2 video compression standard, in: C.-C.J. Kuo, S.-F. Chang, S. Panchanathan (Eds.), Proceedings of the Conference on Multimedia Storage and

23
Archiving Systems III (VV02), vol. 3527, Boston, MA, USA, 1998, pp. 233244. [187] Anil K. Jain, Robert P.W. Dvin, Jianchang Mao, Statistical Pattern Recognition: A Review, IEEE Transaction on Pattern Analysis and Machin Intelligence, vol. 22 (1), pp. 437, 2000.