ST Journal of System Research

vol 2, number 2 December 2001

Image Processing for Digital Still Camera

An Introduction
to the Digital Still Camera
Technology

Massimo Mancuso,
[email protected]
Sebastiano Battiato

Advanced System Technology - Catania Lab

Digital Still Camera & Multimedia Mobile Group
STMicroelectronics,
Stradale Primosole, 50 95125, Catania

This paper gives an overview of the Digital Still Camera technology.

1. INTRODUCTION On the other hand, the image processing algorithms required

The Digital Still Camera is a challenging application for Image in Digital Still Cameras are very demanding as far as quality
Processing techniques.The basics of the algorithms applied to is concerned due to the HVS characteristics for stills and the
the data provided by the image sensor were first applied in quality already provided by “Analog” or silver-halide still
camcorders. In fact, in the beginning, digital still cameras could cameras.
be considered as a derivative of the camcorder technology, Initially, Digital Still Cameras were strictly derived from
modified to capture still pictures. Camcorders. Consequently, the demand in quality for DSC
But two factors contributed significantly in differentiating the greatly differentiated the 2 systems.Today it is possible to say
two systems: that camcorders benefit from the advances made to enhance
- the Human Visual System (HVS) characteristics; the quality provided by DSC: for example, progressive
- the quality of analog cameras. sensors with Bayer CFA [1] developed for DSC now are
It is well known that motion reduces the Human Visual System used in Digital Camcorder.
sensitivity to the high frequency contents - including artifacts Until recently, the DSC market focused in quality
- of a picture. Consequently, if the system is mainly intended improvement and the reduction of the processing time,
for motion picture capture, the algorithms used can take measured by the number of pictures which can be captured
advantage of that. Generally speaking, the complete system in a second. Quality improvement is obtained by increasing
benefits from the reduced sensitivity of the HVS: simpler the resolution of the sensor and by using more sophisticated
algorithms translate into less hardware, lower power image processing algorithms. Now, with a sensor resolution
consumption, and lower cost. reaching over 4 million pixels, and a picture acquisition speed

ARTICLE 1 1
Copyright  STMicroelectronics 2001
 ST Journal of System Research
vol 2, number 2 December 2001

Image Processing for Digital Still Camera

of a fraction of a second, DSC competition among the A digital still camera uses an electronic sensor to acquire the
different manufacturers is driven by new features (video clip spatial variations in light intensity and then use image
acquisition, MP3 player, scanning of film negative etc). processing algorithms to reconstruct a color picture from
The rest of the paper is organized as follows. The next the data provided by the sensor (see Fig. 1).
section describes how digital cameras acquire images using
CCD/CMOS sensors. Section 2 reports, in detail, the image Two technologies exist to manufacture imaging sensors: CCD
generation pipeline from the input real scene to the final high (Charge Coupled Device) or CMOS (Complementary Metal
quality picture. Section 3 is entirely devoted to the picture Oxide Semiconductor). While CMOS sensors will almost
compression stage while section 4 reviews the various certainly improve and become more popular in the future, they
picture storage options, the display function and the various probably won’t replace CCD sensors in higher-end digital
features provided by actual DSCs. A final section closes the cameras.The CCD is a collection of tiny light-sensitive diodes,
paper showing possible evolution of the related scenario. which convert photons (light) into electrons (electrical charge).

Figure 1:
image sensor illumination
DSC working principale

imaging color filter

array array
lens

scene

anti-
input color vignette,
processor engine spatial - encoding
distortion

recovery engine

2. SENSORS AND DATA ACQUISITION These diodes are called photosites. Each photosite is sensitive to
The working principle of the DSC is quite different from that light: the brighter the light that hits a single photosite, the greater
of the conventional camera. Conventional cameras use a the electrical charge that will accumulate at that site. Both CCD
chemical reaction to capture the image, fixing it on film and CMOS image sensors start at the same point: to convert
through an emulsion. The emulsion is composed by salt light into electrons at the photosites. A simplified way to think
containing silver whose particles are sensitive to the about the sensor used in a digital camera (or camcorder) is to
quantum effect of light. Spatial variations of light intensity think of it as having a 2-D array of thousands or millions of tiny
impacting the film appear as a picture. solar cells, each of which transforms the light from one small

ARTICLE 1 2
 ST Journal of System Research
vol 2, number 2 December 2001

Image Processing for Digital Still Camera

portion of the image into electrons. Both CCD and CMOS silicon production line, so they tend to be extremely
devices perform this task using a variety of technologies. inexpensive compared to CCD sensors.
The next step is to read the value (accumulated charge) of CCD sensors have been mass-produced for a longer period of
each cell in the image. In a CCD device, the charge is actually time, so they are more mature.They tend to have higher quality
transported across the chip and read at one corner of the pixels. Based on these differences, CCDs tend to be used in
array. An analog-to-digital converter turns each pixel’s value cameras that focus on high-quality images with lots of pixels
into a digital value. In most CMOS devices, there are several and excellent light sensitivity. CMOS sensors usually have lower
transistors at each pixel, which amplify and move the charge quality, lower resolution and lower sensitivity. However, CMOS
using more traditional wires. The CMOS approach is more cameras are much less expensive and have great battery life.
flexible because each pixel can be read individually. CCDs use Over time, CMOS sensors will improve to the point where
a special manufacturing process to create the ability to they reach near parity with CCD devices in most applications,
transport charge across the chip without distortion. This but they are not there yet. In any case, the sensor architecture
process leads to very high-quality sensors in terms of fidelity and physical layout is essentially independent of the technology;
and light sensitivity. CMOS chips, on the other hand, use a for the purpose of this tutorial both CCD and CMOS can be
normal manufacturing process to create the chip: the same thought as nearly identical devices.

Figure 2:
A sensor color is obtained by a mono sensor
plus suitable RGB color filters. Each pixel/color
filter has a small lens a) placed on top,
improving significantly light gathering at pixel
site. On the right b) it is showed the typical
layout of a single CMOS sensor.

process used to make most microprocessors. Because of the Each sensitive element of the sensor (known as pixel) is
manufacturing differences, there are several noticeable sensitive to one color component only. This is obtained
differences between CCD and CMOS sensors. through the deposition of color filters on top of a
CCD sensors create high-quality, low-noise images. CMOS monochrome sensor. Filters are divided into primary and
sensors, traditionally, are more susceptible to noise. complementary colors. The first ones exhibit excellent
Because each pixel on a CMOS sensor has several transistors color reproduction, but are less sensitive than the
located next to it, the light sensitivity of a CMOS chip is lower. complementary. Primary filters are arranged into a pattern
Many of the photons hitting the chip hit the transistors instead known as Bayer pattern. Picture quality is strictly related to
of the photodiode. the number of pixels composing the sensor: the higher the
CMOS sensors traditionally consume little power. better.The amount of detail that the camera can capture is
Implementing a sensor in CMOS yields a low-power sensor. called the resolution, and it is measured in pixels. In general,
CCDs, on the other hand, use a special process that consumes the more pixels your camera has, the more detail it can
lots of power. CCDs consume as much as 100 times more capture. But increasing the sensor resolution without
power than an equivalent CMOS sensor. increasing its size reduces the area of individual pixels, and
CMOS chips can be fabricated on just about any standard therefore their sensitivity. Micro-lenses can be used to

ARTICLE 1 3
 ST Journal of System Research
vol 2, number 2 December 2001

Image Processing for Digital Still Camera

increase the sensor sensitivity by focusing more light on the colors naturally so images are not affected by surrounding
pixel (see Fig.2). light.
Notice in Figure 2 the particular arrangement of the color To do that classical techniques use either simple global
filter array over the pixels, in what is called a Bayer pattern. measure of energy of the scene analyzing the relative
Half of all pixels are green versus a quarter each for blue and distribution of the various chromatic channels or try to
red pixels. This particular arrangement relies on the higher adapt the white color to the particular light condition
sensitivity of our eyes to the green color. Therefore, what (sunset, ...). Auto-white-balancing is sufficient for most
comes out of the sensor is one color component, red, green conditions, but if there is no near white color in the picture,
or blue, per sensor pixel.To construct a color image from the colors that are not originally white may appear white in the
sensor’s output, an RGB triplet must be computed for each image and the white balance of the image may not be
pixel, in a filtering operation known as color interpolation. correct. Also, Auto-white-balancing may not have the desired
effect when shooting under white fluorescent or other
The reconstruction process must guarantee the rendering of fluorescent lights. In such cases, some cameras offer the
a high quality images avoiding typical artifacts that, due to the possibility to use a white surface and quick reference white
acquisition process, could be present. For this reason balance to achieve the correct white balance, or use preset
powerful and smart algorithms are applied to enhance quality white balance to select a color temperature for the incident
in a sort of chain known as DSC pipeline (see Figures 3 and light. Alternatively, it is possible to use preset white
5).They are detailed in the following section. balancing to reproduce more red in a picture of a sunset, or
capture a warmer artistic effect under artificial lighting.

3. IMAGE PROCESSING AND DSC AutoExposure determines the amount of light hitting the
sensor and the sensor itself is used for light metering. The
3.1 Pre Capture Algorithms exposure - the amount of light that reaches the image
During the Pre-Capture phase, just before the actual sensor-determines how light or dark the resulting
picture is captured, the sensor is read continuously and the photograph will be. When the shutter opens, light strikes
output is analyzed in order to determine three parameters the image sensor inside the camera. If too much light strikes
which will determine the quality of the final picture: white it, the photograph will be overexposed-washed out and
balancing, exposure and focus. faded. Too little light produces an underexposed
photograph-dark and lacking in details, especially in shadow
Auto-White-Balancing (AWB for short) compensates areas. To measure the light reflecting from the scene, a
automatically for the dominant “color” of the scene. The camera uses built in light meters.The part of the scene they
human eye is able to compensate colors automatically measure makes all of the difference in the world. Most read
through a characteristic known as Color Constancy, by which the entire image area but give more emphasis to the
the color white, in particular, is always perceived as white bottom part of the scene because this reduces the
independently of the spectral characteristics of the light possibility that the bright sky will cause the picture to be
source illuminating the scene.When a scene is captured on underexposed.They also emphasize the center of the image
a picture, the illuminating context is lost, color constancy area based on the assumption the major subject is placed
does not hold anymore, and white balancing is required to there. This is called a center-weighted system. Some system
compensate colors. AWB relies on the analysis of the allows the user to select a small area of the scene and
picture in order to match the white with a reference white meter it directly using a spot meter. In this mode, only the
point. White balance adjustment attempts to reproduce part of the scene in the center of the viewfinder is metered.

ARTICLE 1 4
 ST Journal of System Research
vol 2, number 2 December 2001

Image Processing for Digital Still Camera

Auto-Focus techniques are more proprietary and vary from the digital camera are prone to and can cope with or
one manufacturer to the other. The Auto-Focus algorithm remove these errors (or interference). Visible noise in a
directly affects picture sharpness. Essentially, it consists on digital image is often affected by temperature (high worse,
extracting a measure of the high frequency content of the low better) and ISO sensitivity (high worse, low better).
picture and changing the focus setting until this measure Color Correction simply adjusts by mathematical operations
reaches a maximum. The article “In-focus image recovery the RGB components of a color separation and creates a
through focus measure analysis” by Salvatore Curti examines new RGB output based on the relative values for the input
in detail one such algorithm. components. Also called color matrixing or color mixing.

3.2 Post Capture Algorithms The key point of the image processing pipeline used in digital
Once the picture is taken a number of different techniques still cameras are the algorithms themselves but, also the kind
such as Defect Correction, Noise Reduction and Color Correction of data they are applied to. Classical image processing
are applied to compensate/enhance the sensor output data. algorithms are applied to color images simply working
Defect correction manages pixel defects related to the sensor independently at the same manner in each color planes.As far
and/ or to the memory storing the picture.When systems on as noise reduction is concerned, several references can be
a chip solution for DSC are considered, both sensor and found on algorithms able to reduce noise without smoothing
memory can be part of a more complex device. Exploiting edges. Normally they are applied to gray level images.To use
the redundancy of image data, these defects can be corrected them on color pictures, it is common practice, to replicate
in a complete transparent way for the DSC manufacturer. the filter for the chromatic components (R, G, B). This will
Noise Reduction is performed to limit the visible effects of an lead to a higher complexity and to possible false color
electronic error (or interference) in the final image from a introduction along the edges.
digital camera. Noise is a function of how well the sensor In a DSC, a better solution is working in the Bayer pattern
(CCD/CMOS) and digital signal processing systems inside domain so that all the subsequent algorithms will benefit of

Figure 3:
Typical DSC
Gamma Video Enc. Imaging Pipeline
Color Calibration
Color Space Conf

Sensor interoperability

CDS APPLICATIONS Serial Link

GA NVMEM
ADC
AWB, AE,
AF Scaling
Pix. Corr Rate CTRL
Noise Red. CODEC
CFA Interp. File Formatting

Image
Image formatting
generation

“shoebox”

ARTICLE 1 5
 ST Journal of System Research
vol 2, number 2 December 2001

Image Processing for Digital Still Camera

the “noisy-free” data, and the complexity will be lower than Often such operations, together with chromatic
working on a RGB domain. Similar consideration can be subsampling, is one of the first steps performed before
developed for the scaling algorithm. Working in the Bayer image/video compression.This is simply due to the fact that
domain requires a little effort to readapt ideas and the human eye is more sensitive to luminance than
techniques to the new particular environment but allows chrominance.
improving significantly quality of the final image reducing at All these algorithms contribute to the final quality of the
the same time the associated computational complexity. image, and constitute one of the key features distinguishing
one camera from another.
The Color Interpolation algorithm converts the sensor
output to a true color image (e.g. RGB 24 bits/pixel), 4. COMPRESSION
reconstructing for each pixel the three chromatic Managing high quality (e.g. 24 bit/pixel) digital images requires
components. Using simple interpolation techniques (i.e. having enough on-board memory to do it. But in order to
median or bicubic filtering) introduces aliasing and/or store them physically for later usage, an ad-hoc compression
unpleasing artifacts; for this reasons each manufacturer algorithm “must” be used in order to reduce the amount of
applies proprietary solutions. Figure 4 illustrates the quality physical bytes needed maintaining a very high quality. The
difference between a straight bicubic interpolation and an subject of digital data compression divides neatly into two
ST proprietary algorithm. categories: lossless compression, in which exact recovery of
the original data is ensured; and lossy compression, in which
Typically, the camera includes additional image processing only an approximate reconstruction is available. Table 1
algorithms (proprietary or not, depending on the specific summarizes the main difference between the most classical
application) to sharpen, scale, etc.. A brief mention to the uncompressed format, the TIFF (Tagged Image File Format)
scaling algorithm used to downsample the input image in format, that requires 24 bit/pixel and the Jpeg format able to
order to achieve the desired resolution (e.g an alternative compress image without noticeable difference (visually
way to achieve compression) saving computational lossless compression) using about 4 bit/pixel. Such kinds of
resources. Before compression, usually, the data is performance are achieved using redundant and irrelevant data
transformed into a suitable color space splitting the present in the original uncompressed image. The
luminance component from the chromatic components. redundancy is related with repetitious, inherent in statistical

Figure 4: Images obtained using different color interpolation algorithms: classical bicubic (on the left), ST proprietary algorithm (on the right)

ARTICLE 1 6
 ST Journal of System Research
vol 2, number 2 December 2001

Image Processing for Digital Still Camera

nature, having to do with similarities, correlation and The first constraint stems from the fact that JPEG
predictability of the data. It can be either “spatial” (e.g. compression does not explicitly guarantee a compressed
characterized by correlation of neighboring pixels) or file size, nevertheless, the system must be predictable
“psycho-visual” (e.g. can be removed without complaint regarding the number of pictures that can be captured
from a human observer). The irrelevant, instead, relates to and stored. From a user perspective, it is not acceptable
an observer viewing an image: function of image resolution, to tell the user the picture he just took cannot be saved.
noise, detail, and viewing conditions. Commonly used algorithms are based on iterating the
The degree of redundancy determines how much compression several times until the target compression
compression can be made. factor is obtained.This, of course, will have impact on the
processing time per picture.
A good quality image having, for example, dimension of
1024x768 pixels, requires 3 bytes for each pixel to represent The second constraint is quite obvious, since quality is a
all its colors (in an RGB color space, also called true-color). differentiating factor among DSCs. In JPEG, the primary
This implies that: 1024x768x3x8 = 18.874 Mbit are needed means of controlling quality is the tuning of the
to represent the image. quantization tables. Everything else being equal, selecting
To overcome this the lossy compression, the process of the right quantization tables can lead to a 10-20%
reducing the amount of information needed to represent and improvement in compression factor or the
image without visibly deteriorating, is vastly used. Some corresponding increase in quality (2 DB on the average)
cameras, also, allow exportation of data directly in “RAW” for a fixed size.
format. In this case the data is technically formatted in
proprietary ways, describes the picture in the bayer check Papers in this issue address these two issues in particular.
board pattern mentioned before. Such a feature can be used
from a professional photograph working with original input
data in order to apply your own enhancement techniques. 5. IMAGE DISPLAY, STORAGE
AND NEW FEATURES
Format bits/pixel Quality Comment
RAW 10-12 Lossless Bayer Pattern
TIFF 24 Lossless Very large files 5.1 Image Display
High quality Image output is the final stage of the image processing
JPEG 2-4 Visually Lossless Small/Medium size
with Good/High quality steps described so far. The basic forms of outputting the
Table 1 - Main image formats. picture are, the LCD of the camera, the TV screen. Having
the possibility of pre-viewing the picture immediately
The standard JPEG [3] allows managing properly the right after it is taken is a feature that digital still camera users
trade-off between final quality and desired compression size. greatly appreciate. But the LCD, jointly with the flash-gun,
In the DSC environment, the JPEG compression stage has to is one of the power-hungry feature of the DSC. Some
obey two constraints: cameras have no LCD panel, and instead use a simple
• the compressed file size must not exceed a fixed size, optical viewfinder. Other cameras have both an LCD
determine from the camera settings such as picture panel and an optical viewfinder, in order to save the
quality and size, in other words, the compression factor battery. The same LCD panel, can also act as the
is fixed viewfinder. It also allows reviewing images in memory,
• the picture quality must be maximized for the given deleting images and taking more pictures in their place.
compression factor, Some newer cameras have advanced features for the

ARTICLE 1 7
 ST Journal of System Research
vol 2, number 2 December 2001

Image Processing for Digital Still Camera

LCD (i.e. zooming in on parts of the image and see them In order to transfer the files from a flash memory device to a
in greater detail). Others offer an “MPEG movie” feature, personal computer without using cables, one needs a drive or
allowing one to take short movies. It is important to other suitable reader-device.These devices behave much like
mention that the display function requires the presence of floppy drives and are inexpensive to buy.
a PAL/NTSC video encoder on chip. Micro hard disks are appearing as mass memory for the
digital still cameras. Users will benefit of the augmented
5.2 Storage capacity by storing video clip.
Early generations of digital cameras had fixed storage inside
the camera. To get the pictures out, they needed to be 5.3 New Features
hooked up directly to a computer by cables so that the Features allowing a number of different image effects are
images could be transferred. Although most of today’s becoming increasingly common. This allows the selection of
cameras are capable of the connecting to a serial, parallel, monochrome, negative and sepia modes. Apart from their
SCSI, and/or USB ports, they usually are provided with some use for artistic effect, the monochrome mode is useful for
sort of removable storage device. Digital Still Cameras use capturing images of documents for subsequent optical
flash memory to store pictures. The memory is available character recognition (OCR). Some digital cameras also
through removable flash-card of different capacity and type. provide a ‘sports’ mode - which adds sharpness to the
The main are: Compact Flash, Smart Media and Memory captured images of moving objects - and a ‘night shooting’
Stick. These three are all small, removable, flash memory mode that allows for long exposures.
devices that have no moving parts.They are fast, inexpensive Panoramic modes differ in their degree of complexity. At the
ways of store photos in order to transfer to a computer or simpler end of the spectrum is the option for a letterbox
printer later. aspect image that simply trims off the top and the bottom

PAL/NTSC
Frame Encoder
Buffer
LCD Preview
Display
Imager Picture Image
Proc. CO-DEC LCD Controller
& Drivers

Autofocus/ Shutter Image Drivers &

Drivers & Actuators Regulators Microcontroller Storage Media:
System Timing üFlash Memory
Photometry Interface & Processing üHDD
Memory Interface
User Interface
Audio Acquisition Audio Processing Data I/O Interface Audio Out

Power Management Converters & Control

üSerial Data I/O
DRAM
üIRDA
Battery üRS232
üUSB
üIEEE 1394
Figure 5: DSC: functional block diagram

ARTICLE 1 8
 ST Journal of System Research
vol 2, number 2 December 2001

Image Processing for Digital Still Camera

edges of a standard image - taking up less storage space in infrastructure enabling the access to services such as, on line
the process. More esoteric is the ability to produce pseudo- photo album, printing on demand, etc. The increased
panoramic shots by capturing a series of images and then bandwidth and service capability of the 2G and 3G wireless
combining them into a single panoramic landscape using communication is expected to contribute to this trend.
special-purpose software. Services such as Multimedia Messaging and Mobile Internet
A self-timer is a common feature, typically providing a 10 - will contribute to make image acquisition more pervasive
second delay between the time the shutter is activated and (imaging enabled mobile phones), but at the same time will
when the picture is taken and all modern day digital cameras make easier to access to remote imaging services through a
have a built-in automatic flash, with a manual override option. link between DSC and wireless linked mobile devices.
Typical settings allow to have a working range of up about The new JPEG2000 standard [4] is promising a lot as far as
20’, providing also a number of different modes, such as auto compression efficiency and features to be supported at a cost
lowlight and backlight flash, fill flash for bright lighting shadow of a higher complexity with respect to JPEG. It provides a new
reduction, force-off for indoor and mood photography and image representation with a rich set of features, all supported
red-eye reduction. Another feature commonly available with within the same compressed bit-stream that can address a
film cameras that is now available on their digital variety of existing and emerging compression applications. In
counterparts is the ability to watermark a picture with a date particular, the Part 1 of the standard addresses some of the
and time, or indeed some other chosen text.Yet, the recent shortcomings of baseline JPEG by supporting the following
innovation of built-in microphones provides for sound set of features: improved compression efficiency, lossy to
annotation, in standard “WAV” format. After recording, this lossless compression, multiple resolution representation,
sound can be sent to an external device for playback, or embedded bit-stream (progressive decoding and SNR
played back on headphones using an ear socket. A couple of scalability), tiling, region-of-interest (ROI) coding, error
other features which demonstrate the digital camera’s close resilience, random codestream access and processing,
coupling with other aspects of PC technology are a function improved performance to multiple compression/
that allows thumbnail images to be emailed directly by decompression cycles, a more flexible file format.
camera-resident software and the ability to capture short The complexity and the fact that in the range of visually loss-
video clips that can be stored in MPEG-like format. less compression factor the compression efficiency on
JPEG2000 vs. JPEG is in the range 10~15% higher, both make
6. Conclusions JPEG2000 attractive for its flexibility and features that must
The rapid growth of the digital still camera market segment be exploited by a killer application.
is due to 3 factors mainly: the wider diffusion of personal
computers, the increasing quality of the digital pictures and References
the introduction of new features enabled by the digital [1] B.E. Bayer, Color imaging array, U.S. Patent 3,971,065-1976;
technology. To produce photo-realistic images, DSCs require [2] W.K. Pratt, Digital Image Processing,
significant amount of image processing. In addition, standard New York:Wiley- Interscience, Ch. 10, 1978;
image processing features like image capture and playback,
[3] W.B. Pennebaker, J.L Mitchell,
multimedia features such as video/speech capture and audio
JPEG Still Image Data Compression Standard,
playbacks are becoming popular in consumers DSCs.
New York, NY,Van Nostrand Reinhold, 1993;
The high rate of innovation in the image processing
technology used in DSCs includes changing the product [4] Information Technology - JPEG2000 Image Coding
features. The new features are not only related to functions System, ISO/IEC International Standard 15444-1,
implemented on board, but also to the availability of an ITU Recommendation T.800, 2000.

ARTICLE 1 9