Image Signal Processor

Image Signal Processor

Introduction
Objectives
ISP Pipeline
Bayer Domain Processing
Dead Pixel Correction
Black Level Compensation
Lens Shading Correction
Anti-aliasing Noise Filter
HDR/Tone Mapping
Auto White Balance Gain Control
Color Filter Array Interpolation
RGB Domain Processing
Gamma Correction
Color Correction Matrix
Color Space Conversion
YUV Domain Processing
Noise Filter for Luma/Chroma
Edge Enhancement
False Color Suppression
Hue/Saturation Control
Brightness/Contrast Control
ISP Tuning Algorithm
Lens Shading Correction
Color Correction Matrix
Auto Exposure/Auto White Balance
ISP Tuning Tool
References

Introduction
Image Signal Processor (ISP) is an application processor to do digital image processing,
specifically for conversion from RAW image (acquired from Imaging Sensors) to RGB/YUV image
(to further processing or display).
Objectives
This project aims to provide an overview of ISP and stimulate the whole ISP pipeline and some
tuning functions. The proposed ISP pipeline consists of following modules, dead pixel correction
(DPC), black level compensation (BLC), lens shading correction (LSC), anti-aliasing noise filter
(ANF), auto white balance gain control (AWB), color filter array interpolation (CFA), gamma
correction (GC), color correction matrix (CCM), color space conversion (CSC), noise filter for luma
and chroma (NF), edge enhancement (EE), false color suppression (FCS), hue/saturation/control
(HSC) and brightness/contast control (BCC).

Some advanced functions like wide/high dynamic range (W/HDR) and temporal/spatial noise filter
(T/SNF) will be implemented furthermore.

ISP Pipeline

Bayer Domain Processing

Dead Pixel Correction
From the perspective of sensor manufacturing, an image sensor may have a certain number of
defective pixels due to the uncertainty of manufacturing process, such as dusts, manufacturing
faults, incompletion of exposure etc. In other words, the chance of having defective pixels only
depends on the manufacturing process and cannot be avoidable ultimately but can be eliminated
as much as possible.

Based on the value of defects, the defective pixels can be classified into three types, dead (always
low), hot (always high) and stuck (always a value between low and high). Based on the status of
defective pixels, they are divided into two types as static dead pixel (always constant) and
dynamic dead pixel (changed by some conditions like exposure or temperature).

There are two steps to realize the dead pixel correction function. The first one is to detect
defective pixels. The second one is to replace the defects with interpolated values.

For stuck defects, it is easy to find the locations by capturing black or white images. Then the
locations of known defects are stored into ISP memory for correction.

For dynamic defects, it is often estimated by using the neighborhood of defective pixels.

The differences between center pixel and neighbored 8 pixels are calculated.

One threshold thres is predefined to charge the pixel is dead or not.

Once the automatic dead pixel detection is done, the correction can be executed. Here lists two
interpolation methods. One is mean filter, another one is gradient-based filter.

For mean filter, it is pretty easy and simple

For gradient-based filter, first calculate the gradient of adjacent pixels on different directions.

From above figure, there are four directions gradient calculated.

The output pixel is the average of adjacent pixels on the selected direction.

Black Level Compensation

From the sensor characterization, the lowest output voltage of sensor is the black level voltage.
Due to circuit design, the black level voltage cannot be zero. This voltage can be measured under
the condition by setting exposure time and all gain (analog and digital) to smallest value. This
leads to the fact that the image is not pure dark even the exposure time and gain are set to
lowest value. The black level compensation is also called optical black correction. So the black
level voltage should be corrected to result in a pure black image. The adjustment is simple as
shown below.

where , , , , , are configurable parameters.

Lens Shading Correction

For camera module, the sensor receives the image via lens. However, at the edge of lens, the light
has a large angle with the optical axis of lens. The intensity of light degrades with the increase of
distance from optical axis. Meanwhile, the refractive index of different colors are different. As a
result, lens shading causes two shadings, luma shading and color shading.

Luma shading causes the intensity of image is not uniform. The corners of image are dark
compared to the center of image.

Color shading causes the color distortion of image. This usually reflects the inconsistency of the
color in image center and corners.
Lens shading correction aims to correct the luma and color distortion. The correction method is
simple, which is to multiply a gain on each pixel at different locations. It is impossible to store all
gains on all pixel locations. The easiest way is to do interpolation based on certain points.

There are several steps to get the parameters.

1. Take a image with uniform light condition. And divide the image into blocks. Four
points of each block have a correction coefficient. Store the coefficients into look-up table.
2. Based on the pixel location, calculating the pixel falls into which block. Then get four
coefficient of the block from LUT.
3. Calculating the correction gain of the pixel by interpolation.
4. Multiplying the correction gain by the pixel.

Usually step 1 is done by tuning software.

Investigate several LSC algorithms

Implement one of LSC algorithm

Anti-aliasing Noise Filter

HDR/Tone Mapping

Auto White Balance Gain Control

White balance (WB) is the process of removing unrealistic color casts, so that objects which
appear white in person are rendered white in your photo. Proper camera white balance has to
take into account the "color temperature" of a light source, which refers to the relative warmth or
coolness of white light. Our eyes are very good at judging what is white under different light
sources, but digital cameras often have great difficulty with auto white balance (AWB) — and can
create unsightly blue, orange, or even green color casts. Understanding digital white balance can
help you avoid these color casts, thereby improving your photos under a wider range of lighting
conditions.

Color temperature describes the spectrum of light which is radiated from a "blackbody" with that
surface temperature. A blackbody is an object which absorbs all incident light — neither
reflecting it nor allowing it to pass through. A rough analogue of blackbody radiation in our day to
day experience might be in heating a metal or stone: these are said to become "red hot" when
they attain one temperature, and then "white hot" for even higher temperatures. Similarly,
blackbodies at different temperatures also have varying color temperatures of "white light."
Despite its name, light which may appear white does not necessarily contain an even distribution
of colors across the visible spectrum:

Why is color temperature a useful description of light for photographers, if they never deal with
true blackbodies? Fortunately, light sources such as daylight and tungsten bulbs closely mimic
the distribution of light created by blackbodies, although others such as fluorescent and most
commercial lighting depart from blackbodies significantly. Since photographers never use the
term color temperature to refer to a true blackbody light source, the term is implied to be a
"correlated color temperature" with a similarly colored blackbody. The following table is a rule-of-
thumb guide to the correlated color temperature of some common light sources:
 Color Temperature (K) Light Source

1000 - 2000 Candlelight

2500-3500 Tungsten Bulb (household variety)

3000-4000 Sunrise/Sunset (clear sky)

4000-5000 Fluorescent Lamps

5000-5500 Electronic Flash

5000-6500 Daylight with Clear Sky (sun overhead)

6500-8000 Moderately Overcast Sky

9000-10000 Shade or Heavily Overcast Sky

To realize auto white balance function, it requires to estimate the color temperature. Under
different color temperature, the gain differs and should be calculated based on the situation. The
estimation of color temperature is complex and done by software or firmware. For hardware,
only applying separate gain on different channels on bayer image. Usually the gain consists of
white balance gain and exposure gain. The image statistics are acquired on bayer domain and
calculated by firmware.

Where , , is the average of different channels on bayer domain. is the digital

gain applied to all channels to keep the image brightness constant.

Color Filter Array Interpolation

A color image requires at least three color samples at each pixel location. Computer images often
use red, green, and blue. A camera would need three separate sensors to completely measure
the image. Using multiple sensors to detect different parts of the visible spectrum requires
splitting the light entering the camera so that the scene is imaged onto each sensor. Precise
registration is then required to align the three images. These additional requirements add a large
expense to the system. Thus, many cameras use a single sensor array with a color filter array.
The color filter array allows only one part of the spectrum to pass to the sensor so that only one
color is measured at each pixel. This means that the camera must estimate the missing two color
values at each pixel. This process is known as demosaicing.

Several patterns exist for the filter array. The most common array is the Bayer color filter array.
The Bayer array measures the green image on a quincunx grid and the red and blue images on
rectangular grids. The green image is measured at a higher sampling rate because the peak
sensitivity of the human visual system lies in the medium wavelengths, corresponding to the
green portion of the spectrum.
The simplest approach to demosaicing is bilinear interpolation, in which the three color planes
are independently interpolated using symmetric bilinear interpolation from the nearest
neighbors of the same color. As expected, bilinear interpolation generates significant artifacts,
especially across edges and other high-frequency content, since it doesn’t take into account the
correlation among the RGB values. Practical demosaicing algorithms take such correlation into
account, either with better linear filters [4], or with nonlinear filters that adapt interpolation
smoothness to a measure of image activity or edginess.

Malvar et al proposed a simple linear demosaicing filter with better performance and low
complexity than others. This demosaicing algorithm has been widely used in ISP and adapted in
Matlab.

Malvar algorithm is designed based on bilinear interpolation filter. The bilinear demosaicing the
green value at a pixel location that falls in a red or blue pixel, is computed by the
average of the neighboring green values in a cross pattern.

Where .

Specificially, to interpolate G values at a red location, a portion of estimated luminance change is

added.

Where is the gradient of R at the location, computed by

Where .

Malvar demosaicing is a gradient-corrected bilinear interpolated approach, with a gain parameter

to control how much correction is applied. For interpolating G at blue pixels, the same formula is
used, but corrected by .

For interpolating R at green pixels, is determined by a 9-point region.

For interpolating R at blue pixels, is determined by a 5-point region.

In order to determine the appropriate values for gain parameters , Wiener approach is
used. First, calculating the values leads to minimum mean-square error interpolation. Then
approximating the optimal Wiener coefficients by integer multiples of small powers of , with the
final result , and .

The final filters are listed below.

RGB Domain Processing

Gamma Correction

Gamma correction is a nonlinear operation used to encode and decode linear luminance or RGB
values to match the non-linear characteristics of display devices.

Gamma correction helps to map data into a more perceptually uniform domain, so as to optimize
perceptual performance of a limited signal range, such as a limited number of bits in each RGB
component. Gamma correction is, in the simplest cases, defined by
Where is a gain multiply to input. is called gamma compression and is called
gamma expansion.

Nowadays, gamma correction is not only used as a method to match the nonlinear response of
display devices, but a tool to adjust the input by mapping to nonlinear response to extend or
compress the dynamic range of image.

Gamma correction is often realized in hardware by a look-up table, which is easy to change and
program.

Color Correction Matrix

There are many variations that cause difficulties in accurately reproducing color in imaging
systems. These can include:
 Spectral characteristics of the optics (lens, filters)
Lighting source variations like daylight, fluorescent, or tungsten
Characteristics of the color filters of the sensor

Color correction is a process used in stage lighting, photography, television, cinematography, and
other disciplines, which uses color gels, or filters, to alter the overall color of the light. Typically
the light color is measured on a scale known as color temperature, as well as along a green–
magenta axis orthogonal to the color temperature axis. Different color temperature results in
different color correction matrix.

Color correction matrix is used to correct the differences between source imaging system and
destination display or receiver system. For ISP system, the color space of sensor is different from
the color space of display device or our human eyes. CCM is dedicated to transfer the sensor RGB
color space to sRGB color space.

The images before color correction and after color correction are shown.

3D Lookup table is also used to map one color space to another. A 3D LUT is a 3D lattice of
output RGB color values that can be indexed by sets of input RGB color values. Each axis of the
lattice represents one of the three input color components and the input color thus defines a
point inside the lattice. Since the point may not be on a lattice point, the lattice values must be
interpolated; most products use trilinear interpolation. Cubes may be of various sizes and bit
depths. Often 33×33×33 cubes are used as 3D LUTs. The most common practice is to use RGB 10-
bit/component log images as the input to the 3D LUT. Output is usually RGB values that are to be
placed unchanged into a display device's buffer.
Color Space Conversion

YCbCr is one of color spaces, widely used in modern color TV system. Usually RGB is transferred
to luma (Y) and chroma (CbCr) via conversion matrix. The transmitter sends the encoded luma
and chroma signal on one coaxial. The advantage of YCbCr color space is that the luma (Y) and
chroma (CbCr) are separated. So that luma (Y) signal is compatible with color and black/white
TVs.

There are three common color spaces, like BT.601 (for SDTV), BT.709 (for HDTV), BT.2020 (for
UHDTV). Under different color space, the conversion matrix is different.

BT.601color space conversion

The fixed 8bit point color space conversion matrix is

The fixed 8bit point RGB2YCrbCr conversion matrix is

The inverse conversion matrix is

BT.709 color space conversion

The fixed 8bit point color space conversion matrix is

The fixed 8bit point RGB2YCrbCr conversion matrix is

The inverse conversion matrix is

BT.2020 color space conversion

The fixed 8bit point color space conversion matrix is

The fixed 8bit point RGB2YCrbCr conversion matrix is

The inverse conversion matrix is

YUV Domain Processing

Noise Filter for Luma/Chroma

Edge Enhancement

Edge enhancement is an image processing filter that enhances the edge contrast of an image or
video in an attempt to improve its acutance (apparent sharpness). The filter works by identifying
sharp edge boundaries in the image, such as the edge between a subject and a background of a
contrasting color, and increasing the image contrast in the area immediately around the edge.
This has the effect of creating subtle bright and dark highlights on either side of any edges in the
image, called overshoot and undershoot, leading the edge to look more defined when viewed
from a typical viewing distance.

Edge enhancement applied to an image can vary according to a number of properties; the most
common algorithm is unsharp masking, which has the following parameters:

Gain. This controls the extent to which contrast in the edge detected area is enhanced.
Radius or aperture. This affects the size of the edges to be detected or enhanced, and the
size of the area surrounding the edge that will be altered by the enhancement. A smaller
radius will result in enhancement being applied only to sharper, finer edges, and the
enhancement being confined to a smaller area around the edge.
Threshold. Where available, this adjusts the sensitivity of the edge detection mechanism. A
lower threshold results in more subtle boundaries of color being identified as edges. A
 threshold that is too low may result in some small parts of surface textures, film grain or
noise being incorrectly identified as being an edge.

In some cases, edge enhancement can be applied in the horizontal or vertical direction only, or to
both directions in different gains.

Edge extraction is done on Y channel, to extract the edge map for image sharpening. The filter is
selected as x .

The filter is applied on x region.

The extracted edge map is given by

Where , ranges from -2 to 2 with interval of 1.

The edge map (EM) is further modified through a lookup table (EMLUT) as
Where is the gain for different thresholds. When , the pixel is noise not the
edge. There are two methods to calculate the Y value:

The is calculated by the gradient of neighbor pixels, which is almost same in DPC. Four
directions are calculated.

Find the minimum of , then takes the value of that direction.

The is further clipped before adding to the original Y channel.

To avoid over-shoot on edges, the is further clipped for output.

False Color Suppression

False colors (speckles) could be introduced during the demosaicing phase where very fine detail
is resolved. False color suppression is similar to color smoothing. The luminance channel is not
affected by this suppression. False colors are generally more apparent in images from cameras
without anti-aliasing filters. Nowadays, despite the false color introduced by demosaicing, other
factors also leads to false color like lens, optical filter, dynamic compression etc. False color
suppression is used to reduce or eliminate the false color introduced during sensor imaging or
isp pipeline. Another function of false color suppression is used to control the chroma saturation
level while doing sharpening and chroma noise under different light condition, like low light or
dark light condition.

False color suppression is conducted on chroma (Cb, Cr) channel. Luma (Y) is not influenced. The
false color suppression is also based on the piece-wise linear function.
The edge map acquired from Edge Enhancement is taken by absolute value.

Then clipping the absolute edge map into .

The chroma gain is calculated by

Where .

Once the gain is computed, then doing false color suppression on chroma channel when edge is
larger than .

Hue/Saturation Control

Hue, saturation, and brightness are aspects of color in the RGB scheme. Hue is one of the main
properties (called color appearance parameters) of a color, defined technically , as "the degree to
which a stimulus can be described as similar to or different from stimuli that are described as
red, green, blue, and yellow" (which in certain theories of color vision are called unique hues).
Hue can typically be represented quantitatively by a single number, often corresponding to an
angular position around a central or neutral point or axis on a colorspace coordinate diagram
(such as a chromaticity diagram) or color wheel, or by its dominant wavelength or that of its
complementary color. Most sources of visible light contain energy over a band of wavelengths.
Hue is the wavelength within the visible-light spectrum at which the energy output from a source
is greatest. This is shown as the peak of the curves in the accompanying graph of intensity versus
wavelength. The saturation of a color is determined by a combination of light intensity and how
much it is distributed across the spectrum of different wavelengths. The saturation is
represented by the steepness of the slopes of the curves. Here, the red curve represents a color
having low saturation, the green curve represents a color having greater saturation, and the blue
curve represents a color with fairly high saturation. As saturation increases, colors appear more
"pure." As saturation decreases, colors appear more "washed-out". Saturation is the purity or
vividness of a color, expressed as the absence of white. A color with 100% saturation contains no
white. A color with 0% saturation corresponds to a shade of gray.

Hue and saturation adjustment are both applied on chroma channel. Hue adjustment involves
the change of hue angle, which is implemented by function.

Where is realized by lookup table, computed by software and programmed into memory.

The final are clipped to .

Saturation adjustment is easy to implement.

Where is the saturation ratio.

Brightness/Contrast Control

Brightness is a relative expression of the intensity of the energy output of a visible light source. It
can be expressed as a total energy value (different for each of the curves in the diagram), or as
the amplitude at the wavelength where the intensity is greatest (identical for all three curves). In
the RGB color model, the amplitudes of red, green, and blue for a particular color can each range
from 0 to 100 percent of full brilliance. These levels are represented by the range of decimal
numbers from 0 to 255, or hexadecimal numbers from 00 to FF. brightness is the perception
elicited by the luminance of a visual target. It is not necessarily proportional to luminance. This is
a subjective attribute/property of an object being observed and one of the color appearance
parameters of color appearance models. Brightness refers to an absolute term and should not
be confused with Lightness. Contrast in visual perception is the difference in appearance of two
or more parts of a field seen simultaneously or successively (hence: brightness contrast, lightness
contrast, color contrast, simultaneous contrast, successive contrast, etc.). The contrast is defined
as the ratio of the luminance of the brightest color (white) to that of the darkest color (black) that
the system is capable of producing. A high contrast ratio is a desired aspect of any display. It has
similarities with dynamic range.

Brightness and contrast are both applied on luma channel.

Where ranges from .

Where is the adjustment ratio, ranges from .

ISP Tuning Algorithm

Lens Shading Correction

Color Correction Matrix

Auto Exposure/Auto White Balance

ISP Tuning Tool

Load RAW/RGB/YUV Images (.raw/.rgb888/.yuv444)

Load configuration files (.csv)

Individually run each algorithm

Sequentially execute all algorithm

References
1. http://www.babelcolor.com/index_htm_files/AN-
9a%20How%20to%20derive%20and%20use%20a%20Color%20Correction%20Matrix.pdf
2. https://blog.csdn.net/u014470361/article/details/88692390
3. https://zhuanlan.zhihu.com/p/34562544
4. https://en.wikipedia.org/wiki/Color_space
5. https://zh.wikipedia.org/wiki/YCbCr
6. https://en.wikipedia.org/wiki/Edge_enhancement
7. http://www.asicfpga.com/site_upgrade/asicfpga/isp/edge_enhancement1.html
8. https://rawpedia.rawtherapee.com/Demosaicing
9. http://help.corel.com/paintshop-pro/v20/main/en/documentation/index.html#page/Corel_P
aintShop_Pro/Adjusting_hue_and_saturation.html
10. https://whatis.techtarget.com/definition/hue-saturation-and-brightness
11. https://en.wikipedia.org/wiki/Display_contrast
12. https://en.wikipedia.org/wiki/Brightness