Chapter 1

VIDEO FORMATION,
PERCEPTION, AND
REPRESENTATION

In this rst chapter, we describe what is a video signal, how is it captured and
perceived, how is it stored/transmitted, and what are the important parameters
that determine the quality and bandwidth (which in turn determines the data rate)
of a video signal. We rst present the underlying physics for color perception
and speci cation (Sec. 1.1). We then describe the principles and typical devices
for video capture and display (Sec. 1.2). As will be seen, analog videos are cap-
tured/stored/transmitted in a raster scan format, using either progressive or in-
terlaced scans. As an example, we review the analog color television (TV) system
(Sec. 1.4), and give insights as to how are certain critical parameters, such as frame
rate and line rate, chosen, what is the spectral content of a color TV signal, and how
can di erent components of the signal be multiplexed into a composite signal. Fi-
nally, Section 1.5 introduces the ITU-R BT.601 video format (formerly CCIR601),
the digitized version of the analog color TV signal. We present some of the consider-
ations that have gone into the selection of various digitization parameters. We also
describe several other digital video formats, including high-de nition TV (HDTV).
The compression standards developed for di erent applications and their associated
video formats are summarized.
The purpose of this chapter is to give the readers background knowledge about
analog and digital video, and to provide insights to common video system design
problems. As such, the presentation is intentionally made more qualitative than
quantitative. In later chapters, we will come back to certain problems mentioned
in this chapter and provide more rigorous descriptions/solutions.
1.1 Color Perception and Speci cation
A video signal is a sequence of two dimensional (2D) images projected from a
dynamic three dimensional (3D) scene onto the image plane of a video camera. The
1
2 Video Formation, Perception, and Representation Chapter 1

color value at any point in a video frame records the emitted or re ected light at a
particular 3D point in the observed scene. To understand what does the color value
mean physically, we review in this section basics of light physics and describe the
attributes that characterize light and its color. We will also describe the principle
of human color perception and di erent ways to specify a color signal.
1.1.1 Light and Color
Light is an electromagnetic wave with wavelengths in the range of 380 to 780
nanometer (nm), to which the human eye is sensitive. The energy of light is mea-
sured by ux, with a unit of watt, which is the rate at which energy is emitted. The
radiant intensity of a light, which is directly related to the brightness of the light
we perceive, is de ned as the ux radiated into a unit solid angle in a particular
direction, measured in watt/solid-angle. A light source usually can emit energy in
a range of wavelengths, and its intensity can be varying in both space and time. In
this book, we use C ( ; t; ) to represent the radiant intensity distribution of a light,
X
which speci es the light intensity at wavelength , spatial location = (X; Y; Z ) X
and time t.
The perceived color of a light depends on its spectral content (i.e. the wavelength
composition). For example, a light that has its energy concentrated near 700 nm
appears red. A light that has equal energy in the entire visible band appears white.
In general, a light that has a very narrow bandwidth is referred to as a spectral
color. On the other hand, a white light is said to be achromatic.
There are two types of light sources: the illuminating source, which emits an
electromagnetic wave, and the re ecting source, which re ects an incident wave.1
The illuminating light sources include the sun, light bulbs, the television (TV)
monitors, etc. The perceived color of an illuminating light source depends on the
wavelength range in which it emits energy. The illuminating light follows an additive
rule, i.e. the perceived color of several mixed illuminating light sources depends on
the sum of the spectra of all light sources. For example, combining red, green, and
blue lights in right proportions creates the white color.
The re ecting light sources are those that re ect an incident light (which could
itself be a re ected light). When a light beam hits an object, the energy in a certain
wavelength range is absorbed, while the rest is re ected. The color of a re ected light
depends on the spectral content of the incident light and the wavelength range that
is absorbed. A re ecting light source follows a subtractive rule, i.e. the perceived
color of several mixed re ecting light sources depends on the remaining, unabsorbed
wavelengths. The most notable re ecting light sources are the color dyes and paints.
For example, if the incident light is white, a dye that absorbs the wavelength near
700 nm (red) appears as cyan. In this sense, we say that cyan is the complement of
1 The illuminating and re ecting light sources are also referred to as primary and secondary light
sources, respectively. We do not use those terms to avoid the confusion with the primary colors
associated with light. In other places, illuminating and re ecting lights are also called additive
colors and subtractive colors, respectively.
Section 1.1. Color Perception and Speci cation 3

Figure 1.1. Solid line: Frequency responses of the three types of cones on the human
retina. The blue response curve is magni ed by a factor of 20 in the gure. Dashed Line:
The luminous eÆciency function. From [10, Fig. 1].

red (or white minus red). Similarly, magenta and yellow are complements of green
and blue, respectively. Mixing cyan, magenta, and yellow dyes produces black,
which absorbs the entire visible spectrum.
1.1.2 Human Perception of Color
The perception of a light in the human being starts with the photo receptors located
in the retina (the surface of the rear of the eye ball). There are two types of
receptors: cones that function under bright light and can perceive the color tone,
and rods that work under low ambient light and can only extract the luminance
information. The visual information from the retina is passed via optic nerve bers
to the brain area called the visual cortex, where visual processing and understanding
is accomplished. There are three types of cones which have overlapping pass-bands
in the visible spectrum with peaks at red (near 570 nm), green (near 535 nm), and
blue (near 445 nm) wavelengths, respectively, as shown in Figure 1.1. The responses
of these receptors to an incoming light distribution C () can be described by:
Z
Ci = () ()
C  ai  d; i = r; g; b; (1.1.1)
where ar (); ag (); ab () are referred to as the frequency responses or relative ab-
sorption functions of the red, green, and blue cones. The combination of these
three types of receptors enables a human being to perceive any color. This implies
that the perceived color only depends on three numbers, Cr ; Cg ; Cb , rather than
the complete light spectrum C (). This is known as the tri-receptor theory of color
vision, rst discovered by Young [14].
4 Video Formation, Perception, and Representation Chapter 1

There are two attributes that describe the color sensation of a human being:
luminance and chrominance. The term luminance refers to the perceived brightness
of the light, which is proportional to the total energy in the visible band. The term
chrominance describes the perceived color tone of a light, which depends on the
wavelength composition of the light. Chrominance is in turn characterized by two
attributes: hue and saturation. Hue speci es the color tone, which depends on
the peak wavelength of the light, while saturation describes how pure the color is,
which depends on the spread or bandwidth of the light spectrum. In this book, we
use the word color to refer to both the luminance and chrominance attributes of a
light, although it is customary to use the word color to refer to the chrominance
aspect of a light only.
Experiments have shown that there exists a secondary processing stage in the
human visual system (HVS), which converts the three color values obtained by the
cones into one value that is proportional to the luminance and two other values that
are responsible for the perception of chrominance. This is known as the opponent
color model of the HVS [3, 9]. It has been found that the same amount of energy
produces di erent sensations of the brightness at di erent wavelengths, and this
wavelength-dependent variation of the brightness sensation is characterized by a
relative luminous eÆciency function, ay (), which is also shown (in dashed line)
in Fig. 1.1. It is essentially the sum of the frequency responses of all three types
of cones. We can see that the green wavelength contributes most to the perceived
brightness, the red wavelength the second, and the blue the least. The luminance
(often denoted by Y) is related to the incoming light spectrum by:
Z
Y = () ()
C  ay  d: (1.1.2)
In the above equations, we have neglected the time and space variables, since we
are only concerned with the perceived color or luminance at a xed spatial and
temporal location. We also neglected the scaling factor commonly associated with
each equation, which depends on the desired unit for describing the color intensities
and luminance.
1.1.3 The Trichromatic Theory of Color Mixture
A very important nding in color physics is that most colors can be produced by
mixing three properly chosen primary colors. This is known as the trichromatic
theory of color mixture, rst demonstrated by Maxwell in 1855 [9, 13]. Let Ck ; k =
1; 2; 3 represent the colors of three primary color sources, and C a given color. Then
the theory essentially says
X
C= Tk Ck ; (1.1.3)
k=1;2;3

where Tk 's are the amounts of the three primary colors required to match color
C . The Tk 's are known as tristimulus values. In general, some of the Tk 's can be
Section 1.1. Color Perception and Speci cation 5

negative. Assuming only T1 is negative, this means that one cannot match color
C by mixing C1 ; C2 ; C3 , but one can match color C + jT1 jC1 with T2 C2 + T3 C3 :
In practice, the primary colors should be chosen so that most natural colors can
be reproduced using positive combinations of primary colors. The most popular
primary set for the illuminating light source contains red, green, and blue colors,
known as the RGB primary. The most common primary set for the re ecting light
source contains cyan, magenta, and yellow, known as the CMY primary. In fact,
RGB and CMY primary sets are complement of each other, in that mixing two
colors in one set will produce one color in the other set. For example, mixing red
with green will yield yellow. This complementary information is best illustrated by
a color wheel, which can be found in many image processing books, e.g., [9, 4].
For a chosen primary set, one way to determine tristimulus values of any color
is by rst determining the color matching functions, mi (), for primary colors, Ci ,
i=1,2,3. These functions describe the tristimulus values of a spectral color with
wavelength , for various  in the entire visible band, and can be determined by
visual experiments with controlled viewing conditions. Then the tristimulus values
for any color with a spectrum C () can be obtained by [9]:
Z
Ti = () ()
C  mi  d; i = 1; 2; 3: (1.1.4)
To produce all visible colors with positive mixing, the matching functions associated
with the primary colors must be positive.
The above theory forms the basis for color capture and display. To record the
color of an incoming light, a camera needs to have three sensors that have frequency
responses similar to the color matching functions of a chosen primary set. This can
be accomplished by optical or electronic lters with the desired frequency responses.
Similarly, to display a color picture, the display device needs to emit three optical
beams of the chosen primary colors with appropriate intensities, as speci ed by
the tristimulus values. In practice, electronic beams that strike phosphors with
the red, green and blue colors are used. All present display systems use a RGB
primary, although the standard spectra speci ed for the primary colors may be
slightly di erent. Likewise, a color printer can produce di erent colors by mixing
three dyes with the chosen primary colors in appropriate proportions. Most of
the color printers use the CMY primary. For a more vivid and wide-range color
rendition, some color printers use four primaries, by adding black (K) to the CMY
set. This is known as the CMYK primary, which can render the black color more
truthfully.
1.1.4 Color Speci cation by Tristimulus Values
Tristimulus Values We have introduced the tristimulus representation of a color
in Sec. 1.1.3, which speci es the proportions, i.e. the Tk 's in Eq. (1.1.3), of the
three primary colors needed to create the desired color. In order to make the color
speci cation independent of the absolute energy of the primary colors, these values
6 Video Formation, Perception, and Representation Chapter 1

should be normalized so that Tk = 1; k = 1; 2; 3 for a reference white color (equal

energy in all wavelengths) with a unit energy. When we use a RGB primary, the
tristimulus values are usually denoted by R; G; and B.
Chromaticity Values: The above tristimulus representation mixes the luminance
and chrominance attributes of a color. To measure only the chrominance informa-
tion (i.e. the hue and saturation) of a light, the chromaticity coordinate is de ned
as:
Tk
tk = ; k = 1; 2; 3: (1.1.5)
T1 + T2 + T3
Since t1 + t2 + t3 = 1, two chromaticity values are suÆcient to specify the chromi-
nance of a color.
Obviously, the color value of an imaged point depends on the primary colors
used. To standardize color description and speci cation, several standard primary
color systems have been speci ed. For example, the CIE, 2 an international body
of color scientists, de ned a CIE RGB primary system, which consists of colors at
700 (R0 ), 546.1 (G0), and 435.8 (B0) nm.
Color Coordinate Conversion One can convert the color values based on one set
of primaries to the color values for another set of primaries. Conversion of (R,G,B)
coordinate to the (C,M,Y) coordinate is, for example, often required for printing
color images stored in the (R,G,B) coordinate. Given the tristimulus representation
of one primary set in terms of another primary, one can determine the conversion
matrix between the two color coordinates. The principle of color conversion and
the derivation of the conversion matrix between two sets of color primaries can be
found in [9].
1.1.5 Color Speci cation by Luminance and Chrominance At-
tributes
The RGB primary commonly used for color display mixes the luminance and chromi-
nance attributes of a light. In many applications, it is desirable to describe a color
in terms of its luminance and chrominance content separately, to enable more ef-
cient processing and transmission of color signals. Towards this goal, various
three-component color coordinates have been developed, in which one component
re ects the luminance and the other two collectively characterize hue and satura-
tion. One such coordinate is the CIE XYZ primary, in which Y directly measures
the luminance intensity. The (X; Y; Z ) values in this coordinate are related to the
(R; G; B) values in the CIE RGB coordinate by [9]:
2 3 2
X 2:365 0:515 0:005 3 2 R 3
4 Y 5 = 4 0:897 1:426 0:014 5 4 G 5 : (1.1.6)
Z 0:468 0:089 1:009 B
2 CIE stands for Commission Internationale de L'Eclariage or, in English, International Com-
mission on Illumination.
Section 1.2. Video Capture and Display 7

In addition to separating the luminance and chrominance information, another

advantage of the CIE XYZ system is that almost all visible colors can be speci ed
with non-negative tristimulus values, which is a very desirable feature. The problem
is that the X,Y,Z colors so de ned are not realizable by actual color stimuli. As
such, the XYZ primary is not directly used for color production, rather it is mainly
introduced for de ning other primaries and for numerical speci cation of color. As
will be seen later, the color coordinates used for transmission of color TV signals,
such as YIQ and YUV, are all derived from the XYZ coordinate.
There are other color representations in which the hue and saturation of a color
are explicitly speci ed, in addition to the luminance. One example is the HSI coor-
dinate, where H stands for hue, S for saturation, and I for intensity (equivalent to
luminance)3 . Although this color coordinate clearly separates di erent attributes of
a light, it is nonlinearly related to the tristimulus values and is diÆcult to compute.
The book by Gonzalez has a comprehensive coverage of various color coordinates
and their conversions [4].
1.2 Video Capture and Display
1.2.1 Principle of Color Video Imaging
Having explained what is light and how it is perceived and characterized, we are
now in a position to understand the meaning of a video signal. In short, a video
records the emitted and/or re ected light intensity, i.e. C ( ; t; ) from the objects
X
in the scene that is observed by a viewing system (a human eye or a camera). In
general, this intensity changes both in time and space. Here, we assume that there
are some illuminating light sources in the scene. Otherwise, there will be no injected
nor re ected light and the image will be totally dark. When observed by a camera,
only those wavelengths to which the camera is sensitive are visible. Let the spectral
absorption function of the camera be denoted by ac (), then the light intensity
distribution in the 3D world that is \visible" to the camera is:
Z 1
( ; t) = C ( ; t; )ac ()d:
X X (1.2.1)
0
The image function captured by the camera at any time t is the projection of
the light distribution in the 3D scene onto a 2D image plane. Let P (
) represent
the camera projection operator so that the projected 2D position of the 3D point
X is given by = P ( ).1 Further more, let P 1(
) denote the inverse projection
x X
operator, so that = P ( ) speci es the 3D position associated with a 2D point
X x
x : Then the projected image is related to the 3D image by
(P ( ); t) =  ( ; t) or ( ; t) =  P 1( ); t
:
X X x x (1.2.2)
The function ( ; t) is what is known as a video signal. We can see that it describes
x
the radiant intensity at the 3D position that is projected onto in the image
X x
3 The HSI coordinate is also known as HSV, where V stands for \value" of the intensity.
8 Video Formation, Perception, and Representation Chapter 1

plane at time t. In general the video signal has a nite spatial and temporal range.
The spatial range depends on the camera viewing area, while the temporal range
depends on the duration in which the video is captured. A point in the image plane
is called a pixel (meaning picture element) or simply pel.4 For most camera systems,
the projection operator P (
) can be approximated by a perspective projection. This
is discussed in more detail in Sec. 5.1.
If the camera absorption function is the same as the relative luminous eÆciency
function of the human being, i.e. ac() = ay (), then a luminance image is formed.
If the absorption function is non-zero over a narrow band, then a monochrome
(or monotone) image is formed. To perceive all visible colors, according to the
trichromatic color vision theory (see Sec. 1.1.2), three sensors are needed, each with
a frequency response similar to the color matching function for a selected primary
color. As described before, most color cameras use the red, green, and blue sensors
for color acquisition.
If the camera has only one luminance sensor, ( ; t) is a scalar function that
x
represents the luminance of the projected light. In this book, we use the word
gray-scale to refer to such a video. The term black-and-white will be used strictly
to describe an image that has only two colors: black and white. On the other hand,
if the camera has three separate sensors, each tuned to a chosen primary color, the
signal is a vector function that contains three color values at every point. Instead
of specifying these color values directly, one can use other color coordinates (each
consists of three values) to characterize light, as explained in the previous section.
Note that for special purposes, one may use sensors that work in a frequency
range that is invisible to the human being. For example, in X-ray imaging, the
sensor is sensitive to the spectral range of the X-ray. On the other hand, an infra-
red camera is sensitive to the infra-red range, which can function at very low ambient
light. These cameras can \see" things that cannot be perceived by the human eye.
Yet another example is the range camera, in which the sensor emits a laser beam and
measures the time it takes for the beam to reach an object and then be re ected
back to the sensor. Because the round trip time is proportional to the distance
between the sensor and the object surface, the image intensity at any point in a
range image describes the distance or range of its corresponding 3D point from the
camera.
1.2.2 Video Cameras
All the analog cameras of today capture a video in a frame by frame manner with
a certain time spacing between the frames. Some cameras (e.g. TV cameras and
consumer video camcorders) acquire a frame by scanning consecutive lines with a
certain line spacing. Similarly, all the display devices present a video as a consecu-
tive set of frames, and with TV monitors, the scan lines are played back sequentially
as separate lines. Such capture and display mechanisms are designed to take advan-
4 Strictly speaking the notion of pixel or pel is only de ned in digital imagery, in which each
image or a frame in a video is represented by a nite 2D array of pixels.
Section 1.2. Video Capture and Display 9

tage of the fact that the HVS cannot perceive very high frequency changes in time
and space. This property of the HVS will be discussed more extensively in Sec. 2.4.
There are basically two types of video imagers: (1) tube-based imagers such
as vidicons, plumbicons, or orthicons, and (2) solid-state sensors such as charge-
coupled devices (CCD). The lens of a camera focuses the image of a scene onto a
photosensitive surface of the imager of the camera, which converts optical signals
into electrical signals. The photosensitive surface of the tube imager is typically
scanned line by line (known as raster scan) with an electron beam or other electronic
methods, and the scanned lines in each frame are then converted into an electrical
signal representing variations of light intensity as variations in voltage. Di erent
lines are therefore captured at slightly di erent times in a continuous manner. With
progressive scan, the electronic beam scans every line continuously; while with
interlaced scan, the beam scans every other line in one half of the frame time (a
eld) and then scans the other half of the lines. We will discuss raster scan in more
detail in Sec. 1.3. With a CCD camera, the photosensitive surface is comprised of a
2D array of sensors, each corresponding to one pixel, and the optical signal reaching
each sensor is converted to an electronic signal. The sensor values captured in each
frame time are rst stored in a bu er, which are then read-out sequentially one line
at a time to form a raster signal. Unlike the tube based cameras, all the read-out
values in the same frame are captured at the same time. With interlaced scan
camera, alternate lines are read-out in each eld.
To capture color, there are usually three types of photosensitive surfaces or CCD
sensors, each with a frequency response that is determined by the color matching
function of the chosen primary color, as described previously in Sec. 1.1.3. To reduce
the cost, most consumer cameras use a single CCD chip for color imaging. This is
accomplished by dividing the sensor area for each pixel into three or four sub-areas,
each sensitive to a di erent primary color. The three captured color signals can be
either converted to one luminance signal and two chrominance signal and sent out
as a component color video, or multiplexed into a composite signal. This subject is
explained further in Sec. 1.2.4.
Many cameras of today are CCD-based because they can be made much smaller
and lighter than the tube-based cameras, to acquire the same spatial resolution.
Advancement in CCD technology has made it possible to capture in a very small
chip size a very high resolution image array. For example, 1/3-in CCD's with 380 K
pixels are commonly found in consumer-use camcorders, whereas a 2/3-in CCD with
2 million pixels has been developed for HDTV. The tube-based cameras are more
bulky and costly, and are only used in special applications, such as those requiring
very high resolution or high sensitivity under low ambient light. In addition to the
circuitry for color imaging, most cameras also implement color coordinate conversion
(from RGB to luminance and chrominance) and compositing of luminance and
chrominance signals. For digital output, analog-to-digital (A/D) conversion is also
incorporated. Figure 1.2 shows the typical processings involved in a professional
video camera. The camera provides outputs in both digital and analog form, and in
the analog case, includes both component and composite formats. To improve the
10 Video Formation, Perception, and Representation Chapter 1

Figure 1.2. Schematic Block Diagram of a Professional Color Video Camera. From [6,
Fig. 7(a)].

image quality, digital processing is introduced within the camera. For an excellent
exposition of the video camera and display technologies, see [6].
1.2.3 Video Display
To display a video, the most common device is the cathode ray tube (CRT). With
a CRT monitor, an electron gun emits an electron beam across the screen line by
line, exciting phosphors with intensities proportional to the intensity of the video
signal at corresponding locations. To display a color image, three beams are emitted
by three separate guns, exciting red, green, and blue phosphors with the desired
intensity combination at each location. To be more precise, each color pixel consists
of three elements arranged in a small triangle, known as a triad.
The CRT can produce an image having a very large dynamic range so that the
displayed image can be very bright, suÆcient for viewing during day light or from a
distance. However, the thickness of a CRT needs to be about the same as the width
of the screen, for the electrons to reach the side of the screen. A large screen monitor
is thus too bulky, unsuitable for applications requiring thin and portable devices.
To circumvent this problem, various at panel displays have been developed. One
popular device is Liquid Crystal Display (LCD). The principle idea behind the LCD
is to change the optical properties and consequently the brightness/color of the liq-
uid crystal by an applied electric eld. The electric eld can be generated/adapted
by either an array of transistors, such as in LCD's using active matrix thin- lm-
transistors (TFT), or by using plasma. The plasma technology eliminates the need
for TFT and makes large-screen LCD's possible. There are also new designs for
at CRT's. A more comprehensive description of video display technologies can be
found in [6].
The above stated raster scan and display mechanisms only apply to TV cameras
and displays. With movie cameras, the color pattern seen by the camera at any
Section 1.2. Video Capture and Display 11

frame instant is completely recorded on the lm. For display, consecutive recorded
frames are played back using an analog optical projection system.

1.2.4 Composite vs. Component Video

Ideally, a color video should be speci ed by three functions or signals, each de-
scribing one color component, in either a tristimulus color representation, or a
luminance-chrominance representation. A video in this format is known as com-
ponent video. Mainly for historical reasons, various composite video formats also
exist, wherein the three color signals are multiplexed into a single signal. These
composite formats were invented when the color TV system was rst developed and
there was a need to transmit the color TV signal in a way so that a black-and-
white TV set can extract from it the luminance component. The construction of
a composite signal relies on the property that the chrominance signals have a sig-
ni cantly smaller bandwidth than the luminance component. By modulating each
chrominance component to a frequency that is at the high end of the luminance
component, and adding the resulting modulated chrominance signals and the orig-
inal luminance signal together, one creates a composite signal that contains both
luminance and chrominance information. To display a composite video signal on a
color monitor, a lter is used to separate the modulated chrominance signals and the
luminance signal. The resulting luminance and chrominance components are then
converted to red, green, and blue color components. With a gray-scale monitor, the
luminance signal alone is extracted and displayed directly.
All present analog TV systems transmit color TV signals in a composite format.
The composite format is also used for video storage on some analog tapes (such
as the VHS tape). In addition to being compatible with a gray-scale signal, the
composite format eliminates the need for synchronizing di erent color components
when processing a color video. A composite signal also has a bandwidth that is
signi cantly lower than the sum of the bandwidth of three component signals, and
therefore can be transmitted or stored more eÆciently. These bene ts are however
achieved at the expense of video quality: there often exist noticeable artifacts caused
by cross-talks between color and luminance components.
As a compromise between the data rate and video quality, S-video was invented,
which consists of two components, the luminance component and a single chromi-
nance component which is the multiplex of two original chrominance signals. Many
advanced consumer level video cameras and displays enable recording/display of
video in S-video format. Component format is used only in professional video
equipment.

1.2.5 Gamma Correction

We have said that the video frames captured by a camera re ect the color values of
the imaged scene. In reality, the output signals from most cameras are not linearly
12 Video Formation, Perception, and Representation Chapter 1

related to the actual color values, rather in a non- linear form:5

vc = Bc ; c (1.2.3)
where Bc represents the actual light brightness, and vc the camera output voltage.
The value of c range from 1.0 for most CCD cameras to 1.7 for a vidicon camera
[7]. Similarly, most of the display devices also su er from such a non-linear relation
between the input voltage value vd and the displayed color intensity Bd, i.e.
Bd = vd : d (1.2.4)
The CRT displays typically have a d of 2.2 to 2.5 [7]. In order to present true colors,
one has to apply an inverse power function on the camera output. Similarly, before
sending real image values for display, one needs to pre-compensate the gamma e ect
of the display device. These processes are known as gamma correction.
In TV broadcasting, ideally, at the TV broadcaster side, the RGB values cap-
tured by the TV cameras should rst be corrected based on the camera gamma and
then converted to the color coordinates used for transmission (YIQ for NTSC, and
YUV for PAL and SECAM). At the receiver side, the received YIQ or YUV values
should rst be converted to the RGB values, and then compensated for the monitor
gamma values. In reality, however, in order to reduce the processing to be done
in the millions of receivers, the broadcast video signals are pre-gamma corrected in
the RGB domain. Let vc represent the R, G, or B signal captured by the camera,
the gamma corrected signal for display, vd, is obtained by
vd = vc = : c d (1.2.5)
In most of the TV systems, a ratio of c= d = 2:2 is used. This is based on the
assumption that a CCD camera with c = 1 and a CRT display with d = 2:2 are
used [7]. These gamma corrected values are converted to the YIQ or YUV values for
transmission. The receiver simply applies a color coordinate conversion to obtain the
RGB values for display. Notice that this process applies display gamma correction
before the conversion to the YIQ/YUV domain, which is not strictly correct. But
the distortion is insigni cant and not noticeable by average viewers [7].
1.3 Analog Video Raster
As already described, the analog TV systems of today use raster scan for video
capture and display. As this is the most popular analog video format, in this
section, we describe the mechanism of raster scan in more detail, including both
progressive and interlaced scan. As an example, we also explain the video formats
used in various analog TV systems.
5 A more precise relation is B = cv c + B ; where c is a gain factor, and B is the cut-o level
c c 0 0
of light intensity. If we assume that the output voltage value is properly shifted and scaled, then
the presented equation is valid.
Section 1.3. Analog Video Raster 13

Progressive Frame Interlaced Frame

Field 1 Field 2

(a) (b)
Figure 1.3. Progressive (a) and Interlaced (b) Raster Scan Formats.

1.3.1 Progressive and Interlaced Scan

Progressive Scan In raster scan, a camera captures a video sequence by sampling
it in both temporal and vertical directions. The resulting signal is stored in a
continuous one dimensional (1D) waveform. As shown in Fig. 1.3(a), the electronic
or optic beam of an analog video camera continuously scans the imaged region from
the top to bottom and then back to the top. The resulting signal consists of a
series of frames separated by a regular frame interval,
t, and each frame consists
of a consecutive set of horizontal scan lines, separated by a regular vertical spacing.
Each scan line is actually slightly tilted downwards. Also, the bottom line is scanned
about one frame interval later than the top line of the same frame. However, for
analysis purposes, we often assume that all the lines in a frame are sampled at
the same time, and each line is perfectly horizontal. The intensity values captured
along contiguous scan lines over consecutive frames form a 1D analog waveform,
known as a raster scan. With a color camera, three 1D rasters are converted into a
composite signal, which is a color raster.

Interlaced Scan The raster scan format described above is more accurately known
as progressive scan (also known as sequential or non-interlaced scan), in which
the horizontal lines are scanned successively. In the interlaced scan, each frame is
scanned in two elds and each eld contains half the number of lines in a frame.
The time interval between two elds, i.e., the eld interval, is half of the frame
interval, while the line spacing in a eld is twice of that desired for a frame. The
scan lines in two successive elds are shifted by half of the line spacing in each
eld. This is illustrated in Fig. 1.3(b). Following the terminology used in the
MPEG standard, we call the eld containing the rst line and following alternating
lines in a frame the top eld, and the eld containing the second line and following
14 Video Formation, Perception, and Representation Chapter 1

alternating lines the bottom eld. 6 In certain systems, the top eld is sampled
rst, while in other systems, the bottom eld is sampled rst. It is important to
remember that two adjacent lines in a frame are separated in time by the eld
interval. This fact leads to the infamous zig-zag artifacts in an interlaced video
that contains fast moving objects with vertical edges. The motivation for using
the interlaced scan is to trade o the vertical resolution for an enhanced temporal
resolution, given the total number of lines that can be recorded within a given time.
A more thorough comparison of the progressive and interlaced scans in terms of
their sampling eÆciency is given later in Sec. 3.3.2.
The interlaced scan introduced above should be called 2:1 interlace more pre-
cisely. In general, one can divide a frame into K  2 elds, each separated in time
by
t=K: This is known as K:1 interlace and K is called interlace order. In a digital
video where each line is represented by discrete samples, the samples on the same
line may also appear in di erent elds. For example, the samples in a frame may be
divided into two elds using a checker-board pattern. The most general de nition
of the interlace order is the ratio of the number of samples in a frame to the number
of samples in each eld.
1.3.2 Characterization of a Video Raster
A raster is described by two basic parameters: the frame rate (frames/second or fps
or Hz), denoted by fs;t; and the line number (lines/frame or lines/picture-height),
denoted by fs;y . These two parameters de ne the temporal and vertical sampling
rates of a raster scan. From these parameters, one can derive another important
parameter, the line rate (lines/second), denoted by fl = fs;tfs;y : 7 We can also
derive the temporal sampling interval or frame interval,
t = 1=fs;t, the vertical
sampling interval or line spacing,
y = picture-height=fs;y , and the line interval,
Tl = 1=fl =
t =fs;y , which is the time used to scan one line. Note that the line
interval Tl includes the time for the sensor to move from the end of a line to the
beginning of the next line, which is known as the horizontal retrace time or just
horizontal retrace, to be denoted by Th . The actual scanning time for a line is
Tl0 = Tl Th : Similarly, the frame interval
t includes the time for the sensor to
move from the end of the bottom line in a frame to the beginning of the top line
of the next frame, which is called vertical retrace time or just vertical retrace, to be
denoted by Tv : The number of lines that is actually scanned in a frame time, known
as the active lines, is fs;y
0 = (
t Tv )=Tl = fs;y Tv =Tl: Normally, Tv is chosen to
be an integer multiple of Tl:
A typical waveform of an interlaced raster signal is shown in Fig. 1.4(a). Notice
that a portion of the signal during the horizontal and vertical retrace periods are
held at a constant level below the level corresponding to black. These are called
6 A more conventional de nition is to call the eld that contains all even lines the even- eld,
and the eld containing all odd lines the odd- eld. This de nition depends on whether the rst
line is numbered 0 or 1, and is therefore ambiguous.
7 The frame rate and line rate are also known as the vertical sweep frequency and the horizontal
sweep frequency, respectively.
 Section 1.3. Analog Video Raster 15

Horizontal retrace Vertical retrace Vertical retrace

for first field from first to second field from second to third field

Blanking level
Black level

White level

(a)

(b)
Figure 1.4. A Typical Interlaced Raster Scan: (a) Waveform, (b) Spectrum.

sync signals. The display devices start the retrace process upon detecting these
sync signals.
Figure 1.4(b) shows the spectrum of a typical raster signal. It can be seen that
the spectrum contains peaks at the line rate fl and its harmonics. This is because
adjacent scan lines are very similar so that the signal is nearly periodic with a
period of Tl: The width of each harmonic lobe is determined by the maximum
vertical frequency in a frame. The overall bandwidth of the signal is determined by
the maximum horizontal spatial frequency.
The frame rate is one of the most important parameters that determine the
quality of a video raster. For example, the TV industry uses an interlaced scan
with a frame rate of 25{30 Hz, with an e ective temporal refresh rate of 50-60 Hz,
while the movie industry uses a frame rate of 24 Hz.8 On the other hand, in the
8 To reduce the visibility of icker, a rotating blade is used to create an illusion of 72 frames/c.
16 Video Formation, Perception, and Representation Chapter 1

Luminance,
RGB
Chrominance,
---> Modulation
Audio
YC1C2
Multiplexing

YC1C2 De- De-

---> Multiplexing Modulation
RGB

Figure 1.5. Analog Color TV systems: Video Production, Transmission, and Reception.

computer industry, 72 Hz has become a de facto standard. The line number used in
a raster scan is also a key factor a ecting the video quality. In analog TVs, a line
number of about 500-600 is used, while for computer display, a much higher line
number is used (e.g., the SVGA display has 1024 lines). These frame rates and line
numbers are determined based on the visual temporal and spatial thresholds under
di erent viewing environments, as described later in Sec. 2.4. Higher frame rate
and line rate are necessary in computer applications to accommodate a signi cantly
shorter viewing distance and higher frequency contents (line graphics and texts) in
the displayed material.
The width to height ratio of a video frame is known as the image aspect ratio
(IAR). For example, an IAR of 4:3 is used in standard-de nition TV (SDTV) and
computer display, while a higher IAR is used in wide-screen movies (up to 2.2) and
HDTVs (IAR=16:9) for a more dramatic visual sensation.

1.4 Analog Color Television Systems

In this section, we brie y describe the analog color TV systems, which is a good
example of many concepts we have talked about so far. One major constraint
in designing the color TV system is that it must be compatible with the previous
monochrome TV system. First, the overall bandwidth of a color TV signal has to t
within that allocated for a monochrome TV signal (6 MHz per channel in the U.S.).
Secondly, all the color signals must be multiplexed into a single composite signal in
a way so that a monochrome TV receiver can extract from it the luminance signal.
The successful design of color TV systems that satisfy the above constraints is one
of the great technological innovations of the 20th century. Figure 1.5 illustrates the
main processing steps involved in color TV signal production, transmission, and
reception. We brie y review each of the steps in the following.
There are three di erent systems worldwide: the NTSC system used in North
Section 1.4. Analog Color Television Systems 17

America as well as some other parts of Asia, including Japan and Taiwan; the
PAL system used in most of Western Europe and Asia including China, and the
Middle East countries; and the SECAM system used in the former Soviet Union,
Eastern Europe, France, as well as Middle East. We will compare these systems
in terms of their spatial and temporal resolution, the color coordinate, as well as
the multiplexing mechanism. The materials presented here are mainly from [9, 10].
More complete coverage on color TV systems can be found in [5, 1].

1.4.1 Spatial and Temporal Resolutions

All three color TV systems use the 2:1 interlaced scan mechanism described in
Sec. 1.3 for capturing as well as displaying video. The NTSC system uses a eld
rate of 59.94 Hz, and a line number of 525 lines/frame. The PAL and SECAM
systems both use a eld rate of 50 Hz and a line number of 625 lines/frame. These
frame rates are chosen to not to interfere with the standard electric power systems in
the involved countries. They also turned out to be a good choice in that they match
with the critical icker fusion frequency of the human visual system, as described
later in Sec. 2.4. All systems have an IAR of 4:3. The parameters of the NTSC,
PAL, and SECAM video signals are summarized in Table 1.1. For NTSC, the line
interval is Tl=1/(30*525)=63.5 s. But the horizontal retrace takes Th=10 s, so
that the actual time for scanning each line is Tl0=53.5 s. The vertical retrace
between adjacent elds takes Tv =1333 s, which is equivalent to the time for 21
scan lines per eld. Therefore, the number of active lines is 525-42=483/frame. The
actual vertical retrace only takes the time to scan 9 horizontal lines. The remaining
time (12 scan lines) are for broadcasters wishing9 to transmit additional data in the
TV signal (e.g., closed caption, teletext, etc.)

1.4.2 Color Coordinate

The color coordinate systems used in the three systems are di erent. For video
capture and display, all three systems use a RGB primary, but with slightly di erent
de nitions of the spectra of individual primary colors. For transmission of the video
signal, in order to reduce the bandwidth requirement and to be compatible with
black and white TV systems, a luminance/chrominance coordinate is employed. In
the following, we describe the color coordinates used in these systems.
The color coordinates used in the NTSC, PAL and SECAM systems are all de-
rived from the YUV coordinate used in PAL, which in turn originates from the XYZ
coordinate. Based on the relation between the RGB primary and XYZ primary, one
can determine the Y value from the RGB value, which forms the luminance compo-
nent. The two chrominance values, U and V, are proportional to color di erences,
B-Y and R-Y, respectively, scaled to have desired range. Speci cally, the YUV
9 The number of active lines cited in di erent references vary from 480 to 495. This number is
calculated from the vertical blanking interval cited in [5].
18 Video Formation, Perception, and Representation Chapter 1

Parameters of Analog Color TV Systems

Table 1.1.

Parameters NTSC PAL SECAM

Field Rate 59.94 50 50
Line Number/Frame 525 625 625
Line Rate (Line/s) 15,750 15,625 15,625
Image Aspect Ratio 4:3 4:3 4:3
Color Coordinate YIQ YUV YDbDr
Luminance Bandwidth (MHz) 4.2 5.0, 5.5 6.0
Chrominance Bandwidth (MHz) 1.5(I),0.5(Q) 1.3(U,V) 1.0 (U,V)
Color Subcarrier (MHz) 3.58 4.43 4.25 (Db),4.41 (Dr)
Color modulation QAM QAM FM
Audio Subcarrier (MHz) 4.5 5.5,6.0 6.5
Composite Signal Bandwidth(MHz) 6.0 8.0,8.5 8.0

coordinate is related to the PAL RGB primary values by [9]:

2
Y
3 2
0:299 0:587 0:114 3 2 R~ 3
4 U 5 = 4 0:147 0:289 0:436 5 4 G~ 5 (1.4.1)
V 0:615 0:515 0:100 B~
and
2
R ~ 3 2 1:000 0:000 1:140 3 2 Y 3
~ = 4 1:000
4 G 5 0:395 0:581 5 4 U 5; (1.4.2)
B ~ 1:000 2:032 0:001 V
where R;~ G;~ B~ are normalized gamma-corrected values, so that (R;~ G;~ B~ ) = (1; 1; 1)
corresponds to the reference white color de ned in the PAL/SECAM system.
The NTSC system uses the YIQ coordinate, where the I and Q components are
rotated versions (by 33o) of the U and V components. This rotation serves to make I
corresponding to colors in the orange-to-cyan range, whereas Q the green-to-purple
range. Because the human eye is less sensitive to the changes in the green-to-purple
range than that in the yellow-to-cyan range, the Q component can be transmitted
with less bandwidth than the I component [10]. This point will be elaborated more
in Sec. 1.4.3. The YIQ values are related to the NTSC RGB system by:
2
Y
3 2
0:299 0:587 0:114 3 2 R~ 3
4 I 5 = 4 0:596 0:275 0:321 5 4 G~ 5 (1.4.3)
Q 0:212 0:523 0:311 B~
Section 1.4. Analog Color Television Systems 19

and
2
R~ 3 2 1:0 0:956 0:620 3 2 Y 3
~ = 4 1:0 0:272 0:647 5 4 I 5
4 G 5 (1.4.4)
B~ 1:0 1:108 1:70 Q
p
With the YIQ coordinate, tan 1(Q=I ) approximates the hue, and I 2 + Q2=Y
re ects the saturation. In a NTSC composite video, the I and Q components are
multiplexed into one signal, so that
p the phase of the modulated signal is tan (Q=I ),
1

whereas the magnitude equals I + Q =Y . Because transmission errors a ect

2 2
the magnitude more than the phase, the hue information is better retained than
saturation in broadcast TV signal. This is desired, as the human eye is more
sensitive to the color hue. The name I and Q come from the fact that the I signal is
In-phase with the color modulation frequency, whereas the Q signal is in Quadrature
(i.e. 1/4 of the way around the circle or 90 degrees out of phase) with the modulation
frequency. The color multiplexing scheme is explained later in Sec. 1.4.4.
Note that because the RGB primary set and the reference white color used in
the NTSC system are di erent from those in the PAL/SECAM system, the same
set of RGB values corresponds to slightly di erent colors in these two systems.
The SECAM system uses the YDbDr coordinate, where the Db and Dr values
are related to the U and V values by [7]
Db = 3:059U; Dr = 2:169V: (1.4.5)
1.4.3 Signal Bandwidth
The bandwidth of a video raster can be estimated from its line rate. First of all,
the maximum vertical frequency results when the white and black lines alternate in
a raster frame, which is equal to fs;y
0 =2 cycles/picture-height, where fs;y
0 represent
the number of active lines. The maximum frequency that can be rendered properly
by a system is usually lower than this theoretical limit. The attenuation factor
is known as the Kell factor, denoted by K , which depends on the camera and
display aperture functions. Typical TV cameras have a Kell factor of K = 0:7: The
maximum vertical frequency that can be accommodated is related to the Kell factor
by
0 =2 (cycles/picture-height):
fv;max = Kfs;y (1.4.6)
Assuming that the maximum horizontal frequency is identical to the vertical one
for the same spatial distance, then, fh;max = fv;maxIAR (cycles/picture-width).
Because each line is scanned in Tl0 seconds, the maximum frequency in the 1D
raster signal is
fmax = fh;max =Tl0 = IAR
Kfs;y
0 =2T 0 Hz:
l (1.4.7)
For the NTSC video format, we have fs;y 0 = 483; T 0 = 53:5 s. Consequently, the
l
maximum frequency of the luminance component is 4.2 megacycles/second or 4.2
20 Video Formation, Perception, and Representation Chapter 1

MHz. Although the potential bandwidth of the chrominance signal could be just
as high, usually it is signi cantly lower than the luminance signal. Furthermore,
the HVS has been found to have much lower threshold for observing changes in
chrominance. Because of the Typically, the two chrominance signals are bandlimited
to have much narrower bandwidth. As mentioned previously, the human eye is
more sensitive to spatial variations in the orange-to-cyan color range, represented
by the I component, than it is for the green-to-purple range, represented by the
Q component. Therefore, the I component is bandlimited to 1.5 MHz, and the
Q component to 0.5 MHz.10 Table 1.1 lists the signal bandwidth of di erent TV
systems.
1.4.4 Multiplexing of Luminance, Chrominance, and Audio
In order to make the color TV signal compatible with the monochrome TV system,
all three analog TV systems use the composite video format, in which the three
color components as well as the audio component are multiplexed into one signal.
Here, we brie y describe the mechanism used by NTSC. First, the two chrominance
components I (t) and Q(t) are combined into a single signal C (t) using quadrature
amplitude modulation (QAM). The color sub-carrier frequency fc is chosen to be
an odd multiple of half of the line rate, fc = 455 f2 = 3.58 MHz. This is chosen
l

to satisfy the following criteria: i) It should be high enough where the luminance
component has very low energy; ii) It should be midway between two line rate har-
monics where the luminance component is strong; and iii) It should be suÆciently
far away from the audio sub-carrier, which is set at 4.5 MHz (286 fl), the same
as in the monochrome TV. Figure 1.6(a) shows how the harmonic peaks of the
luminance and chrominance signals interleave with each other. Finally, the audio
signal is frequency modulated (FM) using an audio sub-carrier frequency of fa =4.5
MHz and added to the composite video signal, to form the nal multiplexed signal.
Because the I component has a bandwidth of 1.5 MHz, the modulated chrominance
signal has a maximum frequency of up 5.08 MHz. In order to avoid the interference
with the audio signal, the chrominance signal is bandlimited in the upper sideband
to 0.6 MHz. Notice that the lower sideband of the I signal will run into the upper
part of the Y signal. For this reason, sometimes the I signal is bandlimited to 0.6
MHz on both sidebands. Finally, the entire composite signal, with a bandwidth of
about 4.75 MHz, is modulated onto a picture carrier frequency, fp, using vestigial
sideband modulation (VSB), so that the lower sideband only extends to 1.25 MHz
below fp and that the overall signal occupies 6 MHz. This process is the same as in
the monochrome TV system. The picture carrier fp depends on the broadcasting
channel. Figure 1.6(b) illustrates the spectral composition of the NTSC compos-
ite signal. The signal bandwidth and modulation methods in the three color TV
systems are summarized in Table 1.1.
At a television receiver, the composite signal rst has to be demodulated to
the baseband, and then the audio and three components of the video signals must
10 In [9], the bandwidth of I and Q are cited as 1.3 and 0.6 MHz, respectively.
 Section 1.4. Analog Color Television Systems 21

Y(f) Luminance Chrominance

fc
(a) (color sub-carrier)

Luminance

I and Q

Audio

Picture Color Audio

(b) Carrier Sub-carrier Sub-carrier

Figure 1.6. Multiplexing of luminance, chrominance, and audio signals in NTSC sys-
tem. (a) The interleaving between luminance and chrominance harmonics; (b) The overall
spectral composition of the NTSC composite signal.
22 Video Formation, Perception, and Representation Chapter 1

Table 1.2. Analog Video Tape Formats

Video Format Tape Format Horizontal Luminance Applications

Lines Bandwidth
composite VHS, 8mm 240 3 MHz Consumer
Umatic SP 330 4 MHz Professional
S-video S-VHS, Hi8 400 5.0 MHz High quality consumer
component Betacam SP 480 4.5 MHz Professional

be demultiplexed. To separate the video and audio signals, a low-pass lter can be
used. This process is the same in a monochrome TV as that in a color TV. To further
separate the chrominance signal from the luminance signal, ideally, a comb lter
should be used, to take advantage of the interleaving of the harmonic frequencies
in these two signals. Most high-end TV sets implement a digital comb lter with
null frequencies at the harmonics corresponding to the chrominance component to
accomplish this. Low-end TV sets however use a simple RC circuit to perform low-
pass ltering with a cut-o frequency at 3 MHz, which would retain the unwanted
I component in the extracted luminance signal, and vice verse. This will lead to
cross-color and cross-luminance artifacts. Cross-color refers to the spurious colors
created by the high frequency luminance signal that is close to the color sub-carrier
frequency. Cross-luminance refers to false high frequency edge patterns caused by
the modulated chrominance information. For a good illustration of the e ects of
di erent lters, see [2]. After extracting the chrominance signal, a corresponding
color-demodulation method is used to separate the two chrominance components.
Finally, the three color components are converted to the RGB coordinate for display.
1.4.5 Analog Video Recording
Along with the development of analog TV systems, various video tape recording
(VTR) technologies have been developed, to allow professional video production
(record and editing) as well as for consumer level recording (home video) and play-
back (VCR). Table 1.2 summarizes common video tape formats.
1.5 Digital Video
1.5.1 Notations
A digital video can be obtained either by sampling a raster scan, or directly using a
digital video camera. Presently all digital cameras use CCD sensors. As with analog
cameras, a digital camera samples the imaged scene as discrete frames. Each frame
Section 1.5. Digital Video 23

comprises of output values from a CCD array, which is by nature discrete both
horizontal and vertically. A digital video is de ned by the frame rate, fs;t, the
line number, fs;y , as well as the number of samples per line, fs;x. From these, one
can nd the temporal sampling interval or frame interval,
t = 1=fs;t, the vertical
sampling interval
y = picture-height=fs;y , and the horizontal sampling interval

x = picture-width=fs;x. In this book, we will use (m; n; k) to represent a digital
video, where integer indices m and n are the column and row indices, and k the
frame number. The actual spatial and temporal locations corresponding to the
integer indices are x = m
x; y = n
y , and t = k
t: For convenience, we use the
notation (x; y; t) to describe a video signal in a general context, which could be
either analog or digital. We will use (m; n; k) only when speci cally addressing
digital video.
In addition to the above parameters, another important parameter of a digital
video is the number of bits used to represent a pixel value (luminance only or three
color values), to be denoted by Nb. Conventionally, the luminance or each of the
three color values is speci ed with 8 bits or 256 levels. Therefore, Nb = 8 for
a monochrome video, while Nb = 24 for a color video. The data rate, R, of a
digital video is determined by R = fs;tfs;xfs;y Nb, with a unit of bits/second (bps).
Usually it is measured in kilobits/second (Kbps) or megabits/second (Mbps). In
general, the spatial and temporal sampling rates can be di erent for the luminance
and chrominance components of a digital video. In this case, Nb should re ect the
equivalent number of bits used for each pixel in the luminance sampling resolution.
For example, if the horizontal and vertical sampling rates for each chrominance
component are both half of that for the luminance, then there are two chrominance
samples for every four Y samples. If each sample is represented with 8 bits, the
equivalent number of bits per sample in the Y resolution is (4*8+2*8)/4=12 bits.
When displaying the digital video on a monitor, each pixel is rendered as a
rectangular region with a constant color that is speci ed for this pixel. The ratio of
the width to the height of this rectangular area is known as the pixel aspect ratio
(PAR). It is related to the IAR of the display area and the image dimension by

PAR = IAR
fs;y =fs;x: (1.5.1)

For proper display of a digitized video, one must specify either PAR or IAR, along
with fs;x and fs;y . The display device should conform to the PAR speci ed for
this video (or derived from the speci ed IAR). Otherwise, the object shape will be
distorted. For example, a person will become fatter and shorter, if the display PAR
is larger than the PAR speci ed for this video. In computer industry, a PAR of
1.0 is normally used. On the other hand, in the TV industry, non-square pixels are
used because of some historical reasons. The rationale behind this is explained in
Sec. 1.5.2.
24 Video Formation, Perception, and Representation Chapter 1

1.5.2 ITU-R BT.601 Digital Video

Spatial Resolution of the BT.601 Signal In an attempt to standardize the digital
formats used to represent di erent analog TV video signals with a quality equivalent
to broadcast TV, the International Telecommunications Union - Radio Sector (ITU-
R) developed the BT.601 recommendation [8]. The standard speci es digital video
formats with both 4:3 and 16:9 aspect ratios. Here, we only discuss the version with
aspect ratio 4:3.11 To convert a raster scan to a digital video, one just needs to
sample the 1D waveform. If a total number of fs;x samples are taken per line, the
equivalent sampling rate is fs = fs;xfs;y fs;t = fs;xfl samples/second. In the BT.601
standard, the sampling rate fs is chosen to satisfy two constraints: i) the horizontal
sampling resolution should match with the vertical sampling resolution as closely
as possible, i.e. let
x 
y ; and ii) the same sampling rate should be used for the
NTSC and PAL/SECAM systems, and it should be multiples of respective line rates
in these systems. The rst criterion calls for fs;x  IAR
fs;y , or fs  IAR
fs;y
2 f ,
s;t
which leads to fs  11 and 13 MHz for the NTSC and PAL/SECAM system. A
number that is closest to both numbers and yet satis es the second criterion is then
chosen, which is
fs = 858fl (NTSC) = 864fl(PAL=SECAM) = 13:5MHz: (1.5.2)
The numbers of pixels per line are fs;x = 858 for NTSC and 864 for PAL/SECAM.
These two formats are known as 525/60 and 625/50 signals, respectively, and are
illustrated in Fig. 1.7. The numbers of active lines are respectively fs;y0 = 480 and
576 in the 525 and 625 line systems, but the number of active pixels/line are the
same, both equal to fs;x 0 = 720 pixels. The rest are samples obtained during the
horizontal and vertical retraces, which fall in the non-active area.
With the BT.601 signal, the pixel width to height ratio is not one, i.e. the
physical area associated with a pixel is not a square. Speci cally, PAR =
x=
y =
IAR
fs;y 0 = 8=9 for 525/60 and 16/15 for 625/50 signals. To display a BT.601
0 =fs;x
signal, the display device must has a proper PAR, otherwise the image will be
distorted. For example, when displayed on a computer screen which has a PAR
of 1, a 525/60 signal will appear stretched horizontally, while a 625/50 signal will
appear stretched vertically. Ideally, one should resample the original signal so that
0 = IAR
fs;y
fs;x 0 . For example, the 525/60 and 625/50 signals should be resampled
to have 640 and 768 active pixels/line, respectively.
Color Coordinate and Chrominance Subsampling Along with the image resolu-
tion, the BT.601 recommendation also de nes a digital color coordinate, known as
YCbCr. The Y, Cb, and Cr components are scaled and shifted versions of the analog
Y, U, and V components, where the scaling and shifting operations are introduced
so that the resulting components take value in the range of (0,255). For a more
detailed explanation on the design this color coordinate, the readers are referred
11 The ITU-R was formerly known as International Radio Consultative Committee (CCIR) and
the 4:3 aspect ratio version of the BT.601 format was called the CCIR601 format.
Section 1.5. Digital Video 25

858 pels 864 pels

720 pels 720 pels

525 lines

480 lines

625 lines

576 lines
Active Active
Area Area

122 16 132 12
pel pel pel pel

525/60: 60 field/s 625/50: 50 field/s

Figure 1.7. BT.601 video formats.

to [9]. Here we only present the transformation matrix for deriving this coordinate
from digital RGB coordinate. Assuming that the RGB values are in the range of
(0{255), the YCbCr values are related to RGB values by:
2 3 2
Y 0:257 0:504 0:098 3 2 R 3 2 16 3
4 Cb 5 = 4 0:148 0:291 0:439 5 4 G 5 + 4 128 5 : (1.5.3)
Cr 0:439 0:368 0:071 B 128
The inverse relation
2
is:
3 2
R 1:164 0:0 1:596 3 2 Y 16 3
4 G 5 = 4 1:164 0:392 0:813 5 4 Cb 128 5 : (1.5.4)
B 1:164 2:017 0:0 Cr 128
In the above relations, R = 255R;~ G = 255G;~ B = 255B~ are the digital equiva-
lent of the normalized RGB primaries, R;~ G;
~ and B~ , as de ned either in the NTSC
or PAL/SECAM system. In the YCbCr coordinate, Y re ects the luminance and
is scaled to have a range of (16{235), Cb and Cr are scaled versions of color dif-
ferences B Y and R Y , respectively. The scaling and shifting is designed so
that they have a range of (16{240). The maximum value of Cr corresponds to
red (Cr = 240 or R = 255; G = B = 0 ), whereas the minimum value yields
cyan (Cr = 16 or R = 0; G = B = 255). Similarly, the maximum and minimum
values of Cb correspond to blue (Cb = 240 or R = G = 0; B = 255) and yellow
(Cb = 16 or R = G = 255; B = 0).
The spatial sampling rate introduced previously refers to the luminance com-
ponent, Y . For the chrominance components, Cb and Cr , usually only half of the
sampling rate is used, i.e. fs;c = fs=2: This leads to half number of pixels in each
line, but the same number of lines per frame. This is known as the 4:2:2 format,
implying there are 2 Cb samples and 2 Cr samples for every 4 Y samples. To fur-
ther reduce the required data rate, BT.601 also de ned the 4:1:1 format, in which
26 Video Formation, Perception, and Representation Chapter 1

4:4:4 4:2:2 4:1:1 4:2:0

For every 2x2 Y Pixels For every 2x2 Y Pixels For every 4x1 Y Pixels For every 2x2 Y Pixels
4 Cb & 4 Cr Pixel 2 Cb & 2 Cr Pixel 1 Cb & 1 Cr Pixel 1 Cb & 1 Cr Pixel
(No subsampling) (Subsampling by 2:1 (Subsampling by 4:1 (Subsampling by 2:1 both
horizontally only) horizontally only) horizontally and vertically)

Y Pixel Cb and Cr Pixel

Figure 1.8. BT.601 chrominance subsampling formats. Note that the two adjacent lines
in any one component belong to two di erent elds.

the chrominance components are subsampled along each line by a factor of 4, i.e.,
there are 1 Cb sample and 1 Cr sample for every 4 Y samples. This sampling
method, however, yields very asymmetric resolutions in the horizontal and vertical
directions. Another sampling format is therefore developed, which subsamples the
Cb and Cr components by half in both the horizontal and vertical directions. In
this format, there are also 1 Cb sample and 1 Cr sample for every 4 Y samples.
But to avoid the confusion with the previously de ned 4:1:1 format, this format
is designated as 4:2:0. For applications requiring very high resolutions, the 4:4:4
format is de ned, which samples the chrominance components in exactly the same
resolution as the luminance components. The relative positions of the luminance
and chrominance samples for di erent formats are shown in Fig. 1.8.12
In Chap. 4, we will discuss solutions for converting videos with di erent spa-
tial/temporal resolutions. The conversion between di erent color subsampling for-
mats is considered in one of the exercise problems.
The raw data rates of a BT.601 signal depends on the color sub-sampling factor.
With the most common 4:2:2 format, there are two chrominance samples per two Y
samples, each represented with 8 bits. Therefore, the equivalent bit rate for each Y
sample is Nb = 16 bits, and the raw data rate is fsNb = 216 Mbps. The raw data
rate corresponding to the active area is fs;tfs;y 0 Nb = 166 Mbps. With the 4:2:0
0 fs;x
format, there are two chrominance samples per four Y samples, and the equivalent
bit rate for each Y sample is Nb = 12 bits. Therefore the raw data rate is 162 Mbps,
with 124 Mbps in the active area. For the 4:4:4 format, the equivalent bit rate for
each Y sample is Nb = 24 bits, and the raw data rate is 324 Mbps, with 249 Mbps
in the active area. The resolutions and data rates of di erent BT.601 signals are
12 For the 4:2:0 format, the Cr and Cb samples may also be positioned in the center of the four
corresponding Y samples, as shown in Fig. 13.14(a).
Section 1.5. Digital Video 27

summarized in Table 1.3.

The BT.601 formats are used in high-quality digital video applications, with the
4:4:4 and 4:2:2 formats typically used for video production and editing, whereas 4:2:0
for video distribution, e.g.,13movies on digital video disks (DVD), video-on-demand
(VOD), etc. The MPEG2 video compression standard was primarily developed
for compression of BT.601 4:2:0 signals, although it can also handle videos in lower
or higher resolutions. A typical 4:2:0 signal with a raw active data rate of 124 Mbps
can be compressed down to about 4-8 Mbps. We will introduce the MPEG2 video
coding algorithm in Sec. 13.5.
1.5.3 Other Digital Video Formats and Applications
In addition to the BT.601 format, several other standard digital video formats have
been de ned. Table 1.3 summarizes these video formats, along with their main
applications, compression methods, and compressed bit rates. The CIF (Com-
mon Intermediate Format) is speci ed by International Telecommunications Union-
Telecommunications Sector (ITU-T), which has about half the resolution of BT.601
4:2:0 in both horizontal and vertical dimensions and is developed for video confer-
encing applications, and the QCIF, which is a quarter of CIF, used for video phone
type applications. Both are non-interlaced. The ITU-T H.261 coding standard was
developed to compress videos in either format to p  64 Kbps, with p = 1; 2; : : : ; 30,
for transport over ISDN (integrated service digital network) lines, which only allow
transmission rates in multiples of 64 Kbps. Typically, a CIF signal with a raw
data rate of 37.3 Mbps can be compressed down to about 128 to 384 Kbps, with
reasonable quality, while a QCIF signal with a raw data rate of 9.3 Mbps can be
compressed to 64-128 Kbps. A later standard, H.263, can achieve better quality
than H.261, at the same bit rate. For example, it is possible to compress a QCIF
picture to about 20 Kbps, while maintaining a quality similar or better than H.261
at 64 Kbps. This enables video phone over a 28.8 Kbps modem line.
In parallel with the e ort of ITU-T, the ISO-MPEG also de ned a series of digital
video formats. The SIF (Source Intermediate Format) is essentially a quarter size
of the active area in the BT.601 signal, and is about the same as CIF. This format
is targeted for video applications requiring medium quality, such as video games
and CD movies. As with BT.601, there are two SIF formats: one with a frame rate
of 30 Hz and a line number of 240, and another with a frame rate of 25 and line
number of 288, both have 352 pixels/line. There is also a corresponding set of SIF-I
format, which is 2:1 interlaced. The MPEG-1 algorithm can compress a typical SIF
video with a raw data rate of 30 Mbps to about 1.1 Mbps with a quality similar to
the resolutions seen on a VHS VCR, which is lower than broadcast television. The
rate of 1.1 Mbps enables the playback of digital movies on CD-ROM's, which have
an access rate of 1.5 Mbps. Distribution of MPEG1 movies on video CD's (VCD)
marked the entrance of digital video into the consumer market in the early 1990's.
13 MPEG standards for Motion Picture Expert Group of the International Standard Organization
or ISO.
28 Video Formation, Perception, and Representation Chapter 1

MPEG2-based DVD's, which started in mid 90's, opened the era of high quality
digital video entertainment. MPEG2 technology is also the corner stone of the next
generation TV system, which will be fully digital, employing digital compression and
transmission technology. Table 1.3 lists the details of the video formats discussed
above, along with their main applications, compression methods, and compressed
bit rates. More on compression standards will be presented in Chap. 13.
The BT.601 format is the standard picture format for digital TV (DTV). To
further enhance the video quality, several HDTV formats have also been standard-
ized by the Society of Motion Picture and Television Engineers (SMPTE), which
are also listed in Table 1.3. A distinct feature of HDTV is its wider aspect ratio,
16:9 as opposed to 4:3 in SDTV. The picture resolution is doubled to tripled in
both horizontal and vertical dimensions. Furthermore, progressive scan is used to
reduce the interlacing artifacts. A high pro le has been developed in the MPEG2
video compression standard for compressing HDTV video. Typically it can reduce
the data rate to about 20 Mbps while retaining the very high quality required. This
video bit rate is chosen so that the combined bit stream with audio, when trans-
mitted using digital modulation techniques, can still t into a 6 MHz terrestrial
channel, which is the assigned channel bandwidth for HDTV broadcast in the U.S.

1.5.4 Digital Video Recording

To store video in digital formats, various digital video tape recorder (DVTR) for-
mats have been developed, which di er in the video format handled and technology
for error-correction-coding and storage density. Table 1.4 lists some standard and
proprietary tape formats. The D1-D5 formats store a video in its raw, uncom-
pressed formats, while others pre-compress the video. Only a conservative amount
of compression is employed so as not to degrade the video quality beyond that ac-
ceptable for the intended application. A good review of digital VTRs can be found
in [11]. A extensive coverage on the underlying physics of magnetic recording and
operation of DVTRs can be found in the book by Watkinson [12].
In addition to magnetic tape recorders, VCD and DVD are two video storage
devices using optical disks. By incorporating MPEG1 and MPEG2 compression
technologies, they can store SIF and BT.601 videos, respectively, with suÆcient
quality. At present, VCD and DVD are read-only, so that they are mainly used
for distribution of pre-recorded video, as opposed to as tools for recording video by
consumers.
Except video recording systems using magnetic tapes, hard-disk based systems,
such as TiVo and ReplayTV, are also on the horizon. These systems enable con-
sumers to record up to 30 hours of live TV programs onto hard-disks in MPEG-2
compressed formats, which can be viewed later with usual VCR features such as
fast forward, slow motion, etc. They also allow instant pause of a live program
that is being watched, by storing the live video from the time of the pause onto the
disk. As the price for hard-disks drops down continuously, hard-disk-based DVTR's
Section 1.5. Digital Video 29

Table 1.3. Digital Video Formats for Di erent Applications

Video Y Color Frame Raw Data
Format Size Sampling Rate (Mbps)
HDTV over air, cable, satellite, MPEG2 video, 20-45 Mbps
SMPTE 296M 1280x720 4:2:0 24P/30P/60P 265/332/664
SMPTE 295M 1920x1080 4:2:0 24P/30P/60I 597/746/746
Video Production, MPEG2, 15-50 Mbps
BT.601 720x480/576 4:4:4 60I/50I 249
BT.601 720x480/576 4:2:2 60I/50I 166
High quality video distribution (DVD,SDTV), MPEG2, 4-8 Mbps
BT.601 720x480/576 4:2:0 60I/50I 124
Intermediate quality video distribution (VCD, WWW), MPEG1, 1.5 Mbps
SIF 352x240/288 4:2:0 30P/25P 30
Video conferencing over ISDN/Internet, H.261/H.263, 128-384 Kbps
CIF 352x288 4:2:0 30P 37
Video telephony over wired/wireless modem, H.263, 20-64 Kbps
QCIF 176x144 4:2:0 30P 9.1

may eventually overtake tape-based systems, which are slower and have less storage
capacity.
1.5.5 Video Quality Measure
To conduct video processing, it is necessary to de ne an objective measure that
can measure the di erence between an original video and the processed one. This
is especially important, e.g., in video coding applications where one must measure
the distortion caused by compression. Ideally such a measure should correlate well
with the perceived di erence between two video sequences. Finding such a measure
however turns out to be an extremely diÆcult task. Although various quality mea-
sures have been proposed, those that correlate well with visual perception are quite
complicated to compute. Most video processing systems of today are designed to
minimize the mean square error (MSE) between two video sequences 1 and 2,
which is de ned as
MSE = e2 = N1
XX
( 1 (m; n; k) 2(m; n; k))2 ; (1.5.5)
k m;n
30 Video Formation, Perception, and Representation Chapter 1

Table 1.4. Digital Video Tape Formats

Tape Video Source Compressed Compression Intended
Format Format Rate Rate Method Application
Uncompressed formats
SMPTE D1 BT.601 4:2:2 216 Mbps N/A N/A Professional

SMPTE D2 BT.601 composite 114 Mbps N/A N/A Professional

SMPTE D3 BT.601 composite 114 Mbps N/A N/A Professional/

Consumer

SMPTE D5 BT.601 4:2:2 270 Mbps N/A N/A Professional

(10 bit)
Compressed formats

Digital Betacam BT.601 4:2:2 166 Mbps 80 Mbps Frame DCT Professional

Betacam SX BT.601 4:2:2 166 Mbps 18 Mbps MPEG2 Consumer

(I and B mode only)

DVCPRO50 BT.601 4:2:2 166 Mbps 50 Mbps frame/ eld DCT Professional

DVCPRO25 (DV) BT.601 4:1:1 124 Mbps 25 Mbps frame/ eld DCT Consumer

where N is the total number of pixels in either sequence. For a color video, the
MSE is computed separately for each color component.
Instead of the MSE, the peak signal to noise ratio (PSNR) in decibel (dB) is
more often used as a quality measure in video coding. The PSNR is de ned as
2
PSNR = 10 log10 max
2
(1.5.6)
e

where max is the peak (maximum) intensity value of the video signal. For the most
common 8 bit/color video, max = 255: Note that for a xed peak value, PSNR
is completely determined by the MSE. The PSNR is more commonly used than
the MSE, because people tend to associate the quality of an image with a certain
range of PSNR. As a rule of thumb, for the luminance component, a PSNR over
40 dB typically indicates an excellent image (i.e., being very close to the original),
between 30 to 40 dB usually means a good image (i.e., the distortion is visible but
acceptable), between 20 and 30 dB is quite poor, and nally, a PSNR lower than
20 dB is unacceptable.
Section 1.6. Summary 31

It is worth noting that to compute the PSNR between two sequences, it is

incorrect to calculate the PSNR between every two corresponding frames and then
taking the average of the PSNR values obtained over individual frames. Rather
one should compute the MSE between corresponding frames, average the resulting
MSE values over all frames, and nally convert the MSE value to PSNR.
A measure that is sometimes used in place of the MSE, mainly for reduced
computation, is the mean absolute di erence (MAD). This is de ned as
MAD = 1
XX
N k m;n
j 1 (m; n; k) 2 (m; n; k)j : (1.5.7)
For example, for motion estimation, the MAD is usually used to nd the best
matching block in another frame for a given block in a current frame.
It is well known that MSE or PSNR does not correlate very well with visual
distortion between two imagery. But these measures have been used almost exclu-
sively as objective distortion measures in image/video coding, motion compensated
prediction, and image restoration, partly because of their mathematical tractability,
and partly because of the lack of better alternatives. Designing objective distortion
measures that are easy to compute and yet correlate well with visual distortion is
still an open research issue. In this book, we will mostly use MSE or PSNR as the
distortion measure.
1.6 Summary
Color Generation, Perception, and Speci cation (Sec. 1.1)
 The color of a light depends on its spectral content. Any color can be created
by mixing three primary colors. The most common primary set includes red,
green, and blue colors.
 The human eye perceives color by having receptors (cones) in the retina that
are tuned to red, green, and blue wavelengths. The color sentation can be
described by three attributes: luminance (i.e., brightness), hue (color tone),
and saturation (color purity). The human eye is most sensitive to luminance,
then to hue, and nally to saturation.
 A color can be speci ed by three numbers: either those corresponding to
the contributions of the three primary colors (i.e., tristimulus values), or a
luminance value and two chrominance values.
Analog Video (Sec. 1.3)
 Analog videos used in broadcasting TV, video camcorder, etc., video display,
are stored in a raster scan format. The visual quality and bandwidth of a
raster scan depends on its frame rate and line number.
32 Video Formation, Perception, and Representation Chapter 1

 Interlaced scan is a mechanism to trade o vertical resolution for enhanced

temporal resolution. But it also leads to interlacing artifacts.
Analog Color TV Systems (Sec. 1.4)
 There are three analog color TV systems world wide: NTSC, PAL, and SE-
CAM. They all use 2:1 interlace, but di er in frame rate, line number, color
coordinate, and luminance and chrominance multiplexing.
 In color TV systems, the luminance and two chrominance components as well
as the associated audio signal are multiplexed into a composite signal, us-
ing modulation (frequency shifting) techniques. The multiplexing methods
are designed so that the color TV system is downward compatible with the
monochrome TV system. Furthermore, the modulation frequencies for indi-
vidual components are chosen to minimize the interference among them.
Digital Video (Sec. 1.5)
 BT.601 is a digital video format, resulting from sampling the analog color TV
signals. The sampling rate is chosen so that the horizontal sampling rate is
similar to the vertical sampling rate, and that the data rates for NTSC and
PAL/SECAM systems are the same.
 The chrominance components can be sampled at a lower rate than the lumi-
nance component. There are di erent color subsampling formats de ned in
BT.601.
 Compression is necessary to reduce the raw data rate of a digital video to
reduce the storage/transmission cost. Di erent video compression standards
have been developed for videos intended for di erent applications.
1.7 Problems
1.1 Describe the mechanism by which the human being perceives color.
1.2 What is the perceived color if you have a light that has approximately the
same energy at frequencies corresponding to red, green, and blue, and are
zero at other frequencies ? What about red and green frequencies only ?
1.3 What is the perceived color if you mix red, green, and blue dyes in equal
proportion ? What about red and green dyes only ?
1.4 For the following colors in the RGB coordinate, determine their values in the
YIQ and YUV coordinates, respectively.
(a) (1,1,1); (b) (0,1,0); (c) (1,1,0); (d) (0, 1,1).
Section 1.8. Bibliography 33

1.5 For the following colors in the digital RGB coordinate, determine their values
in the YCbCr coordinate.
(a) (255,255, 255); (b) (0, 255, 0); (c) (255,255,0); (d) (0, 255, 255).
1.6 In Sec. 1.5.2, we say that the maximum value of Cr corresponds to red, whereas
the minimum value yields cyan . Similarly, the maximum and minimum values
of Cb correspond to blue and yellow, respectively. Verify these statements
using the YCbCr to RGB coordinate transformation.
1.7 In Fig. 1.4, we show the spectrum of a typical raster signal. Why is the
spectrum of the video signal nearly periodic? What does the width of harmonic
lobes depend on?
1.8 What are the pros and cons of progressive vs. interlaced scans? For the same
line number per frame, what is the relation between the maximum temporal
frequency that a progressive raster can have and that of an interlaced raster
which divides each frame into two elds? What about the relation between
the maximum vertical frequencies?
1.9 In Sec. 1.4.3, we estimated the bandwidth of the NTSC signal based on its
scan parameters. Following the same approach, estimate the bandwidth of
the PAL and SECAM signals.
1.10 Describe the process for forming a composite color video signal. How should
you select the color sub-carrier frequency and audio sub-carrier frequency ?
1.11 What are the pros and cons of using component vs. composite format?
1.12 Project: Using an oscilloscope to i) draw the waveform, and ii) measure the
spectrum of a composite video signal output from a TV set or a camcorder.
1.13 Project: Digitize a composite video signal using an A/D converter, and using
Matlab to determine the spectrum. Also perform ltering to separate the
luminance, chrominance and audio signals.
1.8 Bibliography
[1] K. B. Benson, editor. Television Engineering Handbook. McGraw Hill, 1992.
Revised by J.C.Whitaker.
[2] J. F. Blinn. NTSC: nice technology, super color. IEEE Computer Graphics and
Applications Magazine, pages 17{23, Mar. 1993.
[3] R. M. Boynton. Human Color Vision. Holt, Rinhart, Winston, New York, 1979.
[4] R. G. Gonzalez and R. E. Woods. Digital Image Processing. Addison{Wesley,
2nd edition, 1992.
34 Video Formation, Perception, and Representation Chapter 1

[5] B. Grob and C. E. Herndon. Basic Television and Video Systems. Glencoe
McGraw Hill, 6th edition, 1999.
[6] Y. Hashimoto, M. Yamamoto, and T. Asaida. Cameras and display systems.
Proc. of IEEE, pages 1032{1043, July 1995.
[7] B. G. Haskell, A. Puri, and A. N. Netravali. Digital Video: An Introduction to
MPEG-2. Chapman & Hall, New York, 1997.
[8] ITU-R. BT.601-5: Studio encoding parameters of digital television for standard
4:3 and wide-screen 16:9 aspect ratios, 1998. (Formerly CCIR 601).
[9] A. N. Netravali and B. G. Haskell. Digital Pictures - Representation, Compres-
sion and Standards. Plenum Press, 2nd edition, 1995.
[10] D. H. Pritchard. US color television fundamentals. IEEE Trans. Consum.
Electron., CE-23:467{78, 1977.
[11] M. Umemoto, Y. Eto, and T. Fukinuki. Digital video recording. Proc. of IEEE,
pages 1044{1054, July 1995.
[12] J. Watkinson. The Art of Digital Video. Focal Press, Oxford, 2nd edition,
1994.
[13] G. Wyszecki and W. S. Stiles. Color Science. John Wiley, New York, 1967.
[14] T. Young. On the theory of light and colors. Philosophical Transactions of the
Royal Society of London, 92:20{71, 1802.