CSE412

SELECTED TOPICS IN COMPUTER

ENGINEERING

INTRODUCTION
 Course Info
Lectures
– Sat. 5:30pm – 8:30pm

TA
– TBA

Course Assessment
– Final Exam 90 marks
– Midterm, Projects, Assignments, Quizzes … etc. 35 marks
 Textbooks and References
Textbook
1. Z. Li and M. S. Drew, Fundamentals of Multimedia, Prentice Hall, 2 nd
Edition, Springer, 2014. (or 3rd Edition, 2021).

References
1. M. Mandal, Multimedia Signals and Systems, Kluwer Academic
Publishers, 2003.
2. A. Puri, Multimedia Systems, Standards and Networks, Taylor and
Francis, 2007.
3. K. R. Rao, Z. S. Bojkovic, and D. A. Milovanovic, Multimedia
Communication Systems, Prentice Hall, 2002.
4. F. Halsall, Multimedia Communications, Addison Wesley, 2001.
 Tentative Weekly Plan
Week Topic
1 Introduction to Multimedia Systems
2 Video representation and coding
3 Voice Fundamentals and coding
4 Multimedia standards
5 Audio-Video Analysis and Retrieval
6 Content-based Image Retrieval
7 Midterm (Tentative)
8 Content-based Video Retrieval
9 Multimedia Information Security and Digital Rights Management (DRM)
10 Packet Telephony Networks
11 VoIP Systems
12 H.323 Concepts and Configuration – SIP Concepts and Configuration
13 H.264
14 H.265
15 Internet Multimedia Content Distribution
5

What is Multimedia?
• Multimedia is a combination of two or more
categories of information having different
transport signal characteristics
– Text
– Audio and speech
– Images
– Graphics
– Video
 What is a Multimedia System?

A system that involves:

– Generation
– Representation
– Storage
– Transmission
– Search and retrieval
– Delivery

of multimedia information
7

Challenges of Multimedia Computing

Developing a successful multimedia system
is non-trivial.
– Continuous media types such as video need a lot of
space to store and a large bandwidth to transmit.
– They also have tight timing constraints.
– Automatically analyzing, indexing, and organizing
information in audio, image, and video is much
harder than text.
8

Multimedia is Multidisciplinary

Computer
networks,
operating system

Image, audio, Multimedia Computer vision,

speech computing pattern recognition
processing

Human computer Computer

interaction graphics
9

Multimedia Systems
Multimedia systems involve some basic enabling
techniques:
– Multimedia data representation and compression.
– Multimedia data processing and analysis.
– Transmitting multimedia data through communication
networks.
– Multimedia database, indexing, and retrieval.
1
0
Media Processing and Analysis
• In applications such as digital library,
automatic data analysis has to be done to
extract semantic meanings from audios,
images, and videos.
• Based on media processing methods such
as
– object tracking (face, eyes),
– object recognition,
– gesture recognition, etc.,

• we can build more effective human computer

interfaces using multimedia.
1
1
Extracting Objects
1
2
Multiple Object Tracking

Toys Squash

Lab
Double Squash
1
3
Object Recognition
1
4
Media Delivery
• Transmitting multimedia data across the network is another
topic in multimedia computing.
• There are different issues when we transmit video and audio
through packet network.
– Quality of service
– Synchronization
– Error and congestion control
– Session setup
1
5
Multimedia Database and Indexing
• Multimedia database has to deal with large
media files.
• Multimedia data need new data structures,
indexing and searching methods.
• Content-based multimedia retrieval is
normally used in such environments.
 What is a Digital Image?
A digital image is a representation of a two-
dimensional image as a finite set of digital
values, called picture elements or pixels
 What is a Digital Image? (cont…)
• Pixel values typically represent gray levels
or colors.
• Digitization implies that a digital image is
an approximation of a real scene

1 pixel
 What is a Digital Image? (cont…)
Common image formats include:
– 1 sample per point (Grayscale)
– 3 samples per point (Red, Green, and Blue)
– 4 samples per point (Red, Green, Blue, and “Alpha”)
 Light and Electromagnetic Spectrum

• Light is just a particular part of the

electromagnetic spectrum that can be sensed by
human eye
• Electromagnetic spectrum is split up according to
the wavelengths of different forms of energy
 Light and Color

Gamma-ray X-ray Ultra-violet Infra-red Microwave Radio

10-11 10-9 10-7 10-5 10-3 10-1 10 103

Wavelength (cm)
Visible spectrum

400 500 600 700 (nm)

Wavelengths comprising the visible range of the

electromagnetic spectrum
 Additive and Subtractive Colors
Additive Color Process
• The additive color process produces and presents color to
the eye by engaging light directly emitted from a
source, and then mixing up three primary colors from this
emitted light to form all the other colors. The additive color
process engages the three primary colors: Red, Green,
and Blue.(RGB)
• The color model employed in the additive color process is
called the RGB (Red, Green, Blue) color model. Mixing any
one of these three primary colors in equal quantities with
another primary color produces what are known as the
secondary colors: Cyan, Magenta, and Yellow (CMYK), K
for black.
 Additive and Subtractive Colors
Subtractive color Process
• The subtractive color process produces and presents color
to the eye by absorbing (subtraction) some light
wavelengths and reflecting some other. The working of this
color process can be understood by looking at inks, paints
and natural colorants. Each color printed by these on any
media can be seen by the eye because the media absorbs
some part of the light wavelengths and reflects the others.
• The subtractive color process begins with white light, for
example, that is reflected by white paper. When the colors
are placed on the paper, two layers are formed, the colors
and the white light reflected from the paper. When the
viewer views the printed paper, the inks placed on the
paper absorb and subtract certain light wavelengths from
the white and present the corresponding color to the eye.
 Reflected Light
• Colors that we perceive are determined by the nature of
the light reflected from an object
• For example, if white
light is shone onto a
green object all
wavelengths are
absorbed except for green Colours
Absorbed
light is reflected from
the object
RGB & CMY Primaries
 RGB & CMY Primaries
• RGB primary and CMY primary are complement to each
other means:
o white minus red gives cyan;
o white minus green gives magenta
o white minus blue gives yellow;
o red + green + blue = white (additive)
o cyan + magenta + yellow = black (subtractive).
 Color Perception and Specification
Human Perception of Color
• There are two types of photoreceptors located in the retina
cones and rods.
• Cones operate under bright light and can perceive color tone.
• Rods work under low ambient light and can only extract
luminance information.
• There are three types of cones: red (570 nm), green (535 nm),
and blue (445 nm).
 Human Visual System

The human eye

The retina
Color Perception and Specification

Frequency responses of the three types of cones in the

human retina and the luminous efficiency function
 Color Perception and Specification
Human Perception of Color
• The combination of these three colors, rather than the complete
light spectrum enables a human to perceive any color. This is
known as the tri-receptor theory of color vision.
• There are two attributes that describe the color sensation of a
human being: luminance and chrominance.
• Luminance refers to the perceived brightness of light, which is
proportional to the total energy in the visible band.
• Chrominance describes the perceived color tone of light, which
depends on the wavelength composition of the light.
• Chrominance is, in turn, characterized by two attributes: hue and
saturation.
 Color Perception and Specification
Human Perception of Color
• Hue specifies the color tone, which depends on the peak
wavelength of the light.
• Saturation describes how pure the color is, which depends on the
bandwidth of the light spectrum.
• The Human Visual System (HVS) converts the three color values
into one value that is proportional to the luminance and two other
values that are responsible for the perception of chrominance.
 Color Perception and Specification
The Trichromatic Theory of Color Mixture
• Most colors can be produced by mixing three properly chosen
primary colors.
• The most popular primary set for illuminating light sources
eulb dna neerg ,der sniatnoc colors, known as RGB Primary.
• The most common primary set for reflecting light sources
sa nwonk ,sroloc wolley dna atnegam ,nayc sniatnocCMY primary.
• RGB primary and CMY primary are complement to each other.
• To record the color of an incoming light, a camera must have three
sensors that have frequency responses similar to the color matching
functions of a chosen primary set.
• To display a color picture, the display device must emit three optical
beams of the chosen primary set with appropriate intensities, as
specified by the tristimulus values.
 Color Perception and Specification
Color Specification by Tristimulus Values
• For the RGB primary color set, the three tristimulus values
are denoted by R (red), G (green) and B (blue). A color is
then specified by its tristimulus coefficient, defined as

R G B
x ; y ; z ;
RG  B R G  B RG B

and x  y  z  1.
 Color Video
Principles of Color Video
• A video (a sequence of moving images) records the emitted
and/or reflected light intensity from the objects in a scene that is
observed by a video camera.
• The image function (x ,y ,t) )captured by the camera at any time t
is the projection of the light distribution in the 3D scene onto a 2D
image plane.
• In general, a video signal has spatial and temporal dimensions.
The spatial dimension (x ,y) )depends on the viewing area; and the
temporal dimension, t )depends on the duration for which the
scene is captured).
 Color Video
Principles of Color Video
• If the camera has only one luminance sensor, (x ,y ,t) )is a
scalar function representing the luminance of the projected
light. Luminance image is also known as grey scale image.
• On the other hand, if the camera has three separate sensors,
each tuned to a chosen primary color, the signal is a vector
function that contains three color values at every point.
• A black and white image strictly has two colors: black and
white.
• A monochrome image is one which consists of colors of a
narrow band of wavelengths.
 Color Video
Composite and Component Video
• A component video is specified by a tristimulus color
representation or a luminance-chrominance representation.
• The three color components can be multiplexed into a single
signal to form a composite video.
• The composite format is used in analog TV systems where the
chrominance signals are modulated to a higher frequency than the
luminance and adding the resulting modulated chrominance
signals to the luminance signal.
• A filter is used to separate the luminance and chrominance signals
for display in a color monitor.
• With a grey-scale monitor, only the luminance signal is extracted
for display.
 Analog Video Raster
Progressive and Interlaced Scan
• In a raster scan, a camera captures a video sequence by
sampling it in both temporal and spatial directions. The
resulting signal is stored in a continuous 1-D waveform.
• The video signal consists of a series of frames separated by a
regular frame interval. Each frame consists of a consecutive
set of horizontal scan lines, separated by a regular vertical
spacing. The format is known as progressive scan.
• In the interlaced scan, each frame is scanned in two fields,
separated by the field interval. Each field thus contains half
the number of lines in a frame.
• The field containing the first line is called the top field; and that
containing the second line is the bottom field.
 Analog Video Raster
Progressive and Interlaced Scan
Field 1 Field 2

Progressive scan 2:1 Interlaced scan

Progressive and interlaced scan