MULTIMEDIA: MEDIA AND DATA

STREAMS

Multi means many & media is an intervening

medium through which something is transmitted
or carried on. Media is a means for distribution &
presentation of information.
Ex: text, graphics, speech music.

CLASSIFICATION OF MEDIUM

It is according to perception, representation,

presentation, storage, transmission, &
information exchange.

The Perception medium: It is through seeing or

hearing the information. Visual media such as
text, image & video are used for perception of
seeing. For perception of hearing, auditory
media such as music, noise & speech are
relevant.
The Representation Medium: Represents
coding of computer information.
1. A text character is coded in ASCII
2. Audi stream is represented using PCM.
3. An image can be coded as a facsimile or
JPEG format.
4. A combine audio/video may be stored in
MPEG format in computer.
5. Graphics are coded according the video
text standard.
The Presentation Medium: refers to tools &
devices for input (Keyboard, mouse, camera,
microphone) & output of information (paper,
screen, speaker).
The storage medium: refers to a data carrier
which enables storage of data (Microfilm, floppy
disk, HD, CD-ROM).
The transmission medium: The different
carriers that enable continuous data
transmission. Storage media are excluded.
Information is transmitted over networks, wire &
cable transmission co-axial cables & fibre optics.

The information exchange medium: Includes

all information carriers for transmission &
storage.
Definition of multimedia: An MM system is
characterized by computer controlled, integrated
production, manipulation, presentation, storage
& communication of independent information
which is encoded through a continuous (time-
dependent) & discrete (time-independent)
medium.

Representation dimensions of Medium

Media is divided to two types w.r.t. time in their
representation space:
1. Time independent media like text &
graphics that consist exclusively of a
sequence of independent elements without
the time element.
2. Values of media like sound & full motion
video change with time. Processing of
these media is time critical as correctness
of data is time dependent. We call these
continuous media. Ex: Video as a
sequence of sample with periodic behavior.

Traditional data stream characteristics: A

sequence of individual packets transmitted in a
line in a time independent fashion is called a
data stream. Information is divided to individual
units or packets & sent from one computer
(source) to another computer (destination).
Data stream characteristics for continuous
media.
1. The time interval between a complete
transmission of consecutive packets: If is
constant, data stream is strongly periodic.
In ideal case, jitter = 0. When time interval
is not constant, it is weakly periodic. All
other possibilities of transmission w.r.t.
time are periodic. Ex: Of a-periodic data
stream in co-operative application with
shared windows.

<--T--→T- >< -T - >

strongly periodic signal

<t1><t2><t3>< t1><t2><t3>
---------→----------→
T T

Weakly periodic signal

 2. Variation of consecutive packet amount.
This is the second data stream
characteristic.
a) If the amount of data is constant
during the lifetime of a data stream, it
is strongly regular (typical for un-
compressed digital data transmission).
Here, the data sizes of all packets in
the stream are constant.
b) Weakly regular data stream has data
periodically varying with time. The I,
B, P images in the MPEG
compressed image is weakly periodic
as the bit rate of compression is in the
ratio 10:1:2 but each image is
compressed separately.
c) If data amount is neither constant nor
periodic with time.

MULTIMEDIA FILE FORMATS

Text as well as MM information is stored on
computer as files. Some of the formats are:
1. RTF: Rich text Format. Plain text files with
only information codes as ASCII, EBCDIC.
Formatted text files contain info codes as
 well as formatting info. Different word
processors may have different formatting
techniques for which RTF aims to bridge
the gap between them so that file transfer
between different systems become easier.
Main elements of RTF are
• Character set: Windows ANSI, IBM PC
etc.
• Font table
• Color table
• Document formatting
• Section & page formatting
• Paragraph formatting
• General formatting
• Character formatting
• Special characters like backslash,
hyphen etc,.
2. TIFF: Tagged Image File Format: The
traditional file storage uses sequential
storage which begins with a file header
(fixed length) containing info about the
format of data). This method works well for
small files. When editing sequential file, the
data as well as the header contents have to
be first loaded from the disk to main
memory, modified & then stored back at the
disk. An MM file having text, audio, video is
 much larger than a plain text file & this
method is difficult in practice as additional
space has to be created. This limitation is
overcome in TIFF where tags are located
with help of pointers & sequential search is
not needed to locate the tags. Seq. search
made the editing too slow. Tags are used
to store the file/image related details & a
block of bytes known as CHUNK is used to
store the info. The concept of tags & chunk
is also used in RIFF.
RIFF: Resource Interchange File Format:
• Provided a method for encapsulating the
MM files for Windows based applications.
A custom file format can be converted to
RIFF format by encapsulating it in RIFF
structure.
• Chunks are used to store blocks of data,
each chunk has a unique tag. This tag is a
4-character ASCII string that acts as
chunk ID. Various types of chunks are:
(i) RIFF chunk to define the
contents of RIFF file.
(ii) LIST chunk to store info on
data, copyright etc.
(iii) Sub-chunk: To add info to
primary chunk.
VIDEO Medium
Video signal representation: Has three aspects:
1. Visual representation: The televised
image should convey the spatial & temporal
content of the scene. Important measures
are:
• Vertical details & viewing distance: The
geometry of the field occupied by the TV
image is based on the ratio of the picture
width to height W/H which is called the
aspect ratio; conventionally it is 4/3.
• The viewing distance D determines the
angle h subtended by the picture height. The
measure of angle is given by D/H. Smallest
detail that can be reproduced in the image is
pixel. Ideally, each detail of a scene needs a
pixel. However, only 70% vertical detail is
presented by the scanning lines as some
details is lost in between the lines. This ratio
is called the Kell factor.
• Horizontal detail & picture width: Picture
width = 4/3 X Picture height.
• Total detail content of the image:
1. Vertical resolution = no. of
pixels in picture height.
 2. Picture width in pixels = Aspect
ratio X Hor. Resolution.
• Perception of depth: Third spatial dimension
depth depends upon angular separation of
images received by two eyes of viewer. In
flat image TV, degree of depth is perceived
from perspective appearance of subject
matter. Also, focal length of lenses
influences depth perception.
• Luminance & chrominance: Color vision is
achieved thru 3 signals – R,G,B that light
each portion of scene. These 3 signals are
conveyed separately to the input terminals
of the picture tube so that the tube
reproduces at each point the relative
intensities of R, G, B discerned by the
camera. During transmission of the signals
from camera to receiver (display), a different
division of signals in comparison to RGB is
used. The color encoding during
transmission uses luminance & two
chrominance signals.
• Temporal aspects of illumination:
▪ A discrete sequence of individual
pictures can be perceived as a
continuous sequence if the slightly
different individual pictures or
frames can be rapidly shown in
 succession & light is cut off between
the frames.
▪ For perception of continuity, two
things are needed:
1. Rate of image repetition must
be high for smooth motion
from frame to frame.
2. Rate must be high enough so
that persistence of vision
extends between flashes. This
property of human vision is
called boundary of motion
resolution.
● Continuity of motion: Perception of
continuous motion is with frame rate>15
frames /sec. Smooth video motion is at
30 frames/sec. For motion picture it is 24
frames/sec but appears smooth as it is
shown in big screen at dark, away from
the viewer.
• Flicker: Through slow motion, a periodic
fluctuation of brightness perception
occurs called Flicker. To avoid it, min
frame rate should be > 50 refresh
cycles/sec.
o A movie at 16 frames/sec has flicker
for which light wave is interrupted
twice additionally to reduce flicker.
 So, picture flicker rate = 3X16 = 48
Hz.
o In TV, flicker is reduced through use
of display refresh buffer. A full TV
picture is divided to two half pictures
consisting of interleaved scanning
lines. Each half picture is
transmitted in an interleaved
manner. If picture transmission
occurs at 25 Hz, the half pictures
must be scanned at 2X25 = 50 Hz.
This technique is called Interlacing
where each image is divided to two
frames having alternate line. The
even horizontal line are scanned
once & then the odd horizontal
lines. This reduces flicker without
increasing bandwidth.

▪ Blanking:
▪ Horizontal blanking: as the beam scans
from left to right, the electron beam of the
picture tube of TV is turned on & intensity is
modulated. When it flies back from right to
left side of next line, electron beam is
blanked/ turned off. This is called horizontal
blanking & the period for which it is blanked
off is called horizontal blanking period.
 ▪ Vertical blanking : When beam reaches the
right hand side of the bottom most
horizontal line of an image to be scanned of
a frame, it moves back quickly to the left
topmost corner of ext frame. This diagonal
movement is blanked off & called Vertical
blanking.
▪ Still image coding: Line drawings,
photographs, text page etc. can be coded
using
▪ Vector graphics: Every image is
represented as a collection of line
vectors. It is no more I dominant
use except for drafting.
▪ Raster Scan: Image is scanned
from the top left cornet to bottom
right corner in sequential order.
Image is divided to a collection of
horizontal lines.

COLOR MODELS

Color perception: We see an object when

light falling on it within visible range of spectrum
is reflected & detected by the eye. Each color
radiation has three properties: luminance, hue &
saturation – the corresponding human
perception being brightness, color & purity.
▪ Luminance: Property by virtue of which a
radiation appears to have more or less
intensity.
▪ Different hues of same color are produced
by radiation of different wavelength.
▪ Saturation: The colorfulness of a color.
When a color has more white mixed with it, it
is less saturated.
The human eye’s response is not same for all
colors. It responds maximum to yellow. It also
responds more to variations in brightness than
to color. These two factors are used in image
compression by discarding color components
not visually significant.

CHROMATIC MODELS: One parameter defines

luminance or brightness, the other two defines
color of pixels called chrominance. Two
chromatic components are required which are
added to get the final color. These are additive
models.
▪ RGB MODEL:
▪ Additive model
▪ Originally for image capture &
display
▪ R+G+B = 1
▪ CMYK Model
• Cyan, magenta, yellow & black
represent the image
• Subtractive model.

▪ YUV Model
Y stands for luminance , U,V for two color
components called chrominance.
Y = 0.299R + 0.587G + 0.114B
U = 0.596R - 0.247G - 0.322B
V = 0.211R – 0.532G + 0.312B
There are several YUV models another of
which is:
Y = 0.3R + 0.6G + 0.1B
U = B-Y
V = R-Y
If R=G=B=0; Y=U=V=0; it is a monochrome
grayscale image.
Q: Explain one additive colour model (RGB) and
one subtractive colour/chromatic model (YUV)
Q: What is luminance, hue and saturation for
color perception?
Q: Our eyes are more sensitive to luminance
(brightness) or chrominance (colour)? Our eyes
are more sensitive to luminance that is
brightness and less to colour

Q: What is perception of depth?

Q: How do you get perception of continuity?
What are the conditions for perception of
continuity?
Q: What should be the frame rate for continuous
video, for smooth video, for motion picture and
for television viewing? (For TV it is 50
frames/sec)

Q: What is horizontal blanking and vertical

blanking?
Q: Indian TVs have _________ horizontal
scanning lines (625)
Q: What is aspect ratio?

Q: What is kell factor? What is its value? 70% or

0.7 (0.69 is the actual value)

Q: Aspect ratio is: 4:3, 16:9, both 4:3 & 16:9?

Ans: Both 4:3 and 16:9

Q: What is optimum viewing angle while viewing

TV? 15 degree.
 ANIMATION
Display of animation: with raster systems,
animation objects made up of graphical
primitives like lines, polygon etc. must be scan
converted to their pix-map in their frame
buffer.
To show a rotating object, we can scan convert
to pixmap successive views from slightly
different positions, one after another. The scan
conversion must be done at least 10 times/sec
(preferably at 15-20 times/sec) to a reasonably
smooth effect; hence a new image must be
created in no more than 100 ms. From this 100
mS, a small portion is taken for scan conversion.
If 75 mS is taken for scan conversion, only 25
mS is left for erasing & redrawing the complete
object on the display which is not enough. To
prevent this double buffering is done. The frame
buffer is divided to two images each with half of
the bits per pixel of overall frame buffer. Lets
discuss an example with two halves of pixmap –
image0 & image1 .
Ex.
Load look-up table to display values as
background color.
Scan convert object into image0
Load look-up table to display only image0
Repeat
Scan convert object into image1
Load look-up table to display only image1
Rotate object data structure description
Scan convert object into image0
Load look-up table to display only image0
Rotate object data structure description
Until (termination condition)

If the rotating & scan converting takes time >

100mS, animation is slow but transition from one
image to next appears instantaneous.

Transmission of Animation

Animated objects may be symbolically

represented using graphical objects or scan
converted pixmap images. Transmission is done
by Symbolic representation where an animated
object ball is transmitted symbolically as a circle
along with the operating commands like roll the
ball etc. performed on the object & at the
receiver side, the animation is displayed as
discussed above.

METHODS OF CONTROLLING ANIMATION

1. Full explicit control: Simplest control. The
animator provides description of everything
that occurs in the animation. It may specify
the foll:
▪ Scaling, translation, rotation
▪ Provide key-frame information &
interpolation methods to use
between frames.
▪ In an interactive system, may allow
direct manipulation with a mouse,
joystick, data glove or other input
device.
2. Procedural control: Based on
communication between various objects to
determine its properties.
▪ In physically based systems, the
position of one object may influence
the motion of another like balls cannot
pass through walls.
▪ In anchor based systems, the
individual actors may pass their
positions to the other actors to affect
other’s behaviour.
3. Constraint based systems: Some objects in
physical world move in straight lines but
many move in a manner determined by
other objects with which they are in contact.
This compound motion may not be linear at
all. Such motion may be modeled by
constraints specifying an animated
sequence. It is much easier than explicit
control.
4. Tracking line action: Trajectories of objects
in the course of an animation may be
generated by tracking live action. Traditional
animation uses Rotoscoping. A film is made
in which people/animals act out parts of
characters in the animation & then the
animators draw over the film enhancing the
background & replacing human actors with
their animated equivalents.
5. Kinematics & Dynamics:
▪ Kinematics refers to position & velocity
of points. A kinematic description of a
scene: “The cube is at the origin at time
t = 0. It moves with a constant
acceleration in the direction (1, 1, 15)
thereafter.”
▪ On the contrary, dynamics takes into
account the physical laws that govern
kinematics like Newton’s laws of motion
 etc. A particle moves with an
acceleration proportional to the forces
acting on it & the proportionality
constant depends on the mass of the
particle. The dynamic description of the
particle would be “At time t=0 secs, the
cube is at position (0 m, 100m, 0 m).
The cube has a mass of 100 gms. The
force of gravity acts on the cube.” The
result of dynamic simulation of such a
model is that the cube falls.

ANIMATION LANGUAGES

THERE ARE THREE CATEGORIES:

1. Linear list notations: Each event in an
animation is described by a starting &
ending frame number & an action that is to
take place. The actions take parameter
statements.
42, 53, B, ROTATE, “PALM” 1, 30
The above statement means between
frames 42 & 53, rotate the object called
PALM about axis 1 by 30 deg.
determining amount of rotation at each
frame from table B.
Many other such link lists are available.
2. General purpose Languages: Animation
capability may be embedded within a
general purpose programming language.
ASAS is an example of such language
built on LISP. Main entities are vectors,
colors, polygons, solids, groups, points of
views, light etc. ASAS also has wide range
of geometric transformations that operate
on objects. The ASAS program fragment
below describes an animated sequence in
which the object called my_cube is spun
while the camera pans. The fragment is
evaluated at each frame to generate the
entire sequence.
(grasp my_cube); The cube becomes the
current object
(cw 0.05); Spin it clockwise by a small
amount
(grasp camera); Make the camera the
current object
(right panning speed); Move it to the right

3. Graphical languages: With textual

languages, we are unable to visualize the
action just by looking at the script. A real
time pre-viewer would help to solve the
problem for textual animation languages,
however, real time animation is still beyond
the scope of computer hardware.
Graphical animation languages describe
animation more visually. These
languages express & edit the changes
taking place in animation, Rather than
explicitly writing out descriptions of actions,
animator provides picture of the actions.
 COMPUTER-BASED ANIMATION
To animate something is to bring life to some
object. Animation covers all changes that have a
visual effect on the object. It may include time
varying positions (motion dynamics), shape,
color, transparency, structure & texture of an
object (update dynamics) and change in lighting,
camera, position, orientation & focus. A
computer based animation is an animation
performed by a computer using graphical tools
to provide visual effects.

BASIC CONCEPTS
1. Input process: Before computers can be
used, drawings MUST be DIGITIZED
because key frames (frames where entities
are at extreme positions) MUST be drawn.
This can be done thru optical scanning,
tracing drawing with data tablets etc.
Drawings may need to be post processed
(filtered) to clean glitches during input.
2. Composition stage: Here, the foreground &
background figures are combined to
generate the individual frames for final
animation. It can be performed with image
composition technique. These low
 resolution frames of an animation are
placed in a rectangular array to generate a
trail film (pencil test) by using pan-zoom
features available in some frame buffers.
The frame buffer can take a particular
portion of an image (Pan) & then enlarge it
to fit the entire screen (Zoom). This process
can be repeated on several frames of the
animation stored in the single image. If
done fast, it gives the effect of continuity.
3. In-between process: The animation of
movement from one position to another
needs a composition of frames with
intermediate positions (intermediate
frames) in between key frames. This is
called in between process performed thru
INTERPOLATION. The system only stores
the initial & final positions. Earliest
interpolation is linear interpolation (or
lerping).