MPEG

(Video Compression)

Subhasis Chaudhuri
Dept of Electrical Engineering
IIT Mumbai

2-1 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Motion JPEG
(Doing it frame by frame)

• Split the video into individual frames and run JPEG

on each frame separately, this is called motion
JPEG.
• Takes no advantage of any correlation between
successive frames.
• Temporal redundancy should be taken care of.
• Simple and easier to implement!
2-2 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 Illustration of Temporal Redundancy
Good amount of temporal similarity
between adjacent frames in a video sequence.
But ……
not exactly the same either!!!

2-3 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Motion Compensation – Basic Idea

• A video consists of a nearly static background with one or more

moving foreground objects.
• Encode the first frame using JPEG and then use this frame as the
reference image.
• Split the second frame into blocks and compare each block with
an equivalent block in the reference frame.
• Encode only the difference.
• Continue with subsequent frames.

2-4 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Motion Compensation – Block Matching
How do we match the blocks?
•Matching criteria:
•Find the blocks in frames n and n-1 that are very close in intensity –
Called block matching.
•Use minimum mean square error (MMSE) between block.
MSE = ∑ ( I n +1 ( x + d x , y + d y ) − I n ( x, y )) 2
x, y
Matching block size:
•Size affects the coding efficiency – more MVs for smaller blocks,
more MMSE for larger blocks.
•block size in MPEG : 16×16 pixels.

2-5 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Motion Compensation – Motion Vectors
• In absence of static back / foreground, compute the displacement
of the block.
• Motion vector is used by the decoder to determine the location of
a block in the current frame from the previous frame.
• Motion vector should ideally be zero for a static background – but
usually not so.

2-6 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Motion Compensation – Search Space
Search range (How far do we look for a match?) :

• More computation if we need to search everywhere.

• Restrict search to a small neighborhood of the block.
• Small search-space for a slow motion video.
• Need for larger search area for a faster sequence.
• More MMSE (need for better residue coding) if search
space is too restricted.
• Usually the motion is faster horizontally than vertically.
• How about a search range: ±(60 × 30) pixels ?
2-7 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 Encoding the Residuals
• The difference between the block being coded and
its best match in previous frame is known as
residual.
• The residuals should be encoded and transmitted
along with the motion vector to reconstruct the block.
• If the MMSE is very large intra-code the macro-
block for increased efficiency.
• The residuals should be statistically coded.
2-8 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 Video Coding Standards
• MPEG – 1
• MPEG – 2
• MPEG – 4
• MPEG – 7 (Ongoing)
• MPEG – 21 (Someday …)
• Compare these to basic video coding
standards like H.261 or H.263 for low bit rate
coding.
2-9 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 MPEG -- 1

• MPEG: Moving Pictures Experts Group.

• MPEG-1 : Video compression at 1.5 Mb/s.
• It is asymmetric -- the complexity of the encoder is
much higher than that of the decoder.

Horizontal resolution ≤768 pixels

Vertical resolution ≤ 576 pixels
Picture rate ≤ 30 frames/s
Bit rate ≤ 1,856,000 bits/s

2-10 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 MPEG: Video Encoding

The MPEG standards

• do not define an encoding process
• define syntax of the coded stream
• define a decoding process

2-11 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Frame Types in MPEG
• Intra frames (I-frames)
• A I-frame is encoded using only information from within that frame
(intra coded) -- no temporal compression (inter coded).
• Non-intra frames (P-frames and B-frames)
• motion compensated information will be used for coding.
• P frame (predicted frame) use preceeding frame as reference
image
• B frame (bidirectional frame) use both preceeding frame and
following P/I frame as reference images

2-12 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Motion estimation for different frames

2-13 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Reconstructing a reference frame that
will be the same as at the decoder

Frame Variable To
being DCT Quantize length transmit
coded code buffer

Reference Inverse De-

frame DCT quantize

2-14 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 I P B B B P B B B P B B B

1 5 2 3 4 9 6 7 8 13 10 11 12
A typical group of pictures in coding order
I B B B P B B B P B B B P

A typical group of pictures in display order

2-15 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Coding of Macroblock
Y CB CR

0 1
4 5
2 3

♣ ♣ Spatial sampling relationship for MPEG-1

♦ ♣ -- Luminance sample
♣ ♣ ♦ -- Color difference sample
2-16 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 Coding of Macroblock (cont.)

Intra coding of macroblocks

– just as what JPEG does
– MPEG has two default quantization tables, one for intra
coding, another one for non-intra coding of residuals

8 16 19 22 26 27 29 34 16 11 10 16 24 40 51 61
16 16 22 24 27 29 34 37 12 12 14 19 26 58 60 55
19 22 26 27 29 34 34 38 14 13 16 24 40 57 69 56
22 22 26 27 29 34 37 40 14 17 22 29 51 87 80 62
22 26 27 29 32 35 40 48 18 22 37 56 68 109 103 77
26 27 29 32 35 40 48 58 24 35 55 64 81 104 113 92
26 27 29 34 38 46 56 69 49 64 78 87 103 121 120 101
27 29 35 38 46 56 69 83 72 92 95 98 112 100 103 99

MPEG quantization table(for intra coding) JPEG quantization table (luminance)

2-17 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Coding of Macroblock (cont.)
Non-intra coding of macroblocks
• Intra code the macroblock -- in case we fail to find a
reasonable match in motion estimation.
• Use motion estimation to find the nearest match and
get the motion vector. Only luminance samples are
used in motion estimation.
• Treat each DCT block in MB separately. Encode the
residuals by DCT as in intra coding.
• Apply this to all six blocks in the macroblock.
• Motion vectors are coded predictively.

2-18 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Coding of Macroblock (cont.)

• P-frames
• If the block is skipped, send a “skip” code.
• else compare the number of total bits of inter and intra
coding, choose the more efficient one. Mark this block
accordingly.

2-19 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Coding of Macroblock (cont.)
• B-frames
• Encoder forms a prediction error MB from either
of the MBs or from their average.
• Motion vector estimation is performed twice.
• Prediction error is encoded using a block-based
DCT.
• Encoder needs to reorder pictures because B-
frames always arrive late.

2-20 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Building up the Bit Stream
The six layers of MPEG video bit stream
• Sequence Layer: video clip, complete program item.
• Group of Pictures Layer (GOP): include three different
coding ways.
• Frame Layer
• Slice Layer: in case the data is lost or corrupted.
• Macroblock Layer: 16×16 luminance block.
• Block Layer (DCT unit)

2-21 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Structure of the Coded Bit-Stream
• Sequence layer: picture dimensions,
pixel aspect ratio, picture rate, minimum GOP-1 GOP-2 GOP-n
buffer size, DCT quantization matrices
• GOP layer: will have one I picture, start Sequence layer
with I or B picture, end with I or P
picture, has closed GOP flag, timing I B B B P B B.. GOP layer
info, user data
• Picture layer: temporal ref number,
picture type, synchronization info,
Slice layer
resolution, range of motion vectors Slice-1
• Slices: position of slice in picture, Slice-2 mb-1 mb-2 mb-n
quantization scale factor
…
• Macroblock: position, H and V motion Macroblock layer
vectors, which blocks are coded and Slice-N
01
transmitted 23 4 5
Picture layer 8x8 block
2-22 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 MPEG-2

• Supports higher bit rates 80-100 Mbps for HDTV

instead of 1.5 Mbps for MPEG-1.
• Supports a larger number of applications
• Supports interlaced and non-interlaced frame
• Different color subsampling modes e.g., 4:2:2, 4:2:0,
4:4:4
• Flexible quantization schemes – can be changed at
picture level
• Scalable bit-streams
2-23 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 Effects of Interlacing
• Fields or frame pictures can be encoded
• Frame prediction: current frame predicted from previous frame
• Field prediction:
• Top and bottom fields of reference frame predicts first field
• Bottom field of previous frame and top field of current frame
predicts the bottom field of current frame
• 16 X 8 motion compensation mode
• A macroblock may have two of them
• A B picture macroblock may have four!
• Top field of current frame is predicted from two motion vectors
coming from the top and bottom field of reference frame.

2-24 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 A Simplified MPEG encoder
Scale Rate Buffer
factor controller fullness

IN Variable-
Frame Transmit OUT
DCT Quantize length
recorder buffer
coder
DC

Motion Prediction De- Prediction

predictor quantize encoder

Reference Inverse
frame DCT

Motion vectors

2-25 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 MPEG-4
• Objective
• Standardize algorithms for audiovisual coding in
multimedia applications allowing for
• Interactivity
• High compression
• Scalability of audio and video content
• Support for natural and synthetic audio and video
• The Idea
• An audiovisual scene is a coded representation of
audiovisual objects related in space and time
2-26 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 Scopes

• Author : greater production flexibility and reusability

• Network Service Provider : Offering transport
information which can be interpreted on various
network platforms
• End user : Higher levels of interaction with content
within the limits set by the author.

2-27 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Objectives
• Interactivity : Interacting with different audio-visual
objects

• Scalability : Adopting contents to match bandwidth

• Reusability : For both tools and data

2-28 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 MPEG-4: Composition
• A/V object
• Video objects within a scene
• The background
• An instrument or voice
• Coded independently
• A/V scene
• Mixture of natural or synthetic objects
• Individual bit-streams multiplexed and transmitted
• One or more channels
• Each channel may have its own QoS.
2-29 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 Video Object Plane (VOP)
• Video frame union of segmented regions with
arbitrary shape (VOP)
• Shape, motion and texture of VOPs for a particular video
object is encoded into a video object layer (VOL).
• Encode
• VOL identifiers
• Composition information
• Overlapping configuration of VOPs

2-30 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Coding
• Shape coding
• Shape information in alpha planes
• Transparency of shape encoded
• Inter and intra shape coding functions
• After shape coding each VOP in a VO is partitioned
into non-overlapping macroblocks
• Motion coding
• Shift parameter w.r.t. reference window
• Standard macroblock, or
• Contour macroblock
2-31 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 Coding (cont.)
• Texture coding
• Intra-VOPs, residual errors from motion compensation are DCT coded
like MPEG-1
• 4 luminance and 2 chrominance blocks in a macroblock
• P-VOPs (prediction error blocks) may not conform to VOP boundary
• Pixels outside the active area are set to a constant value
• Standard compression
• Efficient prediction of DC and AC components from intra and inter
coded blocks
• Multiplexing
• Shape → motion → texture coded data
• Motion and DCT coefficients can be jointly (H.263) or individually
coded
2-32 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG
 Logical Structure of a Scene

Scene

Person Background Video Synthetic Objects

Dynosaur Jatayu
Shape Voice

2-33 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 Scene Description
• MPEG uses a binary language called BInary Format for
Scenes (BIFS) – similar to VRML.

• The standard differentiates parameters used to improve

the coding efficiency of an object (motion vectors in video
coding), and the ones used as modifiers of an object (its
position in the scene).

• Modification in the latter does not require re-decoding the

primitive media objects.

2-34 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 An MPEG-4 Example

2-35 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG

 An MPEG-4 Scene

2-36 Copyright  Subhasis Chaudhuri, 2003 Multimedia Course MPEG