Chapter 3: Multimedia Data

representation

1
 Multimedia Data Basics
Multimedia systems/applications must deal with the
generation
manipulation
storage
presentation
Communication of information/data
Lets consider some broad implications of the above

2
Discrete v Continuous Media
• All data must be in the form of digital information.
• The data may be in a variety of formats:
 text,
 graphics,
 Images,
 audio,
 video.

3
Synchronization
A majority of this data is large and the different media may
need synchronization:

• The data will usually have temporal relationships as an integral

property.

4
 Static and Continuous Media
Static or Discrete Media:
• Some media is time independent:
 Normal data, text, single images, graphics
are examples.
Continuous Media:
• Time dependent Media:
 Video, animation and audio are examples.

5
 Multimedia Data Representation
Issues to be covered:
• Digital Audio
Digital Audio Synthesis
MIDI — Synthesis and Compression Control
Digital Audio Signal Processing/Audio Effects
• Graphics/Image Formats
Colour Representation/Human Colour Perception
• Digital Video
Chroma Subsampling
6
 Analog and Digital Signal Conversion
The world we sense is full of analog signals:
• Electrical sensors convert the medium they sense into
electrical signals
 E.g. transducers, thermo couples: temperature sensor,
microphones: acoustic sensor
Cameras (Still and Video): light sensor.
 (usually) continuous Analog signals(e.g. Sound and Light)
• Analog: continuous signals must be converted or
digitised for computer processing.
• Digital: discrete digital signals that computer can readily
7
deal with.
 Analog to Digital Converter (ADC)
• Special hardware devices: Analog-to-Digital converters.
E.g. Audio:

Take analog signals from analog sensor (e.g. microphone)

and digitally sample data

8
 Digital to Analog Converter (DAC)
• Playback – a converse operation to Digital-to-Analog :

• Takes digital signal, possible after modification by computer (e.g.

volume change, equalisation)
• Outputs an analog signal that may be played by analog output
device (e.g. loudspeaker, RGB monitor/display) 9
Analog-to-Digital-to-Analog Pipeline
• Begins at the conversion from the analog input and ends at the
conversion from the output of the processing system to the analog
output as shown:

10
Analog-to-Digital-to-Analog Pipeline
• Anti-aliasing filters (major part of Analog Conditioning ) are needed at the input to
remove frequencies above the sampling limit that would result in aliasing.
The anti-aliasing filter at the output removes the aliases that result from the
sampling (see sampling theorem).
• After the anti-aliasing filter, the analog/digital converter (ADC)
quantizes the continuous input into discrete levels.
• After digital processing, the output of the system is given to a digital/analog
converter (DAC) which converts the discrete levels into continuous voltages or
currents.
• This output must also be filtered with a low pass filter to remove the aliases from
the sampling. Subsequent processing can include further filtering, mixing, or
other operations. However, these will not be discussed further in this course.
11
 Multimedia Data: Input and format
How to capture and store each Media format?

Note that text and graphics (and some images) are mainly generated directly by
computer/device (e.g. drawing/painting programs) and do not require digitising:

They are generated directly in some (usually binary) format.

• Printed text and some hand written text can be scanned via Optical Character
Recognition
• Hand written text could also be digitised by electronic pen sensing
• Printed imagery/graphics can be scanned directly to image formats.

12
 Text and Static Data
Source: keyboard, speech input, optical character
recognition, data stored on disk.

• Stored and input character by character:

 Storage: 1 byte per character (text or format character),
e.g. ASCI I; more bytes for Unicode.
 For other forms of data (e.g. Spreadsheet files). May store as text (with
formatting, e.g. CSV – Comma-Separated Values) or may use binary
encoding.
13
 Text and Static Data
• Formatted Text: Raw text or formatted text e.g HTML,
Rich Text Format (RTF), Word or a program language
source (Java, Python, MATLAB etc.)
• Data Not temporal — BUT may have natural implied
sequence e.g. HTML format sequence, Sequence of
Java program statements.
• Size Not significant w.r.t. other Multimedia data formats.
• Compression: convenient to bundle files for archiving
and transmission of larger files. E.g. Zip, RAR, 7-zip.
General purpose compression programs may not work
well for other media types: audio, image, video etc.
14
 Graphics
• Format: constructed by the composition of primitive
objects such as lines, polygons, circles, curves and arcs.
• Input: Graphics are usually generated by a graphics editor
program (e.g. Illustrator, Freehand) or automatically by a
program (e.g. Postscript).

15
 Graphics
• Graphics input devices: keyboard (for text and cursor control),
mouse, trackball or graphics tablet.
• Graphics are usually selectable and editable or revisable (unlike
images).

• Graphics files usually store the primitive assembly

• Do not take up a very high storage overhead.
16
 Graphics
• Graphics standards: OpenGL - Open Graphics Library, a standard
specification defining a cross-language, cross-platform API for
writing applications that produce 2D/3D graphics.

• Animation: can be generated via a sequence of slightly

changed graphics
 2D animation: e.g.Flash — Key frame interpolation:
tweening: motion & shape

17
 Graphics
Animation
3D animation: e.g. Maya.
Change of shape/texture/position, lighting, camera
Graphics animation is compact
– suitable for network transmission (e.g. Flash)

18
 Images
• Still pictures which (uncompressed) are represented as
a bitmap (a grid of pixels).

19
 Images
• Input: scanned for photographs or pictures using a digital scanner or from a
digital camera.
• Input: May also be generated by programs similar to graphics or animation
programs.
• Analog sources will require digitising.
• Stored at 1 bit per pixel (Black and White), 8 Bits per pixel (Grey Scale, Colour
Map) or 24 Bits per pixel (True Colour)
• Size: a 512x512 Grey scale image takes up 1/4 MB, a 512x512 24 bits image
takes 3/4 MB with no compression.
• This overhead soon increases with image size — modern high digital camera
10+ Mega pixels ≈ 29MB uncompressed!
• Compression is commonly applied. 20
 Images
• Can usually only edit individual or groups of pixels in an Image editing
application, e.g. photoshop.

21
 Sound
What is Sound?

Sound Generation
Source — Generates Sound
Air Pressure changes
Electrical — Loud Speaker
Acoustic —Direct Pressure Variations

Sound Reception
Destination — Receives Sound
Electrical — Microphone produces electric signal
Ears — Responds to pressure hear sound
(MPEG Audio —exploits this fact)
22
 Digitising Sound

• Microphone:
Receives sound
Converts to analog signal.
• Computer like discrete entities need to convert Analog-to-Digital
 Dedicated Hardware (e.g. Sound card) Also known as
Digital Sampling

23
 Audio
• Audio signals are continuous analog signals.
Input: microphones and then digitised and stored

• CD Quality Audio requires 16-bit sampling at 44.1 KHz:even higher

audiophile rates (e.g. 24-bit, 96 KHz)
• 1 Minute of Mono CD quality (uncompressed) audio = 5 MB.
StereoCD quality (uncompressed) audio= 10 MB.
• Usually compressed (E.g. MP3, AAC, Flac, OggVorbis)
24
 Video
• Input: Analog Video is usually captured by a video camera
and then digitised, although digital video cameras now
essentially perform both tasks.
• There are a variety of video (analog and digital) formats
• Raw video can be regarded as being a series of single
images. There are typically 25, 30 or 50 frames per second.

25
 Video
Video Size :
• A 512x512 size monochrome video images take 25*0.25 = 6.25MB
for a second to store uncompressed.
• Typical PAL digital video (720×576 pixels per colour frame) ≈ 1.2×25
= 30MB for a second to store uncompressed.
• High Definition video on Blu-ray (upto 1920×1080 = 2Megapixels per
frame) ≈ 6×25 = 150 MB for a second to store uncompressed, i.e.9 GB
for a minute to store uncompressed. (There are higher possible frame
rates!)
• Digital video clearly needs to be compressed. 26
 General Themes across all above

• Sampling/Digitisation
Sampling Artifacts —Aliasing
• Compression requirements
Data formats especially size
Human Perception→compression ideas

27
 Sample Rates and Bit Size
Bit Size —Quantisation
How do we store each sample value (Quantisation)?
8Bit Value (0-255)
16Bit Value (Integer) (0-65535)

Sample Rate
How many Samples to take?
11.025 KHz — Speech (Telephone 8 KHz)
22.05KHz — Low Grade Audio
(WWW Audio, AMRadio)
44.1KHz —CD Quality

28
 Digital Sampling
Sampling process basically involves:
• Measuring the analog signal at regular discrete intervals
• Recording the value at these points

29
Digital Sampling

30
 Nyquist’s Sampling Theorem
The Sampling Frequency is critical to
the accurate reproduction of a digital
version of an analog waveform

Nyquist’s Sampling Theorem

The Sampling frequency for a signal must be at least
twice the highest frequency component in the signal.

31
 Conclusions
• The world we sense is full of analog signals; however,
• the processing of multimedia data requires that these
have the form of digital signals,
• the Nyquist's sampling theorem allow analog signals
to be translated into digital signals to be processed and
stored as they were generated by computers,
• digital signals need to be compressed for their high
memory usage.

32