NEW TRENDS IN IMAGE AND VIDEO COMPRESSION

Luis Torres* Edward J. Delp

Polytechnic University of Catalonia Purdue University
Barcelona, Spain West Lafayette, Indiana USA
[email protected] [email protected]

ABSTRACT presents a brief summary of the technology related to

present still image and video compression standards.
Further developments in the standards will also be
Image and video compression have been the object of
presented.
intensive research in the last thirty years. The field is
now mature as is proven by the large number of Section 3 presents ideas as to how compression
applications that make use of this technology. Digital techniques will evolve and where the state of the art
Video Broadcasting, Digital Versatile Disc, and will be in the future. We will also described new
Internet streaming are only a few of the applications trends in compression research such joint
that use compression technology. Image and video source/channel coding and scalable compression.
standards have played a key role in this deployment. Section 4 will introduce preliminary results of face
Now is time to ask: are there any new ideas that may coding in which a knowledge-based approach will be
advance the current technology? Have we reached a shown as a promising technique for very low bit rate
saturation point in image and video compression video coding. Section 5 describes media streaming
research? Although the future is very difficult to which is a new and exciting area for compression
predict, this paper will try to provide a brief overview research.
to where this exciting area is heading.
2. STANDARDS AND STATE OF THE ART
1. INTRODUCTION
Image and video coding are one of the most 2.1 Still Image Coding
important topics in image processing and digital For many years the Discrete Cosine Transform
communications. During the last thirty years we have (DCT) has represented the state of the art in still
witnessed a tremendous explosion in research and image coding. JPEG is the standard that has
applications in the visual communications field. incorporated this technology [1]. JPEG has been a
There is no doubt that the beginning of the new success and has been deployed in many applications
century revolves around the “information society.” reaching worldwide use. However, for some time it
Technologically speaking, the information society was very clear that a new still image coding standard
will be driven by audio and visual applications that needed to be introduced to serve the new range of
allow instant access to multimedia information. This applications which have emerged in the last years.
technological success would not be possible without The result is JPEG2000 that will be standardized at
image and video compression. The advent of coding the end of 2000. It is currently in the Final
standards, adopted in the past years, has allowed Committee Draft stage [2]. The JPEG2000 standard
people around world to experience the “digital age.” uses the Discrete Wavelet Transform. Tests have
Each standard represents the state of the art in indicated that at low data rates JPEG2000 provides
compression at the particular time that it was about 20% better compression efficiency for the same
adopted. It is important then, to summarise the state image quality than JPEG. JPEG2000 also offers a
of the art for each of the standards1. Section 2 new set of functionalities. These include error
resilience, arbitrarily shaped region of interest,
random access, lossless and lossy coding as well as a
*
This work has been partially supported by Grant TIC98-0422 of fully scalable bit stream. These functionalities
the Spanish Government and the Hypermedia ACTS European introduce more complexity for the encoder. MPEG-4
project to LT and by a grant from Texas Instruments to EJD.
1
has a “still image” mode known as Visual Texture
The reader should be aware that the authors have somewhat
differing views on the role and impact of compression standards.
Coding (VTC) which also uses wavelets but supports
LT feels very strongly that standards are important and necessary. less functionalities than JPEG2000 [3]. For a
EJD is somewhat skeptical of this view. We are still very good comparison between the JPEG2000 standard, JPEG,
friends.
MPEG-4 VTC and other lossless JPEG schemes see applications where the content provider wishes the
[4]. For further discussion on the role of image and user to obtain both the encoder and decoder from
video standards see [5]. them so that the provider can gain economic advan-
tage.
2.2 Video Coding
2.3.1 Still image coding
During the last ten years, the hybrid scheme
combining motion compensated prediction and DCT JPEG2000 represents the state of the art with respect
has represented the state of the art in video coding. to still image coding standards. This is mainly due to
This approach is used by the ITU H.261 and H.263 the 20% improvement in coding efficiency with res-
standards as well as for the MPEG-1 and MPEG-2 pect to the DCT as well as the new set of functio-
standards. However in 1993, the need to add new nalities incorporated. Non-linear wavelet decompo-
content-based functionalities and to provide the user sition may bring further improvement [7]. Other
the possibility to manipulate the audio-visual content improvements will include the investigation of color
was recognized and a new standard effort known as transformations for color images [8] and perceptual
MPEG-4 was launched. In addition to these functio- models [9].
nalities, MPEG-4 provides also the possibility of Although other techniques, such as fractal coding or
combining natural and synthetic content. MPEG-4 vector quantization have also being studied, they
phase 1 became an international standard in 1999 [3]. have not found their way into the standards. Other
MPEG-4 is having difficulties finding wide-spread alternate approaches such as “second generation tech-
use, mainly due to the protection of intellectual niques” [10] raised a lot of interest for the potential of
property and to the need to develop automatic and high compression ratios. However, they have not
efficient segmentation schemes. been able to provide very high quality. Second gene-
The frame-based part of MPEG-4 which incorporates ration techniques and, in particular, segmentation-
error resilience tools, is finding its way in the mobile based image coding schemes, have produced a coding
communications and Internet streaming. H.263, and approach more suitable for content access and
several variants of it [6], are also very much used in manipulation than for strictly coding applications.
mobile communication and streaming and it will be These schemes are the basis of MPEG-4.
interesting to see how these two standards compete in There are many schemes that may increase the coding
these applications. efficiency of JPEG2000. But all these schemes may
The natural video part of MPEG-4 is also based in only improve by a small amount. We believe that the
motion compensation prediction followed by the JPEG2000 framework will be widely used for many
DCT, the fundamental difference is that of adding the applications.
coding of the object shape. Due to its powerful
object-based approach, the use of the most efficient 2.3.2 Video coding
coding techniques, and the large variety of data types All the video coding standards based on motion
that it incorporates, MPEG-4 represents today the prediction and the DCT produce block artifacts at low
state-of-the-art in terms of visual data coding data rate. There has been a lot of work using post-
technology [5]. How MPEG-4 is deployed and what processing techniques to reduce blocking artifacts
applications will make use of its many functionalities [11, 12, 13]. A great deal of work has been done to
is still an open question. investigate the use of wavelets in video coding. This
work has taken mainly two directions. The first one is
2.3 What can be done to improve the standards? to code the prediction error of the hybrid scheme
using the DWT [14]. The second one is to use a full
Can something be done to “significantly” improve
3-D wavelet decomposition [15, 16]. Although these
the performance of compression techniques? How
approaches have reported coding efficiency
will this affect the standards? We believe that no
improvements with respect to the hybrid schemes,
significant improvements are to be expected in the
most of them are intended to provide further
near future. However, compression techniques that
functionalities such as scalability and progressive
require new types of functionalities driven by
transmission.
applications will be developed. For example, Internet
applications may require new types of techniques that One of the approaches that reports major improve-
support scalability modes tied to the network trans- ments using the hybrid approach is the one proposed
port. We may also see proprietary methods developed in [17]. Long-term memory prediction extends
that use variations on standards, such as the video motion compensation from the previous frame to
compression technique used by RealNetworks, for several past frames with the result of increased
coding efficiency. The approach is combined with based" coding approaches. Figure 1 shows this
affine motion compensation. Data rate savings classification according to [20].
between 20 and 50% are achieved using the test It can be seen from this classification that the coding
model of H.263+. The corresponding gains in PSNR community has reached third generation image and
are between 0.8 and 3 dB. video coding techniques. MPEG-4 provides segmen-
It can be said that MPEG-4 and H.263+ represent the tation-based approaches as well as model based video
state of the art in video coding. H.263+ provides a coding in the facial animation part of the standard.
framework for doing frame-based low to moderate
data rate robust compression. MPEG-4 combines 3.2 Coding through recognition and
frame-based and segmentation-based approaches reconstruction
along with the mixing of natural and synthetic
Which techniques fall within the “recognition and
content allowing efficient coding as well as content
reconstruction” fourth generation approaches? The
access and manipulation. There is no doubt that other
answer is coding through the understanding of the
schemes may improve the coding efficiency
established in MPEG-4 and H.263+ but no significant content. In particular, if we know that an image
breakthrough has been presented to date. The basic contains a face, a house, and a car we could develop
recognition techniques to identify the content as a
question remains: what is next? The next section will
previous step. Once the content is recognized, con-
try to provide some clues.
tent-based coding techniques can be applied to
3. NEW TRENDS IN IMAGE AND VIDEO encode each specific object. MPEG-4 provides a
COMPRESSION partial answer to this approach by using specific
techniques to encode faces and to animate them.
Before going any further, the following question has Some researchers have already addressed this pro-
to be raised: if digital storage is becoming so cheap blem. For instance, in [21] a face detection algorithm
and so wide spread and the available transmission is presented which helps to locate the face in a
channel bandwidth is increasing due to the deploy- videoconference application. Then, bits are assigned
ment of cable, fiber optics and ADSL modems, why in such a way that the face is encoded with more
is there a need to provide more powerful compression quality than the background.
schemes? The answer is, with no doubt, mobile video
transmission channels and Internet streaming. For a 3.3 Coding through metadata
discussion on the topic see [18, 19].
If it is clear that understanding the visual content
helps provide advanced image and video coding tech-
niques then the efforts of MPEG-7 may also help in
this context. MPEG-7 strives at specifying a standard
way of describing various types of audio-visual
information. Figure 2 gives a very simplified picture
of the elements that define the standard. The elements
that specify the description of the audio-visual
content are known as metadata.

Im a g e
a n a ly s is

F e a tu re C o n te n t S e a rc h
e x tr a c tio n d e s c r ip t io n e n g in e

Figure 1. Image and video coding classification T o o ls

M P E G -7
3.1 Image and video coding classification
Figure 2. MPEG-7 standard
In order to have a broad perspective, it is important to
understand the sequence of image and video coding Once the audio-visual content is described in terms of
developments expressed in terms of "generation- the metadata, the image is ready to be coded. Notice
that what is coded is not the image itself but the
description of the image (the metadata). An example 3.4 Coding through merging of natural and
will provide further insight. synthetic content
Let us assume that automatic tools to detect a face in In addition to the use of metadata, future video
a video sequence are available. Let us further coding schemes will merge natural and synthetic
simplify the visual content by assuming that we are content. This will allow an explosion of new
interested in high quality coding of a videoconference applications combining these two types of contents.
session. Prior to coding, the face is detected and MPEG-4 has provided a first step towards this
represented using metadata. In the case of faces, combination by providing efficient ways of face
some core experiments in MPEG-7 show that a face encoding and animation. However, more complex
can be well represented by a few coefficients, for structures are needed to model, code, and animate
instance by using the projection of the face on an any kind of object. The needs arisen in [23] are still
eigenspace previously defined. The image face can be valid today. No major step has been made concerning
well reconstructed, up to a certain quality, by coding the modeling of any arbitrary-shaped object. For
only a very few coefficients. In the next section, we some related work see [24].
will provide some very preliminary results using this
Video coding will become multi-modal and cross-
approach.
modal. Speech and audio will come to the rescue of
Once the face has been detected and coded, the video (or viceversa) by combining both fields in an
background remains to be coded. This can be done in intelligent way. To the best of our knowledge, the
many different ways. The simplest case is when the combination of speech and video for video coding
background is roughly coded using conventional purposes has not yet been reported. Some work has
schemes (1st generation coding). If the background is been done with respect to video indexing [25].
not important, then it can not even be transmitted and
the decoder adds some previously stored background 3.5 Other Trends in Video Compression:
to the transmitted image face. Streaming and Mobile Environments
For more complicated video sequences, we need to
The two most important applications in the future
recognize and to describe the visual content. If this is
will be wireless or mobile multimedia systems and
available, then coding is “only” a matter of assigning
streaming content over the Internet. While both
bits to the description of each visual object.
MPEG-4 and H.263+ have been proposed for these
MPEG-7 will provide mechanisms to fully describe a applications, more work needs to be done.
video sequence (in this section, a still image is
In both mobile and Internet streaming one major
considered a particular case of video sequence). This
problem that needs to be addressed is: how does one
means that knowledge of color and texture of objects,
handle errors due to packet loss and should the
shot boundaries, shot dissolves, shot fading and even
compression scheme adapt to these types of errors?
scene understanding of the video sequence will be
H.263+ [26] and MPEG-4 [27] both have excellent
known prior to encoding. All this information will be
error resilience and error concealment functionalities.
very useful to the encoding process. Hybrid schemes
could be made much more efficient, in the motion The issue of how the compression scheme should
compensation stage, if all this information is known adapt is one of both scalability and network transport
in advance. This approach to video coding is quite design. At a panel on the “Future of Video Compre-
new. For further information see [18, 22]. ssion” at the Picture Coding Symposium held in April
1999, it was agreed that rate scalability and temporal
It is also clear that these advances in video coding
scalability were important for media strea-ming
will be possible only if sophisticated image analysis
applications. It also appears that one may want to
tools (not part of the MPEG-7 standard) are deve-
design a compression scheme that is tuned to the
loped. The deployment of new and very advanced
channel over which the video will be transmitted. We
image analysis tools are one of the new trends in
are now seeing work done in this area with techni-
video coding. The final stage will be intelligent
ques such as multiple description coding [28, 29].
coding implemented through semantic coding. Once
a complete understanding of the scene is achieved, MPEG-4 is proposing a new “fine grain scalability”
we will be able to say (and simultaneously encode): mode and H.263+ is also examining how multiple
this is a scene that contains a car, a man, a road, and description approaches can be integrated into the
children playing in the background. However we standards. We are also seeing more work in how the
have to accept that we are still very far from this 5th compression techniques should be “matched” to the
generation schemes. network transport [30, 31, 32].
3.6 Protection of Intellectual Property Rights specific set of eigenfaces is less than a threshold, then
While the protection of intellectual property rights is the face is said to match the training image which
generated this set of eigenfaces. In this case we code
not a compression problem, it will have impact on the
the recognized face using only the five coefficients
standards. We are seeing content providers
used in the reconstruction. It is clear that the
demanding that methods exist for both conditional
corresponding eigenfaces of each person have to be
access and copy protection. MPEG-4 is studying
watermarking and other techniques. The newly transmitted previously to the decoder. However this
announced MPEG-21 [33] will address this in more can be done using conventional still image coding
techniques such as JPEG and no significant
detail.
increment in bit rate is generated.
4. FACE CODING USING RECOGNITION Figure 5 provides some results. Figure 5 shows the
AND RECONSTRUCTION original image Ana, the reconstruction of the detected
face image Ana using the eigenvectors and corres-
This section presents very preliminary results on face
ponding projected coefficients of the PCA using the
coding using recognition and reconstruction of visual training images of Ana and the error done. Figure 6
data. Although the main objective of this research shows the equivalent result for José Mari. Only 5 real
work has been for video face recognition [34] it can
numbers have been used to decode the shown images
be easily extended to face coding. Related work but
which means a very high compression ratio.
in a different context, has been presented in [35]. Our
application assumes that the video sequence to be Our scheme is at a very early stage of development
coded contains image faces whose identity is known and we have not yet designed any bit assignment
previously. A set of training images for each face scheme to encode the face and the background. Our
contained in the video sequence is previously known. purpose here is to show that face coding using
Figure 3 shows five views of the image Ana and recognition and reconstruction is a promising
Figure 4 five views of the image José Mari. These approach and to indicate that much more work needs
images come from the test sequences accepted in to be done in order to have good results. Although the
MPEG-7. presented results are not yet of very high quality, we
believe that image coding using recognition and
reconstruction may be the next step forward in video
coding. Good object models will be needed, though,
to encode any kind of object following this approach.

Figure 3. Five training views of the image Ana

Figure 4. Five training views of the image José Mari Figure 5. Decoded (reconstructed) image Ana. Left:
original image. Center: reconstructed image. Right:
Once these training images have been found (usually Error image.
coming from an image data base), a Principal
Component Analysis (PCA) is performed for each
individual using the corresponding training set of
each person. This means that we obtain a PCA
decomposition for every face image to be coded. The
PCA is done previously to the encoding process. The
first stage of the encoding process is automatic face
segmentation and extraction of the video sequence.
To that end we have used the face detection
algorithm proposed in [36]. Once detected, all faces
Figure 6. Decoded (reconstructed) image José Mari.
have been projected and reconstructed using each set
Left: original image. Center: reconstructed image.
of different eigenvectors (called eigenfaces) obtained
Right: Error image.
in the PCA stage. If the reconstruction error using a
Although not directly related to source video coding,
let us mention that many efforts are being dedicated January 23-29, 1999, San Jose, California, pp.
to provide robust video transmission through a varie- 282-293.
ty of channels, Internet and mobile being the most
[9] N. S. Jayant, J. D. Johnston and R. J. Safranek,
significant. For a good review of the topic see [37].
“Signal compression based on models of human
perception,” Proceedings of the IEEE, vol. 81,
5. CONCLUSIONS
no. 10, October 1993, pp. 1385-1422.
We feel that any advances in compression techniques [10] L. Torres and M. Kunt, Editors, Video coding:
will be driven by applications such as databases, the second generation approach, Kluwer
wireless and Internet streaming. New semantic-based Academic Publishers, Boston, USA, January
techniques so far have promised much but have 1996.
delivered little new results. Much work needs to be in
the area of segmentation of video. [11] K. K. Pong and T. K. Kan, “Optimum loop filter
in hybrid coders,” IEEE Transactions on
Circuits and Systems in Video Technology, vol.
REFERENCES 4, no.2, pp. 158-167, 1997.
[12] T. O’Rourke and R. L. Stevenson, “Improved
[1] W. B. Pennebaker and J. L. Mitchell, JPEG Still image decompression for reduced transform
Image Data Compresion Standard, Van coding artifacts,” IEEE Transactions on
Nostrand Reihold, New York, 1992. Circuits and Systems for Video Technology, vol.
[2] ISO/IEC JTC 1/SC 29/WG 1, ISO/IEC FCD 5, no. 6, pp. 490-499, December 1995.
15444-1: Information Technology – JPEG 2000 [13] R. Llados-Bernaus, M. A. Robertson and R. L.
image coding system: Core coding system, WG Stevenson, “A stochastic technique for the
1 N 1646, March 2000. removal of artifacts in compressed images and
http://www.jpeg.org/FCD15444-1.htm video,” in Recovery Techniques for Image and
[3] ISO/IEC ISO/IEC 14496-2: 1999: Information Video Compression and Transmission, Kluwer,
technology – Coding of audio visual objects – 1998.
Part 2: Visual, December 1999. [14] K. Shen and E. J. Delp, “Wavelet based rate
[4] D. Santa Cruz and T. Ebrahimi, “A study of scalable video compression,” IEEE Transac-
JPEG2000 still image coding versus other tions on Circuits and Systems for Video Techno-
standards,” Proceedings of the European Signal logy, vol. 9, no. 1, February 1999, pp. 109-122.
Processing Conference (EUSIPCO), Tampere, [15] C. I. Podilchuk, N. S. Jayant, and N. Farvardin,
Finland, September 5-8, 2000. “Three-dimensional subband coding of video,”
[5] F. Pereira, “Visual data representation: recent IEEE Transactions on Image Processing, vol. 4,
achievements and future developments,” no. 2, pp. 125-139, February 1995.
Proceedings of the European Signal Processing [16] D. Taubman and A. Zakhor, “Multirate 3-D
Conference (EUSIPCO), Tampere, Finland, subband coding of video,” IEEE Transactions
September 5-8, 2000. on Image Processing, vol. 3, no. 5, pp. 572-588,
[6] G. Côté, B. Erol, M. Gallant, and F. Kossentini, September 1994.
"H.263+: Video coding at low bit rates," IEEE [17] T. Wiegand, E. Steinbach, and B. Girod, “Long
Transactions on Circuit and Systems for Video term memory prediction using affine
Technology, vol. 8, no. 7, November 1998. compensation,” Proceedings of the IEEE
[7] D. Wajcer, D. Stanhill, and Y. Zeevi, “Repre- International Conference on Image Processing,
sentation and coding of images with nonsepa- Kobe, Japan, October 1999.
rable two-dimensional wavelet,” Proceedings of [18] R. Schäfer, G. Heising, and A. Smolic,
the IEEE International Conference on Image “Improving image compression - is it worth the
Processing, Chicago, USA, October 1998. effort?” Proceedings of the European Signal
[8] M. Saenz, P. Salama, K. Shen and E. J. Delp, Processing Conference (EUSIPCO), Tampere,
“An evaluation of color embedded wavelet Finland, September 5-8, 2000.
image compression techniques,” Proceedings of [19] M. Reha Civanlar and A. Murat Teklap, “Real-
the SPIE/IS&T Conference on Visual time Video over the Internet,” Signal
Communications and Image Processing (VCIP), Processing: Image Communication, vol. 15, no.
 1-2, pp. 1-5, September 1999 (Special issue on [30] W. Tan and A. Zakhor, “Real-time Internet
streaming). video using error resilient scalable compression
[20] H. Harashima, K.Aizawa, and T. Saito, “Model- and TCP-friendly transport protocol,” IEEE
based analysis synthesis coding of video- Transactions on Multimedia, vol. 1, no. 2, pp.
telephone images – conception and basic study 172-186, June 1999.
of intelligent image coding,” Transactions [31] H. Radha, Y. Chen, K. Parthasarathy and R.
IEICE, vol. E72, no. 5, pp. 452-458, 1989. Cohen, “Scalable Internet video using MPEG-
[21] J. Karlekar and U. B. Desai, “Content-based 4,” Signal Processing: Image Communication,
very low bit-rate video coding using wavelet vol. 15, no. 1-2, pp. 95-126, September 1999.
transform,” Proceedings of the IEEE Interna- [32] U. Horn, K. Stuhlmüller, M. Link and B. Girod,
tional Conference on Image Processing, Kobe, “Robust Internet video transmission based on
Japan, October 1999. scalable coding and unequal error protection,”
[22] P. Salembier and O. Avaro, “MPEG-7: Signal Processing: Image Communication, vol.
Multimedia content description interface,” 15, no. 1-2, pp. 77-94, September 1999
Workshop on MPEG-21, Noordwijkerhout, the [33] ISO/IECJTC1/SC29/WG11/N3300, MPEG-
Netherlands, March 20-21, 2000. 21 Multimedia Framework, Noordwijkerhout –
http://www.cselt.it/mpeg/events/mpeg21/ March 2000.
[23] D. Pearson, “Developments in model-based [34] L. Torres, L. Lorente and J. Vilà, “Face
coding,” Proceedings of the IEEE, vol. 86, no. recognition using self-eigenfaces,” Proceedings
6, pp. 892-906, June 1995. of the International Syposium on Image/Video
[24] V. Vaerman, G. Menegaz, and J. P. Thiran, “A Communications Over Fixed and Mobile
parametric hybrid model used for multidi- Networks, Rabat, Morocco, pp. 44-47, April
mensional object representation,” Proceedings 2000.
of the IEEE International Conference on Image [35] B. Moghaddam, A. Pentland, “Probabilistic
Processing, Kobe, Japan, October 1999. visual learning for object representation”, IEEE
[25] T. Huang, “From video indexing to multimedia Transactions on Pattern Analysis and Machine
understanding,” Proceedings of the 1999 Inter- Intelligence, vol. 19, no. 7, pp. 696-710, July
national Workshop on Very Low Bitrate Video 1997.
Coding, Kyoto, Japan, October 1999. (Keynote [36] F. Marqués, V. Vilaplana, and A. Buxes,
speech.) “Human face segmentation and tracking using
[26] S. Wenger, G. Knorr, J. Ott, and F. Kossentini, connected operators and partition projection,”
“Error resilience support in H.263+,” IEEE Proceedings of the IEEE International Confere-
Transactions on Circuits and Systems for Video nce on Image Processing, Kobe, Japan, October
Technology, vol. 8, no. 7, pp. 867-877, Novem- 1999.
ber 1998. [37] Special session on Robust Video, Proceedings
[27] R. Talluri, “Error-resilient video coding in the of the IEEE International Conference on Image
ISO MPEG-4 Standard,” IEEE Communications Processing, Kobe, Japan, October 1999.
Magazine, vol. 2, no. 6, pp. 112-119, June 1999.
[28] S. D. Servetto, K. Ramchandran, V.A.
Vaishampayan, and K. Nahrstedt, “Multiple
description wavelet based image coding,” IEEE
Transactions on Image Processing, vol. 9, no. 5,
pp. 813-826, May 2000.
[29] S. D. Servetto and K. Nahrstedt, “Video
streaming over the public Internet: Multiple
description codes and adaptive transport
protocols,” Proceedings of the 1999 Inter-
national Conference on Image Processing,
Kobe, Japan, October 1999.