Data Compression: Lecture 10

Dictionary Techniques
• In many applications, the output of the source consists of recurring
patterns e.g. text source in which certain patterns or words recur
constantly.
• Also, there are certain patterns that simply do not occur, or if they do,
occur with great rarity e.g. we can be reasonably sure that the word
Limpopo1 occurs in a very small fraction of the text sources in
existence.
• A very reasonable approach to encoding such sources is to keep a list,
or dictionary, of frequently occurring patterns.
• When these patterns appear in the source output, they are encoded
with a reference to the dictionary.
• If the pattern does not appear in the dictionary, then it can be
encoded using some other, less efficient, method.
• For this technique to be effective, the class of frequently occurring
patterns, and hence the size of the dictionary, must be much smaller
than the number of all possible patterns.
• Suppose we have a particular text that consists of
four-character words, three characters from the
26 lowercase letters of the English alphabet
followed by a punctuation mark of total 32
alphabets.
• If we were to encode the text source one
character at a time, treating each character as an
equally likely event, we would need 5 bits per
character.
• Treating all 324 ( = 220 = 1,048,576) four-character
patterns as equally likely, we have a code that
assigns 20 bits to each four-character pattern.
• Let us now put the 256 most likely four-character
patterns into a dictionary.
• The transmission scheme works as follows:
– Whenever we want to send a pattern that exists in the
dictionary, we will send a 1-bit flag, say, a 0, followed
by an 8-bit index corresponding to the entry in the
dictionary.
– If the pattern is not in the dictionary, we will send a 1
followed by the 20-bit encoding of the pattern.
– If the pattern we encounter is not in the dictionary,
we will actually use more bits than in the original
scheme, 21 instead of 20.
– But if it is in the dictionary, we will send only 9 bits.
– If the probability of encountering a pattern from the
dictionary is p, then the average number of bits per
pattern R is given by
• For scheme to be useful, R should have a value
less than 20.
• This happens when p ≥ 0.084.
• This does not seem like a very large number.
• For better performance, p should be as large
as possible.
 Static Dictionary
• Choosing a static dictionary technique is most
appropriate when considerable prior
knowledge about the source is available.
• This technique is especially suitable for use in
specific applications.
• In such applications, it is highly efficient to
design a compression scheme based on a
static dictionary containing the recurring
patterns.
 Adaptive Dictionary-The LZ77 Approach
• In this approach, the dictionary is simply a portion
of the previously encoded sequence.
• The encoder examines the input sequence through
a sliding window
• The window consists of two parts, a search buffer
that contains a portion of the recently encoded
sequence, and a look-ahead buffer that contains the
next portion of the sequence to be encoded.
• To encode the sequence in the look-ahead buffer,
the encoder moves a search pointer back through
the search buffer until it encounters a match to
the first symbol in the look-ahead buffer.
• The distance of the pointer from the look-ahead
buffer is called the offset.
• The encoder then examines the symbols following
the symbol at the pointer location to see if they
match consecutive symbols in the look-ahead
buffer.
• The number of consecutive symbols in the search
buffer that match consecutive symbols in the look
-ahead buffer, starting with the first symbol, is
called the length of the match.
• The encoder searches the search buffer for the
longest match.
• Once the longest match has been found, the encoder encodes it with
a triple <o, l, c>, where o is the offset, l is the length of the match,
and c is the codeword corresponding to the symbol in the look-ahead
buffer that follows the match.
• The pointer in the example below is pointing to the beginning of the
longest match.
• The offset o in this case is 7, the length of the match l is 4, and the
symbol in the look-ahead buffer following the match is r.
• The reason for sending the third element in the triple is to take care
of the situation where no match for the symbol in the look-ahead
buffer can be found in the search buffer.
• In this case, the offset and match-length values are set to 0, and the
third element of the triple is the code for the symbol itself.
• If the size of the search buffer is S, the size of
the window (search and look-ahead buffers) is
W, and the size of the source alphabet is A,
then the number of bits needed to code the
triple using fixed-length codes is
 LZ77: Example
• Suppose the sequence to be encoded is

• Suppose the length of the window is 13, the

size of the look-ahead buffer is six, and the
current condition is as follows:
• We look back in the already encoded portion
of the window to find a match for d.
• As we can see, there is no match, so we
transmit the triple <0 0 C(d)>.
• With abrarr in the look-ahead buffer.
• Looking back from the current location, we find a
• match to a at an offset of two. The length of this
match is one.
• Looking further back, we have another match for a at
an offset of four; again the length of the match is one.
• Looking back even further in the window, we have a
third match for a at an offset of seven.
• However, this time the length of the match is four
• So we encode the string abra with the triple <7 4 C(r)>,
and move the window forward by five characters.
• The window now contains the following
characters:
• Now the look-ahead buffer contains the string
rarrad.
• Looking back in the window, we find a match for r
at an offset of one and a match length of one, and
a second match at an offset of three with a match
length of what at first appears to be three.
• It turns out we can use a match length of five
instead of three.
Decoding
 The LZ78 Approach
• The LZ77 approach implicitly assumes that like
patterns will occur close together.
• It makes use of this structure by using the recent
past of the sequence as the dictionary for
encoding.
• This means that any pattern that recurs over a
period longer than that covered by the coder
window will not be captured.
• The worst-case situation would be where the
sequence to be encoded was periodic with a
period longer than the search buffer.
• Consider the sequence below:
• This is a periodic sequence with a period of nine.
• If the search buffer had been just one symbol
longer, this sequence could have been significantly
compressed.
• As it stands, none of the new symbols will have a
match in the search buffer and will have to be
represented by separate codewords.
• As this involves sending along overhead (a 1-bit
flag for LZSS and a triple for the original LZ77
algorithm), the net result will be an expansion
rather than a compression.
• The LZ78 algorithm solves this problem by dropping
the reliance on the search buffer and keeping an
explicit dictionary.
• This dictionary has to be built at both the encoder and
decoder, and care must be taken that the dictionaries
are built in an identical manner.
• The inputs are coded as a double <i c>, with i being an
index corresponding to the dictionary entry that was
the longest match to the input, and c being the code
for the character in the input following the matched
portion of the input.
• As in the case of LZ77, the index value of 0 is used in
the case of no match.
• This double then becomes the newest entry in the
dictionary.
• Thus, each new entry into the dictionary is one new
symbol concatenated with an existing dictionary entry.
 LZW Encoding
• In LZ77, we send a triple <0, l, C>
• In LZ78, only <i, c> are sent
• In LZW only i i.e. index of the dictionary entry is sent
• For this, the dictionary has to be primed with all the
letters of the source alphabet.
• The input to the encoder is accumulated in a pattern p
as long as p is contained in the dictionary.
• If the addition of another letter a results in a pattern p
∗ a (∗ denotes concatenation) that is not in the
dictionary, then the index of p is transmitted to the
receiver, the pattern p ∗ a is added to the dictionary,
and we start another pattern with the letter a.
The LZW algorithm-—encoding
• The encoder first encounters
the letter w.
• This “pattern” is in the
dictionary so we concatenate
the next letter to it, forming the
pattern wa.
• This pattern is not in the
dictionary, so we encode w with • This pattern is not in the
its dictionary index 5, add the dictionary, so we encode a with
its dictionary index value 2, add
pattern wa to the dictionary as the pattern ab to the dictionary
the sixth element of the as the seventh element of the
dictionary, and begin a new dictionary, and start
pattern starting with the letter constructing a new pattern with
a. the letter b.
• We continue in this manner,
• As a is in the dictionary, we constructing two-letter
concatenate the next element b patterns, until we reach the
to form the pattern ab. letter w in the second wabba.
• At this point the output of the
encoder consists entirely of indices
from the initial dictionary: 5 2 3 3 2 1.
The dictionary at this point looks like
Table 5.7.
• This pattern does not exist in the
dictionary, so we include it as the
12th entry in the dictionary and start
a new pattern with the symbol b.
• We also encode wa with its index
value of 6.
• Notice that after a series of two-
letter entries, we now have a three-
letter entry. As the encoding
progresses, the length of the entries
keeps increasing.
 LZW Decoding
• Start with dictionary containing index of single
Letter
• Continue Decoding in a way exactly identical
to encoding
 The Burrows-Wheeler Transform
• The Burrows-Wheeler Transform (BWT)
algorithm also uses the context of the symbol
being encoded, but in a very different way, for
lossless compression.
• the BWT algorithm requires that the entire
sequence to be coded be available to the
encoder before the coding takes place.
• The decoding procedure is not immediately
evident once we know the encoding
procedure.
 BWT Encoding
• Given a sequence of length N, we create N −1 other
sequences where each of these N −1 sequences is a
cyclic shift of the original sequence.
• These N sequences are arranged in lexicographic order.
• The encoder then transmits the sequence of length N
created by taking the last letter of each sorted,
cyclically shifted, sequence.
• This sequence of last letters L, and the position of the
original sequence in the sorted list, are coded and sent
to the decoder. As we shall see, this information is
sufficient to recover the original sequence.
• We start with a sequence of length N and end with a
representation that contains N +1 elements. However,
this sequence has a structure that makes it highly
amenable to compression.
 BWT Decoding
• The first step in obtaining the permutation is
to generate the sequence F consisting of the
first element of each row.
• Therefore, the sequence F is simply the
sequence L in lexicographic order. In our
example this means that F is given as

• We can use L and F to generate the original

sequence.
• Because each row is a cyclical shift, the letter in
the first column of any row is the letter appearing
after the last column in the row in the original
sequence.
• If we know that the original sequence is in the kth
row, then we can begin unraveling the original
sequence starting with the kth element of F.
• The original sequence is sequence number 10, so the first
letter in of the original sequence is F(10) = t.
• To find the letter following t we look for t in the array L.
• There are two t’s in L. Which should we use? The t in F
that we are working with is the lower of two t’s, so we
pick the lower of two t’s in L. This is L(4).
• Therefore, the next letter in our reconstructed sequence
is F(4) = h. The reconstructed sequence to this point is th.
• To find the next letter, we look for h in the L array. Again
there are two h’s. The h at F(4) is the lower of two h’s in
F, so we pick the lower of the two h’s in L. This is the fifth
element of L, so the next element in our decoded
sequence is F(5) = i.
• The decoded sequence to this point is thi.
• The process continues to generate the original sequence.
 Move-to-Front Coding
• Takes advantage of long runs of identical symbols
• In this coding scheme, start with some initial listing of
the source alphabet.
• The symbol at the top of the list is assigned the
number 0, the next one is assigned the number 1, and
so on.
• The first time a particular symbol occurs, the number
corresponding to its place in the list is transmitted.
• Then it is moved to the top of the list.
• If a run of this symbol occurs, transmit a sequence of
0s.
• This way, long runs of different symbols get
transformed to a large number of 0s.
Lossless Image Compression
 Old JPEG-LS
• A predictor combines the values of up to three neighboring
samples (A, B, and C) to form a prediction of the sample
indicated by X
• This prediction is then subtracted from the actual value of
sample X, and the difference is encoded losslessly by either
of entropy coding methods -Huffman or arithmetic.
• Any one of the eight predictors listed in Table 1 can be
used.

38
 Old JPEG-LS

• If compression is performed in a non-real time

environment all 8 modes of prediction can be tried
and the one giving the most compression used.

39
 CALIC
• CALIC: Context Adaptive Lossless Image Compression
• Uses both context and prediction of the pixel values
• Context: to obtain the distribution of the symbol being
encoded
• Prediction: use previous values of the sequence to obtain a
prediction of the value of the symbol being encoded
• In an image, a given pixel generally has a value close to
one of its neighbors
• Which neighbor has the closest value depends on the local
structure of the image

40
 CALIC
• We can get an idea of what kinds of structure may
or may not be in the neighborhood of X by NN N NE

computing NW
N
NE

W X
WW

• If dh>>dv, a large amount of horizontal variations, better to

pick N as the initial prediction of X
• If dv>>dh, a large amount of vertical variations, better to
pick W as the initial prediction of X
• If the differences are moderate or small, the initial prediction
value is a weighted average of neighboring pixels

41
CALIC: Initial Prediction
 CALIC
• We refine this initial prediction using information about the
inter-relationship of the pixels in the neighborhood
• We quantify the information about the neighborhood by
first forming the vector [N, W, NW, NE, NN, WW, 2N-NN,
2W -WW]
• We then compare each component of this vector with our
initial prediction
• If the value of the component is less than the prediction, we
replace the value with a one, otherwise we replace it with a
zero.

43
 CALIC
• We also compute where is
the predicted value of N.
• The range of values for  is divided into 4 intervals (the
combination of two consecutive interval in 8 context intervals
explained in the next slide)
• These 4 intervals along with 144 possibilities for the vector
described above, 144x4=576 contexts for X.
• Based on the context for the pixel, we find a value named
offset and add it to the initial prediction of X.
• The residual (difference between pixel value and the
prediction) has to be encoded using its context.

44
 CALIC
• The residual is mapped to [0,M-1] interval (original pixel
values are assumed to be between 0 and M-1.
• Context for encoding of residual is based on the range that
 fall in:
context 1
context 2

context 8

• The residual is arithmetic coded using the context

45
 CALIC: Encoding
• If a pixel takes on
values between 0
and M −1, then given
a predicted value ˆX ,
the difference X− ˆX
will take on values in
the range −ˆX to M
−1− ˆX .
• We can use this fact
to map the difference
values into the range
0-M −1, using the
following mapping:
 JPEG-LS
• The new JPEG-LS is based on an algorithm named LOCO-
1 developed by HP (similar to CALIC)
• It has both a lossless and lossy (called near lossless) modes
• Initial prediction:

47
 JPEG-LS
• Context:
D1=NE-N D2=N-NW D3=NW-W
• D1, D2 and D3 are mapped to Q1, Q2 and
Q3:

T1, T2 and T3 are

positive
coefficients
defined by user

48
 JPEG-LS
• Q1 and Q2 and Q3 define a context vector Q=(Q1, Q2 ,
Q3)
• Given 9 different values for each component, the context
vector can have 9x9x9=729 possible values
• The number of contexts is reduced by replacing any
context vector Q whose fist nonzero element is negative
with –Q
• Whenever this happens a variable SIGN is set to -1,
otherwise it is set to 1
• This reduces the number of context to 365
• Q is then mapped to a number between 0 and 364.
• This number is used to find a correction value c[Q]
• c[Q] is multiplied by SIGN and added to initial prediction
error
49
 JPEG-LS
• The prediction error rn is mapped into an interval that is the
same size as the range occupied by the original pixel
values

(pixel values are between 0 and M-1)

• Prediction errors are encoded based on Golomb

codes.
50