Design of an Optimal Quantizer

We introduced you to the two basic operations of sampling and quantization to obtain a digital
image. Sampling process does not imply any limit on the values of the samples, as long as the
Nyquist rate is followed, we can always recover back the analog one. But quantization does
introduces losses. This operation converts the continuously valued irradiance of each sample
at the detector (i.e., the brightness) to an integer, i.e., the sampled image is quantized. The entire
process of measuring and quantizing the brightnesses is significantly affected by detector
characteristics such as dynamic range and linearity.
Quantization is a process of representing a large possible set of values with a much smaller set.
The number of elements in the original set in many practical situations is infinite (like the set
of real numbers.)

Prior to storage or transmission of a given parameter, (could be in speech or image or any other
signal processing applications), it must be quantized in order to reduce storage space or
transmission bandwidth. The quantization operation is performed by digital comparators or
sample-and hold circuits. In the process, some quality loss is introduced, which is undesirable.
How to minimize loss (quantization noise) for a given amount of available resources is the
central problem of quantization that we will discuss here.
The resolution, or step size of the quantizer is one of the most important decisions to make:
the difference in brightness between adjacent gray levels. This decides how many bits should
we use? While choosing Step Size, it is important how should we spread the resulting
quantization levels? And how efficient can this entire process be –How much noise we insert
to the quantized signal ?– This is measured through common metrics such as SNR, MSE –
mean square error between original and quantized samples.
What we are studying here is known as scaler quantization: (we quantize each sample
separately), it is different from vector quantization in which a group of samples are quantized
jointly.
The quantizer mapping function is a Non-linear operation? Can you justify why? Also note that
in the above mapping, the variable X can be both a continuous/discrete, however, the output C
will always be discrete . Also, the cardinality of the set C will be always less than that of X

In most systems, for an 𝑚 –bit quantizer, and the dynamic range (fmin to fmax ) of the
measured image sample irradiances the step size 𝑏 between adjacent quantized levels is fixed
𝑓𝑚𝑎𝑥 −𝑓𝑚𝑖𝑛
(“uniform quantization”) and 𝑏 = . We may now define a set of uniformly
2𝑚 −1
spaced levels 𝑓𝑞 that span the dynamic range as :

Here Q represents the non-linear truncation or rounding operation. The form of Q determines
the location of the decision levels where the quantizer jumps from one level to the next. The
image irradiances are reconstructed by assigning all pixels with a particular gray level fq to the
same irradiance value E [x,y],which might be defined by “inverting” the quantization relation.
The reconstruction level is often placed between the decision levels by adding a factor b /2 :

Usually (of course), 𝐸̂ [𝑥, 𝑦] ≠ 𝐸[𝑥, 𝑦], due to the quantization, i.e., there will be quantization
error. The goal of optimum quantization is to adjust the quantization scheme to reconstruct
the set of image irradiances which most closely approximates the ensemble of original values.
The criterion which defines the goodness of fit and the statistics of the original irradiances will
determine the parameters of the quantizer, e.g., the set of thresholds between the levels.
The grey value of the quantized image is an integer value which is related to the input irradiance
at that sample. For uniform quantization, where the steps between adjacent levels are the same
size, the constant of proportionality is the difference in irradiance between adjacent quantized
levels. The difference between the true input irradiance (or brightness) and the corresponding
irradiance of the digital level is the quantization error at that pixel:

Note that the quantization error is bipolar in general, i.e., it may take on positive or negative
values. It often is useful to describe the statistical properties of the quantization error, which
will be a function of both the type of quantizer and the input image. However, if the difference
between quantization steps (i.e., the width of a quantization level) is b, is constant, the
quantization error for most images may be approximated as a uniform distribution with mean
zero and variance 𝑏 2 /12. [Derive it yourself as you learnt for PCM in your Digital
communication classes)

where N is the number pixels in the image. If the irradiance is measured in Watts/ mm2 , 𝜖 2 will have
units of (Watts/ 𝑚𝑚2 )2

The RMS error for one image is a function of the quantizer used, and that the RMS error from
one quantizer will differ for different images. It should also be obvious that it is desirable to
minimize the RMS error in an image. The brute-force method for minimizing quantization error
is to add more bits to the ADC, which increases the cost of the quantizer and the memory
required to store the image.Also recall the concept of the SNR : Here most commonly the
signal and noise of images are measured by photoelectric detectors as differences in electrical
potential in volts; the signal dynamic range is Vf = Vmax−Vmin
What will be the SNR of quantization?
Note that the input signal and the type of quantizer determine the probability density function
of the quantization error in a strict sense, but it has been verified that mostly it can be assumed
to be uniform pdf. Thus in the case of an m-bit uniform quantizer (2^m gray levels) where the
levels are spaced by intervals of width b over the full analog dynamic range of the signal, the
error due to quantization will be (approximately) uniformly distributed over this interval b. If
the nonlinearity of the quantizer is rounding, the mean value of the error is 0; if truncation to
the next lower integer, the mean value is –b/2
Recall that the variance of a signal is a measure of the spread of its amplitude about the mean
value. The variances obviously depend on the statistics (the histograms) of the signal and noise.
The variances depend only on the range of grey values and not on their “arrangement” (i.e.,
numerical“order”or“pictorial”appearance in the image. Since the noise often is determined by
the measurement equipment, a single measurement of the noise variance often is used for many
signal amplitudes. However, the signal variance must be measured each time.
Since the variance depends on the statistics of the signal, it is common (though less rigorous)
to approximate the variance by the square of the dynamic range, which is the “peak-to-peak
signal amplitude” fmax−fmin ≡ ∆f. In most cases, (∆f)^2 is larger (and often much larger) than
the variance
There are two mappings in a quantizer: An Encoder mapping and a Decoder mapping
associated with which there will be two levels: Make a note of two IMPORTANT DESIGN
PAIR parameters we will be talking about: The Decision (or threshold) Levels and the
Reconstruction levels.The concept is known to you for a uniform quantizer.
The plot of the quantization error for a uniform quantizer (UQ) will be something like this:

Note also the difference between two types of quantizer: a mid-rise and a mid-tread Quantizer.
See their characteristics in the plot below: Midrise quantizer: zero is not an output level, while
Midtread quantizer: zero is an output level. Usually, a mid-riser quantizer is used if the number
of representative levels is even and a mid-step quantizer if the number of level is odd.
For a scaler quantizer the construction of intervals can be viewed as a part of the encoder, while
selection of reconstruction values is a part of the Decoder. The quality of reconstruction
depends on proper choice of both. Thus the design of a quantizer consists of three steps: (i)
Divide the range into intervals, assign binary codes to these intervals, and select the
reconstruction levels. For a UQ, all intervals are of the same size except the two outer intervals.
so that the decision boundaries are spaced evenly. Reconstruction values are also spaced evenly
with the same spacing as the decision boundaries. In the inner intervals the reconstruction
values are the mid-points of the intervals. For example:

The values in the region R1 are quantized to t1; The values in the region R2 are quantized to
t2, .. And so on…The decision boundary between R1 and R2 ? ( dotted line (t1+t2)/2
If t1 , t2, t3, …. tn are known and fixed discrete levels. (decision boundaries? Mid-point
But How can we find the levels t1 t2 ….?
What is the best way to quantize into L discrete levels: This actually depends on the PDFs of
source (which is always not uniform). Note the source can be either discrete or continuous.

The generalized mathematical model of a M-level Quantizer Q(x) can be described as below:
Input X – A random variable with a pdf -probability density function 𝑓𝑋 (𝑥) : Typically three
types of distributions are common (Uniform, Laplacian and Gaussian) ; We will define the
problem with a general pdf
Output: has decision boundaries, and reconstruction levels.

What is an optimal quantizer?

An optimal quantization of a random variable X having a probability distribution p(x) is
obtained by a quantizer that minimises a given metric which is usually the Mean Square Error
The mean square error of a Quantizer for a continuous valued signal

The MSE of a quantizer for a discrete valued signal will be

𝑛 2
𝑀𝑆𝐸 = 𝜎𝑞2 =∑ (𝑥𝑘 − 𝑄 (𝑥𝑘 )) 𝑓𝑋 (𝑥𝑘 )
𝑘=1

The MSE for a specific image (MXN ) will be

𝑁 𝑀
1
𝑀𝑆𝐸 = 𝜎𝑞2 = ∑ ∑(𝑥 (𝑖, 𝑗) − 𝑄(𝑥 (𝑖, 𝑗)))2
𝑀𝑁
𝑖=1 𝑗=1
Uniform quantization with a quantization step ; p(x) is a uniform probability distribution of a random
variable X

The general problem

Quite often the sources we deal with do not have a uniform distribution, however we still
want the simplicity of a uniform quantizer. When the distribution is not uniform, it is not a
good idea to find the step size by dividing the range of inputs by the number of levels. If the
pdf is not uniform, an optimal quantization step size can be found that minimizes the
distortion (MSE

Uniform Quantization of A Non-uniformly Distributed Source

Derivation will not be asked in Exam, but this is how it is done:

Can you interpret the result of this type of optimization?

UQ is a special case of Non-uniform quantizer which we will discuss now.

Non uniform quantizers attempt to decrease the average distortion or MSE by assigning more
levels (decrease the step size) to more probable regions. For given M and input pdf, we need
to choose {bi} and {yi} to minimize the distortion. We will see from the derivation shown next
that the optimum value of reconstruction level depends on the boundary and the boundary
depends on the reconstruction levels. ) It is difficult to solve both sets of equations
analytically. Instead, it is easier to find these optimality conditions: – For a given partition
(encoder), what is the optimum codebook (decoder)? – For a given codebook (decoder), what
is the optimum partition (encoder).
An iterative algorithm known as the Lloyd algorithm solves the problem by iteratively
optimizing the encoder and decoder until both conditions are met with sufficient accuracy.
We analyse the optimal NU Quantizer for a Non-uniform source as follows:
The Lloyd-Max algorithm is an algorithm for finding the representative levels and the decision
levels to meet the (1)[ MIDPOINT of RL and (2) CENTROID of PDF conditions;