See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/340458036

Effects of Rounding and Truncating Methods of Quantization Error and SQNR for
Sine Signal

Article in Journal of Applied Science and Technology Trends · March 2020

DOI: 10.38094/jastt113

CITATIONS READS
5 156

2 authors, including:

Karwan Jacksi
University of Zakho
51 PUBLICATIONS 709 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

ICOASE2018 View project

Semantic Similarity based on Document Clustering View project

All content following this page was uploaded by Karwan Jacksi on 06 April 2020.

The user has requested enhancement of the downloaded file.

 Vol. 01, No. 01, pp. 08 –12 (2020)
ISSN: 2708-0757

JOURNAL OF APPLIED SCIENCE AND TECHNOLOGY TRENDS

www.jastt.org

Effects of Rounding and Truncating Methods of Quantization

Error and SQNR for Sine Signal

Alaaldin Hasso1, Karwan Jacksi1,*

1
Department of Computer Science, University of Zakho, Kurdistan Region, Iraq, (alaaldin.hasso, karwan.jacksi)@uoz.edu.krd)

Abstract
Within the Analog to Digital Conversion (ADC), quantization noise is a duplicate of a Quantization Error (QE) which is introduced by
quantization. In signal processing and telecommunication systems, the noise is non-linear and depends on the signal type. During the
analog, Sine signal converts to the digital (ADC) process, the two methods are used Rounding and Truncating in-order to eliminate the
error produced in the digitization process. The rounding method quantize assigns each sample of sine signal to the nearest quantization
level. However, making the Truncating would have assigned each sample of sine signal to the quantization level below it.
This paper compares the rounding and truncating methods of QE for sine signal, signal to quantization noise ratio, correlation coefficient,
and regression equation of a line for both methods. Then, it calculates the residual sum of squares and compares it to the regression
equations of the lines.

Keywords: ADC, SQNR, RSS, Quantization Error, Correlation Coefficient, Regression Equation

Received: January 10 / Accepted: March 01 / Online: March 12

I. INTRODUCTION
This paper reviews the QE, SQNR, correlation coefficient
The wide range of ceaseless sine signal has to be transformed (r), regression equation of a line in section II. Calculating the
to digital data with limited accuracy. This is known as the Residual Sum of Square (RSS) for the rounding and truncating
quantization [1]. The Quantization Error (QE) is the error methods is also detailed in section II. The comparison between
generated from applying the incessant signal value by a limited the regression equations of a line is presented in section III.
set of discrete value levels [2]. Then, a comparison between the two methods is exposed in
During the Analog sinusoidal signal to Digital signal section IV. Finally, the discussions and conclusions are
Converter (ADC), the adequacy of the analog sine signal to be presented in section V and VI.
transformed has endless accuracy [3]. Implementing a
measurable methodology that adopting the QE of the sine signal II. BACKGROUND
is irregular sequence and augments to the original signal [4]. To
decrease the additional digits, the truncating method can be used A. Rounding Method
that allocates each sine signal models to a lower level of QE [5]. This section reviews the QE, SQNR, correlation coefficient,
During the ADC process, as the number of bits rises in the regression equation of a line, and calculates RSS for the
digital world, the experimental and theoretical Signal to rounding method
Quantization Noise Ratio (SQNR) of the sine signal using the From Fig. 1, it can be seen that the rang of QE of the sine
rounding method are similar to theoretical values [6]. However, analog signal is symmetric about zero. Furthermore, it is not
by means of the truncating method, the experimental SQNR resulting from a reduction of several significant bits as well as
value is permanently underneath the theoretical value, which falls in the rang of all positive and negative numbers. The same
means not as much of exact quantization of the sine signal that experiment is achieved with the same values when the number
consequences in a lower SQNR. This was expected because of bits is 4, 8, and 16 [6]. Fig. 1. shows the original analog sine
instead of rounding the data, the truncating method occurred [3]. signal (X2), Quantization (Xq) and QE (Xe) using the rounding

doi: 10.38094/jastt113 8
 Hasso et al. / Journal of Applied Science and Technology Trends Vol. 01, No. 01, pp. 08 –12, (2020)

method for some bits equal to 2 (n=2).

Experimental rounding
𝒐.𝟓 ∆ ∆ 𝒐.𝟓
(−𝟎. 𝟐𝟓 = − =− ≤ QE Rounding ≤ = = 𝟎. 𝟐𝟓) SQNRR = aR + bR *n (4)
𝟐 𝟐 𝟐 𝟐
(1) SQNR R = 2.270698 + 5.9820738 ∗ n
The investigation of the regression equation and the
correlation coefficient (r) of the SQNR on the quantity of bits by Theoretical for sinusoidal signal
means of rounding methods.
SQNR Theo = 1.76 + 6.02 ∗ n (5)

Calculating RSS for the rounding method

To calculate RSS for the rounding method, as shown in
Table I,Error! Reference source not found. according to the
following formula (formula 6) which is taken from [7].

𝐑𝐒𝐒 = [𝐒𝐐𝐍𝐑 𝐑 − 𝐒𝐐𝐍𝐑 𝐓𝐇𝐞𝐨 ]^𝟐 (6)

RSS = 1.64713551
Where
SQNR_R is the experimental rounding observed value
Fig. 1. X2, Xq and Xe using rounding method for (n=2)
SQNR_THeo. is the corresponding theoretical value
The (r) states that how the data is close in a scatter-plot
reduction sideways a straight line. The value of (r) which is
TABLE I. CALCULATING RSS FOR ROUNDING METHOD
nearer to 1 means the nearer the data is described by a linear
equation. When the (r) is equal to 1, this means that the data set
is faultlessly related, however once the (r) is close to zero, this RSS = [SQNR Exp.(R) SQNR Exp. SQNR
No. of bits
means that the data set shows no straight-line relationship [6]. - SQNR (Theo)]2 Round Theo.

0.628849 14.593 13.8 2

𝐒𝐧𝐞
𝐫𝐑 = (2)
√𝐒𝐧𝐧 𝐒𝐞𝐞
0.130321 26.201 25.84 4

3.61E-06 37.8781 37.88 6

1004.9884 1004.9884
rR = = = 0.99999022
√168x6013.091 1005.0866 0.02053489 49.7767 49.92 8

0.04687225 61.7435 61.96 10

The value of the r is (0.99999022) which means that it has a
very strong relationship. 0.74200996 74.8614 74 12

The equation of the regression line in general is: 0.07817616 85.7604 86.04 14

0.00036864 98.0608 98.08 16

𝐞 = 𝐚 + 𝐛𝐧 (3)
RSS = 1.64713551

Where (bR) is the slope of the line, and (aR) is the intercept
B. Truncating method
This section reviews the QE, SQNR, correlation coefficient,
𝑆𝑛𝑒 1004.9884 regression equation of a line, and calculates the RSS for
𝑏𝑅 = = = 5.9820738
𝑆𝑛𝑛 168 truncating method. Fig. 2. illustrates the original sine analog
signal (X2), Quantization (Xq) and QE (Xe) using Truncating
for a number of bits (n=2) which is taken from [3].
𝑎𝑅 = 𝑒̅ − 𝑏𝑛̅ = 56.109362 − 5.9820738 𝑥 9
= 56.109362 − 53.838664 = 2.270698

9
 Hasso et al. / Journal of Applied Science and Technology Trends Vol. 01, No. 01, pp. 08 –12, (2020)

TABLE II. CALCULATING RSS FOR THE TRUNCATING METHOD

RSS = [SQNR Ex.

SQNR Ex. Tr SQNR Theo. No. of Bits
Tr- SQNR Theo]2

38.49582025 7.5955 13.8 2

41.78458881 19.3759 25.84 4

39.08625361 31.6281 37.88 6

34.37594161 44.0569 49.92 8

38.85776896 55.7264 61.96 10

34.07173641 68.1629 74 12
Fig. 2. X2, Xq, and Xe using Truncating for (n=2)
34.56616849 80.1607 86.04 14
From Fig. 2., the rang of QE using the truncating method of 38.05039225 91.9115 98.08 16
the sine analog signal is not symmetric about zero, and it is a
RSS = 299.2886704
result of a reduction of a number of important bits as well as falls
in the rang of all negative numbers. The same experiment for the
truncating method is accomplished in [3] with the same values From Table II RSS for the truncating (RSS)T is
when the number of bits is 4, 8, and 16.
RSST=299.2886704

(− Δ = − 0.5 ≤ QE (Truncating) ≤ 0) (7) III. REGRESSION EQUATION COMPARISONS

A comparison between regression equations of lies for
equations (4, 5, and 9) which are mentioned and detailed in the
The correlation coefficient for truncating (rT) above sections are presented. The comparison is accomplished
using a multi-paradigm numerical computing environment
MATLAB.
𝐒𝐧𝐞
𝒓𝑻 = (8)
√𝐒𝐧𝐧 𝐒𝐞𝐞 %n=Number of Bits
>>n=2:2:16 ;
%S_th=Signal Quantization to Noise Ratio Theory
1015.41 1015.41
𝑟𝑇 = = = 0.9999789 >>S_th=1.76 + 6.02*n ;
√168x6137.507 1015.4314
>>plot(n,S_th,'r');
>>grid on; >> hold on;
%S_R=Signal Quantization to Noise Ratio Experimental using
The value of the correlation coefficient is (0.9999789) which Rounding Method
means it has a strong relationship. >>S_R=2.2702 + 5.98207*n ;
The regression equation of a line for truncating >>plot(n,S_R,'b'); >> hold on;
%S_T=Signal Quantization to Noise Ratio Experimental using
Truncating Method
𝑆𝑛𝑒 1015.41 >>S_T=-4.56972 + 6.04411*n;
𝑏𝑇 = = = 6.0441071 >>plot(n,S_T,'g');
𝑆𝑛𝑛 168
>>xlabel('n'); >> ylabel('SQNR');
>>legend('S_th,r','S_R,b','S_T,g');
>>title('SQNR for Theory, Experimental Rounding, Experimental
𝑎 𝑇 = 𝑒̅ − 𝑏𝑛̅ = 49.827237 − 6,0441071 𝑥 9
Truncating');
= 49.827237 − 54.396963 = −4.569726

Experimental Truncating The lines in Fig. 3 are drawn from the above code which
presents a comparison among the three lines from equations 4, 5
SQNRT = aT + bT * n (9)
and 9.
SQNRT=-4.569726+6.0441071* n

10
 Hasso et al. / Journal of Applied Science and Technology Trends Vol. 01, No. 01, pp. 08 –12, (2020)

SQNR = SQNR =
The Equation of
8 2.270698 + −4.569726 +
Regression a ine
5.9820738 ∗ n 6.0441071 ∗ n

9 RSS 1.64713551 299.2886704

The Standard Error of Estimate (SEE) using the truncating

method is (0.005014927716). This can be confirmed, as the
inverse relationships between R-squared and SEE. In other
words, the precision of the model increases as the SEE
decreases.
Fig. 3. Lines drawn from equations (4, 5, and 9) The SEE and coefficient of variation of the mean deviation
of the quantization error using the rounding method are lower
Similarly, by using MATLAB program as shown in Fig. 3., than using the truncating method for the same signal. This means
as the quantity of bits amplified in the digital world, the that the rounding method gives more accurate results.
experimental and theoretical SQNR of the sine signal using
rounding method is very similar, but as using the truncating The most appropriate statistics for making a comparison
method, the experimental SQNR value was always lower than between these two methods is to calculate the RSS for each
the theoretical value, which leads to less accurate quantization method separately. From the calculations, the RSS using the
of the signal and a lower SQNR. rounding method is much less than the RSS in Table II, using
the truncating method. This means that as the value of RSS
IV. COMPARISON OF ROUNDING AND TRUNCATING METHODS decreases the accuracy of the method increases, and vice versa.
In this section, a detailed comparison for rounding and
V. DISCUSSION
truncating methods for sine signal is presented. The precision of
the model decreases when the standard deviation and mean In an ideal ADC, where the quantization error is uniformly
absolute deviation increase, it means using rounding will be distributed and the signal has a uniform distribution covering all
more accurate as shown in Table III. The R-Squared statistic quantization levels, the SQNR can be calculated as follow:
indicates that the model as fitted explains that (99.71793519%) SQNR=6.02*n (10)
of the variability in SQNR using a rounding method while
(99.70530110%) using the truncating method. The value of the The most common test signals that complete this is the full
correlation coefficient for rounding is (0.99999022) this means amplitude triangle signals [8]. for example, a 16-bit ADC has a
that it has a very strong relationship, while for the truncating maximum SQNR of 6.02 × 16 = 96.3 dB.
method (0.99997890) has only a strong relationship.
But when the input signal is a full-amplitude, the sine wave
distribution of the signal is no longer uniform distribution and
TABLE III. COMPARISON OF ROUNDING AND TRUNCATING METHODS does not cover all quantization level, and the corresponding
equation (5) is given
Rounding Truncating
∆ ∆
SQNR=1.76+6.02*n
1 Rang of QE (- ≤ QE ≤ ) (- Δ = - 0.5 ≤ QE ≤ 0)
2 2
The (1.76) difference in SQNR only occurs due to the sine
Standard wave signal being a full-scale, and again when the input signal
2 Diversion Error 0.03720387388 0.08552677275 has high amplitude and a wide frequency spectrum, the SQNR
(SD) is assumed to be uniformly distributed. In this case, a 16-bit
Mean Absolute ADC has a maximum SQNR of 98.09 dB.
3 0.02485336575 0.057049666
Diversion Error
For the rounding method, the corresponding equation is (4)
R-square (R2)
4 99.71793519 % 99.7053011 % SQNR R = 2.270698 + 5.9820738 ∗ n
Error

5 SEE 0.00213459133 0.005014927716 The 2.270698 difference in SQNR occurs due to a full-scale
sine wave signal. A 16-bit ADC has a maximum SQNR of
Coefficient of 97.9838788 dB
Variation
6
(CV) %
11.66849539 % 12.06942713 % For the truncating method, the corresponding equation is (9)
Error SQNR T = −4.569726 + 6.0441071 ∗ n
Correlation
7 0.99999022 0.9999789 The (-4.569726) difference in SQNR occurs also due to a
Coefficient (r)
full-scale sine signal, and a 16-bit ADC has a maximum SQNR
of 92.1359876 dB

11
 Hasso et al. / Journal of Applied Science and Technology Trends Vol. 01, No. 01, pp. 08 –12, (2020)

The differences in these equations have not occurred, if the truncating will be smaller in size, speedily and less latency,
input sine wave signal has a uniform distribution and covering because fewer electric circuits needed to build.
all quantization levels, as in equation (10)
REFERENCES
VI. CONCLUSION [1] A. V. Oppenheim, “Applications of digital signal processing,”
By using the STATGRAPHICS PROGRAM, and Englewood Cliffs NJ Prentice-Hall Inc 1978 510 P, 1978.
[2] P. Zador, “Asymptotic quantization error of continuous signals and the
MATLAB programs, the value of SQNR using the truncating quantization dimension,” IEEE Trans. Inf. Theory, vol. 28, no. 2, pp.
method is always lower than the theoretical and rounding values, 139–149, 1982.
which leads to less accurate quantization of the sine signal. [3] A. Hasso, K. Jacksi, and K. Smith, “Effect of Quantization Error and
SQNR on the ADC Using Truncating Method to the Nearest Integer Bit,”
The designed filter for truncating method is running to presented at the 2019 International Conference on Advanced Science and
remove the 16-bits least significant bit (LSB) and to keep the Engineering (ICOASE), 2019, pp. 112–117.
most significant 16-bits (MSB) of a 32- bit in the digital world, [4] S. Pavan, R. Schreier, and G. C. Temes, Understanding delta-sigma data
this means that ignoring the lower 16-bits of the digital word of converters. John Wiley & Sons, 2017.
[5] B. Widrow and I. Kollár, “Quantization noise,” Camb. Univ. Press, vol.
the output signals and take only the upper 16 bits. 2, p. 5, 2008.
The problem with the truncating method is that it makes no [6] A. Hasso and S. Mohammed, “The Effect of ADC Quantization Error by
rounding to nearest integer bit,” in 1st international conference on
use of the information contained in the LSB, which gives us a Engineering and innovative Technology, SU-ICETT, Salahaddin
smarter approach to round the upper 16-bits either up or down University, 2016.
1-bit based on the lower 16-bits. Selecting of the 16-bits gives a [7] W. M. Mendenhall and T. L. Sincich, Statistics for Engineering and the
more accurate sign of the original word and also eliminates bias Sciences. CRC Press, 2016.
that is a DC component in our signal spectrum. [8] K. C. Pohlmann, Principles of digital audio. McGraw-Hill, Inc., 1995.

The advantage of the truncating method over the rounding

method is that designing a microprocessor's filter for the

12

View publication stats