Turbo Codes: Some Simple Ideas for Efficient Communications

Claude Berrou

ENST Bretagne, BP 832, 29285 Brest Cedex, France

e-mail : [email protected]

Abstract - Turbo codes are error-correcting codes which enable communication systems to work in conditions
close to the theoretical limits. These codes call on different concepts that were not so familiar a few years ago.
This paper is an attempt to classify (and clarify) the fundamental physical concepts on which turbo coding and
decoding rely.

1. INTRODUCTION

The invention of turbo codes [1] did not result from a linear limpid mathematical demonstration. It was the outcome of
an empirical construction of a global coding/decoding scheme, using existing bricks that had never been put together in
this way before.

Because the foundation of digital communications relies on potent mathematical considerations [2], error coding codes
were believed to belong solely to the world of mathematics. It is true that good codes (BCH, Reed-Solomon, ...) were
defined thanks to algebraic tools, but generally without thinking of the decoding procedure, which had to be invented
afterwards. In the case of turbo codes, the decoder was designed first and the encoder naturally followed. In fact,
considering the feedback structure of the decoder, many codes could have been devised to fit the "turbo" or "message
passing" principle. The parallel concatenation of two Recursive Systematic Convolutional was adopted for its good
behaviour near the theoretical limits. But other multidimensional codes, like the serially concatenated convolutional
code [3] or the rediscovered Low Density Parity Check (LDPC) code [4] can benefit from the "message passing"
decoding method.

The paper is divided into three main parts. In Section 2, we set the problem of coding from a very pragmatic point of
view. Section 3 details the three keys which open the door to turbo codes and finally Section 4 gives some recent results
about a new family of turbo codes: duo-binary turbo codes.

2. WHAT IS A GOOD CODE ?

Fig. 1 represents three possible behaviours for an error-correcting code on an Additive White Gaussian Noise (AWGN)
channel with QPSK modulation and 1/2 coding rate. The information block is assumed to be around 188 bytes (MPEG
application).
BER
5

10-1
5
10-2
5 3
10-3
poor
5
convergence,
10-4 high no coding
5 2 asymptotic
10-5 gain
5
good
10-6 convergence,
5 low
10-7 asymptotic
5 1 gain
10-8

theoretical limit
Eb/N0 (dB)
(R = 1/2, 188 bytes)

Figure 1. Possible behaviours for a coding/decoding scheme.

Behaviour 1 corresponds to the ideal Shannon system. It assumes random coding with minimum distance dmin given by
the Gilbert-Varshamov bound, that is around 380. The theoretical asymptotic gain, approximated by

Ga≈ 10 log(Rd min ) , (1)

would be higher than 22 dB!

Behaviour 2 has good convergence and low dmin. This is for instance what is obtained with turbo coding when the
permutation function is not properly designed. The asymptotic gain, around 7.5 dB for a Bit Error Rate (BER) of 10 -8 in
the example given, is reached at medium BER. Below, the curve is stuck parallel to the "no coding" one. A possible
solution to overcome this flattening involves using an outer code, like the Reed-Solomon (RS) code.

Behaviour 3 has poor convergence and high dmin. This is representative of a decoding procedure that does not take
advantage of all the information available on the receiver side. A typical example is the classical concatenation of an RS
code and a simple convolutional code. Whereas the minimum distance may be very large (still depending on the
interleaver depth), the decoder is clearly sub-optimal because the convolutional (Viterbi) decoder does not take
advantage of the RS redundant symbols.

The search for the perfect codec scheme has always faced the "convergence versus dmin" dilemma. Usually, improving
one of either aspect, by some more or less relevant idea, weakens the other.

Turbo codes have constituted a real breakthrough with regard to the convergence problem. But it is only for some years
that, in addition to good convergence properties, sufficient minimum distances have been achieved, leading now to
powerful standalone channel coding. As can again be observed from Fig. 1, minimum distances as large as those given
by random coding are not necessary. Hence, dmin = 25, for a 188-byte block with 1/2 rate, would be sufficient to reach
the theoretical limit @ BER = 10-8, instead of the value 380 given by random coding. dmin = 32 would be necessary @
BER = 10-10.

To conclude, if a good code has to display the same convergence threshold as random coding, its minimum distance
may, in practical cases, be much more reasonable.

3. TURBO CODES

3.1 Random-Like Coding

Because we want to obtain performance close to the Shannon limit, which relies on random coding, we have to
construct a coding scheme with some strong random property.

Forney, in a well known paper devoted to convolutional codes [5], made a comparative analysis of non-recursive (non-
systematic) and recursive (systematic) codes, coming to the conclusion that both classes were interesting, each from a
particular point of view. However, Viterbi, one year later [6, chap. VIII], no longer mentioned recursive codes in the
comparison he made between systematic and non-systematic codes, and it is quite true that non-recursive systematic
convolutional codes are unattractive, because of their poor minimum distance (which can be as low as 1 when
punctured!). So, systematic codes, recursive or not, were pushed into the background ... and the random property of
convolutional codes was temporarily lost.

Figure 2 considers both types of encoders: the non-recursive non-systematic convolutional (NSC) and the recursive
systematic convolutional (RSC) encoders, constructed from the same generators GX(D)= 1 + D + D3, GY(D)= 1 + D2 +
D3, with 1/2 rate. Assuming the coding of an infinite length sequence with registers initialized in state 0, we can observe
that :
 the NSC encoder has finite impulse response, whereas the RSC encoder yields infinite informative redundancy Y
about a single "1" entering the encoder.

X X

d d s3
s1 s2 s3 s1 s2
(data) (data)
non-recursive
 recursive

non-systematic Y systematic Y

Figure 2. The two forms of convolutional encoders.

 all finite weight1 input sequences produce finite weight sequences at the output of the NSC code. In the case of RSC
codes, only a fraction of finite weight input sequences yield finite weight sequences (i.e. with "poor" redundancy).
These particular sequences, which have to be a multiple of GX(D), are denoted RTZ (Return To Zero) sequences [7].
Their proportion is exactly

p1 2− , (2)

where is the code memory ( = 3 in the example shown). Hence, for sufficiently large values of, only a small
proportion of data sequences feeding an RSC encoder will not be of the RTZ type.

In order to decrease even more this proportion to an extremely low level, multi-dimensional coding can be devised [8].
This relies on a parallel concatenation of N CRSC (Circular Recursive Systematic Convolutional) encoders, thus called
because they use circular or "tail-biting" termination [9-10]. Fig. 3 gives the principle of the encoding, for blocks of k
bits with 1/2 global rate. Thanks to the circular termination technique, which involves making the initial and the final
state of the encoder equal, no additional bit is needed to transform each RSC code into a block code. Moreover, if the
permuted sequence at the input of a particular component encoder is not an RTZ sequence, the whole redundancy set of
the corresponding circle is affected by this sequence, and not only the part after it, as would be the case with a 0-
initialized encoder.
systematic
part
Y11
1
Yk/N k1

k data permutation CRSC

circular
1
trellis
(identity)

Yp1

Y12
permutation CRSC c 2
k1
o
2 d
e circular
w trellis
Yp2
o
r
d

Y1N
N
CRSC k1
permutation

N circular
trellis

YpN

Yk/N
Figure 3. Multiple parallel concatenation of N CRSC codes.

1
The weight of a binary word is the number of bits equal to ‘1’, that is the number of bits differing from the « all zero » word, which
is used as a reference for linear codes.

Yk/N
With this N-dimension encoder and N - 1 permutations drawn at random, the proportion of sequences which are RTZ
for all encoders, is lowered to the value:

− N
p N 2 , (3)

and the other codewords, with proportion 1 - pN, have a minimum distance at least equal to

k
 d min . (4)
2N

This value assumes that only one permuted sequence is not RTZ and that Y takes value "1" one time out of two
statistically, in the corresponding circle.

Choosing N = 8 and = 3 for instance would give p8≈ 10-7 and for a block of 1024 bits, d min would be equal to
64.

Fortunately, from the decoding point of view, it is not necessary to extend the dimension of the composite code to a
large value. By properly designing permutation2, instead of drawing it at random, the composite code with N = 2
only, which is the standard turbo code, offers quasi-optimal performance by itself.

3.2 Non-Regular Permutation

The dilemma in the design of a good permutation resides in the need to obtain a sufficient minimum distance for two
distinct classes of codewords, which require conflicting treatment. The first class contains all codewords with input
weight w≤ 3, and a good permutation for this class is as regular as possible. The second class encompasses all
codewords with input weight w > 3, and non uniformity (controlled disorder) has to be introduced in the permutation
function to obtain a large minimum distance. Fig. 4 illustrates the situation, with the example of a 1/3 rate turbo code,
using component encoders with code memory =3 and periodicity L = 2-1 = 7.

N columns

00...001000000100...00
M
lines

X (a)

C1
1101
data

regular
permutation Y1
on
k = M.N bits
C2 (b)

1101
0000 1101
0000 1101
0000 0000
Y2 0000 1101
0000
0000
(c) 1101

Figure 4. Some possible RTZ (Return To Zero) sequences for both encoders C 1 and C2. (a): with input weight w = 2; (b): with w = 3;
(c): with w = 6 or 9.
For the sake of simplicity, the block of k bits is organized as a rectangle with M lines and N columns ( M≈ N≈ k ).
Regular permutation is used, that is data are written linewise and read columnwise.

Case (a) depicts a situation where encoder C1 (the horizontal one) is fed by an RTZ sequence with input weight w = 2.
Redundancy Y1 delivered by this encoder is poor, but redundancy Y2 produced by encoder C2 (the vertical one) is very
informative for this pattern, which is also an RTZ sequence, but whose span is 7. M instead of 7. The associated
7 .M
minimum distance would be around , which is a large minimum distance for typical values of k. With respect to
2
this w = 2 case, the code is said to be "good" because dmin tends to infinity when k tends to infinity.

Case (b) deals with a w = 3 RTZ sequence. Again, whereas the contribution of redundancy Y1 is not high for this
pattern, redundancy Y2 gives relevant information over a large span, of length 3. M. The conclusions are the same as
previously.

Case (c) shows two examples of sequences with weights w = 6 and w = 9, which are RTZ sequences for both encoders
C1 and C2. They are obtained by a combination of two or three minimal length RTZ sequences. The set of redundancies
is limited and depends neither on M nor on N. These patterns are typical of codewords that limit the minimum distance
of a turbo code, when using regular permutation.

In order to "break" rectangular patterns, some disorder has to be introduced in the permutation rule, but with the
concern that the good properties of regular permutation, with respect to weights 2 and 3, are not lost. This is the crucial
problem in the search for good permutations. There is no optimal solution for the time being, though the literature in
this area is copious. Nevertheless, some good permutations have already been devised for several applications (CCSDS
[11], UMTS [12], DVB-RCS [13], ...), even without the support of a sound definitive theory.

3.3 Iterative Decoding Using Extrinsic Information

Decoding a composite code by a global approach is not possible in practice, because of the tremendous number of states
to consider. A joint probabilistic process by the decoders of C 1 and C2, has to be elaborated. Because of latency
constraints, this joint process is worked out in an iterative manner.

Turbo decoding relies on the fundamental following criterion for message passing algorithms:

when having several probabilistic machines work together on the estimation of a common set of symbols, all the
machines have to give the same decision, with the same probability, about each symbol, as a single (global) decoder
would do.

To make the composite decoder satisfy this criterion, the structure of Fig. 5 is adopted. The components are Soft-
In/Soft-Out (SISO) decoders, and permutation and inverse permutation memories. The node variables of the decoder are
Logarithms of Likelihood Ratios (LLR). An LLR related to a particular datum d is defined as:

æ Pr(d 1) ö
LLR(d ) logçç ÷÷ (5)
è Pr(d 0) ø

(working in the logarithmic domain means that multiplications are replaced by additions).

The role of a SISO decoder is to receive an input LLR and, thanks to local redundancy (i.e. Y1 for DEC1, Y2 for DEC2),
to try to improve it. The output LLR of a SISO decoder may be written as

LLR out (d ) LLR in (d ) z(d ) (6)

where z (d ) is the extrinsic information about d, provided by the decoder. If this works properly, z is negative if d = 0,
and positive if d = 1.

According to the structure of the decoder, after p iterations, the output of DEC1 is
 X
z1
C1 
data 8-state
(x+z2)+z1
SISO
y1 DEC1
decoded
permutation Y1 x
y2 8-state
output
 C2 SISO d^
x DEC2

(x+z1)+z2
−  z2
2
Y

Figure 5. A turbo code and its associated decoder (basic structure).

LLR out, p1 (d ) (x z p−12 (d )) z p1 (d )

and the output of DEC2 is

LLR out, p 2 (d ) (x z p−11 (d )) z p 2 (d )

If the iterative process converges towards fixed points, z p1 (d )− z p−11 (d ) and z p 2 (d )− z p−12 (d ) tend to zero when p
goes to infinity. Therefore, both LLRs become equal, which fulfils the fundamental condition of equal probabilities for
each datum d.

As for the proof of convergence itself, one can refer to various thorough papers dealing with the theoretical aspects of
the subject, like [14] or [15].

In practice, depending on the kind of SISO algorithm chosen, some tuning (mutiplying, limiting) operations on extrinsic
information, are added to the basic structure to ensure convergence within a small number of iterations.

4. SOME RECENT RESULTS WITH DUO-BINARY TURBO CODES

Duo-binary turbo codes are built from a parallel concatenation of two RSC component codes, each with two inputs.
The advantages of this construction, compared to classical turbo codes, are many [16-19]: better convergence of the
iterative decoding, large minimum distances (i.e. large asymptotic gains), less sensitivity to puncturing patterns,
reduced latency, and robustness towards the flaws of the component decoding algorithm, in particular when the MAP
(Maximum A Posteriori) algorithm is simplified to the Max-Log-MAP.

On the basis of 8-state trellises, duo-binary turbo codes have already been adopted for the European standardization of
the return channel for satellite networks (DVB-RCS, ETSI EN 301 790 [13]) and are also in the process of
normalization for the return channel on terrestrial networks (DVB-RCT). Combined with the technique of circular
treillises, this turbo code offers good performance and versatility for encoding blocks with various sizes (from 12 to 216
bytes) and various rates (from 1/3 to 6/7), while keeping reasonable decoding complexity.

Replacing the 8-state component encoder by a 16-state encoder allows better performance at low error rates, at the price
of doubled complexity. Minimum distances are increased by 50% on average, with regard to the 8-state duo-binary
turbo code, and allow the Frame Error Rate (FER) curve to decrease down to 10-8 without noticeable change in the
slope. Such a low FER is obtained at signal to noise ratios that are situated for most cases at the theoretical limit plus
about 1.3 dB, even with the simplified MAP algorithm (Max-Log-MAP).
 FER
5 R = 2/3
10-1
5
10-2
5 8-PSK (pragmatic)
10-3
5 4-PSK

theoretical limit
188 b.

188 b., R = 2/3

10-4
5  =3
10-5 (DVB-RCS)
5
10-6
5
10-7 188 b. 376 b. 188 b.
5  =4  =4  =4
10-8
1 2 3 4 5 6
Eb/N0 (dB)

Figure 6. Performance, in Frame Error Rate (FER), of the 16-state duo-binary turbo code for 188 bytes (QPSK and 8-PSK) and 376
bytes (8-PSK). AWGN and rate 2/3. Max-Log-MAP decoding with 4-bit (QPSK) or 5-bit (8-PSK) input samples and 8 iterations.

Fig. 6 gives some examples of performance for a 2/3 coding rate, with the real constraints of an actual implementation.
With QPSK modulation and 188-byte (MPEG) blocks, the gain achieved is around 1 dB @ FER = 10-7, compared to 8-
state duo-binary turbo codes. No change in the slope appears, though it is strictly parallel concatenation. This is due to
the care taken in the design of the permutation function.

8-PSK modulation is used with so-called pragmatic mapping [20], designed so as to better protect the systematic part of
the codewords. Again, excellent performance can be observed for 188 and 376-byte blocks, with losses, related to the
theoretical limits (3.5 and 3.3 dB respectively), close to those obtained with QPSK.

5. CONCLUSION

Turbo codes now allow engineers to design systems able to work in conditions close to the theoretical capacities, but
there is still some progress left to be made. The example given in Fig. 6 shows that gaining one more dB in the
convergence threshold, while increasing minimum distances by roughly 25%, would be the ultimate answer. That is,
unless somebody with some good ideas and no respect for theoretical limits simply decides to go further.

REFERENCES

[1] C. Berrou, A. Glavieux and P. Thitimajshima, "Near Shannon limit error-correcting coding and decoding: turbo-
codes", Proc. of IEEE ICC '93, Geneva, pp. 1064-1070, May 1993.
[2] C. E. Shannon, "A mathematical theory of communication", Bell System Technical Journal, Vol. 27, July and
October 1948.
[3] J. Hagenauer and P. Hoeher, "Concatenated Viterbi-decoding", Proc. Int. Workshop on Inf. Theory, Gotland,
Sweden, Aug-Sep. 1989.
[4] R. G. Gallager, Low-density parity-check codes, MIT Press, 1963.
[5] G. D. Forney Jr, "Convolutional codes I: algebraic structure", IEEE Trans. Inf. Theory, Vol. IT-16, N° 16, Nov.
1970.
[6] A. J. Viterbi, "Convolutional codes and their performance in communication systems", IEEE Trans. Comm.
Technology, Vol. COM-19, N° 15, Oct. 1971
[7] R. Podemski, W. Holubowicz, C. Berrou and G. Battail, "Hamming distance spectra of turbo-codes", Ann.
Télécomm., Tome 50, N° 9-10, pp. 790-797, Sept-octobre 1995.
[8] C. Berrou, C. Douillard and M. Jézéquel, "Multiple parallel concatenation of circular recursive convolutional
(CRSC) codes", Ann. Télécomm., Tome 54, N° 3-4, pp. 166-172, March-April 1999.
[9] C. Bettstetter, "Turbo decoding with tail-biting trellises", Diplomarbeit, Technischen Universität München, Jul.
1998.
[10] C. Berrou, C. Douillard and M. Jézéquel, "Designing turbo codes for low error rates", IEE Colloquium: "Turbo
codes in digital broadcasting - Could it double capacity?", pp. 1-7, London, Nov. 1999.
[11] CCSDS, Telemetry channel coding, CCSDS 101.0-B-4, Blue Book, http://www.ccsds.org/documents/pdf/CCSDS-
101.0-B-4.pdf, May 1999.
[12] 3GPP Technical Specification Group, Multiplexing and channel coding (FDD), 3GPP Technical Specification, TS
25.212 v2.0.0, 1999.
[13] DVB, "Interaction channel for satellite distribution systems", ETSI EN 301 790, V1.2.2, pp. 21-24, Dec. 2000.
[14] S. M. Aji and R. J. McEliece, "The generalized distributive law", IEEE Trans. Inf. Theory, Vol. 46, pp. 325-343,
Mar. 2000.
[15] Y. Weiss and W. T. Freeman, "On the optimality of solutions of the max-product belief-propagation algorithm in
arbitrary graphs", IEEE Trans. Inf. Theory, Vol. 47, N° 2, pp. 736-744, Feb. 2001.
[16] C. Berrou, "Some clinical aspects of turbo codes", in Proc. of the International Symposium on Turbo Codes &
Related Topics, pp. 26-31, Brest, Sept. 1997.
[17] C. Berrou and M. Jézéquel, "Non binary convolutional codes for turbo coding", Elect. Letters, Vol. 35, N° 1, pp.
39-40, Jan. 1999.
[18] C. Berrou, M. Jézéquel, C. Douillard and S. Kerouédan, "The advantages of non-binary turbo codes", to appear at
ITW 2001, Cairns, Australia, Sept. 2001.
[19] M. Bingeman and A. K. Khandani, "Symbol-based turbo codes", IEEE Comm. Letters, Vol. 3, N° 10, pp. 285-287,
Oct. 1999.
[20] S. Le Goff, A. Glavieux and C. Berrou, "Turbo-codes and high spectral efficiency modulation", Proc. of IEEE
ICC'94, pp. 645-649, New Orleans, May 1994.