Chapter 11

Codes on graphs
In this chapter we will introduce the subject of codes on graphs. This subject forms an intel-
lectual foundation for all known classes of capacity-approaching codes, including turbo codes,
low-density parity-check (LDPC) codes, and repeat-accumulate (RA) codes.
In Chapter 10, we saw that trellis representations of codes are basically time-varying state-
space realizations of codes on a sequential, ordered time axis which is a subinterval of the
integers. Similarly, codes on graphs may be regarded as based on a more generalized notion of
a realization in which the \time axis" is an unordered graph.
We begin by introducing this more general class of realizations, which in system theory are
called \behavioral realizations." For linear codes, it suÆces to consider linear realizations.
We rst consider elementary linear behavioral realizations, in which the constraints are given
by sets of linear equations. We then consider general linear behavioral realizations in which
the constraints are given by linear codes. Then we show how behavioral realizations may be
naturally represented by graphical models of various types: Tanner graphs (factor graphs),
normal graphs, and Markov graphs (Markov random elds). Finally, we show some elementary
but important connections between the properties of realizations and their graphs.

11.1 Elementary realizations of linear block codes

We continue to restrict attention to linear (n; k; d) block codes over a nite eld F q . So far we
have seen several general methods of representing such codes:
P
By a set of k generators fgj ; 1  j  kg. The code C is then the set of all linear combinations
of generators over F q .
i ai gi
 By a set of n k generators fhj ; 1  j  n kg for the dual code C ?. The code C is then
the set of all n-tuples x 2 (F q )n such that hx; hj i = 0 for all j .
 By a trellis (state-space) representation. The code is then the set of all n-tuples correspond-
ing to paths through the trellis.

149
150 CHAPTER 11. CODES ON GRAPHS

We will see that these representations are all special cases of a general class of representa-
tions called behavioral realizations (from the behavioral approach to system theory pioneered
by Willems). In general, a behavioral realization de nes a code by a set of constraints that the
code symbols and other auxiliary state variables must satisfy.
For linear codes, we need consider only linear behavioral realizations, where the variables are
over a eld and the constraints are linear. In the simplest case, the variables are eld elements
and the constraints are linear equations involving the variables. In the general case, the variables
can be vector spaces over the eld, and the constraints are expressed in terms of linear codes.

11.1.1 Elementary linear behavioral realizations

The elements of an elementary linear behavioral realization of a linear (n; k; d) block code over
F q are as follows:

 The n code symbols x = fxi 2 F q ; i 2 I g, where I denotes the symbol index set.
 An additional set of s auxiliary variables s = fsj 2 F q ; j 2 J g, often called state (hidden,
latent, unobserved) variables, where the state variable index set J may be unrelated to I .
 A set of e linear homogeneous equations over F q involving the components of the symbol
and state variables, called the constraint equations.
B
The full behavior generated by the realization is the set of all combinations (x; s) (called
trajectories ) of symb ol and state variables that satisfy all constraint equations. The code C
generated by the realization is the set of all symbol n-tuples x that appear in any trajectory
(x; s) 2 B; i.e., such that there exists some set s of state variables such that (x; s) 2 B.
In general, the e linear homogeneous constraint equations may be written in matrix form as
xA + sB = 0;
where x is a row n-tuple of symbols, s is a row s-tuple of state variable components, and A and
B are n  e and s  e F q -matrices, respectively. The set B of all solutions (x; s) to such a set
of equations is a subspace of the vector space (F q )n+s of dimension dim B  n + s e, with
equality if and only if all equations are linearly independent.
The code C is the projection of B onto its rst n components. The dimension of C is equal to
the dimension of B if and only if codewords corresponding to distinct trajectories are distinct.
We now show that generator matrices and parity-check matrices yield elementary behavioral
realizations.
Example 1. Let G be a k  n generator matrix for C , whose k rows form a set of linearly
independent generators for C . Then C is the set of all n-tuples of the form x = aG for some
information k-tuple a 2 (F q )k . Thus C has an elementary linear behavioral realization with a
state k-tuple a and n constraint equations, namely
x aG = 0:
11.1. ELEMENTARY REALIZATIONS OF LINEAR BLOCK CODES 151
For example, the following generator matrix generates an (8; 4; 4) binary linear block code:
2 3
11110000
66 01011010 77
4Q00111100 5: (11.1)
00001111
This yields a linear behavioral realization with 4 state variables and 8 constraint equations,
namely the following linear homogeneous equations over F 2 :
x0 = a1 ;
x1 = a1 + a2 ;
x2 = a1 + a3 ;
x3 = a1 + a2 + a3 ; (11.2)
x4 = a2 + a3 + a4 ;
x5 = a3 + a4 ;
x6 = a2 + a4 ;
x7 = a4 :

Example 2 . Let H be an (n k)  n generator matrix for C ?. Then C is the set of all n-tuples
that satisfy the n k constraint equations
xH T = 0:
This corresponds to an elementary linear behavioral realization with no state variables.
For example, since the (8; 4; 4) code C is self-dual, the generator matrix (11.1) is also a generator
matrix for C ?. This yields an elementary linear behavioral realization with no state variables
and 4 constraint equations, namely the following linear homogeneous equations over F 2 :
x0 + x1 + x2 + x3 = 0;
x1 + x3 + x4 + x6 = 0; (11.3)
x2 + x3 + x4 + x5 = 0;
x4 + x5 + x6 + x7 = 0:
This is evidently a more compact representation than the generator representation of Example
1| in fact, it can be found by eliminating the state variables from Eqs. (11.2)| and, because
it has no state variables, it is better suited for checking whether a given 8-tuple x is in C . On
the other hand, in the generator representation the state 4-tuple a may be freely chosen and
determines the codeword| i.e., the generator representation is an input-output representation|
so it is better for generating codewords, e.g., in encoding or in simulation.

11.1.2 Graphs of elementary linear behavioral representations

We may draw a graph of an elementary linear behavioral representation as follows. In coding

theory, such a graph is called a Tanner graph or a factor graph.
152 CHAPTER 11. CODES ON GRAPHS

The graph has two types of vertices, namely n + s vertices corresponding to the n symbol and
s state variables, and e vertices corresponding to the e constraint equations. An edge is drawn
between a variable vertex and a constraint vertex if and only if the corresponding variable is
involved in the corresponding constraint equation. Thus the graph is bipartite ; i.e., the vertices
may be partitioned into two sets such that every edge connects a vertex of one type to a vertex
of the other type.
A generic Tanner graph therefore has the form of Figure 1. Here symbol variables are repre-
sented by lled circles, state variables by open circles, and constraints by squares containing a
\+" sign.
~H\ H
~HX\XH\XHXHHHH
...
H\H\XHXXXHXHHX
\\HHHH +
~HH \H +

nXXXHHXHHH\\\ . . .
X
n XXHXHX\HX\ +
...   

n
Figure 1. Generic bipartite Tanner graph, with symbol ( lled circles) and state (open circles)
variables, and with elementary linear constraint equations (squares with \+").

The degree of a variable or equation will be de ned as the degree (number of incident edges)
of the corresponding graph vertex| i.e., the degree of a variable is the number of equations
that it is involved in, and the degree of an equation is the number of variables that it involves.
Clearly the sum of the variable degrees is equal to the sum of the constraint degrees, since both
are equal to the number of edges in the graph.
Example 1 (cont.) The Tanner graph corresponding to the generator representation of the
(8; 4; 4) code de ned by Eqs. (11.2) is shown in Figure 2(a). Since each symbol variable has
degree 1 in this representation, the corresponding symbol vertex is located adjacent to the
unique constraint vertex with which it is associated.
We will say that a symbol variable of degree 1 is normalized. We may denote a symbol variable
of degree 1 by a special \dongle" symbol as in Figure 2(b), denoting a \half-edge" (a hyperedge
of degree 1) incident on the associated constraint vertex. The \dongle" is an input/output
terminal that can connect the realization with the outside world.
11.2. GENERAL LINEAR BEHAVIORAL REALIZATIONS 153

 + ~x0  + x0

  + 
  +
     ~x1      x1
       
a1 nXXXX  + ~x2 a1 nXXXX  + x2
X X
a2 nXZXXXXXXXX + ~x3 a2 nXZXXXXXXXX + x3
X X
a3 nXZXZXZXXXXXX + ~x4 a3 nXZXZXZXXXXXX + x4
Z
X ZX
a4 nHXHXXXZXZXZXXX + ~x5 a4 nHXXHXXZXXZXZXX + x5
HHHXXXXXZX HHXHXXXXZXZ
HHH Z + ~x6 HHH + x6
HH + ~x7 HH + x7

Figure 2. (a) Generator representation for (8; 4; 4) code. (b) Alternative representation.
Example 2 (cont.) The Tanner graph corresponding to the parity-check representation of the
(8; 4; 4) code de ned by Eqs. (11.3) is shown in Figure 3.
x0 ~HH
x1 ~XHXHHXHXHHH
X
x2 ~HHHHHHXXHHXHXHX +

x3 ~XXHXHXHHHHHH +
X
x4 ~XXXXXXHXHXXH +
 
XXXXX
x5 ~  X +


x6 ~
x7 ~
Figure 3. Parity-check representation for (8; 4; 4) code.

11.2 General linear behavioral realizations

We now generalize the elementary realizations above by letting symbol and state variables be m-
tuples over F q , where m may be di erent for each variable. Furthermore, we replace constraint
equations by constraints that certain subsets of variables must lie in certain small linear block
codes over F q .
The elements of a general linear behavioral realization of a linear (n; k; d) block code over F q
are therefore as follows:
154 CHAPTER 11. CODES ON GRAPHS

 A set of symbol mi-tuples fxi 2 P

(F q )mi ; i 2 I g, where I denotes the (clustered) symbol
variable index set. We de ne n = m . I i
 A setPof state j -tuples fsj 2 (F q )j ; j 2 J g, where J denotes the state index set. We de ne
s= .
J j
 A set of linear constraint codes fCk ; k 2 K g over F q , where each code Ck involves a certain
subset of the symbol and state variables, and K denotes the constraint index set.
Again, the full behavior B generated by the realization is the set of all trajectories (x; s) such
that all constraints are satis ed| i.e., such that for each k the values taken by the subset of
variables involved in the constraint code Ck forms a codeword in Ck | and the code C generated
by the realization is the set of all symbol sequences x that appear in any trajectory (x; s) 2 B.
Notice that the constraint imposed by a linear homogeneous equation that involves r variables
is equivalent to a constraint that these variables must lie in a certain (r 1)-dimensional subspace
of (F q )r ; i.e., that they form a codeword in a (r; r 1) linear block code over F q . So an elementary
linear behavioral realization is indeed a special case of this more general type of realization.
The generalization to m-tuple variables allows us to consider state spaces of dimension larger
than 1, which we need for general trellis (state-space) realizations. It also allows us to consider
the clustered symbol variables that arise in sectionalized trellis representations.
We now show how trellis (state-space) realizations may be expressed as general linear behav-
ioral realizations. For simplicity we restrict attention to the unclustered case.
Example 3. Let us consider a conventional linear state-space realization of C on the time axis
I = [0; n) (i.e., an unclustered linear trellis representation).
Here we de ne state spaces j = (F q )j of dimension j for the state time axis J = [0; n],
where 0 = n = 0; i.e., the starting and ending state spaces have size j0 j = jn j = 1.
We then de ne each trellis section (state transition) by a linear subspace Ci  i i+1 F q ; i 2
[0; n), which de nes the set of state transitions (si ; si+1 ) 2 i  i+1 that can possibly occur and
the code symbol xi 2 F q associated with each such transition. In other words, the behavioral
constraint at time i is that the triple (si ; si+1 ; xi ) must lie in a certain small linear block code
Ci of length ni = i + i+1 + 1 over F q . Each set of variables that satis es the constraints at
time i corresponds to a distinct valid trellis branch labelled by the corresponding (si ; si+1 ; xi ),
so the branch complexity at time i is the size jCi j of the constraint code Ci .
The full behavior B of this realization is then the set of all state/symbol sequences (s; x) such
that (si ; si+1 ; xi ) 2 Ci for 1  i  n, which is a set of linear homogeneous constraints. For
each state/symbol sequence (s; x) in B, the state sequence s represents a valid path through the
code trellis, and the symbol sequence x represents the corresponding codeword. If the trellis is
minimal, then each path (s; x) corresponds to a distinct codeword x, so jBj = jC j.
For example, in Chapter 10 we saw how to construct a minimal trellis (state-space) represen-
tation for a linear block code C from a set of trellis-oriented generators fgj ; 1  j  kg for C .
Each generator contributes a one-dimensional component to the state space i during each time
i that it is active.
Continuing with our (8; 4; 4) example, the generators of (11.1) are trellis-oriented and are
active during the intervals [0; 3); [1; 6); [2; 5) and [4; 7), respectively. Therefore we may de ne
the state spaces at times i = 1; 2; 3; 4; : : : by their F 2 components as follows: s1 = (s11 ); s2 =
(s21 ; s22 ); s3 = (s31 ; s32 ; s33 ); s4 = (s42 ; s43 ); : : :, where
11.2. GENERAL LINEAR BEHAVIORAL REALIZATIONS 155

s11 = s21 = s31 = a1 ;

s22 = s32 = s42 = s52 = s62 = a2 ;
s33 = s43 = s53 = a3 ;
(11.4)
s54 = s64 = s74 = a4 :
Here (a1 ; a2 ; a3 ; a4 ) denote the same generator coeÆcients as in (11.2). These can be thought of
as four free variables (inputs, driving variables) that arrive at times 0, 1, 2 and 4, respectively,
and remain active in the state spaces (in memory) through times 3, 6, 5, and 7, respectively.
The code symbols xi are then determined as in (11.2) as functions of si and si+1 by
x0 = a1 = s11 ;
x1 = a1 + a 2 = s11 =s21 + s22 ;
x2 = a1 + a3 = s21 =s31 + s33 ;
(11.5)
x3 = a1 + a2 + a3 = s31 + s32 =s42 + s33 =s43
and so forth, where the slashes indicate that the relevant coeÆcient may be determined from
either si or si+1 . The point of (11.5) is that the code symbol at any time may be determined
locally by a constraint equation involving only the previous and next states.
Figure 4 shows a Tanner graph representing Eqs. (11.4){(11.5) (not including the ai variables).
Each horizontal row contains the state components associated with a single trellis-oriented gen-
erator during its active interval. Each vertical column contains the state components that are
active at a given time. The fact that the code symbol at any time can be determined from
the previous and next states is indicated symbolically by a dashed box around xi and the state
components at times i and i + 1. The corresponding linear constraint is denoted by Ci .
x0 x1 x2 x3 x4 x5 x6 x7

C0
n = s21n = s31n C3
s11 C5 C6 C7
s22n = s32n = s42n = s52n = s62n
C1
s33n = s43n = s53n
C2
s54n = s64n = s74n
C4
Figure 4. Trellis representation of (8; 4; 4) code.
The constraint codes Ci that determine each trellis section are therefore as follows:
 For (s0; s1 ; x0 ), there are only two F 2 -variables, s11 and x0, which satisfy the constraint
s11 = x0 (= a1 ); i.e., the 2-tuple (s11 ; x0 ) is constrained to lie in the (2; 1; 2) binary repetition
code, which we denote by C0 .
 For (s1; s2; x1 ), there are four F2 -variables, s11; s21 ; s22 and x1, which satisfy the two con-
straints s21 = s11 (= a1 ) and x1 = s11 + s22 (= a1 + a2 ); i.e., the 4-tuple (s11 ; s21 ; s22 ; x1 )
is constrained to lie in a certain linear (4; 2) code C1 .
 For (s2; s3 ; x2 ), there are six F 2 -variables (s21; s22 ; s31 ; s32 ; s33; x2 ), which satisfy the three
constraints s31 = s21 (= a1 ), s32 = s22 (= a2 ), and x2 = s21 + s33 (= a1 + a3 ); i.e., they
must lie in a certain linear (6; 3) code C2 . Etc.
156 CHAPTER 11. CODES ON GRAPHS

11.3 Graphs of general linear b ehavioral representations

There are various styles for drawing graphs of general linear behavioral representations. Tanner
graphs represent both variables and constraints by vertices; \normal graphs" represent variables
by edges (or hyperedges) and constraints by vertices; Markov graphs represent variables by
vertices and constraints by edges (or hyperedges, or cliques).

11.3.1 Tanner graphs

We start with generalized Tanner graphs. Again, such a graph is bipartite, with the two types
of vertices representing variable m-tuples and constraint codes, respectively. Again, an edge is
drawn between a variable vertex and a constraint vertex if and only if the corresponding variable
is involved in the corresponding constraint code.
A generic generalized Tanner graph is illustrated in Figure 5. It looks much like Figure 1. It is
bipartite, with symbol m-tuples represented by lled circles, state m-tuples by un lled circles,
and constraints by squares. However, the squares may now be labelled to denote various types
of constraint codes. A \+" sign will continue to denote a zero-sum code on all incident variables.
An \=" sign will denote a repetition code; i.e., all incident variables must be equal.
~H\ H
~HX\XH\XHXHHHH
...
H\H\XHXXXHXHHX
\\HHHH
~HH \H

nXXXHXHHHH\\\ . . .
X
n XXHXHX\XH\
...  

n
Figure 5. Generic generalized Tanner graph, with both symbol ( lled circles) and state (open
circles) variables, and with generalized linear constraint codes (squares).
Example 3 (cont.) Figure 6 is a generalized Tanner graph of the same state-space realization
as was illustrated in Figure 4. Here we aggregate the state spaces into i -tuples si ; 0  i  8,
where the state dimension pro le is f0; 1; 2; 3; 2; 3; 2; 1; 0g. We label each state variable by its
dimension. The state spaces 0 and 8 do not need to be shown, because they have dimension 0
and are not actually involved in any constraints. The constraint code Ci constrains (si ; si+1 ; xi ),
and thus has length i + i+1 + 1. We label each constraint code by its length and dimension.
Since the symbol variables have degree 1, we use the special \dongle" symbol for them.
11.3. GRAPHS OF GENERAL LINEAR BEHAVIORAL REPRESENTATIONS 157
X0 X1 X2 X3 X4 X5 X6 X7

C0 S1 n C1 Sn
2 C2 Sn
3 C3 Sn
4 C4 Sn
5 C5 Sn
6 C6 Sn
7 C7
(2; 1) 1 (4; 2) 2 (6; 3) 3 (6; 3) 2 (6; 3) 3 (6; 3) 2 (4; 2) 1 (2; 1)
Figure 6. Generalized Tanner graph for trellis representation of (8; 4; 4) code.
In general, every linear state-space (trellis) representation has a chain Tanner graph similar
to that of Figure 6, with constraint codes Ci constraining (si ; si+1 ; xi ). In general the symbol
variables xi , like the state variables, may have any dimension; i.e., symbols may be clustered.
Note that in a trellis representation all symbol variables have degree 1, and all nontrivial state
variables have degree 2. Note also that there are no cycles (loops).

11.3.2 Normal graphs

In \normal graphs" (the terminology is not yet standard), constraint codes are again represented
by vertices, but variables are represented by edges (if the variable has degree 2), or by hyperedges
(if the variable has degree other than 2). Consequently a normal graph is a hypergraph (a graph
whose (hyper)edges may have arbitrary degree). We have already introduced a symbol for a
hyperedge of degree 1 (i.e., incident on a single vertex). Hyperedges of degree greater than 2
may be represented by a star of edges interconnecting all incident vertices.
Normal graphs are particularly nice when the degrees of all variables are 2 or less. We de ne a
realization to be normal if the degree of every symbol variable is 1, and the degree of every state
variable is 2. We then represent symbol variables as before by \half-edges" using our special
\dongle" symbols, and state variables by ordinary edges.
Normality may seem to be a stringent restriction. However, note that a conventional state-
space realization as in Figure 6 is already normal. Figure 7 shows a corresponding normal
graph.
X0 X1 X2 X3 X4 X5 X6 X7

C0 S1 C1 S2 C2 S3 C3 S4 C4 S5 C5 S6 C6 S7 C7
(2; 1) 1 (4; 2) 2 (6; 3) 3 (6; 3) 2 (6; 3) 3 (6; 3) 2 (4; 2) 1 (2; 1)
Figure 7. Normal graph for trellis representation of (8; 4; 4) code.
Moreover, any realization (e.g., a generic realization as illustrated in Figure 5) may be trans-
formed into a normal realization by the simple conversion shown graphically in Figure 8. In this
generic normal realization, the equality constraints ensure that the variables corresponding to
all incident edges are equal, and \dongles" indicate symbol variables.
Figure 8 illustrates that every realization has an equivalent normal realization whose normal
graph is essentially the same as the Tanner graph of the original realization. Figure 7 illustrates
that the normal graph may be simpler, if some or all of the states already have degree 2.
Normal graphs have some conceptual advantages over Tanner graphs. When we discuss de-
coding, we will see that they lead to a clean functional separation: symbol variables (half-edges)
are for input-output, state variables (edges) are for communication (message-passing), and con-
straints (vertices) are for computation. Representing states as edges also highlights their role as
158 CHAPTER 11. CODES ON GRAPHS

=H
\\HH
=H XXH\XXHXHHH
H\H\HXXXHXHHX
...
\\HHHH
=H    H
\
HHH\

= XXXXHHH \\ . . .
X
=  XXHXHX\XH\
...   
  
=
Figure 8. Normal graph of a normal realization equivalent to Figure 5.

sites for graph cuts. Finally, there are some beautiful duality theorems for normal graphs that
we will not have time to cover here (basically, that the dual normal graph generates the dual
code). Therefore we will primarily use normal graphs in these notes. However, Tanner graphs
are better established and predominate in the current literature.
We have seen that there are many styles of normal realizations which result in di erent graphs
of the same code. For the (8; 4; 4) code, we have seen the generator representation of Figure 2,
the parity-check representation of Figure 3, and the trellis representation of Figure 7.
Figure 9 shows three further examples for this code. Figure 9(a) is a two-section, four-state
conventional trellis clustering four symbol bits at a time, which as we saw in Chapter 10 is
optimal. Figure 9(b) is a two-section \tail-biting" (circular) trellis with two 2-state state spaces,
which in this case may be obtained merely by splitting the central state space of Figure 9(a).
Note however that while Figure 9(b) has smaller state spaces, it is no longer cycle-free.
' 1 $
4 4 = + = + = + = +
2 4 4 @@ @@
(6; 3) (6; 3) (6; 3) (6; 3) @= @=
& % 1 + +
(a) (b)   (c)  
Figure 9. Normal graphs for (8; 4; 4) code: (a) Two-section, four-state trellis;
(b) Two-section, two-state tail-biting trellis; (c) RM-style realization.
Figure 9(c) is a particular case of a style of representation that may be used for all Reed-Muller
(RM) codes, in which all state variables are binary and all constraint codes are simple (3; 2; 2)
parity-check constraints or (3; 1; 3) repetition constraints. It is believed (but not proved) that
this representation is the most eÆcient possible representation for the (8; 4; 4) code of this type.
It is easy to see how the graphs of Figures 9(a) and 9(b) (as well as a minimal four-section,
four-state trellis) may be obtained by agglomerating subgraphs of Figure 9(c).
11.3. GRAPHS OF GENERAL LINEAR BEHAVIORAL REPRESENTATIONS 159
11.3.3 Block diagrams as directed normal graphs

Conventional block diagrams may often be regarded as normal graphs, with the vertices
(\blocks") representing constraints, and the edges labeled by internal or external variables.
There are various graphical conventions for tying edges together to make a hyperedge.
However, one di erence is that the blocks usually represent input-output (causal) relationships,
and the block diagram is usually a directed graph in which the edges are also labelled with arrows,
indicating the direction of causality. In this respect block diagrams resemble Bayesian networks,
which are directed graphs representing probabilistic cause-and-e ect models.
This style of graphical model can sometimes be superimposed on a normal graph, as follows. If a
constraint code is a linear (n; k) code and has an information set of size k, then the corresponding
k symbols may be regarded as \inputs" to the constraint, and the remaining n k symbols as
\outputs" determined by the inputs. Arrows may be drawn on the edges to represent such
input-output relationships. If arrows can be drawn consistently on all edges in this way, then a
normal graph may be converted to a directed normal graph (block diagram).
For example, Figure 10 shows how the parity-check representation for the (8; 4; 4) code, shown
in Figure 3, may be converted to directed normal graph form. This could be useful if, for
example, we wanted to use such a graph to implement an encoder. However, this example is
somewhat misleading, as parity-check representations cannot always be converted to encoders
in this way.
-=H
H
- = XHXHXHHH
HHXHXXHXH
- = HH HHHXHXHj-Xz +
HH  HH

 = 9XXX HH -Hj: +
XXXHXHH
- = XXXX XX-XzHj +

XX
 = 9 XXXXz-: +
 

 = = 

 = 
Figure 10. Encoder from parity-check normal graph for (8; 4; 4) code.

11.3.4 Markov graphs

In Markov graphs (Markov random elds), a choice is made opposite to that made for normal
graphs: variables are represented by vertices, and constraints are represented by edges (if the
constraint has degree 2), or by hyperedges (if the constraint has degree greater than 2). More-
over, a hyperedge is usually represented by a clique, namely a set of ordinary edges between
every pair of variables incident on the hyperedge. (This style of graph representation has the
disadvantage that it sometimes generates inadvertent cliques.)
160 CHAPTER 11. CODES ON GRAPHS

Markov graphs are particularly nice when the degrees of all constraints are 2 or less. Such a
realization is called a pairwise Markov graph. We may then represent constraints by ordinary
edges. Pairwise constraints often arise naturally in physical models.
Any generic realization, as illustrated in Figure 5, may be transformed into a pairwise Markov
realization by the simple conversion that is shown graphically in Figure 11. Here each constraint
code has been replaced by a state \supervariable" whose alphabet is the set of all codewords
in the constraint code. Each edge then represents the constraint that the associated ordinary
variable must be equal to the corresponding component of the supervariable.
~H\ H
~HX\XH\HXHXHHH
...
H\H\XHXXXHXHHX n
\\HHHH
~HH \H n

nXXXHXHHHH\\\ . . .
X
n XXHXHX\XH\ n
...  

n
Figure 11. Pairwise Markov graph of a pairwise MRF equivalent to Figure 5.
For example, if the constraint code has degree 3 and constrains three incident variables
(x1 ; x2 ; x3 ) to satisfy the parity check x1 + x2 + x3 = 0, then the constraint code has four
codewords: f000; 110; 101; 011g. We then de ne a supervariable x123 to have these codewords
as its alphabet, and constrain x1 to equal the rst component of x123 , etc.
Pairwise Markov graphs also have some nice properties that are exploited in the statistical
physics and statistical inference literature. Unfortunately, a nontrivial realization cannot be
both normal and pairwise Markov, because if the degrees of all variables and constraints are 2
or less, then the associated graph can be only a collection of disconnected simple chains or loops.

11.4 Graph-theoretic properties of realizations

A graph illustrates dependency relationships. We now develop some elementary but important
connections between graph properties and dependencies.

11.4.1 Connectedness and independence

Suppose that a code C is the Cartesian product of two codes, C = C1  C2 . In other words, C
consists of the pairs of codewords (c1 ; c2 ) such that c1 2 C1 ; c2 2 C2 . Then a realization of C
may be constructed from independent realizations of C1 and C2 . A graph of such a realization
is a disconnected graph, with two component subgraphs representing C1 and C2 , respectively.
11.4. GRAPH-THEORETIC PROPERTIES OF REALIZATIONS 161
Conversely, if a graph of a realization of C is disconnected, then C is evidently the Cartesian
product of the codes realized by the component subgraphs. In short, a code C has a realization
whose graph is disconnected if and only if C is the Cartesian product of shorter codes. Thus
disconnectedness is a graph-theoretic expression of independence.

11.4.2 Cut sets and conditional independence

A cut set of a connected graph is a minimal set of edges such that removal of the set partitions
the graph into two disconnected subgraphs.
Notice that a connected graph is cycle-free if and only if every edge is by itself a cut set.
In a normal graph, a cut set consists of a set of ordinary (state) edges, and may be speci ed
by the corresponding subset   J of the state index set J . If the cut set consists of a single
edge, then the cut set may be identi ed with the corresponding state variable Sj . If the cut set
consists of several edges, then it may be identi ed with the set of all corresponding state spaces
Sj ; j 2Q. We will regard the Cartesian product of all these state spaces
Q as a superstate variable
S = j 2 Sj . Note that the size of the alphabet of S is jS j = j 2 jSj j, the product of the
sizes of its component state spaces.
Figure 12 gives a high-level view of a realization with a cut set . Since removal of a cut set
partitions a graph into two disconnected subgraphs, it follows that the symbol variables, the
constraint codes, and the states not in  are partitioned by the cut set into two disjoint subsets
connected only by the states in .
XjP XjF

S
CP CF

Figure 12. High-level view of a realization with a cut set .

We label these two components arbitrarily as the \past" P and the \future" F relative to the
cut set . The subsets of symbol variables associated with the past and future components are
denoted by XjP and XjF , respectively. The constraints and internal variables in the past and
future components are aggregated into constraints CP and CF that jointly constrain S , the
common superstate variable, and XjP and XjF , respectively.
Figure 12 makes it clear that a cut set in the graph corresponds to a certain conditional
independence (Markov) property: given that the state variables S = fSj ; j 2 g in the cut set
have a certain set of values s = fsj ; j 2 g, the possible values of XjF depend only on s and
not otherwise on the \past" P , and vice versa. In other words, the superstate variable value s
is a suÆcient statistic for the past with respect to the set of possible futures, and vice versa.
More concretely, let XjP (s ) and XjF (s ) denote the sets of possible past and future symbol
values that are consistent with a given superstate value s , in view of the constraints CP and CF .
Then the set of possible codewords consistent with s is the Cartesian product XjP (s )XjF (s ).
(In e ect, xing the value of the superstate removes the corresponding edge and disconnects the
graph.) The set C of all possible codewords is then the union of such Cartesian products over
all superstates: [
C= XjP (s )  XjF (s ): (11.6)
s 2S
162 CHAPTER 11. CODES ON GRAPHS

11.4.3 The cut-set b ound

Observe that (11.6) is a generic expression for a code C generated by a two-section trellis with
central state space S , and that Figure 12 is a generic normal graph for such a two-section trellis.
This observation leads directly to a lower bound on the size of S :
Theorem 11.1 (Cut-set bound) Given a graphical representation of a code
Q C and a cut set
, the size jS j = j 2 jSj j of the alphabet of the superstate variable S = fSj ; j 2 g is lower-
bounded by the minimal state space size in a conventional trellis in which the symbol variables
are divided into \past" and \future" in the same way.

If C is linear, then this lower bound on jS j is given by the state space theorem for linear codes.
For example, by the cut-set bound and the Muder bound, given any graphical representation
of the (8; 4; 4) binary code and any cut set that partitions the code symbols into two subsets of
size 4, the size of the alphabet of the superstate variable S must be at least 4. We see that for
the obvious cut sets in each of the graphs of Figure 9, this lower bound is satis ed with equality.
We can draw some important general conclusions from the cut-set bound.
First, consider cycle-free graphical representations. In a cycle-free realization, every edge (state
variable) is a cut set, and therefore the size of every state space is lowerbounded by the minimal
size of a state space in some conventional trellis in which the symbol variables are partitioned into
\past" and \future" in the same way. Therefore we cannot expect any great reduction in state
space sizes due to using general cycle-free graphs rather than conventional trellis representations.
On the other hand, signi cant reductions in state space size are possible if we use graphs with
cycles. Then cut sets will generally correspond to multiple state variables, and the complexity
mandated by the cut-set lower bound may be spread out across these multiple state spaces.
For example, consider tail-biting trellis representations. A tail-biting trellis, illustrated in
Figure 13 (also in Figure 9(b)), is a trellis in which the topology of the time axis is that of a
circle rather than an interval.

Figure 13. Normal graph of a tail-biting trellis representation.

In a tail-biting trellis representations, all cut sets involve two state variables. Therefore the
minimum complexity mandated by the cut-set bound may be spread out over two state spaces,
each of which may be as small as the square root of the cut-set lower bound.
For example, Figures 9(a) and 9(b) indicate how the state complexity (maximum state space
size) can be reduced from 4 to 2 by going from a conventional two-section trellis representation of
the (8; 4; 4) code to a tail-biting representation. This is not terribly impressive. However, it has
been shown similarly that the state complexity of a 12-section representation of the (24; 12; 8)
Golay code can be reduced from 256 to 16 by going to a tail-biting representation. Larger gains
are possible with larger codes.
We conclude that it may be possible to construct graphical representations of large, complex
codes without state spaces growing too large, if and only if we use graphs with cycles.