Figures
Uploaded by
Erica RosellaFigures
Uploaded by
Erica RosellaFigures from Information Theory, Inference, and Learning Algorithms, by David J.C.
MacKay, provided
for the use of teachers.
http://www.inference.phy.cam.ac.uk/mackay/itila/
0.3
0.25
0.2
0.15
0.1
0.05
0
0 1 2 3 4 5 6 7 8 9 10
r
0.12
0.1
0.08
0.06
0.04
0.02
0
0 5 10 15 20 25
r
1
H2 (x)
0.8
0.6
0.4
0.2
0
0 0.2 0.4 0.6 0.8 1 x
0 -0 P (y = 0 | x = 0) = 1 − f ; P (y = 0 | x = 1) = f ;
x @
R
@
y
1 -1 P (y = 1 | x = 0) = f ; P (y = 1 | x = 1) = 1 − f.
(1 − f )
0 -0
@
@f
@R
@
1 -1
(1 − f )
Source
6
s ŝ
?
Encoder Decoder
6
t r
- Noisy
channel
s 0 0 1 0 1 1 0
z}|{ z}|{ z}|{ z}|{ z}|{ z}|{ z}|{
t 000 000 111 000 111 111 000
n 000 001 000 000 101 000 000
r 000 001 111 000 010 111 000
s 0 0 1 0 1 1 0
z}|{ z}|{ z}|{ z}|{ z}|{ z}|{ z}|{
t 000 000 111 000 111 111 000
n 000 001 000 000 101 000 000
r 0 00
|{z} 0 01
|{z} 1 11
|{z} 0 00
|{z} 0 10
|{z} 1 11
|{z} 0 00
|{z}
ŝ 0 0 1 0 0 1 0
corrected errors ?
undetected errors ?
s encoder t channel r decoder ŝ
f = 10%
- - -
0.1
R5 R1
0.1 R1 0.01 R3
0.08
1e-05 more useful codes
pb
0.06
0.04 1e-10
R3
0.02
R5 more useful codes
R61 R61
0 1e-15
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Rate Rate
t5 1
s1 s2 1 0
s3 0
t7 s4 t 1 0
6 0
(a) (b)
r5
r1 r2
r3
r7 r4 r
6
(a)
1 0* 1
1 1* 1 0 1 0
0 0 1*
1 0 0 1 0 0 1 0
0
(b) (c) (d)
1 1
1 0 - 1 1
1* 1*
0* 0 0 0* 0 0
0
(e) (e )
0.1
R5 R1
0.1 R1 0.01 H(7,4)
0.08
1e-05 more useful codes
H(7,4) pb
0.06 BCH(511,76)
0.04 BCH(31,16) 1e-10
R3
0.02 BCH(15,7)
BCH(1023,101)
R5 more useful codes
0 1e-15
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Rate Rate
0.1
R5 R1
0.1 R1 0.01
0.08
1e-05
H(7,4) pb
0.06
0.04 1e-10
R3 achievable not achievable
0.02
R5
achievable not achievable
0 1e-15
C C
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Rate Rate
x
a
b
c
d
e
f
g
h
i
j
k
l
m
n
o
p
q
r
s
t
u
v
w
x
y
z
–
abcdefghijklmnopqrstuvwxyz– y
x x
a a
b b
c c
d d
e e
f
g f
g
h h
i i
j j
k k
l l
m m
n n
o o
p p
q q
r r
s s
t t
u u
v v
w w
x x
y y
z z
– –
abcdefghijklmnopqrstuvwxyz– y abcdefghijklmnopqrstuvwxyz– y
(a) P (y | x) (b) P (x | y)
i ai pi
1 a 0.0575 a
2 b 0.0128 b
3 c 0.0263 c
4 d 0.0285 d
5 e 0.0913 e
6 f 0.0173 f
7 g 0.0133 g
8 h 0.0313 h
9 i 0.0599 i
10 j 0.0006 j
11 k 0.0084 k
12 l 0.0335 l
13 m 0.0235 m
14 n 0.0596 n
15 o 0.0689 o
16 p 0.0192 p
17 q 0.0008 q
18 r 0.0508 r
19 s 0.0567 s
20 t 0.0706 t
21 u 0.0334 u
22 v 0.0069 v
23 w 0.0119 w
24 x 0.0073 x
25 y 0.0164 y
26 z 0.0007 z
27 – 0.1928 –
u
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 10 nB
0.3
0.25
0.2
0.15
0.1
0.05
0
0 1 2 3 4 5 6 7 8 9 10
u
u P (u | nB = 3, N )
0 0
1 0.063
2 0.22
3 0.29
4 0.24
5 0.13
6 0.047
7 0.0099
8 0.00086
9 0.0000096
10 0
A B
x2 e−x log x1 x log x
-1 0 1 2 3 -1 0 1 2 3 0 1 2 3 0 1 2 3
λf (x1 ) + (1 − λ)f (x2 )
f (x∗ )
x1 x2
∗
x = λx1 + (1 − λ)x2
0.5
P (fˆ)
0.4
0.3
0.2
0.1
0 0.2 0.4 0.6 0.8 1
fˆ
0.15
0.1
0.05
0
0 5 10 15 20
ewpage
0.25
P(x|lambda=2)
P(x|lambda=5)
0.2 P(x|lambda=10)
0.15
0.1
0.05
0
2 4 6 8 10 12 14 16 18 20 x
0.2
P(x=3|lambda)
P(x=5|lambda)
P(x=12|lambda)
0.15
0.1
0.05
0
1 10 100 λ
1.4e-06
1.2e-06
1e-06
8e-07
6e-07
4e-07
2e-07
0
1 10 100
3
100
10
1
x 1.5
1
λ
2
2.5
H0 is true H1 is true
pa = 1/6 pa = 0.25 pa = 0.5
8 8 8
1000/1 1000/1 1000/1
6 6 6
4 100/1 4 100/1 4 100/1
2 10/1 2 10/1 2 10/1
0 1/1 0 1/1 0 1/1
-2 1/10 -2 1/10 -2 1/10
-4 1/100 -4 1/100 -4 1/100
0 50 100 150 200 0 50 100 150 200 0 50 100 150 200
8 8 8
1000/1 1000/1 1000/1
6 6 6
4 100/1 4 100/1 4 100/1
2 10/1 2 10/1 2 10/1
0 1/1 0 1/1 0 1/1
-2 1/10 -2 1/10 -2 1/10
-4 1/100 -4 1/100 -4 1/100
0 50 100 150 200 0 50 100 150 200 0 50 100 150 200
8 8 8
1000/1 1000/1 1000/1
6 6 6
4 100/1 4 100/1 4 100/1
2 10/1 2 10/1 2 10/1
0 1/1 0 1/1 0 1/1
-2 1/10 -2 1/10 -2 1/10
-4 1/100 -4 1/100 -4 1/100
0 50 100 150 200 0 50 100 150 200 0 50 100 150 200
(a) 0 0.2 0.4 0.6 0.8 1 (b) 0 0.2 0.4 0.6 0.8 1
P (pa | s = aba, F = 3) ∝ p2a (1 − pa ) P (pa | s = bbb, F = 3) ∝ (1 − pa )3
H0 is true H1 is true
pa = 1/6 pa = 0.25 pa = 0.5
8 8 8
1000/1 1000/1 1000/1
6 6 6
4 100/1 4 100/1 4 100/1
2 10/1 2 10/1 2 10/1
0 1/1 0 1/1 0 1/1
-2 1/10 -2 1/10 -2 1/10
-4 1/100 -4 1/100 -4 1/100
0 50 100 150 200 0 50 100 150 200 0 50 100 150 200
Where the prize is
door door door
1 2 3
pnone pnone pnone
none
3 3 3
Which doors opened by earthquake
2
p3 p3 p3
3
3 3 3
1,2
1,3
2,3
p1,2,3 p1,2,3 p1,2,3
1,2,3
3 3 3
0.05 H0
H1
0.04
0.03
0.02 140
0.01
0
0 50 100 150 200 250
P (D|H1 , α)
α
P (D|H0 )
.37 .25
1.0 .48
2.7 .82
7.4 1.3
20 1.8
55 1.9
148 1.7
403 1.3
1096 1.1
P (D0 |H1 , α)
α
P (D0 |H0 )
.37 .32
1.0 .61
2.7 1.0
7.4 1.6
20 2.2
55 2.3
148 1.9
403 1.4
1096 1.2
1 1
10 h(p) = log2 p h(p) H2 (p) H2 (p)
p 0.8
8
0.001 10.0 0.011
0.6
6 0.01 6.6 0.081
4 0.1 3.3 0.47 0.4
2 0.2 2.3 0.72 0.2
0
0.5 1.0 1.0 0
0 0.2 0.4 0.6 0.8 1 p 0 0.2 0.4 0.6 0.8 1 p
1+
1
1+ 2+ 5− - 2+
1+
2 @
2+ R
@ 5−
3+ weigh
3+
4+ 126 3
- 3+ 4+ 6− - 4+
1+ 5−
345 A 4 @
R
@ 6−
2+ 6− A
3+ 7− A 7−
4+ 8− A 1
U
A 7− 8− - 8−
5+ 7 @
6+ R
@ ?
7+ 4−
8+ 3
6+ 3− 4− - 3−
9+ 1−
4 @
10+ 2− R
@ 6+
11+ weigh 3− weigh
2−
12+ 1234 4− 126 1
- - 1− 2− 5+ - 1−
1− 5678 B 5+ 345 A 2 @
2− B 6+ A R
@ 5+
3− B 7+ A 7+
4− B 8+ A 7
B U
A 7+ 8+ - 8+
5− B 1 @
6− R
@ ?
B
7− B 9+
8− B 9
B + 9+ 10+ 11+ - 10+
9− 9
10 @
B R
@ 11+
10− B 10+
11− 11+ weigh
B 10−
12− BNB 12+ 9 10 11 9
- 9− 10− 11− - 9−
9− 123 A 10 @
10− A R
@ 11−
11− A 12+
12− A 12
U
A 12+ 12− - 12−
1 @
R
@ ?
×j
A × ××× × ××× ×× ×× ××× ×
B × ××× × × ××× ×× × ×× ××× ×
C × ××× × ××× ×× ×× ××× ×
×j
D ××× × ××× ×× ×× ××× ×
×j
E ××× × ××× ×× ×× ××× ×
×j
F × ××× ××× ×× ×× ××× ×
Sj
G × × × ××× × ××× ×× ×× ××× ×
H × ××× ××× ×× × ××× ×
1 2 3 4 5 6 7 8
move # 1 2 32 48 49
question G3 B1 E5 F3 H3
outcome x=n x=n x=n x=n x=y
63 62 32 16 1
P (x)
64 63 33 17 16
h(x) 0.0227 0.0230 0.0443 0.0874 4.0
Total info. 0.0227 0.0458 1.0 2.0 6.0
1 aaail
2 aaaiu
3 aaald
..
.
129 abati
..
.
2047 azpan
2048 aztdn
..
.
..
.
16 384 odrcr
..
.
..
.
32 737 zatnt
..
.
32 768 zxast
−6 −4 −2.4 −2 log2 P (x)
-
S0 S 1
16
6 6 6
e,f,g,h d a,b,c
(a)
3 {a,b,c,d,e,f,g,h}
{a,b,c,d,e,f,g}
2.5 {a,b,c,d,e,f}
{a,b,c,d,e}
Hδ (X)
2 {a,b,c,d}
1.5 {a,b,c}
1 {a,b}
0.5
{a}
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
(b) δ
log2 P (x)
−14 −12 −10 −8 −6 −4 −2 0
-
S0.01 S0.1
6 6 6 6 6
1111 1101, 1011, . . . 0110, 1010, . . . 0010, 0001, . . . 0000
(a) (b)
4
N=4
3.5
Hδ (X 4 ) 3
2.5
1.5
0.5
0
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 δ
10
N=10
8
Hδ (X 10 )
6
0
0 0.2 0.4 0.6 0.8 1 δ
1
N=10
N=210
0.8 N=410
1 N
N Hδ (X )
N=610
N=810
N=1010
0.6
0.4
0.2
0
0 0.2 0.4 0.6 0.8 1 δ
x log2 (P (x))
...1...................1.....1....1.1.......1........1...........1.....................1.......11... −50.1
......................1.....1.....1.......1....1.........1.....................................1.... −37.3
........1....1..1...1....11..1.1.........11.........................1...1.1..1...1................1. −65.9
1.1...1................1.......................11.1..1............................1.....1..1.11..... −56.4
...11...........1...1.....1.1......1..........1....1...1.....1............1......................... −53.2
..............1......1.........1.1.......1..........1............1...1......................1....... −43.7
.....1........1.......1...1............1............1...........1......1..11........................ −46.8
.....1..1..1...............111...................1...............1.........1.1...1...1.............1 −56.4
.........1..........1.....1......1..........1....1..............................................1... −37.3
......1........................1..............1.....1..1.1.1..1...................................1. −43.7
1.......................1..........1...1...................1....1....1........1..11..1.1...1........ −56.4
...........11.1.........1................1......1.....................1............................. −37.3
.1..........1...1.1.............1.......11...........1.1...1..............1.............11.......... −56.4
......1...1..1.....1..11.1.1.1...1.....................1............1.............1..1.............. −59.5
............11.1......1....1..1............................1.......1..............1.......1......... −46.8
.................................................................................................... −15.2
1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 −332.1
log2 P (x)
−N H(X)
-
TN β
6 6 6 6 6
1111111111110. . . 11111110111
0000100000010. . . 00001000010
0100000001000. . . 00010000000
0001000000000. . . 00000000000
0000000000000. . . 00000000000
1
N
Hδ (X N )
H0 (X)
H +
H
H −
0 1 δ
1
N=1
N=2
N =1 N =2
N=1000
1 1
0.8 δ N Hδ (X) 2Hδ (X) δ N Hδ (X) 2Hδ (X)
0–0.2 1 2 0–0.04 1 4
0.6
0.2–1 0 1 0.04–0.2 0.79 3
0.2–0.36 0.5 2
0.4
0.36–1 0 1
0.2
0
0 0.2 0.4 0.6 0.8 1
50
40
30
20
10
0
0 0.2 0.4 0.6 0.8 1 1.2 1.4
0
0 0.2 0.4 0.6 0.8 1 1.2 1.4
0.5
0.4
0.3
0.2
0.1
0
0 0.2
0000
000
0001
00
The total symbol code budget
0010
001
0011
0
0100
010
0101
01
0110
011
0111
1000
100
1001
10
1010
101
1011
1
1100
110
1101
11
1110
111
1111
C0
C3
C4
C6
0000
0000 0000 0000
000 0001
000 000 000
00
0001 0001 0001
00
00
00
0010 0010 0010 0010
001 001 001 001
0011
0011 0011 0011
0
0100
0
0 0
0100 0100 0100
010
010 010 010
0101
0101
01 0101 01 0101
01 01
0110 0110 0110 0110
011
011 011 011
0111
0111 0111 0111
1000 1000
1000 1000
100
100
10 100 100
1001 1001 1001 1001
10
10 10
1010
1010 1010 1010
101
1011
101
1011 101
1011
101
1011
1 1
1 1
1100
110
1100 1100 1100
110
1101
1101
11
110
1101
110
1101
11 11
11
1110
111
1110
1110 1110
111
1111
1111 111
1111
111
1111
1
ai pi log2 pi li c(ai )
a
a 0.0575 4.1 4 0000 n
b 0.0128 6.3 6 001000 b
c 0.0263 5.2 5 00101 g
d 0.0285 5.1 5 10000 c
e 0.0913 3.5 4 1100 s
f 0.0173 5.9 6 111000 −
g 0.0133 6.2 6 001001 d
h 0.0313 5.0 5 10001 h
i 0.0599 4.1 4 1001 i
j 0.0006 10.7 10 1101000000 k
k 0.0084 6.9 7 1010000 x
l 0.0335 4.9 5 11101 y
m 0.0235 5.4 6 110101 u
n 0.0596 4.1 4 0001 o
o 0.0689 3.9 4 1011 e
p 0.0192 5.7 6 111001 j
q 0.0008 10.3 9 110100001 z
r 0.0508 4.3 5 11011
q
s 0.0567 4.1 4 0011
v
w
t 0.0706 3.8 4 1111
m
u 0.0334 4.9 5 10101
v 0.0069 7.2 8 11010001 r
f
w 0.0119 6.4 7 1101001 p
x 0.0073 7.1 7 1010001 l
y 0.0164 5.9 6 101001 t
z 0.0007 10.4 10 1101000001
– 0.1928 2.4 2 01
x P (x)
a 0.0575
b 0.0128
c 0.0263
d 0.0285
e 0.0913
f 0.0173
g 0.0133
h 0.0313
i 0.0599
j 0.0006
k 0.0084
l 0.0335
m 0.0235
n 0.0596
o 0.0689
p 0.0192
q 0.0008
r 0.0508
s 0.0567
t 0.0706
u 0.0334
v 0.0069
w 0.0119
x 0.0073
y 0.0164
z 0.0007
− 0.1928
0000
000
0001
00
The total symbol code budget
0010
001
0011
0
0100
010
0101
01
0110
011
0111
1000
100
1001
10
1010
101
1011
1
1100
110
1101
11
1110
111
1111
symbol probability Huffman Rival code’s Modified rival
codewords codewords code
a pa cH (a) cR (a) cR (c)
b pb cH (b) cR (b) cR (b)
c pc cH (c) cR (c) cR (a)
0.00
6
0.25 0
601
? ? 01101
0.50
6
0.75 1
?
1.00
0.00
?a1
6
P (x1 = a1 ) a2 a1
6
a2
a2 a5
?
P (x1 = a1 ) + P (x1 = a2 )
.. ..
. .
P (x1 = a1 ) + . . . + P (x1 = aI−1 )
6aI
?
1.0
00000
0000
00001
000
00010
0001
00011
00
00100
0010
00101
001
a 00110
0011
00111
0
01000
0100
01001
010
01010
0101
01011
01
01100
0110
01101
011
01110
0111
ba 01111 10010111
10000 10011000
10001
1000 bbbaa 10011001
bba 100 10011010
10010
1001 10011011
bbba 10011 bbba bbbab 10011100 10011
b 10
10100 B 10011101
bb bbb bbbb 10101
1010 B 10011110
10110
101 B bbba2
OCC 10011111
1011 B
bbb2 10111
1 C10100000
bb2 11000
1100 C
11001 100111101
b2 110
11010
1101
11011
11
11100
1110
2 11101
111
11110
1111
11111
00000
0000
aaaa 00001
000
aaa 00010
0001
aa aaab 00011
00
00100
aaba 0010
aab 00101
aabb 001
a 00110
aa2 0011
00111
abaa 0
aba abab 01000
0100
ab abba 01001
abb abbb 010
01010
ab2 0101
01011
01
a2 01100
0110
01101
baaa 011
baa baab 01110
0111
ba baba 01111
bab babb 10000
ba2 1000
bbaa 10001
bba 100
bbab 10010
1001
bbba 10011
10
b 10100
bb 1010
bbb bbbb 10101
101
10110
1011
10111
1
bb2 11000
1100
11001
b2 110
11010
1101
11011
11
11100
1110
2 11101
111
11110
1111
11111
probability density
is maximized here
√
Nσ
almost all
probability mass is here
Arabic Roman
1 2 3 M X
4 5 6 C I 2
7 8 9
0 2
ewpage
Source string’s interval Binary intervals
P (x|H) ?
6
?
?
H(X, Y )
H(X)
H(Y )
H(X | Y ) I(X; Y ) H(Y |X)
H(Y)
H(X|Y) I(X;Y) H(Y|X) H(X,Y)
H(X)
I(X;Y) A I(X;Y|Z)
H(Y)
H(X,Y|Z)
H(Y|X,Z)
H(X|Y,Z)
H(X)
H(Z|X) H(Z|Y) H(Z|X,Y)
H(Z)
H(X, Y )
H(X)
H(Y )
H(X | Y ) I(X; Y ) H(Y |X)
I(X; Y )
0.4
0.3
0.2
0.1
0
0 0.25 0.5 0.75 1
p1
I(X; Y )
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0 0.25 0.5 0.75 1
p1
ABCDEFGHIJKLMNOPQRSTUVWXYZ-
1 A
B
PP- B
A
B
q
PC C
D
E
D
1
F
G
E
PP- E
H
I
q
PF
J
K
L
M
1 G N
H
PP- H
O
P
Q
.. q
PI R
S
T
U
. V
W
1
Y X
- Z
Y
ZP
Z
Pq
-
P-
pBM 6
achievable
-
C R
0000
1000
0100
1100
0010
1010
0110
1110
0001
1001
0101
1101
0011
1011
0111
1111
0000
1000
0100
1100
0010
1010
0110
1110
0001
1001
00
10
01
11
0101
1101
0 1 00 0011
10 1011
0 01 0111
1 11 1111
N =1 N =2 N =4
0000
1000
0100
1100
0010
1010
0110
1110
0001
1001
0101
1101
0011
1011
0111
1111
0000
1000
0100
1100
0010
1010
0110
1110
0001
1001
00
10
01
11
0101
1101
0 1 00 0011
10 1011
0 01 0111
1 11 1111
N =1 N =2 N =4
AN
Y AN
Y
' $ ' $
Typical y Typical y
& % & %
6
Typical y for a given typical x
(a) (b)
x(1) x(2) x(1) x(2)
00
10
01
11
00
10
01
11
00
10
01
11
00 00 00 - m̂ = 1
?0 ?0 ?0 - m̂ = 1
10 10 10
0? 0? 0? - m̂ = 1
- m̂ = 0
-
0 1 ?? ?? ??
1? 1? 1? m̂ = 2
-
0 01 01 01
m̂ = 2
-
? ?1 ?1 ?1
1 11 11 11 m̂ = 2
Q (a) (b) (c)
N =1 N =2
1
Z
0.9 BSC
BEC
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
pb 6
R(pb )
1 2
3
-
C R
AN
X
-
-
6 qq
6qq qq q 2N H(X)
qqq
qq q
qq q
q q q
q q q
q q q
q q q
q q q
q q q
q q q
q q q 2N H(X,Y ) dots
q q q
q q q
q q q
q q q
q qq
q qq
qq6q
qqq 6
AN q q q 2N H(Y |X)
Y qqq
qqq?
?
qqq
qqq
q-qq
qqq
qqq
2N H(Y ) qqq
qqq
qqq
qqq
qqq
qqq
- qqq
qqq
2N H(X|Y ) qq qq qq
qqq
qqq
qq
? qq
?
x(3) x(1) x(2) x(4) x(3) x(1) x(2) x(4)
qqq qqq q qq
qqq qqq qq
q qq qq q - ŝ(ya ) = 0
qq qqq qqq q
ya
q qq qq q
q qq qq q q qq qq q - ŝ(yb ) = 3
q qq qq q yb q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q
q qq qq q q qq qq q - ŝ(yd ) = 0
q qq qq q yd q qq qq q
q qq qq q q qq qq q
q qq qq q yc q qq qq q - ŝ(yc ) = 4
q qq qq q qq qq
qq qq
(a) (b)
⇒
(a) A random code . . . (b) expurgated
pb 6
achievable
-
R
C
0.3
0.25 Optimum
Simple
0.2
pb
0.15
0.1
0.05
0
0 0.5 1 1.5 2 2.5
R
Er (R)
C
R
1
0.5 p1
1
0.5
0 0
0 0.5 p0 1
5.5
5
4.5
4
3.5
3
2.5
2
1.5
1
(a) 0.5
1 10 100 1000
2.5
1.5
0.5
(b) 0
1 2 3 4 5 6 7 8 9 10
φ1 (t)
φ2 (t)
φ3 (t)
x(t)
x0
x1
y
w
(a)
g
-
(b)
..
.
1.4
1.2
1
capacity
0.8
0.6
0.4
0.2
0
0 1 2 3 4 5 6
bandwidth
1 1 1 1 1 1 1 1 1 1 1 1
0 0 0 ? ? 1 1 0 1 1 1 0 0 0
1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 ? 1 0 1 1 1 1 1 1 1
0 0 0 ? 0 0 1 0 0 0 0 0 0
0 0 0 ? 1 0 0 0 0 0 0 0 0
(a) 1 1 1 (b) (c) 1 1 1 (d) 1 1 1 (e) 1 1 1
1 0 1 1 1 1
1 1 0 (1)(1)(1)
1 1 1 1 1 1
1 0 1 1 1 1
1 0 0 0 0 0
1 0 0 0 0 0
(d ) 1
0 1 1 (e ) 1 1 1
0
- C1 -
-
- π C -
2
1.2
1
0.8
0.6
0.4
0.2
0
0 0.5 1 1.5 2 2.5
0.1
0.01
0.1 1
ewpage
string length N ' 200
number of strings S ' 223
number of possible 2N ' 2200
strings
string length N ' 200
number of strings S ' 223
size of hash function M ' 30 bits
size of hash table T = 2M
' 230
ewpage
Hash
function Hash table
Strings - hashes
M bits
-
6
h(x(2) ) → 2
N bits
-
6 x(1)
@
@
(2)
x
x(3) R
@
h(x(1) ) → 1
@
..
.
@
S R
@
h(x(3) ) → 3
2M
(s)
x
A
.. A
. A
A
A
? U
A h(x(s) ) → s
?
350
w A(w) 300
250
0 1
5 12 200
8 30 150
9 20 100
10 72 50
11 120 0
12 100 0 5 8 10 15 20 25 30
13 180
14 240
15 272 100
16 345
17 300
18 200 10
19 120
20 36
1
Total 2048
0 5 8 10 15 20 25 30
w A(w)
0 1
3 7
4 7
7 1
Total 16
8
7
6
5
4
3
2
1
0
0 1 2 3 4 5 6 7
t
t
t
...
1 2
t t
... ...
1 2 1 2
t t
... ...
1 2 1 2
00000 0 00000 0 00000 0 00000 0 00000
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 100000 0 0000
00000 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0000
uN vN wN xN
N
N
1 0 1 0 1 0 1 0 0 1 0 0 1 1 0 1 0 0 0 1 0 1 1 0
0 0 1 1 1 0 1 1 1 1 0 0 0 1 1 0 0 1 1 0 1 0 0 0
1 0 1 1 1 0 1 1 1 0 0 1 0 1 1 0 0 0 1 1 0 1 0 0
0 0 0 0 1 0 1 1 1 1 0 0 1 0 1 1 0 1 0 0 1 0 0 0
0 0 0 0 0 0 1 1 0 0 1 1 1 1 0 1 0 0 0 0 0 1 0 0
1 1 0 0 1 0 0 0 1 1 1 1 1 0 0 0 0 0 1 0 1 1 1 0 M
1 0 1 1 1 1 1 0 0 0 1 0 1 0 0 0 0 1 0 0 1 1 1 0
1 1 0 0 1 0 1 1 0 0 0 1 1 0 1 0 1 1 1 0 1 0 1 0
1 0 0 0 1 1 1 0 0 1 0 1 0 0 0 0 1 0 1 1 1 1 0 1
0 1 0 0 0 1 0 0 0 0 1 0 1 0 1 0 0 1 1 0 1 0 1 0
010111110111111110111010
101110101001001101000011
6e+52
5e+52
4e+52
3e+52
2e+52
1e+52
0
0 100 200 300 400 500
1e+60
1e+40
1e+20
1
1e-20
1e-40
1e-60
1e-80
1e-100
1e-120
0 100 200 300 400 500
1
Capacity
R_GV
0.5
0
0 0.25 0.5
f
t
...
1 2
1
0.8
R
0.6
0.4
0.2
0
0 2 4 6 8 10 12
C
1e+25
1e+20
1e+15
1e+10
100000
1
0 2 4 6 8 10 12
C
1
pb 0.01
21 N=3
315
0.0001
61525
1e-06
1e-08
1e-10
1e-12
10^13
1e-14
0 0.2 0.4 0.6 0.8 1
R
1
d=10
d=20
1e-05 d=30
d=40
1e-10 d=50
d=60
1e-15
1e-20
0.0001 0.001 0.01 0.1
RB
(A) (B)
H(X ,X )
encode
x (A) - t(A)
RA H H(X (B) ) Achievable
H
j
H
*
C
encode (B)
(a) x(B) -t
RB
H(X (B) | X (A) )
(b) RA
H(X (A) | X (B) ) H(X (A) )
x(A) -
P (y|x(A) , x(B) ) - y
x (B) -
(a)
RB
1
y: x(A)
0 1 1/2 Achievable
0 0 1
x(B)
(b) 1 1 2 (c)
1/2 1 RA
x(A) - - y (B)
(A) (B) (A) (B)
P (y ,y |x ,x )
y (A) x(B)
(a)
y (A) : x(A) y (B) : x(A)
0 1 0 1
0 0 0 0 0 1
x(B) x(B)
(b) 1 1 0 1 0 0
0.8
0.6 Achievable
R(B)
0.4
0.2
0
(c) 0 0.2 0.4 0.6 0.8 1
R(A)
(A)
y
*
x
Hj
H (B)
y
RB
6
C (B)
@
@
@
@
@ -
C (A) RA
R
CA
CB
fA fB f
R
CA
CB
fA fB f
0.6
0.4
0.2
0
0 0.2 0.4 0.6 0.8 1
1 2 3 4
4 3 2 1
Commander
Commander
Jim
Commander
Jim
A
B
A
(a) (b) B
A
N
M P
B
1 1 1 1
A
1 2 3
2 5
5
B
A
B
1 1 1 1
5 5 3 1
A 1 2 3
2 2 1
2 5
1 1
5
B 1
J H2
2*
Hj
H
1
HH
4*
H 2 M H1
*
H
j
H j
H
AH
KH B
H
j
H 2*
H
j
H *
1 1
IHH N 3
j
H *
1 L 3
(a) 2* J H2H
j
4 1
HH
4 2 M H1
0
* H
j *
Hj
AH 2 KH B
H
j *
H
j *
1 1 1 N 3
H
I
H
j *
1 L 3
6 2
JH
(b) 2
* H
j
4 1
4 Hj
H 2 M H1
H
0
* H
5 * j
AH H
2 K B
H
j H
j *
1 1 * 1 N 3
3
I H
H
j *
1 2 3
L
6
(c) 2* H2
4 1
J
Hj
4* 2* M H1H
0 3 j
H
H H B
A
j 1 2
H *K H j *
1 1 N 3
I H
H
j *
1 2 3
L
(d) 6 2
2*
4 1
J
5 1
4 2 H
0
* H
3
*M H j
H 2 KH
B
A
Hj * H
j *
1 1 1 4 3
I H
H
j N
1 2 3
L
(e) 6 2
2*
4 1
J
5 1
4 2 H
0
* H
3
*M H j 6
HH 2 KH
H
A
j
1 1
* j
1 4 3
B
I H
H
j N
1 2 3
L
2
1+f
H_2(f)
0
0 0.25 0.5 0.75 1
0.7
0.6
0.5
0.4
0.3 R(f) = H_2(f)/(1+f)
0.2
0.1
0
0 0.25 0.5 0.75 1
1 s1 s2 s3 s4 s5 s6 s7 s8
1f 1f 1f 1f 1f 1f 1f f
1 0 1 01 01 01 01 01 01 01 1
@ @ @ @ @ @ @
@@ @@ @@ @@ @@ @@ @@
0f - f R - f R - f R - f R - f R - f R - f R - f
0
(a) 0 (c) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
sn sn+1 (from)
1j 1j 1 0
@0 1 1 0 1
(b) (d) A= (to)
@ 0 1 1
@
j R
@ j
-
0 0 0
sn sn+1 sn sn+1
1 m
11 m
1 - 11 n
11 n
11
11 A
1 11 A
1
1 0 A0 1 A0
A A
1 1m A 1m 1 0 1n A 1n
A @0 A1
A 10 @ A
A @A
m UA R@U
A
0m n 0n
0 0 1 1 0 0 1
1 0 @0 0 @0
@ @
00 @R @
00 @R @
0 00m - m
00 00n n
00
0
1 1 0 0 0 1 0 0
0 0 0 1 0 0 1 1
B A=
1 0 0
C A=
0 1 1 0 0
0 0 1 1 0 0 1 0
M1 = 2 M1 = 2 M2 = 3 M1 = 2 M2 = 3 M3 = 5
1 1 1 1 1 2
1h 1h 1h 1h 1h 1h
@ @ @
@ @ @
h
0 - 0h h
0 - 0h R- 0h
@ h
0 - 0h - 0h
@
R R- 0h
@
1 1 2 1 2 3
1
n Mn Mn /Mn−1 log2 Mn n
log2 Mn
1 2 1.0 1.00
2 3 1.500 1.6 0.79
3 5 1.667 2.3 0.77
4 8 1.600 3.0 0.75
5 13 1.625 3.7 0.74
6 21 1.615 4.4 0.73
7 34 1.619 5.1 0.73
8 55 1.618 5.8 0.72
9 89 1.618 6.5 0.72
10 144 1.618 7.2 0.72
11 233 1.618 7.9 0.71
12 377 1.618 8.6 0.71
100 9×1020 1.618 69.7 0.70
200 7×1041 1.618 139.1 0.70
300 6×1062 1.618 208.5 0.70
400 5×1083 1.618 277.9 0.69
- t
z1 hd z0
6
-⊕ s
z1 hd z0 t
?
-⊕ - s
ewpage
2
0
1
0 1 0 1
0 0 1 1 0
0
1 1 1 0
3
1 0
0 1 0 0 2
0 0 1 0 1
0
0 0 0 1 1
1 0
1 1 1 1
0
0
ewpage
D A T A S C H M O S A S S
U F O S T I E U P I L I A
F A T H E R T I M E S O R B
F R O V E E R E T H E R
M I S S A P P E A S E
S T O O L S S T A I R
T I L T S U N L U C K I L Y
U T A H S T E A L E R A S
D O V E C O T E S C N O T E
R U L E R M A N N E R
G A R G L E R M I R Y
I D I O T C A S T T E A
L I D O B R O T H E R R A T
D E E S A O R T A A E R O
S U R E S T E E P H E L M
B A N G E R B A K E R I E S
V A O R I O L
P A R L I A M E N T C A T S
L L S M E L K O
V A L E N T I N E S E T N A
N O B E T
C A N O E R H A P S O D Y
H E U E
J E N N I F E R S T E P S
E O T X P
D U E T N U T C R A C K E R
S T W O A A U R
P H I L B A T T L E S T A R
E E E E I E T
B R I S T L E S A U S T E N
0.1
to theand
of
I
0.01 is
Harriet
0.001
0.0001 information
probability
1e-05
1 10 100 1000 10000
0.1
the
of a is
x
0.01
probability
information
0.001
Shannon
Bayes
0.0001
0.00001
1 10 100 1000
alpha=1
0.1
alpha=10
0.01
alpha=100
0.001
alpha=1000
0.0001
0.00001 book
1 10 100 1000 10000
0.1
0.01
0.001
0.0001
0.00001
1 10 100 1000 10000
1000
900
800
700
600
500
(a)
0 10 20 30 40 50 60 70 80
1000 1000
sex
900 900
sex
800 800
700 700
no sex
no sex
600 600
500 500
0 200 400 600 800 1000 1200 1400 1600 0 50 100 150 200 250 300 350
(b) (c)
G = 1000 G = 100 000
20 50
45
40 with sex
15
with sex 35
mG 30
10 without sex 25
20
15
5
10 without sex
5
0 0
0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1
f f
(a) mG = 4 (b) mG = 1
1000 1000
900 900
800 800
Fitnesses
700 sexual fitness 700 sexual fitness
parthen fitness parthen fitness
600 600
500 500
0 50 100 150 200 250 0 50 100 150 200 250
100 100
80 80
Percentage
60 60
40 40
20 20
0 0
0 50 100 150 200 250 0 50 100 150 200 250
ewpage
Data:
Assignment Update Assignment Update Assignment Update
Run 1
Run 2
10 10
8 8
6 6
(a) (b)
4 4
2 2
0 0
0 2 4 6 8 10 0 2 4 6 8 10
(a) (b)
Large σ . . .
...
. . . small σ
m1
m2
m2 m1
4
Data density
3
Mean locations
2
1 -2-1 0 1 2
0
-1
-2-1 0 1 2
-2
-3
-4
0 0.5 1 1.5 2 2.5 3 3.5 4
0.8
0.6 Data density
Mean locns.
0.4
0.2 -2 -1 0 1 2
0
-0.2
-0.4 -2 -1 0 1 2
-0.6
-0.8
0 0.5 1 1.5 2
Earthquake Burglar
j j
@
R
@
j jAlarm
Radio @
R
@
j
Phonecall
0.06
1
0.05
0.9
0.04 0.8
0.03 0.7
0.6
0.02 sigma
0.5
0.01 0.4
10 0.3
0.8 0.2
0.6 0.1
sigma 0 0.5 1 1.5 2
0.4
2
0.2 1.5
1 mean
0.5 mean
0
-0.5
-0.5 0 0.5 1 1.5 2 2.5
0.06
1
0.05
0.9
0.04 0.8
0.03 0.7
0.6
0.02 sigma
0.5
0.01 0.4
10 0.3
0.8 0.2
0.6 0.1
sigma 0 0.5 1 1.5 2
0.4
2
(a1) 0.2 1
1.5 (a2) mean
0.5 mean
0
4.5 0.09
sigma=0.2 mu=1
4 sigma=0.4 0.08 mu=1.25
sigma=0.6 mu=1.5
3.5 0.07
3 0.06
Posterior
2.5 0.05
2
0.04
1.5
0.03
1
0.02
0.5
(b) 0
(c) 0.01
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 0
mean 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
t=0 t=1 t=2 t=3 t=9
t=0 t=1 t = 10 t = 20 t = 30 t = 35
t=0 t = 10 t = 20 t = 30
t=0 t = 10 t = 20 t = 26 t = 32
t=0 t=5 t = 10 t = 20
b
(x3 , y3 )Q
QA
QAQ
A Q
A Q
A
A
b A
(x1 , y1 ) A
Ab
(x2 , y2 )
A B C D-G
-30 -20 -10 0 10 20
Scientist xn
A −27.020
B 3.570
C 8.191
D 9.898
E 9.603
F 9.945
G 10.056
4
0
0 1 2 3 4 5
0.3
0.25
0.2
0.15
0.1
0.05
0
0 1 2 3 4 5 6 7 8 9 10
1
0.1
0.01
0.001
0.0001
1e-05
1e-06
0 1 2 3 4 5 6 7 8 9 10
r
0.25
0.2
0.15
0.1
0.05
0
0 5 10 15
1
0.1
0.01
0.001
0.0001
1e-05
1e-06
1e-07
0 5 10 15
r
0.5
0.4
0.3
0.2
0.1
0
-2 0 2 4 6 8
0.1
0.01
0.001
0.0001
-2 0 2 4 6 8
1 0.8
0.9 0.7
0.8
0.6
0.7
0.6 0.5
0.5 0.4
0.4 0.3
0.3
0.2
0.2
0.1 0.1
0 0
0 2 4 6 8 10 -4 -2 0 2 4
0.1 0.1
0.01 0.01
0.001 0.001
0.0001 0.0001
0 2 4 6 8 10 -4 -2 0 2 4
x l = ln x
2.5 0.8
0.7
2 0.6
1.5 0.5
0.4
1 0.3
0.2
0.5
0.1
0 0
0 1 2 3 -4 -2 0 2 4
1
0.1
0.1
0.01
0.01
0.001 0.001
0.0001 0.0001
0 1 2 3 -4 -2 0 2 4
v ln v
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0 1 2 3 4 5
0.1
0.01
0.001
0.0001
0 1 2 3 4 5
u = (20, 10, 7) u = (0.2, 1, 2) u = (0.2, 0.3, 0.15)
8 8 8
4 4 4
0 0 0
-4 -4 -4
-8 -8 -8
-8 -4 0 4 8 -8 -4 0 4 8 -8 -4 0 4 8
5
4
3
2
1
0
0 0.25 0.5 0.75 1
0.6
0.5
0.4
0.3
0.2
0.1
0
-6 -4 -2 0 2 4 6
I = 100
1
0.1
1
10
0.1 100
1000
0.01
0.001
0.0001
1 10 100
I = 1000
1
0.1
1
0.1 10
100
1000
0.01
0.001
0.0001
1e-05
1 10 100 1000
0.06
1
0.05
0.9
0.04 0.8
0.03 0.7
0.6
0.02 sigma
0.5
0.01 0.4
10 0.3
0.8 0.2
0.6 0.1
sigma 0 0.5 1 1.5 2
0.4
2
(a1) 0.2 1
1.5 (a2) mean
0.5 mean
0
0.09 0.09
mu=1
0.08 mu=1.25 0.08
mu=1.5
P(sigma|D,mu=1)
0.07 0.07
0.06 0.06
0.05 0.05
0.04 0.04
P(sigma|D)
0.03 0.03
0.02 0.02
(c) 0.01
(d) 0.01
0 0
0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
(a)
Repetition code R3
(b)
Simple parity code P3
(c)
(7, 4) Hamming code
t Likelihood Posterior probability
0000000 0.0275562 0.25
0001011 0.0001458 0.0013
0010111 0.0013122 0.012
0011100 0.0030618 0.027
0100110 0.0002268 0.0020
0101101 0.0000972 0.0009
0110001 0.0708588 0.63
0111010 0.0020412 0.018
1000101 0.0001458 0.0013
1001110 0.0000042 0.0000
1010010 0.0030618 0.027
1011001 0.0013122 0.012
1100011 0.0000972 0.0009
1101000 0.0002268 0.0020
1110100 0.0020412 0.018
1111111 0.0000108 0.0001
n Likelihood Posterior marginals
P (yn | tn = 1) P (yn | tn = 0) P (tn = 1 | y) P (tn = 0 | y)
1 0.1 0.9 0.061 0.939
2 0.4 0.6 0.674 0.326
3 0.9 0.1 0.746 0.254
4 0.1 0.9 0.061 0.939
5 0.1 0.9 0.061 0.939
6 0.1 0.9 0.061 0.939
7 0.3 0.7 0.659 0.341
+
+
=
(a)
(b)
x1 x2 x3
g g g
@
@ @@
f1 f2 f3 f4 f5
xn
rm→n (xn ) = fm (xn )
fm
Evidence
P(D|H )
1
P(D|H2)
D
C
1
Create
Gather alternative
DATA MODELS
- Fit each MODEL
to the DATA
@
Gather Create new
more data Assign preferences to the models
alternative MODELS
6 6
@
@
R
@
Choose what Decide whether
data to ? to create new
gather next models
Choose future
actions
P (w | D, Hi )
σw|D
P (w | Hi )
wMP
w
σw
P (D | H3 )
P (D | H2 )
D
P (D | H1 )
P (w | D, H1 )
P (w | D, H2 ) P (w | H1 )
P (w | D, H3 )
P (w | H2 )
D P (w | H3 )
w w w
σw|D
σw
1?
or 2?
1?
or 2?
∗ ∗
H1 : L(H1 ) L(w(1) | H1 ) L(D | w(1) , H1 )
∗ ∗
H2 : L(H2 ) L(w(2) | H2 ) L(D | w(2) , H2 )
∗ ∗
H3 : L(H3 ) L(w(3) | H3 ) L(D | w(3) , H3 )
H11 H10
v m v m
d d
H01 H00
v m v m
d d
3 3
P*(x) P*(x)
2.5 2.5
2 2
1.5 1.5
1 1
0.5 0.5
0 0
-4 -2 0 2 4 -4 -2 0 2 4
(a) (b)
P ∗ (x)
64 log(2)
Entropy
0
0 1 2 3 4 5 6
(a) Temperature (b)
-6.2 -6.2
-6.4 -6.4
-6.6 -6.6
-6.8 -6.8
-7 -7
(a) (b)
-7.2 -7.2
10 100 1000 10000 100000 1000000 10 100 1000 10000 100000 1000000
(a) (b)
cQ∗ (x) cQ∗ (x)
∗
P ∗ (x) P (x)
x x x
P ∗ (x) Q∗ (x)
φ(x)
x
P(x)
Q(x)
phi(x)
-5 0 5 10 15
P(x)
cQ(x)
-4 -3 -2 -1 0 1 2 3 4
P ∗(x)
x(1)
Q(x; x(1) )
L
(b) Metropolis (c) Independent sampling
100 iterations 100 iterations
12 12
10 10
8 8
6 6
4 4
2 2
0 0
0 5 10 15 20 0 5 10 15 20
400 iterations 400 iterations
40 40
35 35
30 30
(a) 25 25
20 20
15 15
10 10
5 5
0 0
0 5 10 15 20 0 5 10 15 20
1200 iterations 1200 iterations
90 90
80 80
70 70
60 60
50 50
40 40
30 30
20 20
10 10
0 0
0 5 10 15 20 0 5 10 15 20
x2 x2
P (x)
(t)
P (x1 | x2 )
x(t)
(a) (b)
x1 x1
x2 x2
x(t+2)
x(t+1)
P (x2 | x1 )
x(t)
(c) (d)
x1 x1
Q(x; x(1) )
P ∗(x)
x(1) x
Q(x; x(2) )
P ∗(x)
x(2) x
p(0) (x)
0 5 10 15 20
p(1) (x)
0 5 10 15 20
p(2) (x)
0 5 10 15 20
p(3) (x)
0 5 10 15 20
(10)
p (x)
0 5 10 15 20
p(100) (x)
0 5 10 15 20
p(200) (x)
0 5 10 15 20
p(400) (x)
0 5 10 15 20
p(0) (x)
0 5 10 15 20
p(1) (x)
0 5 10 15 20
p(2) (x)
0 5 10 15 20
p(3) (x)
0 5 10 15 20
(10)
p (x)
0 5 10 15 20
p(100) (x)
0 5 10 15 20
p(200) (x)
0 5 10 15 20
p(400) (x)
0 5 10 15 20
1 2
3a,3b,3c 3d,3e
5,6 8
5,6,7
10
1
0 1 2 3 4 5 6 7 8 9 10 11
0 X B−1
(1)
(2)
(3)
1.1 10 3.5
1000
1 10000
100000 3
0.9 8 theory
2.5
0.8
6
0.7 2
0.6 1000 1.5
10000 4
0.5 100000
theory 1
0.4 2
0.5
0.3
0.2 0 0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6
Hamiltonian Monte Carlo Simple Metropolis
1 1
(a) (c)
0.5 0.5
0 0
-0.5 -0.5
-1 -1
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
1 1
(b) 0.5 (d)
0.5
0
0
-0.5
-0.5
-1
-1.5 -1
-1.5 -1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
Gibbs sampling Overrelaxation
(a) 1 1
0.5 0.5
0 0
-0.5 -0.5
-1 -1
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
(b) 0
-0.2
-0.4
-0.6
-0.8
-1
-1 -0.8-0.6-0.4-0.2 0
(c)
Gibbs sampling
3
2
1
0
-1
-2
-3
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Overrelaxation
3
2
1
0
-1
-2
-3
0 200 400 600 800 1000 1200 1400 1600 1800 2000
N Mean energy and fluctuations Mean square magnetization
0.5 1
Mean Square Magnetization
0 0.8
-0.5 0.6
Energy
-1 0.4
-1.5
16 0.2
-2
0
1 10 1 10
Temperature Temperature
0.5 1
Mean Square Magnetization
0 0.8
-0.5 0.6
Energy
-1 0.4
100 -1.5
0.2
-2
0
1 10 1 10
Temperature Temperature
N = 100 N = 4096
0 0
-0.5 -0.5
Energy
-1 -1
-1.5 -1.5
-2 -2
2 2.5 3 3.5 4 4.5 5 2 2.5 3 3.5 4 4.5 5
(a) 0.28 0.05
0.26
0.045
0.24
0.22 0.04
sd of Energy
0.2
0.035
0.18
0.03
0.16
0.14 0.025
0.12
0.02
0.1
0.08 0.015
2 2.5 3 3.5 4 4.5 5 2 2.5 3 3.5 4 4.5 5
(b) 1 1
Mean Square Magnetization
0.8 0.8
0.6 0.6
0.4 0.4
0.2 0.2
0 0
2 2.5 3 3.5 4 4.5 5 2 2.5 3 3.5 4 4.5 5
(c) 1.6 1.8
1.4 1.6
1.4
1.2
1.2
Heat Capacity
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2 0.2
0 0
2 2.5 3 3.5 4 4.5 5 2 2.5 3 3.5 4 4.5 5
(d)
0.45
Heat Capacity
0.4 Var(E)
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0.1 1 10
Temperature
J = +1 J = −1
0.5 0.5
0 0
-0.5 -0.5
Energy
-1 -1
-1.5 -1.5
-2 -2
1 10 1 10
Temperature Temperature
J = +1 J = −1
0.5 0.5
0 0
-0.5 -0.5
-1 -1
Energy
Energy
-1.5 -1.5
-2 -2
-2.5 -2.5
-3 -3
1 10 1 10
(a) Temperature (d) Temperature
0.08 0.03
0.07
0.025
0.06
0.02
sd of Energy
sd of Energy
0.05
0.04 0.015
0.03
0.01
0.02
0.005
0.01
0 0
1 10 1 10
(b) Temperature (e) Temperature
1.6 0.25
1.4
0.2
1.2
Heat Capacity
Heat Capacity
1 0.15
0.8
0.6 0.1
0.4
0.05
0.2
0 0
1 10 1 10
(c) Temperature (f) Temperature
T J = +1 T J = −1
20 50
6 5
4 2
3 0.5
2
b b b b b b
b b b b b b
b b b b b b
b b b b b b
T
2.5
2.4
2.3
2
T
J = −1 J = +1
b b b
b H b H b H
H b H b H b
b H b H b H
H b H b H b
b H b H b H
H b H b H b
b H b H b H
H H H
-1 +1
+1 +1 +1 +1
(a) (b)
Ferromagnets of width 8 Antiferromagnets of width 8
-1 -1
Triangular Triangular
-2 Rectangular -2 Rectangular
-3
Free Energy -3
-4
-4
-5
-5
-6
-6
-7
-7 -8
0 2 4 6 8 10 0 2 4 6 8 10
Temperature Temperature
0.7
0.6
0.5
Entropy
0.4
0.3
0.2 Triangular(-)
Rectangular
Triangular(+)
0.1
0
0 2 4 6 8 10
Temperature
0
-0.5
-1
-1.5
-2 Triangular(-)
Rectangular(+/-)
Triangular(+)
-2.5
-3
-3.5
1 10
Rectangular Ferromagnet Triangular Ising Models
1.2 1.2
1 width 4 1 width 4 (-)
width 8 width 8 (-)
width 4 (+)
0.8 0.8 width 8 (+)
Heat Capacity
0.6 0.6
0.4 0.4
0.2 0.2
0 0
-0.2 -0.2
1 10 1 10
Temperature Temperature
Rectangular Ferromagnet Triangular Ising Models
7 16
6 14
5 12 width 4 (-)
width 8 (-)
10 width 4 (+)
4 width 8 (+)
8
var(E) 3 width 4
width 8 6
2
4
1 2
0 0
-1 -2
1 10 1 10
Temperature Temperature
+ + + + +
− − + − +
+ − − − +
s2 s3
250 250 250 250
200 200 200 200
150 150 150 150
100 100 100 100
50 50 50 50
0 0 0 0
0 5 10 15 20 0 5 10 15 20 0 5 10 15 20 0 5 10 15 20
(i) (ii) (i) (ii)
(a) (b)
0 0 0 0 0 0
-50 -50 -50 -50 -50 -50
-100 -100 -100 -100 -100 -100
-150 -150 -150 -150 -150 -150
-200 -200 -200 -200 -200 -200
-250 -250 -250 -250 -250 -250
0 10 20 0 10 20 0 10 20 0 10 20 0 10 20 0 10 20
T0 = −50 T0 = −100 T0 = −200 T0 = −50 T0 = −100 T0 = −200
(a) (b)
0 0 0 0 0
-50 -50 -50 -50 -50
-100 -100 -100 -100 -100
-150 -150 -150 -150 -150
-200 -200 -200 -200 -200
-250 -250 -250 -250 -250
0 10 20 0 10 20 0 10 20 0 10 20 0 10 20
T0 = −50 T0 = −100 T0 = −200 T0 = −50 T0 = −1000
(a) (b) (c)
1
h = 0.00
h = 0.40
h = 0.80
0.5
-0.5
-1
0 1 2 3 4 5 6 7 8
1
1
0.5
0.5
0 0
Free Energy Energy
-2
0
-2.5
-3
-0.5
-3.5
-4 -1
-4.5
-1.5 mean field theory
-5 mean field theory real 8x8 system
real 8x8 system
-5.5
-2
-6
0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8
Entropy Heat Capacity, dE/dT
1.6
0.7 mean field theory
1.4 real 8x8 system
0.6
1.2
0.5 1
0.4 0.8
0.3 0.6
0.2 mean field theory 0.4
real 8x8 system
0.2
0.1
0
0
-0.2
0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8
Fluctuations, var(E)
6
mean field theory
5 real 8x8 system
-1
0 1 2 3 4 5 6 7 8
(a) (b) (c)
σ σ σ
1 1 1
0.9 0.9 0.9
0.8 0.8 0.8
0.7 0.7 0.7
0.6 0.6 0.6
0.5 0.5 0.5
0.4 0.4 0.4
0.3 0.3 0.3
0.2 0.2 0.2
0 0.5 1 1.5 2 0 0.5 1 1.5 2 0 0.5 1 1.5 2
µ µ µ
(d) (e)
σ σ
1 1
0.9 0.9
0.8 0.8
0.7 0.7
0.6 0.6
0.5 0.5
0.4 0.4
0.3 0.3
0.2 0.2
0 0.5 1 1.5 2 0 0.5 1 1.5 2
µ µ
(f)
σ
1
0.9
0.8
0.7
0.6
0.5
... 0.4
0.3
0.2
0 0.5 1 1.5 2
µ
Upper bound
2
1
≤ exp(µa − H2e (µ)) µ ∈ [0, 1]
1 + e−a
1 Lower bound
1
≥ g(ν) exp (a − ν)/2 − λ(ν)(a2 − ν 2 )
0 1 + e−a
-5 0 5 where λ(ν) = [g(ν) − 1/2] /2ν.
(a) (b)
s1 sK
y1 yN
x1 x1
-4 -2 0 2 4 -4 -2 0 2 4
4 4
2 2
0 x2 0 x2
-2 -2
-4 -4
(a) (b)
x1
-8 -6 -4 -2 0 2 4 6 8 30
8
6 20
4
10
2
0 x2 x2 0
-2
-10
-4
-6 -20
-8
-30
-30 -20 -10 0 10 20 30
(c) (d) x1
6 6 6
80 3000
70
60 200
50 2000
40
30 100
90
80 1000
900
70 800
20
60 700
50 600
40 500
10 400
9 30
8 300
7
6 20
5 200
3 10
9
8 100
90
7 80
2
6 70
5 60
4 50
1 40
3
metres feet inches
-3 -2 -1 0 1 2 3
1.2
0.8
0.6
0.4
0.2
0
1e-10 1e-09 1e-08 1e-07
1
0.01
0.0001
1e-06
1e-08
1e-10
1e-10 1e-09 1e-08 1e-07
σn
3.5
2.5
1.5
0.5
0
-6 -4 -2 0 2 4 6
(µn − µ1 )
ewpage
0 0.2 0.4 0.6 0.8 1
1
0.8
pB+
0.6
1
0.4 0.8
0.6
0.2
0.4
0.2
0 0.2 0
0.4 0.6
0 0.8 1
0 0.2 0.4 0.6 0.8 1
pA+
1
pB+
0 pA+
0 1
1
pB+
0 pA+
0 1
6
y
w0
b EA
EA
w1 E A wI
b b Eb Ab
x1 . . . xI
1
0.5
10
5
-10 0 x2
-5 0 -5
x1 5 10 -10
w = (0, 2)
1 1
5 w2 60.5 0.5
10 10
0 5 5
-10 0 x2 -10 0 x2
-5 0 -5 -5 0 -5
x1 5 10 -10 x1 5 10 -10
1
w = (1, 4) w = (5, 4)
4 0.5
10
0 5
-10 0 x2
-5 0 -5
x1 5 10 -10
1 1
w = (−2, 3)
3 0.5 0.5
10 10
5 0 5
-10 0 x2 -10 0 x2
-5 0 -5 -5 0 -5
x1 5 10 -10 x1 5 10 -10
1
w = (0, 2) w = (2, 2)
2 0.5
10
5
-10 0 x2
-5 0 -5
x1 5 10 -10
1 1
w = (5, 1)
1 0.5 0.5
10 10
5 5
-10 0 x2 -10 0 x2
-5 0 -5 -5 0 -5
x1 5 10 -10 x1 5 10 -10
1
w = (1, 0) w = (3, 0) -
0 0.5
10
-10
5
0 x2
w1
-5 0 -5
x1 5 10 -10
1
w = (−2, −1)
−1 0.5
10
0 5
-10 0 x2
-5 0 -5
x1 5 10 -10
w = (2, −2)
−2
−3 −2 −1 0 1 2 3 4 5 6
α = 0.01 α = 0.1 α=1
2 2
2
1 1
0
0 0
-2
-1 -1
-4 w0
w1
w2 -2 w0 -2 w0
-6 w1 w1
(a) -8 -3
w2
-3
w2
-10 -4 -4
-12 -5 -5
1 10 100 1000 10000 100000 1 10 100 1000 10000 100000 1 10 100 1000 10000 100000
3 3 3
2.5 2.5 2.5
2 2 2
1.5 1.5 1.5
1 1 1
(b) 0.5 0.5 0.5
0 0 0
-0.5 -0.5 -0.5
-0.5 0 0.5 1 1.5 2 2.5 3 -0.5 0 0.5 1 1.5 2 2.5 3 -0.5 0 0.5 1 1.5 2 2.5 3
7 7 7
G(w) G(w) G(w)
6 M(w) 6 M(w) 6 M(w)
5 5 5
4 4 4
3 3 3
(c)
2 2 2
1 1 1
0 0 0
1 10 100 1000 10000 100000 1 10 100 1000 10000 100000 1 10 100 1000 10000 100000
10 10 10
8 8 8
6 6 6
4 4 4
(d)
2 2 2
0 0 0
0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10
{tn }N - Learning - w - w - {t̂n }N
n=1 algorithm n=1
6
6
{xn }N
n=1 {xn }N
n=1
w2
x2
(0)
x1 w1
x(1)
(a) (b)
(1)
w2
x2 (0,0)
(0,1)
x1 (1,0) w1
x(2) x(1)
(a) (b)
(1,1)
w2
(0,0,1)
x2
(0,1,1)
(1,0,1)
x (3)
x1
w1
(0,1,0)
x(2) x(1)
(b)
(a)
(1,0,0)
(1,1,0)
(a)
(b)
(a) (b)
(c)
N=K
1
1
0.75
0.75 N=2K
0.5
0.5 0.25
0.25 70 0
60
50
0 K
40
10 30
20
20 10
30 N=K
(a) N 40 (b) 50 N 100 150
50 K=N/2
60
70 50 60 70
10 20 30K 40 2400
2300 log T(N,K)
log 2^N
1
2200
0.75
2100
0.5 2000
0.25 1900
N=K N=2K
(c) 0 (d) 1800
1800 1900 2000 2100 2200 2300 2400
0 0.5 1 1.5 2 2.5 3
N/K
ewpage
Data set Likelihood Probability of parameters
0 w2
N =0 (constant) 5
0 w2 -5
-5
-5 0 w1 5 -5 0 5
w1
N =2 5
10
x2
5
0 w2
0.5 5 5
0
0 w2 0 w2 -5
-5
-5 -5
-5 0 -5 0
-10 w1 5 w1 5 -5 0 5
-10 -5 0
x1
5 10 w1
N =4 5
10
0.1
x2
5
0 w2
0.05 5 5
0
0 w2 0 w2 -5
-5
-5 -5
-5 0 -5 0
-10 w1 5 w1 5 -5 0 5
-10 -5 0
x1
5 10 w1
N =6 5
10
x2
5
0.05
0 w2
5 5
0
0 w2 0 w2 -5
-5
-5 -5
-5 0 -5 0
-10 w1 5 w1 5 -5 0 5
-10 -5 0
x1
5 10 w1
10
w2 wMP
2
A A 5 Samples from 1
P(w|D,H)
B B
0 w1
(a) (b) 0 5 10 (c)
P ∗ (x)
x(1)
Q(x; x(1) )
−ηg
(a) (b) (c)
Dumb Metropolis Gradient descent Langevin
10
5
0
-5
-10
-15
-20
-25
Langevin -30
0 2000 4000 6000 8000 10000
5
0
-5
-10
-15
-20
-25
-30
-35
HMC -40
0 2000 4000 6000 8000 10000
ψ(a, s2 )
(b)
(a)
5 10
4
3
2
1 5
A
0
(a) (b)
-1
-2
-3 B
0 1 2 3 4 5 6 0
0 5 10
(a) (b)
(a)
(b) →
. 0 0 0 0 -2 2 -2 2 2 -2 0 0 0 2 0 0 -2 0 2 2 0 0 -2 -2
0 . 4 4 0 -2 -2 -2 -2 -2 -2 0 -4 0 -2 0 0 -2 0 -2 -2 4 4 2 -2
0 4 . 4 0 -2 -2 -2 -2 -2 -2 0 -4 0 -2 0 0 -2 0 -2 -2 4 4 2 -2
0 4 4 . 0 -2 -2 -2 -2 -2 -2 0 -4 0 -2 0 0 -2 0 -2 -2 4 4 2 -2 (c) →
0 0 0 0 . 2 -2 -2 2 -2 2 -4 0 0 -2 4 -4 -2 0 -2 2 0 0 -2 2
-2 -2 -2 -2 2 . 0 0 0 0 4 -2 2 -2 0 2 -2 0 -2 0 0 -2 -2 0 4
2 -2 -2 -2 -2 0 . 0 0 4 0 2 2 -2 4 -2 2 0 -2 4 0 -2 -2 0 0
-2 -2 -2 -2 -2 0 0 . 0 0 0 2 2 2 0 -2 2 4 2 0 0 -2 -2 0 0 (d) → →
2 -2 -2 -2 2 0 0 0 . 0 0 -2 2 2 0 2 -2 0 2 0 4 -2 -2 -4 0
2 -2 -2 -2 -2 0 4 0 0 . 0 2 2 -2 4 -2 2 0 -2 4 0 -2 -2 0 0
-2 -2 -2 -2 2 4 0 0 0 0 . -2 2 -2 0 2 -2 0 -2 0 0 -2 -2 0 4
0 0 0 0 -4 -2 2 2 -2 2 -2 . 0 0 2 -4 4 2 0 2 -2 0 0 2 -2 (e) →
0 -4 -4 -4 0 2 2 2 2 2 2 0 . 0 2 0 0 2 0 2 2 -4 -4 -2 2
0 0 0 0 0 -2 -2 2 2 -2 -2 0 0 . -2 0 0 2 4 -2 2 0 0 -2 -2
2 -2 -2 -2 -2 0 4 0 0 4 0 2 2 -2 . -2 2 0 -2 4 0 -2 -2 0 0
0 0 0 0 4 2 -2 -2 2 -2 2 -4 0 0 -2 . -4 -2 0 -2 2 0 0 -2 2 (f) →
0 0 0 0 -4 -2 2 2 -2 2 -2 4 0 0 2 -4 . 2 0 2 -2 0 0 2 -2
-2 -2 -2 -2 -2 0 0 4 0 0 0 2 2 2 0 -2 2 . 2 0 0 -2 -2 0 0
0 0 0 0 0 -2 -2 2 2 -2 -2 0 0 4 -2 0 0 2 . -2 2 0 0 -2 -2
2 -2 -2 -2 -2 0 4 0 0 4 0 2 2 -2 4 -2 2 0 -2 . 0 -2 -2 0 0 (g) →
2 -2 -2 -2 2 0 0 0 4 0 0 -2 2 2 0 2 -2 0 2 0 . -2 -2 -4 0
0 4 4 4 0 -2 -2 -2 -2 -2 -2 0 -4 0 -2 0 0 -2 0 -2 -2 . 4 2 -2
0 4 4 4 0 -2 -2 -2 -2 -2 -2 0 -4 0 -2 0 0 -2 0 -2 -2 4 . 2 -2
-2 2 2 2 -2 0 0 0 -4 0 0 2 -2 -2 0 -2 2 0 -2 0 -4 2 2 . 0 (h) →
-2 -2 -2 -2 2 4 0 0 0 0 4 -2 2 -2 0 2 -2 0 -2 0 0 -2 -2 0 .
(i) → (j) → (k) → →
(l) → → (m) → →
(a) (b) → →
. -1 1 -1 1 x x -3 3 x x -1 1 -1 x -1 1 -3 x 1 3 -1 1 x -1
-1 . 3 5 -1 -1 -3 -1 -3 -1 -3 1 x 1 -3 1 -1 -1 -1 -1 -3 5 3 3 -3
1 3 . 3 1 -3 -1 x -1 -3 -1 -1 x -1 -1 -1 1 -3 1 -3 -1 3 5 1 -1
-1 5 3 . -1 -1 -3 -1 -3 -1 -3 1 -5 1 -3 1 -1 -1 -1 -1 -3 5 x 3 -3
1 -1 1 -1 . 1 -1 -3 x x 3 -5 1 -1 -1 3 x -3 1 -3 3 -1 1 -3 3
(c) → →
x -1 -3 -1 1 . -1 1 -1 1 3 -1 1 -1 -1 3 -3 1 x 1 x -1 -3 1 3
x -3 -1 -3 -1 -1 . -1 1 3 1 1 3 -3 5 -3 3 -1 -1 x 1 -3 -1 -1 1
-3 -1 x -1 -3 1 -1 . -1 1 -1 3 1 x -1 -1 1 5 1 1 -1 x -3 1 -1
3 -3 -1 -3 x -1 1 -1 . -1 1 -3 3 1 1 1 -1 -1 3 -1 5 -3 -1 x 1
x -1 -3 -1 x 1 3 1 -1 . -1 3 1 -1 3 -1 x 1 -3 5 -1 -1 -3 1 -1
x -3 -1 -3 3 3 1 -1 1 -1 . -3 3 -3 1 1 -1 -1 -1 -1 1 -3 -1 -1 5 (d) →
-1 1 -1 1 -5 -1 1 3 -3 3 -3 . -1 1 1 -3 3 x -1 3 -3 1 -1 3 -3
1 x x -5 1 1 3 1 3 1 3 -1 . -1 3 -1 1 1 1 1 3 -5 -3 -3 3
-1 1 -1 1 -1 -1 -3 x 1 -1 -3 1 -1 . x 1 -1 3 3 -1 1 1 -1 -1 -3
x -3 -1 -3 -1 -1 5 -1 1 3 1 1 3 x . x 3 -1 -1 3 1 -3 -1 -1 1
-1 1 -1 1 3 3 -3 -1 1 -1 1 -3 -1 1 x . -5 -1 -1 -1 1 1 -1 -1 1
1 -1 1 -1 x -3 3 1 -1 x -1 3 1 -1 3 -5 . 1 1 1 -1 -1 1 1 -1 (e) →
-3 -1 -3 -1 -3 1 -1 5 -1 1 -1 x 1 3 -1 -1 1 . 1 1 -1 -1 -3 1 -1
x -1 1 -1 1 x -1 1 3 -3 -1 -1 1 3 -1 -1 1 1 . -3 3 -1 1 -3 -1
1 -1 -3 -1 -3 1 x 1 -1 5 -1 3 1 -1 3 -1 1 1 -3 . x -1 -3 1 -1
3 -3 -1 -3 3 x 1 -1 5 -1 1 -3 3 1 1 1 -1 -1 3 x . -3 -1 -5 1
-1 5 3 5 -1 -1 -3 x -3 -1 -3 1 -5 1 -3 1 -1 -1 -1 -1 -3 . 3 x -3
1 3 5 x 1 -3 -1 -3 -1 -3 -1 -1 -3 -1 -1 -1 1 -3 1 -3 -1 3 . 1 -1
x 3 1 3 -3 1 -1 1 x 1 -1 3 -3 -1 -1 -1 1 1 -3 1 -5 x 1 . -1
(f) → →
-1 -3 -1 -3 3 3 1 -1 1 -1 5 -3 3 -3 1 1 -1 -1 -1 -1 1 -3 -1 -1 .
Desired memories:
→ → → → →
→ → → → →
Desired memories Attracting stable states
moscow------russia moscow------russia
lima----------peru lima----------peru
london-----england londog-----englard (1)
tokyo--------japan tonco--------japan (1)
edinburgh-scotland edinburgh-scotland
ottawa------canada → W → (2)
oslo--------norway oslo--------norway
stockholm---sweden stockholm---sweden
paris-------france paris-------france
wzkmhewn--xqwqwpoq (3)
paris-------sweden (4)
ecnarf-------sirap (4)
1
1
0.8 0.99
0.6 0.98
0.4 0.97
0.2 0.96
0 0.95
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.09 0.1 0.11 0.12 0.13 0.14 0.15
√
IN
ai
I
Place in tour Place in tour 1 2 3 4
1 2 3 4 1 2 3 4 A
A A
B
City B City B
C
C C
D
D D (b)
B B 1 2 3 4
A A A
D D B
C C C
(a1) (a2) −dBD
D
(c)
(a) (b)
u u- u u u e e e e
(b) (c) (d) (z)
(a)
? - u- - u- - u -... - e e e
? u ? u u e e
e
→ →
ewpage
u ...
u u
u u u ...
u u u u
u u u u u
..
.. .. .
. .
(a) (b)
Outputs
Hiddens
Inputs
10
y
t Output
5
√
@ Hσout ∼ 1/σin
@ σout
Output
0
@
ttttt @tHidden layer
@
@
-5
σbias
- @ σin
t
@t Input -10 ∼ σbias /σin
1 x
-2 -1 0 1 2 3 4
Input
0.4
0.2
-0.2
-0.4
-0.6
-0.8
-1
-1.2
-1.4
-2 -1 0 1 2 3 4 5
1
-1
-2
1
0.5
-1 0
-0.5
0 -0.5
0.5 -1
1
Test Error
(a)
Training Error
(d)
Model Control Parameters
Log Probability(Training Data | Control Parameters)
(b)
(e)
Model Control Parameters
(c)
Error rate
Rejection rate
3.0 4.0
2.0
2.0
1.0
0.0
t
t
0.0
−2.0
−1.0
−2.0 −4.0
−3.0 −1.0 1.0 3.0 5.0 −3.0 −1.0 1.0 3.0 5.0
x x
0 2 0 2
(a) 2 exp − (x−x )
2(1.5)2
(x−x )
(b) 2 exp − 2(0.35) 2
4.0 6.0
4.0
2.0
2.0
0.0
t
0.0
−2.0
−2.0
−4.0 −4.0
−3.0 −1.0 1.0 3.0 5.0 −3.0 −1.0 1.0 3.0 5.0
x x
2 0 0 2
(c) 2 exp − sin (π(x−x
2(0.5) 2
)/3.0)
(d) 2 exp − (x−x
2(1.5)
)
2 + xx0
7.0 7.0
5.0 5.0
3.0 3.0
1.0 1.0
−1.0 −1.0
(a) −3.0 (b) −3.0
0.0 2.0 4.0 6.0 0.0 2.0 4.0 6.0
3.5
× 3
2.5
2 θ3
1.5
0.5
×
0.5 1 1.5 2 2.5 3 3.5 4
(c) r1
H =
tn (a) The prior distribution over codewords
P (t) ∝ [Ht = 0].
The variable nodes are the transmitted
P bits {tn }.
Each node represents the factor [ n∈N (m) tn = 0 mod 2].
P (rn | tn ) (b) The posterior distribution over codewords,
tn
P (t | r) ∝ P (t)P (r | t).
Each upper function node represents a likelihood factor P (rn | tn ).
(c) The joint probability of the noise n and syndrome z,
P (nn )
nn
P (n, z) = P (n) [z = Hn].
The top variable nodes are now the noise bits {nn }.
The added variable nodes at the base are the syndrome values
{zm }. P
zm Each definition zm = n Hmn nn mod 2 is enforced by a factor.
0.1
Probability of decoder error
0.01
0.001
low-density
0.0001 parity-check code
1e-05 Shannon limit
1e-06 GV
C
0 0.2 0.4 0.6 0.8 1
Rate
(a1) (b1)
0.4 0.4
0.35 0.35
0.3 P(y|‘0’) P(y|‘1’) 0.3 P(y|‘0’) P(y|‘1’)
0.25 0.25
0.2 0.2
0.15 0.15
0.1 0.1
0.05 0.05
0 0
(a2) -4 -2 0 2 4 (b2) -4 -2 0 2 4
1 1
0.1
0.1
0.01
0.01
0.001
0.001
0.0001 N=816
N=96
N=408 0.0001
1e-05 (N=96) j=4
(N=204) j=3
N=204 j=5
1e-06 1e-05 j=6
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 1 1.5 2 2.5 3 3.5 4
(a) (b)
3 3 3
(a) (b)
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0 5 10 15 20 25 30
x
f f f f f f iteration
r@@?
R @@?
R i−1
@
@
R
@
fx iteration
?r i
1
difference set cyclic codes 0.1
N 7 21 73 273 1057 4161
0.01
M 4 10 28 82 244 730
K 3 11 45 191 813 3431
0.001 Gallager(273,82)
d 4 6 10 18 34 66 DSC(273,82)
k 3 5 9 17 33 65
0.0001
1.5 2 2.5 3 3.5 4
M -
g -
@ 6
@ 0
A B @
T @ M
@
@
C D E g6
??
N -
Octal name
- t(a)
z7 hd z6 hd z5 hd z4 hd z3 hd z2 hd z1 hd z0 ` s
? ? ? ? ?
-⊕ -⊕ -⊕ - ⊕ - ⊕ - t(b)
(a) (1, 353)8
-⊕ - ⊕ - ⊕ - ⊕ - t(b)
6 6 6 6
z7 hd z6 hd z5 hd z4 hd z3 hd z2 hd z1 hd z0 s
? ? ? ? ?
-⊕ -⊕ -⊕ -⊕ - ⊕ - t(a)
(b) (247, 371)8
-⊕ - ⊕ - ⊕ - ⊕ - t(b)
6 6 6 6
z7 hd z6 hd z5 hd z4 hd z3 hd z2 hd z1 hd z0
? ? ? ? 6 - t(a)
-⊕ -⊕ -⊕ -⊕ -⊕ ` s
(c) 1, 247
371 8
11
10
01
00
(a)
transmit 0 0 0 0 1 1 1 0 1 1 0 0 0 0 0 0
source 0 0 1 0 0 0 0 0
11
10
01
00
(b)
transmit 0 0 0 0 1 1 0 1 0 1 0 0 0 1 0 1
source 0 0 1 0 0 0 0 0
11
10
01
00
transmit 0 0 0 0 1 1 1 0 1 1 0 0 0 0 0 0
source 0 0 1 1 1 0 0 0
-⊕ - t(b)
6
z2 hd z1 hd z0 s
? ?
- ⊕ - ⊕ - t(a)
p
(a) (5, 7)8
-⊕ - t(b)
6
z2 hd z1 hd z0
? 6 - t(a)
-⊕ -⊕ ` s
p
(b) (5/7)8
-⊕ - t(b)
6
z4 hd z3 hd z2 hd z1 hd z0
? ? ? 6 - t(a)
-⊕ -⊕ -⊕ -⊕ ` s (21/37)8
1111
1110
1101
1100
1011
1010
1001
1000
0111
0110
0101
0100
0011
0010
0001
0000
received 0 1 1 0 1 0 1 0 0 0 0 1 1 1 1 0
1111
1110
1101
1100
1011
1010
1001
1000
0111
0110
0101
0100
0011
0010
0001
0000
transmit 1 1 1 0 1 0 1 0 0 0 0 1 1 1 1 0
source 1 1 1 1 0 0 1 1
1111
1110
1101
1100
1011
1010
1001
1000
0111
0110
0101
0100
0011
0010
0001
0000
transmit 1 1 1 0 1 0 1 0 0 0 0 1 1 1 0 1
source 1 1 1 1 0 0 1 0
1111
1110
1101
1100
1011
1010
1001
1000
0111
0110
0101
0100
0011
0010
0001
0000
(a) (b)
- C1 -
-
- π C -
2
ewpage
(a)
(b)
(a)
1 1 1 1 1
0 0 0 0 0
1 1 1 1 1 1
0 0 0 0 0 0
(b)
1
total
undetected
0.1
0.01
N=204
0.001
408
0.0001
816
3000
N=30000 9999
1e-05
1 2 3 4 5
2000 3000
1 1800
total
detected 2500
1600
undetected
0.1
1400
2000
1200
0.01
1000 1500
0.001 800
1000
600
0.0001 400
500
200
1e-05
0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 0 0
0 20 40 60 80 100 120 140 160 180 0 20 40 60 80 100 120 140 160 180
(a) (ii.b) Eb /N0 = 0.749 dB (ii.c) Eb /N0 = 0.846 dB
1000 1000
100 100
10 10
1 1
10 20 30 40 50 60 70 80 90100 10 20 30 40 50 60 70 80 90 100
(iii.b) (iii.c)
GT = H =
{A, p} =
3
T
G = H =
3 3
3
3 3
T
G = A, p =
3 3
3 3
(a) (b)
(a)
(b)
s1 s2 s3
a)
1 0 1 1
1
b)
0 1 1
1
c)
1 1 0
1 0
d)
1 1
1 0
e)
1 1
1 0 1
f)
0.5
rho
tau
0.4
0.3
0.2
0.1
0
0 10 20 30 40 50
140
delta=0.01
120 delta=0.1
100 delta=0.9
80
60
40
20
0
0.01 0.1
11000
delta=0.01
10800 delta=0.1
delta=0.9
10600
10400
10200
10000
0.01 0.1
c
10000 10500 11000 11500 12000
10000 10500 11000 11500 12000
10000 10500 11000 11500 12000
var(E) N=24
log Z
var(E) N=8
N beta epsilon
N log (2)
beta beta
(a) (c)
log Z
N beta epsilon
N log (2)
beta
(b)
var(E) N=24
log Z
var(E) N=8
N beta epsilon
N log (2)
beta beta
(a) (b)
You might also like
- Answers To Exercises - Mathematical Statistics With Applications (7th Edition) PDF71% (7)Answers To Exercises - Mathematical Statistics With Applications (7th Edition) PDF20 pages
- Solution Manual To Introduction To Mathematical Statistics 6ed Hogg McKean and Craig89% (9)Solution Manual To Introduction To Mathematical Statistics 6ed Hogg McKean and Craig111 pages
- Introduction To Probability and Statistics SOLUTION PDFNo ratings yetIntroduction To Probability and Statistics SOLUTION PDF22 pages
- E2 201: Information Theory (2019) Solutions To Homework 3No ratings yetE2 201: Information Theory (2019) Solutions To Homework 311 pages
- Hybrid Least Squares For Learning Functions From Highly Noisy DataNo ratings yetHybrid Least Squares For Learning Functions From Highly Noisy Data30 pages
- Reference: "Detection, Estimation and Modulation Theory" by H.L. Van TreesNo ratings yetReference: "Detection, Estimation and Modulation Theory" by H.L. Van Trees18 pages
- Quiz 1: CS 726, Spring 2025: Prof. Sunita Sarawagi January, 2025No ratings yetQuiz 1: CS 726, Spring 2025: Prof. Sunita Sarawagi January, 20254 pages
- CSC 5132: Information Theory and Coding Techniques: Lecturer: Dr. P. SibandaNo ratings yetCSC 5132: Information Theory and Coding Techniques: Lecturer: Dr. P. Sibanda2 pages
- MATH 11300 Probability AUTUMN 2010: Problem Sheet 6 - SolutionsNo ratings yetMATH 11300 Probability AUTUMN 2010: Problem Sheet 6 - Solutions4 pages
- New Method of On-Line Estimation of Noise Coy Ariances and RNo ratings yetNew Method of On-Line Estimation of Noise Coy Ariances and R6 pages
- 1960 Max: Qu, Antixing For Minimum Distortion 7No ratings yet1960 Max: Qu, Antixing For Minimum Distortion 76 pages
