1

13. The Weak Law and the Strong

Law of Large Numbers
James Bernoulli proved the weak law of large numbers (WLLN)
around 1700 which was published posthumously in 1713 in his
treatise Ars Conjectandi. Poisson generalized Bernoullis theorem
around 1800, and in 1866 Tchebychev discovered the method bearing
his name. Later on one of his students, Markov observed that
Tchebychevs reasoning can be used to extend Bernoullis theorem
to dependent random variables as well.

In 1909 the French mathematician Emile Borel proved a
deeper theorem known as the strong law of large numbers that further
generalizes Bernoullis theorem. In 1926 Kolmogorov derived
conditions that were necessary and sufficient for a set of mutually
independent random variables to obey the law of large numbers.
PILLAI
2
Let be independent, identically distributed Bernoulli random
Variables such that


and let represent the number of successes
in n trials. Then the weak law due to Bernoulli states that [see
Theorem 3-1, page 58, Text]



i.e., the ratio total number of successes to the total number of
trials tends to p in probability as n increases.

A stronger version of this result due to Borel and Cantelli
states that the above ratio k/n tends to p not only in probability, but
with probability 1. This is the strong law of large numbers (SLLN).
i
X
, 1 ) 0 ( , ) ( q p X P p X P
i i
= = = =
n
X X X k + + + =
2 1

{ } .
2
c
c
n
pq
p P
n
k
s >
(13-1)
PILLAI
3
What is the difference between the weak law and the strong
law?
The strong law of large numbers states that if is a
sequence of positive numbers converging to zero, then



From Borel-Cantelli lemma [see (2-69) Text], when (13-2) is
satisfied the events can occur only for a finite
number of indices n in an infinite sequence, or equivalently, the
events occur infinitely often, i.e., the event k/n
converges to p almost-surely.

Proof: To prove (13-2), we proceed as follows. Since


} {
n
c
{ } .
1
< >

= n
n
p P
n
k
c
(13-2)
{ }
n
p
n
k
c <
4 4
4
n np k p
n
k
c c > >
PILLAI
{ }
=
n n
k
n
A p c >
A
4
we have



and hence




where



By direct computation



{ } { } ( ) c c c c < + > = >

=
p P p P n n k p np k
n
k
n
k
n
n
k
) ( ) (
4 4 4 4
0
4
{ }
4 4
0
4
) ( ) (

n
k p np k
p P
n
k
n
n
k
c
c

=

s >
(13-3)
k n k
n
i
i n
q p k X P k p

|
.
|

\
|
=
=
)
`

= =
k
n
1
) (
} { } {
1
4 4
1 0
4
) ( ) ( ) ( ) ( ) (

= = =
= =
n
i
i
n
i
i
n
k
n
p X E np X E k p np k
PILLAI
5







since

Substituting (13-4) also (13-3) we obtain



Let so that the above integral reads
and hence


, 3
)] 1 ( 3 [ ) )( 1 ( 3 ) (
) ( ) ( ) 1 ( 3 ) ( ) ( ) 1 ( 4 ) (
) ( } ) {(
2
2 3 3
2
1 1
2
1 1
3
1
4
1 1 1 1
4
1
pq n
pq n n n pq n n pq q p n
Y E Y E n n Y E Y E n n Y E
Y Y Y Y E Y E
j
n
i
n
j
i j
n
i
n
j
i
n
i
i
n
i
n
k
n
j
n
l
l j k i
n
i
i
=
+ s + + =
+ + =
= =


= = = = =
= = = = =
(13-4)
can coincide with
j, k or l, and the second variable
takes (n-1) values
n i = 1
1 2 / 1 , 1 3 3 ) (
2 2 3 3 3
< s < + = + pq pq q p q p q p
{ }
2 4
3
k
n
pq
P p
n
c
c
> s
(13-5)
0
{ }

3/ 2
1/ 8 3/ 2
1
1 1
1 1
3 3 1
3 (1 2) 9 ,
( )
n n
k
n
P p pq pq x dx
n n
pq pq



= =
> s s +
= + = <

}
1/ 8
1
n
c =
PILLAI
6
thus proving the strong law by exhibiting a sequence of positive
numbers that converges to zero and satisfies (13-2).
We return back to the same question: What is the difference
between the weak law and the strong law?.
The weak law states that for every n that is large enough, the
ratio is likely to be near p with certain probability that
tends to 1 as n increases. However, it does not say that k/n is bound
to stay near p if the number of trials is increased. Suppose (13-1) is
satisfied for a given in a certain number of trials If additional
trials are conducted beyond the weak law does not guarantee that
the new k/n is bound to stay near p for such trials. In fact there can
be events for which for in some regular manner.
The probability for such an event is the sum of a large number of
very small probabilities, and the weak law is unable to say anything
specific about the convergence of that sum.
However, the strong law states (through (13-2)) that not only
all such sums converge, but the total number of all such events
1/ 8
1/
n
n c =
1
( ) / /
n
i
X n k n
i
=
=
,
0
n
, / c + > p n k
0
n n >
.
0
n c
PILLAI
7
where is in fact finite! This implies that the probability
of the events as n increases becomes and remains
small, since with probability 1 only finitely many violations to
the above inequality takes place as
Interestingly, if it possible to arrive at the same conclusion
using a powerful bound known as Bernsteins inequality that is
based on the WLLN.
Bernsteins inequality : Note that


and for any this gives
Thus





} { c > p
n
k
. n
c + > p n k /
) ( c c + > > p n k p
n
k
, 0 > . 1
)) ( (
>
+ c p n k
e
( )
(
( )
(
( )

=
+

+ =
+
+ =

s
s
= >
n
k
k n k
k
n
p n k
k n k
n
p n k
k
n
p n k
n
p n k
k n k
k
n
q p e
q p e
q p p P
n
k
0
)) ( (
) (
)) ( (
) (


} {
c
c
c
c
c
PILLAI
8







Since for any real x,




Substituting (13-7) into (13-6), we get


But is minimum for and hence


Similarly

( )
. ) (
) ( ) ( } {
0
n p q n
k n p k q
n
k
n
qe pe e
qe pe e p P
k
n
n
k
c
c
c


=

+ =
= >

2
x x
e x e + s
.
) ( ) (
2 2 2 2 2
2 2 2 2



e qe pe
e p q e q p qe pe
p q
p q p q
s + =
+ + + s +

. } {
2
c
c
n n
e p P
n
k

s >
c n n
2
2 / c =
(13-6)
(13-7)
0. , } {
4 /
2
> s >

c c
c n
e p P
n
k
(13-8)
4 /
2
} {
c
c
n
e p P
n
k

s <
PILLAI
9
and hence we obtain Bernsteins inequality


Bernsteins inequality is more powerful than Tchebyshevs inequality
as it states that the chances for the relative frequency k /n exceeding
its probability p tends to zero exponentially fast as
Chebyshevs inequality gives the probability of k /n to lie
between and for a specific n. We can use Bernsteins
inequality to estimate the probability for k /n to lie between
and for all large n.
Towards this, let

so that


To compute the probability of the event note that its
complement is given by
2
/ 4
{ } 2 .
n
k
n
P p e
c
c

> s
(13-9)
. n
c p c + p
c + p
c p
} { c c + < s = p p y
n
k
n
2
/ 4
( ) { } 2
c n
n
n
k
P y P p e
c
c

= > s
,
n
m n
y

c
n
m n
c
n
m n
y y

=

=
= ) (
PILLAI
10
and using Eq. (2-68) Text,



This gives



or,


Thus k /n is bound to stay near p for all large enough n, in probability,
a conclusion already reached by the SLLN.
Discussion: Let Thus if we toss a fair coin 1,000 times,
from the weak law


2
2
/ 4

/ 4
2
( ) {1 ( )} 1 1 as
1
m
n n
n m n m
e
P y P y m
e
c
c

= =
= >

. as 1 } , { > + s s m m n all for p p P

n
k
c c
{ } .
40
1
01 . 0
2
1
s >
n
k
P
. 1 . 0 = c
2
2
2
/ 4
/ 4

/ 4
2
( ) ( ) 2 .
1
m
c c n
n n
n m
n m n m
e
P y P y e
e
c
c
c

=
= =
s s =

PILLAI
11
Thus on the average 39 out of 40 such events each with 1000 or more
trials will satisfy the inequality or, it is quite possible
that one out of 40 such events may not satisfy it. As a result if we
continue the coin tossing experiment for an additional 1000 more
trials, with k representing the total number of successes up to the
current trial n, for it is quite possible that for few
such n the above inequality may be violated. This is still consistent
with the weak law, but not so often says the strong law. According
to the strong law such violations can occur only a finite number of
times each with a finite probability in an infinite sequence of trials,
and hence almost always the above inequality will be satisfied, i.e.,
the sample space of k /n coincides with that of p as

Next we look at an experiment to confirm the strong law:

Example: 2n red cards and 2n black cards (all distinct) are shuffled
together to form a single deck, and then split into half. What is
the probability that each half will contain n red and n black cards?
} { 1 . 0
2
1
s
n
k
, 2000 1000 = n
. n
12
Solution: From a deck of 4n cards, 2n cards can be chosen in
different ways. To determine the number of favorable draws of n red
and n black cards in each half, consider the unique draw consisting
of 2n red cards and 2n black cards in each half. Among those 2n red
cards, n of them can be chosen in different ways; similarly for
each such draw there are ways of choosing n black cards.Thus
the total number of favorable draws containing n red and n black
cards in each half are among a total of draws. This gives
the desired probability to be






For large n, using Stinglings formula we get
|
|
.
|

\
|
n
n
2
4
|
|
.
|

\
|
n
n 2
|
|
.
|

\
|
n
n 2
|
|
.
|

\
|
n
n
2
4
|
|
.
|

\
|
n
n 2
|
|
.
|

\
|
n
n 2
n
p
.
) ! ( )! 4 (
) ! 2 (
4
4
2
4
2 2
n n
n
p
n
n
n
n
n
n
n
=
|
.
|

\
|
|
.
|

\
|
|
.
|

\
|
~
PILLAI
13



For a full deck of 52 cards, we have which gives


and for a partial deck of 20 cards (that contains 10 red and 10 black
cards), we have and
One summer afternoon, 20 cards (containing 10 red and 10 black
cards) were given to a 5 year old child. The child split that partial
deck into two equal halves and the outcome was declared a success
if each half contained exactly 5 red and 5 black cards. With adult
supervision (in terms of shuffling) the experiment was repeated 100
times that very same afternoon. The results are tabulated below
in Table 13.1, and the relative frequency vs the number of trials
plot in Fig 13.1 shows the convergence of k /n to p.
n e n n e n n
e n n
p
n n n n
n n
n

2
] 2 [ ) 4 ( ) 4 ( 2
] ) 2 ( ) 2 ( 2 [
4 4 4
4 2 2
t t t
t
= ~

, 13 = n
221 . 0 ~
n
p
5 = n
. 3568 . 0 ~
n
p
PILLAI
14

Expt Number of
successes
Expt Number of
successes
Expt Number of
successes
Expt Number of
successes
Expt Number of
successes
1 0 21 8 41 14 61 23 81 29
2 0 22 8 42 14 62 23 82 29
3 1 23 8 43 14 63 23 83 30
4 1 24 8 44 14 64 24 84 30
5 2 25 8 45 15 65 25 85 30
6 2 26 8 46 16 66 25 86 31
7 3 27 9 47 17 67 25 87 31
8 4 28 10 48 17 68 25 88 32
9 5 29 10 49 17 69 26 89 32
10 5 30 10 50 18 70 26 90 32
11 5 31 10 51 19 71 26 91 33
12 5 32 10 52 20 72 26 92 33
13 5 33 10 53 20 73 26 93 33
14 5 34 10 54 21 74 26 94 34
15 6 35 11 55 21 75 27 95 34
16 6 36 12 56 22 76 27 96 34
17 6 37 12 57 22 77 28 97 34
18 7 38 13 58 22 78 29 98 34
19 7 39 14 59 22 79 29 99 34
20 8 40 14 60 22 80 29 100 35
Table 13.1
PILLAI
15
0.3437182
The figure below shows results of an experiment of
100 trials.
Fig 13.1
n
n
p