Foundations of Machine Learning Katholische Universität Eichstätt-Ingolstadt

summer semester 2024 Felix Voigtlaender, Thomas Jahn

discussed on: 25/04/2024

Exercise 0: Probability Theory

Exercise 0.1 (10 points). Suppose we play a game where we start with c dollars. On each of n
independent turns, you either double or halve your money, with equal probability. What is your
expected fortune after n turns? Formalize the situation using random variables.
n o
Solution. Let X1 , . . . , Xn ∈ 21 , 2 be the outcomes of the individual turns where “ 21 ” stands
for “halving the money” and “2” stands for “doubling the money”. By assumption P(Xi = 2) =
P(Xi = 12 ) = 12 for all i ∈ {1, . . . , n}. The fortune Sn at the end of the game will be
n
Y
Sn = c Xi .
i=1

Since X1 , . . . , Xn are jointly independent, we have

n n   n
2 0.5 5

independence Y LOTUS Y
E[Sn ] = c E[Xi ] = c + =c .
i=1 i=1 2 2 4

iid
Exercise 0.2. Let X1 , X2 , . . . ∼ U ([0, 1]) (i.e., X1 , X2 , · · · stochastically independent and all
uniformly distributed in [0, 1]). Compute P(max {X1 , . . . , Xn } ≤ λ) for λ ∈ R and n ∈ N.

Solution. Consider

P(max {X1 , . . . , Xn } ≤ λ) = P(X1 ≤ λ ∧ . . . ∧ Xn ≤ λ)

iid
= P(X1 ≤ λ) · . . . · P(Xn ≤ λ)
iid
= P(X1 ≤ λ)n

0 if λ < 0,


=

λn if λ ∈ [0, 1],


1 if λ > 1.

Exercise 0.3. Let X ∼ N (0, 1) and Y ∼ Bin(n, p) be independent random variables. Compute
E[(X + Y )2 ]. You may use the known formulas for the expectations and variances of X and Y .

Solution. Consider

E[(X + Y )2 ] = E[X 2 + 2XY + Y 2 ]

E linear
= E[X 2 ] + 2E[XY ] + E[Y 2 ]
X,Y independent
= E[X 2 ] + 2E[X]E[Y ] + E[Y 2 ]
= V[X] + E[X]2 + 2E[X]E[Y ] + V[Y ] + E[Y ]2
= 1 + 02 + 2 · 0 · np + np(1 − p) + (np)2
= 1 + np(1 − p) + n2 p2 .
Here we used
E[X 2 ] = V[X] + E[X]2 = 1 + 02 = 1,
E[Y 2 ] = V[Y ] + E[Y ]2 = np(1 − p) + n2 p2 .
If the expectations and values of the normal distribution and the binomial distribution are not
assumed to be known, one can do it like this:

E[(X + Y )2 ] = E[X 2 + 2XY + Y 2 ]

E linear
= E[X 2 ] + 2E[XY ] + E[Y 2 ]
X,Y independent
= E[X 2 ] + 2E[X]E[Y ] + E[Y 2 ]
!
LOTUS
Z ∞
1 2 x2
= x √ exp − dx
−∞ 2π 2
n
!! !
Z ∞
1 x2 n k
 
n−k
X
+ x √ exp − · k p (1 − p)
−∞ 2π 2 k=0 k
n
!
2 n
 
k n−k
X
+ k p (1 − p)
k=0 k
taking into account that
! " !#∞
Z ∞
1 x2 1 −x2
x √ exp − dx = − √ exp = 0,
−∞ 2π 2 2π 2 −∞
! " !#∞ !
Z ∞
2 1 x2 −x x2 Z ∞
−1 x2
x √ exp − dx = √ exp − − √ exp − dx = 0 − (−1) = 1,
−∞ 2π 2 2π 2 −∞ −∞ 2π 2
n n n
n k n k n−1 k
     
n−k n−k
p (1 − p)n−k
X X X
k p (1 − p) = k p (1 − p) = n
k=0 k k=1 k k=1 k − 1
n  m  
n−1 m

k−1 (n−1)−(k−1)
pj (1 − p)m−j
X X
= np p (1 − p) = np
k=1 k−1 j=0 j
= np(p + (1 − p))m = np,
n n
n k n k
   
k2 p (1 − p)n−k p (1 − p)n−k
X X
= (k(k − 1) + k)
k=0 k k=0 k
n n
n n−1 n−2 k n k
   
p (1 − p)n−k + p (1 − p)n−k
X X
= k(k − 1) · · k
k=2 k k − 1 k − 2 k=0 k
n 
!
n−2

2 k−2 n−k
X
= n(n − 1)p p n + np
k=2 k −2
n 
!
n − 2 k−2 (n−2)−(k−2)

2
X
= n(n − 1)p p n + np
k=2 k−2
2
= n(n − 1)p + np,
 2
  
uv ′ = uv − u′ v with u = x, v ′ = x √12π exp − x2 , u′ = 1, v = − √12π exp − 12 x2 .
R R
where
iid
Exercise 0.4. Let X1 , X2 , . . . ∼ U ([0, 1]) (i.e., X1 , X2 , · · · stochastically independent and all
uniformly distributed in [0, 1]). For n ∈ N, let Zn := n1 ni=1 Xi2 . What can you conclude about
P

the convergence of the Zn using the strong law of large numbers?

Solution. The random variables X12 , X22 . . . are iid (Lemma 2.23) with
Z 1
1
E[Xi2 ] = x2 dx = < ∞.
0 3
The strong law of large numbers (Theorem 2.31) says that
1
 
P lim Zn = = 1,
n→∞ 3
i.e., Zn converges almost surely to 13 .