Assignment 0 (Sol.

)
Reinforcement Learning
Prof. B. Ravindran

Note: This is an ungraded assignment. Marks scored in this assignment will not be counted
towards the final score.

1. There are n bins of which the kth contains k − 1 blue balls and n − k red balls. You pick a bin
at random and remove two balls at random without replacement. Find the probability that:
• the second ball is red;
• the second ball is red, given that the first is red.

(a) 1/3, 2/3

(b) 1/2, 1/3
(c) 1/2, 2/3
(d) 1/3, 1/3

Sol. (c)
Pnof (k − 1) + (n
Let Ci be the colour of the ith ball. In each bin, there are a total P− n
k) = (n − 1)
balls. Of these half are blue and the other half are red (verify k=1 k − 1 = k=1 n − k).
The probability of the second ball being red is equal to the probability of the second ball being
red given that the first ball was either red or blue.
For a particular bin we have,

(n − k) (n − k − 1) (k − 1) (n − k) n−k
P (C2 = red) = + =
(n − 1) (n − 2) (n − 1) (n − 2) n−1

Considering all bins, we have

n
X n−k 1
P (C2 = red) = =
n(n − 1) 2
k=1

The probability of the second ball being red given that the first ball was red,

P (C2 = red, C1 = red)

P (C2 = red|C1 = red) =
P (C1 = red)

Now, P (C1 = red) = 21 .

1
 For a particular bin,
(n − k) (n − k − 1)
P (C2 = red, C1 = red) =
(n − 1) (n − 2)

Considering all bins, we have

n (n−k)(n−k−1)
n(n−1)(n−2)
X
P (C2 = red|C1 = red) = 1
k=1 2

Simplifying, we have
2
P (C2 = red|C1 = red) =
3
2. A medical company touts its new test for a certain genetic disorder. The false negative rate is
small: if you have the disorder, the probability that the test returns a positive result is 0.999.
The false positive rate is also small: if you do not have the disorder, the probability that the
test returns a positive result is only 0.005. Assume that 2% of the population has the disorder.
If a person chosen uniformly from the population is tested and the result comes back positive,
what is the probability that the person has the disorder?

(a) 0.803
(b) 0.976
(c) 0.02
(d) 0.204

Sol. (a)
Let,
• T be the probability of the test being positive,
• D be the probability of a person having the disorder
From the data provided, we have:
• P (T |D) = 0.999
• P (T |¬D) = 0.005
• P (D) = 0.02
• P (¬D) = 1 − P (D) = 0.98
We want to calculate the probability of a person chosen uniformly at random having the
disorder given that the test came back positive, i.e.,
P (D|T )

Now from the Bayes’ theorem, we have

P (T |D)P (D) P (T |D)P (D)
P (D|T ) = =
P (T ) P (T |D)P (D) + P (T |¬D)P (¬D)

Substituting known values, we have

0.999 ∗ 0.02
P (D|T ) = = 0.803
0.999 ∗ 0.02 + 0.005 ∗ 0.98

2
3. In an experiment, n coins are tossed, with each one showing up heads with probability p
independently of the others. Each of the coins which shows up heads is then tossed again.
What is the probability of observing 5 heads in the second round of tosses, if we toss 15 coins
in the first round and p = 0.4?
(Hint: First find the mass function of the number of heads observed in the second round.)

(a) 0.372
(b) 0.055
(c) 0.0345
(d) 0.0488

Sol. (b)
The same result will be observed if we toss each of the coins twice and count the number of
coins showing two consecutive heads. The probability of showing two consecutive heads, given
that the probability of showing heads in a toss is p, is equal to p2 . Hence, the number of heads
X observed after the second round of tosses can be given by
 
n 2r
P (X = r) = p (1 − p2 )n−r
x

Now substituting the given values into the above pmf we have,
 
15
P (X = 5) = 0.410 (1 − 0.42 )10 = 0.055
5

4. An airline knows that 5 percent of the people making reservations on a certain flight will not
show up. Consequently, their policy is to sell 52 tickets for a flight that can hold only 50
passengers. What is the probability that there will be a seat available for every passenger who
shows up?

(a) 0.5101
(b) 0.81
(c) 0.6308
(d) 0.7405

Sol. (d)
The probability that there will be a seat available for every passenger who shows up is equal to
the probability that less than or equal to 50 passengers show up. This is the same as 1 minus
the probability that exactly 52 or 51 passengers show up. Thus, the required probability

= 1 − 0.9552 − 52(0.95)51 (0.05) = 0.7405.

5. Let X have mass function

{x(x + 1)}−1

if x = 1, 2, ...,
f (x) =
0 otherwise,

and let α ∈ R. For what values of α is it the case that E(X α ) < ∞ ?

3
 (a) α < 21
(b) α<1
(c) α>1
(d) α > 43

Sol. (b)
We have
∞
X xα
E[X α ] =
x=1
x(x + 1)
This expression is finite only if α < 1.
6. Is the following a distribution function?
 −1/x
e x>0
F (x) =
0 otherwise
If so, give the corresponding density function. If not, mention why it is not a distribution
function.

(a) No, not a monotonic function

(b) Yes, x−2 e−1/x , x > 0
(c) No, not right continuous
(d) Yes, x−1 e−1/x , x > 0

Sol. (b)
F (x) → 1 as x → ∞ and F (x) = 0 for x < 0 by definition. The monotonicity and continuity
of F follow from the corresponding properties of e−1/x . Also, since all continuous functions
are right continuous, F is right continuous. Thus, the given function is a distribution function.
Its corresponding density function is given by differentiating:
 −2 −1/x
x e x>0
f (x) =
0 otherwise

7. Let Am×n be a matrix of real numbers. The matrix AAT has an eigenvector x with eigenvalue
b. Then the eigenvector y of AT A which has eigenvalue b is equal to

(a) xT A
(b) AT x
(c) x
(d) Cannot be described in terms of x

Sol. (b)
(AAT )x = bx
Multiplying by AT on both sides and rearranging,
(AT )(AAT x) = AT (bx)
(AT A)(AT x) = b(AT x)
Hence, AT x is an eigenvector of AT A, with eigenvalue b.

4
 8. Let An×n be a row stochastic matrix - in other words, all elements are non-negative and the
sum of elements in every row is 1. Let b be an eigenvalue of A. Which of the following is true?

(a) |b| > 1

(b) |b| <= 1
(c) |b| >= 1
(d) |b| < 1

Sol. (b)
Note that Ax = bx where x is an eigenvector and b is an eigenvalue. Let xmax be the largest
element of x. Let Aij denote the ith row, j th column element of A, and xj denote the j th
element of x. Now,
j=n
X
Aij xj = bxi
j=1

Let us consider the case where

Pj=nxi corresponds to the maximum element of x. The RHS is
equal to bxmax . Now, since j=1 Aij = 1 and Aij > 0, the LHS is less than or equal to xmax .
Hence,
bxmax ≤ xmax
|b| ≤ 1

9. Let u be a n × 1 vector, such that uT u = 1. Let I be the n × n identity matrix. The n × n

matrix A is given by (I − kuuT ), where k is a real constant. u itself is an eigenvector of A,
with eigenvalue -1. What is the value of k?

(a) -2
(b) -1
(c) 2
(d) 0

Sol. (c)
(I − kuuT )u = −u
u − ku(uT u) = −u
2u − ku = 0
Hence, k = 2
10. Which of the following are true for any m × n matrix A of real numbers

(a) The rowspace of A is the same as the columnspace of AT

(b) The rowspace of A is the same as the rowspace of AT
(c) The eigenvectors of AAT are the same as the eigenvectors of AT A
(d) The eigenvalues of AAT are the same as the eigenvalues of AT A

5
 Sol. (a) & (d)
Since the rows of A are the same as the columns of AT , the rowspace of A is the same as
the columnspace of AT . The eigenvalues of AAT are the same as the eigenvalues of AT A,
because if AAT x = λx we get AT A(AT x) = λ(AT x). (b) is clearly not necessary. (c) need not
hold, since although the eigenvalues are same, the eigenvectors have a factor of AT multiplying
them.

11. The Singular Value Decomposition (SVD) of a matrix R is given by U SV T . Consider an

orthogonal matrix Q and A = QR. The SVD of A is given by U1 S1 V1T . Which of the following
is/are true?

(a) U = U1
(b) S = S1
(c) V = V1

Sol. (b) & (c)

The matrix V1 represents the eigenvectors of AT A. Now

AT A = (RT QT )(QR)

Since Q is orthogonal, QT Q = I. Therefore,

AT A = (RT IR)

AT A = (RT R)
Since these matrices are equal, their eigenvectors will be the same. V represents the eigen-
vectors of RT R and V1 the eigenvectors of AT A. Hence, V = V1 . Also, since S represents
the eigenvalues of AT A (as well as AAT , since the set of eigenvalues is the same for both),
S = S1 . However, U need not be equal to U1 , since AAT = (QR)(RT QT ) = Q(RRT )QT .