10-701/15-781 Machine Learning

Mid-term Exam Solution

Your Name:
Your Andrew ID:

True or False (Give one sentence explanation) (20%)

1. (F) For a continuous random variable x and its probability distribution function p(x), it holds
that 0 p(x) 1 for all x.
2. (F) Decision tree is learned by minimizing information gain.
3. (F) Linear regression estimator has the smallest variance among all unbiased estimators.
4. (T) The coefficients assigned to the classifiers assembled by AdaBoost are always non-negative.
5. (F) Maximizing the likelihood of logistic regression model yields multiple local optimums.
6. (F) No classifier can do better than a naive Bayes classifier if the distribution of the data is known.
7. (F) The back-propagation algorithm learns a globally optimal neural network with hidden layers.
8. (F) The VC dimension of a line should be at most 2, since I can find at least one case of 3 points
that cannot be shattered by any line.
9. (F) Since the VC dimension for an SVM with a Radial Base Kernel is infinite, such an SVM must
be worse than an SVM with polynomial kernel which has a finite VC dimension.

10. (F) A two layer neural network with linear activation functions is essentially a weighted combination of linear separators, trained on a given dataset; the boosting algorithm built on linear
separators also finds a combination of linear separators, therefore these two algorithms will give
the same result.

Linear Regression (10%)

We are interested here in a particular 1-dimensional linear regression problem. The dataset corresponding to this problem has n examples (x1 ; y1 ), . . . , (xn ; yn ) where xi and yi are real numbers for all i. Let
w = [w0 , w1 ]T be the least squares solution we are after. In other words, w minimizes
n

J(w) =

1X
(yi w0 w1 xi )2 .
n
i=1

You can assume for our purposes here that the solution is unique.
1. (5%) Check each statement that must be true if w = [w0 , w1 ]T is indeed the least squares solution.
1 Pn

( )
i=1 (yi w0 w1 xi )yi = 0
n
P
n
1

) = 0
( )
i=1 (yi w0 w1 xi )(yi y
n
P
n
1

) = 0
()
i=1 (yi w0 w1 xi )(xi x
n
1 Pn

i=1 (yi w0 w1 xi )(w0 + w1 xi ) = 0 ()

n
where x
and y are the sample means based on the same dataset. (hint: take the derivative of
J(w) with respect to w0 and w1 )
(sol.) Taking the derivative with respect to w1 and w0 gives us the following conditions of optimality
n

2X

J(w) =
(yi w0 w1 xi ) = 0
w0
n
i=1

2X
J(w) =
(yi w0 w1 xi )xi = 0
w1
n
i=1

This means that the prediction error (yi w0 w1 xi ) does not co-vary with any linear function
of the inputs (has a zero mean and does not co-vary with the inputs). (xi x
) and (w0 + w1 xi )
are both linear functions of inputs.
2. (5%) There are several numbers (statistics) computed from the data that we can use to estimate
w . There are
x
=
y =

1
n
1
n

Pn

i=1 xi

Pn

Cxx =
Cxy =
Cyy =
Suppose we only care about the value

( )
i=1 yi
Pn
2
)
()
i=1 (xi x
Pn
)(yi y) ()
i=1 (xi x
Pn
)2
( )
i=1 (yi y
of w1 . Wed like to determine

1
n
1
n
1
n

w1 on the basis of ONLY two

numbers (statistics) listed above. Which two numbers do we need for this? (hint: use the answers
to the previous question)
3

(sol.) We need Cxx (spread of x) and Cxy (linear dependence between x and y). No justification
was necessary as these basic points have appeared in the course. If we want to derive these more
mathematically, we can, for example, look at one of the answers to the previous question:
n

1X
(yi w0 w1 xi )(xi x
) = 0,
n
i=1

which we can rewritte as

" n
#
" n
#
" n
#
X
X
1X
1
1
yi (xi x
) w0
(xi x
) w1
xi (xi x
) = 0
n
n
n
i=1

i=1

By using the fact that 1/n

i (xi

i=1

x
) = 0 we see that

i=1

i=1

1X
1X
yi (xi x
) =
(yi y)(xi x
) = Cxy
n
n
n
n
1X
1X
xi (xi x
) =
(xi x
)(xi x
) = Cxx
n
n
i=1

i=1

Substituting these back into our equation above gives

Cxy w1 Cxx = 0

AdaBoost (15%)

Consider building an ensemble of decision stumps Gm with the AdaBoost algorithm,

!
M
X
f (x) = sign
m Gm (x) .
m=1

Figure 1 dispalys a few labeled point in two dimensions as well as the first stump we have chosen. A
stump predicts binary 1 values, and depends only on one coordinate value (the split point). The little
arrow in the figure is the normal to the stump decision boundary indicating the positive side where the
stump predicts +1. All the points start with uniform weights.

x2

+1

+1

1
1

+1

x1
Figure 1: Labeled points and the first decision stump. The arrow points in the positive direction from
the stump decision boundary.

1. (5%) Circle all the point(s) in Figure 1 whose weight will increase as a result of incorporating the
first stump (the weight update due to the first stump).
(sol.) The only misclassified negative sample.
2. (5%) Draw in the same figure a possible stump that we could select at the next boosting iteration.
You need to draw both the decision boundary and its positive orientation.
(sol.) The second stump will also be a vertical split between the second positive sample (from left
to right) and the misclassified negative smaple, as drawn in the figure.
3. (5%) Will the second stump receive higher coefficient in the ensemble than the first? In other
words, will 2 > 1 ? Briefly explain your answer. (no calculation should be necessary).
(sol.) 2 > 1 because the point that the second stump misclassifies will have a smaller relative
weight since it is classified correctly by the first stump.

Neural Nets (15%)

Consider a neural net for a binary classification which has one hidden layer as shown in the figure. We
use a linear activation function h(z) = cz at hidden units and a sigmoid activation function g(z) =

1
1+ez

at the output unit to learn the function for P (y = 1|x, w) where x = (x1 , x2 ) and w = (w1 , w2 , . . . , w9 ).

1
x1

w3
w4

bias

w2

w1

bias

w7
w8

w
x2 w5
6

w9

1. (5%) What is the output P (y = 1 | x, w) from the above neural net? Express it in terms of xi , c
and weights wi . What is the final classification boundary?
(sol.)
g(w7 + w8 h(w1 + w3 x1 + w5 x2 ) + w9 h(w2 + w4 x1 + w6 x2 ))
1
=
1 + exp((w7 + cw8 w1 + cw9 w2 + (cw8 w3 + cw9 w4 )x1 + (cw8 w5 + cw9 w6 )x2 ))
The classification boundary is :
w7 + cw8 w1 + cw9 w2 + (cw8 w3 + cw9 w4 )x1 + (cw8 w5 + cw9 w6 )x2 = 0
2. (5%) Draw a neural net with no hidden layer which is equivalent to the given neural net, and
of this new neural net in terms of c and wi .
write weights w
(sol.)
3. (5%) Is it true that any multi-layered neural net with linear activation functions at hidden layers
can be represented as a neural net without any hidden layer? Briefly explain your answer.
(sol.) Yes. If linear activation functions are used for all the hidden units, output from hidden units
will be written as linear combination of input features. Since these intermediate output serves as
input for the final output layer, we can always find an equivalent neural net which does not have
any hidden layer as seen in the example above.

Kernel Method (20%)

Suppose we have six training points from two classes as in Figure (a). Note that we have four points
from class 1: (0.2, 0.4), (0.4, 0.8), (0.4, 0.2), (0.8, 0.4) and two points from class 2: (0.4, 0.4), (0.8, 0.8).
Unfortunately, the points in Figure (a) cannot be separated by a linear classifier. The kernel trick is to
find a mapping of x to some feature vector (x) such that there is a function K called kernel which
satisfies K(x, x0 ) = (x)T (x0 ). And we expect the points of (x) to be linearly separable in the feature
space. Here, we consider the following normalized kernel:
K(x, x0 ) =

xT x0
||xT ||||x0 ||

1. (5%) What is the feature vector (x) corresponding to this kernel? Draw (x) for each training
point x in Figure (b), and specify from which point it is mapped.

(x) =

x
||x||

2. (5%) You now see that the feature vectors are linearly separable in the feature space. The
maximum-margin decision boundary in the feature space will be a line in R2 , which can be
written as w1 x + w2 y + c = 0. What are the values of the coefficients w1 and w2 ? (Hint: you
dont need to compute them.)
(sol.)
(w1 , w2 ) = (1, 1)
3. (3%) Circle the points corresponding to support vectors in Figure (b).
4. (7%) Draw the decision boundary in the original input space resulting from the normalized linear
kernel in Figure (a). Briefly explain your answer.

VC Dimension and PAC Learning(10%)

The VC dimension, V C(H), of hypothesis space H defined over instance space X is the size of the
largest largest number of points (in some configuration) that can be shattered by H. Suppose with
probability (1 ), a PAC learner outputs a hypothesis within error  of the best possible hypothesis in
H. It can be shown that the lower bound on the number of training examples m sufficient for successful
learning, stated in terms of V C(H) is
1
m (4 log2 (2/) + 8V C(H) log2 (13/)).

Consider a learning problem in which X = R is the set of real numbers, and the hypothesis space is the
set of intervals H = {(a < x < b)|a, b R}. Note that the hypothesis labels points inside the interval
as positive, and negative otherwise.
1. (5%) What is the VC dimension of H?
(sol.) V C(H) = 2. Suppose we have two points x1 and x2 , and x1 < x2 . They can always be
shattered by H, no matter how they are labled.
(a) if x1 positive and x2 negative, choose a < x1 < b < x2 ;
(b) if x1 negative and x2 positive, choose x1 < a < x2 < b;
(c) if both x1 and x2 positive, choose a < x1 < x2 < b;
(d) if both x1 and x2 negative, choose a < b < x1 < x2 ;
However, if we have three points x1 < x2 < x3 and if they are labeled as x1 (positive) x2 (negative)
and x3 (positive), then they cannot be shattered by H.
2. (5%) What is the probability that a hypothesis consistent with m examples will have error at least
?
(sol.) Use the above result. Substitute V C(H) = 2 into the inequality
1
m (4 log2 (2/) + 8 2 log2 (13/)),

we have
m 4 log2 (2/) + 8 2 log2 (13/)
m 16 log2 (13/) 4 log2 (2/)
2m/4
2/
(13/)4
13 4

2m/41

Logistic Regression (10%)

We consider the following models of logistic regression for a binary classification with a sigmoid function
g(z) =

1
:
1+ez

Model 1: P (Y = 1 | X, w1 , w2 ) = g(w1 X1 + w2 X2 )
Model 2: P (Y = 1 | X, w1 , w2 ) = g(w0 + w1 X1 + w2 X2 )
We have three training examples:
x(1) = [1, 1]T

x(2) = [1, 0]T

x(3) = [0, 0]T

y (1) = 1

y (2) = 1

y (3) = 1

1. (5%) Does it matter how the third example is labeled in Model 1? i.e., would the learned value
of w = (w1 , w2 ) be different if we change the label of the third example to -1? Does it matter in
Model 2? Briefly explain your answer. (Hint: think of the decision boundary on 2D plane.)
(sol.) It does not matter in Model 1 because x(3) = (0, 0) makes w1 x1 + w2 x2 always zero and
hence the likelihood of the model does not depend on the value of w. But it does matter in Model
2.
2. (5%) Now, suppose we train the logistic regression model (Model 2) based on the n training
examples x(1) , . . . , x(n) and labels y (1) , . . . , y (n) by maximizing the penalized log-likelihood of the
labels:
X
i

log P (y (i) | x(i) , w)

||w||2 =
log g(y (i) wT x(i) ) ||w||2
2
2
i

For large (strong regularization), the log-likelihood terms will behave as linear functions of w.
1
log g(y (i) wT x(i) )) y (i) wT x(i)
2
Express the penalized log-likelihood using this approximation (with Model 1), and derive the
in terms of and training data {x(i) , y (i) }. Based on this, explain how w
expression for MLE w
(i)

(i)

behaves as increases. (We assume each x(i) = (x1 , x2 )T and y (i) is either 1 or -1 )
(sol.)
log l(w)

X1
2
X
1

y (i) wT x(i)

log l(w)
w1
2

(i)

y (i) x1 w1 = 0

1X

log l(w)
w2
2

(i)

y (i) x2 w2 = 0

1 X (i) (i)
w=
y x
2
i

10

||w||2
2

11

12