0% found this document useful (0 votes)
75 views10 pagesMidterm 2002
Machine Learning Notes
Uploaded by
Muhammad MurtazaCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
0% found this document useful (0 votes)
75 views10 pagesMidterm 2002
Machine Learning Notes
Uploaded by
Muhammad MurtazaCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
/ 10
15-781 Midterm, Fall 2002
YOUR ANDREW USERID IN CAPITAL LETTERS:
© There are 5 questions.
© Questions 1-5 are worth 20 points each.
¢ The maximum possible total score is 100,
© Unless otherwise stated there is no need to show your working. 1
Decision Trees (20 points)
Master Yoda is concerned about the number of Jedi apprentices that have turned to the
Dark Side, so he’s decided to train a decision tree on some historical data to help identify
problem cases in the future. ‘I'he following table summarizes whether or not each of 12
initiates turned to the Dark Side based on their age when their Jedi training began, whether
or not they completed their training, their general disposition, and their species.
Dark Side | Age Started Training | Completed Training | Disposition | _Sp
0 3 1 Happy Human
0 9 I Happy Gungan
0 6 0 Happy Wookie
0 6 T Sad | Mon Calamari
0 7 0 Sad Human
0 8 T ‘Angry Tuman
0 5 I Angry Ewok
I 9 0 Happy Ewok
1 & a Sad Hinman
1 8 0 Sad Human
I 6 0 Angry Wookie
1 7 0 Angry Mon Calamari
(3 points) What is the initial entropy of Dark Sid
(8 points) Which attribute would the decisi
for the root of the tree?
on-tree building algorithm choose to use
(8 points) What is the information gain of the attribute you chose to split on in the
previous question?
(8 points) Draw the full decision tree that would be learned for this data (with no
pruning). (Hint: The tree will have no more than three splits. The correct split at
each point should be clear from just the groups it splits the data into, without having
to actually compute the information gain for each possible split.) ‘e)
f)
(2 points) Consider the possibility that the input data above is noisy and not completely
accurate, so that the decision tree you learned may not accurately reflect the function
you want to learn. If you were to evaluate the three initiates represented by the data
points below, on which one would you be most confident of your prediction, and why’?
Name_| Age Started Training | Completed Training | Disposition | __ Species
Ardath 5 0 Angry Human
Barbar 8 0 Angry Gungan
Caldar 8 0 Happy | Mon Calamari
(8 points) Assume we train a decision tree to predict Z from A, B, and C using the
following data (with no pruning):
Z[A[BIC
o[oo{o
ofofofi
ofofoti
ofoli{o
ofolifa
Tpo}i}t
oft foto
1j1jo}.
1jijiijo
1jijijo
o;ifiti
dyajiie
What would be the training set error for this dataset? Express your answer as the
number of records out of 12 that would be misclassified.
(3 points) Consider a dec
mn tree built from an arbitrary set of data. If the output
is diserete-valued and can take on k different possible values, what is the maximum
training set error (expressed as a fraction) that any data set could possibly have? 2 Probability and Bayes Classifiers (20 points)
Lhis figure illustrates a simple class of probability density functions over pairs of real-valued
variables. We call it the Rectangle PDF.
re
| pean. |
(sy) © Rect, tes iss) ee |
means Par
P(x,y) i Ifo Se Saye and Yo Sy Syn} | seer:
= 0 otherwise Y fase) oe
(2 points) Assuming (x,y) ~ Rect(0,0,0.5,2) (as shown in |
(a) the diagram to the right), compute the value of the density _y=1.0 |
r= i= 4) |
0.5
yo
xel/2 xel xslt
(b) (8 points) Under the same assumptions, compute the density p(y =
(c) (8 points) Under the same assumptions, compute the density p(x = +)
(a) (3 points) Under the same assumptions, compute the density pl = "y= ‘e)
)
(s)
Maximum Likelihood Estimation of Rectangles
drawn
Assume we have R datapoints (2,41), (2, yo)-(tp. ye) where each datapoint i
independently from Rect( to, thos.This Yui)
Suppose we want to find the MLE parameters (jo, to, tnisYai) that maximize the
likelihood of the datapoints. It turns out (no proof given or required) that these MLE
values define the bounding box of the datapoints:
ming
mingys
= marry
margyy
Now, suppose that we use the rectangle distribution as the density estimator for each
class of a Bayes Classifier that we're about to learn. ‘Ihe data is
6
5 c
=p [cas] A
1j4] A 4 e
3}1| A
2/2, B 3 2
EN] B
3(2[ Cc fa . of c
405l © A
5]2[ c yu .
[5] C
0
0 12 3 4 5 6
x—>
Assuming we use the Rectangle Bayes Classifier learned from the data, what value will
the classifier give for:
(3 points) P(Class = Ax = 1.5,
(8 points) P(Class = Alr = 2.5, y
(8 points) P(Class = Aly = 5; 3 Cross Validation (20 points)
Suppose we are learning a classifier with binary output values
real-valued input X. Here is our data:
and Y=1. ‘There is one
[v
0
2/0
1 oe ew
3 [0 4
6a |
711 yo| s+ eee .
St PO rt
eran Ko} o 12 3 45 6 7 8 9 10
9 {1 x—>
10 [1
Assume we will learn a decision tree on this data. Assume that when the decision tree
splits on the real valued attribute x, it puts the split threshold halfway between the attributes
that surround the split. For example, using information gain as the splitting criterion, the
decision tree would initially choose to split at x = 5, which is halfway between the x = 4
and x = 6 datapoints
Let Algorithm D12 be the method of learning a decision tree with only two leaf nodes
(ie. only one split).
Let Algorithm DT* be the method of learning a decision tree fully with no pruning.
(a) (5 points) What will be the training set error of DT2 on onr data? In this part, and
all future parts, you can express your answer as the number of misclassifications out
of 10.
(b) (5 points) What will be the leave-one-out-cro
alidation error of DT2 on our data?
{c) (5 points) What will be the training set error of DT* on our data?
(a) (5 points) What will be the leave-one-ont-cross-validation error of DT* on our data? 4
Computational learning theory (20 points)
‘True or false: For a-d, if false, give a counter example. If true, give a 1 sentence justification.
(a)
(3 points) Within the setting of the PAC model it is impossible to assure with proba-
bility 1 that the concept will be learned perfectly (ie., with true error=0), regardless
of how many training examples are provided.
(3 points) If the Halving Algorithm has made exactly |logo|#|| mis
tains the target concept, then it must have learned a hypothesis with true error:
regardless of what training sequence we presented and what hypothesis space H it
considered
(3 points) It is impossible for the Halving Algorithm to lean any concept without it
making at least VC(H) mistakes, regardless of what training sequence we present, and
what hypothesis space H it considers.
(3 points) The PAC bounds make a worst case assumption about the probability di
tribution over the instances X, but it is possible to learn from fewer examples for some
distributions over }
a ‘onsider the class of concepts H2p defined by conjunctions
of wwo arbitrary perceptrons. ore preciseiy, each nypoti
esis h(x) : X + {0,1} in H2p is of the form h(x) = pi(z)
AND po(2), where pi(2x) and p2(x) are any two-input pe
coptrons. The fignre illnstrates one sich possi
two dimensions
Je clasefier in
{e) (4 points) Draw a set of three points in the plane that cannot be shattered by H2p.
‘f) (4 points) What is the VC dimension of H2p? (Partial credit will be given if you can
bound it, so show your reasoning!) Regression and neural networks (20 points)
(8 points) Derive a gradient descent training algorithm that minimizes the sum of
squared errors for a variant of a perceptron where the output o of the unit depends on
its inputs 2; as follows:
= Wy + WT + Wat + Woe. + WT +. + Wyo, + WyTh
Give your answer in the form w; + w; +... for 1
