Scribd
Upload
85 views4 pages

Learning From Data Solutions To Selected Exercises: N N M J J N 2

This document contains solutions to selected exercises on learning from data. It includes: 1) Derivations of the normal equations for linear regression with and without regularization. 2) Applications of probability rules like the sum and product rule to calculate probabilities of events. 3) Proofs that the variance of the sum of independent random variables is equal to the sum of their individual variances, and that the expected value of the product of independent random variables is equal to the product of their expected values. 4) Derivations of properties of expectations, variances and covariances.

Uploaded by

righthearted
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
85 views4 pages

Learning From Data Solutions To Selected Exercises: N N M J J N 2

This document contains solutions to selected exercises on learning from data. It includes: 1) Derivations of the normal equations for linear regression with and without regularization. 2) Applications of probability rules like the sum and product rule to calculate probabilities of events. 3) Proofs that the variance of the sum of independent random variables is equal to the sum of their individual variances, and that the expected value of the product of independent random variables is equal to the product of their expected values. 4) Derivations of properties of expectations, variances and covariances.

Uploaded by

righthearted
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Learning from Data

Solutions to Selected Exercises

1.1 The error function is given by

2
M
N
X
X
1
t n
wj xjn
E(w) =
2 n=1
j=0
To obtain the normal equations we take the partial derivatives with
respect to the weights and equate to zero. Application of the chain rule
gives

N
M
X
X
E
t n
=
wj xjn xin
wi
n=1
j=0
Equating this expression to zero gives

N
N
M X
N
M
X
X
X
X
t n

wj xjn xin =
tn xin +
xjn xin wj = 0
n=1

n=1

j=0

j=0 n=1

So we get
M X
N
X

(xn )i+j wj =

j=0 n=1

N
X

tn (xn )i

i = 0, . . . , M

n=1

1.2 The error function is given by


E(w) =

N
M
X 2
1X
w
(y(xn , w) tn )2 +
2 n=1
2 j=0 j

To obtain the normal equations we take the partial derivatives with


respect to the weights and equate to zero. Application of the chain rule
gives


N
X
E
y(xn , w)
=
(y(xn , w) tn )
+ wi
wi
wi
n=1

This gives the normal equations


N
X




N
X
y(xn , w)
y(xn , w)
y(xn , w)
+wi =
tn
wi
wi
n=1
n=1


Plugging in y(xn , w) =
M X
N
X

PM

j=0

i = 0, . . . , M

wj xjn we get

(xn )i+j wj + wi =

j=0 n=1

N
X

tn (xn )i

i = 0, . . . , M

n=1

1.3 Use the sum and product rules of probability.


Probability of drawing an apple:
X
p(a) =
p(a, box)
box
X
=
p(a|box)p(box)
box
= p(a|r)p(r) + p(a|b)p(b) + p(a|g)p(g)
= 0.3 0.2 + 0.5 0.2 + 0.3 0.6 = 0.34
Probability of green box given orange
p(o|g)p(g)
p(g, o)
=P
p(o)
box p(o|box)p(box)
0.18
=
= 0.5
0.36

p(g|o) =

1.5
var[f (x)] = E[(f (x) E[f (x)])2 ]
2

(1.38;expand)
2

= E[f (x) 2f (x)E[f (x)] + E[f (x)] ]


(push expectation inward)
= E[f (x)2 ] 2E[f (x)]E[f (x)] + E[f (x)]2
= E[f (x)2 ] E[f (x)]2

(1.39)

The important thing to notice is that E[f (x)] is a constant, and so


E[2f (x)E[f (x)]] = 2E[f (x)]E[f (x)]

and

E[E[f (x)]2 ] = E[f (x)]2

1.6
cov[x, y] = E[xy] E[x]E[y]

(1.41)

Well show that E[xy] = E[x]E[y] if x and y are independent.


XX
E[xy]
xy p(x, y)
x

XX
x

xy p(x)p(y)

x p(x)

y p(y) = E[x]E[y]

1.10 We dont need independence for the first one:


XX
E[x + z]
(x + z) p(x, z)
x

XX
x

x p(x, z) +

XX
z

z p(x, z)

X X
X X
p(x, z)
z
p(x, z) +
x

x p(x) +

z p(z) = E[x] + E[z]

In exercise 1.5 we have shown that var[x + z] = E[(x + z)2 ] E[x + z]2 .
Since E[x + z] = E[x] + E[z] as we have shown above, we get
var[x + z] = E[(x + z)2 ] E[x]2 2E[x]E[z] E[z]2
Left to determine E[(x + z)2 ]:
XX
(x + z)2 p(x, z)
E[(x + z)2 ]
x

XX

XX
x

(independence/expand)

(x2 + 2xz + z 2 ) p(x)p(z)

x2 p(x)p(z) +

x p(x)

XX
x

p(z) + 2

2xz p(x)p(z) +

X
x

x p(x)

XX
z

X
z

z p(z) +

z 2 p(x)p(z)

z 2 p(z)

= E[x2 ] + 2E[x]E[z] + E[z 2 ]


So finally we get
var[x + z] = E[x2 ] + 2E[x]E[z] + E[z 2 ] E[x]2 2E[x]E[z] E[z]2
= (E[x2 ] E[x]2 ) + (E[z 2 ] E[z]2 ) = var[x] + var[z]

X
x

p(x)

1.13
#
N
N
1 X
1 X
2
E
(xn ) =
E[(xn )2 ]
N n=1
N n=1
"

N
1 X
1
var(xn ) = N 2 = 2
N n=1
N

You might also like