10-705/36-705 Intermediate Statistics

Larry Wasserman http://www.stat.cmu.edu/~larry/=stat705/ Fall 2011

Week
August 29 September 5 September 12 September 19 September 26 October 3 October 10 October 17 October 24 October 31 November 7 November 14 November 21 November 28 December 5

Class I Syllabus
Review No Class Convergence Convergence Addendum Likelihood Minimax Summary Asymptotics Testing Testing Nonparametric Test III The Bootstrap No Class Prediction Multiple Testing

Class II
Review, Inequalities OP Convergence Suciency Point Estimation Asymptotics Review Testing Condence Intervals Nonparametric No Class Bayesian Inference No Class Prediction Causation

Day III

Class IV
Inequalities VC Theory Test I Suciency Minimax Theory Asymptotics Test II Mid-semester Break Condence Intervals Review The Bootstrap Bayesian Inference No Class Model Selection Individual Sequences

HW HW HW HW HW

1 2 3 4 5

[soln] [soln] [soln] [soln] [soln]

HW 6 [soln] HW 7 [soln] HW 8 [soln] HW 9 [soln] HW 10 [soln]

Practice Final

10-705/36-705: Intermediate Statistics, Fall 2010

Professor Ofce Email Phone Ofce hours Class Time Location TAs Larry Wasserman Baker Hall 228 A [email protected] 268-8727 Mondays, 1:30-2:30 Mon-Wed-Fri 12:30 - 1:20 GHC 4307 Wanjie Wang and Xiaolin Yang Website http://www.stat.cmu.edu/larry/=stat705

Objective This course will cover the fundamentals of theoretical statistics. Topics include: point and interval estimation, hypothesis testing, data reduction, convergence concepts, Bayesian inference, nonparametric statistics and bootstrap resampling. We will cover Chapters 5 10 from Casella and Berger plus some supplementary material. This course is excellent preparation for advanced work in Statistics and Machine Learning. Textbook Casella, G. and Berger, R. L. (2002). Statistical Inference, 2nd ed. Background I assume that you are familiar with the material in Chapters 1 - 4 of Casella and Berger. Other Recommended Texts Wasserman, L. (2004). All of Statistics: A concise course in statistical inference. Bickel, P. J. and Doksum, K. A. (1977). Mathematical Statistics. Rice, J. A. (1977). Mathematical Statistics and Data Analysis, Second Edition. Grading 10% : Test I (Sept. 16) on the material of Chapters 14 20% : Test II (October 14) 20% : Test III (November 7) 30% : Final Exam (Date set by the University) 20% : Homework

Exams All exams are closed book. Do NOT buy a plane ticket until the nal exam has been scheduled. Homework Homework assigments will be posted on the web. Hand in homework to Mari Alice Mcshane, 228 Baker Hall by 3 pm Thursday. No late homework. Reading and Class Notes Class notes will be posted on the web regularly. Bring a copy to class. The notes are not meant to be a substitute for the book and hence are generally quite terse. Read both the notes and the text before lecture. Sometimes I will cover topics from other sources. Group Work You are encouraged to work with others on the homework. But write-up your nal solutions on your own.

Course Outline
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. Quick Review of Chapters 1-4 Inequalities Vapnik-Chervonenkis Theory Convergence Sufciency Likelihood Point Estimation Minimax Theory Asymptotics Robustness Hypothesis Testing Condence Intervals Nonparametric Inference Prediction and Classication The Bootstrap Bayesian Inference Markov Chain Monte Carlo Model Selection

Lecture Notes 1 Quick Review of Basic Probability (Casella and Berger Chapters 1-4) 1 Probability Review

Chapters 1-4 are a review. I will assume you have read and understood Chapters 1-4. Let us recall some of the key ideas.

1.1

Random Variables
P (X A) = P ({ : X() A})

A random variable is a map X from a probability space to R. We write

and we write X P to mean that X has distribution P . The cumulative distribution function (cdf ) of X is FX (x) = F (x) = P (X x). If X is discrete, its probability mass function (pmf ) is pX (x) = p(x) = P (X = x). If X is continuous, then its probability density function function (pdf ) satises P (X A) = pX (x)dx =
A A

p(x)dx

and pX (x) = p(x) = F (x). The following are all equivalent: X P, X F, X p.

Suppose that X P and Y Q. We say that X and Y have the same distribution if P (X A) = Q(Y A) for all A. In other words, P = Q. In that case we say that X and Y are equal in d d distribution and we write X = Y . It can be shown that X = Y if and only if FX (t) = FY (t) for all t.

1.2

Expected Values

The mean or expected value of g(X) is E (g(X)) = g(x)dF (x) = g(x)dP (x) = g(x)p(x)dx if X is continuous if X is discrete. j g(xj )p(xj )

Recall that: 1. E(
k j=1 cj gj (X))

k j=1 cj E(gj (X)).

2. If X1 , . . . , Xn are independent then

n

E
i=1

Xi

=
i

E (Xi ) .

3. We often write = E(X). 4. 2 = Var (X) = E ((X )2 ) is the Variance. 5. Var (X) = E (X 2 ) 2 . 6. If X1 , . . . , Xn are independent then
n

Var
i=1

ai X i

=
i

a2 Var (Xi ) . i

7. The covariance is Cov(X, Y ) = E((X x )(Y y )) = E(XY ) X Y and the correlation is (X, Y ) = Cov(X, Y )/x y . Recall that 1 (X, Y ) 1. The conditional expectation of Y given X is the random variable E(Y |X) whose value, when X = x is E(Y |X = x) = y p(y|x)dy where p(y|x) = p(x, y)/p(x). The Law of Total Expectation or Law of Iterated Expectation: E(Y ) = E E(Y |X) = The Law of Total Variance is Var(Y ) = Var E(Y |X) + E Var(Y |X) . The nth moment is E (X n ) and the nth central moment is E ((X )n ). The moment generating function (mgf ) is MX (t) = E etX . Then, MX (t)|t=0 = E (X n ) . d If MX (t) = MY (t) for all t in an interval around 0 then X = Y . 2
(n)

E(Y |X = x)pX (x)dx.

1.3

Exponential Families
k

A family of distributions {p(x; ) : } is called an exponential family if p(x; ) = h(x)c() exp

i=1

wi ()ti (x) .

e x 1 = e exp{log x}. x! x! Example 2 X U (0, ) is not an exponential family. The density is p(x) = P (X = x) = 1 pX (x) = I(0,) (x) where IA (x) = 1 if x A and 0 otherwise.

Example 1 X Poisson() is exponential family since

We can rewrite an exponential family in terms of a natural parameterization. For k = 1 we have p(x; ) = h(x) exp{t(x) A()} where A() = log For example a Poisson can be written as p(x; ) = exp{x e }/x! where the natural parameter is = log . Let X have an exponential family distribution. Then E (t(X)) = A (), Var (t(X)) = A (). h(x) exp{t(x)}dx.

Practice Problem: Prove the above result.

1.4

Transformations
FY (y) = P(Y y) = P(g(X) y) =

Let Y = g(X). Then pX (x)dx

A(y)

where Then pY (y) = FY (y). If g is monotonic, then Ay = {x : g(x) y}. dh(y) dy

pY (y) = pX (h(y)) where h = g 1 . 3

Example 3 Let pX (x) = ex for x > 0. Hence FX (x) = 1 ex . Let Y = g(X) = log X. Then FY (y) = P (Y y) = P (log(X) y) y = P (X ey ) = FX (ey ) = 1 ee and pY (y) = ey ee for y R. Example 4 Practice problem. Let X be uniform on (1, 2) and let Y = X 2 . Find the density of Y . Let Z = g(X, Y ). For exampe, Z = X + Y or Z = X/Y . Then we nd the pdf of Z as follows: 1. For each z, nd the set Az = {(x, y) : g(x, y) z}. 2. Find the CDF FZ (z) = P (Z z) = P (g(X, Y ) z) = P ({(x, y) : g(x, y) z}) = 3. The pdf is pZ (z) = FZ (z). Example 5 Practice problem. Let (X, Y ) be uniform on the unit square. Let Z = X/Y . Find the density of Z. pX,Y (x, y)dxdy.
Az
y

1.5

Independence
P(X A, Y B) = P(X A)P(Y B)

Recall that X and Y are independent if and only if

for all A and B. Theorem 6 Let (X, Y ) be a bivariate random vector with pX,Y (x, y). X and Y are independent i pX,Y (x, y) = pX (x)pY (y). X1 , . . . , Xn are independent if and only if
n

P(X1 A1 , . . . , Xn An ) =

i=1

P(Xi Ai ).

Thus, pX1 ,...,Xn (x1 , . . . , xn ) = n pXi (xi ). i=1 If X1 , . . . , Xn are independent and identically distributed we say they are iid (or that they are a random sample) and we write X1 , . . . , X n P or X1 , . . . , X n F 4 or X1 , . . . , Xn p.

1.6

Important Distributions
1 2 2 p(x) = e(x) /(2 ) . 2 1 (2)d/2 || 1 exp (x )T 1 (x ) . 2

X N (, 2 ) if If X Rd then X N (, ) if p(x) =

X 2 if X = p

p j=1

2 Zj where Z1 , . . . , Zp N (0, 1).

X Bernoulli() if P(X = 1) = and P(X = 0) = 1 and hence p(x) = x (1 )1x X Binomial() if p(x) = P(X = x) = n x (1 )nx x x {0, . . . , n}. x = 0, 1.

X Uniform(0, ) if p(x) = I(0 x )/.

1.7

Sample Mean and Variance

X= 1 n Xi ,
i

The sample mean is

and the sample variance is S2 =

1 n1

(Xi X)2 .

Let X1 , . . . , Xn be iid with = E(Xi ) = and 2 = Var(Xi ) = 2 . Then E(X) = , Var(X) = 2 , n E(S 2 ) = 2 .

Theorem 7 If X1 , . . . , Xn N (, 2 ) then (a) X N (, ) n (b)

(n1)S 2 2
2

2 n1

(c) X and S 2 are independent 5

1.8

Delta Method
Y N (g(), 2 (g ())2 ).

If X N (, 2 ), Y = g(X) and 2 is small then

To see this, note that Y = g(X) = g() + (X )g () + (X )2 g () 2

for some . Now E((X )2 ) = 2 which we are assuming is small and so Y = g(X) g() + (X )g (). Thus E(Y ) g(), Hence, g(X) N g(), (g ())2 2 . Var(Y ) (g ())2 2 .

Appendix: Useful Facts

Facts about sums

n i=1

i=

n(n+1) . 2 n(n+1)(2n+1) . 6 a , 1r

n 2 i=1 i

Geometric series: a + ar + ar2 + . . . =

for 0 < r < 1.

a(1rn ) . 1r

Partial Geometric series a + ar + ar2 + . . . + arn1 = Binomial Theorem

n x=0

n x a = (1 + a)n , x

x=0

n x nx a b = (a + b)n . x

Hypergeometric identity
x=0

a x

b nx

a+b . n

Common Distributions
Discrete
Uniform X U (1, . . . , N ) X takes values x = 1, 2, . . . , N P (X = x) = 1/N E (X) = E (X 2 ) = Binomial X Bin(n, p) X takes values x = 0, 1, . . . , n P (X = x) = Hypergeometric X Hypergeometric(N, M, K) P (X = x) = Geometric X Geom(p) P (X = x) = (1 p)x1 p, x = 1, 2, . . . E (X) = Poisson X Poisson() P (X = x) =
e x x! x n x x

xP (X = x) = x2 P (X = x) =

1 xN = x

1 N (N +1) N 2

(N +1) 2

1 x2 N =

1 N (N +1)(2N +1) N 6

px (1 p)nx

(M )(N M ) x Kx N (K )

x(1 p)x1 = p

d x dp

1 ((1 p)x ) = p pp2 = p .

x = 0, 1, 2, . . .

E (X) = Var (X) = MX (t) =

tx e x x=0 e x!

=e

x=0

(et )
x!

= e ee = e(e 1) .

 E (X) = et e(e 1) |t=0 = . Use mgf to show: if X1 Poisson(1 ), X2 Poisson(2 ), independent then Y = X1 + X2 Poisson(1 + 2 ).

Continuous Distributions
Normal X N (, 2 ) p(x) =
1 2 1 exp{ 22 (x )2 }, x R

mgf MX (t) = exp{t + 2 t2 /2}. E (X) = Var (X) = 2 . e.g., If Z N (0, 1) and X = + Z, then X N (, 2 ). Show this... Proof. MX (t) = E etX = E et(+Z) = et E etZ = et MZ (t) = et e(t) which is the mgf of a N (, 2 ). Alternative proof: FX (x) = P (X x) = P ( + Z x) = P = FZ x x 1
2
2 /2

= et+t

2 2 /2

pX (x) = FX (x) = pZ

1 1 = exp 2 2 1 1 exp = 2 2 which is the pdf of a N (, 2 ).

x x

1
2

Gamma X (, ). pX (x) = () = 2 = p
1 x1 ex/ , ()

x a positive real.

1 1 x/ x e dx. 0

Important statistical distribution: 2 = ( p , 2). p 2

p i=1

Xi2 , where Xi N (0, 1), iid.

Exponential X exponen()
1 pX (x) = ex/ , x a positive real.

exponen() = (1, ). e.g., Used to model waiting time of a Poisson Process. Suppose N is the number of phone calls in 1 hour and N P oisson(). Let T be the time between consecutive phone calls, then T exponen(1/) and E (T ) = (1/). If X1 , . . . , Xn are iid exponen(), then
i

Xi (n, ).

Memoryless Property: If X exponen(), then P (X > t + s|X > t) = P (X > s).

Linear Regression
Model the response (Y ) as a linear function of the parameters and covariates (x) plus random error ( ). Yi = (x, ) + i where (x, ) = X = 0 + 1 x1 + 2 x2 + . . . + k xk .

Generalized Linear Model

Model the natural parameters as linear functions of the the covariates. Example: Logistic Regression. e x P (Y = 1|X = x) = . 1 + e T x In other words, Y |X = x Bin(n, p(x)) and (x) = T x where (x) = log p(x) 1 p(x)
T

Logistic Regression consists of modelling the natural parameter, which is called the log odds ratio, as a linear function of covariates.

Location and Scale Families, CB 3.5

Let p(x) be a pdf. Location family : {p(x|) = p(x ) : R} Scale family : Location Scale family : p(x|) = 1 x f : >0 1 f x : R, > 0

p(x|, ) =

(1) Location family. Shifts the pdf. e.g., Uniform with p(x) = 1 on (0, 1) and p(x ) = 1 on (, + 1). e.g., Normal with standard pdf the density of a N (0, 1) and location family pdf N (, 1). (2) Scale family. Stretches the pdf. e.g., Normal with standard pdf the density of a N (0, 1) and scale family pdf N (0, 2 ). (3) Location-Scale family. Stretches and shifts the pdf. e.g., Normal with standard pdf the density of a N (0, 1) and location-scale family pdf 1 N (, 2 ), i.e., p( x ) .

10

Multinomial Distribution
The multivariate version of a Binomial is called a Multinomial. Consider drawing a ball from an urn with has balls with k dierent colors labeled color 1, color 2, . . . , color k. Let p = (p1 , p2 , . . . , pk ) where j pj = 1 and pj is the probability of drawing color j. Draw n balls from the urn (independently and with replacement) and let X = (X1 , X2 , . . . , Xk ) be the count of the number of balls of each color drawn. We say that X has a Multinomial (n, p) distribution. The pdf is p(x) = n p x1 . . . p xk . k x1 , . . . , x k 1

Multivariate Normal Distribution

We now dene the multivariate normal distribution and derive its basic properties. We want to allow the possibility of multivariate normal distributions whose covariance matrix is not necessarily positive denite. Therefore, we cannot dene the distribution by its density function. Instead we dene the distribution by its moment generating function. (The reader may wonder how a random vector can have a moment generating function if it has no density function. However, the moment generating function can be dened using more general types of integration. In this book, we assume that such a denition is possible but nd the moment generating function by elementary means.) We nd the density function for the case of positive denite covariance matrix in Theorem 5. Lemma 8 (a). Let X = AY + b Then MX (t) = exp (b t)MY (A t). (b). Let c be a constant. Let Z = cY. Then MZ (t) = MY (ct). (c). Let Y= Then Y1 , Y2 t= t1 t2

t1 MY1 (t1 ) = MY . 0

(d). Y1 and Y2 are independent if and only if t1 MY t2 = MY t1 0 MY . 0 t2

11

We start with Z1 , . . . , Zn independent random variables such that Zi N1 (0, 1). Let Z = (Z1 , . . . , Zn ) . Then E(Z) = 0, cov(Z) = I, MZ (t) = exp tt t2 i = exp . 2 2 (1)

Let be an n 1 vector and A an n n matrix. Let Y = AZ + . Then E(Y) = cov(Y) = AA . (2)

Let = AA . We now show that the distribution of Y depends only on and . The moment generating function MY (t) is given by MY (t) = exp( t)MZ (A t) = exp t + t (A A)t 2 = exp t + t t 2 .

With this motivation in mind, let be an n1 vector, and let be a nonnegative denite nn matrix. Then we say that the n-dimensional random vector Y has an n-dimensional normal distribution with mean vector , and covariance matrix , if Y has moment generating function t t MY (t) = exp t + . (3) 2 We write Y Nn (, ). The following theorem summarizes some elementary facts about multivariate normal distributions. Theorem 9 (a). If Y Nn (, ), then E(Y) = , cov(Y) = . (b). If Y Nn (, ), c is a scalar, then cY Nn (c, c2 ). (c). Let Y Nn (, ). If A is p n, b is p 1, then AY + b Np (A + b, AA ). (d). Let be any n 1 vector, and let be any n n nonnegative denite matrix. Then there exists Y such that Y Nn (, ). Proof. (a). This follows directly from (2) above. (b) and (c). Homework. (d). Let Z1 , . . . , Zn be independent, Zi N (0, 1). Let Z = (Z1 , . . . , Zn ) . It is easily veried that Z Nn (0, I). Let Y = 1/2 Z + . By part b, above, Y Nn (1/2 0 + , ). We have now shown that the family of normal distributions is preserved under linear operations on the random vectors. We now show that it is preserved under taking marginal and conditional distributions.

12

Theorem 10 Suppose that Y Nn (, ). Let Y= Y1 , = Y2 1 , = 2 11 12 . 21 22

where Y1 and 1 are p 1, and 11 is p p. (a). Y1 Np (1 , 11 ), Y2 Nnp (2 , 22 ). (b). Y1 and Y2 are independent if and only if 12 = 0. (c). If 22 > 0, then the condition distribution of Y1 given Y2 is Y1 |Y2 Np (1 + 12 1 (Y2 2 ), 11 12 1 21 ). 22 22 Proof. (a). Let t = (t1 , t2 ) where t1 is p 1. The joint moment generating function of Y1 and Y2 is 1 MY (t) = exp(1 t1 + 2 t2 + (t1 11 t1 + t1 12 t2 + t2 21 t1 + t2 22 t2 )). 2 Therefore, t1 MY 0 1 0 = exp(1 t1 + t1 11 t1 ), MY 2 t2 1 = exp(2 t2 + t2 22 t2 ). 2

By Lemma 1c, we see that Y1 Np (1 , 11 ), Y2 Nnp (2 , 22 ). (b). We note that t1 0 MY (t) = MY MY 0 t2 if and only if t1 12 t2 + t2 21 t1 = 0. Since is symmetric and t2 21 t1 is a scalar, we see that t2 21 t1 = t1 12 t2 . Finally, t 12 t2 = 0 for all t1 Rp , t2 Rnp if and only if 12 = 0, and the result follows from Lemma 1d. (c). We rst nd the joint distribution of X = Y1 12 1 Y2 and Y2 . 22 X Y2 X Y2 = I 12 1 22 0 I Y1 Y2

Therefore, by Theorem 2c, the joint distribution of X and Y2 is Nn 1 12 1 2 11 12 1 21 0 22 22 , 2 0 22

and hence X and Y2 are independent. Therefore, the conditional distribution of X given Y2 is the same as the marginal distribution of X, X|Y2 Np (1 12 1 2 , 11 12 1 21 ). 22 22 13

Since Y2 is just a constant in the conditional distribution of X given Y2 we have, by Theorem 2c, that the conditional distribution of Y1 = X + 12 1 Y2 given Y2 is 22 Y1 |Y2 Np (1 12 1 2 + 12 1 Y2 , 11 12 1 21 ) 22 22 22 Note that we need 22 > 0 in part c so that 1 exists. 22 Lemma 11 Let Y Nn (, 2 I), where Y = (Y1 , . . . , Yn ), = (1 , . . . , n ) and 2 > 0 is a scalar. Then the Yi are independent, Yi N1 (, 2 ) and ||Y||2 YY = 2 2 n 2 2 .

Proof. Let Yi be independent, Yi N1 (i , 2 ). The joint moment generating function of the Yi is n 1 1 MY (t) = (exp(i ti + 2 t2 )) = exp( t + 2 t t) i 2 2 i=1 which is the moment generating function of a random vector that is normally distributed with mean vector and covariance matrix 2 I. Finally, Y Y = Yi2 , = 2 and i Yi / N1 (i /, 1). Therefore Y Y/ 2 2 ( / 2 ) by the denition of the noncentral 2 n distribution. We are now ready to derive the nonsingular normal density function. Theorem 12 Let Y Nn (, ), with > 0. Then Y has density function pY (y) = 1 (2)n/2 ||1/2 1 exp (y ) 1 (y ) . 2

Proof. We could derive this by nding the moment generating function of this density and showing that it satised (3). We would also have to show that this function is a density function. We can avoid all that by starting with a random vector whose distribution we know. Let Z Nn (0, I). Z = (Z1 , . . . , Zn ) . Then the Zi are independent and Zi N1 (0, 1), by Lemma 4. Therefore, the joint density of the Zi is n 1 1 1 1 exp zi2 = exp z z . pZ (z) = (2)1/2 2 (2)n/2 2 i=1

Let Y = 1/2 Z + . By Theorem 2c, Y Nn (, ). Also Z = 1/2 (Y ), and the transformation from Z to Y is therefore invertible. Furthermore, the Jacobian of this inverse transformation is just |1/2 | = ||1/2 . Hence the density of Y is pY (y) = pZ (1/2 (y )) = 1 ||1/2 (2)n/2 1 ||1/2 1 exp (y ) 1 (y ) . 2 14

We now prove a result that is useful later in the book and is also the basis for Pearsons tests.
2

Theorem 13 Let Y Nn (, ), > 0. Then (a). Y 1 Y 2 ( 1 ). n (b). (Y ) 1 (Y ) 2 (0). n Proof. (a). Let Z = 1/2 Y Nn (1/2 , I). By Lemma 4, we see that Z Z = Y 1 Y 2 ( 1 ). n (b). Follows fairly directly.

The Spherical Normal

For the rst part of this book, the most important class of multivariate normal distribution is the class in which Y Nn (, 2 I). We now show that this distribution is spherically symmetric about . A rotation about is given by X = (Y ) + , where is an orthogonal matrix (i.e., = I). By Theorem 2, X Nn (, 2 I), so that the distribution is unchanged under rotations about . We therefore call this normal distribution the spherical normal distribution. If 2 = 0, then P (Y = ) = 1. Otherwise its density function (by Theorem 4) is pY (y) = 1 1 exp 2 ||y ||2 . n/2 n (2) 2

By Lemma 4, we note that the components of Y are independently normally distributed with common variance 2 . In fact, the spherical normal distribution is the only multivariate distribution with independent components that is spherically symmetric.

15

Lecture Notes 2 1 Probability Inequalities

Inequalities are useful for bounding quantities that might otherwise be hard to compute. They will also be used in the theory of convergence. Theorem 1 (The Gaussian Tail Inequality) Let X N (0, 1). Then P(|X| > ) If X1 , . . . , Xn N (0, 1) then 1 2 P(|X n | > ) en /2 . n Proof. The density of X is (x) = (2)1/2 ex P(X > ) = = By symmetry, P(|X| > ) Now let X1 , . . . , Xn N (0, 1). Then X n = n1 where Z N (0, 1) and 2e

2 /2

2e

2 /2

. Hence,

(s)ds

s (s)ds e
2 /2

(s)ds =

( )

2 /2

. Xi N (0, 1/n). Thus, X n = n1/2 Z 1 2 n ) en /2 . n

d

n i=1

P(|X n | > ) = P(n1/2 |Z| > ) = P(|Z| >

Theorem 2 (Markovs inequality) Let X be a non-negative random variable and suppose that E(X) exists. For any t > 0, P(X > t) Proof. Since X > 0, E(X) =
0 t

E(X) . t

(1)

x p(x)dx =
0

x p(x)dx +
t t

xp(x)dx

x p(x)dx t

p(x)dx = t P(X > t).

Theorem 3 (Chebyshevs inequality) Let = E(X) and 2 = Var(X). Then, P(|X | t) 2 t2 and P(|Z| k) 1 k2 (2)

where Z = (X )/. In particular, P(|Z| > 2) 1/4 and P(|Z| > 3) 1/9. Proof. We use Markovs inequality to conclude that P(|X | t) = P(|X |2 t2 ) The second part follows by setting t = k. If X1 , . . . , Xn Bernoulli(p) then and X n = n1 n Xi Then, Var(X n ) = Var(X1 )/n = i=1 p(1 p)/n and Var(X n ) p(1 p) 1 P(|X n p| > ) = 2 2 n 4n 2 since p(1 p) 1 for all p. 4 2 E(X )2 = 2. t2 t

Hoedings Inequality

Hoedings inequality is similar in spirit to Markovs inequality but it is a sharper inequality. We begin with the following important result. Lemma 4 Supppose that E(X) = 0 and that a X b. Then E(etX ) et
2 (ba)2 /8

Recall that a function g is convex if for each x, y and each [0, 1], g(x + (1 )y) g(x) + (1 )g(y). Proof. Since a X b, we can write X as a convex combination of a and b, namely, X = b + (1 )a where = (X a)/(b a). By the convexity of the function y ety we have X a tb b X ta etX etb + (1 )eta = e + e . ba ba Take expectations of both sides and use the fact that E(X) = 0 to get EetX b ta a tb e + e = eg(u) ba ba (3)

where u = t(b a), g(u) = u + log(1 + eu ) and = a/(b a). Note that g(0) = g (0) = 0. Also, g (u) 1/4 for all u > 0. By Taylors theorem, there is a (0, u) such that g(u) = g(0) + ug (0) + Hence, EetX eg(u) et
2 (ba)2 /8

u2 u2 u2 t2 (b a)2 g () = g () = . 2 2 8 8

Next, we need to use Cherno s method. Lemma 5 Let X be a random variable. Then P(X > ) inf et E(etX ).
t0

Proof. For any t > 0, P(X > ) = P(eX > e ) = P(etX > et ) et E(etX ). Since this is true for every t 0, the result follows. Theorem 6 (Hoedings Inequality) Let Y1 , . . . , Yn be iid observations such that E(Yi ) = and a Yi b where a < 0 < b. Then, for any > 0, P |Y n | 2e2n
2 /(ba)2

(4)

Proof. Without los of generality, we asume that = 0. First we have P(|Y n | ) = P(Y n ) + P(Y n ) = P(Y n ) + P(Y n ). 3

Next we use Chernos method. For any t > 0, we have, from Markovs inequality, that
n

P(Y n ) = P

= P e = etn

i=1 P t n Yi i=1

Yi n etn

=P e

Pn

i=1

Yi

en
Pn
i=1

etn E et

Yi

E(etYi ) = etn (E(etYi ))n .

i

From Lemma 4, E(etYi ) et

2 (ba)2 /8

. So
2 n(ba)2 /8

P(Y n ) etn et This is minimized by setting t = 4 /(b a)2 giving P(Y n ) e2n

2 /(ba)2

Applying the same argument to P(Y n ) yields the result. Example 7 Let X1 , . . . , Xn Bernoulli(p). Chebyshevs inequality yields P(|X n p| > ) According to Hoedings inequality, P(|X n p| > ) 2e2n which decreases much faster. Corollary 8 If X1 , X2 , . . . , Xn are independent with P(a Xi b) = 1 and common mean , then, with probability at least 1 , |X n | where c = (b a)2 . c log 2n 2 (5)
2

1 . 4n 2

The Bounded Dierence Inequality

So far we have focused on sums of random variables. The following result extends Hoedings inequality to more general functions g(x1 , . . . , xn ). Here we consider McDiarmids inequality, also known as the Bounded Dierence inequality.

Theorem 9 (McDiarmid) Let X1 , . . . , Xn be independent random variables. Suppose that sup

x1 ,...,xn ,xi

g(x1 , . . . , xi1 , xi , xi+1 , . . . , xn ) g(x1 , . . . , xi1 , xi , xi+1 , . . . , xn ) ci

(6)

for i = 1, . . . , n. Then P g(X1 , . . . , Xn ) E(g(X1 , . . . , Xn ))

n i=1

exp

n 2 i=1 ci

(7)

Proof. Let Vi = E(g|X1 , . . . , Xi )E(g|X1 , . . . , Xi1 ). Then g(X1 , . . . , Xn )E(g(X1 , . . . , Xn )) = Vi and E(Vi |X1 , . . . , Xi1 ) = 0. Using a similar argument as in Hoedings Lemma we have, 2 2 E(etVi |X1 , . . . , Xi1 ) et ci /8 . (8) Now, for any t > 0,
n

P (g(X1 , . . . , Xn ) E(g(X1 , . . . , Xn )) ) = P =P e
t Pn
i=1 Vi

i=1

Vi
Pn
i=1

e
i=1

et E et

Vi

= et E et et et

Pn1

Vi

E etVn X1 , . . . , Xn1
i=1

2 c2 /8 n

E et

Pn1

Vi

. . . Pn 2 2 et et i=1 ci . The result follows by taking t = 4 /

n 2 i=1 ci . n i=1

Example 10 If we take g(x1 , . . . , xn ) = n1

xi then we get back Hoedings inequality.

Example 11 Suppose we throw m balls into n bins. What fraction of bins are empty? Let Z be the number of empty bins and let F = Z/n be the fraction of empty bins. We can write Z = n Zi where Zi = 1 of bin i is empty and Zi = 0 otherwise. Then i=1
n

= E(Z) =
i=1

E(Zi ) = n(1 1/n)m = nem log(11/n) nem/n

and = E(F ) = /n em/n . How close is Z to ? Note that the Zi s are not independent so we cannot just apply Hoeding. Instead, we proceed as follows.

Dene variables X1 , . . . , Xm where Xs = i if ball s falls into bin i. Then Z = g(X1 , . . . , Xm ). If we move one ball into a dierent bin, then Z can change by at most 1. Hence, (6) holds with ci = 1 and so 2 P(|Z | > t) 2e2t /m . Recall that he fraction of empty bins is F = Z/m with mean = /n. We have P(|F | > t) = P(|Z | > nt) 2e2n
2 t2 /m

Bounds on Expected Values

E |XY | E(X 2 )E(Y 2 ). (9)

Theorem 12 (Cauchy-Schwartz inequality) If X and Y have nite variances then

The Cauchy-Schwarz inequality can be written as

2 2 Cov2 (X, Y ) X Y .

Recall that a function g is convex if for each x, y and each [0, 1], g(x + (1 )y) g(x) + (1 )g(y). If g is twice dierentiable and g (x) 0 for all x, then g is convex. It can be shown that if g is convex, then g lies above any line that touches g at some point, called a tangent line. A function g is concave if g is convex. Examples of convex functions are g(x) = x2 and g(x) = ex . Examples of concave functions are g(x) = x2 and g(x) = log x. Theorem 13 (Jensens inequality) If g is convex, then Eg(X) g(EX). If g is concave, then Eg(X) g(EX). (11) Proof. Let L(x) = a + bx be a line, tangent to g(x) at the point E(X). Since g is convex, it lies above the line L(x). So, Eg(X) EL(X) = E(a + bX) = a + bE(X) = L(E(X)) = g(EX). (10)

Example 14 From Jensens inequality we see that E(X 2 ) (EX)2 . 6

Example 15 (Kullback Leibler Distance) Dene the Kullback-Leibler distance between two densities p and q by p(x) D(p, q) = p(x) log dx. q(x) Note that D(p, p) = 0. We will use Jensen to show that D(p, q) 0. Let X f . Then D(p, q) = E log q(X) p(X) log E q(X) p(X) = log p(x) q(x) dx = log p(x) q(x)dx = log(1) = 0.

So, D(p, q) 0 and hence D(p, q) 0. Example 16 It follows from Jensens inequality that 3 types of means can be ordered. Assume that a1 , . . . , an are positive numbers and dene the arithmetic, geometric and harmonic means as aA = aG aH Then aH aG aA . Suppose we have an exponential bound on P(Xn > ). In that case we can bound E(Xn ) as follows. Theorem 17 Suppose that Xn 0 and that for every P(Xn > ) c1 ec2 n for some c2 > 0 and c1 > 1/e. Then, E(Xn ) where C = (1 + log(c1 ))/c2 . Proof. Recall that for any nonnegative random variable Y , E(Y ) = Hence, for any a > 0,
2 E(Xn ) 0 a
2

1 (a1 + . . . + an ) n = (a1 . . . an )1/n 1 . = 1 1 ( + . . . + a1 ) n a1 n

> 0, (12)

C . n

(13)

P(Y t)dt.

=
0

a 2 P(Xn

t)dt =

2 P(Xn

t)dt +

2 P(Xn

t)dt a +

2 P(Xn t)dt.

Equation (12) implies that P(Xn >

2 E(Xn ) a + a

t) c1 ec2 nt . Hence,
a

2 P(Xn t)dt = a +

P(Xn 7

t)dt a + c1

ec2 nt dt = a +

c1 ec2 na . c2 n

Set a = log(c1 )/(nc2 ) and conclude that

2 E(Xn )

log(c1 ) 1 1 + log(c1 ) + = . nc2 nc2 nc2 1 + log(c1 ) . nc2

Finally, we have E(Xn )

2 E(Xn )

Now we consider bounding the maximum of a set of random variables. Theorem 18 Let X1 , . . . , Xn be random variables. Suppose there exists > 0 such that 2 E(etXi ) et /2 for all t > 0. Then E Proof. By Jensens inequality, exp tE max Xi E exp t max Xi
1in n 1in

max Xi

2 log n.

(14)

1in

= E Thus, E The result follows by setting t =

1in

max exp {tXi }

i=1

E (exp {tXi }) net

2 2 /2

1in

max Xi

log n t 2 + . t 2

2 log n/.

OP and oP

In statisics, probability and machine learning, we make use of oP and OP notation. Recall rst, that an = o(1) means that an 0 as n . an = o(bn ) means that an /bn = o(1). an = O(1) means that an is eventually bounded, that is, for all large n, |an | C for some C > 0. an = O(bn ) means that an /bn = O(1). We write an bn if both an /bn and bn /an are eventually bounded. In computer sicence this s written as an = (bn ) but we prefer using an bn since, in statistics, often denotes a parameter space. Now we move on to the probabilistic versions. Say that Yn = oP (1) if, for every > 0, P(|Yn | > ) 0. 8

Say that Yn = oP (an ) if, Yn /an = oP (1). Say that Yn = OP (1) if, for every > 0, there is a C > 0 such that P(|Yn | > C) . Say that Yn = OP (an ) if Yn /an = OP (1). Lets use Hoedings inequality to show that sample proportions are OP (1/ n) within the the true mean. Let Y1 , . . . , Yn be coin ips i.e. Yi {0, 1}. Let p = P(Yi = 1). Let pn = 1 n
n

Yi .
i=1

We will show that: pn p = oP (1) and pn p = OP (1/ n). We have that 2 P(|pn p| > ) 2e2n 0 and so pn p = oP (1). Also, C P( n|pn p| > C) = P |pn p| > n 2e2C < n(pn p) = OP (1) and so 1 n .
2

if we pick C large enough. Hence,

pn p = O P

Now consider m coins with probabilities p1 , . . . , pm . Then

m

P(max |pj pj | > )

j

j=1 m

P(|pj pj | > ) 2e2n

2

union bound

Hoeding
2

j=1
2

= 2me2n = 2 exp (2n Supose that m en where 0 < 1. Then P(max |pj pj | > ) 2 exp (2n
j 2

log m) .

n ) 0.

Hence, max |pj pj | = oP (1).

j

Lecture Notes 3 1 Uniform Bounds

n i=1

Sometimes we want to say more than this.

Recall that, if X1 , . . . , Xn Bernoulli(p) and pn = n1 inequality, 2 P(|pn p| > ) 2e2n .

Xi then, from Hoedings

Example 1 Suppose that X1 , . . . , Xn have cdf F . Let 1 Fn (t) = n

n

i=1

I(Xi t).

We call Fn the empirical cdf. How close is Fn to F ? That is, how big is |Fn (t) F (t)|? From Hoedings inequality, P(|Fn (t) F (t)| > ) 2e2n . But that is only for one point t. How big is supt |Fn (t) F (t)|? We would like a bound of the form P sup |Fn (t) F (t)| >
t
2

something small.

Example 2 Suppose that X1 , . . . , Xn P . Let 1 Pn (A) = n

n i=1

I(Xi A).

How close is Pn (A) to P (A)? That is, how big is |Pn (A) P (A)|? From Hoedings inequality, 2 P(|Pn (A) P (A)| > ) 2e2n .

But that is only for one set A. How big is supAA |Pn (A) P (A)| for a class of sets A? We would like a bound of the form P sup |Pn (A) P (A)| >
AA

something small.

Example 3 (Classication.) Suppose we observe data (X1 , Y1 ), . . . , (Xn , Yn ) where Yi {0, 1}. Let (X, Y ) be a new pair. Suppose we observe X. Now we want to predict Y . A classier h is a function h(x) which takes values in {0, 1}. When we observe X we predict Y with h(X). The classication error, or risk, is the probability of an error: R(h) = P(Y = h(X)). 1

The training error is the fraction of errors on the observed data (X1 , Y1 ), . . . , (Xn , Yn ): 1 R(h) = n By Hoedings inequality, How do we choose a classier? One way is to start with a set of classiers H. Then we dene h to be the member of H that minimizes the training error. Thus h = argminhH R(h). An example is the set of linear classiers. Suppose that x Rd . A linear classier has the form h(x) = 1 of T x 0 and h(x) = 0 of T x < 0 where = (1 , . . . , d )T is a set of parameters. Although h minimizes R(h), it does not minimize R(h). Let h minimize the true error R(h). A fundamental question is: how close is R(h) to R(h )? We will see later than R(h) is close to R(h ) if suph |R(h) R(h)| is small. So we want P sup |R(h) R(h)| >
h n

I(Yi = h(Xi )).

i=1

P(|R(h) R(h)| > ) 2e2n .

something small.

More generally, we can state out goal as follows. For any function f dene P (f ) = f (x)dP (x), 1 Pn (f ) = n
n

f (Xi ).
i=1

Let F be a set of functions. In our rst example, each f was of the form ft (x) = I(x t) and F = {ft : t R}. We want to bound P sup |Pn (f ) P (f )| > We will see that the bounds we obtain have the form P sup |Pn (f ) P (f )| >
f F f F

c1 (F)ec2 n

where c1 and c2 are positive constants and (F) is a measure of the size (or complexity) of the class F. Similarly, if A is a class of sets then we want a bound of the form P sup |Pn (A) P (A)| >
AA 1 n i=1

c1 (A)ec2 n

where Pn (A) = n I(Xi A). Bounds like these are called uniform bonds since they hold uniformly over a class of functions or over a class of sets. 2

Finite Classes
max sup |fj (x)| B.
x

Let F = {f1 , . . . , fN }. Suppose that

1jN

We will make use of the union bound. Recall that

N

P A1

AN

P(Aj ).
j=1

Let Aj be the event that |Pn (fj ) P (f )| > . From Hoedings inequality, P(Aj ) 2 2 2en /(2B ) . Then P sup |Pn (f ) P (f )| >
f F

= P(A1
N

AN )
N

Thus we have shown that

j=1

P(Aj )

2en
j=1

2 /(2B 2 )

= 2N en

2 /(2B 2 )

P sup |Pn (f ) P (f )| >

f F

2en

2 /(2B 2 )

where = |F|. The same idea applies to classes of sets. Let A = {A1 , . . . , AN } be a nite collection of sets. By the same reasoning we have P sup |Pn (A) P (A)| >
AA

2en

2 /(2B 2 )

where = |F| and Pn (A) = n1 n I(Xi A). i=1 To extend these ideas to innite classes like F = {ft : t R} we need to introduce a few more concepts.

Shattering
1. A = {(, t] : t R}. 2. A = {(a, b) : a b}. 3. A = {(a, b) (c, d) : a b c d}. 3

Let A be a class of sets. Some examples are:

4. A = all discs in Rd . 5. A = all rectangles in Rd . 6. A = all half-spaces in Rd = {x : T x 0}. 7. A = all convex sets in Rd . Let F = {x1 , . . . , xn } be a nite set. Let G be a subset of F . Say that A picks out G if AF =G for some A A. For example, let A = {(a, b) : a b}. Suppose that F = {1, 2, 7, 8, 9} and G = {2, 7}. Then A picks out G since A F = G if we choose A = (1.5, 7.5) for example. Let S(A, F ) be the number of these subsets picked out by A. Of course S(A, F ) 2n . Example 4 Let A = {(a, b) : a b} and F = {1, 2, 3}. Then A can pick out: , {1}, {2}, {3}, {1, 2}, {2, 3}, {1, 2, 3}. So s(A, F ) = 7. Note that 7 < 8 = 23 . If F = {1, 6} then A can pick out: , {1}, {6}, {1, 6}. In this case s(A, F ) = 4 = 22 .

We say that F is shattered if s(A, F ) = 2n where n is the number of points in F . Let Fn denote all nite sets with n elements. Dene the shatter coecient sn (A) = sup s(A, F ).
F Fn

Note that sn (A) 2n .

The following theorem is due to Vapnik and Chervonenis. The proof is beyond the scope of the course. (If you take 10-702/36-702 you will learn the proof.)

Class A VC dimension VA A = {A1 , . . . , AN } log2 N Intervals [a, b] on the real line 2 2 Discs in R 3 d Closed balls in R d+2 Rectangles in Rd 2d d Half-spaces in R d+1 Convex polygons in R2 Convex polygons with d vertices 2d + 1 Table 1: The VC dimension of some classes A. Theorem 5 Let A be a class of sets. Then P sup |Pn (A) P (A)| >
AA

8 sn (A) en

2 /32

(1)

This partly solves one of our problems. But, how big can sn (A) be? Sometimes sn (A) = 2n for all n. For example, let A be all polygons in the plane. Then sn (A) = 2n for all n. But, in many cases, we will see that sn (A) = 2n for all n up to some integer d and then sn (A) < 2n for all n > d.

The Vapnik-Chervonenkis (VC) dimension is d = d(A) = largest n such that sn (A) = 2n . In other words, d is the size of the largest set that can be shattered.

Thus, sn (A) = 2n for all n d and sn (A) < 2n for all n > d. The VC dimensions of some common examples are summarized in Table 1. Now here is an interesting question: for n > d how does sn (A) behave? It is less than 2n but how much less? Theorem 6 (Sauers Theorem) Suppose that A has nite VC dimension d. Then, for all n d, s(A, n) (n + 1)d . (2)

We conclude that: Theorem 7 Let A be a class of sets with VC dimension d < . Then P sup |Pn (A) P (A)| >
AA

8 (n + 1)d en

2 /32

(3)

Example 8 Lets return to our rst example. Suppose that X1 , . . . , Xn have cdf F . Let 1 Fn (t) = n
n

i=1

I(Xi t).

We would like to bound P(supt |Fn (t) F (t)| > ). Notice that Fn (t) = Pn (A) where A = (, t]. Let A = {(, t] : t R}. This has VC dimension d = 1. So P(sup |Fn (t) F (t)| > ) = P sup |Pn (A) P (A)| >
t AA

8 (n + 1) en

2 /32

In fact, there is a tighter bound in this case called the DKW (Dvoretsky-Kiefer-Wolfowitz) inequality: 2 P(sup |Fn (t) F (t)| > ) 2e2n .
t

Bounding Expectations

Eearlier we saw that we can use exponential bounds on probabilities to get bounds on expectations. Let us recall how that works. Consider a nite collection A = {A1 , . . . , AN }. Let Zn = max |Pn (Aj ) P (Aj )|.
1jN

We know that But now we want to bound

P(Zn > ) 2me2n . max |Pn (Aj ) P (Aj )| .

2

(4)

E(Zn ) = We can rewrite (4) as

1jN

2 P(Zn >

or, in other words, Recall that, in general, if Y 0 then

) 2N e2n .

2 P(Zn > t) 2N e2nt . 0

E(Y ) =

P(Y > t)dt.

Hence, for any s,

2 E(Zn ) = 0 s 2 P(Zn > t)dt + s s 2 P(Zn > t)dt 2 P(Zn > t)dt

=
0

s+

2 P(Zn > t)dt s

s + 2N = s + 2N = s+ Let s = log(N )/(2n). Then

2 E(Zn ) s +

e2nt dt

e2ns 2n

N 2ns e . n

1 log N + 2 N 2ns log N e = + = . n 2n n 2n

Finally, we use Cauchy-Schwartz: E(Zn ) In summary: log N . 1jN n For a single set A we would have E|Pn (A) P (A)| O(1/ n). The bound only increases logarithmically with N . E max |Pn (Aj ) P (Aj )| =O
2 E(Zn )

log N + 2 =O 2n

log N n

Lecture Notes 4 1 Random Samples

Let X1 , . . . , Xn F . A statistic is any function T = g(X1 , . . . , Xn ). Recall that the sample mean is n 1 Xn = Xi n i=1 and sample variance is
2 Sn 2

1 = n1

i=1

(Xi X n )2 .

Let = E(Xi ) and = Var(Xi ). Recall that E(X n ) = , Var(X n ) = 2 , n

2 E(Sn ) = 2 .

Theorem 1 If X1 , . . . , Xn N (, 2 ) then X n N (, 2 /n). Proof. We know that MXi (s) = es+ So, MX n (t) = E(etX n ) = E(e n
2 2 t n
t 2 s2 /2

Pn

i=1

Xi

)
2 t2 /(2n2 )

= (EetXi /n )n = (MXi (t/n))n = e(t/n)+ = exp t + which is the mgf of a N (, 2 /n). 2

Example 2 (Example 5.2.10). Let Z1 , . . . , Zn Cauchy(0, 1). Then Z n Cauchy(0, 1). Lemma 3 If X1 , . . . , Xn N (, 2 ) then Tn = Xn tn1 N (0, 1). S/ n

Let X(1) , . . . , X(n) denoted the ordered values: X(1) X(2) X(n) . Then X(1) , . . . , X(n) are called the order statistics. 1

Convergence

Let X1 , X2 , . . . be a sequence of random variables and let X be another random variable. Let Fn denote the cdf of Xn and let F denote the cdf of X. 1. Xn converges almost surely to X, written Xn X, if, for every > 0, P( lim |Xn X| < ) = 1.
n a.s.

(1)

2. Xn converges to X in probability, written Xn X, if, for every > 0, P(|Xn X| > ) 0 as n . In other words, Xn X = oP (1). 3. Xn converges to X in quadratic mean (also called convergence in L2 ), written qm Xn X, if E(Xn X)2 0 (3) as n . 4. Xn converges to X in distribution, written Xn
n

(2)

X, if (4)

lim Fn (t) = F (t)

at all t for which F is continuous. Convergence to a Constant. A random variable X has a point mass distribution if there exists a constant c such that P(X = c) = 1. The distribution for X is denoted by c P P and we write X c . If Xn c then we also write Xn c. Similarly for the other types of convergence. Theorem 4 Xn X if and only if, for every
n mn as

> 0,

lim P(sup |Xm X| ) = 1.

Example 5 (Example 5.5.8). This example shows that convergence in probability does not imply almost sure convergence. Let S = [0, 1]. Let P be uniform on [0, 1]. We draw S P . Let X(s) = s and let X1 = s + I[0,1] (s), X4 = s + I[0,1/3] (s),
P

X2 = s + I[0,1/2] (s), X3 = s + I[1/2,1] (s) X5 = s + I[1/3,2/3] (s), X6 = s + I[2/3,1] (s)

etc. Then Xn X. But, for each s, Xn (s) does not converge to X(s). Hence, Xn does not converge almost surely to X. 2

Example 6 Let Xn N (0, 1/n). Intuitively, Xn is concentrating at 0 so we would like to say that Xn converges to 0. Lets see if this is true. Let F be the distribution function for a point mass at 0. Note that nXn N (0, 1). Let Z denote a standard normal random variable. For t < 0, Fn (t) = P(Xn < t) = P( nXn < nt) = P(Z < nt) 0 since nt . For t > 0, Fn (t) = P(Xn < t) = P( nXn < nt) = P(Z < nt) 1 since nt . Hence, Fn (t) F (t) for all t = 0 and so Xn 0. Notice that Fn (0) = 1/2 = F (1/2) = 1 so convergence fails at t = 0. That doesnt matter because t = 0 is not a continuity point of F and the denition of convergence in distribution only requires convergence at continuity points. Now consider convergence in probability. For any > 0, using Markovs inequality, P(|Xn | > ) = P(|Xn | >
2 2

2 E(Xn ) 2

1 n 2

as n . Hence, Xn 0. The next theorem gives the relationship between the types of convergence. Theorem 7 The following relationships hold: qm P (a) Xn X implies that Xn X. P (b) Xn X implies that Xn X. P (c) If Xn X and if P(X = c) = 1 for some real number c, then Xn X. P as (d) Xn X implies Xn X. In general, none of the reverse implications hold except the special case in (c). Proof. We start by proving (a). Suppose that Xn X. Fix Markovs inequality, P(|Xn X| > ) = P(|Xn X|2 >
2 qm

> 0. Then, using

E|Xn X|2
2

0.

Proof of (b). Fix > 0 and let x be a continuity point of F . Then Fn (x) = P(Xn x) = P(Xn x, X x + ) + P(Xn x, X > x + ) P(X x + ) + P(|Xn X| > ) = F (x + ) + P(|Xn X| > ). 3

Also, F (x ) = P(X x ) = P(X x , Xn x) + P(X x , Xn > x) Fn (x) + P(|Xn X| > ). Hence, F (x ) P(|Xn X| > ) Fn (x) F (x + ) + P(|Xn X| > ). Take the limit as n to conclude that F (x ) lim inf Fn (x) lim sup Fn (x) F (x + ).
n n

This holds for all > 0. Take the limit as and conclude that limn Fn (x) = F (x). Proof of (c). Fix > 0. Then, P(|Xn c| > ) = = =

0 and use the fact that F is continuous at x

P(Xn < c ) + P(Xn > c + ) P(Xn c ) + P(Xn > c + ) Fn (c ) + 1 Fn (c + ) F (c ) + 1 F (c + ) 0 + 1 1 = 0.

Proof of (d). This follows from Theorem 4. Let us now show that the reverse implications do not hold. Convergence in probability does not imply convergence in quadratic mean. Let U Unif(0, 1) and let Xn = nI(0,1/n) (U ). Then P(|Xn | > ) = P( nI(0,1/n) (U ) > ) = P(0 U < 1/n) =
P 1/n 0

2 1/n 0. Hence, Xn 0. But E(Xn ) = n quadratic mean.

du = 1 for all n so Xn does not converge in

Convergence in distribution does not imply convergence in probability. Let X N (0, 1). Let Xn = X for n = 1, 2, 3, . . .; hence Xn N (0, 1). Xn has the same distribution function as X for all n so, trivially, limn Fn (x) = F (x) for all x. Therefore, Xn X. But P(|Xn X| > ) = P(|2X| > ) = P(|X| > /2) = 0. So Xn does not converge to X in probability. The relationships between the types of convergence can be summarized as follows:

q.m. a.s. prob distribution

4

Example 8 One might conjecture that if Xn b, then E(Xn ) b. This is not true. Let Xn be a random variable dened by P(Xn = n2 ) = 1/n and P(Xn = 0) = 1 (1/n). P Now, P(|Xn | < ) = P(Xn = 0) = 1 (1/n) 1. Hence, Xn 0. However, E(Xn ) = [n2 (1/n)] + [0 (1 (1/n))] = n. Thus, E(Xn ) . Example 9 Let X1 , . . . , Xn Uniform(0, 1). Let X(n) = maxi Xi . First we claim that X(n) 1. This follows since
P

P(|X(n) 1| > ) = P(X(n) 1 ) = Also

P(Xi 1 ) = (1 )n 0.

P(n(1 X(n) ) t) = P(X(n) 1 (t/n)) = (1 t/n)n et . So n(1 X(n) ) Exp(1). Some convergence properties are preserved under transformations. Theorem 10 Let Xn , X, Yn , Y be random variables. Let g be a continuous function. P P P (a) If Xn X and Yn Y , then Xn + Yn X + Y . qm qm qm (b) If Xn X and Yn Y , then Xn + Yn X + Y . (c) If Xn X and Yn c, then Xn + Yn X + c. P P P (d) If Xn X and Yn Y , then Xn Yn XY . (e) If Xn X and Yn c, then Xn Yn cX. P P (f ) If Xn X, then g(Xn ) g(X). (g) If Xn X, then g(Xn ) g(X). Parts (c) and (e) are know as Slutzkys theorem Parts (f) and (g) are known as The Continuous Mapping Theorem. It is worth noting that Xn X +Y. X and Yn Y does not in general imply that Xn +Yn

The Law of Large Numbers

The law of large numbers (LLN) says that the mean of a large sample is close to the mean of the distribution. For example, the proportion of heads of a large number of tosses of a fair coin is expected to be close to 1/2. We now make this more precise. Let X1 , X2 , . . . be an iid sample, let = E(X1 ) and 2 = Var(X1 ). Recall that the sample mean is dened as X n = n1 n Xi and that E(X n ) = and Var(X n ) = 2 /n. i=1 5

Theorem 11 (The Weak Law of Large Numbers (WLLN)) P If X1 , . . . , Xn are iid, then X n . Thus, X n = oP (1). Interpretation of the WLLN: The distribution of X n becomes more concentrated around as n gets large. Proof. Assume that < . This is not necessary but it simplies the proof. Using Chebyshevs inequality, Var(X n ) 2 = 2 P |X n | > 2 n which tends to 0 as n . Theorem 12 The Strong Law of Large Numbers. Let X1 , . . . , Xn be iid with mean . as Then X n . The proof is beyond the scope of this course.

The Central Limit Theorem

The law of large numbers says that the distribution of X n piles up near . This isnt enough to help us approximate probability statements about X n . For this we need the central limit theorem. Suppose that X1 , . . . , Xn are iid with mean and variance 2 . The central limit theorem (CLT) says that X n = n1 i Xi has a distribution which is approximately Normal with mean and variance 2 /n. This is remarkable since nothing is assumed about the distribution of Xi , except the existence of the mean and variance. Theorem 13 (The Central Limit Theorem (CLT)) Let X1 , . . . , Xn be iid with mean and variance 2 . Let X n = n1 n Xi . Then i=1 Zn Xn Var(X n ) n(X n ) = Z

where Z N (0, 1). In other words,

z n

lim P(Zn z) = (z) =

1 2 ex /2 dx. 2

Interpretation: Probability statements about X n can be approximated using a Normal distribution. Its the probability statements that we are approximating, not the random variable itself.

A consequence of the CLT is that X n = OP 1 n .

In addition to Zn N (0, 1), there are several forms of notation to denote the fact that the distribution of Zn is converging to a Normal. They all mean the same thing. Here they are: 2 n 2 X n N 0, n n(X n ) N 0, 2 n(X n ) N (0, 1). Recall that if X is a random variable, its moment generating function (mgf) is X (t) = EetX . Assume in what follows that the mgf is nite in a neighborhood around t = 0. Xn N , Lemma 14 Let Z1 , Z2 , . . . be a sequence of random variables. Let n be the mgf of Zn . Let Z be another random variable and denote its mgf by . If n (t) (t) for all t in some open interval around 0, then Zn Z. Proof of the central limit theorem. Let Yi = (Xi )/. Then, Zn n1/2 i Yi . = Let (t) be the mgf of Yi . The mgf of i Yi is ((t))n and mgf of Zn is [(t/ n)]n n (t). Now (0) = E(Y1 ) = 0, (0) = E(Y12 ) = Var(Y1 ) = 1. So, (t) = (0) + t (0) + = 1+0+ t2 t3 (0) + (0) + 2! 3! Zn N (0, 1)

t2 t3 + (0) + 2 3! 2 3 t t = 1 + + (0) + 2 3!
n

Now, n (t) = = = t n 2 t t3 1+ + (0) + 2n 3!n3/2 1+

2 /2

t2 2

t3 3!n1/2

(0) +

et

which is the mgf of a N(0,1). The result follows from Lemma 14. In the last step we used the fact that if an a then an n 1+ ea . n The central limit theorem tells us that Zn = n(X n )/ is approximately N(0,1). However, we rarely know . We can estimate 2 from X1 , . . . , Xn by
2 Sn =

1 n1

i=1

(Xi X n )2 .

This raises the following question: if we replace with Sn , is the central limit theorem still true? The answer is yes. Theorem 15 Assume the same conditions as the CLT. Then, n(X n ) N (0, 1). Tn = Sn Proof. We have that Tn = Zn Wn where Zn = and Wn = Now Zn
P

n(X n ) . Sn

N (0, 1) and Wn 1. The result follows from Slutzkys theorem.

There is also a multivariate version of the central limit theorem. Recall that X = (X1 , . . . , Xk )T has a multivariate Normal distribution with mean vector and covariance matrix if 1 1 exp (x )T 1 (x ) . f (x) = k/2 ||1/2 (2) 2 In this case we write X N (, ). Theorem 16 (Multivariate central limit theorem) Let X1 , . . . , Xn be iid random vectors where Xi = (X1i , . . . , Xki )T with mean = (1 , . . . , k )T and covariance matrix . Let X = (X 1 , . . . , X k )T where X j = n1 n Xji . Then, i=1 n(X ) N (0, ).

The Delta Method

If Yn has a limiting Normal distribution then the delta method allows us to nd the limiting distribution of g(Yn ) where g is any smooth function. Theorem 17 (The Delta Method) Suppose that n(Yn ) N (0, 1) and that g is a dierentiable function such that g () = 0. Then n(g(Yn ) g()) N (0, 1). |g ()| In other words, Yn N , 2 n implies that g(Yn ) N g(), (g ())2 2 . n

Example 18 Let X1 , . . . , Xn be iid with nite mean and nite variance 2 . By the central limit theorem, n(X n )/ N (0, 1). Let Wn = eX n . Thus, Wn = g(X n ) where g(s) = es . Since g (s) = es , the delta method implies that Wn N (e , e2 2 /n). There is also a multivariate version of the delta method. Theorem 19 (The Multivariate Delta Method) Suppose that Yn = (Yn1 , . . . , Ynk ) is a sequence of random vectors such that n(Yn ) N (0, ). Let g : Rk R and let . g(y) = . . .
g yk

g y1

Let Then

denote

g(y) evaluated at y = and assume that the elements of n(g(Yn ) g()) X11 X21 , X12 X22
n

are nonzero.

N 0,

Example 20 Let , ..., X1n X2n

n

be iid random vectors with mean = (1 , 2 )T and variance . Let 1 X1 = n X1i ,

i=1

1 X2 = n 9

X2i
i=1

and dene Yn = X 1 X 2 . Thus, Yn = g(X 1 , X 2 ) where g(s1 , s2 ) = s1 s2 . By the central limit theorem, X 1 1 n N (0, ). X 2 2 Now g(s) = and so
T g s1 g s2

s2 s1

= (2 1 )

11 12 12 22

2 1

= 2 11 + 21 2 12 + 2 22 . 1 2

Therefore,

n(X 1 X 2 1 2 )

N 0, 2 11 + 21 2 12 + 2 22 . 2 1

10

Addendum to Lecture Notes 4

Here is the proof that Tn = where
2 Sn

n(X n ) Sn
n

N (0, 1)

1 = n1
P

i=1

(Xi X n )2 .

2 Step 1. We rst show that Rn 2 where 2 Rn =

1 n

i=1

(Xi X n )2 . 1 n
n 2

Note that
2 Rn

1 = n

Xi2
i=1

Xi
i=1

Dene Yi = Xi2 . Then, using the LLN (law of large numbers) 1 n Next, by the LLN, 1 n
2 n

Xi2 =
i=1

1 n

i=1

Yi E(Yi ) = E(Xi2 ) = 2 + 2 .
n

i=1

Xi .

Since g(t) = t is continuous, the continuous mapping theorem implies that 1 n Thus
P n 2

Xi
i=1

2 .

2 Rn (2 + 2 ) 2 = 2 .

Step 2. Note that

2 Sn = P

n n1

2 Rn . P

2 2 Since, Rn 2 and n/(n 1) 1, we have that Sn 2 .

Step 3. Since g(t) = P implies that Sn .

t is continuous, (for t 0) the continuous mapping theorem 1

Step 4. Since g(t) = t/ is continuous, the continuous mapping theorem implies that P Sn / 1. Step 5. Since g(t) = 1/t is continuous (for t > 0) the continuous mapping theorem P implies that /Sn 1. Since convergence in probability implies convergence in distribution, /Sn 1. Step 5. Note that Tn = n(X n ) Sn Vn Wn . 1. By Slutzkys

Now Vn Z where Z N (0, 1) by the CLT. And we showed that Wn theorem, Tn = Vn Wn Z 1 = Z.

Lecture Notes 5 1 Statistical Models

A statistical model P is a collection of probability distributions (or a collection of densities). An example of a nonparametric model is P= A parametric model has the form P= p(x; ) :
2 /2

p:

(p (x))2 dx < .

where Rd . An example is the set of Normal densities {p(x; ) = (2)1/2 e(x) For now, we focus on parametric models. The model comes from assumptions. Some examples: Time until something fails is often modeled by an exponential distribution. Number of rare events is often modeled by a Poisson distribution. Lengths and weights are often modeled by a Normal distribution.

}.

These models are not correct. But they might be useful. Later we consider nonparametric methods that do not assume a parametric model

Statistics

Let X1 , . . . , Xn p(x; ). Let X n (X1 , . . . , Xn ). Any function T = T (X1 , . . . , Xn ) is itself a random variable which we will call a statistic. Some examples are: order statistics, X(1) X(2) X(n) 1

 sample mean: X =

1 n

Xi ,
i (Xi

sample variance: S 2 =

1 n1

x)2 ,

sample median: middle value of ordered statistics, sample minimum: X(1) sample maximum: X(n) sample range: X(n) X(1) sample interquartile range: X(.75n) X(.25n) Example 1 If X1 , . . . , Xn (, ), then X (n, /n). Proof: MX = E[etx ] = E[e
n P Xi t/n

]=
i

E[eXi (t/n) ]
n

= [MX (t/n)] = This is the mgf of (n, /n).

1 1 t/n

1 = 1 /nt

Example 2 If X1 , . . . , Xn N (, 2 ) then X N (, 2 /n). Example 3 If X1 , . . . , Xn iid Cauchy(0,1), p(x) = for x R, then X Cauchy(0,1). Example 4 If X1 , . . . , Xn N (, 2 ) then (n 1) 2 S 2 (n1) . 2 The proof is based on the mgf. 1 (1 + x2 )

Example 5 Let X(1) , X(2) , . . . , X(n) be the order statistics, which means that the sample X1 , X2 , . . . , Xn has been ordered from smallest to largest: X(1) X(2) X(n) . Now, FX(k) (x) = P (X(k) x) = P (at least k of the X1 , . . . , Xn x)
n

=
j=k n

P (exactly j of the X1 , . . . , Xn x) n [FX (x)]j [1 FX (x)]nj j

=
j=k

Dierentiate to nd the pdf (See CB p. 229): pX(k) (x) = n! [FX (x)]k1 p(x) [1 FX (x)]nk . (k 1)!(n k)!

Suciency

(Ch 6 CB) We continue with parametric inference. In this section we discuss data reduction as a formal concept. Sample X n = X1 , , Xn F . Assume F belongs to a family of distributions, (e.g. F is Normal), indexed by some parameter . We want to learn about and try to summarize the data without throwing any information about away. If a statistic T (X1 , , Xn ) contains all the information about in the sample we say T is sucient. 3

3.1

Sucient Statistics

Denition: T is sucient for if the conditional distribution of X n |T does not depend on . Thus, f (x1 , . . . , xn |t; ) = f (x1 , . . . , xn |t).

Example 6 X1 , , Xn Poisson(). Let T =

n i=1

Xi . Then, P (X n = xn and T = t) . P (T = t) if T (xn ) = t if T (X n ) = t

pX n |T (xn |t) = P(X n = xn |T (X n ) = t) = But

Hence,

0 P (X n = xn and T = t) = P (X n = xn )
n P

P (X = x ) =
i=1

e xi en xi en t = = . xi ! (xi !) (xi !)

Now, T (x ) =

xi = t and so P (T = t) = en (n)t t! since T Poisson(n). t! xi )!nt

Thus, P (X n = xn ) = P (T = t) ( which does not depend on . So T = statistics are: T = 3.7

i i i

Xi is a sucient statistic for . Other sucient

Xi , T = (

Xi , X4 ), and T (X1 , . . . , Xn ) = (X1 , . . . , Xn ).

3.2

Sucient Partitions

It is better to describe suciency in terms of partitions of the sample space. Example 7 Let X1 , X2 , X3 Bernoulli(). Let T = Xi .

xn (0, 0, 0) (0, 0, 1) (0, 1, 0) (1, 0, 0) (0, 1, 1) (1, 0, 1) (1, 1, 0) (1, 1, 1)

t t=0 t=1 t=1 t=1 t=2 t=2 t=2 t=3 4 elements

p(x|t) 1 1/3 1/3 1/3 1/3 1/3 1/3 1

8 elements

1. A partition B1 , . . . , Bk is sucient if f (x|X B) does not depend on . 2. A statistic T induces a partition. For each t, {x : T (x) = t} is one element of the partition. T is sucient if and only if the partition is sucient. 3. Two statistics can generate the same partition: example:
i

Xi and 3

Xi .

4. If we split any element Bi of a sucient partition into smaller pieces, we get another sucient partition. Example 8 Let X1 , X2 , X3 Bernoulli(). Then T = X1 is not sucient. Look at its partition:

xn (0, 0, 0) (0, 0, 1) (0, 1, 0) (0, 1, 1) (1, 0, 0) (1, 0, 1) (1, 1, 0) (1, 1, 1) 8 elements

t t=0 t=0 t=0 t=0 t=1 t=1 t=1 t=1 2 elements

p(x|t) (1 )2 (1 ) (1 ) 2 (1 )2 (1 ) (1 ) 2

3.3

The Factorization Theorem

Theorem 9 T (X n ) is sucient for if the joint pdf/pmf of X n can be factored as p(xn ; ) = h(xn ) g(t; ). Example 10 Let X1 , , Xn Poisson. Then en Xi = p(x ; ) = (xi !)
n P P 1 en i Xi . (xi !)

Example 11 X1 , , Xn N (, 2 ). Then p(x ; , ) = (a) If known: p(xn ; ) = 1 2 2

n 2

1 2 2

n 2

exp

(xi x)2 + n(x )2 2 2

exp

(xi x)2 2 2

exp

n(x )2 2 2
g(T (xn )|)

h(xn )

Thus, X is sucient for . (b) If (, 2 ) unknown then T = (X, S 2 ) is sucient. So is T = ( Xi , Xi2 ).

3.4

Minimal Sucient Statistics (MSS)

We want the greatest reduction in dimension. Example 12 X1 , , Xn N (0, 2 ). Some sucient statistics are: T (X1 , , Xn ) = (X1 , , Xn )
2 2 T (X1 , , Xn ) = (X1 , , Xn ) m n

T (X1 , , Xn ) = T (X1 , , Xn ) =

Xi2 ,
i=1 i=m+1

Xi2

Xi2 .

Denition: T is a Minimal Sucient Statistic if the following two statements are true: 1. T is sucient and 2. If U is any other sucient statistic then T = g(U ) for some function g. In other words, T generates the coarsest sucient partition. Suppose U is sucient. Suppose T = H(U ) is also sucient. T provides greater reduction than U unless H is a 1 1 transformation, in which case T and U are equivalent. Example 13 X N (0, 2 ). X 2, X 4, e
X2

X is sucient. |X| is sucient. |X| is MSS. So are

. Xi .

Example 14 Let X1 , X2 , X3 Bernoulli(). Let T =

xn (0, 0, 0) (0, 0, 1) (0, 1, 0) (1, 0, 0) (0, 1, 1) (1, 0, 1) (1, 1, 0) (1, 1, 1)

p(x|t)

u u=0 u=1 u=1 u=1 u = 73 u = 73 u = 91

p(x|u) 1 1/3 1/3 1/3 1/2 1/2 1

t=0 1 t = 1 1/3 t = 1 1/3 t = 1 1/3 t = 2 1/3 t = 2 1/3 t = 2 1/3 t=3 1

u = 103 1

Note that U and T are both sucient but U is not minimal.

3.5

How to nd a Minimal Sucient Statistic

p(y n ; ) . p(xn ; )

Theorem 15 Dene R(xn , y n ; ) = Suppose that T has the following property: R(xn , y n ; ) does not depend on if and only if T (y n ) = T (xn ). Then T is a MSS. Example 16 Y1 , , Yn iid Poisson (). en p(y ; ) = yi
n P yi

p(y n ; ) yi xi , = p(xn ; ) yi !/ xi ! Yi is a minimal

which is independent of i sucient statistic for .

yi =

xi . This implies that T (Y n ) =

The minimal sucient statistic is not unique. But, the minimal sucient partition is unique.

Example 17 Cauchy. p(x; ) = Then p(y n ; ) = p(xn ; ) The ratio is a constant function of if T (Y n ) = (Y(1) , , Y(n) ). It is technically harder to show that this is true only if T is the order statistics, but it could be done using theorems about polynomials. Having shown this, one can conclude that the order statistics are the minimal sucient statistics for . Note: Ignore the material on completeness and ancillary statistics. 1 . (1 + (x )2 )
n i=1 n

{1 + (xi )2 } {1 + (yj )2 }

j=1

Lecture Notes 6 1 The Likelihood Function

Denition. Let X n = (X1 , , Xn ) have joint density p(xn ; ) = p(x1 , . . . , xn ; ) where . The likelihood function L : [0, ) is dened by L() L(; xn ) = p(xn ; ) where xn is xed and varies in . 1. The likelihood function is a function of . 2. The likelihood function is not a probability density function. 3. If the data are iid then the likelihood is
n

L() =
i=1

p(xi ; )

iid case only.

4. The likelihood is only dened up to a constant of proportionality. 5. The likelihood function is used (i) to generate estimators (the maximum likelihood estimator) and (ii) as a key ingredient in Bayesian inference. Example 1 These 2 samples have the same likelihood function: (X1 , X2 , X3 ) Multinomial (n = 6, , , 1 2) X = (1, 3, 2) X = (2, 2, 2) = = L() = 6! 1 3 (1 2)2 4 (1 2)2 1!3!2! 6! 2 2 L() = (1 2)2 4 (1 2)2 2!2!2!

Example 2 X1 , , Xn N (, 1). Then, L() = 1 2

n 2

1 exp 2

i=1

(xi )2

n exp (x )2 . 2

Example 3 Let X1 , . . . , Xn Bernoulli(p). Then L(p) pX (1 p)nX for p [0, 1] where X =

i

Xi .

Theorem 4 Write xn y n if L(|xn ) L(|y n ). The partition induced by is the minimal sucient partition. Example 5 A non iid example. An AR(1) time series auto regressive model. The model is: X1 N (0, 2 ) and Xi+1 = Xi + ei+1 ei N (0, 2 ).
iid

It can be show that we have the Markov property: o(xn+1 |xn , xn1 , , x1 ) = p(xn+1 |xn ). The likelihood function is L() = p(xn ; ) = p(x1 ; )p(x2 |x1 ; ) p(xn |x1 , . . . , xn1 ; ) = p(xn |xn1 ; )p(xn1 |xn2 ; ) p(x2 |x1 ; )p(x1 ; )
n

=
i=1

1 exp 2

1 (xn+i1 xni )2 . 2 2

Likelihood, Suciency and the Likelihood Principle

The likelihood function is a minimal sucient statistic. That is, if we dene the equivalence relation: xn y n when L(; xn ) L(; y n ) then the resulting partition is minimal sucient. Does this mean that the likelihood function contains all the relevant information? Some people say yes it does. This is sometimes called the likelihood principle. That is, the likelihood principle says that the likelihood function contains all the infomation in the data. This is FALSE. Here is a simple example to illustrate why. Let C = {c1 , . . . , cN } be important). Let = N 1 a nite set of constants. For simplicity, asssume that cj {0, 1} (although this is not
N j=1 cj .

Suppose we want to estimate . We proceed as follows. 2

Let S1 , . . . , Sn Bernoulli() where is known. If Si = 1 you get to see ci . Otherwise, you do not. (This is an example of survey sampling.) The likelihood function is Si (1 )1Si .

The unknown parameter does not appear in the likelihood. In fact, there are no unknown parameters in the likelihood! The likelihood function contains no information at all. But we can estimate . Let 1 = N
N

cj Sj .
j=1

Then E() = . Hoedings inequality implies that P(| | > ) 2e2n Hence, is close to with high probability. Summary: the minimal sucient statistic has all the information you need to compute the likelihood. But that does not mean that all the information is in the likelihood.
2 2

Lecture Notes 7 1 Parametric Point Estimation

X1 , . . . , Xn p(x; ). Want to estimate = (1 , . . . , k ). An estimator = n = w(X1 , . . . , Xn ) is a function of the data. Methods: 1. Method of Moments (MOM) 2. Maximum likelihood (MLE) 3. Bayesian estimators Evaluating Estimators: 1. 2. 3. 4. Bias and Variance Mean squared error (MSE) Minimax Theory Large sample theory (later).

Some Terminology
E () = (x1 , . . . , xn )p(x1 ; ) p(xn ; )dx1 dxn

Bias: E () the distribution of n is called its sampling distribution the standard deviation of n is called the standard error denoted by se(n ) n is consistent if n later we will see that if bias 0 and Var(n ) 0 as n then n is consistent an estimator is robust if it is not strongly aected by perturbations in the data (more later) 1
P

Method of Moments
n

Dene m1 = m2 = 1 n 1 n Xi ,
i=1 n

1 () = E(Xi ) 2 () = E(Xi2 ) . . . . . . k () = E(Xik ).

Xi2 ,
i=1

1 mk = n Let = (1 , . . . , k ) solve:

Xik ,
i=1

mj = j (),

j = 1, . . . , k.

Example 1 N (, 2 ) with = (, 2 ). Then 1 = and 2 = 2 + 2 . Equate: 1 n to get = X, Example 2 Suppose X1 , . . . , Xn Binomial(k, p) where both k and p are unknown. We get kp = X n , giving p= X , k k= X
1 n n

Xi = ,
i=1

1 n

Xi2 = 2 + 2
i=1

1 = n
2

i=1

(Xi X n )2 .

1 n

i=1

Xi2 = kp(1 p) + k 2 p2
2

X . (Xi X)2 i

Maximum Likelihood
L() = p(X1 , . . . , Xn ; ).

Let maximize

Same as maximizing () = log L(). Often it suces to solve () = 0, j Example 3 Binomial. L(p) =
i

j = 1, . . . , k.
i

pXi (1 p)1Xi = pS (1 p)nS where S =

Xi . So

(p) = S log p + (n S) log(1 p) and p = X. Example 4 X1 , . . . , Xn N (, 1). L() e(Xi )

i
2 /2

en(X) ,

n () = (X )2 2

and = X. For N (, 2 ) we have L(, ) and 1 (, ) = n log 2 2

2 2

1 1 exp 2 2

i=1

(Xi )2

i=1

(Xi )2 .

Set = 0, to get 1 = n
n 2

=0 2
n

Xi ,
i=1

1 = n

i=1

(Xi X)2 .

Example 5 Let X1 , . . . , Xn Uniform(0, ). Then L() = and so = X(n) . 1 I( > X(n) ) n

The mle is equivariant. if = g() then = g(). Suppose g is invertible so = g() and = g 1 (). Dene L () = L() where = g 1 (). So, for any , L () = L() L() = L () and hence = g() maximizes L (). For non invertible functions this is still true if we dene L () = sup L().
: ()=

Example 6 Binomial. The mle is p = X. Let = log(p/(1 p)). Then = log(p/(1 p)). Later, we will see that maximum likelihood estimators have certain optimality properties.

Bayes Estimator

Regard as random. Start with prior distribition (). Note that f (x|)() = f (x, ). Now Compute the posterior distribition by Bayes theorem: (|x) = where m(x) = This can be written as (|x) L()(). f (x|)()d. f (x|)() m(x)

Now compute a point estimator from the posterior. For example: = E(|x) = (|x)d = f (x|)()d . f (x|)()d

This approach is controversial. We will discuss the controversey and the meaning of the prior later in the course. For now, we just think of this as a way to dene an estimator. Example 7 Let X1 , . . . , Xn Bernoulli(p). Let the prior be p Beta(, ). Hence (p) = and () =
0

( + ) ()()

t1 et dt.

Set Y =

Xi . Then (p|X) pY 1 pnY p1 1 p1 pY +1 1 pnY +1 .

likelihood prior

Therefore, p|X Beta(Y + , n Y + ). (See page 325 for more details.) The Bayes estimator is p= where p= , + = + . ++n Y + Y + = = (1 )pmle + p (Y + ) + (n Y + ) ++n

This is an example of a conjugate prior. Example 8 Let X1 , . . . , Xn N (, 2 ) with 2 known. Let N (m, 2 ). Then
2 n E(|X) = 2 X + 2 + 2 + n
2

2 n

and Var(|X) = 2 2 /n 2 . 2 + n

MSE

The mean squared error (MSE) is E ( )2 = The bias is B = E () and the variance is V = Var (). Theorem 9 We have M SE = B 2 + V. Proof. Let m = E (). Then M SE = E ( )2 = E ( m + m )2 = E ( m)2 + (m )2 + 2E ( m)(m ) = E ( m)2 + (m )2 = V + B 2 . ((x1 , . . . , xn ) )2 f (x1 ; ) f (xn ; )dx1 . . . dxn .

An estimator is unbiased if the bias is 0. In that case, the MSE = Variance. There is often a tradeo between bias and variance. So low bias can imply high variance and vice versa. Example 10 Let X1 , . . . , Xn N (, 2 ). Then E(X) = , The MSEs are 2 E(X ) = , n
2

E(S 2 ) = 2 .

See p 331 for calculations.

2 4 E(S ) = . n1
2 2 2

Best Unbiased Estimators

What is the smallest variance of an unbiased estimator? This was once considered an important question. Today we consider it not so important. There is no reason to require an estmator to be unbiased. Having small MSE is more important. However, for completeness, we will briey consider the question. An estimator W is UMVUE (Uniform Minimum Variance Unbiased Estimator) for () if (i) E (W ) = () for all and (ii) if E (W ) = () for all then Var (W ) Var (W ). The Cramer-Rao inequality gives a lower bound on the variance of any unbaised estimator. The bound is: Var (W )
2 d EW d

log f (X; )

( ())2 . In ()

There is also a link with suciency. Theorem 11 The Rao-Blackwell Theorem. Let W be an unbiased estimator of () and let T be a sucient statistic. Dene W = (T ) = E(W |T ). Then W is unbiased and Var (W ) Var (W ) for all . Note that is a well-dened estimator since, by suciency, it does not depend on . Proof. We have E (W ) = E (E(W |T )) = E (W ) = () so W is unbiased. Also, Var (W ) = Var (E(W |T )) + E (Var(W |T )) = Var (W ) + E (Var(W |T )) Var (W ).

Ignore the material on completeness.

Lecture Notes 8 1 Minimax Theory

Suppose we want to estimate a parameter using data X n = (X1 , . . . , Xn ). What is the best possible estimator = (X1 , . . . , Xn ) of ? Minimax theory provides a framework for answering this question.

1.1

Introduction

Let = (X n ) be an estimator for the parameter . We start with a loss function L(, ) that measures how good the estimator is. For example: L(, ) = ( )2 L(, ) = | | L(, ) = | |p L(, ) = 0 if = or 1 if = L(, ) = I(| | > c) L(, ) = log
p(x; ) b p(x; )

squared error loss, absolute error loss, Lp loss, zeroone loss, large deviation loss, KullbackLeibler loss.

p(x; )dx

If = (1 , . . . , k ) is a vector then some common loss functions are

k

L(, ) = || || =

j=1 k

(j j )2 ,
1/p

L(, ) = || ||p =

j=1

|j j |

When the problem is to predict a Y {0, 1} based on some classier h(x) a commonly used loss is L(Y, h(X)) = I(Y = h(X)). For real valued prediction a common loss function is L(Y, Y ) = (Y Y )2 .

The risk of an estimator is R(, ) = E L(, ) = L(, (x1 , . . . , xn ))p(x1 , . . . , xn ; )dx. 1 (1)

When the loss function is squared error, the risk is just the MSE (mean squared error): R(, ) = E ( )2 = Var () + bias2 . (2)

If we do not state what loss function we are using, assume the loss function is squared error.

The minimax risk is Rn = inf sup R(, )

b

where the inmum is over all estimators. An estimator is a minimax estimator if sup R(, ) = inf sup R(, ).
b

Example 1 Let X1 , . . . , Xn N (, 1). We will see that X n is minimax with respect to many dierent loss functions. The risk is 1/n. Example 2 Let X1 , . . . , Xn be a sample from a density f . Let F be the class of smooth densities (dened more precisely later). We will see (later in the course) that the minimax risk for estimating f is Cn4/5 .

1.2

Comparing Risk Functions

To compare two estimators, we compare their risk functions. However, this does not provide a clear answer as to which estimator is better. Consider the following examples. Example 3 Let X N (, 1) and assume we are using squared error loss. Consider two estimators: 1 = X and 2 = 3. The risk functions are R(, 1 ) = E (X )2 = 1 and R(, 2 ) = E (3 )2 = (3 )2 . If 2 < < 4 then R(, 2 ) < R(, 1 ), otherwise, R(, 1 ) < R(, 2 ). Neither estimator uniformly dominates the other; see Figure 1. Example 4 Let X1 , . . . , Xn Bernoulli(p). Consider squared error loss and let p1 = X. Since this has zero bias, we have that R(p, p1 ) = Var(X) = Another estimator is p2 = p(1 p) . n

Y + ++n 2

3 2 1 0 0 1 2 3 4 5 R(, 2 ) R(, 1 )

Figure 1: Comparing two risk functions. Neither risk function dominates the other at all values of . where Y =
n i=1

Xi and and are positive constants.1 Now,

2

R(p, p2 ) = Varp (p2 ) + (biasp (p2 ))2 = Varp Y + ++n + Ep Y + ++n

2

np(1 p) + = ( + + n)2 Let = = n/4. The resulting estimator is p2 = and the risk function is R(p, p2 ) =

np + p ++n

Y + n/4 n+ n n . 4(n + n)2

The risk functions are plotted in gure 2. As we can see, neither estimator uniformly dominates the other. These examples highlight the need to be able to compare risk functions. To do so, we need a one-number summary of the risk function. Two such summaries are the maximum risk and the Bayes risk. The maximum risk is R() = sup R(, ) (3)

This is the posterior mean using a Beta (, ) prior.

Risk

Figure 2: Risk functions for p1 and p2 in Example 4. The solid curve is R(p1 ). The dotted line is R(p2 ). and the Bayes risk under prior is B () = R(, )()d. (4)

Example 5 Consider again the two estimators in Example 4. We have R(p1 ) = max and R(p2 ) = max
p

1 p(1 p) = 0p1 n 4n

n n 2 = . 4(n + n) 4(n + n)2

Based on maximum risk, p2 is a better estimator since R(p2 ) < R(p1 ). However, when n is large, R(p1 ) has smaller risk except for a small region in the parameter space near p = 1/2. Thus, many people prefer p1 to p2 . This illustrates that one-number summaries like maximum risk are imperfect. These two summaries of the risk function suggest two dierent methods for devising estimators: choosing to minimize the maximum risk leads to minimax estimators; choosing to minimize the Bayes risk leads to Bayes estimators. An estimator that minimizes the Bayes risk is called a Bayes estimator. That is, B () = inf B ()
e

(5)

where the inmum is over all estimators . An estimator that minimizes the maximum risk is called a minimax estimator. That is, sup R(, ) = inf sup R(, )
e

(6)

where the inmum is over all estimators . We call the right hand side of (6), namely, Rn Rn () = inf sup R(, ),
b

(7)

the minimax risk. Statistical decision theory has two goals: determine the minimax risk Rn and nd an estimator that achieves this risk. Once we have found the minimax risk Rn we want to nd the minimax estimator that achieves this risk: sup R(, ) = inf sup R(, ). (8)
b

Sometimes we settle for an asymptotically minimax estimator sup R(, ) inf sup R(, ) n
b

(9)

where an bn means that an /bn 1. Even that can prove too dicult and we might settle for an estimator that achieves the minimax rate, sup R(, )

inf sup R(, ) n

b

(10)

where an

bn means that both an /bn and bn /an are both bounded as n .

1.3

Bayes Estimators

Let be a prior distribution. After observing X n = (X1 , . . . , Xn ), the posterior distribution is, according to Bayes theorem, P( A|X n ) = p(X1 , . . . , Xn |)()d = p(X1 , . . . , Xn |)()d
A

L()()d L()()d
A

(11)

where L() = p(xn ; ) is the likelihood function. The posterior has density (|xn ) = p(xn |)() m(xn ) (12)

where m(xn ) = p(xn |)()d is the marginal distribution of X n . Dene the posterior risk of an estimator (xn ) by r(|xn ) = L(, (xn ))(|xn )d. 5 (13)

Theorem 6 The Bayes risk B () satises B () = r(|xn )m(xn ) dxn . (14)

Let (xn ) be the value of that minimizes r(|xn ). Then is the Bayes estimator. Proof.Let p(x, ) = p(x|)() denote the joint density of X and . We can rewrite the Bayes risk as follows: B () = = = R(, )()d = L(, (xn ))p(x|)dxn ()d L(, (xn ))(|xn )m(xn )dxn d r(|xn )m(xn ) dxn .

L(, (xn ))p(x, )dxn d =

L(, (xn ))(|xn )d m(xn ) dxn =

If we choose (xn ) to be the value of that minimizes r(|xn ) then we will minimize the integrand at every x and thus minimize the integral r(|xn )m(xn )dxn . Now we can nd an explicit formula for the Bayes estimator for some specic loss functions. Theorem 7 If L(, ) = ( )2 then the Bayes estimator is (xn ) = (|xn )d = E(|X = xn ). (15)

If L(, ) = | | then the Bayes estimator is the median of the posterior (|xn ). If L(, ) is zeroone loss, then the Bayes estimator is the mode of the posterior (|xn ). Proof.We will prove the theorem for squared error loss. The Bayes estimator (xn ) minimizes r(|xn ) = ( (xn ))2 (|xn )d. Taking the derivative of r(|xn ) with respect to (xn ) and setting it equal to zero yields the equation 2 ( (xn ))(|xn )d = 0. Solving for (xn ) we get 15. Example 8 Let X1 , . . . , Xn N (, 2 ) where 2 is known. Suppose we use a N (a, b2 ) prior for . The Bayes estimator with respect to squared error loss is the posterior mean, which is
b2 n X+ (X1 , . . . , Xn ) = 2 + 2 2+ b b n
2

2 n

a.

(16)

1.4

Minimax Estimators

Finding minimax estimators is complicated and we cannot attempt a complete coverage of that theory here but we will mention a few key results. The main message to take away from this section is: Bayes estimators with a constant risk function are minimax. Theorem 9 Let be the Bayes estimator for some prior . If R(, ) B () for all then is minimax and is called a least favorable prior. Proof.Suppose that is not minimax. Then there is another estimator 0 such that sup R(, 0 ) < sup R(, ). Since the average of a function is always less than or equal to its maximum, we have that B (0 ) sup R(, 0 ). Hence, B (0 ) sup R(, 0 ) < sup R(, ) B ()

(17)

(18)

which is a contradiction. Theorem 10 Suppose that is the Bayes estimator with respect to some prior . If the risk is constant then is minimax. Proof.The Bayes risk is B () = . Now apply the previous theorem. R(, )()d = c and hence R(, ) B () for all

Example 11 Consider the Bernoulli model with squared error loss. In example 4 we showed that the estimator n Xi + n/4 p(X n ) = i=1 n+ n has a constant risk function. This estimator is the posterior mean, and hence the Bayes estimator, for the prior Beta(, ) with = = n/4. Hence, by the previous theorem, this estimator is minimax. Example 12 Consider again the Bernoulli but with loss function L(p, p) = Let p(X n ) = p =
n i=1

(p p)2 . p(1 p) 1 p(1 p) p(1 p) n 1 n

Xi /n. The risk is (p p)2 p(1 p) = =

R(p, p) = E

which, as a function of p, is constant. It can be shown that, for this loss function, p(X n ) is the Bayes estimator under the prior (p) = 1. Hence, p is minimax. 7

What is the minimax estimator for a Normal model? To answer this question in generality we rst need a denition. A function is bowl-shaped if the sets {x : (x) c} are convex and symmetric about the origin. A loss function L is bowl-shaped if L(, ) = ( ) for some bowl-shaped function . Theorem 13 Suppose that the random vector X has a Normal distribution with mean vector and covariance matrix . If the loss function is bowl-shaped then X is the unique (up to sets of measure zero) minimax estimator of . If the parameter space is restricted, then the theorem above does not apply as the next example shows. Example 14 Suppose that X N (, 1) and that is known to lie in the interval [m, m] where 0 < m < 1. The unique, minimax estimator under squared error loss is (X) = m emX emX emX + emX .

This is the Bayes estimator with respect to the prior that puts mass 1/2 at m and mass 1/2 at m. The risk is not constant but it does satisfy R(, ) B () for all ; see Figure 3. Hence, Theorem 9 implies that is minimax. This might seem like a toy example but it is not. The essence of modern minimax theory is that the minimax risk depends crucially on how the space is restricted. The bounded interval case is the tip of the iceberg. Proof That X n is Minimax Under Squared Error Loss. Now we will explain why X n is justied by minimax theory. Let X Np (, I) be multivariate Normal with mean vector = (1 , . . . , p ). We will prove that = X is minimax when L(, ) = || ||2 . Assign the prior = N (0, c2 I). Then the posterior is |X = x N c2 x c2 , I . 1 + c2 1 + c2 (19)

The Bayes risk for an estimator is R () = R(, )()d which is minimized by the posterior mean = c2 X/(1 + c2 ). Direct computation shows that R () = pc2 /(1 + c2 ). Hence, if is any estimator, then pc2 = R () R ( ) 2 1+c = R( , )d() sup R( , ).

(20) (21)

We have now proved that R() pc2 /(1 + c2 ) for every c > 0 and hence R() p. But the risk of = X is p. So, = X is minimax. 8 (22)

-0.5

0.5

Figure 3: Risk function for constrained Normal with m=.5. The two short dashed lines show the least favorable prior which puts its mass at two points.

1.5

Maximum Likelihood

For parametric models that satisfy weak regularity conditions, the maximum likelihood estimator is approximately minimax. Consider squared error loss which is squared bias plus variance. In parametric models with large samples, it can be shown that the variance term dominates the bias so the risk of the mle roughly equals the variance:2 R(, ) = Var () + bias2 Var (). The variance of the mle is approximately Var() Hence, nR(, )
1 nI()

(23)

where I() is the Fisher information. (24)

1 . I()

For any other estimator , it can be shown that for large n, R(, ) R(, ). So the maximum likelihood estimator is approximately minimax. This assumes that the dimension of is xed and n is increasing.

1.6

The Hodges Example

Here is an interesting example about the subtleties of optimal estimators. Let X1 , . . . , Xn N (, 1). The mle is n = X n = n1 n Xi . But consider the following estimator due to i=1
2

Typically, the squared bias is order O(n2 ) while the variance is of order O(n1 ).

Hodges. Let Jn = and dene n = 1 n1/4 , 1 n1/4 (25)

X n if X n Jn / 0 if X n Jn .

(26)

Suppose that = 0. Choose a small so that 0 is not contained in I = ( , + ). By the law of large numbers, P(X n I) 1. In the meantime Jn is shrinking. See Figure 4. Thus, for n large, n = X n with high probability. We conclude that, for any = 0, n behaves like X n. When = 0, P(X n Jn ) = P(|X n | n1/4 ) = P( n|X n | n1/4 ) = P(|N (0, 1)| n1/4 ) 1. (27) (28)

Thus, for n large, n = 0 = with high probability. This is a much better estimator of than X n . We conclude that Hodges estimator is like X n when = 0 and is better than X n when = 0. So X n is not the best estimator. n is better. Or is it? Figure 5 shows the mean squared error, or risk, Rn () = E(n )2 as a function of (for n = 1000). The horizontal line is the risk of X n . The risk of n is good at = 0. At any , it will eventually behave like the risk of X n . But the maximum risk of n is terrible. We pay for the improvement at = 0 by an increase in risk elsewhere. There are two lessons here. First, we need to pay attention to the maximum risk. Second, it is better to look at uniform asymptotics limn sup Rn () rather than pointwise asymptotics sup limn Rn ().

10

Jn [ n [ n
1/4

I ] n ] n
1/4 1/4

) +

n1/2 Jn n1/2 0

1/4

Figure 4: Top: when = 0, X n will eventually be in I and will miss the interval Jn . Bottom: when = 0, X n is about n1/2 away from 0 and so is eventually in Jn .

0.000
1

0.005

0.010

0.015

Figure 5: The risk of the Hodges estimator for n = 1000 as a function of . The horizontal line is the risk of the sample mean.

11

36-705/10-705 Summary of Minimax Theory Larry Wasserman October 5, 2011 1. Loss L(, ) where = (X1 , . . . , Xn ). Remember that is a function of X1 , . . . , Xn . 2. Risk R(, ) = E [L(, )] = 3. If L(, ) = ( )2 then R(, ) = E ( )2 = MSE = bias2 + variance. 4. Maximum risk: we dene how good an estimator is by its maximum risk sup R(, ).

L(, (x1 , . . . , xn ))p(x1 , . . . , xn ; )dx1 dxn .

5. Minimax risk: Rn = inf sup R(, ).

b

6. An estimator is minimax if sup R(, ) = Rn .

7. The Bayes risk for an estimator , with respect to a prior is B () = R(, )()d.

8. An estimator is the Bayes estimator with respect to a prior if B ( ) = inf B ().

b

In other words, minimizes B () over all estimators. 9. The Bayes risk can we re-written as B () = r() m(x1 , . . . , xn )dx1 dxn

where m(x1 , . . . , xn ) = p(x1 , . . . , xn ; )()d and r() = L(, )p(|x1 , . . . , xn )d. Hence, to minimize B ( ) is suces to minimize r(). 10. Key Theorem: Suppose that (i) is the Bayes estimator with respect to some prior and (ii) R(, ) is constant. Then is minimax. 1

11. Bounds. Sometimes it is hard to nd Rn so it is useful to nd a lower bound and an upper bound on the minimax risk. The following result is helpful: Theorem: Let by a Bayes estimator with respect to some prior . Let be any estimator. Then: B ( ) Rn sup R(, ). (1)

Proof of the Lower Bound. Let be the Bayes estimator for some prior . Let be any other estimator. Then, B ( ) B () = R(, )()d sup R(, ).

Take the inf over all and conclude that B ( ) inf sup R(, ) = Rn .
b n

Hence, Rn B ( ). Proof of the Upper bound. Choose any estimator . Then Rn = inf sup R(, ) sup R(, ).
b

12. How to prove that X n is minimax for the Normal model. Let X1 , . . . , Xn N (, 2 ) where 2 is known. Let L(, ) = ( )2 .

(a) First we show that Rn = 2 /n. We do this by getting a lower bound and an upper bound on Rn . (b) Lower Bound. Let = N (0, c2 ). The posterior p(|X1 , . . . , Xn ) is N (a, b2 ) where 1 nX/ 2 and b2 = 1 a= 1 n n . + 2 + 2 c2 c2 The Bayes estimator minimizes r() = ( )2 p(|x1 , . . . , xn )d. This is minimized by = p(|x1 , . . . , xn )d = E(|X1 , . . . , Xn ). But E(|X1 , . . . , Xn ) = a. So the Bayes esimator is nX/ 2 = 1 n . + 2 c2 Next we compute R( , ). This means we need to compute the MSE of . The bias is 2 /( 2 + nc2 ). The variance of is nc4 2 /( 2 + nc2 )2 . So R(, ) = bias2 + variance = 2 4 nc4 2 2 4 + nc4 2 + 2 = . ( 2 + nc2 )2 ( + nc2 )2 ( 2 + nc2 )2

Let us now compute the Bayes risk of this estimator. It is B ( ) = = By (1), this proves that Rn 2 . 2 2 + n c R(, )()d = 4 c2 + nc4 2 = ( 2 + nc2 )2 4 2 ()d + nc4 2 ( 2 + nc2 )2

2 . 2 +n c2

(c) Upper Bound. Choose = X n . Then R(, ) = 2 /n. By (1), Rn sup R(, ) =

2 . n

(d) Combining the lower and upper bound we see that 2 2 Rn . 2 n +n c2 This bound is true for all c > 0. If take the limit as c then we get that 2 Rn = . We have succeeded in nding the minimax risk Rn . n (e) The last step is to nd a minimax estimator. We have to nd an estimator whose maximum risk is Rn . But we already saw that X has maximum risk equal to Rn . Hence X n is minimax.

Lecture Notes 9 Asymptotic (Large Sample) Theory 1 Review of o, O, etc.

1. an = o(1) mean an 0 as n . 2. A random sequence An is op (1) if An 0 as n . 3. A random sequence An is op (bn ) if An /bn 0 as n . P 4. np op (1) = op (np ), so n op (1/ n) = op (1) 0. 5. op (1) op (1) = op (1).
P P

1. an = O(1) if |an | is bounded by a constant as n . 2. A random sequence Yn is Op (1) if for every limn P (|Yn | > M ) < as n . 3. A random sequence Yn is Op (bn ) if Yn /bn is Op (1). 4. If Yn Y , then Yn is Op (1). 5. If n(Yn c) Y then Yn = OP (1/ n). (potential test qustion: prove this) 6. Op (1) Op (1) = Op (1). 7. op (1) Op (1) = op (1). > 0 there exists a constant M such that

Distances Between Probability Distributions

Let P and Q be distributions with densities p and q. We will use the following distances between P and Q. 1. Total variation distance V (P, Q) = supA |P (A) Q(A)|. 1

2. L1 distance d1 (P, Q) =

|p q|. ( p q)2 . p log(p/q).

3. Hellinger distance h(P, Q) =

4. Kullback-Leibler distance K(P, Q) = 5. L2 distance d2 (P, Q) = (p q)2 .

Here are some properties of these distances: 1. V (P, Q) = 1 d1 (P, Q). (prove this!) 2 2. h2 (P, Q) = 2(1 pq). 2V (P, Q).

3. V (P, Q) h(P, Q) 4. h2 (P, Q) K(P, Q). 5. V (P, Q) h(P, Q) 6. V (P, Q)

K(P, Q).

K(P, Q)/2.

Consistency
n
P

n = T (X n ) is consistent for if

as n . In other words, n = op (1). Here are two common ways to prove that n consistent.

Method 1: Show that, for all > 0, P(|n | ) 0.

Method 2. Prove convergence in quadratic mean: MSE(n ) = Bias2 (n ) + Var(n ) 0. If bias 0 and var 0 then n which implies that n . Example 1 Bernoulli(p). The mle p has bias 0 and variance p(1 p)/n 0. So p p and is consistent. Now let = log(p/(1p)). Then = log(p/(1 p)). Now = g(p) where g(p) = log(p/(1 p)). By the continuous mapping theorem, so this is consistent. Now consider p= Then bias = E(p) p = and var = So this is consistent. Example 2 X1 , . . . , Xn Uniform(0, ). Let n = X(n) . By direct proof (we did it earlier) we have n . Method of moments estimators are typically consistent. Consider one parameter. Recall that () = m where m = n1
P n i=1 P P P qm p

X +1 . n+1 p1 0 n(1 + n)

p(1 p) 0. n

Xi . Assume that 1 exists and is continuous. So

= 1 (m). By the WLLN m (). So, by the continuous mapping Theorem, n = 1 (m) 1 (()) = .
P

Consistency of the MLE

Under regularity conditions (see page 516), the mle is consistent. Let us prove this in a special case. This will also reveal a connection between the mle and Hellinger distance.

Suppose that the model consists of nitely many distinct densities {p0 , p1 , . . . , pN }. The likelihood function is L(pj ) =
i=1 n

pj (Xi ).

The mle p is the density pj that maximizes L(pj ). Without loss of generality, assume that the true density is p0 . Theorem 3 P(p = p0 ) 0 as n . Proof. Let us begin by rst proving an inequality. Let P L(pj ) 2 > en j /2 L(p0 )
n j n

= h(p0 , pj ). Then, for j = 0, pj (Xi ) 4 > en j /2 p0 (Xi ) E pj (Xi ) p0 (Xi )

n

= P
i=1

pj (Xi ) 2 > en j /2 p0 (Xi )

n

=P
i=1 n

e = e

n 2 /4 j

E
i=1

pj (Xi ) p0 (Xi )
n

=e

n 2 /4 j i=1

n 2 /4 j

pj p0

=e

n 2 /4 j

h2 (p0 , pj ) 1 2
2 2

=e
2

n 2 /4 j

2 j

2 p0 pj and also that log(1 x) x for x > 0. We used the fact that h2 (p0 , pj ) = 2 2 Let = min{ 1 , . . . ,
N }.

= en j /4 exp n log 1 Then

2 j

en j /4 en j /2 = en j /2 .

P(p = p0 ) P
N

L(pj ) 2 > en j /2 for some j L(p0 ) P L(pj ) 2 > en j /2 L(p0 )

2 2 /2

j=1 N

j=1

en j /2 N en

0.

We can prove a similar result using Kullback-Leibler distance as follows. Let X1 , X2 , . . . be iid F . Let 0 be the true value of and let be some other value. We will show that 4

L(0 )/L() > 1 with probability tending to 1. We assume that the model is identiable; this means that 1 = 2 implies that K(1 , 2 ) > 0 where K is the Kullback-Leibler distance. Theorem 4 Suppose the model is identiable. Let 0 be the true value of the parameter. For any = 0 P as n . Proof. We have 1 ( (0 ) ()) = n
p

L(0 ) >1 L()

1 n

i=1

log p(Xi ; 0 )

1 n

log p(Xi ; )
i=1

E(log p(X; 0 )) E(log p(X; )) = = (log p(x; 0 ))p(x; 0 )dx log p(x; 0 )dx (log p(x; ))p(x; 0 )dx p(x; 0 ) p(x; ) = K(0 , ) > 0.

So P L(0 ) >1 L() = P ( (0 ) () > 0) = P 1 ( (0 ) ()) > 0 n 1.

This is not quite enough to show that n 0 . Example 5 Inconsistency of an mle. In all examples so far n , but the number of parameters is xed. What if the number of parameters also goes to ? Let Y11 , Y12 N (1 , 2 ) Y21 , Y22 N (2 , 2 ) . . . . . . Yn1 , Yn2 N (n , 2 ). 5

Some calculations show that =

2

i=1 j=1

(Yij Y i )2 . 2n

It is easy to show (good test question) that 2 . 2

2 p

Note that the modied estimator 2 2 is consistent. The reason why consistency fails is because the dimension of the parameter space is increasing with n. Theorem 6 Under regularity conditions on the model {p(x; ) : }, the mle is consistent.

Score and Fisher Information

The score and Fisher information are the key quantities in many aspects of statistical inference. (See Section 7.3.2 of CB.) Suppose for now that R. L() = p(xn ; ) () = log L() S() =

() score function.

Recall that the value that maximizes L() is the maximum likelihood estimator (mle). Equivalently, maximizes (). Note that = T (X1 , . . . , Xn ) is a function of the data. Often, we get by dierentiation. In that case solves S() = 0. Well discuss the mle in detail later.

Some Notation: Recall that E (g(X)) g(x)p(x; )dx.

Theorem 7 Under regularity conditions, E [S()] = 0. In other words, log p(x1 , . . . , xn ; ) p(x1 , . . . , xn ; )dx1 . . . dxn = 0.

That is, if the expected value is taken at the same as we evaluate S, then the expectation is 0. This does not hold when the s mismatch: E0 [S(1 )] = 0. Proof. E [S()] = = = log p(xn ; ) p(xn ; ) dx1 dxn p(xn ; ) p(xn ; ) dx1 dxn p(xn ; ) p(xn ; ) dx1 dxn
1

= 0.

Example 8 Let X1 , . . . , Xn N (, 1). Then

n

S() =
i=1

(Xi ). and

Warning: If the support of f depends on , then

cannot be switched.

The next quantity of interest is the Fisher Information or Expected Information. The information is used to calculate the variance of quantities that arise in inference problems 7

such as the mle . It is called information because it tells how much information is in the likelihood about . The denition is: I() = E [S()2 ] = E [S()2 ] E [S()]
2

= Var (S()) since E [S()] = 0 2 = E 2 () easiest way to calculate

We will prove the nal equality under regularity conditions shortly. I() grows linearly in n, so for an iid sample, a more careful notation would be In () In () = E = nE 2 l() = E 2
2 n

i=1

2 log p(Xi ; ) 2

log p(X1 ; ) = nI1 (). 2

Note that the Fisher information is a function of in two places: The derivate is w.r.t. and the information is evaluated at a particular value of . The expectation is w.r.t. also. The notation only allows for a single value of because the two quantities should match. A related quantity of interest is the observed information, dened as 2 In () = 2 () =
P n

i=1

2 log p(Xi ; ). 2

1 By the LLN n In () I1 (). So observed information can be used as a good approximation

to the Fisher information. Let us prove the identity: E [S()2 ] = E note that p=1 p =0 p =0 8 p p=0 p E p p = 0.
2 2

() . For simplicity take n = 1. First

Let

= log p and S =

= p /p. Then

= (p /p) (p /p)2 and p p

2

V (S) = E(S 2 ) (E(S))2 = E(S 2 ) = E = E = E p p

2

E p p

p p p p
2

= E( ). Why is I() called Information? Later we will see that Var() 1/In (). The Vector Case. Let = (1 , , K ). L() and () are dened as before. S() =
() i i=1, ,K

a vector of dimension K

Information I() = Var[S()] is the variance-covariance matrix of S() = [Iij ]ij=1, ,k where Iij = E 2 () . i j

I()1 is the asymptotic variance of . (This is the inverse of the matrix, evaluated at the proper component of the matrix.)

Example 9 X1 , , Xn N (, )
n

L(, ) =
i=1

1 exp 2

1 (xi )2 2

n 2

exp

1 (xi )2 2

n 1 (, ) = K log (xi )2 2 2 1 (xi ) S(, ) = n 1 2 + 2 2 (xi )2 1 n (xi ) 2 I(, ) = E 1 n 1 2 (xi ) 2 2 3 (xi ) 2 n 0 = n 0 2 2 You can check that E (S) = (0, 0)T .

6
If

Eciency and Asymptotic Normality

n(n ) N (0, v 2 ) then we call v 2 the asymptotic variance of n . This is not the same

as the limit of the variance which is limn nVar(n ). Consider X n . In this case, the asymptotic variance is 2 . We also have that limn nVar(X n ) = 2 . In this case, they are the same. In general, the latter may be larger (or even innite). Example 10 (Example 10.1.10) Suppose we observe Yn N (0, 1) with probability pn and Wn Bernoulli(pn )
2 Yn |Wn N (0, Wn + (1 Wn )n ).

2 Yn N (0, n ) with probability 1 pn . We can write this as a hierachical model:

Now, Var(Yn ) = VarE(Yn |Wn ) + EVar(Yn |Wn )

2 2 = Var(0) + E(Wn + (1 Wn )n ) = pn + (1 pn )n .

10

2 Suppose that pn 1, n and that (1 pn )n . Then Var(Yn ) . Then

P(Yn a) = pn P(Z a) + (1 pn )P(Z a/n ) P(Z a) and so Yn N (0, 1). So the asymptotic variance is 1.

Suppose we want to estimate (). Let v() = where I() = Var log p(X; ) = E 2 log p(X; ) . 2 | ()|2 I()

We call v() the Cramer-Rao lower bound. Generally, any well-behaved estimator will have a limiting variance bigger than or equal to v(). We say that Wn is ecient if n(Wn ()) N (0, v()).

Theorem 11 Let X1 , X2 , . . . , be iid. Assume that the model satises the regularity conditions in 10.6.2. Let be the mle. Then n( () ()) So () is consistent and ecient. We will now prove the asymptotic normality of the mle. Theorem 12 Hence, n = + OP Proof. By Taylors theorem 0 = () = () + ( ) () + . 11 1 n . n(n ) N 0, 1 I() . N (0, v()).

Hence n( ) Now 1 1 A = () = n n n

1 n 1 n n

() ()

A . B n(S 0)

S(, Xi ) =
i=1

where S(, Xi ) is the score function based on Xi . Recall that E(S(, Xi )) = 0 and Var(S(, Xi )) = I(). By the central limit theorem, A WLLN, B E( ) = I(). By Slutskys theorem A B So n( ) N 0, 1 I() . I()Z = I() Z I() =N 0, 1 I() .
P

N (0, I()) =

I()Z where Z N (0, 1). By the

Theorem 11 follows by the delta method: implies that n( (n ) ()) N (0, ( ())2 /I()). n(n ) N (0, 1/I())

The standard error of is se = The estimated standard error is se = The standard error of = () is se = | ()| = nI() 12 | ()| . In () 1 In () . 1 = nI() 1 . In ()

The estimated standard error is se = | ()| In () .

ent+n ln , l() = nt + n ln , S() = E[l ()] =

n , 2

Example 13 X1 , , Xn iid Exponential (). Let t = x. So: p(z; ) = ex , L() =

n

N , n .

nt =

1 t

1 , X

l () =

n , 2

I() =

Example 14 X1 , . . . , Xn Bernoulli(p). The mle is p = X. The Fisher information for n = 1 is I(p) = So Informally, pN estimated standard error of the mle is se = p(1 p) . n p, p(1 p) n . n(p p) 1 . p(1 p) N (0, p(1 p)).

The asymptotic variance is p(1 p)/n. This can be estimated by p(1 p)/n. That is, the

Now suppose we want to estimate = p/(1 p). The mle is = p/(1 p). Now p 1 = p 1 p (1 p)2 The estimated standard error is se( ) = p(1 p) 1 = n (1 p)2 p . n(1 p)3

7
If

Relative Eciency
n(Wn ()) n(Vn ()) 13

2 N (0, W ) 2 N (0, V )

then the asymptotic relative eciency (ARE) is ARE(Vn , Wn ) =

2 W . 2 V

Example 15 (10.1.17). Let X1 , . . . , Xn Poisson(). The mle of is X. Let = P(Xi = 0). So = e . Dene Yi = I(Xi = 0). This suggests the estimator 1 Wn = n Another estimator is the mle Vn = e .
b n

Yi .
i=1

The delta method gives e2 Var(Vn ) . n We have n(Wn ) n(Vn ) N (0, e (1 e )) N (0, e2 ).

So ARE(Wn , Vn ) = e 1. 1

Since the mle is ecient, we know that, in general, ARE(Wn , mle) 1.

Robustness

The mle is ecient only if the model is right. The mle can be bad if the model is wrong. That is why we should consider using nonparametric methods. One can also replace the mle with estimators that are more robust. 14

that we have a perturbed model Xi is N (, 2 ) with probability 1 and Xi is Cauchy with probability . Then, Var(X n ) = . Consider the median Mn . We will show that ARE(median, mle) = .64.

Suppose we assume that X1 , . . . , Xn N (, 2 ). The mle is n = X n . Suppose, however

But, under the perturbed model the median still performs well while the mle is terrible. In other words, we can trade eciency for robustness. Let us now nd the limiting distribution of Mn . Let Yi = I(Xi + a/ n). Then Yi Bernoulli(pn ) where a a 1 pn = P ( + a/ n) = P () + p() + o(n1/2 ) = + p() + o(n1/2 ). 2 n n
i

Also,

Yi has mean npn and standard deviation n = npn (1 pn ). Yi n+1 . 2

Note that, a Mn + n Then, a P( n(Mn ) a) = P Mn + n = P Now,

n+1 2 i

if and only if
i

=P
i n+1 2

Yi .

n+1 2

Yi npn n

npn n

npn 2ap() n Z a 2p()

and hence Z P( n(Mn ) a) P(Z 2ap()) = P a 2p() so that n(Mn ) For a standard Normal, (2p(0))2 = .64. 15 N 0, 1 (2p())2 . =P

Lecture Notes 10 Hypothesis Testing 1 Introduction

(See Chapter 8 and Chapter 10.3.) Null hypothesis: H0 : 0 Alternative hypothesis: H1 : 1 where 0 1 = . Example 1 X1 , . . . , Xn Bernoulli(p). H0 : p = 1 2 1 H1 : p = . 2

The question is not whether H0 is true or false. The question is whether there is sucient evidence to reject H0 , much like a court case. Our possible actions are: reject H0 or retain (dont reject) H0 . Decision Retain H0 Reject H0 Type I error (false positive)

H0 true

H1 true

Type II error (false negative)

Warning: Hypothesis testing should only be used when it is appropriate. Often times, people use hypothesis tetsing when it would be much more appropriate to use condence intervals (which is the next topic).

Constructing Tests
1. Choose a test statistic W = W (X1 , . . . , Xn ). 2. Choose a rejection region R. 3. If W R we reject H0 otherwise we retain H0 .

Example 2 X1 , . . . , Xn Bernoulli(p). H0 : p = Let W = n1

n i=1

1 2

1 H1 : p = . 2

Xi . Let R = {xn : |w(xn ) 1/2| > }. So we reject H0 if |W 1/2| > .

We need to choose W and R so that the test has good statistical properties. We will consider the following tests: 1. Neyman-Pearson Test 2. Wald test 3. Likelihood Ratio Test (LRT) 4. the permutation test 5. the score test (optional) Before we discuss these methods, we rst need to talk about how we evaluate tests.

Evaluating Tests

Suppose we reject H0 when X n = (X1 , . . . , Xn ) R. Dene the power function by () = P (X n R). We want () to be small when 0 and we want () to be large when 1 . The general strategy is: 2

1. Fix [0, 1]. 2. Now try to maximize () for 1 subject to () for 0 . We need the following denitions. A test is size if sup () = .
0

A test is level if
0

sup () .

A size test and a level test are almost the same thing. The distinction is made bcause sometimes we want a size test and we cannot construct a test with exact size but we can construct one with a smaller error rate. Example 3 X1 , . . . , Xn N (, 2 ) with 2 known. Suppose H0 : = 0 , H1 : > 0 .

This is called a one-sided alternative. Suppose we reject H0 if W > c where W = Then () = P X n 0 >c / n Xn >c+ 0 = P / n / n 0 = P Z >c+ / n 0 = 1 c+ / n X n 0 . / n

where is the cdf of a standard Normal. Now sup () = (0 ) = 1 (c).

To get a size test, set 1 (c) = so that c = z where z = 1 (1 ). Our test is: reject H0 when W = X n 0 > z . / n

Example 4 X1 , . . . , Xn N (, 2 ) with 2 known. Suppose H0 : 0 , H1 : = 0 .

This is called a two-sided alternative. We will reject H0 if |W | > c where W is dened as before. Now () = P (W < c) + P (W > c) X n 0 X n 0 < c + P >c = P / n / n 0 0 = P Z < c + +P Z >c+ / n / n 0 0 = c + +1 c+ / n / n 0 0 + c = c + / n / n since (x) = 1 (x). The size is (0 ) = 2(c). To get a size test we set 2(c) = so that c = 1 (/2) = 1 (1 /2) = z/2 . The test is: reject H0 when |W | = X n 0 > z/2 . / n

The Neyman-Pearson Test

Let C denote all level tests. A test in C with power function is uniformly most powerful (UMP) if the following holds: if is the power function of any other test in C then () () for all 1 . 4

Consider testing H0 : = 0 versus H1 : = 1 . (Simple null and simple alternative.) Theorem 5 Suppose we set R= x = (x1 , . . . , xn ) : f (X1 , . . . , Xn ; 1 ) >k f (X1 , . . . , Xn ; 0 ) P0 (X n R) = . In other words, reject H0 if L(1 ) > k. L(0 ) This test is a UMP level test. This is theorem 8.3.12 in the book. The proof is short; you should read the proof. Notes: 1. Ignore the material on union-intersection tests and monotonote likelihood ratios (MLR). 2. In general it is hard to nd UMP tests. Sometimes they dont even exist. Still, we can nd tests with good properties. = xn : L(1 ) >k L(0 )

where k is chosen so that

5
Let

The Wald Test

W = n 0 . se N (0, 1). Hence, an asymptotic

Under the uusal conditions we have that under H0 , W level test is to reject when |W | > z/2 . For example, with Bernoulli data, to test H0 : p = p0 , W = You can also use W = p p0
p0 (1p0 ) n

p p0

p(1b) b p n

In other words, to compute the standard error, you can replace with an estimate or by the null value 0 .

The Likelihood Ratio Test (LRT)

This test is simple: reject H0 if (xn ) c where (xn ) = sup0 L() L(0 ) = sup L() L()

where 0 maximizes L() subject to 0 . Example 6 X1 , . . . , Xn N (, 1). Suppose H0 : = 0 , After some algebra (see page 376), n = exp (X n 0 )2 . 2 So R = {x : c} = {x : |X 0 | c } where c = 2 log c/n. Choosing c to make this level gives: reject if |W | > z/2 where W = n(X 0 ) which is the test we constructed before. Example 7 X1 , . . . , Xn N (, 2 ). Suppose H0 : 0 , Then (xn ) = L(0 , 0 ) L(, ) H1 : = 0 . H1 : = 0 .

where 0 maximizes the likelihood subject to = 0 . In the homework, you will prove that (xn ) < c corresponds to rejecting when |Tn | > k for some constant k where Tn = X n 0 . S/ n 6

Under H0 , Tn has a t-distribution with n 1 degrees of freedom. So the nal test is: reject H0 if |Tn | > tn1,/2 . This is called Students t-test. It was invented by William Gosset working at Guiness Breweries and writing under the pseudonym Srudent. Theorem 8 Consider testing H0 : = 0 versus H1 : = 0 where R. Under H0 , 2 log (X n ) Hence, if we let Wn = 2 log (X n ) then P0 (W > 2 ) 1, as n . Proof. Using a Taylor expansion: () () + ()( ) + and so 2 log (xn ) = 2 () 2 (0 ) 2 () 2 () ()( )2 = ()( )2 () = In (0 )( n( 0 ))2 = An Bn . In (0 ) Now An 1 by the WLLN and
P

2 . 1

()

( )2 = () + 2

()

( )2 2

Bn

N (0, 1). The result follows by Slutskys theorem.

Example 9 X1 , . . . , Xn Poisson(). We want to test H0 : = 0 versus H1 : = 0 . Then 2 log (xn ) = 2n[(0 ) log(0 /)]. We reject H0 when 2 log (xn ) > 2 . 1, 7

Now suppose that = (1 , . . . , k ). Suppose that H0 xes some of the parameters. Then 2 log (X n ) where = dim() dim(0 ). Example 10 Consider a multinomial with = (p1 , . . . , p5 ). So L() = py1 py5 . 5 1 Suppose we want to test H0 : p1 = p2 = p3 and p4 = p5 versus the alternative that H0 is false. In this case = 4 1 = 3. The LRT test statistic is (x ) =
n 5 i=1 5 i=1
j p0j

Y Y

pj j

where pj = Yj /n, p10 = p20 = p30 = (Y1 + Y2 + Y3 )/n, p40 = p50 = (1 3p10 )/2. These 2 . 3,

calculations are on p 491. Make sure you understand them. Now we reject H0 if 2(X n ) >

p-values

When we test at a given level we will reject or not reject. It is useful to summarize what levels we would reject at and what levels we woud not reject at. The p-value is the smallest at which we would reject H0 . In other words, we reject at all p. So, if the pvalue is 0.03, then we would reject at = 0.05 but not at = 0.01. Hence, to test at level when p < . 8

Theorem 11 Suppose we have a test of the form: reject when W (X n ) > c. Then the p-value when X n = xn is p(xn ) = sup P (W (X n ) W (xn )).
0

Example 12 X1 , . . . , Xn N (, 1). Test that H0 : = 0 versus H1 : = 0 . We reject when |W | is large, where W = n(X n 0 ). So p = P0 | n(X n 0 )| > w = P (|Z| > w) = 2(|w|). Theorem 13 Under H0 , p Unif(0, 1). Important. Note that p is NOT equal to P (H0 |X1 , . . . , Xn ). The latter is a Bayesian quantity which we will discuss later.

The Permutation Test

This is a very cool test. It is distribution free and it does not involve any asymptotic approximations. Suppose we have data X1 , . . . , X n F and Y1 , . . . , Ym G. We want to test: H0 : F = G versus H1 : F = G. Let Z = (X1 , . . . , Xn , Y1 , . . . , Ym ). Create labels L = (1, . . . , 1, 2, . . . , 2).
n values m values

A test statistic can be written as a function of Z and L. For example, if W = |X n Y n | then we can write W =
N i=1 Zi I(Li = 1) N i=1 I(Li = 1) N i=1 Zi I(Li = 2) N i=1 I(Li = 2)

where N = n + m. So we write W = g(L, Z). Dene p= 1 N! I(g(L , Z) > g(L, Z))

where L is a permutation of the labels and the sum is over all permutations. Under H0 , permuting the labels does not change the distribution. In other words, g(L, Z) has an equal chance of having any rank among all the permuted values. That is, under H0 , Unif(0, 1) and if we reject when p < , then we have a level test. Summing over all permutations is infeasible. But it suces to use a random sample of permutations. So we do this: 1. Compute a random permutation of the labels and compute W . Do this K times giving values W1 , . . . , WK . 2. Compute the p-value 1 K
K

I(Wj > W ).
j=1

The Score Test (Optional)

n

Recall that the score statistic is S() = logf (X1 , . . . , Xn ; ) = logf (Xi ).

i=1

Recall that E S() = 0 and V S() = In (). By the CLT, Z= S(0 ) In (0 ) 10 N (0, 1)

under H0 . So we reject if |Z| > z/2 . The advantage of the score test is that it does not require maximizing the likelihood function. Example 14 For the Binomial, S(p) = and so Z= p p0
p0 (1p0 ) n

n(pn p) , p(1 p)

In (p) =

n p(1 p)

This is the same as the Wald test in this case.

11

Lecture Notes 11 Interval Estimation (Condence Intervals)

Chapter 9 and Chapter 10.4

Introduction

Find Cn = [L(X1 , . . . , Xn ), U (X1 , . . . , Xn )] so that P L(X1 , . . . , Xn ) U (X1 , . . . , Xn ) In other words:

1 for all .

inf P L(X1 , . . . , Xn ) U (X1 , . . . , Xn )

1 .

We say that Cn has coverage 1 or that Cn is a 1 condence interval. Note that Cn is random and is xed (but unknown). More generally, a 1 condence set Cn is a (random) set Cn such that

inf P Cn (X1 , . . . , Xn )

1 .

Again, Cn is random, is not. Example 1 Let X1 , . . . , Xn N (, ). Suppose that is known. Let L = L(X1 , . . . , Xn ) = X c and U = U (X1 , . . . , Xn ) = X + c. Then P (L U ) = P (X c X + c) c n n(X ) c n = P (c < X < c) = P < < c n c n = P <Z< = (c n/) (c n/) = 1 2(c n/) = 1

if we choose c = z/2 / n. So, if we dene Cn = X n z/2 n then P ( Cn ) = 1 for all . 1

Example 2 Xi N (i , 1) for i = 1, . . . , n. Let Cn = { Rn : ||X ||2 2 }. n, Then P ( Cn ) = P (||X ||2 > 2 ) = P (2 > 2 ) = . / n, n n, Four methods: 1. Probability Inequalities 2. Inverting a test 3. Pivots 4. Large Sample Approximations Optimal condence intervals are condence intervals that are as short as possible but we will not discuss optimality.

Using Probability Inequalities

Intervals that are valid for nite samples can be obtained by probability inequalities. Example 3 Let X1 , . . . , Xn Bernoulli(p). By Hoedings inequality: P(|p p| > ) 2e2n . Let
n
2

1 log 2n 1 log 2n
n, p

2 . 2

Then P |p p| > Hence, P(p C) 1 where C = (p .

n ).

Example 4 Let X1 , . . . , Xn F . Suppose we want a condence band for F . We can use VC theory. Remember that P sup |Fn (x) F (x)| >
x

2e2n . 2 .

Let
n

1 log 2n

Then P sup |Fn (x) F (x)| >

x

1 log 2n

Hence, PF (L(t) F (t) U (t) for all t) 1 for all F , where L(t) = Fn (t) We can improve this by taking L(t) = max Fn (t)
n, n,

U (t) = Fn (t) +

n.

0 ,

U (t) = min Fn (t) +

n,

1 .

Inverting a Test

For each 0 , construct a level test of H0 : = 0 versus H1 : = 0 . Dene 0 (xn ) = 1 if we reject and 0 (xn ) = 0 if we dont reject. Let A(0 ) be the acceptance region, that is, A(0 ) = {xn : 0 (xn ) = 0}. Let C(xn ) = { : xn A()} = { : (xn ) = 0}. Theorem 5 For each , P ( C(xn )) = 1 . Proof. 1 P ( C(xn )) is the probability of rejecting when is true which is .

The converse is also true: if C(xn ) is a 1 condence interval then the test: reject H0 if 0 C(xn ) / is a level test. Example 6 Suppose we use the LRT. We reject H0 when L(0 ) L() So C= : L() L() c . c.

See Example 9.2.3 for a detailed example involving the exponential distribution. Example 7 Let X1 , . . . , Xn N (, 2 ) with 2 known. The LRT of H0 : = 0 rejects when |X 0 | z/2 . n So A() = and so C(X n ) if and only if |X | z/2 . n In other words, C = X z/2 . n If is unknown, then this becomes S C = X tn1,/2 . n (Good practice question.) xn : |X 0 | < z/2 n

Pivots
For example, if X1 , . . . , Xn N (, 1) then X n N (0, 1/n)

A function Q(X1 , . . . , Xn , ) is a pivot if the distribution of Q does not depend on .

so Q = X n is a pivot. Let a and b be such that P (a Q(X, ) b) 1 for all . We can nd such an a and b because Q is a pivot. It follows immediately that C(x) = { : a Q(x, ) b} has coverage 1 . Example 8 Let X1 , . . . , Xn N (, 2 ). ( known.) Then n(X ) N (0, 1). Z= We know that P (z/2 Z z/2 ) = 1 and so P Thus C = X z/2 . n If is unknown, then this becomes S C = X tn1,/2 n because T = n(X ) tn1 . S 5 z/2 n(X ) z/2 = 1 .

Example 9 Let X1 , . . . , Xn Uniform(0, ). Let Q = X(n) /. Then P(Q t) = so Q is a pivot. Let cn = 1/n . Then P(Q cn ) = . Also, P(Q 1) = 1. Therefore, 1 = P(c Q 1) = P c 1 1 c X(n) X(n) = P X(n) c = P so a 1 condence interval is X(n) , X(n) 1/n X(n) 1
i

P(Xi t) = tn

Large Sample Condence Intervals

We know that, under regularity conditions, n se N (0, 1)

where n is the mle and se = 1/ In (). So this is an asymptotic pivot and an approximate condence interval is n z/2 se. By the delta method, a condence interval for () is (n ) z/2 se()| (n )|. By inverting the LRT and using the 2 limiting distribution we get the LRT large sample condence set: C= : 2 log 6 L() L() 2 k, .

Then P ( C) 1 for each . Example 10 Let X1 , . . . , Xn Bernoulli(p). Using the Wald statistic pp so an approximate condence interval is p z/2 Using the LRT we get C= p : 2 log pY (1 p)nY pY (1 p)nY 2 . 1, p(1 p) . n
p(1b) b p n

N (0, 1)

These intervals are dierent but, for large n, they are nearly the same. A nite sample interval can be constructed by inverting a test.

A Pivot For the cdf

Let X1 , . . . , Xn F . We want to construct two functions L(t) L(t, X) and U (t) U (t, X) such that PF (L(t) F (t) U (t) for all t) 1 for all F . Let Kn = sup |Fn (x) F (x)|
x

where 1 Fn (x) = n

i=1

I(Xi x) =

#{Xi x} n

is the empirical distribiton function. We claim that Kn is a pivot. To see this, let Ui = F (Xi ). Then U1 , . . . , Un Uniform(0, 1). So Kn = sup |Fn (x) F (x)|
x

1 = sup n x = sup
x

1 n 1 n

i=1 n

I(Xi x) F (x) I(F (Xi ) F (x)) F (x) I(Ui F (x)) F (x) I(Ui t) t

i=1 n

= sup
x

sup
0t1

1 n

i=1 n

i=1

and the latter has a distribution depending only on U1 , . . . , Un . We could nd, by simulation, a number c such that 1 P sup 0t1 n A condence set is then C = {F : sup |Fn (x) F (x)| < c}.
x n

i=1

I(Ui t) t > c

= .

Lecture Notes 12 Nonparametric Inference

This is not in the text. Suppose we want to estimate something without assuming a parametric model. Some examples are: 1. Estimate the cdf F . 2. Estimate a density function p(x). 3. Estimate a regression function m(x) = E(Y |X = x). 4. Estimate a functional T (P ) of a distribution P for example T (P ) = E(X) = x p(x)dx.

The cdf and the Empirical Probability

We already solved this problem when we did VC theory. Given X1 , . . . , Xn F where Xi R we use, Fn (x) = We saw that P sup |F (x) F (x)| >
x

1 n

i=1

I(Xi x). 2e2n .

P
2

Hence, sup |F (x) F (x)| 0

x

and sup |F (x) F (x)| = OP

x

1 n

It can be shown that this is the minimax rate of convergence. In other words, More generally, for Xi Rd , we set 1 Pn (A) = n
n

i=1

I(Xi A).

We saw that, for any class A with VC dimension v, P sup |Pn (A) P (A)| >
AA

c1 nv ec2 n .

Density Estimation

X1 , . . . , Xn are iid with density p. For simplicity assume that Xi R. What happens if we try to do maximum likelihood? The likelihood is
n

L(p) =
i=1

p(Xi ).

We can make this as large as we want by making p highly peaked at each Xi . So supp L(p) = and the mle is the density that puts innite spikes at each Xi . We will need to put some restriction on p. For example pP= p : p 0, p = 1, |p (x)|2 dx C .

The most commonly used nonparametric density estimator is probably the histogram. Another common estimator is the kernel density estimator. A kernel K is a symmetric density function with mean 0. The estimator is 1 pn (x) = n where h > 0 is called the bandwidth. The bandwidth controls the smoothness of the estimator. Larger h makes fn smoother. As a loss function we will use L(p, p) = The risk is R = E (L(p, p)) = where b(x) = E(p(x)) p(x) 2 E(p(x) p(x))2 dx = (b2 (x) + v(x))dx (p(x) p(x))2 dx.
n

i=1

1 K h

x Xi h

is the bias and v(x) = Var(p(x)). Let Yi = Then pn (x) = n1

n i=1

1 K h

x Xi h

Yi and
n

E(p(x)) = E = E = = =

1 n

Yi
i=1

= E(Yi )

Xi x 1 K h h ux 1 K p(u)du h h K(t)p(x + ht)dt K(t) p(x) + htp (x) + where u = x + ht

h2 t2 p (x) + o(h2 ) dt 2 h2 = p(x) K(t)dt + hp (x) tK(t)dt + p (x) t2 K(t)dt + o(h2 )dt 2 2 h = (p(x) 1) + (hp (x) 0) + p (x) + o(h2 ) 2 where = t2 K(t)dt. So E(p(x)) p(x) + and b(x) Thus b2 (x)dx = h4 2 4 (p (x))2 dx. h2 p (x). 2 h2 p (x) 2

Now we compute the variance. We have v(x) = Var 1 n

n

Yi
i=1

VarYi E(Yi2 ) (E(Yi ))2 = . n n

Now E(Yi2 ) = E = = 1 2 Xi x K h2 h 1 2 ux p(u)du K h2 h

1 K 2 (t)p(x + ht)dt u = x + ht h p(x) p(x) K 2 (t)dt = h h where = K 2 (t)dt. Now (E(Yi ))2 So v(x) = and v(x)dx Finally, R Note that h bias , variance h bias , variance . If we choose h = hn to satisfy hn 0, then we see that pn (x) p(x). If we minimize over h we get h= 4nC
1/5 P

p(x) +

h2 p (x) 2

= f 2 (x) + O(h2 ) f 2 (x). 1 nh p(x) nh

E(Yi2 ) (E(Yi ))2 p(x) p(x) + f 2 (x) = +o n n nh nh . nh = Ch4 + . nh nh

h4 2 4

(p (x))2 dx +

nhn

=O

1 n

1/5

This gives R= for some constant C1 . Can we do better? The answer, based on minimax theory, is no. Theorem 1 There is a constant a such that inf sup R(f, p)
p f F b

C1 n4/5

a n4/5

So the kernel estimator achieves the minimax rate of convergence. The histogram converges at the sub-optimal rate of n2/3 . Proving these facts is beyond the scope of the course. There are many practical questions such as: how to choose h in practice, how to extend to higher dimensions etc. These are discussed in 10-702 as well as other courses.

Regression

We observe (X1 , Y1 ), . . . , (Xn , Yn ). Given a new X we want to predict Y . If our prediction is m(X) then the predictive loss os (Y m(X))2 . Later in the course we will discuss prediction in detail and we will see that the optimal predictor is the regression function m(x) = E(Y |X = x) = The kernel estimator is mn (x) = yp(y|x)dy.

n xXi i=1 Yi K h n K xXi i=1 h

The properties are similar to kernel density estimation. Again, you will study this in more detail in some other classes.

Functionals

Let X1 , . . . , Xn F . Let F be all distributions. A map T : F R is called a statistical functional. 5

Notation. Let F be a distribution function. Let f denote the probability mass function if F is discrete and the probability density function if F is continuous. The integral is interpreted as follows: g(x)dF (x) =
j

g(x)dF (x)

g(xj )p(xj ) if F is discrete if F is continuous.

g(x)p(x)dx

A statistical functional T (F ) is any function of of the cdf F . Examples include the mean = x dF (x), the variance 2 = (x )2 dF (x), the median m = F 1 (1/2), and the

largest eigenvalue of the covariance matrix . The plug-in estimator of = T (F ) is dened by n = T (Fn ). A functional of the form a(x)dF (x) is called a linear functional. The empirical cdf Fn (x) a(x)dF (x) is a linear functional a(x)dF (x) is:
n

is discrete, putting mass 1/n at each Xi . Hence, if T (F ) = then the plug-in estimator for linear functional T (F ) = T (Fn ) = 1 a(x)dFn (x) = n

a(Xi ).
i=1

Let se be an estimate of the standard error of T (Fn ). In many cases, it turns out that n = T (Fn ) N (T (F ), se2 ). In that case, an approximate 1 condence interval for T (F ) is then n z/2 se. We can use the Wald statistic W = to do a hypothesis test. n 0 se

Example 2 (The mean) Let = T (F ) =

x dF (x). The plug-in estimator is = x dFn (x) = X n . The standard error is se = Var(X n ) = / n. If denotes an es timate of , then the estimated standard error is se = / n. A Normal-based condence interval for is X n z/2 / n. Example 3 (The variance) Let 2 = Var(X) = estimator is
2

x2 dF (x)

x dF (x) . The plug-in

2 = 1 = n = 1 n

x2 dFn (x)
n

xdFn (x)
n 2

(1) (2) (3)

i=1 n

Xi2

1 n

Xi
i=1

i=1

(Xi X n )2 .

Example 4 (Quantiles) Let F be strictly increasing with density f . Let T (F ) = F 1 (p)

1 be the pth quantile. The estimate of T (F ) is Fn (p). We have to be a bit careful since Fn is 1 1 not invertible. To avoid ambiguity we dene Fn (p) = inf{x : Fn (x) p}. We call Fn (p)

the pth sample quantile.

How do we estimate the standard error? There are two approaches. One is based on something called the inuence function which is a nonparametric version of the score function. We wont cover that in this course. The second approach is to use the bootstrap which we will discuss in an upcoming lecture.

Optional: The Inuence Function

If you are curious what the inuence is, I will describe it here. This section is optional and you can skip it if you prefer. The inuence function is dened by LF (x) = lim
0

T (1 )F + x T (F ) 7

where x denote a point mass distribution at x: x (y) = 0 if y < x and x (y) = 1 if y x. The empirical inuence function is dened by L(x) = lim
0

T (1 )Fn + x T (Fn )

The inuence function is the nonparametric version of the score function. More precisely, it behaves like the score divided by the Fisher information, L = score/information = S/I. Theorem 5 If T is Hadamard dierentiable1 with respect to d(F, G) = supx |F (x) G(x)| then where 2 = n(T (Fn ) T (F )) N (0, 2 )

L2 (x)dF (x). Also, F (T (Fn ) T (F )) se N (0, 1)

where se = / n and 1 = n

L2 (Xi ).
i=1

We call the approximation (T (Fn ) T (F ))/se N (0, 1) the functional delta method or the nonparametric delta method. From the normal approximation, a large sample condence interval is: T (Fn ) z/2 se. Example 6 (The mean) Let = T (F ) = and se2 = 2 /n where 2 = n1
n i=1 (Xi

x dF (x). The plug-in estimator is =

x dFn (x) = X n . Also, T ((1 )F + x ) = (1 ) + x. Thus, L(x) = x, L(x) = xX n X n )2 . A pointwise asymptotic nonparametric 95

percent condence interval for is X n 2 se.

1

Hadamard dierentiability is a smoothness condition on T .

Example 7 (Quantiles) Let F be strictly increasing with positive density f , and let T (F ) = F 1 (p) be the pth quantile. The inuence function is
p1 , p() p , p()

L(x) =

x x > .

The asymptotic variance of T (Fn ) is 2 1 = n n

L2 (x)dF (x) =

p(1 p) . nf 2 ()

Lecture Notes 13 The Bootstrap

This is mostly not in the text.

Introduction

Can we estimate the mean of a distribution without using a parametric model? Yes. The key idea is to rst estimate the distribution function nonparametrically. Then we can get an estimate of the mean (and many other parameters) from the distribution function. How can we get the standard error of that estimator? The answer is: the bootstrap. The bootstrap is a nonparametric method for nding standard errors and condence intervals.

Notation. Let F be a distribution function. Let p denote the probability mass function if F is discrete and the probability density function if F is continuous. The integral is interpreted as follows: g(x)dF (x) =
j

g(x)dF (x)

g(xj )p(xj ) if F is discrete if F is continuous.

(1)

g(x)p(x)dx

For 0 < < 1 dene z by P(Z > z ) = where Z N (0, 1). Thus z = 1 (1 ) = 1 ().

Review of The Empirical Distribution Function

P(X x) is a distribution function on the real line. We can estimate F with the empirical distribution function Fn , the cdf that puts mass 1/n at each data point Xi . Recall that the empirical distribution function Fn is dened by 1 Fn (x) = n
n

The bootstrap uses the empirical distribution function. Let X1 , . . . , Xn F where F (x) =

i=1

I(Xi x)

(2)

where

From (1) it follows that Cantelli Theorem,

g(x)dFn (x) = n1

1 if X x i I(Xi x) = 0 if X > x. i
n i=1

(3)

g(Xi ). According to the Glivenko

as

sup |Fn (x) F (x)| 0.

x

(4)

Hence, Fn is a consistent estimator of F . In fact, the convergence is fast. According to the DvoretzkyKieferWolfowitz (DKW) inequality, for any P sup |F (x) Fn (x)| >
x

> 0,
2

2e2n .
n)

(5) 0. Hence, supx |F (x)

If

Fn (x)| = OP (n1/2 ).

= cn / n where cn , then P(supx |F (x) Fn (x)| >

Statistical Functionals
x dF (x), the variance 2 = (x )2 dF (x), m = F 1 (1/2), and the largest

Recall that a statistical functional T (F ) is any function of of the cdf F . Examples include the mean =

eigenvalue of the covariance matrix . The plug-in estimator of = T (F ) is dened by n = T (Fn ). (6)

Let se be an estimate of the standard error of T (Fn ). (We will see how to get this later.) In many cases, it turns out that T (Fn ) N (T (F ), se2 ). In that case, an approximate 1 condence interval for T (F ) is then T (Fn ) z/2 se. 2 (8) (7)

Example 1 (The mean) Let = T (F ) =

x dF (x). The plug-in estimator is = x dFn (x) = X n . The standard error is se = Var(X n ) = / n. If denotes an es timate of , then the estimated standard error is se = / n. A Normal-based condence interval for is X n z/2 / n. Example 2 A functional of the form that a(x)dF (x) is dened to be a(x)dF (x) is called a linear functional. (Recall
j

a(x)p(x)dx in the continuous case and

a(xj )p(xj ) in

the discrete case.) The empirical cdf Fn (x) is discrete, putting mass 1/n at each Xi . Hence, if T (F ) = T (F ) = a(x)dF (x) is a linear functional then the plug-in estimator for linear functional a(x)dF (x) is: T (Fn ) = 1 a(x)dFn (x) = n
n

a(Xi ).
i=1 2

(9) x dF (x) . The plug-in

Example 3 (The variance) Let 2 = Var(X) = estimator is

x2 dF (x)
2

2 = 1 = n = 1 n

x2 dFn (x)
n

xdFn (x)
n 2

(10) (11) (12)

i=1 n

Xi2

1 n

Xi
i=1

i=1

(Xi X n )2 .

Example 4 (The skewness) Let and 2 denote the mean and variance of a random variable X. The skewness which measures the lack of symmetry of a distribution is dened to be = E(X )3 = 3 (x )3 dF (x)
n i=1

To nd the plug-in estimate, rst recall that = n1 The plug-in estimate of is = (x )3 dFn (x) (x )2 dF
n (x)

(x )2 dF (x)

3/2

.
n i=1 (Xi

(13) )2 .

Xi and 2 = n1

3/2

1 n

n i=1 (Xi 3

)3

(14)

Example 5 (Correlation) Let Z = (X, Y ) and let = T (F ) = E(X X )(Y Y )/(x y ) denote the correlation between X and Y , where F (x, y) is bivariate. We can write T (F ) = a(T1 (F ), T2 (F ), T3 (F ), T4 (F ), T5 (F )) where T1 (F ) = T4 (F ) = and a(t1 , . . . , t5 ) = (t4 t2 )(t5 t2 ) 2 1 t3 t1 t2 . (16) x dF (z) T2 (F ) = y dF (z) y 2 dF (z) T3 (F ) = xy dF (z) (15)

x2 dF (z) T5 (F ) =

Replace F with Fn in T1 (F ), . . . , T5 (F ), and take = a(T1 (Fn ), T2 (Fn ), T3 (Fn ), T4 (Fn ), T5 (Fn )). We get = (17)

n i=1 (Xi n i=1 (Xi

X n )(Yi Y n ) )2
n i=1 (Yi

(18) )2

Xn

Yn

which is called the sample correlation. Example 6 (Quantiles) Let F be strictly increasing with density f . Let T (F ) = F 1 (p)
1 be the pth quantile. The estimate of T (F ) is Fn (p). We have to be a bit careful since Fn is 1 1 not invertible. To avoid ambiguity we dene Fn (p) = inf{x : Fn (x) p}. We call Fn (p)

the pth sample quantile.

The Bootstrap

Let Tn = g(X1 , . . . , Xn ) be a statistic and let VarF (Tn ) denote the variance of Tn . We have added the subscript F to emphasize that the variance is itself a function of F . In other words VarF (Tn ) = where = E(Tn ) = g(X1 , . . . , Xn )dF (x1 )dF (x2 ) dF (xn ). 4 (g(X1 , . . . , Xn ) )2 dF (x1 )dF (x2 ) dF (xn )

If we knew F we could, at least in principle, compute the variance. For example, if Tn = n1

n i=1

Xi , then 2 VarF (Tn ) = = n x2 dF (x) n xdF (x)

(19)

In other words, the variance of = T (Fn ) is itself a function of F . We can write VarF (Tn ) = U (F ) for some U . Therefore, to estimate VarF (Tn ) we can use VarF (Tn ) = U (Fn ). This is the bootstrap estimate of the standard error. To repeat: we estimate U (F ) = VarF (Tn ) with U (Fn ) = VarFn (Tn ). In other words, we use a plug-in estimator of the variance. b But how can we compute VarFn (Tn )? We approximate it with a simulation estimate b denoted by vboot . Specically, we do the following steps:

Bootstrap Variance Estimation 1. 2. 3. 4.

Draw X1 , . . . , Xn Fn . Compute Tn = g(X1 , . . . , Xn ). Repeat steps 1 and 2, B times to get Tn,1 , . . . , Tn,B . Let B B 1 1 vboot = T Tn,b B b=1 B r=1 n,r

(20)

By the law of large numbers, vboot VarFn (Tn ) as B . The estimated standard b error of Tn is seboot = vboot . The following diagram illustrates the bootstrap idea: Real world: Bootstrap world: F Fn = X1 , . . . , X n = Tn = g(X1 , . . . , Xn )

as

= X1 , . . . , Xn

= Tn = g(X1 , . . . , Xn )

Bootstrap for the Median Given data X = (X(1), ..., X(n)):

= median(X)

Tboot = vector of length B for(i in 1:N){ Xstar = sample of size n from X (with replacement)

Tboot[i] = median(Xstar) } se = sqrt(variance(Tboot))

Figure 1: Pseudo-code for bootstrapping the median.

O(1/ n)

O(1/ B)

VarF (Tn )

VarFn (Tn ) b

vboot .

(21)

How do we simulate from Fn ? Since Fn gives probability 1/n to each data point, drawing n points at random from Fn is the same as drawing a sample of size n with replacement from the original data. Therefore step 1 can be replaced by:
1. Draw X1 , . . . , Xn with replacement from X1 , . . . , Xn .

Example 7 Figure 1 shows pseudo-code for using the bootstrap to estimate the standard error of the median.

The Parametric Bootstrap

So far, we have estimated F nonparametrically. There is also a parametric bootstrap. If F depends on a parameter and is an estimate of , then we simply sample from F b 6

instead of Fn . This is just as accurate, but much simpler than, the delta method. Here is more detail. Suppose that X1 , . . . , Xn p(x; ). Let be the mle. Let = g(). Then = g(). To get the standard error of we need to compute the Fisher information and then do the delta method. The bootstrap allows us to avoid both steps. We just do the following: 1. Compute the estimate from the data X1 , . . . , Xn .
2. Draw a sample X1 , . . . , Xn f (x; ). Compute 1 and 1 = g(1 ) from the new data. Repeat B times to get 1 , . . . , B .

3. Compute the standard deviation se = 1 B

B

b=1

(j )2

where

1 B

j .
b=1

(22)

No need to get the Fisher information or do the delta method.

Bootstrap Condence Intervals

There are several ways to construct bootstrap condence intervals. They vary in ease of calculation and accuracy. Normal Interval. The simplest is the Normal interval n z/2 seboot where seboot is the bootstrap estimate of the standard error. (23)

Pivotal Intervals. Let = T (F ) and n = T (Fn ). We can also construct an approxi mate condence interval for using the (approximate) pivot n( ) as follows: C= H 1 1 n n 7
2

H 1 , n 2 n

(24)

where 1 H(r) = B where H(r) = P

I
j=1

n(j ) r n(j ) r .
P

(25)

n(n ) r ,

H(r) = Pn I

(26)

Theorem 8 Under appropriate conditions on T , supu |H(u) H(u)| 0 as n and supu |H(u) H(u)| 0 as B .
P

Now we can show that the condence interval has coverage that is approximately equal to 1 . Applying Theorem 8 we have H 1 1 H 1 2 P( C) = P n n 2 n n = P H 1 n(n ) H 1 1 2 2 1 1 1 = H H H H 2 2 H H 1 1 H H 1 2 2 1 1 H H 1 H H 2 2 = 1 = 1 . 2 2

Remarks About The Bootstrap

1. The bootstrap is nonparametric but it does require some assumptions. You cant assume it is always valid. (See the appendix.) 2. The bootstrap is an asymptotic method. Thus the coverage of the condence interval is 1 + rn where the remainder rn 0 as n . 3. There is a related method called the jackknife where the standard error is estimated by leaving out one observation at a time. However, the bootstrap is valid under weaker conditions than the jackknife. See Shao and Tu (1995).

4. Another way to construct a bootstrap condence interval is to set C = [a, b] where a is

the /2 quantile of 1 , . . . , B and b is the 1/2 quantile. This is called the percentile

interval. This interval seems very intuitive but does not have the theoretical support of the interval in (24). However, in practice, the percentile interval and the interval in (24) are often quite similar. 5. There are many cases where the bootstrap is not formally justied. This is especially true with discrete structures like trees and graphs. Nonethless, the bootstrap can be used in an informal way to get some intuition of the variability of the procedure. But keep in mind that the formal guarantees may not apply in these cases. For example, see Holmes (2003) for a discussion of the bootstrap applied to phylogenetic tres. 6. There is an improvement on the bootstrap called subsampling. In this case, we draw samples of size m < n without replacement. Subsampling produces valid condence intervals under weaker conditions than the bootstrap. See Politis, Romano and Wolf (1999). 7. There are many modications of the bootstrap that lead to more accurate condence intervals; see Efron (1996).

Examples

Example 9 (The Median) The top left plot of Figure 2 shows the density for a 2 distribution with 4 degrees of freedom. The top right plot shows a histogam of n = 50 draws from this distribution. Let = T (P ) be the median. The true value is = 3.36. The samlpe
median turns out to be n = 3.22. We computed B = 1000 bootstrap values 1 , . . . , B shown

in the histogram (bottom left plot). The estimated standard error is 0.35. This is smaller than the true standard error which is 0.44. Next we conducted a small simulation. We drew a sample of size n and computed the 95 percent bootstrap condence interval. We repeated this process N = 100 times. The bottom right plot shows the 100 intervals. The vertical line is the true value of . The percentage

0.00 0

0.05

0.10

0.15

10

12

10

12

14

2.0

2.5

3.0

3.5

4.0

4.5

Figure 2: Top left: density of a 2 with 4 degrees of freedom. The vertical line shows the median. Top right: n = 50 draw from the distribution. Bottom left: B = 1000 boostrap
values 1 , . . . , B . Bottom right: Bootstrap condence intervals from 100 experiments.

of intervals that cover is 0.83 which shows that the bootstrap interval undercovers in this case. Example 10 (Nonparametric Regression) The bootstrap is often used informally to get a sense of the variability of a procedure. Consider the data (X1 , Y1 ), . . . , (Xn , Yn ) in the top left plot of Figure 3. To estimate the regression function m(x) = E(Y |X = x) we use a kernel regression estimator given by m(x) = Xi )/h)/
j n i=1

K((x Xj )/h) and K(x) = e

x2 /2

Yi wi (x) where wi (x) = K((x

is a Gaussian kernel. The estimated curve

is shown in the top right plot. We now create B = 1, 000 boostrap replications resulting in curves m , . . . , m in the bottom left plot. At each x, we nd the .025 and .975 quantile of 1 B 10

the bootstrap replications. This reults in the upper and lower band in the bottom right plot. The bootstrap reveals greater variability in the estimated curve around x = 0.5. The reason why we call this an informal use of the bootstrap is that the bands shown in the lower right plot are not rigorous condence bands. There are several reasons for this. First, we used a percentile interval (described in the earlier list of remarks) rather than the interval dened by (24). Second, we have not adjusted for the fact that we are making simultaneous bands over all x. Finally, the theory of the bootstrap does not directly apply to nonparametric smoothing. Roughly speaking, we are really creating approximate condence intervals for E(m(x)) rather than for m(x). Despite these shortcomings, the bootstrap is still regarded as a useful tool here but we must keep in mind that it is being used in an informal way. Some authors refer to the bands as variability bands rather than condence bands for this reason.

Example 11 (Estimating Eigenvalues) Let X1 , . . . , Xn be random vectors where Xi Rp and let be the covariance matrix of Xi . A common dimension reduction technique is principal components which involves nding the spectral decomposition = EE T where the columns of E are the eigenvectors of and is a diagonal matrix whose diagonal elements are the ordered eigenvalues 1 p . The data dimension can be reduced to q < p by projecting each data point onto the rst q eigenvalues. We choose q such that
p j=q+1

2 is j

small. Of course, we need to estimate the eigenvectors and eigenvalues. For now, let us focus on estimating the largest eigenvalue and denote this by . An estimate of is the largest principal component of the sample covariance matrix 1 S= n
n

i=1

(Xi X)(Xi X)T .

(27)

It is not at all obvious how can can estimate the standard error of or how to nd a condence interval for . In this example, the bootstrap works as follows. Draw a sample of size n with
replacement from X1 , . . . , Xn . The new sample is denoted by X1 , . . . , Xn . Compute the

sample covariance matrix S of the new data and let denote the largest eigenvector of S . Repeat this process B times where B is typically about 10,000. This yields bootstrap values 11

0.5

0.0

0.5

1.0

0.5

0.0

0.5

1.0

0.5

0.0

0.5

1.0

0.5

0.0

0.5

1.0

Figure 3: Top left: the data (X1 , Y1 ), . . . , (Xn , Yn ). Top right: kernel regression estimator. Bottom left: 1,000 bootstrap replications. Bottom right: 95 percent variability bands. .

12

 1 , . . . , B . The standard deviation of 1 , . . . , B is an estimate of the standard error of the

original estimator . Figure 4 shows a PCA analysis of US arrest data. The last plot shows bootstrap replications of the rst principal component.

Example 12 (Median Regression) Consider the linear regression model Yi = XiT + i . Instead of using least squares to estimate , dene to minimize median|Yi XiT |. (29) (28)

The resulting estimator is more resistant to outliers than the least squares estimator. But how can we nd the standard error of ? Using the bootstrap approach, we resample the pairs
of data to get the bootstrap sample (X1 , Y1 ), . . . , (Xn Yn ) and then we get the corresponding

bootstrap estimate . We can repeat this many times and use the standard deviation of the bootstrap estimates to estimate the standard error of . Figure 5 shows bootstrap replications of ts from regression and robust regression (minimizing L1 error instead of squared error) in a dataset with an outlier. Warning! The bootstrap is not magic. Its validity requires some conditions to hold. When the conditions dont hold, the bootstrap, like any method, can give misleading answers.

13

out

5 0.3
Mississippi North Carolina South Carolina

2.0

0.2

West Virginia Vermont Georgia AlaskaAlabamaArkansas Kentucky Murder Louisiana Tennessee South Dakota Montana North Dakota Maryland Assault Maine Wyoming Virginia Idaho New Mexico Florida New Hampshire Michigan Indiana Nebraska Iowa Missouri Kansas Oklahoma Rape Texas Delaware Oregon Pennsylvania Wisconsin Minnesota Illinois Ohio Arizona Nevada New York Colorado Washington Connecticut New Jersey Island Utah Massachusetts Rhode California Hawaii UrbanPop

1.5

Variances

PC2

0.0

0.1

1.0

0.2

0.1

0.5

0.2 0.0

0.1

0.0 PC1

0.1

0.2

0.3

Histogram of v

150

Frequency

100

first

50

0.50

0.55

0.60

0.65 v

0.70

0.75

0.6

0.4

0.2

0.0

0.2

0.4

0.6

Figure 4: US Arrest Data

14

y 1 1.0 0 1 2

0.5

0.0 x

0.5

1.0

y 1 1.0 0 1 2

0.5

0.0 x

0.5

1.0

Figure 5: Robust Regression

15

Lecture Notes 14 Bayesian Inference

Relevant material is scattered throughout the book: see sections 7.2.3, 8.2.2, 9.2.4 and 9.3.3. We will also cover some material that is not in the book.

Introduction

So far we have been using frequentist (or classical) methods. In the frequentist approach, probability is interpreted as long run frequencies. The goal of frequentist inference is to create procedures with long run guarantees. Indeed, a better name for frequentist inference might be procedural inference. Moreover, the guarantees should be uniform over if possible. For example, a condence interval traps the true value of with probability 1 , no matter what the true value of is. In frequentist inference, procedures are random while parameters are xed, unknown quantities. In the Bayesian approach, probability is regarded as a measure of subjective degree of belief. In this framework, everything, including parameters, is regarded as random. There are no long run frequency guarantees. Bayesian inference is quite controversial. Note that when we used Bayes estimators in minimax theory, we were not doing Bayesian inference. We were simply using Bayesian estimators as a method to derive minimax estimators.

The Mechanics of Bayes

Let X1 , . . . , Xn p(x|). In Bayes we also include a prior (). It follows from Bayes theorem that the posterior distribution of given the data is (|X1 , . . . , Xn ) = where m(X1 , . . . , Xn ) = p(X1 , . . . , Xn |)()d. 1 p(X1 , . . . , Xn |)() m(X1 , . . . , Xn )

Hence, (|X1 , . . . , Xn ) L()() where L() = p(X1 , . . . , Xn |) is the likelihood function. The interpretation is that (|X1 , . . . , Xn ) represents your subjective beliefs about after observing X1 , . . . , Xn . A commonly used point estimator is the posterior mean = E(|X1 , . . . , Xn ) = (|X1 , . . . , Xn )d = L()() . L()()

For interval estimation we use C = (a, b) where a and b are chosen so that
b a

(|X1 , . . . , Xn ) = 1 .

This interpretation is that P ( C|X1 , . . . , Xn ) = 1 . This does not mean that C traps with probability 1 . We will discuss the distinction in detail later. Example 1 Let X1 , . . . , Xn Bernoulli(p). Let the prior be p Beta(, ). Hence (p) = and () =
0

( + ) ()()

t1 et dt.

Set Y =

i Xi . Then

(p|X) pY 1 pnY p1 1 p1 pY +1 1 pnY +1 .

likelihood prior

Therefore, p|X Beta(Y + , n Y + ). (See page 325 for more details.) The Bayes estimator is p= where Y + Y + = = (1 )pmle + p (Y + ) + (n Y + ) ++n p= = + . ++n

, + This is an example of a conjugate prior.

Example 2 Let X1 , . . . , Xn N (, 2 ) with 2 known. Let N (m, 2 ). Then

2 n E(|X) = X+ 2 + 2 2+ n
2

2 n

and Var(|X) = 2 2 /n 2 . 2 + n

Where Does the Prior Come From?

This is the million dollar question. In principle, the Bayesian is supposed to choose a prior that represents their prior information. This will be challenging in high dimensional cases to say the least. Also, critics will say that someones prior opinions should not be included in a data analysis because this is not scientic. There has been some eort to dene noninformative priors but this has not worked out so well. An example is the Jereys prior which is dened to be () I().

You can use a at prior but be aware that this prior doesnt retain its atness under transformations. In high dimensional cases, the prior ends up being highly inuential. The result is that Bayesian methds tend to have poor frequentist behavior. Well return to this point soon. It is common to use at priors even if they dont integrate to 1. This is posible since the posterior might still integrate to 1 even if the prior doesnt.

Large Sample Theory

There is a Bayesian central limit theorem. In nice models, with large n, (|X1 , . . . , Xn ) N , 1 In () (1)

where n is the mle and I is the Fisher information. In these cases, the 1 Bayesian intervals will be approximately the same as the frequentist condence intervals. That is, an approximate 1 posterior interval is C = z/2 In ()

which is the Wald condence interval. However, this is only true if n is large and the dimension of the model is xed. Here is a rough derivation of (1). Note that
n

log (|X1 , . . . , Xn ) =
i=1

log p(Xi |) + log () log C

where C is the normalizing constant. Now the sum has n terms which grows with sample size. The last two terms are O(1). So the sum dominates, that is,
n

log (|X1 , . . . , Xn ) Next, we note that

i=1

log p(Xi |) = (). ( )2 () . 2

() () + ( ) () + Now () = 0 so () () + Thus, approximately,

( )2 () . 2 ( )2 2 2

(|X1 , . . . , Xn ) exp where 2 = Let

i

1 ()

= log p(Xi |0 ) where 0 is the true value. Since 0 , () (0 ) =

i i

=n

1 n

i i

nI1 (0 ) nI1 () = In ()

and therefore, 2 1/In (). 4

Bayes Versus Frequentist

In general, Bayesian and frequentist inferences can be quite dierent. If C is a 1 Bayesian interval then P ( C|X) = 1 . This does not imply that frequentist coverage = inf P ( C) = 1 ..

Typically, a 1 Bayesian interval has coverage lower than 1 . Suppose you wake up everyday and produce a Bayesian 95 percent interval for some parameter. (A dierent parameter everyday.) The fraction of times your interval contains the true parameter will not be 95 percent. Here are some examples to make this clear. Example 3 Normal means. Let Xi N (i , 1), i = 1, . . . , n. Suppose we use the at prior (1 , . . . , n ) 1. Then, with = (1 , . . . , n ), the posterior for is multivariate Normal with mean X = (X1 , . . . , Xn ) and covariance matrix equal to the identity matrix. Let =
n i=1

2 . Let Cn = [cn , ) where cn is chosen so that P( Cn |X1 , . . . , Xn ) = .95. i P ( Cn ) 0, as n .

How often, in the frequentist sense, does Cn trap ? Stein (1959) showed that

Thus, P ( Cn ) 0 even though P( Cn |X1 , . . . , Xn ) = .95. Example 4 Sampling to a Foregone Conclusion. Let X1 , X2 , . . . N (, 1). Suppose we continue sampling until T > k where T = n|X n | and k is a xed number, say, k = 20.

The sample size N is now a random variable. It can be shown that P(N < ) = 1. It

can also be shown that the posterior (|X1 , . . . , XN ) is the same as if N had been xed in advance. That is, the randomness in N does not aect the posterior. Now if the prior () is smooth then the posterior is approximately |X1 , . . . , XN N (X n , 1/n). Hence, if Cn = X n 1.96/ n then P( Cn |X1 , . . . , XN ) .95. Notice that 0 is never in Cn since, when we stop sampling, T > 20, and therefore 1.96 20 1.96 X n > > 0. n n n 5 (2)

Hence, when = 0, P ( Cn ) = 0. Thus, the coverage is Coverage = inf P ( Cn ) = 0.

This is called sampling to a foregone conclusion and is a real issue in sequential clinical trials. Example 5 Here is an example we discussed earlier. Let C = {c1 , . . . , cN } be a nite set of constants. For simplicity, asssume that cj {0, 1} (although this is not important). Let = N 1
N j=1 cj .

Suppose we want to estimate . We proceed as follows. Let S1 , . . . , Sn

Bernoulli() where is known. If Si = 1 you get to see ci . Otherwise, you do not. (This is an example of survey sampling.) The likelihood function is Si (1 )1Si .

The unknown parameter does not appear in the likelihood. In fact, there are no unknown parameters in the likelihood! The likelihood function contains no information at all. The posterior is the same as the prior. But we can estimate . Let 1 = N
N

cj Sj .
j=1

Then E() = . Hoedings inequality implies that P(| | > ) 2e2n log(2/)/(2n 2 ).
2 2

Hence, is close to with high probability. In particular, a 1 condence interval is

Bayesian Computing

If = (1 , . . . , p ) is a vector then the posterior (|X1 , . . . , Xn ) is a multivariate distribution. If you are interested in one parameter, 1 for example, then you need to nd the marginal posterior: (1 |X1 , . . . , Xn ) = (1 , . . . , p |X1 , . . . , Xn )d2 dp . 6

Usually, this integral is intractable. In practice, we resort to Monte Carlo methods. These are discussed in 36/10-702.

Bayesian Hypothesis Testing

Bayesian hypothesis testing can be done as follows. Suppose that R and we want to test H0 : = 0 and H1 : = 0 .

If we really believe that there is a positive prior probability that H0 is true then we can use a prior of the form a0 + (1 a)g() where 0 < a < 1 is the prior probability that H0 is true and g is a smooth prior density over which represents our prior beliefs about when H0 is false. It follows from Bayes theorem that P ( = 0 |X1 , . . . , Xn ) = where m = choice of g. Sometimes, people like to summarize the test by using the Bayes factor B which is dened to be the posterior odds divided by the prior odds: B= where posterior odds = = = P ( = 0 |X1 , . . . , Xn ) 1 P ( = 0 |X1 , . . . , Xn )
aL(0 ) aL(0 )+(1a)m (1a)m aL(0 )+(1a)m

aL(0 ) ap(X1 , . . . , Xn |0 ) = aL(0 ) + (1 a)m ap(X1 , . . . , Xn |0 ) + (1 a) p(X1 , . . . , Xn |)g()d

L()g()d. It can be shown that P ( = 0 |X1 , . . . , Xn ) is very sensitive to the

posterior odds prior odds

aL(0 ) (1 a)m 7

and prior odds = and hence a P ( = 0 ) = P ( = 0 ) 1a L(0 ) . m

B=

Example 6 Let X1 , ldots, Xn N (, 1). Lets test H0 : = 0 versus H1 : = 0. Suppose we take g() to be N (0, 1). Thus, 1 2 g() = e /2 . 2 Let us further take a = 1/2. Then, after some tedious integration to compute m(X1 , . . . , Xn ) we get P ( = 0 |X1 , . . . , Xn ) = = L(0) L(0) + m enX enX
2 2

/2

/2

2 n enX /(2(n+1)) n+1

On the other hand, the p-value for the usual test is p = 2( n|X|). Figure 1 shows the posterior of H0 and the p-value as a function of X when n = 100. Note that they are very dierent. Unlike in estimation, in testing there is little agreement between Bayes and frequentist methods.

Conclusion

Bayesian and frequentist inference are answering two dierent questions. Frequentist inference answers the question: How do I construct a procedure that has frequency guarantees? Bayesian inference answers the question: How do I update my subjective beliefs after I observe some data? In parametric models, if n is large and the dimension of the model is xed, Bayes and frequentist procedures will be similar. Otherwise, they can be quite dierent. 8

0.0 0.0

0.2

0.4

0.6

0.8

1.0

0.1

0.2

0.3

0.4

0.5

Figure 1: Solid line: P ( = 0|X1 , . . . , Xn ) versus X. Dashed line: p-value versus X.

Lecture Notes 15 Prediction

This is mostly not in the text. Some relevant material is in Chapters 11 and 12.

Introduction

We observe training data (X1 , Y1 ), . . . , (Xn , Yn ). Given a new pair (X, Y ) we want to predict Y from X. There are two common versions: 1. Y {0, 1}. This is called classication, or discrimination, or pattern recognition. (More generally, Y can be discrete.) 2. Y R. This is called regression. For classication we will use the following loss function. Let h(x) be or prediction of Y when X = x. Thus h(x) {0, 1}. The function h is called a classier. The classication loss is I(Y = h(X)) and the the classication risk is R(h) = P(Y = h(X)) = E(I(Y = h(X))). For regression, suppose our prediction of Y when X = x is g(x). We will use the squared error prediction loss (Y g(X))2 and the risk is R(g) = E(Y g(X))2 .

Regression

Theorem 1 R(g) is minimized by m(x) = E(Y |X = x) = y p(y|x)dy.

Proof. Let g(x) be any function of x. Then R(g) = E(Y g(X))2 = E(Y m(X) + m(X) g(X))2 = E(Y m(X))2 + E(m(X) g(X))2 + 2E((Y m(X))(m(X) g(X))) E(Y m(X))2 + 2E((Y m(X))(m(X) g(X))) = E(Y m(X))2 + 2EE (Y m(X))(m(X) g(X)) X = E(Y m(X))2 + 2E (E(Y |X) m(X))(m(X) g(X)) = E(Y m(X))2 + 2E (m(X) m(X))(m(X) g(X)) = E(Y m(X))2 = R(m).

Hence, to do make predictions, we need to estimate m(x) = E(Y |X = x). The simplest apprach is to use a parametric model. In particular, the linear regression model assumes that m(x) is a linear function of x. (More precisely, we seek the best linear predictor.) Suppose that Xi Rp so that Xi = (Xi1 , . . . , Xip )T . Then the linear regression model is
p

m(x) = 0 +
j=1

j xj .

We can write Yi = 0 +

j Xij + i ,
j=1

i = 1, . . . , n

where

1, . . . , n

are iid with mean 0.

If we use the convention that Xi1 = 1 then we can write the model more simply as
p

Yi =
j=1

j Xij +

= T Xi + i ,

i = 1, . . . , n

(1)

where = (1 , . . . , p )T and Xi = (Xi1 , . . . , Xip )T . Let us dene Y = (Y1 , . . . , Yn )T , = ( 1, . . . ,

T n)

and let X be the n p matrix with

X(i, j) = Xij . Then we can write (1) as Y = X + . The least squares estimator is the that minimizes
n

i=1

(Yi XiT )2 = ||Y X||2 .

Theorem 2 Suppose that X T X is invertible. Then the least squares estimator is = (X T X)1 X T Y. The tted values or predicted values are Y = (Y1 , . . . , Yn )T where Yi = XiT . Hence, Y = X = HY where H = X(X T X)1 X T is called the hat matrix. Theorem 3 The matrix H is symmetric and idempotent: H 2 = H. Moreover, HY is the projection of Y onto the column space of X. This is discussed in more detail in 36-707 and 10/36-702. Theorem 4 Suppose that the linear model is correct.1 Also, suppose that Var( i ) = 2 . Then,
1

n( )

N (0, 2 X T X).

This model is virtually never correct, so view this result with caution.

Under the (questionable) assumption that the linear model is correct, we can also say the following. A consistent estimator of 2 is 2 = and RSS np N (0, 1)

n(j j ) sj

where the standard error sj is the j th diagonal element of 2 X T X. To test H0 : j = 0 versus H1 : j = 0 we reject if |j |/s)j > z/2 . An approximate 1 condence interval for j is j z/2 sj . Theorem 5 Suppose that the linear model is correct and that the least squares estimator is the maximum likelihood estimator.
1, . . . , n

N (0, 2 ). Then

Linear Prediction When the Model is Wrong

When the model is wrong (and it always is) the least squares estimator still has the following good property. Let minimize R() = E(Y X T )2 . We call
(x)

= xT the best linear predictor.

Theorem 6 Under weak conditions, R() R( ) 0. Hence, the least squares estimator approximates the best linear predictor. Lets prove this in the case with one covariate. Then R() = E(Y X)2 = E(Y )2 2E(XY ) + 2 E(X 2 ). 4
P

Minimizing with respect to we get = E(XY ) E(X 2 )

assuming that 0 < E(X 2 ) < and E(XY ) < . Now =

i

Xi Yi = 2 i Xi

1 n 1 n

Xi Yi . 2 i Xi

By the law of large numbers and the continuous mapping theorem: . Since R() is a continuous function of , it follows from the continuous mapping theorem that R() R( ). In fact, = and R() = R( ) + ( )R ( ) + o( ) and so R() R( ) + OP (1/ n). The message here is that least squares estimates the best linear predictor: we dont have to assume that the truth is linear.
1 n 1 n i P P

Xi Yi E(XY ) + OP (1/ n) = = + OP (1/ n) 2 2 ) + O (1/ n) E(X P i Xi

Nonparametric Regression

Suppose we want to estimate m(x) where we only assume that m is a smooth function. The kernel regression estimator is m(x) =
i

Yi wi (x)

1.0

q q q q q q q

q q q q q q qq q q q q

qq q q q

0.5

q q q q q q

q q q q

q q q q q q q

0.0

q q q q q q q q q q q q q q q q q q q q q q q q q q q q

q q q q q q q q q q q q

0.5

q qq q q q q q q

1.0

q q q q

1.0

0.5

0.0 x

0.5

1.0

Figure 1: A kernel regression estimator. where K wi (x) =

j

||xXi || h ||xXj || h

Here K is a kernel and h is a bandwidth. The properties are simialr to that of kernel density estimation. The properties of m are discussed in more detail in the 36-707 and in 10-702. An example is shown in Figure 1.

Classication

The best classier is the so-called Bayes classier dened by: hB (x) = I(m(x) 1/2) where m(x) = E(Y |X = x). Theorem 7 For any h, R(h) R(hB ). Proof. For any h, R(h) R(hB ) = P(Y = h(X)) P(Y = hB (X)) = = We will show that P(Y = h(x)|X = x) P(Y = hB (x)|X = x) 0 for all x. Now P(Y = h(x)|X = x) P(Y = hB (x)|X = x) = h(x)P(Y = 1|X = x) + (1 h(x))P(Y = 0|X = x) hB (x)P(Y = 1|X = x) + (1 hB (x))P(Y = 0|X = x) P(Y = h(x)|X = x)p(x)dx P(Y = hB (x)|X = x)p(x)dx

(P(Y = h(x)|X = x) P(Y = hB (x)|X = x)) p(x)dx.

= (h(x)(1 m(x)) + (1 h(x))m(x)) (hB (x)(1 m(x)) + (1 hB (x))m(x)) = 2(m(x) 1/2)(hB (x) h(x)) 0 since hB (x) = 1 if and only if m(x) 1/2.

The most direct approach to classication is empirical risk minimization (ERM). We start with a set of classiers H. Each h H is a function h : x {0, 1}. The training error or empirical risk is 1 R(h) = n We choose h to minimize R: h = argminhH R(h). For example, a linear classier has the form h (x) = I( T x 0). The set of linear
n

I(Yi = h(Xi )).

i=1

classiers is H = {h : Rp }.

Theorem 8 Suppose that H has VC dimension d < . Let h be the empirical risk minimizer and let h = argminhH R(h) be the best classier in H. Then, for any > 0,
2

P(R(h) > R(h ) + 2 ) c2 nd enc2 for some constnts c1 and c2 . Proof. Recall that

P(sup |R(h) R(h)| > ) c2 nd enc2 .

hH

But when suphH |R(h) R(h)| we have R(h) R(h) + R(h ) + R(h ) + 2 .

Empirical risk minimization is dicult because R(h) is not a smooth function. Thus, we often use other approaches. One idea is to use a surrogate loss function. To expain this idea, it will be convenient to relabel the Yi s as being +1 or -1. Many classiers then take the form h(x) = sign(f (x)) 8

for some f (x). For example, linear classiers have f (x) = xT . Th classication loss is then L(Y, f, X) = I(Y f (X) < 0) since an error occurs if and only if Y and f (X) have dierent signs. An example of surrogate loss is the hinge function (1 Y f (X))+ . Instead of minimizing classication loss, we minimize (1 Yi f (Xi ))+ .

The resulting classier is called a support vector machine. Another approach to classication is plug-in clasication. We replace the Bayes rule hB = I(m(x) 1/2) with h(x) = I(m(x) 1/2) where m is an estimate of the regression function. The estimate m can be parametric or nonparametric. A common parametric estimator is logistic regression. Here, we assume that ex m(x; ) = . 1 + exT Since Yi is Bernoulli, the likeihood is
n
T

L() =
i=1

m(Xi ; )Yi (1 m(Xi ; ))1Yi .

We compute the mle numerically. See Section 12.3 of the text. What is the relationship between classication and regression? Generally speaking, classication is easier. This follows from the next result. Theorem 9 Let m(x) = E(Y |X = x) and let hm (x) = I(m(x) 1/2) be the Bayes rule. Let g be any function and let hg (x) = I(g(x) 1/2). Then R(hg ) R(hm ) 2 9 |g(x) m(x)|2 dP (x).

Proof. We showed earlier that R(hg ) R(hm ) = and that P(Y = hg (x)|X = x) P(Y = hm (x)|X = x) = 2(m(x) 1/2)(hm (x) hg (x)). Now 2(m(x) 1/2)(hm (x) hg (x)) = 2|m(x) 1/2| I(hm (x) = hg (x)) 2|m(x) g(x)| since hm (x) = hg (x) implies that |m(x) 1/2| |m(x) g(x)|. Hence, R(hg ) R(hm ) = 2 2 2 |m(x) 1/2|I(hm (x) = hg (x))dP (x) |m(x) g(x)|dP (x) |g(x) m(x)|2 dP (x) [P(Y = hg (x)|X = x) P(Y = hm (x)|X = x)] dP (x)

where the last setp follows from the Cauchy-Schwartz inequality.

Hence, if we have an estimator m such that

|m(x) m(x)|2 dP (x) is small, then the

excess classication risk is also small. But the reverse is not true.

10

Lecture Notes 16 Model Selection

Not in the text.

Introduction

Sometimes we have a set of possible models and we want to choose the best model. Model selection methods help us choose a good model. Here are some examples. Example 1 Suppose you use a polynomial to model the regression function: m(x) = E(Y |X = x) = 0 + 1 x + + p xp . You will need to choose the order of polynomial p. We can think of this as a sequence of models M1 , . . . , Mp , . . . indexed by p. Example 2 Suppose you have data Y1 , . . . , Yn on age at death for n people. You want to model the distribution of Y . Some popular models are: 1. M1 : the exponential distribution: f (y; ) = ey . 2. M2 : the gamma distribution: f (y; a, b) = (ba /(a))y a1 eby . 3. M3 : the log-normal distribution: we take log Y N (, 2 ). Example 3 Suppose you have time series data Y1 , Y2 , . . .. A common model is the AR (autoregressive model): Yt = a1 Yt1 + a2 Yt2 + + ak Ytk + where
t t

N (0, 2 ). The number k is called the order of the model. We need to choose k.

Example 4 In a linear regression model, you need to choose which variables to include in the regression. This is called variable selection. This problem is discussed at length in 36-707 and 10-702.

The most common model selections methods are: 1. AIC (and related methods like Cp ). 2. Cross-validation. 3. BIC (and related methods like MDL, Bayesian model selection). We need to distinguish between 2 goals: 1. Find the model that gives the best prediction (without assuming that any of the models are correct). 2. Assume one of the models is the true model and nd the true model. Generally speaking, AIC and cross-validation are used for goal 1 while BIC is used for goal 2.

AIC

Suppose we have models M1 , . . . , Mk where each model is a set of densities: Mj = p(y; j ) : j j .

We have data Y1 , . . . , Yn drawn from some density f . We do not assume that f is in any of the models.

Let j be the mle from model j. An estimate of f , based on model j is fj (y) = p(y; j ). The quality of fj (y) as an estimate of f can be measured by the Kullback-Leibler distance: K(f, fj ) = = p(y) log p(y) fj (y) dy p(y) log fj (y)dy.

p(y) log p(y)dy

The rst term does not depend on j. So minimizing K(f, fj ) over j is the same as maximizing Kj = p(y) log p(y; j )dy.

We need to estimate Kj . Intuitively, you might think that a good estimate of Kj is 1 Kj = n where
j (j ) n

log p(Yi ; j ) =
i=1

j (j )

is the log-likelihood funcion for model j. However, this estimate is very biased

because the data are being used twice: rst to get the mle and second to estimate the integral. Akaike showed that the bias is approximately dj /n where dj = dimension(j ). Therefore we use Kj = Now, dene AIC(j) = 2nKj =
j (j ) j (j )

dj dj = Kj . n n 2dj .

Notice that maximizing Kj is the same as maximizing AIC(j) over j. Why do we multiply by 2n? Just for historical reasons. We can multiply by any constant; it wont change which model we pick. In fact, dierent texts use dierent versions of AIC. AIC stands for Akaike Informaion Criterion. Akaike was a famous Japanese statistician who died recently (August 2009).

Theoretical Derivation of AIC

Let us now look closer to see where the formulas come from. Recall that Kj = p(y) log p(y; j )dy. 3

For simplicity, let us focus on one model and drop the subscript j. We want to estimate K= Our goal is to show that K where 1 K= n and d is the dimension of .
n

p(y) log p(y; )dy.

d K n log p(Yi ; )

i=1

Some Notation and Background. Let 0 minimize K(f, p(; )). So p(y; 0 ) is the closest density in the model to the true density. Let (y, ) = log p(y; ) and s(y, ) = log p(y; )

be the score and let H(y, ) be the matrix of second derivatives. Let Zn = n( 0 ) and recall that Zn where J = E[H(Y, 0 )] and V = Var(s(Y, 0 )). In class we proved that V = J 1 . But that proof assumed the model was correct. We are not assuming that. Let Sn = By the CLT, nSn Hence, in distribution JZn nSn . (1) N (0, V ) 1 n
n

N (0, J 1 V J 1 )

s(Yi , 0 ).
i=1

Here we used the fact that Var(AX) = A(VarX)AT . Thus Var(JZn ) = J(J 1 V J 1 )J T = V. We will need one other fact. Let Let Q= (Q is a called a quadratic form.) Then E(Q) = trace(A) + T A.
T

be a random vector with mean and covariance .

A.

The details. By using a Taylor series K 1 p(y) log p(y; 0 ) + ( 0 )T s(y, 0 ) + ( 0 )T H(y, 0 )( 0 ) dy 2 1 T Z JZn 2n n

= K0 where

K0 =

p(y) log p(y; 0 )dy,

The second term dropped out because, like the score function, it has mean 0. Again we do a Taylor series to get 1 K n
n

i=1

1 (Yi , 0 ) + ( 0 )T s(Yi , 0 ) + ( 0 )T H(Yi , 0 )( 0 ) 2 1 T Z Jn Zn 2n n

= K0 + An + ( 0 )T Sn

T Zn Sn 1 T K0 + An + Zn JZn 2n n

where 1 Jn = n

i=1

H(Yi , 0 ) J, 5

and 1 An = n Hence, K K An + where we used (1). We conclude that E(K K) E(An ) + E Hence,
T Zn JZn n

i=1

( (Yi , 0 ) K0 ).

T nZn Sn Z T JZn An + n n n

=0+

trace(J J 1 V J 1 ) trace(J 1 V ) = . n n

KK use

trace(J 1 V ) . n

If the model is correct, then J 1 = V so that trace(J 1 V ) = trace(I) = p. Thus we would p . n

KK

You can see that there are a lot of approximations and assumptions being used. So AIC is a very crude tool. Cross-validation is much more reliable.

Cross-Validation

There are various avors of cross-validation. In general, the data are split into a training set and a test set. The models are t on the training set and are used to predict the test set. Usually, many such splits are used and the result are averaged over splits. However, to keep things simple, we will use a single split. Suppose again that we have models M1 , . . . , Mk . Assume there are 2n data points. Split the data randomly into two halves that we will denote D = (Y1 , . . . , Yn ) and T =
(Y1 , . . . , Yn ). Use D to nd the mles j . Then dene

Kj =

1 n

log p(Yi ; j ).
i=1

Note that E(Kj ) = Kj ; there is no bias because j is independent of Yj . We will assume that | log p(y; )| B < . By Hoedings inequality, P(max |Kj Kj | > ) 2ke2n
j
2 /(2B 2 )

Let
n

2B 2 log(2k/) . n
n)

Then P(max |Kj Kj | >

j

If we choose j = argmaxj Kj , then, with probability at least 1 , Kb max Kj 2 j

j

2B 2 log(2k/) = max Kj O j n

log k n

So with high probability, you choose close to the best model. This argument can be improved and also applies to regression, classication etc. Of course, with regression, the loss function is E(Y m(X))2 and the cross-validation score is then 1 n For classication we use 1 n
n

i=1 n

(Yi m(Xi ))2 .

I(Yi = h(Xi )).

i=1

We have made essentially no assumptions or approximations. (The bounded on log f can be relaxed.) The beauty of cross-validation is its simpicity and generality. It can be shown that AIC and cross-validation have very similar behavior. But, cross-validation works under weaker conditions.

BIC

BIC stands for Bayesian Information Criterion. It is also known as the Schwarz Criterion after Gideon Schwarz. It is virtually identical to the MDL (minimum description length) criterion. 7

We choose j to maximize BICj =

j (j )

dj log n. 2

This is the same as AIC but the penalty is harsher. Thus, BIC tends to choose simpler models. Here is the derivation. We put a prior j (j ) on the parameter j . We also put a prior probability pj that model Mj is the true model. By Bayes theorem P (Mj |Y1 , . . . , Yn ) p(Y1 , . . . , Yn |Mj )pj . Furthermore, p(Y1 , . . . , Yn |Mj ) = p(Y1 , . . . , Yn |Mj , j )j (j )dj = L(j )j (j )dj .

We know choose j to maximize P (Mj |Y1 , . . . , Yn ). Equivalently, we choose j to maximize log L(j )j (j )dj + log pj .

Some Taylor series approximations show that log L(j )j (j )dj + log pj
j (j )

dj log n = BICj . 2

What happened to the prior? It can be shown that the terms involving the prior are lower order than the term that appear in formula for BICj so they have been dropped. BIC behaves quite dierently than AIC or cross-validation. It is also based on dierent assumptions. BIC assumes that one of the models is true and that you are trying to nd the model most likely to be true in the Bayesian sense. AIC and cross-validation are trying to nd the model that predict the best.

Model Averaging
Suppose we want to predict a new observation Y . Let D =

Bayesian Approach.

{Y1 , . . . , Yn } be the observed data. Then p(y|D) =

j

p(y|D, Mj )P(Mj |D) 8

where P(Mj |D) = L(j )j (j )dj L(s )s (s )ds s eBICj . BICs se

Frequentist Approach. There is a large and growing literaure on frequenist model averaging. It is discussed in 10-702.

7
Let

Simple Normal Example

Y1 , . . . , Yn N (, 1).

We want to compare two models: M0 : N (0, 1), Hypothesis Testing. We test H0 : = 0 versus = 0. The test statistic is Z= Y 0 Var(Y ) We reject H0 if |Z| > z /2. For = 0.05, we reject H0 if |Z| > 2, i.e., if 2 |Y | > . n = n Y. and M1 : N (, 1).

AIC. The likelihood is proportional to

n

L() = where S 2 =
i (Yi

e(Yi )
i=1

2 /2

= en(Y )

2 /2

enS

2 /2

Y )2 . Hence, n(Y )2 nS 2 () = . 2 2 9

Recall that AIC =

|S|. The AIC scores are

nS 2 nY AIC0 = (0) 0 = 2 2 and AIC1 = () 1 = since = Y . We choose model 1 if AIC1 > AIC0 that is, if
2

nS 2 1 2

nS 2 nY nS 2 1> 2 2 2 or 2 |Y | > . n Similar to but not the same as the hypothesis test.

BIC. The BIC scores are 0 nY nS 2 BIC0 = (0) log n = 2 2 2 and 1 nS 2 1 BIC1 = () log n = log n. 2 2 2 We choose model 1 if BIC1 > BIC0 that is, if |Y | > log n . n
2

Hypothesis testing AIC/CV/Cp BIC

controls type I errors nds the most predictive model nds the true model (with high probability)

10

Lecture Notes 17 1 Multiple Testing and Condence Intervals

Suppose we need to test many null hypotheses H0,1 , . . . , H0,N where N could be very large. We cannot simply test each hypotheses at level because, if N is large, we are sure to make lots of type I errors just by chance. We need to do some sort of multiplicity adjustment.

Familywise Error Control. Suppose we get a p-value pj for each null hypothesis. Let I = {i : H0,i is true}. If we reject H0,i for any i I then we have made an error. Let R = {j : we reject H0j } be the set of hypotheses we reject. We say that we have controlled the familywise error rate at level if P(R I = ) . The easiest way to control the familywise error rate is the Bonferroni method. The idea is to reject H0,i if and only if pi < /N . Then P(making a false rejection) = P pi < =
iI iI

for some i I N P pi < N since pi Unif(0, 1) for i I N

|I| . N

So we have overall control of the type I error. However, it can have low power. The Normal Case. Suppose that we have N sample means Y1 , . . . , YN each based on n Normal observations with variance 1. So Yj N (j , 1/n). To test H0,j : j = 0 we can use 1

the test statistic Tj =

nYj . The p-value is pj = 2(|Tj |).

If we did uncorrected testing we rject when pj < , which means, |Tj | > z/2 . A useful approximation is: z So we reject when |Tj | > 2 log(2/). 2 log(1/).

Under the Bonferroni correction we reject when pj < /N which coresponds to |Tj | > 2 log(2N/). log N .

Hence, the familywise rejection threshold grows like

False Discovery Control. The Bonferroni adjustment is very strict. A weaker type of control is based on the false discovery rate.1 Suppose we reject a set of hyptheses R. Dene the false discovery proportion FDP = |R I| |R|

where the ratio is dened to be 0 in case both the numerator and denominator are 0. Our goal is to nd a method for choosing R such that FDR = E(FDP) . The Benjamini-Hochberg method works as follows: 1. Find the ordered p-values P(1) < < P(N ) . 2. Let j = max{i : P(i) < i/N }. Let T = P(j) . 3. Let R = {i : Pi T }.
1

Reference: Benjamini and Hochberg (1995).

Let us see why this controls the FDR. Consider, in general, rejecting all hypothesis for which Pi < t. Let Wi = 1 if H0,i is true and Wi = 0 otherwise. Let G be the empirical distribution of the p-values and let G(t) = E(G(t)). In this case, FDP = Hence, E(FDP) = =
1 E( N 1 E( N N 1 i=1 Wi E(I(Pi < t)) N N 1 i=1 E(I(Pi < t)) N N i=1 Wi I(Pi < t)) N i=1 I(Pi < t)) N i=1 Wi I(Pi < t) N i=1 I(Pi < t) 1 N 1 N N i=1 Wi I(Pi < t) . N i=1 I(Pi < t)

t|I| t t . G(t) G(t) G(t)

Let t = P(i) for some i; then G(t) = i/N . Thus, FDR P(i) N/i. Setting this equal to we get P(i) < i/N is the Benjamini-Hochberg rule. FDR control typically has higher power than familywise control. But they are controlling dierent things. You have to decide, based on the context, which is appropriate. Example 1 Figure 1 shows an example where Yj N (j , 1) for j = 1, . . . , 1, 000. In this example, j = 3 for 1 j 50 and j = 0 for j > 50. The gure shows the test statistics, the p-values, the sorted log p-values with the Bonferroni threshold and the sorted log p-values with the FDR threshold (using = 0.05). Bonferroni rejects 7 hypotheses while FDR rejects 22. Multiple Condence Intervals. A similar problem occurs with condence intervals. If we construct a condence interval C for one parameter then P( C) 1 . But if we construct condence intervals C1 , . . . , CN for N parameters 1 , . . . , N then we want to ensure that P(j Cj , for all j = 1, . . . , N ) 1 . To do this, we construct each condence interval Cj at level 1 /N . Then P(j Cj for some j) /
j

P(j Cj ) /

= . N

pvalue 0 200 400 600 800 1000

0.0 0

0.2

0.4

0.6

0.8

1.0

200

400

600

800

1000

Index

Index

sorted log pvalues

12

q q q

12 200 400 600 800 1000

sorted log pvalues

qqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqq qqqqq qqqqq qqqqq qqqqq qqqqq qq qq qq qq qq qq qqq qqq qqq qqq qqq qqq qq qq qq qq qq qq q q qq qq qq qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q

qqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqq qqqqq qqqqq qqqqq qqqqq qqqqq qq qq qq qq qq qq qqq qqq qqq qqq qqq qqq qq qq qq qq qq qq q q qq qq qq qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q

8 6 4 2

8 6 4 2

200

400

600

800

1000

Bonferroni

FDR

Figure 1: Top left: 1,000 test statistics. Top right: the p-values. Bottom left: sorted log p-values and Bonferroni threshold. Bottom right: sorted log p-values and FDR threshold.

Causation

Most of statistics and machine learning is concerned with prediction. A typical question is: what is a good prediction of Y given that I observe that X = x? Causation is concerned with questions of the form: what is a good prediction of Y given that I set X = x? The dierence between passively observing X = x and actively intervening and setting X = x is signicant and requires dierent techniques and, typically, much strnger assumptions. Consider this story. A mother notices that tall kids have a higher reading level than short kids. (This is because the tall kids are older.) The mother puts her small child on a device and stretches the child until he is tall. She is dismayed to nd out that his reading level has not changed. Te mother is correct that height and reading skill are associated. Put another way, you can use height to predict reading skill. But that does not imply that height causes reading skill. This is what statisticians mean when they say: correlation is not causation. On the other hand, consider smoking and lung cancer. We know that smoking and lung cancer are associated. But we also believe that smoking causes lung cancer. In this case, we recognize that intervening and forcing someone to smoke does change his probability of getting lung cancer. The dierence between prediction (association/correlation) and causation is this: in prediction we are interested in P(Y A|X = x) which means: the probability that Y A given that we observe that X is equal to x. For causation we are interested in P(Y A|set X = x) which means: the probability that Y A given that we set X equal to x. Prediction is about passive observation. Causation is about active intervention. Most of statistics and

machine learning concerns prediction. But sometimes causation is the primary focus. The phrase correlation is not causation can be written mathematically as P(Y A|X = x) = P(Y A|set X = x). Despite the fact that causation and association are dierent, people mix them up all the time, even people trained in statistics and machine learning. On TV recently there was a report that good health is associated with getting seven hours of sleep. So far so good. Then the reporter goes on to say that, therefore, everyone should strive to sleep exactly seven hours so they will be healthy. Wrong. Thats confusing causation and association. Another TV report pointed out a correlation between people who brush their teeth regularly and low rates of heart disease. An interesting correlation. Then the reporter (a doctor in this case) went on to urge people to brush their teeth to save their hearts. Wrong! To avoid this confusion we need a way to discuss causation mathematically. That is, we need someway to make P(Y A|set X = x) formal. There are two common ways to do this. One is to use counterfactuals. The other is to use causal graphs. These approaches are equivalent. There are two dierent languages for saying the same thing. Causal inference is tricky and should be used with great caution. The main messages are: 1. Causal eects can be estimated consistently from randomized experiments. 2. It is dicult to estimate causal eects from observational (non-randomized) experiements. 3. All causal conclusions from observational studes should be regarded as very tentative. Causal inference is a vast topic. We will only touch on the main ideas here. Counterfactuals. Consider two variables Y and X. Suppose that X is a binary variable that represents some treatment. For example, X = 1 means the subject was treated and X = 0 means the subject was given placebo. The response variable Y is real-valued. We can address the problem of predicting Y from X by estimating E(Y |X = x). To address causal questions, we introduce counterfactuals. Let Y1 denote the response we observe if the subject is treated, i.e. if we set X = 1. Let Y0 denote the response we observe if the 6

subject is not treated, i.e. if we set X = 0. If we treat a subject, we observe Y1 but we do not observe Y0 . Indeed, Y0 is the value we would have observed if the subject had been treated. The unobserved variable is called a counterfactual. We have enlarged our set of variables from (X, Y ) to (X, Y, Y0 , Y1 ). Note that Y = XY1 + (1 X)Y0 . A small dataset might look like this: X 1 1 1 1 0 0 0 0 Y 1 1 0 1 1 0 1 1 Y0 * * * * 1 0 1 1 Y1 1 1 0 1 * * * * (1)

The asterisks indicate unobserved variables. To answer causal questions, we are interested in the distribution p(y0 , y1 ). We can interpret p(y1 ) as p(y|set X = 1) and we can interpret p(y0 ) as p(y|set X = 0). In particular, we might want to estimate the mean treatment eect or mean causal eect = E(Y1 ) E(Y0 ) = E(Y |set X = 1) E(Y |set X = 0). The parameter has the following intepretation: is the mean response if we forced everyone to take the treatment minus mean response if we forced everyone not to take the treatment. Suppose now that we observe a sample (X1 , Y1 ), . . . , (Xn , Yn ). Can we estmate ? No. In general, there is no consistnt estimator of . We can estimate = E(Y |X = 1)E(Y |X = 0) but is not equal to .

However, suppose that we did a randomized experiment where we randomly assigned each person to treatment of placebo by the ip of a coin. In this case, X will be independent of (Y0 , Y1 ). In symbols: random treatment assignment implies : (Y0 , Y1 ) Hence, in this case, = E(Y |X = 1) E(Y |X = 0) = E(Y1 |X = 1) E(Y0 |X = 0) since Y = XY1 + (1 X)Y0 = E(Y1 ) E(Y0 ) = since (Y0 , Y1 ) X. X.

Hence, random assignment makes equal to and can be consistently estimated. If X is randomly assigned then correlation = causation. This is why people spend millions of dollars doing randomized experiements. In some cases it is not feasible to do a randomized experiment. Smoking and lung cancer is an example. Can we estimate causal parameters from observational (non-randomized) studies? The answer is: sort of. In an observational stsudy, the treated and untreated groups will not be comparable. Maybe the healthy people chose to take the treatment and the unhealthy people didnt. In other words, X is not independent of (Y0 , Y1 ). The treatment may have no eect but we would still see a strong association between Y and X. In other words, might be large even though = 0. To account for the dierences in the groups, we might measure confounding variables. These are the variables that aect both X and Y . By denition, there are no such variables in a randomized experiment. The hope is that if we measure enough confounding variables Z = (Z1 , . . . , Zk ), then, perhaps the treated and untreated groups will be comparable, conditional on Z. Formally, we hope that X is indpendent of (Y0 , Y1 ) conditional on Z. If this is true,

we can estimate since = E(Y1 ) E(Y0 ) = = = E(Y1 |Z = z)p(z)dz E(Y0 |Z = z)p(z)dz E(Y0 |X = 0, Z = z)p(z)dz E(Y |X = 0, Z = z)p(z)dz (2)

E(Y1 |X = 1, Z = z)p(z)dz E(Y |X = 1, Z = z)p(z)dz

where we used the fact that X is indpendent of (Y0 , Y1 ) conditional on Z in the third line and the fact that Y = (1 X)Y1 + XY0 in the fourth line. The latter quantity can be estimated by 1 = n
n

i=1

1 m(1, Zi ) n

m(0, Zi )
i=1

where m(x, z) is an estimate of the regression function m(x, z) = E(Y |X = x, Z = z). This is known as adjusting for confounders and is called the adjusted treatment eect. It is instructive to compare the casual eect = E(Y1 ) E(Y0 ) = E(Y |X = 1, Z = z)p(z)dz E(Y |X = 0, Z = z)p(z)dz

with the predictive quantity = E(Y |X = 1) E(Y |X = 0) = E(Y |X = 1, Z = z)p(z|X = 1)dz E(Y |X = 0, Z = z)p(z|X = 0)dz

which are mathematically (and conceptually) quite dierent. We need to treat cautiously. It is very unlikely that we have successfully measured all the relevant confounding variables so should be regarded as a crude approximation to at best. Causal Graphs. Another way to capture the dierence between P (Y A|X = x) and P (Y A|set X = x) is to represent the distribution using a directed graph and then we capture the second statement by performing certain operations on the graph. 9

A Directed Acyclic Graph (DAG) is a graph for a set of variables with no cycles. The graph denes a set of distributions of the form p(y1 , . . . , yk ) = p(yj |parents(yj )

where parents(yj ) are the parents of yj . A causal graph is a DAG with extra information. A DAG is a causal graph if it correctly encodes the eect of setting a variable to a xed value. Consider the graph G in Figure (2). Here, X denotes treatment, Y is response and Z is a confounding variable. To nd the causal distribution p(y|set X = x) we do the following steps: 1. Form a new graph G by removing all arrow into X. Now set X equal to x. This corresponds to replacing the joint distribution p(x, y, z) = p(z)p(x|z)p(y|x, z) with the new distribution p (y, z) = p(z)p(y|x, z). The factor p(x|z) is removed because we know regard x as a xed number. 2. Compute the distribution of y from the new distribution: p(y|set X = x) p (y) = Now we have that = p(y|set X = 1) p(y|set X = 0) = p(z)p(y|1, z)dz p(z)p(y|0, z)dz p (y, z)dz = p(z)p(y|x, z)dz.

This is precisely the same equation as (2). Both approaches lead to the same thing. If there were unobserved confounding variables, then the formula for would involve these variables and the causal eect would be non-estimable (as before). In a randomized experiment, there would be no arrow from Z to X. (Thats the point of randomization). In that case the above calculations shows that = E(Y |X = 1) E(Y |X = 0) just as we saw with the counterfactual approach. To understand the dierence between p(y|x) and p(y|set x) more clearly, it is helpful to consider two dierent computer programs. Consider the DAG in Figure 2. The 10

Figure 2: Conditioning versus intervening. probability function for a distribution consistent with this DAG has the form p(x, y, z) = p(x)p(y|x)p(z|x, y). The following is pseudocode for generating from this distribution.

For i

1, . . . , n :

xi < pX (xi ) yi < pY |X (yi |xi ) zi < pZ|X,Y (zi |xi , yi ) Suppose we run this code, yielding data (x1 , y1 , z1 ), . . . , (xn , yn , zn ). Among all the times that we observe Y = y, how often is Z = z? The answer to this question is given by the conditional distribution of Z|Y . Specically, P(Z = z|Y = y) = p(y, z) P(Y = y, Z = z) = P(Y = y) p(y) p(x) p(y|x) p(z|x, y) x p(x, y, z) = x = p(y) p(y) p(y|x) p(x) p(x, y) = p(z|x, y) = p(z|x, y) p(y) p(y) x x =
x

p(z|x, y) p(x|y).

Now suppose we intervene by changing the computer code. Specically, suppose we x Y at the value y. The code now looks like this:

11

set Y for i

= =

y 1, . . . , n

xi < pX (xi ) zi < pZ|X,Y (zi |xi , y) Having set Y = y, how often was Z = z? To answer, note that the intervention has changed the joint probability to be p (x, z) = p(x)p(z|x, y). The answer to our question is given by the marginal distribution p (z) =
x

p (x, z) =
x

p(x)p(z|x, y).

This is p(z|set Y = y). Example 2 You may have noticed a correlation between rain and having a wet lawn, that is, the variable Rain is not independent of the variable Wet Lawn and hence pR,W (r, w) = pR (r)pW (w) where R denotes Rain and W denotes Wet Lawn. Consider the following two DAGs: Rain Wet Lawn Rain Wet Lawn.

The rst DAG implies that p(w, r) = p(r)p(w|r) while the second implies that p(w, r) = p(w)p(r|w) No matter what the joint distribution p(w, r) is, both graphs are correct. Both imply that R and W are not independent. But, intuitively, if we want a graph to indicate causation, the rst graph is right and the second is wrong. Throwing water on your lawn doesnt cause rain. The reason we feel the rst is correct while the second is wrong is because the interventions implied by the rst graph are correct. Look at the rst graph and form the intervention W = 1 where 1 denotes wet lawn. Following the rules of intervention, we break the arrows into W to get the modied graph: Rain set Wet Lawn =1 12

with distribution p (r) = p(r). Thus P(R = r | W := w) = P(R = r) tells us that wet lawn does not cause rain. Suppose we (wrongly) assume that the second graph is the correct causal graph and form the intervention W = 1 on the second graph. There are no arrows into W that need to be broken so the intervention graph is the same as the original graph. Thus p (r) = p(r|w) which would imply that changing wet changes rain. Clearly, this is nonsense. Both are correct probability graphs but only the rst is correct causally. We know the correct causal graph by using background knowledge. Learning Casual Structure? We could try to learn the correct causal graph from data but this is dangerous. In fact it is impossible with two variables. With more than two variables there are methods that can nd the causal graph under certain assumptions but they are large sample methods and, furthermore, there is no way to ever know if the sample size you have is large enough to make the methods reliable. Randomization Again. We can use DAGs to represent confounding variables. If X is a treatment and Y is an outcome, a confounding variable Z is a variable with arrows into both X and Y ; see Figure 3. It is easy to check, using the formalism of interventions, that the following facts are true: In a randomized study, the arrow between Z and X is broken. In this case, even with Z unobserved (represented by enclosing Z in a circle), the causal relationship between X and Y is estimable because it can be shown that E(Y |X := x) = E(Y |X = x) which does not involve the unobserved Z. In an observational study, with all confounders observed, eect. If Z is unobserved then we cannot estimate the causal eect because E(Y |X := x) = in this case. P(Y = y|X = x) = P(Y = y|X := x) which is just another way of saying that causation is not association. E(Y |X = x, Z = z)dFZ (z) involves the unobserved Z. We cant just use X and Y since we get E(Y |X := x) = E(Y |X = x, Z = z)p(z) which is just the adjusted treatment

13

Figure 3: Randomized study; Observational study with measured confounders; Observational study with unmeasured confounders. The circled variables are unobserved.

Individual Sequence Prediction

2

The goal is to predict yt from y1 , . . . , yt1 with no assumptions on the sequence.

The data

are not assumed to be iid; they are not even assumed to be random. This is a version of online learning. For simplicity assume that yt {0, 1}. Suppose we have a set of prediction algorithms (or experts): F = {F1 , . . . , FN } Let Fj,t is the prediction of algorithm j at time t based on y t1 = (y1 , . . . , yt1 ). At time t: 1. You see y t1 and (F1,t , . . . , FN,t ). 2. You predict Pt . 3. yt is revealed. 4. You suer loss (Pt , yt ). We will focus on the loss (pt , yt ) = |pt yt | but the theory works well for any convex loss. The cumulative loss is 1 Lj (y ) = n
n n

i=1

|Fj,t yt |

1 Sj (y n ) n

where Sj (y n ) =

n i=1

|Fj,t yt |. The maximum regret is Rn = tmax

y {0,1}t

LP (y n ) min Lj (y n )
j

Reference: Prediction, Learning, and Games. Nicol` Cesa-Bianchi and Gbor Lugosi, 2006. o a

14

and the minimax regret is Vn = inf tmax Let Pt (y t1 ) =

N j=1

P y {0,1}t

LP (y n ) min Lj (y n ) .
j

wj,t1 Fj,t where wj,t1 = exp {Sj,t1 } Zt

and Zt =

N j=1

exp {Sj,t1 } . The wj s are called exponential weights. 8 log N/n. Then LP (y n ) min Lj (y n )
1jN

Theorem 3 Let =

log N . 2n
Zn+1 Z1

Proof. The idea is to place upper and lower bounds on log Upper bound: We have log Zn+1 Z1
N

then solve for LP (y n ).

= log
j=1

exp {nLj,n }

log N

log max exp {nLj,n } log N

j

= n min Lj,n log N.

j

(3)

Lower bound: Note that log Zt+1 Zt = log = log E e

N j=1

wj,t1 e|Fj,t yt | wj,t1 .

N j=1 |Fj,t yt |

This is a formal expectation with respect to the distribution over j probability proportional to e|Fj,t yt | ). Recall Hoedings bound for mgf: if a X b log E(esX ) sE(X) + s2 (b a)2 . 8

15

So: log E e|Fj,t yt | E|Fj,t yt | + 2 8 2 = |EFj,t yt | + 8 = |Pt (y t1 ) yt | + Summing over t: log Combining (3) and (4) we get n min Lj (y n ) log N log
j

2 . 8

Zn+1 Z1

nLP (y n ) + Zn+1 Z1

n 2 . 8 n 2 . 8

(4)

nLP (y n ) +

Rearranging the terms we have: LP (y n ) min Lj (y n ) +

j

n log N + . 8

Set =

8 log N/n to get LP (y n ) min Lj (y n )

1jN

log N . 2n

The result held for a specic time n. We can make the result uniform over time as follows. If we set t = 8 log N/t then we have: LP (y n ) min Lj (y n ) +
j

1 + 12n log N 8

for all n and for all y1 , y2 , . . . , yn . every F and every y n there is a Gj such that
n

Now suppose that F is an innite class. A set G = {G1 , . . . , GN } is an r-covering if, for

t=1

|Ft (y t1 ) Gj,t (y t1 )| r.

Let N (r) denote the size of the smallest r-covering. 16

Theorem 4 (Cesa-Bianchi and Lugosi) We have that Vn (F) inf

r>0

r + n

log N (r) 2n

Cesa-Bianchi and Lugosi also construct a predictor that nearly achieves the bound of the form Pt = where
(k) Pt k=1 (k)

ak P t

is a predictor based on a nite subset of F.

Using batchication it is possible to use online learning for non-online learning. Suppose we are given data: (Z1 , . . . , Xn ) where Zi = (Xi , Yi ) and an arbitrary algorithm A that takes data and outputs classier H. We used uniform convergence theory to analyze H but online methods provide an alternative analysis. H0 , H1 , . . . , Hn . Let 1 Mn = n To choose a nal classier: 1. usual batch method: use the last one Hn 2. average: H =
1 n n i=1 n 3

We apply A sequentially to get classiers

(Ht1 (Xt ), Yt )
i=1

Ht1
t

3. selection: choose Ht to minimize 1 t (Ht (Xt ), Yt ) +

i=1

1 log 2(n t)

n(n + 1)

Analyzing Hn requires assumptions on A, uniform convergence etc. This is not needed for the other two methods. Theorem 5 If is convex: P R(H) Mn + For any , P R(H) Mn +
3

2 log n

36 log n

2(n + 1)

Reference: Cesa-Bianchi, Conconi and Gentile (2004).

17

Homework 1 36-705 Due: Thursday Sept 8 by 3:00 From Casella and Berger: 1. Chapter 1, problem 1.47. 2. Chapter 1, problem 1.49. 3. Chapter 2, problem 2.1. 4. Chapter 2, problem 2.3. 5. Chapter 2, problem 2.7a. 6. Chapter 2, problem 2.15. 7. Chapter 2, problem 2.30. 8. Chapter 3, problem 3.32. 9. Chapter 4, problem 4.4. 10. Chapter 4, problem 4.5.

Homework 2 36-705 Due: Thursday Sept 15 by 3:00 1. Let Xn be a sequence of random variables such that Xn 0 for all n. Suppose that P(Xn > t) ( 1 )k where k > 1. Derive an upper bound on E(Xn ). t 2. Let X1 , . . . , Xn Unif(0, 1). Let Y = max1in Xi . (i) Bound E(Y ) using the method we derived in lecture notes 2.

(ii) Find an exact expression for E(Y ). Compare the result to part (i). 3. An improvement on Hoedings inequality is Bernsteins inequality. Let X1 , . . . , Xn be iid, with mean , Var(Xi ) = 2 and |Xi | c. Then Bernsteins inequality says that P |X n | > 2 exp n2 2 2 + 2c /3 .

(When is suciently small, this bound is tighter than Hoedings inequality.) Let X1 , . . . , Xn Uniform(0, 1) and An = [0, 1/n]. Let pn = P(Xi An ) and let 1 pn = n
n

IAn (Xi ).
i=1

(i) Use Hoedings inequality and Bernsteins inequality to bound P(|pn pn | > ). (ii) Show that the bound from Bernsteins inequality is tighter. (iii) Show that Hoedings inequality implies pn pn = O inequality implies pn pn = OP (1/n).
1 n

but that Bernsteins

4. Show that Xn = oP (an ) and Yn = OP (bn ) implies that Xn Yn = oP (an bn ).

Homework 3 36-705 Due: Thursday Sept 22 by 3:00 1. Let A be a class of sets. Let B = {Ac : A A}. Show that sn (B) = sn (A). 2. Let Let A and B be classes of sets. Let C= Show that sn (C) sn (A)sn (B). 3. Show that sn+m (A) sn (A)sm (A). 4. Let A= A = [a, b] [c, d] : a, b, c, d R, abcd . A B : A A, B B .

Find VC dimension of A.

Homework 4 36-705 Due: Thursday September 29 by 3:00 1. 5.33 2. 5.34 3. 5.35 4. 5.36 5. 5.39

Homework 5 36-705 Due: Thursday October 6 by 3:00 1. 6.2 2. 6.4 3. 6.9 (b) and (e). 4. Write (x1 , . . . , xn ) (y1 , . . . , yn ) to mean that the likelihood function based on (x1 , . . . , xn ) is proportional to the likelihood function based on (y1 , . . . , yn ). The equivalence relation induces a partition of the sample space: (x1 , . . . , xn ) and (y1 , . . . , yn ) are in the same element of the partition if and only if (x1 , . . . , xn ) (y1 , . . . , yn ). Show that is a minimal sucient partition. 5. 7.1 6. 7.5 (a). 7. 7.8. 8. 7.9. 9. In class, we found the minimax estimator for the Bernoulli. Here, you will ll in the details. Let X1 , . . . , Xn Bernoulli(p). Let L(p, p) = (p p)2 . (a) Let p be the Bayes estimator using a Beta(, ) prior. Find the Bayes estimator. (b) Compute the risk function. (c) Compute the Bayes risk. (d) Find and to make the risk constant and hence nd the minimax estimator.

Homework 6 36-705 Due: Thursday October 20 by 3:00 1. 10.1 2. 10.2 3. 10.4 4. 10.18 5. 10.19

Homework 7 36-705 Due: Thursday October 27 by 3:00 1. 8.13 (a,b) 2. 8.14 3. 8.15 4. 8.17 5. 8.20 6. 10.31 (a,b,c,e) 7. Show that, when H0 is true, then the p-value has a Uniform (0,1) distribution.

Homework 8 36-705 Due: Thursday November 10 2010 by 3:00 1. 9.1. 2. 9.4(a) 3. 9.33(a) 4. Let X1 , . . . , Xn Uniform(0, ). Find the 1 likelihood ratio condence interval for . Note: the limiting 2 theory does not apply to this example. You need to nd the cuto value directly. 5. Let X1 , . . . , Xn p and assume that 0 Xi 1. The histogram density estimator is dened as follows. Divide [0, 1] into m bins B1 = [0, 1/m], B2 = (1/m, 2/m], . . . ,. Let h = 1/m and let j = n1 n I(Xi Bj ). Let i=1 p(x) = j h

when x Bj . Find the asymptotic MSE. Find the best h. Find the rate of convergence of the estimator.

Homework 9 10/36-705 Due: Thursday Nov 17 by 3:00 1. Let X1 , . . . , Xn p and let ph denote the kernel density estimator with bandwidth h. Let R(h) = E[L(h)] denote the risk, where L(h) = (a) Dene R(h) = E[L(h)] where L(h) = (ph (x))2 dx 2 ph (x)p(x)dx. (ph (x) p(x))2 dx.

Show that minimizing R(h) over h is equivalent to minimizing R(h). (b) Let Y1 , . . . , Yn be a second sample from p. Dene R(h) = 2 (ph (x)) dx n
2 n

ph (Yi )
i=1

where ph is still based on X1 , . . . , Xn . Show that ER(h) = R(h). (Hence, R(h) can be used as an estimate of the risk.) 2. Again, let ph denote the kernel density estimator. Use Hoedings inequality to nd a bound on P(|ph (x) ph (x)| > ) where ph (x) = E(ph (x)).
3. Let X1 , . . . , Xn Bernoulli(). Let n = n1 n Xi . Let X1 , . . . , Xn denote a i=1 n bootstrap sample. Let = n1 i=1 Xi . Find the following four quantities:

E( |X1 , . . . , Xn ),

E( ),

V( |X1 , . . . , Xn ),

V( ).

4. The bootstrap estimate of Var(n ) is V( |X1 , . . . , Xn ). (In other words, when B , the bootstrap estimate of variance converges to V( |X1 , . . . , Xn ).) Show that the bootstrap is consistent, in the sense that V( |X1 , . . . , Xn ) Var(n ) 1.
P

Homework 10 36/10-705 Due: Thursday December 1 2010 by 3:00 1. 7.23 2. 9.27 3. Suppose that V = k (Zj + j )2 where Z1 , . . . , Zk are independent, standard Normal j=1 random variables. We say that V has a non-central 2 distribution with non-centrality 2 parameter = j j and k degrees of freedom. We write V 2 (). k (a) Show that if V 2 () then E(V ) = k + and Var(V ) = 2(k + 2). k (b) Let Yi N (i , 1) for i = 1, . . . , n. (1 , . . . , k ) using a at prior. (c) Find the posterior distribution of = (d) Find the mean of the posterior. (e) Find the bias and variance of . (f) Show that is not a consistent estimator of . (Technically, the parameter is changing with n. You may assume that is bounded as n increases.) Hint: you may use the fact that if V 2 (), then (V E(V ))/ Var(V ) N (0, 1). k Find the posterior distribution of = 2 . i

(g) Find cn so that P ( Cn |X1 , . . . , Xn ) = 1 where Cn = [cn , ).

(h) Construct an unbiased estimator of . Compare this to the Bayes estimator.

(i) Find a (frequentist) condence interval An = [an , ) such that P ( An ) = 1 for all . Compare this to the Bayes posterior interval Cn .

2011 Fall 10705 Homework 1 Solutions

2.7a.

P y X y 2 P Y y P X y P 1 X y

y y 1 y f X x dx y 1 y f x dx 1 X

y 1

y 1

Differentiationgives

2 9 y fY y 1 1 1 9 y

y 1 y 1

3.32a. Notethat c*1

h x exp t x dx .Wehave
i i i

log c* log c*1 j j j h x exp t x dx

i i i

c*

3.31(a) interchange integration and differentiation

c* h x

exp i ti x dx j i

c* h x exp j t j x t j x dx t j x h x c* exp j t j x dx t j x

and

2 2 log c* log c*1 j 2 j 2

2 *

2 * 1 c log c*1 c 2 j j c* h x

(3.31 (b))

2 exp i ti x dx 2 t j x 2 j i

t j 2 x h x c* dx 2 t j x

t j 2 x 2 t j x Var t j x
3.32b.

x; , x 1

exp x / 1 1 exp x 1 ln x

Thenaturalparametersandsufficientstatisticsare [ 1 / , 1], t x [ x, log x ] .Further,

log c* log log log 1 / 1 log

Therefore

log c* log 1 / 1 1 1 1

36-705 Intermediate Statistics HW2

Problem 1
As X 0 with Prob 1, we have E[X] =
0 0 a a

(1 F (t))dt =

P (X > t)dt =
0

P (X > t)dt +

P (X > t)dt

With P (X > t) 1 and P (X > t) (1/t)k , we have the upper bound of E[X],
a

E[X] =
0

P (X > t)dt +
a

P (X > t)dt

1dt +
0 a

(1/t)k dt = a +

1 . (k 1)ak1

Set the derivative of it to be 0, we get 1 + 1/ak = 0, so we get a = 1. The upper bound of k 1 E[X] is 1 + k1 = k1 .

Problem 2
The cumulative density function of Y is
n

P (Y y) = P (Xi y, 1 i n) = So, the expected value of Y is

1 1

i=1

P (Xi y) = y n ,

0 y 1.

E[Y ] =
0

P (Y > t)dt =
0

(1 tn )dt = 1

1 n = . n+1 n+1

Problem 3
(i) Note that pn = P (Xi An ) = 1/n. Let Yi = IAn (Xi ), then E[Yi ] = E[IAn (Xi )] = P (Xi An ) = 1/n, 1 Yn = n
n n

i=1

1 Yi = n

IAn (Xi ) = pn ,
i=1

therefore: 1. Hoedings inequality: Yi = 0 or 1, thus the bound is 0 Yi 1, and nally P (|n pn | ) = P (|Yn E[Y ]| ) 2 exp{ p 2n 2 2 } = 2e2n ; (1 0)2

2. Bernsteins inequality: still 0 Yi 1, hence |Yi | 1, and the variance is V ar(Yi ) = 1/n(1 1/n) = as Yi Bernoulli(1/n). So, we have P (|n pn | ) = P (|Yn E[Y ]| ) 2 exp{ p n2 }. 2(n 1)/n2 + 2 /3 n1 , n2

(ii) When is small and n is large, 2(n 1)/n2 + 2 /3 will be very small, in the order of 1/n, so 2(n 1)/n2 + 2 /3 < 1/2, and so we have 2 exp{ n2 2 } 2e2n . 2 + 2 /3 2(n 1)/n

Therefore, Bernsteins inequality is tighter than Hoedings inequality. (iii) Use Hoedings inequality, P(
2 |n pn | p 2 C) = P (|n pn | C/ n) 2e2n(C/ n) = 2e2C . p 1/ n

n n So, for any , when C is large enough, there is P ( p1/pn C) , therefore, Hoedings inequality implies pn pn = Op (1/ n). Use Bernsteins inequality, we have

P(

nC 2 /n2 |n pn | p C) = P (|n pn | C/n) 2 exp{ p }. 1/n 2(n 1)/n2 + 2C/3n

Simplify the exponential part, we have nC 2 /n2 C 2 /n C2 3C = = , 2 + 2C/3n 2 + 2C/3n 2(n 1)/n 2(n 1)/n 2(n 1)/n + 2C/3 2
3C |n pn | p C) 2e 2 . 1/n

for large n and large C. So, in all, we have P(

For any , there is C large enough, such that the probability is smaller than . So, Bernsteins inequality implies pn pn = Op (1/n).

Problem 4

36-705 Intermediate Statistics HW3

Problem 1
Notice that (A F ) (Ac F ) = (A Ac ) F = . On the other hand, (A F ) (Ac F ) = (A Ac ) F = F = F . This shows A and Ac pick dierent parts of F , that is, (Ac F ) = F \(A F ). For any nite set F w/ n elements, say the total number of distinct A F is m1 , then for every distinct A F , there is a corresponding Ac B such that Ac F picks the other part of F . Then the total number of distinct Ac F is also m1 . So we have S(A, F ) = S(B, F ), taking sup on both sides, we have, sn (A) = sn (B)

Problem 2
C = {AB : A A, B B}. Notice that for C C, C F = (AB)F = (AF )(BF ), therefore, A F C F F , and B F C F F . For any nite set F w/ n elements, say the total number of distinct A F is m1 and the total number of distinct B F is m2 . Then, the total number of distinct C F w/ C = A B, i.e. the total number of distinct intersections (A F ) (B F ) is at most m1 m2 (the maximum number of distinct pairs). That is S(C, F ) S(A, F )S(B, F ), taking sup on both sides, sn (C)
F Fn

sup [S(A, F )S(B, F )]

F Fn F Fn

sup S(A, F ) sup S(B, F ) = sn (A)sn (B).

Problem 3
Let Fn+m = Fn Fm where Fn with n elements and Fm with m elements are disjoint and Fn+m have m + n elements. For A A, A Fn+m = A (Fn Fm ) = (A Fn ) (A Fm ). Therefore, AFn AFn+m Fn+m , and AFm AFn+m Fn+m . For any nite set Fn w/ n elements and Fm w/ m elements, say the total number of distinct AFn is n1 and 1

the total number of distinct A Fm is m1 . Then, the total number of distinct A Fn+m w/ Fn+m = Fn Fm , which are subsets of distinct unions (A Fn ) (A Fm ) is at most n1 m1 (the maximum number of distinct pairs). That is S(A, Fn+m ) S(A, Fn )S(A, Fm ), taking sup on both sides, sn+m (A)
Fn Fn

sup
Fn Fn ,Fm Fm Fm Fm

[S(A, Fn )S(A, Fm )]

sup S(A, Fn ) sup S(A, Fm ) = sn (A)sm (A).

Problem 4
A is the set of single intervals or joint of two separate intervals on the real line. 1. Let F4 = {1, 0, 1, 2} with 4 elements. Then 1).[2, 1.5] F = ,

2).[1.5, 0.5] F = {1},

3).[1.5, 0.5] F = {1, 0},

9).[0.5, 1.5] F = {1},

8).[0.5, 2.5] F = {0, 1, 2},

7).[0.5, 1.5] F = {0, 1},

6).[0.5, 0.5] F = {0},

5).[1.5, 2.5] F = {1, 0, 1, 2},

4).[1.5, 1.5] F = {1, 0, 1},

10).[0.5, 2.5] F = {1, 2}, 11).[1.5, 2.5] F = {2}, 12).[1.5, 0.5] [0.5, 1.5] F = {1, 1}, 14).[0.5, 0.5] [1.5, 2.5] F = {0, 2}, 13).[1.5, 0.5] [1.5, 2.5] F = {1, 2},

So s(A, F4 ) = = 16 and s4 (A) = 16. The VC dimension of A, d(A) = max{n : sn (A) = 2n } 4. 2. For set Fn , st. n 5, eg F5 = {1, 0, 1, 2, 3}, it is impossible A F5 = {1, 0, 2}, since any interval covering {1, 1} will also cover {0}, similarly, the interval covering {0, 2} will also cover {1}. This is suce to show that the VC dimension of A is less than 5. So we have 4 d(A) < 5, that is d(A) = 4. 2

24

16).[1.5, 0.5] [0.5, 2.5] F = {1, 1, 2}

15).[1.5, 0.5] [1.5, 2.5] F = {1, 0, 2},

Test 1 Solutions

Problem 1
X1 and X2 are iid Unif(0,2), then, fX1 ,X2 (x1 , x2 ) = fX1 (x1 )fX2 (x2 ) = I(x1 (0, 2))I(x2 (0, 2)) = 0 < x1 < 2, 0 < x2 < 2 0, ow
1 4

FY (y) = P (Y y) = P (X1 X2 y) = P (X1 X2 + y) =

fX1 ,X2 (x1 , x2 )dx1 dx2

where, A = {(x1 , x2 ) R2 : x1 x2 + y}. Since the integral is over a function which takes 1 value 4 over a square and 0 everywhere else, the value of the integral is equal to 1 of the area 4 of the region determined by the intersection of A with the square 0 < x1 < 2, 0 < x2 < 2. The four dierent cases are shown in the last page. The cdf is, 0 y 2 (2+y)2 2 < y < 0 8 FY (y) = (2y)2 1 0y2 8 1 y>2 Dierentiate it wrt y to get the pdf, dFY fY (y) = = dy
2y 4 2+y 4

0y2 2 y < 0 ow

Problem 2

n Let Xi iid Bernoulli(p) for i = 1, 2, ..., n. Then X = i=1 Xi has Binomial(n, p) distribution and the MGF of X, MX = n MXi = (MXi )n . We know the MGF of i=1 Bernoulli distribution is

MX1 = E[etX1 ] = et p + e0 (1 p) 1

Then we have MX = (MX1 )n = (et p + (1 p))n

Problem 3
We know that, E(g(X)|Y ) = Then, E(E(g(X)|Y )) = = = = = [ E(g(X)|Y )p(y)dy g(x)p(x|y)dx

g(x)p(x|y)dx]p(y)dy g(x)p(x|y)p(y)dydx

g(x)[

p(x, y)dy]dx

g(x)p(x)dx

= E(g(X))

Problem 4
We know that X U nif (2, 1) and Y = e|X| , then Y = eX eX 0x1 2 x < 0

and 1 y e2 . The attached gure shows how the function looks like. The cdf is 0 P ( log(y) x log(y)) = log y 1 dx = 2 log y 3 log y 3 FY (y) = P (Y y) = P (e|X| y) = 1 y+1 P ( log(y) x 1) = log y 1 dx = log 3 3 1 Dierentiate the cdf with respect to y, we get the pdf, 2 3y 1 y e 1 e y < e2 pY (y) = 3y 0 ow 2

y<1 1ye e y < e2 y e2

Intermediate Statistics HW4

5.33 Since lim n FX n x FX x , lim x FX x 0 ,forany ,wecanfindanmandan N1 suchthat

P X n m 1 / 2 forn> N1 .Then,since lim n P Yn c m 1 ,wecanfindan N 2 suchthat

P Yn c m 1 / 2 forn> N 2 .

Note that P A B 1 P A P B , then P X n Yn c P X n m, Yn c m P X n m P Yn c m 1 1 / 2 1 / 2 1 1

for n max N1 , N 2 .Thus lim n P X n Yn c 1 .

5.36(a)

E Y E E Y | N E 2 N 2 E N 2 2 Var Y E Var Y | N Var E Y | N E 4 N Var 2 N 4 4 8

(b)

M Y / t E et Y / e t / E etY / tY e t / E E e | N e t / E 1 2t /

total expectation
e t / 1 2t /
n 0 n n

n e
n!

1 2t / 1 2t / e t / e e1 2t / e n! n0
e e e
t

Poisson 1 2t /

1 2 t /

2 8 1 2 t / 8

2 1 1 8 1 2 t / 8 t 2 2 t 3 2 2t 2 8 8 8

2 3 2 2t 2t 2t t 1 1 8 8 8 8 3 t 2t 2 8 2

Taylor expansion

e as

t2 2

whichisthemgfofN(0,1).

36-705 Intermediate Statistics HW5

Problem 1 (C&B 6.2)

Problem 2 (C&B 6.4)

Problem 3 (C&B 6.9)

(b)

(e) 1

Problem 4
Refer to Notes 6 p.2 for the denition of minimal sucient partition. Let be the parameter of the distribution and f be the joint pdf. f (x1 , ..., xn |) f (y1 , ..., yn |)

is independent of if and only if (x1 , ..., xn ) (y1 , ..., yn ). Therefore, by C&B Theorem 6.2.13, is a minimal sucient partition for .

Problem 5 (C&B 7.1)

1 1 1. x = 0, the likelihood L() = 3 I( = 1)+ 1 I( = 2)+0I( = 3) = 3 I( = 1)+ 1 I( = 4 4 2), therefore, the MLE = 1; 1 2. x = 1, L() = 3 I( = 1) + 1 I( = 2), = 1; 4

3. x = 2, L() = 1 I( = 2) + 1 I( = 3), = 2 or = 3; 4 4
1 4. x = 3, L() = 1 I( = 1) + 1 I( = 2) + 2 I( = 2), = 3; 6 4

5. x = 4, L() = 1 I( = 1) + 1 I( = 3), = 3. 6 4 2

Finally,

Problem 6 (C&B 7.5(a))

X = 0, 1; 1 = 2 or 3 X = 2; 3 X = 3, 4.

Problem 7 (C&B 7.8)

Problem 8 (C&B 7.9)

3

Problem 9
(a) Bayes estimator under square error loss L(p, p) = (p p)2 is the posterior mean. iid Xi Bernoulli(p), p Beta(, ) are conjugate, the posterior is p|X Beta( + P + i Xi i Xi ). Therefore, Bayes estimator p = ++n . i Xi , + n (b) Risk function for p R(p, p) = Ep [L(p, p)] = M SE() p = (E[] p)2 + V [] p p + np np(1 p) p)2 + = ( ++n ( + + n)2 ((1 p) p)2 np(1 p) = + 2 ( + + n) ( + + n)2 (c) Bayes risk for p B(, p) = R(p, p)(p)dp 4

= = = = =

1 2 ) + np np2 ](p)dp [( + )2 (p 2 ( + + n) + 1 n 2 + n( + [( + )2 )] ( + + n)2 ( + )2 ( + + 1) + ( + )2 ( + + 1) ( + )2 n n( + 1) 1 + ] [ ( + + n)2 + + 1 + ( + )( + + 1) 1 n + ] [ 2 + +1 ( + + n) ( + )( + + 1) ( + )( + + 1)( + + n)

(d) The risk R(p, p) = ((1 p) p)2 np(1 p) 1 + = {p2 [(+)2 n]+p[n2(+)]+2 } 2 2 ( + + n) ( + + n) ( + + n)2

is a 2nd order polynomial of p. To make it constant, set ( + )2 n = 0; = n 2( + ) = 0. Thus pm =

P + i Xi ++n

= =

n ; 2 n . 2

P n/2+ i Xi n+n

is the minimax estimator.

36705 Intermediate StatisticsChapter 10 Homework 6 Solutions

Asymptotic Evaluations
Problem 1 C & B 10.1
10.1 First calculate some moments for this distribution. EX = /3, E X 2 = 1/3, VarX = 1 2 . 3 9

So 3Xn is an unbiased estimator of with variance Var(3Xn ) = 9(VarX)/n = (3 2 )/n 0 as n . So by Theorem 10.1.3, 3Xn is a consistent estimator of . 10.3 a. The log likelihood is 1 n 36-705 Intermediate Statistics HW6 (xi Oct 22, 2010 )/. log (2) 2 2 root with the plus sign, as it has to be nonnegative.

Problem 2 C & B 10.2equal to zero, and a little algebra will show that the MLE is the root Dierentiate and set

Problem 1 (200.points) this equation are (1 of 2 + W = The roots of

1 + 4W )/2, and the MLE is the

By theorem secondlecture 4, of the log likelihood is (2 x2 + n)/(23 ), yielding an expected b. The 10 in Pderivative i P P Wn , an 1 = an Wn 1 = . Fisher information of P P bn 0 = an Wn + bn 1 + 0 = . 2
I() = E 2 2n + n Xi + n = , 3 2 22

Problem Theorem 10.1.12 the variancepoints each) and by 2 (20 points, 10 of the MLE is 1/I().
10.4 a. Write Problem likelihood is10.4 a The log 3 C & B

root of 2 + W = 0. The roots of 2this equation2 are 2 (1 1 + 4W )/2, and the 2 2 EXi i = 0, VarXi i = 2 (2 + ), EXi = + , VarXi = 2 2 (22 + 2 ), 1 MLE is the root with the plus sign, as it has to be nonnegative. and Cov(Xi , Xi i ) = 0. Applying the formulas Example 5.5.27, the asymptotic mean of b The second derivative of the log likelihood is (2 x2 + n)/(23 ), yielding an expeci and variance are tatied Fisher information of 2 2 Xi Yi ( + 2 2 2 Xi2Yi n n2n + n ) = Xi + = E E 2 I() 1 and Var = 2 +2 2 )]2 n(2 + 2 ) , Xi X2 [n( 23 i 2
b.

From normality and independence Dierentiate and set equal to zero, and a little algebra will show that the MLE is the

Xi Yi Xi (Xii ) + Xi i n 2 = 1 2 =1+ 2 . 2 Xlog(2) Xi (xi ) /. Xi i

So by Theorem plus sign, n isit has to be nonnegative. . root with the 10.1.3, 3X as a consistent estimator of 10.3 a.The second derivative of the log likelihood is (2 x2 + n)/(23 ), yielding an expected b. The log likelihood is i 1 n Fisher information of (xi )/. log (2) 2 22 2n 10.1 First calculateand setmoments zero,this2 littlei algebra= show that the MLE is the root some equal to = E distribution. will + n , for and a X + n Dierentiate I() 2 3 2are (1 1224W )/2, and the MLE is the of + W = 0. The roots of this equation + 2 root with the plus sign, EXit= /3, be nonnegative. VarX = 1 . as has to E X 2 = 1/3, and by Theorem 10.1.12 the variance of the MLE 1/I(). 3 3 9 is 2 b. The second derivative of the log likelihood is (2 xi + n)/(2 ), yielding an expected 10.4 a. Write information of Fisher So 3Xn is an unbiased estimator of with variance X (X + ) Xi Yi i i 2 i = 1 + Xi i . 2 = 2 2n + n 2 2 X i + n X Xi i X = I() i= E Var(3Xn ) = 9(VarX)/n = (3 2 )/n 0 ,as n . 3 2 22 From normality and independence So by by Theorem 10.1.12Xn is a consistent estimator of . and Theorem 10.1.3, 3 variance of the MLE is 1/I(). the 2 2 EXlog = 0, VarXi i = 2 (2 + 2 ), EXi = 2 + 2 , VarXi = 2 2 (22 + 2 ), i i likelihood is 10.4 a. Write 10.3 a. The XY Xi (Xi + ) Xi i i 2 i n log (2) i1 2 = 1 + )/.. (xi X 2 and Cov(Xi , Xi i ) = 0. Applying the formulas of=Example 5.5.27, the asymptotic mean Xi Xi 2 i Problem 3 C & B 10.4 2 and variance are From normality andset equal to zero, and a little algebra will show that the MLE is the root Dierentiate and independence equation and of 2 + W = 0. The roots of this Xi Yi are n 2 (2 +1+)4W )/2, 2 the MLE is the (1 2 Xi Yi Xi Yi2 2 2 2 2 EXi iE 0, VarXi 1 and 2 E 2 be EXi = 2 + 2 = + + 2 2 i as ( Var root with the plus sign,= it has to ),nonnegative. 2, VarXi== 2 2(2 2 ), 2 Xii2 Xi X [n( + 2 )]2 n( + ) 2 b.and Cov(X , X ) = 0. Applying the formulasisof Example+ n)/(23 ), yielding an expected The second derivative of the log likelihood (2 xi 5.5.27, the asymptotic mean i i i b. andFisher information of and variance are Yi =2+ 22 i 2n 2 n 2 2 ( i Xi XY n X + ) + n + I() i i 2 = EXi Xi 2 i 3 2 n 2 (2 + 2 ) 2, XV ar Y 2 == 2 2 i Yi E 2 1 and Var [n( 2 +2 )]2 2 n( 2 = ) 2 2 + Xi 2 )]2 n( + 2 ) 2X Xi with approximate mean and variance /(n ). [n( + i and by Theorem 10.1.12 the variance of the MLE is 1/I(). b. Yi i 10.4 a. Write X + + ) = (X Xi Yi Xi i i Xi Xi i i 2 = 2 =1+ 2 . Xi X Xi with approximate mean and variance 2 /(n2i ). 10-2 Solutions Manual for Statistical Inference From normality and independence c. 2 1 Y EXi i = 0, VarXi i = 2 (2 + 2i),= + i1 = 2+ 2 , VarXi = 2 2 (22 + 2 ), EX 2 i n Xi n Xi and Cov(Xi , Xi i )mean Applying the 2 /(n2 ). of Example 5.5.27, the asymptotic mean with approximate = 0. and variance formulas and variance are 2 10-2 Solutions 10.5 a. The integral of ETn is unbounded near zero. Manual for Statistical Inference We have 1 2 2 12 c. Xi Yi n ) 2 2 X2 1 (x) i Yi nn( + 1 /2 1 2 1 and e Var E ETn > 2 dx >Yi 2 K 21 2 2= = , 2 dx i 2 2 Problem 4, C &XB22 0 x2 10.18 Xi [n( + 0 )]x n( + ) = 2 + i

n Xi n Xi 2 ) as 2 (x), b. wheredensity of0x1 e(x)f/22 then variance 2 /(n2 ). Denote the K = max approximate mean and 1 with n(, Y b. If we delete the interval (, ), then iintegrand is bounded, that is, over the range of the = + i 2 10.5 a. The 2integral of ETn is unbounded near Xi 2 X (1 )f (x) + f (x) zero. We have Xi integration 1/x < 1/ . i 1 worksfor <20. Then 1 c. with approximatesimilar and variance 21 1 ). Assume > 0. A mean argument n 1 n 2 = 1) = and /(n (x)2 /2 2 and, Let Y Bernoulli(), then P (Y > P (Y = 0) = 1 dx > ETn e K dx = , 2 2 2 2 2 0 2 P ( < X < ) = P [ n( ) < n(Xx ) < n( )] < P [Z < n( 0 )], x V ar(Xi ) = E(V ar(Xi |Y )) + V ar(E(Xi |Y )) 2 2 where K = max0x1 e(x) /2 where Z n(0, 1). For < , the probability goes to 0 as n . b. If |Y delete is dierentiable ), 0) = 2 Note that, V ar(Xiwe that () 2 interval (,at = 0 the, integrand is bounded, that is, 10.7 We need to assume = 1) = the and V ar(Xi |Y =then, the true value of the parameter. Then over the range of integration 1/x2 < 1/ 10.1.12. we apply Theorem 5.5.24 to Theorem 2 . 2 2 > 0. A similar argument (1 b) for 10.9 We will doc. more general problemi that includes a) and ) special0. Then a Assume E(V ar(X |Y )) = + works as < cases. Suppose we want to estimate t e /t! = P (X = t). Let P ( < X < ) = P [ n( ) < n(X ) < n( )] < P [Z < n( )], 1 if X1 = t T = T (X1 , . . . , Xn ) = 2 0 if X1 = t. where Z n(0, 1). For < , the probability goes to 0 as n . 10.7 We need to assume that T is dierentiable at = 0 , the true value of the Then ET = P (T = 1) = P (X1 = t), so()is an unbiased estimator. Since i is a complete parameter. Then X sucient statistic forTheorem 5.5.24 UMVUE. The10.1.12. is 0 for y = Xi < t, and for we apply , E(T | Xi ) is to Theorem UMVUE y t, 10.9 We will do a more general problem that includes a) and b) as special cases. Suppose we want to estimate|y)t e /t! (X P= t| t). Let E(T = P = 1 (X = Xi = y) P (X1 = t, Xi = y) 1 if X1 = t T = T2 = (X , . . . , Xn ) = P ( Xi = y) 1 0 if X1 = t.

Also E(Xi |Y = 1) = , E(Xi |Y = 0) = and E(E(Xi |Y )) = + (1 ), V ar(E(Xi |Y )) = = (1 )( )2 By the fact that Xi s are iid, 1 V ar(X) = V ar( n (Xi )) =
i Y

(E(Xi |Y ) E(E(Xi |Y )))2 P (Y )

= ( (1 ))2 + ( (1 ))2 (1 )

1 2 ( + 2 (1 ) + (1 )( )2 ) n

Since the mean and variance of Cauchy distribution do not exist, any contaminate of cauchy distribution will make ( )2 and 2 innite. So V ar(Xi ) will be innite.

Problem 5, C & B 10.19

Xi n(, 2 ). a). 1 V ar( (X)) = V ar( n = = 1 ( n2 1 V ar( n2

Xi ) =
i

Xi )
i

V ar(Xi ) + 2
i i<j

Cov(Xi , Xj ))

n(n 1) 2 1 (n 2 + 2 ) 2 n 2 1 = ( 2 + (n 1) 2 ) n So, as n , V ar(X) b). 0.

1 V ar(X) = 2 V ar( n = = 1 ( n2

Xi )
i

V ar(Xi ) + 2
i n n i<j

Cov(Xi , Xj )) Cov(Xi , Xj ))

1 (n 2 + 2 n2

i=1 j=i+1

1 = 2 (n 2 + 2 n = c). We know 2 2

|ij| 2 )
i=1 j=i+1

1 n 1 2 + 2 (n ) n n 1 1 Cov(X1 , Xi ) V ar(X1 )V ar(Xi )

Corr(X1 , Xi ) =

And since i iid n(0, 1) we can use 1 for all i s, X2 = X1 + 1 X3 = (X1 + 1 ) + 1 ...
i2

Xi = i1 X1 +
j=0

j 1

So,
i2

Cov(X1 , Xi ) = Cov(X1 ,
i2

i1

X1 +
j=0

j 1 )

= i1 Cov(X1 , X1 ) +
j=1

Cov(X1 , j 1 )

= = Also,

i1

V ar(X1 )

i1 2

i2

V ar(Xi ) = 2(i1) V ar(X1 ) +

j=0

2j V ar(1 )

= 2(i1) 2 + = Given 2 =
1 , 12

1 1 2

1 2(i1) 1 2

Corr(X1 , Xi ) = i1

36-705 Intermediate Statistics Test 2 Solution

(1) Let X1 , . . . , Xn Bernoulli() where 0 < < 1. Let 1 Wn = n
n

i=1

Xi (1 Xi ).

(a) Show that there is a number such that Wn converges in probability . Solution: As Xi is either 0 or 1, so Xi (1 Xi ) = 0 with probability 1. Hence, Wn has point mass probability 1 at 0. So, E[Wn ] = 0, V ar(Xn ) = 0. Obviously, if we set = 0, we have P (|Wn | > ) = 0, for any > 0. Hence Wn converges to in probability. (b) Find the limiting distribution of n(Wn ). Solution: Wn has point mass 1 at 0, so n(Wn ) also has point mass 1 at 0. The limiting distribution is P ( n(Wn ) = 0) = 1.

(2) Let X1 , . . . , Xn Normal(, 1). (a) Let T = (X1 , . . . , Xn1 ). Show that T is not sucient. Solution: As T = (X1 , . . . , Xn1 ), the conditional distribution of (X1 , . . . , Xn |T = (t1 , . . . , tn1 )) is
n1 f (Xn ) i=1 f (ti ) f (X1 = t1 , . . . , Xn1 = tn1 , Xn ) f (X1 , . . . , Xn , T = t) = f (Xn ), = = n1 f (T ) f (X1 = t1 , . . . , Xn1 = tn1 ) i=1 f (ti )

where

(Xn )2 1 f (Xn ) = e 2 . 2

Obviously, the conditional pdf f (X1 , . . . , Xn |T = (t1 , . . . , tn1 )) depends on , so T is not sucient. (b) Show that U = n Xi is minimal sucient. i=1 Solution: From the solution above, the ratio between probability is f (x1 , . . . , xn |T ) = e f (y1 , . . . , yn |T )
Pn 2 2 Pn i=1 (xi yi ) + i=1 (xi yi ) 2

Obviously, when n xi = n yi , the ratio does not depend on , which means that i=1 i=1 T (X n ) = n xi is sucient. To make sure that the ratio does not depend on , there i=1 must be n xi = n yi , so T (X n ) = n xi is also minimal. i=1 i=1 i=1 In all, T (X n ) = n xi is minimal sucient statistic. i=1

(3) Let X1 , . . . , Xn be drawn from a uniform distribution on the set [0, 1] where > 0. (a) Find the method of moments estimator of . Solution: The probability density function for X is fX (x) = So, the expectation for X is
1

[2, 2 + ]

1 1+

0 x 1, 2 x 2 + , ow.

EX =
0

x dx + 1+

2+ 2

2 + 4 + 1 x dx = . 1+ 2(1 + )

To nd the moment estimator, let E[X] = and we get the solution 1 = X 2 + X 2 2X + 3, 2 = X 2 X 2 2X + 3. 1 n

n

Xi = X,
i=1

As > 0, only the 1 is kept, and the estimator is =X 2+ X 2 2X + 3.

(b) Find the mean squared error of . Solution: The form is too complicated, so we use Delta method to nd the approximate MSE of . In part (a), we have that 2 + 4 + 1 E[X] = . 2(1 + )

The variance for X can also be calculated, as V ar(X) = E[X 2 ] (E[X])2 = which ends up as ( + 2)3 7 (E[X])2 , 3(1 + )

4 + 43 + 182 + 28 + 1 V ar(X) = . 12(1 + )2

With CLT, we know that 1 2 + 4 + 1 1 4 + 43 + 182 + 28 + 1 , X N( ) n 2(1 + ) n 12(1 + )2 Let g(x) = x 2 + x2 2x + 3, so = g(X), with g( g( and the approximate MSE is M SE() = (E[( )])2 + V ar() 2( + 1)2 2 1 4 + 43 + 182 + 28 + 1 ) = ( (1 + )2 + 2 n 12(1 + )2 1 +1 4 + 43 + 182 + 28 + 1 = ( )2 n (1 + )2 + 2 3 (c) Show that is consistent. Solution: According to the result in part (b), the mean squared error of goes to 0 in in probability, which means that is consistent. the order of O(1/n), so 2 + 4 + 1 ) = , 2(1 + )

2 + 4 + 1 2( + 1)2 )= , 2(1 + ) (1 + )2 + 2

(4) Let X1 , . . . , Xn N (, 1). Let = e + 1. (a) Find the maximum likelihood estimator of and show that it is consistent. Solution: The likelihood function for is
n

L(; X n ) =
i=1

(xi )2 1 e 2 , 2

hence the log-likelihood function is n l(; X ) = log 2 2

n n

i=1

(xi )2 . 2

Take derivative of l(), we have l() = To nd the MLE of , let

l() n

i=1

xi n.

= 0, and the solution is 1 = n

n

xi .
i=1

As this is the only solution for the derivative function, so this is global maximum, and 1 MLE for is = n n xi . i=1 As MLE of function g() is function of MLE g(), so MLE for is = en
1

Pn

i=1

xi

+ 1.

Obviously, the distribution of is N (, 1/n), so in probability. As is according to continuous mapping theorem, g() = in a continuous function of , probability, which means that MLE is consistent. (b) Consider some loss function L(, ). Dene what it means for an estimator to be a minimax estimator for . Solution: We say is a minimax estimator of , if for any other estimator , there is sup R(, ) sup R(, ),

where R(, ) = E L(, ) = L(, (xn ))f (xn ; )dxn .

(c) Let be a prior for . Find the Bayes estimator for under the loss L(, ) = ( )2 / . Solution: To nd the Bayes estimator, for any xn , we want to choose (xn ) to minimize r(|xn ) =

L(, (xn ))(|xn )d.

Introduce Loss function L(, ) = n r(|x ) with respect to , we have r(|xn ) = Let
r(|xn )

( )2

in the equation, and take the derivative of 2( ) (|xn )d.

= 0, the equation is

2( ()) (|xn )d = 0,

which is equivalent with hence the solution is (xn ) = 1/

1 (|xn )d

(|xn )d = 0,

1 (|xn )d = 1/E[1/ |xn ].

2011 Fall 10705/36705 Homework 7 Solutions

10.31a.ByCLT,wehave p1 p1 , p1 1 p1 / n1 , p2 p2 , p2 1 p2 / n2 .Stackingthem
together,andconsideringthattheyareindependent,wehave

p1 d p 2
d

p1 p1 1 p1 / n1 , 0 p2

0 .UsingDeltasmethod,itiseasytoshowthat p2 1 p2 / n2

p1 p2 p1 p2 , p1 1 p1 / n1 p1 1 p1 / n2 .Under H 0 : p1 p2 p . p istheMLEofp,
thus p p .CombiningthesefactswithSlutzkeystheorem,weget
p

Therefore, T 12 .
d

7.Showthat,whenH0 istrue,thenthepvaluehasaUniform(0,1)distribution. Proof: First,accordingtoC&BTheorem2.1.10,thecdfofacontinuousr.v.followsUniform(0,1).

36-705 Intermediate Statistics HW8 Problem 1 (C & B 9.1)

Solution:

Problem 2 (C & B 9.4(a))

Solution: (a).

Problem 3 (C & B 9.33(a))

Solution:

Problem 4
Solution: The likelihood function for is L(; X1 , . . . , Xn ) = So, the MLE is = X(n) . The likelihood ratio for data is
n X(n) n IX(n) L() = n = n IX(n) . IX(n) L()

1 IX , n (n)

X(n) = max{X1 , . . . , Xn }.

Hence, using LRT, we accept the H0 when

n X(n) L() = n IX(n) > C. L()

Choose a proper C to make sure that the test has size . For Uniform distribution, the size could be calculated as P (
n X(n)

IX(n) C) = P (X(n) C 1/n ) = 2

(C 1/n )n = C. n

So, take C = to make sure that the LRT is with size . In this sense, the acceptance region is A() = {X1 , , Xn : X(n) > 1/n },
(n) and the corresponding 1 condence interval is (X(n) , 1/n ).

Problem 5
j 1 Solution: Given x, say that x Bj , then the estimator is p(x) = h = nh n I(Xi Bj ). i=1 For any Xi , the distribution of I(Xi Bj ) is Bernoulli Distribution with parameter p = P (X Bj ) = Bj p(t)dt. As X1 , , Xn are i.i.d samples, so I(X1 Bj ), , I(Xn Bj ) are also i.i.d samples. Hence, we have the expectation and variance for j as

E[j ] =
Bj

p(t)dt,

and var(j ) =

1 n

Bj

p(t)dt(1

p(t)dt).
Bj

Hence, the bias and variance for the estimator is bias((x)) = p and var((x)) = p So, the MSE for this single point is M SE(x) = b2 + v = ( 1 h p(t)dt p(x))2 + 1 nh2 p(t)dt(1 p(t)dt).
Bj

1 h

Bj

p(t)dt p(x),

1 nh2

Bj

p(t)dt(1

p(t)dt)
Bj

Bj

Bj

Now try to estimate the MSE term by term. Taylor expansion shows that p(t)dt = hp(x)+p (x)
Bj Bj

(tx)dt+
Bj

(t x)2 1 p ()dx = hp(x)+hp (x)(h(j )x)+O(h3 ). x 2 2

In the bin Bj , the integration over bias square is (

Bj

1 h

Bj

p(t)dt p(x))2 dx =

1 p (x)2 (h(j ) x)2 dx + O(h3 ), 2 Bj 3

and by the mean value theorem, (

Bj

1 h

Bj

p(t)dt p(x))2 dx p (j )2 x

h3 1 x (h(j ) x)2 dx = p (j )2 . 2 12 Bj

Hence, we have
1 m m Bj

bias(x) dx =
0 j=1

bias(x) dx

p (j ) x
j=1

3 2h

12

h2 p (xj ) dx . 12
2

For the variance part, in the bin Bj , it does not change, so the integration is vdt = h(
Bj

1 nh2

Bj

p(t)dt(1

p(t)dt)) =
Bj

1 ( nh

Bj

p(t)dt (

p(t)dt)2 ),
Bj

and on [0, 1] interval, the variance is

1 m

vdt =
0 j=1

1 ( nh

1 1 p(t)dt ( p(t)dt) ) = nh nh Bj Bj
2 Bj

(
j=1 Bj

p(t)dt)2 .

With mean value theorem, we have that

1

p(t)dt = p(j )h, so it becomes x 1 (1 nh p2 (x)dx).

vdt =
0

1 1 nh nh

j=1

h2 p(j )2 x

So, the approximation of MSE on the density function is M SE = If we take C1 = b2 + v p (xj )2 dx h2 1 + (1 12 nh p2 (x)dx).

p (xj )2 dx/12, and C2 = (1

p2 (x)dx), then the approximate MSE is 1 , nh

M SE C1 h2 + C2

so the best h should be O(n1/3 ), and the corresponding convergence rate is n2/3 .

2011Fall10705/36705Test3Solutions
(1) Let X 1 ,

, X n ~ Bernoulli ( ) , ( 0,1)

(a) FindMLE ,scorefunction,andFisherinformation.

Solution:

L ( ; X n ) = X i (1 )

n X i

1 n Xi n i =1 log L X i n = S ( ) = (1 ) 2 log L n I n ( ) = E = 2 (1 )

(b) Findthelimitingdistributionof = e .
Solution: AccordingtoThm11Lecture9:

n ( ) N

(()
()

| ' ( ) |2 2 0, = N ( 0, e (1 ) ) I1 ( )

2 N e , e (1 ) n d

(c) FindtheWaldtestfor H 0 : = 1/ 2, H1 : 1/ 2 . Solution:

(1 ) se = n

()

ThereforetheWaldtestis:rejectwhen

1/ 2 1/ 4n

> z /2 .

(2) Let X 1 ,

, X n ~ N ( ,1) .

(a) FindthelevelNeymanPearsontestfor H 0 : = 1, H1 : = 2 . Solution: TheNeymanPearsontestrejectswhen T X n > k .

( )

1 2 exp ( X i 2 ) L (1 ) 2 i = exp n X 3 / 2 = T ( xn ) = ( ) L ( 0 ) 1 2 exp ( X i 1) 2 i

Toget k

P0 (T ( x ) > k ) = P0 exp n ( X 3 / 2 ) > k = P0 ( X > c ) = .

Since X ~ N (1,1 / n ) ,weknow P0 X > 1 +

( (

z = .Thereforethetestis:rejectwhen n

X > 1+

z n

(b) InwhatsenseisNeymanPearsonoptimal: Solution: NeymanPearsonisoptimalbecauseitisUMP:amongalltheleveltests,NeymanPearson hasthelargestpowerfunctionforall 1 i.e.hasminimumtypeIIerror. (c) FindtheLRTof H 0 : = 1, H1 : 1 . Solution:

0 = 1,MLE = X
1 1 2 2 exp ( X i 0 ) exp ( X i 1) 2 i = 2 i = (Xn) = 2 2 1 1 L exp X i exp ( X i X ) 2 i 2 i 2 2 1 2 n = exp ( X i 1) ( X i X ) = exp ( X 1) 2 2 i L 0

( ) ()

WeknowunderH0 n X 1 ~ N ( 0,1) n X 1 H0when n X 1

~ 12 .ThereforetheLRTis:reject

2 > 1, (orequivalently n X 1 > z /2 ).

(3) Let X 1 ,

, X n ~ Uniform ( 0, ) , > 0

(a) FindthelikelihoodandMLE. Solution:

L=

I (X
i

) =

I X (n)

= X ( n ) = max i X i
(b) FindtheformofLRTfor H 0 : = 1, H1 : 1 Solution: TheLRTrejectsH0if X n c .

( )

(X

)=

L 0

( ) = I ( X ( ) 1) = X I X 1 ) ( ) ( ( ) 1 L ( ) I ( X( ) X( ) ) X
n n n n n ( n) n n

Therefore,if X ( n ) > 1 ,alwaysrejectH0.OtherwiserejectH0if X (nn ) issmallerthansome value. (c) Findtheformoflikelihoodratioconfidenceinterval. Solution:

L ( ) c C = : L L ( ) = L
n n

()

() 1 I ( X( ) )
1 X (nn )

X (nn )

I X (n)

When < X ( n ) ,thisratioisalwayszero.When X ( n ) ,thisratioismonotonically decreasingwith .Therefore,Cshouldhavetheform X ( n ) , U .

(4) Let X 1 ,

, X n ~ p ( x; )

(a) LetC(Xn)bea1confidenceintervalfor.Considertesting H 0 : = 0 , H1 : 0 . SupposewerejectH0if 0 C X n .ShowthatthisdefinesaleveltestforH0. Solution:

( )

inf P C ( X n ) 1

( ) sup 1 P ( C ( X ) ) sup P ( C ( X ) )
1 inf P C ( X n )

n

whichisthedefinitionofaleveltest. (b) X 1 , for. Solution:

, X n ~ Uniform ( 0, ) , > 0 .Let Cn = X ( n ) , X ( n ) / 1/ n .ShowthatCnisa1CI

For ,

P ( Cn ) = P X ( n ) X ( n ) / 1/ n = P 1/ n X ( n ) = 1 P 1/ n X ( n ) . = 1 P (

1/ n

X1 )

1 = 1 1/ n = 1

(c) Use(a)and(b)todefinealeveltestof H 0 : = 1, H1 : 1. Finditspowerfunction. Solution: Thetestis:rejectH0if 1 X ( n ) , X ( n ) / 1/ n X ( n ) > 1, or , X ( n ) < 1/ n .

 ( ) = P X ( n ) > 1 X ( n ) < 1/ n = P X ( n ) > 1 + P X ( n ) < 1/ n

= 1 P X ( n ) < 1 + P X ( n ) < 1/ n
n

)
)
n

= 1 ( P ( X 1 < 1) ) + P ( X 1 < 1/ n ) 1 1/ n = 1/ n < 1 n 1 1< 1 + n

36705 Intermediate Statistics Homework 9 Solutions

Problem 1
a). R(h) = E(L(h)) = E (h (x))2 dx 2E p ph (x)p(x)dx + (p(x))2 dx

Since the last term of the rhs has nothing to do with h, dierentiate R(h) with respective to h, d (p(x))2 /dh = 0. Then,

min R(h) = min E(L(h)) = min E

h h h

(h (x))2 dx 2E p

ph (x)p(x)dx

= min R(h)
h

b). ER(h) = E = EX = EX 2 (h (x)) dx p n

2 n

E(h (Yi )) p
i=1

(h (x))2 dx 2EX,Y (h (Y1 )) p p (h (x))2 dx 2EX p (h (y)p(y)dy) p

(1) (2)

The second expectation in (1) is with respect both Xs and Ys, while the second expec tation in (2) is with respect to Xs. So with prove that ER(h) = R(h) = E (h (x))2 dx p 2E ph (x)p(x)dx.

Problem 2
Kernel density estimator is dened as 1 ph (x) = n
n i=1

1 K h 1

Xi x h

1 Let Yi = h K

Xi x h

. Since kernel K is a symmetric density with expectation 0. Then

x x a < Yi < b with a = min(K( Xih ))/h and b = max(K( Xih ))/h. We can apply Hoeding inequality that
2n 2

P (|h (x) ph (x)| > ) = P (| p

1 n

i=1

Yi E(Y )| > ) 2e (ba)2

Problem 3
E( |X1 , ..., Xn ) = n1 E[Y Binomial(n, X)] = n1 nX = X E( ) = E(E( |X1 , ..., Xn )) = EX = Var( |X1 , ..., Xn ) = n2 nX(1 X) = n1 X(1 X) Var( ) = Var(E( |X1 , ..., Xn )) + E(Var( |X1 , ..., Xn )) = = n1 (1 ) + n1 (EX EX ) = = n1 2(1 ) n2 (1 ) = 2n 1 = (1 ) n2 = Var(X) + E(n1 X(1 X)) =
2

= n1 ((1 ) + n1 (1 ) 2 ) =

Problem 4
V( |X1 , . . . , Xn ) X(1 X) = p(1 p) Var(n ) P X p by law of large number Then, P X(1 X) p(1 p) by Theorem 10 in Lecture 4 So X(1 X) P 1 p(1 p) 2

36-705 Intermediate Statistics HW10 Problem 1 (C & B 7.23)

Solution:

Problem 2 (C & B 9.27)

Solution:

Problem 3
Solution: (a) Say that Xj = Zj + j , then Xj N (j , 1). The moment generating function for 2 Xj N (, 1) is MX (t) = et+t /2 . So, we have the moments for X as E[X] = , and V ar(X 2 ) = E[X 4 ] (E[X 2 ])2 = 3 + 122 + 4 (1 + 2 )2 = 2 + 42 . Because Xj s are independent, so we have
k k 2 E[Xj ] j=1 k

E[X 2 ] = 1 + 2 ,

E[X 3 ] = 3 + 3 ,

E[X 4 ] = 3 + 62 + 4 ,

E[V ] =

=
j=1 k

2 1 + j = k + ,

V ar[V ] =
j=1

2 ar[Xj ]

=
j=1

2 2 + 4j = 2(k + 2).

(b) The posterior distribution of is

(yi i )2 1 f () = f (y|)f () = n e 2 . i=1 2

So, the posterior distribution of is N (y, In ). 2

2 (c) In this case, the distribution of = i 2 is 2 (), where = i yi . i n 2 (d) According to the result in (a), the posterior mean of is = n + i yi . 2 2 2 (e) Let W = i yi and thus = n + W . By denition, W n ( i i ). Therefore

bias() = E() = n + (n +

2 = E(n + W ) i 2 ) i
i

2 i
i

2 i
i

= 2n V ar() = V ar(n + W ) = V ar(W ) = 2n + 4

i

2 i

(f) According to the hint, W N (E(W ), V ar(W )) = N (n + i 2 , 2n + 4 i 2 ). i i Now consider the probability P (| | > ) for an arbitrarily small . This probability will never approach 1 since = W + n i 2 N (2n, 2n + 4 i 2 ). In other words, the i i density of will never concentrate around zero. Therefore, is not consistent. 2 2 (g) From (c), = i i is n (), then the 1 condence interval for is Cn = 2 [2 ( i yi ), +). n, (h) From (e), bias() = 2n, then E( 2n) = 0. = 2n = W n is an unbiased 2 0, so is not consistent either. estimator. V ar() = 2n + 4 i i . As n , V ar() 2 2 2 (i) From (e) we have W n ( i i ) = n ( ). Suppose we want to test H0 : = 0 vs. H1 : = 0 , then the rejection region of a level test is R = {W : W 2 (0 )}. n, By inverting this test, we have a size 1 condence interval An = { : W 2 ( )}. n, The interval in (g) is actually Bayesian credible interval where the parameter is random and the interval is determined by the posterior distribution of . The interval in (i) is the frequentist condence interval which we assume it is xed and the interval is determined from the distribution of the estimator of .

Practice Final Exam 1. Let X1 , . . . , Xn be iid from a distribution with mean and variance 2 . Let
2 Sn

1 = n1

i=1 P

(Xi X n )2

where X n = n1

n i=1

Xi . Prove that Sn .

2. Let > 0. Let S denote the square in the plane whose four corners are (, ), (, ), (, ) and (, ). Let X1 , . . . , Xn be iid data from a uniform distribution over S . (Note that each Xi R2 .)

( , )

( , )

( , )

( , )

(a) Find a minimal sucient statistic. (b) Find the maximum likelihood estimate (mle). (c) Show that the mle is consistent.

3. Let X1 , . . . , Xn Poisson() and let Y1 , . . . , Ym Poisson(). Assume that the two samples are independent.

(a) Find the Wald test for testing H0 : = versus H1 : = .

(b) Find the likelihood ratio test for testing H0 : = versus H1 : = .

What is the (approximate) level critical value? (d) Find the BIC criterion for deciding between the two models: Model I: = . Model II: = . 4. Let X1 , . . . , Xn Unif(0, ). (a) Let = aX n where a > 0 is a constant. Find the risk of under squared error loss. (b) Find the posterior mean using the (improper) prior () 1/. (c) Suppose now that 0 B where B > 0 is given. Hence the parameter space is = [0, B]. Let be the Bayes estimator (assuming squared error loss) assuming that the prior puts all its mass at = 0. In other words, the prior is a point mass at = 0. Prove that the posterior mean is not minimax. (Hint: You need only nd some other estimator such that sup R(, ) < sup R(, ). (c) Find an approximate 1 condence interval for .

5. Suppose that (Y, X) are random variables where Y {0, 1} and X R. Suppose that X|Y = 0 Unif(5, 5) and that Further suppose that P(Y = 0) = P(Y = 1) = 1/2. (a) Find m(x) = P(Y = 1|X = x). (b) Let A = {(a, b) : a, b R , a b}. Find the VC dimension of A. 2 X|Y = 1 Unif(1, 1).

(c) Let H = {hA : A A} where hA (x) = 1 if x A and hA (x) = 0 if x A. Show / that the Bayes rule h is in H. (d) Let h be the empirical risk minimizer based on data (X1 , Y1 ), . . . , (Xn , Yn ). Show that R(h) R(h ) with high probability.

6. Let X1 , X2 be iid Uniform(0, 1). Find the density of Y = X1 + X2 . 7. Let X1 , . . . , Xn be iid data from a uniform distribution over the disc of radius in R2 . Thus, Xi R2 and 1 if ||x|| 2 f (x; ) = 0 otherwise where ||x|| = x2 + x2 . 2 1

(a) Find a minimal sucient statistic. (b) Find the maximum likelihood estimate (mle). (c) Show that the mle is consistent. 8. Let X Binomial(n, p) and Y Binomial(m, q). Assume that X and Y are independent.

(a) Find the Wald test for testing H0 : p = q versus H1 : p = q.

(b) Find the likelihood ratio test for testing H0 : p = q versus H1 : p = q.

(c) Find an approximate 1 condence interval for = p q.

9. Let X f (x; ) where . Let L(, ) be a loss fuctions. (a) Dene the following terms: risk function, minimax estimator, Bayes estimator. (b) Show that a Bayes estimator with constant risk is minimax.

10. Let X1 , . . . , Xn N (, 1). Let be a N (0, 1) prior: 1 2 () = e /2 . 2 (a) Find the posterior distribution for . (b) Find the posterior mean . (c) Find the mean squared error R(, ) = E ( )2 . 11. Let X1 , . . . , Xn Poisson(). (a) Find the mle . (b) Find the score function. (c) Find the Fisher informtion. (d) Find the limiting distribution of the mle. (e) Show that is consistent. (f) Let = e . Find the limiting distribution of = e .
b

(g) Show that is a consistent estimate of . 12. Let X1 , . . . , Xn be a sample from f (x; ) = (1/2)(1 + x) where 1 < x < 1 and 1 < < 1. (a) Find the mle . Show that it is consistent. (b) Find the method of moments estimator and show that it is consistent.