0 Conditional Distribution For jointly continuous random variables x and Y, Sex vis)=[fnx(els)de vx such that f(x) >0 4.4. Conditional Expectation We can also ask what the expected behaviour of one random variable is, given knowledge of the value of a second random variable and this gives tise to the idea of conditional expectation. Definition 4.10 (Conditional Expectation): The conditional expectation in discrete and continuous cases corresponds to an expectation with respect 10 the appropriate conditional probability distribution: Diserete E[Y|X=s]= DvP (Y= 91K =x) Continuous ELL = 2]=[° ohne (V1 = 2), Note that before x is known to take the value x, E[}’| X] is itself a random variable being a function of the random variable x . We might be interested in the distribution of the random variable E[Y| x). Theorem 4.1 (Tower Property of Conditional Expectation): For any two random variables X, and X, sR Nae LL Nw DIOLS, Ps 0 RHE ak PATRI wee wn aoucsdet com pee 20 (a marran ms Scanned with CamScanneree ELEN 1 42))=EL%] Exercise 44.1: Suppose that © ~ Ufo, 1} and (x |@)~8in(2, @) Find B[4|@] and hence or otherwise show that E[X]=1 4.5. Conditional Expectations of Functions of Random Variables By extending the theorem on marginal expectations we can relate the conditional and marginal expectations of functions of random variables (in particular, their variances). Theorem 4.2: (Marginal Expectation of a ‘Transformed Random Variables): For any random variables X; and X;, and for any function 40 {EL A(%)) X2]]= ELA) Theorem 4.3 (Marginal Variance): For any random variables X, and x, Var( X,) = [ Var(%; |2)]}+Var( M1 4%2]) 46, Independence of Random Variables Whilst the previous sections have been conceméd with the information that ‘one random variable carries about another, it would seem that there must be pairs of random variables which each provide no information whatsoever about the other. It is, for example, difficult to imagine that the value obtain when a die is rolled in Coventry will tell us much about the outcome of a coin toss taking place at the same time in Lancaster. There are two equivalent statements of a property termed stochastic independence which capture precisely this idea The following two definitions are equivalent for both discrete and continuous random variables Definition 4.11: (Stochastic Independence): Definition 1 Random variables X,,X2,...X, are stochastically independent iff nals) “FTF 0) Definition 2: Random variables. ),,,....%, are stochastically. independent iff foe s)=T1 f(s) If X; and X; are independent then their conditional densities are equal to their marginal densities 4.7. Covariance and Correlation Having established that sometimes one random variable does convey information about another and in other cases knowing the value of a random variable tells us nothing useful about another random variable itis useful to have mechanisms for characterising the relationship between pairs (or larger ‘groups) of random variables Definition 4.12 (Covariance and Correlation): Covariance: For random variables ’ and ¥ defined on the same probability space THAT, Ft Paria Sara Haat Khar rau lafaaipracanvcom: Webate: wor istndctesom Scanned with CamScanner Joint and Conditional Distributions Re Dab 11016, Ph (268759, Cas MTR vam ee Put Your Own Notes48. Ty Coo. Y] FLX He (Hr) EL] -ney Correlation: For random variables x probability space Cov[.x,¥] and Y defined on the same CovfX,7] ola] Wvar[x]Ver[?] provided that ¢y >0 and oy >0. ): Let X and Y have finite Theorem 4A (Cauchy-Schwart Tneqt second moments. Then f°] 2X]=1 for some constant ¢ e[xy]) =|e[x yp se[ x ( elevy’ sel, with equality if and only if PLY ‘Transformation of Random Variables: (x) Theorem 4.5 (Distribution of a Function of a Random Variable be a random variable and Y= g(X) where g is injective (ic. it maps at ‘most one x to any value y). Then ) de dy fel») = fe(27"09)} given that (2"'(y)) exists and (g°'(y)) >0 vy or (g"(y)) <0 vy. If g is not bijective (one-to-one) there may be values of for which there exists no x such that y= g(x). Such points clearly have density zero, When the conditions of this theorem are not satisfied it is necessary to be a little more careful. The most general approach for finding the density of a transformed random variable is to explicitly construct the distribution function of the transformed random variable and then to use the standard approach to turn the distribution function into a density (this approach is discussed in Larry Wasserstein’s “All of Statistics”) Exercise 4.8.1: Let X be distributed exponentially with parameter 0, that ee x20 Il eco Find the density function of ne a ee ae 0 Gi) y=x"%, poo 0. for x<0 (ii) ¥=e(X) with g(X)=|x for Osxst 1 forx>t ( Deli 0016, Ps (12657527, Cal 99918) 59117, RROD Scanned with CamScanner2> ips ccademy Theorem 4.6 (Probability Integral Transformation): If X is a random UT variable with continuous Fy (x), then U = Fy (X) is uniformly distributed Pat Your Own Notes over the interval (0, 1). —— Conversely if U is uniform over (0,1), then X =F! (U) has distribution funetion Fy 49. Moment-Generating-Function Technique ‘The following technique is but one example of a situation in which the ‘moment generating function proves invaluable. Function of Vail. For Y= g() compute iy-efe"] -efee()] m(pe ele" }=e[ 4] If the result is the MGF of a known distribution then it will follow that ¥ hhas that distribution, Sums of Independent random variables. For ¥ =F, Xj, where the X; are independent random variables for which the MGE exists V—A<10 reef "Ja Ten (fr her ‘Thus [],my, (0) may be used to identify the distribution of ¥ as above Scanned with CamScannerCHAPTER 5 FERENCE 5.1. Sample Statisties Suppose we select a sample of size n from a population of size. For each i in {1,..n}, let X; be a random variable denoting the outcome of the i observation of a variable of interest. For example, X; might be the height of the i person sampled. Under the assumptions of simple random sampling, the X, are independent and identically distributed (i). Therefore, if the distribution of a single unit sampled from, the population can be characterized by a distribution with density function. f, the marginal density function of each X, is also f and their joint density function g is a simple product of their marginal densities =H) F(¥2)-S (rn) BOM Qnty In order to make inferences about a population parameter, we use sample data to form an estimate of the population parameter. We calculate our estimate using an estimator or sample statistic, which is a function of the X, We have already seen examples of sample statistics, for example the sample mean where » is the size of the sample is an estimator of the population mean, eg, fordiserete ur SaP[e= where NV is the number of tinct values which it is possible for an X; to take, 5.2. Sampling Distributions Since an estimator 6 is a function of random variables, it follows that is itself a random variable and possesses its own distribution. The probability | distribution of an estimator itself is called a sampling distribution, Proposition 5.1 (Distribution of The Sample Mean): Let ¥ denote the sample mean of a random sample of size n from a normal distribution with ‘mean 1 and variance o?. Then rf) Scanned with CamScanner- Theorem 5.1 (Central Limit Theorem): Let f be a density function with 2 mean and finite variance a? Let X be the sample mean of a random sample of size n from f and let ] J] x-e[7]_¥. Fale vn ‘Then the distribution of Z, approaches the standard normal distribution as nv. This is often written as: Z,—“+N(0,1) with “> denoting convergence in distribution. Thus, ifthe sample size is “large enough”, the sample mean can be assumed to follow a normal distribution regardless of the population distribution. In practice, this assumption is often taken to be valid for a sample size n>30 The Chi-Squared Distribution ‘The chi-squared distribution is a special case of the gamma distribution. The smple variance Ss-3p 1 mI of a standard normal distribution is 7 with n-1 degrees of freedom. Definition 5.1: If X is a random variable with density r(f SF x)= 14 (8 (3) ° otherwise then X is defined to have a y? distribution with & degrees of freedom (1?) where k is a positive integer. ‘Thus the 12 density is a gamma density with r=f Result: Ifthe RVs X,, =... are independently normally distributed with ‘means 41, and variances o? then 2) has a z2 distribution. Theorem 5.2: If X,,.Xq is a random sample from @ normal distribution with mean js and variance o? then (i) ¥ and 4,(x,-X)° are independent, 7 CA SENDA HORTON, SHAT Ba Scanned with CamScannerCorollary 5.1: If s? = _Y'(x, isthe sample variance of a random sample of size n from a normal distribution with mean 1 and variance a? , then The ¢ distribution is closely related to the normal distribution and is needed ! for making inferences about the mean of a normal distribution when the variance is also unknown, Definition 5.2: If Z ~N(0, 1), U~z3 and Z and U are independent of one i other, then z Ve where 1 denotes a f distribution with & degrees of freedom, The density of the ¢ distribution is: ‘The MGF for the ¢ distribution does not exist, but E[x]=0 as the distribution is symmetric (actually the expectation only exists when > 1 although an extension known as the Cauchy principle value can be defined more generally), and it can be shown that var[x]=—4., for k>2 kd Theorem 5.3: If X~ then | L() 2 we ¥ as kro, That is, as k+0 the density approaches the density of a standard normal. The F Distribution The F Distribution is useful for making inferences about the ratio of two ‘unknown variances. ‘HAL, rat Flor Sa Sara Hau Khas, Neo LT New DATI0OK, Ps (11) 27ST, Ca PTANGN ADDICT, SHAT ‘rma nftadarnndea com: Web: ww easton om Scanned with CamScannerSiayse U and ¥ ate independently itu wih SERRE ‘Put Your Own. Definition 5. U~22, and y~ 33 . Then the random variable Notes xe U ft is distributed sccotng to an F distbution with m and n depress of freedom. The density of X is given by (net) S()=Faa()= HG sae rf rf [mya aay fies ; U7) Returning to the case of two independent random samples_froity normal Populations of common variance, but differing means River y, ~M(Hie") Foostay ~M(H2s9?) 5.3. Point Estimation ‘The sample mean and variance are examples of point estimators, because the estimates they produce are single point values, rather than a range of values. For a given parameter there are an infinite number of possible estimators, hhence the question arises: what makes a “good” estimator? Definition 5.4 (Unbiasedness): Let X be a random variable with pdf £(x:0), where O«MER? is some unknown parameter, p21. Let Xjond’y be a random sample from the distribution of and let 6 denote a statistic. 6 is an unbiased estimator of 6 if F[i]-0 voen where the expectation is with respect to (x; 8). If 6 is not unbiased, we say that 6 is a biased estimator of 0, with sies(3)-[8]-0 IF 6is(@) +0 when n>. then we say that 6 is asymptotically unbiased, Example 8.1: Consider the following estimator of the population variance nus Khas New LT, New DoB-1 0016, Pa (01) 255757, Ct 99915504 & SPIES, SHONTID Ea Scanned with CamScannerTy We find that As this decays to zero as | 1 itis an asymptotically unbiased estimator, However, we can see that | the sample variance 2 DSK 3] So this estimator is biased with bias equal to Consistency: In order to define a consistent estimator, we first define convergence in probability Definition 5.5 (Convergence in Probabi ): Let {X,} be a sequence of random variables and let be a random variable. We say that x, converges in probability © X if Ve>0 lim P[|X,—X|>€]=0, or equivalently tim P[]x—X]8(a) Consistency according to the definition above may be hard to prove, but it tums out that a sufficient (though not necessary) condition for consistency is that sias(8) +0 and Var(@)-+0 as n> Definition 5.7 (Consistency in Mean-Squared Error): If 6 is an estimator of 6, then the mean squared error of 6 is defined as )-#{(0-0)'] and 6 is said to be consistent in MSE if Mse(@) +0, as the size of the wst( sample on which 6 is based increases to infinity. (8) =Vae(@)+[ias(6))° Interval Estimation This section describes confidence intervals which are intervals constructed such that they contain @ with some level of confidence. Definition 5.8 (Confidence Interval}: Let Xj; be a random sample from a distribution with pdf (x; ) where @ i8 ait unknown parameter in the parameter space ©. If Land U arestatistes such that Pl Dai6016 Fh (1) 26575, CalDOPIANOM& DOPLETON, SRIBTID Scanned with CamScannerHence P(-1.96Zq a8 mr, In practice, if n2120, the quantiles of the standard normal can be used instead (they are essentially indistinguishable from those of the ¢ distribution in this regime). Bxercise 5.4.3: Suppose we observe the following data on lactic. acid concentrations in cheese 086 153 157 181 099 109 129° 1.78 ~ 1.29 1.58 Assuming that these data are a random sample from a-normal distribution, calculate a 90% Cl for the mean of this distribution. CI for Differences in Means of Normal Populations with Equal Variance: Suppose we have two independent random samples Kieoky =Mb49") Tt, Met) In Section 5.2 we saw that ~tayam-d> Where 3 isthe pooled variance 2 (=U)sh +(ns -I)F ‘Which can be shown to be an unbiased estimator of o* ‘Therefore the exact 100(1-a)% CI for 4, —1, in this case is In the case of unequal variances, an exact Cl cannot be derived. Exercise 5.4.4: Consider a study to compare the nicotine contents of two brands of cigarettes. 10 cigarettes of Brand A had an average nicotine content of 3.1 milligrams with a standard error of 0.5 mg, while 8 cigarettes of Brand B had an average nicotine content of 2.7 mg with a standard error of 0.7 mg, Assuming that the 2 data sets are independent random samples from normal populations with equal variances, construct a 95% confidence interval for the difference between the mean nicotine contents of the two brands. Confidence Interval for the Variance of a Normal Population: Suppose Xiu, are iid normal with unknown 41 and o?. From the Corollary of ‘Theorem 5.3 we know that (n=)? TBA, et Flor Sa ner Khas, Near LIT New DAWFTOOG, Pe Scanned with CamScannerBaipsccademy oo ——= He aus Gun Notes le Where P(0< x22, Hence an exact 100(1-«)% CI for o? is given by (eu sv) (aaianet “Maree Note that the 72 distribution is asymmetric, hence this Cl is asymmetric, unlike the previous Cls we have defined, This illustrates an important point: any interval which has the appropriate coverage probability can be used as a confidence interval and there is no unique (1~a) confidence interval for a statistic (imagine shifting the left endpoint of the interval slightly towards the cent; the right endpoint could then be moved a small amount away from the centre to compensate). Symmetry is one of the properties which is often considered desirable when dealing with confidence intervals, but it’s certainly not essential CI for the Ratio of Variance of Normal Populations: Suppose we have ‘wo independent random samples Kok, —M(sic8) Hoth, ~Mae8) Where pyn6; and oy are unknown. Lt o-let In Section 5.2 we found that Thus we have PF cim-tt-e2 <2 < Fajita) P|) 0c as Fahne ame a from which we derive the following exact Cl for 2 : . Note this Cl is also asymmetric. Example 5.2: Consider the study described in Exercise 5.4.4. A 95% Cl for cee pletict a (8s a 85 asa) (25 anthem) -( 122,246) Which includes 1, so the assumption of equal variances seems reasonable in TA, eat Fes) Soa vas Ks, Near LIT, New Dai-I0016 Ph 11)2689757, Cle SESE POTTING SRIETOD emai noddatcademrcom: Webte wwpacaden.com Scanned with CamScannerCHAPTER 6 MAXIMUM LIKELIHOOD ESTIMATION 6.1. Likelihood Function and ML Estimator Suppose we have a simple random sample X;,...X, from a density /(x; 0) parameterized by some possibly unknown parameter 6. The joint pdf of the entire data set is £(x58) with x =(x),0.4%)" The likelihood function £(0; x) is this joint pdf viewed as a function of 0 with x fixed to the observed data. It is often more convenient to work with the log-likelihood 1(8; x)=log[£(0; x)] Which in the simple random sampling case above yields: 1(8; x) = Yilow f(x: 0) ‘The maximum likelihood estimator (MLE) is the value 0 which maximises 1(0; x). The MLE also maximises 1(0; x) because log(.) is monotonic function. Usually it is easier to maximize 1(0; x) , so we work with this Example 6.1; Suppose X,,...%, is a random sample from the exponential distribution with pdf foe x>0, 0 otherwise F(x) <0, s0 8 does comeapord to a maximum, Exercise 6.2: Suppose Xj...4', form a random sample from the Poisson distribution Poisson (8). Derive the ML estimator for 0 BATT, est Fw Ta aus Khas, Near LT, New Dei 11016 Pu (1) 265757, Cals 999164 & SOPLSLTIN ASHBLAT Scanned with CamScanner1 6.4. Properties of MLEs ‘The maximum likelihood estimator has a number of useful properties. Theorem 6.1: Suppose @ and 4 represent two alternative parameterizations and that 4 is @ one-to-one function of O (more formally, there exists a bijective mapping, g, from 9 to4), so we can write $=8(0), 0=A(6) for appropriate g and h= g"! Then if @ is the MLE of 6, then the MLE of 4 is ¢(3). Corollary 6.1 (Invariance of MLE): Let 6,,...0, be a MLE for 6,,..,0, If 7(9)=(7,(0),--.7;(8)) isa sufficiently regular transformation of the PARAMETER space @ , then 7(0)=(7 (0). (8)) isa MLE of 7(0) Lemma 6.1: Suppose there exists an unbiased estimator, 6, which attains the Cramer-Rao bound. Suppose that 6, the MLE. isa solution to a [HA rt Pty Sa Sarl Haus Ks Near LT, New Doi 11016 Ps (UD) 265007, Cah PRIN A DDSTONTG RTD ‘ui niece cams Weber stead am 4 Scanned with CamScanner1 CHAPTER 7 HYPOTHESIS TESTING Introduction | A hypothesis is a falsifiable claim about the real world; a statement which if true will explain some observable quantity. Statisticians will be interested in hypotheses like (i) “The probabilities of a male panda or a female panda being born are equal’, (ii) “The number of flying bombs falling on a given area of London during World War II follows a Poisson distribution”, (ii) “The mean systolic blood pressure of 35-year-old men is no higher than that of 40-year-old women”, (iv) “The mean value of ¥ = log (systolic blood pressure) is independent of X= age" (ie. B[Y[X =x] = constant) These hypotheses can be translated into statements about parameters within | a probability model: @ aap. (i) + (within the general probability model p, >0. Yn ~Poi(n) for some 220", ie. : py =P(N=n) =D" exp(-2)/n! beni Pa =1) | ii) *0, 50," and iv)“, -0° (assuming the linear model BY |x]=y +B") Definition 7.1 (Hypothesis Test): A hypothesis test is a procedure for deciding whether to accept a particular hypothesis as a reasonable simplifying assumption, or (o reject it as unreasonable in the light of the data, Definition 7.2 (Null Hypothesis assumption we are considering making. The null hypothesis Ho is the default Definition 7.3 (Alternative Hypothesis): The alternative hypothesis #, is the alternative explanation(s) we are considering for the data. Ordinarily, the null hypothesis is what we would assume fo be true in the absence of data which suggests that itis not. Ordinarily, Hy will explain the data in at least as simple a way as 1, Definition 7.4 (Type I error): A type I error is made if Hy is rejected when +, is true. In some situations this type of error is known as a false positive. Definition 7.5 (Type I error): A type II error is made if Hy is accepted when Hg is false. This may also be termed a false negative Scanned with CamScannerExample 7.1: In the first example above (pandas) the null hypothesis is Ho: i= Pa ‘The alternative hypothesis in the first example would usually be Hy: p,# Pz, though it could also be (for example) @ Hin py, oF Gil) Aye -P2 each of these alternative hypotheses makes a slightly different statement about the collection of situations which we believe are possible. A statistician needs to decide which type of hypothesis test is appropriate in any real situation, for some specified 6 + 0. 7.2. Simple Hypothesis Tests The simplest type of hypothesis testing occurs*when the probability distribution giving rise to the data is specified completely under the null and altemnative hypotheses, ~ Definition 7.6 (Simple Hypotheses): A simple hypothesis is of the form H,:0=0,, where is the parameter vector, which parameterizes the probabilistic model for the data. A simple hypottiesis specifies the precise value of the parameter vector (ice. the probability distribution of the data is specified completely) Definition 7.7 (Composite Hypotheses): A: composite hypothesis is of the form #1, :0€0,, ic. the parameter O,lies-in a specified subset 2 of the parameter space. This type of hypothesis specifies an entire collection of values for the parameter vector and so specifies a class of probabilistic models from which the data may have arisen. Definition 7.8 (Simple Hypothesis Test): A simple hypothesis test tests a simple null hypothesis 7, :0=0, against a simple alternative #,:0= 0, , where @ parameterizes the distribution of our experimental random variables X= X).2p..u0Xy Although simple hypothesis tests seem appealing, there are many situations in which a statistician cannot reduce the problem at hand to a clear dichotomy between two fully-specified models for the data-generating process There may be many seemingly sensible approaches to testing a given hypothesis. A reasonable criterion for choosing between them is to attempt to minimize the chance of making a mistake: incorrectly rejecting a true null hypothesis, or incorrectly accepting a false null hypothesis, Definition 7.9 (Size): A test of size a i hypothesis Hy fone which rejects the mull 1p in favour of the alternative #, :0=6, ifand if XeC, Where P(XeC, |@=0))=0 for some subset C,, of the sample space S of X. The size, a, of a testis the probability of rejecting 7, when Hy is in fact true; ie. size is the probability of type Terror if Mg is true. We want a to be small (a.=0.05, say. AM, at oe) Sia oes, Near LL New Dai, Pus (11 AES7S2, Cok DOPOTIDEM & BOVIS BSBA [27 | Scanned with CamScannerDefinition 7.10 (Critical Region): The set C,, in Definition 7.9 is called the critical region or rejection region of the test. Definition 7.11 (Power & Power Functio with critical region C, is the function B(0)=P(XeC, |), and the power of a simple test is P=B(0), ie. the probability that we reject Mo in favour of H, when 1 is true. Thus a simple test of power B has probability 1-p of a type I error occurring when H, is true, Clearly for a fixed size a of test, the larger the power i of atest the better. : The power function of a test However, there is an inevitable trade-off between small size and high power (as in a jury trial: the more careful one is not to convict an innocent defendant, the more likely one is to free a guilty one by mistake), A hypothesis test typically uses a test statistic 7(X), whose distribution is known under Hy, and such that extreme values of 7(X) are more compatible with #, that Hy Many useful hypothesis tests have the following form: Definition 7.12 (Simple Likelihood Ratio Test): A simple likelihood ratio test (SLRT) of Hy :0=0) against #7,:0=0, rejects Hy iff | [eles cz {xO < 4} Koco tas) <} Where £(0:.) isthe likelihood of@ given the data x, andthe number 4, is chosen so that the size of the test is a. ‘nN (0,1). Show that the =0 against H, :0=1 can be written, Exercise 7.2.1: Suppose that Xj, likelihood ratio for testing Hy 2(a)=e0[ oz a} Hence show that the corresponding SLRT of size a rejects Hy when the test statistic T(X)=* satisfies > @"'(1-a)/V2. ‘A numberof points should be bome in mind + For a simple hypothesis test, both Hy and #1, are ‘point hypotheses’, each specifying a particular value for the parameter 0 rather than a region of the parameter space. ‘+ Inpractice, no hypothesis will be precisely true, so the whole foundation of ‘classical hypothesis testing seems suspect! Actually, there is @ tendency to overuse hypothesis testing: it is appropriate only when one really does wish to compare competing, well-defined hypotheses. In many problems the use of point estimation with an appropriate confidence interval is much easier to justify + Regarding likelihood as a measure of compatibility between data and model, an SLRT compares the compatibility of 6 and 0, with the observed data x, and accepts Hp iff the ratio is sufficiently large. + One reason for the importance of likelihood ratio tests is the following theorem, which shows that out of all tests of a given size, an SLRT (if one exists) is “best" in a certain sense. DAH, Fst Po) ia Sar Ha Scanned with CamScanner2> ips academy Element Theorem 7.1 (The Neyman-Pearson Lemma): Given random variables Xi Aan Xy» With joint density f(x|0), the simple likelihood ratio test of a fixed size a for testing Hy :0= 0p against H,:0=0, is at least as powerful as any other test of the same size. Proof: Let 4 be a positive constant and C, a subset of the sample space satisfying ix the size of the test to be a. (a) P(XeC|0=%)=a, 2 (Box) _ F180) 5 2852) f(*1)) ‘Suppose that there exists another test of size a, defined by the critical region Cl, ic (b) XeGQe Reject Hy iff xeC, where P(x eC; |0=6))= Let B, = CpG. By = CoN. By =CENG, Note that 8, UB) =Co, 8, UB, jy and B,,2, & B, are disjoint. Let the power of the likelihood ratio test be I = P(X © Cy |0=6,), and the power of the other test be f, = P(X€C,|0=6)) We want to show that fy —f, 20 But Fo th= fc, FCI )ae-f,, F (#10 )de = Faye, SVE fa ya, (Orde =f, F0 Oi)def,, £2181) de AlsoB,