Lec1: Bayes Rules holds if and only if with prob one, i.e.

^I = 1 ∑ ( i ) f ( x ) dx
Va r Y ∨X ( Y |X ) =0 n
g x
P(B∨A )P ( A ) n i=1 f ( x i )
P( A∨B)= Y =E Y ∨X (Y ∨ X ) with prob one.
P (B ) PDF method to get pdf of Y =g (X ):
Choose to minimize the variance of the weights g(x)
P ( B )=P ( B| A ) P ( A ) + P ( B|¬ A ) P ( ¬ A ) Given , , find f (x)
f X ( x ) Y =g ( X ) x=g ( y ) −1
f (x)
f ( y| x ) f ( x )
| |
Continuous: , where
f X∨Y ( x| y ) = Y ∨X X
d −1 Lec3: Standard error (SE):
f Y ( y) f Y ( y )=f X ( g−1 ( y ) ) g ( y)
dy
If the function g( X) is not monotonic, you must break the range of
SE ( θ^ ) =√ Var ( θ^ )
∞
f Y ( y )= ∫ f Y ∨X ( y|x ) ⋅ f ( x ) dxX
Y into intervals where g is monotonic and sum the contributions Mean squared error (MSE):
−∞ from each interval.
^ ) 2=Var ( θ^ ) + Bias ( θ^ )2,
MSE ( θ^ )=E ( θ−θ
Marginal PDF:
1 y−b
( )
If
Y =aX +b , then f Y ( y ) = fX .
Bias ( θ^ ) =E ( θ^ ) −θ
∞
|a| a
f X ( x )=∫ f X ,Y ( x , y ) dy
,

−∞ Method of pdf to get multivariate: Sensible criteria to evaluate θ^

∞ multivariate continuous, Small SE: Stable; Small bias: accurate; Small MSE:
X =( X 1 , … , X n )
f Y ( y )= ∫ f X , Y ( x , y ) dx Consistent
^ X 1 , X n)→θ , in probability, as n → ∞
−∞ θ(
Y = ( Y 1 , … , Y n ) =g (X )
Independence of Two Random Variables: Robust (gives a good estimate, even if some bad data present in the

P ( X=x and Y = y )=P ( X =x ) ⋅ P ( Y = y .) g is one-to-one so that its inverse exists and is denoted by sample)

for any x, y The law of large numbers: To evaluate integration

x=g ( y )=w ( y )=( w1 ( y ) ,… , wn ( y ) )
−1
Or:
n
f X , Y ( x , y )=f X ( x ) ⋅f Y ( y ) ^I = 1 ∑ g ( x ) f has same domain as g and integrates to 1

| |
∂ w1 ∂ w1 n i=1 f (x)
…
Cov ( X ,Y ) =E [ ( X −E ( X ) ) ( Y −E ( Y ) ) ] ∂ y1 ∂ yn
Joint PDF:
J= ⋮ ⋱ ⋮ ,
Cov ( X , Y ) =E ( XY )−E ( X ) E ( Y )
Z=MAX ( X ,Y ) CDF : { [ F }Y ( z ) ]
n
∂ wn ∂ wn
Zero covariance: can still dependent in a nonlinear manner. E.g.: …
∂ y1 ∂ yn
2, Z= n
X ∼Uniform (−1 ,1 ) , Y = X MIN ( X ,Y ) CDF : 1−[ 1−F ( z ) ]
f Y ( y )=f X ( g−1 ( y ) )|J|
E [Z ]=0 , E [ Y ] =1 / 3 αβ −α z − β z
Exp: f Z= X +Y ( z )= ( e −e )
Conditional Distribution: for s.t. for some , and otherwise. β−α
y y=g ( x ) x f Y ( y )=0
Properties of conditional expectation: Estimators:

is a function of x and is free of Y. Moment Generating functions: MSE:

E( Y |X ) ( h ( Y ) ∨x )
2
M X ( t )=E ( e tX ) , t ∈ R ^ ) =[ bias ( θ^ ) ] + Var ( θ^ )
MSE ( θ^ )=E ( θ−θ
If X ,Y independent:
EY ∨ X ( h ( Y )|x ) =EY ( h ( Y ) )
If the mgf exists in an open interval containing zero, then Method of moments estimator (MME)

E(h(X)X =x)=h(x). Let g(x) = EYX(h(Y)x), then g(X) is a random

variable and denoted by EYX(h(Y)X)

M (Xk ) ( 0 )=E ( X k ) μk =E ( X k )
Law of total expectation: EX ¿ In particular, For any constants , If X 1 , … , X n as sample, then k th sample moment:
a , b M a+ bX ( t ) =e at M X ( bt )
E X [ E Y ∨X (Y | X ) ]=E Y (Y ) 1
n

∑
,
If independent then mk = X ki μk =mk , for k =1, ... , K
X ,Y M X +Y ( t )=M X ( t ) M Y (t ) n i=1
Variance decomposition:

Joint Moment Generating functions: where K is the number of parameters to be estimated.

Va r Y ( Y )=Va r X [ EY ∨ X ( Y |X ) ]+ E X [Va r Y ∨ X (Y ∨X )]
M X , …, X ( t 1 , … , t n ) =E ( e t 1 X 1+⋯+t n X n
) 1
n
1
n
Note. 1.
Va r Y ( Y ) E X [ Va r Y ∨ X ( Y | X ) ]
and the 1 n
σ 2M = ∑
n i=1
X 2i − X 2= ∑ ( X i−X ) =S ¿2
n i=1
2

Lec2: Monte Carlo integration:

equality holds if and only if with g(x) is to be integrated, f(x) draw set of random numbers,
EY ∨ X ( Y |X )=EY ( Y )
(Given pdf) Approach: 1. Calculate , use them

( ) E [ X ] / E [ X 2]
1
probability one. ^I =∫ g ( x ) f ( x ) dx=E g ( x ) ,
0 f (x) f (x)
2. and the equality to represent params, substitute to X , S n.There can be different
Va r Y ( Y ) ≥ Va r X [ EY |X ( Y |X ) ]
MMEs.
where X 1 , ∼ f ( x )
Maximum likelihood estimator (MLE):
 Let such that ， To test for vs .
^
θ=argmax [ L ( θ| X ) ]=argmax [ l ( θ|X ) ] θ1 θ1 >θ 0 H 0 H ¿1 : θ=θ 1
Approach: find , and factorize, get
L ( X 1 , X 2 , … , X n|θ )
Approach: Get ,using , => . By L-P Lemma we have the MP test is to reject ¿
F ( θ^ ) , F ( θ^ 2 ) f X(x) Var ( θ^ ) H1
parameter out.

Cramér-Rao Lower Bound: Uniform:

L ( θ0 ) for some constant … U [ 0 ,θ ] :T =max \{ X i \}
θ^ of a parameter θ , the variance of θ^ is at
when
for any unbiased estimator Λ= <c c.
least as large as the inverse of the Fisher Information, I ( θ ):
L ( θ^ ) :
U [ α , β ] T =( min \{ X i \} , max \{ X i \} )
1
Var ( θ^ ) ≥
<=> 2 for some constant .
, where I ( θ ) is defined as: Λ ( x ) <c X > c0 c0
I (θ) Poisson:∑ Xi ,
H 0 vs
[( )] [ ]
Therefore, the rejection region of the MP test for i
2
∂ ∂2
I ( θ )=E lo g f ( X|θ ) =−E lo g f¿ ( X|θ ) is: 2 1
∂ θ2 H 1 : θ=θ 1 R=\{ x : x > c0 \}
−n λ T
∂θ g ( T ; θ ) =e ⋅ λ , h ( X )=
∏ Xi !
, For given significant level α , the cur-off point c 0 is determined by i

f (X ; θ) is the probability density function of X , parameterized 2 , then calculate Normal: σ known :T =X ,

α =P \{ X > C0 ∨H 0 \}
byθ. 求两阶导数
−n
Confidence Interval (Interval Estimator):

Efficiency: Comparation: typical approach:

=>
α =f ( c 0 ,θ 0 ) c 0=g ( α 0 , θ0 )
is free of
θ1
. g ( T ; μ )= ( 2 π σ ) ex p
2 2
( 2−nσ ( μ−T ) ) ,
2
2

~
( )
So thee test with rejection region
~ Var ( θ )
n
−1
eff ( θ^ , θ )= 2 h ( X )=ex p 2∑ ( i
X −X )2
Var ( θ^ ) R=\{ x : x ≻θ0 log α \} 2σ i=1
Asymptotic confidence intervals: Is also the uniformly most powerful test for H 0=θ 0 ,…
σ un known :T =( X , s 2 )
√ ¿ ( θ ) ( θ−θ
^ ) d N ( 0 ,1 ) , n → ∞ vs ¿
H 1 : θ=θ 1
→ n −T
1
T =∑ X i , g ( T ; λ )=
Exp: λ
e , h ( X )=1
1−α ≈ P (−z ( α / 2 ) < √ ¿ ( θ^ ) ( θ−θ
Chi-square testing:

0) < z ( α / 2) )
^ i=1 λn
n
1
σ :V 1= 2 ∑ ( X i −μ ) ,V 1 ∼ χ 2 ( n )
2 2

θ^ ± z ( α / 2 )
1 σ 0 i=1 Gamma: T=
(∏ X , ∑ X )
i
i
i
i ,

√¿ ( θ^ ) Generalized likelihood ratio test: −∑ X i

1
i

n ∏
α−1
max L ( θ )
( Xi)
β
Hypothesis Testing:
L ( θ0 ) g (T ; θ)= e , h( X )
power=1−P ( Type II Error ) =1−β Λ= , Λ=
θ ∈Θ
, Λ ∈ [0 , 1]
0
( Γ (α) β ) i
L( θ)^ max L ( θ )
UMP testing: Uniformly most powerful (UMP) test. If a test of Useful equations:
θ ∈Θ
H 0 :θ=θ 0 vs H 1 : θ=θ 1 is a most powerful test for every 1. Small values of are evidence against H 0; n n
1 2
θ1 >θ 0 among all tests of size , then the test is uniformly most 2. Under some smoothness conditions, the asymptotic null distribution 2
Sn = ∑
n−1 i=1
( X i−X ) =S +(∑ X )
2 ¿2

powerful (UMP) for testing H :θ=θ vs H : θ>θ of is 2 distribution, where is the number i=1
0 0 1 1
−2 log ( Λ ) χ r −r r1 0
unbiased
UMP: W ={ x 1 :n : x< C } ,
of free parameters in Θ r is the number of free parameters in Θ
1 0 0 n n
¿2 1 1
where C is chosen such that P μ ( X <C )=α .
;
S = ∑
n i=1
X 2
i −X 2
= ∑
n i =1
( X i− X )
2 biased

3. Decision rule: reject H when

1
0
2

{ }
2
−2σlog ( Λ ) > χ r−r ( α ) σ
σ E ( X )= + μ
2 2
Thus, C=μ1 +uα ,W = x 1: n : x< μ 1+u α . 0
n
√n √n
Design a LR Test:
Cov ( X , Y ) =Cov ( Y , X );
is the rejection region for the likelihoodlikelihood
ratioratio
test
test statistic:
Cov ( X , X )=Var ( X )

( )
Approach to finding a UMP test: Use the Neyman-Pearson Lemma to
L( θH ) Cov ( aX ,bY )=ab ⋅Cov ( X ,Y );
2
find a MP test of H 0 ≔θ0 vs H 1 ≔ θ1 for some θ1 >θ 0 λ=−2 log 0
∼ χ n , nis number of unknown
Cov ( X ,c )=0, c is constant
If the form of the test is the same for all θ , then the test is UMP. L ( θ^ )
1
Neyman-Pearson lemma: Rej if exceeds the critical value of
t n ( α )=−t n ( 1−α )
2
H0 λ χ α,1
L(θ0)
Λ ( x )= Sufficient Statistics:
L ( θ1 ) Solution Approach:

Approach:
T is θ SS ↔ f ( X 1 , X 2 , … , X n ; θ ) =h ( X 1 , X 2 ,⋯ , X n ) g ( T ; θ )
 Let be independent and Let i.i.d. , Define:
∼ N ( μ ,θ 2 )
2
X1 … X k X i ∼ χn X1 , … , Xn

2 n
Y = X 1 +…+ X k ∼ χ n +…+n Sample mean: 1
1 k
X= ∑X
n i=1 i
Let then
2 2
Z ∼ N (0 , 1) X =Z = χ 1
n
Sample variance: 1
2 2
X ∼ χ ( m ) ,Y ∼ χ ( n )
: S=2
n ∑
n−1 i=1
( X i−X )
2

Common MGFs :
1 2 2
X ∼ N ( μ , σ 2 ): M X ( t )=e
( ) ( )
μt+ σ t 2
2 X/m X m n 1. σ and
∼ F ( m, n ) , ∼ Beta , X ∼ N μ,
Y /n X +Y 2 2 n
λ , for
X ∼ Exp ( λ ): M X ( t )= t <λ
λ−t T distribution: √ n ( X−μ ) / σ ∼ N ( 0 , 1 )
and be independent, then 2. and 2 n are independent.
X ∼ Poisson ( λ ) M X ( t )=e λ (e −1)
: t
2
Z ∼ N (0 , 1) U ∼ χn X SN
Z
∼t ( n−1 ) S2n
( )
−α
t: 3.
X ∼ Γ ( α , β ) M X ( t )= 1− for t< β √ U / n n ∼ χ n−1
2
β σ
2
t (x)=t(−x ), symmetric about 0
As , , by LLN
:
X ∼ Bin ( n , p ) M X ( t )=( 1− p+ p e t )
n
n → ∞ t n → N ( 0 ,1 ) 4. √ n ( X−μ ) ∼ t
2 n−1
Sn
bt at
t n has heavier tail than N (0 ,1)
e −e
X ∼ U ( a , b ) M X ( t )= F distribution F
m ,n
5. X 1 −X , … , X n−X not independent. Sum to 0. Degree
t ( b−a )
n n-1
mean: , n>2 variance:
Conversions: n−2

(∑ )
2
n n n
2 n ( m+ n−2 )
Sn = ∑ X i ∼ N μi , ∑ σ
Normal: 2
i ,n> 4
i=1 i=1 i=1 m ( n−2 )2 ( n−4 )

(∑ )
n n Gamma:

Sn=∑ X i ∼ Poisson
Poisson:
λi α Converge almost surely.
i=1 i=1
( ) β α −1 −β x
f x ;α , β = x e for x >0 ,∧α , β >0.
Γ (α ) (
P lim |Z n−Z|< ϵ =1
n →∞ )
1
( )
Exponential: , all are the same.
Sn ∼ Γ n , λi Converge in probability.
λ E ( X )=α / β , D ( X )=α / β
2

lim P (|Z n−Z|< ϵ )=1

x−a α n→∞
CDFs: U ( a , b ) : F ( x )= , x ∈ [ a , b] MME: E [ X ] = =X ,
b−a β Converge in distribution.

α X
[ ( )]
2
1 x−μ Var ( X )= =S → β= 2 , α = X lim F n ( z )=F ( z )
N ( μ , σ ) F ( x )= 2 1+ erf
2 : 2
β S n→∞
σ √2

( )
−α
t
: − λx M X ( t )= 1− for t< β
exp ( λ ) F ( x )=1−e for x ≥ 0 β

( )
k Method for Integration by Parts:
λi 1
Poisson ( λ ) F ( k )=e− λ ∑
: Note:
exp ( λ)≡ Γ 1 ,
i! λ ∫ u dv =u ⋅v−∫ v du
, is easy to integral
i=0 v
k −k

() ( )
−α Look at the tables:
Bin(n , p) F ( k )=∑ n p ( 1− p ) t
, i n−i 2
M X ( t )= 1− to M X ( t )=( 1−2t ) z ( α / 2 ) is upper 2 quantile in N ( 0 , 1 )
i=0 i β
Others: P-value: In null-hypothesis significance testing, the
Chi-square distribution 2:
χ
( ) k 1 p-value is the probability of obtaining test results at least as extreme as
n
X ∼ χ2 ( k ) → X ∼ Γ ,
Special case of Gamma distribution,
2 2 the result actually observed, under the assumption that the null

hypothesis is correct.
For a chi-square distribution converted to a gamma distribution:

( n2 , 12 ) τ.
Find the form of an approximate confidence interval for
χ 2n =Gamma Distributions of sample mean and sample variance.
τ^ ∼ N τ , ( nτ^ )
( 1−α ) 100 % asymptotic C.I. is τ^ ± Z α / 2 ⋅ ( τ^
√n )

If two estimators not unbiased, use their MSE (divide) Exact Pivots :
Exponential:

( )
n
nS nS
Y = λ ∑ X i ; CI : 2
,
i=1 χ 1−α / 2 , n χ 2α / 2 ,n
Y −nλ
Poisson: Z= CI : ( nλ ± z α / 2 √ nλ )
√ nλ
Binomial:

Z=
X −np
√np ( 1−p )
, CI : p ± z α / 2 (
p ( 1− p )
n √ )