.

Bayesian Methods for Data Analysis

ENAR Annual Meeting

Tampa, Florida March 26, 2006

ENAR - March 2006

Course contents
Introduction of Bayesian concepts using single-parameter models. Multiple-parameter models and hyerarchical models. Computation: approximations to the posterior, importance sampling and MCMC. Model checking, diagnostics, model t. Linear hierarchical models: random eects and mixed models. Generalized linear models: logistic, multinomial, Poisson regression. Hierarchical models for spatially correlated data.
ENAR - March 2006 2

rejection and

Course emphasis
Notes draw heavily on the book by Gelman et al., Bayesian Data Analysis 2nd. ed., and many of the gures are borrowed directly from that book. We focus on the implementation of Bayesian methods and interpretation of results. Little theory, but some is needed to understand methods. Lots of examples. Some are not directly drawn from biological problems, but still serve to illustrate methodology. Biggest idea to get across: inference by simulation.
ENAR - March 2006 3

 Software: R (or SPlus) early on, and WinBUGS for most of the examples after we discuss computational methods. All of the programs used to construct examples can be downloaded from www.public.iastate.edu/ alicia. On my website, go to Teaching and then to ENAR Short Course.

ENAR - March 2006

Single parameter models

There is no real advantage to being a Bayesian in these simple models. We discuss them to introduce important concepts: Priors: non-informative, conjugate, other informative Computations Summary of posterior, presentation of results Binomial, Poisson, Normal models as examples Some real examples

ENAR - March 2006

Binomial model
Of historical importance: Bayes derived his theorem and Laplace provided computations for the Binomial model. Before Bayes, question was: given , what are the probabilities of the possible outcomes of y ? Bayes asked: what is P r(1 < < 2|y )? Using a uniform prior for , Bayes showed that
2 1

P r(1 < < 2|y )

y (1 )ny d p(y )

with p(y ) = 1/(n + 1) uniform a priori.

ENAR - March 2006 6

 Laplace devised an approximation to the integral expression in numerator. First application by Laplace: estimate the probability that there were more female births in Paris from 1745 to 1770. Consider n exchangeable Bernoulli trials y1, ..., yn. yi = 1 a success, yi = 0 a failure. Exchangeability:summary is # of successes in n trials y . For the probability of success, Y B (n, ): p(y |) =
ENAR - March 2006

n y (1 )ny . y
7

 = y/n. MLE is sample proportion: Prior on : uniform on [0,1] (for now) p() 1. Posterior: p(|y ) y (1 )ny . As a function of , posterior is proportional to a Beta(y + 1, n y + 1) density. We could also do the calculations to derive the posterior: p(|y ) =
n y n y

y (1 )ny y (1 )ny d
8

ENAR - March 2006

n! = (n + 1) y (1 )ny y !(n y )! = = (n + 1)! y (1 )ny y !(n y )! (n + 2) y+11(1 )ny+11 (y + 1)(n y + 1)

= Beta(y + 1, n y + 1) Point estimation

y +1 Posterior mean E (|y ) = n +2 Note posterior mean is compromise between prior mean (1/2) and sample proportion y/n. Posterior mode y/n Posterior median such that P r( |y ) = 0.5. Best point estimator minimizes the expected loss (more later).
ENAR - March 2006 9

 Posterior variance (y + 1)(n y + 1) V ar(|y ) = . (n + 2)2(n + 3) Interval estimation 95% credibe set (or central posterior interval) is (a, b) if:
a b

p(|y )d = 0.025 and

0 0

p(|y )d = 0.975

A 100(1 )% highest posterior density credible set is subset C of such that C = { : p(|y ) k ()} where k () is largest constant such that P r(C |y ) 1 . For symmetric unimodal posteriors, credible sets and highest posterior density credible sets coincide.
ENAR - March 2006 10

 In other cases, HPD sets have smallest size. Interpretation (for either): probability that is in set is equal to 1 . Which is what we want to say!.

Credible set and HPD set

ENAR - March 2006

11

 Inference by simulation: Draw values from posterior: easy for closed form posteriors, can also do in other cases (later) Monte Carlo estimates of point and interval estimates Added MC error (due to sampling) Easy to get interval estimates and estimators for functions of parameters. Prediction: Prior predictive distribution
1

p(y ) =
0

n y 1 ny (1 ) d = . y n+1

A priori, all values of y are equally likely

ENAR - March 2006 12

 Posterior predictive distribution, to predict outcome of a new trial y given y successes in previous n trials
1

P r( y = 1|y ) =
0 1

P r( y = 1|y, )p(|y )d P r( y = 1|)p(|y )d

0 1

= =
0

y+1 p(|y )d = E (|y ) = n+2

ENAR - March 2006

13

Binomial model: dierent priors

How do we choose priors? In a purely subjective manner (orthodox) Using actual information (e.g., from literature or scientic knowledge) Eliciting from experts For mathematical convenience To express ignorance Unless we have reliable prior information about , we prefer to let the data speak for themselves. Asymptotic argument: as sample size increases, likelihood should dominate posterior

ENAR - March 2006

14

Conjugate prior
Suppose that we choose a Beta prior for : p(|, ) 1(1 ) 1 Posterior is now p(|y ) y+1(1 )ny+ 1 Posterior is again proportional to a Beta: p(|y ) = Beta(y + , n y + ). For now, , considered xed and known, but they can also get their own prior distribution (hierarchical model).
ENAR - March 2006 15

 Beta is the conjugate prior for binomial model: posterior is in the same form as the prior. To choose prior parameters, think as follows: observe successes in + prior trials. Prior guess for is /( + ).

ENAR - March 2006

16

Conjugate priors
Formal denition: F a class of sampling distributions and P a class of prior distributions. Then P is conjugate for F if p() P and p(y |) F implies p(|y ) P . If F is exponential family then distributions in F have natural conjugate priors. A distribution in F has form p(yi|) = f (yi)g () exp(()T u(yi)). For an iid sequence, likelihood function is p(y |) g ()n exp(()T t(y )),
ENAR - March 2006 17

for () the natural parameter and t(y ) a sucient statistic. Consider prior density p() g () exp(()T ) Posterior also in exponential form p(|y ) g ()n+ exp(()T (t(y ) + ))

ENAR - March 2006

18

Conjugate prior for binomial proportion

y is # of successes in n exchangeable trials, so sampling distribution is binomial with prob of success : p(y |) y (1 )ny Note that p(y |) y (1 )n(1 )y (1 )n exp{y [log log(1 )]} Written in exponential family form } p(y |) (1 ) exp{y log 1
n
ENAR - March 2006 19

where g ()n = (1 )n, y is the sucient statistic, and the logit log /(1 ) is the natural parameter.

Consider prior for : Beta(, ): p() 1(1 ) 1

If we let = 1 and = + 2, can write p() (1 ) or p() (1 ) exp{ log } 1

ENAR - March 2006 20

 Then posterior is in same form as prior: p(|y ) (1 )n+ exp{(y + ) log } 1

Since p(y |) u(1 )ny then prior Beta(, ) suggests that a priori we believe in approximately successes in + trials. Our prior guess for the probability of success is /( + ) By varying ( + ) (with /( + ) xed), we can incorporate more or less information into the prior: prior sample size Also note: +y ++n is always between the prior mean /( + ) and the MLE y/n. E (|y ) =
21

ENAR - March 2006

 Posterior variance V ar(|y ) = = ( + y )( + n y ) ( + + n)2( + + n + 1) E (|y )[1 E (|y )] ++n+1

As y and n y get large: E (|y ) y/n var(|y ) (y/n)[1 (y/n)]/n which approaches zero at rate 1/n. Prior parameters have diminishing inuence in the posterior as n .

ENAR - March 2006

22

Placenta previa example

Condition in which placenta is implanted low in uterus, preventing normal delivery. In study in Germany, found that 437 out of 980 births with placenta previa were females so observed ratio is 0.446 Suppose that in general population, proportion of female births is 0.485. Can we say that the proportion of female babies among placenta previa births is lower than in the general population? If y = number of female births, is probability of a female birth, and p() is uniform, then p(|y ) y (1 )n1
ENAR - March 2006 23

and therefore |y Beta(y + 1, n y + 1).

Thus a posteriori, is Beta(438, 544) and 438 E (|y ) = 438 + 544 V ar(|y ) = 438 544 (438 + 544)2(438 + 544 + 1)

Check sensitivity to choice of priors: try several Beta priors with increasingly more information about .

Fix prior mean at 0.485 and increase prior sample size + .

ENAR - March 2006 24

Prior mean 0.5 0.485 0.485 0.485 0.485 0.485 0.485

+ 2 2 5 10 20 100 200

Post. median 0.446 0.446 0.446 0.446 0.447 0.450 0.453

Post 2.5th pctile 0.415 0.415 0.415 0.415 0.416 0.420 0.424

Post 97.5th pctile 0.477 0.477 0.477 0.477 0.478 0.479 0.481

Results robust to choice of prior, even very informative prior. Prior mean not in posterior 95% credible set.

ENAR - March 2006

25

Conjugate prior for Poisson rate

y (0, 1, ...) is counts, with rate > 0. Sampling distribution is Poisson yi exp() p(yi|) = yi ! For exchangeable (y1, ..., yn)
exp(n) yi exp() ny p(y |) = = . n y ! y ! i i=1 i i=1 n

Consider Gamma(, ) as prior for : p() 1 exp() Then posterior is also Gamma:
+1 p(|y ) ny exp(n)1 exp() ny exp(n )
ENAR - March 2006 26

 For = ny + and = n + , posterior is Gamma(, ) Note: Prior mean of is / Posterior mean of is ny + E (|y ) = n+ If sample size n then E (|y ) approaches MLE of . If sample size goes to zero, then E (|y ) approaches prior mean.

ENAR - March 2006

27

Mean of a normal distribution

y N (, 2) with 2 known For {y1, ..., yn} an iid sample, the likelihood is: p(y |) = i 1 1 exp( 2 (yi )2) 2 2 2

Viewed as function of , likelihood is exponential of a quadratic in : p(y |) exp( 1 2 2

2 (2 2yi + yi )) i

Conjugate prior for must belong to family of form p() = exp(A2 + B + C )

ENAR - March 2006 28

that can be parameterized as 1 p() exp( 2 ( 0)2) 20

2 and then p() is N (0, 0 ) 2 (0, 0 ) are hyperparameters. For now, consider known.

If p() is conjugate, then posterior p(|y ) must also be normal with 2 parameters (n, n ). Recall that p(|y ) p()p(y |) Then 1 p(|y ) exp( [ 2
i (yi

)2

( 0)2 + ]) 2 0
29

ENAR - March 2006

 Expand squares, collect terms in 2 and in : 1 p(|y ) exp( [ 2

2 i yi

i yi 2

2 20 + 2 0 + ]) 2 0

2 2 1 (0 + 0 2) + 2)n2 2(ny 0 exp( [ ]) 2 2 2 0 2 2 + 0( 2/n) 1 (( 2/n) + 0 ) 2 y 0 exp( [ 2( )]) 2 2 2 2 2 ( /n)0 ( /n) + 0

Then p(|y ) is normal with

2 2 Mean: n = ( y 0 + 0( 2/n))/(( 2/n) + 0 ) 2 2 2 Variance: n = (( 2/n)0 )/(( 2/n) + 0 )

Posterior mean
ENAR - March 2006 30

 Note that n =
2 y 0

+ 0( /n) = 2 ( 2/n) + 0

n 1 y + 2 2 0 0 1 n + 2 2 0

2 (by dividing numerator and denominator into ( 2/n)0 )

Posterior mean is weighted average of prior mean and sample mean.

2 Weights are given by precisions n/ 2 and 1/0 .

As data precision increases (( 2/n) 0) because 2 0 or because n , n y . Also, note that n =

ENAR - March 2006

2 y 0 + 0( 2/n) 2 ( 2/n) + 0
31

2 0 2/n ( 2 = 0( 2 2) + y 2) ( /n) + 0 ( /n) + 0 2 2 Add and subtract 00 /(( 2/n) + 0 ) to see that 2 0 n = 0 + ( y 0)( 2 ) 2 ( /n) + 0

Posterior mean is prior mean shrunken towards observed value. Amount of shrinkage depends on relative size of precisions Posterior variance: Recall that
2 2 1 (( 2/n) + 0 ) 2 y 0 + 0( 2/n) p(|y ) exp( [ 2( )]) 2 2 2 2 2 ( /n)0 ( /n) + 0
ENAR - March 2006 32

 Then 1 2 n = =
2 (( 2/n) + 0 ) 2 ( 2/n)0 n 1 + 2 2 0

Posterior precision = sum of prior and data precisions. Posterior predictive distribution Want to predict next observation y . Recall p( y |y ) We know that
ENAR - March 2006 33

p( y |)p(|y )d

 Given , y N (, 2) 2 |y N (n, n ) Then p( y |y ) 1 ( y )2 1 ( n)2 exp{ } exp{ }d 2 2 2 2 n 1 ( y )2 ( n)2 exp{ [ + ]}d 2 2 2 n

Integrand is kernel of bivariate normal, so ( y , ) have bivariate normal joint posterior. Marginal p( y |y ) must be normal.
ENAR - March 2006 34

 Need E ( y |y ) and var( y |y ): E ( y |y ) = E [E ( y |y, )|y ] = E (|y ) = n, because E ( y |y, ) = E ( y |) = var( y |y ) = E [var( y |y, )|y ] + var[E ( y |y, )|y ] = E ( 2|y ) + var(|y )
2 = 2 + n

because var( y |y, ) = var( y |) = 2 Variance includes additional term n as a penalty for us not knowing the true value of . Recall n =
ENAR - March 2006

1 + n y 2 0 2 0 1 + n 2 2 0
35

2 In n, prior precision 1/0 and data precision n/ 2 are equivalent. Then:

For n large, ( y , 2) determine posterior 2 For 0 = 2/n, prior has the same weight as adding one more observation with value 0. 2 2 When 0 with n xed, or when n with 0 xed: p(|y ) N ( y , 2/n) Good approximation in practice when prior beliefs about are vague or when sample size is large.

ENAR - March 2006

36

Normal variance
Example of a scale model Assume that y N (, 2), known. For iid y1, ..., yn: 1 2 2 n/2 p(y | ) ( ) exp( 2 2 p(y | ) ( ) for suent statistic 1 v= n
ENAR - March 2006

(yi )2)
i=1

2 n/2

nv exp( 2 ) 2

(yi )2
i=1
37

 Likelihood is in exponential family form with natural parameter ( 2) = 2. Then, natural conjugate prior must be of form p( 2) ( 2) exp(( 2))

Consider an inverse Gamma prior: p( 2) ( 2)(+1) exp(/ 2)

For ease of interpretation, reparameterize as a scaled-inverted 2 2 distribution, with 0 d.f. and 0 scale.
2 Scale 0 corresponds to prior guess for 2. 2 Large 0: lots of condence in 0 as a good value for 2.
ENAR - March 2006 38

2 If 2 inv 2(0, 0 ) then 2 2 0 0 0 p( 2) ( 2 )( 2 +1) exp( 2 ) 0 2

2 0 0 0 Gamma( 2 , 2 ).

Corresponds to Inv

2 0/(0 2) Prior mean is 0 4 /0: large 0, large prior precision. Prior variance behaves like 0

Posterior p( 2|y ) p( 2)p(y | 2) ( )

ENAR - March 2006

2 n/2

2 nv 2 ( 0 +1) 00 exp( 2 )( 2 ) 2 exp( 2 ) 2 0 2

39

1 +1) 2 ( 2 ( ) exp(

1 2 ) 1 1 2 2

with 1 = 0 + n and
2 1

2 nv + 00 = n + 0

p( 2|y ) is also a scaled inverted 2 Posterior scale is weighted average of prior guess and data estimate Weights given by prior and sample degrees of freedom Prior provides information equivalent to 0 observations with average 2 squared deviation equal to 0 .
2 As n , 1 v.

ENAR - March 2006

40

Poisson model
Appropriate for count data such as number of cases, number of accidents, etc. For a vector of n iid observations: t(y)en yi e = p(y |) = p(y |) = i , n yi ! y ! i=1 where is the rate, y = 0, 1, ... and t(y ) = statistic for .
n i=1 yi

the sucient

We can write the model in the exponential family form: p(y |) en et(y) log where () = log is natural parameter.
ENAR - March 2006 41

 Natural conjugate prior must have form p() e e log or p() e 1 that looks like a Gamma(, ). Posterior is Gamma( + ny , + n)

ENAR - March 2006

42

Poisson model - Digression

In the conjugate case, can often derive p(y ) using p(y ) = p()p(y |) p(|y )

In case of Gamma-Poisson model for a single observation: ( + y ) Poisson() Gamma(, ) p(y ) = = Gamma( + y, + 1) ()y !(1 + )(+y) 1 y = ( + y 1, y )( ) ( ) +1 +1 the density of a negative binomial with parameters (, ).
ENAR - March 2006 43

 Thus we can interpret negative binomial distributions as arising from a mixture of Poissons with rate , where follows a Gamma distribution: p(y ) = Poisson (y |) Gamma (|, )d

Negative binomial is robust alternative to Poisson.

ENAR - March 2006

44

Poisson model (contd)

Given rate, observations assumed to be exchangeable. Add an exposure to model: observations are exchangeable within small exposure intervals. Examples: In a city of size 500,000, dene rate of death from cancer per million people, then exposure is 0.5 Intersection in Ames with trac of one million vehicles per year, consider number of trac accidents per million vehicles per year. Exposure for intersection is 1. Exposure is typically known and reects the fact that in each unit, the number of persons or cars or plants or animals that are at risk is dierent.
ENAR - March 2006 45

 Now p(y |) for xi the exposure of unit i.

Py

exp(

xi)

With prior p() = Gamma (, ), posterior is p(|y ) = Gamma ( +

i

yi , +
i

xi)

ENAR - March 2006

46

Non-informative prior distns

A non-informative prior distribution Has minimal impact on posterior Lets the data speak for themselves Also called vague, at, diuse So far, we have concentrated on conjugate family. Conjugate priors can also be almost non-informative.
2 If y N (, 1), natural conjugate for is N (0, 0 ), and posterior is 2 N (1, 1 ), where 2 1 0/0 + ny / 2 2 1 = , = 1 2 + n/ 2 2 + n/ 2 1/0 1/0
ENAR - March 2006 47

 For 0 : 1 y 2 1 2/n Same result could have been obtained using p() 1. The uniform prior is a natural non-informative prior for location parameters (see later). It is improper: p()d = d =

yet leads to proper posterior for . This is not always the case. Poisson example: y1, ..., yn iid Poisson(), consider p() 1/2.
1/2 d 0

= improper
48

ENAR - March 2006

 Yet p(|y ) =

=
1 + proportional to a Gamma( 2

P y n 1/2 e P y 1/2 n e P y +1/21 n

i i i i i i

i yi , n),

proper

Uniform priors arise when we assign equal density to each : no preferences for one value over the other. But they are not invariant under one-to-one transformations. Example: = exp{} so that = log { } is inverse transformation.
ENAR - March 2006 49

d 1 d = is Jacobian Then, if p() is prior for , p( ) = 1p(log ) is corresponding prior for transformation. For p() c, p( ) 1, informative. Informative prior is needed to arrive at same answer in both parameterizations.

ENAR - March 2006

50

Jereys prior
In 1961, Jereys proposed a method for nding non-informative priors that are invariant to one-to-one transformations Jereys proposed p() [I ()]1/2 where I () is the expected Fisher information: d2 I () = E [ 2 log p(y |)] d If is a vector, then I () is a matrix with (i, j ) element equal to d2 E [ log p(y |)] didj
ENAR - March 2006 51

and p() |I ()|1/2 Theorem: Jereys prior is locally uniform and therefore non-informative.

ENAR - March 2006

52

Jereys prior for binomial proportion

If y B (n, ) then log p(y |) y log + (n y ) log(1 ) and d2 2 2 log p ( y | ) = y ( n y )(1 ) d2 Taking expectations: d2 I () = E [ 2 log p(y |)] = E (y )2 + (n E (y ))(1 )2 d n 2 2 = n (n n)(1 ) = . (1 ) Then 1 1 p() [I ()]1/2 1/2(1 )1/2 Beta( , ). 2 2
ENAR - March 2006 53

Jereys prior for normal mean

y1, ..., yn iid N (, 2), 2 known. p(y |) exp( 1 2 2 d2 n log p ( y | ) = d2 2 log p(y |) constant with respect to . Then I () constant and p() constant. 1 2 2 (y )2)

(y )2

ENAR - March 2006

54

Jereys prior for normal variance

y1, ..., yn iid N (, 2), known. 1 p(y | ) exp( 2 (y )2) 2 1 log p(y | ) n log 2 (y )2 2 d2 n 3 2 log p ( y | ) = ( y ) i d 2 2 2 4
n

Take negative of expectation so that: I ( ) = n 3 n 2 + n = 2 2 4 2 2

and therefore the Jereys prior is p( ) 1.

ENAR - March 2006 55

Invariance property of Jereys

The Jereys prior is invariant to one-to-one transformations = h() Then: p() = p()| d dh() 1 | = p()| | d d

Under invariance, choose p() such that p() constructed as above would match what would be obtained directly. Consider p() = [I ()]1/2 and evaluate I () at = h1(): d2 I () = E [ 2 log p(y |)] d d2 d = E [ 2 log p(y | = h1())[ ]2] d d
ENAR - March 2006 56

d 2 = I ()[ ] d
d Then [I ()]1/2 = [I ()]1/2 d as required.

ENAR - March 2006

57

Multiparameter models - Intro

Most realistic problems require models with more than one parameter Typically, we are interested in one or a few of those parameters Classical approach for estimation in multiparameter models: 1. Maximize a joint likelihood: can get nasty when there are many parameters 2. Proceed in steps Bayesian approach: base inference on the marginal posterior distributions of the parameters of interest. Parameters that are not of interest are called nuisance parameters.
ENAR - March 2006 1

Nuisance parameters
Consider a model with two parameters (1, 2) (e.g., a normal distribution with unknown mean and variance) We are interested in 1 so 2 is a nuisance parameter The marginal posterior distribution of interest is p(1|y ) Can be obtained directly from the joint posterior density p(1, 2|y ) p(1, 2)p(y |1, 2) by integrating with respect to 2: p(1|y ) = p(1, 2|y )d2

ENAR - March 2006

Nuisance parameters (contd)

Note too that p(1|y ) = p(1, |2, y )p(2|y )d2

The marginal of 1 is a mixture of conditionals on 2, or a weighted average of the conditional evaluated at dierent values of 2. Weights are given by marginal p(2|y )

ENAR - March 2006

Nuisance parameters (contd)

Important dierence with frequentists! By averaging conditional p(1, |2, y ) over possible values of 2, we explicitly recognize our uncertainty about 2. Two extreme cases: 1. Almost certainty about the value of 2: If prior and sample are very informative about 2, marginal p(2|y ) will be concentrated around 2. In that case, some value p(1|y ) p(1| 2, y ) 2. Lots of uncertainty about 2: Marginal p(2|y ) will assign relatively 2 no high probability to wide range of values of 2. Point estimate longer reliable. Important to average over range of values of 2.
ENAR - March 2006 4

Nuisance parameters (contd)

In most cases, integral not computed explicitly Instead, use a two-step simulation approach 1. Marginal simulation step: Draw value 2 of 2 from p(2|y ) for k = 1, 2, ... (k ) 2. Conditional simulation step: For each 2 , draw a value of 1 from (k ) the conditional density p(1|2 , y ) Eective approach when marginal and conditional are of standard form. More sophisticated simulation approaches later.
(k )

ENAR - March 2006

Example: Normal model

yi iid from N (, 2), both unknown Non-informative prior for (, 2) assuming prior independence: p(, 2) 1 2

Joint posterior: p(, 2|y ) p(, 2)p(y |, 2) 1 n2 exp( 2 2

ENAR - March 2006

(yi )2)
i=1
6

 Note that
n

(yi )2 =
i=1 i

2 (yi 2yi + 2) 2 yi 2ny + n2 i

= =
i

(yi y )2 + n( y )2

by adding and subtracting 2ny 2.

ENAR - March 2006

Example: Normal model (contd)

Let

1 s = n1
2

(yi y )2
i

Then can write posterior for (, 2) as p(, |y )

2 n2

1 exp( 2 [(n 1)s2 + n( y )2]) 2

Sucient statistics are ( y , s2)

ENAR - March 2006

Example: Conditional posterior p(| 2, y )

Conditional on 2: p(| 2, y ) = N ( y , 2/n) We know this from earlier chapter (posterior of normal mean when variance is known) We can also see this by noting that, viewed as a function of only: p(| 2, y ) exp( n 2 ) ( y ) 2 2

that we recognize as the kernel of a N ( y , 2/n)

ENAR - March 2006 9

Example: Marginal posterior p( 2|y )

To get p( 2|y ) we need to integrate p(, 2|y ) over : p( 2|y ) n2 exp(
n2

1 2 2 [( n 1) s + n ( y ) ])d 2 2 n exp( 2 ( y )2)d 2

(n 1)s2 ) exp( 2 2

2 ( n 1) s 2 /n n2 exp( ) 2 2 2

Then

(n 1)s2 p( |y ) ( ) exp( ) 2 2 which is proportional to a scaled-inverse 2 distribution with degrees of freedom (n 1) and scale s2.
2 2 (n+1)/2
10

ENAR - March 2006

Recall classical result: conditional on 2, the distribution of the scaled sucient statistic (n 1)s2/ 2 is 2 n1 .

ENAR - March 2006

11

Normal model: analytical derivation

For the normal model, we can derive the marginal p(|y ) analytically:

p(|y ) =

p(, 2|y )d 2 1 1 n/2+1 ( 2) exp( 2 [(n 1)s2 + n( y )2])d 2 2 2

Use the transformation

A 2 2 where A = (n 1)s2 + n( y )2. Then z= d 2 A = 2 dz 2z

ENAR - March 2006

12

and

p(|y )
0

z n A ( ) 2 +1 2 exp(z )dz A z z
n 1 2

n/2

exp(z )dz

ENAR - March 2006

13

Normal model: analytical derivation

p(|y ) A
n/2

n 1 2

exp (z )dz

Integrand is unnormalized Gamma(n/2, 1), so integral is constant w.r.t. Recall that A = (n 1)s2 + n( y )2. Then p(|y ) An/2 [(n 1)s2 + n( y )2]n/2 n( y )2 n/2 [1 + ] 2 (n 1)s the kernel of a tdistribution with n 1 degrees of freedom, centered at y and with scale parameter s2/n
ENAR - March 2006 14

 For the non-informative prior p(, 2) 2, the posterior distribution of is a non-standard t. Then, p( the standard t distribution. y |y ) = tn1 s/ n

ENAR - March 2006

15

Normal model: analytical derivation

We saw that y p( |y ) = tn1 s/ n

Notice similarity to classical result: for iid normal observations from N (, 2), given (, 2), the pivotal quantity y |, 2 tn1 s/ n A pivot is a non-trivial function of the data and the parameter(s) whose distribution, given , is independent of . Property deduced from sampling distribution as above. Baby example of pivot property: y N (, 1). Pivot is x = y . Given , x N (0, 1), independent of .
ENAR - March 2006 16

Posterior predictive for future obs

Posterior predictive distribution for future observation y is a mixture: p( y |y ) = p( y |y, 2, )p(, 2|y )dd 2

First factor in integrand is just normal model, and it does not depend on y at all. To simulate y from posterior predictive distributions, do the following: 1. Draw 2 from Inv-2(n 1, s2) 2. Draw from N ( y , 2/n) 3. Draw y from N (, 2)
ENAR - March 2006 17

 Can derive the posterior predictive distribution in analytic form. Note that p( y | 2, y ) = After some algebra: 1 2 p( y | , y ) = N ( y , (1 + ) ) n
2

p( y | 2, )p(| 2, y )d 1 n 2 exp( 2 ( y ) ) exp( 2 ( y )2)d 2 2

Using same approach as in deriving posterior distribution of , we nd that 1 p( y |y ) tn1( y , (1 + )1/2s) n

ENAR - March 2006 18

Normal data and conjugate prior

Recall that using a non-informative prior, we found that p(| 2, y ) N ( y , 2/n) p( 2|y ) Inv 2(n 1, s2) Then, factoring p(, 2) = p(| 2)p( 2) the conjugate prior for 2 would also be scaled inverse 2 and for (conditional on 2) would be normal. Consider | 2 2 Jointly: p(, 2) 1( 2)(0/2+1) exp(
ENAR - March 2006

N(0, 2/0)
2 ) Inv-2(0, 0

1 2 2 [ + ( ) ]) 0 0 0 0 2 2
19

Normal data and conjugate prior

Note that and 2 are not independent a priori. Posterior density for (, 2):
2 Multiply likelihood by N-Inv-2(0, 2/0; 0, 0 ) prior Expand the two squares in Complete the square by adding and subtracting term depending on y and 0 2 2 Then p(, 2|y ) N-Inv-2(n, n /n; n, n ) where

n = n
2 nn

0 n 0 + y 0 + n 0 + n = 0 + n, n = 0 + n 0 n 2 2 = 00 + (n 1)s + ( y )2 0 + n
20

ENAR - March 2006

Normal data and conjugate prior

Interpretation of posterior parameters: n a weighted average as earlier 2 nn : sum of the sample sum of squares, prior sum of squares, and additional uncertainty due to dierence between sample mean and prior mean.

ENAR - March 2006

21

Normal data and conjugate prior

Conditional posterior of : As before | 2, y N(n, 2/n).
2 Marginal posterior of 2: As before 2|y Inv-2(n, n ). 2 Marginal posterior of : As before |y tn (|n, n /n).

Two ways to sample from joint posterior distribution: 1. Sample from t and 2 from Inv-2 2. Sample 2 from Inv-2 and given 2, sample from N

ENAR - March 2006

22

Semi-conjugate prior for normal model

Consider setting independent priors for and 2: 2
2 N(0, 0 ) 2 Inv-2(0, 0 )

Example: mean weight of students, visual inspection shows weights between 100 and 200 pounds. p(, 2) = p()p( 2) is not conjugate and does not lead to posterior of known form Can factor as earlier:
2 | 2, y N(n, n )
ENAR - March 2006 23

with n =

1 n + y 0 2 2 0 , n 1 + 2 2 2

2 n

1 2 0

1 n + 2

NOTE: Even though and 2 are independent a priori, they are not independent in the posterior.

ENAR - March 2006

24

Semi-conjugate prior and p( 2|y )

The marginal posterior p( 2|y ) can be obtained by integrating the joint p(, 2|y ) w.r.t. : p( 2|y )
2 2 N(|0, 0 ) Inv2( 2|0, 0 ) N(yi|, 2)d

Integration can be performed by noting that integrand as function of is proportional to normal density. Keeping track of normalizing constants that depend on 2 is messy. Easier to note that: p(, 2|y ) 2 p( |y ) = p(| 2, y )
ENAR - March 2006 25

so that
2 2 2 2 2 ) N ( y | , ) ) Inv ( | , N ( | , i 0 0 0 0 2 p( |y ) 2) N(|n, n

which is still a mess.

ENAR - March 2006

26

Semi-conjugate prior and p( 2|y )

From earlier page:
2 2 N(|0, 0 ) Inv 2( 2|0, 0 ) N(yi|, 2) p( |y ) 2) N(|n, n 2

Factors that depend on must cancel, and therefore we know that p( 2|y ) does not depend on in the sense that we can evaluate p( 2|y ) for a grid of values of 2 and any arbitrary value of . Choose = n and then denominator simplies to something 1 proportional to n . Then
2 2 ) Inv 2( 2|0, 0 ) N(yi|, 2) p( 2|y ) n N(|0, 0

that can be evaluated for a grid of values of 2.

ENAR - March 2006 27

Implementing inverse CDF method

We often need to draw values from distributions that do not have a nice standard form. Example is expression 3.14 in text, for p( 2|y ) in the normal model with semi-conjugate priors for and 2. One approach is the inverse cdf method (see earlier notes), implemented numerically. We assume that we can evaluate the pdf (even in unnormalized form) for a grid of values of in the appropriate range

ENAR - March 2006

28

Implementing inverse CDF method

To generate 1000 draws from a distribution p(|y ) with parameter , do 1. Evaluate p(|y ) for a grid of m values of . Let the evaluations be denoted by (p1, p2, ..., pm) m1 2. Compute the CDF over the grid: (p1, p1 + p2, ...., i=1 pi, 1) and denote those (f1, f2, ..., fm) 3. Generate M uniform random variables u in [0, 1]. 4. If u [fi1, fi], draw i.

ENAR - March 2006

29

Inverse CDF method - Example

1 2 3 4 5 6 7 8 9 10 Prob() 0.03 0.04 0.08 0.15 0.20 0.30 0.10 0.05 0.03 0.02 CDF (F) 0.03 0.07 0.15 0.30 0.50 0.80 0.90 0.95 0.98 1.00

Consider a parameter with values in (1, 10) with probability distribution and cumulative distribution functions as above. Under distribution, values of between 4 and 7 are more likely.
ENAR - March 2006 30

Inverse CDF method - Example

To implement method do: 1. Draw u U (0, 1). For example, in rst draw u = 0.45 2. For u (fi1, fi), draw i. 3. For u = 0.45 (0.3, 0.5), draw = 5. 4. Alternative approach: (a) Flip another coin v U (0, 1). (b) Pick i1 if v 0.5 and pick i if v > 0.5 5. In example, for u = 0.45, would either choose = 4 or would choose = 4 with probability 1/2. 6. Repeat many times M . If M very large, we expect that about 50% of our draws will be = 4, 5, or 6, about 2% will be 10, etc.

ENAR - March 2006

31

Example: Football point spreads

Data di, i = 1, ..., 672 are dierences between predicted outcome of football game and actual score. Normal model: di N (, 2) Priors: N (0, 22) p( 2) 2 To draw values of (, 2) from posterior, do First draw 2 from p( 2|y ) using inverse cdf method Then draw from p(| 2, y ), normal. To draw 2, evaluate p( 2|y ) on the grid [150, 250].
ENAR - March 2006 32

Example: Rounded measurements (prob. 3.5)

Sometimes, measurements are rounded, and we do not observe true values. yi are observed rounded values, and zi are unobserved true measurements. If zi N (, 2), then y |, 2 i Prior: p(, 2) 2 We are interested in posterior inference about (, 2) and in dierences between rounded and exact analysis.
ENAR - March 2006 33

yi + 0.5

yi 0.5

Joint posterior contours

exact normal
4 sigma2 0 8 1 2 3

10 mu

11

12

13

rounded
4 sigma2 0 8 1 2 3

10 mu

11

12

13

Multinomial model - Intro

Generalization of binomial model, for the case where observations can have more than two possible values. Sampling distribution: multinomial with parameters (1, ..., k ), the probabilities associated to each of the k possible outcomes. Example: In a survey, respondents may: Strongly Agree, Agree, Disagree, Strongly Disagree, or have No Opinion when presented with a statement such as Instructor for Stat 544 is spectacular.

ENAR - March 2006

34

Multinomial model - Sampling distn

Formally: y = k 1 vector of counts of #s of observations in each outcome j : probability of j th outcome k k = 1 and j =1 j j =1 yj = n Sampling distribution:
j p(y |) k j =1 j

ENAR - March 2006

35

Multinomial model - Prior

Conjugate prior for (1, ..., k ) is Dirichlet distribution, a multivariate generalization of the Beta:
j p(|) k j =1 j

with

j > 0j,

and

0 =
j =1 k

j > 0j,

and
j =1

j = 1

ENAR - March 2006

36

 The j can be thought of as prior counts associated to j th outcome, so that 0 would then be a prior sample size. For the Dirichlet: j E (j ) = , 0 j (0 j ) V ar(j ) = 2 0(0 + 1) ij Cov (i, j ) = 2 0(0 + 1)

ENAR - March 2006

37

Dirichlet distribution
The Dirichlet distribution is the conjugate prior for the parameters of the multinomial model. If (1, 2, ...K ) D(1, 2, ..., K ) then (0) 1 p(1, ...k ) = j j j , j (j ) where j 0, j j = 1, j 0 and 0 = j j . Some properties: E (j ) =
ENAR - March 2006

j 0
38

j (0 j ) 2 ( + 1) 0 0 ij Cov (i, j ) = 2 0(0 + 1) V ar(j ) = Note that the j are negatively correlated. Because of the sum to one restriction, the pdf of the K -dimensional random vector can be written in terms of K 1 random variables. The marginal distributions can be shown to be Beta. Proof for K = 3: (1, 2, 3) 11 21 p(1, 2) = 1 2 (1 1 2)31 j (0)
ENAR - March 2006 39

 To get marginal for 1, integrate p(1, 2) with respect to 2 with limits of integration 0 and 1 1. Call normalizing constant Q and use change of variable: 2 v= 1 1 so that 2 = v (1 1) and d2 = (1 1)dv . The marginal is then:
1

p(1) = Q
0

1 1 1 (v (1 1))21

(1 1 v (1 1))31(1 1)dv =
1 1 Q1 (1 1

1 )

2 +3 1 0

v 21(1 v )31dv (2)(3) (2 + 3)

40

1 1 = Q1 (1 1)2+31

ENAR - March 2006

 Then, 1 Beta(1, 2 + 3). This generalizes to what is called the clumping property of the Dirichlet. In general, if (1, ..., k ) D(1, ..., K ): p(i) = Beta(1, 0 i)

ENAR - March 2006

41

Multinomial model - Posterior

Posterior must have Dirichlet form also:
j p(|y ) k j =1 j j k j =1 j

1 yj j y +j 1

n =

j (j

+ yj ) = 0 + n is total number of observations.

Posterior mean (a point estimate) of j is: E (j |y ) = =

ENAR - March 2006

j + yj 0 + n # of obs. of jth outcome total # of obs

42

 For j = 1 j : uniform non-informative prior on all vectors of j such that j j = 1 For j = 0 restriction. j : the uniform prior is on the log(j ) with same

In either case, posterior is proper if yj 1 j .

ENAR - March 2006

43

Multinomial model - samples from p(|y )

Gamma method: Two steps for each j : 1. Draw x1, x2, ..., xk from independent Gamma(, (j + yj )) for any common k 2. Set j = xj / i=1 xi Beta method: Relies on properties of Dirichlet: Marginal p(j |y ) = Beta(j + yj , n (j + yj )) Conditional p(j |j , y ) = Dirichlet

ENAR - March 2006

44

Multinomial model - Example

Pre-election polling in 1988: n = 1,447 adults in the US y1 = 727 supported G. Bush (the elder) y2 = 583 supported M. Dukakis y3 = 137 supported other or had no opinion

If no other information available, can assume that observations are exchangeable given . (However, if information on party aliation is available, unreasonable to assume exchangeability) Polling is done under complex survey design. Ignore for now and assume simple random sampling. Then, (y1, y2, y3) M ult(1, 2, 3) Of interest: 1 2, the dierence in the population in support of the two major candidates.
ENAR - March 2006 45

Multinomial model - Example

Non-informative prior, for now: set 1 = 2 = 3 = 1 Posterior distribution is Dirichlet(728, 584, 138). Then: E (1|y ) = 0.502, E (2|y ) = 0.403 Other quantities obtain by simulation (see next) To derive p(1 2|y ) do: 1. Draw m values of (1, 2, 3) from posterior 2. For each draw, compute 1 2 Results and program: see quantiles of posterior distributions of 1 and 2, credible set for 1 2 and Prob(1 > 2|y ).
ENAR - March 2006 46

Proportion voting for Bush (the elder)

0.46

0.48

0.50 theta_1

0.52

0.54

Dierence between two candidates

0.0

0.05

0.10 theta_1 - theta_2

0.15

0.20

Proportion voting for Dukakis

0.36

0.38

0.40 theta_2

0.42

0.44

Example: Bioassay experiment

Does mortality in lab animals increase with increased dose of some drug? Experiment: 20 animals randomly allocated to four doses (treatments), and number of dead animals within each dose recorded. Dose xi -0.863 -0.296 -0.053 0.727 ni 5 5 5 5 No. of deaths yi 0 1 3 5

Animals exchangeable within dose.

ENAR - March 2006 47

Bioassay example (contd)

Model yi Bin (ni, i) i are not exchangeable because probability of death depends on dose. One way to model is with linear relationship: i = + xi. Not a good idea because i (0, 1). Transform i: logit (i) = log
ENAR - March 2006

i 1 i
48

 Since logit (i) (, +), can use linear model. regression) E [logit (i)] = + xi. Likelihood: if yi Bin (ni, i), then p(yi|ni, i) (i)yi (1 i)niyi . But recall that

(Logistic

i log 1 i

= + xi,

so that

exp( + xi) . i = 1 + exp( + xi)

ENAR - March 2006

49

 Then p(yi| , , ni, xi) exp( + xi) 1 + exp( + xi)

yi

exp( + xi) 1 1 + exp( + xi)

ni yi

Prior: p(, ) 1. Posterior: p(, |y, n, x) k i=1 p(yi |, , ni , xi ). We rst evaluate p(, |y, n, x) on the grid (, ) [5, 10] [10, 40] and use inverse cdf method to sample from posterior.
ENAR - March 2006 50

Bioassay example (contd)

Posterior evaluated on 200 200 grid of values of (, ). 1 1 2 . . . . 200 p(1|y ) p(2|y ) ... ... p(200|y ) 2 ... ... 200 p(1|y ) p(2|y ) . . . . p(200|y )

Entry (j, i) in grid is p(, | = i, = j , y ). Sum of column j entries is p(j |y ) because: p(j |y ) =
ENAR - March 2006

p(j , |y )d
51

p(j |, y )p( |y )d p(j |, y )

Sum of row i entries is p(i|y ).

ENAR - March 2006

52

Bioassay example (contd)

We sample from p(|y ), and then from p( |, y ) (or the other way around). To sample from p(|y ): 1. Obtain empirical p(| = j , y ), j = 1, ...200 by summing over the . 2. Use inverse cdf method to sample from p(|y ). To sample from p( |, y ): 1. Given a draw , choose appropriate column in grid 2. Use inverse cdf method on p( | = , y ).

ENAR - March 2006

53

Bioassay example (contd)

LD50 is dose at which probability of death is 0.5, or E so that yi ni = i = 0.5

0.5 = logit1( + xi) logit(0.5) = + xi 0 = + xi

Then LD50 is computed as xi = / .

ENAR - March 2006 54

Posterior distribution of the probability of death for each dose

box plot: theta

1.0
[3]

[4]

0.75

0.5

[2]

0.25

[1]

0.0

Posterior distributions of the regression parameters

Alpha: posterior mean = 1.297, with 95% credible set [-0.365, 3.755] Beta: posterior mean = 11.52, with 95% credible set [3.572, 25.79]

alpha sample: 2000

0.6

beta sample: 2000

0.1 0.075 0.05 0.025 0.0

0.4

0.2

0.0

-2.5

0.0

2.5

5.0

-20.0

0.0

20.0

40.0

Posterior distribution of the LD50 in the log scale

Posterior mean: -0.1064 95% credible set: [-0.268, 0.1141]

LD50 sample: 2000

6.0

4.0

2.0

0.0

-1.0

-0.5

0.0

0.5

Multivariate normal model - Intro

Generalization of the univariate normal model, where now observations are d 1 vectors of measurements For the ith sample unit: yi = (yi1 , yi2 , ..., yid ), and yi N (, ), with (, ) d 1 and d d respectively For a sample of n iid observations: p(y |, ) ||n/2 exp[ 1 2 (yi ) 1(yi )]
i

1 ||n/2 exp[ tr(yi ) 1(yi )] 2 1 ||n/2 exp[ tr1S ] 2

ENAR - March 2006 55

with S=
i

(yi )(yi )

S is the d d matrix of sample squared deviations and cross-deviations from .

ENAR - March 2006

56

Multivariate normal model - Known

We want the posterior distribution of the mean p(|y ) under the assumption that the variance matrix is known. Conjugate prior for is p(|0, 0) = N (0, 0), as in the univariate case Posterior for : p exp (|y, ) 1 ( 0) 1( 0) + 2 (yi ) 1(yi )
i

that is a quadratic function of . Completing the square in the exponent: p(|y, ) = N (n, n)
ENAR - March 2006 57

with n = (10 + n1y )(1 + n1)1 n = (1 + n1)1

ENAR - March 2006

58

Multivariate normal model - Known

Posterior precision is equal to the sum of prior and sample precisions:

1 1 1 n = 0 + n

Posterior mean is weighted average of prior and sample means:

1 1 1 1 1 n = ( + n y )( 0 0 0 + n )

From usual properties of multivariate normal distribution, can derive marginal posterior distribution of any subvector of or conditional posterior distribution of (1) given (2).
ENAR - March 2006 59

 Let = ((1) , (2) ) and correspondingly, let n (2) n

(1)

n = and n =

n n (2,1) (2,2) n n

(1,1)

(1,2)

ENAR - March 2006

60

Multivariate normal model - Known

Marginal of subvector (1) is p((1)|, y ) = N (n , n Conditional of subvector (1) given (2) is
1|2 (2) (2) 1 |2 p((1)|(2), , y ) = N ((1) + ( ) , ) n n (1) (1,1)

).

where
1 |2

= =

,2) (1 n ,1) (1 n

,2) (2 n

1|2

,2) (1 n

,2) (2 n

,1) (2 n

We recognize 1|2 as the regression coecient in a regression of (1) on (2)

ENAR - March 2006 61

Multivariate normal - Posterior predictive

We seek p( y |y ), and note that p( y , |y ) = N ( y ; , )N (; n, n) Exponential in p( y , |y ) is a quadratic form in ( y , ), so ( y , ) have a normal joint posterior distribution. Marginal p( y |y ) is also normal with mean and variance: E( y |y ) = E(E( y |, y )|y ) = E(|y ) = n var( y |y ) = E(var( y |, y )|y ) + var(E( y |, y )|y ) = E(|y ) + var(|y ) = + n .

ENAR - March 2006

62

Multivariate normal - Sampling from p( y |y )

With known, to draw a value y from p( y |y ) note that p( y |y ) = p( y |, y )p(|y )d

Then: 1. Draw from p(|y ) = N (n, n) 2. Draw y from p( y |, y ) = N (, ) Alternatively (better), draw y directly from p( y |y ) = N (n, + n)

ENAR - March 2006

63

Sampling from a multivariate normal

Want to sample y d 1 from N (, ). Two common approaches Using Cholesky decomposition of : 1. Get A, a lower triangular matrix such that AA = 2. Draw (z1, z2, ..., zd) iid N (0, 1) and let z = (z1, ..., zd) 3. Compute y = + Az

ENAR - March 2006

64

Sampling from a multivariate normal

Using sequential conditional sampling: Use fact that all conditionals in a multivariate normal are also normal. 1. Draw y1 from N (1, (11)) 2. Then draw y2|y1 from N (2|1, 2|1) 3. Etc. For example, if d = 2, 1. Draw y1 from N ((1), 2(1)) 2. Draw y2 from N (
(2)

(12) ( (12))2 (1) 2(2) + 2(2) (y1 ), 2(1) )

65

ENAR - March 2006

Non-informative prior for

A non-informative prior for is obtained by letting the prior precision 0 go to zero. With the uniform prior, the posterior for is proportional to the likelihood. Posterior is proper only if n > d; otherwise, S is not of full column rank. If n > d, p(|, y ) = N ( y , /n)

ENAR - March 2006

66

Multivariate normal - unknown ,

The conjugate family of priors for (, ) is the normal-inverse Wishart family. The inverse Wishart is the multivariate generalization of the inverse 2 distribution
1 If |0, 0 Inv-Wishart0 ( 0 ) then (0 +d+1)/2

p(|0, 0) ||

1 exp tr(01) 2

for positive denite and symmetric. For the Inv-Wishart with 0 degrees of freedom and scale 0: E () = (0 d 1)10
ENAR - March 2006 67

Multivariate normal - unknown ,

Then conjuage prior is
1 Inv-Wishart0 (0 )

| N(0, /0) which corresponds to p(, ) exp ||(0+d)/2+1 1 0 tr(01) ( 0) 1( 0) 2 2

ENAR - March 2006

68

Multivariate normal - unknown ,

Posterior must also be in the Normal-Inv-Wishart form Results from univariate normal generalize directly:
1 |y Inv-Wishartn ( n ) |, y N (n, /n) |y Mult-tnd+1(n, n/(n(n d + 1))) y |y Mult-t

Here: n = n n 0 0 + y 0 + n 0 + n = 0 + n

n = 0 + n
ENAR - March 2006 69

n S

n0 = 0 + S + ( y 0)( y 0) 0 + n =
i

(yi y )(yi y )

ENAR - March 2006

70

Multivariate normal - sampling ,

To sample from p(, |y ) do: (1) Sample from p(|y ) (2) Sample from p(|, y ) To sample from p( y |y ), use drawn (, ) and obtain draw y from N (, ) Sampling from Wishart (S ): 1. Draw independent d 1 vectors 1, 2, ..., from a N (0, S ) 2. Let Q = i=1 ii Method works when > d. Inv-Wishart. If Q Wishart then Q1 =

We have already seen how to sample from a multivariate normal given mean vector and covariance matrix.
ENAR - March 2006 71

Multivariate normal - Jereys prior

If we start from the conjugate normal-inv-Wishart prior and let : 0 0 0 1 |0| 0 then resulting prior is Jereys prior for (, ): p(, ) ||(d+1)/2

ENAR - March 2006

72

Example: bullet lead concentrations

In the US, four major manufacturers produce all bullets red. One of them is Cascade. A sample of 200 round-nosed .38 caliber bullets with lead tips were obtained from Cascade. Concentration of ve elements (antimony, copper, arsenic, bismuth, and silver) in the lead alloy was obtained. Data for for Cascade are stored in federal.data in the courses web site. Assuming that the 200 5 1 observation vectors for Cascade can be represented by a multivariate normal distribution (perhaps after transformation), we are interested in the posterior distribution of the mean vector and of functions of the covariance parameters.
ENAR - March 2006 1

 Particular quantities of interest for inference include: The mean trace element concentrations (1, ..., 5) The correlations between trace element concentrations (11, 12, ..., 45). The largest eigenvalue of the covariance matrix In the data le, columns correspond to antimony, copper, arsenic, bismuth, and silver, in that order. For this example, the antimony concentration was divided by 100

ENAR - March 2006

Bullet lead example - Model and prior

We concentrate on the correlations that involve copper (four of them). Sampling distribution: y |, N (, )

We use the conjugate Normal-Inv-Wishart family of priors, and choose parameters for p(|0, 0) rst: 0,ii = (100, 5000, 15000, 1000, 100) 0,ij = 0 i, j 0 = 7
ENAR - March 2006 3

 For p(|, mu0, 0): 0 = (200, 200, 200, 100, 50) 0 = 10 Low values for 0 and 0 suggest little condence in prior guesses 0, 0 We set 0 to be diagonal apriori: we have some information about the variances of the ve element concentrations, but no information about their correlation. Sampling from posterior distribution: We follow the usual approach:
1 1. Draw from Inv-Wishartn ( n ) 2. Draw from N (n, /n)
ENAR - March 2006 4

3. For each draw, compute: (a) the ratio between 1 (largest eigenvalue of ) and 2 (second largest eigenvalue of ) (b) the four correlations copper,j = copper,j /(copper j ), for j {antimony, arsenic, bismuth, silver}. We are interested in the posterior distributions of the ve means, the eigenvalues ratio, and the four correlation coecients

ENAR - March 2006

Scatter plot of the posterior mean concentrations

ENAR - March 2006

Results from R program

# antimony c(mean(sample.mu[,1]),sqrt(var(sample.mu[,1]))) [1] 265.237302 1.218594 quantile(sample.mu[,1],probs=c(0.025,0.05,0.5,0.95,0.975)) 2.5% 5.0% 50.0% 95.0% 97.5% 262.875 263.3408 265.1798 267.403 267.7005 # copper c(mean(sample.mu[,2]),sqrt(var(sample.mu[,2]))) [1] 259.36335 5.20864 quantile(sample.mu[,2],probs=c(0.025,0.05,0.5,0.95,0.975)) 2.5% 5.0% 50.0% 95.0% 97.5% 249.6674 251.1407 259.5157 268.0606 269.3144 # arsenic c(mean(sample.mu[,3]),sqrt(var(sample.mu[,3]))) [1] 231.196891 8.390751 quantile(sample.mu[,3],probs=c(0.025,0.05,0.5,0.95,0.975))
ENAR - March 2006 7

2.5% 5.0% 50.0% 95.0% 97.5% 213.5079 216.7256 231.6443 243.5567 247.1686 # bismuth c(mean(sample.mu[,4]),sqrt(var(sample.mu[,4]))) [1] 127.248741 1.553327 quantile(sample.mu[,4],probs=c(0.025,0.05,0.5,0.95,0.975)) 2.5% 5.0% 50.0% 95.0% 97.5% 124.3257 124.6953 127.2468 129.7041 130.759 # silver c(mean(sample.mu[,5]),sqrt(var(sample.mu[,5]))) [1] 38.2072916 0.7918199 quantile(sample.mu[,5],probs=c(0.025,0.05,0.5,0.95,0.975)) 2.5% 5.0% 50.0% 95.0% 97.5% 36.70916 36.93737 38.1971 39.54205 39.73169

ENAR - March 2006

Posterior of 1/2 of

ENAR - March 2006

Summary statistics of posterior dist. of ratio

c(mean(ratio.l),sqrt(var(ratio.l))) [1] 5.7183875 0.8522507 quantile(ratio.l,probs=c(0.025,0.05,0.5,0.95,0.975)) 2.5% 5.0% 50.0% 95.0% 97.5% 4.336424 4.455404 5.606359 7.203735 7.619699

ENAR - March 2006

10

Correlation of copper with other elements

ENAR - March 2006

11

Summary statistics for correlations

# j = antimony c(mean(sample.rho[,1]),sqrt(var(sample.rho[,1]))) [1] 0.50752063 0.05340247 quantile(sample.rho[,1],probs=c(0.025,0.05,0.5,0.95,0.975)) 2.5% 5.0% 50.0% 95.0% 97.5% 0.4014274 0.4191201 0.5076544 0.5901489 0.6047564 # j = arsenic c(mean(sample.rho[,3]),sqrt(var(sample.rho[,3]))) [1] -0.56623609 0.04896403 quantile(sample.rho[,3],probs=c(0.025,0.05,0.5,0.95,0.975)) 2.5% 5.0% 50.0% 95.0% 97.5% -0.6537461 -0.6448817 -0.570833 -0.4857224 -0.465808 # j = bismuth c(mean(sample.rho[,4]),sqrt(var(sample.rho[,4]))) [1] 0.63909311 0.04087149 quantile(sample.rho[,4],probs=c(0.025,0.05,0.5,0.95,0.975))
ENAR - March 2006 12

2.5% 5.0% 50.0% 95.0% 97.5% 0.5560137 0.5685715 0.6429459 0.7013519 0.7135556 # j = silver c(mean(sample.rho[,5]),sqrt(var(sample.rho[,5]))) [1] 0.03642082 0.07232010 quantile(sample.rho[,5],probs=c(0.025,0.05,0.5,0.95,0.975)) 2.5% 5.0% 50.0% 95.0% 97.5% -0.09464575 -0.0831816 0.03370379 0.164939 0.1765523

ENAR - March 2006

13

S-Plus code
# # # # # # # # Cascade example We need to define two functions, one to generate random vectors from a multivariate normal distribution and the other to generate random matrices from a Wishart distribution. Note, that a function that generates random matrices from a inverse Wishart distribution is not necessary to be defined since if W ~ Wishart(S) then W^(-1) ~ Inv-Wishart(S^(-1))

# A function that generates random observations from a # Multivariate Normal distribution: rmnorm(a,b) # the parameters of the function are # a = a column vector of kx1 # b = a definite positive kxk matrix # rmnorm_function(a,b) { k_nrow(b) zz_t(chol(b))%*%matrix(rnorm(k),nrow=k)+a zz } # A function that generates random observations from a # Wishart distribution: rwishart(a,b) # the parameters of the function are # a = df # b = a definite positive kxk matrix # Note a must be > k

ENAR - March 2006

14

rwishart_function(a,b) { k_ncol(b) m_matrix(0,nrow=k,ncol=1) cc_matrix(0,nrow=a,ncol=k) for (i in 1:a) { cc[i,]_rmnorm(m,b) } w_t(cc)%*%cc w } # #Read the data # y_scan(file="cascade.data") y_matrix(y,ncol=5,byrow=T) y[,1]_y[,1]/100 means_rep(0,5) for (i in 1:5){ means[i]_mean(y[,i]) } means_matrix(means,ncol=1,byrow=T) n_nrow(y) # #Assign values to the prior parameters # v0_7 Delta0_diag(c(100,5000,15000,1000,100)) k0_10 mu0_matrix(c(200,200,200,100,50),ncol=1,byrow=T) # Calculate the values of the parameters of the posterior # mu.n_(k0*mu0+n*means)/(k0+n) v.n_v0+n k.n_k0+n

ENAR - March 2006

15

ones_matrix(1,nrow=n,ncol=1) S = t(y-ones%*%t(means))%*%(y-ones%*%t(means)) Delta.n_Delta0+S+(k0*n/(k0+n))*(means-mu0)%*%t(means-mu0) # # Draw Sigma and mu from their posteriors # samplesize_200 lambda.samp_matrix(0,nrow=samplesize,ncol=5) #this matrix will store #eigenvalues of Sigma rho.samp_matrix(0,nrow=samplesize,ncol=5) mu.samp_matrix(0,nrow=samplesize,ncol=5) for (j in 1:samplesize) { # Sigma SS = solve(Delta.n) # The following makes sure that SS is symmetric for (pp in 1:5) { for (jj in 1:5) { if(pp<jj){SS[pp,jj]=SS[jj,pp]} } } Sigma_solve(rwishart(v.n,SS)) # The following makes sure that Sigma is symmetric for (pp in 1:5) { for (jj in 1:5) { if(pp<jj){Sigma[pp,jj]=Sigma[jj,pp]} } } # Eigenvalue of Sigma lambda.samp[j,]_eigen(Sigma)$values # Correlation coefficients for (pp in 1:5){ rho.samp[j,pp]_Sigma[pp,2]/sqrt(Sigma[pp,pp]*Sigma[2,2]) } # mu mu.samp[j,]_rmnorm(mu.n,Sigma/k.n)

ENAR - March 2006

16

} # Graphics and summary statistics sink("cascade.output") # Calculate the ratio between max(eigenvalues of Sigma) # and max({eigenvalues of Sigma}\{max(eigenvalues of Sigma)}) # ratio.l_sample.l[,1]/sample.l[,2] # "Histogram of the draws of the ratio" postscript("cascade_lambda.eps",height=5,width=6) hist(ratio.l,nclass=30,axes = F, xlab="eigenvalue1/eigenvalue2",xlim=c(3,9)) axis(1) dev.off() # Sumary statistics of the ratio of the eigenvalues c(mean(ratio.l),sqrt(var(ratio.l))) quantile(ratio.l,probs=c(0.025,0.05,0.5,0.95,0.975)) # # correlations with copper # postscript("cascade_corr.eps",height=8,width=8) par(mfrow=c(2,2)) # "Histogram of the draws of corr(antimony,copper)" hist(sample.rho[,1],nclass=30,axes = F,xlab="corr(antimony,copper)", xlim=c(0.3,0.7)) axis(1) # "Histogram of the draws of corr(arsenic,copper)" hist(sample.rho[,3],nclass=30,axes = F,xlab="corr(arsenic,copper)") axis(1)

ENAR - March 2006

17

# "Histogram of the draws of corr(bismuth,copper)" hist(sample.rho[,4],nclass=30,axes = F,xlab="corr(bismuth,copper)", xlim=c(0.5,.8)) axis(1) # "Histogram of the draws of corr(silver,copper)" hist(sample.rho[,5],nclass=25,axes = F,xlab="corr(silver,copper)", xlim=c(-.2,.3)) axis(1) dev.off() # Summary statistics of the dsn. of correlation of copper with # antimony c(mean(sample.rho[,1]),sqrt(var(sample.rho[,1]))) quantile(sample.rho[,1],probs=c(0.025,0.05,0.5,0.95,0.975)) # arsenic c(mean(sample.rho[,3]),sqrt(var(sample.rho[,3]))) quantile(sample.rho[,3],probs=c(0.025,0.05,0.5,0.95,0.975)) # bismuth c(mean(sample.rho[,4]),sqrt(var(sample.rho[,4]))) quantile(sample.rho[,4],probs=c(0.025,0.05,0.5,0.95,0.975)) # silver c(mean(sample.rho[,5]),sqrt(var(sample.rho[,5]))) quantile(sample.rho[,5],probs=c(0.025,0.05,0.5,0.95,0.975)) # # Means # mulabels=c("Antimony","Copper","Arsenic","Bismuth","Silver") postscript("cascade_means.eps",height=8,width=8) pairs(sample.mu,labels=mulabels) dev.off()

ENAR - March 2006

18

# Summary statistics of the dsn. of mean of # antimony c(mean(sample.mu[,1]),sqrt(var(sample.mu[,1]))) quantile(sample.mu[,1],probs=c(0.025,0.05,0.5,0.95,0.975)) # copper c(mean(sample.mu[,2]),sqrt(var(sample.mu[,2]))) quantile(sample.mu[,2],probs=c(0.025,0.05,0.5,0.95,0.975)) # arsenic c(mean(sample.mu[,3]),sqrt(var(sample.mu[,3]))) quantile(sample.mu[,3],probs=c(0.025,0.05,0.5,0.95,0.975)) # bismuth c(mean(sample.mu[,4]),sqrt(var(sample.mu[,4]))) quantile(sample.mu[,4],probs=c(0.025,0.05,0.5,0.95,0.975)) # silver c(mean(sample.mu[,5]),sqrt(var(sample.mu[,5]))) quantile(sample.mu[,5],probs=c(0.025,0.05,0.5,0.95,0.975)) q()

ENAR - March 2006

19

Advanced Computation
Approximations based on posterior modes Simulation from posterior distributions Markov chain simulation Why do we need advanced computational methods? Except for simpler cases, computation is not possible with available methods: Logistic regression with random eects 2 Normal-normal model with unknown sampling variances j Poisson-lognormal hierarchical model for counts.
ENAR - March 26, 2006 1

Strategy for computation

If possible, work in the log-posterior scale. Factor posterior distribution: p(, |y ) = p( |, y )p(|y ) Reduces to lower-dimensional problem May be able to sample on a grid Helps to identify parameters most inuenced by prior Re-parametrizing sometimes helps: Create parameters with easier interpretation Permit normal approximation (e.g., log of variance or log of Poisson rate or logit of probability)

ENAR - March 26, 2006

Normal approximation to the posterior

It is often reasonable to approximate a complicated posterior distribution using a normal (or a mixture of normals) approximation. Approximation may be a good starting point for more sophisticated methods. Computational strategy: 1. Find joint posterior mode or mode of marginal posterior distributions (better strategy if possible) 2. Fit normal approximations at the mode (or use mixture of normals if posterior is multimodal) Notation: p(|y ): joint posterior of interest (target distribution)
ENAR - March 26, 2006 3

 q (|y ): un-normalized density, typically p()p(y |). = (, ), typically lower dimensional than . Practical advice: computations are often easier with log posteriors than with posteriors.

ENAR - March 26, 2006

Finding posterior modes

To nd the mode of a density, we maximize the function with respect to the parameter(s). Optimization problem. Modes are not interesting per se, but provide the rst step for analytically approximating a density. Modes are easier to nd than means: no integration, and can work with un-normalized density Multi-modal posteriors pose a problem: only way to nd multiple modes is to run mode-nding algorithms several times, starting from dierent locations in parameter space Whenever possible, shoot for nding the mode of marginal and
ENAR - March 26, 2006 5

conditional posteriors. If = (, ), and p(, y ) = p( |, y )p(|y ) then 1. Find mode y) 2. Then nd by maximizing p( | = , Many algorithms to nd modes. Most popular include Newton-Raphson and Expectation-Maximization (EM).

ENAR - March 26, 2006

Taylor approximation to p(|y )

is Second order expansion of log p(|y ) around the mode |y ) + 1 ( ) log p(|y ) log p( 2 d2 log p(|y ) 2 d ). (
=

Linear term in expansion vanishes because log posterior has a zero derivative at the mode. Considering log p(|y ) approximation as a function of , we see that: |y ) is constant and 1. log p( 1 ) log p(|y ) ( 2 d2 log p(|y ) 2 d ) (
=

is proportional to a log-normal density

ENAR - March 26, 2006 7

 in interior of parameter space: Then, for large n and [I ( )]1) p(|y ) N(, d2 I () = 2 log p(|y ), d the observed information matrix. with

ENAR - March 26, 2006

Normal and normal-mixture approximation

For one mode, we saw that V ) pnapprox(|y ) = N(, with )]1 V = [L ( )]1 or [I ( ) is called the curvature of the log posterior at the mode. L ( Suppose now that p(|y ) has K modes. Approximation to p(|y ) is now a mixture of normals: pnapprox(|y )
k
ENAR - March 26, 2006 9

k , V ) k N( k

 The k reect the relative mass associated to each mode k |y )|V |1/2. It can be shown that k = q ( k The mixture-normal approximation is then pnapprox(|y )
k

k |y ) exp{ 1 ( k ) V 1( k )}. q ( k 2

A more robust approximation can be obtained by using a tkernel instead of the normal kernel: ptapprox(|y )
k

k |y )[ + ( k ) V 1( k )](d+ )/2, q ( k

with d the dimension of and relatively low. For most problems, = 4 works well.
ENAR - March 26, 2006 10

Sampling from a mixture approximation

To sample from the normal-mixture approximation: 1. First choose one of the K normal components, using relative probability masses k as multinomial probabilities. 2. Given a component, draw a value from either the normal or the t density. Reasons not to use normal approximation: When mode of parameter is near edge of parameter space (e.g., in SAT example) When even transformation of parameter makes normal approximation crazy. Can do better than normal approximation using more advanced methods
ENAR - March 26, 2006 11

Simulation from posterior - Rejection sampling

Idea is to draw values of p(|y ), perhaps by making use of an instrumental or auxiliary distribution g (|y ) from which it is easier to sample. The target density p(|y ) need only be known up to the normalizing constant. Let q (|y ) = p()p(y ; ) be the un-normalized posterior distribution so that q (|y ) . p(|y ) = p()p(y ; )d Simpler notation: 1. Target density: f (x)
ENAR - March 26, 2006 12

2. Instrumental density: g (x) 3. Constant M such that f (x) M g (x) The following algorithm produces a variable Y that is distributed according to f : 1. Generate X g and U U[0,1] 2. Set Y = X if U f (X )/M g (X ) 3. Reject draw otherwise. Proof: The distribution function of Y is given by: P (Y y ) = P X y |U f (X ) M g (X )
f (X ) M g (X )

P X y, U = P U
ENAR - March 26, 2006

f (X ) M g (X )
13

Then P (Y y ) =
y f (x)/M g (x) dug (x)dx 0 f (x)/M g (x) dug (x)dx 0 y M 1 f (x)dx . 1 f (x)dx M y f (x)dx,

Since the last expression equals

we have proven the result.

In the Bayesian context, q (|y ) (the un-normalized posterior) plays the role of f (x) above. When both f and g are normalized densities: 1. The probability of accepting a draw is 1/M , and expected number of draws until one is accepted is M .
ENAR - March 26, 2006 14

2. For each f , there will be many instrumental densities g1, g2, .... Choose the g that requires smallest bound M 3. M is necessarily larger than 1, and will approach minimum value 1 when g closely imitates f . In general, g needs to have thicker tails than f for f /g to remain bounded for all x. Cannot use a normal g to generate values from a Cauchy f . Can do the opposite, however. Rejection sampling can be used within other algorithms, such as the Gibbs sampler (see later).

ENAR - March 26, 2006

15

Rejection sampling - Getting M

Finding the bound M can be a problem, but consider the following implementation approach recalling that q (|y ) = p()p(y ; ). Let g () = p() and draw a value from the prior. Draw U U (0, 1). ), where is the mode of p(y ; ). Let M = p(y ; Accept the draw if u p()p(y ; ) p(y ; ) q (|y ) = . = M p() p(y ; )p() p(y ; )

Those from the prior that are likely under the likelihood are kept in the posterior sample.
ENAR - March 26, 2006 16

Importance sampling
Also known as SIR (Sampling Importance Resampling), the method is no more than a weighted bootstrap. Suppose we have a sample of draws (1, 2, ..., n) from the proposal distribution g (). Can we convert it to a sample from q (|y )? For each i, compute i = q (i|y )/g (i) wi = i/
j

j .

Draw from the discrete distribution over {1, ..., n} with weight wi on i, without replacement.
ENAR - March 26, 2006 17

 The sample of re-sampled s is approximately a sample from q (|y ). Proof: suppose that is univariate (for convenience). Then:
n

Pr( a)

=
i=1

wi1,a(i) n1
i i 1,a (i ) n1 i i

( |y ) Eg qg ( ) 1,a (i ) ( |y ) Eg qg ( ) q (|y )d a q (|y )d a

p(|y )d.

18

ENAR - March 26, 2006

 The size of the re-sample can be as large as desired. The more g resembles q , the smaller the sample size that is needed for the re-sample to approximate well the target distribution p(|y ). A consistent estimator of the normalizing constant is n1
i

i.

ENAR - March 26, 2006

19

Importance sampling in a dierent context

Suppose that we wish to estimate E [h()] for p(|y ) (e.g., a posterior mean). If N values i can be drawn from p(|y ), can compute Monte Carlo estimate: 1 E [h()] = h(i). N Sampling from p(|y ) may be dicult. But note: h()p(|y ) g ()d. g ()
20

E [h()|y ] =
ENAR - March 26, 2006

 Generate values from g (), and estimate [h()] = 1 E N h(i) p(i|y ) g (i)

The w(i) = p(i|y )/g (i) are the same importance weights from earlier. Will not work if tails of g are short relative to tails of p.

ENAR - March 26, 2006

21

Importance sampling (contd)

Importance sampling is most often used to improve normal approximation to posterior. Example: suppose that you have a normal (or t) approximation to the posterior and use it to draw a sample that you hope approximates a sample from p(|y ). Importance sampling can be used to improve sample: 1. Obtain large sample of size L: (1, ..., L) from the approximation g (). 2. Construct importance weights w(l) = q (l|y )/g (l) 3. Sample k < L values from (1, ..., L) with probability proportional to the weights, without replacement.
ENAR - March 26, 2006 22

 Why without replacement? If weights are approximately constant, can re-sample with replacement. If some weights are large, re-sample would favor values of with large weights repeteadly unless we sample without replacement. Dicult to determine whether importance sampling draws approximate draws from posterior Diagnostic: monitor weights and look for outliers Note: draws from g () can be reused for other p(|y )! Given draws (1, ..., m) from g (), can investigate sensitivity to prior by re-computing importance weights. For posteriors p1(|y ) and p2(|y ), compute (for the same draws of ) pj (i|y ) w (i) = , g (i)
j
ENAR - March 26, 2006

j = 1, 2.
23

Markov chain Monte Carlo

Methods based on stochastic process theory, useful for approximating target posterior distributions. Iterative: must decide when convergence has happened Quite general, can be implemented where other methods fail Can be used even in high dimensional examples Three methods: Metropolis-Hastings, Metropolis and Gibbs sampler. M and GS special cases of M-H.

ENAR - March 26, 2006

24

Markov chains
A process Xt in discrete time t = 0, 1, 2, ..., T where E (Xt|X0, X1, ..., Xt1) = E (Xt|Xt1) is called a Markov chain. A Markov chain is irreducible if it is possible to reach all states from any other state: pn(j |i) > 0, pm(i|j ) > 0, m, n > 0

where pn(j |i) denotes probability of getting to state j from state i in n steps.
ENAR - March 26, 2006 25

 A Markov chain is periodic with period d if pn(i|i) = 0 unless n = kd for some integer k . If d = 2, the chain returns to i in cycles of 2k steps. If d = 1, the chain is aperiodic. Theorem: Finite-state, irreducible, aperiodic Markov chains have a limiting distribution: limn pn(j |i) = , with pn(j |i) the probability that we reach j from i after n steps or transitions.

ENAR - March 26, 2006

26

Properties of Markov chains (contd)

Ergodicity: If a MC is irreducible and aperiodic and has stationary distribution then we have ergodicity: a n = 1 n a(Xt)
t

E {a(X )} as n . a n is an ergodic average. Also, rate of convergence can be calculated and is geometric.

ENAR - March 26, 2006

27

Numerical standard error

Sequence {X1, X2, ..., Xn} is not iid. The asymptotic standard error of an is approximately
2 a {1 + 2 n

i(a)}
i

where i is the ith lag autocorrelation in the function a{Xt}.

2 First term a /n is the usual sampling variance under iid sampling.

Second term is a penalty for the fact that sample is not iid and is usually bigger than 1. Often, the standard error is computed assuming a nite number of lags.
ENAR - March 26, 2006 28

Markov chain simulation

Idea: Suppose that sampling from p(|y ) is hard, but that we can generate (somehow) a Markov chain {(t), t T } with stationary distribution p(|y ). Situation is dierent from the usual stochastic process case: Here we know the stationary distribution. We seek an algorithm to transition from (t) to (t+1)) and that will take us to the stationary distribution. Idea: start from some initial guess 0 and let the chain run for n steps (n large), so that it reaches its stationary distribution. After convergence, all additional steps in the chain are draws from the stationary distribution p(|y ).
ENAR - March 26, 2006 29

 MCMC methods all based on the same idea; dierence is just in how the transitions in the MC are created. In MCMC simulation, we generate at least one MC for each parameter in the model. Often, more than one (independent) chain for each parameter

ENAR - March 26, 2006

30

The Gibbs Sampler

An iterative algorithm that produces Markov chains with joint stationary distribution p(|y ) by cycling through all possible conditional posterior distributions. Example: suppose that = (1, 2, 3), and that the target distribution is p(|y ). Steps in the Gibbs sampler are: 1. 2. 3. 4.
0 0 0 Start with a guess (1 , 2 , 3 ) 1 0 0 Draw 1 from p(1|2 = 2 , 3 = 3 , y) 1 1 0 Draw 2 from p(2|1 = 1 , 3 = 3 , y) 1 1 1 Draw 3 from p(3|1 = 1 , 2 = 2 , y)

Steps above complete one iteration of the GS Repeat the steps above n times, and after convergence (see later), draws (n+1, n+2, ...) are sample from stationary distribution p(|y ).
ENAR - March 26, 2006 31

The Gibbs Sampler (contd)

Baby example: = (1, 2) are bivariate normal with mean y = (y1, y2), 2 2 variances 1 = 2 = 1 and covariance . Then: p(1|2, y ) N (y1 + (2 y2), 1 2) and p(2|1, y ) N (y2 + (1 y1), 1 2) In this case, would not need GS, just for illustration See gure: y1 = 0, y2 = 0, and = 0.8.

ENAR - March 26, 2006

32

Example: normal

Gibbs sampling in the bivariate

Just as illustration, we consider the trivial problem of sampling from the posterior distribution of a bivariate normal mean vector. Suppose that we have a single observation vector (y1, y2) where y1 y2 N 1 2 , 1 1 . (1)

With a uniform prior on (1, 2), the joint posterior distribution is 1 |y 2

KSU - April 2005

y1 y2

1 1

(2)
1

 The conditional distributions are 1|2, y 2|1, y N(y1 + (2 y2), 1 2) N(y2 + (1 y1), 1 2)

We generate four independent chains, starting from four dierent corners of the 2-dimensional parameter space.

KSU - April 2005

After 50 iterations..

After 1000 iterations, and eliminating the path lines.

The non-conjugate normal example

Let y N (, 2), with (, 2) unknown. Consider the semi-conjugate prior:
2 N (0, 0 ) 2 2 Inv 2(0, 0 ).

Joint posterior distribution is p(, 2) ( ) e

( 1 2 2 n 2 2

P(y ) ) (
i 2

1 ( )2 ) 0 2 20

( )

2 0 0 0 ( ) 2 ( 2 +1)) 2 2

ENAR - March 26, 2006

33

Non-conjugate example (contd)

Derive full conditional distributions p(| 2, y ) and p( 2|, y ).

For : 1 1 1 p(| 2, y ) exp{ [ 2 (yi )2 + 2 ( 0)2]} 2 0 1 2 2 + 2)2 2(n0 y + 20)]} exp{ [(n0 2 2 2 1 n0 + 2 2 n0 y + 20 exp{ ]} 2 [ 2 2 + 2 2 2 0 n0
2 N (n, n ),
ENAR - March 26, 2006 34

where n =

n 1 y + 2 2 0 0 1 n + 2 2 0

and

2 n

n 2

1 . + 12
0

ENAR - March 26, 2006

35

Non-conjugate normal example (contd)

The full conditional for 2 is p( |, y )
2
n+ 1 2 ( 2 0 +1) ( ) exp{ [

2 (yi )2 + 00 ]}

2 Inv 2(n, n ),

where n = 0 + n
2 2 0 = nS 2 + 00 ,

and S = n1 (yi )2.

ENAR - March 26, 2006

36

Non-conjugate normal example (contd)

For the non-conjugate normal model, Gibbs sampling consists of following steps: 1. Start with a guess for 2, 2(0). 2. Draw (1) from a normal 2 2(0) (n( 2(0)), n ( )), where
n

with

mean

and

variance

n( 2(0)) =

2(0)

y + 12 0
0

n 2(0)

+ 12
0

and
2 2(0) ( ) n

1 = n 1 . + 2 2(0)
0
37

ENAR - March 26, 2006

2 3. Next draw 2(1) from Inv 2(n, n ((1))), where 2 2 n ((1)) = nS 2((1)) + 00 ,

and S 2((1)) = n1 4. Repeat many times. (yi (1))2.

ENAR - March 26, 2006

38

A more interesting example

Poisson counts, with a change point: consider a sample of size n of counts (y1, ..., yn) distributed as Poisson with some rate, and suppose that the rate changes, so that For i = 1, ..., m , For i = m + 1, ..., n , and m is unknown. Priors on (, , m): Ga(, ) Ga(, ) yi Poi() yi Poi()

m U[1,n]
ENAR - March 26, 2006 39

 Joint posterior distribution:

m n

p(, , m|y )
i=1

eyi
i=m+1

eyi 1e 1e n1

y1 +1e(m+ )y2 + 1e(nm+)

with y1 = m i=1 yi and y2 = n i=m+1 yi .

Note: if we knew m, problem would be trivial to solve. Thus Gibbs, where we condition on all other parameters to determine Markov chains, appears to be the right approach.

To implement Gibbs sampling, need full conditional distributions. Pick and choose pieces that depend on each parameter
ENAR - March 26, 2006 40

 Conditional of : p(|m, , y )

m i=1 yi +1

exp((m + ))

Ga( +
i=1

yi , m + )

Conditional of :

p(|, m, y )

n i=m+1 yi + 1

exp((n m + ))

Ga( +
i=m+1

yi , n m + )

Conditional of m = 1, 2, ..., n: p(m|, , y ) = c1q (m|, )

ENAR - March 26, 2006 41

1 y1 c +1 exp((m

+ ))

y2 + 1

exp((n m + )).

Note that all terms in joint posterior depend on m, so conditional of m is proportional to joint posterior. Distribution does not look like any standard form so cannot sample directly. Need to obtain normalizing constant, easy to do for relatively small n and for this discrete case:
n

c=
k=1

q (k |, , y ).

To sample from p(m|, , y ) can use inverse cdf on a grid, or other methods.

ENAR - March 26, 2006

42

Non-standard distributions
It may happen that one or more of the full conditionals is not a standard distribution What to do then? Try direct simulation: grid approximation, rejection sampling Try approximation: normal or t approximation, need mode at each iteration (see later) Try more general Markov chain algorithms: Metropolis or MetropolisHastings.

ENAR - March 26, 2006

43

Metropolis-Hastings algorithm
More exible transition kernel: rather than requiring sampling from conditional distributions, M-H permits using many other proposal densities Idea: instead of drawing sequentially from conditionals as in Gibbs, M-H jumps around the parameter space The algorithm is the following: 1. Given a draw t in iteration t, sample a candidate draw from a proposal distribution J (|) 2. Accept the draw with probability p(|y )/J (|) r= . p(|y )/J (| )
ENAR - March 26, 2006 44

3. Stay in place (do not accept the draw) with probability 1 r, i.e., (t+1) = (t). Remarkably, the proposal distribution (text calls it jump distribution) can have just about any form. When proposal distribution is symmetric, i.e. J (|) = J (|), Metropolis-Hastings acceptance probability is p(|y ) r= . p(|y ) This is the Metropolis algorithm
ENAR - March 26, 2006 45

Proposal distributions
Convergence does not depend on J , but rate of convergence does. Optimal J is p(|y ) in which case r = 1. Else, how do we choose J ? 1. It is easy to get samples from J 2. It is easy to compute r 3. It leads to rapid convergence and mixing: jumps should be large enough to take us everywhere in the parameter space but not too large so that draw is accepted (see gure from Gilks et al. (1995)). Three main approaches: random walk M-H (most popular), independence sampler, and approximation M-H
ENAR - March 26, 2006 46

Independence sampler
Proposal distribution Jt does not depend on t. Just nd a distribution g () and generate values from it Can work very well if g (.) is a good approximation to p(|y ) and g has heavier tails than p. Can work awfully bad otherwise Acceptance probability is p(|y )/g () r= p(|y )/g ()
ENAR - March 26, 2006 47

 With large samples (central limit theorem operating) proposal could be normal distribution centered at mode of p(|y ) and with variance larger than inverse of Fisher information evaluated at mode.

ENAR - March 26, 2006

48

Random walk M-H

Most popular, easy to use. Idea: generate candidate using random walk. Proposal is often normal centered at current draw: J (|(t1)) = N (|(t1), V ) Symmetric: think of drawing values (t1) from a N (0, V ). Thus, r simplies. Dicult to choose V : 1. V too small: takes long to explore parameter space
ENAR - March 26, 2006 49

2. V too large: jumps to extremes are less likely to be accepted. Stay in the same place too long 3. Ideal V : posterior variance. Unknown, so might do some trial and error runs Optimal acceptance rate (from some theory results) are between 25% and 50% for this type of proposal distribution. Gets lower with higher dimensional problem.

ENAR - March 26, 2006

50

Approximation M-H
Idea is to improve approximation of J to p(|y ) as we know more about . E.g., in random walk M-H, can perhaps increase acceptance rate by considering J (|(t1)) = N (|(t1), V(t1) ) variance also depends on current draw Proposals here typically not symmetric, requires full r expression.

ENAR - March 26, 2006

51

Starting values
If chain is irreducible, choice of 0 will not aect convergence. With multiple chains (see later), can choose over-dispersed starting values for each chain. Possible algorithm: 1. Find posterior modes 2. Create over-dispersed approximation to posterior at mode (e.g., t4) 3. Sample values from that distribution Not much research on this topic.

ENAR - March 26, 2006

52

How many chains?

Even after convergence, draws from stationary distribution are correlated. Sample is not i.i.d. An iid sample of size n can be obtained by keeping only last draw from each of n independent chains. Too inecient. Compromise: If autocorrelation at lag k and larger is negligible, then can generate an almost i.i.d. sample by keeping only every k th draw in the chain after convergence. Need a very long chain. To check convergence (see later) multiple chains can be generated independently (in parallel). Burn-in: iterations required to reach (approximately). Not used for inference.
ENAR - March 26, 2006

stationary

distribution

53

Convergence
Impossible to decide whether chain has converged, can only monitor behavior. Easiest approach: graphical displays (trace plots in WinBUGS). A bit more formal (and most popular in terms of use): and Rubin (1992). ) of Gelman (R

To use the G-R diagnostic, must generate multiple independent chains for each parameter. The G-R diagnostic compares the within-chain sd to the between-chain sd.
ENAR - March 26, 2006 54

 Before convergence, the within-chain sd is an underestimate of the sd(|y ), and between-chain sd is overestimate. If chains converge, all draws are from stationary distribution, so within and between chain variances should be similar. Consider scalar parameter and let ij be j th draw in ith chain, with i = 1, ..., n and j = 1, ..., J Between-chain variance: B= n J 1 ( j ..)2, j = 1 n ij , .. = 1 j J

Within-chain variance: W =
ENAR - March 26, 2006

1 J (n 1)

(ij j )2.
55

 An unbiased estimate of var( |y ) is weighted average: 1 n1 W+ B var ( |y ) = n n Early in iterations, var ( |y ) overestimates true posterior variance Diagnostic measures potential reduction in the scale if iterations are continued: ( |y ) ) = ( var ) (R W which goes to 1 as n .

ENAR - March 26, 2006

56

Hierarchical models - Introduction

Consider the following study: J counties are sampled from the population of 99 counties in the state of Iowa. Within each county, nj farms are selected to participate in a fertilizer trial Outcome yij corresponds to ith farm in j th county, and is modeled as p(yij |j ). We are interested in estimating the j . Two obvious approaches: 1. Conduct J separate analyses and obtain p(j |yj ): all s are independent 2. Get one posterior p(|y1, y2, ..., yJ ): point estimate is same for all s.
ENAR - March 26, 2006 1

 Neither approach appealing. Hierarchical approach: view the s as a sample from a common population distribution indexed by a parameter . Observed data yij can be used to draw inferences about s even though the j are not observed. A simple hierarchical model: p(y, , ) = p(y |)p(|)p().

p(y |) = p(|) = p() =

Usual sampling distribution Population dist. for or prior Hyperprior

ENAR - March 26, 2006

Hierarchical models - Rat tumor example

Imagine a single toxicity experiment performed on rats. is probability that a rat receiving no treatment develops a tumor. Data: n = 14, and y = 4 develop a tumor. From the sample: tumor rate is 4/14 = 0.286. Simple analysis: y | |,
ENAR - March 26, 2006

Bin(n, ) Beta(, )
3

 Posterior for is p(|y ) = Beta( + 4, + 10) Where can we get good guesses for and ? One possibility: from the literature, look at data from similar experiments. In this example, information from 70 other experiments is available. In j th study, yj is the number of rats with tumors and nj is the sample size, j = 1, .., 70. See Table 5.1 and Figure 5.1 in Gelman et al.. Model the yj as independent binomial data given the study-specic j and sample size nj
ENAR - March 26, 2006 4

From Gelman et al.: hierarchical representation of tumor experiments

 Representation of hierarchical model in Figure 5.1 assumes that the Beta(, ) is a good population distribution for the j . Empirical Bayes as a rst step to estimating mean tumor rate in experiment number 71: 1. Get point estimates of , from earlier 70 experiments as follows: 70 2. Mean tumor rate is r = (70)1 j =1 yj /nj and standard deviation is [(69)1 j (yj /nj r )2]1/2. Values are 0.136 and 0.103 respectively. Using method of moments: = 0.136, + = /0.136 + 2 = 0 . 103 ( + )2( + + 1) Resulting estimate for (, ) is (1.4, 8.6).
ENAR - March 26, 2006 5

 Then posterior is p(|y ) = Beta(5.4, 18.6) with posterior mean 0.223, lower than the sample mean, and posterior standard deviation 0.083. Posterior point estimate is lower than crude estimate; indicates that in current experiment, number of tumors was unusually high. Why not go back now and use the prior to obtain better estimates for tumor rates in earlier 70 experiments? Should not do that because: 1. Cant use the data twice: we use the historical data to get the prior, and cannot now combine that prior with the data from the experiments for inference 2. Using a point estimate for (, ) suggests that there is no uncertainty about (, ): not true
ENAR - March 26, 2006 6

3. If the prior Beta(, ) is the appropriate prior, shouldnt we know the values of the parameters prior to observing any data? Approach: place a hyperprior on the tumor rates j with parameters (, ). Can still use all the data to estimate the hyperparameters. Idea: Bayesian analysis on the joint distribution of all parameters (1, ..., 71, , |y )

ENAR - March 26, 2006

Hierarchical models - Exchangeability

J experiments. In experiment j , yj p(yj |j ). To create joint probability model, use idea of exchangeability Recall: a set of random variables (z1, ..., zK ) are exchangeable if their joint distribution p(z1, ..., zK ) is invariant to permutations of their labels. In our set-up, we have two opportunities for assuming exchangeability: 1. At the level of data: conditional on j , the yij s are exchangeable if we cannot distinguish between them 2. At the level of the parameters: unless something (other than the data) distinguishes the s, we assume that they are exchangeable as well.
ENAR - March 26, 2006 8

 In rat tumor example, we have no information to distinguish between the 71 experiments, except sample size. Since nj is presumably not related to j , we choose an exchangeable model for the j . Simplest exchangeable distribution for the j : p(|) = J j =1 p(j |). The hyperparameter is typically unknown, so marginal for must be obtained by averaging: p() = J j =1 p(j |)p()d.

Exchangeable distribution for (1, ..., J ) is written in the form of a mixture distribution
ENAR - March 26, 2006 9

 Mixture model characterizes parameters (1, ..., J ) as independent draws from a superpopulation that is determined by unknown parameters Exchangeability does not hold when we have additional information on covariates xj to distinguish between the j . Can still model exchangeability with covariates: p(1, ..., J |x1, ..., xJ ) = J j =1 p(j |xj , )p(|x)d,

with x = (x1, ..., xJ ). In Iowa example, we might know that dierent counties have very dierent soil quality. Exchangeability does not imply that j are all the same; just that they can be assumed to be draws from some common superpopulation distribution.
ENAR - March 26, 2006 10

Hierarchical models - Bayesian treatment

Since is unknown, posterior is now p(, |y ). Model formulation: p(, ) = p()p(|) = joint prior p(, |y ) p(, )p(y |, ) = p(, )p(y |) The hyperparameter gets its own prior distribution. Important to check whether posterior is proper when using improper priors in hierarchical models!

ENAR - March 26, 2006

11

Posterior predictive distributions

There are two posterior predictive distributions potentially of interest: 1. Distribution of future observations y corresponding to an existing j . Draw y from the posterior predictive given existing draws for j . j s drawn 2. Distribution of new observations y corresponding to new from the same superpopulation. First draw from its posterior, then for a new experiment, and then draw y draw from the posterior . predictive given the simulated In rat tumor example: 1. More rats from experiment #71, for example 2. Experiment #72 and then rats from experiment #72

ENAR - March 26, 2006

12

Hierarchical models - Computation

Harder than before because we have more parameters Easiest when population distribution p(|) is conjugate to the likelihood p(|y ). In non-conjugate models, must use more advanced computation. Usual steps: 1. Write p(, |y ) p()p(|)p(y |) 2. Analytically determine p(|y, ). Easy for conjugate models. 3. Derive the marginal p(|y ) by p(|y ) =
ENAR - March 26, 2006

p(, |y )d,
13

or, if convenient, by p(|y ) =

p(, |y ) p( |y,) .

Normalizing constant in denominator may depend on as well as y . Steps for computation in hierarchical models are: 1. Draw vector from p(|y ). If is low-dimensional, can use inverse cdf method as before. Else, need more advanced methods 2. Draw from conditional p(|, y ). If the j are conditionally independent, p(|, y ) factors as p(|, y ) = j p(j |, y ) so components of can be drawn one at a time. 3. Draw y from appropriate posterior predictive distribution.

ENAR - March 26, 2006

14

Rat tumor example

Sampling distribution for data from experiments j = 1, ..., 71: yj Bin(nj , j ) Tumor rates j assumed to be independent draws from Beta: j Beta(, ) Choose a non-informative prior for (, ) to indicate prior ignorance. Since hyperprior will be non-informative, and perhaps improper, must check integrability of posterior.
ENAR - March 26, 2006 15

 Defer choice of p(, ) until a bit later.

Joint posterior distribution: p(, , |y ) p(, )p(|, )p(y |, , ) ( + ) 1 y j (1 j ) 1j j j (1 j )nj yj . ()( )

p(, ) j

Conditional of : notice that given (, ), the j are independent, with Beta distributions: ( + + nj ) +y 1 p(j |, , y ) = j j (1 j ) +nj yj 1. ( + yj )( + nj yj )
ENAR - March 26, 2006 16

 Marginal posterior distribution of hyperparameters is obtained using p(, |y ) = p(, , |y ) p(|y, , )

Substituting into expression above, we get p(, |y ) p(, )j ( + ) ( + yj )( + nj yj ) ()( ) ( + + nj )

Not a standard distribution, but easy to evaluate and only in two dimensions. Can evaluate p(, |y ) over a grid of values of (, ), and then use inverse cdf method to draw from marginal and from conditional.
ENAR - March 26, 2006 17

 But what is p(, )? As it turns out, many of the obvious choices for the prior lead to improper posterior. (See solution to Exercise 5.7, on the course web site.) Some obvious choices such as p(, |y ) 1 lead to non-integrable posterior. Integrability can be checked analytically by evaluating the behavior of the posterior as , (or functions) go to . An empirical assessment can be made by looking at the contour plot of p(, |y ) over a grid. Signicant mass extending towards innity suggests that the posterior will not integrate to a constant. Other choices leading to non-integrability of p(, |y ) include:
ENAR - March 26, 2006 18

 A at prior on prior guess and degrees of freedom p( , + ) 1. + A at prior on the log(mean) and log(degrees of freedom) p(log(/ ), log( + )) 1. A reasonable choice for the prior is a at distribution on the prior mean and square root of inverse of the degrees of freedom: p( , ( + )1/2) 1. + This is equivalent to p(, ) ( + )5/2.
ENAR - March 26, 2006 19

 Also equivalent to p(log(/ ), log( + ) ( + )5/2. We use the parameterization (log(/ ), log( + )). Idea for drawing values of and from their posterior is the usual: evaluate p(log(/ ), log( + )|y ) over grid of values of (, ), and then use inverse cdf method. For this problem, easier to evaluate the log posterior and then exponentiate. Grid: From earlier estimates, potential centers for the grid are = 1.4, = 8.6. In the new parameterization, this translates into u = log(/ ) = 1.8, v = log( + ) = 2.3.
ENAR - March 26, 2006 20

 Grid too narrow leaves mass outside. Try [2.3, 1.3] [1, 5]. Steps for computation: 1. Given draws (u, v ), transform back to (, ). 2. Finally, sample j from Beta( + yj , + nj yj )

ENAR - March 26, 2006

21

From Gelman et al.:

Contours of joint posterior distribution of alpha and beta in reparametrized scale

Posterior means and 95% credible sets for i

Normal hierarchical model

What we know as one-way random eects models Set-up: J independent experiments, in each want to estimate a mean j . Sampling model: yij |j , 2 N(j , 2) for i = 1, ..., nj , and j = 1, ..., J . Assume for now that 2 is known.
1 If y .j = n j i yij ,

then
2 y .j |j N(j , j )

ENAR - March 26, 2006

22

2 with j = 2/nj .

Sampling model for y .j is quite general. For nj large, y .j is normal even if yij is not. Need now to think of priors for the j . What type of posterior estimates for j might be reasonable? 1. 2. j = y .j : reasonable if nj large 2 j = y .. = ( j 2)1 j j y .j : pooled estimate, reasonable if we believe that all means are the same.

To decide between those two choices, do an F-test for dierences between groups. ANOVA approach (for nj = n):
ENAR - March 26, 2006 23

Source Between groups Within groups

df J-1 J(n-1)

SS SSB SSE

MS SSB / (J-1) SSE / J(n-1)

E(MS) n 2 + 2 2

where 2, the variance of the group means can be estimated as 2 = M SB M SE n

If M SB >>> M SE then 2 > 0 and F-statistic is signicant: do not j = y pool and use .j . Else, F-test cannot reject H0 : 2 = 0, and must pool. Alternative: why not a more general estimator: j = j y .j + (1 j ) y..
ENAR - March 26, 2006 24

for j [0, 1]. Factor (1 j ) is a shrinkage factor. All three estimates have a Bayesian justication: j = y 1. .j is posterior mean of j if sample means are normal and p(j ) 1 j = y 2. .. is posterior mean if 1 = ... = J and p() 1. 3. j = j y .j + (1 j ) y.. is posterior mean if j N(, 2) independent of other s and sampling distribution for y .j is normal Latter is called the Normal-Normal model.

ENAR - March 26, 2006

25

Normal-normal model
Set-up (for 2 known): y .j |j 1, ..., J |, p(, 2) It follows that p(1, ..., J ) =
j

2 N(j , j )

N(, 2)
j

N(j ; , 2)p(, 2)dd 2

The joint prior can be written as p(, ) = p(| )p( ). For , we will
ENAR - March 26, 2006 26

consider a contidional at prior, so that p(, ) p( ) Joint posterior p(, , |y ) p(, )p(|, )p(y |) p(, )
j

N(j ; , )
j

2 N( y.j ; j , j )

Conditional distribution of group means For p(| ) 1, the conditional distributions of the j given , , y .j are independent, and j , Vj ) p(j |, , y .j ) = N(
ENAR - March 26, 2006 27

with

2 j + 2y .j j = , 2 2 j +

Vj1

1 1 = 2+ 2 j

Marginal distribution of hyperparameters To get p(, |y ) we would typically either integrate p(, , |y ) with respect to 1, ..., J , or would use the algebraic approach p(, |y ) = p(, , |y ) p(|, , y )

In the normal-normal model, data provide information about , , as shown below. Consider the marginal posterior: p(, |y ) p(, )p(y |, )
ENAR - March 26, 2006 28

where p(y |, ) is the marginal likelihood: p(y |, ) = =

j

p(y |, , )p(|, )d
2 N( y.j ; j , j ) j

N(j ; , )d1, ..., dJ

Integrand above is the product of quadratic functions in y .j and j , so they are jointly normal. Then the y .j |, are also normal, with mean and variance: E ( y.j |, ) = E [E ( y.j |j , , )|, ] = E (j |, ) = var( y.j |, ) = E [var( y.j |j , , )|, ]
ENAR - March 26, 2006 29

+var[E ( y.j |j , , )|, ]

2 = E (j |, ) + var(j |, ) 2 = j + 2

Therefore p(, |y ) p( )
j 2 N( y.j ; , j + 2)

We know that p(, |y ) p( )p(| )

j 2 (j + 2)1/2 exp

1 2 ( y ) .j 2 + 2) 2(j

Now x , and think of p( y.j |) as the likelihood in a problem in

ENAR - March 26, 2006 30

which is the only parameter. Recall that p(| ) 1. From earlier, we know that p(|y, ) = N( , V) where =
j 2 y .j /(j + 2)

2 2 j 1/(j + )

V =
j

1 2 + 2 j

To get p( |y ) we use the old trick: p( |y ) =

ENAR - March 26, 2006

p(, |y ) p(|, y ) p( )
j 2 N( y.j ; , j + 2)

N(; , V )
31

 Expression holds for any , so set = . Denominator is then just 1/2 V . Then
1 /2 p( |y ) p( )V j 2 N( y.j ; , j + 2).

What do we use for p( )? Safe choice is always a proper prior. For example, consider
2 ), p( ) = Inv 2(0, 0

where
2 0 can be best guess for 0 can be small to reect prior uncertainty.
ENAR - March 26, 2006 32

 Non-informative (and improper) choice: Beware. Improper prior in hierarchical model can easily lead to non-integrable posterior. In normal-normal model, natural non-informative p( ) 1 or equivalently p(log( )) 1 results in improper posterior. p( ) 1 leads to proper posterior p( |y ).

ENAR - March 26, 2006

33

Normal-normal model - Computation

1. Evaluate the one-dimensional p( |y ) on a grid. 2. Sample from p( |y ) using inverse cdf method. 3. Sample from N( , V) j , Vj ) 4. Sample j from N(

ENAR - March 26, 2006

34

Normal-normal model - Prediction

Predicting future data y from the current experiments with means = (1, ..., J ): 1. Obtain draws of (, , 1, ..., J ) 2 2. Draw y from N(j , j ) and Predicting future data y from a future experiment with mean sample size n : 1. Draw , from their posterior from the population distribution p(|, ) (also known as 2. Draw the prior for ) 3. Draw y from N(, 2), where 2 = 2/n

ENAR - March 26, 2006

35

Example: eect of diet on coagulation times

Normal hierarchical model for coagulation times of 24 animals randomized to four diets. Model: Observations: yij with i = 1, ..., nj , j = 1, ..., J . Given , coagulation times yij are exchangeable Treatment means are normal N (, 2), and variance 2 is constant across treatments Prior for (, log , log ) . A uniform prior on log leads to improper posterior. p(, , log , log |y ) j N(j |, 2) j i N(yij |j , 2) j = n1 Crude initial estimates: j
ENAR - March 26, 2006

yij = y .j , = J 1

y .j = y ..,
36

2 .j )2, 2 = J 1 j = (nj 1)1 (yij y 1)1 ( y.j y ..)2.

2 j , and 2 = (J

Data The following table contains data that represents coagulation time in seconds for blood drawn from 24 animals randomly allocated to four dierent diets. Dierent treatments have dierent numbers of observations because the randomization was unrestricted.

Diet
A B C D

Measurements
62,60,63,59 63,67,71,64,65,66 68,66,71,67,68,68 56,62,60,61,63,64,63,59

ENAR - March 26, 2006

37

Conditional maximization for joint mode

Conjugacy makes conditional maximization easy. Conditional modes of treatment means are j , Vj ) j |, , , y N( with
2 2 / + n y / j .j j = , 2 2 1/ + nj /

and Vj1 = 1/ 2 + nj / 2. j in place of For j = 1, ..., J , maximize conditional posteriors by using current estimate j .
ENAR - March 26, 2006 38

 Conditional mode of : |, , , y N( , 2/J ), = J 1 j .

Conditional maximization: replace current estimate with .

ENAR - March 26, 2006

39

Conditional maximization
Conditional mode of log : rst derive conditional posterior for 2: 2|, , , y Inv 2(n, 2), with 2 = n1 (yij j )2.

The mode of the Inv 2 is 2n/(n + 2). To get mode for log , use transformation. Term n/(n + 2) disappears with Jacobian, so conditional mode of log is log . Conditional mode of log : same reasoning. Note that 2|, , 2, y Inv 2(J 1, 2), with 2 = (J 1)1 (j )2. After accounting for Jacobian of transformation, conditional mode of log is log .
ENAR - March 26, 2006 40

Conditional maximization
Starting from crude estimates, conditional maximization required only three iterations to converge approximately (see table) Log posterior increased at each step Values in nal iteration is approximate joint mode. When J is large relative to the nj , joint mode may not provide good summary of posterior. Try to get marginal modes. In this problem, factor: p(, , log , log |y ) = p(|, log , log , y )p(, log , log |y ). Marginal of , log , log is three-dimensional regardless of J and nj .
ENAR - March 26, 2006 41

Conditional maximization results

Table shows iterations for conditional maximization. Crude estimate 61.000 66.000 68.000 61.000 64.000 2.291 3.559 -61.604 First iteration 61.282 65.871 67.742 61.148 64.010 2.160 3.318 -61.420 Stepwise ascent Second Third iteration iteration 61.288 61.290 65.869 65.868 67.737 67.736 61.152 61.152 64.011 64.011 2.160 2.160 3.318 3.312 -61.420 -61.420 Fourth iteration 61.290 65.868 67.736 61.152 64.011 2.160 3.312 -61.420

Parameter 1 2 3 4 log p(params.|y )

ENAR - March 26, 2006

42

Marginal maximization
Recall algebraic trick: p(, log , log |y ) = p(, , log , log |y ) p(|, log , log , y ) j N(j |, 2)j i N(yij |j , 2) j , Vj ) j N(j |

in place of : Using p(, log , log |y ) j N(j |, 2)j iN(yij |j , 2)j Vj and Vj as earlier. with
ENAR - March 26, 2006 43

1 /2

 Can maximize p(, log , log |y ) using EM. Here, (j ) are the missing data. Steps in EM: 1. Average over missing data in E-step 2. Maximize over (, log , log ) in M-step

ENAR - March 26, 2006

44

EM algorithm for marginal mode

Log joint posterior: log p(, , log , log |y ) n log (J 1) log 1 1 2 2 2 (j ) ( y ) ij j 2 j 2 2 j i E-step: Average over using conditioning trick (and p(| rest). Need two expectations: Eold[(j )2] = E[(j )2|old, old, old, y ] = [Eold(j )]2 + varold(j ) j )2 + Vj = ( Similarly: j )2 + Vj . Eold[(yij j )2] = (yij
ENAR - March 26, 2006 45

EM algorithm for marginal mode (contd)

M-step: Maximize the expected log posterior (expectations just taken in E-step) with respect to (, log , log ). Dierentiate and equate to zero to get maximizing values (new , log new , log new ). Expressions are: 1 new j . = J j 1/2 1 j )2 + Vj ] new = [(yij n j i and new = 1 J 1
j

1/2 j new )2 + Vj ] [( .

ENAR - March 26, 2006

46

EM algorithm for marginal mode (contd)

Beginning from joint mode, algorithm converged in three iterations. Important to check that log posterior increases at each step. Else, programming or formulation error! Results: Parameter Value at joint mode 64.01 2.17 3.31 First iteration 64.01 2.33 3.46 Second iteration 64.01 2.36 3.47 Third iteration 64.01 2.36 3.47

ENAR - March 26, 2006

47

Using EM results in simulation

In a problem with standard form such as normal-normal, can do the following: 1. Use EM to nd marginal and conditional modes 2. Approximate posterior at the mode using N or t approximation 3. Draw values of parameters from papprox, and act as if they were from p. Importance re-sampling can be used to improve accuracy of draws. For each draw, compute importance weights, and re-sample draws (without replacement) using probability proportional to weight. In diet and coagulation example, approximate approach as above likely to produce quite reasonable results. But must be careful, specially with scale parameters.
ENAR - March 26, 2006 48

Gibbs sampling in normal-normal case

The Gibbs sampler can be easily implemented in the normal-normal example because all posterior conditionals are of standard form. Refer back to Gibbs sampler discussion, and see derivation of conditionals in hierarchical normal example. Full conditionals are: j |all N (J of them) |all N 2|all Inv 2 2|all Inv 2.

Starting values for the Gibbs sampler can be drawn from, e.g., a t4 approximation to the marginal and conditional mode.
ENAR - March 26, 2006 49

 Multiple parallel chains for each parameter permit monitoring convergence using potential scale reduction statistic.

ENAR - March 26, 2006

50

Results from Gibbs sampler

Summary of posterior distributions in coagulation example. Posterior quantiles and estimated potential scale reductions computed from the second halves of then Gibbs sampler sequences, each of length 1000. Potential scale reductions for and were computed on the log scale. The hierarchical variance 2, is estimated less precisely than the unitlevel variance, 2, as is typical in hierarchical models with a small number of batches.

ENAR - March 26, 2006

51

Posterior quantiles from Gibbs sampler

Burn-in = 50% of chain length. Parameter 1 2 3 4 log p(, log , log |y ) log p(, , log , log |y ) 2.5% 58.92 63.96 65.72 59.39 55.64 1.83 1.98 -70.79 -71.07 Posterior quantiles 25.0% 50.0% 75.0% 60.44 61.23 62.08 65.26 65.91 66.57 67.11 67.77 68.44 60.56 61.14 61.71 62.32 63.99 65.69 2.17 2.41 2.7 3.45 4.97 7.76 -66.87 -65.36 -64.20 -66.88 -65.25 -64.00 97.5% 63.69 67.94 69.75 62.89 73.00 3.47 24.60 -62.71 -62.42

ENAR - March 26, 2006

52

Checking convergence
Parameter 1 2 3 4 log p(, log , log |y ) log p(, , log , log |y ) R 1.001 1.001 1.000 1.000 1.005 1.000 1.012 1.000 1.000 upper 1.003 1.003 1.002 1.001 1.005 1.001 1.013 1.000 1.001

ENAR - March 26, 2006

53

Chains for diet means

theta1

58

62

66

500

600

700 iteration

800

900

1000

theta2

63

65

67

500

600

700 iteration

800

900

1000

theta3

65

67

69

500

600

700 iteration

800

900

1000

theta3

59

61

63

65

500

600

700 iteration

800

900

1000

ENAR - March 26, 2006

54

Posteriors for diet means

56

58

60

62

64

66

62

64

66 theta2

68

70

theta1

64

66

68 theta3

70

72

58

60

62

64

theta4

ENAR - March 26, 2006

55

Chains for , ,

mu

50

60

70

80

90 100

500

600

700 iteration

800

900

1000

sigma2

10

15

500

600

700 iteration

800

900

1000

tau2

2000

6000

10000

500

600

700 iteration

800

900

1000

ENAR - March 26, 2006

56

Posteriors for , ,

50

55

60

65 mu

70

75

80

1.5

2.0

2.5

3.0 sigma

3.5

4.0

4.5

10 tau

15

20

25

ENAR - March 26, 2006

57

Hierarchical modeling for meta-analysis

Idea: summarize and integrate the results of research studies in a specic area. Example 5.6 in Gelman et al.: 22 clinical trials conducted to study the eect of beta-blockers on reducing mortality after cardiac infarction. Considering studies separately, no obvious eect of beta-blockers. Data are 22 22 tables: in j th study, n0j and n1j are numbers of individuals assigned to control and treatment groups, respectively, and y0j and y1j are number of deaths in each group. Sampling model for j th experiment: two independent Binomials, with probabilities of death p0j and p1j .
ENAR - March 26, 2006 58

 Possible quantities of interest: 1. dierence p1j p0j 2. probability ratio p1j /p0j 3. odds ratio j = [p1j /(1 p1j )]/[p0j /(1 p0j )] We parametrize in terms of the log odds ratios: j = log j because the posterior is almost normal even for small samples.

ENAR - March 26, 2006

59

Normal approximation to the likelihood

Consider estimating j by the empirical logits: yj = log y1j n1 j y 1 j log y0j n0 j y0 j

Approximate sampling variance (e.g., using a Taylor expansion):

2 j =

1 1 1 1 + + + y 1 j n1 j y1 j y 0 j n0 j y 0 j

See Table 5.4 for the estimated log-odds ratios and their estimated standard errors.
ENAR - March 26, 2006 60

Raw data Study j 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Control deaths total 3 39 14 116 11 93 127 1520 27 365 6 52 152 939 48 471 37 282 188 1921 52 583 47 266 16 293 45 883 31 147 38 213 12 122 6 154 3 134 40 218 43 364 39 674 Treated deaths total 3 38 7 114 5 69 102 1533 28 355 4 59 98 945 60 632 25 278 138 1916 64 873 45 263 9 291 57 858 25 154 33 207 28 251 8 151 6 174 32 209 27 391 22 680

Logodds, yj 0.0282 -0.7410 -0.5406 -0.2461 0.0695 -0.5842 -0.5124 -0.0786 -0.4242 -0.3348 -0.2134 -0.0389 -0.5933 0.2815 -0.3213 -0.1353 0.1406 0.3220 0.4444 -0.2175 -0.5911 -0.6081

sd, j 0.8503 0.4832 0.5646 0.1382 0.2807 0.6757 0.1387 0.2040 0.2740 0.1171 0.1949 0.2295 0.4252 0.2054 0.2977 0.2609 0.3642 0.5526 0.7166 0.2598 0.2572 0.2724

ENAR - March 26, 2006

61

Goals of analysis
Goal 1: if studies can be assumed to be exchangeable, we wish to estimate the mean of the distribution of eect sizes, or overall average eect. Goal 2: the average eect size in each of the exchangeable studies Goal 3: the eect size that could be expected if a new, exchangeable study, were to be conducted.

ENAR - March 26, 2006

62

Normal-normal model for meta-analysis

2 2 First level: sampling distribution yj |j , j N(j , j )

Second level: population distribution j |, N(, 2) Third level: prior for hyperparameters , p(, ) = p(| )p( ) 1 mostly for convenience. Can incorporate information if available.

ENAR - March 26, 2006

63

Study j 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

2.5% -0.58 -0.63 -0.64 -0.44 -0.44 -0.62 -0.62 -0.43 -0.56 -0.48 -0.47 -0.42 -0.65 -0.34 -0.54 -0.50 -0.46 -0.53 -0.51 -0.51 -0.67 -0.69

Posterior quantiles of eect j normal approx. (on log-odds scale) 25.0% 50.0% 75.0% -0.32 -0.24 -0.15 -0.36 -0.28 -0.20 -0.34 -0.26 -0.18 -0.31 -0.25 -0.18 -0.28 -0.21 -0.12 -0.36 -0.27 -0.19 -0.44 -0.36 -0.27 -0.28 -0.20 -0.13 -0.36 -0.27 -0.20 -0.35 -0.29 -0.23 -0.31 -0.24 -0.17 -0.28 -0.21 -0.12 -0.37 -0.27 -0.20 -0.22 -0.12 0.00 -0.32 -0.26 -0.17 -0.30 -0.23 -0.15 -0.28 -0.21 -0.11 -0.30 -0.22 -0.13 -0.31 -0.22 -0.13 -0.32 -0.24 -0.17 -0.40 -0.30 -0.23 -0.40 -0.30 -0.22

97.5% 0.13 -0.03 0.06 -0.04 0.13 0.04 -0.16 0.08 -0.05 -0.12 -0.01 0.09 0.02 0.30 0.00 0.06 0.15 0.15 0.17 0.04 -0.09 -0.07

ENAR - March 26, 2006

64

Estimand Mean, Standard deviation, Predicted eect, j

2.5% -0.38 0.01 -0.57

Posterior quantiles 25.0% 50.0% 75.0% -0.29 -0.25 -0.21 0.08 0.13 0.18 -0.33 -0.25 -0.17

97.5% -0.12 0.32 0.08

Marginal posterior density p( |y )

0.0 0.0

0.02

0.04

0.5

1.0 tau

1.5

2.0

ENAR - March 26, 2006

65

Conditional posterior means E (j |, y )

0.4

19 18 14

0.2

17 Estimated Treatment Effects 5 0.0 1 12 8 16 20 11 4 10 15 9 13 22 7 3 6 21 2 0.0 0.5 1.0 tau 1.5 2.0

ENAR - March 26, 2006

-0.6

-0.4

-0.2

66

Conditional posterior standard deviations sd(j |, y )

0.8

19 6 Estimated standard deviations 0.6 3 18 2 0.4 13 17 15 5 9 16 22 20 21 12 8 14 11 10 4 7

0.2 0.0

0.5

1.0 tau

1.5

2.0

ENAR - March 26, 2006

67

Histogram of 1000 simulations of 2, 18, 7, and 14

-1.0

-0.6

-0.2

0.2

-0.8

-0.4

0.0 Study 18

0.4

Study 2

-0.8

-0.4 Study 7

0.0

-0.6

-0.2

0.2

0.6

Study 14

ENAR - March 26, 2006

68

Overall mean eect, E (|, y )

E(mu|tau,y)

-0.26 0.0

-0.25

-0.24

-0.23

0.5

1.0 tau

1.5

2.0

ENAR - March 26, 2006

69

Example: Mixed model analysis

Sheeld Food Company produces dairy products. Government cited company because the actual content of fat in yogurt sold by Sheeld appers to be higher than the label amount. Sheeld believes that the discrepancy is due to the method employed by the goverment to measure fat content, and conducted a multilaboratory study to investigate. Four laboratories in the United States were randomly chosen. Each laboratory received 12 carefully mixed samples of yogurt, with instructions to analyze 6 using the government method and 6 using Sheelds method.

 Fat content in all samples was known to be very close to 3%. Because of technical diculties, none of the labs managed to analyze all six samples using the government method within the alloted time. Method Government Lab 1 5.19 5.09 Lab 2 4.09 3.0 3.75 4.04 4.06 3.02 3.32 2.83 2.96 3.23 3.07 Lab 3 4.62 4.32 4.35 4.59 3.08 2.95 2.98 2.74 3.07 2.70 Lab 4 3.71 3.86 3.79 3.63 2.98 2.89 2.75 3.04 2.88 3.20

Sheeld

3.26 3.38 3.24 3.41 3.35 3.04

 We tted a mixed linear model to the data, where Method is a xed eect with two levels Laboratory is a random eect with four levels Method by laboratory interaction is random with six levels yijk = i + j + ( )ij + eijk , and i = + i and eijk N(0, 2). Priors: p(i) 1
2 2 p(j | | ) 2 p(( )ij | ) 2 N(0, ) 2 N(0, )

The three variance components were assigned diuse inverted gamma priors.

alpha[1] chains 1:3 5.5 5.0 4.5 4.0 3.5 3.0 5001 6000 iteration 8000 10000

alpha[2] chains 1:3

4.5

4.0 3.5 3.0 2.5

2.0

5001

6000

8000

10000

iteration

beta[1] chains 1:3 sample: 15000

beta[2] chains 1:3 sample: 15000

4.0 3.0 2.0 1.0 0.0

3.0
2.0
1.0
0.0
-1.0

0.0

1.0

-1.0

-0.5

0.0

0.5

beta[3] chains 1:3 sample: 15000

beta[4] chains 1:3 sample: 15000

4.0 3.0 2.0 1.0 0.0

4.0 3.0 2.0 1.0 0.0

-2.0

-1.0

0.0

1.0

-2.0

-1.0

0.0

beta[1] chains 1:3

beta[3] chains 1:3

1.0 0.5 0.0 -0.5 -1.0

0

1.0 0.5 0.0 -0.5 -1.0

20
lag

40

20
lag

40

beta[4] chains 1:3

1.0 0.5 0.0 -0.5 -1.0

beta[2] chains 1:3

1.0 0.5 0.0 -0.5 -1.0

20

40

20

40

lag

lag

beta[1] chains 1:3

1.0 0.5 0.0 -0.5 -1.0
1.0 0.5 0.0 -0.5 -1.0

beta[2] chains 1:3

20

40

20 lag

40

lag
beta[3] chains 1:3
1.0 0.5 0.0 -0.5 -1.0

beta[4] chains 1:3

1.0 0.5 0.0 -0.5 -1.0

0
20
40

20
lag

40

lag

beta[1] chains 1:3

1.5
1.5

beta[2] chains 1:3

1.0

1.0

0.5

0.5

0.0

0.0

5001

6000

8000

5001

6000

8000

iteration

iteration

beta[3] chains 1:3

1.5
beta[4] chains 1:3

1.0

1.0
0.5
0.0

0.5

0.0

5001

6000

8000

5001

6000

8000
iteration

iteration

box plot: alpha

box plot: beta

5.0

[1]

1.0
[1]

0.5
[2]

[3]

4.0
[2]

[4]

0.0

3.0 -0.5

2.0

-1.0

Model checking
What do we need to check? Model t: does the model t the data? Sensitivity to prior and other assumptions Model selection: is this the best model? Robustness: do conclusions change if we change data?

Remember: models are never true; they may just t the data well and allow for useful inference. Model checking strategies must address various parts of models: priors sampling distribution hierarchical structure
ENAR - March 26, 2006 1

 other model characteristics such as covariates, form of dependence between response variable and covariates, etc. Classical approaches to model checking: Do parameter estimates make sense Does the model generate data like observed sample Are predictions reasonable Is model best in some sense (e.g., AIC, likelihood ratio, etc.)

Bayesian approach to model checking Does the posterior distribution of parameters correspond with what we know from subject-matter knowledge? Predictive distribution for future data must also be consistent with substantive knowledge Future data generated by predictive distribution compared to current sample
ENAR - March 26, 2006 2

 Sensitivity to prior and other model components Most popular model checking approach has a frequentist avor: we generate replications of the sample from posterior and observe the behavior of sample summaries over repeated sampling.

ENAR - March 26, 2006

Posterior predictive model checking

Basic idea is that data generated from the model must look like observed data. Posterior predictive model checks based on replicated data y rep generated from the posterior predictive distribution: p(y rep|y ) = p(y rep|)p(|y )d.

y rep is not the same as y . Predictive outcomes y can be anything (e.g., a regression prediction using dierent covariates x ). But y rep is a replication of y . y rep are data that could be observed if we repeated the exact same experiment again tomorrow (if in fact the s in our analysis gave rise to the data y ).
ENAR - March 26, 2006 4

 Denition of a replicate in a hierarchical model: p(|y ) p(|, y ) p(y rep|) : rep from same units p(|y ) p(|) p(y rep|) : rep from new units Test quantity: T (y, ) is a discrepancy statistic used as a standard. We use T (y, ) to determine discrepancy between model and data on some specic aspect we wish to check. Example of T (y, ): proportion of standardized residuals outside of (3, 3) in a regression model to check for outliers In classical statistics, use T (y ) a test-statistic that depends only on the data. Special case of Bayesian T (y, ). For model checking:
ENAR - March 26, 2006 5

 Determine appropriate T (y, ) Compare posterior predictive distribution of T (y rep, ) to posterior distribution of T (y, )

ENAR - March 26, 2006

Bayes p-values
p-values attempt to measure tail-area probabilities. Classical denition: class p value = Pr(T (y rep) T (y )|) Probability is taken over distribution of y rep with xed typically used to compute the pvalue. Point estimate Posterior predictive p-values: Bayes p value = Pr(T (y rep, ) T (y, )|y )
ENAR - March 26, 2006 7

 Probability taken over joint posterior distribution of (, y rep): Bayes p value = IT (yrep,)T (y,)p(|y )p(y rep|)ddy rep

ENAR - March 26, 2006

Relation between p-values

Small example: Sample y1, ..., yn N (, 2) Fit N (0, 2) and check whether = 0 is good t In real life, would t more general N (, 2) and would decide whether = 0 is plausible. Classical approach: Test statistic is sample mean T (y ) = y p value = P r(T (y rep) T (y )| 2) = P r( y rep y | 2)
ENAR - March 26, 2006 9

 = Pr = P This is special case: parameters. Bayes approach:

ny S

rep

tn1

ny S

ny 2 | S

not always possible to get rid of nuisance

p value = P r(T (y rep) T (y )|y ) = Note that IT (yrep)T (y)p(y rep| 2) = P (T (y rep) T (y )| 2)
ENAR - March 26, 2006 10

IT (yrep)T (y)p(y rep| 2)p( 2|y )dy repd 2

= classical p-value Then: Bayes p value = E {p valueclass|y } where the expectation is taken with respect to p( 2|y ). In this example, classical pvalue and Bayes pvalue are the same. In general, Bayes can handle easily nuisance parameters.

ENAR - March 26, 2006

11

Interpreting posterior predictive pvalues

We look for tail-area probabilities that are not too small or too large. The ideal posterior predictive p-value is 0.5: the test quantity falls right in the middle of its posterior predictive. Posterior predictive p-values are actual posterior probabilities Wrong interpretation: P r(model is true |data).

ENAR - March 26, 2006

12

Example: Independence of Bernoulli trials

Sequence of binary outcomes y1, ..., yn modeled as iid Bernoulli trials with probability of success . Uniform prior on leads to p(|y ) s(1 )ns with s = Sample: 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0 Is the assumption of independence warranted? Consider discrepancy statistic T (y, ) = T (y ) = number of switches between 0 and 1.
ENAR - March 26, 2006 13

yi .

 In sample, T (y ) = 3. Posterior is Beta(8,14). To test assumption of independence, do: 1. For j = 1, ..., M draw j from Beta(8,14). rep,j rep,j , ..., y20 } independent Bernoulli variables with 2. Draw {y1 probability j . 3. In each of the M replicate samples, compute T (y ), the number of switches between 0 and 1. p = Prob(T (y rep) T (y )|y ) = 0.98.
B

If observed trials were independent, expected number of switches in 20 trials is approximately 8 and 3 is very unlikely.

ENAR - March 26, 2006

14

Posterior predictive dist. of number of switches

ENAR - March 26, 2006

15

Example: Newcombs speed of light

Newcomb obtained 66 measurements of the speed of light. (Chapter 3). One measurement of -44 is a potential outlier under a normal model. From data: y = 26.21 and s = 10.75. Model: yi|, 2 N (, 2), (, 2) 2. Posteriors p( 2|y ) = Inv 2(65, s2) p(| 2, y ) = N( y , 2/66) or p(|y ) = t65( y , s2/66).
ENAR - March 26, 2006 16

 For posterior predictive checks, do for i = 1, ..., M 1. Generate ((i), 2(i)) from p(, 2|y ) rep(i) rep(i) from N ((i), 2(i)) , ..., y66 2. Generate y1

ENAR - March 26, 2006

17

Example: Newcombs data

ENAR - March 26, 2006

18

Example: Replicated datasets

ENAR - March 26, 2006

19

 Is observed minimum value -44 consistent with model? Dene T (y ) = min{yi} get distribution of T (y rep). Large negative value of -44 inconsistent with distribution of rep T (y rep) = min{yi } because observed T (y ) very unlikely under distribution of T (y rep). Model inadequate, must account for long tail to the left.

ENAR - March 26, 2006

20

Posterior predictive of smallest observation

ENAR - March 26, 2006

21

 Is sampling variance captured by model? Dene T (y ) = s2 y 1 = i(yi y )2 65

obtain distribution of T (y rep). Observed variance of 115 very consistent with distribution (across reps) of sample variances Probability of observing a larger sample variance is 0.48: observed variance in middle of distribution of T (y rep) But this is meaningless: T (y ) is sucient statistic for 2, so model MUST have t it well. Symmetry in center of distribution Look at dierence in the distance of 10th and 90th percentile from the mean. Approx 10th percentile is 6th order statistic y(6), and approx 90th percentile is y(61).
ENAR - March 26, 2006 22

Dene T (y, ) = |y(61) | |y(6) | Need joint distribution of T (y, ) and of T (y rep, ). Model accounts for symmetry in center of distribution Joint distribution of T (y, ) and T (y rep, ) about evenly distributed along line T (y, ) = T (y rep, ). Probability that T (y rep, ) > T (y, ) about 0.26, plausible under sampling variability.

ENAR - March 26, 2006

23

Posterior predictive of variance and symmetry

ENAR - March 26, 2006

24

Choice of discrepancy measure

Often, choose more than one measure, to investigate dierent attributes of the model. Can smoking behavior in adolescents be predicted using information on covariates? (Example on page 172 of Gelman et al.) Data: six observations of smoking habits in 2,000 adolescents every six months Two models: Logistic regression model Latent-class model: propensity to smoke modeled as nonlinear function of predictors. Conditional on smoking, t same logistic regression model as above. Three dierent test statistics chosen:
ENAR - March 26, 2006 25

 % who never smoked % who always smoked % of incident smokers: began not smoking but then switched to smoking and continued doing so. Generate replicate datasets from both models. Compute the three test statistics in each of the replicated datasets Compare the posterior predictive distributions of each of the three statistics with the value of the statistic in the observed dataset Results: Model 2 slightly better than 1 at predicting % of always smokers, but both models fail to predict % of incident smokers.
Test variable % never smokers % always-smokers % incident smokers

T (y ) 77.3 5.1 8.4

95% CI for T (y rep ) [75.5, 78.2] [5.0, 6.5] [5.3, 7.9]

p-value 0.27 0.95 0.005

95% CI for T (y rep ) [74.8, 79.9] [3.8, 6.3] [4.9, 7.8]

p-value 0.53 0.44 0.004

ENAR - March 26, 2006

26

Omnibus tests
It is often useful to also consider summary statistics: -discrepancy: T (y, ) =
2 (yi E (yi | ))2 var (yi | )

Deviance: T (y, ) = 2 log p(y |) Deviance is proportional to mean squared error if model is normal and with constant variance. In classical approach, insert null or mle in place of in T (y, ) and compare test statistic to reference distribution derived under asymptotic arguments. In Bayesian approach, reference distribution for test statistic is automatically calculated from posterior predictive simulations.
ENAR - March 26, 2006 27

 A Bayesian 2 test is carried out as follows: Compute the distribution of T (y, ) for many draws of from posterior and for observed data. Compute the distribution of T (y rep, ) for replicated datasets and posterior draws of Compute Bayesian p-value: probability of observing a more extreme value of T (y rep, ). Calculate p-value empirically from simulations over and y rep.

ENAR - March 26, 2006

28

Criticisms of posterior predictive checks

Too conservative because data get used twice Posterior predictive p-values are not really p-values: distribution under null hypothesis is not uniform, as it should be What is high/low if pp p-values not uniform? Some Bayesians object to frequentist slant of using unobserved data for anything Lots of work recently on improving posterior predictive p-values: Bayarri and Berger, JASA, 2000, is good reference In spite of criticisms, pp checks are easy to use in very general cases, and intuitively appealing.
ENAR - March 26, 2006 29

Pps can be conservative

One criticism for pp model checks is that they tend to reject models only in the face of extreme evidence. Example (from Stern): y N (, 1) and N (0, 9). Observation: yobs = 10 Posterior p(|y ) = N (0.9yobs, 0.9) = N (9, 0.9) Posterior predictive dist: N (9, 1.9) Posterior predictive p-value is 0.23, we would not reject the model

Eect of prior is minimized because 9 > 1 and posterior predictive mean is close to observed datapoint. To reject, we would need to observe y 23.
ENAR - March 26, 2006 30

Graphical posterior predictive checks

Idea is to display the data together with simulated data. Three kinds of checks: Direct data display Display of data summaries or parameter estimates Graphs of residuals or other measures of discrepancy

ENAR - March 26, 2006

31

Model comparison
Which model ts the data best? Often, models are nested: a model with parameters is nested within a model with parameters (, ). Comparison involves deciding whether adding to the model improves its t. Improvement in t may not justify additional complexity. It is also possible to compare non-nested models. We focus on predictive performance and on model posterior probabilities to compare models.

ENAR - March 26, 2006

32

Expected deviance
We compare the observed data to several models to see which predicts more accurately. We summarize model t using the deviance D(y, ) = 2 log p(y |). It can be shown that the model with the lowest expected deviance is best in the sense of minimizing the (Kullback-Leibler) distance between the model p(y |) and the true distribution of y , f (y ). We compute the expected deviance by simulation: Davg(y ) = E|y (D(y, )|y ).
ENAR - March 26, 2006 33

 An estimate is

1 Davg(y ) = M

D(y, j ).
j

ENAR - March 26, 2006

34

Deviance information criterion - DIC

Idea is to estimate the error that would be expected when applying the model to future data. Expected mean square predictive error: 1 pred Davg = E n rep rep (yi E (yi |y ))2 .
i

A model that minimizes the expected predictive deviance is best in the sense of out-of-sample predictive power. pred An approximation to Davg is the Deviance Information Criterion or DIC pred = 2D avg(y ) D (y ), DIC = D avg
ENAR - March 26, 2006 35

 where D (y ) = D (y, (y )) and (y ) is a point estimator of such as the posterior mean.

ENAR - March 26, 2006

36

Example: SAT study

We compare three models: no pooling, complete pooling and hierarchical model (partial pooling). Deviance is D(y, ) = 2
j 2 log N(yj |j , j ) 2 ((yj j )2/j ) j

= log(2J 2) +

The DIC for the three models were: 70.3 (no pooling), 61.5 (complete pooling), 63.4 (hierarchical model). Based on DIC, we would pick the complete pooling model.
ENAR - March 26, 2006 37

 We still prefer the hierarchical model because assumption that all schools have identical mean is strong.

ENAR - March 26, 2006

38

Bayes Factors
Suppose that we wish to decide between two models M1 and M2 (dierent prior, sampling distribution, parameters). Priors on models are p(M1) and p(M2) = 1 p(M1). The posterior odds favoring M1 over M2 are p(M1|y ) p(y |M1) p(M1) = . p(M2|y ) p(y |M2) p(M2) The ratio p(y |M1)/p(y |M2) is called a Bayes factor. It tells us how much the data support one model over the other.
ENAR - March 26, 2006 39

 The BF12 is computed as BF12 = p(y |1, M1)p(1|M1)d1 . p(y |2, M2)p(2|M2)d2

Note that we need p(y ) under each model to compute p(i|Mi). Therefore, the BF is only dened when the marginal distribution of the data p(y ) is proper, and therefore, when the prior distribution in each model is proper. For example, if y N (, 1) with p() 1, we get p(y ) (2 )
1 /2

1 exp{ (y )2}d = 1, 2

which is constant for any y and thus is not a proper distribution. This creates an indeterminacy because we can increase or decrease the BF simply by using a dierent constant in the numerator and denominator.
ENAR - March 26, 2006 40

Computation of Bayes Factors

Diculty lies in the computation of p(y ) p(y ) = p(y |)p()d.

The simplest approach is to draw many values of from p() and get a Monte Carlo approximation to the integral. May not work well because the s from the prior may not come from the parameter space region where the sampling distribution has most mass. A better Monte Carlo approximation is similar to importance sampling.
ENAR - March 26, 2006 41

Note that p(y )

1

= =

h() d p(y ) h() p(|y )d. p(y |)p()

h() can be anything (e.g., a normal approximation to the posterior). Draw values of from the posterior and evaluate the integral numerically. Computations can get tricky because denominator can be small. As n , 2, M2) log(p(y | 1, M1) 1 (d1 d2) log(n), log(BF ) log(p(y | 2
ENAR - March 26, 2006 42

i is the posterior mode under model Mi and di is the dimension where of i. Notice that the criterion penalizes a model for additional parameters. Using log(BF ) is equivalent to ranking models using the BIC criterion, given by 1 BIC = log(p(y |, M ) + d log(n). 2

ENAR - March 26, 2006

43

Ordinary Linear Regression - Introduction

Question of interest: how does an outcome y vary as a function of a vector of covariates X . We want the conditional distribution p(y |, x), where observations (y, x)i are assumed exchangeable. Covariates (x1, x2, ..., xk ) can be discrete or continuous, and typically x1 = 1 for all n units. The matrix X , n k is the model matrix Most common version of the model is the normal linear model
k

E (yi|, X ) =
j =1

j xij

ENAR - March 2006

Ordinary Linear Regression - Model

In ordinary linear regression models, the conditional variance is equal across observations: var(yi|, X ) = 2. Thus, = (1, 2, ..., k , 2). Likelihood: For the ordinary normal linear regression model, we have p(y |X, , 2) = N (X, 2I ) with In and n n identity matrix. Priors: A non-informative prior distribution for (, 2) is p(, 2) 2 (more on informative priors later)
ENAR - March 2006 2

 Joint posterior:
2 2 n ( ) 2 1 exp

p(, |y )

1 2 (y X ) (y X ) 2

Joint posterior p(, |y )

2 2 n ( ) 2 1 exp

1 2 (y X ) (y X ) 2

Consider p(, 2|y ) = p( | 2, y )p( 2|y ) Conditional posterior for :

ENAR - March 2006

Expand and complete the squares in

3

p( | 2, y ) (viewed as function of ). We get: 1 X X p( | , y ) exp 2 X X 2 2

2

1 ) X X ( ) exp 2 ( 2 for = (X X )1X y Then: 2(X X )1) | 2, y N (,

ENAR - March 2006

Ordinary Linear Regression - p( 2|y )

The marginal posterior distribution for 2 is obtained by integrating the joint posterior with respect to : p( 2|y ) = p(, 2|y )d
n

( 2) 2 1

1 exp 2 (y X ) (y X ) d 2

By expanding square in integrand, and adding and subtracting 2y X (X X )1X y , we can write: arge p( |y ) =
ENAR - March 2006

2 n ( ) 2 1

exp
n

1 ) ) + ( ) X X ( ) (y X (y X 2 2 1 2 ( n k ) S 2 2 exp

( 2) 2 1 exp

1 ) X X ( ) d ( 2 2

Integrand is kernel of k dimensional normal, so result of integration is proportional to ( 2)k/2. Then p( 2|y ) ( 2)
nk 2 +1

exp 2(n k )S 2

proportional to an Inv-2(n k, S 2).

ENAR - March 2006

Ordinary Linear Regression - Notes

and S 2 are the MLEs of and 2 Note that When is the posterior proper? p(, 2|y ) is proper if 1. n > k 2. rank(X ) = k : columns of X are linearly independent or |X X | = 0 To sample from the joint posterior: 1. Draw 2 from Inv-2(n k, S 2) 2(X X )1) 2. Given 2, draw vector from N(, S 2, (X X )1 once, before starting repeated For eciency, compute , drawing.

ENAR - March 2006

Regression - Posterior predictive distribution

In regression, we often want to predict the outcome y for a new set of ) covariates x . Thus, we wish to draw values from p( y |y, X By simulation: 1. Draw 2 from Inv-2(n k, S 2) 2(X X )1) 2. Draw from N(, 3. Draw y i for i = 1, ..., m from N( xi, 2)

ENAR - March 2006

Regression - Posterior predictive distribution

We can derive p( y |y ) in steps, rst considering p( y |y, 2). Note that p( y |y, 2) = p( y |, 2)p( | 2, y )d

is normal because exponential in integrand is quadratic function in (, y ). To get mean and variance use conditioning trick: E ( y | 2, y ) = E E ( y |, 2, y )| 2, y | 2, y ) = E (X = X
ENAR - March 2006 9

where inner expectation averages over y conditional on and outer averages over . var( y | 2, y ) = E var( y |, 2, y )| 2, y + var E ( y |, 2, y )| 2, y | 2, y ] = E [ 2I | 2, y ] + var[X (X X )1X ) = 2(I + X (X X )1X is Var has two terms: 2I is sampling variation, and 2X due to uncertainty about To complete specication of p( y |y ) must integrate p( y |y, 2) with respect to marginal posterior distribution of 2. Result is: S 2[I + X , (X X )1X ]) p( y |y ) = tnk (X
ENAR - March 2006 10

Regression example: radon measurements in Minnesota

Radon measurements yi were taken in three counties in Minnesota: Blue Earth, Clay and Goodhue. 14 houses in each of Blue Earth and Clay county and 13 houses in Goodhue were sampled. Measurements were taken in the basement and in the rst oor. We t an ordinary regression model to the log radon measurements, without an intercept. We dene dummy variables as follows: X1 = 1 if county is Blue Earth

and is 0 otherwise. Similarly, X2 and X3 are dummies for Clay and Goodhue counties, respectively. X4 = 1 if measurement was taken in the rst oor. The model is log(yi) = 1x1i + 2x2i + 3x3i + 4x4i + ei, with e N(0, 2). Thus E (y |Blue Earth, basement) = exp(1) E (y |Blue Earth, rst oor) = exp(1 + 4) E (y |Clay, basement) = exp(2)

E (y |Clay, rst oor) = exp(2 + 4) E (y |Goodhue, basement) = exp(3) E (y |Goodhue, rst oor) = exp(3 + 4) We used noninformative priors N(0, 1000) for the regression coecients and a noninformative prior Gamma(0.01, 0.01) for the error variance.

beta[1]

3.0
2.5 2.0 1.5 1.0
0.5
3001

3500

4000
iteration

4500

5000

beta[2]
3.0

2.5 2.0 1.5 1.0

0.5

3001

3500

4000

4500

5000

iteration

beta[3]
3.0

2.5

2.0

1.5

1.0

3001

3500

4000

4500

5000

iteration

beta[4] 1.0 0.0 -1.0 -2.0 3001 3500 4000 iteration 4500 5000

box plot: beta

3.0
[1]

[2]

[3]

2.0

1.0
[4]

0.0

-1.0

becb sample: 2000

becff sample: 2000

0.3
0.2
0.1
0.0
0.0

0.3
0.2
0.1
0.0

5.0

10.0

15.0

0.0

5.0

10.0

15.0

ccb sample: 2000

0.4 0.3 0.2 0.1 0.0
0.4 0.3 0.2 0.1 0.0

ccff sample: 2000

0.0

5.0

10.0

0.0

5.0

10.0

gcb sample: 2000

0.3
0.3

gcff sample: 2000

0.2

0.2

0.1

0.1

0.0

0.0

0.0

5.0

10.0

15.0

0.0

5.0

10.0

15.0

Regression - Posterior predictive checks

For regression models, there are well-known methods for checking model and assumptions using estimated residuals. Residuals:
i

= yi xi

Two useful test statistics are: Proportion of outliers among residuals Correlation between squared residuals and tted values y Posterior predictive distributions of both statistics can be obtained via simulation Dene a standardized residual as ei = (yi xi )/
ENAR - March 2006 13

 If normal model is correct, standardized residuals should be about N (0, 1) and therefore, |e|i > 3 suggests that ith observation may be an outlier. To derive the posterior predictive distribution for the proportion of outliers q and for , the correlation between (e2, y ), do: 1. 2. 3. 4. 5. 6. Draw ( 2, ) from the joint posterior distribution Draw y rep from N (X, 2I ) given the existing X Run the regression of y rep on X , and save residuals Compute Compute proportion of large standardized residuals q Repeat for another y rep

Approach is frequentist in nature: we act as if we could repeat the experiment many times. Inspection of posterior predictive distribution of and of q provides
ENAR - March 2006 14

information about model adequacy.

ENAR - March 2006

15

 Results of posterior predictive checks suggest that there are no outliers and that the correlation between the squared residuals and the tted values is negligible. The 95% credible set for the proportion of absolute residuals (in the 41 observations) above 3 was (0, 4.88) with a mean of 0.59% and a median of 0. The 95% credible set for was (0.31, 0.32) with a mean of 0.011. Notice that the posterior distribution of the proportion of outliers is skewed. This sometimes happens when the quantity of interest is bounded and most of the mass of its distribution is close to the boundary.

Regression with unequal variances

Consider now the case where y N (X, y ), with y = 2I . Covariance matrix y is n n and has n(n + 1)/2 distinct parameters, and cannot be estimated from n observations. Must either specify y or it must be assigned an informative prior distribution. Typically, some structure is imposed on y to reduce the number of free parameters.

ENAR - March 2006

16

Regression - known y
As before, let p( ) 1 be the non-informative prior Since y is positive denite and symmetric, it has an upper-triangular 1 /2 1 /2 1 /2 square root matrix (Cholesky factor) y such that y y = y so that if: y = X + e, e N (0, y ) then
1 /2 1 /2 1/2 y = X + e, y y y 1 /2 e N (0, I ) y

With y known, just proceed as in ordinary linear regression, but use the transformed y and X as above and x 2 = 1. Algebraically, this
ENAR - March 2006 17

is equivalent to computing = (X 1X )1X 1y y y V

1 1 = (X X ) y

Note: By using the Cholesky factor you avoid computing the n n 1 inverse y .

ENAR - March 2006

18

Prediction of new y with known y

Even if we know y , prediction of new observations is more complicated: must know covariance matrix of old and new data. Example: heights of children from same family are correlated. To predict height of a new child when a brother is in the old dataset, we must include that correlation in the prediction. . y are n new observations given a n k matrix of regressors X Joint distribution of y , y is: y N |X, X, y y |y, , y
ENAR - March 2006

X X

y y y y y y y

N (y , Vy )
19

where
1 y = X + y y yy (y X ) 1 Vy = y y y y yy y y

ENAR - March 2006

20

Regression - unknown y
To draw inferences about (, y ), we proceed in steps: p( |y , y ) and then p(y |y ). Let p( ) 1 as before We know that p(y |y ) = p(, y |y ) p(y )p(y |, y ) p( |y , y ) p( |y , y ) p(y ) N(y |, y ) , N( |, V ) derive

V ) depend on y . where (,
ENAR - March 2006 21

. Expression for p(y |y ) must hold for any , so we set = Note that V ) |V |1/2 p( |y , y ) = N(, . Then for = 1 ) 1(y X )] p(y |y ) p(y )|V |1/2|y |1/2 exp[ (y X y 2 In principle, p(y |y ) could be evaluated for a range of values of y . However: 1. It is dicult to determine a prior for an n n unstructured matrix. V depend on y and it is very dicult to draw values of y from 2. , p(y |y ). Need to put some structure on y
ENAR - March 2006 22

Regression - y = 2Qy
Suppose that we know y upto a scalar factor 2. Non-informative prior on , 2 is p(, 2) 2 Results follow directly from ordinary linear regression, by using 1/2 1/2 transformation Qy y and Qy X , which is equivalent to computing = (X Q1X )1X Q1y y y V
1 1 = (X Q y X)

) ) Q1(y X s2 = (n k )1(y X y To estimate the joint posterior distribution p(, 2|y ) do

ENAR - March 2006 23

1. Draw 2 from Inv-2(n k, s2) V 2 ) 2. Draw from n(, In large datasets, use Cholesky factor transformation and unweighted 1 regression to avoid computation of Q y .

ENAR - March 2006

24

Regression - other covariance structures

Weighted regression In some applications, y = diag( 2/wi), for known weights and 2 unknown.
1 Inference is the same as before, but now Q y = diag(wi )

Parametric model for unequal variances: Variances may depend on weights in non-linear fashion: ii = 2v (wi, ) for unknown parameter (0, 1) and known function v such as v = wi Note that for that function v : = 0 ii = 2 i
ENAR - March 2006 25

 = 1 ii = 2/wi i In practice, to uncouple from the scale of the weights, we multiply weights by a factor so that their product equals 1.

ENAR - March 2006

26

Parametric model for unequal variances

Three parameters to estimate: , 2, . Possible prior for is uniform in [0, 1] Non-informative prior for , 2 is 2 Joint posterior distribution:
2 N ( y ; ( X ) , v (wi, )) p(, 2, |y ) p()p(, 2)n i i i=1

For a given , we are back in the earlier weighted regression case with
1 Q y = diag(v (w1 , ), ..., v (wn , ))
ENAR - March 2006 27

 This suggests the following scheme 1. 2. 3. 4. Draw from marginal posterior p(|y ) 1/2 1/2 Compute Qy y and Qy X Draw 2 from p( 2|, y ) as in ordinary regression Draw from p( | 2, , y ) as in ordinary regression

Marginal posterior distribution of : p(|y ) = = p(, 2, |y ) p(, 2|, y ) p(, 2, |y ) p( | 2, , y )p( 2|, y ) p() 2i N(yi|(X )i, 2v (wi, )) V ) Inv-2(n k, s2) N(,

and 2 = s2. Expression must hold for any (, 2), so we set =

ENAR - March 2006 28

 and s2 depend on . Recall that Also recall that weights are scaled to have product equal to 1. Then p(|y ) p()|V |1/2(s2)(nk)/2 To carry out computation First sample from p(|y ): 1. Evaluate p(|y ) for a range of values of [0, 1] 2. Use inverse cdf method to sample 1/2 1/2 Given , compute y = Qy y and X = Qy X Compute = (X X )1X y V
ENAR - March 2006

= (X X )1
29

) (y X ) s2 = (n k )1(y X Draw 2 from Inv-2(n k, s2) 2 V ) Draw from N(,

ENAR - March 2006

30

Including prior information about

Suppose we wish to add prior information about a single regression coecient j of the form:
2 j N(j 0, ), j 2 with j 0, known. j

Prior information can be added in the form of an additional data point. An ordinary observation y is normal with mean x and variance 2. As a function of j , the prior can be viewed as an observation with 0 on all xs except xj
ENAR - March 2006 31

2 . variance j

To include prior information, do the following: 1. Append one more data point to vector y with value j 0. 2. Add one row to X with zeroes except in j th column. 2 3. Add a diagonal element with value to y . j Now apply computational methods for non-informative prior. Given y , posterior for obtained by weighted linear regression.

ENAR - March 2006

32

Adding prior information for several s

Suppose that for the entire vector , N(0, ). Proceed as before: add k data points and draw posterior inference by weighted linear regression applied to observations y, explanatory variables X and variance matrix : y = y 0 , X = X Ik , = y 0 0

Computation can be carried out conditional on rst and then inverse cdf for or using the Gibbs sampling.
ENAR - March 2006 33

Prior information about 2

Typically we do not wish to include prior information about 2. If we do, we can use the conjugate prior
2 2 Inv-2(n0, 0 ).

The marginal posterior of 2 is

2 2 + nS n 0 0 ). 2|y Inv-2(n0 + n, n0 + n

If prior information on is also incorporated, S 2 is replaced by corresponding value from regression of y on X and , and n is replaced by length of y.
ENAR - March 2006 34

Inequality constraints on
Sometimes we wish to impose inequality constraints such as 1 > 0 or 2 < 3 < 4. The easiest way is to ignore the constraint until the end: Simulate (, 2) from posterior discard all the draws that do not satisfy the constraint. Typically a reasonably ecient way to proceed unless constraint eliminates a large portion of unconstrained posterior distribution. If so, data tend to contradict the model.
ENAR - March 2006 35

Generalized Linear Models

Generalized linear models are an extension of linear models to the case where relationship between E (y |X ) and X is not linear or normal assumption is not appropriate. Sometimes a transformation suces to return to the linear setup. Consider the multiplicative model
1 b2 b3 yi = xb i1 xi2 xi3 i

A simple log transformation leads to log(yi) = b1 log(xi1) + b2 log(xi2) + b3 log(xi3) + ei When simple approaches do not work, we use GLIMs.
ENAR - March 26, 2006 1

 There are three main components in the model: 1. Linear predictor = X 2. Link function g (.) relating linear predictor to mean of outcome variable: E (y |X ) = = g 1( ) = g 1(X ) 3. Distribution of outcome variable y with mean = E (y |X ). Distribution can also depend on a dispersion parameter : p(y |X, , ) = n i=1 p(yi |(X )i , ) In standard GLIMs for Poisson and binomial data, = 1. In many applications, however, excess dispersion is present.

ENAR - March 26, 2006

Some standard GLIMs

Linear model: Simplest GLIM, with identity link function g () = . Poisson model: Mean and variance and link function log() = X , so that = exp(X ) = exp( ) For y = (y1, ..., yn): p(y | ) = n i=1 with i = (X )i.
ENAR - March 26, 2006 3

1 exp( exp(i))(exp(i))yi y!

 Binomial model: Suppose that yi Bin(ni, i), ni known. Standard link function is logit of probability of success : i g (i) = log 1 i For a vector of data y : p(y | ) = n i=1 ni yi exp(i) 1 + exp(i)
yi

= (X )i = i

1 1 + exp(i)

ni yi

Another link used in econometrics is the probit link: 1(i) = i with (.) the normal cdf.
ENAR - March 26, 2006 4

 In practice, inference from logit and probit models is almost the same, except in extremes of the tails of the distribution.

ENAR - March 26, 2006

Overdispersion
In many applications, the model can be formulated to allow for extra variability or overdispersion. E.g. in Poisson model, the variance is constrained to be equal to the mean. As an example, suppose that data are the number of fatal car accidents at K intersections over T years. Covariates might include intersection characteristics and trac control devices (stop lights, etc). To accommodate overdispersion we model the (log)rate as a linear combination of covariates and add a random eect for intersection with its own population distribution.

ENAR - March 26, 2006

Setting up GLIMs
Canonical link functions: Canonical link is function of mean that appears in exponent of exponential family form of sampling distribution. All links discussed so far are canonical except for the probit. Oset: Arises when counts are obtained from dierent population sizes or volumes or time periods and we need to use an exposure. Oset is a covariate with a known coecient. Example: Number of incidents in a given exposure time T are Poisson with rate per unit of time. Mean number of incidents is T . Link function would be log() = i, but here mean of y is not but T .
ENAR - March 26, 2006 7

 To apply the Poisson GLIM, add a column to X with values log(T ) and x the coecient to 1. This is an oset.

ENAR - March 26, 2006

Interpreting GLIMs
In linear models, j represents the change in the outcome when xj is changed by one unit. Here, j reects changes in g () when xj is changed. The eect of changing xj depends of current value of x. To translate eects into the scale of y , measure changes relative to a baseline y0 = g 1(x0 ). A change in x of x takes outcome from y0 to y where g (y0) = x0 y0 = g 1(x0 )
ENAR - March 26, 2006 9

and y = g 1(g (y0) + (x) )

ENAR - March 26, 2006

10

Priors in GLIM
We focus on priors for although sometimes is present and has its own prior. Non-informative prior for : With p( ) 1, posterior mode = MLE for Approximate posterior inference can be approximation to posterior at mode. Conjugate prior for : As in regression, express prior information about in terms of hypothetical data obtained under same model. Augment data vector and model matrix with y0 hypothetical observations and X0n0k hypothetical predictors.
ENAR - March 26, 2006 11

based

on

normal

 Non-informative prior for in augmented model. Non-conjugate priors: Often more natural to model p( |0, 0) = N (0, 0) with (0, 0) known. Approximate computation based on normal approximation (see next) particularly suitable. Hierarchical GLIM: Same approach as in linear models. Model some of the as exchangeable with common population distribution with unknown parameters. Hyperpriors for parameters.

ENAR - March 26, 2006

12

Computation
Posterior distributions of parameters can be estimated using MCMC methods in WinBUGS or other software. Metropolis within Gibbs will often be necessary: in GLIM, most often full conditionals do not have standard form. An alternative is to approximate the sampling distribution with a cleverly chosen approximation. Idea: ) perhaps conditional Find mode of likelihood (, hyperparameters Create pseudo-data with their pseudo-variances (see later) Model pseudo-data as normal with known (pseudo-)variances.
ENAR - March 26, 2006

on

13

Normal approximation to likelihood

2 Objective: nd zi and i such that normal likelihood 2 N (zi|(X )i, i )

is good approximation to GLIM likelihood p(yi|(X )i, ). ) be mode of (, ) so that Let (, i is the mode of i. For L the loglikelihood, write p(y1, ..., yn) = ip(yi|i, ) = i exp(L(yi|i, ))
ENAR - March 26, 2006 14

 Approximate factor in exponent by normal density in i: 1 L(yi|i, ) 2 (zi i)2, 2i

2 where (zi, i ) depend on (yi, i, ).

2 Now need to nd expressions for (zi, i ).

2 To get (zi, i ), match rst and second order terms in Taylor approx 2 2 around i to (i, i ) and solve for zi and for i .

Let L = L/i: 1 L = 2 (zi i) i

ENAR - March 26, 2006 15

2 Let L = 2L/i :

L = Then zi
2 i

1 2 i

) L (yi| i, = i ) L (yi| i, = 1 ) L (yi| i,

Example: binomial model with logit link: exp(i) L(yi, |i) = yi log 1 + exp(i) +(ni yi) log
ENAR - March 26, 2006

1 1 + exp(i)
16

= yii ni log(1 + exp(i)) Then L L = yi ni = ni exp(i) 1 + exp(i)

exp(i) (1 + exp(i))2

Pseudo-data and pseudo-variances: zi (1 + exp( i))2 = i + exp( i) 1 (1 + exp( i))2 ni exp( i) yi exp( i) ni 1 + exp( i)

2 i =

ENAR - March 26, 2006

17

Models for multinomial responses

Multinomial data: outcomes y = (yi, ..., yK ) are counts in K categories. Examples: Number of students receiving grades A, B, C, D or F Number of alligators that prefer to eat reptiles, birds, sh, invertebrate animals, or other (see example later) Number of survey respondents who prefer Coke, Pepsi or tap water. In Chapter 3, we saw non-hierarchical multinomial models:
j p(y |) k j =1 j

with j : probability of j th outcome and n.

ENAR - March 26, 2006

k j =1 j

= 1 and

k j =1 yj

18

 Here: we model j as a function of covariates (or predictors) X with corresponding regression coecients j . For full hierarchical structure, the j are modeled as exchangeable with some common population distribution p( |, ). Model can be developed as extension of either binomial or Poisson models.

ENAR - March 26, 2006

19

Logit model for multinomial data

Here i = 1, ..., I is number of covariate patters. E.g., in alligator example, 2 sizes four lakes = 8 covariate categories. Let yi be a multinomial random variable with sample size ni and k possible outcomes. Then yi Mult (ni; i1, ..., ik ) with
i yi = ni , and k j ij

= 1.

ij is the probability of j th outcome for ith covariate combination. Standard parametrization: log of the probability of j th outcome relative
ENAR - March 26, 2006 20

to baseline category j = 1: log ij i1 = ij = (Xj )i,

with j a vector of regression coecients for j th category. Sampling distribution:

k p(y | ) I i=1 j =1

exp(ij )
k l=1 exp(il )

yij

For identiability, 1 = 0 and thus i1 = 0 for all i. j is eect of changing X on probability of category j relative to category 1.
ENAR - March 26, 2006 21

 Typically, indicators for each outcome category are added to predictors to indicate relative frequency of each category when X = 0. Then ij = j + (Xj )i with 1 = 1 = 0 typically.

ENAR - March 26, 2006

22

Example from WinBUGS - Alligators

Agresti (1990) analyzes feeding choices of 221 alligators. Response is one of ve categories: sh, invertebrate, reptile, bird, other. Two covariates: length of alligator (less than 2.3 meters or larger than 2.3 meters) and lake (Hancock, Oklawaha, Traord, George). 2 4 = 8 covariate combinations (see data) For i, j a combination of size and lake, we have counts in ve possible categories yij = (yij 1, ..., yij 5). Model p(yij |ij , nij ) = Mult (yij |nij , ij 1, ..., ij 5)
ENAR - March 26, 2006 23

with ijk = and

exp(ijk )
k l=1 exp(ijl )

ijk = k + ik + jk . Here, k is baseline indicator for category k ik is coecient for indicator for lake jk is coecient for indicator for size

ENAR - March 26, 2006

24

box plot: alpha

0.0
[2]

[5]

[4]
[3]

-2.0

-4.0

-6.0

Mean alpha[2] alpha[3] alpha[4] alpha[5] -1.838 -2.655 -2.188 -0.7844

Std 0.5278 0.706 0.58 0.3691

2.5% -2.935 -4.261 -3.382 -1.531

Median -1.823 -2.593 -2.153 -0.7687

97.5% -0.8267 -1.419 -1.172 -0.1168

box plot: beta

5.0
[2,2]

[3,2]

[3,3]
[2,3]

[4,2]

2.5

[3,4] [3,5]

[4,3] [4,4]
[2,4] [2,5]

[4,5]

0.0

-2.5

-5.0

Mean beta[2,2] beta[2,3] beta[2,4] beta[2,5] beta[3,2] beta[3,3] beta[3,4] beta[3,5] beta[4,2] beta[4,3] beta[4,4] beta[4,5] 2.706 1.398 -1.799 -0.9353 2.932 1.935 0.3768 0.7328 1.75 -1.595 -0.7617 -0.843

Std 0.6431 0.8571 1.413 0.7692 0.6864 0.8461 0.7936 0.5679 0.6116 1.447 0.8026 0.5717

2.5% 1.51 -0.2491 -5.1 -2.555 1.638 0.37 -1.139 -0.3256 0.636 -4.847 -2.348 -1.97

Median 97.5% 2.659 1.367 -1.693 -0.867 2.922 1.886 0.398 0.7069 1.73 -1.433 -0.7536 -0.8492 4.008 3.171 0.5832 0.4413 4.297 3.842 1.931 1.849 3.023 0.9117 0.746 0.281

box plot: gamma

4.0

[2,4]

2.0

[2,3]

[2,5]

0.0
[2,2]

-2.0

-4.0

Mean
gamma[2,2] gamma[2,3] gamma[2,4] gamma[2,5] -1.523 0.342 0.7098 -0.353

Std
0.4101 0.5842 0.6808 0.4679

2.5%
-2.378 -0.788 -0.651 -1.216

Median
-1.523 0.3351 0.7028 -0.3461

97.5%
-0.7253 1.476 2.035 0.518

box plot: p[1,,]

box plot: p[2,,]

0.8
[1,2,1]

0.8
[1,1,1]

[2,1,2]

[2,2,1]

0.6

0.6

[2,1,1]

0.4

[1,1,5]
[1,2,5]

0.4
[1,2,4]

[2,2,2]
[2,2,3]

0.2

[1,1,2]
[1,1,4]

[1,2,3]

0.2

[2,1,3]
[2,1,5]
[2,2,4]
[2,2,5]

[1,1,3]
[1,2,2]
[2,1,4]

0.0

0.0

box plot: p[3,,]

box plot: p[4,,]

0.8
[3,1,2]

0.8

[4,2,1]

0.6

0.6

[4,1,1]
[4,1,2]

[3,2,1]

0.4
[3,1,1]
[3,1,5]
[3,2,2]

[3,2,3]

[3,2,5]

0.4

[3,2,4]

[4,2,2]

0.2

[3,1,3]

0.2
[3,1,4]
[4,1,4]

[4,1,5]

[4,2,4]

[4,2,5]

[4,2,3]

[4,1,3]

0.0

0.0

p[1,1,1] p[1,1,2] p[1,1,3] p[1,1,4] p[1,1,5] p[1,2,1] p[1,2,2] p[1,2,3] p[1,2,4] p[1,2,5] p[2,1,1] p[2,1,2] p[2,1,3] p[2,1,4] p[2,1,5] p[2,2,1] p[2,2,2] p[2,2,3] p[2,2,4] p[2,2,5] p[3,1,1] p[3,1,2] p[3,1,3] p[3,1,4] p[3,1,5] p[3,2,1]

0.5392 0.09435 0.04585 0.06837 0.2523 0.5679 0.02322 0.06937 0.1469 0.1927 0.2578 0.5968 0.08137 0.0095 0.05445 0.4612 0.2455 0.1947 0.0302 0.06833 0.1794 0.5178 0.09403 0.03554 0.1732 0.2937

0.07161 0.04288 0.02844 0.03418 0.06411 0.09029 0.01428 0.04539 0.07437 0.07171 0.07217 0.08652 0.0427 0.01186 0.03517 0.08211 0.06879 0.07042 0.02921 0.03984 0.05581 0.09136 0.04716 0.02504 0.06253 0.07225

0.4046 0.03016 0.007941 0.01923 0.1389 0.3959 0.005301 0.01191 0.03647 0.0793 0.1367 0.4159 0.01939 0.00533 0.009716 0.3055 0.1263 0.07776 7.653E-4 0.01357 0.08555 0.3334 0.02652 0.006063 0.07053 0.1618

0.5384 0.08735 0.04043 0.06237 0.2496 0.5684 0.02013 0.05918 0.1344 0.1852 0.2496 0.6 0.0728 0.04105 0.0476 0.4632 0.2426 0.189 0.02009 0.06097 0.1732 0.5185 0.08517 0.02996 0.167 0.2901

0.676 0.2067 0.1213 0.1457 0.3808 0.7342 0.06186 0.1768 0.3182 0.349 0.4265 0.7551 0.1841 0.143 0.1398 0.6175 0.3867 0.3502 0.1122 0.1643 0.296 0.6933 0.2219 0.1055 0.3261 0.4469

Interpretation of results
Because we set to zero several model parameters, interpreting results is tricky. For example: Beta[2,2] ha posterior mean 2.714. This is the effect of lake Oklawaha (relative to Hancock) on the alligators preference for invertebrates relative to fish. Since beta[2,2] > 0, we conclude that alligators in Oklawaha eat more invertebrates than do alligators in Hancock (even though both may prefer fish!). Gamma[2,2] is the effect of size 2 relative to size on the relative preference for invertebrates. Since gamma[2,2] < 0, we conclude that large alligators prefer fish more than do small alligators.

The alpha are baseline counts for each type of food relative to fish.

Hierarchical Poisson model

Count data are often modeled using a Poisson model. If y Poisson() then E (y ) = var(y ) = . When counts are assumed exchangeable given and the rates can also be assumed to be exchangeable, a Gamma population model for the rates is often chosen. The hierarchical model is then yi i
ENAR - March 26, 2006

Poisson(i) Gamma(, ).
25

 Priors for the hyperparameters are often taken to be Gamma (or exponential): with (a, b, c, d) known. The joint posterior distribution is
1 i p(, , |y ) iy exp{i } i exp{i }i

Gamma(a, b) Gamma(c, d),

a1 exp{b} c1 exp{d} To carry out Gibbs sampling we need to nd the full conditional distributions.
ENAR - March 26, 2006 26

 Conditional for i is
i +1 p(i| all) y exp{i( + 1)}, i

which is proportional to a Gamma with parameters (yi + , + 1). The full conditional for is
1 a1 p(| all) i exp{b}. i

The conditional for does not have a standard form. For : p( | all) i exp{i} c1 exp{d} c1 exp{ (
i
ENAR - March 26, 2006 27

i + d)},

which is proportional to a Gamma with parameters (c, Computation:

i i

+ d).

Given , , draw each i from the corresponding Gamma conditional. Draw using a Metropolis step or rejection sampling or inverse cdf method. Draw from the Gamma conditional. See Italian marriages example.

ENAR - March 26, 2006

28

Italian marriages Example

Gill (2002) collected data on the number of marriages per 1,000 people in Italy during 1936-1951. Question: did the number of marriages decrease during WWII years? (1939 1945). Model: Number of marriages yi are Poisson with year-specific means i. Assuming that rates of marriages are exchangeable across years, we model the i as Gamma(, ). To complete model specification, place independent Gamma priors on (, ), with known hyper-parameter values.

WinBUGS code: model { for (i in 1:16) { y[i] ~ dpois(l[i]) l[i] ~ dgamma(alpha, beta) } alpha ~ dgamma(1,1) beta ~ dgamma(1,1) warave <- (l[4]+l[5]+ l[6]+l[7]+l[8]+l[9]+l[10]) / 7 nonwarave<- (l[1]+l[2]+l[3]+l[11]+l[12]+l[13]+l[14]+l[15]+l[16]) / 9 diff <- nonwarave - warave } list(y = c(7,9,8,7,7,6,6,5,5,7,9,10,8,8,8,7))

Results
lambda: 95% credible sets
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16]

0.0

5.0

10.0

lambda: box plot

15.0
[2] [3] [1] [4] [5] [6] [7] [8] [9] [10] [11]

[12]

[13]

[14]

[15] [16]

10.0

5.0

0.0

Difference between non-war and war years marriage rate

Nonwar - war average 0.4 0.3 0.2 0.1 0.0 -5.0 0.0 5.0

Overall marriage rate: If i ~ Gamma(, ), then E(i | y) = / .

overallrate overallstd mean sd 7.362 1.068 2.281 0.394 2.5% median 5.508 7.285 1.59 2.266 97.5% 9.665 3.125

overallrate chains 1:3 sample: 3000 0.6 0.4 0.2 0.0 4.0 6.0 8.0 10.0

overallstd chains 1:3 sample: 3000 1.5 1.0 0.5 0.0 1.0 2.0 3.0 4.0

Poisson regression
When rates are not exchangeable, we need to incorporate covariates into the model. Often we are interested in the association between one or more covariate and the outcome. It is possible (but not easy) to incorporate covariates into the PoissonGamma model. Christiansen and Morris (1997, JASA) propose the following model: Sampling distribution, where ei is a known exposure: yi|i Poisson(iei). Under model, E (yi/ei) = i.
ENAR - March 26, 2006 29

 Population distribution for the rates: i| Gamma(, /i), with log(i) = xi , and = (0, 1, ..., k1, ). is thought of as an unboserved prior count. Under population model, E (i) = /i = i CV2(i) = =
ENAR - March 26, 2006

2 i 1 2 i 1
30

 For k = 0, i is known. For k = 1, i are exchangeable. For k 2, i are (unconditionally) nonexchangeable. In all cases, standardized rates i/i are exchangeable, and have expectation 1. The covariates can include random eects. To complete specication of model, we need priors on . Christensen and Morris (1997) suggest: and independent a priori. Non-informative prior on s associated to xed eects. For a proper prior of the form: p( |y0)
ENAR - March 26, 2006

Gamma(, ),

are

y0 , 2 ( + y0)
31

where y0 is the prior guess for the median of . and the MLE of ) provide Small values of y0 (for example, y0 < less information. When the rates cannot be assumed to be exchangeable, it is common to choose a generalized linear model of the form:
i p(y | ) i exp{i}y i

i exp{ exp(i)}[exp(i)]yi , for i = xi and log(i) = i. The vector of covariates can include one or more random eects to accommodate additional dispersion (see epylepsy example).
ENAR - March 26, 2006 32

 The second-level distribution for the s will typically be at (if covariate is a xed eect) or normal
2 j Normal(j 0, ) j 2 if j th covariate is a random eect. The variance represents the j between batch variability.

ENAR - March 26, 2006

33

Epilepsy example
From Breslow and Clayton, 1993, JASA. Fifty nine epilectic patients in a clinical trial were randomized to a new drug: T = 1 is the drug and T = 0 is the placebo. Covariates included: Baseline data: number of seizures during eight weeks preceding trial Age in years. Outcomes: number of seizures during the two weeks preceding each of four clinical visits. Data suggest that number of seizures was signicantly lower prior to fourth visit, so an indicator was used for V4 versus the others.
ENAR - March 26, 2006 34

 Two random eects in the model: A patient-level eect to introduce between patient variability. A patients by visit eect to introduce between visit within patient dispersion.

ENAR - March 26, 2006

35

Epilepsy study Program and results

model { for(j in 1 : N) { for(k in 1 : T) { log(mu[j, k]) <- a0 + alpha.Base * (log.Base4[j] - log.Base4.bar) + alpha.Trt * (Trt[j] - Trt.bar) + alpha.BT * (BT[j] - BT.bar) + alpha.Age * (log.Age[j] - log.Age.bar) + alpha.V4 * (V4[k] - V4.bar) + b1[j] + b[j, k] y[j, k] ~ dpois(mu[j, k]) b[j, k] ~ dnorm(0.0, tau.b); # subject*visit random effects } b1[j] ~ dnorm(0.0, tau.b1) # subject random effects BT[j] <- Trt[j] * log.Base4[j] # interaction log.Base4[j] <- log(Base[j] / 4) log.Age[j] <- log(Age[j]) diff[j] <- mu[j,4] mu[j,1] } # covariate means: log.Age.bar <- mean(log.Age[]) Trt.bar <- mean(Trt[]) BT.bar <- mean(BT[]) log.Base4.bar <- mean(log.Base4[]) V4.bar <- mean(V4[]) # priors: a0 ~ dnorm(0.0,1.0E-4) alpha.Base ~ dnorm(0.0,1.0E-4) alpha.Trt ~ dnorm(0.0,1.0E-4);

alpha.BT ~ dnorm(0.0,1.0E-4) alpha.Age ~ dnorm(0.0,1.0E-4) alpha.V4 ~ dnorm(0.0,1.0E-4) tau.b1 ~ dgamma(1.0E-3,1.0E-3); sigma.b1 <- 1.0 / sqrt(tau.b1) tau.b ~ dgamma(1.0E-3,1.0E-3); sigma.b <- 1.0/ sqrt(tau.b) # re-calculate intercept on original scale: alpha0 <- a0 - alpha.Base * log.Base4.bar - alpha.Trt * Trt.bar - alpha.BT * BT.bar - alpha.Age * log.Age.bar - alpha.V4 * V4.bar }

Results
Parameter alpha.Age alpha.Base alpha.Trt alpha.V4 alpha.BT sigma.b1 sigma.b Mean 0.4677 0.8815 -0.9587 -0.1013 0.3778 0.4983 0.3641 Std 0.3557 0.1459 0.4557 0.08818 0.2427 0.07189 0.04349 2.5th -0.2407 0.5908 -1.794 -0.273 -0.1478 0.3704 0.2871 Median 0.4744 0.8849 -0.9637 -0.09978 0.3904 0.4931 0.362 97.5th 1.172 1.165 -0.06769 0.07268 0.7886 0.6579 0.4552

Individual random effects

box plot: b1

2.0
[10]
[8]
[4]
[1] [2]

[25]

[35]
[32] [33]

[56]
[46]
[43] [42]

[36]

[49]

[3]

1.0

[18]
[11]

[22]

[28]
[24]

[37]

[40]

[53]
[47]

[55]

[59]

[5]

[6]

[7]

[9]

[12] [14] [13] [15]

[21] [20] [19]

[27]

[45] [44]
[30][31] [29]

[39]
[34]

[48]
[41]

[23]

[50] [51]

[26]

[54]

[57]
[52]

0.0

[17] [16]

[38]

[58]

-1.0

-2.0

Diff in seizure counts between V4 and V1

[1] [2] [3] [4]

[8]

[7]

[6]

[5]

[49]

[9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59]

[25]

-100.0

-50.0

0.0

Patients 5 and 49 are different. For #5, the number of events increases from visits 1 through 4. For #49, there is a significant decrease.
box plot: b[5, ] box plot: b[49, ]

1.0
[5,2]

[5,4]

1.0
[49,1]

0.5
[5,3] [5,1]

0.5
[49,2]

[49,3]

[49,4]

0.0

0.0 -0.5

-1.0

-0.5

Convergence and autocorrelation

We ran three parallel chains for 10,000 iterations Discarded the first 5,000 iterations from each chain as burn-in Thinned by 10, so there are 1,500 draws for inference.
sigma.b chains 1:3
1.0 0.5 0.0 -0.5 -1.0
1.0 0.5 0.0 -0.5 -1.0

sigma.b1 chains 1:3

20

40

20

40

lag

lag

sigma.b chains 1:3

1.5

sigma.b1 chains 1:3

1.5

1.0

1.0

0.5

0.5

0.0

0.0
5001 6000
8000

5001 6000

8000

iteration

iteration

Example: hierarchical Poisson model

The USDA collects data on the number of farmers who adopt conservation tillage practices. In a recent survey, 10 counties in a midwestern state were randomly sampled, and within county, a random number of farmers were interviewed and asked whether they had adopted conservation practices. The number of farmers interviewed in each county varied from a low of 2 to a high of 100. Of interest to USDA is the estimation of the overall adoption rate in the state.

 We tted the following model: yi i Poisson(i) Gamma(, ),

i = i Ei

where yi is the number of adopters in the ith county, Ei is the number of farmers interviewed, i is the expected adoption rate, and the expected number of adopters in a county can be obtained by multiplying i by the number of farms in the county. The hierarchical structure establishes that even though counties may vary signicantly in terms of the rate of adoption, they are still exchangeable, so the rates are generated from a common population distribution. Information about the overall rate of adoption across the state is

contained in the posterior distribution of (, ). We tted the model using WinBUGS and chose non-informative priors for the hyperparameters. Observed data:

County 1 2 3 4 5

Ei 94 15 62 126 5

yi 5 1 5 14 3

County 6 7 8 9 10

Ei 31 2 2 9 100

yi 19 1 1 4 22

Posterior distribution of rate of adoption in each county

box plot: theta

1.5
[5]

[7]

[8]

1.0
[6]

[9]

0.5
[2]

[10]
[3]

[4]

[1]

0.0

node

mean

sd 0.02414 0.08086 0.03734 0.03085 0.2859 0.1342 0.3549 0.3689 0.2055 0.0486

2.5% 0.02127 0.009868 0.03248 0.06335 0.131 0.3651 0.0482 0.04501 0.1352 0.1386

median 97.5% 0.05765 0.08742 0.08618 0.1132 0.4954 0.5962 0.3646 0.3805 0.3988 0.2211 0.1147 0.306 0.1767 0.1831 1.242 0.8932 1.343 1.41 0.9255 0.33

theta[1] 0.06033 theta[2] 0.1074 theta[3] 0.09093 theta[4] 0.1158 theta[5] 0.5482 theta[6] 0.6028 theta[7] 0.4586 theta[8] 0.4658 theta[9] 0.4378 theta[10] 0.2238

Posterior distribution of number of adopters in each county, given exposures

box plot: lambda

40.0

[10]

30.0
[4]

[6]

20.0

[1]

[3]

10.0
[5]
[2]

[9]

[7]

[8]

0.0

node lambda[1] lambda[2] lambda[3] lambda[4] lambda[5] lambda[6] lambda[7] lambda[8] lambda[9] lambda[10]

mean 5.671 1.611 5.638 14.59 2.741 18.69 0.9171 0.9317 3.94 22.38

sd 2.269 1.213 2.315 3.887 1.43 4.16 0.7097 0.7377 1.85 0.1028

2.5% 2.0 0.148 2.014 7.982 0.6551 11.32 0.09641 0.09002 1.217 13.86

median 97.5% 5.419 1.311 5.343 14.26 2.477 18.48 0.7292 0.761 3.589 22.11 10.78 4.59 10.95 23.07 6.21 27.69 2.687 2.819 8.33 33.0

node alpha beta mean

mean

sd

2.5%

median

97.5%

0.8336 0.3181 0.3342 0.7989 1.582 2.094 1.123 0.4235 1.897 4.835 0.4772 0.2536 0.1988 0.4227 1.14

Poisson model for small area deaths

Taken from Bayesian Statistical Modeling, Peter Congdon, 2001

Congdon considers the incidence of heart disease mortality in 758 electoral wards in the Greater London area over three years (1990-92). These small areas are grouped administratively into 33 boroughs.

Regressors: at ward level: xij , index of socio-economic deprivation. at borough level: wj , where


wi =

1 0

for inner London boroughs for outer suburban boroughs

We assume borough level variation in the intercepts and in the impacts of deprivation; this variation is linked to the category of borough (inner vs outer).

Model
First level of the hierarchy Oij |ij Poisson(ij )

log(ij ) = log(Eij ) + 1j + 2j (xij x ) + ij

ij N (0, )
Death counts Oij are Poisson with means ij . log(Eij ) is an oset, i.e. an explanatory variable with known coecient (equal to 1).

 j = (1j , 2j ) are random coecients for the intercepts and the impacts of deprivation at the borough level. ij is a random error for Poisson over-dispersion. We tted two models: one without and another with this random error term.

Model (contd)
Second level of the hierarchy


j =
where

1j 2j


N2(j , )

11 12 = 21 22 1j = 11 + 12wj 2j = 21 + 22wj 11, 12, 21, and 22 are the population coecients for the intercepts, the impact of borough category, the impact of deprivations, and, respectively, the interaction impact of the level-2 regressors and level-1 regressors. Hyperparameters   1 1 Wishart (R) , ij N (0, 0.1) i, j {1, 2} 2 Inv-Gamma(a, b)

Computation of Eij
Suppose that p is an overall disease rate. Then, Eij = nij p and ij = pij /p. The s are said: 1. externally standardized if p is obtained from another data source (such as a standard reference table); 2. internally standardized if p is obtained from the given dataset, e.g.

X
p =

Oij nij

X
ij ij

In our example we rely on the latter approach. Under (a), the joint distribution of the Oij is a product Poisson; under (b) is multinomial. P However, since likelihood inference is unaected by whether we condition on ij Oij , the product Poisson likelihood is commonly retained.

Model 1: without overdispersion term (i.e. without ij )

node 11 12 21 22 1 2 Deviance

mean -0.075 0.078 0.621 0.104 0.294 0.445 945.800

std. dev. 0.074 0.106 0.138 0.197 0.038 0.077 11.320

Posterior quantiles 2.5% median 97.5% -0.224 -0.075 0.070 -0.128 0.078 0.290 0.354 0.620 0.896 -0.282 0.103 0.486 0.231 0.290 0.380 0.318 0.437 0.618 925.000 945.300 969.400

Model 2: with overdispersion term

node 11 12 21 22 1 2 Deviance mean -0.069 0.068 0.616 0.105 0.292 0.431 802.400 std. dev. 0.074 0.106 0.141 0.198 0.037 0.075 40.290 Posterior quantiles 2.5% median 97.5% -0.218 -0.069 0.076 -0.141 0.068 0.278 0.335 0.615 0.892 -0.276 0.104 0.499 0.228 0.289 0.376 0.306 0.423 0.600 726.600 800.900 883.500

Including ward-level random variability ij reduces the average Poisson GLM deviance to 802, with a 95% credible interval from 726 to 883. This is in line with the expected value of the GLM deviance for N = 758 areas if the Poisson Model is appropriate.

Hierarchical models for spatial data

Based on the book by Banerjee, Carlin and Gelfand Hierarchical Modeling and Analysis for Spatial Data, 2004. We focus on Chapters 1, 2 and 5. Geo-referenced data arise in agriculture, climatology, economics, epidemiology, transportation and many other areas. What does geo-referenced mean? In a nutshell, we know the geographic location at which an observation was collected. Why does it matter? Sometimes, relative location can provide information about an outcome beyond that provided by covariates. Example: infant mortality is typically higher in high poverty areas. Even after incorporating poverty as a covariate, residuals may still be spatially correlated due to other factors such as nearness to pollution sites, distance to pre and post-natal care centers, etc.
ENAR - March 26, 2006 1

 Often data are also collected over time, so models that include spatial and temporal correlations of outcomes are needed. We focus on spatial, rather than spatio-temporal models. We also focus on models for univariate rather than multivariate outcomes.

ENAR - March 26, 2006

Types of spatial data

Point-referenced data: Y (s) a random outcome (perhaps vectorvalued) at location s, where s varies continuously over some region D. The location s is typically two-dimensional (latitude and longitude) but may also include altitude. Known as geostatistical data. Areal data: outcome Yi is an aggregate value over an areal unit with well-dened boundaries. Here, D is divided into a nite collection of areal units. Known as lattice data even though lattices can be irregular. Point-pattern data: Outcome Y (s) is the occurrence or not of an event and locations s are random. Example: locations of trees of a species in a forest or addresseses of persons with a particular disease. Interest is often in deciding whether points occur independently in space or whether there is clustering.
ENAR - March 26, 2006 3

 Marked point process data: If covariate information is available we talk about a marked point process. Covariate value at each site marks the site as belonging to a certain covariate batch or group. Combinations: e.g. ozone daily levels collected in monitoring stations with precise location, and number of children in a zip code reporting to the ER on that day. Require data re-alignment to combine outcomes and covariates.

ENAR - March 26, 2006

Models for point-level data

The basics Location index s varies continuously over region D. We often assume that the covariance between two observations at locations si and sj depends only on the distance dij between the points. The spatial covariance is often modeled as exponential: Cov (Y (si), Y (si )) = C (dii ) = 2edii , where ( 2, ) > 0 are the partial sill and decay parameters, respectively. Covariogram: a plot of C (dii ) against dii .
ENAR - March 26, 2006 5

 For i = i , dii = 0 and C (dii ) = var(Y (si)). Sometimes, var(Y (si)) = 2 + 2, for 2 the nugget eect and 2 + 2 the sill.

ENAR - March 26, 2006

Models for point-level data (contd)

Covariance structure Suppose that outcomes are normally distributed and that we choose an exponential model for the covariance matrix. Then: Y |, N(, ()), with Y ()ii = {Y (s1), Y (s2), ..., Y (sn)} = cov(Y (si), Y (si ))

= ( 2, 2, ). Then ()ii = 2 exp(dii ) + 2Ii=i ,

ENAR - March 26, 2006 7

with ( 2, 2, ) > 0. This is an example of an isotropic covariance function: the spatial correlation is only a function of d.

ENAR - March 26, 2006

Models for point-level data, details

Basic model: Y (s) = (s) + w(s) + e(s), where (s) = x (s) and the residual is divided into two components: w(s) is a realization of a zero-centered stationary Gaussian process and e(s) is uncorrelated pure error. The w(s) are functions of the partial sill 2 and decay parameters. The e(s) introduces the nugget eect 2. 2 interpreted as pure sampling variability or as microscale variability, i.e., spatial variability at distances smaller than the distance between two outcomes: the e(s) are sometimes viewed as spatial processes with rapid decay.
ENAR - March 26, 2006 9

The variogram and semivariogram

A spatial process is said to be: Strictly stationary if distributions of Y (s) and Y (s + h) are equal, for h the distance. Weakly stationary if (s) = and Cov (Y (s), Y (s + h)) = C (h). Instrinsically stationary if E [Y (s + h) Y (s)] = 0, and E [Y (s + h) Y (s)]2 = V ar[Y (s + h) Y (s)] = 2 (h), dened for dierences and depending only on distance. 2 (h) is the variogram and (h) is the semivariogram
ENAR - March 26, 2006 10

Examples of semi-variograms
Semi-variograms for the linear, spherical and exponential models.

ENAR - March 26, 2006

11

Stationarity
Strict stationarity implies weak stationarity but the converse is not true except in Gaussian processes. Weak stationarity implies intrinsec stationarity, but the converse is not true in general. Notice that intrinsec stationarity is dened on the dierences between outcomes at two locations and thus says nothing about the joint distribution of outcomes.

ENAR - March 26, 2006

12

Semivariogram (contd)
If (h) depends on h only through its length ||h||, then the spatial process is isotropic. Else it is anisotropic. There are many choices for isotropic models. The exponential model is popular and has good properties. For t = ||h||: (t) = 2 + 2(1 exp(t)) if t > 0, = 0 otherwise. See gures, page 24. The powered exponential model has an extra parameter for smoothness: (t) = 2 + 2(1 exp(t)) if t > 0
ENAR - March 26, 2006 13

 Another popular choice is the Gaussian variogram model, equal to the exponential except for the exponent term, that is exp(2t2)). Fitting of the variogram has been traditionally done by eye: Plot an empirical estimate of the variogram akin to the sample variance estimate or the autocorrelation function in time series Choose a theoretical functional form to t the empirical Choose values for ( 2, 2, ) that t the data well. If a distribution for the outcomes is assumed and a functional form for the variogram is chosen, parameter estimates can be estimated via some likelihood-based method. Of course, we can also be Bayesians.

ENAR - March 26, 2006

14

Point-level data (contd)

For point-referenced data, frequentists focus on spatial prediction using kriging. Problem: given observations {Y (s1), ..., Y (sn)}, how do we predict Y (so) at a new site so? Consider the model Y = X + , where and where = 2H () + 2I. Here, H ()ii = (, dii ).
ENAR - March 26, 2006 15

N(0, ),

 Kriging consists in nding a function f (y ) of the observations that minimizes the MSE of prediction Q = E [(Y (so) f (y ))2|y ].

ENAR - March 26, 2006

16

Classical kriging (contd)

(Not a surprising!) Result: f (y ) that minimizes Q is the conditional mean of Y (s0) given observations y (see pages 50-52 for proof): + ) 1(y X E [Y (so)|y ] = xo 1 V ar[Y (so)|y ] = 2 + 2 , where do1), ..., don)) = ( 2(, 2(, = (X 1X )1X 1y ). = 2H ( Solution assumes that we have observed the covariates xo at the new site.
ENAR - March 26, 2006 17

 If not,in the classical framework Y (so), xo are jointly estimated using an EM-type iterative algorithm.

ENAR - March 26, 2006

18

Bayesian methods for estimation

The Gaussian isotropic kriging model is just a general linear model similar to those in Chapter 15 of textbook. Just need to dene the appropriate covariance structure. For an exponential covariance structure with a nugget eect, parameters to be estimated are = (, 2, 2, ). Steps: Choose priors and dene sampling distribution Obtain posterior for all parameters p(|y ) Bayesian kriging: get posterior predictive distribution for outcome at new location p(yo|y, X, xo).
ENAR - March 26, 2006 19

 Sampling distribution (marginal data model) y | N(X, 2H () + 2I ) Priors: typically chosen so that parameters are independent a priori. As in the linear model: Non-informative prior for is uniform or can use a normal prior too. Conjugate priors for variances 2, 2 are inverse gamma priors. For , appropriate prior depends on covariance model. For simple exponential where (si sj ; ) = exp(||si sj ||), a Gamma prior can be a good choice.
ENAR - March 26, 2006 20

 Be cautious with improper priors for anything but .

ENAR - March 26, 2006

21

Hierarchical representation of model

Hierarchical model representation: rst condition on the spatial random eects W = {w(s1), ..., w(sn)}: y |, W N(X + W, 2I ) N(0, 2H ()).

W |, 2

Model specication is then completed by choosing priors for , 2 and for , 2 (hyperparameters). Note that hierarchical model has n more parameters (the w(si)) than the marginal model. Computation with the marginal model preferable because 2H ()+ 2I tends to be better behaved than 2H () at small distances.
ENAR - March 26, 2006 22

Estimation of spatial surface W |y

Interest is sometimes on estimating the spatial surface using p(W |y ). If marginal model is tted, we can still get marginal posterior for W as p(W |y ) = p(W | 2, )p( 2, |y )d 2d.

Given draws ( 2(g), (g)) from the Gibbs sampler on the marginal model, we can generate W from p(W | 2(g), (g)) = N(0, 2(g)H ((g))). Analytical marginalization over W is possible only if model has Gaussian form.
ENAR - March 26, 2006 23

Bayesian kriging
Let Yo = Y (so) and xo = x(so). Kriging is accomplished by obtaining the posterior predictive distribution p(yo|xo, X, y ) = = p(yo, |y, X, xo)d p(yo|, y, xo)p(|y, X )d.

Since (Yo, Y ) are jointly multivariate normal (see expressions 2.18 and 2.19 on page 51), then p(yo|, y, xo) is a conditional normal distribution. Given MCMC draws of the parameters ((1), ..., (G)) from the posterior (g ) distribution p(|y, X ), we draw values yo for each (g) as
(g ) yo p(yo|(g), y, xo).
ENAR - March 26, 2006 24

 Draws {yo , yo , ..., yo , } are a sample from the posterior predictive distribution of the outcome at the new location so. To predict Y at a set of m new locations so1, ..., som, it is best to do joint prediction to be able to estimate the posterior association among m predictions. Beware of joint prediction at many new locations with WinBUGS. It can take forever.

(1)

(2)

(G)

ENAR - March 26, 2006

25

Kriging example from WinBugs

Data were rst published by Davis (1973) and consist of heights at 52 locations in a 310-foot square area. We have 52 s = (x, y ) coordinates and outcomes (heights). Unit of distance is 50 feet and unit of elevation is 10 feet. The model is height = + , where and where = 2H ().
ENAR - March 26, 2006 26

N(0, ),

 Here, H ()ij = (si sj ; ) = exp(||si sj ||). Priors on (, , ). We predict elevations at 225 new locations.

ENAR - March 26, 2006

27

model { # Spatially structured multivariate normal likelihood height[1:N] ~ spatial.exp(mu[], x[], y[], tau, phi, kappa) for(i in 1:N) { mu[i] <- beta } # Priors beta ~ dflat() tau ~ dgamma(0.001, 0.001) sigma2 <- 1/tau # priors for spatial.exp parameters phi ~ dunif(0.05, 20) # prior decay for correlation at min distance (0.2 x 50 ft) is 0.02 to 0.99 # prior range for correlation at max distance (8.3 x 50 ft) is 0 to 0.66 kappa ~ dunif(0.05,1.95)

# exponential correlation function

# Spatial prediction # Single site prediction for(j in 1:M) { height.pred[j] ~ spatial.unipred(beta, x.pred[j], y.pred[j], height[]) } # Only use joint prediction for small subset of points, due to length of time it takes to run for(j in 1:10) { mu.pred[j] <- beta } height.pred.multi[1:10] ~ spatial.pred(mu.pred[], x.pred[1:10], y.pred[1:10], height[]) }

Data

Click on one of the arrows for the data

Initial values
Click on one of the arrows for inital values for spatial.exp model

beta 1200.0 1.00E+3 800.0 600.0 400.0 1 500 1000 iteration 1500 2000

phi 0.8 0.6 0.4 0.2 0.0 1 500 1000 iteration 1500 2000

kappa 2.0 1.5 1.0 0.5 1 500 1000 iteration 1500 2000

(samples)means for height.pred

(1) < 700.0 (16) 700.0 - 750.0

N
(40) 750.0 - 800.0 (80) 800.0 - 850.0 (61) 850.0 - 900.0 (26) 900.0 - 950.0 (1) >= 950.0

0.002km

Hierarchical models for areal data

Areal data: often aggregate outcomes over a well-dened area. Example: number of cancer cases per county or proportion of people living in poverty in a set of census tracks. What are the inferential issues? 1. Identifying a spatial pattern and its strength. If data are spatially correlated, measurements from areas that are close will be more alike. 2. Smoothing and to what degree. Observed measurements often present extreme values due to small samples in small areas. Maximal smoothing: substitute observed measurements by the overall mean in the region. Something less extreme is what we discuss later. 3. Prediction: for a new area, how would we predict Y given measurements in other areas?
ENAR - March 26, 2006 28

Dening neighbors
A proximity matrix W with entries wij spatially connects areas i and j in some fashion. Typically, wii = 0. There are many choices for the wij : Binary: wij = 1 if areas i, j share a common boundary, and is 0 otherwise. Continuous: decreasing function of intercentroidal distance. Combo: wij = 1 if areas are within a certain distance. W need not be symmetric. Entries are often standardized by dividing into j wij = wi+. If entries are standardized, the W will often be asymmetric.
ENAR - March 26, 2006 29

 The wij can be thought of as weights and provide a means to introduce spatial structure into the model. Areas that are closer by in some sense are more alike. For any problem, we can dene rst, second, third, etc order neighbors. For distance bins (0, d1], (d1, d2], (d2, d3], ... we can dene 1. W (1), the rst-order proximity matrix with wij = 1 if distance between i abd j is less than d1. (2) 2. W (2), the second-order proximity matrix with wij = 1 if distance between i abd j is more than d1 but less than d2.
(1)

ENAR - March 26, 2006

30

Areal data models

Because these models are used mostly in epidemiology, we begin with an application in disease mapping to introduce concepts. Typical data: Yi observed number of cases in area i, i = 1, ..., I Ei expected number of cases in area i. The Y s are assumed to be random and the E s are assumed to be known and to depend on the number of persons ni at risk. An internal standardized estimate of Ei is Ei = ni r = ni
ENAR - March 26, 2006

i yi i ni

,
31

corresponding to a constant disease rate across areas. An external standardized estimate is Ei =

j

nij rj ,

where rj is the risk for persons of age group j (from some existing table of risks by age) and nij is the number of persons of age j in area i.

ENAR - March 26, 2006

32

Standard frequentist approach

For small Ei, Yi|i Poisson(Eii), with i the true relative risk in area i. The MLE is the standard mortality ratio i = SM Ri = Yi . Ei

The variance of the SM Ri is var(Yi) i var(SM Ri) = = , 2 Ei Ei

ENAR - March 26, 2006 33

estimated by plugging i to obtain Yi var(SM Ri) = 2 . Ei

To get a condence interval for i, rst assume that log(SM Ri) is approximately normal.

From a Taylor expansion: V ar[log(SM Ri)] =

ENAR - March 26, 2006

1 2 V ar (SM Ri ) SM Ri
2 Yi 1 Ei 2= . 2 Yi Ei Yi
34

 An approximate 95% CI for log(i) is SM Ri 1.96/(Yi)1/2. Transforming back, an approximate 95% CI for i is (SM Ri exp(1.96/(Yi)1/2), SM Ri exp(1.96/(Yi)1/2)). Suppose we wish to test whether risk in area i is high relative to other areas. Then test H0 : i = 1 versus Ha : i > 1. This is a one-sided test.
ENAR - March 26, 2006 35

 Under H0, Yi Poisson(Ei) so the pvalue for the test is p = Prob(X Yi|Ei)
Yi 1

= 1 Prob(X Yi|Ei) = 1
x=0 x exp(Ei)Ei . x!

If p < 0.05 we reject H0.

ENAR - March 26, 2006

36

Hierarchical models for areal data

To estimate and map underlying relative risks, might wish to t a random eects model. Assumption: true risks come from a common underlying distribution. Random eects models permit borrowing strength across areas to obtain better area-level estimates. Alas, models may be complex: High-dimensional: one random eect for each area. Non-normal if data are counts or binomial proportions. We have already discussed hierarchical Poisson models, so material in the next few transparencies is a review.
ENAR - March 26, 2006 37

Poisson-Gamma model
Consider Yi|i i|a, b Poisson(Eii) Gamma(a, b).

Since E (i) = a/b and V ar(i) = a/b2, we can x a, b as follows: A priori, let E (i) = 1, the null value. Let V ar(i) = (0.5)2, large on that scale. Resulting prior is Gamma(4, 4). Posterior is also Gamma: p(i|yi) = Gamma(yi + a, Ei + b).
ENAR - March 26, 2006 38

 A point estimate of i is E (i|y ) = E (i|yi) = = yi + a Ei + b

[E (i )]2 yi + V ar(i) (i ) Ei + VE ar (i ) yi Ei ( E ) i

Ei +

E (i ) V ar (i )

E (i ) V ar (i ) E (i ) (i ) Ei + VE ar (i )

= wiSM Ri + (1 wi)E (i), where wi = Ei/[Ei + (E (i)/V ar(i))]. Bayesian point estimate is a weighted average of the data-based SM Ri and the prior mean E (i).
ENAR - March 26, 2006 39

Poisson-lognormal models with spatial errors

The Poisson-Gamma model does not allow (easily) for spatial correlation among the i. Instead, consider the Poisson-lognormal model, where in the second stage we model the log-relative risks log(i) = i: Yi|i Poisson(Ei exp(i))

i = xi + i + i, where xi are area-level covariates. The i are assumed to be exchangeable and model between-area variability: i N(0, 1/h).
ENAR - March 26, 2006 40

 The i incorporate global extra-Poisson variability in the log-relative risks (across the entire region). The i are the spatial parameters; they capture regional clustering. They model extra-Poisson variability in the log-relative risks at the local level so that neighboring areas have similar risks. One way to model the i is to proceed as in the point-referenced data case. For = (1, ..., I ), consider |, NI (, H ()), and H ()ii = cov (i, i ) with hyperparameters . Possible models for H () include the exponential, the powered exponential, etc.
ENAR - March 26, 2006 41

 While sensible, this model is dicult to t because Lots of matrix inversions required Distance between i and i may not be obvious.

ENAR - March 26, 2006

42

CAR model
More reasonable to think of a neighbor-based proximity measure and consider a conditionally autoregressive model for : i, 1/(cmi)), i N( where i =
i=j

wij (i j ),

and mi is the number of neighbors of area i. Earlier we called this wi+. The weights wij are (typically) 0 if areas i and j are not neighbors and 1 if they are. CAR models lend themselves to the Gibbs sampler. Each i can be sampled from its conditional distribution so no matrix inversion is
ENAR - March 26, 2006 43

needed: p(i| all) Poi(yi|Eie

xi +i +i

1 ) N(i|i, ). mic

ENAR - March 26, 2006

44

Diculties with CAR model

The CAR prior is improper. Prior is a pairwise dierence prior identied only up to a constant. The posterior will still be proper, but to identify an intercept 0 for the log-relative risks, we need to impose a constraint: i i = 0. In simulation, constraint is imposed numerically by recentering each vector around its own mean. h and c cannot be too large because i and i become unidentiable. We observe only one Yi in each area yet we try to t two random eects. Very little data information. Hyperpriors for h, c need to be chosen carefully.
ENAR - March 26, 2006 45

 Consider h Gamma(ah, bh), c Gamma(ac, bc).

To place equal emphasis on heterogeneity and spatial clustering, it is tempting to make ah = ac and bh = bc. This is not correct because 1. The h prior is dened marginally, where the c prior is conditional. 2. The conditional prior precision is cmi. Thus, a scale that satises sd(i) = 1 1 sd(i) 0 . 7 m h ) c

with m the average number of neighbors is more fair (Bernardinelli et al. 1995, Statistics in Medicine).

ENAR - March 26, 2006

46

Example: Incidence of lip cancer in 56 areas in Scotland

Data on the number of lip cancer cases in 56 counties in Scotland were obtained. Expected lip cancer counts Ei were available. Covariate was the proportion of the population in each county working in agriculture. Model included only one random effect bi to introduce spatial association between counties. Two counties are neighbors if they have a common border (i.e., they are adjacent). Three counties that are islands have no neighbors and for those, WinBUGS sets the random spatial effect to be 0. The relative risk for the islands is thus based on the baseline rate 0 and on the value of the covariate xi. We wish to smooth and map the relative risks RR.

Model model { # Likelihood for (i in 1 : N) { O[i] ~ dpois(mu[i]) log(mu[i]) <- log(E[i]) + alpha0 + alpha1 * X[i]/10 + b[i] RR[i] <- exp(alpha0 + alpha1 * X[i]/10 + b[i]) relative risk (for maps) } # CAR prior distribution for random effects: b[1:N] ~ car.normal(adj[], weights[], num[], tau) for(k in 1:sumNumNeigh) { weights[k] <- 1 } # Other priors: alpha0 ~ dflat() alpha1 ~ dnorm(0.0, 1.0E-5) tau ~ dgamma(0.5, 0.0005) sigma <- sqrt(1 / tau) }

# Area-specific

# prior on precision # standard deviation

Data
list(N = 56, O = c( 9, 39, 11, 9, 15, 8, 26, 7, 6, 20, 13, 5, 3, 8, 17, 9, 2, 7, 9, 7, 16, 31, 11, 7, 19, 15, 7, 10, 16, 11, 5, 3, 7, 8, 11, 9, 11, 8, 6, 4, 10, 8, 2, 6, 19, 3, 2, 3, 28, 6, 1, 1, 1, 1, 0, 0), E = c( 1.4, 8.7, 3.0, 2.5, 4.3, 2.4, 8.1, 2.3, 2.0, 6.6, 4.4, 1.8, 1.1, 3.3, 7.8, 4.6, 1.1, 4.2, 5.5, 4.4, 10.5,22.7, 8.8, 5.6,15.5,12.5, 6.0, 9.0,14.4,10.2, 4.8, 2.9, 7.0, 8.5,12.3,10.1,12.7, 9.4, 7.2, 5.3, 18.8,15.8, 4.3,14.6,50.7, 8.2, 5.6, 9.3,88.7,19.6, 3.4, 3.6, 5.7, 7.0, 4.2, 1.8), X = c(16,16,10,24,10,24,10, 7, 7,16, 7,16,10,24, 7,16,10, 7, 7,10, 7,16,10, 7, 1, 1, 7, 7,10,10, 7,24,10, 7, 7, 0,10, 1,16, 0, 1,16,16, 0, 1, 7, 1, 1, 0, 1, 1, 0, 1, 1,16,10), num = c(3, 2, 1, 3, 3, 0, 5, 0, 5, 4, 0, 2, 3, 3, 2, 6, 6, 6, 5, 3, 3, 2, 4, 8, 3, 3, 4, 4, 11, 6, 7, 3, 4, 9, 4, 2, 4, 6, 3, 4, 5, 5, 4, 5, 4, 6, 6, 4, 9, 2,

4, 4, 4, 5, 6, 5), adj = c( 19, 9, 5, 10, 7, 12, 28, 20, 18, 19, 12, 1, 17, 16, 13, 10, 2, 29, 23, 19, 17, 1, 22, 16, 7, 2, 5, 3, 19, 17, 7, 35, 32, 31, 29, 25, 29, 22, 21, 17, 10, 7, 29, 19, 16, 13, 9, 7, 56, 55, 33, 28, 20, 4, 17, 13, 9, 5, 1, 56, 18, 4, 50, 29, 16, 16, 10, 39, 34, 29, 9, 56, 55, 48, 47, 44, 31, 30, 27, 29, 26, 15, 43, 29, 25, 56, 32, 31, 24,

45, 33, 18, 4, 50, 43, 34, 26, 25, 23, 21, 17, 16, 15, 9, 55, 45, 44, 42, 38, 24, 47, 46, 35, 32, 27, 24, 14, 31, 27, 14, 55, 45, 28, 18, 54, 52, 51, 43, 42, 40, 39, 29, 23, 46, 37, 31, 14, 41, 37, 46, 41, 36, 35, 54, 51, 49, 44, 42, 30, 40, 34, 23, 52, 49, 39, 34, 53, 49, 46, 37, 36, 51, 43, 38, 34, 30, 42, 34, 29, 26, 49, 48, 38, 30, 24, 55, 33, 30, 28, 53, 47, 41, 37, 35, 31, 53, 49, 48, 46, 31, 24, 49, 47, 44, 24, 54, 53, 52, 48, 47, 44, 41, 40, 38, 29, 21, 54, 42, 38, 34, 54, 49, 40, 34, 49, 47, 46, 41,

52, 51, 49, 38, 34, 56, 45, 33, 30, 24, 18, 55, 27, 24, 20, 18), sumNumNeigh = 234)

Results
The proportion of individuals working in agriculture appears to be associated to the incidence of cancer

alpha1 sample: 10000

4.0 3.0 2.0 1.0 0.0

-0.25

0.0

0.25

0.5

0.75

There appears to be spatial correlation among counties:

box plot: b

2.0
[1]

[3]
[2]

[5]

[13] [12]
[7]

[9] [10]

[15] [17] [19]

1.0

[4]

[16]
[14]

[18]

[20] [21]

[25] [26]

[23] [22] [24]

[28] [27] [29]

[36]

0.0

[32] [33] [35] [56] [31] [40] [37] [30] [38] [39] [43] [34] [51] [52] [41] [50] [44] [46] [47] [48] [55] [53] [54] [45] [42] [49]

-1.0

-2.0

(samples)means for RR

(26) < (15)

1.0 1.0 2.0 3.0 4.0 2.0 3.0 4.0

(9) (4) (2) >=

200.0km

values for O

(38) < 10.0 (14) 10.0 20.0 20.0 30.0

(2)

(2) >= 30.0

200.0km

Example: Lung cancer in London

Data were obtained from an annual report by the London Health Authority in which the association between health and poverty was investigated. Population under study are men 65 years of age and older. Available were: Observed and expected lung cancer counts in 44 census ward districts Ward-level index of socio-economic deprivation. A model with two random effects was fitted to the data. One random effect introduces region-wide heterogeneity (between ward variability) and the other one introduces regional clustering. The priors for the two components of random variability, are sometimes known as convolution priors. Interest is in: Determining association between health outcomes and poverty. Smoothing and mapping relative risks. Assessing the degree of spatial clustering in these data.

Model model { for (i in 1 : N) { # Likelihood O[i] ~ dpois(mu[i]) log(mu[i]) <- log(E[i]) + alpha + beta * depriv[i] + b[i] + h[i] RR[i] <- exp(alpha + beta * depriv[i] + b[i] + h[i]) relative risk (for maps) # Exchangeable prior on unstructured random effects h[i] ~ dnorm(0, tau.h) } # CAR prior distribution for spatial random effects: b[1:N] ~ car.normal(adj[], weights[], num[], tau.b) for(k in 1:sumNumNeigh) { weights[k] <- 1 } # Other priors: alpha ~ dflat() beta ~ dnorm(0.0, 1.0E-5) tau.b ~ dgamma(0.5, 0.0005)

# Area-specific

sigma.b <- sqrt(1 / tau.b) tau.h ~ dgamma(0.5, 0.0005) sigma.h <- sqrt(1 / tau.h) propstd <- sigma.b / (sigma.b + sigma.h) } Data click on one of the arrows to open data Inits click on one of the arrows to open initial values

Note that the priors on the precisions for the exchangeable and the spatial random effects are Gamma(0.5, 0.0005). That means that a priori, the expected value of the standard deviations is approximately 0.03 with a relatively large prior standard deviation. This is not a fair prior as discussed in class. The average number of neighbors is 4.8 and this is not taken into account in the choice of priors.

Results
Parameter Mean Std 2.5th perc. -0.408 0.0133 0.052 97.5th perc. -0.024 0.0838 0.874

Alpha Beta Relative size of spatial std

-0.208 0.0474 0.358

0.1 0.0179 0.243

sigma.b sample: 1000

20.0 15.0 10.0 5.0 0.0
10.0 7.5 5.0 2.5 0.0

sigma.h sample: 1000

-0.2

0.0

0.2

0.4

0.6

-0.2

0.0

0.2

0.4

0.6

Posterior distribution of RR for first 15 wards

node RR[1] RR[2] RR[3] RR[4] RR[5] RR[6] RR[7] RR[8] RR[9] RR[10] RR[11] RR[12] RR[13] RR[14] RR[15] mean 0.828 1.18 0.8641 0.8024 0.7116 1.05 1.122 0.821 1.112 1.546 0.7697 0.865 1.237 0.8359 0.7876 sd 0.1539 0.2061 0.1695 0.1522 0.1519 0.2171 0.1955 0.1581 0.2167 0.2823 0.1425 0.16 0.3539 0.1807 0.1563 2.5% 0.51 0.7593 0.5621 0.5239 0.379 0.7149 0.7556 0.4911 0.7951 1.072 0.4859 0.6027 0.8743 0.5279 0.489 median 97.5% 0.8286 1.188 0.8508 0.7873 0.7206 1.015 1.116 0.8284 1.07 1.505 0.7788 0.8464 1.117 0.8084 0.7869 1.148 1.6 1.247 1.154 0.9914 1.621 1.589 1.132 1.702 2.146 1.066 1.26 2.172 1.3 1.141

values for O

(17) < (19)

5.0 5.0 10.0 15.0 20.0

N
(3) (3) 10.0 15.0 -

(2) >= 20.0

2.5km

values for b

(2) < -0.1 (2) -0.1 - -0.05

N
(18) -0.05 - 1.38778E-17 (19) 1.38778E-17 (2) 0.05 0.1 0.1 0.05

(1) >=

2.5km

values for RR

(8) < (20)

0.8 0.8 1.0 1.2 1.0 1.2

N
(11) (5) >=

2.5km