Stat-340 Assignment 4 2014 Spring Term

Part 1 - Breakfast cereals - Easy

Visit http://lib.stat.cmu.edu/datasets/1993.expo/ to read about the break-
fast cereal dataset. Some times the above web site is busy and doesnt respond.
Do a Google search using the search terms 1993 data expo cereal and several
alternate sites come up such as http://ftp.uni-bayreuth.de/math/statlib/
datasets/1993.expo/readme and http://ftp.uni-bayreuth.de/math/statlib/
datasets/1993.expo/cereal.
In this part of the assignment, you will learn how to:
do simple bootstrapping to estimate the SE and condence interval of a
statistic
use a simple macro variable to control number of iterations
interpret the bootstrap sampling distribution
We will start again with the basic cereal data.
1. Input the data into SAS as in Assignment 1.
2. Deal with the coding for missing values as in Assignment 1.
3. Use Proc Univariate to estimate the mean, standard deviation, and the
standard deviation using the Gini robust procedure (see below) of the
number of calories per serving over all cereals.
Has SAS compute the standard error for the sample mean and 95% con-
dence interval for the population mean. Hint: refer to previous assign-
ments on how to do this. What is the dierence between the standard
1
deviation and the standard error of the sample mean? Interpret the 95%
condence interval for the population mean.
While the formula for the standard error of the sample mean when data is
collected using an SRS is well known, the formula for the standard error of
the sample standard deviation and 95% condence interval for the popula-
tion standard deviation are less well known but could be computed (http:
//web.eecs.umich.edu/~fessler/papers/files/tr/stderr.pdf). Yes,
standard errors of sample standard deviations do exist think carefully
about what this represents! Proc Univariate computes a 95% condence
interval for the population standard deviation, but does not give you a
standard error for the sample standard deviation. What is the dier-
ence between the sample standard deviation and the standard error of the
sample standard deviation (if you knew it). Interpret the 95% condence
interval for the population standard deviation.
Proc Univariate also gives you estimates of other statistics, but does NOT
provide estimates of the standard error of these other statistics nor any
condence intervals for the corresponding population parameter.
For example, the usual estimate of the standard deviation is EXTREMELY
sensitive to outliers. Various robust methods of estimating the standard
deviation have been proposed, among them the Gini estimate of the stan-
dard deviation see http://support.sas.com/documentation/cdl/en/
procstat/63104/HTML/default/viewer.htm#procstat_univariate_sect031.
htm for details on its computation.
You can get the Gini estimate of the standard deviation by specifying the
robustscale option on the Proc Univariate procedure invocation.
4. There are many other statistics that you could compute where it is unclear
on how to compute a standard error or a condence interval. Bootstrap-
ping is one method that can be used to estimate standard errors and
condence intervals in a wide range of situations.
CAUTION: This assignment illustrates how to do a simple bootstrap
in the case of a SRS. If the sample or experimental design is more complex,
the the bootstrap method must be modied to account for the sampling
design. You will learn how to do this in more advanced courses in statistics.
For a VERY brief overview of the bootstrapping procedure, read http://
en.wikipedia.org/wiki/Bootstrapping_(statistics). Statistics Canada
has a nice paper on how to bootstrap in complex surveys at http://www5.
statcan.gc.ca/bsolc/olc-cel/olc-cel?lang=eng&catno=11-522-X200600110416.
As outlined in class, the idea behind a bootstrap when data are collected
using a SRS design, is to resample, with replacement, from the sample,
compute the statistic on this resampled set, repeat these two steps many
times, and look at the distribution of the statistic over the bootstrap sam-
ples. The distribution of the statistic over the bootstrap samples should
mimic the distribution of the statistic over repeated samples from the
c 2013 Carl James Schwarz 2
population (why?) and hence the standard deviation of the statistics over
the bootstrap samples should be an estimate of the standard error of
the statistic (why?). This last point is THE CRUCIAL aspect of boot-
strapping and, inter alia, is also the FUNDAMENTAL CONCEPT OF
STANDARD ERRORS in general.
5. We will start by generating the bootstrapped samples using Proc Survey-
Select. Your code will look something like:
%let nboot=5; /* number of bootstrap reps */
proc surveyselect data=cereal out=bootsample
method=urs outhits rep=&nboot samprate=1 seed=2342332;
run;
All computer generated random numbers rely on a complex mathemati-
cal calculation that takes the current value of a random number and then
generates the next value in sequence. In fact, the computer generated ran-
dom numbers are actually NOT random at all because the same sequence
of values will always be generated. Refer to http://en.wikipedia.org/
wiki/Random_number_generation for a brief explanation of how comput-
ers generate pseudo-random numbers. This sequence of random numbers
must be initialized with a number that is traditionally called the seed. If
you rerun the program with the same seed value, you will get the same se-
quence of random numbers which makes it easier to debug your program.
But then for your nal run, you should replace the seed by a suitable value
(check SAS documentation.)
What does method=urs and outhits do? Why do we want samprate=1?
What does the %let nboot=5; do? Why do we debug our program with a
small value for the number of bootstrap samples? Notice how we refer to
the macro variable in subsequent statements through the use of &nboot.
In general, this is good programming practice to create macro variables
for things like the number of simulations rather than hard coding them in
your code. This way, all you have to do is change one statement in your
SAS code and changes are made throughout your program.
6. Look at the output from Proc SurveySelect. How many observations were
read in and how many observations were created? Does this make sense?
Print out the rst 20 records. Notice what happens if the same cereal is
selected multiple times.
Also notice that the Replicate variable has been added to the dataset to
group the records belonging to the same bootstrap sample together.
7. Tabulate the number of observations in each replicate using Proc Tabulate.
Hint: your table statement will look something like
c 2013 Carl James Schwarz 3
proc tabulate ****;
class ****;
table replicate, n*f=5.0;
Does the output make sense?
8. Now we need to compute our statistics for each bootstrap replicate and
output the results to a dataset. By now, you should be able to guess the
general structure of the code:
proc univariate data=bootsamp noprint;
by *****;
var *****;
output out=bootstat
mean=mean_cereal
stddev=std_cereal
std_gini=gini_std_cereal;
run;
What is the purpose of the NOprint statement? The usefulness of this
should become clearer when we increase the number of bootstrap samples
to 1000.
9. Print out the rst 10 records of the bootstat dataset. Do you understand
what this dataset represents?
10. We now nd the standard deviation, 2.5
th
and 97.5
th
percentiles of the
SAMPLE MEANS from the replicated bootstrap samples. Why? Your
code will look something like:
proc univariate data=bootstat noprint;
var *****;
output out=SE_boot_mean stddev=SE_boot_mean pctlpts=2.5 97.5 pctlpre=cl_mean;
run;
What does the pctlpts= option do? What does the pctlpre do?
The above method of nding the 95% condence interval for the POPU-
LATION mean based on the bootstrap sampling distribution is a simple
(but naive) way to do this. There are better methods that you will learn
about in more advanced classes in statistics.
11. Print out the SE_boot_mean dataset. Compare the standard error of the
sample mean from the bootstrap estimates and the condence limit for
the population mean from the bootstrap estimates to the standard error
of the sample mean and 95% condence interval for the population mean
computed from the values computed by Proc Univariate at the start of
c 2013 Carl James Schwarz 4
this assignment. Of course, with only 5 bootstrap samples, the se and
95% condence interval from bootstrapping are not very good.
Increase the number of replicates to 1000 and see how these bootstrap
values now compare to the original estimates from Proc Univariate which
are based on formulae.
12. We also want to display our bootstrap sampling distribution of the sam-
ple mean. Use Proc SGplot to display a histogram and a kernel density
smoother of the individual bootstrap values of the sample mean. Hint:
you did something like this in earlier assignments.
What is the dierence between a sampling distribution (that you just
created above) and the histogram of the calories per serving? THIS
IS A CRUCIAL CONCEPT TO UNDERSTAND IF
YOU WANT TO BE A STATISTICIAN!
What do you notice about the shape of the sampling distribution above?
Why does it have this shape? Hint: what does the Central Limit Theorem
say about sampling distributions? What is the asymptotic distribution of
Maximum Likelihood Estimates as the sample size increases?
13. We would like to annotate our histogram with the condence limits from
the bootstrap sample. We could take the values and manually add ref-
erence lines though the use of the REFLINE statement in Proc Sgplot
but then we would have to manually change this every time we reran the
program. We would like to annotate the graph automatically.
Full details are available at http://support.sas.com/resources/papers/
proceedings11/277-2011.pdf. You will get a crash course here. Note,
the process of annotating plots is NOT part of the course and you wont
be responsible for it on an exam, but I want to show you the power of
SAS.
Place the following code segment BEFORE your SGplot.
data sgannods;
set SE_boot_mean;
function= line;
y1space=datapercent; x1space=datavalue;
y2space=datapercent; x2space=datavalue;
x1=cl_mean2_5; y1=0; x2=cl_mean2_5; y2=50; output;
x1=cl_mean97_5; y1=0; x2=cl_mean97_5; y2=50; output;
run;
proc print data=sgannods;
title2 annotation instructions;
run;
This creates instructions (annotation instructions) for Proc SGplot to add
things to the graph. Here I want vertical lines stretching from the top
c 2013 Carl James Schwarz 5
to 1/2 up the graph showing the 2.5
th
and 97.5
th
percentile (which cor-
respond to the 95% condence interval - why?). The function statement
says that I want to draw a line; the xxSPACE statements give the type
of co-ordinate system to use to draw the line, i.e. either the actual data
values for the X axis or percentage of the data area in the Y (vertical)
direction. Finally I give the (x
1
, y
1
) and (x
2
, y
2
) co-ordinates for the start
and end of the line.
To apply these annotations to the plot, modify the rst statement to read
proc sgplot .... sganno=sgannods;
where sgannods is the name of the annotation dataset.
14. Repeat this for the sample standard deviation and the robust (Gini) stan-
dard deviation. Hint: You dont have to replicate the entire program, just
repeat the code after you compute the SE_boot_mean dataset - why?
If you were doing this in a real world situation, you would write a macro
to do this replicated code. While this is not part of the course, have a
look at the nal solutions (next week) to see how this could be done.
15. Create a table (by hand, but good for you if you can do this in SAS
through various merges etc.) showing
The three statistics (mean, standard deviation, Gini standard devia-
tion) computed from the original dataset along with the reported SE
(only available for the mean), and reported 95% condence interval
(only available for the mean and simple standard deviation).
The bootstrap estimates of the standard error and 95% condence
limits for the three statistics.
This table will form part of your report (below).
Hand in the following using the electronic assignment submission system:
Your SAS code that did the above analysis.
A PDF le containing all of your SAS output.
A one page (maximum) double spaced PDF le containing a short write
up on this analysis explaining the results of this analysis suitable for a
manager who has had one course in statistics. You should include the
following:
A (very) brief description of the dataset.
c 2013 Carl James Schwarz 6
Your table of results. Compare the SE of the mean computed ana-
lytically and via bootstrapping. Compare the 95% CI for the mean
and standard deviation computed analytically and via bootstrapping.
Discuss the shape of the sampling distribution.
If you have space, your graphs of the sampling distributions.
You will likely nd it easiest to do the write up in a word processor and
then print the result to a PDF le for submission. Pay careful attention
to things like number of decimal places reported and dont just dump
computer output into the report without thinking about what you want.
c 2013 Carl James Schwarz 7
Part 2 - Road Accidents with Injury - Intermediate
We return back to the road accident database. The database is a (near) com-
plete census of accidents in Great Britain in 2010. However, census information
is expensive to collect and in many cases a sampling approach is preferable.
Sampling is, of course, the raison detre of Statistics!
A common exercise for a statistician is to predict the amount of sampling
required. What this usually entails, as a rst step, is to provide a graph of
the precision of the estimate as a function of sample size. Then one can decide
the tradeo in spending money to reduce the standard error, i.e. how much is
information worth?
In some cases, this is relatively easy to do because the standard error of
an estimate exists in closed form. What happens if there is no formula for the
standard error? Again this can often be done using bootstrapping.
In this part of the assignment, you will learn how to:
how to select samples from a population
compute a synthetic index variable from each sample
nd the se of this synthetic index variable from replicate samples
plot the se of the synthetic index as a function of sample size
see that that your plot above is consistent with generally accepted princi-
ples of how the se of a statistic should decline with sample size.
Again consider the accident database.
1. Download the accident *.csv le into your directory.
From the accident dataset, you will need the AccidentID, the accident
severity, the number of vehicles, and the number of casualties variables.
Create a variable, the AccidentIndex, dened as the product of the number
of vehicles and the number of casualties divided by the accident severity.
Think about what a large value of the AccidentIndex implies vs. lower
values of the AccidentIndex.
c 2013 Carl James Schwarz 8
Discard the rest of the variables. Dont forget that you will have to process
the imported data to deal with missing values in some of the codes.
2. Print out the rst 10 records of the accident dataset to ensure that youve
read the data properly.
3. Select 10 samples of size 100 from the accident database. This will serve as
the bootstrap sample notice that in this exercise, we are NOT selecting
with replacement. Dont forget to use a macro variable to control the
number of replicate samples selected we will debug our program with 10
replicate samples at each sample size, but eventually will increase this to
1000 replicate samples at each sample size.
4. Compute the mean and 90
th
percentile of the accident index for EACH
replicate sample above. You will nd it convenient to also count the
number of records in each replicate when nding the statistics as this will
identify the sample size selected from the accident list.
Your code will look something like:
proc univariate ***** NOPRINT;
by *****;
var ***** ;
output out=reps100 n=sampsize mean=***** pctlpts=90 pctlpre=p;
Print out the rst few records of the reps100 dataset. Do you understand
what these represent?
5. Now that you have the estimates from each replicated sample, we nd
the standard deviation of the sample means and the standard deviation
of the 90th percentile across the replicate samples this represents the
approximate standard error for the mean and the 90th percentile (why?)
based on samples of size 100. Use the ID statement to keep the sample
size with the se. You code will look something like:
proc means data=reps100 noprint;
var **** ****;
id sampsize;
output out=se100 std=***** *****;
run;
Note that Ive used Proc Means in this second summarization, but you
could have also used Proc Univariate. There is often multiple ways to do
the same thing in SAS and the choice is personal preference and ease of
use in extracting information.
6. Repeat the above for sample sizes 200, 400, 1000, 2000 and 4000 by copy-
ing the code blocks above and making the appropriate substitutions. In
c 2013 Carl James Schwarz 9
more advanced classes of SAS you see how to do all of this in one big
macro rather than repeating code 6 times. This will eventually give you
6 estimates of the standard error for the two statistics. Each of the -
nal datasets should have a DIFFERENT name for example, the dataset
containing the empirical estimates of the standard errors for sample sizes
of 100 was called se100 above. When you repeated the code block, the
nal datasets should be named se200, se400 etc.
7. Combine the 6 estimates of the standard error into a single dataset using
the SET statement. Your dataset at this point should have the structure:
Sampe size se_p90 se_mean
100 xxx xxx
200 xxx xxx
400 xxx xxx
....
8. In many cases, standard errors decline as a function of

n. We want to
see if the se of mean accident index and the 90th percentile of the accident
index also decline as a function of

n.
If the se declines as a function of

n, this means that
SE = C/

n = Cn
0.5
for some constant C. Take logarithms of both sides to give:
log SE = log C 0.5 log n
(why?). This now looks like a linear regression of log SE vs. log n with
the intercept being log C and the slope of 0.5 (why?).
Create derived variables for the log(sample size), log(se_p90), and log(se_mean).
9. Use Proc Reg to nd the relationship between the log(se) of each statistic
and log(n). You can do both ts in the same procedure your code will
look something like:
p90: model log_se_p90 = log_n;
mean: model log_se_mean = log_n;
Have a look at the residual plots for both models do you notice anything
odd about the plots for the model for the 90th percentile?
Send the parameter estimates to a table for inclusion in your report using
the ODS Output facility.
Use Proc Transpose to create a table that looks like:
c 2013 Carl James Schwarz 10
Parameter Intercept Slope
P90 xxx xxx
Mean xxx xxx
Look at the estimated slope. Is the decline in standard error with sample
size consistent with the

n rule? How did you tell?
10. Plot the log(se) values against log(sample size) for both estimates on the
same graph using Proc SGplot. You should have 2 scatter statements to
plot the actual values, two series, and two reg statements for the separate
regression lines within SGplot. You can label each regression line using
the curvelabel= option on the reg statement.
Have a look at the series plots for both models do you notice anything
odd about the plots for the model for the 90th percentile? Why do you
think this has happened?
It is tedious to add the regression equation to the plot (but you could do
this using the annotate feature of SAS as shown in the previous question).
It is not necessary to annotate the plot with the equations.
11. Finally, after you have debugged your program, increase the number of
replicate samples at each sample size to 100 from 10. If you set up the
macro variable correctly, this will require one simple change.
Hand in the following using the online submission system:
Your SAS code.
A PDF le containing the the output from your SAS program.
A one page (maximum) double spaced PDF le containing a short write
up on this analysis suitable for a manager of trac operations who has
had one course in statistics. You should include:
A (very) brief description of the dataset.
A graph showing the decline in log(se) as a function of log(n) with an
accompanying table of the t and an explanation of the implications
of the slope when investigating the improvement in precision as a
function of sample size.
c 2013 Carl James Schwarz 11
Part 03: Review and preparing for term test
In this part of this assignment, you will work on a few short exercises designed
to review some of the material from the rst three assignments and introduce
some new things about the Data step.
Put all of the code from all of the sub-parts in one single SAS le. There is NO
writeup for this part of the assignment.
1. Problems with input data
Outdoor temperature is measured in degrees Celcius (http://en.wikipedia.
org/wiki/Celsius) in Canada and degrees Fahrenheit (http://en.wikipedia.
org/wiki/Fahrenheit) in the US. Look at http://www.stat.sfu.ca/
~cschwarz/Stat-340/Assignments/Assign04/assign04-part03-ds01.
txt which has the temperatures in degrees Celcius or degrees Fahrenheit
in a few cities in early January.
Read in the data and convert all temperatures to degrees Celcius. Note
that
C = (F 32)
5
9
Print out the nal dataset that contains the city and temperature (1 dec-
imal place) but not the observation number. Make sure that the label
for the temperature indicates it is in degrees Celcius.
2. Column input
So far you have used the list input style of SAS. In this style of input,
variables are separated by at least one delimiter (typically a blank) and
there is no requirement that data values be aligned in the input le.
In some cases (particularly in dealing with data that was collected many
years ago), space on the input record was at a premium and data was often
crunched together without spaces between values. For example, an old
style of input medium was the punch card (http://en.wikipedia.org/
wiki/Punched_card) in which you had at most 80 columns of data.
Look at http://www.stat.sfu.ca/~cschwarz/Stat-340/Assignments/
Assign04/assign04-part03-ds02.txt. The rst two records are the
variable names and a character counter so you can see where the various
columns in the data are. The data variables are:
Make of car in columns 1-5.
Model of car in columns 6-12.
Miles per gallon (mpg), Imperial measurement of fuel economy, in
columns 13-14.
c 2013 Carl James Schwarz 12
Weight of the car in columns 15-18.
Price of the car in columns 19-22.
In column input, you specify the columns that contain the variable. For
example, to read in the make and model of the car, your code would look
something like:
data *****;
infile *****;
length make ***** model ****;
input make $ 1-5 model $ 6-12;
Write SAS code to read in all of the variables from the car data (using
the URL method) and print out the nal dataset.
3. Split-Apply-Combine (SAC) paradigm
The Split-Apply-Combine (SAC) paradigm is a common task, implemented
in SAS using the By statement, various procedures, and the ODS OUT-
PUT or OUTPUT OUT= commands within the procedure.
For example, suppose you wanted to compare the average amount of sugar
by shelf in the cereal data. Your code would look something like:
data cereal; /* define the grouping variable */
/* read in cereal data */
/* make sure shelf and sugar are defined */
run;
proc sort data=cereal by shelf; run; /* sort by grouping variable */
proc univariate data=cereal .... cibasic ;
by shelf; /* separate analysis for each shelf */
var sugars;
ods output .....;
run;
proc sgplot data=....; /* plot the estimate and 95% ci */
....
run;
Youve also analyzed the proportion of survival by passenger class in the
Titanic, the number of accidents per day across the months, etc.
Go back to the accident dataset, and make a side-by-side condence in-
terval plot of the proportion of fatal accidents by MONTH. You will have
to create a month variable from the date, create a fatality indicator, use
Proc Freq or Proc Genmod to estimate the proportion of fatalities in each
month, and nally Proc SGplot to plot the nal estimates and condence
c 2013 Carl James Schwarz 13
intervals.
Are you surprised by the results? There is NO write up for this part.
c 2013 Carl James Schwarz 14
Commments from the marker.
Here are the comments from the marker from previous years assignments.
Part 01 - Cereal
Most papers began with a quick blurb about the dataset itself, followed by a
detailed discussion of the mean, sd, gini, etc without even a mention of the
fact that it was calories/serving that was the variable of interest. Most people
who lost marks on this question lost it because they didnt mention the variable
they were analyzing.
Other mistakes included not using enough replicates in bootstrapping. A
number of people used only 5 replicates, and deemed it sucient. Some even
presented histograms from those bootstrap runs, and claimed that things looked
"normal".
CJS - use 5-10 replicates to TEST your program, but dont forget to increase
the number of replicate bootstrap samples to around 1000.
Part 02 - Accidents
Once again, plenty of papers didnt mention the accident index, or even a
method by which the graphs were generated. After a quick blurb about this
data having 150000 data points and hailing from the UK government, they
delved right into analyzing the graph.
I had very few truly satisfactory slope interpretations. A lot of people seem
to default into the formulaic: If X increases by one unit, then Y increases by
blah units, which in this case is very confusing. I didnt penalize this in most
cases, but I still dont like it. In the instances where the interpretation was sim-
ply the equivalent of "precision increases with sample size", I took marks o.
That statement in itself is not news this experiment was performed specically
to assess the nature of the relationship.
CJS - Yes, it is true that the formal denition of a slope is the change in Y
c 2013 Carl James Schwarz 15
per unit change in X. But statisticians are not robots who just repeat textbook
denitions! You should always put your work in terms of the project.
c 2013 Carl James Schwarz 16