PSI

Tutorial: Chi-Squared Analysis

The chi-squared analysis is a method of determining the relationship between a data set and a theoretical
model. This analysis can tell us three things:

1. What is the best-fit line?
2. How good is the fit?
3. Does the best-fit line agree with our theoretical model?

We will examine each of these in this tutorial. As you work through the tutorial, do not hesitate to ask
questions and collaborate with your classmates. Solutions will be posted on the course website.

Part 1: Maximum Likelihood Estimation: Finding the best-fit line

Suppose you have some set of data, like in the sketch shown
at the right. We want to know, what is the probability of
getting that set of data given some particular model? We will
use a line of reasoning known as Maximum Likelihood
ydata
Estimation (MLE).

Because of the central limit theorem, we can assume that the
ymodel
measurements are Gaussian in distribution. What matters is
the distance between the data point and the model we are
testing compared to the standard deviation at every
measured point.

1. What is the probability that a single point y will fall on
the line given by the model? (You can ignore the
normalization factor for now.)



2. Sketch this distribution and label where the ymodel is on the plot.












3. Will the probability distribution look the same for each data point? What will be the differences?

4. What is the probability of getting the whole data set given this particular model (i.e., fit line)?











Here we define chi-squared (2) to simplify things a bit:

!

!
!

!! !!"#$%!

!!!

The probability found above in #4 is known as the likelihood. Our most probable model is the one with
the greatest likelihood of producing the data set we collected.

Likelihood ~ !

!!!

5. When is the likelihood the greatest? What is the relative value of chi-squared at this value?






Part 2: How good is the fit? Testing our family of models

We have shown that the best-fit line is the one that generates the lowest-value for chi-squared. But how
do we know if this fit is any good? Consider the plots shown below. On the right we have a best-fit line
found by minimizing chi-squared. However, the linear model (ymodel = Ax) does not agree very well with
the data. On the left, we plot the same data, but use a different family of models (ymodel = Ax2) to fit the
data. Here we will show how you can use the minimum value of chi-squared to test whether or not the
family of models represents the data.




ymodel = Ax2
ymodel = Ax

x
x2

6. In a reasonable fit to the data, we expect most of the data points to be within one sigma of the
best-fit line. What would be chi-squared if each data point was one sigma from the best-fit line?

Stop here and use the Matlab script <ChiSquared.m> to analyze your data.

Part 3: Determining uncertainty in the fit: Testing how
well our model agrees with the best-fit line

In Matlab, we made a plot of chi-squared vs. slope (A) for a
range of A values. Your chi-squared vs. slope plot probably
looks something like the figure at the right.

Our model predicts that the slope of the F/mr vs. 2 graph
should be 1. We used Matlab to find the minimum A value on
this plot, and it is not exactly 1.

The Likelihood that our model (ymodel = Axi. ) could generate
the set of data we collected is given by:

!

Likelihood = exp
!!!

!! !! !!
2!!!

If we plot Likelihood vs. slope, we will get something that
looks like a Gaussian (see the figure at right). This is the
distribution describing the probability that our data fits a
given family of models. The most likely model is the mean of
the distribution. Note that the standard deviation (A) of
this distribution is different from the standard deviation (i)
of the data in the sample.

The question now is whether or not our model fits the data. This is tedious to do analytically, so we will
answer the question using an analogy. Think back to our original Gaussian for one variable:

!
!! !
1 !!
1
G(x) = exp
= exp
= exp ! !
!
2!
2
!
2

Where we simplify the argument in the exponent to be in terms of z, where ! ! =

!!!
!

Physics 15a/16

Harvard University, Fall 2011

7. We know what a typical Gaussian looks like (see figure at
left). Instead of graphing G(x) vs. x, sketch z2 vs. x. Line up the x-
axis with the graph of the Gaussian.


8. Mark minimum value of z2 on the graph and call it zmin2


Figure 13: 68% of the area under the Gaussian curve falls within one of the mean (left); 95% falls within

two of the mean (right).

Of course, we dont have to stop there; 99.7% of the area
falls within 3 of the mean, and so forth. (You
could look up all these figures in a table of numerical integrals of the Gaussian distribution function.) But
point is that these figures establish confidence
for practical purposes, 68% and 95% will suce.9 The key
intervals. If a set of measurements obeys a Gaussian distribution
with mean 100 and standard deviation

1, then we can be 68% confident that any given measurement from that set lies between 99 and 101, and

95% confident that it lies between 98 and 102. In particular, we can predict that the next measurement
made in the same way will also fall into those ranges with the same probabilities.
This is exactly what we wanted: a concise but specific
quantitative description of the random error
associated with a measurement. It works so well that well use it to define what we mean by uncertainty
such a convenient quantity to work with, well
going forward. Since one standard deviation turns out to be
2
use that:
9. What values of z correspond to x = 1 and x = 2? Mark these points on your sketch and

on the Gaussian.
The uncertainty in a reported value is defined as the interval
in which we are 68% confident the true value lie.

definition, for a single measurement taken from a data set, the uncertainty is equal to
From this
the standard
deviation of the data set. Rather than specifying the endpoints of the interval, we often
express uncertainties as the central value plus or minus a certain amount. So in our example, rather

than specifying that the measurement is between 99 and 101 about 68% of the time, we can say that the
is 100 with an uncertainty of 1, or write it even more concisely as 100 1.
measurement
A bit of notation: if a measured quantity is represented by x, then the uncertainty in x is sometimes
written as x. is the Greek lowercase letter delta, and in this context it means uncertainty of. So if
Look
back
at length
your tograph
f chi-squared
s. scm
lope
n M
you
measure
some
be L = o
100
1 cm, then L =v100
and iL
= 1atlab.
cm. It should look similar to what we just drew,
but we are using 2 (which includes many data points) instead of z2 (which is for a single data point).
3.6
Standard Deviation and Standard Error

Even 10.
with What
this definition
ofm
uncertainty,
can still
into
confusion
it comes to repeated
is the
inimum wevalue
of run
2?
Wsome
hy is
it not when
zero?
measurements. Consider the example we saw back in Section 2.3, when we took the set of 50 measurements
a length (Figure 8) and expanded it to a set including 5000 data points (Figure 10). In our original data
of
Actually, there is a good reason we dont bother with 4 and beyond, and rarely with 3: Treating almost every
distribution
that comes up as a Gaussian is an approximation. For most distributions, its very good approximation near the

center of the distribution, and not so good way o on the tails. This agrees with common sensethe Gaussian is infinitely
in the sense that it never goes to zero, but of course there are physical measurements where you will obviously never get
wide
a negative value, for example.
11. What is the corresponding value of A? This is our most likely model (best A value).

18



12. Now lets think about the uncertainty in our best A. What values of 2 correspond to A 1 and A
2 ? Give your answer in terms of 2min . (Dont try to do this analytically. Look back at what we
did with z2 and see if you can figure out the answer.) Mark these points on the graphs on the first
page.







9

13. Does our model A value fall within the A1 range? A2 ?

14. How does our actual minimum value of 2 compare to the number of data points?

15. Generally, if each data point is (on average) within one sigma of the model, we say the model is
pretty good. If the data is more than two sigma away from the model, we will reject the family of
models. Can we reject our model based on this analysis?