Notes PDF

Probability Theory and
Statistics
With a view towards the natural sciences
Lecture notes
Niels Richard Hansen

Department of Mathematical Sciences
University of Copenhagen
November 2010
2
Preface
The present lecture notes have been developed over the last couple of years for a
course aimed primarily at the students taking a Master’s in bioinformatics at the
University of Copenhagen. There is an increasing demand for a general introductory
statistics course at the Master’s level at the university, and the course has also
become a compulsory course for the Master’s in eScience. Both educations emphasize
a computational and data oriented approach to science – in particular the natural
sciences.
The aim of the notes is to combine the mathematical and theoretical underpinning
of statistics and statistical data analysis with computational methodology and prac-
tical applications. Hopefully the notes pave the way for an understanding of the
foundation of data analysis with a focus on the probabilistic model and the method-
ology that we can develop from this point of view. In a single course there is no
hope that we can present all models and all relevant methods that the students will
need in the future, and for this reason we develop general ideas so that new models
and methods can be more easily approached by students after the course. We can,
on the other hand, not develop the theory without a number of good examples to
illustrate its use. Due to the history of the course most examples in the notes are
biological of nature but span a range of different areas from molecular biology and
biological sequence analysis over molecular evolution and genetics to toxicology and
various assay procedures.
Students who take the course are expected to become users of statistical methodology
in a subject matter field and potentially also developers of models and methodology
in such a field. It is therefore intentional that we focus on the fundamental principles
and develop these principles that by nature are mathematical. Advanced mathemat-
ics is, however, kept out of the main text. Instead a number of math boxes can
be found in the notes. Relevant, but mathematically more sophisticated, issues are
treated in these math boxes. The main text does not depend on results developed in
i
ii
the math boxes, but the interested and capable reader may find them illuminating.
The formal mathematical prerequisites for reading the notes is a standard calculus
course in addition to a few useful mathematical facts collected in an appendix. The
reader who is not so accustomed to the symbolic language of mathematics may,
however, find the material challenging to begin with.
To fully benefit from the notes it is also necessary to obtain and install the statisti-
cal computing environment R. It is evident that almost all applications of statistics
today require the use of computers for computations and very often also simula-
tions. The program R is a free, full fledge programming language and should be
regarded as such. Previous experience with programming is thus beneficial but not
necessary. R is a language developed for statistical data analysis and it comes with
a huge number of packages, which makes it a convenient framework for handling
most standard statistical analyses, for implementing novel statistical procedures, for
doing simulation studies, and last but not least it does a fairly good job at producing
high quality graphics.
We all have to crawl before we can walk – let alone run. We begin the notes with
the simplest models but develop a sustainable theory that can embrace the more
advanced ones too.
Last, but not least, I owe a special thank to Jessica Kasza for detailed comments on
an earlier version of the notes and for correcting a number of grammatical mistakes.
November 2010
Niels Richard Hansen
Contents
1 Introduction 1
1.1 Notion of probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Statistics and statistical models . . . . . . . . . . . . . . . . . . . . . 4
2 Probability Theory 13
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Sample spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Probability measures . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Probability measures on discrete sets . . . . . . . . . . . . . . . . . . 21
2.5 Descriptive methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.1 Mean and variance . . . . . . . . . . . . . . . . . . . . . . . . 28
2.6 Probability measures on the real line . . . . . . . . . . . . . . . . . . 32
2.7.1 Histograms and kernel density estimation . . . . . . . . . . . 43
2.7.2 Mean and variance . . . . . . . . . . . . . . . . . . . . . . . . 49
2.7.3 Quantiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.8 Conditional probabilities and independence . . . . . . . . . . . . . . 59
2.9 Random variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.9.1 Transformations of random variables . . . . . . . . . . . . . . 63
2.10 Joint distributions, conditional distributions and independence . . . 70
iii
iv Contents
2.10.1 Random variables and independence . . . . . . . . . . . . . . 70

2.10.2 Random variables and conditional distributions . . . . . . . . 75
2.10.3 Transformations of independent variables . . . . . . . . . . . 81
2.11 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
2.12 Local alignment - a case study . . . . . . . . . . . . . . . . . . . . . 91
2.13 Multivariate distributions . . . . . . . . . . . . . . . . . . . . . . . . 97
2.13.1 Conditional distributions and conditional densities . . . . . . 106
2.15 Transition probabilities . . . . . . . . . . . . . . . . . . . . . . . . . 111
3 Statistical models and inference 117

3.1 Statistical Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
3.2 Classical sampling distributions . . . . . . . . . . . . . . . . . . . . . 127
3.3 Statistical Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
3.3.1 Parametric Statistical Models . . . . . . . . . . . . . . . . . . 131
3.3.2 Estimators and Estimates . . . . . . . . . . . . . . . . . . . . 132
3.3.3 Maximum Likelihood Estimation . . . . . . . . . . . . . . . . 136
3.4 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
3.4.1 Two sample t-test . . . . . . . . . . . . . . . . . . . . . . . . 163
3.4.2 Likelihood ratio tests . . . . . . . . . . . . . . . . . . . . . . . 169
3.4.3 Multiple testing . . . . . . . . . . . . . . . . . . . . . . . . . . 172
3.5 Confidence intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
3.5.1 Parameters of interest . . . . . . . . . . . . . . . . . . . . . . 181
3.6 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
3.6.1 Ordinary linear regression . . . . . . . . . . . . . . . . . . . . 190
3.6.2 Non-linear regression . . . . . . . . . . . . . . . . . . . . . . . 204
3.7 Bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
3.7.1 The empirical measure and non-parametric bootstrapping . . 214
3.7.2 The percentile method . . . . . . . . . . . . . . . . . . . . . . 216
4 Mean and Variance 219

Contents v
4.1 Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

4.1.1 The empirical mean . . . . . . . . . . . . . . . . . . . . . . . 224
4.2 More on expectations . . . . . . . . . . . . . . . . . . . . . . . . . . 225
4.3 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
4.4 Multivariate Distributions . . . . . . . . . . . . . . . . . . . . . . . . 234
4.5 Properties of the Empirical Approximations . . . . . . . . . . . . . . 239
4.6 Monte Carlo Integration . . . . . . . . . . . . . . . . . . . . . . . . . 245
4.7 Asymptotic Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
4.7.1 MLE and Asymptotic Theory . . . . . . . . . . . . . . . . . . 256
4.8 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
A R 267
A.1 Obtaining and running R . . . . . . . . . . . . . . . . . . . . . . . . 267
A.2 Manuals, FAQs and online help . . . . . . . . . . . . . . . . . . . . . 268
A.3 The R language, functions and scripts . . . . . . . . . . . . . . . . . 269
A.3.1 Functions, expression evaluation, and objects . . . . . . . . . 269
A.3.2 Writing functions and scripts . . . . . . . . . . . . . . . . . . 270
A.4 Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
A.5 Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
A.5.1 Bioconductor . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
A.6 Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
A.7 Other resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
B Mathematics 277
B.1 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
B.2 Combinatorics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
B.3 Limits and infinite sums . . . . . . . . . . . . . . . . . . . . . . . . . 279
B.4 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
B.4.1 Gamma and beta integrals . . . . . . . . . . . . . . . . . . . 282
B.4.2 Multiple integrals . . . . . . . . . . . . . . . . . . . . . . . . . 283
vi Contents
1
Introduction
1.1 Notion of probabilities
Flipping coins and throwing dice are two commonly occurring examples in an in-
troductory course on probability theory and statistics. They represent archetypical
experiments where the outcome is uncertain – no matter how many times we roll
the dice we are unable to predict the outcome of the next roll. We use probabilities
to describe the uncertainty; a fair, classical dice has probability 1/6 for each side to
turn up. Elementary probability computations can to some extent be handled based
on intuition, common sense and high school mathematics. In the popular dice game
Yahtzee the probability of getting a Yahtzee (five of a kind) in a single throw is for
instance
6 1
5
= 4 = 0.0007716.
6 6
The argument for this and many similar computations is based on the pseudo theorem
that the probability for any event equals
number of favourable outcomes
.
number of possible outcomes
Getting a Yahtzee consists of the six favorable outcomes with all five dice facing the
same side upwards. We call the formula above a pseudo theorem because, as we will
show in Section 2.4, it is only the correct way of assigning probabilities to events
under a very special assumption about the probabilities describing our experiment.
The special assumption is that all outcomes are equally probable – something we
tend to believe if we don’t know any better, or can see no way that one outcome
should be more likely than others.
However, without some training most people will either get it wrong or have to give
up if they try computing the probability of anything except the most elementary
1
2 Introduction
events – even when the pseudo theorem applies. There exist numerous tricky prob-
ability questions where intuition somehow breaks down and wrong conclusions can
be drawn if one is not extremely careful. A good challenge could be to compute the
probability of getting a Yahtzee in three throws with the usual rules and provided
that we always hold as many equal dice as possible.
0.7
0.6
0.5
0.4
0.3
0 100 200 300 400 500
Figure 1.1: The relative frequency of times that the dice sequence ¥ ¨ § comes
out before the sequence ¦ ¥ ¨ as a function of the number of times the dice game
has been played.
The Yahtzee problem can in principle be solved by counting – simply write down all
combinations and count the number of favorable and possible combinations. Then
the pseudo theorem applies. It is a futile task but in principle a possibility.
In many cases it is, however, impossible to rely on counting – even in principle. As
an example we consider a simple dice game with two participants: First I choose
a sequence of three dice throws, ¥ ¨ §, say, and then you choose ¦ ¥ ¨, say. We
throw the dice until one of the two sequences comes out, and I win if ¥ ¨ § comes
out first and otherwise you win. If the outcome is
¨ª¦§©¥§¦¨©¥¨§
then I win. It is natural to ask with what probability you will win this game. In
addition, it is clearly a quite boring game, since we have to throw a lot of dice and
simply wait for one of the two sequences to occur. Another question could therefore
be to ask how boring the game is? Can we for instance compute the probability for
Notion of probabilities 3
having to throw the dice more than 100, or perhaps 500, times before any of the
two sequences shows up? The problem that we encounter here is first of all that the
pseudo theorem does not apply simply because there is an infinite number of favor-
able as well as possible outcomes. The event that you win consists of the outcomes
being all finite sequences of throws ending with ¦ ¥ ¨ without ¥ ¨ § occurring
somewhere as three subsequent throws. Moreover, these outcomes are certainly not
equally probable. By developing the theory of probabilities we obtain a framework
for solving problems like this and doing many other even more subtle computations.
And if we cannot compute the solution we might be able to obtain an answer to
our questions using computer simulations. Moreover, the notes introduce probabil-
ity theory as the foundation for doing statistics. The probability theory will provide
a framework, where it becomes possible to clearly formulate our statistical questions
and to clearly express the assumptions upon which the answers rest.
400
300
Frequency
200
100
0
0−10
10−20
20−30
30−40
40−50
50−60
60−70
70−80
80−90
90−100
100−110
110−120
120−130
130−140
140−150
150−160
160−170
170−180
180−190
190−200
200−210
210−220
220−230
230−240
240−250
250−260
260−270
270−280
280−290
290−300
300−310
310−320
320−330
330−340
340−350
350−360
360−370
370−380
380−390
390−400
400−410
410−420
420−430
430−440
440−450
450−460
460−470
470−480
480−490
490−500
Figure 1.2: Playing the dice game 5000 times, this graph shows how the games are
distributed according to the number of times, n, we had to throw the dice before
one of the sequences ¥ ¨ § or ¦ ¥ ¨ occurred.
Enough about dice games! After all, these notes are about probability theory and
statistics with applications to the natural sciences. Therefore we will try to take
examples and motivations from real biological, physical and chemical problems, but
it can also be rewording intellectually to focus on simple problems like those from
a dice game to really understand the fundamental issues. Moreover, some of the
questions that we encounter, especially in biological sequence analysis, are similar
in nature to those we asked above. If we don’t know any better – and in many cases
we don’t – we may regard a sequence of DNA as simply being random. If we don’t
4 Introduction
have a clue about whether the sequence encodes anything useful, it may just as well
be random evolutionary leftovers. Whether this is the case for (fractions of) the
intergenic regions, say, of eukaryote genomes is still a good question. But what do
we actually mean by random? A typical interpretation of a sequence being random
is to regard the DNA sequence as the outcome of throwing a four sided dice, with
sides A, C, G, and T, a tremendously large number of times.
One purpose of regarding DNA sequences as being random is to have a background
model. If we have developed a method for detecting novel protein coding genes, say,
in DNA sequences, a background model for the DNA that is not protein coding is
useful. Otherwise we cannot tell how likely it is that our method finds false protein
coding genes, i.e. that the method claims that a segment of DNA is protein coding
even though it is not. Thus we need to compute the probability that our method
claims that a random DNA sequence – with random having the meaning above – is
a protein coding gene. If this is unlikely, we believe that our method indeed finds
truly protein coding genes.
A simple gene finder can be constructed as follows: After the start codon, ATG, a
number of nucleotides occur before one of the stop codons, TAA, TAG, TGA is reached
for the first time. Our protein coding gene finder then claims that if more than 99
(33 codons) nucleotides occur before any of the stop codons is reached then we have
a gene. So what is the chance of getting more than 99 nucleotides before reaching a
stop coding for the first time? The similarity between determining how boring our
little dice game is should be clear. The sequence of nucleotides occurring between a
start and a stop codon is called an open reading frame, and what we are interested
in is thus how the lengths of open reading frames are distributed in random DNA
sequences.
The probability theory provides the tools for computing probabilities. If we know
the probability measure – the assignment of probabilities to events – the rest is in
principle just mathematical deduction. Sometimes this is what we need, but in many
cases we are actually interested in questions that go beyond what we can obtain by
pure deduction. Situations where we need the interplay between data – experimental
data or computational data – and the probability theory, are where statistics comes
into the picture. Hopefully the reader will realize upon reading these notes that the
four sided dice model of DNA is rarely useful quantitatively, but may provide some
qualitative insight into more practical problems we really want to solve.
1.2 Statistics and statistical models
Where probability theory is a deductive and mathematical science, statistics – and

in particular applied statistics – is an inductive science, where we want to obtain
knowledge and understanding about general relations and patterns from data. In
other words, if we have observed certain events as outcomes of an experiment how
Statistics and statistical models 5
do we find the mechanisms that generated those outcomes, and how do we under-
stand those mechanisms? What brings the topics of probability theory and statistics
together is the use of probability theory to model the generating mechanisms, and
the corresponding mathematical framework that comes along provides us with meth-
ods for doing inductive inference and for understanding and interpreting the results.
Probability measures have the ability to capture unpredictable or uncontrollable
variation (randomness) together with systematic variability, and are therefore ideal
objects to model almost any kind of experimental data – with “experiments” to be
understood in the widest possible sense. Theoretical statistics offers a range of sta-
tistical models, which for the present section can be thought of families of probability
measures, and methods for transferring data into probability measures. The transfer
of data into a probability measure is a cornerstone in statistics, which is often called
statistical inference. The primary aim of the notes is to give a solid understanding of
the concept of a statistical model based on probability theory and how the models
can be used to carry out the inductive step of doing statistical inference.
To get a good understanding of what a statistical model is, one needs to see some
examples. We provide here a number of examples that will be taken up and recon-
sidered throughout the notes. In this section we describe some of the background
and the experiments behind the data. In the rest of the notes these examples will
be elaborated on to give a fully fledged development of the corresponding statistical
models and inference methods as the proper theoretical tools are developed.
Example 1.2.1 (Neuronal interspike times). Neuron cells in the brain are very well
studied and it is known that neurons transmit electrochemical signals. Measurements
of a cells membrane potential show how the membrane potential can activate voltage-
gated ion channels in the cell membrane and trigger an electical signal known as a
spike.
At the most basic level it is of interest to understand the interspike times, that
is, the times between spikes, for a single neuron in a steady state situation. The
interspike times behave in an intrinsically stochastic manner meaning that if we
want to describe the typical interspike times we have to rely on a probabilistic
description.
A more ambitious goal is to relate interspike times to external events such as visual
stimuli and another goal is to relate the interspike times of several neurons. ⋄
Example 1.2.2 (Motif counting). Special motifs or patterns in the genomic DNA-
sequences play an important role in the development of a cell – especially in the way
the regulation of gene expression takes place. A motif can be a small word, in the
DNA-alphabet, or a collection of words as given by for instance a regular expres-
sion or by other means. One or more proteins involved in the expression regulation
mechanisms are then capable of binding to the DNA-segment(s) that corresponds
to the word or word collection (the binding site), and in doing so either enhance or
suppress the expression of a protein coding gene, say.
6 Introduction
The typical datasets we consider in relation to motifs are computational in nature.

Of course the genomes are biological, but we employ one or several computational
approaches that give us the location, say, of all the occurrences of the motif. It is
rarely the case that the computational procedure will find only biologically meaning-
ful binding sites, and we want a statistical model of the random occurrence of motifs
in DNA-sequences. In particular we are interested in the distribution of how many
computationally predicted binding sites we will find in a sequence of length n, how
the distribution depends upon n and perhaps also on the nucleotide composition
of the DNA-sequences. As for the open reading frames, if we are comfortable with
the four sided dice model of DNA, the problem is a probability problem. But again
we are not, and we would like instead to model directly the predicted binding site
occurrences.
A slight variation of counting the number of occurrences is to just record whether a
motif is present in a sequence or not, or whether it is present at a given location or
window in the sequence. ⋄
Example 1.2.3 (Forensic Statistics). In forensic science one of the interesting prob-
lems from the point of view of molecular biology and statistics is the ability to iden-
tify the person who committed a crime based on DNA-samples found at the crime
scene. One approach known as the short tandem repeat (STR) analysis is to con-
sider certain specific tandem repeat positions in the genome and count how many
times the pattern has been repeated. The technique is based on tandem repeats with
non-varying flanking regions to identify the repeat but with the number of pattern
repeats varying from person to person. These counts of repeat repetitions are useful
as “genetic fingerprints” because the repeats are not expected to have any function
and the mutation of repeat counts is therefore neutral and not under selective pres-
sure. Moreover, the mutations occur (and have occurred) frequently enough so that
there is a sufficient variation in a population for discriminative purposes. It would
be of limited use if half the population, say, have a repeat count of 10 with the other
half having a repeat count of 11.
Without going into too many technical details, the procedure for a DNA-sample
from a crime scene is quite simple. First one amplifies the tandem repeat(s) using
PCR with primers that match the flanking regions, and second, one extracts the
sequence for each of the repeats of interest and simply count the number of repeats.
Examples of STRs used include TH01, which has the pattern AATG and occurs in
intron 1 of the human tyrosine hydroxylase gene, and TPOX, which has the same
repeat pattern but is located in intron 10 of the human thyroid peroxidase gene.
One tandem repeat is not enough to uniquely characterize an individual, so several
tandem repeats are used. A major question remains. Once we have counted the
number of repeats for k, say, different repeat patterns we have a vector (n1 , . . . , nk )
of repeat counts. The Federal Bureau of Investigation (FBI) uses for instance a
standard set of 13 specific STR regions. If a suspect happens to have an identical
vector of repeat counts – the same genetic fingerprint – we need to ask ourselves
what the chance is that a “random” individual from the population has precisely this
genetic fingerprint. This raises a number of statistical questions. First, what kind of
random procedure is the most relevant – a suspect is hardly selected completely at
random – and second, what population is going to be the reference population? And
even if we can come up with a bulletproof solution to these questions, it is a huge
task and certainly not a practical solution to go out and count the occurrences of
the fingerprint (n1 , . . . , nk ) in the entire population. So we have to rely on smaller
samples from the population to estimate the probability. This will necessarily involve
model assumptions – assumptions on the probabilistic models that we will use.
One of the fundamental model assumptions is the classical Hardy-Weinberg equilib-
rium assumption, which is an assumption about independence of the repeat counts
at the two different chromosomes in an individual. ⋄
Example 1.2.4 (Sequence evolution). We will consider some of the most rudimen-
tary models for evolution of biological sequences. Sequences evolve according to a
complicated interaction of random mutations and selection, where the random mu-
tations can be single nucleotide substitutions, deletions or insertions, or higher order
events like inversions or crossovers. We will only consider the substitution process.
Thus we consider two sequences, DNA or proteins, that are evolutionary related via
a number of nucleotide or amino acid substitutions. We will regard each nucleotide
position (perhaps each codon) or each amino acid position as unrelated to each other,
meaning that the substitution processes at each position are independent. We are
interested in a model of these substitution processes, or alternatively a model of the
evolutionary related pairs of nucleotides or amino acids. We are especially interested
in how the evolutionary distance – measured for instance in calendar time – enters
into the models.
The models will from a biological point of view be very simplistic and far from
realistic as models of real sequence evolution processes. However, they form the
starting point for more serious models, if one, for instance, wants to enter the area
of phylogenetics, and they are well suited to illustrate and train the fundamental
concepts of a statistical models and the methods of statistical inference.
Obtaining data is also a little tricky, since we can rarely go out and read of evolution-
ary related sequences where we know the relation “letter by letter” – such relations
are on the contrary established computationally using alignment programs. How-
ever, in some special cases, one can actually observe real evolution as a number
of substitutions in the genome. This is for instance the case for rapidly evolving
RNA-viruses.
Such a dataset was obtained for the H strain of the hepatitis C virus (HCV) (Ogata
et al., Proc. Natl. Acad. Sci., 1991 (88), 3392-3396). A patient, called patient H, was
infected by HCV in 1977 and remained infected at least until 1990 – for a period
of 13 years. In 1990 a research group sequenced three segments of the HCV genome
obtained from plasma collected in 1977 as well as in 1990. The three segments, de-
noted segment A, B and C, were all directly alignable without the need to introduce
8 Introduction
Position 42 275 348 447 556 557 594 652 735 888 891 973 979 1008 1011 1020 1050 1059 1083 1149 1191 1195 1224 1266
H77 G C C A G C C C T C T G G C G C T T C T T T T A
H90 A T T G A T T T C T C A A T A T A C T C C A C G
Table 1.1: The segment position and nucleotides for 24 mutations on segment A of
the hepatitis C virus.
insertions or deletions. The lengths of the three segments are 2610 (A), 1284 (B)
and 1029 (C) respectively.
H90 H90 H90
A C G T A C G T A C G T
A 1 11 1 A 0 5 0 A 1 2 0
H77 C 4 1 20 H77 C 1 0 8 H77 C 1 2 5
G 13 3 1 G 1 1 1 G 4 0 0
T 3 19 1 T 2 6 0 T 1 3 1
segment A segment B segment C
Table 1.2: Tabulation of all mutations in the three segments A, B and C of the
hepatitis C virus genome from the 1977 H strain to the 1990 H strain.
In Table 1.2.4 we see the position for the first 24 mutations as read from the 5’-end
of segment A out of the total of 78 mutations on segment A. In Table 1.2.4 we have
tabulated all the mutations in the three segments. ⋄
Example 1.2.5 (Toxicology). “Alle Dinge sind Gift und nichts ist ohne Gift; allein
die Dosis macht, dass ein Ding kein Gift ist.” Theophrastus Phillipus Auroleus Bom-
bastus von Hohenheim (1493-1541). All compounds are toxic in the right dose – the
question is just what the dose is.
The dose-response experiment is a classic. In a controlled experiment, a collection of
organisms is subdivided into different groups and each group is given a specific dose
of a compound under investigation. We focus here on whether the concentration is
lethal, and thus subsequently we count the number of dead organisms in each group.
How can we relate the dose to the probability of death? What is the smallest con-
centration where there is more than 2% chance of dying? What is the concentration
where 50% of the organisms die – the so-called LD50 value?
We want a model for each of the groups that captures the probability of dying in that
particular group. In addition, we want to relate this probability across the groups
to the concentration – or dose – of the compound.
Figure 1.3 shows the data from an experiment with flies that are given different doses
of the insecticide dimethoat. The figure also shows a curve that is inferred from the
given dataset, which can be used to answer the questions phrased above. A major
topic of the present set of notes is to develop the theory and methodology for how
such a curve is inferred and how we assess the uncertainty involved in reporting e.g.
the inferred value of LD50.
1.0
0.8
Probability of death
0.6
0.4
0.2
0.0
−4 −3 −2 −1 0
log(concentration)
Figure 1.3: The relative frequency of dead flies plotted against the log(concentration)
of the insecticide dimethoat together with a curve inferred from the data that provide
a relation between the probability of fly death and the concentration
Obviously it would be unethical to carry out a controlled experiment with a lethal

outcome on a group of humans. If we are interested in the dose-response relation
regarding lethality for humans of a given compound, we can only use observational
data. That is, we can try to measure the dose that groups of humans are exposed
to for other reasons and relate that to their survival. It is important to be aware of
the fundamentally different nature of such a data collection strategy compared to
the controlled experiment. In particular, for the controlled experiment we design the
experiment so that the groups that are exposed to different doses are identical in
all other aspects. This allows us to conclude that it is actually the concentration of
the compound that is the cause of the death. For the observational data there could
potentially be a number of other differences between groups that are exposed to dif-
ferent dose levels. Observed differences in response can very well be caused by other
– observed or unobserved – alternative differences between the groups. This is a se-
rious weakness of observational data, and substantial amounts of research have been
done to facilitate causal conclusions from the analysis of observational data. Though
these developments are very useful, the conclusions from a statistical analysis of
observational data will never provide strong evidence of a causal explanation unless
it is backed up by serious subject matter theory and/or controlled experiments. ⋄
Example 1.2.6 (Assay measurements). A very large body of biological or biochem-
ical experiments are known as assays. The term covers a wide range of measurement
techniques that share the purpose of either measuring quantitatively the amount of
a substance in a sample, or detecting qualitatively if the substance is present or not.
There is a bewildering terminology associated with assays due to the many different
special techniques and areas of application. We cannot cover everything here but
only give one concrete example.
10 Introduction
ELISA (Enzyme-linked immunosorbent assay) is a so-called immunoassay used to

detect or quantify the presence of an antibody or antigen in a sample. A character-
istic property of this assay is that the final measurement is a measurement of a light
intensity at a given wavelength. Thus it is an indirect measurement of the concen-
tration of the substance of interest given by the resulting light intensity. Changes in
experimental conditions result in “noise” in the measurements so that repeated mea-
surements will give different results. Different biological replications will also result
in biological variation that will add to the measurement noise. All in all we attempt
to capture these variations with a probabilistic model so that we understand the
natural variation in our experiment even under identical conditions (up to what we
can control). In some situations – with the correct experimental design – it may be
possible and desirable to decompose the measurement noise from the biological vari-
ation. In other situations the two kinds of variation add up to a common variation
term where the decomposition is not possible.
From a statistical point of view there is, however, another reason to be interested
in assays. Because the measurement is an indirect measurement we will need a
method to relate the actual observation to the concentration of the substance. This
relation is known as the standard curve. In an ideal world there would be a single
curve relating the concentration to the measurement plus perhaps some measurement
noise. However, in reality such a single, globally valid curve is not likely to be found.
Instead it is inferred from measurements involving a standard solution series with
known concentrations. This leads to a problem known as regression where we want
to infer the relation between two variables where we know one and measure the other
with measurement noise. In the context of e.g. assays it is common to talk about
calibration, as our measurements are used to calibrate the equipment before doing
the experiment of actual interest. ⋄
Example 1.2.7 (Location and scale). A more abstract example of a statistical

model is the repeated measurement of a single, quantitative variable. It could be
a clinical measure like blood pressure, a biochemical measure like a protein con-
centration from an assay as described above, but it could also be a computational
result like a predicted minimal free energy for a set of (equal length) RNA-molecules.
It is customary to summarize such repeated measurements by a typical value, the
location, together with the spread, or scale, of the measurements.
There is an underlying statistical model, the location-scale model, for this kind of
data reduction, which is important in itself, but also forms a very important building
block for many other situations with multiple different groups of repeated measure-
ments, regression analysis, where the measurements depend on another observed
variable, or combinations with a complicated experimental design. Moreover, the
use of a normalization of the measurements to so-called z-values or z-scores is per-
haps best understood within the framework of the statistical model – especially the
assumptions upon which the normalization rests. It is also an integral part of the
location-scale model that the normalized z-values have a residual distribution, which
is sometimes taken to be the standard normal distribution, but which may in princi-
ple be anything. Specification of the residual distribution or clearly announcing the
lack of such a specification is a part of the location-scale model. ⋄
Example 1.2.8 (Local alignment). Local alignment of DNA-sequences or proteins

is one of the classical tools for bioinformaticians and molecular biologists. Various
implementations that solve the algorithmic problem, going back to the seminal paper
by Smith and Waterman, of optimally locally aligning two sequences – with a given
scoring scheme – have been around for quite some time. We will assume some famil-
iarity with the algorithmic side or at least a knowledge of what a local alignment is.
There is also an extensive literature on the probabilistic and statistical aspects of
analyzing the outcome of a local alignment. The central problem is to understand
the distribution of the optimal local alignment score if we align functionally and/or
evolutionary unrelated proteins, say. Otherwise, we have no way of judging whether
a given optimal local alignment expresses a true relationship between the proteins
or whether it is what we should expect to obtain for unrelated proteins.
If we take a bunch of equal length unrelated proteins, we are essentially in the frame-
work of the location-scale model as discussed. As we will illustrate, there is also a
quite good understanding of the residual distribution, which for relevant scoring
schemes can be taken to be the Gumbel distribution. The location and scale parame-
ters will then reflect such things as residue composition, the specific scoring scheme
– do we use BLOSUM60 or PAM250, the gap penalties etc. – and the length of the
proteins. We can hardly restrict our attention to aligning proteins of a single length
only, so we will be particularly interested in how the parameters depend upon pro-
tein length. This turns the statistical model into a regression model where we not
only want to understand the location and scale for a single given length but actually
how they vary with the lengths of the proteins. More ambitiously we can aim for
a model that also captures the dependence upon residue composition and scoring
scheme. ⋄
Example 1.2.9 (Gene expression). A gene expression microarray is an experimental

technique for measuring the expression level of all genes in a cell (culture) at a
given time. It is technically speaking a multiplex assay – see the discussion about
assays above – as we measure simultaneously the concentration of a large number
of substances at the same time.
To make a long story short, one locates on a plate or slide (the microarray) a number
of DNA-fragments – either small synthetic oligonucleotide probes or spots of larger
DNA sequences obtained by PCR-amplification of cDNA clones. Then one extracts
mRNA from cells, which is typically reverse transcribed into cDNA, and the cDNA
is poured onto the slide. The cDNA-molecules bind to the DNA-molecules on the
slide that are complementary by hydrogen bonds, and if the cDNA has been color
labeled in advance, it is possible to read of how much cDNA is attached to a given
probe on the plate. This gives an (indirect) measure of the expression level of the
corresponding gene in the cell. The initial measurement that comes out of such
12 Introduction
an experiment is an image, which shows light intensities measured over the slide.
The image is preprocessed to yield a single light intensity measurement for each
probe on the slide. It should be emphasized that there are numerous details in
the experimental design that can give rise to errors of very different nature. We
are not going to focus on these details, but one remark is appropriate. For this
technology it is not common to infer standard curves from data for each array.
Consequently it is not possible to obtain absolute concentration measurements only
relative measurements. In the comparison between measurements on different arrays,
elaborate normalization techniques are necessary to replace the lack of knowledge
of standard curves. Still we will only be able to discuss relative differences between
measurements across arrays and not absolute differences.
The resulting dataset from a microarray experiment is a high-dimensional vector of
measurements for each gene represented on the array. A typical experimental design
will include several arrays consisting of measurements for several individuals, say,
that may be grouped for instance according to different disease conditions. The mi-
croarray technology presents us with a lot of statistical challenges. High-dimensional
data, few replications per individual (if any), few individuals per group etc., and all
this give rise to high uncertainty in the statistical procedures for inference from
the data. This is then combined with the desire to ask biologically quite difficult
questions. Indeed a challenging task. We will in these notes mostly use microarray
data to illustrate various simple models on a gene-by-gene level. That is, where we
essentially consider each gene represented on the array independently. This is an im-
portant entry to the whole area of microarray data analysis, but these considerations
are only the beginning of a more serious analysis.
The use of microarrays for gene expression measurements is a well developed tech-
nique. It is, however, challenged by other experimental techniques. At the time of
writing the so-called next-generation sequencing techniques promise to replace mi-
croarrays in the near future. Though fundamentally different from a technical point
of view, the nature of the resulting data offer many of the same challenges as men-
tioned above – high-dimensional data with few replications per individual, and still
the biological questions are difficult. The hope is that the sequencing based tech-
niques have smaller measurement noise and that the need for normalization is less
pronounced. In other words, that we get closer to a situation where a single standard
curve is globally valid. Whether this is actually the case only time will show. ⋄
2
Probability Theory
2.1 Introduction
Probability theory provides the foundation for doing statistics. It is the mathemat-
ical framework for discussing experiments with an outcome that is uncertain. With
probability theory we capture the mathematical essence of the quantification of un-
certainty by abstractly specifying what properties such a quantification should have.
Subsequently, based on the abstract definition we derive properties about probabil-
ities and give a number of examples. This approach is axiomatic and mathematical
and the mathematical treatment is self-contained and independent of any interpre-
tation of probabilities we might have. The interpretation is, however, what gives
probability theory its special flavor and makes it applicable. We give a mathemat-
ical presentation of probability theory to develop a proper language, and to get
accustomed to the vocabulary used in probability theory and statistics. However, we
cannot and will not try to derive everything we need along the way. Derivations of
results are made when they can illustrate how we work with the probabilities and
perhaps illuminate the relation between the many concepts we introduce.
2.2 Sample spaces
We will throughout use E to denote a set, called the sample space, such that elements
x ∈ E represent the outcome of an experiment we want to conduct. We use small
letters like x, y, z to denote elements in E. An event, A ⊆ E, is a subset of E, and
we will use capital letters like A, B, C to denote events. We use the word experiment
in a wide sense. We may have a real wet lab experiment in mind or another classical
empirical data collection process in mind. But we may also have a database search or
some other algorithmic treatment of existing data in mind – or even an experiment
13
14 Probability Theory
carried out entirely on a computer, that is, a computer simulation.

The notion of a general sample space may seem abstract and therefore difficult. In
practice the choice of sample space is often quite natural. If we throw a dice the sam-
ple space is the set {¥, ¦, §, ¨, ©, ª }. When modeling random nucleic acids (DNA)
the sample space is {A, C, G, T}. These are both examples of finite sample spaces. A
very common, infinite sample space is the set of real numbers, R, which we use as
sample space in most cases where we measure a single, quantitative variable. That
could be temperature, pH-value, pressure, concentration of a chemical compound,
weight or height of an individual, and many, many other things.
A more complicated sample space is found if we for instance consider mass spec-
trometry. In mass spectrometry one measures the amount of ions in a sample as
a function of the ratio between the mass and the charge (the m/z-ratio) of ions,
which produces a mass spectrum. The sample space is a set of functions – the set of
potential mass spectra. We will typically regard the mass spectrum to be a contin-
uous function of the m/z-ratio, and the sample space is thus the set of continuous
functions on an appropriate interval of m/z-values.
Mass spectrum
40
30
Intensity
20
10
0 5000 10000 15000 20000
m/z
Figure 2.1: A raw mass spectrum. The sample is regarded as a (peaky and oscillating)
continuous intensity of molecules as a function of the ratio between the mass and
charge of the molecule (the m/z-ratio).
Microarray experiments constitute another class of experiments where the outcome

takes its value in a complicated sample space. When the microarray is exposed
to a sample of fluorescent RNA-molecules (in fact, cDNA reverse transcribed from
the RNA) specific molecules prefer to bind to specific probes, and by subsequently
Probability measures 15
scanning the array one obtains a light intensity as a function of the position on the
slide. The light intensity in the vicinity of a given probe is a measure of the amount
of RNA that binds to the probe and thus a measure of the (relative) amount of
that particular RNA-molecule in the sample. The sample space is a set of two-
dimensional functions – the set of potential light intensities as a function of position
on the slide. In most cases the actual outcome of the experiment is not stored, but
only some discretized version or representation of the outcome. This is mostly due
to technical reasons, but it may also be due to preprocessing of the samples by
computer programs. For microarrays, the light intensities are first of all stored as
an image of a certain resolution, but this representation is typically reduced even
further to a single quantity for each probe on the array.
Though the choice of sample space in many cases is given by the experiment we
consider, we may face situations where several choices seem appropriate. The raw
data from a microarray experiment is an image, but do we want to model the ac-
tual image produced from the scanning? Or are we satisfied with a model of the
summarized per probe measurements? In particular when we encounter complicated
measurement processes we may in practice need different preprocessing or adapta-
tion steps of the collected data before we actually try to model and further analyze
the data. It is always good practice then to clearly specify how these preprocessing
steps are carried out and what the resulting final sample space and thus the actual
data structure is.
2.3 Probability measures
If we are about to conduct an uncertain experiment with outcome in the sample space
E we use probabilities to describe the result of the experiment prior to actually
performing the experiment. Since the outcome of the experiment is uncertain we
cannot pinpoint any particular element x ∈ E and say that x will be the outcome.
Rather, we assign to any event A ⊆ E a measure of how likely it is that the event
will occur, that is, how likely it is that the outcome of the experiment will be an
element x belonging to A.
Definition 2.3.1. A probability measure P assigns to all events A ⊆ E a value

P (A) ∈ [0, 1]. The probability measure P must fulfill that:
(i) P (E) = 1,
(ii) if A1 , . . . , An are disjoint events
P (A1 ∪ . . . ∪ An ) = P (A1 ) + . . . + P (An ).
Property (ii) above is known as additivity of a probability measure. It is crucial that

the events are disjoint for additivity to hold.
We will throughout use the name probability distribution, or just distribution, inter-
changeably with probability measure. We can immediately derive a number of useful
and direct consequences of the definition.
For any event A, we can write E = A∪Ac with A and Ac disjoint, hence by additivity
1 = P (E) = P (A) + P (Ac ).
From this we obtain

P (Ac ) = 1 − P (A).
In particular, for the empty event ∅ we have
P (∅) = 1 − P (E) = 0.
If A, B ⊆ E are two events such that A ⊆ B we have that
B = A ∪ (B\A)
where A and B\A are disjoint. By additivity again we obtain that
P (B) = P (A) + P (B\A),
hence since P (B\A) ≥ 0 it follows that if A ⊆ B then
P (A) ≤ P (B). (2.1)
Finally, if A, B ⊆ E are any two events – not necessarily disjoint – then with C =
A ∩ B we have that A = (A\C) ∪ C with (A\C) and C disjoint, thus by additivity
P (A) = P (A\C) + P (C).
Moreover,
A ∪ B = (A\C) ∪ B
with the two sets on the right hand side being disjoint, thus by additivity again
P (A ∪ B) = P (A\C) + P (B)
= P (A) + P (B) − P (A ∩ B). (2.2)
Intuitively speaking, the result states that the probability of the union A ∪ B is the
sum of the probabilities for A and B, but when the sets are not disjoint we have
“counted” the probability of the intersection A ∩ B twice. Thus we have to subtract
it.
We summarize the results derived.
Result 2.3.2. Properties of probability measures:

Math Box 2.3.1 (General Probability Measures). In Definition 2.3.1 we only

require that the probability measure P should be additive in the sense that for a
finite sequence A1 , . . . , An of disjoint sets we have that
P (A1 ∪ . . . ∪ An ) = P (A1 ) + . . . + P (An ).
Probability measures are usually required to be σ-additive, meaning that for any
infinite sequence A1 , A2 , A3 , . . . of disjoint sets it holds that
P (A1 ∪ A2 ∪ A3 ∪ . . .) = P (A1 ) + P (A2 ) + P (A3 ) + . . . . (2.3)
It is a perfectly natural requirement and as it stands, it may seem as a quite innocent

extension. If P for instance is a probability measure on a countably infinite set E
given by point probabilities (p(x))x∈E , it may be observed that (2.3) is fulfilled
automatically by the definition of P from point probabilities. Requiring that P
is σ-additive is, however, a more serious business when dealing with probability
measures on non-discrete sample spaces.
It is, moreover, problematic to assign a probability to every subset of E. In general
one needs to restrict attention to a collection of subsets E, which is required to
contain E and to fulfill
• if A ∈ E then Ac ∈ E
• if A1 , A2 , . . . ∈ E then A1 ∪ A2 ∪ . . . ∈ E
A collection E of subsets fulfilling those requirements is called a σ-algebra and the

sets in E are called measurable. If one doesn’t make this restriction a large num-
ber of “natural” probability measures don’t exist. The most commonly occurring
σ-algebras contain so many sets that it requires sophisticated mathematical argu-
ments to show that there indeed exist sets not in the σ-algebra. From a practical,
statistical point of view it is unlikely that we ever encounter non-measurable sets,
which we really want to assign a probability. However, in some areas of mathemati-
cal statistics and probability theory, measurability problems are present all the time
– especially problems with verifying that sets of interest are actually measurable.
• For any event A ⊆ E,

P (Ac ) = 1 − P (A).
• P (∅) = 0
• For two events A ⊆ B ⊆ E,
P (B) = P (A) + P (B\A) and P (A) ≤ P (B).
• For any two events A, B ⊆ E,
P (A ∪ B) = P (A) + P (B) − P (A ∩ B).

Note that the abstract definition of a probability measure doesn’t say anything about
how to compute the probability of an event in a concrete case. But we are on the
other hand assured that if we manage to come up with a probability measure, it
assigns a probability to any event, no matter how weird it may be. Even in cases
where there is no chance that we would ever be able to come up with an analytic
computation of the resulting probability. Moreover, any general relation or result
derived for probability measures, such as those derived above, apply to the concrete
situation.
The assignment of a probability P (A) to any event A is a quantification of the
uncertainty involved in our experiment. The closer to one P (A) is the more certain
it is that the event A occurs and the closer to zero the more uncertain. Some people,
especially those involved in gambling, find it useful to express uncertainty in terms
of odds. Given a probability measure P we define for any event A the odds that A
occurs as
P (A) P (A)
ξ(A) = = . (2.4)
1 − P (A) P (Ac )
Thus we assign to the event A ⊆ E a value ξ(A) ∈ [0, ∞], and like the probability
measure this provides a quantification of the uncertainty. The larger ξ(A) is the more
certain it is that A occurs. A certain event (when P (A) = 1) is assigned odds ∞. It
follows that we can get the probabilities back from the odds by the formula
1
P (A) = . (2.5)
1 + ξ(Ac )
Odds are used in betting situations because the odds tell how fair1 bets should
be constructed. If two persons, player one and player two, say, make a bet about
whether event A or event Ac occurs, how much should the loser pay the winner for
this to be a fair bet? If player one believes that A occurs and is willing to bet 1
kroner then for the bet to be fair player two must bet ξ(Ac ) kroner on event Ac . For
gambling, this is the way British bookmakers report odds – they say that odds for
event A are ξ(Ac ) to 1 against. With 1 kroner at stake and winning you are paid
ξ(Ac ) back in addition to the 1 kroner. Continental European bookmakers report
the odds as ξ(Ac ) + 1, which include what you staked.
The frequency interpretation states that the probability of an event A should
equal the long run frequency of the occurrence of event A if we repeat the exper-
iment indefinitely. That is, suppose that we perform n independent and identical
experiments all governed by the probability measure P and with outcome in the
sample space E, and suppose that we observe x1 , . . . , xn . Then we can compute the
relative frequency of occurrences of event A
n
1X
εn (A) = 1A (xi ). (2.6)
n
i=1
1
Fair means that on average the players should both win (and loose) 0 kroner, cf. the frequency
interpretation.
Here 1A is the indicator function for the event A, so that 1A (xi ) equals 1 if xi ∈ A
and 0 otherwise. We sometimes also write 1(xi ∈ A) instead of 1A (xi ). We see
that εn (A) is the fraction of experiments in which the event A occurs. As n grows
large, the frequency interpretation says that εn (A) must be approximately equal to
P (A). Note that this is not a mathematical result! It is the interpretation of what
we want the probabilities to mean. To underpin the interpretation we can show
that the mathematical theory based on probability measures really is suitable for
approximating relative frequencies from experiments, but that is another story. We
also need to make a precise definition of what we mean by independent in the first
place.
The frequency interpretation provides the rationale for using odds in the construction
of fair bets. If the two players repeat the same bet n times with n suitably large, and
if the bet is fair according to the definition above with player one betting 1 kroner
on event A each time, then in the i’th bet, player one wins
ξ(Ac )1A (xi ) − 1Ac (xi ),
because this equals ξ(Ac ) if event A comes out and −1 if Ac comes out. Considering
all n bets, and excluding the case P (A) = 0, player one will on average win
1 Xh i
n
ξ(Ac )1A (xi ) − 1Ac (xi ) = ξ(Ac )εn (A) − εn (Ac )
n
i=1
≃ ξ(Ac )P (A) − P (Ac )
P (Ac )
= P (A) − P (Ac ) = 0.
P (A)
Likewise, player two will on average win approximately 0 kroner if n is sufficiently

large. This is why we say that the bet is fair. Note, however, that the total sum that
player one wins/loses is
n h
X i
ξ(Ac )1A (xi ) − 1Ac (xi ) ,
i=1
which may deviate substantively from 0 – and the deviation will become larger as n
increases.
The Bayesian interpretation allows us to choose the probability measure de-
scribing the experiment subjectively. In this interpretation the probability measure
is not given objectively by the experiment but reflects our minds and our knowledge
of the experiment before conducting it. One theoretical justification of the Bayesian
interpretation is that we play a mind game and make everything into a betting sit-
uation. Thus we ask ourselves which odds (for all events) we believe to be fair in
a betting situation before conducting the experiment. Note that this is an entirely
subjective process – there is no theory dictating what fairness means – but we are
nevertheless likely to have an opinion about what is fair and what is not. It is pos-
sible to show that if we make up our minds about fair odds in a consistent manner2
we necessarily end up with a probability measure defined by (2.5). This is the so-
called Dutch book argument. The probabilities don’t represent the long term relative
frequencies when repeating the experiment. After having conducted the experiment
once we have gained new information, which we might want to take into account
when deciding what we then believe to be fair odds.
The two interpretations are fundamentally different of nature and this has conse-
quently lead to different opinions in the statistical literature about how to develop a
suitable and well-founded statistical theory. The methodologies developed based on
either interpretation differ a lot – at least in principle. On the other hand there are
many practical similarities and most Bayesian methods have a frequency interpre-
tation and vice versa. Discussions about which of the interpretations, if any, is the
correct one is of a philosophical and meta mathematical nature – we cannot prove
that either interpretation is correct, though pros and cons for the two interpreta-
tions are often based on mathematical results. We will not pursue this discussion
further. The interested reader can find a balanced an excellent treatment in the book
Comparative Statistical Inference, Wiley, by Vic Barnett.
Throughout, probabilities are given the frequency interpretation, and methodology
is justified from the point of view of the frequency interpretation. This does not by
any means rule out the use of Bayesian methodology a priori, but the methodology
must then be justified within the framework of the frequency interpretation.
Example 2.3.3 (Coin Flipping). When we flip a coin it lands with either heads or
tails upwards. The sample space for describing such an experiment is
E = {heads, tails}
with the four events:

∅, {heads}, {tails}, E.
Any probability measure P on E must fulfill P (∅) = 0 and P (E) = 1 by definition
together with
P (tails) = 1 − P (heads),
hence the probability measure is completely determined by p = P (heads). Note here
the convention that when we consider an event consisting of only a single outcome
like {heads} we usually drop the curly brackets. This extremely simple experiment is
not so interesting in itself, but it serves the purpose of being a fundamental building
block for many more interesting sample spaces and probability measures. Maybe it
is not coin flipping directly that we use as a building block but some other binary
experiment with two possible outcomes. There are several often encountered ways
of coding or naming the outcome of a binary experiment. The most commonly used
2
Not being consistent means that if someone knows our choice of fair odds he can construct a
bet in such a way that he will win money with certainty.
Probability measures on discrete sets 21
sample space for encoding the outcome of a binary experiment is E = {0, 1}. When
using this naming convention we talk about a Bernoulli experiment. In the coin
flipping context we can let 1 denote heads and 0 tails, then if x1 , . . . , xn denote the
outcomes of n flips of the coin we see that x1 + . . . + xn is the total number of heads.
Moreover, we see that
P (x) = px (1 − p)1−x (2.7)
because either x = 1 (heads) in which case px = p and (1 − p)1−x = 1, or x = 0
(tails) in which case px = 1 and (1 − p)1−x = 1 − p. ⋄
Exercises
Exercise 2.3.1. If P (A) = 0.5, P (B) = 0.4 and P (A∪B) = 0.6, compute P ((A∩B)c ).
Exercise 2.3.2. Compute the odds ξ(A) of the event A if P (A) = 0.5, if P (A) = 0.9
and if P (Ac ) = 0.9. Compute the probability P (A) of the event A if ξ(A) = 10 and if
ξ(A) = 2.
Exercise 2.3.3. Consider the sample space E = {1, . . . , n} and let xi (k) = 1 if i = k
and = 0 if i 6= k. Show that
P (k) = P (1)x1 (k) · P (2)x2 (k) · . . . · P (n)xn (k) .
2.4 Probability measures on discrete sets
If E is an infinite set we may be able to arrange the elements in E in a sequential

way;
E = {x1 , x2 , x3 , . . .}.
That is, we may be able to give each element in E a positive integer index such
that each i ∈ N corresponds to exactly one element xi ∈ E. If we can do so, we say
that E is countably infinite. One example of a countably infinite set is the positive
integers themselves, N = {1, 2, 3, . . .}. Also the integers Z = {. . . , −2, −1, 0, 1, 2, . . .}
is a countably infinite set, and perhaps more surprising the rational numbers Q is
a countably infinite set. The set of real numbers is, however, not countable. This is
proved using the famous diagonal argument by George Cantor.
It is particularly simple to define probability measures on finite and countably infinite
sets and this is the subject of the present section. Defining probability measures on
the real line is a more subtle problem, which we defer to the following section.
Definition 2.4.1. We call a set E discrete if it is either finite or countably infinite.
The DNA, RNA and amino acid alphabets are three examples of finite and hence
discrete sets.
Example 2.4.2. Considering any finite set E we can define the set
E ∗ = {x1 x2 . . . xn | n ∈ N, xi ∈ E},
which is the set of all sequences of finite length from E. We claim that E ∗ is discrete.
If E is the DNA-alphabet, say, it is clear that there is an infinite number of DNA-
sequences of finite length in E ∗ , but it is no problem to list them as a sequence in
the following way:
A, C, G, T, AA, AC, AG, AT, CA, . . . , TG, TT, AAA, AAC, . . .
Hence we first list all sequences of length one, then those of length two, those of
length three, four, five and so on and so forth. We can encounter the use of E ∗ as
sample space if we want a probabilistic model of all protein coding DNA-sequences.
⋄
Example 2.4.3. In genetics we find genes occurring in different versions. The ver-
sions are referred to as alleles. If a gene exists in two versions it is quite common to
refer to the two versions as a and A or b and B. Thus if we sample an individual
from a population and check whether the person carries allele a or A we have the
outcome taking values in the sample space
E = {a, A}.
However, we remember that humans are diploid organisms so we actually carry two
copies around. Thus the real outcome is an element in the sample space
E = {aa, Aa, AA}.
The reason that aA is not an outcome also is simply that we cannot distinguish
between aA and Aa. The information we can obtain is just that there is an a and
an A, but not in any particular order.
If we also consider another gene simultaneously, the sample we get belongs to the
sample space
E = {aabb, aaBb, aaBB, Aabb, AaBb, AaBB, AAbb, AABb, AABB}
that contains nine elements. ⋄
To define a probability measure on E in principle requires that we assign a number,

P (A), to every event A ⊆ E. For a finite set of size n there are 2n different events so
even for sets of quite moderate size, n = 20, say, to list 220 = 1048576 probabilities
is not a very practical way of defining a probability measure – not to mention the
problems we would get checking that the additivity property is fulfilled. Fortunately
there is another and somewhat more tractable way of defining a probability measure
on a discrete set.
Definition 2.4.4. If E is discrete we call (p(x))x∈E a vector of point probabilities

indexed by E if 0 ≤ p(x) ≤ 1 and
X
p(x) = 1. (2.8)
x∈E
We define a probability measure P on E with point probabilities (p(x))x∈E by

X
P (A) = p(x) (2.9)
x∈A
for all events A ⊆ E.
The sum as written above over all x ∈ E can be understood in the following way:
Since E is discrete it is either finite or countably infinite. In the former case it is
just a finite sum so we concentrate on the latter case. If E is countable infinite we
know that we can order the elements as x1 , x2 , x3 , . . .. Then we simply define
X ∞
X
p(x) = p(xn ).
x∈E n=1
It even holds that the infinite sum on the right hand side is a sum of all positive
numbers. The careful reader may get worried here, because the arrangement of the
elements of E is arbitrary. What if we choose another way of listing the elements
will that affect the sum? The answer to this is no,
P because the terms are all positive,
in which case we can live happily with writing x∈E p(x) without being specific on
the order of summation. The same does not hold if the terms can be positive as well
as negative, but that is not a problem here.
Example 2.4.5 (Random Amino Acids). As mentioned in the introduction a se-

quence of nucleic acids may be regarded as being random if we don’t know any
better. Likewise, a sequence of amino acids (a protein) is regarded as being random
if we don’t know any better. In these notes a protein is simply a finite sequence from
the sample space

E = A, R, N, D, C, E, Q, G, H, I, L, K, M, F, P, S, T, W, Y, V .
To specify a probability measure on E we have to specify point probabilities p(x)

for x ∈ E according to 2.4.4. A valid choice is
p(x) = 0.05
for all x ∈ E. Clearly p(x) ∈ [0, 1] and since there are 20 amino acids
X
p(x) = 20 × 0.05 = 1.
x∈E
R Box 2.4.1 (Vectors). A fundamental data structure in R is a vector of e.g.

integers or reals. A vector of real valued numbers can be typed in by a command
like
> x <- c(1.3029,4.2,5.3,3.453,342.34)
This results in a vector x of length five. Even a single number is always regarded
as a vector of length one in R. The c above should therefore be understood as a
concatenation of five vectors of length one. If we define another vector of length
two by
> y <- c(1.3,2.3)
we can concatenate x and y to give the vector
> z <- c(x,y)
Then z is a vector of length seven containing the numbers

1.3029, 4.2, 5.3, 3.453, 342.34, 1.3, 2.3. A probability vector in R is simply a
vector of positive real numbers that sum to one.
Under this probability measure all amino acids are equally likely, and it is known as
the uniform distribution on E, cf. the example below. A more reasonable probability
measure on E is given by the relative frequencies of the occurrence of the amino acids
in real proteins, cf. the frequency interpretation of probabilities. The Robinson-
Robinson probabilities come from a survey of a large number of proteins. They read
Amino acid Probability

A 0.079
R 0.051
N 0.045
D 0.054
C 0.019
E 0.063
Q 0.043
G 0.074
H 0.022
I 0.051
L 0.091
K 0.057
M 0.022
F 0.039
P 0.052
S 0.071
T 0.058
W 0.013
Y 0.032
V 0.064
Using the Robinson-Robinson probabilities, some amino acids are much more prob-
able than others. For instance, Leucine (L) is the most probable with probability
0.091 whereas Tryptophan (W) is the least probable with probability 0.013. ⋄
Example 2.4.6 (Uniform distribution). If E is a finite set containing n elements
we can define a probability measure on E by the point probabilities
1
p(x) =
n
P
for all x ∈ E. Clearly p(x) ∈ [0, 1] and x∈E p(x) = 1. This distribution is called
the uniform distribution on E.
If P is the uniform distribution on E and A ⊆ E is any event it follows by the
definition of P that
X1 |A|
P (A) = =
n n
x∈A
with |A| denoting the number of elements in A. Since the elements in E are all
possible but we only regard those in A as favorable, this result gives rise to the
formula
number of favourable outcomes
P (A) = ,
number of possible outcomes
which is valid only when P is the uniform distribution. Even though the formula
looks innocent, it can be quite involved to apply it in practice. It may be easy to
specify the sample space E and the favorable outcomes in the event A, but counting
the elements in A can be difficult. Even counting the elements in E can sometimes
be difficult too. ⋄
R Box 2.4.2 (Biostrings). The package Biostrings comes as part of the

Bioconductor standard bundle of packages for R. With
> library(Biostrings)
you get access to some representations of biological sequences in R. For

instance
> someDNA <- DNAString("ACGTACGTACTT")
constructs a DNAString object. You can also make an RNAString or a

generic biostring using BString. There is a letter function available to
extract letters at specific positions, a matchPattern and a countPattern
function, and there is an alphabetFrequency function. There is also a read-
FASTA function.
The package is developed for handling biological sequences in particular,
but may also be suitable in other situations for handling and analyzing
large stings in R. The BString class is definitely more flexible than the
built in R strings, making the manipulation of especially large strings easier
and more efficient.
0.20
0.20
0.15
0.15
Probability
Probability
0.10
0.10
0.05
0.05
0.00
0.00
0 1 2 3 4 5 6 7 8 9 11 13 15 17 19 21 23 25 0 1 2 3 4 5 6 7 8 9 11 13 15 17 19 21 23 25
n n
Figure 2.2: The point probabilities for the Poisson distribution with λ = 5 (left) and
λ = 10 (right).
Example 2.4.7 (The Poisson Distribution). The (infinite) Taylor expansion of the
Descriptive methods 27
exponential function is given as

∞
X λn
exp(λ) = , λ ∈ R.
n=0
n!
If λ > 0 the numbers

λn
p(n) = exp(−λ)
n!
are positive and
∞
X ∞
X λn
λn
exp(−λ) = exp(−λ) = exp(−λ) exp(λ) = 1.
n! n!
n=0 n=0
Hence (p(n))n∈N0 can be regarded as point probabilities for a probability measure

on the non-negative integers N0 . This distribution is called the Poisson distribution.
A typical application of the Poisson distribution is as a model for the number of times
a rather unlikely event occurs in a large number of replications of an experiment.
One can, for instance, use the Poisson distribution as a model of the number of (non-
overlapping) occurrences of a given sequence pattern in a long text string. This has
found applications in biological sequence analysis and data mining of text collections.
⋄
2.5 Descriptive methods
If we have observations x1 , . . . , xn from a probability measure P on a discrete sample

space E, we want to summarize the observations in a form that is somewhat com-
prehensible. A tabulation of the data consists of counting the number of occurrences,
nx , of the outcome x ∈ E among the n outcomes x1 , . . . , xn . In other words
n
X
nx = 1(xi = x)
i=1
is the frequency of the occurrence of x in the sample. Note the relation between the
tabulation of (absolute) frequencies and the computation of the relative frequencies
εn (x) is
nx
εn (x) = .
n
For variables with values in a discrete sample space we have in general only the
possibility of displaying the data in a summarized version by tabulations. Depending
on the structure of the sample space E the tables can be more or less informative. If
E is a product space, that is, if the data are multivariate, we can consider marginal
tables for each coordinate as well as multivariate tables where we cross-tabulate
0.20
0.20
0.15
0.15
Probability
Probability
0.10
0.10
0.05
0.05
0.00
0.00
0 2 4 6 8 10 0 1 2 3 4 5 6 7 8 9 11 13 15 17 19 21 23 25
n n
Figure 2.3: Example of a theoretical bar plot. Here we see the Poisson point prob-
abilities (λ = 4) plotted using either the plot function with type="h" (left) or the
barplot function (right).
two or more variables. Tables in dimension three and above are quite difficult to
comprehend.
For the special case with variables with values in Z we can also use bar plots to display
the tabulated data. We compare the bar plot with a bar plot of the corresponding
point probabilities p(n) as a function of n ∈ Z to get a picture of where the measure
is concentrated and what the “shape” of the measure is. Usually, the frequencies and
point probabilities are plotted as horizontal bars. It is preferable to plot the relative
frequencies in the bar plot so that the y-axis becomes comparable to the y-axis for
the theoretical bar plot of the point probabilities.
2.5.1 Mean and variance
For probability measures on a discrete sample space E ⊆ R we can define the mean
and variance in terms of the point probabilities.
Definition 2.5.1. If P is a probability measure on E with point probabilities (p(x))x∈E
that fulfill X
|x|p(x) < ∞
x∈E
then we define the mean under P as
X
µ= xp(x).
x∈E
R Box 2.5.1 (Bar plots). We can use either the standard plot function in
R or the barplot function to produce the bar plot. The theoretical bar plot
of the point probabilities for the Poisson distribution with parameter λ = 4
is seen in Figure 2.3. The figure is produced by:
> plot(0:10, dpois(0:10, 4), type = "h", ylab = "Probability",

+ xlab = "n", ylim = c(0, 0.2), lwd = 2)
> barplot(dpois(0:10, 4), ylab = "Probability", xlab = "n",
+ space = 2, ylim = c(0, 0.2), names.arg = 0:10)
We can make R generate the 500 outcomes from the Poisson distribution
and tabulate the result using table, and then we can easily make a bar plot
of the frequencies. Figure 2.4 shows such a bar plot produced by:
> tmp <- table(rpois(500, 4))

> plot(tmp, ylab = "Frequency", xlab = "n")
> barplot(tmp, ylab = "Frequency", xlab = "n", space = 2)
One should note that the empirical bar plot – besides the y-axis – should
look approximately like the theoretical bar plot. One may choose to nor-
malize the empirical bar plot (use relative frequencies) so that it becomes
directly comparable with the theoretical bar plot. Normalization can in par-
ticular be useful, if one wants to compare two bar plots from two samples of
unequal size. Note that if we simulate another 500 times from the Poisson
distribution, we get a different empirical bar plot.
If, moreover, X
x2 p(x) < ∞
x∈E
we define the variance under P as
X
σ2 = (x − µ)2 p(x).
x∈E
The variance is on a quadratic scale, and we define the standard deviation as

√
σ = σ2.
Example 2.5.2. Let P be the uniform distribution on {1, . . . , n} ⊆ R. In this case

all the sums that we will need to consider are sums over {1, . . . , n} only, so they are
finite and well defined. We find that
n
X n
k 1X n+1
µ = = k=
n n 2
k=1 k=1
100
100
80
80
60
60
Frequency
Frequency
40
40
20
20
0
0
0 1 2 3 4 5 6 7 8 9 10 11 12 0 1 2 3 4 5 6 7 8 9 10 11 12
n n
Figure 2.4: Example of an empirical bar plot. This is a plot of the tabulated values
of 500 variables from the Poisson distribution (λ = 4) plotted using either the plot
function (left) or the barplot function (right).
where we use the formula 1 + 2 + 3 + . . . + (n − 1) + n = (n + 1)n/2. Note that

µ = (n + 1)/2 is exactly the midpoint between 1 and n, but note also the µ is not an
integer if n is even. Thus the mean can very well be located outside of the sample
space.
Regarding the variance we find that
n
X n
(k − µ)2 1X 2
σ2 = k − 2kµ + µ2
=
n n
k=1 k=1
n n
!
1 X X (n + 1)(2n + 1) (n + 1)2
2 2
= k − 2µ k + nµ = −
n 6 4
k=1 k=1
where we have used the formula 12 +22 +32 +. . .+(n−1)2 +n2 = n(n+1)(2n+1)/6.
Rearranging the result into a single fraction yields that the variance for the uniform
distribution is
(n + 1)(n − 1)
σ2 = .
12
⋄
If have a dataset x1 , . . . , xn of observations from E the sample mean or average is

n
1X
µ̂n = xk .
n
k=1
If the observations are all realizations from the same probability measure P the
sample mean is an estimate of the (unknown) mean under P – provided there is a
mean. Likewise, the sample variance
n
1X
σ̃n2 = (xk − µ̂n )2
n
k=1
is an estimate of the variance under P provided there is a variance.
Exercises
Ï Exercise 2.5.1. Figure out what the following R-command does:

> tmp <- paste(sample(c("A","C","G","T"),1000,replace=T),
+ sep="",collapse="")
Hint: You may want to try it out and to consult the R-help for the functions sample
and paste.
Ï Exercise 2.5.2. An important function of DNA is as a blueprint for proteins. This
translation of DNA to a protein works in triples of DNA-letters. A triple of letters
from DNA is referred to as a codon. DNA-sequences that encode proteins start with
the start codon ATG and stops with one of the three stop codons TAA , TAG, or TGA. In
between there is a number of complete codons. Figure out how to find the number of
codons in tmp before the first stop codon.
Hint: You can do regular pattern matching in R using grep or regexpr. With greg-
expr you obtain a list of positions in the string.
Ï Exercise 2.5.3. Compute for 1000 replications the number of codons in a random
DNA-sequence that occur before the first stop codon. Explain why you sometimes get
the length −1 (if you do that).
Hint: You may want to consider the replicate function.
Ï Exercise 2.5.4. For the 1000 replications, produce a plot showing, as a function of n,
the number of times that n codons occur before the first stop codon. That is, produce
a bar plot of the relative frequencies of the number of codons before the first stop
codon.
Hint: You can use the table function to summarize the vector produced in Exercise
2.5.3.
Ï Exercise 2.5.5. How can we simulate under the probability measure on E =
{A, C, G, T} given by the point probabilities
P (A) = 0.3, P (C) = 0.1, P (G) = 0.4, P (T) = 0.2,
or
P (A) = 0.3, P (C) = 0.4, P (G) = 0.1, P (T) = 0.2.
What happens to the number of codons before the occurrence of the first stop codon?
Exercise 2.5.6. An open reading frame in a DNA-sequence is a segment of codons
between a start and a stop codon. A long open reading frame is an indication that
the open reading frame is actually coding for a protein since long open reading frames
would be unlikely by chance. Discuss whether you believe that an open reading frame
of length more than 33 is likely to be a protein coding gene.
Exercise 2.5.7. Compute the mean, variance and standard deviation for the uniform
distribution on {1, . . . , 977}.
⋆ Exercise 2.5.8. Show by using the definition of mean and variance that for the
Poisson distribution with parameter λ > 0
µ = λ and σ 2 = λ. (2.10)
What is the standard deviation?

Exercise 2.5.9. The geometric distribution on N0 is given by the point probabilities
p(k) = p(1 − p)k
for p ∈ (0, 1) and k ∈ N0 . Show that

∞
X
p(k) = 1.
k=0
Compute the mean under the geometric distribution for p = 0.1, 0.2, . . . , 0.9.
Hint: If you are not able to compute a theoretical formula for the mean try to compute
the value of the infinite sum approximately using a finite sum approximation – the
computation can be done in R.
Ï Exercise 2.5.10. Plot the point probabilities for the Poisson distribution, dpois,
with λ = 1, 2, 5, 10, 100 using the type="h". In all five cases compute the probability
of the events A1 = {n ∈ N0 | − σ ≤ n − µ ≤ σ}, A2 = {n ∈ N0 | − 2σ ≤ n − µ ≤ 2σ},
A3 = {n ∈ N0 | − 3σ ≤ n − µ ≤ 3σ}.
Ï Exercise 2.5.11. Generate a random DNA sequence of length 10000 in R with each
letter having probability 1/4. Find out how many times the pattern ACGTTG occurs in
the sequence.
Ï Exercise 2.5.12. Repeat the experiment above 1000 times. That is, for 1000 se-
quences of length 10000 find the number of times that the pattern ACGTTG occurs
in each of the sequences. Compute the average number of patterns occurring per se-
quence. Make a bar plot of the relative frequencies of the number of occurrences and
compare with a theoretical bar plot of a Poisson distribution with λ chosen suitably.
2.6 Probability measures on the real line
Defining a probability measure on the real line R yields, to an even larger extent than
in the previous section, the problem: How are we going to represent the assignment
of a probability to all events in a manageable way? One way of doing so is through
distribution functions.
Definition 2.6.1. For a probability measure P on R we define the corresponding

distribution function F : R → [0, 1] by
F (x) = P ((−∞, x]).
That is, F (x) is the probability that under P the outcome is less than or equal to x.
Probability measures on the real line 33
We immediately observe that since (−∞, y] ∪ (y, x] = (−∞, x] for y < x and that
the sets (−∞, y] and (y, x] are disjoint, the additive property implies that
F (x) = P ((−∞, x]) = P ((−∞, y]) + P ((y, x]) = F (y) + P ((y, x]),
or in other words
P ((y, x]) = F (x) − F (y).
We can derive more useful properties from the definition. If x1 ≤ x2 then (−∞, x1 ] ⊆
(−∞, x2 ], and therefore from (2.1)
F (x1 ) = P ((−∞, x1 ]) ≤ P ((−∞, x2 ]) = F (x2 ).
Two other properties of F are consequences of what is known as continuity of prob-

ability measures. Intuitively, as x tends to −∞ the set (−∞, x] shrinks towards the
empty set ∅, which implies that
lim F (x) = P (∅) = 0.

x→−∞
Similarly, when x → ∞ the set (−∞, x] grows to the whole of R and
lim F (x) = P (R) = 1.

x→∞
Likewise, by similar arguments, when ε > 0 tends to 0 the set (−∞, x + ε] shrinks
towards (−∞, x] hence
lim F (x + ε) = P ((−∞, x]) = F (x).

ε→0,ε>0
We collect three of the properties derived for distribution functions.
Result 2.6.2. A distribution function F has the following three properties:
(i) F is increasing: if x1 ≤ x2 then F (x1 ) ≤ F (x2 ).
(ii) limx→−∞ F (x) = 0 and limx→∞ F (x) = 1.
(iii) F is right continuous at any x ∈ R: limε→0,ε>0 F (x + ε) = F (x)
It is of course useful from time to time to know that a distribution function satisfies
property (i), (ii), and (iii) in Result 2.6.2, but that these three properties completely
characterize the probability measure is more surprising.
Result 2.6.3. If F : R → [0, 1] is a function that has property (i), (ii), and (iii)
in Result 2.6.2 there is precisely one probability measure P on R such that F is the
distribution function for P .
1.0
1.0
0.5
0.5
−8 −4 0 4 8 −4 0 4 8
Figure 2.5: The logistic distribution function (left, see Example 2.6.4). The Gumbel
distribution function (right, see Example 2.6.5). Note the characteristic S-shape of
both distribution functions.
This result not only tells us that the distribution function completely characterizes P
but also that we can specify a probability measure just by specifying its distribution
function. This is a useful result but also a result of considerable depth, and a formal
derivation of the result is beyond the scope of these notes.
Example 2.6.4 (Logistic distribution). The logistic distribution has distribution

function
1
F (x) = .
1 + exp(−x)
The function is continuous, and the reader is encouraged to check that the properties
of the exponential function ensure that also (i) and (ii) for a distribution function
hold for this function. ⋄
Example 2.6.5 (The Gumbel distribution). The distribution function defined by
F (x) = exp(− exp(−x))
defines a probability measure on R, which is known as the Gumbel distribution. We

leave it for the reader to check that the function indeed fulfills (i), (ii) and (iii). The
Gumbel distribution plays a role in the significance evaluation of local alignment
scores, see Section 2.12. ⋄
If our sample space E is discrete but actually a subset of the real line, E ⊆ R, like N
or Z, we have two different ways of defining and characterizing probability measures
on E: through point probabilities or through a distribution function. The connection
is given by X
F (x) = P ((−∞, x]) = p(y).
y≤x
1.0
1.0
0.5
0.5
0 5 10 15 20 25 0 5 10 15 20 25
Figure 2.6: The distribution function for the Poisson distribution, with λ = 5 (left)
and λ = 10 (right).
Example 2.6.6. The distribution function for the Poisson distribution with param-
eter λ > 0 is given by
⌊x⌋
X λn
F (x) = exp(−λ)
n!
n=0
where ⌊x⌋ is the largest integer smaller than x. It is a step function with steps at
n
each of the non-negative integers n ∈ N0 and step size at n being p(n) = exp(−λ) λn!
⋄
A number of distributions are defined in terms of a density. Not all probability

measures on R have densities, e.g. those distributions that are given by point proba-
bilities on N. However, for probability measures that really live on R, densities play
to a large extent the same role as point probabilities do for probability measures on
a discrete set.
Definition 2.6.7. A probability measure P is said to have density f : R → [0, ∞) if
Z
P (A) = f (y)dy
A
for all events A ⊆ R. In particular, for a < b,

Z b
P ([a, b]) = f (y)dy.
a
The distribution function for such a probability measure is given by

Z x
F (x) = f (y)dy.
−∞
The reader may be unfamiliar with doing integrations over an arbitrary event A. If
f is a continuous function and A = [a, b] is an interval it should be well known that
the integral !
Z b Z
f (y)dy = f (y)dy
a [a,b]
is the area under the graph of f from a to b. It is possible for more complicated sets
A to assign a kind of generalized area to the set under the graph of f over A. We
will not go into any further details. An important observation is that we can specify
a distribution function F by
Z x
F (x) = f (y)dy (2.11)
−∞
if f : R → [0, ∞) is simply a positive function that fulfills that

Z ∞
f (y)dy = 1. (2.12)
−∞
Indeed, if the total area from −∞ to ∞ under the graph of f equals 1, the area under
f from −∞ to x is smaller (but always positive since f is positive) and therefore
Z x
F (x) = f (y)dy ∈ [0, 1].
−∞
When x → −∞ the area shrinks to 0, hence limx→∞ F (x) = 0 and when x → ∞

the area increases to the total area under f , which we assumed to equal 1 by (2.12).
Finally, a function given by (2.11) will always be continuous from which the right
continuity at any x follows.
That a probability measure P on R is given by a continuous density f means that the
probability of a small interval around x is proportional to the length of the interval
with proportionality constant f (x). Thus if h > 0 is small, so small that f can be
regarded as almost constantly equal to f (x) on the interval [x − h, x + h], then
Z x+h
P ([x − h, x + h]) = f (y)dy ≃ 2hf (x) (2.13)
x−h
where 2h is the length of the interval [x − h, x + h]. Rearranging, we can also write
this approximate equality as
P ([x − h, x + h])
f (x) ≃ .
2h
0.4
1.0
0.3
0.2
0.5
0.1
−4 −2 0 2 4 −4 −2 0 2 4
Figure 2.7: The density (left) and the distribution function (right) for the normal
distribution.
Example 2.6.8 (The Normal Distribution). The normal or Gaussian distribution

on R is the probability measure with density
2
1 x
f (x) = √ exp − .
2π 2
R∞
It is not entirely trivial to check that −∞ f (x)dx = 1, but this is indeed the case.
Using that f (x) = f (−x) we can first observe that
Z ∞ Z ∞
f (x)dx = 2 f (x)dx.
−∞ 0
Using the substitution y = x2 /2, and noting that

1 1
dx = dy = √ dy,
x 2y
we find that
Z ∞ Z ∞ Z ∞
2 1
f (x)dx = 2 f (x)dx = √ √ exp(−y)dy
−∞ 0 2π 0 2y
Z ∞
1 1 Γ(1/2)
= √ √ exp(−y)dy = √ ,
π 0 y π
√
where Γ(1/2) is the Γ-function evaluated in 1/2. So up to showing that Γ(1/2) = π,
cf. Appendix B, we have showed that f integrates to 1.
The distribution function is by definition
Z x 2
1 y
Φ(x) = √ exp − dy,
2π −∞ 2
and it is unfortunately not possible to give a (more) closed form expression for this
integral. It is, however, common usage to always denote this particular distribution
function with a Φ.
The normal distribution is the single most important distribution in statistics. There
are several reasons for this. One reason is that a rich and detailed theory about the
normal distribution and a large number of statistical models based on the normal
distribution can be developed. Another reason is that the normal distribution ac-
tually turns out to be a reasonable approximation of many other distributions of
interest – that being a practical observation as well as a theoretical result known
as the Central Limit Theorem, see Result 4.7.1. The systematic development of the
statistical theory based on the normal distribution is a very well studied subject in
the literature. ⋄
1.0
1.0
0.5
0.5
0 2 4 6 8 0 2 4 6 8
Figure 2.8: The density (left) and the distribution function (right) for the exponential
distribution with intensity parameter λ = 1 (Example 2.6.9).
Example 2.6.9 (The Exponential Distribution). Fix λ > 0 and define
f (x) = λ exp(−λx), x ≥ 0.
Let f (x) = 0 for x < 0. Clearly, f (x) is positive, and we find that
Z ∞ Z ∞
f (x)dx = λ exp(−λx)dx
−∞ 0
∞

= − exp(−λx) = 1.
0
For the last equality we use the convention exp(−∞) = 0 together with the fact that
exp(0) = 1. We also find the distribution function

Z x
F (x) = f (y)dy
−∞
Z x
= λ exp(−λy)dy
0
x

= − exp(−λy) = 1 − exp(−λx)
0
for x ≥ 0 (and F (x) = 0 for x < 0). The parameter λ is sometimes called the
intensity parameter. This is because the exponential distribution is often used to
model waiting times between the occurrences of events. The larger λ is, the smaller
will the waiting times be, and the higher the intensity of the occurrence of the events
will be. ⋄
It is quite common, as for the exponential distribution above, that we only want to
specify a probability measure living on an interval I ⊆ R. By “living on” we mean
that P (I) = 1. If the interval is of the form [a, b], say, we will usually only specify the
density f (x) (or alternatively the distribution function F (x)) for x ∈ [a, b] with the
understanding that f (x) = 0 for x 6∈ [a, b] (for the distribution function, F (x) = 0
for x < a and F (x) = 1 for x > b).
1.0
1.0
0.5
0.5
−1 0 1 2 −1 0 1 2
Figure 2.9: The density (left) and the distribution function (right) for the uniform
distribution on the interval [0, 1] (Example 2.6.10).
Example 2.6.10 (The Uniform Distribution). Let [a, b] ⊆ R be an interval and

define the function f : R → [0, ∞) by
1
f (x) = 1 (x).
b − a [a,b]
That is, f is constantly equal to 1/(b − a) on [a, b] and 0 outside. Then we find that
Z ∞ Z b
f (x)dx = f (x)dx
−∞ a
Z b
1
= dx
a b−a
1
= × (b − a) = 1.
b−a
Since f is clearly positive it is a density for a probability measure on R. This probabil-
ity measure is called the uniform distribution on the interval [a, b]. The distribution
function can be computed (for a ≤ x ≤ b) as
Z x
F (x) = f (y)dy
Z−∞x
1
= dy
a b−a
x−a
= .
b−a
In addition, F (x) = 0 for x < a and F (x) = 1 for x > b. ⋄
R Box 2.6.1. Distribution functions and densities for a number of standard prob-
ability measures on R are directly available within R. The convention is that if a
distribution is given the R-name name then pname(x) gives the distribution function
evaluated at x and dname(x) gives the density evaluated at x. The normal distri-
bution has the R-name norm so pnorm(x) and dnorm(x) gives the distribution and
density function respectively for the normal distribution. Likewise the R-name for
the exponential function is exp so pexp(x) and dexp(x) gives the distribution and
density function respectively for the exponential distribution. For the exponential
distribution pexp(x,3) gives the density at x with intensity parameter λ = 3.
Example 2.6.11 (The Γ-distribution). The Γ-distribution with shape parameter

λ > 0 and scale parameter β > 0 is the probability measure on [0, ∞) with density

1 λ−1 x
f (x) = λ x exp − , x>0
β Γ(λ) β
where Γ(λ) is the Γ-function evaluated in λ, cf. Appendix B. The Γ-distribution with
λ = 1 is the exponential distribution. The Γ-distribution with shape λ = f /2 for
f ∈ N and scale β = 2 is known as the χ2 -distribution with f degrees of freedom. The
σ 2 χ2 -distribution with f degrees of freedom is the χ2 -distribution with f degrees of
freedom and scale parameter σ 2 , thus it is the Γ-distribution with shape parameter
λ = f /2 and scale parameter β = 2σ 2 . ⋄
4
2.0
3
1.5
2
1.0
1
0.5
0.0
0.0 0.5 1.0 0.0 0.5 1.0
Figure 2.10: The density for the B-distribution (Example 2.6.12) with parameters
(λ1 , λ2 ) = (4, 2) (left) and (λ1 , λ2 ) = (0.5, 3) (right)
Example 2.6.12 (The B-distribution). The density for the B-distribution (pro-
nounced β-distribution) with parameters λ1 , λ2 > 0 is given by
1
f (x) = xλ1 −1 (1 − x)λ2 −1
B(λ1 , λ2 )
for x ∈ [0, 1]. Here B(λ1 , λ2 ) is the B-function, cf. Appendix B. This two-parameter
class of distributions on the unit interval [0, 1] is quite flexible. For λ1 = λ2 = 1
we obtain the uniform distribution on [0, 1], but for other parameters we can get a
diverse set of shapes for the density – see Figure 2.10 for two particular examples.
Since the B-distribution always lives on the interval [0, 1] it is frequently encountered
as a model of a random probability – or rather a random frequency. In population
genetics for instance, the B-distribution is found as a model for the frequency of
occurrences of one out of two alleles in a population. The shape of the distribution,
i.e. the proper values of λ1 and λ2 , then depends upon issues such as the mutation
rate and the migration rates. ⋄
From a basic calculus course the intimate relation between integration and differen-
tiation should be well known.
Result 2.6.13. If F is a differentiable distribution function the derivative
f (x) = F ′ (x)
is a density for the distribution given by F . That is

Z x
F (x) = F ′ (y)dy.
−∞
0.3
0.2
0.2
0.1
0.1
−8 −4 0 4 8 −4 −2 0 2 4 6 8
Figure 2.11: The density for the logistic distribution (left, see Example 2.6.14) and
the density for the Gumbel distribution (right, see Example 2.6.15). The density
for the Gumbel distribution is clearly skewed, whereas the density for the logistic
distribution is symmetric and quite similar to the density for the normal distribution.
Example 2.6.14 (Logistic distribution). The density for the logistic distribution is
found to be
exp(−x)
f (x) = F ′ (x) = .
(1 + exp(−x))2
Example 2.6.15 (Gumbel distribution). The density for the Gumbel distribution
is found to be
f (x) = F ′ (x) = exp(−x) exp(− exp(−x)).
Exercises
⋆
Exercise 2.6.1. Argue that the function
F (x) = 1 − exp(−xβ ), x≥0
for β > 0 is a distribution function. It is called the Weibull distribution with parameter
β. Find the density on the interval [0, ∞) for the Weibull distribution.
⋆ Exercise 2.6.2. Argue that the function
F (x) = 1 − xβ0 x−β , x ≥ x0 > 0
for β > 0 is a distribution function on [x0 , ∞). It is called the Pareto distribution on
the interval [x0 , ∞). Find the density on the interval [x0 , ∞) for the Pareto distribu-
tion.
Ï Exercise 2.6.3. Write two functions in R, pgumb and dgumb, that computes the
distribution function and the density for the Gumbel distribution.
Ï Exercise 2.6.4. Let
1
fλ (x) = x2 λ+ 12
(1 + 2λ )
for x ∈ R and λ > 0. Argue that fλ (x) > 0 for all x ∈ R. Use numerical integration
in R, integrate, to compute
Z ∞
c(λ) = fλ (x)dx
−∞
√
for λ = 12 , 1, 2, 10, 100. Compare the results with π and 2π. Argue that c(λ)−1 fλ (x)
is a density and compare it, numerically, with the density for the normal distribution.
The probability measure with density c(λ)−1 fλ (x) is called the t-distribution with
shape parameter λ, and it is possible to show that
√ 1
c(λ) = 2λB(λ, )
2
where B is the B-function.
In the summary of univariate real observations we are essentially either summarizing

the data as a density approximation or via quantiles. The quantiles are often more
informative than aiming for the distribution function, say, directly. As encountered
in Section 2.3, we introduce
n
1X
εn (A) = 1A (xi )
n
i=1
to denote the fraction or relative frequency of observations that belong to the event
A ⊆ R.
2.7.1 Histograms and kernel density estimation
The underlying assumption in this section is that we have a data set of observations
x1 , . . . , xn and that these observations are generated by a distribution P that pos-
sesses a density f . A histogram is then an approximation to the density f , thus it

is a function
fˆ : R → [0, ∞).
When plotting this function the convention is, however, to plot rectangular boxes
instead of the step-function that it really is.
Definition 2.7.1. The histogram with break points q1 < q2 < . . . < qk , chosen so
that
q1 < min xi ≤ max xi < qk ,
i=1,...,n i=1,...,n
is the function fˆ given by

1
fˆ(x) = εn ((qi , qi+1 ]) for qi < x ≤ qi+1 . (2.14)
qi+1 − qi
together with fˆ(x) = 0 for x 6∈ (q1 , qn ]. Usually one plots fˆ as a box of height fˆ(qi+1 )
located over the interval (qi , qi+1 ], and this is what most people associate with a
histogram.
The function fˆ is constructed so that

Z ∞ k−1 Z
X qi+1
1
fˆ(x)dx = εn ((qi , qi+1 ]) dx
−∞ qi+1 − qi
i=1 qi
k−1
X
= εn ((qi , qi+1 ])
i=1
= εn ((q1 , qn ]) = 1
where we use the fact that all the data points are contained within the interval
(q1 , qn ]. Since the function fˆ integrates to 1 it is a probability density. The purpose of
the histogram is to approximate the density of the true distribution of X – assuming
that the distribution has a density.
Sometimes one encounters the unnormalized histogram, given by the function
f˜(x) = nεn (qi , qi+1 ] = n(qi+1 − qi )fˆ(x) for qi < x ≤ qi+1 .
Here f˜(x) is constantly equal to the number of observations falling in the interval
(qi , qi+1 ]. Since the function doesn’t integrate to 1 it cannot be compared directly
with a density, nor is it possible to compare unnormalized histograms directly for
two samples of unequal size. Moreover, if the break points are not equidistant, the
unnormalized histogram is of little value.
It can be of visual help to add a rug plot to the histogram – especially for small or
moderate size dataset. A rug plot is a plot of the observations as small “tick marks”
along the first coordinate axis.
Example 2.7.2. We consider the histogram of 100 and 1000 random variables whose
distribution is N (0, 1). The are generated by the computer. We choose the breaks
to be equidistant from −4 to 4 with a distance of 0.5, thus the break points are
−4 −3.5 −3 −2.5 ... 2.5 3 3.5 4.
We find the histograms in Figure 2.12. Note how the histogram corresponding to
the 1000 random variables approximates the density more closely. ⋄
100 observations 1000 observations

0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
−4 −2 0 2 4 −4 −2 0 2 4
Figure 2.12: The histograms for the realization of 100 (left) and 1000 (right) sim-
ulated N (0, 1) random variables. A rug plot is added to both histograms, and we
compare the histograms with the corresponding density for the normal distribution.
Example 2.7.3. Throughout this section we will consider data from a microarray
experiment. It is the so-called ALL dataset (Chiaretti et. al., Blood, vol. 103, No.
7, 2004). It consists of samples from patients suffering from Acute Lymphoblastic
Leukemia. We will consider only those patients with B-cell ALL, and we will group
the patients according to presence or absence of the BCR/ABL fusion gene.
On Figure 2.13 we see the histogram of the log (base 2) expression levels3 for six
(arbitrary) genes for the group of samples without BCR/ABL.
On Figure 2.14 we have singled out the signal from the gene probe set with the
poetic name 1635 at, and we see the histograms for the log expression levels for the
two groups with or without BCR/ABL. The figure also shows examples of kernel
density estimates. ⋄
3
Some further normalization has also been done that we will not go into here.
R Box 2.7.1 (Histograms). A histogram of the data in the numeric vector

x is produced in R by the command
> hist(x)
This plots a histogram using default settings. The break points are by default
chosen by R in a suitable way. It is possible to explicitly set the break points
by hand, for instance
> hist(x,breaks=c(0,1,2,3,4,5))
produces a histogram with break points 0, 1, 2, 3, 4, 5. Note that R gives an

error if the range of the break points does not contain all the data points in
x. Note also that the default behavior of hist is to plot the unnormalized
histogram if the break points are equidistant, which they are using the
default breakpoints. Specifying non-equidistant breakpoints always gives the
normalized histogram. One can make hist produce normalized histograms
by setting freq=FALSE when calling hist. Adding a rug plot is done by
rug(x).
1.2
0.6
1.0
0.5
1.5
0.8
0.4
1.0
0.6
0.3
0.4
0.2
0.5
0.2
0.1
0.0
0.0
0.0
4.5 5.0 5.5 6.0 6.0 6.5 7.0 7.5 8.0 8.5 5.5 6.0 6.5 7.0
1.5
0.6
1.5
0.5
1.0
0.4
1.0
0.3
0.5
0.2
0.5
0.1
0.0
0.0
0.0
7.0 7.2 7.4 7.6 7.8 8.0 8.2 6.0 6.5 7.0 7.5 6.5 7.0 7.5 8.0 8.5 9.0 9.5
Figure 2.13: Examples of histograms for the log (base 2) expression levels for a
random subset of six genes from the ALL dataset (non BCR/ABL fusion gene).
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
Density
Density
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
5 6 7 8 9 10 11 5 6 7 8 9 10 11
x x
0.7
0.7
Default (bw=0.25) Default (bw=0.35)
Bandwidth bw=1 Bandwidth bw=1
0.6
0.6
Bandwidth bw=0.1 Bandwidth bw=0.1
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
5 6 7 8 9 10 11 5 6 7 8 9 10 11
Figure 2.14: Histograms and examples of kernel density estimates (with a Gaussian
kernel) of the log (base 2) expression levels for the gene 1635 at from the ALL
microarray experiment with (right) or without (left) presence of the BCR/ABL
fusion gene.
The histogram is a crude and in some ways unsatisfactory approximation of the den-
sity f . The plot we get is sensitive to the choice of break points – the number as well
as their location. Moreover, the real density is often thought to be a rather smooth
function whereas the histogram by definition is very non-smooth. The use of kernel
density estimation remedies some of these problems. Kernel density estimation or
smoothing is computationally a little harder than computing the histogram, which
for moderate datasets can be done by hand. However, with modern computers, uni-
variate kernel density estimation can be done just as rapidly as drawing a histogram
– that is, apparently instantaneously.
Kernel density estimation is based on the interpretation of the density as given by
(2.13). Rearranging that equation then says that for small h > 0
1
f (x) ≃ P ([x − h, x + h]),
2h
and if we then use P ([x − h, x + h]) ≃ εn ([x − h, x + h]) from the frequency inter-
pretation, we get that for small h > 0
1
f (x) ≃ εn ([x − h, x + h]).
2h
The function
1
fˆ(x) = εn ([x − h, x + h])
2h
is in fact an example of a kernel density estimator using the rectangular kernel . If
we define the kernel Kh : R × R → [0, ∞) by
1
Kh (x, y) = 1 (y),
2h [x−h,x+h]
we can see by the definition of εn ([x − h, x + h]) that
n
1X 1
fˆ(x) = 1 (xi )
n 2h [x−h,x+h]
i=1
n
1X
= Kh (x, xi ).
n
i=1
One may be unsatisfied with the sharp cut-offs at ±h – and how to choose h in
the first place? We may therefore choose smoother kernel functions but with similar
properties as Kh , that is, they are largest for x = y, they fall off, but perhaps more
smoothly than the rectangular kernel, when x moves away from y, and they integrate
to 1 over x for all y.
Definition 2.7.4. A function
K : R × R → [0, ∞)
is called a kernel if Z ∞
K(x, y)dx = 1
−∞
for all y. If K is a kernel we define the kernel density estimate for our dataset
x1 , . . . , xn ∈ R by
n
ˆ 1X
f (x) = K(x, xi ).
n
i=1
Observe that by the definition of a kernel

Z ∞ n Z
1X ∞
fˆ(x)dx = K(x, xi )dx = 1.
−∞ n −∞
i=1
Popular choices of kernels include the Gaussian kernel

1 (x − y)2
Kh (x, y) = √ exp − ,
h 2π 2h2
and the Epanechnikov kernel

3 (x − y)2
Kh (x, y) = 1− 1[x−h,y+h] (y).
h h2
Both of these kernels as well as the rectangular kernel are examples of choosing a
“mother” kernel, k : R → [0, ∞), and then taking

1 x−y
Kh (x, y) = k .
h h
Note that the mother kernel k needs to be a probability density itself. In this case
the optional parameter, h > 0, is called the bandwidth of the kernel. This parameter
determines how quickly the kernel falls off, and plays in general the same role as h
does for the rectangular kernel. Indeed, if we take the mother kernel k(x) = 1[−1,1] (x),
the kernel Kh defined above is precisely the rectangular kernel. Qualitatively we can
say that large values of h gives smooth density estimates and small values gives
density estimates that wiggle up and down.
As a curiosity we may note that the definition of the kernel density estimate fˆ does
not really include the computation of anything! Every computation is carried out at
evaluation time. Each time we evaluate fˆ(x) we carry out the evaluation of K(x, xi ),
i = 1, . . . , n – we cannot do this before we know which x to evaluate at – and then the
summation. So the definition of the kernel density estimate fˆ can mostly be viewed
as a definition of what we intend to do at evaluation time. However, implementations
such as density in R do the evaluation of the kernel density estimate once and for
all at call time in a number of pre-specified points.
2.7.2 Mean and variance
The mean and variance for probability measures on discrete subsets of R were defined
through the point probabilities, and for probability measures on R given in terms of
a density there is an analogous definition of the mean and variance.
Definition 2.7.5. If P is a probability measure on R with density f that fulfills
Z ∞
|x|f (x)dx < ∞
−∞
R Box 2.7.2 (Kernel density estimation). The density function computes

the evaluation of the density estimate for a dataset in a finite number of
points. Default choice of kernel is the Gaussian, and the band width is
computed automatically by some rule of thumb. The default number of
evaluations is 512.
> plot(density(rnorm(100), n = 1024, bw = 1))
produces a plot of the density estimate for the realization of 100 standard
normally distributed random variables using bandwidth 1 and 1024 points
for evaluation.
then we define the mean under P as

Z ∞
µ= xf (x)dx.
−∞
If, moreover,
Z ∞
x2 f (x)dx < ∞
−∞
we define the variance under P as

Z ∞
2
σ = (x − µ)2 f (x)dx.
−∞
Example 2.7.6. Consider the exponential distribution with intensity parameter

λ > 0. The density is
f (x) = λ exp(−λx)
for x ≥ 0 (and f (x) = 0 for x < 0). We find by partial integration that
Z ∞
µ = xλ exp(−λx)dx
0
∞ Z ∞

= x exp(−λx) + exp(−λx)dx
0 0
1 ∞ 1

= − exp(−λx) = .
λ 0 λ
⋄
Example 2.7.7. Consider the uniform distribution on [a, b]. Then the density is
1
f (x) = 1 (x).
b − a [a,b]
We find that the mean is

Z b b
1 1
µ = x dx = x2
a b − a 2(b − a) a
2
1b −a 2 1 (b − a)(b + a) 1
= = = (a + b).
2 b−a 2 b−a 2
We see that µ is the midpoint between a and b. ⋄
Example 2.7.8. The Γ-distribution on [0, ∞) with shape parameter λ > 0 and
scale parameter β > 0 has finite mean
Z ∞
1 λ−1 x
µ = xx exp − dx
β λ Γ(λ) 0 β
Z ∞
1 λ x
= x exp − dx
β λ Γ(λ) 0 β
β λ+1 Γ(λ + 1) λΓ(λ)
= =β = βλ.
β λ Γ(λ) Γ(λ)
A similar computation reveals that the variance is β 2 λ. In particular, the mean and
variance of the σ 2 χ2 -distribution with f degrees of freedom is σ 2 f and 2σ 4 f . ⋄
Example 2.7.9. A density on R is called symmetric if f (x) = f (−x). Since | − x| =

|x| we have that for a symmetric density
Z ∞ Z ∞
|x|f (x)dx = 2 xf (x)dx,
−∞ 0
so the mean is defined if and only if

Z ∞
xf (x)dx < ∞.
0
In that case, using that (−x)f (−x) = −xf (x),

Z ∞ Z ∞ Z 0
µ = xf (x)dx = xf (x)dx + xf (x)dx
−∞ 0 −∞
Z ∞ Z ∞
= xf (x)dx − xf (x)dx = 0.
0 0
The mean of a probability measure on R R ∞given in terms of a symmetric density is

thus 0 – provided that it is defined. If 0 xf (x)dx = ∞ we say that the mean is
undefined. ⋄
Example 2.7.10. The density for the normal distribution is

2
1 x
f (x) = √ exp − ,
2π 2
which is clearly seen to be symmetric, cf. Example 2.7.9. Moreover, with the substi-
tution y = x2 /2 we have that dy = xdx, so
Z ∞ Z ∞
1
xf (x)dx = √ exp(−y)dy
0 2π 0
1
= √ < ∞.
2π
Thus by Example 2.7.9 the normal distribution has mean µ = 0.
Regarding the variance, we use the symmetry argument again and the same substi-
tution to obtain
Z ∞ Z ∞
2 2
σ = x f (x)dx = 2 x2 f (x)dx
−∞ 0
Z ∞p
2 2Γ(3/2)
= √ 2y exp(−y)dy = √ =1
2π 0 π
√
where we use that that Γ(3/2) = π/2, cf. Appendix B. ⋄
As for discrete probability measures the sample mean and sample variance for a
dataset of observations from a distribution P work as estimates of the unknown
theoretical mean and variance under P – provided they exist.
2.7.3 Quantiles
While histograms and other density estimates may be suitable for getting an idea
about the location, spread and shape of a distribution, other descriptive methods
are more suitable for comparisons between two datasets, say, or between a dataset
and a theoretical distribution. Moreover, the use of histograms and density estimates
builds on the assumption that the distribution is actually given by a density. As an
alternative to density estimation we can try to estimate the distribution function
directly.
Definition 2.7.11. The empirical distribution function is the function defined by
F̂n (x) = εn ((−∞, x])
for x ∈ R.
The empirical distribution functions qualifies for the name “distribution function”,
because it is really a distribution function. It is increasing, with limits 0 and 1 when
x → −∞ and x → ∞ respectively, and it is right continuous. But it is not continuous!
It has jumps at each xi for i = 1, . . . , n and is constant in between. Since distribution
functions all have a similar S-shape, empirical distribution functions in themselves
are not really ideal for comparisons either. We will instead develop methods based
on the quantiles, which are more suitable. We define the quantiles for the dataset
first and then subsequently we define quantiles for a theoretical distribution in terms
of its distribution function. Finally we show how the theoretical definition applied
to the empirical distribution function yields the quantiles for the dataset.
R Box 2.7.3 (Empirical distribution functions). If x is a numeric vector in

R containing our data we can construct a ecdf-object (empirical cumulative
distribution function). This requires the stats package:
> library(stats)
Then
> edf <- ecdf(x)
gives the empirical distribution function for the data in x. One can evaluate
this function like any other function:
> edf(1.95)
gives the value of the empirical distribution function evaluated at 1.95. It is

also easy to plot the distribution function:
> plot(edf)
produces a nice plot.
If x1 , . . . , xn ∈ R are n real observations from an experiment, we can order the

observations
x(1) ≤ x(2) ≤ . . . ≤ x(n) ,
thus x(1) denotes the smallest observation, x(n) the largest and x(i) the observation
with i − 1 smaller observations. If q = i/n for i = 1, . . . , n − 1, then x ∈ R is called a
q-quantile (for the dataset) if x(i) ≤ x ≤ x(i+1) . In other words, for a given q = i/n
there is a whole range of q-quantiles, namely the interval [x(i) , x(i+1) ]. Informally we
can say that x is a q-quantile if the fraction of the observations that are ≤ x is q –
except that x(i+1) is taken as a q-quantile also.
If (i − 1)/n < q < i/n for i = 1, . . . , n the proper definition of the q-quantile is x(i) .
This is the only definition that assures monotonicity of quantiles in the sense that
if x is a q-quantile and y is a p-quantile with q < p then x < y.
Some quantiles have special names, e.g. a 0.5-quantile is called a median, and the
upper and lower quartiles are the 0.75- and 0.25-quantiles respectively. Note the
ambiguity here. If n is even then all the x’s in the interval [x(n/2) , x(n/2+1) ] are
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
1.8 1.9 2.0 2.1 2.2 1.9 2.0 2.1 2.2 2.3
Figure 2.15: The empirical distribution function for the log (base 2) expression levels
for the gene 40480 s at from the ALL microarray experiment with (right) or without
(left) presence of the BCR/ABL fusion gene.
medians, whereas if n is odd the median is uniquely defined as x((n+1)/2) . This

ambiguity of e.g. the median and other quantiles can be a little annoying in practice,
and sometimes one prefers to define a single (empirical) quantile function
Q : (0, 1) → R
such that for all q ∈ (0, 1) we have that Q(q) is a q-quantile. Whether one prefers
x(n/2) , x(n/2+1) , or perhaps (x(n/2) + x(n/2+1) )/2 as the median if n is even is largely
a matter of taste.
Quantiles can also be defined for theoretical distributions. We prefer here to consider
the definition of a quantile function only.
Definition 2.7.12. If F : R → [0, 1] is a distribution function for a probability
measure P on R, then Q : (0, 1) → R is a quantile function for P if
F (Q(y) − ε) ≤ y ≤ F (Q(y)) (2.15)
for all y ∈ (0, 1) and all ε > 0.
The definition is essentially an attempt to define an inverse of the distribution func-

tion. However, the distribution function may not have a real inverse but there will
always be a quantile function. This is not at all obvious but we will not pursue the
general construction of a quantile function. From a practical point of view it is a
greater nuisance that the choice of a quantile function is not necessarily unique. As
mentioned above the empirical distribution function is an example where the corre-
sponding choice of quantile function is not unique and in R the function quantile
R Box 2.7.4 (Quantiles). If x is a numeric vector then
> quantile(x)
computes the 0%, 25%, 50%, 75%, and 100% quantiles. That is, the mini-
mum, the lower quartile, the median, the upper quartile, and the maximum.
> quantile(x,probs=c(0.1,0.9))
computes the 0.1 and 0.9 quantile instead, and by setting the type param-
eter to an integer between 1 and 9, one can select how the function handles
the non-uniqueness of the quantiles. If type=1 the quantiles are the gener-
alized inverse of the empirical distribution function, which we will deal with
in Section 2.11. Note that with type being 4 to 9 the result is not a quantile
for the empirical distribution function – though it may still be a reason-
able approximation of the theoretical quantile for the unknown distribution
function.
has in fact 9 different types of quantile computations. Not all of these computations
give quantiles for the empirical distribution function, though, but three of them do.
Definition 2.7.13. If F is a distribution function and Q a quantile function for F

the median, or second quartile, of F is defined as
q2 = median(F ) = Q(0.5).
In addition we call q1 = Q(0.25) and q3 = Q(0.75) the first end third quartiles of F .
The difference
IQR = q3 − q1
is called the interquartile range.
Note that the definitions of the median and the quartiles depend on the choice
of quantile function. If the quantile function is not unique these numbers are not
necessarily uniquely defined. The median summarizes in a single number the location
of the probability measure given by F . The interquartile range summarizes how
spread out around the median the distribution is.
The definition of a quantile function for any distribution function F and the quantiles
defined for a given dataset are closely related. With
F̂n (x) = εn ((−∞, x]) (2.16)
the empirical distribution function for the data then any quantile function for the
distribution function F̂n also gives empirical quantiles as defined for the dataset.
We will as mentioned use quantiles to compare two distributions – that being either
two empirical distributions or an empirical and a theoretical distribution. In princi-
ple, one can also use quantiles to compare two theoretical distributions, but that is
not so interesting – after all, we then know whether they are different or not – but
the quantiles may tell something about the nature of a difference.
Definition 2.7.14. If F1 and F2 are two distribution functions with Q1 and Q2
their corresponding quantile functions a QQ-plot is a plot of Q2 against Q1 .
R Box 2.7.5 (QQ-plots). If x and y are numeric vectors then
> qqplot(x,y)
produces a QQ-plot of the empirical quantiles for y against those for x.
> qqnorm(x)
results in a QQ-plot of the empirical quantiles for x against the quantiles for the
normal distribution.
Normal Q−Q Plot Poisson Q−Q Plot

20
2
1
15
Sample Quantiles
Sample Quantiles
0
10
−1
5
−2
−2 −1 0 1 2 5 10 15
Theoretical Quantiles Theoretical Quantiles
Figure 2.16: A QQ-plot for 100 simulated data points from the standard normal
distribution (left) and a QQ-plot for 100 simulated data points from the Poisson
distribution with parameter λ = 10 (right). In the latter case the sample points are
for visual reasons “jittered”, that is, small random noise is added to visually separate
the sample quantiles.
When making a QQ-plot with one of the distributions, F2 , say, being empirical, it
is common to plot

2i − 1
Q1 , x(i) , i = 1, . . . , n − 1.
2n
That is, we compare the smallest observation x(1) with the 1/2n’th quantile, the
second smallest observation x(2) with the 3/2n’th quantile and so on ending with
the largest observation x(n) and the 1 − 1/2n’th quantile.
If the empirical quantile function Q1 is created from a dataset with n data points all
being realizations from the distribution function F with quantile function Q1 then
the points in the QQ-plot should lie close to a straight line with slope 1 and intercept
0. It can be beneficial to plot a straight line, for instance through suitably chosen
quantiles, to be able to visualize any discrepancies from the straight line.
11
11
10
10
9
9
8
8
7
7
6
6
5
Figure 2.17: Comparing the empirical distributions of the six genes with probe
set names 1635 at,1636 g at,39730 at,40480 s at, 2039 s at, 36643 at for those with
BCR/ABL (right) and those without (left) using boxplots
Another visualization that is quite useful – specially for comparing three or more
empirical distributions – is the box plot, which is also based on quantiles. Histori-
cally it was also useful for visualizing just a single empirical distribution to get a
rough idea about location and scale, but with current computational power there
are more informative plots for single distributions. As a comparative tool for many
distributions, box plots are on the other hand quite effective.
A box plot using quantile function Q is given in terms of a five-dimensional vector
(w1 , q1 , q2 , q3 , w2 )
with w1 ≤ q1 ≤ q2 ≤ q3 ≤ w2 . Here
q1 = Q(0.25), q2 = Q(0.5), q3 = Q(0.75)

R Box 2.7.6 (Box plots). For a numeric vector x we get a single box plot by
> boxplot(x)
If x is a dataframe the command will instead produce (in one figure) a box plot of
each column. By specifying the range parameter (= whisker coefficient), which by
default equals 1.5, we can change the length of the whiskers.
> boxplot(x,range=1)
produces a box plot with whisker coefficient 1.
are the three quartiles and
w1 = min {xi | xi ≥ q1 − c(q3 − q1 )}

w2 = max {xi | xi ≤ q3 + c(q3 − q1 )}
are called the whiskers. The parameter c > 0 is the whisker coefficient. The box plot
is drawn as a vertical box from q1 to q3 with “whiskers” going out to w1 and w2 . If
data points lie outside the whiskers they are often plotted as points.
Exercises
⋆ Exercise 2.7.1. Let P be the probability measure on R with density
f (x) = cx2 1[0,1] (x),
with c a constant that assures that

Z 1
f (x)dx = 1.
0
Argue that f is a density and compute the constant c. Then compute the mean and
the variance under P .
Ï Exercise 2.7.2. Compute the mean and variance for the Gumbel distribution.
Hint: You are welcome to try to compute the integrals – it’s difficult. Alternatively,
you can compute the integrals numerically in R using the integrate function.
Exercise 2.7.3. Find the quantile function for the Gumbel distribution.
Exercise 2.7.4. Find the quantile function for the Weibull distribution, cf. Exercise
2.6.1. Make a QQ-plot of the quantiles from the Weibull distribution against the
quantiles from the Gumbel distribution.
Conditional probabilities and independence 59
2.8 Conditional probabilities and independence
If we know that the event A has occurred, but don’t have additional information
about the outcome of our experiment, we want to assign a conditional probability
to all other events B ⊆ E – conditioning on the event A. For a given event A we
aim at defining a conditional probability measure P ( · |A) such that P (B|A) is the
conditional probability of B given A for any event B ⊆ E.
Definition 2.8.1. The conditional probability measure P (· |A) for an event A ⊆ E

with P (A) > 0 is defined by
P (B ∩ A)
P (B|A) = (2.17)
P (A)
for any event B ⊆ E.
What we claim here is that P ( · |A) really is a probability measure, but we need to
show that. By the definition above we have that
P (E ∩ A) P (A)
P (E|A) = = = 1,
P (A) P (A)
and if B1 , . . . , Bn are disjoint events

P (B1 ∪ . . . ∪ Bn ) ∩ A
P B1 ∪ . . . ∪ Bn A =
P (A)

P (B1 ∩ A) ∪ . . . ∪ (Bn ∩ A)
=
P (A)
P (B1 ∩ A) P (Bn ∩ A)
= + ... +
P (A) P (A)
= P (B1 |A) + . . . P (Bn |A),
where the third equality follows from the additivity property of P . This shows that
P (· |A) is a probability measure and we have chosen to call it the conditional prob-
ability measure given A. It should be understood that this is a definition – though
a completely reasonable and obvious one – and not a derivation of what conditional
probabilities are. The frequency interpretation is in concordance with this definition:
With n repeated experiments, εn (A) is the fraction of outcomes where A occurs and
εn (B ∩ A) is the fraction of outcomes where B ∩ A occurs, hence
εn (B ∩ A)
εn (A)
is the fraction of outcomes where B occurs among those outcomes where A occurs.
When believing in the frequency interpretation this fraction is approximately equal
to P (B|A) for n large.
Associated with conditional probabilities there are two major results known as the
Total Probability Theorem and Bayes Theorem. These results tell us how to compute
some probabilities from knowledge of other (conditional) probabilities. They are easy
to derive from the definition.
If A1 , . . . , An are disjoint events in E and if B ⊆ E is any event then from the
definition of conditional probabilities
P (B|Ai )P (Ai ) = P (B ∩ Ai )
and since the events A1 , . . . , An are disjoint, so are B ∩ A1 , . . . , B ∩ An , hence by

additivity
P (B ∩ A1 ) + . . . + P (B ∩ An ) = P ((B ∩ A1 ) ∪ . . . ∪ (B ∩ An )) = P (B ∩ A).
where A = A1 ∪ . . . ∪ An . This result tells us how to compute P (B ∩ A) from

knowledge of the conditional probabilities P (B|Ai ) and the probabilities P (Ai ) –
this is the Total Probability Theorem.
Result 2.8.2 (Total Probability Theorem). If A1 , . . . , An are disjoint events in E,

if A = A1 ∪ . . . ∪ An , and if B ⊆ E is any event then
P (B ∩ A) = P (B|A1 )P (A1 ) + . . . + P (B|An )P (An ). (2.18)
If A = E and P (B) > 0 we have that
P (Ai ∩ B) P (B|Ai )P (Ai )

P (Ai |B) = = ,
P (B) P (B)
and if we use the Total Probability Theorem to express P (B) we have derived the
Bayes Theorem.
Result 2.8.3 (Bayes Theorem). If A1 , . . . , An are disjoint events in E with E =

A1 ∪ . . . ∪ An and if B ⊆ E is any event with P (B) > 0 then
P (B|Ai )P (Ai )
P (Ai |B) = (2.19)
P (B|A1 )P (A1 ) + . . . + P (B|An )P (An )
for all i = 1, . . . , n.
The Bayes Theorem is of central importance when calculating with conditional prob-
abilities – whether or not we adopt a frequency or a Bayesian interpretation of
probabilities. The example below shows a classical setup where Bayes Theorem is
required.
Example 2.8.4. Drug tests are, for instance, used in cycling to test if cyclists
take illegal drugs. Suppose that we have a test for the illegal drug EPO, with the
property that 99% of the time it reveals (is positive) if a cyclist has taken EPO. This
Conditional probabilities and independence 61
sounds like a good test – or does it? The 0.99 is the conditional probability that the
test will be positive given that the cyclist uses EPO. To completely understand the
merits of the test, we need some additional information about the test and about
the percentage of cyclists that use the drug. To formalize, let E = {tp, fp, tn, fn} be
the sample space where tp = true positive, fp = false positive, tn = true negative,
and fn = false negative. By tp we mean that the cyclist has taken EPO and the test
shows that (is positive), by fp that the cyclist hasn’t taken EPO but the test shows
that anyway (is positive), by fn that the cyclist has taken EPO but the test doesn’t
show that (is negative), and finally by tn we mean that the cyclist hasn’t taken
EPO and that the test shows that (is negative). Furthermore, let A1 = {tp, fn} (the
cyclist uses EPO) and let A2 = {fp, tn} (the cyclist does not use EPO). Finally, let
B = {tp, fp} (the test is positive). Assume that the conditional probability that the
test is positive given that the cyclist is not using EPO is rather low, 0.04 say. Then
what we know is:
P (B|A1 ) = 0.99
P (B|A2 ) = 0.04.
If we have high thoughts about professional cyclists we might think that only a
small fraction, 7%, say, of them use EPO. Choosing cyclists for testing uniformly at
random gives that P (A1 ) = 0.07 and P (A2 ) = 0.93. From Bayes Theorem we find
that
0.99 × 0.07
P (A1 |B) = = 0.65.
0.99 × 0.07 + 0.04 × 0.93
Thus, conditionally on the test being positive, there is only 65% chance that he
actually did take EPO. If a larger fraction, 30%, say, use EPO, we find instead that
0.99 × 0.3
P (A1 |B) = = 0.91,
0.99 × 0.3 + 0.04 × 0.7
which makes us more certain that the positive test actually caught an EPO user.
Besides the fact that “99% probability of revealing an EPO user” is insufficient
information for judging whether the test is good, the point is that Bayes Theorem
pops up in computations like these. Here we are given some information in terms of
conditional probabilities, and we want to know some other conditional probabilities,
which can then be computed from Bayes Theorem. ⋄
Having defined the conditional probabilities P (B|A) for A and B events in E it is

reasonable to say that the event B is independent of A if P (B|A) = P (B) – that is,
B is independent of A if the probability of B doesn’t change even though we know
that A has occurred. This implies that
P (B)P (A) = P (B|A)P (A) = P (B ∩ A)
from which we see that, if P (B) > 0,
P (A ∩ B)
P (A|B) = = P (A).
P (B)
Hence if B is independent of A then A is also independent of B. This reasoning leads

us to the following definition of independence of two events, which doesn’t refer to
conditional probabilities explicitly.
Definition 2.8.5. Two events A, B ⊆ E are said to be independent if
P (A ∩ B) = P (A)P (B).
2.9 Random variables
It will be a considerable advantage to be able to talk about the unrealized outcome

of an experiment as a variable, whose value has not yet been disclosed to us. Be-
fore the experiment is conducted the outcome is not known and it is described by a
probability distribution, but when the experiment is over, a particular value in the
sample space will be the result. The notion of a random variable, sometimes called a
stochastic variable, allows us to talk about the outcome, that will take some partic-
ular value when the experiment is over, before actually conducting the experiment.
We use capital letters like X, Y and Z to denote random variables. In these notes a
random variable is not a precisely defined mathematical object but rather a useful
notational convention. The notion of a random variable is often hard to grasp – how
can a variable be random? The formalities can be equally difficult – what is the
correct mathematical definition of a random variable? The pragmatic approach take
here is that, at the end of the day we should have a useful notation for the variables
we intend to measure or observe and the computations we intend to do on these
measurements.
What we consider is an experiment with sample space E and governing probability
measure P , and we will say that the outcome X is a random variable that takes
values in E and has distribution P . For an event A ⊆ E we will use the notational
convention
P(X ∈ A) = P (A).
It is important to understand that there is always a probability measure associated
with a random variable – this measure being the distribution of the random variable.
Sometimes we cannot tell or know exactly what the distribution is, but we rather
have several potential probability measures in mind as candidates for the distribution
of the random variable. We have more to say about this in the following chapters
on statistics, which deal with figuring out, based on realizations of the experiment,
what the distribution of the random variable(s) was.
Note that sometimes (but not in these notes) a random variable is assumed to take
values in R. We do not make this restriction, and if X takes values in R we will
usually explicitly write that X is a real valued random variable.
Example 2.9.1. A binary experiment with sample space E = {0, 1} is called a
Bernoulli experiment. A random variable X representing the outcome of such a
Random variables 63
binary experiment is called a Bernoulli variable. The probability
p = P(X = 1)
is often called the success probability. ⋄
2.9.1 Transformations of random variables
A transformation is a map from one sample space into another sample space. Ran-
dom variables can then be transformed using the map, and we are interested in the
distribution of resulting random variable. Transformations are the bread-and-butter
for doing statistics – it is crucial to understand how random variables and distribu-
tions on one sample space give rise to a range of transformed random variables and
distributions. Abstractly there is not much to say, but once the reader recognizes
how transformations play a key role throughout these notes, the importance of being
able to handle transformations correctly should become clear.
If E and E ′ are two sample spaces, a transformation is a map
h : E → E′
that assigns the transformed outcome h(x) in E ′ to the outcome x in E. If X is a

random variable, the h-transformed random variable of X, denoted by h(X), is the
random variable whose value is h(x) if X = x. We use the notation
h−1 (A) = {x ∈ E | h(x) ∈ A}
for A ⊆ E ′ to denote the event of outcomes in E for which the transformed outcome
ends up in A.
Example 2.9.2. Transformations are done to data all the time. Any computation
is essentially a transformation. One of the basic ones is computation of the sample
mean. If X1 and X2 are two real valued random variables, their sample mean is
1
Y = (X1 + X2 ).
2
The general treatment of transformations to follow will help us understand how we
derive the distribution of the transformed random variable Y from the distribution
of X1 and X2 . ⋄
Definition 2.9.3. If P is a probability measure on E, the transformed probability

measure, h(P ), on E ′ is given by
h(P )(A) = P (h−1 (A)) = P ({x ∈ E | h(x) ∈ A})
for any event A ⊆ E ′ . If X is a random variable with distribution P , the distribution

of h(X) is h(P ).
We observe from the definition that for A ⊆ E ′
P(h(X) ∈ A) = h(P )(A) = P (h−1 (A)) = P(X ∈ h−1 (A)).
This notation, P(h(X) ∈ A) = P(X ∈ h−1 (A)), is quite suggestive – to find the
distribution of h(X) we “move” h from the variable to the set by taking the “inverse”.
Indeed, if h has an inverse, i.e. there is a function h−1 : E ′ → E such that
h(x) = y ⇔ x = h−1 (y)
for all x ∈ E and y ∈ E ′ , then
h(X) = y ⇔ X = h−1 (y).
Example 2.9.4 (Indicator random variables). Let X be a random variable taking

values in the sample space E and A ⊆ E any event in E. Define the transformation
h : E → {0, 1}
by
1 if x ∈ A
h(x) = 1A (x) = .
0 if x ∈ Ac
Thus h is the indicator function for the event A. The corresponding transformed
random variable
Y = h(X) = 1A (X)
is called an indicator random variable. We sometimes also write
Y = 1(X ∈ A)
to show that Y indicates whether X takes its value in A or not. Since Y takes values
in {0, 1} it is a Bernoulli variable with success probability
p = P(Y = 1) = P(X ∈ A).
⋄
Example 2.9.5. If E is a discrete set and P is a probability measure on E given
by the point probabilities p(x) for x ∈ E then if h : E → E ′ the probability measure
h(P ) has point probabilities
X
q(y) = p(x), y ∈ h(E).
x:h(x)=y
Indeed, for y ∈ h(E) the set h−1 (y) is non-empty and contains precisely those points
whose image is y. Hence
X X
q(y) = h(P )({y}) = P (h−1 (y)) = p(x) = p(x).
x∈h−1 (y) x:h(x)=y
Random variables 65
As a consequence, if h : E → R and if
X
|h(x)|p(x) < ∞
x∈E
we can compute the mean under h(P ) as

X X X X X X
µ= yq(y) = y p(x) = h(x)p(x) = h(x)p(x),
y∈h(E) y∈h(E) x:h(x)=y y∈h(E) x:h(x)=y x∈E
where we have used that the double sum is a sum over every x ∈ E – just organized so
that for each y ∈ h(E) we first sum all values of h(x)p(x) for those x with h(x) = y,
and then sum these contributions over y ∈ h(E).
Likewise, if X
h(x)2 p(x) < ∞
x∈E
we can compute the variance under h(P ) as

X X
σ2 = (h(x) − µ)2 p(x) = h(x)2 p(x) − µ2 .
x∈E x∈E
Example 2.9.6 (Sign and symmetry). Let X be a real valued random variable
whose distribution is given by the distribution function F and consider h(x) = −x.
Then the distribution function for Y = h(X) = −X is
G(x) = P(Y ≤ x) = P(X ≥ −x) = 1 − P(X < −x) = 1 − F (−x) + P(X = −x).
We say that (the distribution of) X is symmetric if G(x) = F (x) for all x ∈ R. That
is, X is symmetric if X and −X have the same distribution. If F has density f we
know that P(X = −x) = 0 and it follows that the distribution of h(X) has density
g(x) = f (−x) by differentiation of G(x) = 1 − F (−x), and in this case it follows
that X is symmetric if f (x) = f (−x) for all x ∈ R. ⋄
Example 2.9.7. Let X be a random variable with values in R and with distribution
given by the distribution function F . Consider the random variable |X| – the absolute
value of X. We find that the distribution function for |X| is
G(x) = P(|X| ≤ x) = P(−x ≤ X ≤ x) = F (x) − F (−x) + P(X = −x).
for x ≥ 0.
If we instead consider X 2 , we find the distribution function to be
√ √ √ √ √
G(x) = P(X 2 ≤ x) = P(− x ≤ X ≤ x) = F ( x) − F (− x) + P(X = − x)
for x ≥ 0. ⋄
Example 2.9.8 (Median absolute deviation). We have previously defined the in-
terquartile range as a measure of the spread of a distribution given in terms of
quantiles. We introduce here an alternative measure. If X is a real valued random
variable whose distribution has distribution function F and median q2 we can con-
sider the transformed random variable
Y = |X − q2 |,
which is the absolute deviation from the median. The distribution function for Y is
Fabsdev (x) = P(Y ≤ x) = P(X ≤ q2 + x) − P(X < q2 − x)

= F (q2 + x) − F (q2 − x) + P(X = q2 − x).
The median absolute deviation is defined as
MAD = median(Fabsdev ).
The median absolute deviation is like the interquartile range a number that repre-
sents how spread out around the median the distribution is.
For a symmetric distribution we always have the median q2 = 0 and therefore we
have Fabsdev = 2F − 1. Using the definition of the median we get that if x is MAD
then
1
2F (x − ε) − 1 ≤ ≤ 2F (x) − 1
2
from which is follows that F (x − ε) ≤ 3/4 ≤ F (x). In this case it follows that MAD
is in fact equal to the upper quartile q3 . Using symmetry again one can observe that
the lower quartile q1 = −q3 and hence for a symmetric distribution MAD equals half
the interquartile range. ⋄
Example 2.9.9 (Location and Scale). Let X denote a real valued random variable
with distribution given by the distribution function F : R → [0, 1]. Consider the
transformation h : R → R given by
h(x) = σx + µ
for some constants µ ∈ R and σ > 0. Then the distribution of
Y = h(X) = σX + µ
has distribution function

x−µ x−µ
G(x) = P(h(X) ≤ x) = P X ≤ =F .
σ σ
If F is differentiable we know that the distribution of X has density
f (x) = F ′ (x).
Random variables 67
We observe that G is also differentiable and applying the chain rule for differentiation
we find that the distribution of Y has density

′ 1 x−µ
g(x) = G (x) = f .
σ σ
The random variable Y is a translated and scaled version of X. The parameter µ is

known as the location parameter and σ as the scale parameter. We observe that from
a single distribution given by the distribution function F we obtain a two-parameter
family of distributions from F by translation and scaling.
If the distribution of X has density f , mean 0 and variance 1 we can compute the
mean and variance of the distribution of Y . From Definition 2.7.5 it follows, using
the substitution y = x−µ
σ , that the mean is
Z ∞ Z ∞
1 x−µ
x f dx = (y + µ)f (y)dy
−∞ σ σ
Z−∞
∞ Z ∞
= yf (y)dy + µ f (y)dy = µ.
−∞ −∞
Likewise, using the same substitution, the variance is

Z ∞ Z ∞ 2
21 x−µ 2 x−µ x−µ 1
(x − µ) f dx = σ f dx
−∞ σ σ σ σ σ
Z−∞
∞
= σ2 y 2 f (y)dy = σ 2 .
−∞
It follows that the normal distribution with location parameter µ and scale parameter
σ has density

1 (x − µ)2 1 (x − µ)2
√ exp − =√ exp − . (2.20)
σ 2π 2σ 2 2πσ 2 2σ 2
The abbreviation
X ∼ N (µ, σ 2 ).
is often used to denote a random variable X, which is normally distributed with lo-
cation parameter µ and scale parameter σ. In the light of Example 2.7.10 the normal
distribution N (0, 1) has mean 0 and variance 1, thus the N (µ, σ 2 ) normal distribu-
tion with location parameter µ and scale parameter σ has mean µ and variance σ 2 .
⋄
Example 2.9.10 (QQ-plots). When we want to compare an empirical distribution

to a theoretical distribution we are most often interested in a comparison where we
just know the shape of the distribution but not the location and scale. If X has
distribution with distribution function F , quantile function Q and our dataset is
R Box 2.9.1. For some of the standard distributions on the real line R, one can
easily supply additional parameters specifying the location and scale in R. For the
normal distribution pnorm(x,1,2) equals the density at x with location parameter
µ = 1 and scale parameter σ = 2. Similarly, plogis(x,1,2) gives the density for
the logistic distribution at x with location parameter µ = 1 and scale parameter
σ = 2.
Log normal QQplot Log normal QQplot

9
9.0
8
8.5
Sample Quantiles
Sample Quantiles
7
8.0
6
7.5
5
7.0
−2 −1 0 1 2 −2 −1 0 1 2
Normal QQplot Normal QQplot

500
600
400
500
300
Sample Quantiles
Sample Quantiles
400
200
300
100
200
100
−2 −1 0 1 2 −2 −1 0 1 2
Figure 2.18: QQplots for gene 1635 at from the ALL dataset. Here we see the expres-
sion levels and log (base 2) expression levels against the normal distribution with
(right) or without (left) presence of the BCR/ABL fusion gene.
x1 , . . . , xn each being realizations of a random variable with the same distribution

as
µ + σX
for some unknown scale σ > 0 and location µ ∈ R. If we make a QQ-plot of the
empirical quantile function against Q it will still result in points that are close to a
Random variables 69
straight line, but with slope σ and intercept µ. This is because Qµ,σ (y) defined by
Qµ,σ (y) = µ + σQ(y)
is actually a quantile function for the distribution of µ + σX. To see this, recall that
the distribution function for µ + σX is

x−µ
Fµ,σ (x) = F ,
σ
hence for any y ∈ (0, 1) and ε > 0

σQ(y) + µ − µ − ε
Fµ,σ (Qµ,σ (y) − ε) = F = F (Q(y) − ε/σ)
σ
≤ y
≤ F (Q(y)) = Fµ,σ (Qµ,σ (y)).
This adds considerable value to the QQ-plot as it can now be used to justify the
choice of the shape of the distribution without having to discuss how the unknown
scale and location parameters are chosen or perhaps estimated from data. ⋄
Exercises
Exercise 2.9.1. Compute the interquartile range and the median absolute deviation
for the N (0, 1) standard normal distribution.
Exercise 2.9.2. Find the density for the distribution of exp(X) if X is normally
distributed. This is the log-normal distribution. Make a QQ-plot of the normal distri-
bution against the log-normal distribution.
⋆ Exercise 2.9.3. Assume that X is a real valued random variable, whose distribution
has density f and distribution function F . Let h : R → [0, ∞) be the transformation
h(x) = x2 . Show that
√ √
P(h(X) ≤ y) = 1 − F (− y) + F ( y),
and then show that the distribution of h(X) has density

√ √
f (− y) + f ( y)
g(y) = √
2 y
for y > 0. Argue that if the distribution of X is symmetric then this expression reduces
to √
f ( y)
g(x) = √ .
y
⋆ Exercise 2.9.4. Assume that the distribution of X is the normal distribution. Show
that the distribution of X 2 has density
1 y
g(y) = √ exp −
2πy 2
for y > 0.
2.10 Joint distributions, conditional distributions and

independence
For all practical purposes we have more than one random variable in the game when
we do statistics. A single random variable carries a probability distribution on a
sample space, but if we hope to infer unknown aspects of this distribution, it is
rarely sufficient to have just one observation. Indeed, in practice we have a data set
consisting of several observations as in all the examples previously considered. The
question, we need to ask, is which assumptions we want to make on the interrelation
between the random variables representing the entire data set?
If we have n random variables, X1 , X2 , . . . , Xn , they each have a distribution, which
we refer to as their marginal distributions. Are the marginal distributions enough?
No, not at all! Lets consider the following example.
Example 2.10.1. Let X and Y are two random variables representing a particular
DNA letter in the genome in two different but related organisms. Lets assume for
simplicity that each letter has the uniform distribution as the marginal distribution.
What is the probability of observing (X = A, Y = A)? If the events (X = A) and
(Y = A) were independent, we know that the correct answer is 0.25 × 0.25 = 0.125.
However, we claimed that the organims are related, and the interpretation is that the
occurrence of a given letter in one of the organims, at this particular position, gives
us information about the letter in the other organism. What the correct probability
is depends on many details, e.g. the evolutionary distance between the organims. In
Subsection 2.10.2 we consider two examples of such models, the Jukes-Cantor model
and the Kimura model. ⋄
The probability measure the governs the combined behaviour of the pair (X, Y )
above is often referred to as the joint distribution of the variables. We need to know
the joint distribution to compute probabilities involving more than one the random
variable. The marginal probability distributions, given in general as
Pi (A) = P(Xi ∈ A),
are sufficient if we only want to compute probabilities of events involving single

random variables only.
Joint distributions can be arbitrarily complicated, but we defer the general treatment
to Section 2.13 and focus here on the concept of independence.
2.10.1 Random variables and independence
Whenever we have observations of variables we believe are not interrelated, we will

use the independence assumption, which is formally a specification of the joint dis-
tribution of the variables from the knowledge of their marginal distributions only.
Joint distributions, conditional distributions and independence 71
Definition 2.10.2 (Independence). Let X1 , X2 , . . . , Xn be n random variables such

that Xi takes values in a sample space Ei for i = 1, . . . , n. We say that X1 , . . . , Xn
are independent if
P(X1 ∈ A1 , . . . , Xn ∈ An ) = P(X1 ∈ A1 ) · . . . · P(Xn ∈ An )
for all events A1 ⊆ E1 , . . . , An ⊆ En . Moreover, for given marginal distributions

of the Xi ’s, the right hand side above specifies a unique probability distribution that
defines a joint distribution of the Xi ’s.
With this definition we can specify the distribution of X1 , . . . , Xn by specifying only

the marginal distributions and say the magic words; the Xi ’s are independent. If
the variables all represent the same experiment, hence they all take values in the
same sample space E, we can in addition assume that the marginal distributions are
identical. In that case we say that the random variables X1 , . . . , Xn are independent
and identically distributed, which is typically abbreviated iid.
If we consider n real valued random variables X1 , X2 , . . . , Xn such that the marginal
density for the distribution of Xi is fi , then if the joint density, f , factorizes as
f (x1 , . . . , xn ) = f1 (x1 ) · . . . · fn (xn )
we get that
Z Z
P(X1 ∈ A1 , . . . , Xn ∈ An ) = ··· f1 (x1 ) · . . . · fn (xn )dxn . . . dx1
A1 An
Z Z
= f1 (x1 )dx1 · . . . · fn (xn )dxn
A1 An
= P(X1 ∈ A1 ) · . . . · P(Xn ∈ An ).
A similar computation holds with point probabilities, which replaces the densities
on a discrete sample space, and where the integrals are replaced with sums. We
summarize this as follows.
Result 2.10.3. Let X1 , X2 , . . . , Xn be n random variables. If the sample spaces Ei
are discrete, if the marginal distributions have point probabilities pi (x), x ∈ Ei and
i = 1, . . . , n, and the point probabilities for the distribution of (X1 , . . . , Xn ) factorize
as
P (X1 = x1 , . . . , Xn = xn ) = p(x1 , . . . , xn ) = p1 (x1 ) · . . . · pn (xn )
then the Xi ’s are independent.
If Ei = R, if the marginal distributions have densities fi : R → [0, ∞), i = 1, . . . , n,
then if the (n-dimensional) density for the distribution of (X1 , . . . , Xn ) factorizes as
f (x1 , . . . , xn ) = f1 (x1 ) · . . . · fn (xn )
the Xi ’s are independent.

The result above is mostly used as follows. If we want to construct a joint distribution
of X1 , . . . , Xn and we want the marginal distribution of Xi to be N (µ, σ 2 ), say, for
i = 1, . . . , n and we want X1 , . . . , Xn to be independent, the theorem above says that
this is actually what we obtain by taking the joint distribution to have the density
that is the product of the marginal densities. Using the properties of the exponential
function we see that the density for the joint distribution of n iid N (µ, σ 2 ) distributed
random variables is then
n
!
1 X (xi − µ)2
f (x1 , . . . , xn ) = √ exp − .
σ n ( 2π)n 2σ 2
i=1
Example 2.10.4. One example of a computation where we need the joint distri-
bution is the following. If X1 and X2 are two real valued random variables, we are
interested in computing the distribution function for their sum, Y = X1 + X2 . If the
joint density is f this computation is
Z Z
F (y) = P(Y ≤ y) = P(X1 + X2 ≤ y) = f (x1 , x2 )dx1 dx2 .
{(x1 ,x2 )|x1 +x2 ≤y}
If X1 and X2 are independent we have that f (x1 , x2 ) = f1 (x1 )f2 (x2 ) and the double
integral can we written as
Z ∞ Z y−x2
F (y) = f1 (x1 )dx1 f2 (x2 )dx2 .
−∞ −∞
Whether we can compute this double integral in practice is another story, which
depends on the densities f1 and f2 . ⋄
Example 2.10.5. We consider the random variables X and Y that take values in
E0 × E0 with E0 = {A, C, G, T} and with point probabilities given by the matrix
Y
A C G T
A 0.0401 0.0537 0.0512 0.0400 0.1850
C 0.0654 0.0874 0.0833 0.0651 0.3012
X
G 0.0634 0.0848 0.0809 0.0632 0.2923
T 0.0481 0.0643 0.0613 0.0479 0.2215
0.217 0.2901 0.2766 0.2163
The marginal distribution of X and Y respectively are also given as the row sums
(right column) and the column sums (bottom row). It is in fact this matrix rep-
resentation of discrete, multivariate distributions with “the marginal distributions
in the margin” that caused the name “marginal distributions”. One verifies easily –
or by using R – that all entries in this matrix are products of their marginals. For
instance,
P (X = G, Y = C) = 0.0848 = 0.2923 × 0.2901.
⋄
R Box 2.10.1 (Outer products). To compute all 16 products of pairs of

entries in the vectors rmp <- c(0.185, 0.3012, 0.2923, 0.2215) and cmp
<- c(0.217,0.2901,0.2766,0.2163) is perhaps a little tedious. A convenient
operation in R is the outer product, which does exactly this:
> rmp %o% cmp

[,1] [,2] [,3] [,4]
[1,] 0.0401 0.0537 0.0512 0.0400
[2,] 0.0654 0.0874 0.0833 0.0651
[3,] 0.0634 0.0848 0.0809 0.0632
[4,] 0.0481 0.0643 0.0613 0.0479
Alternatively, rmp %*% t(cmp) or outer(rmp,cmp,"*") does the same job in this
case. The former works correctly for vectors only. The binary operator %o% is a
wrapper for the latter, which works for any function f of two variables as well. If
we want to compute f(rmp[i],cmp[j]), say, for all combinations of i and j, this
is done by outer(rmp,cmp,f). Note that when we use outer with the arithmetic
operators like multiplication, *, we need the quotation marks.
Example 2.10.6 (Hardy-Weinberg equilibrium). When diploid organisms like hu-

mans reproduce, the offspring receives one copy of each chromosome from each of
the parents. Thus if the father carries the aa allele combination of a particular gene
and the mother carries the Aa allele combination, then the offspring can get both of
the allele combinations aa and Aa. The Hardy-Weinberg equilibrium is an assump-
tion about independence between the alleles that are passed on from the mother and
from the father.
In a large population we sample uniformly a random individual and let Xm and
Xf denote the random variables representing the allele that was passed on to the
individual from the mother and the father, respectively. We cannot observe Xm and
Xf , as we cannot from the individual determine which of the observed alleles is from
the mother, and which is from the father.
If we assume that the male and female subpopulations have the same proportion
p ∈ [0, 1] of allele a, the marginal distribution of Xm and Xf is given by
P(Xm = a) = P(Xf = a) = p
and
P(Xm = A) = P(Xf = A) = 1 − p.
If Xf and Xm are independent the offspring allele combination, Y , which we can

observe, has the distribution given by
P(Y = aa) = P(Xf = a, Xm = a) = P(Xf = a)P(Xm = a) = p2
P(Y = Aa) = P(Xf = A, Xm = a) + P(Xf = a, Xm = A) = 2p(1 − p)
P(Y = AA) = P(Xf = A, Xm = A) = (1 − p)2 .
This distribution of the allele combination given in terms of the proportions of the
two alleles in the population is the Hardy-Weinberg equilibrium. In reality this is
not really a question of whether the population is in any sort of equilibrium. It is a
distributional consequence of an independence assumption. ⋄
Example 2.10.7 (Linkage). The alleles of two genes, or markers for that matter,
that are located on the same chromosome may occur in an associated way. For
example, consider a gene that occurs as allele a or A and another gene that occurs
as allele b and B, and we let X1 and X2 be random variables that represent the
allele we find on (one of) the chromosomes in a random individual. The marginal
distribution of X1 and X2 are given by
P(X1 = a) = p and P(X2 = b) = q
where p, q ∈ [0, 1] are the proportions of alleles a and b, respectively, in the popu-
lation. If X1 and X2 are independent, we have that the distribution of (X1 , X2 ) is
given by
P(X1 = a, X2 = b) = pq
P(X1 = a, X2 = B) = p(1 − q)
P(X1 = A, X2 = b) = (1 − p)q
P(X1 = A, X2 = B) = (1 − p)(1 − q).
If the distribution of (X1 , X2 ) deviates from this we have linkage. This is another
way of saying that we have dependence. ⋄
When can we assume independence? This is a very important question, and it is

not easy to answer. In some cases there are good arguments that can justify the
independence assumption. In the example above on the Hardy-Weinberg equilibrium
the independence assumption is often refered to as random mating. This may or may
not be questionable, and we have to think about what possible alternatives there
are. A model of non-random mating will require some more thoughts, and how will
it affect the allele frequencies? Instead of arguing for or against, we could also accept
the random mating assumption, and simply try to check if the resulting allele model
fits the data. After all, the derived allele frequencies form the important part of
the model. Checking the model in this case amounts to fitting the more general
model where the three allele combinations have an arbitrary distribution, and then
we compare the more general model with the model obtained under the Hardy-
Weinberg equilibrium assumption.
In general terms, whenever a model assumption is made it should be justifiable or
at least not contrary to actual belief. This holds for the independence assumption
as well. When this is said there are many situations where the data at hand are
analyzed under an independence assumption that is not valid. The reasons can be
many – from ignorance or lack of skills over lack of better alternatives to deliberate
choices for efficiency reasons, say. In general, to use a model that does not fit the
data actually analyzed can potentially lead to biased or downright wrong results. If
one is in doubt about such things, the best advice is to seek assistance from a more
experienced person. There are no magic words or spells that can make problems
with lack of model fit go away.
2.10.2 Random variables and conditional distributions
If X is a random variable with distribution P , taking values in the sample space

E, and if A ⊆ E, then the conditional distribution of X given that X ∈ A is the
probability measure P ( · | A). We write
P(X ∈ B|X ∈ A) = P (B|A)
for B ⊆ E.
For two random variables we make the following definition.
Definition 2.10.8. If X and Y are two random variables taking values in the sample
spaces E1 and E2 respectively and if A ⊆ E1 , then the conditional distribution of Y
given that X ∈ A is defined as
P(Y ∈ B, X ∈ A)
P(Y ∈ B|X ∈ A) =
P(X ∈ A)
for B ⊆ E2 provided that P(X ∈ A) > 0.
Note that if P(X ∈ A) = 0, the conditional distribution of Y given that X ∈ A is

not defined. Note also that
P(Y ∈ B, X ∈ A) = P(Y ∈ B|X ∈ A)P(X ∈ A), (2.21)
and that this equality holds no matter how we define P(Y ∈ B|X ∈ A) in cases
where P(X ∈ A) = 0 (if P(X ∈ A) = 0 the equality reads that 0 = 0 irrespectively
of the definition of the conditional probability).
Considering discrete sample spaces E1 and E2 the conditional distribution of Y
given X = x can be given in terms of point probabilities p(y|x), y ∈ E2 . If the joint
distribution of X and Y has point probabilities p(x, y) for x ∈ E1 and y ∈ E2 , then
by definition
P(Y = y, X = x) p(x, y)
p(y|x) = P(Y = y|X = x) = =P (2.22)
P(X = x) y∈E2 p(x, y)
where the third equality follows from the fact that the marginal distribution of X
has point probabilities X
p1 (x) = p(x, y).
y∈E2
Math Box 2.10.1 (More about conditional distributions). If P denotes the joint
distribution of (X, Y ) on E1 × E2 , then the conditional probability measure given
the event A × E2 for A ⊆ E1 takes the form
P (B1 × B2 ∩ A × E2 ) P ((B1 ∩ A) × B2 ))
P (B1 × B2 |A × E2 ) = =
P (A × E2 ) P1 (A)
for B1 ⊆ E1 and B2 ⊆ E2 . Here we use P1 (A) to denote the marginal distribution

of X. This conditional probability measure is the conditional distribution of the
bundled variable (X, Y ) given that X ∈ A. The conditional distribution of Y
given that X ∈ A, as defined above, is recognized as the second marginal of the
conditional distribution of (X, Y ) given that X ∈ A. By this we mean that if
P (·|E1 × A) denotes the distribution of (X, Y ) conditionally on X ∈ A, which is a
probability measure on E1 × E2 , then the second marginal of this measure, which
is a probability measure on E2 , coincides with the conditional distribution of Y
given X ∈ A.
With A = {x} and B = {y} the formula given by (2.21) reads
p(x, y) = p(y|x)p1 (x). (2.23)
Example 2.10.9. We consider the random variables X and Y as in Example 2.13.3,

thus the point probabilities are given by the matrix
Y
A C G T
A 0.1272 0.0063 0.0464 0.0051 0.1850
C 0.0196 0.2008 0.0082 0.0726 0.3012
X
G 0.0556 0.0145 0.2151 0.0071 0.2923
T 0.0146 0.0685 0.0069 0.1315 0.2215
0.2170 0.2901 0.2766 0.2163
Here the additional right column and the bottom row shows the marginal distribu-
tions of X and Y respectively. Note that these marginals are the same as considered
in Example 2.10.5, but that the point probabilities for the joint distribution of X
and Y certainly differ from the product of the marginals. Thus X and Y are not
independent. We can compute the conditional distribution of Y given X as having
point probabilities
P(X = x, Y = y) p(x, y)
P(Y = y|X = x) = =P .
P(X = x) y∈E p(x, y)
Note that we have to divide by precisely the row sums above. The resulting matrix
of conditional distributions is
Y
A C G T
A 0.6874 0.0343 0.2507 0.0276
X C 0.0649 0.6667 0.0273 0.2411
G 0.1904 0.0495 0.7359 0.0242
T 0.0658 0.3093 0.0311 0.5938
The rows in this matrix are conditional point probabilities for the distribution of
Y conditionally on X being equal to the letter on the left hand side of the row.
Each row sums by definition to 1. Such a matrix of conditional probabilities is called
a transition probability matrix. In terms of mutations (substitutions) in a DNA-
sequence the interpretation is that fixing one nucleic acid (the X) in the first sequence
we can read of from this matrix the probability of finding any of the four nucleic
acids at the corresponding position in a second evolutionarily related sequence. Note
that the nomenclature in molecular evolution traditionally is that a transition is a
change from a pyrimidine to a pyrimidine or a purine to a purine, whereas a change
from a pyrimidine to a purine or vice versa is called a transversion. ⋄
In probability theory the conditional probabilities are computed from the definition
and formulas above. In statistics, however, conditional probabilities are often used as
the fundamental building blocks when specifying a probability model. We consider
two concrete examples from molecular evolution and then the general concept of
structural equations.
Example 2.10.10 (The Jukes-Cantor model). It is possible to systematically spec-

ify models of molecular evolution where there is a time parameter t ≥ 0 that dictates
how dependent two nucleotides are. The simplest such model is the Jukes-Cantor
model. There is one parameter α > 0 in addition to the time parameter that tells
how many mutations occur per time unit. The conditional probabilities on the DNA
alphabet are
P t (x, x) = 0.25 + 0.75 × exp(−4αt)

P t (x, y) = 0.25 − 0.25 × exp(−4αt), if x 6= y,
where P t (x, y) is the conditional probability P(Y = y|X = x) when X and Y are
separated by the evolutionary time t. ⋄
Example 2.10.11 (The Kimura model). Another slightly more complicated model
of molecular evolution is the Kimura model. The model captures the observed fact
that a substitution of a purine with a purine or a pyrimidine with a pyrimidine
(a transition) is happening in the course of evolution with a different rate than a
substitution of a purine with pyrimidine or pyrimidine with purine (a transversion).
If we let α > 0 be a parameter determining the rate of transitions and β > 0 a
0.35
0.35
Transition Transition
Transversion Transversion
0.30
0.30
0.25
0.25
Conditional Probability
Conditional Probability
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
α = 5 × 10−3, β = 10−3 α = 10−3, β = 5 × 10−3
0 200 400 600 800 1000 0 200 400 600 800 1000
t t
Figure 2.19: Two examples of transition and transversion probabilities for the
Kimura model as a function of evolutionary distance in units of time.
parameter determining the rate of a transversion the Kimura model is given by

P t (x, x) = 0.25 + 0.25 exp(−4βt) + 0.5 exp(−2(α + β)t)
P t (x, y) = 0.25 + 0.25 exp(−4βt) − 0.5 exp(−2(α + β)t), for a transversion
t
P (x, y) = 0.25 − 0.25 exp(−4βt), for a transition
with P t (x, y) as above is the conditional probability P(Y = y|X = x) when X and
Y are separated by the evolutionary time t. ⋄
The two models, the Jukes-Cantor model and the Kimura model, may seem a little
arbitrary when encountered as given above. They are not, but they fit into a more
general and systematic model construction that is deferred to Section 2.15. There it
is shown that the models arise as solutions to a system of differential equations with
specific interpretations.
Both of the previous examples gave the conditional distributions directly in terms
of a formula for the conditional point probabilities. The following examples use a
different strategy. Here the conditional probabilities are given by a transformation.
Example 2.10.12. If X and ε are real valued random variables, a structural equa-
tion defining Y is an equation of the form
Y = h(X, ε)
where h is any function taking values in a sample space E. The interpretation of

the structural equation is as a conditional probability distribution of Y given X.
That is, the conditional probability distribution of Y given X = x is h(x, ε) – a
transformation of the distribution of ε by h(x, ·) where x is fixed.
Structural equations are very useful for model specifications and are used quite
extensively but often more or less implicitly in the literature. One thing to keep
in mind is that the combination of the choice of h and the distribution of ε is
not uniquely deducible from the conditional distributions they specify. Hence the
structural equation itself is deducible from data alone. ⋄
From a statistical point of view there are at least two practical interpretations of
the structural equation model. One is that the values of X are fixed by us – we
design the experiment and choose the values. We could, for instance, administer a
(toxic) compound at different doses to a bunch of flies and observe what happens,
cf. Example 1.2.5. The variable ε captures the uncertainty in the experiment – at
a given dose only a fraction of the flies will die. Another possibility is to observe
(X, Y ) jointly where, again, Y may be the death or survival of an insect but X is
the concentration of a given compound as measured from the dirt sample where the
insect is collected. Thus we do not decide the concentration levels. In the latter case
the structural equation model may still provide a useful model of the observed, con-
ditional distribution. However, we should be careful with the interpretation. If we
intervene and force X to take a particular value, it may not have the expected effect,
as predicted by the structural equation model, on the outcome Y . A explanation,
why this is so, is that X and ε may be dependent. Thus if we want to interpret the
structural equation correctly when we make interventions, the X and ε variables
should be independent. One consequence of this is, that when we design an exper-
iment it is essential to break any relation between the dose level X and the noise
variable ε. The usual way to achieve this is to randomize, that is, to assign the dose
levels randomly to the flies.
The discussion about interpretations of structural equations is only a scratch in
the surface on the whole discussion of causal inference and causal conclusions from
statistical data analysis. We also touched upon this issue in Example 1.2.5. Briefly,
probability distributions are descriptive tools that are able to capture association,
but by themselves they provide no explanations about the causal nature of things.
Additional assumptions or controlled experimental designs are necessary to facilitate
causal conclusions.
We treat two concrete examples of structural equation models, the probit regression
model and the linear regression model.
Example 2.10.13. If Φ denotes the distribution function for the normal distribution
and ε is uniformly distributed on [0, 1] we define
Y = 1(ε ≤ Φ(α + βX)).

Thus the sample space for Y is {0, 1} and h(x, ε) = 1(ε ≤ Φ(α+ βx)) is the indicator
function that ε is smaller than or equal Φ(α + βx). We find that
P(Y = 1) = P(ε ≤ Φ(α + βx)) = Φ(α + βx).
The resulting model of the conditional distribution of the Bernoulli variable Y given
X = x is known as the probit regression model. ⋄
Example 2.10.14. If ε ∼ N (0, σ 2 ) and
Y = α + βX + ε
we have a structural equation model with
h(x, ε) = α + βx + ε.
This model is the linear regression model, and is perhaps the single most impor-
tant statistical model. For fixed x this is a location transformation of the normal
distribution, thus the model specifies that
Y | X = x ∼ N (α + βx, σ 2 ).
In words, the structural equation says that there is a linear relationship between
the Y variable and the X variable plus some additional “noise” as given by ε. One
important observation is that there is an embedded asymmetry in the model. Even
though we could “solve” the equation above in terms of X and write
α 1 1
X=− + Y − ε
β β β
this formula does not qualify for being a structural equation too. The explanation
is that if X and ε are independent then Y and ε are not! ⋄
To illustrate the point with causality and the asymmetry of structural equations lets
elaborate a little on the saying that “mud does not cause rain”. Assume that the
measurable level of mud is Y and that the measurable amount of rain is X and that
the model
Y = α + βX + ε
is a good model of how the mud level increases (β > 0) when it rains. Thus if we
perform the rain dance and it starts raining, then the mud level increases accordingly,
and by making it rain we have caused more mud. However, we cannot turn this
around. It won’t start raining just because we add more mud. The point is that
the correct interpretation of a structural equation is tied closely together with the
quantities that we model, and the equality sign in a structural equation should
be interpreted as an assignment from right to left. Arguably, the more suggestive
notation
Y ← α + βX + ε
could be used – similarly to the assignment operator used in R.
2.10.3 Transformations of independent variables
The first thing to observe is a somewhat “obvious” fact. If h1 : E1 → E1′ and

h2 : E2 → E2′ are two transformations, if X1 and X2 are two independent random
variables taking values in E1 and E2 , respectively, and if A1 ⊆ E1′ and A2 ⊆ E2′ are
two events
P(h1 (X1 ) ∈ A1 , h2 (X2 ) ∈ A2 ) = P(X1 ∈ h−1 (A1 ), X2 ∈ h−1 (A2 ))

= P(X1 ∈ h−1 (A1 ))P(X2 ∈ h−1 (A2 ))
= P(h1 (X1 ) ∈ A1 )P (h2 (X2 ) ∈ A2 ).
Thus we have the following result.
Result 2.10.15. If X1 and X2 are independent random variables taking values in E1

and E2 respectively, and if h1 : E1 → E1′ and h2 : E2 → E2′ are two transformations
then the random variables h1 (X1 ) and h2 (X2 ) are independent.
The result should be intuitive and good to have in mind. In words it says that
marginal transformations of independent random variables result in independent
random variables. Intuitive as this may sound it was worth a derivation, since the
definition of independence is a purely mathematical one. The result is therefore just
as much a reassurance that our concept of independence is not counter intuitive in the
sense that we cannot introduce dependence by marginally transforming independent
variables.
Other typical transformations that we will deal with are summation and taking the
maximum or taking the minimum of independent random variables.
We start with the summation and consider the transformation h : Z × Z → Z given
by
h(x1 , x2 ) = x1 + x2 .
We can use Result 2.8.2 to obtain that
X
p(y) = P(Y = y) = P(X1 + X2 = y) = P(X1 + X2 = y, X2 = x)
x∈Z
Due to independence of X1 and X2 we find that
P(X1 + X2 = y, X2 = x) = P(X1 = y − x, X2 = x) = p1 (y − x)p2 (x),
hence X
p(y) = p1 (y − x)p2 (x). (2.24)
x∈Z
Result 2.10.16. Let X1 and X2 be two independent random variables each taking
values in Z and with (p1 (x))x∈Z and (p2 (x))x∈Z denoting the point probabilities for
the distribution of X1 and X2 respectively. Then the distribution of the random

variable Y = X1 + X2 , which also takes values in Z, has point probabilities
X
p(y) = p1 (y − x)p2 (x). (2.25)
x∈Z
Remark 2.10.17. Finding the distribution of the sum of three or more random
variables can then be done iteratively. That is, if we want to find the distribution of
X1 + X2 + X3 where X1 , X2 and X3 are independent then we rewrite
X1 + X2 + X3 = (X1 + X2 ) + X3
and first find the distribution (i.e. the point probabilities) of Y = X1 + X2 using
Result 2.10.16. Then the distribution of Y + X3 is found again using Result 2.10.16
(and independence of Y and X3 , cf. Result 2.10.15). Note that it doesn’t matter how
we place the parentheses.
Example 2.10.18 (Sums of Poisson Variables). Let X1 and X2 be independent
Poisson distributed with parameter λ1 and λ2 respectively. The point probabilities
for the distribution of X1 + X2 are then given by
y
X λy−x λx
p(y) = exp(−λ1 ) 1
exp(−λ2 ) 2
(y − x)! x!
x=0
y
X λy−x
1 λx2
= exp(−(λ1 + λ2 ))
(y − x)!x!
x=0
y
1 X y!
= exp(−(λ1 + λ2 )) λx λy−x
y! x=0 x!(y − x)! 2 1
Using the binomial formula for the last sum we obtain that
(λ1 + λ2 )y
p(y) = exp(−(λ1 + λ2 )) .
y!
This shows that the distribution of X1 + X2 is a Poisson distribution with parameter
λ1 + λ2 . ⋄
A derivation similar to the derivation for variables taking integer values is possible
for real valued random variables whose distributions are given by densities f1 and
f2 , respectively. The differences being that the sum is replaced by an integral and
the point probabilities by densities.
Result 2.10.19. If X1 and X2 are independent real valued random variables with
distributions having density f1 and f2 , respectively, the density, g, for the distribution
of Y = X1 + X2 is given by
Z ∞
g(y) = f1 (y − x)f2 (x)dx.
−∞
Math Box 2.10.2 (Sums of continuous variables). There is, in fact, a technical
problem just copying the derivation for integer valued variables to real valued
variables when we want to compute the density for the sum of two variables. The
problem is that all outcomes x have probablity zero of occuring. An alternative
derivation is based on Example 2.10.4. A substitution of x1 with x1 − x2 and an
interchange of the integration order yields the formula
Z y Z ∞
F (y) = f1 (x1 − x2 )f2 (x2 )dx2 dx1
−∞ −∞
for the distribution function of the sum. From this we read of directly that the
density is the inner integral
Z ∞
f1 (y − x2 )f2 (x2 )dx2
−∞
as a function of y.
Example 2.10.20. If X1 ∼ N (µ1 , σ12 ) and X2 ∼ N (µ2 , σ22 ) are two independent
normally distributed random variables then
Y ∼ N (µ1 + µ2 , σ12 + σ22 ), (2.26)
thus the sum is also normally distributed. Indeed, to simplify computations assume
that µ1 = µ2 = 0 and σ1 = σ2 = 1, in which case the claim is that Y ∼ N (0, 2). The
trick is to make the following observation
y2 y 2
(y − x)2 + x2 = y 2 − 2xy + 2x2 = +2 x− ,
2 2
which leads to
Z ∞
1 (y−x)2 1 x2
g(y) = √ e− 2 √ e− 2 dx
−∞ 2π 2π
Z
1 − y2 ∞ 1 −(x− y )2
= √ e 4 √ e 2 dx
4π −∞ π
| {z }
=1
1 2
− y4
= √ e .
4π
We used that the integral above for fixed y is the integral of the density for the
N y2 , 12 distribution, which is thus 1. Then observe that g is the density for the
N (0, 2) distribution as claimed. For general µ1 , µ2 , σ1 and σ2 we can make the same
computation with a lot of bookkeeping, but this is not really the interesting message
here. The interesting message is that the sum of two normals is normal. ⋄
Continuing adding normally distributed random variables we just end up with nor-
mally distributed random variables with their means and variances added up ac-
cordingly. If X1 , . . . , Xn are iid N (µ, σ) then we find that
S = X1 + . . . + Xn ∼ N (nµ, nσ 2 ).
Thus from Example 2.9.9 the distribution of the sample mean
1 1
Y = S = (X1 + . . . + Xn )
n n
2
is N (µ, σn ). This property of the normal distribution is quite exceptional. It is so
exceptional that the sample mean of virtually any collection of n iid variables strives
towards being normally distributed. The result explaining this fact is one of the
germs, if not the biggest then at least among the really great ones, in probability
theory. This is the Central Limit Theorem or CLT for short.
Result 2.10.21. If X1 , . . . , Xn are iid with mean µ and variance σ 2 then

1 approx σ2
(X1 + . . . + Xn ) ∼ N µ, . (2.27)
n n
The precise meaning of the formulation is that

1
P √ (X1 + . . . + Xn − nµ) ≤ x → Φ(x)
σ n
for n → ∞. The content is that the distribution of the sample mean is well approx-
imated by the normal distribution if n is sufficiently large. How large n needs to
be depends on the actual distribution of the Xi ’s, but the approximation is often
surprisingly good in practice even for n as small as 20. A formal derivation of the
central limit theorem is beyond the scope of these notes, and it does, moreover, not
shed much light on its usefulness.
Then we turn to taking the maximum or taking the minimum of real valued random
variables. If X1 , . . . , Xn are n real value random variables and
Y = max(X1 , X2 , . . . , Xn )
then Y ≤ x if and only if all the X’s are ≤ x. Hence
P(Y ≤ x) = P(X1 ≤ x, X2 ≤ x, . . . , Xn ≤ x).
If X1 , . . . , Xn are iid with common distribution having distribution function F the
right hand side factorizes into
P(X1 ≤ x)P(X2 ≤ x) · . . . · P(Xn ≤ x)
and therefore, using that the common distribution function for the X’s is F , we get
P(Y ≤ x) = P(X1 ≤ x)P(X2 ≤ x) · . . . · P(Xn ≤ x) = F (x)n .
A similar derivation for the minimum is possible, which gives us the following result.
Result 2.10.22. If X1 , . . . , Xn are n iid real value random variables with distribu-
tion function F then the distribution function, G, for
Y1 = max(X1 , X2 , . . . , Xn )
is given as
G1 (x) = F (x)n
and the distribution function for
Y2 = min(X1 , X2 , . . . , Xn )
is given as
G2 (x) = 1 − (1 − F (x))n
Example 2.10.23. Let X1 , . . . , Xn be n iid Gumbel distributed random variables.

Their common distribution function is
F (x) = exp(− exp(−x)).
From Result 2.10.22 we find that the distribution function for Y = max{X1 , . . . , Xn }
is
G(x) = F (x)n = exp(− exp(−x))n

= exp(−n exp(−x))
= exp(− exp(−(x − log n))).
From this we see that the distribution of Y is also a Gumbel distribution with
location parameter log n. ⋄
Example 2.10.24. Let X1 , . . . , Xn be n iid exponentially distributed random vari-

ables with intensity parameter λ. Their common distribution function is
F (x) = 1 − exp(−λx).
From Result 2.10.22 we find that the distribution function for Y = min{X1 , . . . , Xn }
is
G(x) = 1 − (1 − F (x))n = 1 − exp(−λx)n

= 1 − exp(−nλx).
Thus the distribution of the minimum Y is also an exponential distribution with

intensity parameter nλ. ⋄
2.11 Simulations
The use of computer simulations is an omnipresent tool in current scientific research

ranging from issues such as forecasting of the weather or the economic develop-
ments to understanding the mechanisms working in biology, chemistry and physics.
A (computer) simulation consists of an implementation of our favorite experiment
as a mathematical model on a computer and then “we run the experiment on the
computer”. If the model consists of a set of ordinary differential equations, say, this
amounts to solving the equations numerically using the computer. This provides a
very useful supplement to the mathematical analysis of the model, which is typ-
ically so complex that we have no hope of writing down an easily interpretable
solution on paper. We are not interested in solving differential equations but instead
in simulating the outcome of an experiment with a random component. Redoing
a simulation several times should lead to different results reflecting the probability
measure that models the random phenomenon. This is in principle somewhat of a
problem. Whereas the computer is entirely deterministic – logical and deterministic
behavior is what makes the computer useful in the first place – we ask for outcomes
of running identical simulations that should differ. Solving deterministic differential
equations is directly suitable for a computer implementation, but simulating random
variables is not.
The problem is essentially always solved by the following two step procedure.
• The computer emulates the generation of independent, identically distributed

random variables with the uniform distribution on the unit interval [0, 1].
• The emulated uniformly distributed random variables are by transformation

turned into variables with the desired distribution.
If we had access to a stream of truly independent, identically distributed random

variables with the uniform distribution on [0, 1], perhaps arriving to the computer
from an external truly random source, and if we implement the second step above
correctly we would indeed obtain simulations of random variables with the desired
distribution. In practice we don’t use externally generated random numbers but rely
on a program generating pseudo random numbers. It is even questionable if there
exists such a thing as an external, truly random source of random numbers. The
pseudo random number generator emulates the generation of truly random numbers,
and the philosophy is that if we can’t statistically detect the difference, there is, from
a practical point of view, no difference! But the pseudo random number generator
is certainly, like all other computer programs, a deterministic program.
In these notes we will not discuss in any further details how to generate pseudo
random numbers. This will be regarded as a black box tool that we leave for others
to think about. Two things are, however, important to keep in mind. First of all
the pseudo random number generator may not provide sufficiently good pseudo
Simulations 87
random numbers, which can lead to wrong results. If you use the standard generator
provided by R you are not likely to run into problems, but it is not guaranteed
that all programming languages provide a useful standard pseudo random number
generator. If in doubt you should seek qualified assistance. Secondly, the pseudo
random number generator is always initialized with a seed – a number that tells the
generator how to start. Providing the same seed will lead to the same sequence of
numbers. If we need to run independent simulations we should be cautious not to
restart the pseudo random number generator with the same seed. It is, however, an
advantage when debugging programs that you can always provide the same sequence
of random numbers. Most pseudo random number generators have some default way
of setting the seed if no seed is provided by the user.
R Box 2.11.1 (Pseudo random numbers). In R we can generate pseudo random

numbers using the command runif. For instance
> u <- runif(100)
produces a vector u containing 100 pseudo random numbers. In general runif(n)

produces n pseudo random numbers. Use set.seed(m) to set the seed for the
generator to be m. Every time R is opened a new seed is set based on the current
time, so set.seed is only used if one needs to reproduce the same results several
times.
For the rest of this section we will assume that we have access to a sequence
U1 , U2 , . . . , Un of iid random variables uniformly distributed on [0, 1], which in prac-
tice are generated using a pseudo random number generator. This means that
P(U1 ∈ A1 , U2 ∈ A2 , . . . , Un ∈ An ) = P(U1 ∈ A1 )P(U2 ∈ A2 ) · . . . · P(Un ∈ An )
for all events A1 , . . . , An ⊆ [0, 1], and that
P(Ui ∈ [a, b]) = b − a
for 0 ≤ a ≤ b ≤ 1. It then follows by Result 2.10.15 that if Xi = h(Ui ) for a given

transformation h then the Xi ’s are independent. Thus we have the following result.
Result 2.11.1. Let P0 denote the uniform distribution on [0, 1] and h : [0, 1] →
E a map with the transformed probability measure on E being P = h(P0 ). Then
X1 , X2 , . . . , Xn defined by
Xi = h(Ui )
are n iid random variables each with distribution P .
Simulating discrete random variables basically amounts to Algorithm 2.11.1. There

the implicit transformation used is h : [0, 1] → E given by
h(u) = x if u ∈ I(x) = (a(x), b(x)].

Algorithm 2.11.1 (Simulating discrete random variables). If we want to simulate

random variables taking values in a discrete space E we can proceed as follows:
Assume that the distribution that we want to simulate from has point probabilities
p(x), x ∈ E, and choose for each x ∈ E an interval
I(x) = (a(x), b(x)] ⊆ [0, 1]
such that
• the length, b(x) − a(x), of I(x) equals p(x),

• and the intervals I(x) are mutually disjoint:
I(x) ∩ I(y) = ∅ for x 6= y.
Letting u1 , . . . , un be generated by a pseudo random number generator we define
xi = x if ui ∈ I(x)
for i = 1, . . . , n. Then x1 , . . . , xn is a realization of n iid random variables with

distribution having point probabilities p(x), x ∈ E.
If U is uniformly distributed on [0, 1]
P(h(U ) = x) = P(U ∈ I(x)) = b(x) − a(x) = p(x).
This shows that the algorithm indeed simulates random variables with the desired
distribution.
In practice we need to choose the intervals suitably. If the random variable, X, that
we want to simulate takes values in N a possible choice of I(x) is
I(x) = (P(X ≤ x − 1), P(X ≤ x)].
We see that the length of I(x) equals
P(X ≤ x) − P(X ≤ x − 1) = P(X = x) = p(x)
as required, and the intervals are clearly disjoint. To easily compute these intervals
we need easy access to the distribution function
F (x) = P(X ≤ x).
For the simulation of real valued random variables there is also a generic solution
that relies on the knowledge of the distribution function.
Definition 2.11.2. Let F : R → [0, 1] be a distribution function. A function
F ← : (0, 1) → R
Simulations 89
that satisfies
F (x) ≥ y ⇔ x ≥ F ← (y) (2.28)
for all x ∈ R and y ∈ (0, 1) is called a generalized inverse of F .
There are a few important comments related to the definition above. First of all
suppose that we can solve the equation
F (x) = y
for all x ∈ R and y ∈ (0, 1), yielding an inverse function F −1 : (0, 1) → R of F that
satisfies
F (x) = y ⇔ x = F −1 (y). (2.29)
Then the inverse function is also a generalized inverse function of F . However, not
all distribution functions have an inverse, but all distribution functions have a gen-
eralized inverse and this generalized inverse is in fact unique. We will not show this
although it is not particularly difficult. What matters in practice is whether we can
find the (generalized) inverse of the distribution function. Note also, that we do not
really want to define the value of F ← in 0 or 1. Often the only possible definition is
F ← (0) = −∞ and F ← (1) = +∞.
At this point the generalized inverse is useful because it is, in fact, a quantile function.
To see this, first observe that with x = F ← (y) then
F ← (y) ≤ x ⇔ y ≤ F (x) = F (F ← (y))
by the definition of F ← . On the other hand, suppose that there exists a y ∈ (0, 1)
and an ε > 0 such that F (F ← (y) − ε) ≥ y then again by the definition of F ← it
follows that
F ← (y) − ε ≥ F ← (y),
which cannot be the case. Hence there is no such y ∈ (0, 1) and ε > 0 and
F (F ← (y) − ε) < y
for all y ∈ (0, 1) and ε > 0. This shows that F ← is a quantile function.
Result 2.11.3. The generalized inverse distribution function F ← is a quantile func-
tion.
There may exist other quantile functions besides the generalized inverse of the distri-
bution function, which are preferred from time to time. If F has an inverse function
then the inverse is the only quantile function, and it is equal to the generalized
inverse.
We will need the generalized inverse to transform the uniform distribution into
any distribution we would like. The uniform distribution on [0, 1] has distribution
function
G(x) = x
for x ∈ [0, 1]. By the definition of F ← we have that
F ← (U ) ≤ x ⇔ U ≤ F (x)
for all x ∈ R and hence
P(F ← (U ) ≤ x) = P(U ≤ F (x)) = G(F (x)) = F (x).
Note that it doesn’t matter if we have defined F ← on (0, 1) only as the uniform
random variable U with probability 1 takes values in (0, 1). We have derived the
following result.
Result 2.11.4. If F ← : (0, 1) → R is the generalized inverse of the distribution
function F : R → [0, 1] and if U is uniformly distributed on [0, 1] then the distribution
of
X = F ← (U )
has distribution function F .
The result above holds for all quantile functions, but it is easier and more explicit
just to work with the generalized inverse, as we have done.
Algorithm 2.11.2 (Simulating real valued random variables). If we want to sim-

ulate random variables taking values in R with distribution function F we can
proceed as follows: First we find the generalized inverse, F ← : (0, 1) → R, of F .
Then we let u1 , . . . , un be generated by a pseudo random number generator and
we define
xi = F ← (ui )
for i = 1, . . . , n. Then x1 , . . . , xn is a realization of n iid random variables with
distribution having distribution function F .
Example 2.11.5. The exponential distribution with parameter λ > 0 has distribu-
tion function
F (x) = 1 − exp(−λx), x ≥ 0.
The equation
F (x) = 1 − exp(−λx) = y
is solved for y ∈ (0, 1) by
1
F −1 (y) = − log(1 − y).
λ
Thus the simulation of exponentially distributed random variables can be based on
the transformation
1
h(y) = − log(1 − y).
λ
⋄
Local alignment - a case study 91
R Box 2.11.2 (Inverse distribution functions). Like distribution functions and

densities the (generalized) inverse distribution functions are directly available in R
for a number of standard distributions. The general convention is that qname is the
inverse distribution function for a distribution called name. The inverse distribution
function for the normal distribution evaluated at x is therefore given by qnorm(x).
Exercises
Ï Exercise 2.11.1. Write three R functions: pgumbel, dgumbel, and qgumbel taking
three arguments x,location,scale and returning the value (in x) of the distribution
function, the density, and the inverse distribution function (the quantile function)
respectively for the Gumbel distribution with location and scale parameters given by
location and scale.
Ï Exercise 2.11.2. Write a fourth function, rgumbel, that takes one integer argument,
n, and the location,scale arguments, and returns a vector of length n, which is a
simulated realization of n independent and identically Gumbel distributed random
variables.
⋆ Exercise 2.11.3. Let X have the geometric distribution with success probability p.
Show that the distribution function for X is
F (x) = 1 − (1 − p)⌊x⌋+1 ,
where ⌊x⌋ ∈ Z is the integer fulfilling that x − 1 < ⌊x⌋ ≤ x for x ∈ R. Define likewise
⌈x⌉ ∈ Z as the integer fulfilling that x ≤ ⌈x⌉ < x + 1 for x ∈ R.
Argue that
⌊z⌋ ≥ x ⇔ z ≥ ⌈x⌉
for all x, z ∈ R use this to show that

log(1 − y)
F ← (y) = −1
log(1 − p)
is the generalized inverse for the distribution function for the geometric distribution.
Ï Exercise 2.11.4. Write an R-function, my.rgeom, that takes two arguments, (n,p),
and returns a simulation of n iid geometrically distributed variables with success
parameter p. Note that the operations ⌊·⌋ and ⌈·⌉ are known as floor and ceiling.
2.12 Local alignment - a case study
As a case study of some of the concepts that have been developed up to this point
in the notes, we present in this section some results about the distribution of the
score of optimally locally aligned random amino acid sequences.
This is a classical problem and a core problem in biological sequence analysis whose
solution has to be found in the realms of probability theory and statistics. Moreover,
even though the problem may seem quite specific to biological sequence analysis it
actually holds many of the general issues related to the extraction of data from large
databases and how the extraction procedure may need a careful analysis for a correct
interpretation of the results.
The local alignment problem is essentially not a biological problem but a computer
science problem of finding substrings of two strings that match well. Given two
strings of letters can we compute two substrings of letters – one from each – that
maximize the length of matching letters? In the words ABBA and BACHELOR we can
find the substring BA in both but no substrings of length 3. The stringent requirement
of exact matching is often to hard. If we try to match BACHELOR and COUNCILLOR
we could get an exact match of length 3 of LOR but allowing for two mismatches
we can match CHELOR and CILLOR of length 6. It may also be an idea to allow for
gaps in the matching. If we try to match COUNCILLOR with COUNSELOR we can match
COUN--CILLOR with COUNSE---LOR where we have introduced two gaps in the former
string and three in the latter.
When we introduce mismatches and gaps we will penalize their occurrences and
formally we end up with a combinatorial optimization problem where we optimize
a score over all selections of substrings. The score is a sum of (positive) match
contributions and (negative) mismatch and gap constributions. The computational
solution known as the Smith-Waterman algorithm is a so-called dynamic program-
ming algorithm, which can solve the optimization problem quite efficiently.
In the context of biological sequences, e.g. proteins regarded as a string of letters
from the amino acid alphabet, there are many implementations of the algorithm
and it is in fact not awfully complicated to write a simple implementation yourself.
The actual algorithm is, however, outside of the scope of these notes. In R there is
an implementation of several alignment algorithms, including the Smith-Waterman
algorithm for local alignment, in the pairwiseAlignment function from the package
Biostrings. The widely used BLAST program and friends offer a heuristic that
improves the search time tremendously and makes it feasible to locally align a protein
to the entire database of known protein sequences.
The purpose of the computation of the local alignment is to find parts of the protein
that share evolutionary or functional relations to other proteins in the database. The
algorithm will always produce a best optimal local alignment disregarding whether
the best alignment has any meaning or is just a coincidence due to the fact that a
database contains a large number of different sequences. Thus we need to be able to
tell if a local alignment with a given score is something we would expect to see by
chance or not.
Assume that X1 , . . . , Xn and Y1 , . . . , Ym are in total n + m random variables with
values in the 20 letter amino acid alphabet

E = A, R, N, D, C, E, Q, G, H, I, L, K, M, F, P, S, T, W, Y, V .
We regard the first n letters X1 , . . . , Xn as one sequence (one protein or a protein
sequence database) and Y1 , . . . , Ym as another sequence (another protein or protein
sequence database). The joint distribution of the n + m random variables is specified

by assuming that they are all independent and identically distributed. The distribu-
tion of each single letter in the sequences may be given by the Robinson-Robinson
frequencies from Example 2.4.5 – or any other distribution on the 20 letter amino
acid alphabet. The joint distribution is a probability measure on the product space
E n+m .
The optimal local alignment score can be regarded as a transformation
h : E n+m → R
where h(x1 , . . . , xn , y1 , . . . , ym ) denotes the score obtained by optimally aligning the
first n letters x1 , . . . , xn with the last m letters y1 , . . . , ym . Let
Sn,m = h(X1 , . . . , Xn , Y1 , . . . , Ym )
denote the random variable we obtain by transforming the n + m random variables
by the function h. This gives a distribution on R, but it is really only a distribution
living on a discrete – even finite – subset of R. Indeed, the function h can at most
take 20n+m different values as this is the size of the sample space E n+m . Although
one in principle can compute the distribution of Sn,m by going through each of the
different 20n+m possible values of h and compute the corresponding probability, this
is a futile task for realistic values of n and m.
There are (at least) two alternative approaches. One is to compute the distribution
of Sn,m by simulations as discussed in Section 2.11. Simulations are a universally
applicable tool for studying the distribution of transformations. The downside is first
of all that it can be time-consuming in particular in the setup of local alignment, and
second that you get no or little theoretical insight about the distribution of Sn,m .
Another approach is to find an approximating distribution with a few parameters,
whose values depend on n, m, the distribution of the letters, and the choice of scoring
mechanism for the local alignment.
For optimal local alignment scores a frequently used approximation is a location scale
transformation of a Gumbel distribution. Under certain conditions on the scoring
mechanism and the letter distribution, and for n and m sufficiently large,
P(Sn,m ≤ x) ≃ exp(−Knm exp(−λx)) (2.30)
with λ, K > 0 two parameters not depending upon n and m. We use the notation
≃ to denote approximately equal without specifying what we mean by that in any
detail. In practice, we work and argue as if ≃ means equality, but we keep in mind
that it is only an approximation. Defining the variable
′
Sn,m = λSn,m − log(Knm)
′
we see that Sn,m is a location-scale transformation of Sn,m , hence

′ λ(x + log(Knm))
P(Sn,m ≤ x) ≃ exp −Knm exp −
λ
= exp(− exp(−x)).
BLOSUM62 BLOSUM50 PAM250

gap open gap ext. λ K λ K λ K
12 3 0.300 0.10 0.178 0.028 0.170 0.022
12 2 0.300 0.09 0.158 0.019 0.145 0.012
12 1 0.275 0.05 − − − −
11 3 0.301 0.09 0.167 0.028 0.153 0.017
11 2 0.286 0.07 0.130 0.009 0.122 0.009
11 1 0.255 0.035 − − − −
10 3 0.281 0.06 0.139 0.013 0.129 0.012
10 2 0.266 0.04 0.099 0.007 − −
10 1 0.216 0.014 − − − −
9 3 0.273 0.06 0.107 0.008 0.102 0.010
9 2 0.244 0.030 − − − −
Table 2.1: A selection of values for the λ and K parameters for different choices of
scoring schemes in the local alignment algorithm. The entries marked by − are scor-
ing schemes where the approximation (2.30) breaks down. The values presented here
are from Altschul, S.F. and Gish, W. Local Alignment Statistics. Methods in Enzy-
mology, vol 266, 460-480. They are computed (estimated) on the basis of alignments
of simulated random amino acid sequences using the Robinson-Robinson frequencies.
In other words, the distribution of Sn,m is approximately a Gumbel distribution with

location parameter log(Knm)/λ and scale parameter 1/λ.
On Figure 2.20 we see the density for the location-scale transformed Gumbel distri-
butions, taking n = 1, 000 and m = 100, 000, that approximate the distribution of
the maximal local alignment score using either BLOSUM62 or BLOSUM50 together
with the affine gap penalty function. We observe that for the BLOSUM50 matrix
the distribution is substantially more spread out.
If n = n1 + n2 , the approximation given by (2.30) implies that
P(Sn,m ≤ x) ≃ exp(−Knm exp(−λx))

= exp(−K(n1 + n2 )m exp(−λx))
= exp(−Kn1 m exp(−λx)) exp(−Kn2 m exp(−λx)).
If (2.30) holds for Sn1 ,m and Sn2 ,m too, then
P(Sn1 ,m ≤ x) ≃ exp(−Kn1 m exp(−λx)) and P(Sn2 ,m ≤ x) ≃ exp(−Kn2 m exp(−λx)),
which implies that
P(Sn,m ≤ x) ≃ P(Sn1 ,m ≤ x)P(Sn2 ,m ≤ x).

0.10
0.10
0.05
0.05
0.00
0.00
40 45 50 55 60 65 70 70 80 90 100 110 120 130
Figure 2.20: The density for the Gumbel distribution that approximates the maximal
local alignment score with n = 1, 000 and m = 100, 000 using an affine gap penalty
function with gap open penalty 12 and gap extension penalty 2 together with a
BLOSUM62 (left) or BLOSUM50 (right) scoring matrix.
This (approximate) equality says that the distribution of Sn,m behaves (approxi-
mately) as if Sn,m = max{Sn1 ,m , Sn2 ,m } and that Sn1 ,m and Sn2 ,m are independent.
The justification for the approximation (2.30) is quite elaborate. There are theoret-
ical (mathematical) results justifying the approximation for ungapped local align-
ment. When it comes to gapped alignment, which is of greater practical interest,
there are some theoretical results suggesting that (2.30) is not entirely wrong, but
there is no really satisfactory theoretical underpinning. On the other hand, a number
of detailed simulation studies confirm that the approximation works well – also for
gapped local alignment. For gapped, as well as ungapped, local alignment there exist
additional corrections to (2.30) known as finite size corrections, which reflect the fact
that (2.30) works best if n and m are large – and to some extent of the same order.
Effectively, the corrections considered replace the product nm by a smaller number
n′ m′ where n′ < n and m′ < m. We will not give details here. Another issue is how
the parameters λ and K depend upon the scoring mechanism and the distribution
of the letters. This is not straight forward – even in the case of ungapped local
alignment, where we have analytic formulas for computing λ and K. In the gapped
case, the typical approach is to estimate values of λ and K from simulations.
Exercises
Alice and Bob work in Professor A.K.’s laboratory on a putative serine protease:
>gi|21220235|ref|NP_626014.1| putative secreted serine protease [Streptomyces coelicolor A3(2)]

MKHRRIPRRRVAVVGAGITALVAAGVTFQTANASEAPKTAAPETLSVSAAGELASTLLGDLGADAAGTYY
DAQAKSLVVNVLDQSAAQTVEEAGAKARVVENSLADLKSARTALTKDATIPGTSWATDPTTNKVVVTADR
TVSEAELAKLTKVVDGLGAKAELKRTKGEYKPFVAGGDAITGGGGRCSLGFNVTKGGEPYFITAGHCTES
ISTWSDSSGNVIGENAASSFPDNDYGLVKYTADVDHPSEVNLYNGSSQAISGAAEATVGMQVTRSGSTTQ
VHDGTVTGLDATVNYGNGDIVNGLIQTDVCAEPGDSGGSLFSGDQAIGLTSGGSGDCTSGGETFFQPVTE
ALSATGTQIG
Bob is convinced that the protein is related to the Human protein
>gi|4506141|ref|NP_002766.1| HtrA serine peptidase 1 [Homo sapiens]

MQIPRAALLPLLLLLLAAPASAQLSRAGRSAPLAAGCPDRCEPARCPPQPEHCEGGRARDACGCCEVCGA
PEGAACGLQEGPCGEGLQCVVPFGVPASATVRRRAQAGLCVCASSEPVCGSDANTYANLCQLRAASRRSE
RLHRPPVIVLQRGACGQGQEDPNSLRHKYNFIADVVEKIAPAVVHIELFRKLPFSKREVPVASGSGFIVS
EDGLIVTNAHVVTNKHRVKVELKNGATYEAKIKDVDEKADIALIKIDHQGKLPVLLLGRSSELRPGEFVV
AIGSPFSLQNTVTTGIVSTTQRGGKELGLRNSDMDYIQTDAIINYGNSGGPLVNLDGEVIGINTLKVTAG
ISFAIPSDKIKKFLTESHDRQAKGKAITKKKYIGIRMMSLTSSKAKELKDRHRDFPDVISGAYIIEVIPD
TPAEAGGLKENDVIISINGQSVVSANDVSDVIKRESTLNMVVRRGNEDIMITVIPEEIDP
Bob does a local alignment of the two proteins using the Smith-Waterman algo-
rithm with the BLOSUM50 scoring matrix and affine gap penalty function with gap
open penalty 10 and gap extension penalty 3. He uses the an implementation called
SSearch and finds the (edited) alignment:
>>QUERY sequence (480 aa)

Smith-Waterman score: 68; 29.333% identity (64.000% similar) in 75 aa overlap (263-331:270-344)
Entrez lookup Re-search database General re-search
>QUERY 263- 331: ------------- :
230 240 250 260 270 280 290 300

QUERY GENAASSFPDNDYGLVKYTADVDHPSEVNLYNGSSQAISGAAEATVGMQVTRSGSTTQVHDGTVTGLDATVNYGNGDI--
..: : :. :: .... ..::. .:.. :. ..
QUERY VELKNGATYEAKIKDVDEKADIALIKIDHQGKLPVLLLGRSSELRPGEFVVAIGSPFSLQNTVTTGIVSTTQRGGKELGL
230 240 250 260 270 280 290 300
310 320 330 340 350 360

QUERY VNG---LIQTDVCAEPGDSGGSLFSGD-QAIGLTSGGSGDCTSGGETFFQPVTEALSATGTQIG
:. ::::. . :.::: : . : ..::...
QUERY RNSDMDYIQTDAIINYGNSGGPLVNLDGEVIGINTLKVTAGISFAIPSDKIKKFLTESHDRQAKGKAITKKKYIGIRMMS
310 320 330 340 350 360 370 380
Note that the output shows in the upper right corner that the length of one of the
sequences in 480. The other one has length 360.
Alice is not so keen on Bobs theory and she does a database search against the set of
Human proteins using BLAST with default parameter settings (Blosum62, gap open
11, gap extension 1, Human protein database contains at the time aprox. 13.000.000
amino acids). She finds the best alignment of the serine protease with the Human
serine peptidase far down the list
Score = 61
Identities = 26/95 (27%), Positives = 40/95 (42%), Gaps = 15/95 (15%)
Multivariate distributions 97
Query 243 DVDHPSEVNLYNGSSQA------ISGAAEATVGMQVTRSGSTTQVHDGTVTGLDATVNYG 296

DVD +++ L Q + ++E G V GS + + TG+ +T G
Sbjct 244 DVDEKADIALIKIDHQGKLPVLLLGRSSELRPGEFVVAIGSPFSLQNTVTTGIVSTTQRG 303
Query 297 NGDIVNGL-------IQTDVCAEPGDSGGSLFSGD 324

++ GL IQTD G+SGG L + D
Sbjct 304 GKEL--GLRNSDMDYIQTDAIINYGNSGGPLVNLD 336
Alice believes that this provides her with evidence that Bobs theory is wrong. Who
is A. K. going to believe?
Exercise 2.12.1. Compute the probability – using a suitable model – for each of
the local alignments above of getting a local alignment with a score as large or larger
than the given scores. Pinpoint the information you need to make the computations.
Discuss the assumptions upon which your computations rest. What is the conclusion?
2.13 Multivariate distributions
If X1 , . . . , Xn are n random variables taking values in the sample spaces E1 , . . . , En ,

we can bundle the variables into a single random variable. The bundled variable
X = (X1 , . . . , Xn ) takes values in the product space
E1 × . . . × En = {(x1 , . . . , xn )|x1 ∈ E1 , . . . , xn ∈ En }.
The product space is the set of n-tuples with the i’th coordinate belonging to Ei
for i = 1, . . . , n. Each of the variables can represent the outcome of an experiment,
and we want to consider all the experiments simultaneously. To do so, we need to
define the distribution of the bundled variable X that takes values in the product
space. Thus we need to define a probability measure on the product space. In this
case we talk about the joint distribution of the random variables X1 , . . . , Xn , and
we often call the distribution on the product space a multivariate distribution or a
multivariate probability measure. We do not get the joint distribution automatically
from the distribution of each of the random variables – something more is needed. We
need to capture how the variables interact, and for this we need the joint distribution.
If the joint distribution is P and A is an event having the product form
A = A1 × . . . × An = {(x1 , . . . , xn ) | x1 ∈ A1 , . . . , xn ∈ An }
we use the notation
P (A) = P(X ∈ A) = P((X1 , . . . , Xn ) ∈ A1 × . . . × An ) = P(X1 ∈ A1 , . . . , Xn ∈ An ).
The right hand side is particularly convenient, for if some set Ai equals the entire
sample space Ei , it is simply left out. Another convenient, though slightly technical
point, is that knowledge of the probability measure on product sets specifies the
measure completely. This is conceptually similar to the fact that the distribution
function uniquely specifies a probability measure on R.
Example 2.13.1. Let X1 , X2 , and X3 be three real valued random variables, that
is, n = 3 and E1 = E2 = E3 = R. Then the bundled variable X = (X1 , X2 , X3 ) is a
three dimensional vector taking values in R3 . If A1 = A2 = [0, ∞) and A3 = R then
P((X1 , X2 , X3 ) ∈ [0, ∞) × [0, ∞) × R) = P(X1 ∈ [0, ∞), X2 ∈ [0, ∞)).
Moreover, in this case we rewrite the last expression as
P(X1 ∈ [0, ∞), X2 ∈ [0, ∞)) = P(X1 ≥ 0, X2 ≥ 0).
If A1 = [a, b], A2 = R and A3 = [c, d] for a, b, c, d ∈ R then
P((X1 , X2 , X3 ) ∈ [a, b] × R × [c, d]) = P(X1 ∈ [a, b], X3 ∈ [c, d])

= P(a ≤ X1 ≤ b, c ≤ X3 ≤ d).
Definition 2.13.2 (Marginal distribution). If the bundled variable X = (X1 , . . . , Xn )

has distribution P , the marginal distribution, Pi , of Xi is given by
Pi (A) = P(Xi ∈ A) = P (E1 × . . . × Ei−1 × A × Ei+1 × . . . × En )
for A ⊆ Ei .
If the sample spaces that enter into a bundling are discrete, so is the product space –
the sample space of the bundled variable. The distribution can therefore in principle
be defined by point probabilities.
Example 2.13.3. If two DNA-sequences that encode a protein (two genes) are
evolutionary related, then typically there is a pairing of each nucleotide from one
sequence with an identical nucleotide from the other with a few exceptions due
to mutational events (an alignment). We imagine in this example that the only
mutational event occurring is substitution of nucleic acids. That is, one nucleic acid
at the given position can mutate into another nucleic acid. The two sequences can
therefore be aligned in a letter by letter fashion without gaps, and we are going
to consider just a single aligned position in the two DNA-sequences. We want a
probabilistic model of the pair of letters occurring at that particular position. The
sample space is going to be the product space
E = E0 × E0 = {A, C, G, T} × {A, C, G, T},
and we let X and Y denote the random variables representing the two aligned
nucleic acids. To define the joint distribution of X and Y , we have to define point
probabilities p(x, y) for (x, y) ∈ E. It is convenient to organize the point probabilities
in a matrix (or array) instead of as a vector. Consider for instance the following
matrix
A C G T
A 0.1272 0.0063 0.0464 0.0051
C 0.0196 0.2008 0.0082 0.0726
G 0.0556 0.0145 0.2151 0.0071
T 0.0146 0.0685 0.0069 0.1315
As we can see the probabilities occurring in the diagonal are (relatively) large and
those outside the diagonal are small. If we let A = {(x, y) ∈ E | x = y} denote the
event that the two nucleic acids are identical, then
X
P(X = Y ) = P (A) = p(x, y) = 0.1272 + 0.2008 + 0.2151 + 0.1315 = 0.6746.
(x,y)∈A
This means that the probability of obtaining a pair of nucleic acids with a mutation
is P(X 6= Y ) = P (Ac ) = 1 − P (A) = 0.3254. ⋄
Compared to discrete sample spaces, the situation seems more complicated if we

bundle real valued variables. If we bundle n real valued random variables X1 , . . . , Xn ,
the bundled variable takes values in Rn – the set of n-dimensional real vectors. To
define and handle distributions on Rn easily becomes quite technical. There exists
a multivariate analog of the distribution function, but it is a clumsy object to work
with. The best situation arises when the joint distribution is given by a density.
Definition 2.13.4. If X1 , . . . , Xn are n real valued random variables we say that
the distribution of X = (X1 , . . . , Xn ) has density
f : Rn → [0, ∞)
if Z Z
P(X1 ∈ A1 , . . . , Xn ∈ An ) = ··· f (x1 , . . . , xn )dxn . . . dx1 .
A1 An
Note that from the fact that P(X ∈ Rn ) = 1, a density f must fulfill that
Z ∞ Z ∞
··· f (x1 , . . . , xn )dxn . . . dx1 = 1.
−∞ −∞
Remember also the convention about our formulations: Every time we talk about
random variables, we are in fact only interested in their distribution, which is a prob-
ability measure on the sample space. The content of the definition above is therefore
that the distribution – the probability measure – is given simply by specifying a
density. One of the deeper results in probability theory states that if a function f ,
defined on Rn and with values in [0, ∞), integrates to 1 as above, then it defines a
probability measure on Rn .
If you feel uncomfortable with the integration, remember that if Ai = [ai , bi ] with
ai , bi ∈ R for i = 1, . . . , n then
Z b1 Z bn
P(a1 ≤ X1 ≤ b1 , . . . , an ≤ Xn ≤ bn ) = ··· f (x1 , . . . , xn )dxn . . . dx1
a1 an
can be computed as n successive ordinary integrals.

The integrations can be carried out in any order, so carrying out the integration
over the i’th variable as the last ourter integration we can compute the distribution
function for the distribution of Xi as follows:
Z x Z ∞ Z ∞
Fi (x) = P(Xi ≤ x) = ··· f (x1 , . . . , xn )dxn . . . dxi+1 dxi−1 . . . dx1 dxi .
−∞ −∞ −∞
| {z }
n−1
Consequently we can see that the marginal distribution of Xi also has a density that
is given by the function
Z ∞ Z ∞
xi 7→ ··· f (x1 , . . . , xn )dxn . . . dxi+1 dxi−1 . . . dx1 .
−∞ −∞
| {z }
n−1
We state this as a result.

Result 2.13.5. If f : Rn → [0, ∞) is the density for the joint distribution of
X1 , . . . , Xn then the marginal distribution of Xi has density
Z ∞ Z ∞
fi (xi ) = ··· f (x1 , . . . , xn )dxn . . . dxi+1 dxi−1 . . . dx1 . (2.31)
−∞ −∞
| {z }
n−1
In a similar manner, one can compute the density for the distribution of any subset of
coordinates of an n-dimensional random variable, whose distribution has a density,
by integrating out over the other coordinates.
Example 2.13.6 (Bivariate normal distribution). Consider the function
p 2
1 − ρ2 x − 2ρxy + y 2
f (x, y) = exp −
2π 2
for ρ ∈ (−1, 1). We will first show that this is a density for a probability measure
on R2 , thus that it integrates to 1, and then find the marginal distributions. The
numerator in the exponent in the exponential function can be rewritten as
x2 − 2ρxy + y 2 = (x − ρy)2 + (1 − ρ2 )y 2 ,
hence
Z ∞Z ∞ 2
x − 2ρxy + y 2
exp − dydx
−∞ −∞ 2
Z ∞Z ∞
(x − ρy)2 (1 − ρ2 )y 2
= exp − − dydx
−∞ −∞ 2 2
Z ∞ Z ∞
(x − ρy)2 (1 − ρ2 )y 2
= exp − dx exp − dy.
−∞ −∞ 2 2
0.15 0.15
0.10 0.10
density
density
4 4
0.05 2 0.05 2
0 0
y
0.00 0.00
−4 −4
−2 −2 −2 −2
0 0
x 2 x 2
−4 −4
4 4
Figure 2.21: Two examples of the density for the bivariate normal distribution as
considered in Example 2.13.6 with ρ = 0 (left) and ρ = 0.75 (right).
The inner integral can be computed for fixed y using substitution and knowledge
about the one-dimensional normal distribution. With z = x − ρy we have dz = dx,
hence Z ∞ Z ∞ 2
(x − ρy)2 z √
exp − dx = exp − dz = 2π,
−∞ 2 −∞ 2
where the last equality follows from the fact that the density for the normal distri-
bution on R is 2
1 x
√ exp − ,
2π 2
which integrates to 1.
√ We see that the inner integral does not depend
p upon y and is
2
p another substitution with z = y 1 − ρ (using that
constantly equal to 2π, thus
2
ρ ∈ (−1, 1)) such that dz = 1 − ρ dy gives
Z Z Z
∞ ∞
x2 − 2ρxy + y 2 √ ∞
(1 − ρ2 )y 2
exp − dydx = 2π exp − dy
−∞ −∞ 2 −∞ 2
√ Z ∞ 2
2π z
= p exp − dz
2
1 − ρ −∞ 2
2π
= p
1 − ρ2
√
where we once more use that the last integral equals 2π. This shows that the
(positive) function f integrates to 1, and it is therefore a density for a probability
measure on R2 .
Suppose that the distribution of (X, Y ) has density f on R2 , then we have almost
computed the marginal distributions of X and Y by the integrations above. The
marginal distribution of Y has by Result 2.13.5 density
p Z 2
1 − ρ2 ∞ x − 2ρxy + y 2
f2 (y) = exp − dx
2π −∞ 2

1 y2
= p exp − .
2π(1 − ρ2 )−1 2(1 − ρ2 )−1
√
Here we used, as argued above, that the integration over x gives 2π, and then we
have rearranged the expression a little. From (2.20) in Example 2.9.9 we recognize
this density as the density for the normal distribution with scale parameter σ 2 =
(1 − ρ2 )−1 and location parameter µ = 0. Note that for ρ = 0 we have σ 2 = 1
and for ρ → ±1 we have σ 2 → ∞. The density is entirely symmetric in the x- and
y-variables, so the marginal distribution of X is also N (0, (1 − ρ2 )−1 ).
This probability measure is called a bivariate normal or Gaussian distribution. The
example given above contains only a single parameter, ρ, whereas the general bivari-
ate normal distribution is given in terms of five parameters. The general bivariate
density is given as
p
λ1 λ2 − ρ 2 1 2 2

f (x, y) = exp − λ1 (x − ξ1 ) − 2ρ(x − ξ1 )(y − ξ2 ) + λ2 (y − ξ2 )
2π 2
(2.32)
where ξ1 , ξ2 ∈ R, λ1 , λ2 ∈ (0, ∞) and ρ ∈ (−λ1 λ2 , λ1 λ2 ). For this density it is
possible to go through similar computations as above, showing that it integrates to
1. If the distribution of (X, Y ) is given
by f it islikewise possible
to compute
the
λ2 λ1
marginal distributions, where X ∼ N ξ1 , λ1 λ2 −ρ2 and Y ∼ N ξ2 , λ1 λ2 −ρ2 . ⋄
It is possible to define n-dimensional normal distributions in a similar way to the

bivariate normal distribution considered in Example 2.13.6; see Math Box 2.13.1.
The theory for the multivariate normal distribution is very rich and well developed,
and it has played and still plays an important role in theoretical as well as applied
statistics. The widespread use of analysis of variance (ANOVA) and classical linear
regression is founded on results about the multivariate normal distribution, and
these classical models are also important tools in bioinformatics, for instance in the
modeling and analysis of gene expression data. The theory is, however, covered so
well in the literature, that we will pursue any systematic development in these notes.
Example 2.13.7. The marginal distributions of X and Y computed in Example

2.13.6 were both found to be N (0, (1 − ρ2 )−1 ). The product of the corresponding
marginal densities is

1 − ρ2 x2 + y 2
f1 (x)f2 (y) = exp − ,
2π 2(1 − ρ2 )−1
Math Box 2.13.1 (Multivariate normal distributions). We can define the family
of n-dimensional regular normal or Gaussian distributions via their densities on
Rn for n ≥ 1. The measures are characterized in terms of a vector ξ ∈ Rn and a
positive definite symmetric n × n matrix Λ. Positive definite means that for any
vector x ∈ Rn we have xt Λx > 0. Consequently one can show that the positive
function x 7→ exp(−( 12 x − ξ)t Λ(x − ξ)) have a finite integral over Rn and that
Z r
1 (2π)n
exp(− (x − ξ)t Λ(x − ξ))dx = .
Rn 2 detΛ
Here detΛ is the determinant of the matrix Λ. The density for the n-dimensional
regular normal distribution with parameters ξ and Λ is then
s
detΛ 1 t
f (x) = exp − (x − ξ) Λ(x − ξ) .
(2π)n 2
The full bivariate normal distribution considered in Example 2.13.6 as given by

(2.32) corresponds to
λ1 −ρ
Λ= ,
−ρ λ2
for which we can verify that detΛ = λ1 λ2 − ρ2 .
That Λ is positive definite implies that detΛ > 0 and thus that Λ is invertible. We
denote the inverse by Σ = Λ−1 , and using that detΣ = (detΛ)−1 we can also write
the density as

1 1 t −1
f (x) = p exp − (x − ξ) Σ (x − ξ) .
(2π)n detΣ 2
One can then show that the i’th marginal distribution is the normal distributions
with location parameter ξi and scale parameter Σii .
It is possible to define a multivariate normal distribution for positive semi-definite
matrices Λ that only fulfill that xt Λx ≥ 0, but it requires a little work. If we don’t
have strict inequality, there is no density, and the multivariate normal distribution
is called singular.
which is seen to be equal to f (x, y) from Example 2.13.6 if and only if ρ = 0. Thus if
the distribution of (X, Y ) is given by the density f in Example 2.13.6, then X and
Y are independent if and only if the parameter ρ equals 0. ⋄
Example 2.13.8. (Bivariate Dirichlet distribution) Let λ1 , λ2 , λ > 0 be given. De-

fine for (x, y) ∈ R2 with x, y > 0 and x + y < 1 the function
Γ(λ1 + λ2 + λ) λ1 −1 λ2 −1
f (x, y) = x y (1 − x − y)λ−1 .
Γ(λ1 )Γ(λ2 )Γ(λ)
12 12
10 10
8 8
density
density
6 6
4 4
2 2
0 0
0.2 0.8 0.2 0.8
0.4 0.6 0.4 0.6
x 0.6 0.4 y x 0.6 0.4 y
0.8 0.2 0.8 0.2
Figure 2.22: Two examples of the density for the bivariate Dirichlet distribution as
considered in Example 2.13.8 with λ1 = λ2 = λ = 4 (left) and λ1 = λ2 = 2 together
with λ = 6 (right).
The function f is a priori defined on the open 2-dimensional unit simplex

U2 = {(x, y) ∈ R2 | x, y > 0, x + y < 1}.
By defining f (x, y) = 0 outside U2 , we can show that f integrates to 1. We only
need to integrate over the set U2 as f is defined to be 0 outside, and we find that
Z 1 Z 1−x
xλ1 −1 y λ2 −1 (1 − x − y)λ−1 dydx
0 0
Z 1 Z 1−x
= y λ2 −1 (1 − x − y)λ−1 dy xλ1 −1 dx
0 0
Z 1 Z 1
λ2 −1 λ2 −1
= (1 − x) z (1 − x) (1 − z) (1 − x)dz xλ1 −1 dx
λ−1 λ−1
0 0
Z 1 Z 1
λ2 −1 λ−1
= z (1 − z) dz (1 − x)λ2 +λ−1 xλ1 −1 dx
0 0
Z 1 Z 1
λ2 −1 λ−1
= z (1 − z) dz xλ1 −1 (1 − x)λ2 +λ−1 dx
0 0
where we have used the substitution (1− x)z = y in the first inner integral, for which
(1 − x)dz = dy. Both of the last two integrals can be recognized as B-integrals, cf.
(B.7), and according to (B.6) we find that
Z 1 Z 1
λ2 −1 λ−1 Γ(λ2 )Γ(λ) Γ(λ1 )Γ(λ2 + λ)
z (1 − z) dz xλ1 −1 (1 − x)λ2 +λ−1 dx =
0 0 Γ(λ2 + λ) Γ(λ1 + λ2 + λ)
Γ(λ1 )Γ(λ2 )Γ(λ)
= .
Γ(λ1 + λ2 + λ)
Math Box 2.13.2 (Multivariate Dirichlet distribution). There is an n-dimensional

Dirichlet distribution living on the n-dimensional unit simplex
Un = {(x1 , . . . , xn ) | x1 , . . . , xn > 0, x1 + . . . + xn < 1},
with density
Γ(λ1 + . . . + λn + λ) λ1 −1 λ2 −1
f (x1 , . . . , xn ) = x x2 · · · xnλn −1 (1 − x1 − . . . − xn )λ−1
Γ(λ1 ) · · · Γ(λn )Γ(λ) 1
for (x1 , . . . , xn ) ∈ Un and with the parameters λ1 , . . . , λn , λ > 0. We refrain from

showing that the density f really integrates to 1 over the unit simplex Un .
We note that if the joint distribution of X1 , . . . , Xn is an n-dimensional Dirichlet
distribution, then
(X1 , . . . , Xn , 1 − X1 − . . . − Xn )
is an n + 1-dimensional probability vector. Therefore the Dirichlet distribution is
often encountered as a model of random probability vectors (e.g. frequencies).
We see that the integral of the positive function f is 1, and f is the density for the
bivariate Dirichlet distribution on R2 . Since f is 0 outside the 2-dimensional unit
simplex U2 , the distribution is really a distribution living on the unit simplex. ⋄
Exercises
Exercise 2.13.1. Consider the pair of random variables (X, Y ) as in Example 2.13.3
in the notes. Thus (X, Y ) denotes the pair of aligned nucleotides in an alignment
of two DNA-sequences and the joint distribution of this pair is given by the point
probabilities
A C G T
A 0.12 0.03 0.04 0.01
C 0.02 0.27 0.02 0.06
G 0.02 0.01 0.17 0.02
T 0.05 0.03 0.01 0.12
• Compute the point probabilties for the marginal distributions, P1 and P2 , of

the variables X and Y respectively.
• Compute the point probabilities for the distribution, P , that make X and Y
independent with marginal distribution P1 and P2 respectively.
• Compute the probability under P as given above for the event X = Y . Compare
with the probability for this event as computed in Example 2.13.3.
2.13.1 Conditional distributions and conditional densities
If X and Y are real valued random variables with joint distribution P on R2 having
density f , then the probability of X being equal to x is 0 for all x ∈ R. Thus we
cannot use Definition 2.10.8 to find the conditional distribution of Y given X = x.
It is, however, possible to define a conditional distribution in a sensible way as long
as f1 (x) > 0 where f1 denotes the density for the marginal distribution of X. This
is even possible if we consider not just two real valued random variables, but in fact
if we consider two random variables X and Y with values in Rn and Rm such that
the joint distribution P on Rn+m has density f .
Definition 2.13.9 (Conditional densities). If f is the density for the joint distri-
bution of two random variables X and Y taking values in Rn and Rm , respectively,
then with
Z
f1 (x) = f (x, y)dy
Rm
we define the conditional distribution of Y given X = x to be the distribution with

density
f (x, y)
f (y|x) = .
f1 (x)
for all y ∈ Rm and x ∈ Rn with f1 (x) > 0.
Note that from this definition we find the formula
f (x, y) = f (y|x)f1 (x), (2.33)
which reads that the density for the joint distribution is the product of the densities
for the marginal distribution of X and the conditional distribution of Y given X.
Note the analogy to (2.21) and especially (2.23) for point probabilities. It is necessary
to check that the definition really makes sense, that is, that f (y|x) actually defines
a density for all x ∈ R with f1 (x) > 0. It is positive by definition, and we also see
that
Z R
m f (x, y)dy
f (y|x)dy = R =1
Rm f1 (x)
by (2.31).
If we let Px denote the probability measure with density y 7→ f (y|x) for given x ∈ Rn
with f1 (x) > 0, i.e. the conditional distribution of Y given X = x, then for B ⊆ Rm ,
Z
P(Y ∈ B|X = x) = Px (B) = f (y|x)dy.
B
If moreover A ⊆ Rn
Z Z
P(X ∈ A, Y ∈ B) = P (A × B) = f (x, y)dydx
ZA ZB
= f (y|x)f1 (x)dydx
A B
Z Z
= f (y|x)dy f1 (x)dx
ZA B
= Px (B)f1 (x)dx.
A
The interpretation is that the realization of X and Y can always be thought of

as a two step procedure. First X is realized according to the probability measure
with density f1 , and we determine whether X belongs to the event A or not (the
outer integration over A). Then conditionally on X = x the realization of Y is given
according to the conditional probability measure Px .
A common usage of conditional densities is to define multivariate probability mea-
sures via (2.33). Thus if we are given a marginal density and a conditional density,
then we can define f by (2.33). By successive integration, first over y then over x,
we see that this defines a density on Rn+m .
Example 2.13.10 (Linear regression). Let X be a real valued random variable with
distribution N (0, σ12 ). Define
Y = α + βX + ε (2.34)
where ε ∼ N (0, σ22 ) is independent of X. We read this equation as a structural equa-

tion, see Example 2.10.12 that defines Y as follows. Given X = x the conditional
distribution of Y equals the distribution of α + βx + ε. This is a location transforma-
tion of the normal distribution, thus the conditional distribution of Y given X = x
is N (α + βx, σ22 ). We can then use (2.33) to compute the joint distribution of (X, Y )
as

1 (y − α − βx)2 x2
f (x, y) = exp − −
2πσ1 σ2 2σ 2 2σ12
2 22
1 1 σ1 β + σ22 2 β 1 2
= exp − x − 2 x(y − α) + (y − α) (2.35)
2πσ1 σ2 2 σ12 σ22 σ22 σ22
One alternative interpretation of the defining equation (2.34) of Y is that Y is a

transformation of X and ε. In Exercise 2.13.5 it is shown that this interpretation
leads to the same result. That is, the same joint distribution of X and Y . This is
fortunate as it shows that the distributional interpretation of equation (2.34) is not
ambiguous. ⋄
Exercises
Exercise 2.13.2. In this exercise we consider a dataset of amino acid pairs (x, y).
We think of the data as representing the outcome of a random variable (X, Y ). Here
X and Y represent an amino acid at a given position in two evolutionary related
proteins (same protein, but from two different species, say). The dataset may be
obtained from a (multiple) alignment of (fractions) of proteins. The only mutational
event is substitution of one amino acid for another. In this exercise you are allowed
to think of the different positions as independent, but X and Y are dependent.
• Load the aa dataset into R with data(aa) and use table to cross-tabulate
the data according to the values of x and y. Compute the matrix of relative
frequencies for the occurrence of all pairs (x, y) for (x, y) ∈ E0 × E0 where E0
denotes the amino acid alphabet.
Exercise 2.13.3. Continuing with the setup from the previous exercise we denote
by P the probability measure with point probabilities, p(x, y), being the relative
frequencies computed above. It is a probability measure on E0 × E0
• Compute the point probabilities, p1 (x) and p2 (y), for the marginal distributions
of P and show that X and Y are not independent.
• Compute the score matrix defined as
p(x, y)
Sx,y = log .
p1 (x)p2 (y)
Discuss the interpretation of the values.

• If (X, Y ) have distribution P , SX,Y can be regarded as a random variable with
values in a finite sample space (why?). Compute the mean of SX,Y .
• Assume instead that X and Y are in fact independent with distributions given
by the point probabilities p1 (x) and p2 (y) respectively and compute then the
mean of SX,Y .
⋆
Exercise 2.13.4. Show that if the distribution of (X, Y ) is given by the density f
in (2.35) then the marginal distribution of Y is N (α, β 2 σ12 + σ22 ). Then show that X
and Y are independent if and only if β = 0.
Hint: See Example 2.13.6, formula (2.32).
⋆⋆
Exercise 2.13.5. Let X and ε be independent real valued random variables with
distribution N (0, σ12 ) and N (0, σ22 ) respectively. Show that the density for their joint
distribution is
1 x2 ε2
f (x, ε) = exp − 2 − 2 .
2πσ1 σ2 2σ1 2σ2
Define the transformation
h : R2 → R
by h(x, ε) = (x, α + βy + ε) and find the density for the distribution of h(X, ε).
Compare with Example 2.13.10.
Hint: First find h−1 (I × J) for I, J ⊆ R. You may think of these sets as intervals.
Then use the definitions of transformed distributions and densities to compute
P(X ∈ I, α + βX + ε ∈ J) = P((X, ε) ∈ h−1 (I × J)).
Multivariate datasets from the sample space Rd with d ≥ 2 are more difficult to
visualize than a one-dimensional dataset. It is the same problem as we have with
tabulations. Two-dimensional tables are basically the limit for our comprehension.
Bivariate datasets (d = 2) can also be visualized for instance via scatter plots and
bivariate kernel density estimation, but for multivariate datasets we usually have to
rely on visualizing univariate or bivariate transformations.
If xi = (xi1 , xi2 ) ∈ R2 is bivariate a simple plot of xi2 against xi1 for i = 1, . . . , n
is called a scatter plot. It corresponds to a one-dimensional rug plot, but the two
dimensions actually gives us a better visualization of the data in this case. The scatter
plot may highlight some simple dependence structures in the data. For instance, if
the conditional distribution of Xi2 given Xi1 = xi1 is N (α + βxi1 , σ22 ) as in Example
2.13.10 with α, β ∈ R, then this will show up on the scatter plot (if β 6= 0) as a
tendency for the points to lie around a line with slope β. How obvious this is depends
on how large σ22 is compared to β and how spread out the distribution of Xi1 is.
Example 2.14.1. We simulate n = 100 bivariate iid random variables (Xi1 , Xi2 )
for i = 1, . . . , 100 where the distribution of Xi1 ∼ N (0, 1) and the conditional dis-
tribution of Xi2 given Xi1 = xi1 is N (βxi1 , 1) with β = 0, 0.5, 1, 2. The resulting
scatter plots are shown in Figure 2.23. The dependence between Xi1 and Xi2 is de-
termined by β, and as shown in Exercise 2.13.4, the variables are independent if and
only if β = 0. From the figure we observe how the dependence through β influences
the scatter plot. As β becomes larger, the points clump around a line with slope β.
For small β, in this case β = 0.5 is quite small, the scatter plot is not so easy to
distinguish from the scatter plot with β = 0 as compared to the larger values of β.
Thus weak dependence is not so easy to spot by eye. ⋄
Example 2.14.2 (Iris data). A classical dataset that is found in numerous textbooks
contains characteristic measurements of the Iris flower for three different species, Iris
β=0 β = 0.5
2
1
1
0
0
−1
−1
−2
−2
−2 −1 0 1 2 −2 −1 0 1 2
β=1 β=2
2
2
1
1
0
0
−1
−1
−2
−2
−2 −1 0 1 2 −2 −1 0 1 2
Figure 2.23: Scatter plot of 100 simulated variables as described in Example 2.14.1.
The variables have a linear dependence structure determined by the parameter β
that ranges from 0 to 2 in these plots.
Setosa, Iris Virginica and Iris Versicolor. The data were collected in 1935 by Edgar
Anderson, and is today available in R via data(iris). The date are organized in a
dataframe as follows
Sepal.Length Sepal.Width Petal.Length Petal.Width Species

5.1 3.5 1.4 0.2 setosa
4.9 3.0 1.4 0.2 setosa
4.7 3.2 1.3 0.2 setosa
4.6 3.1 1.5 0.2 setosa
5.0 3.6 1.4 0.2 setosa
5.4 3.9 1.7 0.4 setosa
4.6 3.4 1.4 0.3 setosa
5.0 3.4 1.5 0.2 setosa
4.4 2.9 1.4 0.2 setosa
4.9 3.1 1.5 0.1 setosa
.. .. .. .. ..
. . . . .
There are 50 measurements for each species, thus a total of 150 rows in the data
frame. In Figure 2.24 we see two-dimensional scatter plots of the different measure-
ments against each other for the species Setosa. Restricting our attention to the
two variables Petal length and Petal width, we see in Figure 2.25 scatter plots, two-
Transition probabilities 111
2.5 3.0 3.5 4.0 0.1 0.2 0.3 0.4 0.5 0.6
5.5
Sepal.Length
5.0
4.5
4.0
3.5
3.0
2.5 Sepal.Width
1.8
1.6
Petal.Length
1.4
1.2
1.0
0.6
0.5
0.4
Petal.Width
0.3
0.2
0.1
4.5 5.0 5.5 1.0 1.2 1.4 1.6 1.8
Figure 2.24: Considering the species Setosa in the Iris dataset we see a bivariate
scatter plot of all the pair-wise combinations of the four variables in the dataset.
This is obtained in R simply by calling plot for a dataframe containing the relevant
variables.
dimensional kernel density estimates and corresponding contour curves. The kernel
density estimates are produced by kde2d from the MASS package. ⋄
R Box 2.14.1 (Bivariate kernel density estimation). The package MASS

contains a function, kde2d, for bivariate kernel density estimation. It uses a
bivariate Gaussian kernel, where you can change the bandwidth along each
of the coordinate axes separately. It returns the evaluation of the kernel
density estimate in a quadratic grid. You can use the plotting functions
persp, contour and filled.contour to visualize the estimated density.
2.15 Transition probabilities
Recall that if we have two random variables X and Y taking values in the discrete
sample space E we have the relation
P(X = x, Y = y) = P(Y = y|X = x)P(X = x).

Setosa Versicolor
0.8 3.0 2.0 1.4
1.2
2.5 1.8
0.6
1.0
2.0 1.6
0.8
Petal width
Petal width
0.4 1.5 1.4
0.6
1.0 1.2
0.4
0.2
0.5 1.0
0.2
0.0 0.0 0.8 0.0

0.5 1.0 1.5 2.0 2.5 2.5 3.0 3.5 4.0 4.5 5.0 5.5
Petal length Petal length
2.5
2.0 1.0
1.5
1.0 0.5
0.5
0.5 0.8 2.5 2.0

3.0 1.8
1.0 0.6
3.5 1.6
1.5 0.4 4.0 1.4
4.5 1.2
2.0 0.2
5.0 1.0
2.5 0.0 5.5 0.8
Figure 2.25: For the Iris Setosa and Iris Versicolor we have estimated a two-
dimensional density for the simultaneous distribution of petal length and width.
In both cases we used a bivariate Gaussian kernel with bandwidth 0.8 in both di-
rections.
If we think about the two step interpretation of the joint distribution the relation
above says that the joint distribution is decomposed into a first step where X is
sampled according to its marginal distribution and then Y is sampled conditionally
on X. With such a dynamic interpretation of the sampling process it is sensible to
call P(Y = y|X = x) the transition probabilities. Note the the entire dependence
structure between X and Y is captured by the conditional distribution, and X and
Y are independent if P(Y = y|X = x) = P(Y = y) for all y ∈ E.
It is possible to specify whole families of probability measures on E × E where E is a
finite sample space, e.g. E = {A, C, G, T}, which are indexed by a time parameter t ≥ 0
and perhaps some additional parameters that capture the dependence structure.
This is done by specifying the transition probabilities via a system of differential
equations. We introduce Pt as the distribution of the pair of random variables (X, Y )
on E × E indexed by the time parameter t ≥ 0. We are going to assume that the
marginal distribution of X, which is given by the point probabilities
X
π(x) = Pt (x, y),
y∈E
does not depend on t, and we are going to define the matrix P t for each t ≥ 0 by
Pt (x, y)
P t (x, y) = .
π(x)
That is, P t (x, y) is the conditional probability that x changes into y over the time
interval t (note that we may have x = y). There is a natural initial condition on P t
for t = 0, since for t = 0 we will assume that X = Y , thus

0 1 if x = y
P (x, y) =
0 otherwise
This is the only reasonable assumption if t is to be interpreted as a time parameter.

With this initial condition we specify the matrix of conditional probabilities by the
differential equation
dP t (x, y) X t
= P (x, z)λ(z, y) (2.36)
dt
z∈E
for x, y ∈ E and (λ(z, y))z,y∈E a matrix of mutation intensities satisfying λ(z, y) ≥ 0

for z 6= y and X
λ(z, z) = − λ(z, y). (2.37)
y6=z
Thus to specify Pt for all t ≥ 0 by (2.36) we need only to specify a single matrix
of intensities. In a very short time interval, ∆, the interpretation of the differential
equation is that
P t+∆ (x, y) − P t (x, y) X

≃ P t (x, z)λ(z, y)
∆
z∈E
X
= P t (x, y)λ(y, y) + P t (x, z)λ(z, y).
z6=y
Rearranging yields
X
P t+∆ (x, y) ≃ P t (x, y)(1 + ∆λ(y, y)) + P t (x, z)∆λ(z, y).
z6=y
This equation reads that the conditional probability that x changes into y in the
time interval t + ∆ is given as a sum of two terms. The first term is the probability
P t (x, y) that x changes into y in the time interval t times the factor 1 + ∆λ(y, y).
This factor has the interpretation as the probability that y doesn’t mutate in the
short time interval ∆. The second term is a sum of terms where x changes into some
z 6= y in the time interval t and then in the short time interval ∆ it changes from z
to y with probability ∆λ(z, y). In other words, for z 6= y the entries in the intensity
matrix have the interpretation that ∆λ(z, y) is approximately the probability that
z changes into y in the short time interval ∆, and 1 + ∆λ(y, y) is approximately the
probability that y doesn’t change in the short time interval ∆.
In Exercises 2.15.1 and 2.15.2 it is verified, as a consequence of λ(x, y) ≥ 0 for x 6= y
and (2.37), that the solution to the system of differential equations (2.36) is indeed
a matrix of conditional probabilities, that is, all entries of P t are ≥ 0 and the row
sums are always equal to 1.
It is a consequence of the general theory for systems of linear differential equations
that there exists a unique solution to (2.36) with the given initial condition. In
general the solution to (2.36) can, however, not easily be given a simple analytic
representation, unless one accepts the exponential of a matrix as a simple analytic
expression; see Math Box 2.15.1. There we also show that the solution satisfies the
Chapman-Kolmogorov equations; for s, t ≥ 0 and x, y ∈ E
X
P t+s (x, y) = P t (x, z)P s (z, y). (2.38)
z∈E
For some special models we are capable of obtaining closed form expressions for
the solution. This is for instance the case for the classical examples from molecular
evolution with E = {A, C, G, T} that lead to the Jukes-Cantor model and the Kimura
model introduced earlier via their the conditional probabilities.
Example 2.15.1 (The Jukes-Cantor model). The Jukes-Cantor model is given by
assuming that
−3α if x = y
λ(x, y) =
α if x 6= y
That is, the matrix of intensities is
A C G T
A −3α α α α
C α −3α α α
G α α −3α α
T α α α −3α
The parameter α > 0 tells how many mutations that occur per time unit. The
solution to (2.36) is
P t (x, x) = 0.25 + 0.75 × exp(−4αt)

P t (x, y) = 0.25 − 0.25 × exp(−4αt), if x 6= y,
Math Box 2.15.1 (Transition probability matrices). With E = {x1 , . . . , xn } de-

fine the matrix
 
λ(x1 , x1 ) λ(x1 , x2 ) . . . λ(x1 , xn )
 λ(x2 , x1 ) λ(x2 , x2 ) . . . λ(x2 , xn ) 
 
Λ= .. .. .. ,
 . . . 
λ(xn , x1 ) λ(xn , x2 ) . . . λ(xn , xn )
then the set of differential equations defined by (2.36) can be expressed in matrix
notation as
dP t
= P t Λ.
dt
If Λ is a real number, this differential equation is solved by P t = exp(tΛ), and this
solution is also the correct solution for matrix Λ, although it requires a little more
work to understand how the exponential function works on matrices. One can take
as a definition of the exponential function the usual Taylor expansion
X∞ n n
t Λ
exp(tΛ) = .
n=0
n!
It is possible to verify that this infinite sum makes sense. Moreover, one can simply
differentiate w.r.t. the time parameter t term by term to obtain
X∞
d tn−1 Λn
exp(tΛ) = n
dt n=1
n!
∞ n−1 n−1
!
X t Λ
= Λ
n=1
(n − 1)!
∞ n n
!
X t Λ
= = exp(tΛ)Λ.
n=0
n!
Many of the usual properties of the exponential function carry over to the expo-
nential of matrices. For instance, exp(tΛ + sΛ) = exp(tΛ) exp(sΛ), which shows
that
P t+s = P t P s .
This is the Chapman-Kolmogorov equations, (2.38), in their matrix version.
which is verified by checking that P t fulfills the differential equation. ⋄
Example 2.15.2 (The Kimura model). Another slightly more complicated model
defined in terms of the differential equations (2.36), that admits a relatively nice
closed form solution, is the Kimura model. Here the intensity matrix is assumed to
be
A C G T
A −α − 2β β α β
C β −α − 2β β α
G α β −α − 2β β
T β α β −α − 2β
with α, β > 0. The interpretation is that a substitution of a purine with a purine or a
pyrimidine with a pyrimidine (a transition) is happening with another intensity than
a substitution of a purine with pyrimidine or pyrimidine with purine (a transversion).
We see that the intensity for transversions is λ(x, y) = β and the intensity for
transitions is λ(x, y) = α. The solution is

P t (x, y) = 0.25 + 0.25 exp(−4βt) − 0.5 exp(−2(α + β)t), if λ(x, y) = α
t
P (x, y) = 0.25 − 0.25 exp(−4βt), if λ(x, y) = β,
which is verified by checking that the system of differential equations is fulfilled. It

is tedious but in principle straight forward. ⋄
Exercises
⋆
Exercise 2.15.1. Let P t (x, y) for x, y ∈ E satisfy the system of differential equations
dP t (x, y) X
= P t (x, z)λ(z, y)
dt
z∈E
P
with λ(x, y) ≥ 0 for x 6= y and λ(x, x) = − y6=y λ(x, y). Define
X
sx (t) = P t (x, y)
y
as the “row sums” of P t for each x ∈ E. Show that dsx (t)/dt = 0 for t ≥ 0. Argue
that with the initial condition, P 0 (x, y) = 0, x 6= y and P 0 (x, x) = 1, then sx (0) = 1
and conclude that sx (t) = 1 for all t ≥ 0.
⋆⋆
Exercise 2.15.2. We consider the same system of differential equations as above
with the same initial condition. Assume that λ(x, y) > 0 for all x 6= y. Use the sign
of the resulting derivative
dP 0 (x, y)
= λ(x, y)
dt
at 0 to argue that for a sufficiently small ε > 0, P t (x, y) ≥ 0 for t ∈ [0, ε]. Use this fact
together with the Chapman-Kolmogorov equations, (2.38), to show that P t (x, y) ≥ 0
for all t ≥ 0.
3
Statistical models and inference
Given one or more realizations of an experiment with sample space E, which prob-
ability measure P on E models – or describes – the experiment adequately? This is
the crux of statistics – the inference of a suitable probability measure or aspects of
a probability measure from empirical data. What we consider in the present chapter
is the problem of estimation of one or more parameters that characterize the prob-
ability measure with the main focus on using the maximum likelihood estimator.
3.1 Statistical Modeling
A statistical model is a collection of probability measures on the sample space E.

Before we can estimate the probability measure – the unknown parameters – we need
to decide upon which class of probability measures we believe are reasonable models
of the data that we want to model. In other words, we need to set up the framework
that we will be working within. In different contexts there are different ways to
specify the statistical model, but the goal is to find a suitable class of probability
measures. We give four examples of different scientific problems and corresponding
datasets, which give rise to four different ways of specifying a statistical model.
Example 3.1.1 (Neurons). Neuron cells generate, as explained in Example 1.2.1,
electrical signals, and an observable quantity is the time between such signals – the
so-called interspike times. The activity of the neuron – that is, the rate by which
the electrical signals are fired – depends on a range of different things including the
input signals from other neuron cells, and the whole system is influenced by external
stimuli. The activity of auditory neuron cells is, for instance, influenced by sound.
In this example we consider a dataset with 312 measurements of the interspike times
in seconds for an auditory neuron cell. The measurements are taken in a steady state
situation where there is no external stimuli to influence the activity of the cell.
117
118 Statistical models and inference
Histogram of neuron
0.8
0.6
Density
0.4
0.2
0.0
0 1 2 3 4 5
neuron
Figure 3.1: The histogram of the neuron interspike times.
Our model assumptions will be that the n = 312 measurements, x1 , . . . , x312 , of

interspike times are the realization of 312 iid positive, real valued random variables
X1 , . . . , X312 and the objective is to estimate the common distribution of the inter-
spike times.
Figure 3.1 shows a histogram of the data. The histogram resembles the density for the
exponential distribution, so we will restrict our attention further to the model where
the common distribution is an exponential distribution with density λ exp(−λx) for
x ≥ 0 and an unknown parameter λ > 0.
We know that the theoretical mean of the exponential distribution with parameter
λ > 0 is 1/λ, and we can obtain an ad hoc estimate of λ by equating the theoretical
mean equal to the empirical mean and solve for λ. That is, we get the estimate
312
λ̂ = P312 = 1.147.
i=1 xi
Figure 3.2 shows a QQ-plot of the observations against the quantiles for the expo-
nential distribution, which seems to confirm that the exponential distribution is a
suitable choice.
A more careful analysis will, however, reveal that there is a small problem. The
problem can be observed by scrutinizing the rug plot on the histogram, which shows
that there is a clear gap from 0 to the smallest observations. For the exponential
distribution one will find a gap of roughly the same order of magnitude as the
Statistical Modeling 119
5
4
3
neuron
2
1
0
0 1 2 3 4 5 6
qexp(pp)
Figure 3.2: The QQ-plot of the neuron interspike times against the quantiles for the
exponential distribution.
distances between the smallest observations – for the dataset the gap is considerably
larger.
It is possible to take this into account in the model by including an extra, unknown
parameter µ ≥ 0 such that Xi −µ is exponentially distributed with parameter λ > 0.
This means that Xi has a distribution with density
fµ,λ (x) = λ exp(−λ(x − µ)), for x ≥ µ.
The exponential distribution with parameter λ is a scale transformation with scale

parameter σ = 1/λ of the exponential distribution with λ = 1. The extended model
above is seen to be a scale-location transformation with scale parameter σ = 1/λ
and location parameter µ. At this point we will not pursue any ad hoc methods
for parameter estimation in the extended two-parameter model, but we will make
two remarks. First, the points in the QQ-plot seem to fall on a straight line, which
confirms the choice of the exponential distribution – but only up to a scale-location
transformation. Second, a location transformation of the exponential distribution,
or any other distribution living on the positive half-line for that matter, does not
give a particularly pleasant model.
Let us elaborate. If µ > 0 the model dictates that interspike times are all greater
than µ. It is hardly conceivable that a single dataset can provide evidence for a fixed,
absolute lower bound > 0 on the interspike times let alone provide a sufficiently good
estimate should it be the case that such a bound exists. It is well documented in
the literature that after a spike there is a refractory period where the cell cannot
fire, which can explain why we see the gap between 0 and the smallest observations.
However, if we compute an estimate µ̂ > 0 based on the dataset, we obtain a resulting
model where a future observation of an interspike time smaller than µ̂ is in direct
conflict with the model. Unless we have physiological substantive knowledge that
supports an absolute lower bound we must be skeptical about an estimate. There is
in fact evidence for a lower bound, whose value is of the order of one millisecond – too
small to explain the gap seen in the data. In conclusion, it is desirable to construct a
refined model, which can explain the gap and the exponential-like behavior without
introducing an absolute lower bound. ⋄
Example 3.1.2 (Evolutionary Distance). In this example we consider a dataset
(x1 , y1 ), . . . , (xn , yn )
consisting of n pairs of aligned nucleotides from two homologous DNA-sequences.

The sample space we consider is therefore E = {A, C, G, T} × {A, C, G, T}. We will
assume that the only evolutionary event that occurs is substitution and that the pairs
of aligned variables are realizations of n iid random variables (X1 , Y1 ), . . . , (Xn , Yn ).
In this case we need only to specify a family of probability measures on E.
We have in Section 2.10 introduced two examples of probability measures for such
dependent pairs of nucleotides indexed by a time parameter; the Jukes-Cantor model
and the Kimura model. One of the parameters that enter in all these models is the
time parameter, and we will need to distinguish between two different situations.
We may actually know the evolutionary distance in calendar time between the two
homologous sequences, like for the Hepatitis C dataset considered in Example 1.2.4.
This situation is encountered when we observe an evolving organism, like a virus,
over time. In many applications, for instance in phylogenetics, where we want to
trace back the evolution at the molecular level from present sequences, we do not
know the evolutionary distance in calendar time. Indeed, one of the main objectives
is in this case to estimate the time to a common forefather – at least relatively to
other sequences. In such a case the time parameter enters on an equal footing with
other unknown parameters.
If we consider the Jukes-Cantor model from Example 2.10.10, the conditional prob-
abilities for the second (y) nucleotide given the first (x) are for t, α > 0 given by the
formulas
P t (x, x) = 0.25 + 0.75 × exp(−4αt)

P t (x, y) = 0.25 − 0.25 × exp(−4αt), if x 6= y.
We also need to specify the marginal distribution of the first x-nucleotide. We may
here simply take an arbitrary probability measure on {A, C, G, T} given by point
probabilities (p(A), p(C), p(G), p(T)). This specifies the full distribution of (X1 , Y1 ) by
the point probabilities

p(x)(0.25 + 0.75 × exp(−4αt)) if x = y
P(X1 = x, Y1 = y) = p(x)P t (x, y) = .
p(x)(0.25 − 0.25 × exp(−4αt)) 6 y
if x =
The point probabilities using the Kimura model from Example 3.3.18 are given the
same way just by replacing the conditional probabilities by those from Example
3.3.18, which also introduces the extra parameter β > 0.
If we dwell for a moment on the interpretation of the Jukes-Cantor model, the
conditional probability P t (x, x) is the probability that the a nucleotide does not
change over the time period considered. For the hepatitis C virus dataset considered
in Example 1.2.4, Table 1.2 shows for instance that out of the 2610 nucleotides in
segment A there are 78 that have mutated over the period of 13 years leaving 2532
unchanged. With reference to the frequency interpretation we can try to estimate α
in the Jukes-Cantor model by equation the formula for P t (x, x) equal to 2532/2610.
This gives
log(( 2532 1 4
2610 − 4 ) 3 )
α̂ = − = 7.8 × 10−4 .
4 × 13
⋄
Example 3.1.3 (Forensic Statistics). We want to construct a model of the occur-

rence of repeat counts for STRs, as used in forensics, in a given population; see
Example 1.2.3. For simplicity, we assume first that we only consider a single short
tandem repeat, which we take to be TH01. The fundamental repeat pattern for TH01
is AATG, and our experiment consists of obtaining a sample from the population and
then counting the number of times this pattern is repeated at a particular position
in the genome for the individuals in the sample. We have to remember that since
humans are diploid organisms, we get a pair of counts for each tandem repeat. Since
we cannot distinguish between which of the chromosomes the repeats are located
on, the observations are naturally given as unordered pairs of counts. Observing the
counts 7 and 8, say, from an individual is the same as observing the counts 8 and 7.
We make the convention that we will report the counts with the smallest first. Our
sample space is
E = {(x, y) | x ≤ y, x, y ∈ A}
where A denotes m possible repeat counts observable for TH01. The situation is
the same as with genes where we may have two alleles and we can observe only
three allele combinations. Here the number of different alleles is m and we can
observe m(m + 1)/2 different allele combinations. For TH01 we can take A =
{5, 6, 7, 8, 9, 9.3, 10, 11} for the purpose of this example. We note that 9.3 is also
a possibility, which corresponds to a special variant of the repeat pattern with 9 full
occurrences of the fundamental repeat pattern AATG but with the partial pattern
ATG between the 6th and 7th occurrence.
y
5 6 7 8 9 9.3 10 11
5 0 0 1 0 0 0 0 0
6 19 31 7 16 48 0 0
7 10 9 11 43 0 0
x 8 3 9 17 3 0
9 6 19 1 1
9.3 47 1 0
10 0 0
11 0
Table 3.1: Tabulation of the repeat counts for the short tandem repeat TH01 for the
Caucasian population included in the NIST STR dataset.
For the NIST dataset we can tabulate the occurrences of the different repeat counts
for TH01. Table 3.1 shows the tabulation for the Caucasian population in the dataset.
The dataset also includes data for Hispanics and African-Americans.
To build a model we regard the observations above as being realizations of iid random
variables all with the same distribution as a pair (X, Y ) that takes values in E. The
full model – sometimes called the saturated model – is given as the family of all
probability measures on E. The set of point probabilities
X
Θ = {(p(x, y))(x,y)∈E | p(x, y) = 1}
x,y
on E provides a parametrization of all the probability measures. It is always possible

in principle to work with the full model on a finite sample space – but the number of
parameters easily grows out of control – there is m(m + 1)/2 parameters in general
and 36 for the TH01 case above1 . One possible, simplifying assumption is that the
distribution is given by the Hardy-Weinberg equilibrium. This assumptions enforces
that there is a probability measure P0 on A with point probabilities p0 such that
the point probabilities for the distribution of (X, Y ) is

2p0 (x)p0 (y) if x 6= y
p(x, y) = 2
p0 (x) if x = y
This will bring us down to a model with only 8 parameters. It is an important

question if such a simplifying assumption on the model is appropriate – or if the
data at hand are actually in conflict with the model.
Another important question is what to do when we consider several short tandem
repeats. If we in addition to TH01 consider TPOX, say, we can have another model
for the marginal distribution of the repeat counts (Z, W ). A table corresponding
1
Since the point probabilities sum to 1 there are only m(m+1)/2-1 free parameters
w
5 6 8 9 10 11 12 13
5 0 0 1 0 0 0 0 0
6 0 0 1 0 0 0 0
8 82 45 20 78 15 0
z 9 3 2 16 2 0
10 0 10 2 0
11 19 5 0
12 0 1
13 0
Table 3.2: Tabulation of the repeat counts for the short tandem repeat TPOX for
the Caucasian population included in the NIST STR dataset.
to the TPOX counts in the NIST dataset for the Caucasian population is found
in table 3.2. The question is whether we can assume that (X, Y ) and (Z, W ) are
independent? ⋄
Example 3.1.4 (Gene expression). We have in Example ?? considered data from

the ALL gene expression dataset. There are thousands of genes whose expression
levels are measured for different individuals giving rise to a huge dataset. Additional
information about the individuals, who all suffered from leukemia, is also collected
and one scientific objective is to identify genes for different groups of individuals
that are systematically differently expressed between the two groups. One especially
interesting grouping is to divide the individuals into two groups according to the
BCR/ABL fusion gene. Some of the individuals have this fusion gene, which is a
join of two genes – the BCR and ABL genes – which is a result of a large scale
mutation. The gene is well known to be an oncogene, that is, a gene associated with
cancer, and in this particular case with leukemia.
The objective is indeed a challenge. The technology behind gene expression data is
described in more detail in Example 1.2.9 and there are several sources of variation
that does not make it straight forward to detect such genes in the two groups by
eye. In this example we will focus on a less ambitious goal, which is just to establish
a suitable model of the gene expression measurement for a single gene. Our favorite
gene is the one with the probe set label 1635 at. There is in total 79 individuals
in the dataset, which we divide into two groups, which we call group 1 (BCR/ABL
present) and group 2 (BCR/ABL not present) with 37 and 42 individuals in the two
groups, respectively.
The basic model assumptions will be that the our observations,
x1,1 , . . . , x1,37
x2,1 , . . . , x2,42
are the realization of 79 independent random variables
X1,1 , . . . , X1,37
X2,1 , . . . , X2,42 .
However, we will not insist that all the variables have the same distribution. On the
contrary, the purpose is to figure out if there is a difference between the two groups.
We will, however, assume that within either of the two groups the variables do have
the same distribution.
The measurement of the gene expression level can be viewed as containing two com-
ponents. One component is the signal, which is the deterministic component of the
measurement, and the other component is the noise, which is a random component.
In the microarray setup we measure a light intensity as a distorted representation of
the true concentration of a given RNA-molecule in the cell. If we could get rid of all
experimental uncertainty and, on top of that, the biological variation what would
remain is the raw signal – an undistorted measurement of the expression level of a
particular RNA-molecule under the given experimental circumstances. Even if the
technical conditions were perfect giving rise to no noise at all, the biological vari-
ation would still remain. Indeed, this variation can also be regarded as part of the
insight into and understanding of the function of the biological cell. The bottom line
is that we have to deal with the signal as well as the noise. We are going to consider
two modeling paradigms for this signal and noise point of view; the additive and the
multiplicative noise model.
Let X denote a random variable representing the light intensity measurement of our
favorite gene. The additive noise model says that
X = µ + σε (3.1)
where µ ∈ R, σ > 0 and ε is a real valued random variable whose distribution has
mean value 0 and variance 1. This is simply a scale-location model and it follows from
Example 2.9.9 that the mean value of the distribution of X is µ and the variance
is σ 2 . Thus µ is the mean expression of the gene – the signal – and σεi captures
the noise that makes X fluctuate around µ. It is called the additive noise model
for the simple fact that the noise is added to the expected value µ of X. It is quite
common to assume in the additive noise model that the distribution of ε is N (0, 1).
The model of X is thus N (µ, σ 2 ), where µ and σ are the parameters. The parameters
(µ, σ) can take any value in the parameter space R × (0, ∞).
For the multiplicative noise model we assume instead that
X = µε (3.2)
with ε a positive random variable. We will in this case insist upon µ > 0 also. If
the distribution of ε has mean value 1 and variance σ 2 it follows from Example
2.9.9 that the distribution of X has mean value µ and variance µ2 σ 2 . If we specify
the distribution of ε this is again a two-parameter model just as the additive noise
model. The most important difference from the additive noise is that the standard
deviation of X is µσ, which scales with the mean value µ. In words, the noise gets
larger when the signal gets larger.
The formulation of the multiplicative noise model in terms of mean and variance
is not so convenient. It is, however, easy to transform from the multiplicative noise
model to the additive noise model by taking logarithms. Since X = µε then
log X = log µ + log ε,
which is an additive noise model for log X. If one finds a multiplicative noise model
most suitable it is quite common to transform the problem into an additive noise
model by taking logarithms. Formally, there is a minor coherence problem because
if ε has mean 1 then log ε does not have mean value 0 – it can be shown that it will
always have mean value smaller than 0. If we on the other hand assume that log ε
has mean value 0 it holds that ε will have mean value greater than 1. However, this
does not interfere with the fact that if the multiplicative noise model is a suitable
model for X then the additive noise model is suitable for log X and vice versa – one
should just remember that if the mean of X is µ then the mean of log X is not equal
to log µ but it is in fact smaller than log µ.
Returning to our concrete dataset we choose to take logarithms (base 2), we assume
an additive noise model for the logarithms with error distribution having mean 0 and
variance 1, and we compute the ad hoc estimates of µ (the mean of log X) and σ 2
(the variance of log X) within the groups as the empirical means and the empirical
variances.
group 1 group 2
1 P37 1 P42
µ̂1 = 37 i=1 log x1,i = 8.54 µ̂2 = 42 i=1 log x2,i = 7.33
1 P37 1 P42
σ̃1 = 37 i=1 (log x1,i − µ̂1 )2 = 0.659
2 σ̃2 = 42 i=1 (log x2,i − µ̂2 )2 = 0.404
2
To deal with our objective – is there a difference between the groups – it is of

relevance to compute the difference of the estimated means
µ̂1 − µ̂2 = 1.20.
However, the size of this number does in reality not tell us anything. We have to
interpret the number relatively to the size of the noise as estimated by σ̃12 and σ̃22 .
The formalities are pursued later.
One remark is appropriate. If we don’t take logarithms the empirical means
37 42
1 X 1 X
µ̃1 = x1,i = 426.6, µ̃2 = x2,i = 179.1
37 42
i=1 i=1
are reasonable estimates of the means (of X) within the two groups. For the multi-
plicative mode it is best to compute the ratio
µ̃1
= 2.38
µ̃2
for comparing the mean values instead of the difference. Again any interpretation of
this number needs to be relative to the size of the noise. The downside of using ad
hoc estimates and ad hoc approaches shows it face here, because µ̂1 6= log µ̃1 we have
two procedures. Either we take logarithms, compute the means and compute their
difference or we compute the means, take logarithms and compute their difference
and we end up with two different results. Which is the most appropriate? The model
based, likelihood methods introduced later will resolve the problem. ⋄
All four examples above specify a family of probability distributions on the relevant
sample space by a set of parameters. For the exponential distribution used as a
model of the interspike times the parameter is the intensity parameter λ > 0. For
the additive model (or multiplicative in the log-transformed disguise) the parameters
for the gene expression level are µ ∈ R and σ > 0. For the model in the forensic
repeat counting example the parameters are the point probabilities on the finite
sample space E, or if we assume Hardy-Weinberg equilibrium the point probabilities
on the set A. For the Jukes-Cantor model the parameters are the marginal point
probabilities for the first nucleotide and then α > 0 and time t > 0 that determines
the (mutation) dependencies between the two nucleotides.
Three of the examples discussed above are all examples of phenomenological mod-
els. By this we mean models that attempt to describe the observed phenomena (the
empirical data) and perhaps relate observables. For the neuronal interspike times
we describe the distribution of interspike times. For the gene expression data we
relate the expression level to presence or absence of the BCR/ABL fusion gene, and
for the evolution of molecular sequences we relate the mutations to the time be-
tween observations. Good phenomenological modeling involves a interplay between
the modeling step, the data analysis step and the subject matter field. We do not,
however, attempt to explain or derive the models completely from fundamental the-
ory. It seems that once we step away from the most fundamental models of physics
there is either no derivable, complete theory relating observables of interest or it
is mathematically impossible to derive the exact model. The boundaries between
theory, approximations and phenomenology are, however, blurred.
Most statistical models are phenomenological of nature – after all, the whole purpose
of using a probabilistic model is to capture randomness, or uncontrollable variation,
which by nature is difficult to derive from theory. Often the need for a probabilistic
model is due to the fact that our knowledge about the quantities we model is limited.
A few classical probabilistic models are, however, derived from fundamental sampling
principles. We derive some of these models in the next section.
Classical sampling distributions 127
3.2 Classical sampling distributions
The classical probability models, the binomial, multinomial, geometric and hyper-
geometric distributions, that we consider in this section are all distributions on the
positive integers that arise by sampling and counting, and which can be understood
naturally in terms of transformations of an underlying probability measure.
Example 3.2.1 (Binomial distribution). Let X1 , . . . , Xn denote n iid Bernoulli vari-
ables with success parameter p ∈ [0, 1]. The fundamental sample space is E = {0, 1},
the bundled variable X = (X1 , . . . , Xn ) takes values in {0, 1}n , and the distribution
of X is
Yn Pn Pn
P (X = x) = pxi (1 − p)1−xi = p i=1 xi (1 − p)n− i=1 xi ,
i=1
where x = (x1 , . . . , xn ). Let
h : E → E ′ = {0, 1, . . . , n}
be given as
n
X
h(x1 , . . . , xn ) = xi .
i=1
The distribution of
n
X
Y = h(X) = Xi
i=1
is called the binomial distribution with probability parameter p and size parameter
n. We use the notation Y ∼ Bin(n, p) to denote that Y is binomially distributed with
size parameter n and probability parameter p. We find the point probabilities
P of the
distribution of h(X) as follows: For any vector x = (x1 , . . . , xn ) with ni=1 xi = k it
follows that
P(X = x) = pk (1 − p)n−k .
Thus all outcomes that result in the same value of h(X) are equally probable. So
X
n k
P(h(X) = k) = P (X = x) = p (1 − p)n−k
k
x:h(x)=k
P
where nk denotes the number of elements x ∈ E with h(x) = ni=1 xi = k. From
Section B.2 we get that
n n!
= .
k k!(n − k)!
⋄
Example 3.2.2 (Multinomial distribution). If our fundamental sample space con-
tains m elements instead of just 2 we can derive an m-dimensional version of the bino-
mial distribution. We will assume that the fundamental sample space is {1, 2, . . . , m}
but the m elements could be labeled any way we like. Let X1 , . . . , Xn be n iid ran-
dom variables, let X = (X1 , . . . , Xn ) denote the bundled variable with outcome in
E = {1, 2, . . . , m}n , and consider the transformation
h : E → E ′ = {0, 1, 2, . . . , n}m
defined by
n
X
h(x) = (h1 (x), . . . , hm (x)), hj (x) = 1(xi = j).
i=1
That is, hj (x) counts how many times the outcome j occurs. The distribution of
h(X) = (h1 (X), . . . , hm (X))
is called the multinomial distribution with probability parameter p = (p1 , . . . , pm ).

We observe that for x ∈ E n with h(x) = (k1 , . . . , km ) then
P(X = x) = pk11 pk22 . . . pkmm ,
hence
X
n
P(h(X) = (k1 , . . . , km )) = P (X = x) = pk1 pk2 . . . pkmm ,
k1 . . . km 1 2
x:h(x)=(k1 ,...,km )
n

where k1 ...k m
denotes the number of ways to label n elements such that k1 are
labeled 1, k2 are labeled 2, etc. From Section B.2 we get that

n n!
= .
k k1 !k2 ! · · · km !
⋄
Example 3.2.3 (Geometric distribution). Let X denote a Bernoulli variable taking
values in E = {0, 1} such that P (X = 1) = p. We can think of X representing the
outcome of flipping a (skewed) coin with 1 representing heads (success). We then let
X1 , X2 , X3 , . . . be independent random variables each with the same distribution as
X corresponding to repeating coin flips independently and indefinitely. Let T denote
the first throw where we get heads, that is X1 = X2 = . . . = XT −1 = 0 and XT = 1.
Due to independence the probability of T = t is p(1 − p)t−1 . If we introduce the
random variable
Y = T − 1,
which is the number of tails (failures) we get before we get the first head (success)
we see that
P (Y = k) = p(1 − p)k .
This distribution of Y with point probabilities p(k) = p(1 − p)k , k ∈ N0 , is called
the geometric distribution with success probability p ∈ [0, 1]. It is a probability
distribution on the non-negative integers N0 . ⋄
Classical sampling distributions 129
The distributions derived above can be interpreted as sampling distributions in the

following sense. If we have a population of individuals that have one of two properties
(ill or healthy, male or female, liberal or conservative) and we sample n individuals
completely at random meaning that we sample from the uniform distribution on
the population n times with replacement, the number of individuals among the n
sampled individuals with property one (being ill, say) follows a binomial distribution
B(n, p) with p being the fraction of individuals with property one. And the number
of times we have to sample before we get the first individual with property one is a
geometric distribution. If there are more than two properties we get the multinomial
distribution instead of the binomial distribution. We might for instance divide people
according to a more refined view on their political observance – as socialists, liberals
and conservatives, say – giving rise to a multinomial distribution with three property
labels.
In some cases with finite population sampling the samples are taken without replace-
ment, which result in a different distribution.
Example 3.2.4 (Hypergeometric distribution). We consider here the situation
where we sample individuals or objects from a finite population, and where these
individuals can have a finite number of different properties, which we for convenience
label {1, . . . , m}. The actual sample space E consists of the entire population and
each individual has precisely one of the properties. Let Nj denote the total number
of individuals with property j that are in E – with j ∈ {1, . . . , m} – and
m
X
N= Nj
j=1
is then the total number of individuals in the population. If we independently sample

X1 , . . . , Xn from E completely at random without replacement the probability of
getting X1 = x1 the first time is 1/N and the probability for getting X2 = x2 6= x1
the second time is 1/(N −1) etc. Since we assume that we make independent samples
the probability of getting x1 , . . . , xn ∈ E in n samples is
1 1
p(x1 , . . . , xn ) = = (n) .
N (N − 1)(N − 2) . . . (N − n) N
We define the transformation
h : E n → {0, 1, . . . , n}m
given by
n
X
h(x) = (h1 (x), . . . , hm (x)), hj (x) = 1(property(xi ) = j)
i=1
for x = (x1 , . . . , xn ) ∈ E n . Thus hj (x) is the number of elements in x = (x1 , . . . , xn )

that have property j. The distribution of h(X1 , . . . , Xn ) is the hypergeometric distri-
bution. Since the point probabilities for all (x1 , . . . , xn ) are 1/N (n) the transformation
problem is, like for the binomial and multinomial distribution, a combinatorial prob-
lem. How many outcomes (x1 , . . . , xn ) in E n result in h(x1 , . . . , xn ) = (k1 , . . . , km )
where k1 ≤ N1 , . . . , km ≤ Nm and k1 + . . . km = n? As derived in Section B.2 there
are
Nj
kj
different ways to pick out kj different elements in E with property j. Thus there are

N1 N2 Nm
...
k1 k2 km
ways to pick out k1 elements with property 1, k2 with property 2 etc. There are,
moreover, n! different ways to order these variables, and we find that the point
probabilities for the hypergeometric distribution are given by
number of (x1 , . . . , xn ) with h(x1 , . . . , xn ) = (k1 , . . . , km )

P(h(X) = (k1 , . . . , km )) =
N (n)
n! N
k1
1 N2 Nm
k2 . . . km
=
N (n)
N1 N 2 Nm
k1 k2 . . . km
= N

n
(k1 ) (k2 ) (k )
n N1 N2 . . . Nm m
= .
k1 . . . km N (n)
Here the second equality follows by the definition of the binomial coefficients and
the third by the definition of the multinomial coefficients.
The last formula above for the point probabilities looks similar to the formula for
the multinomial distribution. Indeed, for N1 , . . . , Nm sufficiently large compared to
n it holds that
(k1 ) (k ) (k ) k 1 k 2 k m
N1 N2 2 . . . Nm m N1 N2 Nm
≃ ...
N (n) N N N
and the hypergeometric distribution is well approximated by the multinomial dis-

tribution with pj = Nj /N . This makes a lot of sense intuitively. If we sample with
replacement the variables X1 , . . . , Xn are iid with
P(property(Xi ) = j) = Nj /N.
If N1 , . . . , Nm are large compared to n it hardly makes a difference whether we

sample with or without replacement, and in such cases we will usually prefer to
work with the multinomial distribution. ⋄
Statistical Inference 131
Exercises
Exercise 3.2.1. Consider as in Example 3.2.3 the iid Bernoulli variables

X1 , X2 , X3 , . . . with success probability p. Denote by T the number of variables before
the n’th 1 occurs. That is, XT = 1 and the number of 1’s among the first variables
X1 , . . . , XT −1 is precisely n − 1. Show that the number of 0’s, Y = T − n, before the
n’th 1 has point probabilities

k+n−1 n
P(Y = k) = p (1 − p)k
k
for k ∈ N0 . The distribution is known as the negative binomial distribution.
3.3 Statistical Inference
3.3.1 Parametric Statistical Models
In the previous sections we have introduced various examples of parametrized fam-

ilies (Pθ )θ∈Θ of probability measures on a sample space E. The set Θ is called the
parameter space and θ the parameter. These are the abstract notations that we will
use. The interpretation is that we consider an experiment taking values in E and the
measures Pθ for θ ∈ Θ as candidate models of the outcome of the experiment. We
just don’t know the right one. Based on a sample x ∈ E we are trying to figure out
which of the measures are good candidates for having produced x. In many cases the
sample x is actually a realization x1 , . . . , xn of n independent (and perhaps identi-
cally distributed) random variables X1 , . . . , Xn taking values in a sample space E0 .
In this case E = E0n and Pθ is given in terms of the marginal distributions of the
Xi ’s.
Example 3.3.1 (Multinomial model). Let E0 = {1, 2, . . . , m}, let θ = (p(1), . . . , p(m))
denote a vector of point probabilities on E0 and let
( m
)
X
Θ = (p1 , . . . , pm ) | p(i) ≥ 0, pi = 1 .
i=1
If Pθ denotes the distribution of n iid random variables X1 , . . . , Xn , with each of them

taking values in E0 and having distribution with point probabilities θ, then (Pθ )θ∈Θ
is a statistical model on E = E0n . Note that this statistical model is parametrized by
all probability measures on E0 in terms of the point probabilities θ. From Example
3.2.2 we observe that by defining
n
X
Nj = 1(Xi = j)
i=1
the distribution of the vector

(N1 , . . . , Nm )
is the multinomial distribution with probability parameters θ = (p1 , . . . , pm ) when

the distribution of X1 , . . . , Xn is Pθ . The model is known as the multinomial model.
⋄
Example 3.3.2. Let E0 = [0, ∞), let λ ∈ (0, ∞), and let Pλ be the distribution of n
iid exponentially distributed random variables X1 , . . . , Xn with intensity parameter
λ, that is, the distribution of Xi has density
fλ (x) = λ exp(−λx)
for x ≥ 0. With Θ = (0, ∞) the family (Pλ )λ∈Θ of probability measures is a statistical
model on E = E0n = (0, ∞)n .
Note that we most often use other names than θ for the concrete parameters in
concrete models. For the exponential distribution we usually call the parameter λ.
⋄
As the two previous examples illustrate the parameter space can take quite differ-
ent shapes depending on the distributions that we want to model. It is, however,
commonly the case that Θ ⊆ Rd for some d. In the examples d = m and d = 1
respectively.
As mentioned, we search for probability measures in the statistical model that fit
a dataset well. The purpose of estimation is to find a single ϑ̂ ∈ Θ, an estimate of
θ, such that Pϑ̂ is “the best” candidate for having produced the observed dataset.
We have up to this point encountered several ad hoc procedures for computing
estimates for the unknown parameter. We will in the following formalize the concept
of an estimator as the procedure for computing estimates and introduce a systematic
methodology – the maximum likelihood method. The maximum likelihood method
will in many cases provide quite reasonable estimates and it is straight forward to
implement the method, since the likelihood function that we will maximize is defined
directly in terms of the statistical model. Moreover, the maximum likelihood method
is the de facto standard method for producing estimates in a wide range of models.
3.3.2 Estimators and Estimates
As indicated by the title of this section there is a distinction between an estimator

and an estimate.
Definition 3.3.3. An estimator is a map
θ̂ : E → Θ.
For a given observation x ∈ E the value of θ̂ at x,
ϑ̂ = θ̂(x),
is called the estimate of θ.

The estimator is a map from E into Θ. It is the procedure by which we compute

the estimate once we have data. This implies that if X is a random variable taking
values in E and having distribution Pθ , then we can consider the transformed random
variable θ̂(X). This random variable is usually also called θ̂, thus θ̂ means a map as
well as a random variable and in both cases we refer to θ̂ as an estimator. When
we regard θ̂ as a random variable the estimate ϑ̂ = θ̂(x) becomes the realization of
θ̂ when x is the realization of X. When E = E0n the observation x ∈ E is a vector
of realizations x1 , . . . , xn of the random variables X1 , . . . , Xn . The estimator is then
the transformation θ̂(X1 , . . . , Xn ) of the n random variables and the estimate is the
realization ϑ̂ = θ̂(x1 , . . . , xn ).
When we view θ̂ as a random variable it has a distribution. Finding and under-
standing this distribution is an important problem, since it is the distribution of the
estimator that determines the properties of the estimator. In fact, a property of an
estimator is always going to mean a property about the distribution of that estima-
tor. Note that the distribution of θ̂ depends upon which measure Pθ we consider.
A good estimator is loosely speaking an estimator whose distribution under Pθ is
concentrated around θ for all θ ∈ Θ.
Before we can try to identify a single parameter in Θ it is necessary to deal with the
following identifiability problem.
Definition 3.3.4. The parameter θ is said to be identifiable if the map θ 7→ Pθ
is one-to-one. That is, for two different parameters θ1 and θ2 the corresponding
measures Pθ1 and Pθ2 differ.
If we don’t have identifiability we have no chance of distinguishing between param-

eters for which the corresponding probability measures are identical. This can be
a serious problem, which should be carefully investigated for every model under
consideration.
Example 3.3.5. If Pλ denotes the exponential distribution with λ ∈ (0, ∞) the
intensity parameter, then λ is identifiable since for λ1 6= λ2 we have exp(−λ1 x) 6=
exp(−λ2 x) for all x > 0, thus the two corresponding distribution functions differ.
We can choose another way to parametrize the exponential distribution, where we
let Θ = R\{0} and Pλ be the exponential distribution with intensity parameter λ2 .
That is, the density for Pλ is
fλ (x) = λ2 exp(−λ2 x)
for λ ∈ R. We observe that λ and −λ give rise to the same probability measure
namely the exponential distribution with intensity parameter λ2 . ⋄
The example above illustrates the problem with non-identifiability of the parameter.
We will never be able to say whether λ or −λ is the “true” parameter since they
both give rise to the same probability measure. The example is on the other hand
a little stupid because we would probably not choose such a silly parametrization.
But in slightly more complicated examples it is not always so easy to tell whether
the parametrization we choose makes the parameter identifiable. In some cases the
natural way to parametrize the statistical model leads to non-identifiability.
Example 3.3.6 (Jukes-Cantor). If we consider the Jukes-Cantor model from Ex-
ample 3.1.2 and take both α > 0 and t > 0 as unknown parameters, then we have
non-identifiability. This is easy to see, as the two parameters always enter the con-
ditional probabilities via the product αt. So κα and t/κ gives the same conditional
probabilities for all κ > 0.
This does make good sense intuitively since t is the time parameter (calendar time)
and α is the mutation intensity (the rate by which mutations occur). A model with
large t and small α is of course equivalent to a model where we scale the time down
and the intensity up.
If we fix either of the parameters we get identifiability back, so there is a hope that
we can estimate α from a present, observable process where we know t, and then with
fixed α we may turn everything upside down and try to estimate t. This argument
is based on the “molecular clock”, that is, that the mutation rate is constant over
time. ⋄
Example 3.3.7 (ANOVA). Consider the additive noise model
Xi = µi + σi εi
for the vector X = (X1 , . . . , Xn ) of random variables where εi ∼ N (0, 1) are iid.
The sample space is in this case E = Rn . We regard the scale parameters σi > 0 as
known and fixed. The parameter space for this model is then Θ = Rn and
θ = (µ1 , . . . , µn ).
The parameter is identifiable.

In Example 3.1.4 we considered a situation where Xi was the log-expression level
for a gene, and where we had two groups. This grouping can be coded by a factor,
which is a map from the index set
g : {1, . . . , n} → {1, 2}
such that g(i) = 1 denotes that the i’th observation belongs to group 1 and g(i) = 2
otherwise. The model we considered in Example 3.1.4 correspond to saying that
µi = αg(i)
where (α1 , α2 ) ∈ R2 . We have reduced the parameter space to a 2-dimensional

parameter space.
The grouping indicates whether a particular fusion gene was present or absent for
the individual. There are, however, other potentially important groupings in the
dataset. It is for instance also known whether the individual is a male or a female.
If we define another factor
b : {1, . . . , n} → {1, 2}
such that b(i) = 1 if the i’th individual is a female and b(i) = 2 otherwise we can
consider the model
µi = αg(i) + βb(i)
where µi is broken down into the addition of an αg(i) -component and a βb(i) -component
where (α1 , α2 ) ∈ R2 as above and similarly (β1 , β2 ) ∈ R2 . The parameter space is
Θ = R4
and the parameter is θ = (α1 , α2 , β1 , β2 ). This is a convenient and intuitive way to

parametrize the model, but the parameter for this model is not identifiable as given
here. This is because for all κ ∈ R
µi = αg(i) + βb(i)
= (αg(i) + κ) + (βb(i) − κ).
We can make the additional restriction that
β1 + β2 = 0
and then the parameter becomes identifiable. This restriction effectively reduces the
number of free parameters by 1 and gives a model with only 3 free parameters. An
alternative, identifiable parametrization is given by (α, β, γ) ∈ R3 and
µi = γ + α1(g(i) = 1) + β1(t(i) = 1).
The interpretation of this model is simple. There is a common mean value γ and
then if the fusion gene is present the level changes with α and if the individual is
female the level changes with β.
The additive model above should be compared with a (fully identifiable) model with
four free parameters:
µi = νg(i),b(i)
where the parameter is (ν1,1 , ν1,2 , ν2,1 , ν2,2 ) ∈ R4 . ⋄
The estimation of the unknown parameter for a given statistical model only makes
sense if the parameter is identifiable. If we try to estimate the parameter anyway and
come up with an estimate ϑ̂, then the estimate itself does not have any interpretation.
Only the corresponding probability measure Pϑ̂ can be interpreted as an approxima-
tion of the true probability measure. Without identifiability, we cannot really define
or discuss properties of the estimators directly in terms of the parameter. One will
need to either reparameterize the model or discuss only aspects/transformations of
the parameter that are identifiable. Identifiability is on the other hand often a messy
mathematical problem. In subsequent examples we will only discuss identifiability
issues if there are problems with the parameter being identifiable.
Though identifiability is important for interpretations, probability models with non-
identifiable parametrization and corresponding estimators are routinely studied and
applied to certain tasks. One example is the neural networks or more generally
many so-called machine learning models and techniques. The primary purpose of
such models is to work as approximation models primarily for prediction purposes
– often referred to as black box prediction. The estimated parameters that enter in
the models are not in themselves interpretable.
3.3.3 Maximum Likelihood Estimation
We remind the reader about two ways of representing a probability measure on E.

If E is discrete we can give a probability measure in terms of point probabilities,
thus if Pθ for θ ∈ Θ is a parametrized family of probability measures on a discrete
sample space E, Pθ is defined by the point probabilities pθ (x) for x ∈ E. Likewise,
if E ⊆ Rn for some n we can give a parametrized family Pθ , θ ∈ Θ, of probability
measures on E by densities fθ (x) for x ∈ E.
Definition 3.3.8. Assume that x ∈ E is a given observation. The likelihood function

is the function
Lx : Θ → R
defined as follows: If E is discrete and Pθ has point probabilities (pθ (x))x∈E we define
Lx (θ) = pθ (x). (3.3)
If E ⊆ Rn and Pθ has density fθ : E → [0, ∞) we define
Lx (θ) = fθ (x). (3.4)
We have two different but similar definitions of the likelihood function depending
on whether E is a discrete or a continuous sample space. In fact, there is a more
abstract framework, measure and integration theory, where these two definitions are
special cases of a single unifying definition. It is on the other hand clear that despite
the two different definitions, Lx (θ) is for both definitions a quantification of how
likely the observed value of x is under the probability measure Pθ .
We will often work with the minus-log-likelihood function
lx (θ) = − log Lx (θ) (3.5)
instead of the likelihood function itself. There are several reasons for this. For prac-
tical applications the most notable reason is that the likelihood function often turns
out to be a product of a large number of probabilities (or densities). Since probabil-

ities are less than 1 such a product becomes very small and the risk of running into
underflow problems on computers becomes substantial. Taking the logarithm turns
products into sums and the problem essentially disappears. From a theoretical point
of view the minus-log-likelihood function is simply nicer to work with for a number
of standard models, and it also plays an important role in the further analysis of the
maximum likelihood estimator to be defined below.
Definition 3.3.9 (Tentative). The maximum likelihood estimator, abbreviated MLE,

is the function
θ̂ : E → Θ
such that θ̂(x) is the value of θ at which the likelihood function attains its global
maximum (the minus-log-likelihood function attains its global minimum). We write
θ̂(x) = argmax Lx (θ) = argmin lx (θ). (3.6)

θ∈Θ θ∈Θ
The definition is tentative because there are a number of problems that we need to
take into account. In a perfect world Definition 3.3.9 would work, but in reality the
likelihood function Lx may not attain a global maximum (in which case θ̂(x) is not
defined) and there may be several θ’s at which Lx attains its global maximum (in
which case the choice of θ̂(x) is ambiguous). The problem with non-uniqueness of
the global maximum is not a real problem. The real problem is that there may be
several local maxima and when searching for the global maximum we are often only
able to justify that we have found a local maximum. The problem with non-existence
of the a global maximum is also a real problem. In some situations there exists a
unique θ̂(x) for all x ∈ E such that
Lx (θ) < Lx (θ̂(x)) for all θ 6= θ̂(x),
but quite frequently there is only a subset A ⊆ E such that there exists a unique
θ̂(x) for all x ∈ A with
Lx (θ) < Lx (θ̂(x)) for all θ 6= θ̂(x).
If P(X ∈ Ac ) is small we can with high probability maximize the likelihood func-
tion to obtain an estimate. We just have to remember that under some unusual
circumstances we cannot. When studying the properties of the resulting estimator
we therefore need to consider two things: (i) how θ̂(x) behaves on the set A where
it is defined, and (ii) how probable it is that θ̂(x) is defined.
If the parameter space is continuous we can use calculus to find the MLE analytically.
We differentiate the (minus-log-) likelihood function w.r.t. the parameter and try to
find stationary points, that is, θ’s where the derivative is zero. If Θ = R or an open
interval and lx is twice differentiable it holds that θ̃ ∈ Θ is a local minimizer for
lx (θ) if (and only if)
dlx
(θ̃) = 0 (3.7)
dθ
d2 lx
(θ̃) > 0. (3.8)
dθ 2
From this we can conclude that if there is a unique solution θ̃ ∈ Θ to (3.7) that
also fulfills (3.8) then θ̂(x) = θ̃. To see way, assume that there is a θ0 ∈ Θ such
that lx (θ0 ) < lx (θ̃). The second condition, (3.8), assures that the derivative is > 0
for θ ∈ (θ̃, θ1 ), say, and if θ0 > θ̃ there is a θ2 ∈ (θ1 , θ0 ) where the derivative is <
0. Somewhere in (θ1 , θ2 ) the derivative must take the value 0 – a local maximum –
which contradicts that θ̃ is the unique solution to (3.7).
The equation
dlx
(θ) = 0
dθ
is known as the likelihood equation, and if the MLE, θ̂(x), exists it must satisfy this
equation2 . So as a starting point one can always try to solve this equation to find
a candidate for the MLE. If Θ ⊆ Rd is a multidimensional parameter space there
exists a similar approach, see Math Box 3.3.1, in which case the likelihood equation
becomes a set of equations. Having found a candidate for θ̂(x) that solves the like-
lihood equation(s), there can, however, be substantial problems when d ≥ 2 with
ensuring that the solution is a global maximum. There are some special classes of
models where it is possible to show that when there is a solution to the likelihood
equation then it is the global minimizer for lx and hence the MLE. Typically such ar-
guments rely on convexity properties of lx . For many other models there are no such
result and it is hardly ever possible to solve the likelihood equations analytically.
One therefore needs to rely on numerical methods. To this end there are several ways
to proceed. The Newton-Raphson algorithm solves the likelihood equation numeri-
cally whereas the gradient descent algorithm and its variants are direct numerical
minimization algorithms of lx . The Newton-Raphson algorithm requires that we can
compute the second derivative of lx whereas gradient descent requires only the first
derivative. See Numerical Optimization by Jorge Nocedal and Stephen J. Wright for
a authoritative treatment of optimization methods.
These algorithms are general purpose optimization algorithms that do not take into
account that we try to maximize a likelihood function. In special cases some specific
algorithms exists for maximizing the likelihood, most notably there are a number of
models where one will encounter the so-called EM-algorithm. It is possible that the
reader of these notes will never have to actually implement a numerical optimization
algorithm, and that is not the subject of these notes anyway. A fairly large and ever
growing number of models are available either in R or via alternative programs.
Multivariate numerical optimization is a specialist’s job! It is good to know, though,
2
except in cases where a global maximum is attained at the boundary of the parameter set Θ
Algorithm 3.3.1 (Newton-Raphson). We consider the case Θ ⊆ R and want to

solve the likelihood equation
dlx
(θ) = 0.
dθ
A first order Taylor expansion of the derivative of lx from θ0 amounts to
dlx dlx d2 lx
(θ) ≃ (θ0 ) + 2 (θ0 )(θ − θ0 ).
dθ dθ d θ
For a given θ0 ∈ Θ we solve the linear equation
dlx d2 lx
(θ0 ) + 2 (θ0 )(θ − θ0 ) = 0
dθ d θ
instead of the likelihood equation. The solution is
−1
d2 lx dlx
θ1 = θ0 − 2
(θ 0 ) (θ0 ).
d θ dθ
Taylor expanding from θ1 instead and solving the corresponding linear equation
leads to a θ2 and we can iteratively continue this procedure, which result in a
sequence (θn )n≥0 defined by
−1
d2 lx dlx
θn = θn−1 − (θ n−1 ) (θn−1 ).
d2 θ dθ
If the initial guess θ0 is sufficiently close to ϑ̂ (the MLE) then θn converges rapidly
towards ϑ̂. Note that a fixed point of the algorithm, that is, a θ such that if we put
in θ in the formula above we get back θ, is a solution to the likelihood equation.
what kind of optimization is going on when one computes the MLE in practice,
and it is also good to know that all numerical algorithms, whether they are general
purpose or problem specific, typically rely on an initial guess θ0 of the parameter.
The algorithm will then iteratively “improve” upon the guess. It is good practice
to choose a number of different initial guesses or come up with qualified initial
guesses to prevent that the algorithm either diverges or converges towards a wrong
local minimum close to a “bad” initial guess. Note that there is no way to assure in
general that an algorithm has found a global minimizer.
Example 3.3.10 (Exponential distribution). We consider the model with E =

[0, ∞)n , Θ = (0, ∞), and Pλ the probability measure under which X1 , . . . , Xn are iid
exponentially distributed with intensity parameter λ. The density for the exponential
distribution with intensity parameter λ is
fλ (x) = λ exp(−λx).
Math Box 3.3.1 (Multivariate optimization). If Θ ⊆ Rd we have θ = (θ1 , . . . , θd )

and a multivariate analog of the likelihood equation. If lx (θ) is differentiable the
derivative is
∂lx ∂lx
Dlx (θ) = (θ), . . . , (θ)
∂θ1 ∂θd
and the likelihood equation reads
Dlx (θ) = 0.
In reality this is a system of d equations, and a solution is a stationary point.

A global minimizer in the interior of Θ of the minus-log-likelihood function is
necessarily a stationary point.
The matrix of second derivatives is
 ∂2 l ∂ 2 lx

 2 (θ) ... ∂θd ∂θ1 (θ) 
x

 ∂θ1 

D2 lx (θ) = .. .. ,
 . . 

 ∂ 2 lx (θ) ∂ 2 lx 

∂θ1 ∂θd ... ∂θd2
(θ)
and if θ̃ is a solution of the likelihood equation, Dlx (θ̃) = 0, then if D2 lx (θ̃) is

positive definite the point θ̃ is a local minimizer of the minus-log-likelihood function.
In general it is very difficult to say anything about global properties like existence
and uniqueness of the MLE unless D2 lx (θ) is positive definite for all θ ∈ Θ. In this
case lx (θ) is a convex function and a solution to the likelihood equation is a global
minimizer.
The gradient is ∇lx (θ) = Dlx (θ)T , and the multivariate analog of the Newton-
Raphson algorithm for numerically solving the likelihood equation reads
θn = θn−1 − D2 lx (θn−1 )−1 ∇lx (θn−1 ).
For an initial guess θ0 it yields a sequence (θn )n≥0 that – hopefully – converges to
the MLE ϑ̂ = θ̂(x).
Having observed x1 , . . . , xn the likelihood function is

n n
!
Y X
n
Lx (λ) = λ exp(−λxi ) = λ exp −λ xi
i=1 i=1
and the minus-log-likelihood function is

n
X
lx (λ) = λ xi − n log λ.
i=1
If we differentiate we obtain the likelihood equation

X n
dlx n
(λ) = xi − = 0.
dλ λ
i=1
170
170 1.4
160
160 1.2
150
σ2
1.0
150
140
0.8
1.4
140
1.2
−0.4 1.0
−0.2 0.6
0.8
0.0
130
0.2
0.6 −0.4 −0.2 0.0 0.2 0.4
0.4
µ
Figure 3.3: The minus-log-likelihood function (left) and a contour plot (right) for
the scale-location parameters (µ, σ 2 ) based on n = 100 simulations of iid N (0, 1)-
distributed variables. The MLE is µ̂ = 0.063 and σ̃ 2 = 0.816. The profile minus-
log-likelihood function of σ 2 is given by evaluating the minus-log-likelihood function
along the straight line, as shown on the contour plot, given by µ = µ̂.
There is a unique solution to the likelihood equation,

n
λ̂ = Pn ,
i=1 xi
2
and since dd2lλx (λ) = λn2 > 0 together with uniqueness of the solution λ̂ to the like-
lihood equation, λ̂ is a global minimizer of the minus-log-likelihood function and
hence the MLE. ⋄
The example above shows that the ad hoc estimate of λ considered in Example 3.1.1
for neuronal interspike data is in fact also the maximum likelihood estimate.
Example 3.3.11 (Normal distribution). Let X1 , . . . , Xn be n iid random variables
with the N (µ, σ 2 ) distribution. The statistical model is given by E = Rn , Θ =
R × (0, ∞), θ = (µ, σ 2 ) and Pθ the probability measure that has density
Yn
1 (xi − µ)2
fθ (x1 , . . . , xn ) = √ exp −
2πσ 2 2σ 2
i=1
n
!
1 1 X
= √ exp − 2 (xi − µ)2 .
( 2πσ 2 )n 2σ
i=1
The minus-log-likelihood function for observing x = (x1 , . . . , xn ) ∈ E is therefore

n
1 X n √
lx (µ, σ 2 ) = 2
(xi − µ)2 + log σ 2 + n log 2π.
2σ 2
i=1
We want to minimize this function of the two-dimensional parameter (µ, σ 2 ). We first

fix σ 2 and regard lx (µ, σ 2 ) as a function of µ only. Then we have a one dimensional
parameter and we find the likelihood equation to be
n
dlx 2 1 X
(µ, σ ) = − 2 (xi − µ) = 0
dµ 2σ
i=1
which is a simple, linear equation in µ with the solution

n
1X
µ̂ = xi .
n
i=1
Differentiating once more gives that
d2 lx n
2
(µ, σ 2 ) = 2 > 0
d µ 2σ
so µ̂ is a global minimizer of lx (µ, σ 2 ) for fixed σ 2 . We conclude that no matter what

the value of σ 2 is the likelihood is maximized as a function of µ at µ̂. If we therefore
fix µ = µ̂ and consider the minus-log-likelihood function
n
2 1 X n √
lx (µ̂, σ ) = 2 (xi − µ̂)2 + log σ 2 + n log 2π
2σ 2
i=1
as a function of σ 2 only, the corresponding likelihood equation becomes (note that

we differentiate w.r.t. the parameter σ 2 )
n
dlx 2 1 X n
2
(µ̂, σ ) = − 4
(xi − µ̂)2 + 2 = 0.
dσ 2σ 2σ
i=1
This is again a quite simple equation, and by multiplying with σ 4 > 0 we can
rearrange the equation into
n
X
σ2n = (xi − µ̂)2 .
i=1
The solution is
n
1X
σ̃ 2 = (xi − µ̂)2 .
n
i=1
dlx 2 2 < σ̃ 2 and > 0
Similarly we may show that the derivative dσ 2 (µ̂, σ ) is < 0 for σ
2 2 2 2
for σ > σ̃ . Thus lx (µ̂, σ ) is monotonely decreasing up to σ̃ and then monotonely
increasing thereafter. Thus the solution σ̃ 2 is the unique global minimizer for lx (µ̂, σ 2 )
and in conclusion (µ̂, σ̃ 2 ) is the unique global maximizer for the likelihood function.
⋄
138
profile minus−log−likelihood function
136
134
132
0.6 0.8 1.0 1.2 1.4
σ2
Figure 3.4: Example of the profile minus-log-likelihood function for σ 2 with 100
simulated N (0, 1) variables.
For the statistical model above – the normal distribution with unknown mean and
variance – where we have two unknown parameters, we were able to derive the MLE
by reducing the two-dimensional optimization problem to successive one-dimensional
optimization problems. The technique is useful – analytically as well as numerically.
Instead of throwing ourselves into a difficult optimization problem of several vari-
ables, we may try to solve the problem one variable at a time. The likelihood as
a function of one parameter optimized over all other parameters is known as the
profile likelihood. For instance,
n
1 X n √
lx (µ̂, σ 2 ) = 2
(xi − µ̂)2 + log σ 2 + n log 2π
2σ 2
i=1
considered above is the profile minus-log-likelihood of σ 2 .

Example 3.3.12 (Neuron interspike time models revisited). As remarked after Ex-
ample 3.3.10 we actually used the maximum likelihood estimator for estimation of λ
in the simple model based on the exponential distribution of the neuronal interspike
times considered in Example 3.1.1. We consider here the extended location-scale
model also suggested in Example 3.1.1.
Having observed x1 , . . . , xn the likelihood function is
n n
!
Y X
Lx (µ, λ) = λ exp(−λ(xi − µ)) = λn exp −λ (xi − µ)
i=1 i=1
for µ < min{x1 , . . . , xn }. For µ ≥ min{x1 , . . . , xn } the likelihood function is 0 be-

cause at least one of the factors is 0. This effectively restricts the full parameter
space [0, ∞) × (0, ∞) to [0, min{x1 , . . . , xn }) × (0, ∞).
Fixing µ ∈ [0, min{x1 , . . . , xn }) computations identical to those in Example 3.3.10
show that
n
λ̂(µ) = Pn
i=1 i − µ)
(x
is the unique maximizer of the likelihood function. The profile likelihood function
then becomes
n
n n
Lx (µ, λ̂(µ)) = λ̂(µ) exp(−n) = Pn exp(−n)
i=1 (xi − µ)
and we see that it is monotonely increasing in µ.
The conclusion is that this is a nasty model. The idea behind the likelihood method is
to find parameters such that the probability measure fits the data in the best possible
way. Here it turns out that the closer µ gets to the upper bound min{x1 , . . . , xn }, the
better the model fits the data. We cannot, however, take µ equal to min{x1 , . . . , xn }
as this would make the likelihood drop to the all time minimum of 0! There is no
maximum likelihood estimator for the model and we take this as a bad sign. Just as
there are conceptual problems with µ > 0, as discussed in Example 3.1.1, there are
also inferential problems, and we cannot produce a sensible estimator of µ. ⋄
In the next example we use another trick – a reparameterization – which makes it

possible to find the minimum of the minus-log-likelihood function quite easily.
Example 3.3.13 (Multinomial model). Consider the multinomial model from Ex-
ample 3.3.1. The likelihood function for observing x = (x1 , . . . , xn ) ∈ {1, . . . , m}n
is
Yn m
Y n
Lx (θ) = p(xi ) = pj j
i=1 j=1
with
n
X
nj = 1(ni = j)
i=1
Qn
since the product i=1 p(xi ) contains the pj -factor precisely nj times.
We will here consider the parameter space Θ consisting of θ = (p1 , . . . , pm ) where
pj > 0 and p1 + . . .+ pm = 1 so that none of the point probabilities can be equal to 0.
We make a reparameterization in terms of the parameter β = (β1 , . . . , βm ) ∈ (0, ∞)m
via
eβj
pj (β) = Pm β
i=1 e
i
for j = 1, . . . , m. We may note that the β parameter is in fact not identifiable though
the θ parameter is. It just turns out that the optimization in the β-parametrization
is simpler.
The minus-log-likelihood function in terms of the new parametrization reads

   
Xm m−1
Y Xm m
X
lx (β) = n log  eβj  − log eβj nj = n log  eβj  − βj nj .
j=1 j=1 j=1 j=1
If we fix all parameters but βj we find by differentiation that βj must fulfill that
neβj
Pm β − nj = 0
j=1 e
j
or that
eβj nj
pj (β) = Pm = ,
j=1 eβj n
which has a solution in βj if and only if nj > 0. This shows that in the θ-parametrization
there is a unique minimizer of the minus-log-likelihood equation given by
nj
p̂j =
n
if n1 , . . . , nm > 0. Strictly speaking we have showed that this solution is the only
possible minimizer – we haven’t showed that it actually is a minimizer3
Using the reparameterization we have found that if n1 , . . . , nm > 0 the likelihood
function Lx (θ) attains a unique maximum over the set Θ in
n nm
1
,..., ,
n n
which is therefore the maximum likelihood estimate. The vector (n1 , . . . , nm ) is the
realization of the random variable (N1 , . . . , Nm ) and the maximum likelihood esti-
mator is
N1 Nm
θ̂ = (p̂1 , . . . , p̂m ) = ,...,
n n
This maximum likelihood estimator is very reasonable, since the estimator for pj ,
Nj
p̂j = ,
n
is the relative frequency of observations being equal to j. ⋄
The previous examples gave explicit expressions for the maximum likelihood esti-
mator. It is not always the case that we can come up with a nice analytic solution
to the optimization problem – not even by clever reparameterizations or using the
profile method. In fact, it is rather the exception than the rule that the maximum
3
If one is familiar with convexity concepts it is possible to show that lx is convex in the β-
parametrization, and convexity implies that a solution to the likelihood equation is a global mini-
mizer.
185
180
profile minus−log−likelihood function
175
170
165
160
0.6 0.8 1.0 1.2 1.4
Figure 3.5: The profile minus-log-likelihood function for ρ with 100 simulated Gum-
bel variables.
likelihood estimator is given by a closed form analytic expression. The next example
will show an analysis where we use almost all of the techniques considered previously
and though we cannot find a complete analytic solution we are able to derive precise
conditions for the existence of a unique maximum to the likelihood function and we
can derive a single equation in a one-dimensional parameter that we need to solve
numerically.
Example 3.3.14 (Gumbel distribution). If X1 , . . . , Xn are iid random variables

with distribution being a location-scale transformation of the Gumbel distribution,
then the statistical model is given by the sample space E = Rn , parameter space
Θ = R × (0, ∞), and parameters θ = (µ, σ). The probability measure Pθ has density
n
Y
1 xi − µ xi − µ
fµ,σ (x1 , . . . , xn ) = exp − − exp − .
σ σ σ
i=1
The minus-log-likelihood function for observing x = (x1 , . . . , xn ) is therefore

n
X n
X
xi − µ xi − µ
lx (µ, σ) = n log σ + + exp − . (3.9)
σ σ
i=1 i=1
It is not so easy to find the minimum of this function analytically let alone to show
that it has a minimum. But there is a way around dealing with lx (µ, σ) directly via
a reparameterization. Introduce the parameters (η, ρ) ∈ R × (0, ∞) given by

µ 1
(η, ρ) = , .
σ σ

Clearly then (µ, σ) = ηρ , ρ1 , so there is a one-to-one correspondence between the
parameters (µ, σ) and the parameters (η, ρ). One of the beautiful and useful prop-
erties of the maximum likelihood estimation principle is, that it is invariant under
reparameterizations. This means that we can just as well minimize lx in this new
parametrization, which turns out to be more convenient.
We find that in the (η, ρ)-parametrization
n
X n
X
lx (η, ρ) = (ρxi − η) + exp (−ρxi + η) − n log ρ
i=1 i=1
Xn Xn
= ρ xi + exp(η) exp(−ρxi ) − nη − n log ρ
i=1 i=1
n
X
= ρnx + exp(η) exp(−ρxi ) − nη − n log ρ
i=1
P
where we have introduced the notation x = n1 ni=1 xi , and one can already see the
simplification of the expression for the minus-log-likelihood function. Moreover, if
we fix the ρ parameter and differentiate w.r.t. η, we obtain
X n
dlx
(η, ρ) = exp(η) exp(−ρxi ) − n.
dη
i=1
To find the minimum of the minus-log-likelihood function for any fixed ρ we equate
the derivative to 0. This gives the equation
n
X
exp(η) exp(−ρxi ) = n,
i=1
whose solution is !
n
1X
η̂(ρ) = − log exp(−ρxi ) .
n
i=1
Differentiation once more w.r.t. η yields that
X n
d2 lx
2
(η, ρ) = exp(η) exp(−ρxi ) > 0,
dη
i=1
which shows that not only does the minus-log-likelihood function attain a local
minimum at η̂(ρ) as a function of η for given ρ but actually a global minimum.
1.4 240
240
220 1.2
220
200
1.0
ρ
200
180
0.8
−0.4 1.4 180

−0.2 1.2
0.0 1.0 0.6
0.2 0.8
160
−0.4 −0.2 0.0 0.2 0.4
0.4 0.6
η
Figure 3.6: The minus-log-likelihood function (left) and a contour plot (right) for the
(η, ρ) reparameterization of the scale-location model based on n = 100 simulations of
iid Gumbel distributed variables. The MLE is η̂ = 0.025 and ρ̂ = 0.944. The profile
minus-log-likelihood function of ρ is given by evaluating the minus-log-likelihood
function along the curved line, as shown on the contour plot, given by the equation
η̂(ρ) = η.
The minimizer η̂(ρ) depends upon ρ, which makes things more complicated as com-
pared to the normal distribution where the minimizer of µ for fixed σ 2 does not
depend upon σ 2 . To get any further we plug the expression of η̂(ρ) back into the
minus-log-likelihood function giving the profile minus-log-likelihood function
n
!
1X
lx (η̂(ρ), ρ) = ρnx + n + n log exp(−ρxi ) − n log ρ,
n
i=1
which is a function of ρ alone. Differentiation gives

Pn
dlx i=1 xi exp(−ρxi ) n
(η̂(ρ), ρ) = nx − n P n − .
dρ i=1 exp(−ρxi ) ρ
Equating this derivative to 0 yields the equation
Pn
Pi=1 xi exp(−ρxi ) 1
n + = x, (3.10)
i=1 exp(−ρxi ) ρ
whose solution does not seem to have a nice analytic expression. We can note that
for ρ approaching 0, the left hand side behaves as x + 1/ρ > x and for ρ approaching
∞ the left hand side behaves as min{x1 , . . . , xn }. If there are at least two different
observations the latter is strictly smaller than x and since the left hand side is
continuous in ρ there is in this case always a solution to the equation.
A second differentiation of the profile minus-log-likelihood function – and some al-

gebraic manipulations – give
n
X Pn
d2 lx i=1 xi exp(−ρxi ) 2 exp(−ρxi ) n
2
(η̂(ρ), ρ) = n xi − Pn Pn + 2 > 0.
dρ i=1 exp(−ρxi ) i=1 exp(−ρxi ) ρ
i=1
Since this shows that the derivative is strictly increasing there can be only one
solution to the equation above.
The conclusion of our analysis is, that if n ≥ 2 and at least two of the observations are
different there is precisely one solution to the equation above, hence there is a unique
global minimum for the profile minus-log-likelihood function, and consequently there
is a unique global minimizer for the full minus-log-likelihood function.
From a practical point of view we need to solve (numerically) the equation (3.10)
and then plug this solution into η̂(ρ). This gives the maximum likelihood estimate
of (η, ρ). The Newton-Raphson algorithm for solving the equation reads
d2 lx dlx
ρn = ρn−1 − (η̂(ρn−1 ), ρn−1 )−1 (η̂(ρn−1 ), ρn−1 )
dρ2 dρ
for n ≥ 1 and an initial guess ρ0 . ⋄
The previous example elaborated on the idea of using the profiled minus-log-likelihood
function. Combined with a reparameterization the profile method reduced the bi-
variate optimization to the solution of a univariate equation. In the next example
we show how a reparameterization can help simplify the computations considerably
though they can be carried out in the original parametrization if desired.
Example 3.3.15 (Evolutionary models). We consider the model from Example
3.1.2 with (X1 , Y1 ), . . . , (Xn , Yn ) being n iid random variables each taking values
in E0 × E0 with E0 = {A, C, G, T}. We take the distribution of (Xi , Yi ), when the
evolutionary distance in time is t, to be given by the point probabilities
Pt,p,θ (x, y) = p(x)Pθt (x, y).
Here p is a vector of point probabilities on E0 , so that p(x) is the probability of x,
and Pθt (x, y) is the conditional probability that x mutates into y in time t. These
conditional probabilities depend on the additional parameter θ ∈ Θ. The main focus
is on the estimation of θ. Having observed z = ((x1 , y1 ), . . . , (xn , yn )) ∈ E = (E0 ×
E0 )n the full likelihood function becomes
n
Y n
Y
Lz (t, p, θ) = Pt,p,θ (xi , yi ) = p(xi )Pθt (xi , yi )
i=1 i=1
and the minus-log-likelihood function is

n
X n
X
lz (t, p, θ) = − log p(xi ) − log Pθt (xi , yi ).
i=1 i=1
We observe that the first term depends upon p only and the second term on (t, θ). We
can therefore optimize each term separately to find the maximum likelihood estima-
tor. In addition we see that the first term is simply the minus-log-likelihood function
for the multinomial distribution, and naturally the MLE of the marginal point prob-
ability p(x) is the relative frequency of x among the observations x1 , . . . , xn .
Turning to the second term we know from Example 3.3.6 that there may be a
problem with identifiability of t and the additional parameters, and we consider
here the situation where we know (or fix) t. Then the second term,
n
X
˜lz (θ) = − log Pθt (xi , yi ),
i=1
is regarded as a function of θ only. We also introduce the variables

n
X
nz (x, y) = 1(xi = x, yi = y),
i=1
thus nz (x, y) is the number of observed mutations of x with y. Then we can rewrite
X
˜lz (θ) = − nz (x, y) log Pθt (x, y)
x,y
Pn t
since there are exactly nz (x, y) terms in the sum i=1 log Pθ (xi , yi ) that equals
log Pθt (x, y).
For the rest of this example we will consider the special model, the Jukes-Cantor
model, where
Pαt (x, x) = 0.25 + 0.75 × exp(−4αt)

Pαt (x, y) = 0.25 − 0.25 × exp(−4αt), if x 6= y,
for some (known) t > 0 and α > 0 the unknown additional parameter. Introducing
X X
n1 = nz (x, x) and n2 = nz (x, y)
x x6=y
we find that
˜lz (α) = −n1 log(0.25 + 0.75 × exp(−4αt)) − n2 log(0.25 − 0.25 × exp(−4αt)).
If we differentiate we obtain
d˜lz 3n1 t exp(−4αt) n2 t exp(−4αt)
(α) = −
dα 0.25 + 0.75 × exp(−4αt) 0.25 − 0.25 × exp(−4αt)

3n1 n2
= 4t exp(−4αt) −
1 + 3 exp(−4αt) 1 − exp(−4αt)
dl̃z
and the likelihood equation dα (α) = 0 is equivalent to the equation
3n1 (1 − exp(−4αt)) = n2 (1 + 3 exp(−4αt)).
This equation has a (unique) solution if and only if 3n1 > n2 in which case
1 3(n1 + n2 ) 1 3n
α̂ = log = log
4t 3n1 − n2 4t 3n1 − n2
is the maximum likelihood estimator. Moreover, we see from the expression of the
derivative of ˜lz (α) that (given 3n1 > n2 ) then ddα
l̃z
(α) < 0 if α < α̂ and ddα
l̃z
(α) > 0
˜
if α > α̂. This shows that lz (α) is monotonely decreasing up to α̂ and monotonely
increasing thereafter. Hence α̂ is the global minimum of the minus-log-likelihood
function.
Working with the minus-log-likelihood and in particular differentiation in the α-
parametrization is hideous. It is much easier to make a reparameterization by
γ = γ(α) = 0.25 − 0.25 exp(−4αt)
such that
1
α = α(γ) = log(1 − 4γ)
4t
for γ ∈ (0, 0.25). In the γ-parameter the minus-log-likelihood becomes
lz (γ) = −n1 log(1 − 3γ) − n2 log γ
for γ ∈ (0, 0.25). The differentiation is easier yielding the derivative

3n1 n2
lz′ (γ) = − ,
1 − 3γ γ
and solving the likelihood equation is always possible for γ ∈ (0, 1/3) and gives
n2
γ̂ = .
3n
The solution is in (0, 0.25) if and only if 3n1 > n2 in which case we get
1 n2 1 3n
α̂ = α(γ̂) = log 1 − 4 = log .
4t 3n 4t 3n1 − n2
⋄
Example 3.3.16. We turn to the data from the Hepatitis C virus evolution, as
considered in Example 1.2.4, and we want to estimate the α parameter in the Jukes-
Cantor model. The two quantities that enter in the estimator are n2 , the total
number of mutations, and n1 , the remaining number of non-mutated nucleotide
pairs. Observe that n1 = n − n2 . We will consider two situations. Either we pool
all of the three segments of the virus and make one estimate, or we estimate α
separately for the segments A, B, and C. The following table shows the result.
600
500
400
Minus−log−likelihood
300
200
Segment A
Segment B
100
Segment C
0.0000 0.0005 0.0010 0.0015 0.0020 0.0025 0.0030
Figure 3.7: The (partial) minus-log-likelihood functions ˜lz (α) for the hepatitis C virus
data, as considered in Example 3.3.16, using the Jukes-Cantor model. We estimate
α separately for the three segments A, B, and C of the genome. The corresponding
maximum likelihood estimates are marked on the plots.
Segment
A B C A+B+C
n1 2532 1259 1009 4800
n2 78 25 20 123
α̂ 7.8 × 10−4 5.0 × 10−4 5.0 × 10−4 6.5 × 10−4
The time is measured in years so that t = 13 and the estimated mutation rates are
thus per year mutation rates. Seemingly, segment A shows a different mutation rate
than segments B and C do – but conclusions like this have to be based on knowledge
about the uncertainty of the estimates. We deal with this in Chapter ??. Plugging
the estimated mutation rate α̂A = 7.8 × 10−4 for segment A into the expression for
Pαt (x, y), we find for instance that for x 6= y
Pα̂1 (x, y) = 0.25 − 0.25 × exp(−4 × 7.8 × 10−4 ) = 0.000779.
This is an estimate of the chance that any specific single nucleotide in segment A
will mutate within a year.
Still dividing the data according to the three segments, the frequency vectors nA , nB , nC ∈
N40 and relative frequency vectors pA , pB , pC ∈ R4 (MLEs of p) are
A C G T Total
nA 483 786 763 578 2610
pA 0.185 0.301 0.292 0.221 1
nB 257 398 350 279 1284
pB 0.200 0.310 0.273 0.217 1
nC 233 307 286 203 1029
pC 0.226 0.298 0.278 0.197 1
To investigate whether the model is actually a good model for the data, we reconsider
the table of observed mutations for segment A together with the expected number
of mutations. Since all mutations are equally probable for the Jukes-Cantor model –
the probability being 0.25 − 0.25 × exp(−4 × 13 × 7.8 × 10−4 ) = 0.0099 for segment
A – the expected number of mutations from nucleotide x is nA (x) × 0.0099.
H90 H90
A C G T A C G T
A 1 11 1 A 4.8 4.8 4.8
H77 C 4 1 20 H77 C 7.8 7.8 7.8
G 13 3 1 G 7.6 7.6 7.6
T 3 19 1 T 5.7 5.7 5.7
Observed mutations - segment A Expected mutations - segment A
Looking at these two tables we are suspicious about whether the Jukes-Cantor model
actually is adequate. There are many more transitions than transversions, where the
Jukes-Cantor model predicts the same number. ⋄
Example 3.3.17 (Logistic regression). This example introduces a class of models

for analyzing the effect of a continuous factor/covariate on the distribution of a
single Bernoulli variable. This is a quite classical situation with a dichotomous dose-
response experiment. The response is a 0-1 variable, like dead/alive or red/green,
but the probability of the variable depends upon a continuous dose, for instance the
concentration of a toxic compound, pesticide or insecticide.
In an experiment, carried out by Jørgen Jespersen at Statens Skadedyrslaboratorium,
260 flies (Musca domestica) were exposed to the insecticide dimethoat in varying
concentrations. The experiment was organized with 13 groups of 20 flies, and flies in
the same group were given the same concentration. The results can be summarized
in the following table:
Concentration log(concentration) Deaths Survivors

0.016 −4.135 0 20
0.0226 −3.790 0 20
0.032 −3.442 0 20
0.0453 −3.094 0 20
0.064 −2.749 5 15
0.0905 −2.402 4 16
0.128 −2.056 5 15
0.181 −1.709 15 5
0.256 −1.363 14 6
0.362 −1.016 18 2
0.515 −0.664 20 0
0.724 −0.323 20 0
1.024 0.024 20 0
Thus we have observed 260 Bernoulli variables X1 , . . . , X260 (death = 1 and survival
= 0), which we can safely assume independent. But it would obviously be wrong to
assume that they have the same distribution. On the contrary, we are interested in
figuring out the effect of the concentration of dimethoat on the death rate of the
flies, that is, on the distribution of the Bernoulli variables.
We will parametrize the probability of death as a function of dose (concentration),
and we introduce
exp(α + βy)
p(y) = ,
1 + exp(α + βy)
where p(y) ∈ (0, 1) and α, β > 0. The logistic regression model is then defined by
letting the probability of Xi = 1 given the dose yi be p(yi ). The function y 7→ p(y)
is known as the logistic function, which is why this model of the probability as a
function of dose level is known as logistic regression. It is common not to use the con-
centration directly as the dose level but instead use the log(concentration). Thus y =
log(concentration). The observation consists in general of a vector x = (x1 , . . . , xn )
of 0-1 variables, with n = 260 in the fly death example, and the statistical model of
logistic regression has sample space E = {0, 1}n , parameter θ = (α, β) and parame-
ter space Θ = R2 .
Observing that
1
1 − p(y) =
1 + exp(α + βy)
we can rewrite to find
p(y)
log = α + βy.
1 − p(y)
The left hand side is the logarithm of the odds that the fly die, so the model says
that the log odds for dying depends linearly upon the dose level (log(concentration)
for the flies).
The likelihood function for observing x = (x1 , . . . , xn ) ∈ E is

n
Y n
Y
xi 1−xi exp(αxi + βyi xi )
Lx (α, β) = p(yi ) (1 − p(yi )) = .
1 + exp(α + βyi )
i=1 i=1
The minus-log-likelihood function becomes

n
X
lx (α, β) = log(1 + exp(α + βyi )) − αxi − βyi xi
i=1
n
X
= log(1 + exp(α + βyi )) − αS − βSS,
i=1
where
n
X n
X
S= xi and SS = y i xi .
i=1 i=1
Note that you can compute the likelihood function from the table above. You do not
need the observations, only the summary given in the table. It is actually sufficient
to know just S = 121, SS = −151.9 and the log(concentrations).
We fix α and differentiate w.r.t. β and find
n
X n
X
dlx yi exp(α + βyi )
(α, β) = − SS = yi p(yi ) − SS
dβ 1 + exp(α + βyi )
i=1 i=1
n
X n
X
d2 lx yi2 exp(α
+ βyi )
(α, β) = = yi2 p(yi )(1 − p(yi )).
dβ 2 (1 + exp(α + βyi ))2
i=1 i=1
Likewise, we fix β and differentiate w.r.t. α

n
X n
X
dlx exp(α + βyi )
(α, β) = −S = p(yi ) − S
dα 1 + exp(α + βyi )
i=1 i=1
n
X n
X
d2 lx exp(α + βyi )
(α, β) = = p(yi )(1 − p(yi )).
dα2 (1 + exp(α + βyi ))2
i=1 i=1
Since both of the second derivatives are > 0, we conclude that lx (α, β) as a function
of one of the parameters (and the other fixed) can have at most one local minimum,
which is then a global minimum. This does not prove the uniqueness of minima of
lx regarded as a function of two variables.
We can approach the bivariate optimization by general purpose optimization algo-
rithms, and use for instance the Newton-Raphson algorithm as discussed in Math
Box 3.3.1. As an alternative, which illustrates the use of the one-dimensional Newton-
Raphson algorithm along the coordinate axis, consider the following alternating
4.0
500
500
3.5
400
Minus−log−lik
400
3.0
300
elihood
2.5
β
300
200
100
2.0
200
4.0
3.5 7
3.0 6
1.5
5
2.5
4 100
be
ta
2.0
ha
3 alp
1.0
1.5
2 1 2 3 4 5 6 7
1.0 1 α
4.0 4.0
500 500
3.5 3.5
400 400
3.0 3.0
2.5 2.5
β
300 300
2.0 2.0
200 200
1.5 1.5
100 100
1.0 1.0
1 2 3 4 5 6 7 1 2 3 4 5 6 7
α α
Figure 3.8: The minus-log-likelihood function (upper left) using the logistic regression
model for the fly data. It does not have a pronounced minimum, but rather a long
valley running diagonally in the α-β-parametrization. The contour plots (upper right
and bottom) show how two different algorithms converge towards the minimizer (the
MLE) (α̂, β̂) = (5.137, 2.706) for different starting points (α0 , β0 ) = (2, 2), (4, 3) and
(4, 1.5). The broken line is the bivariate Newton-Raphson algorithm and the black
line is the alternating Newton-Raphson algorithm, as discussed in Example 3.3.17,
which is slower an moves in zig-zag along the α and β axes.
Newton-Raphson algorithm. With initial guess (α0 , β0 ) we define the sequence (αn , βn )n≥0
by
n
!−1 n !
X X
αn = αn−1 − pn−1 (yi )(1 − pn−1 (yi )) pn−1 (yi ) − S
i=1 i=1
and then
n
!−1 n
!
X X
βn = βn−1 − yi2 p′n−1 (yi )(1 − p′n−1 (yi )) yi p′n−1 (yi ) − SS
i=1 i=1
where
exp(αn−1 + βn−1 y) exp(αn + βn−1 y)
pn−1 (y) = p′n−1 (y) = .
1 + exp(αn−1 + βn−1 y) 1 + exp(αn + βn−1 y)
The algorithm amounts to making a one-dimensional Newton-Raphson step first
along the α axis, then along the β axis and so on and so forth. It is not a particular
fast algorithm – if the sequence of parameters converge they do it slowly – but
curiously for this example the alternating Newton-Raphson algorithm is more stable
than the raw bivariate Newton-Raphson. This means that it will actually converge
for a large set of initial guesses. Convergence of the raw bivariate Newton-Raphson
is quite sensitive to making a good initial guess. There are ways to deal with such
problems – via moderations of the steps in the Newton-Raphson algorithm. But
that is beyond the scope of these notes. If any of the algorithms converge then the
resulting point is, for the logistic regression model, always a global minimizer. For
the flies the MLE is
(α̂, β̂) = (5.137, 2.706).
1.0
0.8
0.6
0.4
0.2
0.0
−4 −3 −2 −1 0
log(concentration)
Figure 3.9: The logistic curve for the probability of fly death as a function of
log(concentration) of dimethoat using the MLE parameters (α̂, β̂) = (5.137, 2.706)
Sometimes the likelihood function does not attain a maximum, and neither algorithm
will converge. This happens if there are two concentrations, c0 < c1 , such that all
flies that received a dose below c0 survived and all flies that received a dose above
c1 died, and we have no observations with dose levels in between. This is not really
a problem with the model, but rather a result of a bad experimental design. ⋄
Example 3.3.18 (Kimura model). Instead of considering the Jukes-Cantor model

in Example 3.3.15 we can consider the Kimura model given by the conditional prob-
abilities

P t (x, y) = 0.25 + 0.25 exp(−4βt) − 0.5 exp(−2(α + β)t), if λ(x, y) = α
t
P (x, y) = 0.25 − 0.25 exp(−4βt), if λ(x, y) = β.
We regard t to be fixed and (α, β) ∈ (0, ∞) × (0, ∞) as the unknown parameters. If

we introduce the three numbers
X
n1 = nz (x, x) = nz (A, A) + nz (C, C) + nz (G, G) + nz (T, T)
x
X
n2 = nx (x, y) = nz (A, G) + nz (C, T) + nz (G, A) + nz (T, C)
x,y:λ(x,y)=α
X
n3 = nz (x, y) = n − n1 − n2
x,y:λ(x,y)=β
being the number of nucleotide pairs with no mutations, the number of transitions
and the number of transversions respectively, then the (partial) minus-log-likelihood
function becomes
˜lz (α, β) = −n1 log(0.25 + 0.25 exp(−4βt) + 0.5 exp(−2(α + β)t))

−n2 log(0.25 + 0.25 exp(−4βt) − 0.5 exp(−2(α + β)t))
−n3 log(0.25 − 0.25 exp(−4βt)).
Direct computations with the minus-log-likelihood function are even more hideous in
the (α, β)-parametrization for the Kimura model than for the Jukes-Cantor model.
A reparameterization is possible but we refrain from the further theoretical analysis
of the model. Instead we turn to the Hepatitis C virus data, which we analyzed
in Example 3.3.16 using the Jukes-Cantor model, we apply a standard multivariate
numerical optimization algorithm, for instance optim in R, for computing the max-
imum likelihood estimates of α and β. The results for this example are summarized
in the following table:
Segment
A B C
n1 2532 1259 1009
n2 63 20 14
n3 15 5 6
α̂ 1.9 × 10−3 1.2 × 10−3 1.2 × 10−3
β̂ 2.2 × 10−4 1.5 × 10−4 2.3 × 10−4
2.4 2.4 2.4

150
126
420
2.2 2.2 2.2
148
124
2.0 2.0 2.0

415
146 122
1.8 1.8 1.8
β × 10−4
β × 10−4
β × 10−4
410
144 120
1.6 1.6 1.6
1.4 1.4 1.4 118

405 142
1.2 1.2 1.2

116
140
400
1.0 1.0 1.0
1.0 1.5 2.0 2.5 3.0 1.0 1.5 2.0 2.5 3.0 1.0 1.5 2.0 2.5 3.0
α × 10−3 α × 10−3 α × 10−3
Figure 3.10: The contour plots of the minus-log-likelihood function for the hepatitis C
data using the Kimura model. The three plots represent the three different segments
A, B, and C (from left to right). The corresponding MLEs are found by numerical
optimization using optim in R.
The MLEs of the pA , pB and pC vectors do not change, and we can compute the
table of expected mutations
H90 H90
A C G T A C G T
A 1 11 1 A 1.4 11.7 1.4
H77 C 4 1 20 H77 C 2.3 2.3 19.0
G 13 3 1 G 18.4 2.2 2.2
T 3 19 1 T 1.7 14.0 1.7
Observed mutations - segment A Expected mutations - segment A
Compared to the Jukes-Cantor model, the expectations based on the estimated

Kimura model seem to be in greater concordance with the observed expectations.
⋄
Exercises
Exercise 3.3.1. Consider the Jukes-CantorPmodel in Example 3.3.15. Define the

variable Zi = 1(Xi 6= Yi ). Argue that N1 = ni=1 Zi follows a binomial distribution
with success parameter
p(α) = 0.25 − 0.25 exp(−4αt).
For the binomial distribution we know that the MLE is

1
p̂ = N1 .
n
Show that the MLE of α in Example 3.3.15 exists if and only if there is a solution to
the equation p(α) = p̂ and show that the resulting solution is then precisely α̂.
Color blindness is a so-called X-linked recessive trait. This means that the gene re-
sponsible for color blindness is located on the X chromosome and that color blindness
is recessive. Females who have two X chromosomes are thus color blind if the allele
resulting in color blindness (the CB-allele in the following) is present on both X
chromosomes whereas males, who have only a single X chromosome, are color blind
whenever the CB-allele is present on their X chromosome.
We denote the proportion of males in a population that have the CB-allele by p ∈
(0, 1). This is the parameter we want to estimate.
We assume in the following that we have observations from m randomly selected
males, x1 , . . . , xm ∈ {0, 1}, such that xi = 1 if male number i is color blind (has the
CB-allele). We assume that the observations are independent.
Exercise 3.3.2. Argue that the probability of observing x = (x1 , . . . , xm ) equals
pmb (1 − p)mB
Pm
where mb = i=1 xi is the number of males with the CB-allele and mB = m − mb
is the number of males without the CB-allele, find the likelihood function and the
minus-log-likelihood function for p and show that the MLE equals
mb
p̂ = .
m
Assume in addition that we have observations from f randomly selected females,
y1 , . . . , yf ∈ {0, 1}, where yi = 1 if female number i is color blind, that is, if she has
two CB-alleles. We will assume that the allele distribution in the total population
satisfies the Hardy-Weinberg equilibrium, which means that the proportion of females
with 2 CB-alleles is p2 , the proportion with 1 is 2p(1 − p) and the proportion with 0
is (1 − p)2 . We assume that the observations are independent and also independent
of the male observations above.
Exercise 3.3.3. Argue that the probability that yi = 1 equals p2 and the probability
that yi = 0 equals 2p(1 − p) + (1 − p)2 = (1 − p)(1 + p). Letting y = (y1 , . . . , yf ) argue
that the probability of observing (x, y) equals
pmb +2fb (1 + p)fB (1 − p)mB +fB

Pf
where fb = i=1 yi is the number of females with two CB-alleles (that are color
blind) and fB = f − fb is the number of females with at most one CB-allele. Having
observed (x, y) find the likelihood function and the minus-log-likelihood function for
p and show that the MLE equals
p
−mB + m2B + 4n(mb + 2fb )
p̂ =
2n
where n = 2(fb + fB ) + mb + mB = 2f + m is the total number of X chromosomes in
the sample of males and females. In a study we find fb = 40, fB = 9032, mb = 725
and mB = 8324. Compute the MLE of p.
For the following exercises we consider n independent random variables X1 , . . . , Xn
and we assume that the distribution of Xi is the Poisson distribution with mean
value parameter
λi = eβyi +α
where y1 , . . . , yn ∈ R are known but α, β ∈ R are unknown. We have the observation

x = (x1 , . . . , xn ) ∈ Nn0 and the objective is to estimate α, β.
Exercise 3.3.4. Show that the minus-log-likelihood function is

n
X
lx (α, β) = eβyi +α − βxi yi − αxi + log xi !
i=1
Exercise 3.3.5. Fix β and show that for fixed β the minimum of the minus-log-
likelihood function in α is Pn
xi
α̂(β) = log Pni=1 βy .
i=1 e
i
Exercise 3.3.6. Show that the profile minus-log-likelihood function in β is

n Pn !
X j=1 xi
lx (α̂(β), β) = xi − βxi yi − log Pn βyj
xi + log xi !
i=1 j=1 e
and that the minimizer of the profile minus-log-likelihood solves the equation
Pn Pn
yi eβyi xi yi
Pi=1
n βy
= Pi=1
n .
i=1 e i=1 xi
i
Exercise 3.3.7. Implement the Newton-Raphson algorithm for solving the equation
in β above and implement then a function for estimation of (α, β) for a given dataset
x1 , . . . , xn and y1 , . . . , yn .
3.4 Hypothesis testing
If we have a statistical model on a sample space E given by a parametrized family

of probability measures (Pθ )θ∈Θ and if Θ0 ⊆ Θ is a subset of the parameter space
we might naturally ask if the smaller model (Pθ )θ∈Θ0 is in fact an adequate model?
A statistical test is a procedure so that we, for a given observation x ∈ E, can
decide whether it is conceivable that the unknown parameter, θ, is to be found in
Θ0 or whether we have to use the entire parameter space Θ.
This is formalized as follows. We call the hypothesis that θ ∈ Θ0 the null-hypothesis
and write
H 0 : θ ∈ Θ0 .
We call θ ∈ Θ\Θ0 the alternative.

A procedure for deciding whether the null-hypothesis is true can be stated as a
division of the sample space E into two disjoint events A and R such that E = A∪R.
We call A the acceptance region and R the rejection region. If we observe x ∈ R
then we say that we reject the null-hypothesis and if x ∈ A we say that we accept
the null-hypothesis. The power for θ ∈ Θ of the test is defined as
β(θ) = Pθ (R).
The level of the test is defined as
α = max β(θ) = max Pθ (R)

θ∈Θ0 θ∈Θ0
as the maximal probability for rejecting the null-hypothesis over all possible choices
of θ from the null-hypothesis. That is, α gives the largest probability of rejecting
the null-hypothesis by mistake. For θ ∈ Θ\Θ0 the power, β(θ), is the probability of
correctly rejecting the null-hypothesis under the specific alternative θ.
A good test has small level α and large power β(θ) for all θ ∈ Θ\Θ0 . However, these
two requirements are at odds with one another. If we enlarge the acceptance set,
say, the level as well as the power goes down and if we enlarge the rejection set, the
level as well as the power goes up.
In practice we always specify the acceptance and rejection regions via a test statistic.
A test statistic is a function h : E → R, such that with a given choice of threshold
c ∈ R the acceptance region is defined as
A = {x ∈ E | h(x) ≤ c}.
This is called a one-sided test. Sometimes we encounter two-sided tests defined as
A = {x ∈ E | c1 ≤ h(x) ≤ c2 }
Hypothesis testing 163
for c1 , c2 ∈ R. Often the two-sided tests are symmetric with c1 = −c2 in which case
we can just as well rephrase the test as a one-sided test with the test statistic |h|.
Note that for the one-sided test we have A = h−1 ((−∞, c]), hence
Pθ (A) = Pθ (h−1 ((−∞, c])) = h(Pθ )((−∞, c])
by the definition of what the transformed probability measure h(Pθ ) is. The point
is that to compute the power and level of a test based on a test statistic h we need
to know the transformed probability measure h(Pθ ).
If we have a one-sided test statistic h we reject if h(x) is too large. If we have
settled for a level α-test we use the distribution of the test statistic to compute the
corresponding threshold, which is the (1−α)-quantile for the distribution, and which
we often refer to as the level α critical value or just the critical value for short. If we
reject a hypothesis we often say that the conclusion is statistically significant and
the critical value is sometimes called the significance level.
Example 3.4.1. In Section 2.12 we considered local alignment of protein sequences.

The problem can be formalized as a problem of testing a hypothesis. The setup is
that the two sequences of letters X1 , . . . , Xn and Y1 , . . . , Ym are sampled from some
(joint) distribution. Our null hypothesis is that the two sequences are independent.
The optimal local alignment score is our test statistic and large values are critical,
thus it is a one-sided test. The problem is to find the distribution of the test statistic
so that we can compute the critical value. To get a level α test we need to compute
the 1 − α quantile for the distribution of the test statistic.
In Section 2.12 we discussed the use of the Gumbel distribution as an approximation
to the distribution of the optimal local alignment score. In this case the critical value
becomes
1 log(1 − α)
c(α) = − log − .
λ Knm
⋄
3.4.1 Two sample t-test
We develop in this section a classical test statistic, the two sample t-test, where we
use Example 3.1.4 as inspiration. In that example we consider gene expression of a
particular gene for two different groups of individuals. The sample space is R79 and
the observations are the log-expression measurements.
The full model consists of specifying that the measurements are all assumed inde-
pendent and that the distribution of the log-expressions are N (µ1 , σ12 ) in group 1
and N (µ2 , σ22 ) in group 2. Equivalently we can specify the model by saying that we
observe the realization of Xi,j for i = 1 and j = 1, . . . , 37 or i = 2 and j = 1, . . . , 42,
and that
Xi,j = µi + σi εi,j
where εi,j are 79 iid random variables with the N (0, 1)-distribution.
In a slightly more general framework there are n observations in the first group
and m in the second and thus n + m observations in total. The full model has a
4-dimensional parameter vector
θ = (µ1 , σ12 , µ2 , σ22 ) ∈ Θ = R × (0, ∞) × R × (0, ∞).
The interesting null-hypothesis is
H 0 : µ1 = µ2 ,
that is,
Θ0 = {(µ1 , σ12 , µ2 , σ22 ) ∈ Θ | µ1 = µ2 }.
As in Example 3.1.4 we consider the estimators

n m
1X 1 X
µ̂1 = X1,j µ̂2 = X2,j ,
n m
j=1 j=1
which are also the maximum-likelihood estimators in this model according to Ex-
ample 3.3.11.
If we consider the difference of the estimators we get that
n m
1X 1 X
µ̂1 − µ̂2 = X1,j − X2,j
n m
j=1 j=1
Xn m
1 1 X
= (µ1 + σ1 ε1,j ) − (µ2 + σ2 ε2,j )
n m
j=1 j=1
n m
σ1 X σ2 X
= µ1 − µ2 + ε1,j + −ε2,j .
n m
j=1 j=1
Since all the εi,j ’s are assumed independent and N (0, 1)-distribution we can, using
the result in Math Box 2.10.20, find that

σ2 σ2
µ̂1 − µ̂2 ∼ N µ1 − µ2 , 1 + 2 .
n m
If we choose µ̂1 − µ̂2 as the test statistic – using a symmetric, two-sided test so that
we reject if |µ̂1 − µ̂2 | > c – the normal distribution above tells us how to compute
the power of the test and in particular the level. Under the null-hypothesis it holds
that µ1 − µ2 = 0, but the distribution of the test statistic still depends upon the
unknown parameters σ12 and σ22 . If we thus want to choose the critical value c such
that the probability of wrongly rejecting the null-hypothesis is ≤ α we choose the
σ2 σ2
1 − α/2-quantile for the normal distribution with mean 0 and variance n1 + m2 . Since
this is a scale transformation of the N (0, 1)-normal distribution we can compute the
quantile by Example 2.9.10 as
r
σ12 σ22
c(α) = + z1−α/2
n m
where z1−α/2 is the 1 − α/2-quantile for the N (0, 1)-distribution. For α = 0.05 the
quantile is 1.96 and for α = 0.01 the quantile is 2.58.
We encounter a practical problem. Even if we have decided that a level 5% test
is the relevant test to use we cannot compute the corresponding threshold for the
given test statistic because it depends upon the unknown parameters σ12 and σ22 . A
widespread solution is to use the plug-in principle and simply plug in the estimators
of the unknown parameters in the formula for the threshold. The resulting threshold
becomes r
σ̃12 σ̃22
c̃α = + z1−α/2 ,
n m
and we reject the null-hypothesis if |µ̂1 − µ̂2 | > c̃α .
Returning to the computations in Example 3.1.4 and taking α = 0.05 the threshold
becomes r
0.659 0.404
c̃α = + 1.96 = 0.32
37 42
and since the test statistic in this case equals 1.20 we reject the hypothesis that
the group means are equal. Had we taken α = 0.01 the threshold would be 0.46
and we would still reject. Another way of phrasing this is that the difference in the
estimated means is large compared to the variance in the data – so large that it is
very unlikely that it will be this large by chance if the mean value parameters are
equal.
Because we use estimated values for the unknown variance parameters the compu-
tations above are not exact. Though we aim for a level of the test being α it is not
necessarily precisely α. The problem is most pronounced when n and m are small,
2-5, say, in which case one can make a gross mistake.
We will show later that the empirical variance,
n
1X
σ̃ 2 = (Xi − µ̂),
n
i=1
which is also the maximum-likelihood estimator in the normal model, systematically

underestimates the variance parameter by a factor (n − 1)/n. This is something
that will be made more precise later on. For the present discussion it leads to the
alternative variance estimator
n
1 X
σ̂ 2 = (Xi − µ̂),
n−1
i=1
which is by far the most used estimator of the variance in practice. This is for
instance the estimator computed by var in R. When n = 5 the new estimator is a
factor 5/4 = 1.25 larger – a considerable amount. For large n the difference between
the two estimators becomes negligible. Effectively, using the larger variance estimator
increases the threshold and consequently our conclusions will be more conservative –
we are less likely to reject the hypothesis that the mean value parameters are equal.
R Box 3.4.1 (T-test). The two sample t-test can be carried out using the
t.test function in R. There are two essentially different ways for using the
function. If the data for the two groups are stored in the two vectors x and
y you compute the t-test by
> t.test(x,y)
If the two vectors contain the gene expression measurements from the two
groups considered in Example 3.1.4 the output is
Welch Two Sample t-test
data: x and y
t = 7.1679, df = 67.921, p-value = 7.103e-10
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.8678568 1.5374938
sample estimates:
mean of x mean of y
8.536492 7.333816
Alternatively, the data for the two groups may be stored in a data frame
myData with one column labeled value, say, containing the measure-
ments/observations and another column labeled groups, say, which is a
factor with two levels. Then we can compute the t-test by
> t.test(value~group,data=myData)
The default setting for t.test is to compute the t-test without assuming
equal variances. We can specify equal variances by setting var.equal=TRUE.
There might, however, still be problems with using the approximation. In the at-
tempt to remedy the problems we can choose to consider the t-test statistic
µ̂1 − µ̂2
T =q 2 .
σ̂1 σ̂22
n + m
Unfortunately we cannot obtain a useful, exact representation of the distribution of

this test statistic. A common approximation of the distribution of the t-test statistic
under the null-hypothesis that µ1 = µ2 is as a t-distribution with degrees of freedom
(σ̂12 /n + σ̂22 /m)2

df = . (3.11)
(σ̂12 /n)2 /(n − 1) + (σ̂22 /m)2 /(m − 1)
The t-distribution was considered in Exercise 2.6.4. The degrees of freedom equals
twice the shape parameter of the t-distribution, that is df = 2λ in the notation of
Exercise 2.6.4. The t-distribution does not have other parameters and once we fix
the degrees of freedom we can compute the (1 − α/2)-quantile w1−α/2 and reject the
hypothesis if
|T | > w1−α/2 ,
which is equivalent to rejecting if
r
σ̂12 σ̂22
|µ̂1 − µ̂2 | > + w1−α/2 .
n m
For α = 0.05 the 1 − α/2 = 0.975-quantile for the t-distribution with 2 degrees of
freedom is 4.3, with 5 degrees of freedom it is 2.57 and with 20 degrees of freedom
it is 2.09. It always holds that w1−α/2 ≥ z1−α/2 and the effective change of using
the approximation with the t-distribution is once again that our conclusions get
more conservative – we are less likely to reject the hypothesis that the mean value
parameters are equal.
If we are willing to make one extra assumption, it is possible to derive exact results.
If σ12 = σ22 and if we denote this common parameter by σ 2 we have reduced the
model to a model with three parameters (µ1 , µ2 , σ 2 ) ∈ R × R × (0, ∞). Under this
assumption
2 1 1
µ̂1 − µ̂2 ∼ N µ1 − µ2 , σ + .
n m
We choose to estimate σ 2 using the so-called pooled variance estimator defined as

 
Xn Xm
1  (X1,j − µ̂1 )2 +
σ̂ 2 = (X2,j − µ̂2 )2  . (3.12)
n+m−2
j=1 j=1
In this setup we introduce the t-test statistic

r
µ̂1 − µ̂2 n + m µ̂1 − µ̂2
T =q = ,
σ̂ 2 1 + 1 nm σ̂
n m
It can be shown that under the null-hypothesis that µ1 = µ2 the distribution of T

is exactly a t-distribution with degrees of freedom n + m − 2.
If the degrees of freedom for the tests – with or without the assumption of equal
variances – is large, the quantiles for the t-distribution are close to the corresponding
quantiles for the normal distribution. For doing a single test the practical difference
by using the normal distribution is minor for n ≥ 20.
To summarize the conclusions from the derivations above, we consider the null-
hypothesis
H 0 : µ1 = µ2 .
If we assume equal variances we compute the two-sample t-test
r
nm µ̂1 − µ̂2
T = , (3.13)
n+m σ̂
where σ̂ 2 is the pooled variance estimate given by (3.12) and we reject the hypothesis
if |T | > w1−α/2 where w1−α/2 is the 1 − α/2 quantile for the t-distribution with
n + m − 2 degrees of freedom.
If we do not assume equal variances we compute the two-sample t-test
µ̂1 − µ̂2
T =q 2 (3.14)
σ̂1 σ̂22
n + m
and we reject the hypothesis if |T | > w1−α/2 where w1−α/2 is the 1 − α/2 quantile
for the t-distribution with degrees of freedom given by (3.11). In this case where we
do not assume equal variances the test is often referred to as the Welch two-sample
t-test.
The relevant quantiles can be computed in R using the quantile function qt for the
t-distribution. However, the t-test can also be computed using the t.test function.
This function reports the conclusion of the test in terms of a p-value. If the computed
t-test statistic is equal to t for the concrete dataset then the p-value is
p = P(|T | > |t|)
where T has the t-distribution with df degrees of freedom. Alternatively, because the
t-distribution is symmetric
p = 2(1 − Fdf (|t|))
where Fdf is the distribution function for the t-distribution with df degrees of free-
dom. The null-hypothesis is rejected at level α if and only if the p-value is ≤ α.
There is one natural question left. Should we choose the Welch t-test or should
we assume equal variances? Technically there is no problem in using the Welch t-
test though the t-distribution used is not exact. If the estimated variances are close
to each other there will only be minor differences between the Welch t-test and the
equal variance t-test, and if they are not, the equal variance t-test is not appropriate.
Should we actually make a formal statistical test of the hypothesis that the variance
parameters σ12 and σ22 are equal? If we reject, then we use the Welch t-test, and
otherwise we use the equal variance t-test. Such a procedure has been criticized in
0.6
0.5
0.4
0.3
0.2
0.1
0.0
5 6 7 8 9 10 11
Figure 3.11: The densities for the normal distributions with estimated parameters
for the gene expression data for gene 1635 at. We reject the hypothesis that the
mean value parameters are equal.
the literature. An introduction of a test of equal variances, which dictates which

t-test to use destroys properties of the subsequent t-tests and it turns out to be
a better idea simply to use the Welch t-test. We should, however, be careful with
the conclusions if the variances really are different. If we reject the hypothesis of
equal means it is not so clear how we interpret that one of the means is larger than
the other if also the variances differ. For the gene expression example the estimated
means for group 1 is larger than for group 2 and we have rejected the hypothesis that
they are equal. However, the estimated variance for group 1 is also larger than for
group 2, which implies that the larger mean alone does not imply that the normal
distribution we have fitted for group 1 is unequivocally “to the right” of the normal
distribution fitted to group 2.
3.4.2 Likelihood ratio tests
In the following definition we suppose that X is a random variable with distribution

Pθ for some θ ∈ Θ0 and that we have a likelihood function Lx (θ) for θ ∈ Θ when
observing X = x.
Definition 3.4.2. Observing X = x
maxθ∈Θ0 Lx (θ)
Q(x) =
maxθ∈Θ Lx (θ)
is called the likelihood ratio test statistic. Since Θ0 ⊆ Θ, we have Q(x) ∈ (0, 1]. Small
values of Q(x) are critical.
To use the test statistic above we need to know its distribution under Pθ for θ ∈ Θ0 .
It is in general impossible to find this distribution, but in many situations of practical
importance we can find a useful approximation. We state this as a theorem, though
it is not precise in terms of the prerequisites required for the approximation to be
valid.
Result 3.4.3. If Θ is a d-dimensional parameter space and Θ0 d0 -dimensional, the
distribution of
−2 log Q(X)
can be approximated by a χ2 -distribution with d − d0 degrees of freedom. Large values
of −2 log Q(x) are critical.
Remark 3.4.4. The “dimension” of Θ and Θ0 is a little too involved to define in
a precise mathematical sense. It essentially covers the more intuitive idea of “the
number of free parameters”. In practice, it is often easy to compute the dimension
drop d − d0 as this is simply the number of (different) restrictions that we put on the
parameters in Θ to get Θ0 . For instance, in the example above with the 2-sample t-
test the dimension of Θ is 3 (or 4), the dimension of Θ0 is 2 (or 3) and the dimension
drop is 1.
If θ̂(x) denotes the maximum-likelihood estimate and θ̂0 (x) the maximum-likelihood
estimate under the null hypothesis it follows that
−2 log Q(x) = 2(− log Lx (θ̂0 (x)) + log Lx (θ̂(x))) = 2(lx (θ̂0 (x)) − lx (θ̂(x)))
Having computed −2 log Q(x) we often report the test by computing a p-value. If Fdf
denotes the distribution function for the χ2 -distribution with df degrees of freedom
the p-value is
p = 1 − Fdf (−2 log Q(x)).
This is the probability for observing a value of −2 log Q(X) under the null-hypothesis
that is as large or larger than the observed value −2 log Q(x).
Example 3.4.5. For the Kimura model in Example 3.3.18 we observe that the
null-hypothesis
H0 : α = β
is equivalent to the Jukes-Cantor model. If we consider Segment A of the virus
genome we can compute ˜lx (α̂, β̂) = 399.2 and under the null-hypothesis ˜lx (α̂0 , α̂0 ) =
436.3. We find that
−2 log Q(x) = 74.2.
Under the null-hypothesis we make a single restriction on the parameters, and the
p-value using a χ2 -distribution with 1 degree of freedom is 7.0 × 10−18 , which is
effectively 0. This means that we will by all standards reject the null-hypothesis over
the alternative. That is to say, we reject that the Jukes-Cantor model is adequate
for modeling the molecular evolution of Segment A. ⋄
One word of warning. We computed ˜lx above instead of the full minus-log-likelihood.
We did so because the remaining part of the full minus-log-likelihood does not involve
the parameters α and β and is unaffected by the hypothesis. All terms that remain
constant under the full model and the hypothesis can always be disregarded as the
difference in the computation of −2 log Q is unaffected by these terms. However,
be careful always to disregard the same terms when computing lx (θ̂0 (x)) as when
computing lx (θ̂(x)).
Example 3.4.6. Continuing Example 3.3.18 we will investigate if the three segments
can be assumed to have the same parameters. Thus for the full model we have
six parameters αA , βA , αB , βB , αC , βC , two for each segment. We set up the null-
hypothesis
H0 : αA = αB = αC , βA = βB = βC .
We find that −2 log Q(x) = 6.619, and since the full model has 6 free parameters
and the model under the null-hypothesis has 2, we compute the p-value using the
χ2 -distribution with 4 degrees of freedom. The p-value is 0.157 and we do not reject
the hypothesis that all three segments have the same parameters. ⋄
The methodology of statistical testing is very rigid. Formally we have to set up the
hypothesis prior to considering the data, since testing a hypothesis that is formu-
lated based on the data almost automatically demolishes the assumptions that are
used to derive the distribution of the test statistics. This makes statistical testing
most appropriate for confirmatory analyses where we know what to expect prior to
the actual data analysis and want to confirm and document that our beliefs are cor-
rect. On the other hand, exploratory data analysis where we don’t know in advance
what to expect is an important part of applied statistics. Statistical testing is used
anyway as an exploratory tool to investigate a range of different hypotheses, and
there are numerous algorithms and ad hoc procedures for this purpose. The merits
of such procedures are extremely difficult to completely understand. It is, however,
always important to understand how to correctly interpret a hypothesis test and
the conclusions we can draw. Disregarding whether we use test statistics for a less
formal, exploratory analysis or a formal confirmatory analysis we have to remember
that if we accept a null-hypotheses we do in fact have little evidence that the hy-
pothesis is true. What we can conclude is that there is no evidence in the data for
concluding that the hypothesis is false, which is a considerably vaguer conclusion!
A hypothesis may be screamingly wrong even though we are unable to document
it. If the test we use has little power against a particular alternative, it will be very
difficult to detect such a deviation from the null-hypothesis. On the other hand,
if we reject a null-hypothesis we may be rather certain that the null-hypothesis is
false, but “how false” is it? If we have a large dataset we may be able to statistically
detect small differences that are of little practical relevance. Statistical significance
provides documentation for rejecting a hypothesis, e.g. that there is a difference of
the mean values for two groups, but does not in itself document that the conclusion
is important or significant in the usual sense of the word, e.g. that the difference of
the mean values is of any importance.
3.4.3 Multiple testing
One of the problems with formal statistical testing is that the more tests we do the
less reliable are our conclusions. If we make a statistical test at a 5%-level there
is 5% chance the we by mistake reject the hypothesis even though it is true. This
is not a negligible probability but 5% has caught on in the literature as a suitable
rule-of-thumb level. The problem is that if we carry out 100 tests at a 5%-level then
we expect that 1 out of 20 tests, that is, 5 in total, reject the null-hypothesis even if
it is true in all the 100 situations. What is perhaps even worse is that the probability
of rejecting at least one of the hypothesis is in many cases rather large. If all the
tests are independent, the number of tests we reject follows a binomial distribution
with parameters n = 100 and p = 0.05, in which case the probability of rejecting at
least one hypothesis if they are all true is 1 − (1 − 0.05)100 = 99.4%.
If we carry out 100 two-sample t-tests on different4 datasets and find that at a
5% level we rejected in 4 cases the hypothesis that the means are equal, does this
support a conclusion that the means are actually different in those 4 cases? No, it
does not. If we reject in 30 out of the 100 cases we are on the other hand likely
to believe that for a fair part of the 30 cases the means are actually different. The
binomial probability of getting more than 10 rejections is 1.1% and getting more
than 20 rejections has probability 2.0 × 10−8 . But for how many and for which of
the 30 cases can we conclude that there is a difference? A natural thing is to order
(the absolute value of) the test statistics
|t(1) | ≤ . . . ≤ |t(100) |
and then take them from the top and down.
Example 3.4.7. We consider the ALL microarray dataset introduced in Example

2.7.3. There are a total of 12625 genes represented on the array. We do a t-test for
each gene where we test if there is a difference in the mean value between those with
the BCR/ABL fusion gene and those without. To facilitate the comparison with a
single t-distribution we carry out the tests under the assumption of equal variances
in the two groups. The top 10 list of t-tests are as follows:
4
Formally we need independent datasets for some of the subsequent quantitative computations
to be justified but qualitatively the arguments hold in a broader context.
Histogram of p−values for multiple t−tests
500
4
2
400
0
Sample Quantiles
300
Frequency
−2
200
−4
−6
100
−8
0
−4 −2 0 2 4 0.0 0.2 0.4 0.6 0.8 1.0
Theoretical Quantiles P−value
Figure 3.12: QQ-plot of the 12,625 t-test statistics for the ALL dataset (left) and
histogram of the corresponding p-values.
name t-test statistics p-value

1636 g at −9.26 3.76e − 14
39730 at −8.69 4.79e − 13
1635 at −7.28 2.45e − 10
1674 at −6.90 1.28e − 09
40504 at −6.57 5.27e − 09
37015 at −6.19 2.74e − 08
40202 at −6.18 2.79e − 08
32434 at −5.78 1.54e − 07
37027 at −5.65 2.60e − 07
39837 s at −5.50 4.74e − 07
We find our earlier considered gene, 1635 at, as number three from the top on this
list. Figure 3.12 shows that QQ-plot of the computed t-test statistics against the
t-distribution with 77 degrees of freedom together with a histogram of the p-values.
The QQ-plot bends in a way that indicates that there are too many large and small
values in the sample. This is confirmed by the histogram of p-values, which shows
that there are in the order of several hundred p-values too many in the range from
0 to 0.01. ⋄
The conclusion in the example above is that there are a number of the cases where
we should reject the hypothesis – even in the light of the fact that we do 12625
tests. The real question is that if we continued the list above, when should we stop?
What should the threshold for the t-test statistic be in the light of the multiple tests
carried out?
If we want to control the probability of including just a single wrong rejection we

talk about controlling the family wise error rate. If all the tests are independent and
each at level α the probability of not rejecting a single one if they are all true is
1 − (1 − α)n if we make n tests. If we want to keep this value at a 5% level, say, we
solve and find that
α = 1 − (1 − 0.05)1/n = 1 − (0.95)1/n .
With n = 12625 as in the example this gives an α = 4.1 × 10−6 . Thus each test has
to be carried out at the level 4.1 × 10−6 to be sure not to reject a true hypothesis.
This can be a very conservative procedure.
Current research suggests that for large, multiple testing problems focus should
change from the family wise error rate to other quantities such as the false discovery
rate, which is the relative number of falsely rejected hypotheses out of the total num-
ber of rejected hypotheses. The book Multiple testing procedures with applications
to genomics by Dudoit and van der Laan (Springer, 2008) treats this and a number
of other issues in relation to multiple testing problems. A very pragmatic viewpoint
is that the multiple testing problem is a simple matter of choosing a suitable thresh-
old to replace the critical value used for a single test. How to do this appropriately
and how to interpret the choice correctly can be a much more subtle problem, but
ordering the tests according to p-value is almost always a sensible thing to do.
Exercises
Exercise 3.4.1. Consider the setup for Exercise 3.3.4 and the null-hypothesis
H0 : β = 0
Interpret the hypothesis and compute a formula for −2 log Q(x) for testing this hy-
pothesis. What is the approximating distribution of this test statistic under the null-
hypothesis?
Exercise 3.4.2. Make a simulation study to investigate the distribution of the test
statistics −2 log Q(X) for the hypothesis considered in Example 3.4.5. That is, use the
estimated Jukes-Cantor model to simulate new datasets, 200 say, compute the corre-
sponding −2 log Q(x) statistics for each dataset and compare the resulting empirical
distribution with the χ2 -distribution with 1 degree of freedom.
Confidence intervals 175
3.5 Confidence intervals
The formal statistical test answers a question about the parameter in terms of the
data at hand. Is there evidence in the data for rejecting the given hypothesis about
the parameter or isn’t there? We can only deal with such a question in the light of
the uncertainty in the data even if the hypothesis is true. The distribution of the test
statistic captures this, and the test statistic needs to be sufficiently large compared
to its distribution before we reject the hypothesis.
There is another, dual way of dealing with the uncertainty in the data and thus
the uncertainty in the estimated parameters. If we consider a real valued parameter
then instead of formulating a specific hypothesis about the parameter we report an
interval, such that the values of the parameter in the interval are conceivable in the
light of the given dataset. We call the intervals confidence intervals.
If (Pθ )θ∈Θ is a parametrized family of probability measures on E, and if we have an
observation x ∈ E, then an estimator θ̂ : E → Θ produces an estimate θ̂(x) ∈ Θ. If
the observation came to be as a realization of an experiment that was governed by
one probability measure Pθ in our parametrized family (thus the true parameter is
θ), then in most cases θ̂(x) 6= θ – but it is certainly the intention that the estimate
and the true value should not be too far apart. We attempt here to quantify how far
away from the estimate θ̂(x) it is conceivable that the true value of the parameter
is.
Example 3.5.1. Let X1 , . . . , Xn be iid Bernoulli distributed with success probabil-
ity p ∈ [0, 1]. Our parameter space is [0, 1] and the unknown parameter is the success
probability p. Our sample space is {0, 1}n and the observation is an n-dimensional
vector x = (x1 , . . . , xn ) of 0-1-variables. We will consider the estimator
n
1X
p̂ = Xi ,
n
i=1
which Pis the relative frequency of 1’s (and the MLE as well). The distribution of
np̂ = ni=1 Xk is a binomial distribution with parameters (n, p), which implicitly5
gives the distribution of p̂.
If we in this example take z(p) and w(p) to be the 0.025- and 0.975-quantiles for the
binomial distribution with parameters (n, p) we know that
Pp (z(p) ≤ np̂ ≤ w(p)) ≃ 0.95.
The reason that we don’t get exact equality above is that the binomial distribution
is discrete, so the distribution function has jumps and we may not be able to obtain
exact equality. If we now define
I(p̂) = {p ∈ [0, 1] | z(p) ≤ np̂ ≤ w(p)}
5
The distribution of p̂ is a distribution on {0, 1/n, 2/n, . . . , 1} – a set that changes with n – and
the convention is to report the distribution in terms of np̂, which is a distribution on Z.
1.0
1.0
0.8
0.8
0.6
0.6
p
p
^
^
0.4
0.4
0.2
0.2
0.0
0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
p p
Figure 3.13: These figures show 95%-confidence intervals for the parameter p in the
binomial distribution for different possible realizations of the estimator p̂ (on the
y-axis) with n = 10 (left) and n = 30 (right). For a given estimate p̂(x) = y we can
read of which p (those on the line) that could produce such an estimate. Note the
cigar shape.
we have Pp (p ∈ I(p̂)) ≃ 0.95. We can find the interval I(p̂) by reading it off from a
figure as illustrated in Figure 3.13. Note that the probability statement is a statement
about the random interval I(p̂). It says that this random interval will contain the
true parameter with probability 0.95 and we call I(p̂) a 95%-confidence interval.
It will be shown in a later chapter that the variance of np̂ is np(1 − p), and if we
approximate the binomial distribution B(n, p) with the N (np, np(1−p)) distribution,
which will also be justified later, we arrive at the following convenient approximation
p ! p !
p̂(1 − p̂) p̂(1 − p̂)
z(p) ≃ n p − 1.96 √ and w(p) ≃ n p + 1.96 √
n n
If we plug this approximation into the formula for the confidence interval we get
" p p #
p̂(1 − p̂) p̂(1 − p̂)
I(p̂)) = p̂ − 1.96 √ , p̂ + 1.96 √ .
n n
The approximation does not work well if n is too small or if the true p is too close
to 0 or 1. ⋄
For a single parameter as in Example 3.5.1 we report an interval. How large the
interval is depends upon how certain – or confident – we want to be that the true
parameter is in the interval. In the example we chose a 95%-confidence interval, which

has become a de facto standard. For general parameters the following definition tells
what a confidence set is.
Definition 3.5.2. A confidence set for the parameter θ given the observation x ∈ E
is a subset I(x) ⊆ Θ. If for each x ∈ E we have given a confidence set I(x) we say
that (I(x))x∈E are (1− α)-confidence sets for the unknown parameter if for all θ ∈ Θ
Pθ (θ ∈ I(X)) ≥ 1 − α. (3.15)
We call 1 − α the coverage of the confidence sets.
If Θ ⊆ R and I(x) is an interval we call I(x) a confidence interval. We emphasize

again that Pθ (θ ∈ I(X)) is a probability statement prior to conducting the experi-
ment about whether the random confidence set I(X) will contain the parameter, and
not whether the parameter belongs to the confidence set I(x) after having observed
the realization x of X. This is a very subtle point about the interpretation of confi-
dence sets. It is the observation and therefore the confidence set that is a realization
of the random experiment and not the unknown parameter. For a given realization
x we simply can’t tell whether I(x) contains θ or not, since θ is unknown. But if we
a priori to making the experiment decide upon a family of (1 − α)-confidence sets
that we will choose among depending on the observation, then we know that the
probability that the confidence set we end up with really contains θ is at least 1 − α
no matter what θ is. If α is chosen small, α = 0.05 say, then we are pretty confident
that θ is actually in I(x) and if α = 0.01 we are even more so.
Example 3.5.3. We consider the statistical model for just group 1 in Example
3.1.4, which states that we observe
Xj = µ + σεj
for j = 1, . . . , 37 where the εj ’s are iid N (0, 1). Assume for the sake of simplification
that σ is known. The parameter is then just µ and the parameter space is R, the
sample space is R37 and we observe x = (x1 , . . . , x37 ).
1 P37
As usual µ̂ = 37 j=1 Xj and we introduce the statistic
37
1 X
h(X, µ0 ) = µ̂ − µ0 = (µ − µ0 ) + σ εj .
37
j=1
2
If µ = µ0 the distribution of this statistic is N (0, σ37 ) and if we recall that 1.96 is
the 0.975-quantile for the N (0, 1) normal distribution

σ
Pµ0 |h(X, µ0 )| ≤ 1.96 √ = 0.95
37
We find that
σ
I(x) = {µ0 ∈ R | |h(X, µ0 )| ≤ 1.96 √ }
37
σ σ
= {µ0 ∈ R | −1.96 √ ≤ µ̂ − µ0 ≤ 1.96 √ }
37 37

σ σ
= µ̂ − 1.96 √ , µ̂ + 1.96 √
37 37
If we plug in the estimated value of the standard deviation we get the 95%-confidence
interval [8.27, 8.80]. Since the standard deviation is estimated and not known we
violate the assumptions for the derivations above. Because there is an approximation
involved we say that the confidence interval has nominal coverage 95% whereas the
actual coverage may be lower. If n is not too small the approximation error is minor
and the actual coverage is close to the nominal 95%. Below we treat a method that
is formally correct – also if the variance is estimated. ⋄
Note the similarity of the construction above with a hypothesis test. If we formulate
the simple null-hypothesis
H 0 : µ = µ0 ,
we can introduce the test statistics h(x, µ0 ) as above and reject the test if this
test statistic is larger in absolute value than 1.96 √σ37 . The 95%-confidence interval
consists precisely of those µ0 where the two-sided level 5% test based on the test
statistic h(x, µ0 ) will be accepted.
This duality between confidence intervals and statistical tests of simple hypotheses
is a completely general phenomena. Consider a simple null-hypothesis
H0 : θ = θ0
then if A(θ0 ) is an acceptance region for a level α test, the set
I(x) = {θ0 | x ∈ A(θ0 )}
is a (1 − α)-confidence set. This is because θ0 ∈ I(x) if and only if x ∈ A(θ0 ), hence
Pθ0 (X ∈ A(θ0 )) = Pθ0 (θ0 ∈ I(X)).
This equality also implies that if I(x) for x ∈ E form (1 − α)-confidence sets then
the set
A(θ0 ) = {x ∈ E | θ0 ∈ I(x)}
forms an acceptance region for a level α test.
If Θ ⊆ R we naturally ask what general procedures we have available for producing
confidence intervals. We consider here intervals that are given by the test statistic
h(x, θ0 ) = θ̂(x) − θ0 ,
for any estimator θ̂ of the unknown parameter. With α ∈ (0, 1) the fundamental
procedure is to find formulas for zα (θ0 ) and wα (θ0 ), the α/2 and 1 − α/2 quantiles,
for the distribution of θ̂(x) − θ0 under the probability measure Pθ0 and define the
confidence set
I(x) = {θ0 ∈ Θ | zα (θ0 ) ≤ θ̂(x) − θ0 ≤ wα (θ0 )}.
This is what we did for the binomial and the normal distributions above. Unfortu-
nately these are special cases, and even if we know the quantiles as a function of θ0
it may not be practically possible to compute I(x) above. In general the set does not
even have to be an interval either! A computable alternative is obtained by plugging
in the estimate of θ in the formulas for the quantiles above, which gives
I(x) = [θ̂(x) − wα (θ̂(x)), θ̂(x) − zα (θ̂(x))].
Exact distributions and thus quantiles are hard to obtain, and in most cases we have
to rely on approximations. Below we present three of the most basic constructions
of approximate (1 − α)-confidence intervals.
• Suppose we have a formula se(θ0 ) for the standard deviation of θ̂ under Pθ0
– often referred to as the standard error of θ̂. Assume, furthermore, that the
distribution of θ̂(x)−θ0 can be approximated by the N (0, se(θ0 )2 )-distribution.
With zα the 1 − α/2 quantile for the N (0, 1)-distribution the general construc-
tion is
I(x) = {θ0 ∈ Θ | −se(θ0 )zα ≤ θ̂(x) − θ0 ≤ se(θ0 )zα }.
As above, it may be practically impossible to compute I(x) and it may not
even be an interval. If we plug in the estimate of θ in the formula se(θ0 ) we
arrive at the interval
I(x) = [θ̂(x) − se(θ̂(x))zα , θ̂(x) + se(θ̂(x))zα ].
• If θ̂ is the maximum likelihood estimator and the minus-log-likelihood function

is twice differentiable then
d2 lx
î(x) = (θ̂(x))
dθ 2
is known as the observed Fisher information, or just the observed information
for short. The mean of
d2 lX
(θ0 )
dθ 2
under Pθ0 is denotedqi(θ0 ) and is called
q the Fisher information. Under quite
general conditions 1/ i(θ̂(x)) or 1/ î(x) are both valid estimates of the stan-
dard error of θ̂, and if we proceed as above we get using the latter estimate,
say,  
zα zα 
I(x) = θ̂(x) − q , θ̂(x) + q .
î(x) î(x)
• Estimates ẑα and ŵα of the quantiles zα (θ0 ) and wα (θ0 ) or an estimate, se,
ˆ of
the standard error of θ̂ are found by simulations. This is known as bootstrapping
and the technicalities will be pursued below. In any case, once the estimates
have been computed one proceeds as above and computes either
I(x) = [θ̂(x) − ŵα , θ̂(x) − ẑα ],
or
I(x) = [θ̂(x) − sez
ˆ α , θ̂(x) + sez
ˆ α ].
As most of the practically usable constructions involve approximations, we say that

the resulting confidence intervals have nominal coverage 1 − α. The actual coverage
probability is the function
θ 7→ Pθ (θ ∈ I(X)),
and the sets I(x), x ∈ E, are (1−α)-confidence sets if the actual coverage probability
is larger than 1 − α for all θ. The nominal value 1 − α is what we aim for, the actual
coverage is what we get. Hopefully, if the approximations are not too bad, the actual
coverage is not much smaller than the nominal 1 − α.
Example 3.5.4. For the estimation of the probability parameter p for n iid Bernoulli
random variables the MLE is
n
1X
p̂ = Xi .
n
i=1
The minus-log-likelihood function is
lx (p) = −N log p − (n − N ) log(1 − p)
Pn
where N = i=1 Xi . Differentiation gives
N n−N N n−N N (1 − 2p) + np2
lx′ (p) = − + , î(p) = lx′′ (p) = + = .
p (1 − p) p2 (1 − p)2 p2 (1 − p)2
Since the mean of N is np the Fisher information is
np(1 − 2p) + np2 n((1 − 2p) + p) n
i(p) = 2 2
= 2
= .
p (1 − p) p(1 − p) p(1 − p)
p
If we use 1/ i(p̂) as an estimate of the standard error of p̂ and we proceed with the
standard construction of a (1 − α)-confidence interval we get
" p p #
p̂(1 − p̂) p̂(1 − p̂)
p̂ − √ zα , p̂ + √ zα
n n
where zα is the (1 − α/2)-quantile for the normal distribution N (0, 1). We find that
this interval is identical to the approximate interval considered in Example 3.5.1. For
the binomial distribution we will not use the observed Fisher information because
we have a formula for the Fisher information. The observed information is useful
when we don’t have such a formula. ⋄
2
1
1
µ
µ
0
0
^
^
−1
−1
−2
−2
−2 −1 0 1 2 −2 −1 0 1 2
µ µ
Figure 3.14: These figures show 95%- and 99%-confidence intervals for the parameter
of interest µ in the normal distribution N (µ, σ 2 ) with n = 10 (left) or n = 30 (right)
independent observations. On the figure σ 2 = 1. Reading the figure from the x-axis
the full lines give the 0.025- and 0.975 quantiles for the normal distribution with
√
variance 1/ n and mean value parameter µ, the dashed lines give the 0.005- and
0.995 quantiles. For a given estimate µ̂(x) = y we can read the figure from the y-axis
and read of which µ that could produce such an estimate. This gives the confidence
intervals.
3.5.1 Parameters of interest
In the discussion above we do in reality only treat the situation with a single, uni-
variate real parameter. In situations with more then one parameter we gave abstract
definitions of confidence sets but we did not provide any practical methods. Though
it is possible – also computationally and in practice – to work with confidence sets
for more than a univariate parameter, such sets are notoriously difficult to relate to.
When we have more than one unknown parameter we usually focus on univariate
parameter transformations – the parameters of interest.
In general, if τ : Θ → R is any map from the full parameter space into the real line,
and if we are really interested in τ = τ (θ) and not so much θ, we call τ the parameter
of interest. If θ̂ is an estimator of θ, then τ̂ = τ (θ̂) can be taken as an estimator of
τ . This is the plug-in principle.
Example 3.5.5. We consider the statistical model of X1 , . . . , Xn where we assume

that the variables are iid with the N (µ, σ 2 )-distribution. The parameter space is
Θ = R × (0, ∞) but we are often mostly interested in the mean value parameter µ.
Thus we consider the parameter of interest given by

τ (µ, σ 2 ) = µ
where τ : R × (0, ∞) → R.
When considering any (joint) estimator (µ̂, σ̂ 2 ) of (µ, σ 2 ) the plug-in estimator of µ is
simply µ̂ – that is, nothing happens. The point here is that though the estimator does
not seem to have anything to do with whether or not σ is estimated, the distribution
of µ̂ will in general depend upon the complete, unknown parameter – in this case
(µ, σ) – and not just the unknown parameter of interest itself. We have to take this
into account in the construction of confidence intervals. ⋄
Example 3.5.6. Continuing Example 3.1.4 – and the discussion in Section 3.4.1
– we considered gene expression measures for two groups of individuals. Taking
logarithms we considered the use of an additive noise model, where we assumed that
the variables within each groups are iid N (µi , σi2 ).
For the comparison of gene expression measurements – and many other assay based,
quantitative measurements of concentrations – the fold-change between the groups is
commonly regarded as an interesting parameter. Because we use the additive model
on the log-measurements (base 2), the fold change is defined as
τ (µ1 , σ12 , µ2 , σ22 ) = 2µ1 −µ2 .
Thus a difference in mean of 1 corresponds to a fold-change of 2. A difference in
mean of 2 corresponds to a fold-change of 4 etc. ⋄
Example 3.5.7. If Pθ for θ ∈ Θ is a parametrized family of probability measures
on E, a recurring problem is to estimate the probability of a specific event A ⊆ E.
We define
τ (θ) = Pθ (A).
Our parameter of interest is then the probability of this particular event.
If X1 , . . . , Xn are iid random variables taking values in E and with distribution Pθ
for some θ ∈ Θ, we may estimate τ (θ) directly as the relative frequency
n
1X
εn (A) = 1(Xi ∈ A).
n
i=1
This is not the plug-in estimator – the plug-in estimator requires that we have an
estimator, θ̂, of θ and then use τ (θ̂) = Pθ̂ (A) as an estimator of τ (θ). If necessary,
Pθ̂ (A) can be computed via simulations.
⋄
Example 3.5.8. The logistic regression model for X1 , . . . , Xn with y1 , . . . , yn fixed,
as considered in Example 3.3.17, is given by the point probabilities
exp(α + βyi )
P(Xi = 1) =
1 + exp(α + βyi )
for parameters α, β ∈ R. One interpretation of this parametrization is that the log

exp(α+βy)
odds are linear in y. That is, with p(y) = 1+exp(α+βy) ,
p(y)
log = α + βy.
1 − p(y)
The log odds equal 0 precisely when p(y) = 1/2 and this happens when y = −α/β.
The value of y where p(y) = 1/2 is called LD50 , which means the Lethal Dose for
50% of the subjects considered. In other words, the dose that kills half the flies. We
see that in terms of the parameters in our logistic regression model
α
LD50 = − .
β
It is often of greater interest to estimate LD50 than α or β separately. Thus the

parameter of interest is in this case LD50 . For the flies in Example 3.3.17 we find
that
LD50 = −1.899.
⋄
1.0
0.8
0.6
0.4
0.2
LD50 = − 1.899
0.0
−4 −3 −2 −1 0
log(concentration)
Figure 3.15: In the fly death experiment we estimate the parameter of interest, LD50 ,
to be −1.899. Thus a concentration of 0.1509 = exp(−1.899) is found to be lethal
for half the flies.
If we use the plug-in principle for estimation of the parameter of interest, we can
proceed and try to find the distribution of the statistic
τ̂ − τ0 = τ (θ̂) − τ (θ0 ).
We observe that the distribution of τ (θ̂) is the transformation via τ of the distribu-
tion of θ̂. If we are capable of finding this distribution, we can essentially use the
procedure discussed above. However, we need a minor modification of the definition

of confidence sets when we consider a parameter of interest instead of the complete
parameter.
Definition 3.5.9. A confidence set for a real valued parameter of interest τ = τ (θ)
given the observation x ∈ E is a subset I(x) ⊆ R. If we for each x ∈ E have a
given confidence set I(x) we say that (I(x))x∈E are (1 − α)-confidence sets for the
unknown parameter of interest if for all θ ∈ Θ
Pθ (τ (θ) ∈ I(X)) ≥ 1 − α. (3.16)
We call 1 − α the coverage of the confidence sets.
For the practical construction of confidence intervals we can proceed in ways very
similar to those in the previous section. We summarize the practically applicable
methods below – noting that for the constructions below it is not in general an
assumption that the estimator of τ is the plug-in estimator, but for some of the
constructions we still need an estimator θ̂ of the full parameter.
• We have formulas zα (θ0 ) and wα (θ0 ) for the α/2 and 1 − α/2 quantiles for the
distribution of τ̂ − τ in which case we can compute the interval
I(x) = [τ̂ (x) − wα (θ̂(x)), τ̂ (x) − zα (θ̂(x))].
• We have an estimator se ˆ of the standard error of τ̂ . With zα the 1 − α/2

quantile for the N (0, 1)-distribution we can compute the interval
I(x) = [τ̂ (x) − sez

ˆ α , τ̂ (x) + sez
ˆ α ].
If we have a formula se(θ0 ) for the standard error of τ̂ under Pθ0 we can use
the plug-in estimator se ˆ = se(θ̂(x)). If θ̂ is the maximum-likelihood estimator
and τ̂ = τ (θ̂) is the plug-in estimator, an estimator, se,
ˆ is obtainable in terms
of the Fisher information, see Math Box 4.7.3.
• Estimates ẑα and ŵα of the quantiles zα (θ0 ) and wα (θ0 ) or an estimate, se,
ˆ of
the standard error of τ̂ are found by bootstrapping and we compute
I(x) = [τ̂ (x) − ŵα , τ̂ (x) − ẑα ],
or
I(x) = [τ̂ (x) − sez
ˆ α , τ̂ (x) + sez
ˆ α ].
Example 3.5.10. Just as in Example 3.5.3 we consider the statistical model spec-
ified as
Xj = µ + σεj
Math Box 3.5.1 (Multivariate information). If Θ ⊆ Rd and if the minus-log-

likelihood function as a function of d variables is twice differentiable, the second
derivative is the d × d matrix denoted D2 lX (θ), cf. Math Box 3.3.1.
The Fisher d × d information matrix, I(θ), is the entry-by-entry mean value of
D2 lX (θ). If θ̂ denotes the MLE it is under quite general conditions possible to show
that
approx
θ̂ − θ ∼ N 0, I(θ)−1
under Pθ where I(θ)−1 is the matrix-inverse of I(θ). The distribution is the multi-
variate normal distribution considered in Math Box 2.13.1.
An important consequence is that if v ∈ Rd then τ (θ) = v T θ is a one-dimensional
parameter transformation (take v T = (1, 0, . . . , 0) to select the first coordinate of
θ̂, say) and
approx
v T θ̂ − v T θ ∼ N 0, v T I(θ)−1 v .
The i’the coordinate in θ̂ thus follows approximately a normal distribution with

variance (I(θ)−1 )ii .
More generally, if τ : Θ → R is differentiable
approx
τ (θ̂) − τ (θ) ∼ N 0, Dτ (θ)T I(θ)−1 Dτ (θ) .
The standard error of τ (θ̂) can be estimated in two ways. First, via the plug-in
method q
ˆ = Dτ (θ̂)T I(θ̂)−1 Dτ (θ̂),
se
which requires that we have a formula for I(θ). Alternatively, we can estimate the
Fisher information I(θ) by the observed Fisher information D2 lX (θ̂) in which case
we get q
se
ˆ = Dτ (θ̂)T D2 lX (θ̂)−1 Dτ (θ̂).
for j = 1, . . . , n where the εj ’s are iid N (0, 1), but we do not assume that σ is
known. The parameter of interest is µ and we seek a (1 − α)-confidence
P interval. The
parameter space is R × (0, ∞) and we use the MLE µ̂ = n1 nj=1 Xj and the usual
modification
n
2 1 X
σ̂ = (Xj − µ̂)2
n−1
j=1
of the MLE as the estimator of σ 2 . The computations in Example 3.5.3 combined

with a plug-in of the estimate of the unknown standard deviation lead to the interval

σ̂ σ̂
µ̂ − √ zα , µ̂ + √ zα
n n
with zα the (1 − α/2)-quantile for the N (0, 1)-distribution. This interval is identical
√
to the general approximation constructions when we use σ̂/ n as an estimate of the
standard error of µ̂.

For this particular case we can come up with an exact confidence interval by consid-
ering a special statistic. It is useful to recall the relation between confidence intervals
and the null-hypotheses of the form
H 0 : µ = µ0 .
If we introduce the t-test statistic
√
n(µ̂ − µ0 )
T =
σ̂
it can be shown that under the null-hypothesis H0 it has a t-distribution with n − 1
degrees of freedom – cf. also Section 3.4.1. If wα denotes the (1 − α/2)-quantile for
the t-distribution the resulting confidence interval becomes
√
n|µ̂ − µ0 |
I(x) = {µ0 ∈ R | ≤ wα }
σ̂
σ̂ σ̂
= µ̂ − √ wα , µ̂ + √ wα .
n n
These intervals are exact under the assumption of iid normally distributed observa-
tions, and in this case the actual coverage is (1 − α).
If we return to the gene expression data that we also considered in Example 3.5.3 we
found the approximate 95%-confidence interval [8.27, 8.80]. There are 37 observations
and using the t-distribution with 36 degrees of freedom instead of the approximat-
ing normal distribution the quantile changes from 1.96 to 2.03 and the confidence
interval changes to [8.26, 8.81]. Thus by using the more conservative t-distribution
the confidence interval is increased in length by roughly 3.5%. ⋄
The second construction based on the t-statistic in Example 3.5.10 does not fit in
among the bullet points above. When τ is the parameter of interest the general
t-statistic is
τ̂ (x) − τ
t = t(x, τ ) =
se(θ̂(x))
where se(θ0 ) is the standard error of τ̂ under Pθ0 and θ̂ is an estimator of θ. Using
this statistic requires first of all that we have a formula for the standard error. If we
approximate the distribution of the t-statistic by a N (0, 1)-distribution, the resulting
confidence intervals
I(x) = {τ ∈ R | |t(x, τ )| ≤ zα } = [τ̂ (x) − se(θ̂(x))zα , τ̂ + se(θ̂(x))zα ]
where zα is the 1 − α/2-quantile for the N (0, 1)-distribution are the same as we have
under the second bullet point. When the parameter of interest is the mean value
parameter µ for the normal distribution we found in Example 3.5.10 the formula
σ
se(µ, σ 2 ) = √
n
for the standard error. For this particular example – as we already discussed – it is
possible theoretically to find the exact distribution of the t-statistics. As stated, it is
a t-distribution with n − 1 degrees of freedom. The consequence is that the quantile
zα from the normal distribution is replaced by a quantile wα from the t-distribution.
It happens so that wα ≥ zα but that the quantile for the t-distribution approaches
zα when n gets large. Using the exact t-distribution gives systematically wider con-
fidence intervals. It is not uncommon to encounter the standard confidence interval
construction as above but where the quantile from the normal distribution has been
replaced by the quantile from the t-distribution, also in situations where there is no
theoretical reason to believe that the t-distribution is a better approximation than
the normal distribution. It is difficult to say if such a practice is reasonable, but since
the resulting confidence intervals get systematically larger this way, we can regard
the procedure as being conservative.
Example 3.5.11. Consider the setup in Section 3.4.1 with two groups of indepen-
dent normally distributed observations where the full parameter set is
Θ = R × (0, ∞) × R × (0, ∞).
We consider here the parameter of interest
τ (µ1 , σ12 , µ2 , σ22 ) = µ1 − µ2
being the difference of the two means.

Under the hypothesis
H 0 : µ1 = µ2 + δ
the distribution of the statistic
µ̂1 − µ̂2
is N (δ, σ12 /n+σ22 /m). The parameter δ is the parameter of interest and the standard,
approximate (1 − α)-confidence interval for δ is
" r r #
σ̂12 σ̂22 σ̂12 σ̂22
µ̂1 − µ̂2 − + zα , µ̂1 − µ̂2 + + zα
n m n m
where zα is the (1 − α/2)-quantile for the N (0, 1)-distribution.

For this particular case we can instead use the t-distribution approximation of the
T -test statistic as considered in Section 3.4.1, which will lead to slightly more con-
servative confidence intervals as we replace the quantile zα by the corresponding
quantile for the t-distribution. For the gene expression data considered in Section
3.4.1 the 95%-confidence interval for the difference based on the normal distribution
becomes [0.874, 1.532]. Using the t-distribution with 67.9 degrees of freedom the
interval becomes [0.868, 1.537]. The latter interval is also reported by the t.test
function in R – see R Box 3.4.1. If the fold-change is the parameter of interest
rather than the difference in means, we can obtain (nominal) 95%-confidence inter-
vals as [20.874 , 21.532 ] = [1.83, 2.89] using the quantile from the normal distribution
and [20.868 , 21.537 ] = [1.82, 2.90] using the t-distribution.
If we make the assumption of equal variances in the two groups, the t-distribution
becomes exact. The confidence interval based on the t-distribution with 37 + 42 −
2 = 77 degrees of freedom is [0.874, 1.531]. The general formula for the (1 − α/2)-
confidence interval for the difference δ under the assumption of equal variances is
" r r #
n+m n+m
µ̂1 − µ̂2 − σ̂wα , µ̂1 − µ̂2 + σ̂wα
nm nm
where wα is the (1 − α/2)-quantile for the t-distribution with n + m − 2 degrees of

freedom. Here σ̂ 2 is the pooled variance estimator. ⋄
We have in this section focused on general confidence intervals based on the statistic
τ̂ − τ . The t-statistic is an example of another possible choice of statistic useful for
confidence interval constructions. There are numerous alternatives in the literature
for choosing a suitable statistic h(x, τ ), but we will not pursue these alternatives
here.
3.6 Regression
Regression models form the general class of models that is most important for sta-
tistical applications. It is primarily within the framework of regression models that
we treat relations among several observables. Sometimes these relations are known
up to a small number of parameters from a given scientific theory in the subject
matter field. In other cases we try to establish sensible relations from the data at
hand.
We specify the general regression model for a real valued observable X given y by
the scale-location model
X = gβ (y) + σε
where ε is a random variable with mean 0 and variance 1. Here y ∈ E for some
sample space E and
gβ : E → R
is a function parametrized by β. The full parameter is θ = (β, σ) ∈ Θ, where we
will assume that Θ ⊆ Rd × (0, ∞), thus the β-part of the parametrization is a d-
dimensional real vector. The variable y has many names. Sometimes it is called the
independent variable – as opposed to X which is then called the dependent variable
– in other situations y is called a covariate. We will call y the regressor to emphasize
that the observable X is regressed on the regressor y.
Regression 189
If ε has distribution with density f and if we observe X1 , . . . , Xn such that
Xi = gβ (yi ) + σεi
where ε1 , . . . , εn are iid and y1 , . . . , yn ∈ E then X1 , . . . , Xn become independent but

in general not identically distributed. The value of yi dictates through the function
gβ the mean value of Xi .
We find that the general joint density for the distribution of X1 , . . . , Xn is
n
Y
1 xi − gβ (yi )
fβ,σ (x1 , . . . , xn ) = f . (3.17)
σ σ
i=1
The minus-log-likelihood function becomes

n
X
xi − gβ (yi )
lx (β, σ) = n log σ − log f . (3.18)
σ
i=1
Maximum-likelihood estimation, for the regression model at hand, can thus be boiled
down to a matter of minimizing the function above. We elaborate on this in the case
where the distribution of ε is assumed to be the normal distribution. In that case
n
1 X √
lx (β, σ) = n log σ + 2 (xi − gβ (yi ))2 +n log 2π.
2σ
|i=1 {z }
RSS(β)
We can observe that for fixed σ this minus-log-likelihood is minimized by minimizing

the quantity RSS(β), and the resulting minimizer does not depend upon σ. We can
summarize the conclusion of this analysis as follows. For the regression model with
a normally distributed noise term the MLE exists if and only if
n
X
RSS(β) = (xi − gβ (yi ))2 (3.19)
i=1
has a unique minimizer β̂ in which case

1
σ̃ 2 = RSS(β̂)
n
is the corresponding MLE of the variance σ 2 . We call RSS(β) the residual sum of
squares.
It is quite common to use this MLE of β based on the normal distribution even
in situations where the assumption of a normal distribution does not hold. After
all, minimizing the residual sum of squares RSS(β) is a sensible thing to do. If we
estimate β by minimizing RSS(β) we often talk about the least squares estimator.
400
300
loss
200
100
40 50 60 70 80 90 100
hard
Figure 3.16: Scatter plot of the relation between hardness and abrasion loss for tires
and the straight line estimated by least squares regression.
The least squares estimator and the MLE are identical for the normal distribution
but otherwise not.
The variance is on the other hand typically not estimated by the MLE above – not
even if the assumption of the normal distribution holds. Instead we use the estimator
1
σ̂ 2 = RSS(β̂)
n−d
where d is the dimension of the parameter space for β.
3.6.1 Ordinary linear regression
A linear regression model is obtained when E = R, d = 2, β = (β0 , β1 ) and
gβ (y) = β0 + β1 y.
The parameter β0 is called the intercept and β1 the slope.

The residual sum of squares reads
n
X
RSS(β) = (xi − β0 − β1 yi )2 .
i=1
Regression 191
2.0
1.5
1.5
1.0
density
density
1.0
0.5
0.5
0.0
0.0
0 2 4 6 8 10 12 0 2 4 6 8 10 12
concentration concentration
2.0
1.5
1.5
1.0
density
density
1.0
0.5
0.5
0.0
0.0
0 2 4 6 8 10 12 0 2 4 6 8 10 12
concentration concentration
Figure 3.17: The optical density in the ELISA calibration experiment as a function
of the known concentrations of DNase in four different runs of the experiment.
Under the very mild condition that the yi ’s are not all equal there is a unique
minimizer, and there is even an explicit, analytic formula for the minimizer, see
Math Box 3.6.1.
Example 3.6.1. We consider in this example the relation between the hardness
measured in Shore units of a rubber tire and the abrasion loss in gm/hr. Figure
3.16 shows the scatter plot. From the scatter plot we expect that there is a close
to straight line relation; the harder the tare is the smaller is the loss. There are
30 observations in the dataset and if Xi denotes the loss and yi the hardness for
i = 1, . . . , 30 we suggest the model
Xi = β0 + β1 yi + σεi
where ε1 , . . . , ε30 are iid N (0, 1). Figure 3.16 also shows the straight line estimated
by least squares estimation of the parameters (β0 , β1 ). ⋄
Example 3.6.2 (DNase ELISA calibration). In the development of an ELISA assay

for the recombinant protein DNase in rat serum, a calibration experiment was carried
out as follows. A known dilution series,
0.0488, 0.1953, 0.3906, 0.7812, 1.5625, 3.1250, 6.2500, 12.5000,
in ng/ml, of the DNase protein was made, and each run, out of a total of 11 runs, of
the experiment consisted of measuring the optical density from the ELISA experi-
0.5
0
−0.5
−1
log−density
log−density
−2
−1.5
−3
−2.5
−4
−3 −2 −1 0 1 2 −3 −2 −1 0 1 2
log−concentration log−concentration
0
0
−1
log−density
log−density
−1
−2
−3
−2
−4
−3
−3 −2 −1 0 1 2 −3 −2 −1 0 1 2
log−concentration log−concentration
Figure 3.18: The logarithm of the optical density in the ELISA calibration experi-
ment as a function of the known logarithm of concentrations of DNase in four dif-
ferent runs of the experiment. Note that the transformations make the data points
lie more closely to a straight line than for the untransformed data.
ment with two measurements for each concentration. The first four runs are shown
on Figure 3.17.
From that figure it is not obvious that we can use a linear regression model to capture
the density as a function of concentration. However, taking a look at Figure 3.18 we
see that by applying the log-transformation to both quantities we get points that
approximately lie on a straight line. Thus considering only one run, the first, say,
then if Xi denotes the log-density and yi the log-concentration for i = 1, . . . , 16 we
can try the model
Xi = β0 + β1 yi + σεi .
The resulting estimated lines plotted on Figure 3.18 are estimated in this way – with
a separate estimation for each of the four runs shown on the figure.
⋄
Example 3.6.3 (Beaver body temperature). As with the DNase example above it
is a standard trick to be able to find a suitable transformation of the observables
such that a straight line relation becomes plausible. We consider in this example
the body temperature for a beaver measured every 10 minutes for a period of 16
hours and 40 minutes, that is, 100 measurements in total. We let Xi denote the i’th
Regression 193
38.0
37.5
temp
37.0
0 200 400 600 800 1000
time
Figure 3.19: The body temperature of a beaver measured every 10 minutes together
with a curve estimated by least squares regression.
temperature measurement and ti the time in minutes since the first measurement
(that is t1 = 0, t2 = 10, . . .). We suggest that the body temperature will be periodic
with a period of 24 hours (1440 minutes) and be minimal at 8.30 in the morning.
As the first measurement is taken at 9.30 in the morning we introduce
yi = cos(2π(ti + 60)/1440)
and suggest the model

Xi = β0 + β1 yi + σεi
where ε1 , . . . , ε100 are iid N (0, 1). Figure 3.19 shows a plot of the data (xi , ti ) together
with the curve estimated by least squares linear regression of the xi ’s onto the yi ’s.
⋄
An important technique used in the two latter examples above is that ordinary linear
regression can be useful even in situations where there is no linear relation between
the observable and the regressor. The question is if we can transform the regressor
and/or the observable by some known transformation(s) such that linear regression
is applicable for the transformed values instead.
The most interesting hypothesis to test in relation to the linear regression model
is whether β1 = 0 because this is the hypothesis that the value of the regressor
does not influence the distribution of the observable. There are several formal test
statistics that can be computed for testing this hypothesis. From a summary of the
R Box 3.6.1 (Linear regression). Ordinary linear least squares regression

is done using the lm function in R. If you load the library MASS and read
in the dataset Rubber you will find a data frame named Rubber with three
columns, cf. Example 3.6.1. We can regress the loss on the hardness as
follows:
> library(MASS)
> data(Rubber)
> rubberLm <- lm(loss~hard,data=Rubber)
The resulting object rubberLm can then be “interrogated” by several generic

functions. A call of summary(rubberLm) gives among other things a ta-
ble with the estimated parameters including their estimated standard error
and a t-test for the hypothesis that the parameter is equal to zero. A call
of plot(rubberLm) produces four diagnostic plots, fitted(rubberLm) re-
turns the fitted values, residuals(rubberLm) returns the residuals and
rstandard(rubberLm) gives the standardized residuals.
object returned by the lm function in R we can read of the value of the estimated
standard error se
ˆ of the estimator for β1 , the t-test statistic
β̂1
t=
se
ˆ
for testing the hypothesis H0 : β1 = 0 together with a p-value computed based on
the t-distribution with n − 2 degrees of freedom. The estimated standard error se
ˆ
can also be used for constructing confidence intervals of the form
[β̂1 − sez
ˆ α , β̂1 + sez
ˆ α]
where zα is the (1−α/2)-quantile for the N (0, 1)-distribution. If the error distribution
is N (0, 1) then there is theoretical basis for replacing the quantile for the normal
distribution by the quantile for the t-distribution with n − 2 degrees of freedom.
With β̂0 and β̂1 the least squares estimates we introduce the fitted values
x̂i = β̂0 + β̂1 yi
and the residuals

ei = xi − x̂i = xi − β̂0 − β̂1 yi .
It is important to investigate if the model actually fits the data. That is, we need
methods that can tell if one or more of the fundamental model assumptions are
violated. The residuals should resemble the noise variables σε1 , . . . , σεn , who are iid.
Moreover, there is no relation between the yi ’s and the σεi ’s, and the variance of σεi
Regression 195
Math Box 3.6.1 (Linear regression). The theoretical solution to the minimization
of RSS is most easily expressed geometrically. We denote by x ∈ Rn our vector of
observations, by y ∈ Rn the vector of regressors and by 1 ∈ Rn a column vector
of 1’s. The quantity RSS(β) is the length of the vector x − β0 1 − β1 y in Rn . This
length is minimized over β0 and β1 by the orthogonal projection of x onto the space
in Rn spanned by y and 1.
One can find this projection
√ if we know an orthonormal basis, and such a one is
found by setting a = 1/ n (so that a has length 1) and then replacing y by
y − (y T a)a
b = pPn .
2
i=1 (yi − ȳ)
T
Both a and b have unit length
Pn and they are orthogonal as a b = 0. Note that
T 1
(ya )a = ȳ1 where ȳ = n i=1 yi . We have to assume that at least two of the yi ’s
differ or the sum in the denominator above is 0. If all the yi ’s are equal, the vector
y and 1 are linearly dependent, they span a space of dimension one, and β0 and β1
are not both identifiable. Otherwise the vectors span a space of dimension two.
The projection of x onto the space spanned by a and b is

xT y − nȳx̄ xT y − nȳ x̄
(xT a)a + (xT b)b = x̄ − ȳ Pn 2
1 + Pn 2
y.
i=1 (yi − ȳ) i=1 (yi − ȳ)
| {z } | {z }
β̂0 β̂1
Thus the theoretical solution to the minimization problem can be written
xT y − nȳ x̄
β̂1 = Pn 2
β̂0 = x̄ − ȳβ̂1 .
i=1 (yi − ȳ)
does not depend upon the value yi . We typically investigate these issues by graphical
methods – diagnostic plots – based on the computed residuals. There is one caveat
though. The variance of the i’th residual does in fact depend upon the yi ’s, and it
can be shown to equal σ 2 (1 − hii ) where
1 (yi − ȳ)2
hii = + Pn 2
n j=1 (yj − ȳ)
P
and ȳ = n1 ni=1 yi . The quantity hii is called the leverage of the i’th observation.
The standardized residuals are defined as
ei
ri = √ .
σ̂ 1 − hii
The leverage quantifies how much the observation xi influences the fitted value
x̂i relative to the other observations. A large leverage (close to 1) tells that the
observation heavily influences the fitted value whereas a small value (close to 1/n)
tells that the observation has minimal influence.
Math Box 3.6.2 (Fitted values). We continue here with the setup from Math
Box 3.6.1. If Xi = β0 + β1 yi + σεi we find that the fitted value is
n
X
X̂i = (X T a)ai + (X T b)bi = β0 + β1 yi + σ(aj ai + bj bi )εj .
j=1
If εj ∼ N (0, 1) then σ(aj ai + bj bi )εj ∼ N (0, σ 2 (aj ai + bj bi )2 ) and it follows from

Math Box 2.10.20 that the fitted value has distribution
 
Xn
X̂i ∼ N β0 + β1 yi , σ 2 (aj ai + bj bi )2  .
j=1
P
The quantity usually denoted hii = nj=1 (aj ai + bj bi )2 is know as the leverage and
if we use that a and b are orthonormal we find that
n
X
hii = (aj ai + bj bi )2 = a2i aT a + b2j bT b + 2ai bi aT b
j=1
1 (yi − ȳ)2
= a2i + b2i = + Pn 2
.
n j=1 (yj − ȳ)
It follows that the residual

n
X
ε̂i = Xi − X̂i = σεi − σ(aj ai + bj bi )εj
j=1
n
X
= σ(1 − (a2i + b2i ))εi − σ(aj ai + bj bi )εj
| {z }
j=1,j6=i
hii
has a N (0, σ 2 (1 − hii ))-distribution.
Plots of the residuals or the standardized residuals against the yi ’s or against the
fitted values x̂i ’s are used to check if the mean value specification β0 + β1 yi is ad-
equate. We expect to see an unsystematic distribution of points around 0 at the
y-axis and spread out over the range of the variable we plot against. Systematic
deviations from this, such as slopes or bends, in the point cloud indicate that the
model specification does not adequately capture how the mean of Xi is related to
yi . A plot of the standardized residuals, their absolute value – or the square root
of their absolute value, as the choice is in R – against the fitted values can be used
to diagnose if there are problems with the assumption of a constant variance. We
look for systematic patterns, in particular whether the spread of the distribution
of the points changes in a systematic way over the range of the fitted values. If so,
there may be problems with the constant variance assumption. Finally, we can also
compare the standardized residuals via a QQ-plot to the distribution of the εi ’s. A
QQ-plot of the empirical distribution of the standardized residuals against the nor-
Regression 197
Residuals vs Fitted Scale−Location
1.4
28
26
28 29
100
26
1.2
29
Standardized residuals
1.0
50
Residuals
0.8
0.6
0
0.4
−50
0.2
−100
0.0
100 150 200 250 300 100 150 200 250 300
Fitted values Fitted values
Normal Q−Q Residuals vs Leverage

2.0
0.5
2
28
26 26
29 29
1.5
1
1.0
1
0.5
0
−0.5
−1
Cook’s distance
−1.5
−2 −1 0 1 2 0.00 0.05 0.10 0.15
Theoretical Quantiles Leverage
Figure 3.20: The four standard diagnostic plots produced in R for a linear regression.
mal distribution is thus a diagnostic of whether we can take the εi ’s to be normally

distributed.
The value of the leverages are sometimes also investigated. A large value of the
leverage for a single observation is not in itself problematic, but if the corresponding
observation is an outlier, it can drastically affect the whole fit if the leverage is large.
A large value of the leverage for a single observation can also indicate a mistake in
the recording of the yi -value.
Example 3.6.4. If we want to investigate the model for the tire loss, as considered
in Example 3.6.1, further we can make a summary in R of the result from the call
of lm, cf. R Box 3.6.1, by summary(rubberLm).
Call:
lm(formula = loss ~ hard, data = Rubber)
Residuals:
Min 1Q Median 3Q Max
-86.15 -46.77 -19.49 54.27 111.49
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 550.4151 65.7867 8.367 4.22e-09 ***
hard -5.3366 0.9229 -5.782 3.29e-06 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 60.52 on 28 degrees of freedom

Multiple R-squared: 0.5442, Adjusted R-squared: 0.5279
F-statistic: 33.43 on 1 and 28 DF, p-value: 3.294e-06
From this we read of the estimates β̂0 = 550.4 and β̂1 = −5.334, their standard
errors, the t-value for a test whether the parameter equals 0 and corresponding
p-value computed using the t-distribution with 28 degrees of freedom. As a visual
guidance for the eye, the stars are printed to highlight significance at different levels.
The conclusion is in this case that none of the parameters can be taken equal to 0.
The residual standard error is the estimate of σ.
To assess if the model is adequate we consider the four diagnostic plots shown in
Figure 3.20. These are the default plots produced by a call of plot(rubberLm). The
plot of the residuals against the fitted values shows that the residuals are scattered
randomly around 0 over the range of the fitted values. Thus the straight line seems
to capture the relation between loss and hardness well. The QQ-plot shows that the
distribution of the residuals is reasonably approximated by the normal distribution
– the deviations from a straight line are within the limits of what is conceivable
with only 30 observations. The plot of the square root of the absolute value of the
standardized residuals against the fitted values shows that the variance does not
seem to depend upon the mean. Moreover, there are no clear outliers. The fourth
and final plot shows the standardized residuals plotted against the leverage. We
should in particular be concerned with combinations of large leverage and large
standardized residual as this may indicate an outlier that has considerable influence
on the fitted model.
Confidence intervals for the two estimated parameters, β0 and β1 , can be computed
from the information in the summary table above. All we need is the standard error
and the relevant quantile from the t-distribution or the normal distribution. For
convenience, the R function confint can be used to carry out this computation.
> confint(rubberLm)
2.5 % 97.5 %
(Intercept) 415.657238 685.173020
hard -7.227115 -3.445991
The parameters β0 and β1 are, however, not the only parameters of interest in the
context of regression. If y ∈ R the predicted value
gβ̂ (y) = β̂0 + β̂1 y
is of interest for the prediction of a future loss for a tire of hardness y. If we have a new
data frame in R, called newdata, say, with a column named hard, as our regressor
Regression 199
Prediction
400
300
Loss
200
100
40 50 60 70 80 90 100
Hardness
Figure 3.21: Predicted values, 95%-confidence bands and 95%-prediction bands for
the linear regression model of tire abrasion loss as a function of tire hardness.
in the original dataset, we can use predict(rubberLm,newdata) to compute the

predicted values. Moreover, 95%-confidence intervals for all predicted values are
obtained by predict(rubberLm, newdata, interval = "confidence"). Setting
the argument level to 1 − α (default is 0.95) will give (1 − α)-confidence intervals.
If se(y)
ˆ denotes the estimate of the standard error of gβ̂ (y) the confidence interval
computed by predict is
[gβ̂ (y) − se(y)w
ˆ α , gβ̂ (y) + se(y)w
ˆ α]
with wα the 1 − α/2-quantile for the t-distribution with n − 2 degrees of freedom.

As a final remark we note that to predict a new value
X = β0 + β1 y + σε
the confidence interval computed by e.g. the predict function does only take into
account the uncertainty in the estimate β̂0 + β̂1 y of the mean value β0 + β1 y
of X and not the random component σε. So-called prediction or tolerance inter-
vals are computed by predict(rubberLm, newdata, interval = "prediction").
They are computed as
h p p i
ˆ 2 + σ̂ 2 wα , gβ̂ (y) + se(y)
gβ̂ (y) − se(y) ˆ 2 + σ̂ 2 wα .
Figure 3.21 shows predicted values, 95%-confidence bands and 95%-prediction bands.
⋄
Residuals vs Fitted Residuals vs Fitted

0.2
0.10
27 41
0.1
0.05
0.00
0.0
Residuals
Residuals
−0.05
−0.1
−0.10
−0.15
47
−0.2
31
32
−0.20 48
−0.3
−3 −2 −1 0 1 −2.5 −2.0 −1.5 −1.0 −0.5 0.0 0.5
Fitted values Fitted values
Figure 3.22: Residual plots for the linear regression model of log-density on log-
concentration of run 2 and 3 for the DNase ELISA experiment.
In the example above we discussed predictions. A word of warning is appropriate.

Even though the regression can be extrapolated to yield predictions outside of the
range of the y-values in the dataset such extrapolations should be done with great
care – if not avoided altogether. Predictions for values in the range of the observed
y-values are more reliable. First of all it appears from Figure 3.21 that the confi-
dence interval for the predicted value is most narrow in the center of the y-values,
which is indeed a general phenomena. Second, and more important, we have used
various model diagnostics to justify that the model is appropriate and adequate
for the data at hand. This provides justification of the predictions that the model
provides but only in the range where we have observations. If it cannot be justi-
fied by other means that the model extrapolates well outside of the range of the
observed y-variable, for instance by subject matter knowledge, extrapolations are
model dependent speculations where we have little support for the predictions.
Example 3.6.5. The summary of the lm-object for one of the linear regressions for
the ELISA data considered in Example 3.6.2 reads
Call:
lm(formula = log(density) ~ log(conc), data = myDNase[myDNase[,1] == 2, ])
Residuals:
-0.25254 -0.07480 0.01123 0.11977 0.17263
Regression 201
Coefficients:
(Intercept) -0.85572 0.03505 -24.41 7.09e-13 ***
log(conc) 0.69582 0.02025 34.36 6.39e-15 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

F-statistic: 1181 on 1 and 14 DF, p-value: 6.391e-15
Figure 3.22 shows the residual plot for two of the runs. Both plots are typical for
this dataset and shows a lack of fit for the model. The residuals are positive for the
fitted values in the middle range and negative for the extreme fitted values. The
straight line does not seem to fully capture the relation between the log-density and
the log-concentration. We will return to a refined model in the next section.
The fact that the model does not fit the data does not mean that the linear regression
model is useless. After all, it captures the general relation between the log-density
and log-concentration up to some relatively small curvature, which on the other
hand appears very clearly on the residual plot. The systematic error of the model
for the mean value leads to a larger estimate of the variance than if the model was
more refined. Often this results in conservative conclusions such as wider confidence
intervals for parameters of interest and wider prediction intervals. Whether this is
tolerable must be decided on a case-by-case basis. ⋄
Example 3.6.6. Figure 3.24 shows the residual plot and the QQ-plot for the beaver
data considered in Example 3.6.3. The QQ-plot seems OK, but the residual plot
shows a problem. Around the middle of the residual plot the residuals show a sudden
change from being exclusively negative to being mostly positive. This indicates a
lack of fit in the model and thus that the temperature variation over time cannot be
ascribed to a 24 hour cycle alone.
In fact, there might be an explanation in the dataset for the change of the tem-
perature. In addition to the time and temperature it has also been registered if the
beaver is active or not. There is a 0-1 variable called activ in the dataset, which is
1 if the beaver is active. A refined analysis will include this variable so that we have
a three parameter model of the mean with
β0 + β1 y
the mean temperature if the beaver is inactive and
β0 + βactiv + β1 y
if the beaver is active. Here y = cos(2πt/1440) where t is time in minutes since 8.30
in the morning. The summary output from this analysis is:
Residuals vs Fitted Normal Q−Q

0.6
100 100
42 42
99 99
2
0.4
0.2
1
Residuals
0.0
0
−0.2
−1
−0.4
37.0 37.2 37.4 37.6 37.8 38.0 −2 −1 0 1 2
Fitted values Theoretical Quantiles
Figure 3.23: Residual plot and QQ-plot for the beaver body temperature data.
Call:
lm(formula = temp ~ activ + cos(2 * pi * (time + 60)/1440), data = beaver2)
Residuals:
-0.44970 -0.11881 -0.02208 0.17742 0.39847
Coefficients:
(Intercept) 37.19976 0.04253 874.690 < 2e-16 ***
activ 0.53069 0.08431 6.294 8.95e-09 ***
cos(2 * pi * (time + 60)/1440) -0.24057 0.06421 -3.747 0.000304 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

F-statistic: 198.1 on 2 and 97 DF, p-value: < 2.2e-16
We can read from the summary that the hypothesis that βactiv = 0 is clearly re-
jected. The estimate β̂activ = 0.531 suggests that the beaver being active accounts
for roughly half a grade of increase in the body temperature. Figure 3.24 shows the
fitted mean value and the residual plot. It might be possible to refine the model
even further – perhaps with a smoother transition between the inactive and active
state – but we will not pursue such refinements here. Instead we will focus on an-
Regression 203
Residuals vs Fitted
0.4
42
38.0
0.2
Residuals
37.5
0.0
temp
−0.2
37.0
−0.4
89
90
0 200 400 600 800 1000 37.0 37.2 37.4 37.6 37.8 38.0
time Fitted values
Figure 3.24: Plot of the beaver temperature data and estimated curve (left) and
residual plot (right) when the additional variable activ is included in the analysis.
other potential problem. The observations of the temperature are taken every 10
minutes and this may give problems with the independence assumption. When we
deal with observations over time, this should always be taken into consideration. If
the observations are ordered according to observation time we can investigate the
assumption by a lag-plot of the residuals e1 , . . . , e100 , which is a plot of (ei−1 , ei )
for i = 2, . . . , 100. Figure 3.25 shows that lag-plot of the residuals. This plot should
look like a scatter plot of independent variables, but in this case it does not. On the
contrary, it shows a clear dependence between residual ei and the lagged residual
ei−1 .
Though the residuals show dependence and the model assumptions of independent
εi ’s are questionable, the estimated mean value function can still be used as a rea-
sonable estimate. The problem is in general that all estimated standard errors are
systematically wrong as they are based on the independence assumption, and they
are typically too small, which means that we tend to draw conclusions that are too
optimistic and provide confidence and prediction intervals that are too narrow. The
right framework for systematically correcting for this problem is time series analysis,
which is beyond the scope of these notes. ⋄
0.4
0.2
Residual
0.0
−0.2
−0.4
−0.4 −0.2 0.0 0.2 0.4
Lagged residual
Figure 3.25: The temperature observations are ordered by time and this plot shows
the residuals plotted against the lagged residuals, that is, the residuals from the
previous time point.
3.6.2 Non-linear regression
Dividing regression into linear and non-linear regression is like dividing the animals
in the Zoo into elephants and non-elephants. In the world of non-linear regression
you can meet all sorts of beasts, and we can in this section only touch on a few
simple examples that you can find outside of the world of linear regression.
In this section we will also consider estimation using the least squares method only.
This is only the maximum-likelihood method if the εi -terms are normally distributed.
In the world of regression this is often referred to as non-linear least squares. We can
in general not expect to find closed form solutions to these minimization problems
and must rely on numerical optimization. A word of warning is appropriate here. We
cannot expect that the numerical optimization always goes as smoothly as desirable.
To find the correct set of parameters that globally minimizes the residual sum of
squares we may need to choose appropriate starting values for the algorithm to
converge, and it may very easily be the case that there are multiple local minima,
that we need to avoid.
For non-linear regression there are generalizations of many of the concepts from
linear regression. The fitted values are defined as
x̂i = gβ̂ (yi )

Regression 205
and the residuals are
ei = xi − x̂i = xi − gβ̂ (yi ).
To check the model we make residual plots of the residuals against either the re-
gressors yi or the fitted values x̂i and we look for systematic patterns that either
indicate that the model of the mean via the function gβ̂ is inadequate or that the
constant variance assumption is problematic. Moreover, we can compare the empir-
ical distribution of the residuals to the normal distribution via a QQ-plot. We don’t
have a simple leverage measure, nor do we have a formula for the variance of the
residuals. So even though the residuals may very well have different variances it is
not as easy to introduce standardized residuals that adjust for this. In the context
of non-linear regression the term standardized residual often refers to ei /σ̂ where we
simply divide by the estimated standard deviation.
All standard estimation procedures used will produce estimates of the standard error
for the β-parameters that enter in the model. These estimates are based on the as-
sumption that the εi ’s are normally distributed, but even under this assumption the
estimates will still be approximations and can be inaccurate. The estimated standard
errors can be used to construct confidence intervals for each of the coordinates in the
β-vector or alternatively to test if the parameter takes a particular value. In some
cases – but certainly not always – it is of particular interest to test if a parameter
equals 0, because it is generally a hypothesis about whether a simplified model is
adequate.
Example 3.6.7 (Michaelis-Menten). Enzymes work as catalysts in the conversion

of a substrate into a product. The enzyme alcohol dehydrogenase catalyzes the con-
version of ethanol (the substrate) to acetaldehyde (the product). The data below
are from Bendinskas et al. Journal of Chemical Education, 82(7), 1068 (2005). The
Michaelis-Menten rate equation states that the rate, r, for the conversion is related
to the concentration y of the substrate via
β1 y
r= . (3.20)
β2 + y
With β = (β1 , β2 ) the two unknown parameters and measurements r1 , . . . , rn of the

conversion rate for different substrate concentrations y1 , . . . , yn we set up a non-
linear regression model
R = gβ (y) + σε
where gβ (y) = ββ21+y

y
. We assume in this model that the measurement noise, σε, of
the conversion rate enters additively.
R Box 3.6.2 (Non-linear least squares regression). There are several dif-
ferent possibilities for fitting non-linear regression models in R. The nls
function does non-linear least squares estimation and is quite similar in use
to the lm function for ordinary least squares.
With ethConv a data frame with two columns named rate and conc we
can estimate the Michaelis-Menten curve from Example 3.6.7 by
> nls(rate ~ beta1*conc/(beta2 + conc),

+ data=ethConv,
+ start=c(beta1=1,beta2=1))
Consult Example 3.6.7 on how to summarize the result of this call with the
summary function.
A main difference from lm is that in the formula specification
rate ~ beta1 * conc/(beta2 + conc) we need to explicitly include all
unknown parameters and we need to explicitly give an initial guess of the
parameters by setting the start argument. You can leave out the specifi-
cation of start in which case nls automatically starts with all parameters
equal to 1 – and gives a warning.
If you want to do things beyond what nls can do there are several solutions,
some are tied up with a particular area of applications or with particular
needs.
The drc package is developed for dose-response curve estimation but can
be useful for general non-linear regression problems. The main function,
multdrc, does non-linear least squares estimation but a particular advantage
is that it allows for the simultaneous fitting of multiple dataset. Doing so it
is possible to share some but not all parameters across the different curves.
Substrate concentration Conversion rate

0.007 0.06
0.015 0.11
0.031 0.16
0.068 0.21
0.100 0.23
0.200 0.28
0.300 0.29
0.400 0.28
> ethConvNls <- nls(rate ~ beta1*conc/(beta2 + conc),

+ data=ethConv,
+ start=c(beta1=1,beta2=1))
Regression 207
0.25
0.20
Rate
0.15
0.10
0.0 0.1 0.2 0.3 0.4
Concentration
Figure 3.26: The data and estimated Michaelis-Menten rate curve from Example
3.6.7
We can then call summary on the resulting object to get information on the parameter
estimates, estimated standard errors etc.
> summary(ethConvNls)
Formula: rate ~ beta1 * conc/(beta2 + conc)
Parameters:
beta1 0.309408 0.006420 48.19 5.35e-09 ***
beta2 0.029391 0.002503 11.74 2.30e-05 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Number of iterations to convergence: 8

Achieved convergence tolerance: 5.363e-06
Example 3.6.8 (Four parameter logistic model). One of the flexible and popular
non-linear regression models is known as the four parameter logistic model and is
given by the function
β2 − β1
gβ (y) = + β1
1 + exp(β4 (y − β3 ))
for β = (β1 , β2 , β3 , β3 ) ∈ R4 . For identifiability of the parameters we need to assume

that β1 < β2 , say. The parameters β1 and β2 then give the lower and upper asymp-
totes, respectively, for the graph of the function. The sign of β4 determines if the
function is decreasing or increasing and its absolute value how steep the graph is.
The parameter β3 determines the value of y where the mean response is (β2 + β1 )/2,
which is the mid-point between the minimal value β1 and the maximal value β2 .
We will illustrate the use of this model on the ELISA data considered in Example
3.6.2. We use nls in R with initial values β1 = 0, β2 = 2, β3 = 1 and β4 = −1.5 and
get the following summary of the result.
Formula: density ~ (beta2 - beta1)/(1 + exp(beta4 * (log(conc) - beta3))) +

beta1
Parameters:
beta1 -0.007897 0.017200 -0.459 0.654
beta2 2.377241 0.109517 21.707 5.35e-11 ***
beta3 1.507405 0.102080 14.767 4.65e-09 ***
beta4 -0.941106 0.050480 -18.643 3.16e-10 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The conclusion from this summary is that the parameter β1 can be taken equal to
0. This is sensible as we expect a 0 measurement if there is no DNase in the sample
(the concentration is 0). Taking β1 = 0 a reestimation of the model yields the new
estimates
Parameters:
beta2 2.34518 0.07815 30.01 2.17e-13 ***
beta3 1.48309 0.08135 18.23 1.22e-10 ***
beta4 -0.96020 0.02975 -32.27 8.51e-14 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Regression 209
2
1.5
1
Optical density
1.0
0
0.5
−1
0.0
−3 −2 −1 0 1 2 0.0 0.5 1.0 1.5
Log−concentration Fitted values
Figure 3.27: The data and estimated logistic model for the relation between optical
density and log-concentration for the DNase ELISA considered in Example 3.6.8.
For this model figure 3.27 shows the data and the estimated mean value curve for
run 1 in the dataset together with the residual plot, which shows that the model is
adequate.
When y is the log-concentration an alternative formulation of the logistic model
expressed directly in terms of the concentration is
β2 − β1 β2 − β1
gβ (conc) = + β1 = β4 + β1 .
1 + exp(β4 (log(conc) − β2 )) conc
1+ exp(β2 )
Example 3.6.9 (Expression calibration). It is not the rule to produce standard

curves for microarray experiments, that is, dilution series with known concentrations
are not routinely applied to a microarray for the purpose of estimating the relation
between concentration and the actually observed light intensity. In the process of
understanding and modeling the results from microarray experiments some so-called
spike-in experiments have, however, been done. These are experiments where some
samples of known concentration have been spiked in to the sample that is applied to
the microarray and one can use the results from these known sample concentrations
to infer the relation between concentration and light intensity. We consider here a
spike-in experiment, spikein95, from the Bioconductor SpikeInSubset package.
14
12
Log−intensity (base 2)
10
8
6
−2 0 2 4 6 8 10
Log−concentration (base 2)
Figure 3.28: .
We will use the logistic model as considered above to capture the relation between
log-concentration and log-intensity. The concentrations are in picoMolar.
Using nls in R the summary of the resulting object reads
Formula: log2(value) ~ (beta2 - beta1)/(1 + exp(beta4 * (log2(conc) -

beta3))) + beta1
Parameters:
beta1 7.79500 0.06057 128.69 <2e-16 ***
beta2 12.68086 0.12176 104.15 <2e-16 ***
beta3 5.80367 0.09431 61.54 <2e-16 ***
beta4 -0.95374 0.08001 -11.92 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

From the summary we see that none of the parameters can be taken equal to 0. It is
most notable that β1 is significantly larger than 0 and in fact the estimate β̂1 = 7.795
is far from 0. It suggests that there is a considerable background signal even when
Regression 211
Normal Q−Q Plot
2
1
residuals(spikeIn95Nls, type = "pearson")
0
Sample Quantiles
−1
−2
−2
−3
−4
−4
8 9 10 11 12 −3 −2 −1 0 1 2 3
jitter(fitted(spikeIn95Nls), amount = 0.1) Theoretical Quantiles
Figure 3.29: .
the concentration equals 0. Figure 3.28 shows the data and the estimated curve.
There are only 13 different concentrations but a large number of replications for each
concentration and the x-values have therefore been jittered to aid visualization. We
see from this plot that the logistic curve captures the overall behavior rather well, but
already from this plot it seems that there are problems with variance inhomogeneity
and a number of observations – in particular some of the small observations for the
high concentrations – seem problematic.
Figure 3.29 shows that residual plot and a QQ-plot of the residuals against the
normal distribution. It is clear from the residual plot that an assumption of the
same variance across the different concentrations does not hold. The normal QQ-plot
shows deviations from the normal distribution in both tails. This form of deviation
suggests that the distribution is left-skewed compared to the normal distribution
because there are too many small residuals and too few large residuals. However,
because the variance cannot be assumed constant this is in reality a mixture of
different distributions with different scales. ⋄
Exercises
For the next four exercises you are asked to consider the Rubber data from the MASS
library.
Exercise 3.6.1. In the dataset there is another variable, tens. Plot loss versus tens
and carry out a linear regression of loss on tens. This includes model diagnostics.
What is the conclusion?
Exercise 3.6.2. Compute the residuals when you regress loss on hard and plot the
residuals versus tens. Interpret the result.
Exercise 3.6.3. If we let yi denotes the hardness and zi the tension, consider the
extended model
Xi = β0 + β1 yi + β2 zi + σεi
where ε1 , . . . , ε30 are iid N (0, 1). Use lm (the formula should be loss~hard+tens) to
carry our a linear regression model including model diagnostics where you regress on
hardness as well as tension.
Exercise 3.6.4. Use the function predict to compute 95% prediction intervals for
the 30 observed values using
• the model where we only regress on hardness
• the model where we regress on hardness and tension.
Compare the prediction intervals. It can be useful to plot the prediction intervals
versus the fitted values.
Exercise 3.6.5. Show that the Michaelis-Menten rate equation 3.20 can be rephrased
as
1 β2 1 1
= + .
r β1 y β 1
Argue that this formula suggests a linear regression model for the inverse of the rate
regressed on the inverse of the substrate concentration. Estimate the parameters using
this linear regression model and compare with Example 3.6.7.
3.7 Bootstrapping
The basic idea in bootstrapping for constructing confidence intervals for a parameter
of interest τ when we have an estimator τ̂ of τ = τ (θ) is to find an approximation
of the distribution of τ̂ − τ (θ0 ) – usually by doing simulations that depend upon the
observed dataset x ∈ E. What we attempt is to approximate the distribution of
τ̂ − τ (θ0 )
under Pθ0 , by the distribution of

τ̂ − τ̂ (x)
under a cleverly chosen probability measure Px , which may depend upon x. Since the
distribution is allowed to depend upon the concrete observation x the construction
of the resulting confidence set – that provides information about the uncertainty of
the estimate τ̂ (x) – depends upon the observation itself. Hence what we are going
to suggest is to pull information about the uncertainty of an estimate out from
the very same data used to make the estimate, and for this reason the method is
known as bootstrapping. Supposedly one of the stories in The Surprising Adventures
Bootstrapping 213
of Baron Munchausen by Rudolf Erich Raspe (1736 - 1794) contains a passage

where the Baron pulls himself out of a deep lake by pulling his own bootstraps.
Such a story is, however, not to be found in the original writings by Raspe, but the
stories of Baron Munchausen were borrowed and expanded by other writers, and one
can find versions where the Baron indeed did something like that. To bootstrap is
nowadays, with reference to the Baron Munchausen story, used to describe various
seemingly paradoxical constructions or actions. To boot a computer is for instance
an abbreviation of running a so-called bootstrap procedure that gets the computer
up and running from scratch.
The computation of the distribution of τ̂ − τ̂ (x) under Px is a transformation problem
that is usually solved by computer simulations. We simply use the computer to
generate a large number of new datasets x1 , . . . , xB and relevant quantities such as
quantiles for the distribution of τ̂ − τ̂ (x) or the standard deviation of τ̂ are estimated
from the simulated data.
Algorithm 3.7.1 (The Bootstrap for Confidence Set Construction). We consider

an estimator τ̂ : E → R for the real valued parameter of interest. For a given dataset
x ∈ E the corresponding estimate is τ̂ (x) and with Px a probability measure on E,
which may depend upon x, the bootstrapping algorithm for producing a nominal
(1 − α)-confidence interval proceeds as follows:
• Choose B sufficiently large and simulate B new independent, identically dis-

tributed datasets, x1 , . . . , xB ∈ E, each simulation being from the probability
measure Px .
• Compute, for each dataset xi , i = 1, . . . , B, new estimates τ̂ (xi ) using the
estimator τ̂ .
• Compute ẑα and ŵα as the α/2 and 1 − α/2 quantiles for the empirical
distribution of τ̂ (xi ) − τ̂ (x), i = 1, . . . , B.
• Define I(x) = [τ̂ (x) − ŵα , τ̂ (x) − ẑα ].
A minor modification of the algorithm is given by replacing the last two bullet
points by
PB
• Compute the empirical mean τ = B1 i=1 τ̂ (xi ) and the empirical standard
deviation v
u B
u 1 X
ˆ =t
se (τ̂ (xi ) − τ )2
B − 1 i=1
• Define I(x) = [τ̂ (x) − sez ˆ α ] where zα is the 1 − α/2 quantile for
ˆ α , τ̂ (x) + sez
the N (0, 1)-distribution.
The two most important and commonly encountered choices of Px are known as
parametric and non-parametric bootstrapping respectively. We consider parametric
bootstrapping here and non-parametric bootstrapping in the next section.
Definition 3.7.1. If we have an estimator, θ̂, of θ then parametric bootstrapping is

the bootstrapping method we get by taking
Px = Pθ̂(x) .
Example 3.7.2. As a continuation of Example 3.5.8 we compute confidence inter-

vals for the parameters α and β as well as the LD50 parameter. A simple parametric
bootstrap with B = 1000 yields the following 95% confidence intervals:
Parameters
α β LD50
se
ˆ 0.713 0.358 0.080
[τ̂ (x) − 1.96se,
ˆ τ̂ (x) + 1.96se]
ˆ [3.74, 6.53] [2.00, 3.41] [−2.06, −1.74]
[τ̂ (x) − ŵ0.05 , τ̂ (x) − ẑ0.05 ] [3.47, 6.14] [1.81, 3.19] [−2.06, −1.74]
Note that the confidence interval(s) for LD50 are quite narrow and the estimated
standard deviation is likewise small – at least compared to the parameters α and β.
The parameter LD50 is simply much better determined than the two original param-
eters. Note also that the two different types of intervals differ for α and β but not for
LD50 . This can be explained by a right-skewed distribution of the estimators for the
two former parameters, which turn into a left translation of the second confidence
intervals as compared to the symmetric intervals based on se. ˆ The distribution of
the estimator for LD50 is much more symmetric around LD50 . ˆ ⋄
3.7.1 The empirical measure and non-parametric bootstrapping
Definition 3.7.3. Given a dataset x1 , . . . , xn we define the empirical probability

measure, or simply the empirical measure, εn , on E by
n
1X
εn (A) = 1(xk ∈ A) (3.21)
n
k=1
for all events A ⊆ E.
The empirical measure is the collection of relative frequencies, εn (A), for all events
A ⊆ E, which we encountered when discussing the frequency interpretation in Sec-
tion 2.3. It is also the frequency interpretation that provides the rationale for con-
sidering the empirical measure.
To define non-parametric bootstrapping we need to assume that the dataset consid-
ered consists of realizations of iid random variables. Thus we assume that E = E0n
and that the dataset x = (x1 , . . . , xn ) is a realization of n iid random variables
X1 , . . . , Xn . The empirical measure based on x1 , . . . , xn on E0 is denoted εn .
Bootstrapping 215
Definition 3.7.4. Non-parametric bootstrapping is defined by letting
Px = εn ⊗ . . . ⊗ εn .
That is, random variables X1 , . . . , Xn with distribution Px are iid each having the
empirical distribution εn .
Whereas the procedure for doing parametric bootstrapping is straight forward, the
definition of non-parametric bootstrapping seems a little more difficult. In fact, the
definition of non-parametric bootstrapping is somewhat complicated. In practice it
is really the other way around. How to do parametric bootstrap simulations from
Pθ̂(x) relies upon the concrete model considered, and sometimes it can be difficult
to actually simulate from Pθ̂(x) . It does as a minimum require a simulation algo-
rithm that is model dependent. To do non-parametric bootstrap simulations using
the empirical measure is on the contrary easy and completely model independent.
Simulating from Px is a matter of simulating (independently) from εn , which in turn
is a matter of sampling with replacement from the dataset x1 , . . . , xn . Theorem 3.7.5
below is an adaption of the general Algorithm 2.11.1, which is suitable for simulat-
ing from any empirical measure. The recommended approach is to sample indices
from the dataset uniformly. This approach is efficient and completely generic. The
implementation requires no knowledge about the original sample space whatsoever.
R Box 3.7.1 (Simulation from the empirical measure). If x is a vector of length

n containing the dataset x1 , . . . , xn we can obtain a sample of size B = 1000 from
the empirical measure by
> y <- sample(x,1000,replace=TRUE)
The vector y then contains 1000 simulations from the empirical measure. Note the
parameter replace which by default is FALSE.
Result 3.7.5. Let x1 , . . . , xn be a dataset with values in the sample space E and
corresponding empirical measure. If U is uniformly distributed on {1, . . . , n} then
the distribution of
X = xU
is the empirical measure εn .
Proof: With En = {z ∈ E | xi = z for some i = 1, . . . , n} we find that for z ∈ En

and with Iz = {i ∈ {1, . . . , n} | xi = z} then
n
|Iz | 1X
P(X = z) = P(xU = z) = P(U ∈ Iz ) = = 1(xi = z) = εn (z).
n n
i=1

An alternative formulation of Theorem 3.7.5, making the transformation explicit, is

via the x1 , . . . , xn -dependent map
hx1 ,...,xn : {1, . . . , n} → E
defined by
hx1 ,...,xn (i) = xi .
Then if U has the uniform distribution on {1, . . . , n} the theorem states that the
transformed variable hx1 ,...,xn (U ) has the empirical distribution.
Remark 3.7.6. It follows from the theorem that if U1 , . . . , UB are B iid uniformly
distributed random variables taking values in {1, . . . , n} then X1 , . . . , Xn defined by
Xi = xUi
for i = 1, . . . , n are iid with distribution εn . Taking U1 , . . . , Un to be iid uniformly

from {1, . . . , n} is known as sampling with replacement n times from {1, . . . , n}.
The random variables X1 , . . . , Xn can therefore be regarded as n samples with re-
placement from the set {x1 , . . . , xn }. How we choose to perform the simulation of
U1 , . . . , Un is another issue. It could be done by Algorithm 2.11.1, but the details
are not important. The implementation of that simulation can be done once and for
all and optimized sufficiently.
Non-parametric bootstrapping makes a priori only sense if our observables are iid. In
a regression setup we cannot use non-parametric bootstrapping directly. However,
we can non-parametrically bootstrap the residuals, which gives a bootstrapping al-
gorithm where
Xi = gβ̂ (yi ) + eUi
with U1 , . . . , Un are iid uniformly distributed on {1, . . . , n}. Thus we use the para-
metric estimate of the mean value relation between the observable and the regressor
and then we use the non-parametric, empirical distribution of the residuals to sample
new error variables.
3.7.2 The percentile method
For the construction of confidence intervals it is tempting to proceed in the following

way. If τ̂ (x) is the estimate then with zα and wα the α/2 and 1 − α/2 quantiles of
the distribution of τ̂ under the probability measure Pθ̂(x) we simply take the interval
[zα , wα ] as a (1 − α)-“confidence interval”. This corresponds to reading a figure like
Figure 3.13 vertically compared to the usual horizontal reading. By this we mean
that instead of locating the estimate on the vertical axis and read of horizontally
the values of p in the case of Figure 3.13 that could produce the estimate with high
probability, we simply read of the distribution of p̂ under Pp̂(x) vertically – and locate
relevant quantiles.
Bootstrapping 217
Math Box 3.7.1 (Percentile interval). Assume that there exists a strictly increas-
ing function ϕ : R → R such that for all θ ∈ Θ the distribution of ϕ(τ̂ ) is symmetric
around ϕ(τ ) under Pθ . This means that the distribution of ϕ(τ̂ ) − ϕ(τ ) equals the
distribution of ϕ(τ ) − ϕ(τ̂ ), and in particular it follows that
ϕ(τ ) − zα′ (θ) = wα′ (θ) − ϕ(τ ),
where zα′ (τ ) and wα′ (τ ) are the α/2 and 1 − α/2 quantiles for the distribution of
ϕ(τ̂ ) under Pθ . The construction of standard (1 − α)-confidence intervals for ϕ(τ )
gives
[2ϕ(τ̂ (x)) − wα′ (τ̂ (x)), 2ϕ(τ̂ (x)) − zα′ (τ̂ (x))] = [zα′ , wα′ ].
Using that ϕ is strictly increasing allows us to take the inverse and find that the
corresponding quantiles for the distribution of τ̂ are zα = ϕ−1 (zα′ ) and wα =
ϕ−1 (wα′ ). The confidence interval obtained in this way for τ – by transforming
back and forth using ϕ – is therefore precisely the percentile interval [zα , wα ].
This argument in favor of the percentile method relies on the existence of the func-
tion ϕ, which introduces symmetry and justifies the interchange of the quantiles.
For any practical computation it is completely irrelevant to actually know ϕ. What
is really the crux of the matter is whether there exists such a ϕ that also makes
the distribution of ϕ(τ̂ ) − ϕ(τ ) largely independent of θ.
We may discard this construction by arguing that the [zα , wα ] is simply a misun-
derstanding of the idea in confidence intervals. Confidence intervals are intervals of
parameters for which the observation is reasonably likely. The interval [zα , wα ] is an
interval where the estimator will take its value with high probability if θ = θ̂(x).
This is in principle something entirely different. There are arguments, though, to
justify the procedure. They are based on the existence of an implicit parameter
transformation combined with a symmetry consideration; see Math Box 3.7.1 The
interval [zα , wα ] – and its refinements – is known in the literature as the percentile
confidence interval or percentile interval for short.
Symmetry, or approximate symmetry, of the distribution of the estimators around τ
makes the percentile interval and the classical interval very similar in many practical
cases. But rigorous arguments that justify the use of the percentile intervals are even
more subtle; see Math Box 3.7.1. If the distribution of the estimator is skewed, the
percentile intervals may suffer from having actual coverage which is too small. There
are various ways to remedy this, known as the bias corrected percentile method
and the accelerated bias corrected percentile method, but we will not pursue these
matters here.
There is one argument in favor of the percentile method, and that is invariance
under monotone parameter transformations. This means that if ϕ : R → R is an
increasing function and τ ′ = ϕ(τ ) is our new parameter of interest, then if [zα , wα ]
is a (1 − α)-percentile confidence interval for τ , then [ϕ(zα ), ϕ(wα )] is a (1 − α)-
percentile confidence interval for τ ′ = ϕ(τ ).
Exercises
Exercise 3.7.1. Consider the Poisson model from Exercise 3.3.4. Implement para-
metric bootstrapping for computing confidence intervals for the two parameters α and
β. Compute for the Ceriodaphnia data 95% confidence intervals for the parameters.
The Poisson model can be estimated using glm in R:
glm(organisms~concentration,family=poisson,data=Ceriodaphnia)
use summary on the result to compute estimates of the standard error and compare
the bootstrapped confidence intervals with standard confidence intervals based on the
estimated standard error.
Exercise 3.7.2. Consider the four parameter logistic model for the ELISA data from
Example 3.6.8. Implement a bootstrapping algorithm (parametric or non-parametric)
for the computation of confidence intervals for the predicted value gβ̂ (x). Use the al-
gorithm for computing 95%-confidence intervals for predicted values using the ELISA
data from Example 3.6.8 for different choices of x – make a plot of the confidence
bands. Compare with corresponding confidence bands computed using a linear re-
gression on the log-log-transformed data as in Example 3.6.2.
4
Mean and Variance
We have previously introduced the mean and the variance for probability measures
on R given by a density or given by point probabilities. For the further development
it is beneficial to put these definitions into a more general context. In this chapter
we deal with expectations of real valued random variables in general. We get several
convenient results about how to compute expectations (means) and variances, and
we get access to a better understanding of how the empirical versions approximate
the theoretical mean and variance. We also touch upon higher order moments and
we discuss the multivariate concept of covariance. As an illustration of how some
of these methods can be applied, we discuss Monte Carlo simulations as a general
method based on random simulation for computing integrals numerically and we
discuss aspects of asymptotic theory. The chapter ends with a brief treatment of
entropy.
4.1 Expectations
For the general development and computations of means and variances of real valued
random variables it is useful to introduce some notation. With reference to Sections
2.4 and 2.6 we define the expectation of a real valued random variable by one of the
two following definitions.
Definition 4.1.1. If X is a real valued random variable with density f then
Z ∞
EX = xf (x)dx
−∞
denotes its expectation provided that

Z ∞
|x|f (x)dx < ∞,
−∞
219
220 Mean and Variance
in which case we say that X has finite expectation.
Definition 4.1.2. If X is a discrete random variable taking values in E ⊆ R with

point probabilities (p(x))x∈E then
X
EX = xp(x)
x∈E
denotes its expectation provided that

X
|x|p(x) < ∞.
x∈E
in which case we say that X has finite expectation.
If X1 and X2 are two real valued random variables with a joint distribution having
density f (x1 , x2 ) Result 2.13.5 shows that their marginal densities are
Z Z
f1 (x1 ) = f (x1 , x2 )dx1 f2 (x2 ) = f (x1 , x2 )dx1 .
Provided that the integrals make sense we can compute the expectation of X1 as
Z Z Z
EX1 = x1 f1 (x1 )dx1 = x1 f (x1 , x2 )dx2 dx1
and similarly for the expectation of X2 . What about the expectation of X1 + X2 ?

In principle we have to compute the distribution of X1 + X2 first and then compute
the mean value for that distribution. The computation of the distribution of the
transformed variable X1 + X2 can, however, be bypassed using the following result,
which we state without a derivation.
Result 4.1.3 (Transformations). If h : Rn → R is a real valued function, if the

distribution of X has density f : Rn → [0, ∞) and if h(X) has finite expectation
then
Z
Eh(X) = h(x)f (x)dx
Z ∞ Z ∞
= ··· h(x1 , . . . , xn )f (x1 , . . . , xn )dx1 · · · dxn .
−∞ −∞
| {z }
n
Remark 4.1.4. Whether h has finite expectation can be checked by computing

Z Z ∞ Z ∞
|h(x)|f (x)dx = ··· |h(x1 , . . . , xn )|f (x1 , . . . , xn )dx1 · · · dxn .
−∞ −∞
| {z }
n
Expectations 221
The result in Result 4.1.3 is very useful since we may not be able to find an explicit
analytic expression for the density of the distribution of h(X) – the distribution may
not even have a density – but often the distribution of X is specified in terms of the
density f . The computation of the iterated integrals can, however, be a horrendous
task.
Taking h(x1 , x2 ) = x1 + x2 we find that
Z Z
E(X1 + X2 ) = (x1 + x2 )f (x1 , x2 )dx1 dx2
Z Z Z Z
= x1 f (x1 , x2 )dx2 dx1 + x2 f (x1 , x2 )dx1 dx2
= EX1 + EX2 .
The computations are sensible if X1 +X2 has finite expectation, but using the remark
above and noting that |x1 + x2 | ≤ |x1 | + |x2 | it follows that X1 + X2 indeed has finite
expectation if X1 and X2 have finite expectations. In conclusion, the expectation of
the sum is the sum of the expectations.
A result similar to Result 4.1.3 but for a discrete distribution can also be derived.
In fact, we will do that here. If X is a random variable with values in a discrete set
E, the random variable h(X) takes values in the discrete subset E ′ ⊆ R given by
E ′ = {h(x) | x ∈ E}.
For each z ∈ E ′ we let Az = {x ∈ E | h(x) = z} denote the set of all x’s in E, which
h maps to z. Note that each x ∈ E belongs to exactly one set Az . We say that the
sets Az , z ∈ E ′ , form a disjoint partition of E. The distribution of h(X) has point
probabilities (q(z))z∈E ′ given by
X
q(z) = P (Az ) = p(x)
x∈Az
by Definition 2.9.3, and using Result 4.1.2 the expectation of h(X) can be written
as X X X
Eh(X) = zp(z) = z q(x).
z∈E ′ z∈E ′ x∈Az
Now the function h is constantly equal to z on Az so we get

X X
Eh(X) = h(x)p(x).
z∈E ′ x∈Az
Since the sets Az , z ∈ E ′ , form a disjoint partition of the sample space E the sum
on the r.h.s. above is precisely a sum over all elements in E, hence
X
Eh(X) = h(x)p(x).
x∈E
We have derived the following result.

Result 4.1.5 (Transformations). Let X be a random variable taking values in a

discrete set E with distribution P given by the point probabilities (p(x))x∈E . If h :
E → R is any real valued function then if h(X) has finite expectation
X
Eh(X) = h(x)p(x).
x∈E
Remark 4.1.6. Similar to the continuous case, h(X) has finite expectation if and
only if X
|h(x)|p(x) < ∞.
x∈E
Remark 4.1.7. If X is a Bernoulli variable with success probability p we find that
EX = 1 × P(X = 1) + 0 × P(X = 0) = p. (4.1)
In Section 4.2 we will develop the theoretical details for the assignment of an expec-
tation to a real valued random variable. In summary, a positive real valued random
variable X can be assigned an expectation EX ∈ [0, ∞] – but it may be equal to ∞.
A real valued random variable X with E|X| < ∞ – the expectation of the positive
real valued random variable |X| is finite – can be assigned an expectation EX ∈ R.
If E|X| < ∞ we say that the random variable X has finite expectation. The main
conclusion from Section 4.2 can be stated as the following result.
Result 4.1.8. If X and Y are two real valued random variables with finite expecta-
tion then X + Y has finite expectation and
E(X + Y ) = EX + EY.
Furthermore, if c ∈ R is a real valued constant then cX has finite expectation and
E(cX) = cEX.
Moreover, if X and Y are independent real valued random variables with finite ex-
pectation then
E(XY ) = EX EY.
Example 4.1.9. If X1 , . . . , Xn are iid Bernoulli variables with success probability
p then
X = X1 + . . . + Xn ∼ Bin(n, p).
We can find the expectation for the binomially distributed random variable X by
using Result 4.1.2
Xn
n k
EX = k p (1 − p)n−k ,
k
k=0
but it requires a little work to compute this sum. It is much easier to use Result
4.1.8 together with (4.1) to obtain that
EX = EX1 + . . . + EXn = p + . . . + p = np.
⋄
Expectations 223
Example 4.1.10. Let X1 , . . . , Xn be iid random variables with values in the four
letter alphabet E = {A, C, G, T}. Let w = w1 w2 . . . wm denote a word from this
alphabet with m letters. Assume that m ≤ n (we may think of m ≪ n) and define
n−m+1
X
N= 1(Xi Xi+1 . . . Xi+m−1 = w).
i=1
Thus N is the number of times word w occurs in the sequence. It follows using (4.1)
and Result 4.1.8 that the expectation of N is
n−m+1
X
EN = E1(Xi Xi+1 . . . Xi+m−1 = w)
i=1
n−m+1
X
= P(Xi Xi+1 . . . Xi+m−1 = w)
i=1
Since the X-variables are independent we have that

P(Xi Xi+1 . . . Xi+m−1 = w) = P(Xi = w1 )P(Xi+1 = w2 ) . . . P(Xi+m−1 = wm )
= p(w1 )p(w2 ) . . . p(wm )
= p(A)nw (A) p(C)nw (C) p(G)nw (G) p(T)nw (T)
where p(A), p(C), p(G) and p(T) are the point probabilities for the distribution of the
X-variables and nw (A), nw (C), nw (G) and nw (T) are the number of A’s, C’s, G’s and
T’s in w. Thus
EN = (n − m + 1)p(A)nw (A) p(C)nw (C) p(G)nw (G) p(T)nw (T) . (4.2)
⋄
If X is a real valued random variable and if we consider the family of transformations

hk (x) = xk for k ∈ N the corresponding family of expectations of the transformed
variables hk (X) = X k for k ∈ N have special names. These expectations are known
as the moments of X, and when the distribution of X is given by either a density
or by point probabilities on a discrete subset of R, Results 4.1.3 and 4.1.5 provide
methods for computing these moments.
Definition 4.1.11. If X is a real valued random variable with E|X|k < ∞ we say
that X has finite k’th moment and call
EX k
the k’th moment of X. The central k’th moment of X is
E(X − EX)k .
For k = 2 the central second moment is the variance, and will be treated in greater
detail below.
4.1.1 The empirical mean
The average or empirical mean of x1 , . . . , xn ∈ R is defined as

n
1X
µ̂n = xi .
n
i=1
Occasionally we may also use the notation x to denote the average of n real numbers.
Though the average is always a well defined quantity, we can only really interpret
this quantity if we regard the x’s as a realization of identically distributed random
variables. If those variables have distribution P and if X is a random variable with
distribution P , then we regard µ̂n as an estimate of µ = EX. In Section 4.5 we derive
some results that say more precisely how µ̂n behaves as an estimate of µ = EX.
In this section we will make one observation – namely that the average is also the
expectation of a random variable, whose distribution is the empirical distribution,
εn , given by x1 , . . . , xn . The empirical distribution on R, which is defined in terms of
x1 , . . . , xn , can be seen as a transformation of the uniform distribution on {1, . . . , n}
via
hx1 ,...,xn : {1, . . . , n} → R,
which is defined by
hx1 ,...,xn (i) = xi .
By Result 4.1.5 the expectation of a random variable with distribution εn is therefore

given as
Xn n
X 1
hx1 ,...,xn (i)p(i) = xi = µ̂n
n
i=1 i=1
where p(i) = 1/n are the point probabilities for the uniform distribution on {1, . . . , n}.
Notationally it may be useful to express what the distribution of X is when we
compute its expectation – if that is not clear from the context. Therefore we write
EP X for the expectation, EX, if the distribution of X is P . This allows us for
instance to write
µ̂n = Eεn X,
indicating that if X has the empirical distribution εn given by x1 , . . . , xn , then its

expectation is the average of x1 , . . . , xn .
If X1 , . . . , Xn are iid with distribution P and (finite) expectation µ = EP X1 , and if
x1 , . . . , xn are realizations of X1 , . . . , Xn , then since we regard the empirical distri-
bution εn to be an approximation of P due to the frequency interpretation we may
regard µ̂n as an approximation of µ. This can be regarded as an application of the
plug-in principle.
More on expectations 225
Exercises
Exercise 4.1.1. Consider the setup from Example 4.1.10 with the word w = TATAAA.
Compute the expectation of N when n = 10000 and the random variables have the
uniform distribution on the alphabet.
4.2 More on expectations
This section contains a rather technical development of the results for computing
expectations based on some fundamental results from measure and integration theory
that is beyond the scope of these notes. The section can be skipped in a first reading.
Result 4.2.1 (Fact). There exists an expectation operator E that assigns to any
positive random variable X (i.e. X takes values in [0, ∞)) a number
EX ∈ [0, ∞]
such that:
1. If X is a Bernoulli random variable then
EX = P(X = 1). (4.3)
2. If c ≥ 0 is a positive constant and X is a positive random variable then
E(cX) = cEX. (4.4)
3. If X and Y are two positive random variables then
E(X + Y ) = EX + EY. (4.5)
The number EX is called the expectation or mean of X.
One should note two things. First, the expectation operator assigns an expectation
to any random variable provided it is real valued and positive. Second, that the
expectation may be +∞. The theorem presented as a fact above gives little infor-
mation about how to compute the expectation of a random variable. For this, the
reader is referred to the previous section.
One of the consequences that we can observe right away is, that if X and Y are
two positive random variables such that X ≤ Y (meaning P(X ≤ Y ) = 1), then
Y − X ≥ 0 is a positive random variable with E(Y − X) ≥ 0. Consequently, if
X ≤ Y , then X + Y − X = Y and
EX ≤ EX + E(Y − X) = E(X + X − Y ) = EY. (4.6)

Math Box 4.2.1 (The expectation operator). The expectation operator of a pos-
itive random variable X can be defined in the following way. First, for 0 = s0 <
s1 < . . . < sn some positive real numbers we form a subdivision of the positive half
line [0, ∞) into n + 1 disjoint intervals I0 , I2 , . . . , In given as
[0, ∞) = I0 ∪ I1 ∪ . . . ∪ In−1 ∪ In
k k ... k k
= [0, s1 ] ∪ (s1 , s2 ] ∪ . . . ∪ (sn−1 , sn ] ∪ (sn , ∞)
Then we can compute the average of the si ’s weighted by the probabilities that
X ∈ Ii :
Xn
ξn (X) := si P(X ∈ Ii ).
i=0
If the size of each of the intervals shrinks towards zero as n → ∞ and sn → ∞ then
it is possible to show that ξn (X) always converges to something. Either a positive
real number or +∞. This limit is called the expectation of X and we may write
EX = lim ξn (X).
n→∞
Note that ξn (X) is defined entirely in terms of the distribution of X and so is the
limit.
Moreover, we can also regard
n
X
Xn = si 1(X ∈ Ii )
i=0
as an approximation to the variable X. We round to the largest si smaller than X.

Then if we want the three properties of E in Result 4.2.1 to be fulfilled, it follows
that n n
X X
EXn = si E1(X ∈ Ii ) = si P (X ∈ Ii ) = ξn (X)
i=1 i=1
using for instance that 1(X ∈ Ii ) is a Bernoulli variable. It is definitely beyond the
scope of these notes to show that ξn (X) converges let alone that Result 4.2.1 holds
for the limit. The mathematically inclined reader is referred to the literature on
measure and integration theory.
Moreover, note that if X is any random variable and A an event then 1(X ∈ A) is
a Bernoulli random variable and by (4.3)
E1(X ∈ A) = P(X ∈ A).
Example 4.1.10 in the previous section actually showed how the expectation of a
positive random variable could be computed from the few elementary rules in Result
4.2.1. One question arises. Why does Result 4.2.1 require that X is positive and not
just real valued? The answer lies in the following consideration: If X is a real valued
random variable we can define two positive real valued random variables by
X + = max{X, 0} and X − = max{−X, 0},
which are called the positive and negative part of X respectively. They are both
transformations of X, and one can get X back from these two variables by
X = X + − X −.
This holds because in general for any x ∈ R we have that
x = max{x, 0} + min{x, 0} = max{x, 0} − max{−x, 0}.
A natural and desirable extension of the additivity property, as given in (4.5), is

that
EX = E(X + − X − ) = EX + − EX − . (4.7)
However, for the right hand side to make sense, we can not have that both the terms
are equal to ∞. There is simply no way to make sense out of subtracting ∞ from
∞. On the other hand, by Result 4.2.1 above both terms EX + and EX − are well
defined, and if they are both finite, we can subtract them. Moreover, in analogy with
the reconstruction of X from X + and X − , one can also construct the transformed
variable |X| – the absolute value of X – from X + and X − . Indeed, for x ∈ R we
have that
|x| = max{x, 0} + max{−x, 0},
hence
|X| = X + + X − .
Since |X| is a positive random variable E|X| exists by Result 4.2.1, and by (4.5) we
have that
E|X| = EX + + EX − .
The sum is finite if and only if both terms are finite. This leads us to the definition:
Definition 4.2.2. If X is a real valued random variable we say that it has finite
expectation if
E|X| < ∞.
In this case the expectation of X, EX, is well defined by (4.7), that is
EX = EX + − EX − .
Without too much trouble, one can now show that the properties listed above as
(4.4) and (4.5) for the expectation of a positive random variable carries over in an
appropriate form for the expectation of a general real valued variable. This provides
a proof of the fundamental Result 4.1.8.
The derivation of Result 4.1.8 from the properties of E for positive random variables
is mostly a matter of bookkeeping. First we note that
|X + Y | ≤ |X| + |Y |,
so if E|X| < ∞ and E|Y | < ∞ then by (4.6) E|X + Y | < ∞. So X + Y does have
finite expectation if X and Y do.
To show the additivity we need a little trick. We have that
(X + Y )+ − (X + Y )− = X + Y = X + − X − + Y + − Y − ,
which can be rearranged to give
(X + Y )+ + X − + Y − = X + + Y + + (X + Y )− ,
where all terms on both sides are positive random variables. Then we can apply
Result 4.2.1, (4.5), to obtain
E(X + Y )+ + EX − + EY − = E(X + Y )− + EX + + EY +
and rearranging this equality back yields
E(X + Y ) = E(X + Y )+ − E(X + Y )− = EX + − EX − + EY + − EY − = EX + EY.
If c ∈ R and X is a real valued random variable |cX| = |c||X|, hence
E|cX| = E|c||X| = |c|E|X|,
and we see that cX has finite expectation if and only if |X| has finite expectation.
Moreover, if c ≥ 0
(cX)+ = cX + and (cX)− = cX − ,
thus
E(cX) = E(cX + ) − E(cX − ) = cEX + − cEX − = c(EX + − EX − ) = cEX.
If c < 0
(cX)+ = −cX − (cX)− = −cX +
and
E(cX) = E(−cX − ) − E(−cX + ) = −c(EX − − EX + ) = −c(−EX) = cEX.
The proof of E(XY ) = EX EY for independent variables is skipped since it requires

more knowledge of the underlying definition of the expectation. Note, however, that
if X and Y are independent Bernoulli random variables then XY is a Bernoulli
random variable and
E(XY ) = P(XY = 1) = P(X = 1, Y = 1) = P(X = 1)P(Y = 1) = EX EY

where the third equality follows form independence of X and Y . Without indepen-
dence the conclusion is wrong. This is the starting point for a general proof – relying
on the construction discussed in Math Box 4.2.1.
At this point in the abstract development it is natural to note that the Definitions
4.1.1 and 4.1.2 of the expectation of course coincide with the abstract definition.
In fact, Definitions 4.1.1 and 4.1.2 can be viewed as computational techniques for
actually computing the expectation. As we showed in Example 4.1.10 we may be able
to derive the mean of a random variable without getting even close to understanding
the actual distribution of the random variable. The integer variable N introduced
in Example 4.1.10 has a far more complicated distribution than we can handle, but
its mean can be derived based on the simple computational rules for E. Returning
to Definitions 4.1.1 and 4.1.2 for computing the expectation should be regarded as
a last resort.
Example 4.2.3 (Mixtures). Most of the standard distributions like the normal
distribution are not suited for models of multimodal data. By this we mean data
that cluster around several different points in the sample space. This will show up
on a histogram as multiple modes (peaks with valleys in between). If we for instance
consider gene expressions for a single gene in a group of patients with a given disease,
multiple modes may occur as a result of, yet unknown, subdivisions of the disease on
the gene expression level. In general, we want to capture this phenomena that there
is a subdivision of the observed variable according to an unobserved variable. This
is captured by a triple of independent variables (Y, Z, W ) where Y and Z are real
valued random variables and W is a Bernoulli variable with P(W = 1) = p. Then
we define
X = Y W + Z(1 − W ).
The interpretation is that the distribution of X is given as a mixture of the distri-
bution of Y and Z in the sense that either W = 1 (with probability p) in which
case X = Y or else W = 0 (with probability 1 − p) in which case X = Z. Since
|X| ≤ |Y |W + |Z|(1 − W ) ≤ |Y | + |Z| it follows by Result 4.1.8 that
E|X| ≤ E|Y | + E|Z|.
Thus X has finite expectation if Y and Z have finite expectation. Moreover, Result
4.1.8 implies, since Y and W are independent and Z and W are independent, that
EX = E(Y W ) + E(Z(1 − W )) = EY EW + EZ (1 − E(W )) = pEY + (1 − p)EZ.
The probabilities p and 1 − p of (W = 1) and (W = 0) respectively are called

the mixture proportions, and the equality above says, quite intuitively, that the
expectation of X is a weighted average of the expectation of Y and the expectation
of Z – weighted according to the mixture proportions.
If Y ∼ N (ξ1 , σ12 ) and Z ∼ N (ξ2 , σ22 ) then
EX = pξ1 + (1 − p)ξ2 ,
Math Box 4.2.2. If X is a positive real valued random R ∞ variable with distribution
function F then it has finite expectation if and only if 0 1−F (x)dx < ∞ in which
case Z ∞
EX = 1 − F (x)dx.
0
This makes us in principle capable of computing the expectation of any real valued
random variable X. Both X + and X − are positive random variables, and with F +
and F − denoting their respective distribution functions we get that if
Z ∞ Z ∞
1 − F + (x)dx < ∞ and 1 − F − (x)dx < ∞
0 0
then X has finite expectation and
EX = EX + − EX −
Z ∞ Z ∞
= 1 − F + (x)dx − 1 − F − (x)dx
Z0 ∞ 0
− +
= F (x) − F (x)dx.
0
but note that the distribution of X is far from being a normal distribution. ⋄
4.3 Variance
Definition 4.3.1. If X is a real valued random variable with expectation EX, then
if X has finite second moment, that is, if X 2 has finite expectation, we define the
variance of X as
VX = E(X − EX)2 (4.8)
√
and the standard deviation is defined as VX.
The variance is the expectation of the squared difference between X and its expec-
tation EX. This is a natural way of measuring how variable X is.
Remark 4.3.2. Writing out (X − EX)2 = X 2 − 2XEX + (EX)2 and using Result
4.1.8 we obtain
VX = EX 2 − 2EXEX + (EX)2 = EX 2 − (EX)2 , (4.9)
which is a useful alternative way of computing the variance. The expectation of X 2 ,
EX 2 , is called the second moment of the distribution of X.
Remark 4.3.3. For any µ ∈ R we can write
(X − µ)2 = (X − EX + EX − µ)2
= (X − EX)2 + 2(X − EX)(EX − µ) + (EX − µ)2 ,
Variance 231
from which
E(X − µ)2 = E(X − EX)2 + 2(EX − EX)(EX − µ) + (EX − µ)2

= VX + (EX − µ)2 ≥ VX (4.10)
with equality if and only if EX = µ. The number E(X − µ)2 is the expected squared
difference between µ and X, and as such a measure of how much the outcome
deviates from µ on average. We see that the expectation EX is the unique value of µ
that minimizes this measure of deviation. The expectation is therefore in this sense
the best constant approximation to any outcome of our experiment.
Example 4.3.4. If X is a Bernoulli random variable with success probability p we

know that EX = p. We find using Result 4.1.5 that
VX = E(X − p)2 = (1 − p)2 P(X = 1) + p2 P(X = 0)

= (1 − p)2 p + p2 (1 − p) = (1 − p)p(1 − p + p) = (1 − p)p.
Example 4.3.5. If X is a random variable with mean 0 and variance 1, if µ ∈ R

and σ > 0, then
E(σX + µ) = σE(X) + µ = µ
and
V(σX + µ) = E(σX + µ − µ)2 = E(σ 2 X 2 ) = σ 2 VX = σ 2 .
This shows that the location scale-transformation, as considered in Example 2.9.9,

of X has mean µ and variance σ 2 . This was shown in Example 2.9.9 for the case
where the distribution had a density. In the other direction we find that if EX = µ
and VX = σ 2 then

X −µ X −µ
E = 0 and V = 1.
σ σ
X−µ
We refer to σ as the normalization of X, which has mean 0 and standard deviation
1. ⋄
If x1 , . . . , xn are realizations of n identically distributed random variables with finite

second moment and whose variance is σ 2 , the sample variance or empirical variance
of x1 , . . . , xn is defined as
n
1X
σ̃n2 = (xi − µ̂n )2 (4.11)
n
i=1
where µ̂n is the empirical mean. The empirical variance is an estimate of the variance
σ 2 . Like the empirical mean, the empirical variance is the variance of a random
variable having distribution εn . With the notation as in Section 4.1.1 and using
Result 4.1.5 the variance of X having distribution εn is
Vεn X = Eεn (X − Eεn X)2

Xn
1
= (hx1 ,...,xn (i) − µ̂n )2
n
i=1
n
1X
= (xi − µ̂n )2 = σ̃n2 .
n
i=1
As for the expectation we use the subscript notation, VP X, to denote the variance,
VX, of X if the distribution of X is P . The square root of the empirical variance,
p
σ̃n = σ̃n2 ,
is called the sample standard deviation and is an estimate of the standard deviation,
σ, of the random variables.
Since σ̃n2 is the variance of a random variable X having distribution εn we can use
computational rules for variances. For instance, (4.9) can be used to obtain the
alternative formula
n
1X 2
σ̃n2 2
= Vεn X = Eεn X − (Eεn X) = xi − µ̂2n .
2
(4.12)
n
i=1
It should be remarked that whereas (4.9) can be quite useful for theoretical com-
putations it may not be suitable for numerical computations. This is because both
1 Pn 2 2
n i=1 xi and µ̂n can attain very large numerical values, and subtracting numeri-
cally large numbers can lead to a serious loss of precision.
Example 4.3.6 (Empirical normalization). We consider a dataset x1 , . . . , xn ∈ R
and let X be a random variable with distribution εn (the empirical distribution).
Then by definition
Eεn X = µ̂n and Vεn X = σ en2 .
The normalized dataset, {x′1 , . . . , x′n }, is defined by
xi − µ̂n
x′i =
en
σ
and the normalized empirical distribution, ε′n , is given by the normalized dataset. If
X has distribution εn the normalized random variable
X − µ̂n
X′ =
en2
σ
has distribution ε′n , and we find, referring to Example 4.3.5 that

′ X − µ̂n
Eε′n X = Eεn =0
σen2
Variance 233
and
′ X − µ̂n
Vε′n X = Vεn = 1.
en2
σ
In other words, if we normalize the dataset using the empirical mean and sample
standard deviation, the resulting normalized dataset has empirical mean 0 and sam-
ple standard deviation 1. ⋄
Example 4.3.7. In Example 3.1.4 we considered the two model paradigms – addi-
tive noise and multiplicative noise. The additive noise model for X was formulated
as
X = µ + σε (4.13)
where µ ∈ R, σ > 0 and ε is a real valued random variable with Eε = 0 and Vε = 1.
Using the rules for E developed in this chapter we find that
EX = µ + σEε = µ,
and
VX = σ 2 Vε = σ 2 .
The multiplicative noise model was given as
X = µε (4.14)
with ε a positive random variable, µ > 0. In this case
EX = µEε.
and
VX = µ2 Vε.
The
√ notable property of the multiplicative noise model is that the standard deviation,
VX = µσ, scales with the mean. That is, the larger the expected value of X is the
larger is the noise. For the additive noise model the size of the noise is unrelated to
the mean value.
As discussed in Example 3.1.4 one often transforms the multiplicative noise model
via the logarithm to get
log X = log µ + log ε,
which is an additive noise model. Logarithms and expectations are not interchange-
able, though, and it actually holds that
E log ε < log Eε,
hence if we want E log ε = 0 to assure that E log X = log µ the expectation of X is

always > µ. If we, in addition, assume that log εi ∼ N (0, σ 2 ) as is often done for
the additive noise model the distribution of ε is called the log-normal distribution.
In other words, ε = exp(Z) where Z ∼ N (0, σ 2 ). ⋄
4.4 Multivariate Distributions
If we consider two real valued random variables X and Y , the bundled variable
(X, Y ) takes values in R2 . The mean and variance of each of the variables X and Y
rely exclusively on the marginal distributions of X and Y . Thus they tell us nothing
about the joint distribution of X and Y . We introduce the covariance as a measure
of dependency between X and Y .
Definition 4.4.1. If XY has finite expectation the covariance of the random vari-
ables X and Y is defined as
V(X, Y ) = E ((X − EX)(Y − EY )) (4.15)
and the correlation is defined by
V(X, Y )
corr(X, Y ) = √ . (4.16)
VXVY
The covariance is a measure of the covariation, that is, the dependency between the
two random variables X and Y . The correlation is a standardization of the covariance
by the variances of the coordinates X and Y .
We should note that the covariance is symmetric in X and Y :
V(X, Y ) = V(Y, X).
Furthermore, if X = Y then
V(X, X) = E(X − EX)2 = VX,
that is, the covariance of X with X is simply the variance of X. Moreover,
V(X, Y ) = E ((X − EX)(Y − EY ))

= E(XY − XEY − Y EX + EXEY )
= E(XY ) − EXEY − EXEY + EXEY
= E(XY ) − EXEY, (4.17)
which gives an alternative formula for computing the covariance. Using this last
formula, it follows from Result 4.1.8 that if X and Y are independent then
V(X, Y ) = 0.
On the other hand it is important to know that the covariance being equal to zero
does not imply independence. We also obtain the generally valid formula
E(XY ) = EXEY + V(X, Y )

Multivariate Distributions 235
for the expectation of the product of two random variables.

If we have two random variables, X and Y , with finite variance, the we use (4.9) to
compute the variance of X + Y :
V(X + Y ) = E(X + Y )2 − (E(X + Y ))2

= E(X 2 + Y 2 + 2XY ) − ((EX)2 + (EY )2 + 2EXEY )
= EX 2 − (EX)2 + EY 2 − (EY )2 + 2(EXY − EXEY )
= VX + VY + 2V(X, Y )
using (4.9) again together with (4.17) for the last equality.
We also find that
V(cX) = E(cX − cEX)2 = c2 E(X − EX)2 = c2 VX.
We summaries these derivations as follows.
Result 4.4.2. If X and Y are two random variables with finite variance then the
sum X + Y has finite variance and
V(X + Y ) = VX + VY + 2V(X, Y ). (4.18)
If X is a random variable with finite variance and c ∈ R is a constant then cX has

finite variance and
V(cX) = c2 VX. (4.19)
Remark 4.4.3. We observe that the formula
V(X + Y ) = VX + VY (4.20)
holds if and only if V(X, Y ) = 0, which in particular is the case if X and Y are
independent. Note also that it follows from the theorem that
V(X − Y ) = V(X + (−1)Y ) = VX + V((−1)Y ) + 2V(X, −Y )

= VX + (−1)2 VY + 2V(X, −Y )
= VX + VY − 2V(X, Y ).
Result 4.4.4 (Cauchy-Schwarz). The covariance fulfills the Cauchy-Schwarz in-

equality. If X and Y are real valued random variables with finite variance
√ √
|V(X, Y )| ≤ VX VY .
The inequality is a classical mathematical result, but the derivation is, in fact, quite
elementary given the tools we already have at our disposal, so we provide a derivation
here for completeness.
Proof: A computation shows that for all t ∈ R
0 ≤ E((X − EX) + t(Y − EY ))2

= E(X − EX)2 + 2tE ((X − E(X)(Y − EY )) + t2 E(Y − E(Y )2
= VX + 2tV(X, Y ) + t2 VY.
Defining g(t) = VX + 2tV(X, Y ) + t2 VY as a second order polynomial in t we know

that the polynomial can only be positive if the discriminant of the polynomial is
negative. That is,
4V(X, Y )2 − 4VXVY ≤ 0,
which implies the Cauchy-Schwarz inequality.
The Cauchy-Schwarz inequality implies that the correlation is always a number
between −1 and 1, that is,
−1 ≤ corr(X, Y ) ≤ 1.
If we consider not just two random variables but an n-dimensional vector X =

(X1 , . . . , Xn ) of real valued random variables with finite variance we can compute
the n2 covariances for each pair of variables Xi and Xj . One usually arranges the
covariances in a n × n matrix Σ given by
Σij = V(Xi , Yj ).
That is  

 VX1 V(X1 , X2 ) · · · V(X1 , Xn ) 


 

V(X2 , X1 ) V(X2 ) ··· V(X2 , Xn )
Σ= .. .. .. .. .

 . . . . 


 

V(Xn , X1 ) V(Xn , X2 ) · · · VXn
Note that due to the symmetry of the covariance we have that the covariance matrix
Σ is symmetric:
Σij = Σji .
As a direct consequence of Result 4.4.2 we have the following result about the vari-
ance of the sum of n real valued random variables.
Result 4.4.5. If X1 , . . . , Xn are n real valued random variables with finite variance
and covariance matrix Σ then
n
! n X n
X X
V Xi = Σij
i=1 i=1 j=1
Xn X
= VXi + 2 V(Xi , Xj ).
i=1 i<j
Multivariate Distributions 237
Example 4.4.6. Continuing Example 4.1.10 we want to compute the variance of the
counting variable N . This is a quite complicated task, and to this end it is useful to
introduce some auxiliary variables. Recall that X1 , . . . , Xn are iid random variables
with values in E = {A, C, G, T} and w = w1 w2 . . . wm is an m-letter word. Define for
i = 1, . . . , n − m + 1
Yi = 1(Xi Xi+1 . . . Xi+m−1 = w)
to be the Bernoulli random variable that indicates whether the word w occurs with
starting position i in the sequence of random variables. Then
n−m+1
X
N= Yi .
i=1
To compute the variance of N we compute first the covariance matrix for the vari-
ables Y1 , . . . , Yn−m+1 . If i + m ≤ j then Yi and Yj are independent because the
two vectors (Xi , Xi+1 , . . . , Xi+m−1 ) and (Xj , Xj+1 , . . . , Xj+m−1 ) are independent.
By symmetry
Σij = V(Yi , Yj ) = 0
if |i − j| ≥ m. If |i − j| < m the situation is more complicated, because then the
variables are actually dependent. We may observe that since the X-variables are iid,
then for all i ≤ j with fixed j − i = k < m the variables
(Xi , Xi+1 , . . . , Xi+m−1 , Xj , Xj+1 , . . . , Xj+m−1 )
have the same distribution. Thus if we define
ρ(k) = V(Y1 , Yk )
for k = 0, . . . , m − 1 then, using symmetry again,
Σij = V(Yi , Yj ) = ρ(|i − j|)
for |i − j| < m. Thus for 2m − 1 ≤ n the covariance matrix Σ has the structure
 ρ(0) ρ(1) ρ(2) . . . ρ(m−1) 0 0 ... 0 0 0


 0 


 ρ(1) ρ(0) ρ(1) . . . ρ(m−2) ρ(m−1) 0 ... 0 0 


 ρ(2) ρ(1) ρ(0) . . . ρ(m−3) ρ(m−2) ρ(m−1) . . . 0 0 0  

 


 .
. .
. .
. .. .
. .
. .
. .. .
. .
. 


 . . . . . . . . . . 

 ρ(m−1) ρ(m−2) ρ(m−3) . . . ρ(0) ρ(1) ρ(2) ... 0 0 0 
Σ= 0 ρ(m−1) ρ(m−2) . . . ρ(1) ρ(0) ρ(1) ... 0 0 0 ,

 0 0 ρ(m−1) . . . ρ(2) ρ(1) ρ(0) ... 0 0 0  

 


 .. .. .. .. .. .. .. .. .. .. 


 . . . . . . . . . . 


 0 0 0 ... 0 0 0 . . . ρ(0) ρ(1) ρ(2) 


 
. . . ρ(1) ρ(0) ρ(1) 
 0 0 0 ... 0 0 0 
0 0 0 ... 0 0 0 ... ρ(2) ρ(1) ρ(0)
which is non-zero in a diagonal band. Using Result 4.4.5 we see that there are n−m+1
terms ρ(0) (in the diagonal) and 2(n − m + 1 − k) terms ρ(k) for k = 1, . . . , m − 1,
and therefore
m−1
X
V(N ) = (n − m + 1)ρ(0) + 2(n − m + 1 − k)ρ(k).
k=1
We can also find

ρ(0) = V(Y1 ) = pw (1 − pw )
where pw = p(w1 )p(w2 ) . . . p(wm ) is the “word probability”. The value of ρ(k) for
k = 1, . . . , m − 1 depends, however, crucially and in a complicated way on the
structure of self-overlap in the word w.
If w does not self-overlap, that is, no suffix of the word equals a prefix of the same
word, then E(Y1 Yk ) = P(Y1 Yk = 1) = 0 for k = 1, . . . , m − 1 and we see that
ρ(k) = V(Y1 , Yk ) = E(Y1 Yk ) − EY1 EYk = −p2w .
Therefore, if w does not self-overlap we find the formula

m−1
X
V(N ) = (n − m + 1)pw (1 − pw ) − 2p2w (n − m + 1 − k)
k=1
= (n − m + 1)pw (1 − pw ) − p2w (m − 1)(2n + 2 − 3m)
for the variance of the number of occurrences N of the word w in a sequence of n

iid random variables.
We say that w has a k-shift overlap if
w1+k w2+k . . . wm = w1 w2 . . . wm−k ,
which means that the m − k prefix of the word equals the m − k suffix of the word.
In that case
E(Y1 Yk ) = P(Y1 Yk = 1) = P(X1 . . . Xm = w, Xm+1 . . . Xm+k = wm−k+1 . . . wm )

= pw p(wm−k+1 ) . . . p(wm ),
and therefore
ρ(k) = pw p(wm−k+1 ) . . . p(wm ) − p2w
if w has a k-shift overlap. ⋄
With x1 , . . . , xn ∈ E a dataset where xl = (x1l , . . . , xkl ) ∈ Rk the computation of

the covariance under the empirical measure gives
2
eij,n
σ = Vεn (Xi , Xj ) = Eεn ((Xi − Eεn Xi )(Xj − Eεn Xj ))
n
1X
= (xil − µbi,n )(xjl − µ
bj,n )
n
l=1
where
n
1X
bi,n =
µ xil .
n
l=1
Properties of the Empirical Approximations 239
Math Box 4.4.1 (Multidimensional confidence sets). If Θ ⊆ Rd the θ parameter

is a d-dimensional (column) vector and if Σ(θ) denotes the covariance matrix of θ̂
under Pθ a possible test statistic for testing the null-hypothesis H0 : θ = θ0 is
h(x, θ0 ) = (θ̂(x) − θ0 )t Σ(θ̂(x))−1 (θ̂(x) − θ0 ). (4.22)

Large values are critical. The corresponding multivariate confidence set becomes

I(x) = θ0 ∈ Θ | h(x, θ0 ) ≤ z

= θ0 ∈ Θ | (θ̂(x) − θ0 )T Σ(θ̂(x))−1 (θ̂(x) − θ0 ) ≤ z .
Such a set is known as an ellipsoid in Rd , and if d = 2 the set is an ellipse.
Using (4.17) instead we obtain that

n
2 1X
eij,n
σ = xil xjl − µ
bi,n µ
bj,n . (4.21)
n
l=1
As for the variance this is not a recommended formula to use for the actual compu-
tation of the empirical covariance.
e n is given by
The empirical covariance matrix Σ
e ij,n = σ
Σ 2
eij,n
The empirical correlation becomes

2 Pn
eij,n
σ (xil − µbi,n )(xjl − µ
bj,n )
g ij,n =
corr = pPn l=1 Pn .
ei,n σ
σ ej,n l=1 (xil − µ̂i,n )
2
l=1 (xjl − µ̂j,n )
2
If x1 , . . . , xn are realizations of n identically distributed random variables with values

in Rk the empirical covariances and correlations are estimates of the theoretical
covariances and correlations.
4.5 Properties of the Empirical Approximations
We consider X1 , . . . , Xn independent and identically distributed with distribution P

on E and εn is the corresponding empirical probability measure,
n
X
εn (A) = 1(Xk ∈ A), A ⊆ E,
k=1
Let A be any event, then since 1(Xi ∈ A) is a Bernoulli variable we can use Result
4.2.1 to find that
E1(Xi ∈ A) = P (A)
so
n
! n
1X 1X
Eεn (A) = E 1(Xi ∈ A) = E1(Xi ∈ A)
n n
i=1 i=1
n
X
1
= P (A) = P (A).
n
i=1
Example 4.3.4 gives that
V1(Xi ∈ A) = P (A)(1 − P (A)).
Hence by independence of the random variables X1 , . . . , Xn and Result 4.4.2

n
! n
1X 1 X
Vεn (A) = V 1(Xi ∈ A) = 2 V1(Xi ∈ A)
n n
i=1 i=1
n
1 X 1
= 2
P (A)(1 − P (A)) = P (A)(1 − P (A)).
n n
i=1
We have derived the following result about the empirical probability measure.
Result 4.5.1. With εn the empirical probability measure, and A any event it holds
that
Eεn (A) = P (A) (4.23)
and
1
Vεn (A) = P (A)(1 − P (A)). (4.24)
n
As for all other probability measures the collection of numbers εn (A) for all events
A ⊆ E is enormous even for a small, finite set E. If E is finite we will therefore
prefer the smaller collection of frequencies
n
1X
εn (z) = 1(xi = z)
n
i=1
for z ∈ E – which is also sufficient for completely determining the empirical measure
just like for any other probability measure on a discrete set. If P is given by the
point probabilities (p(z))z∈E Result 4.5.1 tells us that
Eεn (z) = p(z) (4.25)
and
1
Vεn (z) = p(z)(1 − p(z)). (4.26)
n
R Box 4.5.1 (Mean and variance). If x is a numeric vector one can compute the
(empirical) mean of x simply by
> mean(x)
Likewise, the (empirical) variance can be computed by
> var(x)
Using var results in σ̂n2 where we divide by n − 1.

The normalized dataset where we subtract the mean and divide by the variance
can be efficiently computed by
> y <- scale(x)
We recall that the empirical measure can be regarded as a realization of a random

variable. Thus with a different realization we would get a different empirical mean µ̂n
and a different empirical variance σ̃n2 . To evaluate the performance of these empirical
quantities as approximations of the expectation and variance respectively we can
study their distributions when regarded as random variables, that is, when regarded
as estimators. In particular we can compute the expectation and variance of µ̂n and
σ̃n2 .
The computation is an exercise in using the properties of the expectation operator
and independence of the random variables X1 , . . . , Xn . First
n
! n
1X 1X
Eµ̂n = E Xi = EXi = EX,
n n
i=1 i=1
then using independence

n
! n
1X 1 X 1
Vµ̂n = V Xi = 2
VXi = VX,
n n n
i=1 i=1
and finally
n
! n
1X 2 1X
Eσ̃n2 = E Xi − µ̂2n = EXi2 − Eµ̂2n
n n
i=1 i=1
1 n−1
= EX 2 − Vµ̂n − (Eµ̂n )2 = EX 2 − (EX)2 − VX = VX.
n n
Thus we have the following result.
Result 4.5.2. Considering the empirical mean µ̂n and the empirical variance σ̃n2 as
estimators of the mean and variance respectively we have
1
Eµ̂n = EX and Vµ̂n = VX (4.27)
n
together with
n−1
Eσ̃n2 = VX. (4.28)
n
The theorem shows that the expected value of µ̂n equals the true expectation EX
and that the variance of µ̂n decreases as 1/n. Thus for large n the variance of µ̂n
becomes negligible and µ̂n will always be a very close approximation to EX. How
large n should be depends on the size of VX. Regarding the empirical variance
its expectation does not equal the true variance VX. The expected value is always
smaller than VX. The relative deviation is
VX − Eσ̃n2 1
= ,
VX n
which becomes negligible when n becomes large. However, for n = 5, say, the em-
pirical variance undershoots the true variance by 20% on average. For this reason
the empirical variance is not the preferred estimator of the variance. Instead the
standard estimator is
n
n 1 X
σ̂n2 = σ̃n2 = (xi − µ̂n )2 . (4.29)
n−1 n−1
i=1
It follows from Result 4.5.2 and linearity of the expectation operator that
Eσ̂n2 = V(X).
p
The square root σ̂n = σ̂n2 naturally becomes the corresponding estimator of the
standard deviation. Note, however, that the expectation argument doesn’t carry over
to the standard deviations. In fact, it is possible to show that
p
Eσ̂n < V(X)
so σ̂n is still expected to undershoot the standard deviation.

q
The standard deviation of µ̂n is n1 VX, which is usually called the standard error
of the mean, and we call √1 σ̂n (or alternatively √1 σ̃n ) the sample standard error of
n n
the mean.
It is also possible to compute the variance of σ̃n2 but the derivation is long and tedious
so we will skip it. The result is
n−1
Vσ̃n2 = (n − 1)E(X − EX)4
− (n − 3)(VX)2
, (4.30)
n3
which is not a particularly nice formula either. One can observe though that the
variance decreases approximately as 1/n, which shows that also the empirical vari-
ance becomes a good approximation of the true variance when n becomes large. But
regardless of whether we can compute the variance of the empirical variance, we can
compare the variance of σ̃n2 with the variance of σ̂n2 and find that
2
n
V(σ̂n2 ) = V(σ̃n2 ).
n−1
Hence the variance of σ̂n2 is larger than the variance of the empirical variance σ̃n2 .
This is not necessarily problematic, but it should be noticed that what we gain by
correcting the empirical variance so that the expectation becomes right is (partly)
lost by the increased variance.
If we consider an n-dimensional random variable X = (X1 , . . . , Xn ) we can also
derive a result about the expectation of the empirical covariance.
Using (4.21) yields
n
1X
Eσ̃ij,n = EXil Xjl − Eb bj,n
µi,n µ
n
l=1
n n
1 XX
= EXi Xj − EXil Xjm
n2
l=1 m=1
Observing then that due to independence of Xil and Xjm when m 6= l
EXil Xjm = EXi EXj .
There are n(n − 1) such terms in the last sum above. There are n terms equaling
EXi Xj . This gives that
1 n−1
Eσ̃ij,n = EXi Xj − EXi Xj − EXi EXj
n n
n−1 n−1
= (EXi Xj − EXi EXj ) = V(Xi , Xj ).
n n
Thus we have the result.
Result 4.5.3. Considering the empirical covariance as an estimator of the covari-
ance, its expectation is
n−1
Eσ̃ij,n = V(Xi , Xj ).
n
As we can see the empirical covariance also generally undershoots the true covariance
leading to the alternative estimate
n
1 X
σ̂ij,n = (xil − µ
bi,n )(xjl − µ
bj,n) (4.31)
n−1
l=1
of the true covariance with Eσ̂ij,n = V(Xi , Xj ).

To conclude this section we present a simple yet fundamental result that states in
a rather precise way how the empirical mean approximates the true mean when the
number of observations grows to infinity.
Consider the random variable (X − µ)2 1(|X − µ| > ε), which equals (X − µ)2 if
|X − µ| > ε and 0 otherwise. Clearly
ε2 1(|X − µ| > ε) ≤ (X − µ)2 1(|X − µ| > ε) ≤ (X − µ)2
so taking the expectation on both sides yields
ε2 P(|X − µ| > ε) ≤ E(X − µ)2 = VX,
which, slightly rewritten yields the Chebychev inequality.
Result 4.5.4 (Chebychevs inequality). If X is a random variable with finite vari-
ance and expectation µ = EX then for all ε > 0
VX
P(|X − µ| > ε) ≤ 2 .
ε
A most important derivation based on Chebychevs inequality is the Law of Large
2
Numbers. Recall that E(µ̂n ) = µ and V(µ̂n ) = σn , thus if we apply the Chebychev
inequality to µ̂n we find that
V(µ̂n ) σ2
P(|µ̂n − µ| > ε) ≤ = .
ε2 nε2
This gives us the Law of Large Numbers:
Result 4.5.5. If X1 , . . . , Xn are n iid real valued random variables with finite vari-
ance σ 2 and expectation µ, then with
n
1X
µ̂n = Xi
n
i=1
we have for all ε > 0 that
σ2
P(|µ̂n − µ| > ε) ≤ →0
nε2
for n → ∞.
Exercises
Exercise 4.5.1. Compute VX where X is exponentially distributed with intensity

parameter λ > 0. Recall that the expectation or mean is µ = EX = λ−1 and the MLE
of the mean based on iid exponentially distributed random variables X1 , . . . , Xn is
n
1X
µ̂n = Xi .
n i=1
Compute Vµ̂n and the standard error of µ̂n .

Monte Carlo Integration 245
4.6 Monte Carlo Integration
Expectations have a special and classical role to play in probability theory and statis-
tics. There are quite a few examples of distributions on R where we can analytically
compute the expectation. However, there are also many many situations where we
have no chance of computing the expectation analytically. Remember that one of
the analytic tools we have at our disposal is the formula from Definition 4.1.1 or
more generally from Result 4.1.3
Z ∞
Eh(X) = h(x)f (x)dx.
−∞
when X is a real valued random variable, whose distribution has density f and
h : R → R such that h(X) has finite expectation. Thus the success of an analytic
computation of an expectation depends heavily on our ability to compute integrals.
The process can be reversed. Instead of computing the integral analytically we can
rely on the empirical mean as an approximation of the expectation and thus of the
integral. As discussed in Section 4.5 the more (independent) observations we have
the more precise is the approximation, so if we can get our hands on a large number
of iid random variables X1 , . . . , Xn , whose distribution is given by the density f , we
can approximate the integral by the (random) empirical mean
n
1X
h(Xi ).
n
i=1
According to Result 4.5.2 the variance of this random quantity decays like 1/n for
n tending to ∞, and thus for sufficiently large n the empirical mean is essentially
not random anymore. The empirical mean becomes a numerical approximation of
the theoretical integral, and with modern computers and simulation techniques it is
often easy to generate the large number of random variables needed. This technique
is called Monte Carlo integration referring to the fact that we use randomness to do
numerical integration.
Example 4.6.1. We know from Example 2.6.15 that the density for the Gumbel
distribution is
f (x) = exp(−x) exp(− exp(−x)) = exp(−x − exp(−x)).
We should note that the latter formula is more suitable than the former for numerical
computations. The mean value for the Gumbel distribution can then be written as
Z ∞
x exp(−x − exp(−x))dx.
−∞
There is no easy way to compute this integral, but it can be computed numerically
in R using the integrate function. It can also be computed using Monte-Carlo
0.8
0.8
0.7
0.7
Empirical mean
Empirical mean
0.6
0.6
0.5
0.5
0.4
0.4
0 2000 4000 6000 8000 10000 0 2000 4000 6000 8000 10000
n n
Figure 4.1: The average for n simulated Gumbel variables (left) as a function of n
and the average of − log y1 , . . . , − log yn for n simulated exponential variables (right)
as a function of n.
integration by simulating iid Gumbel distributed random variables and computing

the average. Figure 4.1 shows how the average looks as a function of n. We use a
figure like this to investigate how large an n we should choose.
One attempt to compute the integral above analytically would be to make the sub-
stitution y = exp(−x) so that dy = exp(−x)dx and the integrals equals
Z ∞
− log(y) exp(−y)dy.
0
We recognize this integral as the expectation of − log Y where Y has the exponential
distribution. This integral is of course as difficult as the former to compute explicitly,
but we can use Monte-Carlo integration where we take f (x) = exp(−x) the density
for the exponential distribution and h(x) = − log x.
Most likely the two Monte-Carlo algorithms turn out to be identical when it comes
to implementations. ⋄
R∞
Elaborating a little on the idea, we may start with an integral −∞ g(x)dx that we
would like to compute. Initially we have no reference to a probability measure, but
we may take f to be any density, which we for technical reasons assume to be strictly
positive everywhere, that is f (x) > 0 for all x ∈ R. Then
Z ∞ Z ∞
g(x)
g(x)dx = f (x)dx,
−∞ −∞ f (x)
so if we define h(x) = g(x)/f (x), which is well defined as f (x) > 0 for all x ∈ R, we
find that Z
∞
g(X)
g(x)dx = Eh(X) = E
−∞ f (X)
where the distribution of X has density f . Simulating X1 , . . . , Xn as independent
random variables, whose distribution has density f , we get the empirical approxi-
mation
n Z ∞
1 X g(Xi )
≃ g(x)dx,
n f (Xi ) −∞
i=1
which is valid for large n. In this particular setup the Monte Carlo integration is also
known as importance sampling. This is basically because we are free here to choose
f , and a good choice is in general one such that f (x) is large when g(x) is large.
The x′ s where g(x) is large (and g is a reasonably nice function, not oscillating too
rapidly) contribute the most to the integral, and by taking f (x) large when g(x)
is large means that we put a large weight f (x) on the important points x where
we get the largest contribution to the integral. This is a heuristic, not a precise
mathematical result.
2
Example 4.6.2. Take g(x) = 1[a,∞) (x)f0 (x) where f0 (x) = √12π exp − x2 is the
density for the normal distribution with mean 0 and variance 1 and a > 0. We think
of a as quite large, and thus if Y is a random variable with distribution having
density f0 we see that
Z ∞ Z ∞ Z ∞
P(Y ≥ a) = f0 (x)dx = 1[a,∞) (x)f0 (x)dx = g(x)dx.
a −∞ −∞
It is possible to compute the probability P(Y ≥ a) by Monte Carlo integration where

we simulate Y1 , . . . , Yn being iid with the N (0, 1)-distribution and h(x) = 1[a,∞) (x).
However, the empirical mean becomes
n
1X
1(Yi ≥ a)
n
i=1
and if a is large, we may very well risk that no or only a few of the Y ’s are ≥ a. This
is a central problem in computing small probabilities by Monte Carlo integration.
Even though the absolute error will be small almost by definition (a small probability
is close to zero, so even if we get an empirical mean being 0 with high probability
it is in absolute values close to the true probability) the relative error will be very
large. Using importance sampling, taking
2
1 (x − a)2 1 x + a2 − 2xa
f (x) = √ exp − = √ exp − ,
2π 2 2π 2
which is the density for the normal distribution with mean a, we find that simulating
iid X1 , . . . , Xn with the N (a, 1)-distribution the empirical mean becomes

n n
1 X g(Xi ) 1X f0 (Xi )
= 1(Xi ≥ a)
n f (Xi ) n f (Xi )
i=1 i=1
n
1X a2
= 1(Xi ≥ a) exp 2Xi a − .
n 2
i=1
The previous example, though representing a problem of real interest, does not do
justice to the general applicability of Monte Carlo integration and the impact it
has had due to the rapid development of computer technology. One-dimensional
numerical integration of a known f does not in general pose a real challenge. The
real challenge is to do high-dimensional numerical integration. Using the full version
of Result 4.1.3 we have that
Z
Eh(X) = h(x)f (x)dx
where X is a random variable with values in Rn , whose distribution has density

f : Rn → [0, ∞), and where h : Rn → R such that h(X) has finite expectation. Note
that the integral above represents n successive integrals. If n is large, say in the
hundreds or even thousands, it becomes very difficult to write a successful numerical
integration algorithm without relying on (some form of) Monte Carlo integration.
Monte Carlo integration is used in physics, mathematical finance, structural biology
and many other areas to compute quantities such as probabilities or expectations
of random variables that arise from an awfully complicated underlying setup. In
statistics we have encountered bootstrapping as a simulation based method. When
we use bootstrapping for computing an estimate of the standard error the actual
computation is a Monte Carlo integration. In fact, most computations of a numer-
ical quantity based on simulations can be regarded as a Monte-Carlo integration.
It seems, however, that the most influential application of Monte Carlo integration
in statistics is to Bayesian statistical analysis. A particular flavor of Monte Carlo
integration is used here, which is called Markov Chain Monte Carlo or MCMC for
short. The objective is to compute the so-called posterior distribution, or at least
expectations w.r.t. the posterior distribution, but it is often not possible to simu-
late directly from the posterior distribution. This is a problem that physicists also
had for some of their models in statistical physics. It turns out that much like with
importance sampling, we can simulate in a different way and rely on our ability to
compute certain fractions – and by a miracle both in the problems from physics and
in Bayesian statistics we can actually carry this out in practice. The technique of
Markov Chain Monte Carlo integration has certainly provided the Bayesian statisti-
cians with a tool that is indispensable for a proper practical data analysis according
to Bayesian principles. Prior to MCMC practical Bayesian data analysis was often
obstructed by the difficulties of computing the posterior distribution.
Even with a strictly frequentistic interpretation of probabilities the Bayesian method-
ology for high-dimensional parameter estimation has turned out to be useful. In
Chapter 3 the primary approach to estimation was through the minimization of
the minus-log-likelihood function. The density for the posterior distribution is in
effect the likelihood function multiplied by a penalization factor, and the resulting
minus-log becomes a penalized minus-log-likelihood function.
A Bayesian estimator for the parameter suggests itself. Instead of minimizing the
penalized minus-log-likelihood function we can compute the expectation of the poste-
rior as an estimator of the parameter. In principle we get rid of the difficult practical
problem of minimizing a high-dimensional function and replace it with a much sim-
pler problem of computing an average of some simulated random variables, but there
is a caveat. Problems with local minima of the penalized minus-log-likelihood func-
tion can lead to poor simulations and just as for the minimization problem one has
to pay careful attention to whether the simulation algorithm in reality got caught
in an area of the parameter space that is located around a local minimum.
Exercises
Exercise 4.6.1. Let

1
fλ (x) = x2 λ+ 21
c(λ)(1 + 2λ )
denote the density for the t-distribution, cf. Exercise 2.6.4. Random variables from
the t-distribution can be simulated using rt in R where degrees of freedom (df) equals
2λ. Compute using Monte-Carlo integration the mean
Z
xfλ (x)dx
for λ = 2, 4, 10, 100. Plot the average computed as a function of n. Try also to compute
the mean by Monte-Carlo integration for λ = 21 , 1 – what happens?
Exercise 4.6.2. Consider the following integral
Z
1 − 100
P x2
i
i=1 2 −ρ
P99
i=1 xi xi+1 dx
50
e
R100 (2π)
Compute it using Monte Carlo integration with ρ = 0.1. Provide an estimate for the
variance of the result. Can you do it for ρ = 0.2 also? What about ρ = 0.6?
Hint: You should recognize that this can be seen as an expectation of a function of
d = 100 iid N (0, 1)-distributed random variables. It may also be useful to plot the
running mean as a function of the number of simulations n.
4.7 Asymptotic Theory
Asymptotic theory is an important topic in theoretical statistics and has had a

great influence on how practical statisticians actually carry out their data analysis.
In asymptotic theory we consider how the distribution of transformations of n ran-
dom variables behaves when n becomes large. The phenomena that occurs, which
makes this idea of studying the large n sample behavior so important, is that while
we can rarely tell what the actual distribution of the transformation is, we can often
tell very precisely what the limiting distribution is. The limit is, moreover, largely
independent of many of the fine details in the distribution of the variables consid-
ered. In addition, the limit distributions are often useful approximation even for
moderately large n, and this makes asymptotic theory useful in practice.
First we may recall from Result 4.5.5 (the law of large numbers) that
P(|µ̂n − µ| > ε) → 0
for n → ∞ for all ε > 0. The theorem defines a notion of convergence, which we call
convergence in probability . We write
P
µ̂n −→ µ
if P(|µ̂n − µ| > ε) → 0 for n → ∞ for all ε > 0. The much stronger result that we
will discuss in this section also gives us asymptotically the distribution of µ̂n and is
known as the central limit theorem or CLT for short.
Result 4.7.1 (CLT). If X1 , . . . , Xn are n iid real valued random variables with finite
variance σ 2 and expectation µ then for all x ∈ R
√ Z x 2
n(µ̂n − µ) 1 x
P ≤ x → Φ(x) = √ exp − dx
σ 2π −∞ 2
for n → ∞. We write
1
µ, σ 2
as
µ̂n ∼ N
n
and say that µ̂n asymptotically follows a normal distribution.
First note that what we are considering here is the√distribution function of the nor-
malization of the random variable µ̂n . We say that n(µ̂σn −µ) converges in distribution
to the standard normal distribution.
Example 4.7.2. Consider as in Example 4.1.10 n iid random variables, X1 , . . . , Xn ,

with values in E = {A, C, G, T}. Then
n
1X
p̂n (A) = 1(Xi = A)
n
i=1
Asymptotic Theory 251
is the relative frequency of A-occurrences – the empirical mean of the iid Bernoulli
variables 1(X1 = A), . . . , 1(Xn = A). Since
E1(X1 = A) = P(X1 = A) = p(A) and V1(X1 = A) = p(A)(1 − p(A))
we have from Result 4.7.1 that

as 1
p̂n (A) ∼ N p(A), p(A)(1 − p(A)) .
n
Likewise, the same result for p̂n (C), p̂n (G) and p̂n (T) holds. ⋄
Monte-Carlo integration is one situation where the use of the central limit theorem
is almost certainly justified because we want to run so many simulations that the
empirical average is very close to the theoretical average. This will typically ensure
that n is so large that the distribution of the average is extremely well approximated
by the normal distribution. It means that if X1 , . . . , Xn are iid having density f and
we consider the empirical average
n Z
1X
µ̂n = h(Xi ) ≃ µ = h(x)f (x)dx
n
i=1
as
then µ̂n ∼ N µ, n1 σ 2 where
Z
σ 2 = Vh(X1 ) = (h(x) − µ)2 f (x)dx.
We should regard the Monte-Carlo integration as a computer experiment for estima-

tion of the unknown parameter µ. As such, the estimator µ̂n has an approximating
normal distribution, which we can use to give a confidence interval for the parameter,
and the standard (1 − α)-confidence interval is

σ σ
µ̂n − zα √ , µ̂n + zα √
n n
where zα is the (1 − α/2)-quantile for the normal distribution. If we cannot compute
the mean µ analytically we can probably not compute the variance σ 2 analytically
either, hence we need to estimate the standard error using the estimator
n
1 X
σ̂n2 = (Xi − µ̂n )2 ,
n−1
i=1
and we plug this estimate into the formula for the confidence interval above.
Example 4.7.3. If we return to the Monte-Carlo integration of the mean for the
Gumbel distribution as considered in Example 4.6.1 we find that the estimated mean
and standard deviation is
µ̂10000 = 0.562 σ̂10000 = 1.28

0.8
0.7
Empirical mean
0.6
0.5
0.4
0 2000 4000 6000 8000 10000
Figure 4.2: The average for n simulated Gumbel variables as a function of n including
a 95%-confidence band based on the asymptotic normal distribution of the average
for large n.
Here we base the estimates on 10000 simulations. Figure 4.2 shows the same plot
of the average as a function of n as Figure 4.1 but this time with 95%-confidence
bands. The resulting confidence interval is [0.537, 0.587]. ⋄
Example 4.7.4. As in the Example above, but this time considering an m-length
word w as in Example 4.1.10, the relative frequency of w is
n−m+1
1 1 X
p̂w = N= 1(Xi . . . Xi+m−1 = w).
n n
i=1
Note that Ep̂w = pw n−m+1

n where pw is the word probability, see Example 4.1.10,
and one may choose to divide by n − m + 1 instead of n to get expectation pw . For
m ≪ n and especially for n → ∞ this does not really make a difference. Since p̂w
is the empirical mean of dependent random variables, Result 4.7.1 does not apply
as it stands. There are versions of the central limit theorem that can cope with
dependent variables, and the result is that indeed, p̂w also asymptotically has a
normal distribution. In this case
m−1
!
as 1 X
p̂w ∼ N pw , pw (1 − pw ) + 2 ρ(k)
n
k=1
where ρ(k) is defined in Example 4.4.6. If w is not self-overlapping we know that

ρ(k) = −p2w for k = 1, . . . , m − 1, hence for non self-overlapping words

as 1
p̂w ∼ N pw , pw (1 − (2m − 1)pw )
n
since
m−1
X
pw (1 − pw ) + 2 ρ(k) = pw (1 − pw ) − 2(m + 1)p2w
k=1
= pw (1 − pw − 2(m − 1)pw ) = pw (1 − (2m − 1)pw ).
Perhaps a little surprisingly, this asymptotic variance, pw (1−(2m−1)pw ), is actually
always positive. It must be, and a formal derivation of the asymptotic normality of
p̂w will actually prove this. It is a consequence of the non self-overlapping assump-
tion that puts some restrictions on the word probability pw – no matter what the
probabilities for the single letters are! Positivity of pw (1 − (2m − 1)pw ) is equivalent
to
1
pw < ,
2m − 1
which therefore gives us a direct upper bound on the probability of a non self-
overlapping word of length m. ⋄

Result 4.7.5 (The ∆-method). If Zn ∼ N µ, n1 σ 2 and if h : R → R is differen-
as
tiable in µ then
as 1 ′ 2 2
h(Zn ) ∼ N h(µ), h (µ) σ .
n
The ∆-method provides first of all a means for computing approximately the variance
of a non-linear transformation of a real valued random variable – provided that
the transformation is differentiable and that the random variable is asymptotically
normally distributed. If Zn is a real valued random variable with density fn , if
h : R → R, if h(Z) has finite variance, and if µh = Eh(Zn ) we know that
Z ∞
V(h(Zn )) = (h(x) − µh )2 fn (x)dx.
−∞
If Zn also has finite variance and VZn = σ 2 /n, this variance is small for large n and
Zn does not fluctuate very much around its mean µ = EZn . Therefore we can make
the approximation µh ≃ h(µ). Moreover, if h is differentiable we can make a first
order Taylor expansion of h around µ to get
h(x) ≃ h(µ) + h′ (µ)(x − µ),
from which we find
Z ∞ Z ∞
Vh(Zn ) = (h(x) − µh )2 fn (x)dx ≃ (h(µ) + h′ (µ)(x − µ) − h(µ))2 fn (x)dx
−∞ −∞
Z ∞
1
= h′ (µ)2 (x − µ)2 fn (x)dx = h′ (µ)2 VZn = h′ (µ)2 σ 2 .
−∞ n
Math Box 4.7.1 (Multivariate CLT). The notion of convergence in distribution

can be generalized to several dimensions. If Zn ∈ Rd we say that

as 1
Zn ∼ N ξ, Σ
n
where ξ ∈ Rd and Σ is a positive definite d × d matrix if for all v ∈ Rd we have

as 1
v t Zn ∼ N v t ξ, v t Σv .
n
If X1 , . . . , Xn are iid real valued random variables with finite fourth order moment,
and if the X-variables have mean µ and variance σ 2 then

µ̂n as µ 1
∼N , Σ
σ̂n σ n
so the bivariate random variable consisting of the empirical mean and variance con-
verges in distribution to a bivariate normal distribution. The asymptotic covariance
matrix is µ3
2 1 2
Σ=σ µ3 µ4 −1
2 4
where µ3 and µ4 are the third and fourth moments of the normalized random
variable
X1 − µ
Y = .
σ
Thus Y has mean 0 and variance 1 and
µ3 = EY 3 and µ4 = EY 4 .
Note the the values of µ3 and µ4 do not depend upon µ or σ 2 . If the distribution
of the X-variables is N (µ, σ 2 ) it holds that µ3 = 0 and µ4 = 3.
Computing the true variance analytically is in most cases impossible, but differenti-
ating h and computing the approximation is most likely doable. But the ∆-method
actually provides much more information than just a variance approximation. It
also tells that the differentiable transformation preserves the asymptotic normality,
so that you actually have a way to approximate the entire distribution of h(Zn ) if
Zn asymptotically has a normal distribution.
Example 4.7.6. Continuing Examples 4.7.2 and 4.7.4 we consider the function
h : (0, 1) → (0, ∞) given by
x
h(x) = ,
1−x
then for any event, B, we see that h(P (B)) = ξ(B) is the odds for that event. We
Math Box 4.7.2 (Multivariate ∆-method). If h : Rd → Rk is differentiable in ξ

with (d × k matrix of) derivatives
 ∂h1 ∂hk 
∂x1 (ξ) ... ∂x1 (ξ)
 .. .. .. 
Dh(ξ) =  . . . 
∂h1 ∂hk
∂xd (ξ) ... ∂xd (ξ)
as
and if Zn ∼ N ξ, n1 Σ then

as 1
h(Zn ) ∼ N f (ξ), Dh(ξ)t ΣDh(ξ) .
n
The function h needs only to be defined in a neighborhood of ξ for this result to

be true.
As an example take h : R × (0, ∞) → R to be
y
h(x, y) = ,
x
Then
−y
Dh(x, y) = x2 .
1
x
We see that h(µ, σ) is the coefficient of variation, and using the results in Math
Box 4.7.1 and the multivariate ∆-method above we find that

dn = σ̂n ∼ as σ 1 σ4 σ 3 µ3 σ 2 (µ4 − 1)
CV N , − +
µ̂n µ n µ4 µ3 4µ2

1 µ4 − 1
= N CV, CV4 − CV3 µ3 + CV2
n 4
since
µ3 −σ
−σ 1 1
Dh(µ, σ)t Σ Dh(µ, σ) = σ2 µ2 µ µ3
2
µ4 −1
µ2
1
2 4 µ
σ4 σ 3 µ3 σ 2 (µ4 − 1)
= − +
µ4 µ3 4µ2
find the derivative to be
(1 − x) + x 1
h′ (x) = 2
= ,
(1 − x) (1 − x)2
If we consider the iid Bernoulli random variables 1(X1 = A), . . . , 1(Xn = A) with
p(A) ∈ (0, 1) we find, using Example 4.7.2 and the ∆-method in Result 4.7.5, that
the empirical odds for the nucleotide A, ξ̂(A) = p̂(A)/(1 − p̂(A)), is asymptotically
normal,

p̂(A) as p(A) 1 p(A) 1
ξ̂(A) = ∼N , =N ξ(A), ξ(A) .
1 − p̂(A) 1 − p(A) n 1 − p(A) n
If w is a length m word that is not self overlapping and if we consider the (dependent)
Bernoulli random variables 1(X1 . . . Xm = w), . . . , 1(Xn−m+1 . . . Xm = w) we find
instead, using Example 4.7.4 and the ∆-method, that the empirical odds for the
word w, ξ̂(w) = p̂w /(1 − p̂w ), is asymptotically normal,

ˆ p̂w as pw 1 pw (1 − (2m − 1)pw )
ξ(w) = ∼N , .
1 − p̂w 1 − pw n (1 − pw )2
⋄
Exercises
Exercise 4.7.1. Consider the Monte-Carlo integration from Exercise 4.6.1 for the
computation of the mean in the t-distribution. Continue this exercise by computing
estimates for the variance and plot the estimates as a function of n. Try the range
of λ’s 12 , 1, 2, 10. What happens and how can you interpret the result? Choose n and
compute 95%-confidence intervals for the value of the mean for different choices of λ.
4.7.1 MLE and Asymptotic Theory
It is the exception, not the rule, that we analytically can find the distribution of an
estimator. On the contrary, it is often the case that estimators, like most maximum
likelihood estimators, do not even have an explicit analytic expression but are given
as solutions to equations or maximizers of a function. An alternative to knowing the
actual distribution of an estimator is to know a good and useful approximation.
For an estimator, θ̂, of a one-dimensional real parameter θ, we may be able to
compute the expectation, ξ(θ) = Eθ θ̂, and the variance, σ 2 (θ) = Vθ θ̂. A possi-
ble approximation of the distribution, which turns out to be quite well founded, is
N (ξ(θ), σ 2 (θ)) – the normal distribution with mean ξ(θ) and variance σ 2 (θ). We
present this as the following pseudo definition.
Definition 4.7.7. We say that an estimator θ̂ of a one-dimensional parameter θ

approximately follows a normal distribution with parameters ξ(θ) and σ 2 (θ) if
!
θ̂ − ξ(θ)
P ≤ x ≃ Φ(x)
σ(θ)
for x ∈ R. We write
N (ξ(θ), σ 2 (θ))
approx
θ̂ ∼
It is a pseudo definition because we do not try to quantify what we mean by ≃

above. The idea of this section is that it is possible to make mathematical sense
out of the approximation, but it is a technical business just to formulate the re-
sults with rigorous mathematical assumptions let alone to give correct and complete
derivations.
Quite a lot of estimators follow approximately a normal distribution. Next to the
actual quantification of the quality of the approximation, the real obstacle is to com-
pute the approximating parameters. In reality it is the approximating variance that
presents a problem, as most reasonable estimators follow approximately a normal
distribution with ξ(θ) = θ, in which case we say that the estimator is approximately
unbiased – the estimator centers approximately around the true value of θ.
Result 4.7.8. Consider a statistical model (Pθ )θ∈Θ with Θ ⊆ R a nice parameter
set, and with a minus-log-likelihood function lX , which is twice differentiable, then
in a wide range of cases the MLE approximately follows a normal distribution with
ξ(θ) = θ and σ 2 (θ) = i(θ)−1 where
2
d lX
i(θ) = Eθ (θ) ∈ (0, ∞).
dθ 2
That is
N (θ, i(θ)−1 ).
approx
θ̂ ∼
We use the subscript X instead of x when writing the minus-log-likelihood function

lX and its derivatives in θ to emphasize that in this section we will consider them,
for fixed θ, as transformations of X and will not be so interested in them as functions
of θ for a given observation x.
Remark 4.7.9. To somehow justify the theorem we make the following informal
computations. A first order Taylor expansion of the derivative of the minus-log-
likelihood function around θ results in
dlX dlX dlX d2 lX
(θ̂) = (θ + (θ̂ − θ)) = (θ) + (θ)(θ̂ − θ) + residual term.
dθ dθ dθ dθ 2
Since θ̂ is a maximizer of the likelihood function (the minimizer of the minus-log-
likelihood) the derivative is zero (if θ̂ is an interior point in Θ, i.e. not a point on the
boundary). Moreover, ignoring the residual term we then rearrange the equation to
yield
2 −1
d lX dlX
θ̂ ≃ θ − 2
(θ) (θ).
dθ dθ
One major part of the technical difficulties in a formal mathematical derivation has
to do with justifying this approximation. Another major point, which has much more
to do with the statistical model considered and the probabilistic theory, is to verify
that the following two results hold:
d2 lX dlX approx
(θ) ≃ i(θ) and (θ) ∼ N (0, i(θ)).
dθ 2 dθ
If we freely can interchange

R ∞ integration w.r.t. x and differentiation w.r.t. θ in the
following, we find, since −∞ fθ (x)dx = 1, that by differentiation w.r.t. θ
Z
dfθ (x)
0 = dx
dθ
Z
d log fθ (x)
= fθ (x)dx
dθ

dlX (θ)
= −Eθ .
dθ
So from the fact that the integral of the density is 1 we obtain the identity that the
expectation of the minus-log-likelihood function is 0. A second differentiation yields
Z
d2 log fθ (x) d log fθ (x) 2
0 = fθ (x) + fθ (x)dx
dθ 2 dθ
2
d log fθ (x) dlX (θ)
= Eθ + Vθ .
dθ 2 dθ
Thus
d2 log fθ (x) dlX (θ)
i(θ) = −Eθ = Vθ .
dθ 2 dθ
These computations show that the parameters ξ(θ) = θ and σ 2 (θ) = i(θ) are correct
for the approximating normal distribution above. They also show that i(θ) ≥ 0,
since the number equals a variance. We show below for a specialized model setup
with iid observations that one can obtain the two approximations above from the
law of large numbers and the central limit theorem respectively.
In a final step we argue that since
θ̂ ≃ θ − i(θ)−1 Z
with the right hand side a scale-location transformation of the random variable
approx
Z ∼ N (0, i(θ)) then θ̂ ∼ N (θ, i(θ)−1 ).
Since the inverse of i(θ) appears as a variance in the approximating normal distribu-
tion, we can see that i(θ) quantifies how precisely θ is estimated using θ̂. The larger
i(θ) is the more precise is the estimator. This justifies the following definition.
Definition 4.7.10. The quantity i(θ) is called the Fisher information, the expected
information or just the information – after the English statistician R. A. Fisher.
The quantity
d2 lX
(θ)
dθ 2
is called the observed information.
The computations in Remark 4.7.9 suffered from two deficiencies. One was the abil-
ity to formalize a number of analytic approximations, which in fact boils down to
control of the error in the Taylor expansion used and thus detailed knowledge of the
“niceness” of the likelihood function. The other claim was that
d2 lX dlX approx
(θ) ≃ i(θ) and (θ) ∼ N (0, i(θ)).
dθ 2 dθ
This claim is not completely innocent, and does not always hold. But it will hold in
situations where we have sufficient replications – either in an iid setup or a regres-
sion setup. To show why, lets consider the situation where X = (X1 , . . . , Xn ) and
X1 , . . . , Xn are iid such that the minus-log-likelihood function is
n
X
lX (θ) = − log fθ (Xi )
i=1
then i(θ) = ni0 (θ) where

d2 log fθ (X1 ) d log fθ (X1 )
i0 (θ) = −Eθ (θ) = Vθ (θ) .
dθ 2 dθ
The second derivative of the minus-log-likelihood is a sum of iid random variables

2 2
− d logdθfθ2 (X1 ) , . . . , − d logdθfθ2(Xn ) , and the law of large numbers, Result 4.5.5, gives that
n
1 d2 lX 1 X d2 log fθ (Xi ) P d2 log fθ (X1 )
2
(θ) = − → −Eθ (θ) = i0 (θ).
n dθ n dθ 2 dθ 2
i=1
Likewise the first derivative of the minus-log-likelihood is a sum of iid random vari-
ables − d log dθ
fθ (X1 )
, . . . , − d log dθ
fθ (Xn )
, whose mean is 0 and whose variance is i0 (θ), and
the central limit theorem, Result 4.7.1, gives that
n
1 dlX 1 X d log fθ (Xi ) as 1
(θ) = − ∼ N 0, i0 (θ) .
n dθ n dθ n
i=1
Result 4.7.11. In the setup above, if i0 (θ) < ∞ it holds that
1 d2 lX P
2
(θ) → i0 (θ)
n dθ
for n → ∞ and
1 dlX as 1
(θ) ∼ N 0, i0 (θ) .
n dθ n
Remark 4.7.12. If θ̂n denotes the MLE in the previous theorem the conclusion is
that
as 1
θ̂n ∼ N θ, .
ni0 (θ)
Math Box 4.7.3 (Multivariate information). If Θ ⊆ Rd and if the minus-log-

likelihood function as a function of d variables is twice differentiable, the second
derivative is the matrix we denote D2 lX (θ), cf. Math Box 3.3.1.
The Fisher information is
I(θ) = Eθ (D2 lX (θ)).
Like in the one-dimensional setup a similar interchange of integration and differen-
tiation shows that the Fisher information is also the covariance matrix of ∇lX (θ).
If θ̂ denotes the MLE the result then reads that
approx
θ̂ ∼ N 0, I(θ)−1
where I(θ)−1 is the matrix-inverse of I(θ).

An important consequence is that if v ∈ Rd then v t θ̂ is a one-dimensional parameter
transformation (take v = (1, 0, . . . , 0) to select the first coordinate of θ̂, say) and
approx
v t θ̂ ∼ N 0, v t I(θ)−1 v .
The i’th coordinate in θ̂ thus follows approximately a normal distribution with

variance (I(θ)−1 )ii . This number differs from (I(θ)ii )−1 . The latter is in fact the
asymptotic variance parameter if we estimate θi using MLE while keeping the other
parameters fixed, since the Fisher information in that case is i(θi ) = I(θ)ii . The
point is that we need the entire Fisher information matrix I(θ) to compute the
asymptotic variance parameter for each coordinate θ̂i .
To use this distribution in practice we can use the plug-in estimate i0 (ϑ̂n ) as a
substitute for the unknown i0 (θ). This requires that we know a formula for i0 (θ). As
an alternative one can take the observed information evaluated in the MLE
1 d2 lX
(ϑ̂n )
n dθ 2
as an approximation to i0 (θ).
Clearly considering a one-dimensional parameter setup only is not sufficient for most
practical applications. The multivariate version of the Fisher information is a matrix
and the relevant approximations are described in Math Box 4.7.3.
4.8 Entropy
We consider in this section mostly probability measures or random variables on a

discrete sample space E. The definitions and results presented can all be formulated
for continuous distributions given in terms of a density instead of in terms of the
point probabilities as used here.
Entropy 261
Definition 4.8.1 (Entropy). For a probability measure P on E with point probabil-

ities (p(x))x∈E we define the entropy of P as
X
H(P ) = − p(x) log p(x).
x∈E
If X is a random variable with distribution P the entropy of X is defined as the

entropy of P , i.e.
H(X) = H(P ).
We use the convention that 0 log 0 = 0 in the definition. One may also note from
the definition that log p(x) ≤ 0, hence H(P ) ≥ 0, but the sum can be divergent if
E is infinite in which case H(P ) = ∞. We may note that − log p(X) is a positive
random variable and that
H(X) = E(− log p(X)).
The interpretation of H(X) is as a measure of the uncertainty about the outcome

of the random variable X. This is best illustrated by a couple of examples.
0.7
0.6
0.5
0.4
entropy
0.3
0.2
0.1
0.0 0.2 0.4 0.6 0.8 1.0
Figure 4.3: The entropy of a Bernoulli variable as a function of the success probability
p.
Example 4.8.2 (Bernoulli variables). If E = {0, 1} and X is a Bernoulli variable

with P(X = 1) = p then
H(X) = −p log p − (1 − p) log(1 − p).

As a function of the probability p, H(X) takes the value 0 for p = 0 or p = 1. If we

differentiate w.r.t. p for p ∈ (0, 1) we find that
dH(X) 1−p
= − log p − 1 + log(1 − p) + 1 = log(1 − p) − log p = log .
dp p
The derivative is > 0 for p ∈ (0, 1/2), = 0 for p = 1/2, and < 0 for p ∈ (1/2, 1).
Thus H(X) is, as a function of p, monotonely increasing in the interval from 0 to
1/2 where it reaches its maximum. From 1/2 to 1 it decreases monotonely again.
This fits very well with the interpretation of H(X) as a measure of uncertainty. If p
is close to 0, X will quite certainly take the value 0, and likewise if p is close to 1,
X will quite certainly take the value 1. If p = 1/2 we have the greatest trouble to
tell what value X will take, as the two values are equally probable. ⋄
If E is a finite sample space and X takes values in E it holds that H(X) = 0 if

and only if P(X = x) = 1 for some x ∈ E. Moreover, if |E| = n is the number of
elements in E then H(X) is maximal if and only if X has the uniform distribution,
i.e. P (X = x) = 1/n for all x ∈ E. These two cases represent the extreme cases with
minimal and maximal uncertainty, respectively.
5
4
entropy
3
2
1
0
0.0 0.2 0.4 0.6 0.8 1.0
Figure 4.4: The entropy of a geometrically distributed random variable as a function

of the success probability p.
Example 4.8.3 (Geometric distribution). If E = N0 and X follows a geometric

Entropy 263
distribution with success probability p, then

∞
X
H(X) = − p(1 − p)n log(p(1 − p)n )
n=0
∞
X ∞
X
= − log p p(1 − p)n − log(1 − p) np(1 − p)n
n=0 n=0
(1 − p)
= − log p − log(1 − p)
p
p log p + (1 − p) log(1 − p)
= − .
p
For p ∈ (0, 1) this function decreases monotonely, which concurs with the interpre-
tation of the entropy. The closer p is to 1, the more certain we are that X will take a
small value close to or equal to 0. For small values of p the distribution of X becomes
much more spread out, and we are more uncertain what the value of X will be. ⋄
Definition 4.8.4 (Conditional entropy). If X and Y are two random variables
taking values in E1 and E2 respectively we define
X
H(X | Y = y) = − p(x|y) log p(x|y)
x∈E1
with (p(x|y))x∈E1 being the point probabilities for the conditional distribution of X
given Y = y, and we define the conditional entropy of X given Y = y to be
X
H(X | Y ) = p(y)H(X|Y = y)
y∈E2
where (p(y))y∈E2 are the point probabilities for the marginal distribution of Y .
The conditional entropy tells us about the average uncertainty about X given that
we know the value of Y . The gain in information about X, that is, the loss in
uncertainty as measured by entropy, that we get by observing Y tells us something
about how strong the dependency is between the variables X and Y . This leads to
the following definition.
Definition 4.8.5 (Mutual information). Let X and Y be two random variables
taking values in E1 and E2 respectively. The mutual information is defined as
I(X, Y ) = H(X) − H(X|Y ).
A value of I(X, Y ) close to 0 tells us that the variables are close to being independent
– not much knowledge about X is gained by observing Y . A value of I(X, Y ) close to
H(X) tells that the variables are strongly dependent. We therefore regard I(X, Y )
as a quantification of the dependence between X and Y .
Another way to view mutual information is through the relative entropy measure
between two probability measures.
Definition 4.8.6 (Relative Entropy). If P and Q are two probability measures on

E with point probabilities (p(x))x∈E and (q(x))x∈E respectively then provided that
q(x) = 0 implies p(x) = 0, we define the relative entropy of P w.r.t. Q as
X p(x)
D(P | Q) = p(x) log .
q(x)
x∈E
If q(x) = 0 for some x ∈ E with p(x) > 0 then D(P | Q) = ∞. If X and Y are two
random variables with distribution P and Q respectively we define
D(X | Y ) = D(P | Q)
as the relative entropy from X to Y .
From the definition it follows that

X p(x)
D(X | Y ) = p(x) log
q(x)
x∈E
X X
= p(x) log p(x) − p(x) log q(x)
x∈E x∈E
X
= −H(X) − p(x) log q(x),
x∈E
or alternatively that
X
H(X) + D(X | Y ) = − p(x) log q(x). (4.32)
x∈E
Note that the relative entropy is non-symmetric in X and Y .

The concepts of mutual information and relative entropy are related. Let r(x, y)
denote the point probabilities for the probability measure R, then the point proba-
bilities for P and Q are
X X
p(x) = r(x, y) and q(y) = r(x, y)
y x
respectively. Writing out how D(R | P ⊗ Q) is defined we find that

X r(x, y)
D(R | P ⊗ Q) = r(x, y) log
x,y
p(x)q(y)
X X r(x, y)
= − r(x, y) log p(x) + r(x, y) log
x,y x,y
q(y)
!
X X X
= − r(x, y) log p(x) + r(x|y)q(y) log r(x|y)
x y x,y
X X X
= − p(x) log p(x) + q(y) r(x|y) log r(x|y)
x y x
= H(X) − H(X|Y ) = I(X, Y ).
Entropy 265
Thus we have the result:
Result 4.8.7. If X and Y are two random variables with joint distribution R, if X
has (marginal) distribution P and Y has (marginal) distribution Q then
I(X, Y ) = D(R | P ⊗ Q).
We may observe that

p(X)
D(P |Q) = E log
q(X)
is the expectation of the log-likelihood ratio log p(X)
q(X) . It can be shown that D(P |Q) ≥
0 with equality if and only if P = Q.
If Qθ for θ ∈ Θ denotes a parametrized statistical model and if εn is the empirical
probability measured based on x1 , . . . , xn we find that
n
1X
D(εn |Qθ ) = −H(εn ) − log qθ (xi ) = −H(εn ) + lx (θ)
n
i=1
where lx (θ) is the minus-log-likelihood function. Thus minimizing the minus-log-

likelihood function is equivalent to minimizing the relative entropy of the empirical
measure εn w.r.t. Qθ over θ. The relative entropy D(P |Qθ ) is often regarded as a
(asymmetric) distance measure from P to Qθ . The maximum likelihood estimator
can thus be seen as the estimator that takes the probability measure Qθ̂ in the model
that is closest to the empirical measure εn as measured by the relative entropy.
A
The program R is “GNU S”, a freely available environment for statistical computing
and graphics that runs on a variety of platforms including Mac OS X, Linux and
Windows. It is an implementation of the S language developed by John Chambers
and colleagues at the Bell Laboratories.
R consists of the base program with a number of standard packages and a large
and ever growing set of additional packages. The base program offers a Command
Line Interface (CLI), where you can interactively type in commands (R expressions).
One will (or should) quickly learn that the proper use of R is as a high level pro-
gramming language for writing R-scripts, R-functions and R-programs. One may
eventually want to extend the functionality of R by writing an entire R package
and/or implement various time-consuming algorithms in a lower level language like
C with an R-interface. Indeed, many of the base functions are implemented in C or
Fortran. Such advanced use of R is far beyond the scope of this appendix.
This appendix deals with a few fundamental questions that inevitably arise early
on when one wants to use R. Questions like how to obtain R, how to run R, what
is R all about, how to handle graphics, how to load packages and data, how to run
scripts, and similar problems. We also give directions for locating more information
on using and running R. The appendix can not stand alone, and you will for instance
need the manual An Introduction to R – see Section A.2.
A.1 Obtaining and running R
The R program and all additional packages are available for download at the Compre-
hensive R Archive Network (CRAN) at http://cran.r-project.org/. The Dan-
ish mirror is http://mirrors.dotsrc.org/cran/. You can download binaries for
Linux, Windows, and Mac OS X, or you can download the source code and compile
267
268 R
it yourself if you want. R is available as Free Software under the terms of the Free
Software Foundation’s GNU General Public License.
You can also download the packages from CRAN, but once you have installed R, it
is quite easy to download and install packages from within R.
When R is properly installed1 on your computer, you can run it. How you do that
and the appearance of R differ a little from Linux to Windows. On a Linux machine,
you simply type R in a terminal, the program starts and you are now running R.
Whatever you type in next is interpreted by R. Graphics will appear in separate
windows. If you start R in Windows (by locating it from e.g. the Start Menu), R
runs in a new window called the RGui, with a window inside called the R console
containing a command prompt. Don’t be fooled – the fact that it says RGui doesn’t
mean that the Windows version provides a graphical user interface for using R, just
that a few things like running R-scripts, loading and installing packages, and saving
graphics can be done from the menu. It is recommended that you learn how to do
that from the command line anyway.
R runs in a working directory. All interaction with the file system like reading or
writing data files, running scripts or saving graphics will take place relative to the
working directory unless you specify a complete alternative path instead of just a file-
name. You get the current working directory by the command getwd(). You change
the working directory to be path by setwd("path"). In the RGui on Windows, this
can be done from the File menu as well.
You quit R by typing quit() or simply q(). On a Windows machine you can also
close the program in the File menu (Exit). When you quit, you are always asked
if you want to save the workspace. Doing so, all objects and the command history
are stored in the working directory. When starting R it automatically loads the
workspace most recently stored in the default working directory.
A.2 Manuals, FAQs and online help
In general, we refer to the manuals that come with R for detailed information on
R and how to use R. The manuals in PDF-format are located in the subdirec-
tory doc/manual. For the Windows version you can also find them through the
Help menu, and you can always find the most recent version on the R homepage:
http://www.r-project.org/. The most important manual for ordinary use of R is
An Introduction to R.
You can also access the manuals in HTML-format by issuing the help.start()
command in R. The HTML-page that opens contains links to the manuals, links
to help pages grouped according to the package they belong to, as well as a search
1
How one installs the program can be machine and platform dependent. Installation of the
Windows binary should be completely straight forward.
The R language, functions and scripts 269
engine for searching the help pages. Links to some additional material like FAQs are
also given. When running R on Windows you can find the HTML-help page in the
Help menu together with direct links to FAQs, the search engine, etc.
You can access the help pages from within R by the command help. For instance,
help(plot) will give you the help page for the plot function. A shortcut is ?plot.
You can also search for help pages containing the word“plot”by help.search("plot").
Note the quotation marks when using help.search. For a few functions like the
binary operator plus (+), you will need quotation marks when you call the help
function, i.e. help("+").
A.3 The R language, functions and scripts
You may find this section a little difficult, if you are not that familiar with computer
programming and programming languages. However, this may explain a little about
why things are the way they are.
A.3.1 Functions, expression evaluation, and objects
Formally the R language belongs to the family of functional programming languages.

Every command issued is a function call, or expression evaluation, with or without
arguments, and the result of the evaluation is returned. The functions work on
objects. The objects store the data that the functions work on, and there are a
number of data structures for doing this. The objects provide interfaces between the
computer memory, which can not be accessed directly, and the functions. It depends
on the data structure how we access the data stored in the object. For instance,
an object x of type integer is a vector containing integers, i.e. the data structure is
an integer vector. The elements in the vector can be accessed by subscripting, e.g.
x[10] is the 10’th integer in the vector x. Another object y of type list is a list with
each element in the list being some data structure. The first element in the list is
accessed by y[[1]], which could be an integer vector – or even another list. You
can list all existing objects by objects() or alternatively ls().
A thing to remember about the language is that any command typed in is actually
a function evaluation. The function takes some (maybe zero) arguments, and when
evaluated it returns something. Syntactically this means that whenever you want to
execute a command with no arguments, for instance the help.start() as discussed
above, you will still need to provide the parentheses (). If you just type help.start
the R code for the function is printed. Some functions are evaluated because they
have side effects like producing a graph or starting up the HTML-help pages, in
which case the function returns NULL.
R and the standard packages that ships with R provide a large number of commands
– or functions. Many more than a low level language like C. This means that you can
270 R
get R to do rather complicated things by evaluating a few functions, and it is often

much more efficient to use already existing functions than to write you own R code
based on more elementary R functions. The downside is that it can be difficult for
a new user to figure out what functions are available. That is, there is a quite large
body of knowledge that one needs to obtain. The present set of notes introduces
many functions in an appropriate context.
R is an object oriented programming language. This implies that certain functions
are generic meaning that the behaviour of the function depends on the object you
give as argument. A generic function calls the function associated with the data
structure of the object you give as argument. A good example is the function plot.
A large number of objects have their own way to be plotted by default, and the
generic function plot simply calls the correct plotting function for the object. If x
is a numeric vector then plot(x) plots the values in x against the index. If x is the
fitted model returned by the lm function (linear model), then plot(x) will plot the
residuals.
One consequence of the object oriented line of thought is that when you fit a model
to data the result is an object that (hopefully) stores all relevant information. You
don’t want to print all the information on screen. Instead, you can subsequently
“interrogate” the fitted model object by a number of generic functions. The inter-
rogation can in principle proceed the same way no matter what kind of model we
fit, as the resulting fitted model object is able to provide the information we ask for
by the generic function. You can then extract and format the results in a way that
suits you.
A.3.2 Writing functions and scripts
In principle, we can use the functions provided by R and the packages to perform
the computations we want or to produce the graphics we want by typing in function
calls one after the other. If we want, we can even define an entirely new function and
then use it – doing everything from the command line interface. Defining a function,
sq, that squares its argument can be done as follows
> sq <- function(x) x^2
One will, however, very quickly find it to be tedious, boring and inconvenient to
type in one function call after the other and in particular to define functions using
the command line editor – even though the command line editor keeps a history of
your commands.
To avoid working with the command line interface you can use R together with any
text editor for writing a file containing R function calls, which can then be loaded
into R for sequential evaluation. You use source("foo.r") to load the file foo.r.
Graphics 271
Note that you may need to include either the full path or the relative path (relative
to the working directory, cf. Section A.1) if foo.r is not in the working directory.
The usage of source ranges from writing simple scripts that basically collect a
number of function calls over implementations of new functions to entire R programs
that perform a number of different tasks. It is good practice to get used to working
with R this way, i.e. to write R-scripts – then you can always experiment by copy-
pasting to the command line editor, if you are not sure that the whole script will
run, or that you only need to run some parts of the script.
You can in principle use any text editor. There is a quite extensive environment called
ESS (Emacs Speaks Statistics) for the family of Emacs editors. There is more infor-
mation on the homepage http://www.sciviews.org/_rgui/projects/Editors.html,
which gives links to a number of editors that support R script editing with features
such a syntax highlighting and indentation. The RGui for Windows provides in the
File menu its own simple editor for editing scripts, and you can also execute the
source command from the File menu.
A.4 Graphics
R provides a versatile and customisable environment for producing graphics. You

can produce graphics on the screen, print it, and/or save it in a number of formats.
You can do this interactively or from within a script.
In R you define the graphics you want, and a device driver is used to actually produce
the graphics of the appropriate format on the appropriate device. Often you want
the graphics to be displayed on the screen in a window in which case the screen
device (the default device) is needed. A new screen device can be opened by the
dev.new(), command. The functions can take a number of parameters specifying the
size, position, etc. of the window. There are other device drivers like the postscript
and pdf device driver for producing postscript or pdf files, respectively.
If you just use a high-level plotting command like plot, you will not have to worry
about devices in the first place. Just issue the plot command appropriately and you
will see the result plotted in a window. However, the moment that you want plot
in multiple windows, print out the graphics, or save the graphics, you will need to
handle devices. Each time you issue the dev.new() command, you will open a new
device, and the most recently opened will be the active device to which all subsequent
graphics commands are directed. You can get a list with names and numbers of the
open devices by dev.list(), you can get the current active device by dev.cur(),
and you can set the active device to be device number n by dev.set(n). Finally,
you can close down all devices by graphics.off().
Having generated a nice piece of graphics on the screen you want to save or print
it. This is done by copying the content of the active device to a new device, e.g. a
272 R
postscript device. The command dev.copy(device = postscript, file = "my-

graph.ps") will copy the content of the currently active device to the file my-
graph.ps in postscript format. You can also produce e.g. pdf, jpeg, bmp, and bitmap
formats by copying to the device with the corresponding name. Use help(Devices)
for more information. For printing, use dev.print() to print the currently active
device.
Saving graphics can, like executing scripts and changing the working directory, be
done from the File menu when running the RGui on Windows. The R Graphics
window, whose content you want to save, needs to be the active window. Then
simply choose Save in the File menu and choose the appropriate format.
When you use a typical high-level plotting command like plot, you will often provide
labels for the axis and a title for the plot. Labels are given by specifying the additional
parameters xlab and ylab to the plot command and the title by specifying the main
parameter or subsequently by the title command.
Every time you call a high-level plotting function the content of the current device
will be erased. If you want to plot several graphs in the same window, you can do
this by calling points or lines instead of plot. Some high level plotting functions
can also take a parameter, add, which by default equals FALSE. If you set add=TRUE
the graphics will be added to the existing plot. When producing multiple graphs in
the same window, you may often need to explicitly control the range of the x- and
y-axis. You do this by setting the xlim and ylim parameters for the plot function
or another high-level plotting function.
The basic functions for producing graphics in R have by now been extended consid-
erably by packages providing a range of more advanced plotting tools. Two packages
should be mentioned. The lattice package and the ggplot2 package both provide
comparable functionality, and are in particular useful for producing multilayered or
stratified plots.
A.5 Packages
Packages form a very important part of R. A large number of researchers in statistics

choose to implement their work as an R package. At the time of writing the number
of packages available from the CRAN archieve is of the order of thousands. The
biggest problem is actually to figure out which package implement the methods that
one wants to apply. If you don’t know, the CRAN archive provides the possibility of
browsing the packages by topic.
If we want to install and use the ggplot2 package, we proceed as follows:
> install.packages("ggplot2")
installs the package and installs in addition all packages that ggplot2 depends upon.
Packages 273
> t <- seq(0,10,length=101)

> #Compute sinus of the time points in t
> x <- sin(t)
> #Using plot with type="l" produces lines between the points
> plot(t,x,type="l",main="Sinus")
> #Add normally distributed "noise" to the x values
> y <- x + rnorm(101,0,0.2)
> #Plot the noisy values (no annotation) and the mean
> windows()
> plot(t,y,xlim=c(0,10),ylim=c(-2,2),ann=F,pch=20)
> points(t,x,xlim=c(0,10),ylim=c(-2,2),type="l",col="red")
> #Add a title to the plot
> title("Normally distributed variables with oscillating mean")
> #Add a legend to the plot
> legend(0,2,
+ legend=c("Observations","Mean"),col=c("black","red"),
+ lwd=1,lty=c(0,1),pch=c(20,NA))
Table A.1: This example illustrates some uses of R for making graphics. The symbol
# produces comments in R.
> library("ggplot2")
loads the package. Note that you need an Internet connection to install the packages.
One can also load a package by the command require. Using require returns a
logical, which is TRUE if the package is available. This is useful in e.g. scripts for
checking that needed packages have actually been loaded.
A.5.1 Bioconductor
There is an entire software project called Bioconductor based primarily on R, which

provides a number of packages for the analysis of genomic data – in particular for
treating data from microarray chips. Information can be found on
http://www.bioconductor.org/
With
> source("http://www.bioconductor.org/biocLite.R")
> biocLite()
you install a fundamental subset of the Bioconductor libraries. See also

http://www.bioconductor.org/install
To install a specific package from the Bioconductor repository use biocLite("package
name").
274 R
A.6 Literature
How you are actually going to get R to do anything interesting is a longer story. The
present lecture notes contains information embedded in the text via R boxes that
describe functions that are useful in the context they are presented. These boxes can
not entirely stand alone, but must be regarded as directions for further study. An
indispensable reference is the manual An Introduction to R as mentioned above and
the online help pages, whether you prefer the HTML-interface or the help function.
The homepage http://www.r-project.org/ contains a list of, at the time of writ-
ing, 94 books related to R. This author is particularly familiar with three of the
books. An introduction to statistics in R is given in
Peter Dalgaard. Introductory Statistics with R.
Springer, 2002. ISBN 0-387-95475-9,
which treats statistics more thoroughly than the manual. This book combined with
the manual An Introduction to R provides a great starting point for using R to do
statistics. When it comes to using R and S-Plus for more advanced statistical tasks
the bible is
William N. Venables and Brian D. Ripley. Modern Applied Statistics with S.
Fourth Edition. Springer, 2002. ISBN 0-387-95457-0.
A more in-depth book on the fundamentals of the S language is
William N. Venables and Brian D. Ripley. S Programming.
Springer, 2000. ISBN 0-387-98966-8..
There is also a book on using R and Bioconductor for Bioinformatics:
Gentleman, R.; Carey, V.; Huber, W.; Irizarry, R.; Dudoit, S. (Eds.)
Bioinformatics and Computational Biology Solutions Using R and Bioconductor.
Springer, 2005. ISBN: 0-387-25146-4.
A.7 Other resources
The R user community is growing at an increasing rate. The language R has for some
time been far more than an academic language for statistical experiments. Today, R
is just as much a workhorse in practical data analysis and statistics – in business and
in science. The expanding user community is also what drives R forward and users
contribute with packages at many different levels and there is a growing number of
R-related blogs and web-sites. When you want to find an answer to your question,
Google is often your friend. In many cases questions have been asked on one of the
R emailing lists or treated somewhere else, and you will find it if you search. Being
new to R it may be difficult to know what to search for. Two recommended places
to look for information is the R wiki
Other resources 275
http://rwiki.sciviews.org/doku.php
and the list of contributed documents
http://cran.r-project.org/other-docs.html
The latter is a mixed collection of documents on specialized R topics and some
beginners guides that may be more friendly than the manual.
276 R
B
Mathematics
The mathematical prerequisites for reading this introduction to probability theory

and statistics is an elementary understanding of set theory and a few concepts from
calculus such as integration and differentiation. You will also need to understand a
few things about limits and infinite sums. This appendix discusses briefly the most
important mathematical concepts and results needed.
B.1 Sets
A set E is (informally) a collection of elements. If an element x is contained in or

belongs to a the set E we write x ∈ E. If A is a collection of elements all belonging
to E we say that A is a subset of E and write A ⊆ E. Thus A is in itself a set, which
is included in the larger set E. The complement of A, denoted Ac , within E is the
set of elements in E that do not belong to A. We write
Ac = {x ∈ E | x 6∈ A}.
If A, B ⊆ E are two subsets of E we define the union
A ∪ B = {x ∈ E | x ∈ A or x ∈ B}
and the intersection
A ∩ B = {x ∈ E | x ∈ A and x ∈ B}.
We also define
A\B = A ∩ B c ,
which is the set of elements in A that do not belong to B.
277
278 Mathematics
The integers Z, the non-negative integers N0 , the positive integers N (also called the
natural numbers), the rational numbers Q, and the real numbers R are all examples
of sets of numbers. We have the following chain of inclusions
N ⊆ N0 ⊆ Z ⊆ Q ⊆ R.
There is also the even larger set of complex numbers C. We find for instance that
N0 \N = {0},
and that Z\N0 is the set of negative integers. Note that this is the complement of N0
within Z. The complement of N0 within R is a larger set. The set R\Q (which is the
complement of the rational numbers within R) is often called the set of irrational
numbers.
B.2 Combinatorics
In the derivation of the point probabilities for the binomial distribution, Example
3.2.1, we encountered the combinatorial quantity nk . This number is the number of
ways we can pick k out of n elements disregarding the order, since this corresponds
to the number of ways we can pick out k xi ’s to be equal to 1 and the remaining
n − k xi ’s to equal 0 such that the sum x1 + . . . + xn = k. If we take the order
into account there are n possibilities for picking out the first element, n − 1 for the
second, n − 2 for the third and so on, hence there are n(n − 1)(n − 2) · · · (n − k + 1)
ways of picking out k elements. We use the notation
n(k) = n(n − 1)(n − 2) · · · (n − k + 1).
With k = n this argument reveals that there are k(k) = k! orderings of a set of k
elements. In particular, if we pick k elements in order there are k! ways of reordering
the set hence
n n(k) n!
= = .
k k! k!(n − k)!

The numbers nk are known as binomial coefficients. They are often encountered in
combinatorial problems. One useful formula that relies on binomial coefficients is
the following: For x, y ∈ R and n ∈ N
n
X n k n−k
(x + y)n = x y , (B.1)
k
k=0
which is simply known as the binomial formula. Letting x = p and y = 1 − p we see

that
Xn
n k
p (1 − p)n−k = (p + (1 − p))n = 1,
k
k=0
Limits and infinite sums 279
which shows that the point probabilities for the binomial distribution indeed sum
to one (as they necessarily must).
A simple continuation of the argument also gives a formula for the multinomial
coefficients
n
k1 . . . km
encountered in the multinomial distribution in Example 3.2.2. As we argued above
there are n! orderings of the n elements. If we assign labels from the set {1, . . . , m}
by choosing one of the n! orderings and then assign a 1 to the k1 first elements, a
2 to the following k2 elements and so on and so forth we get n! ways of assigning
labels to the elements. However, for any ordering we can reorder within each group
and get the same labels. For a given ordering there are k1 !k2 ! · · · km ! other orderings
that result in the same labels. Hence

n n!
= .
k1 . . . km k1 !k2 ! · · · km !
B.3 Limits and infinite sums
A sequence of real numbers, x1 , x2 , x3 , . . ., often written as (xn )n∈N , can have a limit,
which is a value that xn is close to for n large. We say that xn converges to x if we,
for all ε > 0 can find N ≥ 1 such that
|xn − x| ≤ ε
for n ≥ N . If xn converges to x we write
xn → x, for n → ∞ or lim xn = x.
n→∞
A sequence (xn )n∈N is increasing if
x1 ≤ x2 ≤ x3 . . . .
An increasing sequence is either upper bounded, in which case there is a least upper
bound, and the sequence will approach this least upper bound, or the sequence is
unbounded, in which case the sequence grows towards +∞. An increasing sequence
is therefore always convergent if we allow the limit to be +∞. Likewise, a sequence
is decreasing if
x1 ≥ x2 ≥ x3 . . . ,
and a decreasing sequence is always convergent if we allow the limit to be −∞.
Let (xn )n∈N be a sequence of non-negative reals, i.e. xn ≥ 0, and define
n
X
sn = xk = x1 + x2 + . . . + xn ,
k=1
280 Mathematics
then, since the x’s are non-negative, the sequence (sn )n∈N is increasing, and it has
a limit, which we denote
X∞
xn = lim sn .
n→∞
n=1
It may be +∞. We write
∞
X
xn < ∞
n=1
if the limit is not ∞.
If (xn )n∈N is any sequence of reals we define
x+
n = max{xn , 0} and x−
n = max{−xn , 0}.
Then xn = x+ − + −
n − xn and the sequences (xn )n∈N and (xn )n∈N are sequences of
positive numbers. They are known as the positive respectively the negative part of
the sequence (xn )n∈N . If
X∞
+
s = x+
n <∞
n=1
and
∞
X
s− = x−
n <∞
n=1
then we define the infinite sum
X∞ ∞
X ∞
X
xn = s+ − s− = x+
n − x−
n
n=1 n=1 n=1
and we say that the sum is convergent. If one of the sums, s+ or s− , is +∞ we say
that the sum is divergent. We may also observe that
|xn | = x+ −
n + xn
and we conclude that the sum is convergent if and only if

∞
X
|xn | < ∞.
n=1
A classical infinite sum is the geometric series with xn = ρn−1 for ρ ∈ (−1, 1), then
∞ ∞
!
X X 1
n−1 n
ρ = ρ = . (B.2)
1−ρ
n=1 n=0
Another example is the infinite series representation of the exponential function,

∞
X λn
exp(λ) = (B.3)
n!
n=0
valid for λ ∈ R.
Integration 281
B.4 Integration
Integration as introduced in elementary calculus courses is a way to compute the

area under the graph of a continuous function. Thus if f : R → R is a continuous
function we can introduce a number,
Z b
f (x)dx,
a
which is the area under the graph of f from a to b. This area being computed with
a sign. In some cases it makes sense to let a or b tend to −∞ or ∞ respectively. In
that case we get that
Z ∞
f (x)dx
−∞
is the entire area under the graph from −∞ to ∞. If f is a positive function, this
“area” always makes sense, though it may be infinite.
If f (x) ≥ 0 for all x ∈ R, the sequence of numbers
Z n
In = f (x)dx
−n
is an increasing sequence, hence it has a limit, which may equal +∞, for n → ∞.
We write Z ∞ Z n
f (x)dx = lim In = lim f (x)dx.
−∞ n→∞ n→∞ −n
We say that the function f is integrable over R if this number is finite.

As for the infinite sums above, if f is any continuous function one can define the
two positive functions
f + (x) = max{f (x), 0} and f − (x) = max{−f (x), 0}
such that f (x) = f + (x) − f − (x) and |f (x)| = f + (x) + f − (x). We say that f is inte-
grable over R if the two positive functions f + and f − are integrable or equivalently
if the positive function |f | is integrable. In this case we also have that
Z ∞ Z n
f (x)dx = lim f (x)dx.
−∞ n→∞ −n
Prior to using this identity we have to check if

Z n
lim |f (x)|dx < +∞.
n→∞ −n
282 Mathematics
B.4.1 Gamma and beta integrals
The integral Z ∞
xλ−1 exp(−x)dx
0
is finite for λ > 0. It is known as the Γ-integral (gamma integral), and we define the
Γ-function by
Z ∞
Γ(λ) = xλ−1 exp(−x)dx. (B.4)
0
The Γ-function is a classical and much studied function. It holds that
Γ(λ + 1) = λΓ(λ) (B.5)
for λ > 0, which together with Γ(1) = 1 implies that
Γ(n + 1) = n!
for n ∈ N0 . For non-integer λ the Γ-function takes more special values. One of the
peculiar results about the Γ-function that can give more insight into the values of
Γ(λ) for non-integer λ is the reflection formula, which states that for λ ∈ (0, 1)
π
Γ(λ)Γ(1 − λ) = .
sin(πλ)
For λ = 1/2 we find that Γ(1/2)2 = π, thus

Z ∞
1 √
Γ(1/2) = √ exp(−x)dx = π.
0 x
This can together with (B.5) be used to compute Γ(1/2 + n) for all n ∈ N0 . For
instance,
√
Γ(1/2) π
Γ(3/2) = Γ(1/2 + 1) = = .
2 2
The B-function (B is a capital β – quite indistinguishable from the capital b – and

the pronunciation is thus beta-function) is defined by
Γ(λ1 )Γ(λ2 )
B(λ1 , λ2 ) = (B.6)
Γ(λ1 + λ2 )
for λ1 , λ2 > 0. The B-function has an integral representation as

Z 1
B(λ1 , λ2 ) = xλ1 −1 (1 − x)λ2 −1 dx. (B.7)
0
Integration 283
B.4.2 Multiple integrals
If f : R2 → R is a continuous function then for any a < b, a, b ∈ R, function

Z b
x 7→ f (x, y)dy
a
is also a continuous function from R to R. We can integrate this function over the
interval [c, d] for c < d, c, d ∈ R and get the multiple integral
Z dZ b
f (x, y)dydx.
c a
We can interpret the value of this integral as the volume under the function f over
the rectangle [c, d] × [a, b] in the plane.
It is possible to interchange the order of integration so that
Z dZ b Z bZ d
f (x, y)dydx = f (x, y)dxdy.
c a a c
In general, if f : Rk → R is a continuous function of k variables the k-times iterated

integral
Z b1 Z bk
··· f (x1 , . . . , xk )dxk . . . dx1
a1 ak
is a sensible number, which we can interpret as an k + 1-dimensional volume under

the graph of f over the k-dimensional box
[a1 , b1 ] × . . . × [ak , bk ].
It is notable that the order of the integrations above do not matter.

As for the univariate case we can for positive, continuous functions f : Rk → [0, ∞)
always define
Z ∞ Z ∞ Z n Z n
··· f (x1 , . . . , xk )dxk . . . dx1 = lim ··· f (x1 , . . . , xk )dxk . . . dx1
−∞ −∞ n→∞ −n −n
but the limit may be equal to +∞. For any continuous function f : Rk → R we have
that if Z ∞ Z ∞
··· |f (x1 , . . . , xk )|dxk . . . dx1 < ∞
−∞ −∞
then
Z ∞ Z ∞ Z n Z n
··· f (x1 , . . . , xk )dxk . . . dx1 = lim ··· f (x1 , . . . , xk )dxk . . . dx1 .
−∞ −∞ n→∞ −n −n
Index
p-value, 168 CLT, 84

Fisher information codon, 31
Multivariate, 185, 260 coin flipping, 20
confidence intervals, 175
acceptance region, 162 convergence in distribution, 250
alternative, 162 convergence in probability, 250
assay, 9 coverage, 177, 184
average, 30, 224 actual, 180
nominal, 180
bandwidth, 49
critical value, 163
Bayesian interpretation, 19
BCR/ABL fusion gene, 123 ∆-method, 253
Bernoulli experiment, 21, 62 density, 35
Bernoulli variable, 63 Gumbel distribution, 42
B-distribution, 41 logistic distribution, 42
binomial distribution, 127 distribution, 16
binomial coefficient, 278 distribution function
Biostrings, 26 properties, 33
bootstrap, 212 distribution function
algorithm, 213 definition, 32
non-parametric, 215 dose-response, 8
parametric, 214
ELISA, 10
calibration, 10 empirical distribution function, 52
Cauchy-Schwarz inequality, 235 empirical mean, 224
cause, 9 empirical normalization, 232
Central Limit Theorem, 84 empirical variance, 231
central limit theorem, 250, 254 event, 13
central moment, 223 expectation, 219, 225
central second moment, 223 exponential distribution, 38
Chapman-Kolmogorov equations, 115 exponential distribution
χ2 -distribution, 40 intensity parameter, 39
284
Index 285
Fisher information, 179, 258 MCMC, 248

fitted values, 194 mean, 225
fold-change, 182 median
four parameter logistic model, 207 empirical, 53
frequency interpretation, 18 theoretical, 55
median absolute deviation, 66
Γ-distribution, 40 moment, 223
Gaussian distribution, 37 Monte Carlo integration, 245
geometric distribution, 128 multinomial distribution, 128
Hardy-Weinberg equilibrium, 73 multinomial coefficient, 279
histogram, 44
negative binomial distribution, 131
unnormalized, 44
neural networks, 136
hypergeometric distribution, 129
normal distribution, 37
hypergeometric distribution, 129
null-hypothesis, 162
identically distributed, 71
observational data, 9
identifiability, 133
observed Fisher information, 179
iid, 71
odds, 18
importance sampling, 247
open reading frame, 31
independence, 71
indicator random variable, 64 parameter of interest, 181
information Pareto distribution, 43
expected information, 258 percentile interval, 217
Fisher information, 258 phenomenology, 126
observed information, 258
phylogenetics, 120
intercept parameter, 190
plug-in principle, 165
interquartile range, 55
point probabilities, 23
kernel, 48 Poisson distribution, 26
density estimate, 48 probability distribution, 16
density estimation, 47 Probability measure, 4
rectangular, 48 probability measure
definition, 15
LD50 , 183 properties, 16
level, 162 probit regression, 80
leverage, 195, 196 profile likelihood, 143
linkage, 74
location, 10 quantile function, 54
location-scale transformation, 66 quantiles
log-normal distribution, 233 empirical, 53
quartiles
MAD, 66 empirical, 53
marginal distribution, 70 theoretical, 55
Markov Chain Monte Carlo, 248
maximum likelihood method, 132 random variable, 62
286 Index
randomization, 79 continuous, 39
refractory period, 120 uniform distribution
regression discrete, 25
assay, 10
rejection region, 162 vector, 24
relative frequency, 18
Weibull distribution, 42
reparameterization, 144
Welch two-sample t-test, 168
residual sum of squares, 189
with replacement, 129
residuals, 194
without replacement, 129
non-linear regression, 205
Robinson-Robinson frequencies, 23 Yahtzee, 1
rug plot, 44
sample mean, 30
sample space, 13
discrete, 21
sample standard deviation, 232
sample standard error of the mean, 242
sample variance, 31, 231
scale, 10
scatter plot, 109
significance level, 163
slope parameter, 190
standard curve, 10
standard deviation
integer distributions, 29
standard error of the mean, 242
standardized residuals, 195
statistical test, 162
statistically significant, 163
stochastic variable, 62
structural equation, 78
success probability, 63
symmetric distribution, 65
t-distribution, 43
t-test
one-sample, 186
tabulation, 27
test, 162
test statistic, 162
transformation, 63
transition probabilities, 112
uniform distribution

Notes PDF

Uploaded by

Document Informationclick to expand document information

Document Informationclick to expand document information

Copyright:

Available Formats

Notes PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Notes PDF

Uploaded by

Copyright:

Available Formats

Probability Theory and

Niels Richard Hansen

2.10.1 Random variables and independence . . . . . . . . . . . . . . 70

3 Statistical models and inference 117

4 Mean and Variance 219

4.1 Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

1.1 Notion of probabilities

0 100 200 300 400 500

1.2 Statistics and statistical models

Where probability theory is a deductive and mathematical science, statistics – and

The typical datasets we consider in relation to motifs are computational in nature.

segment A segment B segment C

Obviously it would be unethical to carry out a controlled experiment with a lethal

ELISA (Enzyme-linked immunosorbent assay) is a so-called immunoassay used to

Example 1.2.7 (Location and scale). A more abstract example of a statistical

Example 1.2.8 (Local alignment). Local alignment of DNA-sequences or proteins

Example 1.2.9 (Gene expression). A gene expression microarray is an experimental

2.2 Sample spaces

carried out entirely on a computer, that is, a computer simulation.

0 5000 10000 15000 20000

Microarray experiments constitute another class of experiments where the outcome

2.3 Probability measures

Definition 2.3.1. A probability measure P assigns to all events A ⊆ E a value

(ii) if A1 , . . . , An are disjoint events

P (A1 ∪ . . . ∪ An ) = P (A1 ) + . . . + P (An ).

Property (ii) above is known as additivity of a probability measure. It is crucial that

1 = P (E) = P (A) + P (Ac ).

From this we obtain

In particular, for the empty event ∅ we have

If A, B ⊆ E are two events such that A ⊆ B we have that

where A and B\A are disjoint. By additivity again we obtain that

P (B) = P (A) + P (B\A),

hence since P (B\A) ≥ 0 it follows that if A ⊆ B then

P (A) ≤ P (B). (2.1)

P (A) = P (A\C) + P (C).

Result 2.3.2. Properties of probability measures:

Math Box 2.3.1 (General Probability Measures). In Definition 2.3.1 we only

P (A1 ∪ . . . ∪ An ) = P (A1 ) + . . . + P (An ).

P (A1 ∪ A2 ∪ A3 ∪ . . .) = P (A1 ) + P (A2 ) + P (A3 ) + . . . . (2.3)

It is a perfectly natural requirement and as it stands, it may seem as a quite innocent

A collection E of subsets fulfilling those requirements is called a σ-algebra and the

• For any event A ⊆ E,

• For two events A ⊆ B ⊆ E,

P (B) = P (A) + P (B\A) and P (A) ≤ P (B).

• For any two events A, B ⊆ E,

P (A ∪ B) = P (A) + P (B) − P (A ∩ B).

ξ(Ac )1A (xi ) − 1Ac (xi ),

Likewise, player two will on average win approximately 0 kroner if n is sufficiently

with the four events:

P (k) = P (1)x1 (k) · P (2)x2 (k) · . . . · P (n)xn (k) .

2.4 Probability measures on discrete sets

If E is an infinite set we may be able to arrange the elements in E in a sequential

Definition 2.4.1. We call a set E discrete if it is either finite or countably infinite.

A, C, G, T, AA, AC, AG, AT, CA, . . . , TG, TT, AAA, AAC, . . .

E = {aa, Aa, AA}.

E = {aabb, aaBb, aaBB, Aabb, AaBb, AaBB, AAbb, AABb, AABB}

that contains nine elements. ⋄