SENSE ABOUT SCIENCE

MAKING SENSE OF UNCERTAINTY

Why uncertainty is part of science

CONTRIBUTORS
The contributors met over 2012 to review what is being said
about scientific uncertainty in the media, policy and public
life; to identify the misconceptions in these discussions and
share insights that help to dispel these; and to draft and
edit the guide.
With thanks to everyone who reviewed all or part of the
guide, or gave contributions on specific questions,
including: Dr Suzanne Aigrain, Dr Nina Alphey, Dr Richard
Betts, Professor Sheila Bird, Professor Andy Challinor, Dr
Nathan Green, Professor Joanna Haigh, Professor Mike
Hulme, Christian Hunt, Richard Johnston, Dr Shelly
Lachish, Professor John Mitchell, Professor Peter
Sammonds, Dr Emily Shuckburgh, Dr Juliet Stevens, Dr
Matthew Sullivan, Dr Tom Wells, Dr Ben Wheeler; and for
valuable input from the Royal Society meeting Handling
uncertainty in weather and climate prediction convened by
Professor Tim Palmer in October 2012.
We are grateful to the BBSRC, Clark Bradbury Charitable
Trust, John Innes Centre, NERC, University of Reading and
the Walker Institute for supporting the production and
distribution of this guide.

PETER GIBBS

BBC weather presenter,

Met Office

KATHY MASKELL
Walker Institute
for Climate
System Research

DR MARK NAYLOR
Scottish Government
and Royal Society
of Edinburgh
Personal Fellow

DR EMILY SO

Lecturer, Department of
Architecture, University
of Cambridge

DR FRANK DE VOCHT
Lecturer in Occupational
and Environmental
Health, University
of Manchester

MICHAEL HANLON

PROF. PAUL
HARDAKER

DR ED HAWKINS

Chief Executive, Institute

of Physics; Chairman,
Sense About Science

Climate scientist in
NCAS-Climate,
Department of
Meteorology, University
of Reading

PROF. ANGELA
MCLEAN FRS

DR ELIZABETH
MORRIS OBE

DR KEN MYLNE

PROF. TIM PALMER FRS PROF. SIR MICHAEL

Royal Society Research
RAWLINS

DR KATHERINE ROYSE

Professor, University of
Oxford; Co-Director
Oxford Martin Programme
on Modelling and
Predicting Climate

Chair, National
Institute of
Clinical Excellence

Science Director
Environmental Modelling,
British Geological Survey;
NERC Knowledge
Exchange Fellow

PROF. LEONARD
SMITH

PROF. DAVID
SPIEGELHALTER FRS

DR DAVID
STAINFORTH

PROF. IAN
STEWART FRS

TRACEY BROWN

DR TABITHA
INNOCENT

Journalist

HEATHER MAYFIELD
Deputy Director,
Science Museum

Winton Professor of the

Public Understanding
of Risk, University
of Cambridge

Managing Director,
Sense About Science

Professor of
Mathematical Biology,
University of Oxford

Scott Polar Research

Institute, Cambridge

Senior Research Fellow,

Emeritus Professor of
Grantham Research Institute Mathematics, University
on Climate Change and the of Warwick
Environment, LSE

Scientific Liaison,
Sense About Science

PROF. AVERIL
MACDONALD

JJ Thomson Physical
Laboratory, University
of Reading

Head of Weather
Science Numerical
Modelling, Met Office

Director, Centre for

the Analysis of Time
Series, LSE

DR CHRIS TYLER

Director, Parliamentary
Office of Science and
Technology (POST)

INTRODUCTION
why make sense of uncertainty?

Scientific uncertainty is prominent in research

that has big implications for our society: could
the Arctic be ice-free in summer by 2080?
Will a new cancer drug be worth its side
effects? Is this strain of flu going to be a
dangerous epidemic?
Uncertainty is normal currency in scientific
research. Research goes on because we dont
know everything. Researchers then have to
estimate how much of the picture is known
and how confident we can all be that their
findings tell us whats happening or whats
going to happen. This is uncertainty.
But in public discussion scientific uncertainty is
presented as a deficiency of research. We
want (even expect) certainty safety, effective
public policies, useful public expenditure.
Uncertainty is seen as worrying, and even a
reason to be cynical about scientific research
particularly on subjects such as climate
science, the threat of disease or the prediction
of natural disasters. In some discussions,
uncertainty is taken by commentators to mean
that anything could be true, including things
that are highly unlikely or discredited, or that
nothing is known.

This conflict frustrates us at Sense About

Science, and we know that it frustrates
researchers we work with and the public we
hear from. Some clearer ideas about what
researchers mean by scientific uncertainty
and where uncertainty can be measured and
where it cant would help everyone with how
to respond to the uncertainty in evidence.
This guide has brought together specialists in
many areas climate science, clinical research,
natural hazard prediction, public health,
biostatistics and epidemiology. We asked them
for the reasons why they are not automatically
so troubled by the presence of uncertainty in
the most heated debates.
We have looked at what uncertainty means
and doesnt mean in science, how it is
measured, when it cant be measured and how
that might change through research into the
big questions. Above all we asked how other
people can grapple constructively with
advances in knowledge and changes in
thinking, instead of despairing at those
uncertain scientists.

TRACEY BROWN
TABITHA INNOCENT
SENSE ABOUT SCIENCE

CONTENTS

01

THOSE UNCERTAIN
SCIENTISTS

04

DO WE EVEN NEED
MORE CERTAINTY?

Uncertainty is normal in
scientific research but to policy
makers, journalists and wider
society it sounds like
unreliable. Despite life telling
us otherwise, the assumption in
many debates is that we should
expect certainty.

We need to know when a

decision is, and is not,
affected by whether we know
something completely. This
idea is beginning to shape
the way that scientists and
policy makers use and
communicate uncertainty.

02

05

SO WHAT IS UNCERTAINTY
FOR SCIENTISTS?

PLAYING
ON UNCERTAINTY

There are different types of

uncertainty that are an ordinary
part of scientific research. These
can be addressed or taken into
account in several ways.

Uncertainty does not mean we

know nothing, that evidence
cannot be trusted, that anything
could turn out to be correct or
that decisions cant be made.

03

06

PREDICTIONS
AND MODELS

As numerical models can be

used flexibly and updated as
knowledge changes, they are
routinely used in research that
deals with high levels of
uncertainty. These sometimes get
a bad press, which misreads how
and why they are used.

DELVING
DEEPER

Further resources on scientific

uncertainty, including blogs,
books and guides.

01
it becomes ever more
obvious that none of
them really has a clue

THOSE
UNCERTAIN
SCIENTISTS
In the areas of research that are most often in
the public eye, uncertainty has become a big
point of misunderstanding (even conflict)
between scientists and commentators. A
researcher presents his or her findings, the
radio interviewer (or the politician, journalist
or official) asks: can you be certain?. The
researcher has to answer truthfully 'no' and
then defend their findings, for fear they will be
interpreted as meaningless. In fact, they have
provided important limits to the uncertainty.

when distinguished
scientists say it
isn't actually proven

you can't absolutely prove,

can you, that co2 is responsible
for global warming?

Researchers use uncertainty to express how

confident they are about results, to indicate
what scientists dont yet know, or to
characterise information that is by nature
never black and white. But saying that
something is uncertain in everyday language
has a negative connotation. When a
researcher says the predictions we made on
the basis of our research have a margin of
uncertainty, they mean they are very
confident that the outcome will fall within the
predicted range. But a commentator is likely to
understand from this the piece of research
is unreliable.

like with many neural

disorders and conditions
..there is always uncertainty
in their answers

This is the type of disconnection we see in

media reports of global warming, public
health risks and earthquake prediction.

05

THOSE UNCERTAIN SCIENTISTS

BUT WHY IS UNCERTAINTY SEEN TO

UNDERMINE RESEARCH?
Put crudely, scientists tend to think science is
about things we dont know fully. Journalists
and politicians (and to a large extent, many
people) think it is about things we do know,
and theyre impatient with maybes.

aspire to add something new to what we

know. Uncertainty is what they are interested
in. Scientists design experiments or
data-gather to get new evidence which tests
models and theories in as minimally biased a
way as possible but there are limits on what
can be done, and simplifying assumptions that
have to be made. Research science is moved
forwards by knowledge that is modified with
new evidence and most areas of research
include some knowledge that is settled, and
some that is more uncertain.

The problem here is that not all scientific

knowledge is the same, so treating it as
though it were is misleading.
In the first place, there is rarely such thing as
100% certainty and everything less than this
is uncertain. Scientific inquiry makes some
(potentially imperfect) observations, then
makes predictions to test how widely the
observed pattern holds true from looking at
how a molecule will behave in combating a
disease to understanding the metabolisms of
dinosaurs. Most people know this to some
extent. They know that there are varying risks
in the things they do every day, and that
perfect knowledge is rarely possible (most
knowledge is not like a mathematical proof).
But reports and commentaries like those
heading this section show that complete
certainty is still seen in society as the test of
new knowledge.

Across society, we dont talk much about the

settled things we are confident of the fact
that an antibiotic works, say. We talk about the
interesting, newer issues such as assessing
when antibiotic resistance might occur.
The essential point is that new scientific
knowledge usually includes greater
uncertainty and researchers often dont know
how much of the picture it shows. It doesnt
mean scientists know nothing, and we should
not be exasperated that it is less settled than
the explanation for why the sky is blue.

Secondly, new research needs to be looked at

differently from settled science. Settled science
is concerned with fundamental scientific
principles, well-established and supported by
large bodies of evidence (generally, the science
we all learned in school). Everyone can be
confident about a great deal of knowledge:
that the earth goes round the sun, the germ
theory of infectious disease, Pythagorass
theorem. Some science is widely accepted as
explaining the world around us: it underpins
commonplace inventions such as TV or
pasteurisation, and can get us to the moon,
predict an eclipse and explain why our feet
stay on the ground. And while exceptions are
found that show settled science can be revised
(game-changers, see section 5), research
discoveries rarely change our overall
understanding of the underlying
scientific principles.

MICHAEL
RAWLINS
The term uncertainty is unquestionably
fraught with misinterpretation
especially for non-scientists. I'd prefer the
phrase how confident am I?, the
reciprocal of uncertainty.

ELIZABETH
MORRIS
The word certain is like perfect in the
sense that it describes an end state, so it
is understood that there cannot be
degrees of certainty or perfection, only
degrees of uncertainty or imperfection.
We can say how far we are from the end
state, almost certain for instance.

For the most part, settled science is not what

scientists spend time thinking about,
questioning or researching. Researchers

06

02
satellite data
solves antarctic sea
ice mystery

WHAT DOES
UNCERTAINTY
MEAN FOR
SCIENTISTS?
Scientific researchers have to work out
whether uncertainty can be calculated and
how to do so, and then whether it matters and
what can be done about it.

...astronomers are certain

the asteroid will miss

THERE ARE SEVERAL DIFFERENT

CONCEPTS OF UNCERTAINTY
Uncertainty is a hard thing to discuss because
it is inherently intangible. The difficulty
expressing it is not exclusive to science: it
became familiar after a rather baffling speech
by US politician Donald Rumsfeld in 2002:

the emerging consensus was

that the benefits of hrt
outweigh the risks...

Reports that say that something hasn't

happened are always interesting to me,
because as we know, there are known knowns;
there are things we know we know. We also
know there are known unknowns; that is to say
we know there are some things we do not
know. But there are also unknown unknowns
the ones we don't know we don't know.
Rumsfeld had a point. Some scientific
uncertainty can be known: in a sense,
signposted. There is uncertainty about how
much global temperatures will change if
carbon dioxide emissions continue at current
levels for the next 50 years. But researchers

07

WHAT DOES UNCERTAINTY MEAN FOR SCIENTISTS?

know what information they would need to work this out it is a known unknown. Knowing just
what it is that we dont yet know is very useful because it guides further research to find an answer.
But we also face unknown unknowns questions not yet thought of, and information we do not
realise is missing at all.
These concepts of uncertainty are like the difference between a Victorian map with central Africa
marked unexplored, which indicates known unknown data, and medieval maps made before
Europeans knew that the New World existed, which show sea where the land should be.

DAVID
STAINFORTH

UNCERTAINTIES WE CAN WORK OUT AND UNCERTAINTIES WE CANT

If I drop a tennis ball, what will happen?
There is some uncertainty: I dont know
details about this particular ball. I havent
done experiments to see whether the ball
falls every time. But I still know, almost
certainly, that the ball will fall. I know that
this would be true if I was standing in
Australia, and that the way it falls would be
different if I was standing on the moon.
There isnt 100% certainty because the
situation might be different from what we
expect if I were working with a magician I
might be suspicious about the possibility of
tricks; maybe magnets in the ball and the
ceiling. This could be the case. But without
evidence for it I would be foolish to bet
against the ball falling. Indeed Id bet a lot
that the ball would fall. Now think of a

different question. When I drop the tennis

ball, where will it stop? This is difficult to
tell. It might bounce off a chair, a table or
someones foot. I can calculate some limits
to give me a rough idea of where the ball
might end up but there will remain
substantial uncertainty.
When were thinking about how the climate
might change, the system is more
complicated but the same thinking applies.
There are many details about which we are
extremely uncertain but there is very little
uncertainty over the big picture of serious
warming and significant disruption for
human societies. The ball will fall we just
dont know where it will stop.

08

WHAT DOES UNCERTAINTY MEAN FOR SCIENTISTS?

HOW DO SCIENTISTS WORK OUT UNCERTAINTY?

The best approach to uncertainty is
pragmatism. While it is important to be alert to
the possibility of unknown unknowns of
discovering that ideas about how the world
works are stronger or weaker than we realised
that possibility alone doesnt point us towards
better explanations.

Researchers gather data; if, for example, they

want to find the average height of people in
London, they measure the heights of a sample
of Londoners and calculate the mean average.
There is some chance that the mean average
(the value) the researchers have calculated is
not correct. The real average height of
Londoners could be a bit higher or a
bit lower.

On the other hand, if researchers

are describing the uncertainty in
how they understand a
particular problem, it actually
means quite a lot is known in
this case about the strengths
and weaknesses of the
science. They may know
enough about the uncertainty
to be able to quantify it, e.g.
there is a 20% chance of rain.
So if we hear scientists talking
about the uncertainty in their
findings in some detail we should be
more reassured rather than
more worried!

The researchers need to

express
that
uncertainty
about the value as precisely
as possible, so that their
result is useful to other
researchers and everyone
understands how likely it is
to be right. Using the range
of observed individual heights
they can calculate the range
(the confidence limits) that they
would expect the mean average to
fall within for at least 95 out of 100
repeats of the measurement exercise. This is a
95% confidence interval. If they wanted to
know the limits of that range for a 99%
confidence interval, i.e. where the mean
average is likely to fall in 99 out of 100 repeats
of the experiment, they would expect to give a
bigger range.

There are some well-established principles in

statistics for calculating and expressing
uncertainty (these have been explained in
detail elsewhere, see Resources). They ask
whether an observation or result is in the right
ball-park, or how likely it is that an observation
made in a piece of research is not simply a
random event.
Confidence intervals tell us how reliable
researchers think their observations are.

100%

09

This trade off between precision and

certainty is a fairly intuitive point. If you
throw a ball at a target on the floor and
are asked to draw a circle within which
you are confident it will land, you would
draw a bigger circle around the target if
it had to land in the circle 99 times out
of 100 than if it had to land in the circle
90 times out of 100.)

WHAT DOES UNCERTAINTY MEAN FOR SCIENTISTS?

When researchers are looking at cause and

effect relationships, they usually calculate
uncertainty by asking how likely it is that their
observations represent a real effect, rather
than simply happening by chance. This
question is incorporated into the design of the
study. Researchers start with a description of
what they would expect to see if there was no
effect (a null hypothesis); they then look at
how close the match is between this pattern
and their observations. Researchers can put a
number on the extent to which the results and
the hypothesis match (a p-value). This is
essentially asking, Is our result different
enough from a pattern of no effect that there
really looks to be something going on?.

PRECISE NUMBERS SHOULD

ONLY BE USED WHERE THEY
ARE JUSTIFIED
DAVID
SPIEGELHALTER
In clinical medicine, doctors cannot
predict exactly what will happen to
anyone, and so may use a phrase such
as of 100 people like you, 96 will
survive the operation. Sometimes there
is such limited evidence, say because a
patients condition is completely novel,
that no number can be attached with
any confidence.

Where uncertainty is known, it can be

incorporated into the way that effects are
predicted. For example, there is some
uncertainty in the initial conditions that form
the basis of a hurricane forecast. In the past,
forecasters showed one predicted path of a
hurricane, with caveats about the uncertainty
of the initial conditions. Now they tend to use
the uncertainty to vary the initial conditions
they put into the forecast and show the
resulting range of what the hurricane will do
an ensemble forecast so we can see how
many of the forecasts come up with a
similar path.

MICHAEL
HANLON
When the uncertainty makes the range
of possibilities very broad, we should
avoid trying to come up with a single,
precise number because it creates a
false impression of certainty
spurious precision.

10

WHAT DOES UNCERTAINTY MEAN FOR SCIENTISTS?

USE SEPARATE MEASURES FOR

HOW GOOD THE EVIDENCE IS,
AND HOW CONFIDENT THE
CONCLUSION
Organisations have come up with new
ways of expressing their confidence in
the data. In medicine, the GRADE scale
(a grading system for evidence and
recommendations) is widely used, which
takes into account the quality of the
underlying evidence. And, in climate
science, the most recent
Intergovernmental Panel on Climate
Change (IPCC) assessment used one
verbal scale to express their confidence
in the scientific understanding, and
another to give the likelihood of
something occurring (where virtually
certain corresponds to a level of
more than a 99% likelihood of
happening, for example).

observations. If we observe changes

which are significantly bigger than this,
and we can characterise the uncertainty
with the satellite observations
themselves, then we can more clearly
attribute these changes to long-term
processes such as climate change."

WHAT CAN BE DONE

ABOUT UNCERTAINTY?
Sometimes it is a question of making more
observations or designing experiments or
computational models to improve the
available information. This is not always as
straightforward as it might seem, and it can
take a while to identify the information that
is missing:
The early climate model simulations only
included the effect of greenhouse gases. When
run over the last 100 years or so for which we
have instrumental measurements of surface
temperature, these simulations were warm
compared to observations. In time, the effect of
other factors (industrial aerosols produced by
burning fossil fuels, the effect of volcanic
eruptions, possible variations in solar
irradiance) were added. The agreement of
scientists models with our observations
improved, particularly due to the inclusion of
the effect of industrial aerosols.
JOHN MITCHELL

ELIZABETH
MORRIS

SEPARATE LONG TERM EFFECTS

FROM SHORT TERM EFFECTS
"Glaciologists use satellite data to assess
changes in ice-sheet elevation and
estimate their impact on sea level. We
need to know whether these are
long-term changes (for example
produced by climate change) or just
short-term fluctuations (for example
produced by a particularly heavy
snowfall, or if a particularly warm
summer makes the surface snow
denser). We can understand the
uncertainty in the satellite observations
of long-term change better by defining
the size of the effect that short-term
fluctuations could have on the satellite

11

WHAT DOES UNCERTAINTY MEAN FOR SCIENTISTS?

Sometimes it is only through technological

developments that uncertainty can be reduced. This
has happened in the search for exoplanets
(planets that go round stars outside our solar
system) a step towards detecting whether
there is life on other planets. These planets
were only detected very recently when
instruments and techniques became
sufficiently accurate, but progress has
been rapid. The first exoplanets were
discovered in the 1990s, and
thousands have now been identified.
Their discovery is progressively
reducing the degree of uncertainty we
have about things like the existence of
life on other planets, by improving our
estimates of the number of planets
outside the solar system, and of the
fraction of these that may be able to
support life.

TECHNOLOGICAL LIMITATIONS

But reducing uncertainty is often not possible

or necessary and this is not a barrier to using
scientific knowledge.

The question is not

do we know everything?

For example, Newtons law of gravity is still

sufficient to get us to the moon and back in
spite of not being a complete model of the
way gravity works. Or if, say, all values within
a confidence interval point to the same clinical
decision, further narrowing the uncertainty
may be a waste of public funds. Researchers
can incorporate uncertainty into calculations
or work round it.

it is do we know enough?
or how can we best make
a decision using what we
do know?.

12

WHAT DOES UNCERTAINTY MEAN FOR SCIENTISTS?

Quantify routine uncertainty and

incorporate it into calculations

Risk is also a way of thinking about which

uncertainties to worry about. For example,
there is a greater risk of agricultural chemicals
running off into waterways and damaging
aquatic life when land is bare. We should
therefore be more interested in reducing
uncertainty about the hazard to aquatic life of
those chemicals used on bare land. Risk is
especially important when we want
information that is useful for decision-making.
(But risk itself is a whole other story, told in
some of the resources at the end.)

Uncertainty is considered routine in a lot of

cases it cant be eliminated but its not
necessarily problematic (and a decision can be
made beyond reasonable doubt). For
example, there is a degree of imprecision in
assessing the fetal age from ultrasound scans
during pregnancy (which are used to estimate
the due date). This arises from the natural
variation of size-for-age and from differences
in the skill, experience and equipment of the
ultrasonographer. However, we can quantify
this uncertainty and use it to explain to women
that the calculated due date should be used as
a guide, rather than an exact prediction of the
actual due date.

Mitigate the effects of

uncertain events
Another response to uncertainty is mitigation.
There is currently irresolvable uncertainty
about earthquakes: researchers know where
they are going to occur, but cannot predict
when they will happen.

Calculate risk, a more usable

measure of uncertainty
When uncertainty relates to a real-life
situation with important and perhaps
detrimental consequences we can work out
how the uncertainty affects the risk, which is
more tangible.

In a scientific sense, earthquakes are

unpredictable. But that does not mean that you
cant predict things about them.
PETER SAMMONDS
Using historical data and geology, researchers
can calculate how frequently a particular scale
of earthquake is likely to occur in a specific
location. A one-in-ten year calculation doesnt
say anything about which ten years in 100 will
see an earthquake. But it means researchers
can calculate what kind of reinforcement to
buildings would be needed to make sure they
could withstand this frequency of quakes. If a
serious earthquake is a one-in-200 year
event, a community might only put resources
into reinforcing a building that is designed to
last ten years if its function was sufficiently
important a school or hospital, for example.

There is some confusion between scientific

and everyday uses of the words uncertainty
and risk. In everyday language, we might say
that something that is uncertain is risky. But in
scientific terms, risk broadly means uncertainty
that can be quantified in relation to a
particular hazard and so for a given hazard,
the risk is the chance of its happening. For
example, research shows that oestrogen
therapy (a form of HRT, to relieve the symptoms
of menopause) appears to increase the risk of
women suffering from heart disease (an
associated hazard). Current evidence suggests
the risk is small, but there is a lot of uncertainty
around the estimate and so the real effect
could be larger or smaller than estimated.

13

03
forecasts for global
warming too high

PREDICTIONS
AND MODELS
Modelling is an area that regularly causes
misunderstanding about uncertainty;
recriminations quickly follow events that dont
match predictions.
Many of us look at models as a black box
where data go in and answers come out.
Some scientists find that image annoying
because of the scientific insight that goes into
how models are constructed. Scientific models
differ greatly across fields of research, and
specialists in one field might have no idea
about what is involved in modelling in
another. Suggesting all models are the same is
like saying that all experiments are the same
or all tools are the same.

new computer model predicts

when avian influenza strain
becomes infectious

economy will shrink this year,

treasury forecasts show

SCIENTIFIC MODELS
The scientific models we are concerned with
here are numerical representations of a
process or system to simulate some aspect of
the world: how will a disease epidemic
progress? Will a new bridge hold sufficient
weight? How have genetic traits spread
among human populations? These models
usually involve joining up different ideas or
information about how different parts of a
process work.

which weather forecast

should you believe?

14

PREDICTIONS AND MODELS

These models are a necessary simplification of

the real world and use experimental data or
observations, or physical understanding of
nature, or some combination of these.
Sometimes the components are empirical,
sometimes statistical, and sometimes direct
applications of physical laws. Different
problems require different modelling
approaches and, because not all models are
the same, the uncertainty around models can
be very different.

In epidemiology, one difficulty is that peoples

behaviour directly affects the course of an
epidemic (unlike, say, the weather, which will
be the same regardless of how people respond
to it). How people mix how many people, and
from which age groups tells us a lot in
predicting how a virus will spread. Past
estimates of how much mixing occurs were very
uncertain and epidemiologists knew this was
an important unknown in their models. Then a
recent study recorded and quantified how often
different age groups interact with each other
over an individuals lifetime, for the first time.
We can now use this to inform models of
disease epidemics. By identifying the important
uncertainties, we can try to reduce them.
ANGELA MCLEAN

WHAT MODELS CAN DO

The vast majority of models that researchers
make have a fairly routine use in research and
in everyday clinical applications, and are often
a simple descriptor of a process.
Models are usually used for two reasons, to:

1 ...understand patterns or the

interplay of different factors

Do models reinforce assumptions?

If researchers already know an outcome they

can use a model to work out the possible
ways this could have come about or which
factors have an important influence. Models
can fill in the gaps between observations for
example, in the process of reconstructing
evolutionary trees.

Because models simplify the world they have

to make assumptions, some of which may not
be correct. Are models verified independently
or do they reinforce the same assumptions?
That depends. Some models can be verified
for example the accuracy of weather forecasts
can be tested against what actually happens
with the weather. But some models cannot be
verified in this way. If two models are based
on the same assumptions they are likely to
produce more similar results so an answer
from one is not entirely independent of the
other. Where models based on different
assumptions give a similar answer, these are
more independent and can therefore provide
an indirect way of verifying the original model.
(This question about independence is currently
the subject of a lot of debate
among researchers.)

2 ...make predictions
Researchers start with observations about how
things are now, and use models to predict the
future evolution or to create different future
scenarios that could result from this starting
point. This gives an idea of the range of things
that could happen.
Modelling can be one of the most useful tools
for reducing uncertainty in a crisis.
Researchers turn to models when there is
already a lot of uncertainty about a question,
and they need more information to support
decisions on what to do next. Modelling can
generate a shopping list of things
researchers would like to know more about.

15

PREDICTIONS AND MODELS

MODELS ARE NOT

an answer waiting to
be discovered

intrinsically uncertain
There are many models in routine, everyday
use and often these are already settled science
and have low uncertainty.

There is often more than one way to model

any problem. Most processes and scenarios
are not straightforward to model, and experts
do not all agree on the best approach. This
disagreement is sometimes taken to mean that
everything is contested and unreliable. In fact,
using diverse approaches is a good thing in
any area of scientific research, and is
encouraged. Researchers can then see
whether different models produce a similar
answer independently of each other; if they
do, this helps to quantify the uncertainty.

Early signs of a measles outbreak were

detected in New Zealand in 2011, alongside a
very low level of immunity among young
people. Because the government was slow to
respond, an outbreak started. Fortunately, as
we know a lot about this virus, they already
had models of how a measles outbreak
escalates and had plans in place to control its
spread. They were well able to stop the
outbreak escalating. Because the uncertainty in
the models was relatively low, they generated
confident predictions that informed a useful
intervention. ANGELA MCLEAN

always very interesting (to

most people)
Small changes to models never make the
news. They rarely capture much attention
among scientists either. These tweaks are
often not even well documented but should
be, by the modellers at least because most
of them are not viewed to be scientifically
important enough. But this means that the
effects on data over time can be hard to track
similar to the problem of negative trial
results going unreported and introduce
unknown bias. Some modellers have
suggested there should be a Journal
of Tweaking!

static
Models change as the information researchers
have changes.
We used to think that everyone who caught flu
became ill, but the 2009 H1N1 (swine flu)
pandemic taught us that this clearly isnt true.
The biggest uncertainty throughout the
pandemic was the proportion of people who
would become ill after getting the virus. Models
of the worst-case scenarios predicted this could
be high, which was picked up in alarming
headlines saying that lots of people would die
from swine flu. In the end, roughly one third of
the UK population were infected, which was as
expected. But it was something of a surprise
that only a tiny fraction became ill; it turns out
that for H1N1 pandemic flu it is about 1 in 20.
This makes the distinction between infection
and disease very clear: lots of people were
infected, far fewer were ill.

able to answer everything

The capacity of models to answer increasingly
sophisticated questions is sometimes limited
by technology. For example, current limits with
computing power put some constraints on the
progress climate modellers and others can
make there are not yet powerful enough
computers to consider all possible future
climate scenarios to the same level of detail as
it is possible to forecast the weather. This
means that researchers have to make choices
that restrict models to answering specific parts
of a problem. In the case of climate modelling
this might mean researchers can use a model
to look at climate for a large geographic area,
or for a much smaller area in greater detail,
but not both at the same time.

There had to be a real-life pandemic event for

us to find this out because it is impossible to
discriminate clearly between strains of seasonal
flu virus. Now that we know this, models can
change to make more accurate predictions.
This will improve our ability to manage
pandemic threats. It will also inform the type of
statements scientists and others make about the
public health risk. ANGELA MCLEAN

16

04
we were told no risk,
claim flood families

DO WE EVEN
NEED MORE
CERTAINTY?
The UK government used its stockpile of
antiviral medication in 2009 when a swine flu
pandemic was predicted, at great expense; did
they over-react to the uncertainty inherent in
the swine flu model?

more bse cases

but minister insists
beef is safe

Striving for certainty in scientific research,

even research that affects public policy, can be
a waste of effort and resources. What we need
instead is to talk about just how much
information is enough to make a sound
decision, because if we ask whether we really
need more certainty, sometimes the answer is
a clear no.

scientists blamed
for quake deaths

DO WE EVEN NEED TO KNOW?

In policy-oriented research, there is a
tendency to focus on trying to establish more
complete predictions, for example of future
temperature variations, to feed into decisions.
This leads to too much emphasis on improving
knowledge of all the uncertainty ranges.
Instead, we should identify which details are
most relevant to a decision and extract the
relevant information from models about
processes and trade-offs, working on the
uncertainties that really matter for the
particular problem we are trying
to address. ANDY CHALLINOR

declaring a swine flu pandemic

was a 'monumental error'

17

DO WE EVEN NEED MORE CERTAINTY?

OPERATIONAL KNOWLEDGE
Decision-makers generally look for a higher
level of certainty for an operational decision
(such as introducing body scanners in airports)
than for a decision based on broader ideology
or politics (such as reducing crime rates).

SCIENTIFIC UNCERTAINTY &

PUBLIC POLICY

When are we certain enough? is never going

to be easy to answer. It depends on the desire
to act, robust public discussion, leadership,
willingness to address criticism and the risks
involved in getting things wrong. Decisions
are usually made by policy-makers and
officials, not researchers science doesnt tell
us what to do and what they need is
operational knowledge.

Decisions are not made on the basis of

scientific evidence alone, though it may play
an important role.
Scientific uncertainty is only one of a whole
raft of factors that influence decision making:
the impact on constituents, party votes,
ideological perspective, economic implications
and so on. The impression is sometimes given
that policy makers and politicians are
uncomfortable with handling uncertainty, but
every day they have to make decisions on the
basis of largely uncertain information.
CHRIS TYLER

Engineers work with operational knowledge

all the time this comes down to knowing
enough to be confident about carrying out a
particular task. Take aeroplanes: every
component and every aeroplane is regularly
tested. As a result they are very safe machines
that operate within well-known limits. So even
though it is impossible for an engineer to
predict every variable that an aeroplane will
be subjected to, they can still be flown with
confidence. And engineers can manage any
events that introduce greater uncertainty. For
instance, when the volcanic ash cloud
accumulated over Northern Europe in 2010,
planes were initially grounded as this
increased the uncertainty about when it was
safe to fly. Researchers subsequently
developed remote ash sensors for aeroplanes
and ran experiments to understand the extent
to which engines could tolerate ash, to say
confidently when it was safe to fly.

Sometimes researchers dont have all the

evidence and need to be clear when that is
the case. In areas of scientific uncertainty this
is particularly important given the scope for
the uncertainty itself to become politicised. We
have seen this when uncertainties in the
details of predictions about climate change
and disease epidemics have been amplified to
cast doubt on any and all research.
The reality is that scientists are not, for
instance, debating the evidence for man-made
global warming; they are debating uncertainty
about the extent and timing of changes, and
the most accurate methods for predicting
these. We should recognise this misuse of
uncertainty for what it is, essentially smoke
and mirrors that distract from the
decision-making process.

All the time, multi-million pound decisions are

made using weather forecasts that contain
uncertainty: in energy trading to predict future
energy demands, or by local councils deciding
when to grit roads, for example. If you know
the potential cost or damage, and the
parameters that matter, you can work out when
it pays to take action. LEONARD SMITH

18

DO WE EVEN NEED MORE CERTAINTY?

WE CANT ALWAYS IGNORE

UNCERTAINTY SO WHAT ARE
THE ALTERNATIVES?
Few people would argue that clear
communication about uncertainty is
unimportant, but there are different ideas on
the most effective approach. Although there is
still a long way to go, there are insights
emerging from some areas of research
public health, climate science, economics
where understanding and communicating
uncertainty is especially difficult.
As discussed in section 2, it is misleading to
quantify uncertainty that cannot be quantified
in these cases there is an even greater need
to talk equally clearly about what researchers
do not know as what they do. Unknown
unknowns cannot be identified, much less
quantified, and the best approach is to
recognise this.

CURRENT PLAN
FORECAST

The Bank of England produces forecasts with

quantified uncertainties and contingent
predictions these completely leave aside some
of the uncertainties deemed to be
unquantifiable. They might say, for instance,
this is what will happen unless the Euro
collapses, acknowledging the deeper
uncertainty of whether or not the Euro will
collapse, which they cannot know without
putting a spurious number on this. This has two
direct effects. It makes clear the scale they are
prepared to work on, and what they simply do
not know and cannot say. And this encourages
others using these forecasts financial
institutions, say to adopt an approach for
responding that focuses on resilience. There are
dangers in thinking you can quantify something
which cannot be quantified.
DAVID SPIEGELHALTER

19

DO WE EVEN NEED MORE CERTAINTY?

RESEARCHERS CAN EXPLAIN THE

EVENTS THAT WOULD CHANGE
THEIR MIND

LETS FOCUS ON WHAT WE NEED TO

KNOW, NOT ON THE MOST
COMPLICATED ANSWER

Sometimes changes in uncertainty can be

predicted. In other words there may be
uncertainty around future events but
researchers can work out clear signs of a
change in the level of uncertainty to look out
for. This information helps people faced with
similar, real-world scenarios to respond and
make decisions.

A more complicated solution to a problem is

not necessarily a better one. Keeping things
simple is important when communicating what
public policy decisions are made, and why.

One example, from the UK BSE outbreak in

the early 1990s, would be when the first cat
became ill with mad cow disease. This is when
lots of infectious disease experts stopped eating
beef. However, the fact that a cat getting sick
with a prion disease (TSE) for the first time was
a big danger sign, and represented an increase
in the certainty of health risk to humans, was
not spread as widely as it might have been. I
think being clear about what future events
might change an expert opinion about the level
of uncertainty would be helpful. This boils
down to researchers saying we may be very
uncertain now but are there warning signs that
people should look out for?
ANGELA MCLEAN

Take the debate about whether or not the UKs

mandatory breast cancer screening programme
of women above a certain age can be justified
by a reduced incidence of breast cancer. There
is considerable scientific dispute among experts
who strongly disagree. And, any decisions have
to be communicated to both policy makers and
individuals. A recent review of evidence
deliberately used a very simple model of the
benefits and harms and was selective about
what was included in the model. When they
published their findings, the group
acknowledged the considerable uncertainty in
their estimates and explicitly said that, though
they could not quantify how uncertain they
were about the numbers, they were confident
enough to make policy recommendations.
DAVID SPIEGELHALTER

20

05
are cell phones safe?
researchers still uncertain

PLAYING ON
UNCERTAINTY
On emotive, economically important and
political subjects, uncertainty in research
findings has been played up and played down
according to whether people favour or object
to the implications. And many implausible
products and theories are promoted by
emphasising the uncertainty of mainstream
knowledge, such as alternative medical
therapies that remind customers, scientists
dont really know what causes cancer....

why everything youve been told

about evolution is wrong

IT JUST SHOWS HOW WRONG

SCIENTISTS CAN BE...
uncertainty over mobile phone
and brain cancer links

In 2011, physicists were in the media talking

about data that suggested a particle had
broken the speed of light. Some
commentators immediately took this to
suggest that a fundamental law of physics had
been undermined (it was, in fact, a fault in the
recording equipment, not nearly so widely
reported). A climate change contrarian argued
that if we cannot have confidence in such
fundamental scientific principles, we certainly
cannot have any confidence in the evidence
for global warming. In fact, researchers
willingness to find out whether this particle
might exist is a good example of why we can
be confident in scientific findings: if they're
wrong, they'll be corrected - but only when
evidence that stands up to scrutiny
comes around.

never trust anyone

who is certain
about anything...

21

PLAYING ON UNCERTAINTY

The idea of epigenetics in evolutionary biology suggests an individuals environment, as well as

their genes, can affect the way traits are inherited. When researchers first suggested it, creationists
asked whether this showed that the theory of evolution was wrong. In fact, epigenetics is a
development of evolutionary thinking rather than an invalidation of the theory of evolution.
These presentations of uncertainty take discussions down a blind alley because they distort the
working knowledge we do have. Findings are usually revisions and developments, not game
changers. Uncertainty doesnt mean scientists know nothing.

DAVID
STAINFORTH

GAME CHANGERS ARE EXCEPTIONAL

The history of science is scattered with
game changers, rare discoveries which
substantially change our understanding of
how the world works. These discoveries can
change what we think is happening, beyond
anything previously considered a possibility.
Examples include the realisation that the
earth isnt the centre of the universe,
working out the structure of DNA (and that
it encodes the information required to make
living things), identifying that germs cause
disease, or finding out that particles exist
that are smaller than atoms.
Although such discoveries are important for
science and can open up new opportunities
for society, they neednt necessarily
undermine conclusions based on our earlier
understanding. Where earlier results are
backed up by robust and relevant
observations we can have confidence in
them even if they are later shown to be

based on an incomplete understanding.

Scientists can often tell whether their
conclusions have a significant risk of
changing as our understanding improves, or
are likely to stay the same regardless of any
future discoveries. Sometimes observations
can point to the limits of current theory long
before a new theory is developed. For
instance, measurements of the orbit of the
planet Mercury suggested limitations in
Newtons laws which were only explained
many years later by Einstein.
As a result some uncertainty estimates are
extremely robust, thanks to their foundation
on widely demonstrated phenomena and
methods, but others are much less so. The
difficulty for non-specialists is that there is
rarely an easy way of identifying the
difference between the two. This point
needs to be included in the questions put to
the experts.

22

PLAYING ON UNCERTAINTY

UNCERTAINTY DOES NOT MEAN

THAT ANYTHING COULD BE TRUE
If there is uncertainty in new research that
may become part of a wider public debate,
that uncertainty should not be taken to cast
doubt over the entire body of established
knowledge. This type of thinking seems to
play on the fear that the ground will
disappear from under us if our current
scientific thinking proves to be incomplete:
quite the opposite is true.

PLAYING UP AND PLAYING

DOWN UNCERTAINTY
Exaggerated uncertainty is more often than
not a vague appeal to things being uncertain.
A good example of this is the ongoing public
health controversy about the use of mobile
phones and cancer.

Game changers in research are rare (see

box). But even when these discoveries do
happen, though they often open up new
avenues of research, they are very unlikely to
render all existing knowledge irrelevant.
Game changers can go undetected for some
time, even in active areas of research,
precisely because the knowledge we do have
works well enough that no-one has a strong
reason to question it.

An independent expert committee convened by

the International Agency for Research on
Cancer (World Health Organization) concluded
that the scientific evidence of an increased risk
of brain tumours in cell phone users was
limited and that there was inadequate evidence
to draw conclusions for other types of cancer.
Campaigners claiming that mobile phones do
cause cancer say that because we cannot
entirely rule out an increased risk, we should
take immediate action. At the same time
advocates of mobile technology argue that as
there is no confirmed increase in risk and no
established biological mechanism, no action is
required. The significance of scientific
uncertainty is interpreted differently by the
parties in a debate; particularly when the
conclusion has economic or public health
consequences. FRANK DE VOCHT

They sap convictions by endlessly questioning

data, dismissing experimental innovation,
stressing uncertainties and clamouring for more
research. It takes something which is an
essential part of science healthy skepticism,
curiosity and turns it against itself and makes
it corrosive." ORESKES & CONWAY, 2011
Climate change controversy gets air-time.
Although controversies are allegedly about
science, often such disputes are used as a proxy
for conflicts between alternative visions of what
society should be like and about who has
authority to promote such visions.
MIKE HULME

23

PLAYING ON UNCERTAINTY

UNCERTAINTY IS NOT A WEAKNESS

OF SCIENCE!
1. Scientific research works on the
basis that there are things we
dont know.
Research is not done to prove things by
showing that there is no uncertainty, but to
find the best explanation for our observations
by testing each theory against alternatives.
The most likely explanation, or theory, is the
one that accounts for as much as possible of
what we see. Scientists can also say in
advance what evidence, if found, would refute
their theory. And so to work out which of two
competing alternatives is the better theory, we
shouldnt focus on the amount of uncertainty
but should ask how much of what we observe
can each theory explain.
For example, sunspots (solar activity) have
been suggested as an explanation for global
warming. There is some evidence that solar
activity affects the atmosphere in ways that
cause the temperature to change. However,
this theory cannot account for some important
features of the observed warming patterns
most significantly, while temperature has
increased steadily over the last 50 years,
solar activity has not. As a theory, it doesnt
account very well for our observations of
changing climate.
You should ask anyone who promotes an
alternative idea of what is going on to
indicate the uncertainty levels in their
own theory.

2. Scientists dont draw conclusions

based on a single piece of evidence.
Scientists design experiments to gather data
that answer specific questions; the aim is not
to agree but to explore, test results and retest
them. All measurements have an associated
uncertainty this uncertainty in data is
different to uncertainty about a conclusion.
Different interpretations of the same data are
often possible, and different datasets on the
same problem might reach different
conclusions. This is precisely why uncertainty
in any given piece of evidence does not
necessarily undermine an overall conclusion.

3. Scientific research seeks evidence

not consensus.
Consensus suggests that scientists aim to
agree. This incorrectly implies that scientists try
to minimise uncertainty for the sake of finding
consensus. When researchers in a field assess
the weight of evidence, they dont simply
mean the number of studies on a particular
question, or patients per study, but how
compelling the evidence is and how
thoroughly alternative explanations have been
looked at a result of the scientific process
and peer review of new research.

4. Scientific research is not political,

but the implications of research
can be.
When research has a bearing on public policy
it may seem that a consensus, if that is what
emerges, tells people how to act or what
decision to make. Similarly, if the policy
implications of research findings are clear,
questioning the uncertainty of the science
becomes an easy way to dodge those
implications. The conclusions of research can
be, and often are, used as a proxy for political
arguments. Researchers might well
recommend or discourage a course of action
but this is in the realm of policy making not
research. Scientific evidence will seldom be
the only factor in making a decision or
creating policy.

WHY DOES ANY OF THIS MATTER?

Until we understand scientific uncertainty, we
risk being seduced or confused by
misrepresentation and misunderstanding.
Without an understanding of uncertainty
amongst the public and policymakers alike,
scientists will struggle to talk about uncertainty
in their research and we will all find it hard to
separate evidence from opinion.

24

06
DELVING DEEPER
HANDLING UNCERTAINTY IN SCIENCE
was an interdisciplinary Royal Society meeting in 2010 to look
at how uncertainty is used in different areas of research.
http://rsta.royalsocietypublishing.org/content/369/1956.toc

MERCHANTS OF DOUBT (2011)

is a book by Naomi Oreskes and Erik Conway on the tactical
use of uncertainty by interest groups.

UNCERTAINTY AND STATISTICS:

UNDERSTANDING UNCERTAINTY
is David Spiegelhalters website using everyday examples to help
people make sense of chance, risk, luck, uncertainty and probability.
www.understandinguncertainty.org

HOW TO READ A PAPER (2010)

is a book by Trisha Greenhalgh that explains the different statistical
approaches in research papers; some key insights are summarised
in this collection of BMJ articles.
www.bmj.com/about-bmj/resources-readers/publications/how-read-paper

MAKING SENSE OF STATISTICS

is a guide, produced by Sense About Science with Straight
Statistics, that gives some questions to ask and identifies some pitfalls
to avoid to help people get behind news stories that use statistics.
www.senseaboutscience.org/resources.php/1/making-sense-of-statistics

25

ALL MODELS ARE WRONG

is a blog discussing uncertainty in relation to modelling and
climate science.
www.allmodelsarewrong.com

UNCERTAINTY IN MEDIA AND POLICY-MAKING:

COMMUNICATING UNCERTAINTY
in a Soundbite is a guide from the Science Media Centre for
researchers doing brief news interviews about research.
www.sciencemediacentre.org/wp-content/uploads/2012/09/
Communicating-Uncertainty-in-a-Soundbite.pdf

CLIMATE SCIENCE, THE PUBLIC AND THE NEWS MEDIA

is a report looking at the communication of evidence, including
uncertainty about evidence, in the context of discussions about
climate science.
www.lwec.org.uk/publications/climate-science-public-and-news-media

REDUCING RISK OF FUTURE DISASTERS

is a report from the Government Office for Sciences Foresight
team that discusses policy making in relation to the risk of
natural disasters.
www.bis.gov.uk/foresight/our-work/policy-futures/disasters

POST NOTES
are short briefings for Parliamentarians on scientific subjects and
include many topical areas that deal with scientific uncertainty
and the complexities of integrating uncertain science with actual
policy decisions.
www.parliament.uk/mps-lords-and-offices/offices/bicameral/
post/publications/postnotes/

26

IF YOU ARE
a commentator, medical charity, journalist, employer,
educator, national body, information service, local
authority, parliamentarian, health care provider,
professional association, community
group, NGO, lifestyle writer
or any other civic group in need of
help or comment on a difficult or
controversial area of science call
Sense About Science on
020 7490 9590.

Sense About Science

Registered Charity No. 1146170
Company No. 6771027
14a Clerkenwell Green
London EC1R 0DP
Registered in England and Wales.
www.senseaboutscience.org
This document is prepared by Sense About Science, which
has final responsibility for the content.
It is licensed under Creative commons
Attribution-Noncommercial-No Derivative Works 2.0 UK:
England & Wales Licence.

www.peel-me.com

PUBLISHED IN 2013 BY

design

For more copies or further information,

contact Sense About Science on
[email protected]
or 020 7490 9590.