……………………………………………………………………………………………………………

F
o
r
e
w
o
r
MEASURING STUDENT KNOWLEDGE AND SKILLS d
The PISA 2000 Assessment of Reading, Mathematical
and Scientific Literacy

Foreword

How well are young adults prepared to meet the challenges of the future? Are they able to
analyse, reason and communicate their ideas effectively? Do they have the capacity to continue
learning throughout life? Parents, students, the public and those who run education systems
need to know the answers to these questions.

Many education systems monitor student learning to provide some answers to these questions.
Comparative international analyses can extend and enrich the national picture by providing a
larger context within which to interpret national results. They can provide direction for schools’
instructional efforts and for students’ learning as well as insights into curriculum strengths and
weaknesses. Coupled with appropriate incentives, they can motivate students to learn better,
teachers to teach better and schools to be more effective. They also provide tools to allow central
authorities to monitor achievement levels even when administration is devolved and schools are
run in partnership with communities.

For these reasons, governments and the general public need solid and internationally
comparable evidence on educational outcomes. In response to this demand, the OECD has launched
the Programme for International Student Assessment (PISA). PISA represents a new commitment
by the governments of OECD countries to monitor the outcomes of education systems in terms of
student achievement on a regular basis and within a common framework that is internationally
agreed. PISA aims at providing a new basis for policy dialogue and for collaboration in defining
and operationalising educational goals – in innovative ways that reflect judgements about the
skills that are relevant to adult life. It provides inputs for standard-setting and evaluation; insights
into the mix of factors which contribute to the development of competencies, and how these
factors operate similarly or differently among countries; and a better understanding of the causes
and consequences of observed skill gaps. In supporting a shift in policy focus from the inputs used
in education systems and institutions to learning outcomes, PISA can assist countries in seeking to

3
 ……………………………………………………………………………………………………………

F
o
r bring about improvements in schooling and better preparation for young people as they enter an
e adult life of rapid change and deepening global interdependence.
w
o
r PISA is a collaborative effort, bringing together scientific expertise from the participating countries,
d steered jointly by their governments on the basis of shared, policy-driven interests. Participating countries
take responsibility for the project at the policy level. Experts from participating countries also serve on
working groups that are charged with linking the PISA policy objectives with the best available
substantive and technical expertise in the field of international comparative assessment of educational
outcomes. Through participating in these expert groups, countries ensure that the PISA assessment
instruments are internationally valid and take into account the cultural and curricular contexts of
OECD Member countries; have strong measurement properties; and place an emphasis on authenticity
and educational validity. The frameworks and assessment instruments for PISA 2000 are the product
of a multi-year development process and were adopted in December 1999 by OECD governments,
through the Board of Participating Countries which manages the PISA project.

The PISA 2000 Assessment of Reading, Mathematical and Scientific Literacy is the second volume
in the PISA series Measuring Student Knowledge and Skills. It introduces the PISA approach to
comparative assessment and describes the PISA 2000 assessment instruments in terms of the
content that students need to acquire, the processes that need to be performed, and the contexts
in which knowledge and skills are applied. Each of the assessment domains is illustrated with a
range of sample items. It is published on the responsibility of the Secretary-General of the OECD.

Acknowledgements

This publication was prepared by the Statistics and Indicators Division of the OECD Directorate for
Education, Employment, Labour and Social Affairs, principally Andreas Schleicher and Claudia Tamassia. The
sample assessment materials presented in this publication were developed by international expert panels
and the test developers of the PISA consortium under the direction of Raymond Adams from the Australian
Council of Educational Research. The reading expert panel was chaired by Dr. Irwin Kirsch of the Education
Testing Service in the United States, the mathematics expert panel was chaired by Professor Jan de Lange
from the University of Utrecht, and the science expert panel was chaired by Professor Wynne Harlen from
the Scottish Council for Research in Education. Dr. Geoff Masters from the Australian Council for Educational
Research prepared the review of the PISA literacy concept in the publication. The members of the expert
panels and the PISA test developers are listed in the appendices.

4
……………………………………………………………………………………………………………

T
a
TABLE OF CONTENTS b
l
e
o
7 INTRODUCTION f
7 The Programme for International Student Assessment C
9 The choice of assessment domains o
9 Beyond content n
11 Three dimensions of literacy t
13 Knowledge and understanding e
15 Context of application n
16 PISA: a work in progress t
s
18 CHAPTER 1: ASSESSING READING LITERACY IN PISA
18 The PISA definition of reading literacy and its context
18 Three dimensions of reading literacy
18 Reading tasks
20 Types of text
22 Purposes of text
23 Format of test questions
23 Marking
23 Sample questions
Reading units:
24 UNIT 1 – BEES
27 UNIT 2 – IN POOR TASTE
29 UNIT 3 – A JUST JUDGE
36 UNIT 4 – BULLYING
39 UNIT 5 – MORELAND
40 UNIT 6 – WARRANTY
42 Scoring scheme for the reading sample questions

50 CHAPTER 2: ASSESSING MATHEMATICAL LITERACY IN PISA

50 The PISA definition of mathematical literacy and its context
50 Three dimensions of mathematical literacy
50 Mathematical processes
52 Mathematical content: strands and “big ideas”
54 Situations and contexts
54 Format of test questions and marking
55 Sample questions
Mathematics units:
56 UNIT 1 – PIZZAS
57 UNIT 2 – COINS
58 UNIT 3 – LICHEN
61 UNIT 4 – SHAPES
64 UNIT 5 – BRAKING
67 UNIT 6 – PATIO
68 UNIT 7 – SEALS SLEEP
70 Scoring scheme for the mathematics sample questions
5
 ……………………………………………………………………………………………………………

T
a
b
l
e
o
f 76 CHAPTER 3: ASSESSING SCIENTIFIC LITERACY IN PISA
C 76 The PISA definition of scientific literacy and its context
o 76 Three dimensions of scientific literacy
n 76 Scientific processes
t 78 Scientific concepts
e 78 Situations and areas of application
n
t 78 Format of test questions
s 79 Marking
79 Sample questions
Science units:
80 UNIT 1 – BUSES
82 UNIT 2 – FLIES
85 UNIT 3 – BIODIVERSITY
87 UNIT 4 – CLIMATE CHANGE
90 UNIT 5 – CHOCOLATE
92 UNIT 6 – CALF CLONING
94 Scoring scheme for the science sample questions

97 REFERENCES

103 Appendix 1 – Functional Expert Group Membership

104 Appendix 2 – Test Development Team

6
……………………………………………………………………………………………………………

I
n
t
r
o
d
u
Introduction c
t
i
The Programme for International Student Assessment o
n
The OECD’s Programme for International Student Assessment (PISA) is a new, regular survey of
15-year-olds which assesses aspects of their preparedness for adult life. The first round of the
survey is taking place in 32 countries during 2000, in collaboration between governments and
the OECD. Between 4 500 and 10 000 students will be surveyed in each country.

This publication explains the conceptual framework on which the PISA 2000 assessment is based
in terms of the content that students need to acquire, the processes that need to be performed, and
the contexts in which knowledge and skills are applied. It also illustrates how the assessment is to be
performed by reproducing sample assessment items and explaining what each is measuring.

PISA bases its assessment of student outcomes on three domains of “literacy” – reading literacy,
mathematical literacy and scientific literacy. In each case, international experts from OECD Member
countries have agreed on definitions of literacy and a framework for assessing it, using a common
set of principles (OECD, 1999).

The PISA concept of literacy is much wider than the historical notion of the ability to read and write.
Common usage now accepts that a literate person has a range of competencies. Moreover, it has come
to be accepted that there is no single cut-off point between a person who is fully literate and one who
is illiterate. Literacy is measured on a continuum, not as something that one either does or does not
have. In PISA, literacy is regarded as knowledge and skills for adult life. Its acquisition is a lifelong process
– taking place not just at school or through formal learning, but also through interaction with peers,
colleagues and wider communities. Fifteen-year-olds cannot be expected to have learned in school
everything that they will need to know as adults. They need a solid foundation of knowledge in areas
such as reading, mathematics and science. However, in order to go on learning in these domains and
to apply their learning to the real world, they need to understand some basic processes and principles,
and have the flexibility to use them in different situations. The three PISA domains of literacy therefore
emphasise the ability to undertake a number of fundamental processes in a range of situations, backed
by a broad understanding of key concepts, rather than the possession of specific knowledge.

The countries participating in the first OECD/PISA survey cycle are: Australia, Austria, Belgium,
Brazil, Canada, China, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary,
Iceland, Ireland, Italy, Japan, Korea, Latvia, Luxembourg, Mexico, the Netherlands, New Zealand,
Norway, Poland, Portugal, the Russian Federation, Spain, Sweden, Switzerland, the United Kingdom
and the United States.

7
 ……………………………………………………………………………………………………………

I
n
t
r BOX 1. WHAT IS OECD/PISA?
o
d A SUMMARY OF KEY FEATURES
u
c Basics
t • An internationally standardised assessment, jointly developed by participating countries
i and administered to 15-year-olds in groups in their schools.
o
n • Administered in 32 countries, of which 28 are members of the OECD.
• Between 4 500 and 10 000 students will typically be tested in each country.

Content
• PISA covers three domains: reading literacy, mathematical literacy and scientific literacy.
• PISA aims to define each domain not merely in terms of mastery of the school curriculum,
but in terms of important knowledge and skills needed in adult life. The assessment of
cross-curriculum competencies is an integral part of PISA.
• Emphasis is placed on the mastery of processes, the understanding of concepts and the
ability to function in various situations within each domain.

Methods
• Pencil and paper tests are used, with assessments lasting a total of 2 hours for each
student.
• Test items are a mixture of multiple-choice test items and questions requiring students to
construct their own responses. The items are organised in units based on a passage
setting out a real-life situation.
• A total of about 7 hours of test items is included, with different students taking different
combinations of the test items.
• Students answer a background questionnaire which takes 20-30 minutes to complete,
providing information about themselves. School principals are given a 30-minute
questionnaire asking about their schools.

Assessment cycle
• The first assessment is taking place in 2000, with first results published in 2001, and will
continue thereafter in three-year cycles.
• Each cycle looks in depth at a “major” domain, to which two-thirds of testing time are
devoted; the other two domains provide a summary profile of skills. Major domains are
reading literacy in 2000, mathematical literacy in 2003 and scientific literacy in 2006.

Outcomes
• A basic profile of knowledge and skills among students at the end of compulsory schooling.
• Contextual indicators relating results to student and school characteristics.
• Trend indicators showing how results change over time.
• A knowledge base for policy analysis and research.

8
……………………………………………………………………………………………………………

I
n
The choice of assessment domains t
r
It has long been accepted that an important objective of schooling is the development of a o
“literate” adult population. Historically, this objective has been synonymous with ensuring that all d
adults in a society are able to read and write. Literacy – defined as the ability to read and write – u
c
has been seen as essential to personal fulfilment; full adult participation in social, cultural and t
political life; personal empowerment; and success in securing and maintaining employment. i
o
The notion that schools have a responsibility to create a future society in which all adults are n
also mathematically “literate”, scientifically “literate”, and technologically “literate” is a relatively
recent idea. For much of the past century, the content of school mathematics and science curricula
was dominated by the need to provide foundations for the professional training of mathematicians,
scientists and engineers.

But with the increasing role of science, mathematics and technology in modern life, the objectives
of personal fulfilment, employment, and full participation in society increasingly require an adult
population which is not only able to read and write, but also mathematically, scientifically and
technologically literate. In the words of one commentator, “because of its pervasive role in modern
society, science is far too important to be left to the scientists only” (Zen, 1992, p. 20).

The imperatives in modern society to develop a citizenry which is mathematically, scientifically

and technologically “literate” are very similar to past arguments for achieving basic levels of adult
reading and writing competence: “A shared scientific background is part of a common cultural
base that binds civilised people together” (Zen, 1992); “Individuals who are deprived of the capacity
to make informed choices are rendered more vulnerable in matters of health and environment,
and are impaired in their ability to cope with an increasingly technological world” (Galbraith et al.,
1997); and basic mathematical and scientific literacy “renders individuals less dependent on others,
so that democratic processes, social values, and individual opportunities do not come to be
dominated by knowledgeable élites” (Krugly-Smolska, 1990).

Beyond content
A key feature of the broadened definition of “literacy” in PISA is a more explicit focus on the
knowledge, understanding and skills required for effective functioning in everyday life.

Literacy for effective participation in modern society requires mastery of a body of basic
knowledge and skills. For example, reading literacy depends on the ability to decode text, to
interpret meanings of words and grammatical structures, and to construct meaning at least at a
superficial level. But reading literacy for effective participation in modern society requires much
more than this: it also depends on the ability to read between the lines and to reflect on the
purposes and intended audiences of texts, to recognise devices used by writers to convey messages
and influence readers, and the ability to interpret meaning from the structures and features of
texts. Reading literacy depends on an ability to understand and interpret a wide variety of text
types, and to make sense of texts by relating them to the contexts in which they appear.

9
 ……………………………………………………………………………………………………………

I
n
t Mathematical literacy similarly depends on familiarity with a body of mathematical knowledge
r and skills which includes: basic number facts and operations; working with money; fundamental
o
d ideas about space and shape, including working with measurements; and notions of uncertainty,
u growth and change. But mathematical literacy for effective functioning in modern society requires
c much more than this: it also depends on the ability to think and work mathematically, including
t modelling and problem solving. These competencies include knowing the extent and limits of
i
o mathematical concepts; following and evaluating mathematical arguments; posing mathematical
n problems; choosing ways of representing mathematical situations; and expressing oneself on matters
with a mathematical content. Mathematical literacy depends on an ability to apply this knowledge,
this understanding and these skills in a wide variety of personal, social and work contexts.

Scientific literacy also depends on familiarity with a body of scientific knowledge and skills. This
body of knowledge includes an understanding of fundamental scientific concepts such as food
chains, sustainability, energy conservation, photosynthesis, rates of reaction, adaptation, states of
matter, and inheritance. But scientific literacy for effective functioning in modern society requires
much more than this: it also depends on the ability to use processes of scientific enquiry such as
recognising the nature and limits of such enquiry; identifying evidence required to answer scientific
questions; and drawing, evaluating and communicating conclusions. Scientific literacy depends on
an ability to apply this knowledge, this understanding and these skills in a wide variety of personal,
social, and work contexts.

OECD/PISA adopts definitions of reading literacy, mathematical literacy and scientific literacy
that go beyond mastery of essential knowledge and skills (Box 2). In all three domains, the focus
is not primarily on the mastery of specific curriculum content, but on the ability to reflect on and
use reading, mathematical and scientific knowledge, understanding and skills to achieve personal
goals and to participate effectively in society (OECD, 1999).

BOX 2. OECD/PISA DEFINITIONS OF LITERACY

Reading literacy
The capacity to understand, use and reflect on written texts, in order to achieve one’s
goals, to develop one’s knowledge and potential, and to participate in society.

Mathematical literacy
The capacity to identify, to understand, and to engage in mathematics and make well-
founded judgements about the role that mathematics plays, as needed for an individual’s
current and future private life, occupational life, social life with peers and relatives, and life
as a constructive, concerned, and reflective citizen.

Scientific literacy
The capacity to use scientific knowledge, to identify questions and to draw evidence-based
conclusions in order to understand and help make decisions about the natural world and
the changes made to it through human activity.

10
……………………………………………………………………………………………………………

I
n
Three dimensions of literacy t
r
In each domain, the assessment is organised in three dimensions, corresponding to process o
d
skills, knowledge and understanding, and the context of application. The remainder of this
u
chapter gives an overview of each dimension. Chapters 2 to 4 explain more precisely the c
criteria for assessment in each dimension within individual domains and illustrate these with t
sample items. i
o
n
It should be borne in mind that PISA 2000 gives more emphasis to reading literacy than to the
other two domains. PISA will take place every three years. In each cycle, two-thirds of testing time
will be devoted to assessing one domain in detail. Mathematical literacy will become the major
domain in 2003, and scientific literacy in 2006.

Processes

Reading literacy, mathematical literacy and scientific literacy require an understanding of and
facility with the methods and processes pertinent to each of these domains.

Beyond the ability to retrieve surface meaning from text, reading literacy requires an
understanding and appreciation of the writer’s craft, and an ability to reason about text. Readers
require an understanding of text structure, genre, and register. They must be able to follow chains
of reasoning; to compare and contrast information in a text; draw inferences; identify supporting
evidence; identify and understand irony, metaphor and humour; detect nuances and subtleties of
language; recognise ways in which texts are constructed to persuade and influence; and relate
what they read to their own background experience and knowledge.

Although surface understanding is important, it is not enough. In school and in society, we expect a
reader to be able to analyse, evaluate, and extend the ideas that are presented, just as we expect a writer
to elaborate upon and defend judgements that are expressed.We expect people to know how to get
information and to know how to use it and shape it to suit their needs. For example, readers must learn
to relate what they are reading to their personal experience in order to integrate new ideas with what
they know – perhaps modifying or rejecting the ideas in the process of considering them more fully.
Readers must also learn to test the validity of what they read by comparing it with information from
other sources, as well as to judge the internal logic and consistency of the ideas and information
presented (Applebee et al., 1987, p. 9).
Mathematical literacy similarly involves a range of process skills. The focus here is on students’
abilities to analyse, reason and communicate ideas effectively by posing, formulating and solving
mathematical problems. Mathematical literacy skills include thinking skills (e.g., distinguishing between
different kinds of mathematical statements); argumentation skills (e.g., following and evaluating chains
of mathematical arguments); modelling skills (e.g., translating “reality” into mathematical structures);
problem posing and solving skills; representation skills (e.g., distinguishing between different forms of
representation of mathematical situations); symbolic skills; technical skills (e.g., solving equations);
communication skills; and skills in using mathematical tools and aids.

11
 ……………………………………………………………………………………………………………

I
n
t [In the OECD/PISA definition of mathematics literacy], the term “engage in mathematics” includes
r communicating, taking positions towards, relating to, assessing, and even appreciating mathematics
o
d (OECD, 1999, p. 41).
u
c Scientific literacy depends on an ability to relate evidence or data to claims or conclusions. In
t particular, scientific literacy involves the processes of recognising scientifically investigable questions
i
(e.g., identifying the question or idea being tested, distinguishing questions that can be answered
o
n by scientific investigation from those which cannot); identifying evidence needed in a scientific
investigation (e.g., identifying and recognising what things should be compared, what variables
should be changed or controlled, and what additional information is required); drawing or
evaluating conclusions (e.g., producing a conclusion from a given set of evidence or data, and
identifying assumptions made in reaching a conclusion); and communicating valid conclusions
(e.g., producing an argument based on a situation or on data given, expressed in a manner that
is appropriate and clear to the intended audience).

The American Association for the Advancement of Science uses the term “scientific habits of
mind” in describing processes associated with the application of scientific, mathematical and
technological knowledge to everyday life. Individuals who have developed these habits of mind
should be able to:

(...) use the habits of mind and knowledge of science, mathematics and technology they have acquired
to think about and make sense of many of the ideas, claims, and events that they encounter in everyday
life. Accordingly, science literacy enhances the ability of a person to observe events perceptively, reflect
on them thoughtfully, and comprehend explanations offered for them. In addition, these internal per-
ceptions and reflections can provide the person with a basis for making decisions and taking action
(American Association for the Advancement of Science, 1993, p. 322).
Scientific literacy also includes an understanding of the methods by which science derives
evidence to support claims for scientific knowledge, and of the strengths and limitations of science
in the real world:

An appreciation of what science is and isn’t, of what it can be and cannot be, or of what scientists may
or may not claim for science, is surely an essential element of literacy (Zen, 1992, p. 19).

Another important element of literacy (reading, mathematical and scientific) in PISA is the
ability to stand apart from arguments, evidence or text, to reflect on these, and to evaluate and
criticise claims made. These skills go beyond analysis, problem solving and communication to
evaluation and critical reflection.

In the domain of reading literacy,

(...) reflecting on the content of a text requires that the reader connects information found in a text to
knowledge from other sources. Readers must also assess the claims made in the text against their own
knowledge of the world (...). Readers must be able to develop an understanding of what is said and
12
……………………………………………………………………………………………………………

I
n
intended in a text, and must test that mental representation against what they know and believe on the t
basis of either prior information, or information found in other texts. Readers must call on supporting r
o
evidence from within the text and contrast that with other sources of information, using both general d
and specific knowledge, and the ability to reason abstractly (OECD, 1999, p. 32). u
c
In the domain of scientific literacy, t
i
An important life skill (...) is the capacity to draw appropriate and guarded conclusions from evidence o
and information (...), to criticise claims made by others on the basis of the evidence put forward, and n
to distinguish opinion from evidence-based statements. Science has a particular part to play here since
it is concerned with rationality in testing ideas and theories against evidence from the world around
(OECD, 1999, p. 59).
A brief summary of some of the processes included in the OECD/PISA definitions of reading
literacy, mathematical literacy and scientific literacy is shown in Box 3.

BOX 3. OECD/PISA LITERACY PROCESSES

Reading literacy
“Performing different kinds of reading tasks, such as forming a broad general
understanding retrieving specific information, developing an interpretation or reflecting on
the content or form of the text.”

Mathematical literacy
“Mathematical competencies, e.g., modelling, problem solving; divided into three classes:
i) carrying out procedures; ii) making connection and integration for problem solving; and
iii) mathematisation, mathematical thinking and generalisation.”

Scientific literacy
“Process skills, e.g., recognising scientifically investigable questions, identifying evidence,
drawing, evaluating and communicating conclusions, and demonstrating understanding
of scientific concepts. These do not depend on a pre-set body of scientific knowledge, but
cannot be applied in the absence of scientific content.”

Knowledge and understanding

Literacy requires the development of a body of knowledge and understanding. Reading literacy
requires a knowledge of words, the ability to decode written text, and a knowledge of grammatical
structures. Mathematical literacy requires knowledge of mathematical facts, terms and concepts,
and an understanding of mathematical principles. Scientific literacy requires knowledge of scientific
facts, terms and concepts, and an understanding of scientific principles and laws.

13
 ……………………………………………………………………………………………………………

I
n
t As individuals’ levels of reading literacy, mathematical literacy and scientific literacy develop,
r they are able to draw on an increasingly rich store of knowledge and an increasingly deep
o
d understanding of principles within each domain. The development of knowledge and
u understanding specific to each domain is an important component of the development of
c literacy.
t
i
But literacy involves much more than the mastery of bodies of knowledge. It also involves
o
n an understanding of the methods, processes and limitations of a domain, and the ability to use
knowledge, understanding and skills in everyday contexts. For example,

[reading] literacy is not simply reading, but an ability to use print for personal and social ends. It is a
functional skill in that it requires the application of various skills in common everyday situations
(Venezky et al., 1987, p. 5).
OECD/PISA differs from some other assessment programmes in that it is not primarily an
assessment of the extent to which students have mastered bodies of knowledge and skills
identified in school curricula. It is not an assessment of achievement in school reading,
mathematics and science only. PISA recognises the necessity of curriculum-based knowledge
and understanding for reading literacy, mathematical literacy and scientific literacy (see Box 4),
but tests for these mainly in terms of the acquisition of broad concepts and skills that allow that
knowledge to be applied.

BOX 4. OECD/PISA LITERACY CONTENT

Reading literacy
Reading different kinds of text: continuous text classified by type (e.g., description,
narration, exposition, argumentation or instruction) and documents, classified by structure
(e.g., forms, calls and advertisements, charts and graphs or tables).

Mathematical literacy
Mathematical content: primarily mathematical “big ideas”. In the first cycle, these are
change and growth, and space and shape. In future cycles, chance, quantitative
reasoning, uncertainty and dependency relationships will also be used.

Scientific literacy
Scientific concepts: e.g., structure and properties of matter, chemical and physical
changes, energy transformations, forces and movement, form and function, human
biology, biodiversity or genetic control, chosen from the major fields of physics, biology,
chemistry, etc., and applied in matters to do with science in life and health, science in
Earth and environment, and science in technology.

14
……………………………………………………………………………………………………………

I
n
Context of application t
r
o
Finally, “literacy” incorporates an awareness and appreciation of the contexts in which texts are d
constructed, mathematics is used, and science operates, and an ability to apply the knowledge, u
c
understanding and skills specific to the appropriate domain to a wide range of contexts in the
t
world outside the classroom. This definition of literacy goes beyond the narrower conceptions of i
literacy of the 1970s (sometimes called “functional” or “survival” literacy). The focus of these earlier o
approaches was on minimal skills required to function in adult society. n

Reading literacy as it is defined today includes an understanding of the contexts in which

written texts are created, and the ability to use this contextual understanding to interpret and
reason about texts. Modern definitions also recognise that reading literacy plays a crucial role in
facilitating participation in a wide variety of social contexts. In the OECD/PISA definition,

“participate” includes social, cultural and political engagement. Participation may include a critical
stance, a step towards personal liberation, emancipation and empowerment.The term “society” includes
economic and political as well as social and cultural life (OECD, 1999, p. 21).

Modern definitions of mathematical and scientific literacy similarly emphasise the importance
of recognising and understanding the contexts in which mathematics and science operate and
the forces that shape these fields of human activity:

To ensure the scientific literacy of all students, curricula must present the scientific endeavour as a
social enterprise that strongly influences – and is influenced by – human thought and action (American
Association for the Advancement of Science, 1989, p. 5).

[Being literate] means having the intellectual skills to examine the pros and cons of any technological
development, to examine its potential benefits, and to perceive the underlying political and social
forces driving the development (Fleming, 1989).

Beyond an understanding of the contexts in which written texts occur and mathematics and
science are undertaken, and the way in which these interact with, influence, and are shaped by
context, “literacy” includes an ability to use knowledge, understanding and skills in the three
domains in varied contexts and for a range of purposes:

[Mathematical literacy] involves using mathematics to make sense of the world; to assist in dealing
with real situations which arise in the workplace, personal and community settings.While it necessarily
involves understanding mathematical ideas, notations and techniques, it also involves drawing on
knowledge of particular contexts and circumstances in deciding when to use mathematics, choosing the
mathematics to use, and critically evaluating its use (Cumming, 1997, p. 7).

In OECD/PISA, reading literacy, mathematical literacy and scientific literacy are assumed to
include the ability to apply processes and to use knowledge across a range of contexts:

15
 ……………………………………………………………………………………………………………

I
n
t • Reading literacy includes an ability to read a range of classroom materials and reading materials
r outside classrooms, including reading for personal use (personal letters, fiction, biography, etc.); public
o
d use (official documents, public information, etc.); employment; and education (textbooks, etc.).
u
c • Mathematical literacy includes an ability to apply mathematical knowledge, skills and
t understandings in “authentic” contexts. A context is considered authentic if it resides in the
i actual experiences and practices of the participants in a real-world setting. An important part
o
of the definition of mathematical literacy is doing and using mathematics in a variety of situations.
n
These situations include personal life, school life, work and sports (or leisure in general), local
community and society as encountered in daily life, and scientific contexts.

• Scientific literacy includes an ability to solve problems in real-world situations which can affect
us as individuals (e.g., food and energy use) or as members of a local community (e.g., treatment
of the water supply or siting of a power station) or as world citizens (e.g., global warming,
diminution of biodiversity). Contexts to which scientific literacy could be applied include self
and family (personal), community (public), life across the world (global), and the evolution of
scientific knowledge and its influence on social decisions (historical relevance).

Box 5 summarises some of these contexts.

BOX 5. OECD/PISA CONTEXTS OF LITERACY

Reading literacy
“Reading texts written for different situations, e.g., for personal interest or to meet work
requirements.”
Mathematical literacy
“Using mathematics in different situations, e.g., problems that affect individuals, communities
or the whole world.”

Scientific literacy
“Using science in different situations, e.g., problems that affect individuals, communities
or the whole world.”

PISA: a work in progress

The assessment being carried out in 2000 and illustrated in this publication represents the
fruition of a long collaborative process. PISA is a highly ambitious venture. Its aim is to make an
authentic assessment of how well students are equipped for the future, in a way that is valid
across many different cultures and languages. It has brought together scientific expertise from a
wide range of participating countries, and has worked in association with governments to produce
a survey that will be relevant for policy making. While the first survey will inevitably be able to fulfil
some PISA objectives better than others, the achievement so far has been to produce a robust
framework that can be developed further over the years.

16
1

R
E
A
D
I
N
G
L
I
T
E
R
A
C
Y
 ……………………………………………………………………………………………………………

R
e
a ASSESSING READING LITERACY IN PISA
d
i
n
g
The PISA definition of reading • “Process” – reading tasks: various tasks
L literacy and its context required of readers (such as retrieval of
i information or interpretation of text) used
t Reading literacy is defined in PISA as: in PISA to simulate the types of task that
e
r
students will encounter in real life;
a • Content – types of text: the form in
c
understanding, using, and reflecting
which written material is encountered and
y on written texts, in order to achieve
needs to be understood (many and varied
one’s goals, to develop one’s knowledge forms such as narrative prose, or graphic
and potential, and to participate in presentation), a wide selection of which is
society. represented in the PISA assessment items;
• Context – purpose of text: the situation
Definitions of reading and of reading literacy in which reading takes place (e.g. for private
have changed over time in parallel with social, or occupational use), defined in PISA
economic and cultural changes. Literacy is no according to how the author intended the
longer considered simply the ability to read and text to be used. PISA items are designed to
write. It is viewed as an advancing set of relate to a variety of such contexts.
knowledge, skills, and strategies, which
individuals build on throughout life. Reading tasks
PISA assesses students’ ability to perform a
The PISA definition therefore goes beyond variety of reading tasks. It aims to simulate the
the notion that reading literacy means decoding kinds of task encountered in “authentic” reading
written material and literal comprehension. situations – i.e. in real life. To this end, the
Reading incorporates understanding and assessment measures five aspects of understanding
reflecting on texts. Literacy involves the ability a text. It is expected that all readers, irrespective
of individuals to use written information to fulfil of their overall proficiency, will be able to
their goals, and of complex modern societies demonstrate some level of competency in each
to use written information to function aspect.While there is an interrelationship among
effectively. PISA 2000 employs some 140 the five aspects, since each may require many of
reading literacy items that aim to represent the the same underlying skills, successfully
kinds of literacy that 15-year-olds will face in accomplishing one may not be dependent upon
their future lives. successfully completing any other.The five aspects
of reading assessed in PISA are:
Three dimensions of reading literacy
1. Forming a broad general understanding
Readers respond to a given text in a variety
of ways as they seek to use and understand what This requires the reader to consider the text
they are reading.This dynamic process has many as a whole or in a broad perspective. Students
dimensions, three of which are used to may be asked, for example, to demonstrate
construct the PISA assessment: initial understanding by identifying the main
18
……………………………………………………………………………………………………………

R
e
READING LITERACY a
d
Use information primarily from within the text Drawn upon outside knowledge i
n
g
Consider text Focus on Content Structure
as a whole specific parts of text L
i
t
Independent Understanding
e
pieces of of
r
information relationships
a
c
Broad Retrieve Develop an Reflect on Reflect on y
understanding information interpretation content of text form of text

topic of the text, to explain the purpose of a the new information asked for. This may
map or graph, to match a piece of text to a require discriminating between two similar
question about the broad purpose of the text, pieces of information. By systematically
or to focus on more than one specific reference varying the elements that contribute to
in the text and to deduce the theme from the difficulty, measurement of various levels of
repetition of a particular category of proficient performance associated with this
information. Selecting the main idea implies aspect of comprehension can be achieved.
establishing a hierarchy among ideas and
choosing the most general and overarching 3. Developing an interpretation
ones. Such a task indicates whether the student
This requires readers to extend their initial
can distinguish between key ideas and minor
impressions, processing information in a
details, or can recognise the summary of the
logical manner so that they develop a more
main theme in a sentence or title.
specific or complete understanding of what
they have read. Tasks that might be used to
2. Retrieving information
assess this aspect include comparing and
In the course of daily life, readers often need contrasting information – by integrating two
a particular piece of information. They may or more pieces of information from the text,
need to look up a telephone number. They drawing inferences about the relationship
may want to check the departure time for a between different sources of information, and
bus or train. They may want to find a identifying and listing supporting evidence in
particular fact to support or refute a claim order to infer the author’s intent.
someone has made. To retrieve effectively,
readers must scan, search, locate, and select
4. Reflecting on the content of a text
relevant information. In assessment tasks
that call for retrieving information, students This requires the reader to relate
have to identify the essential elements of a information found in a text to knowledge
message: characters, time, setting, etc.They from other sources. Readers must assess the
must match information given in the claims made in the text against their own
question with either literal or synonymous knowledge of the world or against
information in the text and use this to find information found in other texts in the
19
 ……………………………………………………………………………………………………………

R
e
a assessment or explicitly provided in the Table 1 shows the distribution of reading
d question. In many situations, readers must literacy tasks in the PISA 2000 assessment by
i
know how to justify their own point of view. each of the five aspects defined above. The first
n
g Typical assessment tasks include providing three aspects, which look at how well a student
evidence or arguments from outside the can understand and use information within a
L text, assessing the relevance of particular text, account for 70 per cent of the reading
i pieces of information or evidence, drawing literacy assessment.The remaining tasks require
t comparisons with moral or aesthetic rules wider reflection.
e
(standards), identifying information that
r
a might strengthen the author’s argument, and Types of text
c evaluating the sufficiency of the evidence or
y information provided in the text. At the heart of the organisation of the PISA
reading assessment is a distinction between
5. Reflecting on the form of a text continuous and non-continuous texts. Continuous
texts are typically composed of sentences that
Tasks in this category require readers to stand are, in turn, organised into paragraphs. These
apart from the text, consider it objectively, may be fit into even larger structures such as
and evaluate its quality and appropriateness. sections, chapters, and books. Non-continuous
Knowledge of such things as text structure, texts, or documents as they are known in some
genre and register play an important role in approaches, present information in a variety of
these tasks. Students are required to detect different ways, such as forms, graphs and maps.
nuances in language – for example,
understanding when the choice of an adjective Continuous texts
might colour interpretation.Assessment tasks
include determining the utility of a particular Continuous text types are all in a standard
text for a specified purpose, evaluating an “prose” form. They are classified according to
author’s use of certain textual features in the author’s purpose, with the following five
accomplishing a particular goal, and types used in PISA:
identifying or commenting on the author’s use 1. Description refers to properties of objects
of style and what the author’s purpose and in space, and typically provides an answer
attitude are. to “what” questions.
Table 1. Distribution of reading tasks 2. Narration refers to properties of objects
by aspects of reading literacy in time and typically provides answers to
“when”, or “in what sequence” questions.
Aspect % of PISA reading
literacy assessment 3. Exposition presents information as
composite concepts or mental constructs,
Retrieving information 20 or those elements into which concepts or
Broad understanding 20 mental constructs can be analysed. The
text provides an explanation of how the
Developing an interpretation 30 component elements interrelate in a
Reflecting on content 15 meaningful whole and often answers
“how” questions.
Reflecting on form 15
4. Argumentation presents propositions as to
TOTAL 100
the relationship among concepts, or other
20
……………………………………………………………………………………………………………

R
e
propositions.Argument texts often answer (how to) from process (how something a
“why” questions. works) diagrams. d
i
5. Instruction (sometimes called injunction) 5. Tables and matrices. Tables are row and n
provides directions on what to do and column matrices. Typically, all the entries g
includes procedures, rules, regulations in each column and each row share
and statutes specifying certain behaviours. L
properties, and thus the column and row i
Non-continuous texts labels are part of the information structure t
of the text. Common tables include e
Unlike continuous texts, non-continuous timetables, spreadsheets, order forms and r
ones vary in form, and are thus classified indexes. a
c
according to their structure rather than the y
author’s purpose. The following six types are 6. Maps are non-continuous texts that indicate
used in PISA: the geographic relationships between places.
There is a variety of types of maps. Road
1. Forms are structured and formatted texts maps mark the distances and routes between
which request the reader to respond to identified places.Thematic maps indicate the
specific questions in specified ways.Typical relationships between locations and social
examples are tax forms, immigration or physical features.
forms, visa forms, application forms,
statistical questionnaires, etc. Table 2
2. Calls and advertisements are documents Distribution of reading tasks by types of text
designed to invite the reader to do some- Text type % of PISA reading
thing, e.g., to buy goods or services, attend literacy assessment
gatherings or meetings, elect a person to a Narative 13
public office, etc.The purpose of these docu-
ments is to persuade the reader.They offer Expository 22
something and request attention and action
Descriptive 13
at the same time. Advertisements, invita-
tions, summonses, warnings and notices are Argumentative/Persuasive 13
examples of this document format.
Injunctive 5
3. Charts and graphs are iconic representations TOTAL CONTINOUS TEXTS 66
of data. They are used for the purposes of
scientific argumentation, and also in jour- Charts/Graphs 11
nals and newspapers to display numerical
and tabular public information in a visual Tables 11
format. Diagrams 3
4. Diagrams often accompany technical Maps 3
descriptions (e.g., demonstrating parts of
a household appliance), expository texts, Forms 3
and instructive texts (e.g., illustrating how Advertisements 2
to assemble a household appliance). It is
often useful to distinguish procedural TOTAL NON-CONTINOUS TEXTS 33

21
 ……………………………………………………………………………………………………………

R
e
a The distribution and variety of texts that both practical and intellectual. It also
d students are asked to read for PISA, shown in includes reading to maintain, or develop,
i
Table 2, is an important characteristic of the personal connections to other people.
n
g assessment. Continuous texts represent about Contents typically include personal
two-thirds of the texts contained in the letters, fiction, biography, and
L assessment, of which the largest single category informational texts read for curiosity, as a
i is expository materials. Two-thirds of non- part of leisure or recreational activities.
t continuous texts are tables or charts and graphs.
e 2. Reading for public use is carried out to
The remaining non-continuous texts are maps,
r participate in the activities of the larger
a advertisements, and forms of the type that 15-
c year-olds are expected to be able to read and use. society. It includes the use of official
y documents as well as information about
Purposes of text public events. In general, these tasks are
associated with more or less anonymous
PISA distinguishes four types of reading contact with others.
situation: reading for private use, reading for
public use, reading for work, and reading for 3. Reading for work (occupational) may not
education. yet be required by most 15-year-olds,
but there are two important reasons to
While it is the intent of the PISA reading include such situations in PISA. First,
literacy assessment to measure the kinds of reading in such situations is usually
reading that occur both within and outside closely associated with the accomplish-
classrooms, the manner in which reading ment of some immediate task. Secondly,
situations are defined cannot be based simply some reading abilities will help to equip
on where reading takes place. For example, students for the world of work into
textbooks are read both in schools and in homes, which the PISA target population will
and the processes and purposes of reading these shortly move.
texts differ little from one setting to another.
4. Reading for education, or reading to learn,
In PISA, the reading situations can therefore is normally involved with acquiring
be understood as a general categorisation of information as part of a larger learning
texts based on their intended use, on implicit task. The materials are often not chosen
or explicit relations to others, and on the by the reader, but assigned by a teacher.
general contents. Close attention has therefore The content is usually designed specifically
been paid to the origin and content of texts. for the purpose of instruction.
The goal is to reach a balance between reflecting
the broad definition of reading literacy used in Table 3 shows the distribution of reading
PISA and representing the linguistic and cultural literacy tasks in the assessment across all four
diversity of participating countries. This situations. The occupational situation is given
diversity will help to ensure that no one group less weight because of the likelihood that 15-
is either advantaged or disadvantaged by the year-olds are relatively unfamiliar with this
assessment content. category of text. It is also important to reduce
the potential dependence on specific
1. Reading for private use (personal) is carried occupational knowledge that can result when
out to satisfy an individual’s own interests, occupational texts are selected.
22
……………………………………………………………………………………………………………

R
e
Table 3 obscure the meaning because this assessment is a
Distribution of reading literacy tasks not seen as a test of written expression.The PISA d
by purpose of text i
marking scheme for open-ended items is shown
n
at the end of this chapter. g
Purpose of text % of PISA reading
literacy assessment L
Sample questions
i
Personal 28 The following examples illustrate the range t
e
Educational 28 of tasks and questions used in the PISA r
assessment of reading literacy.These items were a
Occupational 16 used in the PISA field trial but not selected for c
Public 28 the PISA 2000 assessment because of their y
similarity to other item sets in terms of what
TOTAL 100 they measure.The intention in presenting these
tasks and questions is to demonstrate the
connection between the PISA assessment
framework and the items that have been
Format of test questions constructed to represent it.
Just over half of the reading literacy assessment The results from the field trial is indicative only
(55 per cent) will be based on multiple-choice items since they were not based on a probability sample.
and other questions with more or less specified They are based on a judgement sample drawn from
answers requiring little subjective judgement on 32 countries which were given equal weight.
the part of the marker.The rest, 45 per cent, will
require students to construct their own answers Since the PISA 2000 assessment was not
and markers to exercise some judgement in completed by the time this publication was
assessing them. This latter category will account produced, items from the PISA 2000 assessment
for 65 per cent of tasks requiring reflection on could not be included in this publication for
content and form, but only 35 per cent of the other reasons of test security.
three categories of reading task.
Questions use both multiple-choice and
Marking open-ended formats. Not all the questions
relating to each unit are given here, the aim
While multiple-choice items have either a right being to provide a cross-section only. The kinds
or a wrong answer, partial credit models allow of continuous texts shown in Unit 1-4 account
for more complex marking of other items. for about two-thirds of the materials used in
Psychometric models for such marking are well the main assessment. Approximately one-third
established and in some ways are preferable to the of the texts are non-continuous, exemplified
“right or wrong” approach as they make use of here by Units 5 and 6.
more of the information that is in the answers.
Partial credit marking will be used for at least some
of the more complex items in which students
construct their own answers. It is important to
note that markers are advised to ignore spelling
and grammar errors, unless they completely ……………
23
 ……………………………………………………………………………………………………………

R
e
a READING UNIT 1
d
i BEES
n
g
The following information is from a booklet about bees. Refer to the information to answer
L the questions below.
i
t
e
r COLLECTING NECTAR
a
c Bees make honey to survive. It is their only essential food. If there are 60,000 bees in
y a hive about one third of them will be involved in gathering nectar which is then made into
honey by the house bees. A small number of bees work as foragers or searchers. They find
a source of nectar, then return to the hive to tell the other bees where it is.

Foragers let the other bees know where the source of the nectar is by performing a dance
which gives information about the direction and the distance the bees will need to fly. During
this dance the bee shakes her abdomen from side to side while running in circles in the shape
of figure 8. The dance follows the pattern shown on the following diagram.

The diagram shows a bee dancing inside the hive on the vertical face of the honeycomb.
If the middle part of the figure 8 points straight up it means that bees can find the food if
they fly straight towards the sun. If the middle part of the figure 8 points to the right, the
food is to the right of the sun.

The distance of the food from the hive is indicated by the length of time that the bee
shakes her abdomen. If the food is quite near the bee shakes her abdomen for a short time.
If it is a long way away she shakes her abdomen for a long time.

24
……………………………………………………………………………………………………………

R
e
a
d
i
n
g

L
MAKING HONEY i
t
When the bees arrive at the hive carrying nectar they give this to the house bees. The e
house bees move the nectar around with their mandibles, exposing it to the warm dry air r
of the hive. When it is first gathered the nectar contains sugar and minerals mixed with a
about 80% water. After ten to twenty minutes, when much of the excess water has c
evaporated, the house bees put the nectar in a cell in the honeycomb where evaporation y
continues. After three days, the honey in the cells contains about 20% water. At this stage,
the bees cover the cells with lids which they make out of beeswax.

At any one time the bees in a hive usually gather nectar from the same type of blossom
and from the same area. Some of the main sources of nectar are fruit trees, clover and
flowering trees.

GLOSSARY The text in Unit 1 is from an

house bee a worker bee which works inside the hive educational context and is an
mandible mouth-part example of a descriptive-explanatory
text because it characterises the
Reproduced from “Hum Sweet Hum”. food-gathering behaviour of bees. It
© National Foundation for Educational Research 1993. is also an example of a text that
contains both continuous and non-
continuous information in that it
includes a diagram which makes a
major contribution to explaining
Sample Question 1 (Multiple Choice) the dance of the forager bees.

• Aspect: Forming a broad understanding

• Text type: Continuous (description)
• Situation: Educational

What is the purpose of the bees’ dance?

A To celebrate the successful production of honey.
B To indicate the type of plant the foragers have found.
C To celebrate the birth of a new Queen Bee.
D To indicate where the foragers have found food.

Ninety-one per cent of all students participating in the field trial were able to select D as the
correct answer to Sample Question 1.

25
 ……………………………………………………………………………………………………………

R
e
a Sample Question 2 (Multiple Choice)
d
i • Aspect: Developing an interpretation
n • Text type: Continuous (description)
g • Situation: Educational
L
i
t
What is the main difference between nectar and honey?
e A The proportion of water in the substance.
r B The proportion of sugar to minerals in the substance.
a
c C The type of plant from which the substance is gathered.
y D The type of bee which processes the substance.

Sample Question 2 appears to be slightly more difficult, since only 72 per cent of all
students in the PISA field trial answered it correctly. It requires students to develop an
interpretation by following the flow of ideas presented in the paragraph under heading
“Making Honey.”

Sample Question 3 (Open-Constructed Response)

• Aspect: Developing an interpretation
• Text type: Continuous (description)
• Situation: Educational

In the dance, what does the bee do to show how far the food is from
the hive?

In Sample Question 3, readers again need to be able to follow the flow of ideas presented in a
part of the text. In this case it is the paragraph after the diagram beginning “The distance of
the food ...”. Students were required to state two things: the shaking of the abdomen and the
length of time.This question was answered correctly (with full credit) by only 48 per cent of
all students. A further 30 per cent responded by stating only one of these facts, and obtained
partial credit.The question would have been easier if it had instructed students to list two
things stated in the text which bees do in order to indicate how far food is from the hive.
Perhaps because of its current wording, only the more careful readers responded with both
pieces of information.

26
……………………………………………………………………………………………………………

R
e
Sample Question 4 (Closed-Constructed Response) a
d
• Aspect: Retrieving information i
• Text type: Continuous (description) n
• Situation: Educational g

L
Write down three of the main sources of nectar. i
1. ______________________ t
e
2. ______________________ r
3. ______________________ a
c
y
In order to respond correctly to Sample Question 4, students need to locate the last
sentence in the text that states: “Some of the main sources of nectar are fruit trees, clover
and flowering trees.” Given that the information is set out in the text and that the
question specifically asks for three sources, it is somewhat surprising that only 66 per cent
of all students answered correctly. As the question makes no specific reference to the text, it is
possible that students attempted to use their own background knowledge to answer this
question rather than locating the information stated in the last sentence of the text.

READING UNIT 2
IN POOR TASTE

The following letter appeared in a newspaper in 1997. Refer to the letter to answer the
questions below.

IN POOR TASTE
from Arnold Jago

Did you know that in 1996 we spent almost the same amount on chocolate as our
government spent on overseas aid to help the poor?
Could there be something wrong with our priorities?
What are you going to do about it?
Yes, you. The text shown in Unit 2 is one of the shortest
continuous or prose texts in the initial item
Arnold Jago, pool. It is a letter entitled “In Poor Taste” which
Mildura appeared in a newspaper, and it is classed as
an argumentative text because the writer is
Source: The Age, 1st April, 1997. attempting to persuade the reader of his view.
It is drawn from a public context.
27
 ……………………………………………………………………………………………………………

R
e
a Sample Question 5 (Multiple Choice)
d
i • Aspect: Developing an interpretation
n • Text type: Continuous (description)
g • Situation: Public
L
i
Arnold Jago’s aim in the letter is to provoke
t A guilt.
e B amusement.
r
a C fear.
c D satisfaction.
y
Sample Question 5 is a multiple-choice question which was answered correctly by, on average,
81 per cent of all students in the countries participating in the PISA field trial.This letter is
classified as an argumentative text because the writer was attempting to persuade the reader
and it is thought to be a public text since reading it is typically associated with participation
in society at large and also because it deals with information involving public events. It is
designed to discover whether students can understand the way in which the information in the
letter is organised.To this end, students are asked to use the information stated in the letter to
make an inference about the author’s intent or purpose.They are not asked to judge whether
the letter fulfils that intent or purpose (had the question asked the students to consider the
utility of the letter in meeting the author’s goal or intent, then it would have been classified
as reflecting on the form of a text).

Sample Question 6 (Open-Constructed Response)

• Aspect: Reflecting on the content
• Text type: Continuous (argumentation)
• Situation: Public

What kind of response or action do you think Arnold Jago would like
his letter to prompt?

Sample Question 6 proved more difficult, being correctly answered by only 63 per cent of all
students in the PISA field trial. It assesses a different aspect of reading, requiring students to
go beyond the information provided in the letter and to reflect on the content.That is, they
are not given credit for a correct response if they only state that Jago wants them to spend less
on chocolate or to be less greedy. In reflecting on what they read, students need to bring their
outside knowledge to bear on their understanding of the letter and to provide a statement or
sentence indicating that government or individuals should provide larger amounts of overseas
aid or that there should be a shift in priorities.
28
……………………………………………………………………………………………………………

R
e
How did the students answer? Examples (See end of chapter for marking scheme) a
d
i
• “People spending less money on chocolate and more on the overseas sick.” (Score 1 [a])
n
• “That people don’t spend all their money on chocolate rather than overseas.” (Score 1 [a]) g
• “An increase in the spending of the government towards overseas aid to help the poor.” (Score 1 [a]) L
• “People instead of buying and eating chocolate should give to a good cause and not be so self- i
t
indulgent.” (Score 1 [a]) e
• “People stirred up to think more of helping others than indulging in personal pleasures.” r
(Score 1 [b]) a
c
• “People’s awareness that the poor need our help, for people to do something about it.” y
(Score 1 [b])
• “I think he would like letters with written opinions and what they should do to help this problem.”
(Score 0 [e])
• “He may like to see people’s suggestions on how to fund raise for overseas using chocolate.”
(Score 0)
[a], [b] and [e] refer to the scoring guide at the end of this chapter.

READING UNIT 3
A JUST JUDGE
Refer to the following story “A Just Judge” to answer the questions below.

A JUST JUDGE
An Algerian king named Bauakas wanted to find out whether or not it was true, as he
had been told, that in one of his cities lived a just judge who could instantly discern the
truth, and from whom no rogue was ever able to conceal himself. Bauakas exchanged
clothes with a merchant and went on horseback to the city where the judge lived.

At the entrance to the city a cripple approached the king and begged alms of him.
Bauakas gave him money and was about to continue on his way, but the cripple clung to his
clothing.

“What do you wish?” asked the king. “Haven’t I given you money?”

“You gave me alms,” said the cripple, “now grant me one favour. Let me ride with you as
far as the city square, otherwise the horses and camels may trample me.”

29
 ……………………………………………………………………………………………………………

R
e
a
d Bauakas sat the cripple behind him on the horse and took him as far as the city square.
i There he halted his horse, but the cripple refused to dismount.
n
g “We have arrived at the square, why don’t you get off?” asked Bauakas.
L
i “Why should I?” the beggar replied. “This horse belongs to me. If you are unwilling to
t return it, we shall have to go to court.”
e
r Hearing their quarrel, people gathered around them shouting:
a
c “Go to the judge! He will decide between you!”
y
Bauakas and the cripple went to the judge. There were others in court, and the judge
called upon each one in turn. Before he came to Bauakas and the cripple he heard a scholar
and a peasant. They had come to court over a woman: the peasant said she was his wife, and
the scholar said she was his. The judge heard them both, remained silent for a moment, and
then said:

“Leave the woman here with me, and come back tomorrow.”

When they had gone, a butcher and an oil merchant came before the judge. The butcher
was covered with blood, and the oil merchant with oil. In his hand the butcher held some
money, and the oil merchant held onto the butcher’s hand.

“I was buying oil from this man,” the butcher said, “and when I took out my purse to pay
him, he seized me by the hand and tried to take all my money away from me. That is why we
have come to you – I holding onto my purse, and he holding onto my hand. But the money is
mine, and he is a thief.”

Then the oil merchant spoke. “That is not true,” he said. “The butcher came to me to buy
oil, and after I had poured him a full jug, he asked me to change a gold piece for him. When
I took out my money and placed it on a bench, he seized it and tried to run off. I caught him
by the hand, as you see, and brought him here to you.”

The judge remained silent for a moment, then said: “Leave the money here with me, and
come back tomorrow.”

When his turn came, Bauakas told what had happened. The judge listened to him, and
then asked the beggar to speak.

“All that he said is untrue,” said the beggar. “He was sitting on the ground, and as I rode
through the city he asked me to let him ride with me. I sat him on my horse and took him
where he wanted to go. But when we got there he refused to get off and said that the horse
was his, which is not true.”

30
……………………………………………………………………………………………………………

R
e
a
The judge thought for a moment, then said, “Leave the horse here with me, and come d
back tomorrow.” i
n
g
The following day many people gathered in court to hear the judge’s decisions.
L
First came the scholar and the peasant.
i
“Take your wife,” the judge said to the scholar, “and the peasant shall be given fifty strokes t
e
of the lash.” r
a
The scholar took his wife, and the peasant was given his punishment. c
y
Then the judge called the butcher.
“The money is yours,” he said to him. And pointing to the oil merchant he said: “Give
him fifty strokes of the lash.”
He next called Bauakas and the cripple.
“Would you be able to recognise your horse among twenty others?” he asked Bauakas.
“I would,” he replied.
“And you?” he asked the cripple.

“I would,” said the cripple.

“Come with me,” the judge said to Bauakas.
They went to the stable. Bauakas instantly pointed out his horse among the twenty others.
Then the judge called the cripple to the stable and told him to point out the horse. The
cripple recognised the horse and pointed to it. The judge then returned to his seat.

“Take the horse, it is yours,” he said to Bauakas. “Give the beggar fifty strokes of the lash.”

When the judge left the court and went home, Bauakas followed him.

“What do you want?” asked the judge. “Are you not satisfied with my decision?”

“I am satisfied,” said Bauakas. “But I should like to learn how you knew that the woman
was the wife of the scholar, that the money belonged to the butcher, and that the horse was
mine and not the beggar’s.”

“This is how I knew about the woman: in the morning I sent for her and said: ‘Please fill
my inkwell.’ She took the inkwell, washed it quickly and deftly, and filled it with ink; therefore
it was work she was accustomed to. If she had been the wife of the peasant she would not
have known how to do it. This showed me that the scholar was telling the truth.

31
 ……………………………………………………………………………………………………………

R
e
a
d “And this is how I knew about the money: I put it into a cup full of water, and in the
i morning I looked to see if any oil had risen to the surface. If the money had belonged to the
n oil merchant it would have been soiled by his oily hands. There was no oil on the water;
g
therefore, the butcher was telling the truth.
L
i “It was more difficult to find out about the horse.The cripple recognised it among twenty
t others, even as you did. However, I did not take you both to the stable to see which of you
e knew the horse, but to see which of you the horse knew. When you approached it, it turned
r its head and stretched its neck toward you; but when the cripple touched it, it laid back its
a ears and lifted one hoof. Therefore I knew that you were the horse’s real master.”
c
y Then Bauakas said to the judge: “I am
not a merchant, but King Bauakas, I came A third example of a continuous text (Unit 3) is a
here in order to see if what is said of you narrative entitled “A Just Judge”.This story depicts
is true. I see now that you are a wise judge. a sequence of events beginning with an Algerian
Ask whatever you wish of me, and you king named Bauakas who wants to learn whether
shall have it as reward.” it is true that a just judge who is able to discern
truth lives in one of his cities.Whereas “In Poor Taste”
“I need no reward,” replied the judge. (Unit 2) is an example of a short text,“A Just Judge”
“I am content that my king has praised me.” is an example of a long text that students are
expected to be able to read and understand. Although
Source: “A Just Judge” by Leo Tolstoy from Fable 15-year-olds might typically encounter such a text
and Fairytales, translated by Ann Dunningan. in a classroom, it is classified as a personal rather
than educational text, since it is a piece of fiction
composed for personal rather than for formal
Sample Question 7 instructional purposes.
(Multiple Choice)

• Aspect: Reflecting on the content

• Text type: Continuous (narration)
• Situation: Personal/educational

How did the judge know that the woman was the wife of the scholar?
A By observing her appearance, and seeing that she did not look like a
peasant’s wife.
B By the way the scholar and the peasant told their stories in court.
C By the way she reacted to the peasant and the scholar in court.
D By testing her skill in work that she needed to perform for her
husband.

Sample Question 7 is the easiest question associated with this text (answered correctly by 82 per
cent of students in the PISA field trial).The correct answer, D, is very similar to the statement in
the text, which reads,“... therefore it was work she was accustomed to.”

32
……………………………………………………………………………………………………………

R
e
Sample Question 8 (Multiple Choice) a
d
• Aspect: Developing an interpretation i
• Text type: Continuous (narration) n
• Situation: Personal/educational g

L
Why didn’t Bauakas want to be recognised? i
t
A He wanted to see if he would still be obeyed when he was an e
“ordinary” person. r
B He planned to appear in a case before the judge, disguised as a a
c
merchant. y
C He enjoyed disguising himself so he could move about freely and play
tricks on his subjects.
D He wanted to see the judge at work in his usual way, uninfluenced by
the presence of the king.

In Sample Question 8, which is somewhat more difficult, the text states only that the king
exchanged clothes with a merchant and went on horseback to where the judge lived.The
reader must infer the motive for this behaviour from the surrounding text. Seventy per cent of
the students assessed in the PISA field trial selected alternative D, the correct answer to this
question.

Sample Question 9 (Multiple Choice)

• Aspect: Forming a broad understanding
• Text type: Continuous (narration)
• Situation: Personal/educational

What is this story mainly about?

A Major crimes.
B Wise justice.
C A good ruler.
D A clever trick.

Sample Question 9, of similar difficulty to Sample Question 8, asks students to form a broad
understanding of the text by identifying the theme or main idea from a list of alternatives.
Seventy-two per cent of all students answered it correctly by selecting alternative B.

33
 ……………………………………………………………………………………………………………

R
e
a Sample Question 10 (Multiple Choice)
d
i • Aspect: Reflecting on the form
n • Text type: Continuous (narration)
g • Situation: Personal/educational

L Which one of the following best describes the story?

i
t A A folk tale
e B A travel story
r
a
C A historical account
c D A tragedy
y E A comedy

Sample Question 10, which students in the PISA field trial found more difficult, asks students
to show that they understand not only the content of the story but also something about its
style and structure. Sixty-four per cent of all students were able to recognise the rhetorical
structure of this narrative as a folk tale.

Sample Question 11 (Open-Constructed Response)

• Aspect: Reflecting on the content
• Text type: Continuous (narration)
• Situation: Personal/educational

Do you think it was fair of the judge to give the same punishment for
all of the crimes? Explain your answer, referring to similarities or
differences between the three crimes.

A full response to Sample Question 11 requires readers to demonstrate an understanding of the

crimes and to support their opinion with some reflective remarks.This calls for more complex
reflection on the text and the author’s message conveyed by the story. For example, one type of
correct answer might be:“In all three crimes one person wanted to cheat another, so it is fair that
they were punished in the same way.” Another would be: “No, the crimes are not all equal; it is
much more serious to want to steal someone’s wife than it is to steal their money or their horse.”

How did the students answer? Examples (See end of chapter for marking scheme)
• “No, some of the crimes were worse than others.” (Score 1)
• “Yes, they all lied.” (Score 1)
• “I don’t think it was fair to have the same punishment because they were all different cases.” (Score 0)
• “No, because there were different circumstances involved.” (Score 0)
• “Yes, all the three cases had a good and bad person, the so-called baddie should’ve been punished
for doing the wrong thing.” (Score 0)

34
……………………………………………………………………………………………………………

R
e
Sample Question 12 (Open-Constructed Response) a
d
• Aspect: Reflecting on the content i
• Text type: Continuous (narration) n
• Situation: Personal/educational g
For this question you need to compare law and justice in your country L
with the law and justice shown in the story. i
t
Question 12A: e
In the story crimes are punished under the law. What is another way in r
a
which law and justice in your country are SIMILAR to the kind of law c
and justice shown in this story. y
Question 12B:
In the story the judge gives fifty strokes of the lash for all the
crimes. Apart from the kind of punishment, what is one way in which law
and justice in your country are DIFFERENT to the kind of law and
justice shown in this story?

In Sample Question 12, students again need to show that they both understand the story and are
able to use their outside knowledge – this time to suggest features that are similar and different in
their own legal systems. For example, a correct answer stating a similarity would be: “Both sides
are allowed to give their version of the truth.” Another might be:“The same punishment is often
given out for similar crimes.” An incorrect answer might include a vague or inaccurate reference to
the story, or be irrelevant. An example would be:“Even important rulers of countries can be
brought before a judge or taken to court.” The second part of the question, requiring students to
reflect on the content by identifying a difference, is marked in the same manner and 33 per cent
of the students who answered the question got full credit.

How did the students respond? Examples (See end of chapter for marking scheme)
Question 12A:
• “That they were taken to court to discuss the outcome.” (Score 1)
• “The justice system in this story has an impartial person to decide the truth, the judge.” (Score 1)
• “A jury helps to decide”. (Score 0)
• “It was the same.” (Score 0)
Question 12B:
• “A board of 12 judges – a jury – is used instead of a single judge.” (Score 1)
• “There weren’t any lawyers or a jury.” (Score 1)
• “We do the judging inside the courtroom.” (Score 1)
• “The judges don’t use little “tests” like the just judge.” (Score 1)
• “Don’t wear wigs.” (Score 0)

35
 ……………………………………………………………………………………………………………

R
e
a READING UNIT 4
d
i BULLYING
n The following article appeared in a Japanese newspaper in 1996. Refer to it to answer the
g questions below.
L
i
t
e
r PARENTS LACKING AWARENESS OF BULLYING
a
c
y
Only one in three parents polled is aware at their children. The figure was 34 per cent
of bullying involving their children, for the parents of junior high school
according to an Education Ministry survey children and 18 per cent for those of the
released on Wednesday. senior high school students.

The survey, conducted between December Of the parents aware of the bullying, 14 per
1994 and January 1995, involved some cent to 18 per cent said they had been told
19,000 parents, teachers and children at of bullying by teachers. Only 3 per cent to
primary, junior and senior high schools 4 per cent of the parents learned of the
where bullying has occurred. bullying from their children, according to
the survey.
The survey, the first of its kind conducted
by the Ministry, covered students from the The survey also found that 42 per cent of
fourth grade up. According to the survey, primary school teachers are not aware of
22 per cent of the primary school children bullying aimed at their students. The
polled said they face bullying, compared portion of such teachers was 29 per cent at
with 13 per cent of junior high school junior high schools and 69 per cent at senior
children and 4 per cent of senior high high schools.
school students.
Asked for the reason behind bullying, about
On the other hand, some 26 per cent of the 85 per cent of the teachers cited a lack of
primary school children said they have education at home. Many parents singled out
bullied, with the percentage decreasing to a lack of a sense of justice and compassion
20 per cent for junior high school children among children as the main reason.
and 6 per cent for senior high school
students. An Education Ministry official said the
findings suggest that parents and teachers
Of those who replied that they have been should have closer contact with children
bullies, between 39 and 65 per cent said to prevent bullying.
they also have been bullied.
School bullying became a major issue in
The survey indicated that 37 per cent of Japan after 13-year-old Kiyoteru Okouchi
the parents of bullied primary school hanged himself in Nishio, Aichi Prefecture,
children were aware of bullying targeted in the fall of 1994, leaving a note saying

36
……………………………………………………………………………………………………………

R
e
a
that classmates had repeatedly dunked Unit 4 provides another example of continuous d
him in a nearby river and extorted money prose texts. It is an article selected from a i
from him. n
Japanese newspaper. It too is classified as public g
context. It is a type of narrative text (a news
The bullying-suicide prompted the
Education Ministry to issue a report on
story) with rhetorical features of expository prose. L
bullying in March 1995 urging teachers The author raises a problem and in describing the i
problem he presents facts and ideas, explains their t
to order bullies not to come to school. e
relationships and relates them to the problem.The
r
repeated mention of the acts of the Ministry of a
Source: Kyodo, The Japan Times Ltd., Tokyo, Education provides a frame to the expository part c
May 23 1996.
of the article.This is a “news story”, which is y
political-argumentative demonstrating the
ministry’s responsibility and, indeed, responsible
behaviour in finding a solution to the problem.
The text form is frequently used in the media and
Sample Question 13 (Multiple Choice) it was, therefore, found relevant to represent the
reading domain.
• Aspect: Retrieving information
• Text type: Continuous (exposition)
• Situation: Public

What percentage of teachers at each type of school was unaware that

their students were being bullied? Circle the alternative (A, B, C or D)
which best represents this.

A B
Senior High school
Junior High school
Primary school

0 20 40 60 80 100 0 20 40 60 80 100
% of teachers unaware of bullying % of teachers unaware of bullying

C D
Senior High school
Junior High school
Primary school

0 20 40 60 80 100 0 20 40 60 80 100
% of teachers unaware of bullying % of teachers unaware of bullying

37
 ……………………………………………………………………………………………………………

R
e
a Sample Question 13 requires students to retrieve information that is literally stated in
d
i the news article. In order to answer the question correctly, students need to locate the
n information stated in the paragraph beginning “The survey found that 42 per cent of
g teachers...” and match it with one of the four alternatives provided in the question. The
L question was answered correctly by 73 per cent of all students in the PISA field trial. It
i is made more difficult by the fact that information given in a continuous text has to be
t matched with a question expressed in graphic, that is, non-continuous form.
e
r
a
c
y
Sample Question 14 (Open-Constructed Response)
• Aspect: Developing an interpretation
• Text type: Continuous (exposition)
• Situation: Public

Why does the article mention the death of Kiyoteru Okouchi?

Sample Question 14 appears to be slightly easier (answered correctly by 84 per cent of

all students in the PISA field trial). Students are again asked to demonstrate their skill
at developing an interpretation. In order to provide a correct answer, students need to be
able to link together information contained in the last two paragraphs. They are again
asked to demonstrate their skill at interpretation by showing that they can follow the
logical organisation and presentation of ideas within a text. It is worth noting that in
Sample Question 8 (“A Just Judge”), a multiple-choice format is used to obtain evidence
about students’ skill at developing an interpretation, while in Sample Question 14
evidence of this skill is gathered through the constructed response format, by asking
students to generate a short statement connecting the student’s suicide death directly to
the increased awareness of bullying and the public’s concern for this type of behaviour in
Japanese schools.

38
……………………………………………………………………………………………………………

R
e
READING UNIT 5 a
d
MORELAND i
The Moreland Library System gives new library members a bookmark showing its hours of n
opening. Refer to the following bookmark to answer the questions below. g

L
Hours of opening – Effective from February 1 1998 i
Brunswick Campbell Coburg Fawkner Glenroy t
Library Turnbull Library Library Library Library e
Sunday 1pm-5pm Closed 2pm-5pm Closed 2pm-5pm r
Monday 11am-8pm 11am-5.30pm 1pm-8pm 11am-5.30pm 10am-5.30pm a
Tuesday 11am-8pm 11am-8pm 11am-8pm 11am-8pm 10am-8pm c
Wednesday 11am-8pm 11am-5pm 10am-8pm 11am-5pm 10am-8pm y
Thursday 11am-8pm 11am-5.30pm 10am-8pm 11am-5.30pm 10am-8pm
Friday 11am-5pm 11am-5pm 10am-8pm 11am-5pm 10am-5.30pm
Saturday 10am-1pm 10am-1pm 9am-1pm 10am-1pm 9am-1pm

Unit 5 is an example of a non-continuous

text. All of the questions associated with
Sample Question 15 this text proved relatively easy, being
(Closed-Constructed Response) answered correctly by 91 per cent of all
students. Each of the questions is designed
• Aspect: Retrieving information to see whether students are able to retrieve
• Text type: Non-continuous (table) information from this type of display.
• Situation: Public

At What time does the Fawkner Library close on Wednesday?

In order to answer Sample Question 15 correctly, students could simply read across the row for
Wednesday until they found the hours of operation under the heading Fawkner.

Sample Question 16 (Multiple Choice)

• Aspect: Retrieving information
• Text type: Non-continuous (table)
• Situation: Public

Which library is still open at 6 p.m. on Friday evening?

A Campbell Turnbull Library
B Coburg Library
C Fawkner Library
D Glenroy Library
E Brunswick Library
In Sample Question 16, students can again scan across the row labelled “Friday” to determine
that only one library stays open past 5:30 p.m.

39
 ……………………………………………………………………………………………………………

R
e
a READING UNIT 6
d
i WARRANTY
n
g

L
i
t
e
r
a
c
y

Above is the receipt that Sarah received when she bought her new camera. Below is the warranty
card for the camera. Use these documents to answer the questions which follow.

ONE YEAR WARRANTY: (Private Users)

VALID ONLY IN AUSTRALIA

VIDEO HOUSE & COMPANY PTY LTD – ACN 008 458 884 (‘VIDEO HOUSE’) warrants to the initial owner
that the camera is free of any defects in material or workmanship. This warranty is not transferable.
Video House will service, repair or replace at its election, and free of charge, any part which is found
upon inspection by Video House to be defective in material or workmanship during the warranty period(s).

PLEASE PRINT CLEARLY NO. M 409668

Camera – Model........................................................................................................
.................................................................................................................................
Serial No: ..................................................................................................................
SARAH BROWN
Name of Owner: .......................................................................................................
151 GLENLYON STREET,
Address: ....................................................................................................................
BRUNSWICK VIC 3075 Another example of a non-continuous
.................................................................................................................................
text is shown in Unit 6.This text
Date Purchased: .......................................................................................................
actually consists of two parts: one is
Purchase Price: .........................................................................................................

[Rubber Stamp of Dealer]

the receipt, which contains a variety of
information; the other is a warranty
PLEASE NOTE: card which needs to be completed.
Post Immediately – Postage Stamp Necessary
This warranty card should be completed and returned to
Several questions asked about this text
Video House within 10 days of purchase. provide evidence of students’
International Warranty Card issued on request. understanding of, and skill at using,
information presented in this format.

40
……………………………………………………………………………………………………………

R
e
Sample Question 17 (Short Response) a
d
• Aspect: Retrieving information i
• Text type: Non-continuous (certificate) n
• Situation: Private g

Use the details on the receipt to complete the warranty card. L

i
The name and address of the owner have already been filled in. t
e
Sample Question 17 instructs students to use the information contained in the receipt in r
a
order to complete the warranty card.They must retrieve four pieces of information and place c
them appropriately in the spaces provided. Each piece receives a separate mark, which is y
combined to yield a single score. Only 44 per cent of students participating in the PISA field
trial completed all four parts successfully.

Sample Question 18 (Short Response)

• Aspect: Retrieving information
• Text type: Non-continuous (certificate)
• Situation: Private

How long does Sarah have, to return the warranty card?

In Sample Question 18, students have to locate the information on the bottom of the warranty
card which states that it should be returned within 10 days of purchase.This question was
correctly answered by 86 per cent of all students participating in the PISA field trial.

Sample Question 19 (Open-Constructed Response)

• Aspect: Reflecting on content
• Text type: Non-continuous (certificate)
• Situation: Private

The words “Thank you for your business” are printed on the bottom of
the receipt. One possible reason for this is simply to be polite. What is
another possible reason?

Sample Question 19 refers to the small print at the bottom of the receipt, but requires an
answer which goes beyond the information given in the text.The student must bring his or her
outside knowledge to bear on the situation and respond by saying something to the effect that
it is there to help promote a positive business-client relationship.This question was answered
correctly by only 56 per cent of all students participating in the PISA field trial.

41
 ……………………………………………………………………………………………………………

R
e
a SCORING SCHEME FOR THE READING SAMPLE QUESTIONS
d
i
n
g
READING UNIT 1 – BEES
L
i Sample Question 1
t Score 1: Response option D – To indicate where the foragers have found food.
e
r Score 0: Other.
a
c
y Sample Question 2
Score 1: Response option A – The proportion of water in the substance.
Score 0: Other.

Sample Question 3
Score 2: Answers which indicate that information is given BOTH by the shaking of the
abdomen AND by the length of time for which the abdomen is shaken, for example:
• “How long the bee shakes her abdomen for.”
Score 1: Answers which mention the shaking of the abdomen only. (Answer may be partly
inaccurate.) For example:
• “She shakes her abdomen.”
• “She shows how far it is by how fast she shakes her abdomen.”
Score 0: Irrelevant, inaccurate, incomplete or vague answers, for example:
• “How fast the bee runs around in the figure 8.”
• “How big the figure 8 is.”
• “How the bee moves.”
• “The dance.”
• “The abdomen.”

Sample Question 4
Score 1: (in any order from among the following): abc, abe, bde.
a: fruit trees
b: clover
c: flowering trees
d: trees
e: flowers
Score 0: Other.

42
……………………………………………………………………………………………………………

R
e
READING UNIT 2 – IN POOR TASTE a
d
Sample Question 5 i
n
Score 1: Response option A – Guilt.
g
Score 0: Other.
L
Sample Question 6 i
t
Score 1: Answers of type a or b: e
a. Answers which provide a statement or sentence indicating that government or r
individuals should spend more on (overseas) aid, for example: a
• “People donating money to overseas aid.” c
y
• “Donating money to charities.”
• “People should spend less on chocolate and more on the poor.”
b. Answers which provide a statement or sentence indicating that government or
individuals should change their priorities or awareness, for example:
• “Change our priorities.”
• “He would like people to raise their awareness about how we spend our resources.”
Score 0: Answers of type c, d or e:
c. Answers that identify the writer’s strategy, to make the reader feel guilty.
• “Feel guilty / ashamed.”
d. Answers which provide a statement or sentence indicating that less should be spent on
chocolate or be less greedy, for example:
• “Not buy any more chocolate.”
• “Stop eating junk food.”
e. Other, including vague, inappropriate or irrelevant, for example:
• “Spend more on charities.”
• “He would like the government to be sacked.”
• “He would like people to say, ‘I will donate all my money to charity.’”
• “I don’t agree with Arnold Jago.”
• “Agree with him.”

READING UNIT 3 – A JUST JUDGE

Sample Question 7
Score 1: Response option D – By testing her skill in work that she needed to perform for her
husband.
Score 0: Other.
Sample Question 8
Score 1: Response option D – He wanted to see the judge at work in his usual way,
uninfluenced by the presence of the king.
Score 0: Other.
43
 ……………………………………………………………………………………………………………

R
e
a Sample Question 9
d
i Score 1: Response option B – Wise justice.
n
g
Score 0: Other.

L
i Sample Question 10
t Score 1: Response option A – A folk tale.
e
r Score 0: Other.
a
c
y Sample Question 11
Score 1: Answers of type a:
a. Evaluates the fairness of the punishments in relation to each other, in terms of
similarity or difference between offences. Shows accurate understanding of the crimes.
For example:
• “No, it is a much more serious crime to try to steal someone’s wife than to steal their
money or their horse.”
• “All three criminals tried to cheat someone and then lied about it, so it was fair that
they were punished in the same way.”
• “It’s hard to say. The peasant, the oil merchant and the beggar all wanted to steal
something. On the other hand, the things they wanted to steal were not equally
valuable.”
Score 0: Other, including answers of type b, c or d:
b. Shows accurate understanding of the crimes and/or the punishments without
evaluating them. For example:
• “The judge gave fifty strokes to the three criminals. Their crimes were stealing a
woman, stealing money and stealing a horse.”
c. Demonstrates a misunderstanding of the crimes or the punishments. For example:
• “I think the case of the peasant and the scholar was different from the other two
because it was more like a divorce, while the other two were thefts. So the peasant
should not have been punished.”
d. Agrees or disagrees without explanation or with inadequate explanation. May evaluate
the fairness of the punishment per se (i.e. answers as if the question were, “Is fifty
strokes of the lash a just punishment?”). For example:
• “No, fifty lashes are much too harsh a punishment for any of these crimes.”
• “Yes, severe punishments are necessary because that way the criminal won’t try to do
it again.”
• “No, I don’t think the punishments were harsh enough.”
• “He was too severe.”
• “Yes, I think it was fair.”
44
……………………………………………………………………………………………………………

R
e
Sample Question 12 a
d
Question 12A: i
n
Score 1: Describes one similarity. Shows accurate comprehension of the story. Comparison g
with a feature of the national legal system is either explicitly stated or may be readily inferred.
Accurate knowledge of national legal systems is not essential, but take into account what L
background knowledge about the law in your country it would be reasonable to expect of a i
15-year-old. For example: t
e
• “Rulings are made on evidence.” r
• “Both sides are allowed to give their version of the truth.” a
• “Equality before the law (it doesn’t matter who you are).” c
• “There is a judge presiding over the court.” y
• “The same punishment is given for similar offences.”
Score 0: Other, including vague, inaccurate and irrelevant answers.
• “Don’t know right from wrong.”
• “Even important rulers of countries can be called to court.”
• “Punishment.” [excluded by the question]
Question 12B:
Score 1: Describes one difference. Shows accurate comprehension of the story. Comparison
with a feature of the national legal system is either explicitly stated or may be readily inferred.
Accurate knowledge of national legal systems is not essential. (For example “no jury” may be
accepted as a “difference”, although in some legal systems there is no jury.) Take into account
what background knowledge about the law in your country it would be reasonable to expect of a
15-year-old. Examples are:
• “No lawyers.”
• “Judge carries out his own investigation.”
• “It’s very quick, whereas in modern courts usually cases take weeks.”
• “No jury; there doesn’t seem to be any way of appealing.”
• “The punishment is much harsher.” [a qualitative comment on the kind of punishment]
• “The same punishment is given regardless of the offence.”
Score 0: Other, including vague, inaccurate and irrelevant answers.
• “Punishment.”
• “Old fashioned.”
• “Court system.”
• “People do not receive the lash.” [excluded by question]

READING UNIT 4 – BULLYING

Sample Question 13
Score 1: Response option A – A (letter A or graph A).
Score 0: Other.
45
 ……………………………………………………………………………………………………………

R
e
a Sample Question 14
d
i Score 1: Answers which relate bullying (and the suicide) to public concern and/or to the
n survey. The connection may be explicitly stated or readily inferred. Possible answers are:
g
• “To explain why the survey was conducted.”
L • “To give the background to why people are so concerned about bullying in Japan.”
i • “He was a boy who committed suicide because of bullying.”
t • “To show how far bullying can go.”
e • “It was an extreme case.”
r
a Score 0: Vague or inaccurate answers:
c
y • “He was a Japanese schoolboy.”
• “There are many cases like this all over the world.”

READING UNIT 5 – MORELAND

Sample Question 15
Score 1: 5 p.m. or 5 o’clock or 17.00.
Score 0: Other.

Sample Question 16
Score 1: Response option B – Coburg Library.
Score 0: Other.

READING UNIT 6 – WARRANTY

Sample Question 17
• 1a (model)
Score 1: Answers which correctly identify the model:
Rolly Fotonex 250 zoom.
Rolly Fotonex.
Fotonex.
Score 0: Other.
• 1b (serial number)
Score 1: 30910963.
Score 0: Other.
• 1c (date of purchase)
Score 1: 18/10/99.
Score 0: Other.

46
……………………………………………………………………………………………………………

R
e
• 1d (purchase price) a
d
Score 1: ($) 249.08. i
Score 0: Other. For example: ($) 254.74. n
g

Sample Question 18 L
i
Score 1: Answers which indicate 10 days or ten days. For example: t
• “Within 10 days of purchase.” e
r
Score 0: Other. a
c
y
Sample Question 19
Score 1: Answers which refer either explicitly or implicitly to the development of the business-
customer relationship. For example:
• “It’s good for business to be nice to you.”
• “To create a good relationship with the customer.”
• “They want you to come back.”
Score 0: Other. For example:
• “They’re being polite.”
• “They’re glad you bought the camera from them.”
• “They want you to feel special.”
• “To let the customers know they are appreciated.”

47
2

M
A
T
H
E
M
A
T
I
C
A
L
L
I
T
E
R
A
C
Y
 ……………………………………………………………………………………………………………

M
a
t ASSESSING MATHEMATICAL LITERACY IN PISA
h
e
m
a
t The PISA definition of mathematical Three dimensions of mathematical
i literacy and its context literacy
c
a To transform this definition into an
Mathematical literacy is defined in PISA as:
l assessment of mathematical literacy, three broad
dimensions have been identified. These are:
L
i the capacity to identify, to understand, • Processes: the focus is on students’ abilities
t and to engage in mathematics and to analyse, reason and communicate ideas
e effectively by posing, formulating and
r
make well-founded judgements about
the role that mathematics plays, as solving mathematical problems. Processes
a
are divided into three classes: reproduction,
c needed for an individual’s current and
y definitions and computations; connections
future private life, occupational life, and integration for problem solving; and
social life with peers and relatives, and mathematisation, mathematical thinking and
life as a constructive, concerned, and generalisation.
reflective citizen.
• Content: PISA emphasises broad
mathematical themes such as change and
growth, space and shape, chance, quantita-
Several aspects of this definition have specific tive reasoning, and uncertainty and depend-
meaning in the PISA context. As with reading, ency relationships.
the definition revolves around wider uses in • Context: an impor tant aspect of
people’s lives rather than simply carrying out mathematical literacy is doing and using
mechanical operations. The term “literacy” is mathematics in a variety of situations,
therefore used to indicate the ability to put including personal life, school life, work
mathematical knowledge and skill to functional and sports, local community and society.
use rather than just to master it within a school
curriculum. Mathematical processes
In the PISA definition, to “engage in” PISA tasks are designed to encompass a set
mathematics covers not simply physical or social of general mathematical processes that are
actions (such as working out how much change meant to be relevant and pertinent to all
to give someone in a shop) but also wider uses, educational levels:
including taking positions towards and 1. Mathematical thinking, which includes:
appreciating things (such as having an opinion
– posing questions characteristic of
about the government’s spending plans).
mathematics (“Is there...?”, “If so, how
many?”, “How do we find...?”);
Mathematical literacy also implies the ability
to pose and solve mathematical problems in a – knowing the kinds of answers that
variety of contexts, as well as the inclination to mathematics offers to such questions;
do so, which often relies on personal traits such – distinguishing between different kinds
as self-confidence and curiosity. of statements (definitions, theorems,
50
……………………………………………………………………………………………………………

M
a
conjectures, hypotheses, examples, sentation of mathematical objects and t
conditioned assertions); situations, and the interrelations h
e
– understanding and handling the extent between the various representations;
m
and limits of given mathematical concepts. – choosing, and switching between, a
different forms of representation, t
2. Mathematical argumentation, which
according to situation and purpose. i
includes: c
– knowing what mathematical proof is and 6. Symbols and formalism, which includes: a
how it differs from other kinds of l
– decoding and interpreting symbolic and
mathematical reasoning; formal language and understanding its L
– following and assessing chains of math- relationship to natural language; i
ematical arguments of different types; t
– translating from natural language to
– possessing a feel for heuristics (“what e
symbolic/formal language; r
can(not) happen, and why”), and
– handling statements and expressions a
creating mathematical arguments. c
containing symbols and formulae;
3. Modelling, which includes: y
– using variables, solving equations and
– structuring the field or situation to be understanding calculations.
modelled;
– mathematising (that is translating from 7. Communication, which includes:
“reality” to “mathematics”); – expressing oneself, in a variety of ways,
– de-mathematising (that is interpreting on matters with a mathematical compo-
mathematical models in terms of “reality”); nent, in oral as well as in written form;
– tackling the model (working within the – understanding others’ written or oral
mathematics domain); statements about such matters.
– validating the model; 8. Aids and tools, which includes:
– reflecting, analysing, offering a critique – knowing about, and being able to make
of models and their results; use of, various aids and tools (including
– communicating about the model and its information technology tools) that may
results (including the limitations of such assist mathematical activity;
results); – knowing about the limitations of such
– monitoring and controlling the aids and tools.
modelling process.
PISA does not use test items that assess the
4. Problem posing and solving, which above skills individually.When doing “real math-
includes: ematics” it is necessary to draw simultaneously
– posing and formulating mathematical upon many of these skills.
problems;
– solving different kinds of mathematical In order to describe levels of mathematical
problems, in a variety of ways. competency, PISA organises processes into
three classes, defining the type of thinking skill
5. Representation, which includes: needed: i) reproduction, definitions and
– decoding, interpreting and distinguish- computations; ii) connections and integration for
ing between different forms of repre- problem solving; and iii) mathematisation,
51
 ……………………………………………………………………………………………………………

M
a
t mathematical thinking, generalisation and 3. Competency Class 3: Mathematisation,
h insight. In general, these processes are in mathematical thinking, generalisation and
e
ascending order of difficulty, but it does not insight
m
a follow that one must be mastered in order to
progress to the other: it is possible for example In this competency class students are asked
t
i to engage in mathematical thinking without to mathematise situations: to recognise and
c being good at computations. extract the mathematics embedded in the
a situation and use mathematics to solve the
l
1. Competency Class 1: Reproduction,
problem, analyse, interpret, develop their
definitions and computations
own models and strategies, and make
L mathematical arguments, including proofs
i Class 1 covers processes assessed in many
t
and generalisations.
standardised tests, as well in comparative
e These processes involve critical thinking,
r international studies, operationalised mainly
a in multiple-choice format. It deals with analysis and reflection. Students should not
c knowledge of facts, representing, only be able to solve problems but also to
y recognising equivalents, recalling pose problems, to communicate solutions
mathematical objects and properties, appropriately, and to have insight into the
performing routine procedures, applying nature of mathematics as a science.
standard algorithms and developing This level, which goes to the heart of
technical skills. mathematics and mathematical literacy, is
difficult to test. The multiple-choice
2. Competency Class 2: Connections and
format for test items is usually inadequate.
integration for problem solving
Questions with open-ended answers are
Class 2 processes start making connections more suitable, but both the design of such
between the different strands and domains questions and the marking of answers are
in mathematics, and integrate information difficult.
in order to solve simple problems. Although
the problems are supposedly non-routine, Mathematical content: strands and “big
they require relatively minor degrees of ideas”
mathematisation.
School mathematics curricula are usually
Within this competency class students are organised in strands. These strands
also expected to handle different aspects of compartmentalise mathematics and often over-
representation, according to situation and emphasise computation and formulae. At the
purpose. Connections also require students beginning of the 20th century, mathematics
to be able to distinguish and relate different could reasonably be regarded as consisting of
statements such as definitions, claims, exam- some twelve distinct subjects: arithmetic,
ples, conditioned assertions and proofs. geometry, algebra, calculus, and so on. Today,
Decoding and interpreting symbolic and however, between sixty and seventy distinct
formal language, and understanding its rela- subjects would be a reasonable figure. Some
tionship to natural language, form another subjects, such as algebra or topology, have split
aspect of this class. In this class, the problems into various sub-fields; others, such as
are often placed within a context, and engage complexity theory or dynamic systems theory,
students in mathematical decision making. are completely new areas of study. To be
52
……………………………………………………………………………………………………………

M
a
relevant, mathematics must reflect the complex outside world, and to cope with a changing t
patterns in the world around us. world to our best advantage. h
e
Many sub-strands of traditional m
For these and other reasons, PISA took a a
different approach and organised content mathematics curricula are involved in tasks t
around cross-cutting mathematical themes, of this type. They include, for example, i
referred to as “big ideas”. For the purpose of relations, functions and gradients. c
PISA, a selection of big ideas has been made Considering rates of growth for different a
that offers sufficient variety and depth to reveal growth phenomena leads to linear, l
the essentials of mathematics and that relates exponential, logarithmic, periodic, logistic
L
to the traditional curricular strands. The growth curves and their properties and i
following list of mathematical big ideas meets relations. These, in turn, lead to aspects of t
this requirement: change and growth; space and number theory. The connections between e
shape; quantitative reasoning; uncertainty; and these ideas and geometrical representations r
dependency and relationships. can also play a role. a
c
Growth patterns can be expressed in y
In PISA 2000, the focus is on the first two
of these big ideas. They allow a wide range of algebraic forms, which in turn can be
aspects of the curriculum to be represented, represented by graphs. Growth can also be
without giving undue weight to number skills. measured empirically, and questions arise
as to what inferences can be made from
the growth data, how the growth data
1. Change and growth
might be represented, and so on. Data
Every natural phenomenon is a analysis and statistics are relevant
manifestation of change. Examples of curricular strands here.
change include the growth of organisms,
the cycle of seasons, the ebb and flow of 2. Space and shape
tides, cycles in unemployment levels and Patterns are encountered everywhere
the Dow-Jones index. Some growth around us: spoken words, music, video,
processes can be described or modelled by traffic, constructions and art. Shapes are
straightforward mathematical functions: patterns: houses, churches, bridges, starfish,
linear, exponential, periodic, logistic, snowflakes, town plans, clover leaves,
either discrete or continuous. But many crystals and shadows.
processes fall into different categories and
data analysis is quite often essential. In understanding space and shape, students
Observing patterns of change in nature need to look for similarities and
and in mathematics is therefore not differences as they analyse the components
confined to par ticular par ts of the of form and recognise shapes in different
curriculum such as algebra. representations and different dimensions.
This means that they must be able to
PISA examines students’ ability to represent understand the relative positions of
changes in a comprehensible form, to objects. They must be aware of how we
understand the fundamental types of change, see things and why we see them as we do.
to recognise particular types of change when They must learn to navigate through space
they occur, to apply these techniques to the and through constructions and shapes.

53
 ……………………………………………………………………………………………………………

M
a
t Students should therefore be able to for students, or better suited than more
h understand the relations between shapes and scientific contexts. Indeed, some experts
e
images or visual representations, such as that believe that familiarity with the context can be
m
a between a real city and photographs and a hindrance, while research suggests that boys
t maps of the same city. They must also do better on experiential knowledge of
i understand how three-dimensional objects numbers and measures from the daily world
c can be represented in two dimensions, how while girls perform better on items where a
a shadows are formed and must be standard procedure is needed. Secondary school
l
interpreted, what “perspective” is and how students appear to have less need for a
L it functions. Described in this way, the study personally relevant context than primary school
i of space and shape is open-ended and children.
t dynamic, and fits well both into
e mathematical literacy and into the Whatever the distance from the students,
r mathematical competencies defined for this PISA aims to ensure that tasks are based on
a “authentic” contexts which are likely to occur
study.
c
y
in a real-world setting. If mathematics education
In PISA 2000, testing time will be evenly is to prepare students to be active and informed
distributed between the two big ideas: change citizens it has to deal with “real” contexts such
and growth and space and shape. In some items as pollution problems, traffic safety and
at least, marks will not only be awarded for the population growth. This does not exclude,
“correct” answer but also for the different however, artificial fictional contexts based on
strategies used by students to solve the test the stylised representation of problems – such
items. as a traffic situation in a non-existent town.

Situations and contexts Format of test questions and

marking
Students’ mathematical insight and under-
standing need to be assessed in a variety of PISA will assess mathematical literacy
situations, partly to minimise the chance of through a combination of item formats. Some
students finding that tasks are not culturally tasks are assessed through multiple-choice
relevant. questions, typically those associated with
simpler mathematical processes.
One can think of a situation as being at a
certain distance from the student. The closest Open-ended items are preferred for
is private life (daily life), next is school life, work assessing higher-order mathematical processes.
and sports, followed by the local community Such items often require students to show the
and society as encountered in daily life, and steps taken or to explain how the answer was
furthest away are scientific contexts. In this way reached. They allow students to demonstrate
one can define a more or less continuous scale their level of ability by providing solutions of a
of situations. range of mathematical complexity.
Furthermore, since these responses give
It is not always clear to what extent the valuable information about students’ ideas and
students, performances are affected by their thinking, which can be fed back into curriculum
distance from the situations.We cannot say that planning, the marking guides for items included
“closer” contexts are necessarily more attractive in the main study were designed to include a
54
……………………………………………………………………………………………………………

M
a
system of two-digit marking so that the t
frequency of various types of correct and h
e
incorrect response can be recorded. It is
m
important to note that markers are advised to a
ignore spelling and grammar errors, unless they t
completely obscure the meaning, because this i
is not a test of written expression. The PISA c
marking scheme for open-ended items is shown a
l
at the end of this chapter.
L
Sample questions i
t
The following examples illustrate the range e
of tasks and questions used in the PISA r
assessment of mathematical literacy.These items a
c
were used in the PISA field trial but not selected y
for the PISA 2000 assessment because of their
similarity to other item sets in terms of what
they measure.The intention in presenting these
tasks and questions is to demonstrate the
connection between the PISA assessment
framework and the items that were constructed
to represent it.

Since the PISA 2000 assessment was not

completed by the time this publication was
produced, items from the PISA 2000 tests could
not be included in this publication for reasons
of test security.

As will be evident from the examples

shown here, it is not always easy to match
items neatly to the dimensions of mathemat-
ics given in the PISA framework. In particu-
lar, the three competency classes are not
intended to form a strict hierarchy since the
competencies used to solve a given problem
will vary from student to student. Mathemati-
cal problems are also frequently not specific
to particular situations.

…………
55
 ……………………………………………………………………………………………………………

M
a
t MATHEMATICS UNIT 1
h
e PIZZAS
m
a
t
i A pizzeria serves two round pizzas of the same thickness in different sizes. The smaller
c one has a diameter of 30 cm and costs 30 zeds. The larger one has a diameter of 40 cm and
a costs 40 zeds.
l
© PRIM, Stockholm Institute of Education.
L
i
t
e
r
a Sample Question 1 (Open-Constructed Response)
c
• Competency class 2 : Connections and integration for problem solving
y
• Big idea: Change and growth and/or Space and shape
• Situation: Personal

Which pizza is better value for money? Show your reasoning.

In Unit 1, the stimulus and the question based on it are about the conceptual understanding
of the growth rates of areas. A variety of mathematical competencies, of different levels, can be
used to solve the problem.
First, students have to identify the relevant mathematics (part of the mathematisation process)
as a critical step in the modelling process (which means in this case: 30 cm; 30 zeds; 40 cm;
40 zeds). Students can then solve the problem through qualitative reasoning: as the surface
area of a pizza increases faster (quadratic growth) than the price (which appears to be
growing linear), the larger pizza is the better deal.This is a very elegant way of solving the
problem because the reasoning goes to the kernel of the mathematical argument, and is easily
generalisable. Many students will feel more at ease with a quantitative solution, however.They
will calculate the area and amount per zed for each of the two pizzas: the area of the smaller
pizza per zed is about 24 cm2, and that of the bigger one 31 cm2. Other solutions can be
expected: for instance students may visualise the problem by drawing the pizzas to scale and
to reason from the drawings. If they use graph paper they do not need the formula for the
area of a circle, but can still use a “calculation” strategy.
Some students will no doubt answer that the relative prices of the two pizzas are the same.
This is a clear misconception.
For most students, this will be a non-routine problem, and connections will be made between
different strands and “big ideas”: some might even argue that if students use qualitative
reasoning, they are employing competencies of Class 3 (mathematical thinking), while most
solutions will rely on Class 2 competencies.

56
……………………………………………………………………………………………………………

M
a
MATHEMATICS UNIT 2 t
h
COINS e
m
a
t
You are asked to design a new set of coins. All coins will be circular and coloured silver, i
but of different diameters. c
a
Researchers have found out that an ideal coin system meets the following requirements: l

L
i
t
e
r
a
c
• diameters of coins should not be smaller than 15 mm and not be larger than 45 mm. y

• given a coin, the diameter of the next coin must be at least 30% larger.

• the minting machinery can only produce coins with diameters of a whole number of
millimetres (e.g. 17 mm is allowed, 17.3 mm is not).

Sample Question 2 (Open-Constructed Response)

• Competency class 2: Connections and integration for problem solving
• Big idea: Change and growth
• Situation: Occupational

Design a set of coins that satisfy the above requirements. You should
start with a 15 mm coin and your set should contain as many coins as
possible.

The problem set in Sample Question 2 shows the constructive and finite use of mathematics.
Some degree of modelling and argumentation, and symbolic, formal and technical skills are
required, although the calculation skills required are definitely of the lowest level, the
problem lying in the complex way in which the information is presented. Mathematisation is
called for in order to translate the problem from natural language into more mathematical
language.
The format adopted is that of an open-constructed response item as there are ample opportu-
nities for students to adopt different strategies. Many students will start in the order in which
the information is given.This means that the first coin has to be 15 mm in diameter.Then they
57
 ……………………………………………………………………………………………………………

M
a
t will compute the 30 per cent increase: a mathematically elegant solution would be to state that a
h 30 per cent increase will lead to exponential growth with a growth factor of 1.3; which leads to the
e
m sequence: 15 - 19.5 - 25.35 - 32.955 - 42.8415.This would show a sophisticated level of
a mathematical understanding, but would still not be correct because of the requirement for whole
t millimetres. One correct approach is to start with 15, add 30 per cent of 15 and round up to
i 20 mm before again adding 30 per cent. Continuing this procedure by adding 30 per cent of 20,
c
a and so on, the eventual sequence will be 15 - 20 - 26 - 34 - 45. A variety of partially right
l answers are possible, particularly if students do not read the text carefully enough or do not fully
understand the context.
L
i Computational (Class 1) competencies are required, but information must also be integrated so
t this item has been classified as Class 2.The “big idea” superficially may appear to be space and
e shape, but this is superficial, since the problem is about percentual or exponential growth within a
r geometrical context.
a
c
y

MATHEMATICS UNIT 3
LICHEN

A result of global warming is that the ice of some glaciers is melting.Twelve years after
the ice disappears, tiny plants, called lichen, start to grow on the rocks.

Each lichen grows approximately in the shape of a circle.

The relationship between the

diameter of this circle and the age The problem set out in Unit 3 has been modelled already, so
of the lichen can be approximated that the mathematisation process has been reduced to the
with the formula: matching of the text to the mathematical formula. However,
this is not a trivial demand, and calls for symbolic, formal
d = 7.0 x √t-12 for t≥12 and technical competencies.There is clearly a space and
shape component as it involves the growth of circles, but it
where d represents the can be anticipated that the vast majority of students will
diameter of the lichen in solve the problem by substituting the values in the formula,
millimetres, and t represents the almost without realising that they are dealing with shapes.
number of years after the ice has The format of Sample Questions 3 and 4 is clearly that of an
disappeared. open-constructed response item, with limited possibilities for
variation in strategy. Although the situation is scientific and
hence rather “far away” from the student, the context is only
marginally relevant to understanding the problem and
finding a solution. Most students will focus on the formula
and the necessary substitutions and use the context merely to
find out whether it is t or d that they need to substitute.
58
……………………………………………………………………………………………………………

M
a
Sample Question 3 (Open-Constructed Response) t
h
• Competency class 1: Reproduction, definitions, computations e
• Big idea: Change and growth m
• Situation: Scientific a
t
i
c
Using the formula, calculate the diameter of the lichen, 16 years after a
the ice disappeared. l

Show your calculation. L

i
t
e
In Sample Question 3, students with less than a full understanding of the process will still r
have a fair chance of finding the correct answer because the question is relatively a
straightforward.The variation in strategies is very limited, as is the possibility of a partially c
y
correct answer. It seems obvious that the first step is to substitute 16 for t in the formula,
resulting in the square root of 16 - 12, multiplied by 7. If students have found this they have
addressed the essence of the problem, and will be given partial credit. However, they also need
to find the correct answer (14) in order to gain full credit. Although the problem does not
completely satisfy the “reproduction” criteria that play an important role in Competency
Class 1, the substitution required is so fundamental as to form part of the basic tool kit of
any student.

How did the students answer? Examples (See end of chapter for marking scheme)

d = 7.0 x √16 - 12 mm
d = 7.0 x √4 mm Score 2

d = 14 mm

d = 7.0 x √16 - 12 mm
Score 1 (Correct substitution but incorrect answer)
d = 16 mm

d = 7.0 x √16 - 12 Score 1 (Incomplete answer)

d = 7√4

59
 ……………………………………………………………………………………………………………

M
a
t Sample Question 4 (Open-Constructed Response)
h
e
m • Competency class 2: Connections and integration for problem solving
a • Big idea: Change and growth
t • Situation: Scientific
i
c Ann measured the diameter of some lichen and found it was 35
a millimetres.
l
How many years ago did the ice disappear at this spot?
L
i
Show your calculation.
t
e
r
a Sample Question 4 is slightly more difficult, because substituting the value of d instead
c of t causes more computational problems for many students. One may expect trial and
y error answers. Students may repeat the procedure used in Sample Question 3, guessing at
the value of t until they arrive at a reasonable answer. There will again be variations,
ranging from a completely correct answer to a correct substitution but incorrect
computation, or to trial and error.This may lead to an answer close to the correct one,
for instance 36 instead of 37.The question calls for more complex competencies than
Sample Question 3.

How did the students answer? Examples (See end of chapter for marking scheme)

35 = 7.0 x √t - 12
5 = √t - 12 Score 1

5 = √t - √12
Too hard!

35 = 7.0 x √t - 12
352 = 72 x t - 12 Score 1

49t = 1 237
t = 25 years

60
……………………………………………………………………………………………………………

M
a
MATHEMATICS UNIT 4 t
h
SHAPES e
m
a
t
A B C i
c
a
l

L
i
t
e
r
a
Unit 4 clearly relates to familiarity with c
© PRIM, Stockholm Institute of Education. y
fundamental ideas about space and shape,
including working with measurements. It also
depends on the ability to think and work
mathematically.The context is at the furthest
distance from the student, although it is
Sample Question 5 possible that some students will identify it
(Open-Constructed Response) with some real-life problem within their
experience. Sample Questions 5, 6 and 7 are
• Competency class 1: Reproduction, definitions, all of the open-constructed response type
computations
because a range of strategies can be adopted
• Big idea: Space and shape
• Situation: Scientific
in order to answer them.

Which of the figures has the largest area? Give explanations to your
answer.

Sample Question 5 is relatively simple, calling for competencies of Class 1. It deals with
the comparison of three areas, two of them very irregular, and one of them looking like a
circle. As each of the two irregular shapes fits more or less into the circle, it may be
obvious that the largest area is the “almost circle”.This is the “expected” answer, and
anything that resembles this line of reasoning will be given full credit. But one still needs
to be careful in making judgements: answers are clearly correct if they state that: “B is the
largest because it doesn’t have indents in it, which decreases the area. A and C have gaps.”
Or: “B, because it’s a full circle, and the others are like circles with bits taken out.” An
answer could also be expressed in pictures bearing the same message. It is more difficult,
however, if a student answers: “B, because it’s pretty obvious.” Students might indeed find it
obvious, but the question clearly states: “Give explanations to your answer.” The student in
question will not gain full credit.

61
 ……………………………………………………………………………………………………………

M
a
t How did the students answer? (See end of chapter for marking scheme)
h
e • “B, because it has no open areas:” (Score 1)
m
a
t
i
c
a
l • “B. Because it has the largest surface area.” (Score 0)
• “The circle. It’s pretty obvious.” (Score 0)
L
i • “They are all the same.” (Score 0)
t
e
r
a Sample Question 6 (Open-Constructed Response)
c
y • Competency class 2: Connections and integration for problem solving
• Big idea: Space and shape
• Situation: Scientific

Describe a method for estimating the area of figure C.

Sample Question 6 requires some mathematical reasoning and considerable communication

skills. Several approaches are possible. Students might suggest drawing a grid of squares over the
shape and counting the squares in a “smart” way. A more sophisticated solution might use
realignment in order to get as many squares completely full as possible (by putting poorly filled
squares together). Another solution is to cut off the arms and rearrange the pieces so that they fill
a square, and then to measure a side of the square. Since experience shows that students are very
inventive, one should not be surprised to find solutions that use liquids:“Build a model that uses
the shape given as the bottom and build a rim of 1 cm around it. Fill it with water, measure the
amount of water and derive the area from that amount.”
A separate problem is of technical nature: should the question not be:“Describe a method for
estimating the area of figure C in square centimetres”? In order to make an estimation one needs
a unit of measure, which is missing in the present problem, and although many students may not
see this as an obstacle, it might be wise to specify a unit.
How did the students answer? (See end of chapter for marking scheme)
• “You could fill the shape with lots of circles, squares and other basic shapes so there is not a gap.
Work out the area of all of the shapes and add together.” (Score 1)
• “Redraw the shape onto graph paper and count all of the squares it takes up.” (Score 1)
• “Drawing and counting equal size boxes. Smaller boxes = better accuracy.” (Score 1 – Here the
student’s description is brief, but we will be lenient about student’s writing skills and regard
the method offered by the student as correct).
62
……………………………………………………………………………………………………………

M
a
• “Add up the area of each individual arm of the shape.” (Score 0) t
h
• “Find the area of B then find the areas of the cut out pieces and subtract them from the main e
area.” (Score 0) m
a
• “Minus the shape from the circle.” (Score 0) t
• “Use a string and measure the perimeter of the shape. Stretch the string out to a circle and i
measure the area of the circle using π r2.” (Score 0 – Here the method described by the student c
a
is wrong) l

L
i
Sample Question 7 (Open-Constructed Response) t
e
r
• Competency class 2: Connections and integration for problem solving
a
• Big idea: Space and shape c
• Situation: Scientific y

Describe a method for estimating the perimeter of figure C.

Sample Question 7 is of a similar order of complexity as Sample Question 6, and raises many
of the same problems. It tries to establish how well students can measure the perimeter s of an
irregular area. A good way is to use measuring tape or a piece of string or cord over the
outline of the shape and to measure the length of this string. Students with more developed
linear reasoning might suggest approximating the length in straight pieces and subsequently
measuring the sum of the pieces. Or students might argue that they could approximate the
present irregular star by a regular star shape, and measure the length of one arm, and
subsequently multiply the answer by eight.

How did the students answer? Examples (See end of chapter for marking scheme)

• “Wool or string!!!” (Score 1 – Here although the answer is brief, the student did offer a METHOD
for measuring the perimeter)
• “Cut the side of the shape into sections. Measure each then add them together.” (Score 1 – Here
the student did not explicitly say that each section needs to be approximately straight, but we
will give the benefit of the doubt, that is, by offering the METHOD of cutting the shape into
pieces, each piece is assumed to be easily measurable)
• “Measure around the outside.” (Score 0 – Here the student did not suggest any METHOD
of measuring. Simply saying “measure it” is not offering any method of how to go about
measuring it)
• “Stretch out the shape to make it a circle.” (Score 0 – Here although a method is offered by the
student, the method is wrong)

63
 ……………………………………………………………………………………………………………

M
a
t MATHEMATICS UNIT 5
h
e BRAKING
m
a
t
i The approximate distance to stop a moving vehicle is the sum of:
c • the distance covered during the time the driver takes to begin to apply the brakes
a
l (reaction-time distance)
• the distance travelled while the brakes are applied (braking distance).
L
i The ‘snail’ diagram below gives the theoretical stopping distance for a vehicle in good
t braking conditions (a particularly alert driver, brakes and tyres in perfect condition, a dry
e
r
road with a good surface) and how much the stopping distance depends on speed.
a
c
y Distance to stop a vehicle.

Time to stop the vehicle.

Distance traveled during

the braking time.

Distance traveled during

the driver’s reaction time.

Source: La Prévention Routière, Ministère de l’Éducation nationale, de la Recherche et de la Technologie, France.

64
……………………………………………………………………………………………………………

M
a
t
Modelling can take many different forms. In Unit 5, a well-known situation that is relevant to h
e
everyone who travels by road is presented by way of a diagram to show clearly the increase in m
braking distance as a function of speed.The context will be familiar for many students. a
The “distance” from students’ own world is relatively small, although they might not be aware of t
the underlying mathematical model.The context is authentic because it resides in the actual i
experiences and practices of the participants in a real-life setting. Understanding the problem can c
certainly contribute to better “functioning” in our complex society, and the ability to interpret and a
l
use a visual presentation is a skill that falls well within the limits of literacy. It is not dependent
on pure technical skills such as substituting numbers in a known formula. L
The modelling of the braking process has already been done, but modelling and representation i
skills will be used to “de-mathematise” the diagram and translate it into natural language. t
e
Basically, all that is asked of the student is to “read” the diagram, making the connection r
between reaction time, distance and braking action, in relation to speed.This may seem simple a
but students are so used to seeing in a test what has been taught in the classroom that they c
lose the ability to transfer their skills to different contexts.The PISA assessment acknowledges y
this in that all questions are straightforward and use numbers that are directly represented in
the diagram.The questions form a natural sequence, using the same speed throughout. It will
be clear from the nature of the questions that we do not expect a variety of strategies and
styles of answer.The format of Sample Questions 8-12 is therefore that of a closed-con-
structed response item. Although this is definitely a simple problem concerning growth curves,
it is likely that students will never have encountered such a representation. For that reason all
the questions asked about it are assumed to call for Competency Class 2.
The situation (dealing with braking distances diagrammatically) could be used to ask further
questions demanding additional skills. If one takes a speed of 107 kph, for instance, the student
needs to draw a line (maybe) that starts at the appropriate point on the speedometer between
100 and 110, and then to make educated guesses about the correct answer, which can be obtained
in different ways. Much more complex questions could be asked as well. An example would be to
compute the ratios between the distances travelled during the reaction time and the total braking
distance. At 40 kph this ratio is 8.3:18.6, or close to 1:2. At a speed of 180 kph, the ratio is
37.8:245.5, close to 1:6. Students could be asked to explain that difference.

Sample Question 8 (Closed-Constructed Response)

• Competency class 2: Connections and integration for problem solving
• Big idea: Change and growth
• Situation: Personal/public

If a vehicle is travelling at 110 kph, what distance does the vehicle

travel during the driver’s reaction time?
For anyone able to read the diagram, the answer to Sample Question 8 is simple: 22.9 m.

65
 ……………………………………………………………………………………………………………

M
a
t Sample Question 9 (Closed-Constructed Response)
h
e • Competency class 2: Connections and integration for problem solving
m • Big idea: Change and growth
a • Situation: Personal/public
t
i If a vehicle is travelling at 110 kph, what is the total distance travelled
c before the vehicle stops?
a
l Sample Question 9 is equally simple.The answer is 101 metres.
L
i Sample Question 10 (Closed-Constructed Response)
t
• Competency class 2: Connections and integration for problem solving
e
r • Big idea: Change and growth
a • Situation: Personal/public
c If a vehicle is travelling at 110 kph, how long does it take to stop the
y
vehicle completely?
Sample Question 10 is related to the previous question.This question is somewhat less
relevant in most cases of braking but can be found at the outer spiral: 5.84 seconds.

Sample Question 11 (Closed-Constructed Response)

• Competency class 2: Connections and integration for problem solving
• Big idea: Change and growth
• Situation: Personal/public
If a vehicle is travelling at 110 kph, what is the distance travelled while
the brakes are being applied?
Sample Question 11 is different, requiring not just reading of the diagram but also some
simple calculation to show that students can interpret the diagram. Students need to subtract
22.9 from 101 to find the answer, 78.1 metres.

Sample Question 12 (Closed-Constructed Response)

• Competency class 2: Connections and integration for problem solving
• Big idea: Change and growth
• Situation: Personal/public

A second driver, travelling in good road conditions, stops her vehicle in

a total distance of 70.7 metres. At what speed was the vehicle travelling
before the brakes were applied?
Sample Question 12, the last on this stimulus, is somewhat easier as it merely requires
students to read off the speed that accords with the given total braking distance.

66
……………………………………………………………………………………………………………

M
a
MATHEMATICS UNIT 6 t
h
PATIO e
m
a
Sample Question 13 (Open-Constructed Response) t
i
• Competency class 2: Connections and integration for problem solving c
• Big idea: Space and shape a
• Situation: Personal l

L
Nick wants to pave the rectangular patio of his new house. The patio is 5.25 i
metres long and 3.00 metres wide. He needs 81 bricks per square metre. t
e
Calculate how many bricks Nick needs for the whole patio. r
a
c
An important part of the definition of mathematical literacy is doing and using mathematics y
in a variety of “authentic” situations.These situations include personal life and work. Unit 6,
however simple, fits well into the definition and can be encountered in many variations in
daily life or at work.
In order to answer Sample Question 13, the student needs to understand the problem. In this
case the wording is straightforward, and confusion is hardly possible. It is almost a “standard”
problem: the solution is to compute the total area, and multiply this by the number of bricks
needed for one square metre, in this case 81. It could be argued therefore that this problem has
two stages.The mathematisation process (finding the relevant mathematics in the text, which is
5.25 times 3.00 and 81 per square metre) is perhaps a third stage.The question itself is a little
ambiguous because it uses natural language, asking “how many bricks Nick needs”, but this can
be translated into at least three possible correct answers. One answer is 1 276. Or one could
answer by stating how many “whole” bricks are needed (admittedly somewhat less natural):
1 275. Or one could give a “precise” number of bricks (including partial bricks): 1 275.75.The
most likely answer is that 15 square metres takes 1 215 bricks.The three remaining areas are
each of 1/4 square metre. Since 1/4 of 81 is about 20, we must add 60 bricks to the total of
1 215, giving 1 275.The format is therefore that of an open-constructed response item, as
students have the opportunity to adopt a variety of strategies.
The most obvious competencies required are modelling skill, to mathematise the problem,
and problem solving skill to find the correct answer. Communication skill is also needed
because there is more than one possible answer, and (as in almost all problems) technical
skill is needed for correct computation. Representation skill may be useful if students
want to visualise the problem in order to solve it. They might want to draw the patio and
divide it up into blocks of 81 bricks. This question seems to fit neatly into Class 2 as it is
not a reproduction problem, and requires simple problem solving skills, typical for this
level of competencies.

67
 ……………………………………………………………………………………………………………

M
a
t MATHEMATICS UNIT 7
h
e SEALS SLEEP
m
a
t
i A seal has to breathe even if it is asleep in the water. Martin observed a seal for one
c hour. At the start of his observation, the seal was at the surface and took a breath. It then
a
l dove to the bottom of the sea and started to sleep. From the bottom it slowly floated to the
surface in 8 minutes and took a breath again. In three minutes it was back at the bottom of
L the sea again. Martin noticed that this whole process was a very regular one.
i
t
e
r
a
Sample Question 14 (Multiple Choice)
c
y • Competency class 2: Connections and integration for problem solving
• Big idea: Change and growth
• Situation: Personal/Scientific

After one hour the seal was

a at the bottom
b on its way up
c breathing
d on its way down

Sample Question 14 demonstrates that simple mathematics can be used to make sense of the
world around us. Students need to analyse the text carefully and to recognise that the crucial
mathematical idea here is periodicity, which falls within the “big idea” of change and growth.
They need to transform this idea into a mathematical problem, which can be solved in
different ways.This is relatively complex because the information is not neatly organised for
students: first, they are given information about the diving part of the cycle, but not about
the time it takes.They find this information only after the numerical information about the
floating process.This is not what students are usually confronted with.They need to grasp
that a full cycle takes 11 minutes: three minutes to dive, eight minutes to float to the surface,
take a breath, and dive again.The mathematisation of the problem is that at t = 0, the seal
starts to dive, at t = 3, it is at the bottom, and at t = 11 it is at the surface again. And so on.
What is it doing at t = 60? Answer: if five cycles take 55 minutes, at t = 60 we are
5 minutes after the beginning of the cycle, and hence on its way up to the surface. But
students could also mathematise the problem by using less formal mathematics.They could
make a sketch with the surface and the bottom as two parallel lines and draw lines down
(3 mins.) and up (8 mins.), continuing this process until they reach the 60 minute mark.The

68
……………………………………………………………………………………………………………

M
a
competencies used here are modelling, representation and formal skills. Many variations and t
other strategies can be expected. h
e
The problem is not “close” to most students’ worlds. It is somewhat scientific although m
a
students might encounter this or similar problems nearer to home: predicting tides for those t
close to the sea, or arrival times of regular buses.The problem therefore is authentic, although i
it has been pre-modelled.The format is multiple choice, and machine marking is possible. c
a
l

L
i
t
e
r
a
c
y

…………
69
 ……………………………………………………………………………………………………………

M
a
t SCORING SCHEME FOR THE MATHEMATICS SAMPLE QUESTIONS
h
e
m
a
t MATHEMATICS UNIT 1 – PIZZA
i
c Sample Question 1
a Score 1: Answers which give the general reasoning that the surface area of pizza increases more
l
rapidly than the price of the pizza, concluding that the larger pizza is better value. For example:
L • “The diameter of the pizzas is the same number as their price, but the amount of pizza you get
i is found using squarred diameter, so you will get more pizza per zeds from the larger one.”
t
e or
r Answers which calculate the area and amount per zed for each pizza, concluding that the larger
a pizza is better value. For example:
c
y • “Area of smaller pizza is 0.25 x π x 30 x 30 = 225π; amount per zed is 23.6 cm2 area of
larger pizza is 0.25 x π x 40 x 40 = 400π; amount per zed is 31.4 cm2 so larger pizza is
better value.”
Score 0: Other incorrect responses. For example:
• “They are the same value for money.”
or
Answers which are correct but with an incorrect or insufficient reasoning. For example:
• “The larger one.”

MATHEMATICS UNIT 2 – COINS

Sample Question 2
Score 2: 15 - 20 - 26 - 34 - 45. It is possible that the response could be presented as actual
drawings of the coins of the correct diameters. This should be scored 2 as well.
Score 1: The answer gives a set of coins that satisfy the three criteria, but not the set that
contains as many coins as possible, for example:
• “15 - 21 - 29 - 39”, or “15 - 30 - 45.”
or
Answers which give the first four diameters correct, the last one incorrect, for example:
• “15 - 20 - 26 - 34 - .”
or
Answers which give the first three diameters correct, the last two incorrect, for example:
• “15 - 20 - 26 - .”
Score 0: Other incorrect responses.
70
……………………………………………………………………………………………………………

M
a
MATHEMATICS UNIT 3 – LICHEN t
h
e
Sample Question 3
m
Score 2: 14 mm or 14 (units not required). Score 2 should be given as long as the correct a
t
answer is 14 whether solution steps are shown or not. i
c
d = 7.0 x √16 - 12 a
Score 2
d = 14 l
• “14 mm.”
L
i
Score 1: Answers with partial responses, for example: t
• Correct substitution of value in the formula but incorrect answer. e
r
• Incomplete answers. a
Score 0: Other incorrect responses, for example: c
y
• “16.” (Incorrect answer with no solution steps shown).

Sample Question 4
Score 2: Answers which indicate 37 years or 37 (units not required), regardless of the presence
or absence of solution steps, for example:

35 = 7.0 x √t - 12
5 = √t - 12
Score 2
25 = t - 12
t = 37
Score 1: Answers which indicate a correct substitution of values in the formula but an incorrect
answer, for example:

35 = 7.0 x √t - 12
352 = 72 x t - 12
Score 1
35t = 1 752
t = 50

35 = 7.0 x √t - 12
35 = √t - 12 Score 1
1 225 = t - 12
t = 1 237

Score 0: Other incorrect responses.

71
 ……………………………………………………………………………………………………………

M
a
t MATHEMATICS UNIT 4 – SHAPES
h
e Sample Question 5
m
a Score 1: Answers which indicate shape B, supported with plausible reasoning, for example:
t • “B. It doesn’t have indents in it which decreases the area. A and C have gaps.”
i
c • “B, because it’s a full circle, and the others are like circles with bits taken out.”
a Score 0: Answers which indicate shape B, without plausible support.
l

L
Sample Question 6
i Score 1: Answers which indicate any reasonable method, such as:
t
e • “Draw a grid of squares over the shape and count the squares that are more than half filled
r by the shape.”
a • “Cut the arms off the shape and rearrange the pieces so that they fill a square then measure
c the side of the square.”
y
• “Build a 3D model based on the shape and fill it with water. Measure the amount of water
used and the depth of the water in the model. Derive the area from the information.”
Score 0: Other incorrect or incomplete answers. For example:
• “The student suggests to find the area of the circle and subtract the area of the cut out
pieces. However, the student does not mention about HOW to find out the area of the cut
out pieces.”

Sample Question 7
Score 1: Answers which indicate any reasonable method, such as:
• “Lay a piece of string over the outline of the shape then measure the length of string used.”
• “Cut the shape up into short, nearly straight pieces and join them together in a line, then
measure the length of the line.”
• “Measure the length of some of the arms to find an average arm length then multiply by 8
(number of arms) x 2.”
Score 0: Other incomplete or incorrect responses.

MATHEMATICS UNIT 5 – BRAKING

Sample Question 8
Score 1: 22.9 metres (units not required).
Score 0: Other.

Sample Question 9
Score 1: 101 metres (units not required).
Score 0: Other.
72
……………………………………………………………………………………………………………

M
a
Sample Question 10 t
h
Score 1: 5.84 seconds (units not required). e
m
Score 0: Other.
a
t
Sample Question 11 i
Score 1: 78.1 metres (units not required). c
a
Score 0: Other. l

Sample Question 12 L
i
Score 1: 90 kph (units not required). t
Score 0: Other. e
r
a
c
MATHEMATICS UNIT 6 – PATIO y
Sample Question 13
Score 2: Answers which indicate 1 275 or 1 276 (units not required). For example:
• “5.25 x 3 = 15.75 x 81 = 1 276.”
Score 1: Partially correct responses. For example:
• “15.75.” (units not required)
• “5.25 x 3 = 15.75
15.75 x 81 = 9 000”
• “5.25 x 3.0 = 15.75 m2; so 15.75 x 1 275.75 = 1 376 bricks.” (Here the student got the first
part right, but the second part wrong. Give credit for the first part and ignore the second part. So
score as 1)
or
• “1 215 bricks for 5m x 3m.” (This is used for students who are able to calculate the number of
bricks for an integer number of square metres, but not for fractions of square metres. See example
response.)

5m
81 81 81 81 81

81 81 81 81 81 3m

81 81 81 81 81

• “81 x 15 = 1 215; 1 215 + 21 = 1 236.”

Score 0: Other responses.

73
 ……………………………………………………………………………………………………………

M
a
t MATHEMATICS UNIT 7 – SEALS SLEEP
h
e Sample Question 14
m
a Score 1: Response option B – On its way up.
t Score 0: Other.
i
c
a
l

L
i
t
e
r
a
c
y

74
3

S
C
I
E
N
T
I
F
I
C
L
I
T
E
R
A
C
Y
 ……………………………………………………………………………………………………………

S
c
i ASSESSING SCIENTIFIC LITERACY IN PISA
e
n
t
i
f The PISA definition of scientific • Concepts and content: the scientific
i literacy and its context knowledge and conceptual understanding
c that are required in using these processes;
Scientific literacy is defined in PISA as:
L • Context: situations in which the processes
i and understanding are applied – such as the
t the capacity to use scientific knowledge, personal context of health and nutrition or
e to identify questions and to draw
r the global context of climate.
a evidence-based conclusions in order to
c understand and help make decisions Within each dimension, decisions have been
y about the natural world and the changes made about which components need to be
made to it through human activity. included – for example, what kinds of scientific
process are the most important to master. The
An important aspect of scientific literacy is intention has been to focus on literacy as a broad
that it is considered a key outcome of education competence rather than on the mastery of
by age 15 for all students, whether or not they curriculum content only. The arguments and
continue to learn science thereafter. Scientific decisions about these components are briefly
thinking is required by citizens, not just scientists. set out below.
It has been widely accepted in the past that reading
and mathematical skills are important for all Scientific processes
adults, in many life contexts. The inclusion of PISA emphasises the ability to use scientific
scientific literacy as a general competency for life knowledge and to know about science. The
reflects the growing centrality of scientific and assessment of such abilities helps us to under-
technological questions to 21st century life. Note stand how well science education is preparing
that the definition does not imply that tomorrow’s future citizens to participate in societies ever
adults need large reservoirs of scientific more affected by scientific and technological
knowledge, but that the key is to be able to think advances. Students should be equipped with
scientifically about the evidence that they understanding of the nature of science, of its
encounter. procedures, of its strengths and limitations and
of the kinds of question that it can, and cannot,
Three dimensions of scientific literacy answer. Students should also be able to recog-
nise the type of evidence required in a
To transform this definition into an scientific investigation and the extent to which
assessment of scientific literacy, three broad reliable conclusions can be drawn from
dimensions have been identified. These are: evidence. It is considered important for
• Scientific processes or skills: the students to be able to communicate their
mental processes that are involved in understanding and arguments effectively to
addressing a question or issue (such as particular audiences, for otherwise they will
identifying evidence or explaining have no voice in the matters that are debated
conclusions); in society.

76
……………………………………………………………………………………………………………

S
c
It should be possible for all these abilities to 3. Drawing or evaluating conclusions i
be derived from experiencing science at first e
This process involves relating conclusions to n
hand and from investigations and experiments
the evidence on which they are based, or t
in school. However, the concern in PISA is not i
to find out whether students can undertake should be based.
f
scientific investigations for themselves, but i
rather whether their school experiences have It can be assessed, for example, by providing c
culminated in an understanding of scientific students with an account of an investigation
processes and the ability to apply scientific and the conclusions drawn from it and asking L
for an evaluation of these conclusions, or by i
concepts that enable them “to make decisions t
about the natural world and the changes made asking for a conclusion or alternative
e
to it through human activity”. conclusions to be drawn that are consistent r
with given evidence. a
These arguments have led to the c
4. Communicating valid conclusions y
identification of the following scientific
processes for assessment in PISA: The process involved here is that of
expressing, in a way that is appropriate to a
1. Recognising scientifically investigable given audience, the conclusions that can be
questions drawn from available evidence.

This means being able to identify the kinds It can be assessed, for example, by
of question that science can attempt to presenting students with a situation which
answer, or the specific question that is, or requires information or evidence from
might be, tested in a particular situation. different sources to be brought together to
support a given course of action or
It can be assessed, for example, by conclusion. The emphasis here is on the
presenting a situation in which questions clarity of communication rather than the
could be answered scientifically and asking particular conclusion being communicated,
for these to be identified, or presenting provided that this is consistent with scientific
several questions and asking which can be understanding.
answered by scientific investigation.
5. Demonstrating understanding of
2. Identifying evidence needed in a scientific concepts
scientific investigation
This is the process of showing understanding
This process involves identifying or by being able to apply concepts in situations
proposing the evidence required to answer different from those in which they were
the questions posed in a scientific learned. It involves not only recalling
investigation, or the procedures needed to knowledge but also showing the relevance
gather that evidence. of that knowledge or using it in making
predictions or giving explanations.
It can be assessed, for example, by presenting
an investigation and asking students to identify It can be assessed, for example, by asking
the evidence needed or the action to be taken for explanations or predictions about a given
to obtain valid evidence. situation, phenomenon or event.
77
 ……………………………………………………………………………………………………………

S
c
i It is acknowledged and indeed emphasised The areas of application of science have been
e that in using all of these processes some grouped under three broad headings:
n
scientific knowledge is required. This is most 1. Science in life and health,
t
i obvious in Process 5, but it applies to Processes 2. Science in Earth and environment, and
f 1 to 4 equally, for these cannot be described as 3. Science in technology.
i “scientific processes” unless they are applied in
c relation to scientific content. Problems and issues under these headings
can affect us as individuals, as members of a
L Scientific concepts local community or as world citizens; often as
i all three. Moreover, some areas in which science
t The scientific concepts selected in PISA are is applied have a long history, illustrating
e expressed as broad integrating ideas that help
r changes in scientific understanding over time
to explain aspects of our material environment. and providing opportunities for recognising the
a
c
The PISA framework does not attempt to application of science in contexts that are no
y identify all concepts that would meet this longer familiar today.
criterion; assessing them comprehensively
would be impossible within the constraints of The situations that can be used for assessing
limited test “space”. Instead, concepts will be scientific literacy can thus be characterised both
sampled from the following main themes: by the broad area of application and by the aspects
1. Structure and properties of matter of our lives in which they are relevant (Table 1).
2. Atmospheric change
3. Chemical and physical changes Format of test questions
4. Energy transformations
5. Forces and movement The test used to assess scientific literacy is a
6. Form and function series of “units”, each dealing with a particular
7. Human biology problem or issue. Assessment units present
8. Physiological change students with a real-life situation, taken from an
9. Biodiversity authentic source, and a series of questions about
10. Genetic control it. Each question requires the use of one or more
11. Ecosystems
12. The Earth and its place in the universe of the process skills and some scientific
13. Geological change knowledge. The presentation of the stimulus
material (problem or issue) requires the reading
Situations and areas of application of some text, table or diagrammatic
representation. However, since several questions
The PISA definition of scientific literacy are linked to the same stimulus material in each
emphasises the application of processes and unit, the overall time spent reading, rather than
concepts in relation to problems and issues in the answering a question, is no greater than in a series
real world. Students who have acquired some of stand-alone items in a conventional test.
measure of scientific literacy will be able to apply
what they have learned in school and non-school In the following examples, only two of the
situations. A scientific situation is used here to planned items within each unit are presented,
indicate a real-world phenomenon in which in the interests of providing a range of situations.
science can be applied. Note the distinction There will, therefore, appear to be a greater
between a scientific concept (such as atmospheric reading load for each question than is in fact
change) and an aspect of our world in which it is the case for the actual PISA assessment. The
applied (such as weather or climate). commentary on each question indicates the
78
……………………………………………………………………………………………………………

S
c
Table 1 i
Situations used for assessing scientific literacy e
n
Relevance Areas of application t
i
Science in life and health Science in Earth and environment Science in technology f
Personal, Health, disease and Pollution; Production and loss of Biotechnology; Use of i
Community, nutrition; Maintenance soil; Weather and climate. materials and waste c
Global, and sustainable use of disposal; Use of
Historical. species; Interdependence energy; Transportation. L
of physical and biological i
systems. t
e
r
process that is required for answering, the responses give valuable information about a
c
scientific knowledge that is called upon and the students’ ideas and thinking, which can be fed y
area of application. back into curriculum planning, the marking
guides for items in the main study were revised
In the case of Process 5, the aim is to assess to include a system of two-digit marking so that
understanding of a scientific concept through the frequency of various types of correct and
applying it to the situation. For Processes 1 to 4, incorrect responses can be recorded. It is
on the other hand, the scientific knowledge that important to note that markers are advised to
is required is not intended to constitute a major ignore spelling and grammar errors, unless they
obstacle to answering: it is ability in the process completely obscure the meaning, because this
that is being assessed. For some, this appears to is not a test of written expression. The PISA
mean that answering the question is a matter of marking scheme for open-ended items is shown
“common sense” rather than scientific at the end of this chapter.
understanding. However, common sense of this
kind is indeed part of scientific literacy, derived Sample questions
from scientific knowledge which has become
internalised and part of rational thinking.We make The following examples illustrate the range
no apology for this; after all, did not Einstein of tasks and questions used in the PISA
describe science as nothing more than “a assessment of scientific literacy. These items
refinement of everyday thinking”? were used in the PISA field trial but not selected
for the PISA 2000 assessment because of their
Marking similarity to other item sets in terms of what
they measure.The intention in presenting these
About two-thirds of the items used in PISA tasks and questions is to demonstrate the
are in a form that can be unambiguously marked connection between the PISA assessment
as correct or incorrect. Either they are in a fixed- framework and the items that have been
answer form or they require only a few words to constructed to represent it.
be supplied. Other items will require extended
responses and can often be marked as incorrect, Since the PISA 2000 assessment was not
partially correct or correct.The marking scheme completed by the time this publication was
for these open questions provides not only general produced, items from the PISA 2000 tests could
guidelines but examples of answers for each not be included in this publication for reasons
response category. Furthermore, since these of test security. …………
79
 ……………………………………………………………………………………………………………

S
c
i SCIENCE UNIT 1
e
n BUSES
t
i
f
i
A bus is moving along a straight stretch of road. The bus driver, named Ray, has a cup
c
of water resting in a holder on the dashboard:
L
1 2
i
t
e
Water
r
a
c Driving direction
y
Suddenly Ray has to slam on the brakes.

Sample Question 1 (Multiple Choice)

• Process: Demonstrating understanding of scientific concepts
• Concept: Forces and movement
• Situation/Area of application: Science in technology (transportation)

What is most likely to happen to the water in the cup immediately after
Ray slams on the brakes?
A The water will stay horizontal.
B The water will spill over side 1.
C The water will spill over side 2.
D The water will spill but you cannot tell if it will spill over side 1 or
side 2.

Sample Question 1 is the first question in a unit about buses, requiring students to consider
scientific aspects of a commonly experienced form of transport.The question uses this situation
to assess knowledge of the momentum of moving objects and of forces needed to stop
movement.When the bus stops quickly, the water in the cup continues to move in the direction
in which it is already moving and so is likely to spill over the side furthest forward.The force
of reaction against the side of the cup will push the water back, giving rise to the familiar
experience in this situation of a liquid spilling both forward and back. However, to identify
which side it spills over first requires knowledge of the forces acting on it.The fixed answer
form of response is appropriate here since the options are limited.

80
……………………………………………………………………………………………………………

S
c
Sample Question 2 (Open-Constructed Response) i
e
• Process: Drawing or evaluating conclusions n
• Concept: Energy transformations t
• Situation/Area of application: Science in Earth and environment (pollution) i
f
i
Ray’s bus is, like most buses, powered by a diesel engine. These buses c
contribute to environmental pollution.
L
A colleague of Ray works in a city where they use trolley buses. They i
are powered by an electric engine. The voltage needed for such an t
electric engine is provided by overhead lines (like electric trains). The e
electricity is supplied by a power station using coal. r
a
Supporters of the use of trolley buses in the city say that these buses c
don’t contribute to air pollution. y
Are these supporters right? Explain your answer.

Sample Question 2 moves the focus to a different feature of buses, as contributors to air
pollution. Atmospheric pollution is a major concern for the future, and it is important that
students are able to make informed decisions in relation to it.The introductory text to this
question presents a conclusion, drawn by some people, that electrically driven trolley buses do
not contribute to air pollution. Students are required to evaluate the validity of this
conclusion, using the information given in the question and their own knowledge of the
products of burning coal in power stations.To gain credit, students must mention in their
answer the pollution caused by the coal-burning power stations that produce the electricity,
even if they recognise that the electricity users do not themselves cause pollution.

How did the students answer? Examples (See end of chapter for marking scheme)

• “Yes and no. The buses don’t pollute the city which is good, but the power station does pollute
and that’s not very good.” (Score 1)
• “The buses do contribute to the air pollution by using fossil fuels but they’re not as harmful as
normal buses with all their gases.” (Score 1)
• “Well they have no outlet so no harmful smoke goes into the air which can damage the
Ozone layer, and having electricity created by fossil fuels is also more environmental friendly.”
(Score 0)
• “Yes, they are. Because electricity isn’t harmful for the environment we only use up our Earth’s
gas.” (Score 0)
• “I think these supporters are right as diesel engined buses create more pollution than trolley
buses using electricity.” (Score 0)
• “Yes, because while burning coal no pollution gases will be set free.” (Score 0)

81
 ……………………………………………………………………………………………………………

S
c
i SCIENCE UNIT 2
e
n FLIES
t
i
f Read the following information and answer the questions below.
i
c

L A farmer was working with dairy cattle at an agricultural experiment station. The
i population of flies in the barn where the cattle lived was so large that the animals’ health
t
e was affected. So the farmer sprayed the barn and the cattle with a solution of insecticide
r A. The insecticide killed nearly all the flies. Some time later, however, the number of
a flies was again large.The farmer again sprayed with the insecticide.The result was similar
c to that of the first spraying. Most, but not all, of the flies were killed. Again, within a
y short time the population of flies increased, and they were again sprayed with the
insecticide. This sequence of events was repeated five times: then it became apparent
that insecticide A was becoming less and less effective in killing the flies.

The farmer noted that one large batch of

the insecticide solution had been made and The subject of Unit 2, the use of insecticides
used in all the sprayings. Therefore he in agriculture, is one that is gaining in
suggested the possibility that the insecticide importance. Intensive food production has
solution decomposed with age. brought with it greater use of insecticides
and herbicides, whose effect we now know
Source: Teaching About Evolution and the Nature of
tends to diminish with continued use.
Science, National Academy Press, Washington, DC, However, in specific cases, as in that presented
1998, p. 75. in this unit, there could be other reasons
besides the development of resistance in the
organisms targeted.Thus students are
presented here with questions set in a context
that involves genuine and important issues.
Sample Question 3
(Open-Constructed Response)

• Process: Identifying evidence needed in a scientific investigation

• Concept: Chemical and physical changes
• Situation/Area of application: Science in life and health (health, disease and nutrition)

The farmer’s suggestion is that the insecticide decomposed with age.

Briefly explain how this suggestion could be tested.

In Sample Question 3, attention is directed to the farmer’s own suggestion for the decline in
the effectiveness of insecticide A. Students are asked to identify the kind of evidence that
would enable the suggestion to be tested. Scientific knowledge of the meaning of
decomposition and of how this would change the chemical make-up, and thus the effect,

82
……………………………………………………………………………………………………………

S
c
of insecticide A is a necessary background for understanding this question.There are several i
scientifically valid ways of finding out whether change in the insecticide was responsible for e
n
its reduced effect. One is an experimental approach, involving comparison of new and old t
batches in a controlled test. Variables which would need to be controlled in this test are the i
type of fly, age of insecticide and the amount of exposure to insecticide. However, answers of f
this kind are accepted without explicit reference to these controls, since they are often clearly i
c
implied. Partial credit is given for answers which suggest some procedure that is relevant but
is not sufficient to make the comparisons necessary. L
i
t
How did the students answer? Examples (See end of chapter for marking scheme) e
r
• “Some flies could be taken. If they would both be put in a separate box you could use a new a
spray and an older spray and see what the results are.” (Score 2) c
y
• “Make one big batch of spray. Have two groups of flies and spray each group every six months.
Spray group 1 with the big batch, and group 2 a new batch each time.” (Score 2)
• “Try a new bottle of it, then wait till it gets a bit older and the flies come back and then try
again.” (Score 2)
• “Take batches of the insecticide to a laboratory every few months and have its strength tested.”
(Score 2)
• “Do the same thing but buy new insecticide each time, hence proving if his theory is right or
wrong.” (Score 1)
• “Maybe if he sent a fresh batch of the poison to the lab with a batch of the old stuff and get them
retested the results may prove his theory.” (Score 1)
• “He could test it every year to see if it is not old and would still work.” (Score 0)
• “Get a fly from his shed and another shed and spray them each with the insecticide.” (Score 0)

Sample Question 4 (Open-Constructed Response)

• Process: Drawing or evaluating conclusions
• Concept: Physiological change
• Situation/Area of application: Science in life and health (health, disease and nutrition)

The farmer’s suggestion is that the insecticide decomposed with age.

Give two alternative explanations as to why “insecticide A was becoming
less and less effective ...”
Explanation 1:
_____________________________________________________________
Explanation 2:
_____________________________________________________________
83
 ……………………………………………………………………………………………………………

S
c
i The stimulus for this unit presents observations from which alternative conclusions can be
e drawn. Sample Question 4 focuses on possible reasons for the diminishing effect of the
n
t insecticide, apart from the one suggested by the farmer. In the case of answers referring to
i the resistance of the flies, scientific knowledge of physiological change and of the
f possibility of consequent inherited resistance is required. Almost all students assessed in
i the PISA field trial included this as one of their answers. For example: “With the repeated
c
use of the same insecticide the flies were becoming immune to the formula.” The other two
L accepted explanations involve recognising the possibility of changes in the environmental
i conditions and a change in the way the insecticide was applied – conclusions which can
t be drawn from the evidence available. Questions such as this, where there are three (and
e
r in some cases, more) possible correct answers but where only two are required, raise a
a general issue about marking. It can sometimes be argued that one particular answer is
c worth more than others, and therefore that full credit should be given only when this one
y is included. In the case of Sample Question 4, the response referring to the acquired
resistance of the flies might be considered more important than the other two responses.
However, if the question does not signal to the student the preference for one type of
answer, it is only fair to treat all responses equally.

How did the students answer? Examples (See end of chapter for marking scheme)

• Explanation 1: “With the repeated use of the same insecticide the flies were becoming immune
to the formula.”
Explanation 2: “Over time chemicals in the insecticide rose to the top of spray can leaving water
diluted (ineffective) at the bottom.” (Score 2)
• Explanation 1: “The flies were becoming immune to the spray.”
Explanation 2: “Heat may make it decompose and temperature change.” (Score 2)
• Explanation 1: “Maybe the flies developed a defense gene so the insecticide would not work.”
Explanation 2: “He (the farmer) used less each time.” (Score 2)
• Explanation 1: “The temperature got very hot and affected the insecticide.”
Explanation 2: “The farmer did not spray the insecticide on the flies properly.” (Score 2)
• “He might not have sprayed it properly.” (Score 1)
• “The flies could have built up an immunity.” (Score 1)
• “There were different types of flies each time.” (Score 1)
• “The flies could have been breeding.” (Score 0)
• “Because every time he sprayed it it became less and less effective.” (Score 0)
• “When there is more of it in the can it is stronger.” (Score 0)

84
……………………………………………………………………………………………………………

S
c
SCIENCE UNIT 3 i
e
BIODIVERSITY n
t
i
Read the following newspaper article and answer the questions below. f
i
c
BIODIVERSITY IS THE KEY TO MANAGING ENVIRONMENT
L
An ecosystem that retains a high biodiversity (that is, a wide variety of living things) is i
much more likely to adapt to human-caused environment change than is one that has little. t
e
Consider the two food webs shown in the diagram.The arrows point from the organism r
that gets eaten to the one that eats it. These food webs are highly simplified compared with a
food webs in real ecosystems, but they still illustrate a key difference between more diverse c
and less diverse ecosystems. y

Food web B represents a situation with very low biodiversity, where at some levels the
food path involves only a single type of organism. Food web A represents a more diverse
ecosystem with, as a result, many more alternative feeding pathways.
Generally, loss of biodiversity should be regarded seriously, not only because the
organisms that have become extinct represent a big loss for both ethical and utilitarian
(useful benefit) reasons, but also because the organisms that remain have become more
vulnerable (exposed) to extinction in the future.

FOOD WEB A FOOD WEB B

Eucalupt Wattle
Wattle Tea Tree

Beetle Leaf Hopper

Leaf Hopper Butterfly Larvae

Spider
Parasitic Wasp
Parasitic Wasp Honeyeater

Lizard
Robin Robin
Lizard

Snake Raven Raven

Native Cat Native Cat
Snake

Source: Adapted from Steve Malcolm: “Biodiversity is the key to managing environment”,
The Age, 16 August 1994.
……
85
 ……………………………………………………………………………………………………………

S
c
i
e The maintenance of a wide variety of species has far-
n reaching consequences on a global scale and over a fairly
t
i long time span, although its importance may not be
f immediately obvious in everyday life.The impact of the loss
i of a species, which can happen for any of a number of
c reasons, including human activity, may be felt in
unexpected ways. Understanding the chain of events and
L consequences depends on knowledge of the interdependence
i
t of living things and the ability to use this knowledge to
e predict how changes in the population of some living things
r can affect others. Food webs are a useful way of presenting
a and explaining such relationships and are encountered
c universally in science curricula. Unit 3 on biodiversity
y presents two food webs, one representing a more diverse
ecosystem than the other.

Sample Question 5 (Multiple Choice)

• Process: Drawing or evaluating conclusions
• Concept: Ecosystems
• Situation/Area of application: Science in life and health (maintenance and sustainable us of species)

In the article it is stated that “Food web A represents a more diverse

ecosystem with, as a result, many more alternative feeding pathways.”
Look at FOOD WEB A. Only two animals in this food web have three direct
(immediate) food sources. Which two animals are they?
A Native Cat and Parasitic Wasp.
B Native Cat and Raven.
C Parasitic Wasp and Leaf Hopper.
D Parasitic Wasp and Spider.
E Native Cat and Honeyeater.

The first items in the unit (of which Sample Question 5 is one) explore students’ ability to
“read” the food webs and to interpret correctly what they represent. Sample Question 5
requires students to use their knowledge of food webs and the information given in food web A
to identify the conclusion that fits the data.The answer is to be found in the information
given in food web A, and students do not have to recall knowledge of the particular animals
involved.

86
……………………………………………………………………………………………………………

S
c
Sample Question 6 (Multiple Choice) i
e
• Process: Drawing or evaluating conclusions n
• Concept: Ecosystems t
• Situation/Area of application: Science in life and health (maintenance and sustainable us of species) i
f
Food webs A and B are in different locations. Imagine that Leaf Hoppers i
died out in both locations. Which one of these is the best prediction c
and explanation for the effect this would have on the food webs?
L
A The effect would be greater in food web A because the Parasitic Wasp i
has only one food source in web A. t
B The effect would be greater in food web A because the Parasitic Wasp e
r
has several food sources in web A. a
C The effect would be greater in food web B because the Parasitic Wasp c
has only one food source in web B. y
D The effect would be greater in food web B because the Parasitic Wasp
has several food sources in web B.

Sample Question 6 is one of several in the unit which explore students’ understanding of the
impact of change on ecosystems. Other questions require students to generate reasons, in the
context of the given food webs, why loss of diversity is regarded seriously. Sample Question 6
requires the comparison of the two food webs and recognition of the greater impact of
population change on a less diverse ecosystem. A fixed answer format is used, with emphasis on
the reason rather than the choice of food web. Credit is only given for the combination of the
correct web and relevant reason.The careful reading of the alternative answers given in the
question did not appear to be an obstacle in pilot trials, since the question was answered
correctly by 60 per cent of the students in the PISA field trial.

SCIENCE UNIT 4
CLIMATE CHANGE

Read the following information and answer the questions below.

WHAT HUMAN ACTIVITIES CONTRIBUTE TO CLIMATE CHANGE?

The burning of coal, oil and natural gas, as well as deforestation and various agricultural
and industrial practices, are altering the composition of the atmosphere and contributing
to climate change. These human activities have led to increased concentrations of particles
and greenhouse gases in the atmosphere.

87
 ……………………………………………………………………………………………………………

S
c
i
e The relative importance of the main contributors to temperature change is shown in
n
t Figure 1.
i
f Figure 1: Relative importance of the main contributors to changes
i in the temperature of the atmosphere
c
Cooling Relative importance Heating
L
i Carbon dioxide
t
e
r Methane
a
c Particles
y
Particles effects on clouds

Known effect
Possible effect

Figure 1 shows that increased concentrations of carbon dioxide and methane have a
heating effect. Increased concentrations of particles have a cooling effect in two ways,
labelled “Particles” and “Particle effects on clouds”.

Bars extending to the right of the centre line indicate a heating effect. Bars extending to
the left of the centre line indicate a cooling effect. The relative effect of “Particles” and
“Particle effects on clouds” are quite uncertain: in each case the possible effect is somewhere
in the range shown by the light grey bar.

Source: Adapted from US Global Change Research Information Office,

http://www.gcrio.org/ipcc/qa/04.html
The stimulus material for Unit 4 is taken
from information found on the Internet
about the contributions of various factors to
climate change.The Internet is a source of
information that will be used more and more
frequently in the lives of today’s students.
However, some explanation has been added
so that the way in which the data is
presented presents no obstacle in the task set.
The topic is one in which the evidence is
inconclusive in relation to possible action
that might be taken.

88
……………………………………………………………………………………………………………

S
c
Sample Question 7 (Open-Constructed Response) i
e
• Process: Communicating valid conclusions n
• Concept: Atmospheric change t
• Situation/Area of application: Science in Earth and environment (weather and climate) i
f
i
Use the information in Figure 1 to support the view that priority c
should be given to reducing the emission of carbon dioxide from the
L
human activities mentioned. i
t
Sample Question 7 requires students to use the information given to support a particular e
r
course of action, namely to reduce the emission of carbon dioxide as a result of human a
activity. A further question (not shown here) asks students to use the same information in a c
different way, to support the view that the effects of human activity do not constitute a real y
problem. In both cases, the questions assess the ability to communicate conclusions based on
evidence. In combination, the two questions exemplify the caution with which scientific
information has to be applied in such complex cases.The questions depend on knowledge of
the subject matter, such as why cooling and heating are important in climate change and how
carbon dioxide, methane and particles in the air can bring about these effects. However, the
focus is on constructing an argument using the given data and thus assesses the ability to
communicate effectively. In this example, credit is given for answers which identify the
information of relevance to the argument put forward and no credit is given for answers which
fail to focus on the argument for reduction, even if they concern the activities that contribute
to the production of carbon dioxide.

How did the students answer? Examples (See end of chapter for marking scheme)

• “The emission of CO2 causes significant heating to the atmosphere and therefore should be
lessened.” (Score 1)
• “Carbon dioxide in the Earth’s atmosphere heats up the Earth. Normally this wouldn’t be a
problem if the possible effects were proven true. However, they haven’t so the graph shows an
increase in heat. This shows it is necessary to cut down on carbon dioxide emissions.” (Score 1)
(Note: This answer distinguishes between known and possible effects)
• “According to Figure 1 reduction in the emission of carbon dioxide is necessary because it
considerably heats the Earth.” (Score 1)
• “The burning of fossil fuel such as oil, gas and coal are contributing to the build up of gases in
the atmosphere, one of which is carbon dioxide (CO2). This gas affects the temperature of the
Earth which increases causing a greenhouse effect.” (Score 0)
• “The way that humans could help control carbon dioxide levels to drop would be by not driving
a car, don’t burn coal and don’t chop down forests.” (Score 0)

89
 ……………………………………………………………………………………………………………

S
c
i SCIENCE UNIT 5
e
n CHOCOLATE
t
i
f Read the following summary of an article in the newspaper the Daily Mail on March 30, 1998
i and answer the questions below.
c

L A newspaper article recounted the story of a 22-year-old student, named Jessica, who has a
i “chocolate diet”. She claims to remain healthy, and at a steady weight of 50kg, whilst eating 90
t bars of chocolate a week and cutting out all other food, apart from one “proper meal” every five
e days. A nutrition expert commented: “I
r
a am surprised someone can live with a Unit 5 contains several questions bearing on the
c diet like this. Fats give her energy to live understanding of a healthy diet and knowledge of the
y but she is not getting a balanced diet. different kinds of food that are needed.The questions
There are some minerals and nutrients lead to the conclusion that the student’s chocolate diet
in chocolate, but she is not getting is far from healthy and indeed provides more energy
enough vitamins. She could encounter than she needs.This unit is presented here to illustrate
serious health problems in later life.” an engaging and important topic and the variety of
questions that can be asked.
Sample Question 8 In a question not shown here, a further table of
(Open-Constructed Response) information is given from which students have to extract
evidence required to test the claim that Jessica’s weight
• Process: Demonstrating understanding of remains steady.This question assesses the ability to relate
scientific concepts the text to the information given in the table, an aspect of
• Concept: Energy transformations
• Situation/Area of application: Science in reading literacy. A further question not shown here asks
life and health (health, disease and nutrition) for possible reasons why Jessica may be able to maintain a
steady weight despite her excessive intake of energy.
A book with nutritional values
gives the data in Table 1 about
chocolate. Assume that all these data are applicable to the type of
chocolate Jessica is eating all the time. Assume also that the bars of
chocolate she eats have a weight of 100 grams each.
Table 1: Nutritional content of 100 g chocolate

Proteins Fats Carbonhydrates Minerals Vitamins TOTAL ENERGY

Calcium Iron A B C
5g 32 g 51 g 50 mg 4 mg - 0.2 mg - 2 142 kJ

According to the table, 100 g of chocolate contain 32 g of fat and give

2 142 kJ of energy. The nutritionist said: “Fats give her the energy to live.....”.
If someone eats 100 g of chocolate, does all the energy (2 142 kJ) come
from the 32 g of fat? Explain your answer using data from the table.
90
……………………………………………………………………………………………………………

S
c
Sample Question 8 presents information about the nutritional value of chocolate, which is assumed to i
apply to the type consumed by Jessica. It indicates that chocolate contains some minerals and vitamins e
n
as well as fats, protein and carbohydrates.The question posed concerns the energy contributions of t
these components and requires knowledge that energy comes from fats, proteins and carbohydrates and i
not from minerals and vitamins, which have other functions in a healthy diet.The question therefore f
requires not just recall but also application of knowledge in a real-life situation. i
c
Full credit is given for answers which indicate that energy comes not only from the fats but
also from the proteins and carbohydrates (one or both) contained in the chocolate. For L
example: “No, because carbohydrates, for example, provide even more energy than fats.” This i
answer can be regarded as correct because although the amount of energy from 1 g of t
e
carbohydrate is smaller than that from 1 g of fat, the student may mean that more r
carbohydrate is present. Partial credit is given for indicating that energy comes from vitamins a
and/or minerals as well as proteins and carbohydrates:“I don’t think so, because it can also c
come from the carbohydrates, minerals and vitamins.” y

Sample Question 9 (Multiple Choice)

• Process: Demonstrating understanding of scientific concepts
• Concept: Physiological change
• Situation/Area of application: Science in life and health (health, disease and nutrition)

The nutrition experts said that Jessica “... is not getting nearly enough
vitamins”. One of those vitamins missing in chocolate is vitamin C. Perhaps she
could compensate for her shortage of vitamin C by including a food that
contains a high percentage of vitamin C in her “proper meal every five days”.
Here is a list of types of food.
1. Fish.
2. Fruit.
3. Rice.
4. Vegetables.
Which two types of food from this list would you recommend to Jessica
in order to give her a chance to compensate for her vitamin C shortage?
A 1 and 2.
B 1 and 3.
C 1 and 4.
D 2 and 3.
E 2 and 4.
F 3 and 4.
In Sample Question 9, a correct answer depends on knowledge of the sources of one of the
most important components of a healthy diet.This is knowledge that is necessary for students
to make informed decisions about their own and others’ diets.

91
 ……………………………………………………………………………………………………………

S
c
i SCIENCE UNIT 6
e
n CALF CLONING
t
i
f Read the following article about the birth of five calves and answer the questions below.
i
c

L In February 1993 a research team of the National Institute for Agricultural Research in
i Bresson-Villiers (France) succeeded in producing five clones of calves. The production of
t the clones (animals with the same genetic material, even though born to five different
e
r
cows), was a complicated process.
a
c First the researchers removed about thirty egg cells from a cow (let us say the cow’s
y name was Blanche 1). The researchers removed the nucleus from each of the egg cells
taken from Blanche 1.

Then the researchers took an embryo from another cow (let us say Blanche 2). This
embryo contained about thirty cells.

The researchers separated the ball of cells from Blanche 2 into individual cells.

Then they removed the nucleus from each of these individual cells. Each nucleus
was injected separately into one of the thirty cells that came from Blanche 1 (cells from
which the nuclei had been removed).

Finally the thirty injected egg cells were implanted into thirty surrogate cows. Nine
months later, five of the surrogate cows each gave birth to
a calf clone.

One of the researchers said that a large scale In Unit 6, the text provides an
application of this cloning technique could be financially account of an experiment and
beneficial for cattle breeders. its outcome.The idea being
tested in the experiment is not
Source: Corinne Bensimon, Libération, March 1993.
explicitly stated and students
are asked to identify what this
could have been.
92
……………………………………………………………………………………………………………

S
c
Sample Question 10 (Open-Constructed Response) i
e
• Process: Recognising scientifically investigable questions n
• Concept: Genetic control t
• Situation/Area of application: Science in technology (biotechnology) i
f
The results of the French experiments on cows confirmed the main idea i
that was being tested. What main idea could have been tested in this c
experiment?
L
i
A correct answer to Sample Question 10 involves recognising the kind of question that can be t
addressed in a scientific investigation, in this case in relation to subject matter that requires e
knowledge of cell division and of the genetic significance of the nucleus of a cell.The marking r
a
scheme gives credit for plausible answers, such as “That cloning was possible”, which do not c
mention calves or cows. No credit is given for ideas that could be tested but were not tested in y
this particular investigation or for ideas that are impossible to answer scientifically.

How did the students answer? (See end of chapter for marking scheme)
• That cloning was possible. (Score 1)
• That all cells of cows are the same. (Score 0)
• Mass cloning could be achieved. (Score 0)

Sample Question 11 (Complex Multiple Choice)

• Process: Demonstrating understanding of scientific concepts
• Concept: Genetic control
• Situation/Area of application: Science in technology (biotechnology)

Which of the following statements is/are true? Circle Yes or No for each.
Statement:
All five calves have the same type of genes. Yes/No
All five calves have the same sex. Yes/No
The hair of all five calves has the same colour. Yes/No

Here, the question presents statements about the result of cloning in the stimulus article.
However, these statements are not drawn from the experimental data and so are not being
evaluated in relation to the evidence given. Had this been the case, the process assessed would
have been “drawing or evaluating conclusions”. Instead, the student has to apply his or her
knowledge of genetics in answering. Hence, while the concept and area of application involved
are the same as in Sample Question 11, the process is different.

93
 ……………………………………………………………………………………………………………

S
c
i SCORING SCHEME FOR THE SCIENCE SAMPLE QUESTIONS
e
n
t
i
f SCIENCE UNIT 1 – BUSES
i
c Sample Question 1
Score 1: Response option C – The water will spill over side 2.
L
i Score 0: Other.
t
e Sample Question 2
r
a Score 1: Answers which state that the power station or the burning of coal also contributes
c to air pollution. For example:
y • “No, because the power station causes air pollution as well.”
• “Yes, but this is only true for the trolley bus itself; burning coal, however, causes air pollution.”
Score 0: No or yes, without a correct explanation.

SCIENCE UNIT 2 – FLIES

Sample Question 3
Score 2: Answers of type a, b or c:
a. Answers which mention control of all three variables (type of flies, age of insecticide, and
exposure), for example:
• “Compare the results from a new batch of the insecticide with results from the old
batch on two groups of flies of the same species that have not been previously exposed
to the insecticide.”
b. Answers which mention control of two of the three variables (type of flies, age of insecticide,
and exposure), for example:
• “Compare the results from a new batch of the insecticide with the results from the old
batch on the flies in the barn.”
c. Answers which mention control of only one variable of three variables (type of flies, age
of insecticide, and exposure), for example:
• “(Chemically) analyse samples of the insecticide at regular intervals to see if it changes
over time.”
Score 1: Answers of type d or e:
d. “Spray the flies with a new batch of insecticide, but without mentioning comparison with
old batch.”
e. “(Chemically) analyse samples of the insecticide but without mentioning comparison of
analyses over time.”
Note: Score 1 if sending samples of the insecticide to a laboratory is mentioned.
Score 0: Other.
94
……………………………………………………………………………………………………………

S
c
Sample Question 4 i
e
Score 2: Answers which give any two of the following explanations: n
• “Flies with resistance to the insecticide survive and pass on that resistance to later t
generations (also credit for ‘immunity’ although it is recognised that it is not strictly i
f
analogous to ‘resistance’).” i
• “A change in the environmental conditions (such as temperature).” c
• “A change in the way the insecticide was applied (including variation in the amount used).”
L
Score 1: Answers which give one explanation from the above. i
t
Score 0: Other, including new flies moving to the barn from nearby (unsprayed) areas. e
r
a
SCIENCE UNIT 3 – BIODIVERSITY c
y
Sample Question 5
Score 1: Response option A – Native Cat and Parasitic Wasp.
Score 0: Other.

Sample Question 6
Score 1: Response option B – The effect would be greater in food web B because the Parasitic
Wasp has only one food source in web B.
Score 0: Other.

SCIENCE UNIT 4 – CLIMATE CHANGE

Sample Question 7
Score 1: Answers which indicate that:
• Carbon dioxide is relatively the largest contributor to the heating effect and/or the effect
of carbon dioxide is a known effect.
or
• Carbon dioxide is relatively the largest contributor to the heating effect and/or the effect
of carbon dioxide is a known effect, but which also mention that the possible effects of
particles should be considered.
Score 0: Other, including answers which
• Do not indicate that carbon dioxide is relatively the largest contributor to the heating effect.
or
• Do not focus on the fact that the effects of carbon dioxide are known.
or
• Indicate that an increase in temperature will have a bad effect on the Earth.
or
• Focus on the activities that contribute to carbon dioxide production.
95
 ……………………………………………………………………………………………………………

S
c
i SCIENCE UNIT 5 – CHOCOLATE
e
n Sample Question 8
t
i Score 2: Answers which indicate “no” and explain that some energy comes from carbohydrates
f or proteins or carbohydrates + proteins.
i
c Score 1: Answers which indicate “no” and explain that some energy comes from carbohydrates
or proteins or carbohydrates + proteins and also from vitamins and/or minerals.
L Score 0: Answers which
i
t
• Indicate “yes”.
e or
r • Indicate “no” with no explanation.
a or
c • Indicate “no” with irrelevant comment.
y or
• Indicate “no” with the explanation that (only) minerals and/or vitamins will contribute to
energy as well.
or
• Indicate “no” with the explanation that other components of chocolate (without
mentioning them) will contribute as well.

Sample Question 9
Score 1: Response option E – 2 and 4, alternative E.
Score 0: Other.

SCIENCE UNIT 6 – CALF CLONING

Sample Question 10
Score 1: Answers which give an acceptable main idea, for example:
• “Testing whether cloning of calves is possible.”
• “Determining the number of calf clones that could be produced.”
Score 0: Answers which
• Do not mention calves or cloning.
or
• In effect repeat that “a large-scale application of this cloning technique could be financially
beneficial for cattle breeders”.

Sample Question 11
Score 1: Yes,Yes,Yes.
Score 0: Other.

96
……………………………………………………………………………………………………………

R
e
f
e
r
e
n
REFERENCES c
e
s
American Association for the Advancement of Science (1989), Science for all Americans, New
York: Oxford University Press.

American Association for the Advancement of Science (1993), Benchmarks for Science Literacy,
New York: Oxford University Press.

Applebee, A.N., Langer, J.A. & Mullis, I.V.S. (1987), Learning to be Literate in America: Reading,
Writing and Reasoning, Princeton NJ: Educational Testing Service.

Champagne, A.B. & Klopfer, L.E. (1982), “Actions in a time of crisis”, Science Education, 66(4),
pp. 503-514.

Cross, R. (1994), “Scientific literacy and democracy”, Search, 25(9), pp. 283-286.

Cumming, J. (1997), “Developments in numeracy: what is it and how should we teach it?”, Australian
Language Matters, 5(1), pp. 7-8.

Dangerfield, L. (1989), “Scientific literacy: myth or miracle”, The ACT Science Teacher, 14(1), pp. 26-31.

Fleming, R. (1989), “Literacy for a technological age”, Science Education, 73(4), pp. 391-404.

Galbraith, P.L., Carss, M.C., Grice, R.D., Endean, L. & Warry, M. (1997), “Towards scientific literacy
for the third millenium”, International Journal of Science Education, 19(4), pp. 447-467.

Kirsch, I.S. & Murray, T.S. (1998), Adult Literacy in OECD Countries: Technical Report on the First
International Adult Literacy Survey, Washington, DC: US Department of Education.

Krugly-Smolska, E.T. (1990), “Scientific literacy in developed and developing countries”, International
Journal of Science Education, 12, pp. 473-480.

OECD (1999), Measuring Student Knowledge and Skills: A New Framework for Assessment, Paris: OECD.

Paulos, J.A. (1988), Innumeracy: Mathematical Illiteracy and its Consequences, New York: Hill and Wang.

Venezky, R.L., Kaestle, C.F. & Sum, A.M. (1987), The Subtle Danger: Reflections on the Literacy Abilities
of America’sYoung Adults, Princeton, NJ: Educational Testing Service.

Zen, E. (1992), “Scientific literacy: what it is, why it is important, and what can scientists do to improve
the situation?”, The Australian Science Teachers Journal, 38(3), pp. 18-23.
97
 ……………………………………………………………………………………………………………

F
u
r
t
h
e
r
FURTHER READING
R
e
a
d Almond, R.G. & Mislevy, R.J. (1998), Graphical Models and Computerized Adaptive Testing, TOEFL
i Technical Report No.14, Educational Testing Service, Princeton, NJ, March.
n
g
Baker, L. (1991), “Metacognition, reading and science education”, in C. M. Santa and D. E. Alvermann
(eds.), Science Learning: Processes and Applications, International Reading Association, Newark, DE,
pp. 2-13.

Bennett, R.E. (1993), “On the meanings of constructed response”, in R.E. Bennett (ed.), Construction
vs. Choice in Cognitive Measurement: Issues in Constructed Response, Performance Testing, and Portfolio
Assessment, Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 1-27.

Binkley, M. & Linnakylä, P. (1997), “Teaching reading in the United States and Finland”, in M. Binkley,
K. Rust and T. Williams (eds.), Reading Literacy in an International Perspective, U.S. Department of
Education, Washington, DC.

Bruner, J. (1990), Acts of Meaning, Harvard University Press, Cambridge, MA.

Bybee, R.W. (1997), “Towards an understanding of scientific literacy”, in W. Grabe and C. Bolte (eds.),
Scientific Literacy – An International Symposium, IPN, Kiel.

Council of Europe (1996), Modern Languages: Learning, Teaching, Assessment. A Common European
Framework of Reference, CC LANG (95) 5 Rev. IV, Strasbourg.

Council of Ministers of Education, Canada (1994), Student Achievement Indicators Program: Reading
andWriting, Toronto.

de Lange, J. & Verhage, H. (1992), DataVisualization, Sunburst, Pleasantville, NY.

de Lange, J. (1987), Mathematics, Insight and Meaning, OW and OC, Utrecht.

Devlin, K. (1994, 1997), Mathematics,The Science of Patterns, Scientific American Library, New York.

Dole, J., Duffy, G., Roehler, L. & Pearson, P. (1991), “Moving from the old to the new: Research on
reading comprehension instruction”, Review of Educational Research, 16 (2), pp. 239-264.

Ehrlich, M.F. (1996), “Metacognitive monitoring in the processing of anaphoric devices in skilled and
less-skilled comprehenders”, in C. Cornoldi and J. Oakhill (eds.), Reading Comprehension Difficulties:
Processes and Interventions, Lawrence Erlbaum Associates, Mahwah, NJ, pp. 221-249.
98
……………………………………………………………………………………………………………

F
u
Ehrlich, M. F., Kurtz-Costes, B. & Loridant, C. (1993), “Cognitive and motivational determinants of r
reading comprehension in good and poor readers”, Journal of Reading Behavior, 25, pp. 365-381. t
h
Einstein, A. (1933), “Preface to M. Plank”, Where is Science Going?, Allen and Unwin, London. e
r
Elley, W.B. (1992), How in the World do Students Read?, International Association for the Evaluation of
R
Educational Achievement, The Hague. e
a
Frederickson, N. (1984), “The real test bias”, American Psychologist, 39, pp. 193-202. d
i
Freudenthal, H. (1973), Mathematics as an Educational Task, Reidel, Dordrecht. n
g
Freudenthal, H. (1983), Didactical Phenomenology of Mathematical Structures, Reidel, Dordrecht.

Graeber, W. & Bolte, C. (Eds) (1997), Scientific Literacy – An International Symposium, IPN, Kiel.

Gronlund, N.E. (1968), Constructing Achievement Tests, Prentice Hall, Englewood Cliffs.

Grünbaum, B. (1985), “Geometry strikes again”, Mathematics Magazine, 58 (1), pp. 12-18.

Hawking, S.W. (1988), A Brief History of Time, Bantam Press, London.

Hubbard, R. (1989), “Notes from the underground: Unofficial literacy in one sixth grade”, Anthropology
and Education Quarterly, 20, pp. 291-307.

Jones, S. (1995), “The practice(s) of literacy”, in Literacy, Economy and Society: Results of the First
International Adult Literacy Survey, OECD and Statistics Canada, Paris and Ottawa, pp. 87-113.

Kirsch, I. (1995), “Literacy performance on three scales: Definitions and results”, in Literacy, Economy
and Society: Results of the First International Adult Literacy Survey, OECD and Statistics Canada, Paris and
Ottawa, pp. 27-53.

Kirsch, I.S. & Mosenthal, P.B. (1989-1991), “Understanding documents. A monthly column”, Journal
of Reading, International Reading Association, Newark, DE.

Kirsch, I. S. & Mosenthal, P.B. (1994), “Interpreting the IEA reading literacy scales”, in M. Binkley, K. Rust and
M.Winglee (eds.), Methodological Issues in Comparative Educational Studies:The Case of the IEA Reading Literacy
Study, U.S. Department of Education, National Center for Education Statistics,Washington, DC, pp. 135-192.

Kirsch, I., Jungeblut, A. & Mosenthal, P.B. (1998), “The measurement of adult literacy”, inT. S. Murray,
I. S. Kirsch, and L. Jenkins (eds.), Adult Literacy in OECD Countries:Technical Report on the First International
Adult Literacy Survey, U.S. Department of Education, National Center for Education Statistics,Washington, DC.

Langer, J. (1995), Envisioning Literature, International Reading Association, Newark, DE.

Linnakylä, P. (1992), “Recent trends in reading literacy research in Finland”, in P. Belanger, C. Winter
and A. Sutton (eds.), Literacy and Basic Education in Europe on the Eve of the 21st Century, Council of
Europe, Strassbourg, pp. 129-135.

99
 ……………………………………………………………………………………………………………

F
u
r Lundberg, I. (1991), “Reading as an individual and social skill”, in I. Lundberg and T. Hoien (eds.),
t Literacy in aWorld of Change, Center for Reading Research/UNESCO, Stavanger.
h
e Maccarthey, S.J. & Raphael, T.E. (1989), Alternative Perspectives of Reading/Writing Connections,
r
College for Education, Institute for Research on Teaching,. Occasional paper #130, Michigan State
R University.
e
a Millar, R. & Osborne, J. (1998), Beyond 2000: Science Education for the Future, King’s College London
d School of Education, London.
i
n Myers, M. & Paris, S.G. (1978), “Children’s metacognitive knowledge about reading”, Journal of
g Educational Psychology, 70, pp. 680-690.
Paris, S., Wasik, B. &Turner, J. (1991), “The development of strategic readers”, in R. Barr, M. Kamil
and P. Mosenthal (eds.), Handbook of Reading Research, vol. II, Longman, New York.

Senechal, M. (1990), “Shape”, in L.A. Steen (ed.), On the Shoulders of the Giant – New Approaches to
Numeracy, National Academy Press, Washington D.C., pp. 139-182.

Shamos, M.H. (1995), The Myth of Scientific Literacy, Rutgers University Press, New Brunswick.

Smith, M.C. (1996), “Differences in adults’ reading practices and literacy proficiencies”, Reading Research
Quarterly, 31, pp. 196-219.

Sticht,T.G. (ed.) (1975), Reading forWorking: A Functional Literacy Anthology, Human Resources Research
Organization, Alexandria, VA.

Stiggins, R.J. (1982), “An analysis of the dimensions of job-related reading”, Reading World, 82, pp. 237-247.

Streefland, L. (1990), Fractions in Realistic Mathematics Education, A Paradigm of Developmental Research,

Reidel Dordrecht.

Stuart, I. (1990), “Change”, in L.A. Steen (ed.), On the Shoulders of the Giant – New Approaches to
Numeracy, National Academy Press, Washington D.C., pp. 183-218.

Taube, K. & Mejding, J. (1997), “A nine-country study: What were the differences between the low
and high performing students in the IEA Reading Literacy Study?”, in M. Binkley, K. Rust and T. Williams
(eds.), Reading Literacy in the International Perspectives, U.S. Department of Education, National Center
for Education Statistics, Washington, DC, pp. 63-100.

Traub, R.E. (1993), “On the equivalence of the traits assessed by multiple-choice and constructed-response
tests”, in R. E. Bennett (ed.), Construction vs. Choice in Cognitive Measurement: Issues in Constructed Response,
Performance Testing, and Portfolio Assessment, Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 29-44.

Travers, K.J. & Westbury, I. (1989), The IEA Study of Mathematics, vol. 1, Analysis of mathematics
curricula, Pergamon Press, Oxford.

Treffers, A. (1986), Three Dimensions, Reidel, Dordrecht.

100
……………………………………………………………………………………………………………

F
u
Treffers, A. & Goffree, F. (1985), “Rational analysis of realistic mathematics education”, in L. Streefland r
(ed.), Proceedings of the Ninth International Conference for the Psychology of Mathematics Education (PME), t
h
OW and OC, Utrecht, pp. 79-122.
e
r
UNESCO (1993), International Forum on Scientific and Technological Literacy for All, Final Report,
UNESCO, Paris. R
e
Ward,W.C., Dupree, D. & Carlson, S.B. (1987), A Comparison of Free-response and Multiple-choice Questions a
in the Assessment of Reading Comprehension (RR-87-20), Educational Testing Service, Princeton, NJ. d
i
Werlich, E. (1976), A Text Grammar of English, Quelle and Meyer, Heidelberg. n
g
Ziman, J.M. (1980), Teaching and Learning about Science and Society, Cambridge University Press.

101
……………………………………………………………………………………………………………

A
p
p
e
n
d
i
APPENDIX 1 x
FUNCTIONAL EXPERT GROUP MEMBERSHIP

Reading Mathematics Science

Irwin Kirsch, Chair Raimondo Bolletta Peter Fensham
Educational Testing Service Centro Europeo dell’Educazione, Monash University, Melbourne,
Princeton, New Jersey, United States Frascati, Italy Australia
Marilyn Binkley Sean Close Raul Gagliardi
National Center for St Patricks College, University of Geneva, Geneva,
Educational Statistics Dublin, Ireland Switzerland
Washington, DC, United States
Alan Davies Jan de Lange, Chair Wynne Harlen,
University of Edinburgh, Scotland, Utrecht University, Utrecht, Scottish Council for Research
United Kingdom The Netherlands in Education,
Edinburgh, United Kingdom
Stan Jones Maria Luisa Moreno Svein Lie
Statistics Canada Instituto Nacional de Calidad y University of Oslo,
Nova Scotia, Canada Evaluación (INCE) Oslo, Norway
Madrid, Spain
John de Jong Mogens Niss Manfred Prenzel
Swets Language Testing Unit Roskilde University, Roskilde, Institut für die Pädagogik der
Arnhem, The Netherlands Denmark Naturwissenschaften (IPN)
(Institute for Science Education)
Kiel, Germany
Dominique Lafontaine Kyung Mee Park Senta Raizen
Université de Liège, Liège, Chungbuk National University, National Center for Improving
Belgium Seoul, Korea Science Education
Washington, DC, United States
Pirjo Linnakylä Thomas Romberg Donghee Shin
University of Jyvaskyla University of Wisconsin-Madison Korea Institute of Curriculum and
Jyvaskyla, Finland Madison, Wisconsin, United States Evaluation
Seoul, Korea
Martine Rémond Peter Schüller Elizabeth Stage
Institut National de Recherche Federal Ministry of Education University of California
Pédagogique, Paris, France and Cultural Affairs Oakland, California, United States
Vienna, Austria
Wolfgang Schneider
University of Würzburg,
Würzburg, Germany
Ryo Watanabe
National Institute for Educational
Research, Tokyo, Japan …………
103
 ……………………………………………………………………………………………………………

A
p
p
e
n
d
i
x APPENDIX 2
TEST DEVELOPMENT TEAM

Australian Council for National Institute for

Educational Research (ACER) Educational Measurement (CITO)

Margaret Wu Steven Bakker

Co-ordinator, ACER Test Development Co-ordinator, CITO Test Development

Jan Lokan Bart Bossers

Joy McQueen Karin Bugel
Juliette Mendelovits Truus Dekkers
Gayl O’Connor Ico De Roo
Kees Lagerwaard
Erna Van Hest
Gerben van Lent

104