TIMSS

S.Kanageswari Suppiah Shanmugam,

PhD.
TIMSS International Results
TIMSS Released Items
TIMSS Technical Report
http://timssandpirls.bc.edu/timss2011/international-results-mathematics.html
Personal Communication with IEA

TIMSS
Trends in International Mathematics and

Science Study
International
assessment
on
Maths/Science for grade 4/Grade 8
Conducted by International Association
for
Evaluation
of
Educational
Achievement (IEA)
Every four years since 1995 (TIMSS 2011- 5th
cycle)

Development and Administration

Maths/Sc Test Booklets

Student/Teacher/School/Curriculum

Maths Questionnaires
Student Questionnaire(S)
Home Experience
School Experience

Teacher Questionnaire (T)

Education
Professional Development
Teaching Experience

School Questionnaire(P)
Availability of Resources
Types of Programmes
Learning Environment

Curriculum Questionnaire(C)
Organisation of Maths/Sc Curriculum
Content of Maths/Sc Curriculum

Student Testing Time

Student Achievement Booklet Part 1
minutes
Break
Student Achievement Booklet Part 2
minutes
Break
Student Questionnaire
minutes

45

45

30

Booklet
28 item blocks

14 Maths
14 Sc
Assigned to 14 Booklets

Each booklet has 4 blocks

2 maths
2 Sc

Each block has 12- 18 items and in two booklets

(Linking)

MCQ
Structured Response Questions
Provide explanation
Support an answer with reasons
Numerical evidence
Draw diagram
Display data

TIMSS Mathematics
Assessment Framework

TIMSS 2011 Mathematics

Framework
is organized around two dimensions
a content dimension
specifying the domains or subject matter to be
assessed within mathematics (number, algebra,
geometry, and data and chance)
Has several topic areas
Each topic area is presented as a list of
objectives
a cognitive dimension
specifying

the domains or thinking processes to

be assessed (knowing, applying, and reasoning)
The cognitive domains describe the sets of
behaviors expected of students as they engage
with the mathematics content.

Domain
Number
(30%)
Who
le
Nu
m
Frac
tion
s/De
cim
als
Inte
gers
Rati
o/Pr
opor
tion/

Algebra
(30%)
Patterns
Algebrai
c
Expressi
ons
Equation
/Formula
and
Function

Cognitive Domain
Knowing
Using mathematics
(35%)Depends on mathematical knowledge
Familiarity with mathematics concept
Facts - factual knowledge that provides the
basic language of mathematics, and the
essential mathematical facts and
properties that form the foundation for
mathematical thought.
Procedures- entails recall of sets of actions
and how to carry them out &
computational procedures and tools
Knowledge of concepts - make connections
between elements of knowledge, judge the
validity of mathematical

Cognitive Domain
Applying
(40%)
set in
real-life
situations

purely
mathematic
al questions
emphasis
more
familiar
and routine
tasks

application of mathematical tools in

a range of contexts.
The facts, concepts, and procedures
often are very familiar to the
student, with the problems being
routine ones.
apply mathematical knowledge of
facts, skills, and procedures or
understanding of mathematical
concepts to create representations
Problem solving is central but the
problem settings are more routine in
the implemented curriculum.
have been standard in classroom

Cognitive Domain
Reasoning
(25%)

novelty of the
context
the complexity of
the situation,
any solution to the
problem must
involve several
steps,
drawing on
knowledge and
understanding from
different areas of
mathematics
involve transfer of
knowledge and
skills to new
situations
Interactions

for logical, systematic thinking.

includes intuitive and inductive
reasoning
based on patterns and regularities
that can be used to arrive at solutions
to non-routine problems.
Non-routine problems - problems that
are very likely to be unfamiliar to
students.
They make cognitive demands over
and above those needed for solution
of routine problems, even when the
knowledge and skills required for their
solution have been learned.

Cognitive Domain
Knowing
(35%)

Applying
(40%)

Reasoning
(25%)

TOPIC AREA FRACTIONS AND DECIMALS

CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN KNOWING

TOPIC AREA FRACTIONS AND DECIMALS

CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN KNOWING

TOPIC AREA FRACTIONS AND DECIMALS

CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN APPLYING

TOPIC AREA FRACTIONS AND DECIMALS

CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN APPLYING

TOPIC AREA RATIO, PROPORTION AND PERCENT

CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN APPLYING

TOPIC AREA GEOMETRIC SHAPES

CONTENT DOMAIN GEOMETRY
COGNITIVE DOMAIN REASONING

TOPIC AREA INTEGER

CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN REASONING

TOPIC AREA GEOMETRIC SHAPES

CONTENT DOMAIN GEOMETRY
COGNITIVE DOMAIN REASONING

TOPIC AREA DATA

CONTENT DOMAIN DATA AND CHANCE
COGNITIVE DOMAIN REASONING

INTERNATIONAL BENCHMARK ADVANCED (625)

-reason with
information, draw
conclusions, make
generalizations, and
solve linear equations.
-solve a variety of
fraction, proportion, and
percent problems and
justify conclusions.
-express generalizations
algebraically and
model situations.
-solve a variety of
problems involving
equations, formulas,
and functions.
-reason with geometric
figures to solve
problems and with data
from several sources or
unfamiliar
representations to
solve multi-step

Content Domain: Geometry

Cognitive
Reasoning

Description: Solves a word

problem involving filling a
three-dimensional
shape
with rectangular solids

Domain:

INTERNATIONAL BENCHMARK HIGH (550)

-apply understanding
and knowledge in
complex situation
-use information from
several sources to
solve problems
involving different types
of numbers and
operations.
-relate fractions,
Peter,
James, and
decimals,
and Andrew
percentseach had 20 tries at
throwing
balls
into a basket. Complete the missing
to each
other.
boxes
below.
-show
basic procedural
knowledge related to
Name
Number of
Percentage of
algebraic expressions.
Successful Shots Successful
-use properties of
Shots
lines, angles, triangles,
rectangles,
andof 20
Peter
10 out
50 %
rectangular prisms to
solve problems.
James
15 out of 20
-analyse data in a
Andrew
out of 20
80%
variety of graphs.

Content Domain:
Number

Cognitive
Domain: Knowing

Description:
Given the part
and the whole,
can express the
part as a
percentage, and
given the whole
and the %, can
find the part

INTERNATIONAL BENCHMARK
INTERMEDIATE (475)
-apply basic

mathematical
knowledge in a
variety of situations.
-solve problems
involving decimals,
fractions, prop,%
-understand simple
What
does xy + 1 mean?
algebraic
orelationships.
a Add 1 to y, then multiply by
-relate a twox.
odimensional
b Multiply xdrawing
and y by 1.
threeotocaAdd
x to y, then add 1.
dimensional object.
o-read,
d Multiply
x by y, then add 1.
interpret,
and construct
graphs and tables.
-recognise basic
notions of

Content Domain:
Algebra
Cognitive Domain:
Knowing
Description: Knows the
meaning of a simple
algebraic expression
involving multiplication
and addition

INTERNATIONAL BENCHMARK LOW

(440)
Have some
knowledge of
whole numbers
and decimals,
operations,
and
42.65
+ 5.748
basic graphs

Answer: ________

Content Domain:

Number
Cognitive
Domain: Knowing
Description: Adds
a two-place and a
three-place
decimal

Administration in Malaysia
45 countries
Four countries in SEA (Indonesia, Malaysia,

Singapore, Thailand)
180 Schools
5733 Form Two students
Test Booklets
Bahasa Malaysia and English

Student Questionnaire
Bahasa Malaysia
Maths Teacher Questionnaire
English
School Questionnaire
English

Malaysia Performance Trend

Only in Grade 8
Participated since 1999
1999- 16th (39 countries)
2003- 10th (48 countries)
2007- 20th (46 countries)
2011- 26th (45 countries)

- BM
- BM
- BM/ENG
- BM/ENG
Recorded as one of the countries that declined
greatly (40 points or more)

Performance Trend- Content

Domain

Num Geometry Data & Chance / Algebra

Num Overall
Geometry / Data & Chance / Algebra Overall

Performance Trend - Cognitive

Domain

Low performance in Reasoning

Not much difference between

Performance Trend -Content

Domain
by gender

Girls Overall Boys

Performance Trend- Cognitive

Domain by gender

Girls Overall Boys

ACHIEVEMENT FOR MALAYSIA AT

INTERNATIONAL BENCHMARK

At every benchmark, there is a

TIMSS Mathematics
Items

Question Types and Scoring

Procedures
2 question formats- MCQ & constructed-

response
Each MCQ is worth 1 score point.
Constructed-response questions worth 1 or 2
score points, depending on the nature of the
task and the skills required to complete it.
The choice of item format depends on the
mathematics or being assessed, and the format
that best enables students to demonstrate
their proficiency.

Multiple-Choice Questions
(MCQ)

MCQ provide students with 4 response options.

Select one correct response
can be used to assess any of the behaviors in the

cognitive domains.
allow valid, reliable, and economical measurement of a
wide range of content in a relatively short testing time.
do not allow for students explanations or supporting
statements
less suitable for assessing students ability to make
more complex interpretations or evaluations.
linguistic features need to be appropriate.
questions are written clearly and concisely.
response options also are written to minimise the
reading load of the question.
The options that are incorrect are written to be

Constructed-Response Questions
(CRQ)
students are required to construct/give a written

response, rather than select a response from a set

of options
allow students to provide explanations, support an
answer with reasons or numerical evidence, draw
diagrams, or display data,
well-suited for assessing aspects of knowledge
and skills that require students to explain
phenomena or interpret data based on their
background knowledge and experience
real-world setting - the setting is familiar to
students.
need to prepare a Scoring Rubric

Scoring Rubric
describes the essential features of appropriate and

complete responses.
focus on evidence of the type of behavior the
question assesses.
describe evidence of completely correct, partially
correct and completely incorrect responses.
student responses at each level of understanding
provide important guidance to those who will be
rating the students responses.
In scoring, the focus is solely on students
achievement with respect to the topic being
assessed, not on their ability to write well.
students need to communicate clearly

Types of Items- Grade 8

Types of Items- Grade 8

Scoring Rubric

Types of Items- Grade 8

Types of Items- Grade 8

Types of Items- Grade 8

TIMSS Item Writing

Process

Item
Writing
Guidelines

Process

and

What should the student know?

What should the student be able to do?

consider

the timing, grade appropriateness,

difficulty level, potential sources of bias (cultural,
gender, or geographical, ).
Make sure that item validity is not affected by
factors that unnecessarily increase the difficulty of
the item, such as unfamiliar or difficult vocabulary,
grammar, directions, contexts, or stimulus
materials
be sensitive to the possibility of unintentionally
placing particular groups of students at an unfair
disadvantage
diagrams and graphs are drawn accurately (to

MCQ
Stem is the initial part of the item in which the task is

defined.
Options refer to the entire set of labeled response choices
presented under the stem.
Key is the correct response option.
Distracters are the incorrect response options.
ask a direct question with only one correct answer, and
provide plausible distracters
the question must be able to stand alone, and be
answerable without the response options
do not include extraneous information that can confuse
students
avoid questions for which a wrong method yields the
correct answer (e.g., a question about a circle with a radius
of 2, since computing either area or circumference get 4)

CRQ

Write a full credit

answer in terms of
the
language,
knowledge,
and
skills that student
could be expected
to
possess
&
determine whether
to allocate 1 or 2
score points.
Develop a specific
scoring guide

Take Home information

Each of the items needs to contribute to the

overall mathematics test

Some relatively easy items and some
challenging items
Avoid items that almost all students or almost
no students are able to answer correctly

Checklist- MCQ

Checklist-CRQ

Each item
1The Content Domain, topic area, and

objective the item measures

2. The Cognitive Domain
3. The item number (1, 2, 3, etc.)
4. The key (multiple-choice items only) or
5. The scoring guide