0% found this document useful (0 votes)

96 views278 pages

Talis 2008

Uploaded by

Nguyễn Hoàng Diệp

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

96 views278 pages

Talis 2008

Uploaded by

Nguyễn Hoàng Diệp

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 278

TALIS 2008 Technical Report

The OECD’s new Teaching and Learning International Survey (TALIS) has been designed to provide
data and analyses on the conditions needed for effective teaching and learning in schools. As the TALIS 2008
Technical Report
first international survey with this focus, it seeks to fill important information gaps that have been
identified at the national and international levels of education systems.

This TALIS Technical Report describes the development of the TALIS instruments and methods
used in sampling, data collection, scaling and data analysis phases of the first round of the survey.
It also explains the rigorous quality control programme that operated during the survey process,
which included numerous partners and external experts from around the world.

The information in this report complements the first international report from TALIS, Creating
Effective Teaching and Learning Environments: First Results from TALIS (OECD, 2009) and the User
Guide for the TALIS International Database (available for download from www.oecd.org/edu/talis/).

Further reading:
Education at a Glance 2009: OECD Indicators (OECD, 2009)
Teachers Matter: Attracting, Developing and Retaining Effective Teachers (OECD, 2008)
Improving School Leadership (OECD, 2008)
PISA 2006: Science Competencies for Tomorrow’s World (OECD, 2007)

TALIS 2008 Technical Report

The full text of this book is available on line via this link:
http://www.sourceoecd.org/education/9789264079854
Those with access to all OECD books on line should use this link:
http://www.sourceoecd.org/9789264079854
SourceOECD is the OECD’s online library of books, periodicals and statistical databases.
For more information about this award-winning service and free trials ask your librarian, or write to us T e a c h i n g A n d L e a r n i n g I n t e r n a t i o n a l S u r v e y
at [email protected].

ISBN 978-92-64-07985-4
87 2010 01 1P
www.oecd.org/publishing
TALIS 2008
Technical Report

Teaching And Learning International Survey

ORGANISATION FOR ECONOMIC CO-OPERATION
AND DEVELOPMENT

The OECD is a unique forum where the governments of 30 democracies work together to
address the economic, social and environmental challenges of globalisation. The OECD is also at
the forefront of efforts to understand and to help governments respond to new developments
and concerns, such as corporate governance, the information economy and the challenges of an
ageing population. The Organisation provides a setting where governments can compare policy
experiences, seek answers to common problems, identify good practice and work to co-ordinate
domestic and international policies.

The OECD member countries are: Australia, Austria, Belgium, Canada, the Czech Republic,
Denmark, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea,
Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, the Slovak Republic,
Spain, Sweden, Switzerland, Turkey, the United Kingdom and the United States. The Commission of
the European Communities takes part in the work of the OECD.

OECD Publishing disseminates widely the results of the Organisation’s statistics gathering and
research on economic, social and environmental issues, as well as the conventions, guidelines and
standards agreed by its members.

This work is published on the responsibility of the Secretary-General of the OECD. The
opinions expressed and arguments employed herein do not necessarily reflect the official
views of the Organisation or of the governments of its member countries.

Cover illustration:
© Mike Kemp/Rubberball Productions/Getty Images
© Laurence Mouton/PhotoAlto Agency RF Collections/Inmagine ltb.

Corrigenda to OECD publications may be found on line at: www.oecd.org/publishing/corrigenda.

© OECD 2010
You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and multimedia
products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgment of OECD as source
and copyright owner is given. All requests for public or commercial use and translation rights should be submitted to [email protected]. Requests for
permission to photocopy portions of this material for public or commercial use shall be addressed directly to the Copyright Clearance Center (CCC)
at [email protected] or the Centre français d’exploitation du droit de copie (CFC) at [email protected].
3

Foreword
The OECD’s new Teaching and Learning International Survey (TALIS) has been designed to provide data and
analyses on the conditions needed for effective teaching and learning in schools. As the first international
survey with this focus, it seeks to fill important information gaps that have been identified at the national and
international levels.
TALIS is conceptualised as a programme of surveys, with successive rounds designed to address policy-relevant
issues chosen by countries. This Technical Report relates to the first round of TALIS conducted in 2007-08,
which studied lower secondary education in both the public and private sectors. It examined important aspects
of teachers’ professional development; teacher beliefs, attitudes and practices; teacher appraisal and feedback;
and school leadership.
This report describes the development of the TALIS instruments and the methods used in sampling, data
collection, scaling and data analysis, for those wishing to review and replicate the procedures used. Furthermore,
it provides insight into the rigorous quality control programme that operated during all phases of the survey,
involving numerous partners and external experts.
The information in this report complements the User Guide for the TALIS International Database (available for
download from www.oecd.org/edu/talis/). The Guide provides a basic yet thorough introduction to the TALIS
international database and to the results included in the first international report from TALIS, Creating Effective
Teaching and Learning Environments: First Results from TALIS (OECD, 2009).
The database contains the survey responses from more than 70 000 teachers and more than 4 000 school
principals in the 24 countries who participated in the first round of TALIS. The database was used to generate
information and to act as a base for the production of the initial international report from the first round of TALIS
published in June 2009 (OECD, 2009).
TALIS is a collaborative effort by the participating countries, guided by their governments on the basis of shared
policy-driven interests. Representatives of each country form the TALIS Board of Participating Countries, which
determines the policy orientations of TALIS as well as the analysis and results produced from it.
The OECD recognises the significant contributions of Dirk Hastedt, Steffen Knoll, Ralph Carstens and
Friederike Westphal of the IEA Data Processing and Research Center in Hamburg, Germany, who co-
edited this report. The principal authors of specific chapters were for Chapter 1: Dirk Hastedt, Steffen Knoll
and Friederike Westphal; Chapter 4: Barbara Malak-Minklewiez and Suzanne Morony; Chapters 5 and 10:
Jean Dumais and Sylvie LaRoche; Chapter 6: Friederike Westphal and Steffen Knoll; Chapter 7: Ralph Carstens;
Chapter 8: Barbara Malak-Minklewiez, Suzanne Morony and Friederike Westphal; Chapter 9: Alena Becker
and Ralph Carstens; Chapter 11: Svenja Vieluf, Juan Leon and Ralph Carstens. Chapters 2 and 3 and parts of
Chapter 11 were authored by the OECD Secretariat.
The editorial work at the OECD Secretariat was carried out by Eric Charbonnier, Michael Davidson, Ben Jensen,
Niccolina Clements, Soojin Park and Elisabeth Villoutreix. A full list of contributors to the TALIS project is
included in Annex A of this report. This report is published under the responsibility of the Secretary-General of
the OECD.

Barbara Ischinger
Director for Education, OECD

TALIS 2008 Technical Report © OECD 2010

Table of Contents
Foreword...................................................................................................................................................................................................3

List of abbreviations and acronyms.........................................................................................................................15

Chapter 1 Introduction and Summary..................................................................................................................17

Chapter 2 Overview of TALIS 2008 and Framework Development.................................................23

Chapter 3 Development of Teacher and Principal Questionnaires..........................................29

Chapter 4 Translation and Cultural Adaptation......................................................................................45

TALIS 2008 Technical Report © OECD 2010

6
Table of Contents

International translation verification. ........................................................................................................................................51

Processes of translation verification. .................................................................................................................................51
Translation verification report................................................................................................................................................51
Translation verification summary........................................................................................................................................51
Feedback from National Project Managers on translation and translation verification..........................52
International Quality Control Monitor review of translation....................................................................................52
Layout verification: paper and on line data collection.................................................................................................52
References...................................................................................................................................................................................................54

Chapter 5 Sample Design.........................................................................................................................................................55

Chapter 6 Survey Operations Procedures...........................................................................................................65

Chapter 7 On line Data Collection.............................................................................................................................73

Chapter 8 Quality Assurance.............................................................................................................................................89

© OECD 2010 TALIS 2008 Technical Report

Table of Contents
7

Survey Activities Questionnaire....................................................................................................................................................94

Contacting schools........................................................................................................................................................................94
Preparing survey materials.......................................................................................................................................................95
Conversion to on line data collection..............................................................................................................................96
Survey administration..................................................................................................................................................................96
Data entry and verification......................................................................................................................................................97
National quality control monitoring programme.....................................................................................................97
References................................................................................................................................................................................................101

Chapter 9 Creating and Checking the International Database...............................................103

Chapter 10 Estimation Weights, Participation Rates and Sampling Error. ...................115

Abstract. .....................................................................................................................................................................................................116
Overview. .................................................................................................................................................................................................116
Elements of the estimation weight (final weight). ..........................................................................................................116
School base weight (school design weight)...............................................................................................................117
School non-response adjustment factor.......................................................................................................................117
Final school weight....................................................................................................................................................................118
Teacher base weight (teacher design weight)...........................................................................................................118
Teacher non-response adjustment factor. ...................................................................................................................119
Teacher adjustment factor for incidental exclusions...........................................................................................119
Teacher multiplicity adjustment factor.........................................................................................................................119
Final teacher weight..................................................................................................................................................................120
Participation rates................................................................................................................................................................................120
Participation rates for schools.............................................................................................................................................121
Participation rate for teachers. ............................................................................................................................................121
Overall participation rates. ...................................................................................................................................................122
Reporting participation rates...............................................................................................................................................122
Meeting participation rates standard for TALIS. ......................................................................................................122
Sampling error with Balanced Repeated Replication.................................................................................................123
Creating replicates for Balanced Repeated Replication....................................................................................123
Estimating the sampling error.............................................................................................................................................125
Design effect and effective sample size.......................................................................................................................125
References................................................................................................................................................................................................129

Chapter 11 Construction and validation of scales and indices............................................131

TALIS 2008 Technical Report © OECD 2010

8
Table of Contents

Annex A TALIS consortium, experts and consultants..........................................................................207

Annex B Characteristics of National Samples. ...........................................................................................211

Australia.....................................................................................................................................................................................................212
Austria.........................................................................................................................................................................................................213
Belgium (Flanders)..............................................................................................................................................................................214
Brazil............................................................................................................................................................................................................215
Bulgaria......................................................................................................................................................................................................216
Denmark....................................................................................................................................................................................................217
Estonia.........................................................................................................................................................................................................218
Hungary.....................................................................................................................................................................................................219
Iceland........................................................................................................................................................................................................220
Ireland.........................................................................................................................................................................................................221
Italy...............................................................................................................................................................................................................222
Korea............................................................................................................................................................................................................223
Lithuania....................................................................................................................................................................................................224
Malaysia.....................................................................................................................................................................................................225
Malta............................................................................................................................................................................................................226
Mexico........................................................................................................................................................................................................227
Norway.......................................................................................................................................................................................................228
Poland.........................................................................................................................................................................................................229
Portugal......................................................................................................................................................................................................230
Slovak Republic....................................................................................................................................................................................231
Slovenia. ....................................................................................................................................................................................................232
Spain (Excluding La Rioja, Canarias). ....................................................................................................................................233
The Netherlands...................................................................................................................................................................................234
Turkey..........................................................................................................................................................................................................235

Annex C Sampling Forms......................................................................................................................................................237

Sampling Form 1 – Participation...............................................................................................................................................238
Sampling Form 2 – National Target Population...............................................................................................................239
Sampling Form 3 – Stratification...............................................................................................................................................240
Sampling Form 4 – Sampling Frame Description. .........................................................................................................241
Sampling Form 5 – Excluded schools....................................................................................................................................242
Sampling Form 6 – Population counts by strata..............................................................................................................243
Sampling Form 7 – Sample allocation. .................................................................................................................................244

Annex D TALIS Principal and Teacher Questionnaire.............................................................................245

Principal Questionnaire..................................................................................................................................................................246
Background information. .......................................................................................................................................................247
School background information. ......................................................................................................................................248
School management. ................................................................................................................................................................250
Teacher appraisal. .......................................................................................................................................................................253
School resources..........................................................................................................................................................................255
Teacher Questionnaire.....................................................................................................................................................................259
Background information. .......................................................................................................................................................260
Professional development.....................................................................................................................................................262
Teacher appraisal and feedback........................................................................................................................................264
Teaching practices, beliefs and attitudes.....................................................................................................................267
Your teaching in a particular <class> at this school............................................................................................271

© OECD 2010 TALIS 2008 Technical Report

Table of Contents
9

Figures
Figure 1.1 Milestones of the survey. ............................................................................................................................................ 21

Figure 2.1 Policy issues and related indicator domains examined in TALIS............................................................... 26
Figure 2.2 Results of priority-rating exercise............................................................................................................................ 27

Figure 3.1 Survey themes for TALIS.............................................................................................................................................. 32

Figure 3.2 Structure for evaluation of education in schools: data collected in TALIS............................................ 36
Figure 3.3 Principal Questionnaire item on student characteristics, altered for the main survey..................... 41

Figure 4.1 Countries and languages participating in TALIS. .............................................................................................. 48

Figure 5.1 TALIS international target and survey populations.......................................................................................... 57

Figure 6.1 Responsibilities of NPMs and the SC during survey administration........................................................ 68

Figure 7.1 Architectural overview of the SurveySystem...................................................................................................... 76

Figure 9.1 Overview of iterative data processing at the IEA DPC.................................................................................107

Figure 11.1 Illustration of CFA parameters. ...............................................................................................................................137

Figure 11.2 Latent y notation for a one-factor CFA model. ................................................................................................138
Figure 11.3 Illustration of configural invariance of the construct Self-efficacy across two countries. .............140
Figure 11.4 Illustration of metric non-invariance for two countries...............................................................................141
Figure 11.5 Illustration of scalar non-invariance across two countries.........................................................................142
Figure 11.6 Illustration of non-invariance of residual variance across two countries.............................................142

Tables
Table 5.1 Reasons for and magnitude of school exclusion, by country. .................................................................... 61
Table 5.2 Derivation of the required sample size................................................................................................................. 62
Table 5.3 Illustration of systematic random sampling with PPS..................................................................................... 62
Table 5.4 Overview of the national sampling plans............................................................................................................ 63

Table 7.1 Extent to which paper and on line administration modes were used for the Principal
Questionnaire during the main study (sorted in descending order of on line usage
and country name). ....................................................................................................................................................... 84
Table 7.2 Extent to which paper and on line administration modes were used for the Teacher Questionnaire
during the main study (sorted in descending order of on line usage and country name). ............. 85
Table 7.3 Percentage of questionnaires administered on line, by respondent type and gender...................... 85
Table 7.4 Percentage of questionnaires administered on line, by teachers’ age group. ...................................... 86
Table 7.5 Percentage of schools that administered questionnaires on line, by country...................................... 86
Table 7.6 Error typology: type, initial percentage of all analysed login attempts, percentage
of successful corrections, and percentage of residual failed login attempts
(sorted in descending order of magnitude of initial error rate)................................................................... 86

TALIS 2008 Technical Report © OECD 2010

10
Table of Contents

Table 7.7 Maximum, mean, and median duration for completing an on line questionnaire,
as well as percentage of completed questionnaires within one login, one day,
and one week, by country. ........................................................................................................................................ 87

Table 8.1 Preliminary activities of the School Co-ordinator............................................................................................ 99

Table 8.2 Questionnaire distribution and returns................................................................................................................. 99
Table 8.3 Use of forms in survey administration. ................................................................................................................. 99
Table 8.4 Use of specialist translators and reviewers.......................................................................................................100

Table 10.1 Quality ratings and unweighted participation rates......................................................................................126

Table 10.2 Unweighted participation rates, by country.....................................................................................................126
Table 10.3 Weighted participation rates, by country. .........................................................................................................126
Table 10.4 Example of BRR-ready sample design and random assignation of pseudo PSUs. ...........................127
Table 10.5 Estimated design effects and effective sample sizes for selected key variables,
their average and the original and effective sample sizes, by country
(Teacher Questionnaire variables). .......................................................................................................................127
Table 10.6 Estimated design effects and effective sample size for selected key variables,
their average and the original and effective sample sizes, by country
(Principal Questionnaire variables)......................................................................................................................128

Table 11.1 Item wording for school autonomy indices......................................................................................................152

Table 11.2 Selection of indices – rotated component matrix. .........................................................................................152
Table 11.3 Factor loadings used for computation of factor scores for the indices of school autonomy............152
Table 11.4 Reliabilities for indices of school autonomy....................................................................................................153
Table 11.5 Item wording for school resource indices.........................................................................................................153
Table 11.6 Selection of indices-rotated component matrix..............................................................................................154
Table 11.7 Factor loadings used for computation of factor scores for the indices of school resources.............154
Table 11.8 Reliabilities for indices of school resources. ....................................................................................................154
Table 11.9 International means and standard deviations of school- and teacher-level factor score
estimates using equally weighted pooled data. ..............................................................................................155
Table 11.10 Item wording of school leadership indices and dimensions.....................................................................155
Table 11.11 Single items measuring school leadership and management behaviours. ..........................................156
Table 11.12 Reliabilities for indices of school leadership....................................................................................................156
Table 11.13 Model fit for the scale Framing and communicating the school’s goals and curricular
development. .................................................................................................................................................................157
Table 11.14 Model fit for the scale Promoting instructional improvements and professional development.......157
Table 11.15 Model fit for the scale Supervision of instruction in the school. .............................................................158
Table 11.16 Model fit for the scale Accountability role of the principal.......................................................................158
Table 11.17 Model fit for the scale Bureaucratic rule-following. .....................................................................................159
Table 11.18 Multiple group CFA examining different levels of cross-cultural invariance of
the scale Framing and communicating the school’s goals and curricular development. ............159
Table 11.19 Multiple group CFA examining different levels of cross-cultural invariance of
the scale Promoting instructional improvements and professional development............................159
Table 11.20 Multiple group CFA examining different levels of cross-cultural invariance
of the scale Supervision of instruction in the school....................................................................................160

© OECD 2010 TALIS 2008 Technical Report

Table of Contents
11

Table 11.21 Multiple group CFA examining different levels of cross-cultural invariance
of the scale Accountability role of the principal. ...........................................................................................160
Table 11.22 Multiple group CFA examining different levels of cross-cultural invariance of principals’
Bureaucratic rule-following.....................................................................................................................................160
Table 11.23 Factor loadings used for the computation of factor scores for the indicator
Framing and communicating the school’s goals and curricular development..................................160
Table 11.24 Item intercepts used for the computation of factor scores for the indicator
Framing and communicating the school’s goals and curricular development, by country. ........161
Table 11.25 Unique variances used for the computation of factor scores for the indicator
Framing and communicating the school’s goals and curricular development, by country. ........161
Table 11.26 Factor loadings used for the computation of factor scores for the indicator
Promoting instructional improvements and professional development...............................................162
Table 11.27 Item intercepts used for the computation of factor scores for the indicator
Promoting instructional improvements and professional development, by country. .....................162
Table 11.28 Unique variances used for the computation of factor scores for the indicator
Promoting instructional improvements and professional development, by country. .....................163
Table 11.29 Factor loadings used for the computation of factor scores for the indicator
Supervision of instruction in the school.............................................................................................................163
Table 11.30 Item intercepts used for the computation of factor scores for the indicator
Supervision of instruction in the school, by country....................................................................................164
Table 11.31 Unique variances used for the computation of factor scores for the indicator
Supervision of instruction in the school, by country....................................................................................164
Table 11.32 Factor loadings used for the computation of factor scores for the indicator
Accountability role of the principal.....................................................................................................................165
Table 11.33 Item intercepts used for the computation of factor scores for the indicator
Accountability role of the principal, by country. ...........................................................................................165
Table 11.34 Unique variances used for the computation of factor scores for the indicator
Accountability role of the principal, by country. ...........................................................................................166
Table 11.35 Factor loadings used for the computation of factor scores for the indicator
Bureaucratic rule-following.....................................................................................................................................166
Table 11.36 Item intercepts used for the computation of factor scores for the indicator
Bureaucratic rule-following, by country............................................................................................................167
Table 11.37 Unique variances used for the computation of factor scores for the indicator
Bureaucratic rule-following, by country............................................................................................................167
Table 11.38 Factor score determinacy for the scales related to school principal’s leadership, by country. .......168
Table 11.39 Item wording of school climate items and dimensions – Principal Questionnaire.........................168
Table 11.40 Single items measuring aspects of school climate.........................................................................................169
Table 11.41 Item wording of school climate items and dimensions – Teacher Questionnaire............................169
Table 11.42 Reliabilities for school climate indices, by country......................................................................................169
Table 11.43 Model fit and latent correlations for factors influencing school climate, by country.....................170
Table 11.44 Multiple group CFA examining different levels of cross-cultural invariance
of factors influencing school climate. .................................................................................................................170
Table 11.45 Factor loadings used for the computation of factor scores for the indicator
School climate: student delinquency..................................................................................................................170
Table 11.46 Item intercepts used for the computation of factor scores for the indicator
School climate: student delinquency, by country..........................................................................................171
Table 11.47 Unique variances used for the computation of factor scores for the indicator School climate:
student delinquency, by country...........................................................................................................................171

TALIS 2008 Technical Report © OECD 2010

12
Table of Contents

Table 11.48 Factor loadings used for the computation of factor scores for
the indicator School climate: teachers’ working morale.............................................................................172
Table 11.49 Item intercepts used for the computation of factor scores for the indicator
School climate: teachers’ working morale, by country...............................................................................172
Table 11.50 Unique variances used for the computation of factor scores for
the indicator School climate: teachers’ working morale, by country....................................................173
Table 11.51 Factor score determinacy for the indicators School climate: student delinquency
and School climate: teachers’ working morale, by country.....................................................................173
Table 11.52 Model fit of Teacher-student relations (TQ), by country..............................................................................174
Table 11.53 Multiple group CFA examining different levels of cross-cultural invariance
of Teacher-student relations.....................................................................................................................................174
Table 11.54 Factor loadings used for the computation of factor scores for the indicator
Teacher-student relations..........................................................................................................................................174
Table 11.55 Item intercepts used for the computation of factor scores for the indicator
Teacher-student relations, by country. ................................................................................................................175
Table 11.56 Unique variances used for the computation of factor scores for the indicator
Teacher-student relations, by country. ................................................................................................................175
Table 11.57 Factor score determinacy for the indicators Teacher-student relations, by country.........................176
Table 11.58 Item wording of Classroom disciplinary climate items and dimensions..............................................176
Table 11.59 Reliabilities for Classroom disciplinary climate, by country.....................................................................177
Table 11.60 Model fit Classroom disciplinary climate, by country.................................................................................177
Table 11.61 Multiple group CFA examining different levels of cross-cultural invariance
of Classroom disciplinary climate. .......................................................................................................................178
Table 11.62 Factor loadings used for the computation of factor scores for the indicator
Classroom disciplinary climate..............................................................................................................................178
Table 11.63 Item intercepts used for the computation of factor scores for the indicator
Classroom disciplinary climate, by country.....................................................................................................178
Table 11.64 Unique variances used for the computation of factor scores for the indicator
Classroom disciplinary climate, by country.....................................................................................................179
Table 11.65 Factor score determinacy for the indicators Classroom disciplinary climate, by country............179
Table 11.66 Item wording of the scale Self-efficacy...............................................................................................................180
Table 11.67 Reliabilities for Self-efficacy, by country............................................................................................................180
Table 11.68 Model fit for Self-efficacy, by country.................................................................................................................181
Table 11.69 Multiple group CFA examining different levels of cross-cultural invariance of Self-efficacy. .............181
Table 11.70 Factor loadings used for the computation of factor scores for
the indicator Self-efficacy.........................................................................................................................................181
Table 11.71 Item intercepts used for the computation of factor scores for
the indicator Self-efficacy, by country.................................................................................................................182
Table 11.72 Unique variances used for the computation of factor scores for
the indicator Self-efficacy, by country.................................................................................................................182
Table 11.73 Factor score determinacy for the indicators Self-efficacy, by country...................................................183
Table 11.74 Item wording of teachers’ and principals’ beliefs items and dimensions............................................183
Table 11.75 Single items measuring teachers’ and principals’ Beliefs about instruction.......................................184
Table 11.76 Reliabilities for indices of teachers’ Beliefs about instruction, by country. ........................................184
Table 11.77 Model fit and latent correlations for teachers’ Direct transmission beliefs about instruction
and Constructivist beliefs about instruction, by country.............................................................................185

© OECD 2010 TALIS 2008 Technical Report

Table of Contents
13

Table 11.78 Multiple group CFA examining different levels of cross-cultural invariance of teachers’
Direct transmission beliefs about instruction...................................................................................................185
Table 11.79 Multiple group CFA examining different levels of cross-cultural invariance of teachers’
Constructivist beliefs about instruction. .............................................................................................................186
Table 11.80 Factor loadings used for the computation of factor scores for the indicators Direct transmission
beliefs about instruction and Constructivist beliefs about instruction. .................................................186
Table 11.81 Item intercepts used for the computation of factor scores for the indicators Direct transmission
beliefs about instruction and Constructivist beliefs about instruction, by country..........................186
Table 11.82 Unique variances used for the computation of factor scores for
the indicators Direct transmission beliefs about instruction and
Constructivist beliefs about instruction, by country......................................................................................187
Table 11.83 Factor score determinacy for the indicators Direct transmission beliefs about instruction
and Constructivist beliefs about instruction, by country.............................................................................188
Table 11.84 Reliabilities for indices of principals’ Constructivist beliefs about instruction, by country.........188
Table 11.85 Model fit for principals’ Constructivist beliefs about instruction, by country....................................189
Table 11.86 Multiple group CFA examining different levels of cross-cultural invariance of principals’
Constructivist beliefs about instruction. .............................................................................................................189
Table 11.87 Factor loadings used for the computation of factor scores for the indicator principals’
Constructivist beliefs about instruction. .............................................................................................................189
Table 11.88 Item intercepts used for the computation of factor scores for the indicator principals’
Constructivist beliefs about instruction, by country......................................................................................190
Table 11.89 Unique variances used for the computation of factor scores for the indicator principals’
Constructivist beliefs about instruction, by country......................................................................................190
Table 11.90 Factor score determinacy for the indicators principals’ Constructivist beliefs about instruction,
by country.......................................................................................................................................................................191
Table 11.91 Item wording of classroom teaching practices items and dimensions..................................................191
Table 11.92 Single items measuring classroom teaching practices items and dimensions...................................191
Table 11.93 Reliabilities for indices of classroom teaching practices, by country....................................................192
Table 11.94 Reliabilities for indices of classroom teaching practices, by subject.....................................................192
Table 11.95 Model fit and latent correlations for classroom teaching practices, by country...............................193
Table 11.96 Model fit and latent correlations for classroom teaching practices, by subject................................193
Table 11.97 Multiple group CFA examining different levels of cross-cultural invariance of
Classroom teaching practice: structuring. .........................................................................................................194
Table 11.98 Multiple group CFA examining different levels of cross-cultural invariance of
Classroom teaching practice: student-oriented...............................................................................................194
Table 11.99 Multiple group CFA examining different levels of cross-cultural invariance of
Classroom teaching practice: enhanced activities..............................................................................................194
Table 11.100 Factor loadings used for the computation of factor scores for the indicator
Classroom teaching practice: structuring. .........................................................................................................194
Table 11.101 Item intercepts used for the computation of factor scores for the indicator
Classroom teaching practice: structuring, by country..................................................................................195
Table 11.102 Unique variances used for the computation of factor scores for the indicator
Classroom teaching practice: structuring, by country..................................................................................195
Table 11.103 Factor loadings used for the computation of factor scores for the indicator
Classroom teaching practice: student-oriented.............................................................................................196
Table 11.104 Item intercepts used for the computation of factor scores for the indicator
Classroom teaching practice: student-oriented, by country. ...................................................................196

TALIS 2008 Technical Report © OECD 2010

14
Table of Contents

Table 11.105 Unique variances used for the computation of factor scores for the indicator
Classroom teaching practice: student-oriented, by country. ...................................................................197
Table 11.106 Factor loadings used for the computation of factor scores for the indicator
Classroom teaching practice: enhanced activities.......................................................................................197
Table 11.107 Item intercepts used for the computation of factor scores for the indicator
Classroom teaching practice: enhanced activities, by country. .............................................................198
Table 11.108 Unique variances used for the computation of factor scores for the indicator
Classroom teaching practice: enhanced activities, by country. .............................................................198
Table 11.109 Factor score determinacy for the indicators of Classroom teaching practice, by country..........199
Table 11.110 Item wording of Co-operation among staff. ....................................................................................................199
Table 11.111 Single items measuring Co-operation among staff.......................................................................................199
Table 11.112 Reliabilities for indices of Co-operation among staff, by country.........................................................200
Table 11.113 Model fit and latent correlations for indices of Co-operation among staff, by country...............200
Table 11.114 Multiple group CFA examining different levels of cross-cultural invariance
of Exchange and co-ordination for teaching. .................................................................................................201
Table 11.115 Multiple group CFA examining different levels of cross-cultural invariance
of Professional collaboration.................................................................................................................................201
Table 11.116 Factor loadings used for the computation of factor scores for the indicator
Exchange and co-ordination for teaching........................................................................................................201
Table 11.117 Item intercepts used for the computation of factor scores for the indicator
Exchange and co-ordination for teaching, by country...............................................................................202
Table 11.118 Unique variances used for the computation of factor scores for the indicator
Exchange and co-ordination for teaching, by country...............................................................................202
Table 11.119 Factor loadings used for the computation of factor scores for the indicator
Professional collaboration......................................................................................................................................203
Table 11.120 Item intercepts used for the computation of factor scores for the indicator
Professional collaboration, by country. ............................................................................................................203
Table 11.121 Unique variances used for the computation of factor scores for the indicator
Professional collaboration, by country. ............................................................................................................204
Table 11.122 Factor score determinacy for the indicators of co-operation among staff, by country.................204

© OECD 2010 TALIS 2008 Technical Report

List of abbreviations
and acronyms
Abbreviation /Acronym Meaning Comments
CFA Confirmatory Factor Analysis
DPC IEA Data Processing and Research Center Part of IEA
FT Field Trial
IQCM International Quality Control Monitor Contracted by IEA
IDEG OECD TALIS Instrument Development
Expert Group
IEA International Association for the Evaluation
of Educational Achievement
INES OECD Indicators of Education Systems
Programme
ISC International Study Centre In TALIS, the IEA
Data Processing and
Research Center
ISCED International Standard Classification
of Education
MOS Measure of Size
MS Main Study
NAF National Adaptation Form
NDM National Data Manager
NPM National Project Manager
NQM National Quality Monitor
ODC Online Data Collection
P&P Paper and Pencil
PQ Principal/School Questionnaire
SC School Co-ordinator
STF School Tracking Form
TALIS Teaching and Learning International Survey
TLF Teacher Listing Form Form produced
by WinW3S
TQ Teacher Questionnaire
TTF Teacher Tracking Form Form produced
by WinW3S
WinDEM Windows Data Entry Manager Software
WinW3S Windows Within School Sampling Software Software

TALIS 2008 Technical Report © OECD 2010

Chapter 1

Introduction
and Summary
18 Abstract
18 Overview of TALIS
18 Management of the study
19 Standardised procedures to ensure high-quality data
19 National centres and National Project Managers
19 Main tasks of the National Project Managers
20 Phases of the TALIS study
20 Pilot study
20 Field trial
21 Main survey

TALIS 2008 Technical Report © OECD 2010

18
chapter 1 Introduction and Summary

Abstract

This chapter offers an overview of the OECD Teaching and Learning International Survey (TALIS)
and summarises the survey’s main features and objectives. It introduces the main stakeholders in
the management and implementation of TALIS and their responsibilities, discusses standardised
procedures used for the data collection and analyses and outlines the survey’s milestones,
beginning with the pilot study in 2006.

Overview of Talis
TALIS is the first international survey to focus on the working conditions of teachers and the learning environment in
schools. It focuses on four main research areas: school leadership; professional development; teacher appraisal and
feedback; and teaching practices, beliefs and attitudes. Data were collected through two types of questionnaires: a
principal questionnaire that was completed by school principals and a teacher questionnaire that was completed
by the sampled teachers. The questionnaires were designed to be completed by paper-and-pencil or on line.

The OECD Teaching and Learning International Survey (TALIS) Technical Report offers detail and documentation
about the development of the TALIS instruments and the methods used in sampling, data collection, scaling
and data analysis. The Technical Report enables review and replication of the procedures used for TALIS.
Furthermore, it provides insight into the rigorous quality control programme that operated during all phases of
the survey, involving numerous partners of the Consortium as well as external experts.

The first cycle of TALIS was conducted in 24 countries: Australia, Austria, Belgium (Flemish Community),
Brazil, Bulgaria, Denmark, Estonia, Hungary, Iceland, Ireland, Italy, Korea, Lithuania, Malta, Malaysia, Mexico,
the Netherlands, Norway, Poland, Portugal, Spain, the Slovak Republic, Slovenia and Turkey. Sixteen countries
chose to use the on line data collection module (refer to Chapter 6 for more details).

The International Association for the Evaluation of Educational Achievement Data Processing and Research
Center (IEA DPC), together with its consortium members Statistics Canada and the IEA Secretariat, partnered with
the OECD as international contractor in April 2006. The International Study Centre (ISC) completed the work
for the first cycle of TALIS in September 2009 with the publication of the TALIS international report, Creating
Effective Teaching and Learning Environments: First Results from TALIS (OECD, 2009) and the TALIS 2008
Technical Report (OECD, 2010).

The first sections of this chapter outline how the first cycle of TALIS was managed at the international and
national levels. The final two sections outline the three major phases of the survey.

Management of the study

The OECD Secretariat represented and co-ordinated TALIS, holding overall responsibility for managing
the project and monitoring implementation on a day-to-day basis through their communication with the
international contractor. The OECD Secretariat served as the secretariat for the TALIS Board of Participating
Countries (BPC), fostered consensus between the participating countries and acted as interlocutors to the TALIS
BPC and the international contractor.

The TALIS Consortium (see Annex A) included members from the IEA Secretariat (Amsterdam, Netherlands),
Statistics Canada (Ottawa, Canada) and the IEA DPC (Hamburg, Germany), where the ISC was located.

Mr. Juriaan Hartenberg managed financial and contractual relations between the Consortium and the OECD.

© OECD 2010 TALIS 2008 Technical Report

Introduction and Summary chapter 1
19

Standardised procedures to ensure high-quality data

In order to ensure international comparability of the questionnaires and the quality of the international
database, the ISC implemented standardised procedures throughout all stages of survey preparation, survey
administration and data processing. The ISC prepared six manuals that described all the necessary steps to the
successful implementation of TALIS. The manuals were provided in both English and French (the two working
languages of the OECD) and NPMs were requested to follow the procedures as outlined in the manuals.

Statistics Canada conducted the sampling and weighting for all participating countries following the rules and
guidelines outlined in the TALIS Sampling Manual (MS-02-03). For more details about the sample design of
TALIS, see Chapter 5. For more details about sampling weights and participation rates, see Chapter 10.

IEA provided a software package for within-school sampling, data entry and on line data collection (ODC).
Participating countries were asked to use the software, which had operated successfully in numerous IEA
surveys and had been specially adapted for TALIS. IEA Windows Within-School Sampling Software (WinW3S)
enabled NPMs to draw the teacher sample for each nationally sampled school. When the questionnaires
arrived back at the national centres, data were entered using the IEA Windows Data Entry Manager (WinDEM).
Countries applying the ODC module were provided with the IEA SurveySystemDesigner. During a three-day
data management seminar at the beginning of the survey, national data managers were trained in the correct use
of these software packages. This extensive hands-on training familiarised them with the software at the survey
preparation stage.

Quality control was implemented at different levels and during all phases of the survey to closely monitor survey
procedures and data outcomes. At the international level, the IEA Secretariat co-ordinated quality control of
the survey operations procedures whereas at the national level NPMs were asked to nominate quality control
monitors who reported exclusively to the NPM (see Chapter 8 for more details on quality control).

During the first TALIS cycle the ISC held an annual or bi-annual meeting for all NPMs in order to provide an
update on the progress of the survey and to discuss procedures, questionnaires and data.

National centres and National Project Managers

The first task for participating countries was to establish a national centre under the auspices of an
experienced NPM, who became responsible for preparing and co-ordinating the survey at the national level.
In most countries NPMs were supported by a national data manager who managed the technical aspects of
administering the survey. Depending on the size of the country and the organisation of the national centre,
the number of staff members who were involved in the survey varied considerably. Regular communication
between the NPM and the ISC ensured that survey administration proceeded according to the international
schedule.

Main tasks of the National Project Managers

In order to collect TALIS data, national centres distributed questionnaires to teachers and principals
who completed and returned the questionnaires within a defined period of time. This procedure differed
significantly from data collection for achievement studies, which rely on standardised testing sessions. It
was a demanding task to ensure high participation rates at the school and teacher level. In order to achieve
this, NPMs had to cooperate intensively with teachers unions and the ministry. Since TALIS is a new survey,
public relation efforts were necessary to raise awareness among principals and teachers prior to the main
data collection.

TALIS 2008 Technical Report © OECD 2010

20
chapter 1 Introduction and Summary

At national centres NPMs and data managers performed the following tasks:
• established an overall schedule in co-operation with the ISC;
• attended NPM meetings to become acquainted with all TALIS instruments, materials and survey
procedures;
• provided an up-to-date national sampling frame of International Standard Classification of Education (ISCED)
Level 2 schools and discussed national options like oversampling directly with the sampling experts from
Statistics Canada;
• performed within-school sampling and tracking using the IEA WinW3S;
• appointed an experienced translator to produce the national version(s) of the questionnaires based on the
international instruments;1
• documented the required cultural adaptations using the National Adaptation Forms;
• prepared for ODC with the IEA SurveySystemDesigner (if applicable);
• nominated and trained school co-ordinators;
• nominated and trained national quality control monitors;
• monitored the return status of the questionnaires using a software tool provided by the ISC;
• entered data manually using WinDEM or monitored data entry if an external agency was subcontracted;
• performed quality control procedures in WinDEM; and
• completed the survey activities questionnaire after survey administration.

Phases of the TALIS study

The TALIS design included three major components: a pilot study, a field trial and the main survey. For the pilot
study five countries volunteered to test the questionnaires within some self-selected schools. Each country was
requested to run a field trial according to the standardised procedures mentioned in the previous section. All
countries that completed the field trial also performed the main survey.

Pilot study
In order to check the quality and the content of the questionnaires, a small pilot study was conducted in the third
quarter of 2006. Brazil, Malaysia, Norway, Portugal and Slovenia volunteered to test the instruments in their
countries within a survey administration window of three weeks. Each of the participating countries selected
five schools. Within these schools, one principal and five teachers volunteered to respond to the questionnaires.
Therefore every pilot study country gathered data from 5 principals and 25 teachers. Data were entered by the
national centres and processed by the ISC. Each NPM prepared a summary report reflecting the strengths and
weaknesses of the survey procedures and outlining the feedback received from principals and teachers.

To facilitate survey implementation, the first three-day NPM meeting in Paris, France was held two months prior
to the administration of the pilot study. The purposes of the meeting were to present the instruments, materials
and survey operations procedures for TALIS and to introduce NPMs to the survey schedule.

Field trial
The objective of the field trial (FT) was to test the survey instruments and operational procedures in all
participating countries in preparation for the main survey (MS). Austria, Bulgaria and Mexico, which joined
the survey in early 2007 (somewhat later than other countries), also performed field trials. Bulgaria’s field trial
utilised MS instruments, as these were already complete when it joined the survey.

© OECD 2010 TALIS 2008 Technical Report

Introduction and Summary chapter 1
21

In November 2006, the second two-day NPM meeting was held in Hamburg, Germany to discuss the outcomes
of the pilot study and the required changes to the instruments for the FT. After the NPM meeting, the ISC
conducted a three-day data management seminar to train the national data managers in using WinW3S for
within school sampling, WinDEM for data entry and the IEA SurveySystemDesigner for on line data collection.
The 16 countries that had decided to use ODC for the main survey trialled procedures and data collection
during the FT.

All participating countries conducted the FT during a six-week timeframe in the second and third quarter
of 2007.2 Sampling, translation verification and layout verification were performed following main survey
standards. The expected sample size per country was 400 teachers and 20 principals out of 20 schools sampled
by Statistics Canada prior to the FT (see Chapter 5 for more details). Exceptions were made for Iceland and
Malta because of their size: the sample size for Iceland was 10 schools, 10 principals and 100 teachers and the
sample size for Malta was 4 schools, 4 principals and 40 teachers.

Data entry was performed under the supervision of the NPM or data manager using IEA WinDEM software.
National data sets were then submitted to the ISC for data processing and quality checks.

After the FT the ISC collected feedback from NPMs on how the survey had been perceived within the country
and whether modifications to the procedures were needed. One of the challenges of TALIS was to achieve high
participation rates at the school and individual levels. Considerable efforts were exerted in communicating
information about the study. Information about best practices relative to ensuring high participation rates was
gathered and shared with all NPMs prior to the MS preparation and was included in the TALIS Manual for
National Project Managers (MS-01-03) .

Main survey
The third three-day NPM meeting was held in July 2007 in Tallinn, Estonia to prepare NPMs for administrating
the MS. The meeting focused on the outcomes of the FT and the presentation of the finalised MS instruments.
The rules for international and national quality control monitoring (see Chapter 8) were explained since, unlike
the FT, the MS utilised external experts for quality control at the international level.

Figure 1.1
Milestones of the survey

Activity
2006: third quarter Pilot study: conducted in five self-selected countries
2007: first and second quarter FT: testing of software and procedures; mandatory for all participating countries
2007: second and third quarter Finalisation of instruments; fine-tuning of software and procedures
2007: fourth quarter MS: Southern Hemisphere countries
2008: first and second quarter MS: Northern Hemisphere countries
2008: third quarter Data processing, cleaning, weighting
2008: third and fourth quarter Analysis of the MS data; drafting of the International and the Technical Reports
2009: first and second quarter Finalisation and release of the International Report
2009: second quarter International database analyser training at the ISC
2009: second and third quarter Finalisation and release of the Technical Report and the TALIS User Guide

Source: OECD.

TALIS 2008 Technical Report © OECD 2010

22
chapter 1 Introduction and Summary

Southern Hemisphere countries conducted the MS in the fourth quarter of 2007 and had until February 2008 to
submit their data. Northern Hemisphere countries administered the survey within a self-selected period during
the first and second quarter of 2008, with a final data submission deadline of May 31, 2008.

All data were processed and cleaned at the ISC. Weights and weight adjustments were performed at Statistics
Canada in the third quarter of 2008.

In October 2008 the fourth three-day NPM meeting in Dublin, Ireland, took place. The purpose of the meeting
was to review prototype tables for the TALIS international report in plenary and to discuss country data in
individual country sessions. All countries had an opportunity to verify their entries in the National Adaptation
Database (NADB). NPMs provided feedback about the first round of TALIS, allowing their experiences to be
incorporated into the second round of the survey.

We would like to take the opportunity to thank all participants who contributed to the success of the first round
of TALIS.

Notes

1. Although the questionnaire translations were verified by independent experts, NPMs retained overall responsibility for the final
versions.

2. Only Bulgaria conducted the survey in the fourth quarter of 2007 within a timeframe of four weeks.

© OECD 2010 TALIS 2008 Technical Report

Chapter 2

Overview of TALIS 2008

and Framework Development
24 Abstract
24 The rationale for and aims of TALIS
25 The TALIS survey programme design
25 TALIS linked to PISA
25 The conceptual and analytic framework of the
programme
26 Choosing the policy focus for TALIS

TALIS 2008 Technical Report © OECD 2010

24
chapter 2 Overview of TALIS 2008 and Framework Development

Abstract

This chapter presents the rationale and aims of TALIS, as well as the design of the TALIS survey
programme, which was conceptualised as a cycle of surveys, the first of which was conducted
in 2007-08 and is the focus of this report. The chapter presents the conceptual and analytical
frameworks that shaped the development of the programme and offers an overview of the policy
issues and indicators considered and chosen for the first TALIS cycle.

The rationale for and aims of TALIS

TALIS was developed as part of the Organisation for Economic Co-operation and Development (OECD)
Indicators of Education Systems (INES) project, which, over the past 20 years has developed a coherent set of
indicators that provide a reliable basis for the quantitative comparison of the functioning and performance of
education systems in OECD and partner countries. The main product of the INES project is the annual OECD
publication Education at a Glance (OECD, 2008).

Providing information on teachers, teaching and learning is an essential component of the INES programme. At
the INES General Assembly in 2000 in Tokyo, countries called for increased attention to teachers and teaching
in future work. The importance of teachers, including the need for better information on the quality of learning
and how teaching influences learning, was affirmed at the meeting of education sub-Ministers in Dublin in 2003.

Although the INES programme has made considerable progress over the years in developing indicators about
the learning environment and organisation of schools, as well as learning outcomes, significant gaps remain
in the knowledge base on teachers and teaching. To address these deficiencies, a data strategy was developed
that proposed steps towards improving the indicators on teachers, teaching and learning. The strategy that was
developed identified strands of work, one of which was an international survey of teachers, which evolved into
the TALIS programme.

At the same time, the OECD review of teacher policy, which concluded with the report Teachers Matter:
Attracting, Developing and Retaining Effective Teachers (OECD, 2005), identified a need to develop better
national and international information on teachers. The policy framework used in the policy review as well as
the specific data gaps and priorities that it highlighted were instrumental in the design of TALIS.

The overall aim of the TALIS series of surveys is therefore to provide, in a timely and cost-efficient manner, policy
relevant, robust international indicators and analysis on teachers and teaching that help countries to review
and develop policies that create the conditions for effective schooling. Such cross-country analyses provide the
opportunity to examine countries facing similar challenges and to learn from different policy approaches and
the impact they have on the learning environment in schools.

The guiding principles that shaped the development of the survey strategy are:
• Policy relevance. Clarity about the policy issues and a focus on the questions that are most relevant for
participating countries are both essential.
• Value-added. International comparisons should be a significant source of the study’s benefits.
• Indicator-orientation. The results should yield information that can be used to develop indicators.
• Validity, reliability, comparability and rigour. Based on a rigorous review of the knowledge base, the survey
should yield information that is valid, reliable and comparable across participating countries.

© OECD 2010 TALIS 2008 Technical Report

Overview of TALIS 2008 and Framework Development chapter 2
25

• Interpretability. Participating countries should be able to interpret the results in a meaningful way.
• Efficiency and cost-effectiveness. The work should be carried out in a timely and cost-effective way.

The TALIS survey programme design

TALIS is conceived as a sequence of surveys that, over time, could be administered to teachers from all phases
of schooling. The design also incorporates a progressive move to a more fully implemented link of teacher
information to the Programme for International Student Assessment (PISA) operated by the OECD. The specific
plans for successive survey rounds will be reviewed after the first round of the cycle is completed.

Countries participating in the first round of TALIS decided that the main focus of the first round should be teachers
in lower secondary education and their school principals, but also agreed that there should be international
options through which countries could also survey teachers at other levels of education.

The first round of TALIS comprised a core target population and international sampling options. The core
included a representative sample of International Standard Classification of Education (ISCED) Level 2 teachers
and principals of their schools. The international options included the following: Option 1: a representative
sample of teachers of primary education (ISCED Level 1) and the principals of their schools; Option 2: a
representative sample of teachers of upper secondary education (ISCED Level 3) and the principals of their
schools; and Option 3: a representative sample of teachers of 15-year-olds in the schools that took part in
PISA 2006 and the principals of those schools.

The TALIS Board of Participating Countries (BPC) agreed that for an option to be adopted as an international
option, more than half of the participating countries would need to decide to take it up; otherwise the option
would be pursued as a national option. In the event, none of the international options was adopted and
only Iceland pursued a national option of surveying ISCED Level 1 teachers and their principals. In addition,
alongside the TALIS international survey, Mexico conducted a national survey of teachers in Telesecundaria
(distance learning) schools using slightly adapted versions of the TALIS survey instruments.

TALIS linked to PISA

In the process of developing TALIS, several countries expressed a desire to have the survey linked to outcome
measures. Options for achieving this were considered and a link to the PISA outcome measures was seen as the most
obvious route. Following concerns raised by countries about conceptual, methodological and operational issues,
two expert reviews considered what scientifically valid insights could be gained from linking TALIS to PISA.

The assessment from these reviews clarified that while insights to teacher and teaching effectiveness could not
be gained through linking a teacher survey to PISA, there would be value in using the teacher responses to
develop a fuller picture of the learning environment of 15-year-old students in PISA schools and to examine
the relationship with that and school level PISA outcome variables. Nevertheless, some uncertainties remained
and it was therefore decided to include in the first round of TALIS an experimental link to PISA 2006 for those
countries that were interested in taking up the option. As no country pursued this option, further consideration
will be given to the extent of the link between TALIS and PISA in planning future rounds of TALIS.

The conceptual and analytic framework of the programme

A joint taskforce comprising experts from the INES Network A (learning outcomes) and Network C (learning
environment and school organisation) developed the original conceptual framework for the TALIS programme.
The taskforce was asked to develop a data strategy on teachers, teaching and learning in order to fill the data gaps
at the international level and help make the coverage of the INES indicators more complete. A major part of this

TALIS 2008 Technical Report © OECD 2010

26
chapter 2 Overview of TALIS 2008 and Framework Development

strategy was a survey programme that developed into TALIS. Underpinning the data strategy was a conceptual
framework for the development of data and indicators, encompassing the following seven dimensions: i) policies
for maintaining a high-quality teaching force; ii) system-wide characteristics of the teacher workforce; iii) school
policies and antecedents that impact teaching and learning; iv) classroom ecology; v) student characteristics;
vi) teacher characteristics and antecedents; and vii) teaching and learning activities.

In the course of developing TALIS, the components of this conceptual framework were transposed into a policy
framework for questionnaire development. The framework identifies five main policy issues that to a large
degree reflect the policy issues that had been studied in the OECD teacher policy review (OECD, 2005). The
five main policy issues together with the “indicator domains” within them are shown in Figure 2.1.

Full details of the framework are contained in the OECD document Proposal for an international survey of
teachers, teaching and learning (EDU/EC/CERI(2005)5).

Figure 2.1
Policy issues and related indicator domains examined in TALIS

Policy Issue 1 Attracting teachers to the profession

Indicator No. 1 Adequacy of teacher supply and teacher shortages
Indicator No. 2 Profile of new teachers
Indicator No. 3 Motivations and early career experience of new teachers
Indicator No. 4 Effectiveness of recruitment and selection procedures and incentives
Policy Issue 2 Developing teachers within the profession
Indicator No. 5 Profile of teachers’ education and training
Indicator No. 6 Frequency and distribution of education and training
Indicator No. 7 Satisfaction and effectiveness of education and training
Policy Issue 3 Retaining teachers in the profession
Indicator No. 8 Teacher attrition and turnover
Indicator No. 9 Job satisfaction and human resource measures
Indicator No. 10 Recognition, feedback, reward and evaluation of teachers
Policy Issue 4 School policies and effectiveness
Indicator No. 11 School leadership
Indicator No. 12 School climate
Policy Issue 5 Quality teachers and teaching
Indicator No. 13 Teaching practices, beliefs, and attitudes
Indicator No. 14 Quality of teachers (experience, qualifications, responsibilities)
Indicator No. 15 Division of working time

Source: OECD.

Choosing the policy focus for TALIS

In order to narrow down the proposed content of the survey and to assist countries in deciding whether to
participate in TALIS, a priority rating exercise was conducted. Initially this involved all OECD countries but once
the group of participating countries was established, it reviewed and finalised the results. The rating involved
each country assigning 150 rating points across the 15 “indicators” in the framework shown above, with higher
points indicating a higher priority for an indicator to be included in the first round of the TALIS survey.

Figure 2.2 shows the results of the rating exercise, which produced a priority ranking of the 15 indicators.

© OECD 2010 TALIS 2008 Technical Report

Overview of TALIS 2008 and Framework Development chapter 2
27

Figure 2.2
Results of priority-rating exercise

Ranking Indicator number Indicator

1 Indicator No. 10 Recognition, feedback, reward and evaluation of teachers
2 Indicator No. 11 School leadership
3 Indicator No. 13 Teaching practices, beliefs and attitudes
4 Indicator No. 14 Quality of teachers (experience, qualifications, responsibilities)
5 Indicator No. 7 Satisfaction and effectiveness of education and training
6 Indicator No. 5 Profile of teachers’ education and training
7 Indicator No. 12 School climate
8 Indicator No. 15 Division of working time
9 Indicator No. 6 Frequency and distribution of education and training
10 Indicator No. 9 Job satisfaction and human resource measures
11 Indicator No. 3 Motivations and early career experience of new teachers
12 Indicator No. 2 Profile of new teachers
13 Indicator No. 4 Effectiveness of recruitment and selection procedures and incentives
14 Indicator No. 8 Teacher attrition and turnover
15 Indicator No. 1 Adequacy of teacher supply and teacher shortages

Source: OECD.

Participating countries reviewed these results at their meeting in October 2005 and decided on the following
main policy themes for the first round of TALIS: recognition, feedback, reward and evaluation of teachers;
school leadership; and teaching practices, beliefs and attitudes

In addition, “teacher professional development”, drawing on Indicators 5, 6 and 7 and the induction and
mentoring aspects of Indicator 4, was also chosen as an important theme in TALIS. In part this was because of
its synergies with the three main themes and also because it allowed TALIS to serve as a vehicle for countries
of the European Union to collect key information on teachers that the European Commission had identified as
important for monitoring progress towards the Lisbon 2010 goals.

Aspects of the lower–rated indicators were also included where they provided important complementary
analytical value to the main themes. In particular, this meant that aspects of “school climate” (Indicator 12) and
“division of working time” (Indicator 15) and a single item on “job satisfaction” (Indicator 9) were also included
in the survey.

The next chapter discusses the development of the TALIS survey instruments around the chosen themes.

TALIS 2008 Technical Report © OECD 2010

28
chapter 2 Overview of TALIS 2008 and Framework Development

References
OECD (2005), Teachers Matter: Attracting, Developing and Retaining Effective Teachers, OECD, Paris.

OECD (2008), Education at a Glance 2008 – OECD Indicators, OECD, Paris.

© OECD 2010 TALIS 2008 Technical Report

Chapter 3

Development of Teacher
and Principal Questionnaires
30 Abstract
30 The role and membership of the OECD TALIS Instrument
Development Expert Group
31 Timeline and phases of questionnaire development
31 Frameworks guiding questionnaire development
33 Teacher background characteristics and professional
development
34 Principal and school background characteristics
34 School leadership and management
35 Appraisal of and feedback to teachers
37 Teaching practices, attitudes and beliefs
39 Pilot test
39 Field trial

TALIS 2008 Technical Report © OECD 2010

30
chapter 3 Development of Teacher and Principal Questionnaires

Abstract

In order to establish goals for the development of the teacher and principal questionnaires,
the TALIS Board of Participating Countries (BPC) conducted a priority rating exercise (see
Chapter 2). To translate these goals into survey analysis plans and survey questionnaires (i.e. the
Teacher and Principal Questionnaires), an Instrument Development Expert Group (IDEG)
was established in conjunction with the BPC. This chapter explains in detail the frameworks
guiding the questionnaire development for each of the main themes covered by the first TALIS
survey: teacher background characteristics and professional development; principal and school
background characteristics; school leadership and management; appraisal of and feedback to
teachers; and teaching practices, attitudes and beliefs.

The role and membership of the OECD TALIS Instrument Development Expert
Group

The OECD Secretariat led the work of the IDEG, which reported to the TALIS BPC. The specific tasks of the
IDEG were to: review the proposed indicators for the survey to ensure that the variables, indicators and themes
provide a logical basis for instrument development, giving consideration to completeness and coherence;
review the catalogue of existing questions compiled from national and international studies in order to assess
their suitability for measuring the variables within the TALIS analytic framework and to identify other possible
sources of exemplary questions; draft suitable questions for the development of the identified indicators and
research questions; consider and advise on implications for the sample design arising from the questionnaire
development and vice versa; consider and advise on the extent to which the teacher questionnaire in the
main survey should be the same as that in the experimental PISA link; review and revise the questionnaires in
the light of pilot and field trial (FT) results; contribute to the drafting of the survey analysis plans; and present
proposed questionnaires and analysis plans to the BPC.

The persons appointed to the IDEG were chosen for their expertise in the main policy themes selected for TALIS
and for their considerable experience in survey-based education policy research and in instrument development
in the international context. The four chosen experts were:

• Dr. David Baker, Harry and Marion Eberly Professor of Comparative Education and Sociology, Department of
Education Policy Studies, Penn State University, USA. Dr. Baker led the development of the school leadership
sections of TALIS;
• Prof. Aletta Grisay, Consultant, Belgium. Prof. Grisay took a lead role on the teaching attitudes, beliefs and
practices sections of TALIS;
• Prof. Dr. Eckhard Klieme, Head of the Center for Education Quality and Evaluation, German Institute for
International Educational Research, Frankfurt, Germany. Prof. Dr. Klieme led the development of the teaching
attitudes, beliefs and practices sections of TALIS; and
• Prof. Dr. Jaap Scheerens, Professor of Educational Organisation and management, University of Twente, The
Netherlands and chair of the INES Network C, in which the early stages of TALIS were formulated. Prof. Dr.
Scheerens led the development of the teacher appraisal and feedback sections of the TALIS survey.

© OECD 2010 TALIS 2008 Technical Report

Development of Teacher and Principal Questionnaires chapter 3
31

The OECD Secretariat led the drafting of teacher, principal and school background questions as well as
questions related to teachers’ professional development. Representatives of the international contractor, the
International Association for the Evaluation of Educational Achievement (IEA), attended the IDEG meetings to
ensure coherence between instrument development and sample design.

The IDEG’s responsibility was to develop proposed questionnaire content for review by the TALIS National
Project Managers (NPMs) and ultimately the TALIS BPC. TALIS NPMs advised on the validity of the questions
nationally, the value of the questions in the intended analysis, and the clarity of the drafting and sequencing of
the questions. The TALIS BPC approved the questionnaires at each stage of testing and implementation (pilot,
field trial and main survey) and advised on the political relevance of the questionnaire content, its adherence
to the goals of TALIS and the validity of the questions nationally.

Timeline and phases of questionnaire development

The TALIS NPMs and BPC reviewed each successive draft of the questionnaires. The validity of the questionnaires
was first tested through a small-scale pilot in five countries and then more fully in a subsequent FT. While the
pilot test was solely focused on testing the questionnaires, the FT had the wider remit of testing the survey
operations and further testing the validity of the questionnaires.

In summary, the main steps in the development of the questionnaires were:

• TALIS BPC conducted a priority-rating exercise to determine main themes of TALIS – third quarter of 2005.
• IDEG elaborated these themes into research, questions, variables and constructs – fourth quarter of 2005.
• IDEG drafted questionnaire items – January to March 2006.
• TALIS BPC reviewed first draft of questionnaires – April 2006.
• TALIS BPC reviewed second draft of questionnaires – July 2006.
• NPMs conducted further review of second draft of questionnaires – August 2006.
• Pilot version of questionnaires approved – August 2006.
• Pilot conducted – third quarter 2006.
• OECD and IEA meeting to review pilot results – November 2006.
• NPM meeting reviewed pilot results – November 2006.
• IDEG consultations on proposed changes for the FT – November 2006.
• FT version of questionnaire agreed with BPC – December 2006.
• FT conducted – first and second quarter 2007.
• BPC made a plan for assessing the FT results and actions to be taken– June 2007.
• Extended IDEG meeting (with BPC representation) reviewed FT results and proposed changes for main
survey (MS) questionnaires – July 2007.
• Consultations with BPC on finalising MS questionnaires – August 2007.
• MS questionnaires finalised – end August 2007.
• MS began in southern hemisphere countries – October 2007.
• MS began in northern hemisphere countries – February 2008.

Frameworks guiding questionnaire development

The basic framework guiding the development of the questionnaires also formed the basis of the priority-rating
exercise used to determine the main policy themes for TALIS (see Chapter 2). As noted in Chapter 2, the policy

TALIS 2008 Technical Report © OECD 2010

32
chapter 3 Development of Teacher and Principal Questionnaires

themes chosen by the participating countries for the first TALIS were: recognition, feedback, reward and evaluation
of teachers; school leadership; and teaching practices, beliefs and attitudes. Teacher professional development
was also included as an important theme.

The initial phase of questionnaire development involved assessing the outcomes of the priority-rating exercise and
identifying which aspects of the themes could viably be captured through teacher and school principal responses.
Each theme was elaborated in terms of the research and analytical questions that could be investigated and the
variables and constructs that could be measured. In addition, the IDEG reviewed suitable questionnaire items
from other international and national surveys and decided whether they could be used or adapted for TALIS.

From the outset, a goal for questionnaire development was to ensure that the chosen themes were investigated
sufficiently thoroughly to generate valuable policy insights while keeping the questionnaire to a manageable
length. To help address this, particular attention was paid to how the themes interrelate, in order to maximise
the analytical value of the study.

The TALIS analysis plans outlined the research and analytical questions and how they were to be investigated
through analysis of the TALIS data (OECD official document EDU/INES/TALIS(2008)2). The remainder of this
section summarises the frameworks that were developed for each section of the TALIS questionnaires and set
out in the analysis plans.

By way of an overview, Figure 3.1 represents how the different themes of the TALIS survey fit together.

Figure 3.1
Survey themes for TALIS

Professional Teacher Classroom

competence classroom practice level environment
(Knowledge and beliefs)
Structuring Time on task*
Content knowledge Student orientation Classroom
Pedagogical content Enhanced activties disciplinary climate Overall
knowledge job-related
Student Student
attitudes:
learning outcomes
Related Teachers’ School Self-efficacy
beliefs and attitudes professional activities level environment Job satisfaction*
Beliefs about the Co-operation among School climate:
nature of teaching staff: Teacher-student
and learning: • exchange and relations
• direct transmission co-ordination
for teaching
• constructivist beliefs
• professional
collaboration

Teacher background
School background and processes
Professional training / Student background
(e.g. leadership)
experience

Note: Constructs that are covered by the survey are highlighted in blue; single item measures are indicated by an asterisk (*).
Source: Creating Effective Teaching and Learning Environments: First Results from TALIS (OECD, 2009).

© OECD 2010 TALIS 2008 Technical Report

Development of Teacher and Principal Questionnaires chapter 3
33

The shaded elements are those that TALIS measured and the unshaded elements are those that TALIS did not
measure. The arrows illustrate different questions that were addressed in the data analysis. The elements relate
to each other as follows:

• The quality of the learning environment is the most important causal factor for student learning and student
outcomes; important aspects of the school-level environment were addressed in the appraisal and feedback
and school evaluation sections of the survey, the school leadership sections and in measures of school climate.
• The learning environment at the classroom level has an even stronger impact on student learning. However,
as this environment most often varies between subjects and teachers, it is not easy to identify domain-general
indicators. TALIS chose disciplinary climate because this variable has a strong impact on student learning in
different subjects, and because it has been shown that – unlike other features of classroom climate – there is
a high level of agreement between teachers, students and observers with regard to this indicator.
• The quality of the learning environment at the classroom level, in turn, is to a large extent determined by the
instructional methods and classroom practices used by the teacher.
• A large part of teacher activity occurs in the classroom, through instructing classes of students more or less in
isolation from other classes and other teachers. A modern view of teachers also acknowledges professional
activities at the school level, such as cooperating in teams of teachers, building professional learning
communities, participating in school development, and evaluating and changing working conditions. Those
activities shape the learning environment at the school level, i.e. school climate, ethos and culture, which
directly and indirectly (via classroom level processes) impact student learning. In addition to questions on
recognition, feedback, reward and appraisal, TALIS covers various aspects of teacher co-operation.
• Teachers’ classroom practices and other professional activities depend on their personal prerequisites. Earlier
research identified prerequisites as general teacher characteristics, such as personality traits or styles of
interaction. Recent research, however, (see e.g. Campbell et al., 2004) focuses on specific characteristics that
are the result of the teacher’s academic education and socialisation as a professional who understands and
teaches a certain subject area.
• Professional competence is believed to be a crucial factor in determining classroom and school practices.
In addressing this factor, however, TALIS had to limit its scope. Policy makers from participating countries
and the IDEG agreed that it would be impossible to measure professional competence in a broad
sense. Covering professional knowledge as described by Lee Shulman’s most influential triarchic theory
(professional knowledge = content knowledge + pedagogical content knowledge, related to subject
specific didactics + general pedagogical knowledge) is beyond the scope of this survey. Just as Seidel and
Scheerens (in press, p. 4) did, this heterogeneous and “soft” concept was reduced to a few basic factors.
Thus, TALIS focuses on more general types of teacher attitudes and beliefs, which have been shown to
have some influence on the quality of student learning (Seidel & Scheerens, in press, p. 12).

In addition to these pedagogical beliefs and attitudes, TALIS addresses self-related beliefs, namely teacher self-
efficacy (measured by a well-established scale) and job satisfaction (single item rating).

The next sections present the TALIS teacher and principal questionnaires in more detail.

Teacher background characteristics and professional development

Teacher demographic and employment characteristics were included in TALIS as key teacher variables not
only for providing a profile of the ISCED Level 2 teacher population in participating countries but also for the
analysis of the policy themes in TALIS. The survey collected information on age, gender and level of educational
attainment of teachers, as well as their employment status, years of experience as a teacher and a profile of
how they spend their time. Analysis of TALIS data by these characteristics permitted analysis of equity issues.

TALIS 2008 Technical Report © OECD 2010

34
chapter 3 Development of Teacher and Principal Questionnaires

Teacher professional development is an important theme for TALIS, providing information on the amount and
type of professional development undertaken, its perceived impact on teachers’ development, the types of
support for teachers undertaking professional development activities and teachers’ professional development
needs.

In order to capture a broad range of development activities – both formal and informal – professional development
was defined as activities that develop an individual’s skills, knowledge, expertise and other characteristics
as a teacher. This included more organised activities such as courses and workshops as well as less formal
professional development such as reading professional literature such as academic journals.

As well as providing a profile of teachers’ professional development activities, the questions were designed to
allow an analysis of the relationships between professional development activities, appraisal and feedback that
teachers receive, and their teaching practices, beliefs and attitudes. In addition, school principals were asked to
report on the organisation of induction and mentoring practices in their school.

The questions on teachers’ professional development were newly developed for TALIS or adapted from
similar types of questions used in other surveys such as the IEA Progress in International Reading Literacy
Study 2001 teacher questionnaire (IEA, 2001); World Education Indicators Survey of Primary Schools 2004
(UNESCO, 2008), The United States Department of Education Schools and Staffing Survey, conducted by the
National Center for Education Statistics (U.S. Department of Education, 2004).

Principal and school background characteristics

As in the teacher questionnaire, the principal questionnaire included demographic and employment
characteristics (age, gender, educational attainment, length of experience as a principal, profile of how they
divide their time between different tasks) to provide context for the analysis of the main policy themes of TALIS.

The school principal questionnaire also included questions on the background characteristics of the school,
in order to provide important contextual information for the TALIS analysis. The questions provide information
based on school principal reports on the following (sources noted, otherwise items were newly developed for
TALIS): public or private status of the school (adapted from the Programme for International School Assessment
[PISA] 2006 School Questionnaire); size of community in which the school is located (from the PISA 2006
School Questionnaire); number and type of staff in the school (categories based on those collected through
the UNESCO-UIS/OECD/EUROSTAT [UOE] data collection); total school enrolment; socio-economic and
linguistic background of students (adapted from questions in the PISA 2003 School Questionnaire and World
Education Indicators Survey of Primary Schools 2004); school admission policies (from the PISA 2006 School
Questionnaire); school resources (adapted from the PISA 2006 School Questionnaire); and school and teacher
autonomy (adapted from the PISA 2006 School Questionnaire).

School leadership and management

The sections of the TALIS questionnaires that relate to school leadership and management were in part
framed around the Principal Instructional Management Rating Scale (PIMRS), which provided indicators of
principals’ emphasis on instructional leadership job functions associated with leadership in effective schools
(Hallinger, 1994). It was also guided by other work undertaken by the OECD (OECD, 2008b). In addition,
items intended to record different forms of management (in addition to instructional leadership) are based on
work by Quinn et al. (1996), which distinguishes between four different managerial models or styles in which
managers have different values and goals, fulfilment of other roles and, consequently, the way organisations
are steered.

© OECD 2010 TALIS 2008 Technical Report

Development of Teacher and Principal Questionnaires chapter 3
35

Analysis of the questionnaire items enabled the identification of five scales for management behaviour and
styles:

• Management of school goals: Explicit management via the school’s goals and curriculum development.
• Instructional management: actions to improve teachers’ instruction.
• Direct supervision of instruction in the school: actions to directly supervise teachers’ instruction and
learning outcomes.
• Accountable management: managing accountability to shareholders and others.
• Bureaucratic management: management actions mostly aimed at bureaucratic procedures.

In turn, an analysis of the patterns of association across these five scales, yielded two underlying patterns of
management styles among principals in all countries:

• School Management Style A: Instructional Leadership

– Management of school goals.
– Instructional management.
– Direct supervision of instruction in the school.
• School Management Style B: Administrative leadership
– Accountable management.
– Bureaucratic management.

The main scales were constructed from principals’ responses about how often they undertake certain tasks and
activities in the school and their beliefs about their role in the school. In addition, teacher perceptions on school
leadership were obtained through similar questions in the teacher questionnaire.

Appraisal of and feedback to teachers

The framework for the teacher appraisal sections of the TALIS questionnaires and the questionnaire items themselves
were, in the main, developed specifically for TALIS. The framework focuses on teacher appraisal and aspects of
school evaluation that are related to teacher appraisal. Although the main focus was on teacher appraisal, it was
important to set this in the context of school evaluation more generally. School evaluations are often directly
related to teacher appraisal (teacher appraisal is often the direct result or a part of school evaluation) and in some
countries the system of school evaluation is an important policy-malleable aspect of teacher appraisal.

Figure 3.2 depicts the conceptual framework for evaluating education in schools and the main areas in which
data from teachers and school principals were collected. It reflects previous research on the role of evaluation
in the development of schools and teachers and on the design of such evaluations to meet education objectives
(OECD, 2008a; Sammons et al., 1994; Smith and O’Day, 1991).

Data collected in TALIS are at the school and teacher level from school principals and teachers and therefore
focus on the final three aspects of the evaluative framework of school education depicted in Figure 3.2.

TALIS collected data on school evaluations from school principals. The data include the frequency of school
evaluations, including school self-evaluations, and the importance placed upon various areas. Data were
also obtained on the impacts and outcomes of school evaluations, with a focus on the extent to which these
outcomes affect the school principal and the school’s teachers. TALIS also collected data from teachers on the
focus and outcomes of teacher appraisal and feedback. This information makes it possible to see the extent to
which the focus of school evaluations is reflected in teacher appraisal and feedback.

TALIS 2008 Technical Report © OECD 2010

36
chapter 3 Development of Teacher and Principal Questionnaires

Both school evaluation and teacher appraisal and feedback should aim to influence the development and
improvement of schools and teachers. Even a framework for evaluation based on regulations and procedural
requirements would focus on maintaining standards that ensure an identified level of quality of education.
TALIS therefore collected information on changes in teaching practices and other aspects of school education
subsequent to teacher appraisal and feedback. According to the model depicted in Figure 3.2, a focus in school
evaluations on specific areas that reflect stated policy priorities should also be a focus of teacher appraisal and
feedback. This should in turn affect practices in those areas. Since TALIS did not collect information on student
outcomes, teachers’ reports of changes in teaching practices are used to assess the impact of the framework
of evaluation. In addition, teachers’ reports of their development needs provide further information on the
relevance and impact of this framework on teachers’ development.

Figure 3.2
Structure for evaluation of education in schools: data collected in TALIS

Central objectives, policies and programmes, and regulations developed

by policy makers and administrators

School and teacher Student objectives Regulations and

objectives and standards and standards procedures

School evaluations
(Principal questionnaire)

Criteria and focus Impact and outcomes

(Principal questionnaire) (Principal questionnaire)

Teacher appraisal and feedback

(Teacher questionnaire and Principal questionnaire)

Criteria and focus Impact and outcomes

(Teacher questionnaire (Teacher questionnaire
and principal questionnaire) and principal questionnaire)

School and teacher development and improvement

(Teacher questionnaire)

Source: Creating Effective Teaching and Learning Environments (OECD, 2009).

Seventeen different aspects of school evaluation and teacher appraisal and feedback were collected. Respondents
were asked to identify the importance of each of these criteria in the school evaluation or teacher appraisal and
feedback. Links were then made between these criteria and teacher professional development, and the extent
to which teacher appraisal and feedback led to changes in these areas.

© OECD 2010 TALIS 2008 Technical Report

Development of Teacher and Principal Questionnaires chapter 3
37

Data were also collected from teachers on the role of appraisal and feedback in relation to rewards and
recognition within schools. The focus on factors associated with school improvement and teachers’ development
included teachers’ perceptions of the recognition and rewards obtained for their effectiveness and innovation
in their teaching.

In gathering data in TALIS, the following definitions were applied:

• School evaluation refers to an evaluation of the whole school rather than of individual subjects or departments.
• Teacher appraisal and feedback occurs when a teacher’s work is reviewed by either the school principal, an
external inspector or the teacher’s colleagues. This appraisal can be conducted in ways ranging from a more
formal, objective approach (e.g. as part of a formal performance management system, involving set procedures
and criteria) to a more informal, more subjective approach (e.g. informal discussions with the teacher).

Teaching practices, attitudes and beliefs

TALIS examines teacher beliefs, attitudes and practices across and between teachers, schools and countries.
Although TALIS is not aimed at explaining student achievement, achievement growth, student motivation or
motivational change, its study design highlights factors that have been shown to be related to such kinds of
student outcome. Figure 3.1 illustrates the choice of constructs for this section of the survey and their supposed
interactions.

Based on results from the TIMSS video study, Klieme et al. (2006) proposed three basic (second-order)
dimensions of instructional quality: clear and well-structured classroom management (which includes key
components of direct instruction), student orientation (including a supportive climate and individualised
instruction), and cognitive activation (including the use of deep content, higher order thinking tasks and other
demanding activities). These dimensions are to be understood as “latent” factors which are related to, but not
identical with specific instructional practices (see Lipowsky et al., 2008, for a theoretical foundation and an
empirical test of the model). TALIS uses a domain-general version of this triarchic model, identifying structure,
student orientation, and enhanced activities as basic dimensions of teaching practices.

Instructional practices, in turn, depend on what teachers bring to the classroom. Professional competence
is believed to be a crucial factor in classroom and school practices (Shulman, 1987, Campbell et al., 2004;
Baumert and Kunter, 2006). To study this, a number of authors have used, for example, measures of the effects
of constructivist compared with “reception/direct transmission” beliefs on teaching and learning, developed
by Peterson et al. (1989). TALIS uses a domain-general version of two teaching and learning-related indices
(constructivist and direct transmission) to cover teachers’ beliefs and basic understanding of the nature of
teaching and learning.

Teachers’ professional knowledge and actual practices may differ not only among countries but also among
teachers within a country. To gain an understanding of the prevalence of certain beliefs and practices it is
therefore important to examine how they relate to the characteristics of teachers and classrooms. For example,
previous research suggests that the beliefs and practices of female and male teachers may systematically differ
(e.g. Singer, 1996), so that TALIS must control for gender. From the perspective of education policy, however, it is
even more relevant to look at the impact on teachers’ beliefs, practices and attitudes of professional background
factors such as type of training, certification and professional development, subject taught, employment status
(part-time versus full-time) and length of tenure. It is important to note that any of these relationships can
have different causal interpretations. For example, professional development activities may change beliefs and
attitudes, but participation in such activities may itself be due to certain beliefs. As a cross-sectional study,
TALIS can describe such relationships, but it cannot disentangle causal direction. Some of the analyses TALIS

TALIS 2008 Technical Report © OECD 2010

38
chapter 3 Development of Teacher and Principal Questionnaires

provides on these matters are merely exploratory, because so far there is little research, for example, on beliefs
and practices specific to certain subjects.

Good instruction, of course, is not determined just by the teacher’s background, beliefs and attitudes; it should
also be responsive to students’ needs and various student, classroom and school background factors. TALIS looks
at whether teaching practices “adapt” to students’ social and language background, grade level, achievement
level, and class size. TALIS does not allow for examining whether classroom practices are adapted to individual
students but instead looks at macro-adaptivity (Cronbach, 1957), i.e. the adaptation of teaching practices to
characteristics of the class.

Teachers do not act only in the classroom where they instruct students more or less in isolation from other
classes and teachers. A modern view of teaching also includes professional activities on the school level, such
as co-operating in teams, building professional learning communities, participating in school development,
and evaluating and changing working conditions (Darling-Hammond et al. 2005). These activities shape the
learning environment at the school level, i.e. the school climate, ethos and culture, and thus directly and
indirectly (via classroom-level processes) affect student learning. TALIS distinguishes between two kinds of co-
operation by a school’s teaching staff: exchange and co-ordination for teaching (e.g. exchanging instructional
material or discussing learning problems of individual students) versus more general and more innovative kinds
of professional collaboration (e.g. observing other teachers’ classes and giving feedback). It is assumed that both
kinds of co-operative activities will be influenced by school-level context variables such as a school’s teacher
evaluation policies and the school’s leadership.

As is known from research on the effectiveness of schools (Scheerens and Bosker, 1997; Hopkins, 2005; Lee
and Williams, 2006; Harris and Chrispeels, 2006), the quality of the learning environment is the factor affecting
student learning and outcomes that is most readily modified, given that background variables such as cognitive
and motivational capacities, socio-economic background, social and cultural capital are mostly beyond
the control of teachers and schools. TALIS captures students’ background by asking teachers and principals
about the social composition and the relative achievement level of the student population they serve. A more
important task for TALIS is to assess quality, as perceived by teachers, at the classroom as well as the school
level. However, as the environment generally varies between subjects and teachers, it is not easy to identify
domain-general indicators. TALIS uses time on task – i.e. the proportion of lesson time that is actually used for
teaching and learning – as a basic indicator for the quality of the learning environment. Also, classroom climate
is used because of its strong impact on cognitive as well as motivational aspects of student learning in different
subjects. The method used here is adapted from PISA and focuses on the disciplinary aspect. For example, the
statement “When the lesson begins, I have to wait quite a long time for the students to quiet down” indicates
a low level of classroom discipline. It has been shown that classroom discipline, aggregated to the school
level, is a core element of instructional quality. In PISA, it is positively related to the school’s mean student
achievement in many participating countries (Klieme and Rakoczy, 2003). Also, it has been shown that – unlike
other features of classroom instruction – there is a high level of agreement about this indicator among teachers,
students and observers (Clausen, 2002). In addition to the environment at the classroom level, school climate is
used as an indicator for the school environment. Here, school climate is defined as the quality of social relations
between students and teachers (including the quality of support teachers give to students), which is known to
have a direct influence on motivational factors, such as student commitment to school, learning motivation and
student satisfaction, and perhaps a more indirect influence on student achievement (see Cohen, 2006, for a
review of related research). The triarchic model of instructional quality mentioned above (Klieme et al., 2006;
Lipowsky et al., 2008; Rakoczy et al., 2007) suggests specific relations between teaching practices and the two
climate factors: structure-oriented teaching practices should primarily relate to high levels of classroom climate,
while student-oriented practices should be linked with positive social relations.

© OECD 2010 TALIS 2008 Technical Report

Development of Teacher and Principal Questionnaires chapter 3
39

TALIS does not address the ultimate effects of classroom and school-level activities and climate on student
learning and outcomes. However, because TALIS studies teachers (as opposed to the effectiveness of education),
teachers were asked to evaluate what they themselves do. TALIS assessed teachers’ beliefs about their efficacy
by adopting a construct and a related measurement that is widely used in educational research (e.g. Schwarzer,
Schmitz and Daytner, 1999). As a second indicator TALIS used a single item for overall job satisfaction.
Research has shown that teachers’ sense of their efficacy plays a crucial role in sustaining their job satisfaction
(e.g. Caprara et al., 2006). It has also been found to be associated with constructivist goals and student
orientation (Wheatley, 2005) and with successful management of classroom problems and keeping students on
task (e.g. Chacon, 2005; Podell and Soodak, 1993). Thus, previous research suggests that there are significant
relations between teachers’ beliefs, attitudes and practices.

Pilot test
The purpose of the pilot test was to verify that the questionnaires worked in an international context in the
way intended. Five countries volunteered to take part in the pilot test of the questionnaires in November
2006: Brazil, Malaysia, Norway, Portugal and Slovenia (see also section in Chapter 1). Within the participating
countries five schools representing rural and urban areas and different education tracks (if applicable) were
identified for participation in a non-randomised selection process. Within these schools five teachers – males
and females – and the principal were selected. The teachers taught a mixture of subjects.

The pilot test was successful. Statistical analysis of the pilot results showed that the questionnaire items were, in
the main, well suited to support the constructs and analysis planned for TALIS. Throughout the questionnaires,
the statistical analysis and detailed feedback from respondents was helpful in identifying individual items
that could be deleted as well as those that were in need of clarification or rewording, either in the source
questionnaire or in their translation into national languages. Questionnaire length was a major issue raised by
pilot respondents, who reported an average completion time of 53 minutes for the teacher questionnaire and
60 minutes for the principal questionnaire. The wording of several items was amended and some items were
deleted in view of the feedback provided by pilot respondents.

A number of specific changes also arose from the pilot analyses. The questions identifying which subjects the
teachers taught and which they had studied were simplified. There was also a simplification of the questions
on the family background of students in the school, which had proved difficult and time-consuming to answer.
Additionally, the question on the impact of professional development undertaken was modified in order to
strengthen the analytical potential of these questions. The focus shifted to the strength of that impact, providing
a greater alignment between categories of professional development needs, the areas on which teachers receive
feedback and their actual teaching practices. Another change included the adoption of a more generalised
definition of “feedback” to make the questions more relevant in different country contexts. The instructions on
how to identify a target class, about which teachers should respond regarding their teaching practices, were
also improved and clarified.

All of the changes following the pilot test reduced the questionnaire’s length for the FT by approximately 15%.

Field trial
The main purpose of the FT was to test survey operations (data capture, processing and so on) but it also
provided a second opportunity to review the functioning of the questionnaire. The FT was conducted in all
24 participating countries (though the test was conducted later in Bulgaria), aiming for 20 participating schools
and 20 teachers within each school, providing a sizeable number of cases for analysis.

TALIS 2008 Technical Report © OECD 2010

40
chapter 3 Development of Teacher and Principal Questionnaires

The first stage of the analysis examined whether respondents understood the questionnaires by determining
whether they provided appropriate or realistic answers. The analysis focused on missing data, questionnaire
instructions, the presence of contradictory responses and the information provided in “other” categories. The
second stage of the analysis involved more detailed scrutiny of the data, including analysis of the descriptive
statistics, the design of scales and constructs, and analysis of the relationship between relevant indicators,
characteristics and policy issues. The TALIS NPMs and the respondents provided feedback and reports that
aided these analyses.

The changes to the questionnaires introduced following the pilot worked well. The smaller number of
questionnaire items reduced the completion time for both the teacher and principal questionnaires to
approximately 45 minutes. Since the respondents made few negative comments about the questionnaires’
length, this was deemed acceptable.

There were no significant problems with missing data or misunderstanding of questionnaire instructions.
Some issues were addressed by a slight rewording of items or instructions. In addition, an analysis of response
patterns led to some response categories being combined and a few items being deleted. Factor analysis of the
survey responses supported the proposed constructs as set out in the analysis plan, although a small number of
questionnaire items were deemed redundant and therefore deleted.

The most significant issue arising from the FT analysis concerned the questions to school principals about
students’ social and linguistic background, which the TALIS BPC had agreed were important in order to
provide valuable context. Items asking school principals to summarise characteristics of the student population
feature in a number of international surveys and the questions developed for TALIS drew on these. The item on
students’ linguistic background is very similar to a question used in the PISA 2000 and 2003 school background
questionnaire (OECD, 2000 and 2003). An item asking about parents’ education level was used in the UNESCO
World Education Indicators survey of primary schools (UNESCO, 2004).

Nevertheless, the questions proved problematic in the TALIS pilot and despite their being much simplified for
the FT, principals again reported difficulty. There were also concerns about the reliability of the information. The
FT version of the questions asked principals to “estimate the proportion of <ISCED 2> students who:

• Have a first language that is different from the language of instruction.

• Have at least one parent/guardian who has not completed <ISCED 3> or higher.
• Have at least one parent/guardian who has not completed <ISCED 5> or higher.

To address concerns about accuracy of the response and response burden, the MS question was altered to a
“tick box” format rather than asking for specific percentages. Also, because there was evidence that items b and
c were misunderstood, the question wording was amended in the MS to ask what percentage “has completed”
rather than “has not completed”. Figure 3.3 shows the revised question.

To further ensure that the information was obtained successfully, these questions were replicated in the
teacher questionnaire. Here, teachers were asked to estimate these characteristics for the class that was the
focus of the questions on teaching practices. The percentage of missing responses in the main study for these
questions in both the teacher and principal questionnaire ranged from 8% to 12% for the different items in
the questions.

Throughout the development of the questionnaires and in the analysis of the survey data, it was important to
manage the risk of cultural bias in the survey responses. This is a common challenge in international surveys
seeking attitudinal information. In the first instance it was addressed through careful review of the wording

© OECD 2010 TALIS 2008 Technical Report

Development of Teacher and Principal Questionnaires chapter 3
41

of questions and with approved national adaptations of the questions. Analysis of the FT data involved
some investigation of cross-cultural equivalence in the survey responses but most of this was carried out in
the analysis of the main study data. Chapter 10 discusses this in detail with regard to scale and indicator
construction.

Figure 3.3
Principal Questionnaire item on student characteristics, altered for the main survey

Please estimate the broad percentage of students at <ISCED 2> level in this school who have the following characteristics.
It is acceptable to base your replies on rough estimates.
Please mark one choice in each row.
10% or more 20% or more 40% or more
Less than but less but less but less 60%
10% than 20% than 40% than 60% or more
a) Students whose <first language> is different from the
language(s) of instruction or a dialect of this/these. 1 2 3 4 5

b) Students who have at least one parent/guardian who has

completed <ISCED 3> or higher. 1 2 3 4 5

c) Students who have at least one parent/guardian who has

completed <ISCED 5> or higher. 1 2 3 4 5

Source: OECD.

TALIS 2008 Technical Report © OECD 2010

42
chapter 3 Development of Teacher and Principal Questionnaires

References
Baumert, J. and M. Kunter (2006), “Stichwort: Professionelle Kompetenz von Lehrkräften”, Zeitschrift für
Erziehungswissenschaft, No.9 (4), pp. 469-520.

Campbell, A., O. McNamara and P. Gilroy (2004), Practitioner Research and Professional Development in Education,
Chapman, London.

Caprara, G.V., et al. (2006), “Teachers’ Self-Efficacy Beliefs as Determinants of Job Satisfaction and Students’ Academic
Achievement: A Study at the School Level”, Journal of School Psychology, No. 44(6), pp. 473-490.

Chacón, C. (2005), “Teachers’ Perceived Efficacy among English as a Foreign Language Teachers in Middle Schools in
Venezuela”, Teaching and Teacher Education, No. 21, pp. 257-272.

Clausen, M. (2002), Unterrichtsqualität: Eine Frage der Perspektive? Pädagogische Psychologie und Entwicklungspsychologie,
D.H. Rost (ed.), Waxmann, Munster.

Cohen, J. (2006), “Social, Emotional, Ethical and Academic Education: Creating a Climate for Learning, Participation in
Democracy and Well-being”, Harvard Educational Review, Vol. 76, No. 2, Summer, pp. 201-237.

Cronbach, L. (1957), “The Two Disciplines of Scientific Psychology”, American Psychologist, pp. 671‑684.

Darling-Hammond, L., et al. (2005), “Does Teacher Preparation Matter? Evidence about Teacher Certification, Teach for
America, and Teacher Effectiveness”, Education Policy Analysis Archives, No. 13(42).

Hallinger, P. (1994), A Resource Manual for the Principal Instructional Management Rating Scale (PRIMS Manual 2.2), Center
for the Advanced Study of Educational Leadership, Vanderbilt University, Nashville.

Harris, A. and J.H. Chrispeels (eds.) (2006), Improving Schools and Educational Systems: International Perspectives,
Routledge, London.

Hopkins, D. (ed.) (2005), “The Practice and Theory of School Improvement”, in International Handbook of Educational
Change, Springer, Dordrecht.

IEA (2001), “Progress in Reading Literacy Study 2001”, in PIRLS 2001 User Guide for the International Database, Gonzalez,
E.J., & Kennedy, A.M. (2003), Chestnut Hill, Boston College.

Klieme, E., F. Lipowsky, K. Rakoczy and N. Ratzka (2006), “Qualitätsdimensionen und Wirksamkeit von Mathematikunterricht:
Theoretische Grundlagen und ausgewählte Ergebnisse des Projekts ‘Pythagoras’“, in: M. Prenzel and L. Allolio-Näcke (eds.),
Untersuchungen zur Bildungsqualität von Schule, Abschlussbericht des DFG-Schwerpunktprogramms, Waxmann, Munster,
pp. 128-146.

Klieme, E. and K. Rakoczy (2003), “Unterrichtsqualität aus Schülerperspektive: Kulturspezifische Profile, regionale
Unterschiede und Zusammenhänge mit Effekten von Unterricht”; in J. Baumert, et al. (eds.), PISA 2000: Ein differenzierter
Blick auf die Länder der Bundesrepublik Deutschland, Leske & Budrich, Opladen, pp. 334-359.

Lee, J.C.K. and M. Williams (eds.) (2006), School Improvement: International Perspectives, Nova Science Publishers Inc.,
New York.

Lipowsky, F., et al. (2008), “Quality of Geometry Instruction and its Short-term Impact on Students’ Understanding of the
Pythagorean Theorem”, Learning and Instruction, Science Direct Website, Doi:10.1016/j.learninstruc., last consulted 2008.11.01.

Peterson, P.L., et al. (1989), “Teachers’ Pedagogical Content Beliefs in Mathematics”, Cognition and Instruction, No. 6 (1),
pp. 1- 40.

Podell, D. and L. Soodak (1993), “Teacher Efficacy and Bias in Special Education Referrals”, Journal of Educational Research,
No. 86, pp. 247-253.

© OECD 2010 TALIS 2008 Technical Report

Development of Teacher and Principal Questionnaires chapter 3
43

Rakoczy, K., et al. (2007), “Structure as a Quality Feature in Mathematics Instruction of the Learning Environment versus a
Structured Presentation of Learning Content”, in M. Prenzel (ed.), Studies on the Educational Quality of Schools: The Final
Report of the DFG Priority Programme, pp. 101-120, Waxmann, Munster.

Schwarzer, R. Schmitz, G.S. and Daytner, G.T. (1999), The Teacher Self-Efficacy Scale [On line publication http://web.fu-berlin.
de/gesund/skalen/Language_Selection/Turkish/Teacher_Self-Efficacy/teacher_self-efficacy.htm]

Shulman, L. (1987), “Knowledge and Teaching: Foundations of the New Reform”, Harvard Educational Review, No. 57 (1),
pp. 1-22.

Singer, E. (1996), “Espoused Teaching Paradigms of College Faculty”, Research in Higher Education, No. 37 (6), pp. 659-679.

OECD (2008a), Draft Analysis Plan for the OECD Teaching and Learning International Survey (TALIS): Main Study, OECD
committee paper EDU/INES/TALIS(2008)2, OECD, Paris.

OECD (2008b), “Improving school leadership policy and practice”, OECD, Paris.

OECD (2009), Creating Effective Teaching and Learning Environments: First Results from TALIS, OECD, Paris.

Quinn R.E., et al. 1996. Becoming a Master Manager: A Competency Framework (2nd ed.), Wiley & Sons, New York.

Scheerens, J. and R. Bosker (1997), The Foundations of Educational Effectiveness, Elsevier Science Ltd., Oxford.

Seidel, T. and J. Scheerens (in press), Towards a research review and meta-analysis on teaching and learning, Manuscript,
University of Twente.

UNESCO (2008), Technical Report of the World Education Indicators Survey of Primary Schools (forthcoming).

U.S. Department of Education (2004), National Center for Education Statistics, Schools and Staffing Survey 2003-04,
http://nces.ed.gov/surveys/sass/questionnaire.asp

Wheatley, K.F. (2005), “The Case for Reconceptualizing Teacher Efficacy Research”, Teaching and Teacher Education, No. 21,
pp. 747-766.

TALIS 2008 Technical Report © OECD 2010

Chapter 4

Translation and
Cultural Adaptation
46 Abstract
46 Overview
46 Instruments to be translated
47 Cultural and national adaptations
47 Translation procedures
48 Identifying the target language
49 Engaging translators
49 Producing translations
50 Submitting materials for external verification
51 International translation verification
51 Processes of translation verification
51 Translation verification report
51 Translation verification summary
52 Feedback from National Project Managers on translation
and translation verification
52 International Quality Control Monitor review of translation
52 Layout verification: paper and on line data collection

TALIS 2008 Technical Report © OECD 2010

46
chapter 4 Translation and Cultural Adaptation

Abstract

This chapter details the rigorous approach taken to ensure an accurate and appropriate
translation and cultural adaptation of the TALIS survey instruments. Each version of the TALIS
questionnaires was subject to a stringent translation procedure and layout verification process
prior to both the field trial and the main survey. The chapter explains the rationale for this
strict system and describes the responsibilities of the various parties involved in the processes,
including the IEA Secretariat, Instrument Development Expert Group, National Project Managers
and independent language experts, translators and linguistic verifiers.

Overview
The TALIS survey instruments were developed by the Instrument Development Expert Group (IDEG) in English
(see Chapter 3) and translated into French, the other working language of the OECD. Although countries were
free to choose their source language, all participating countries used the international English version as the
sole source for translation and adaptations, adhering to the procedures described in the TALIS Manual for
National Project Managers (MS-01-03). The detailed procedures helped ensure that the 31 national versions of
the instruments were as close as possible to the international original, while allowing for appropriate adaptations
to the national context.

Each version of the TALIS questionnaires was subject to a stringent independent translation and layout
verification process prior to both the field trial (FT) and the main survey (MS). Independent language experts
compared the translated instruments side by side with the international version. The verified instruments with
verifiers’ comments and suggestions were then returned to the National Project Managers (NPMs) for review
and improvement of the translation or adaptation. Questionnaires were then sent to the International Study
Centre (ISC) for layout verification, before they were finalised for data collection.

Instruments to be translated
The international French translations of the manuals and instruments were verified by independent experts
to ensure they were equivalent in meaning to the international English originals, according to the procedures
described later in this chapter. Study participants had the choice of using the international English or French
materials as source documents; all participants produced their translations using the international English
version of the materials. In two cases, participants used the translation produced by another country (introducing
necessary national adaptations) instead of translating from the international version.1

The following materials were required to be translated or adapted:

• Principal and Teacher Questionnaires (MS-11-01, MS-12-01; described in Chapter 3 of this report).
• Principal and Teacher Cover letters (MS-21-01, MS-22-01; only for countries collecting data on line (ODC).
• TALIS School Co-ordinator Manual (MS-03-02).

For both the FT and the MS national translations of the data collection instruments (questionnaires and cover
letters) were independently verified in a process co-ordinated by the IEA Secretariat. For the MS, Australia,
Austria, Ireland and Malta used English-language instruments. These were also submitted for verification:
although they were not translated, they were verified for the appropriateness of the adaptations to the national
context and for layout.

© OECD 2010 TALIS 2008 Technical Report

Translation and Cultural Adaptation chapter 4
47

Cultural and national adaptations

The objective of cultural and national adaptations is to adjust the data collection instruments so they are
appropriate for each participating country. Adaptations were kept to a minimum but in some cases they were
required. The purpose of national and cultural adaptations was to ensure that principals and teachers in each
country were responding to questions equivalent to those received by principals and teachers in all other
countries.

Each country was required to complete electronic National Adaptation Forms (NAF) for each language of survey
administration. The forms themselves provided detailed information on how to make required and optional
adaptations, and were to be completed in English.

To facilitate the adaptation process, the international (English and French) versions of the questionnaires were
highlighted yellow in several places where adaptations were always required. Required national adaptations
included the following:

• Items or information in carets < > on the international version of the questionnaires. Changes to such
information also needed to be described on the NAFs. For example, <ISCED Level> was replaced with the
national name of the level of education according to the International Standard Classification of Education
1997 (UNESCO-UIS, 2006).
• Information in square brackets [ ] indicated required country-specific changes that were not documented
on the NAF. This included provisions to comply with national ethical guidelines for studies of this nature, in
addition to return procedures, the name of the national centre and so on.
• National conventions such as date formats, punctuation and spelling.

For optional adaptations, the ISC required that a rationale be given and that the change be approved. For the
FT, the completed NAFs were sent directly to the IEA Secretariat, together with the translated and adapted
instruments, for translation verification (TV). For the MS, these forms were first submitted to the ISC for review, as
an additional check to ensure that adaptations fell within acceptable guidelines. When the national instruments
differed from the international original this had to be documented on the NAF (with the exception of square-
bracketed items, described above). When the ISC had verified the proposed adaptations, the approved NAFs
were sent together with the instruments to the IEA Secretariat for translation verification.

In the interests of international comparability, some restrictions were introduced defining what kinds of
adaptations to the international instruments were acceptable. Those considered acceptable included an
adaptation of <country specific> terms, an adaptation of valid ranges (if necessary), the removal of questions
or dimensions (only if not applicable) and the addition of questions,2 question parts, dimensions or categories
(only if absolutely necessary). Those adaptations not considered acceptable included the collapsing or removal
of international categories and the modification of the international question stem.

NAFs were updated to reflect any changes at each stage of the verification process: Version I was completed
for translation verification; and Version II was completed before layout verification. NPMs were also required
to send the final version of their NAFs (implementing all changes required during verification) to the ISC before
printing the final version of their instruments.

Translation procedures
The IEA Secretariat devised procedures to guide the translation process, as described in the TALIS Manual for
National Project Managers (MS-01-03). The procedures stipulated the qualifications required for translators and
reviewers who developed the national version of the instruments (described later in this section).

TALIS 2008 Technical Report © OECD 2010

48
chapter 4 Translation and Cultural Adaptation

The translation guidelines highlighted the importance of following the target language rules and the country or
cultural context, while ensuring that the translated text had the same meaning as the source text. This proviso
also applied when adapting from the English used in the international version to the forms of English used in a
different country or cultural context. These documents were designed to guide translators towards developing a set
of instruments that captured the meaning and intent of the international instruments, while safeguarding against
inaccuracies or word-for-word translations that were not appropriate in the national language and context.

For English-speaking countries, the process involved adapting language, terminology and classifications to
local requirements; for countries administering the survey in languages other than English, all materials were
translated and adapted into the local language(s).

The OECD Secretariat prepared and distributed a glossary with the most critical terms and an annotated version
of the questionnaires that clarified the intent behind the questions. It also sent copies to the independent
international translation verifiers contracted by the IEA Secretariat. These documents helped ensure that
translators and verifiers were interpreting the items in the way intended by the questionnaire developers.
Translators contacted the NPM for any clarification of items or intent, and NPMs forwarded these enquiries to
the ISC if they could not resolve them internally.

Translators were advised to take special care to ensure that the translations were linguistically appropriate in the
target language. This extra emphasis was considered important to enhance the credibility of the TALIS survey
among survey respondents.

As explained in earlier in the chapter, during translation, translators were instructed to document any changes
made to the original text in an electronic version of the NAF.

Identifying the target language

In the majority of countries participating in TALIS, one dominant language is used throughout the entire
educational system or is understood by all teachers and principals. This was the language chosen for the
survey. In some countries, educational systems are run autonomously according to region, with regions
potentially representing different language groups. An example is Belgium, which has three official languages.

Figure 4.1
Countries and languages participating in TALIS

Language/s TV rounds Language/s TV rounds Language/s TV rounds

Australia English 2 English 2 Portugal Portuguese 2
Ireland
English 1 Irish 2 Slovak Republic Slovakian 2
Austria German 2 Italy Italian 2 Slovenia Slovenian 2
Slovenian 1 Korea Korean 2 Basque 1
Belgium (Fl.) Dutch 2 Lithuania Lithuanian 2 Catalan 1
Bulgaria Bulgarian 2 Malaysia Bahasa Malaysia 2 Spain Galician 1
Brazil Portuguese 2 Malta English 2 Spanish (Castilian) 2
Denmark Danish 2 Mexico Spanish 2 Valencian 1
Estonia Estonian 2 Netherlands Dutch 2 Turkey Turkish 2
Hungary Hungarian 2 Norway Norwegian Bokmål 2
Iceland Icelandic 2 Poland Polish 2

Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Translation and Cultural Adaptation chapter 4
49

Only one educational system (or region) of Belgium, the Flemish, participated in TALIS. Therefore, the survey
was developed in Dutch only, since it is the language of instruction in that region and all participating teachers
and principals were fluent in it. In other countries such as Ireland, instruments were prepared in two languages
and the sampled teachers had the choice between the English and the Irish versions of the instruments.

The languages of survey administration are listed by country in Figure 4.1. Of the 24 participants, 3 countries
administered the survey in more than one language (from 2 to 5). All participating countries translated the
principal and teacher questionnaires into the languages in Figure 4.1, and participating ODC countries also
translated cover letters for ODC administration.3 Each set of instruments underwent two rounds of translation
verification for both the FT and the MS, and one round if translation verification was conducted only for the MS.
This was the case for English and Slovenian in Austria, since schools using these languages were sampled only
for the MS. Regarding Basque, Catalan, Galician and Valencian in Spain, the NPM decided to administer only
the MS in all of the official languages. Given the small number of sampled schools for the FT, it was considered
to be too time-consuming and costly to administer the FT in all five official languages. The FT was administered
only in Spanish (Castilian).

Engaging translators
NPMs engaged at least two translators for each language of the survey administration, both of whom were
native speakers in the language in which the survey was administered, had an excellent knowledge of English,
and had a familiarity with survey instruments.

The first of these translators was expected to be a language specialist with an excellent understanding of the
country’s cultural context. This person worked on translating the international English text of the instruments
and manuals into the national language. The second translator, known as the reviewer, was someone with
experience in the national educational context and who was familiar with the subject of the study. This person
reviewed and commented on the initial translation for appropriateness to the national educational context,
in addition to accuracy and readability. The NPM then reviewed the translation together with the reviewer’s
comments, and incorporated changes as appropriate into the final document. Using this method, three
independent people compared the translated document against the international English original.

Representatives from countries planning to divide the translation work or to prepare translations for more than
one language were reminded of the importance of ensuring consistency within and between documents. In the
latter case, they were encouraged to engage professionals familiar with all the languages as special reviewers
to make sure that the translations were equivalent.

Producing translations
Each country produced translations for both the FT and the MS.4 The bulk of the translating was done before
the FT, resulting in translated instruments for each country that had been reviewed externally on two separate
occasions. To assist the migration of translations from the FT to the MS, the OECD Secretariat prepared and
distributed to all participants a document that outlined all changes to the questionnaires for the MS. Furthermore,
diagnostic item statistics from the FT, identifying missing data and unrealistic or contradictory responses, were
used to help remedy mistranslated and difficult-to-translate items prior to the main data collection.

The translator received the following materials:

• a basic description of TALIS and a copy of the relevant chapter in the TALIS Manual for National Project
Managers (MS-01-03);
• international versions of the questionnaires and the instructions for them in electronic form;

TALIS 2008 Technical Report © OECD 2010

50
chapter 4 Translation and Cultural Adaptation

• copies of the international questionnaires with annotations by the OECD Secretariat that explained the
intention behind the questions;
• glossary of terms; and
• electronic version of the NAFs.

After the translation was completed, the NPM sent a copy of the materials described in the previous paragraph,
together with a copy of the translated questionnaires and instructions, to the reviewer. The reviewer used these
documents to compare the translation against the international English documents. The reviewer made any
required changes to the translations or NAFs, and these were returned to the NPM to arbitrate the final version.

The TALIS Manual for National Project Managers (MS-01-03) outlines guidelines for translation and cultural
adaptation, which are described below. These guidelines ensured that national translations were consistent with
the international versions of the TALIS instruments, while allowing for cultural adaptations where necessary.
Translators were advised to: find words and phrases in the target language that were equivalent to those in the
international version; ensure that the essential meaning of the text did not change; ensure that the translated
questionnaires asked the same questions as the international versions and that national adaptations were made
appropriately; and be mindful of possible changes in the instrument layout due to translation.

For the purposes of international comparison it was important that the questionnaires be equivalent (as far as
possible) across languages. The translated texts were meant to flow naturally so that it was not obvious that
the document originated in another language. Guidance on language usage for the purposes of translation as
outlined in the NPM Manual comprised the following:
• translations should have the same register (language level, degree of formality) as the source text;
• translated passages should employ correct grammar and usage (for example, subject-verb agreement,
prepositions, verb tenses);
• translated passages should neither clarify, omit nor add information;
• translated passages should employ equivalent qualifiers and modifiers, in the order appropriate for the target
language;
• idiomatic expressions should be translated appropriately, not necessarily word for word; and
• spelling, punctuation and capitalisation in the target text should be appropriate for the target language and
the country or cultural context.

Submitting materials for external verification

The TALIS instruments were subject to rigorous independent verification to ensure – as far as possible – that the
instruments used in each country asked the same questions of the same concepts, and thus were internationally
comparable. Two aspects of the instruments were verified, and NPMs submitted NAFs and translated instruments
by email for both translation and cultural adaptations. The latter were also requested for those countries that
administered the survey in English.

For the FT, the translation verifier alone reviewed the NAFs prior to translation verification. For the MS, the ISC
reviewed the NAFs prior to translation verification, as it required ISC approval before passing to the next stage.
This process was designed to eliminate any unacceptable adaptations to the instruments.

During the final stage, verifying layout before printing paper instruments, staff at the ISC compared the layout and
formatting of the national instruments against the international source documents. For countries participating in
ODC, staff at the ISC reviewed the finalised paper instruments against the on line version of the questionnaires,
to ensure the two were isomorphic. These procedures are described in more detail later in the chapter.

© OECD 2010 TALIS 2008 Technical Report

Translation and Cultural Adaptation chapter 4
51

International translation verification

The IEA Secretariat co-ordinated the translation verification, engaging the services of native-speaking linguistic
verifiers through cApStAn Linguistic Quality Control, based in Brussels, Belgium. These verifiers were
experienced in balancing cultural and national “appropriateness” of the target version with “faithfulness”
to the source version. Verifiers gave expert feedback on the translations and adaptations. The IEA Secretariat
recommended that NPMs carefully consider all verifier recommendations and take care that the original
meaning of the phrases was retained in their translation; however, it was emphasised that the final decision
regarding document content rested with the NPM. NPMs were asked to explain major differences of opinions
between themselves and verifiers (see section below).

Processes of translation verification

Translation verifiers received the international (English or French) questionnaires in PDF format, which gave
them an accurate preview of the intended format. The translated questionnaires and NAFs were received as
Microsoft® Word files. Verifiers inserted their comments and changes directly into the translations using the
“track changes” feature. The verified documents complete with annotations and comments became known as
the Translation Verification Report (TVR).

Verifiers were instructed to i) check the accuracy and comparability of the translations of the instruments,
ensuring that the translation had not affected the meaning or difficulty of the text, the questions were not
made easier or more difficult when translated and no information was omitted or added in the translated
text; ii) document ALL deviations in the participating country’s translation, including additions, deletions and
mistranslations, according to specific guidelines; and iii) suggest alternative translations, if necessary, that
would improve the comparability.

Translation verification report

Verifiers returned the TVRs to the IEA Secretariat, which then forwarded them to NPMs. Comments were
assigned codes to indicate the severity of the error identified, ranging from Code 1, indicating a major change
or error that must be addressed, to Code 4 indicating a change that was acceptable. Verifiers also noted whether
changes had been appropriately documented on the NAF. Codes were categorised as follows:

• Major Change or Error: Examples included incorrect order of choices in a multiple-choice question;
omission of a question; incorrect translation resulting in the answer being suggested by the question; an
incorrect translation which changed the meaning or difficulty of the question; incorrect order of questions.
• Minor Change or Error: Examples include spelling errors that did not affect comprehension.
• Suggestion for Alternative: The translation might be adequate, but the verifier suggested different wording.
• Acceptable Change: Change is acceptable and appropriate. An example would be capitalisation or date
format as used in the language of translation.

In order to draw the attention of NPMs to unknown or unclassifiable irregularities, verifiers used “Code 1?”
when they were unsure of which code to use. Code 1 errors required further follow-up with the IEA Secretariat
(see next section).

Translation verification summary

As part of the process of translation verification, NPMs were asked to record and respond to Code 1 (or “Code
1?”) verifier suggestions in a separate document, titled “TV Summary”. Sixteen of twenty-four participating
countries submitted their document and provided the IEA Secretariat with further information about the nature

TALIS 2008 Technical Report © OECD 2010

52
chapter 4 Translation and Cultural Adaptation

of interventions flagged as serious by the verifier. Of the eight countries that did not submit the form, two had no
errors marked Code 1 and in another two the IQCM indicated on the TVR that all verifier suggestions had been
implemented. Whenever possible, this information was sent to the ISC before instrument finalisation. The IEA
Secretariat asked NPMs to justify any Code 1 interventions that were not implemented – they were not obliged
to accept the verifiers’ suggestions, but they were required to explain important points of difference. The IEA
Secretariat also forwarded this feedback to verifiers, both for the FT and MS, in a process that both verifiers and
NPMs described as useful.

Feedback from National Project Managers on translation and translation

verification
This section gives a brief overview of NPM feedback regarding translation and translation verification. It is based
on an analysis of the on line Survey Activities Questionnaire (SAQ) and outlined in more detail in Chapter 7.

NPMs were asked to complete the SAQ, describing their experience with TALIS. The majority (67%) reported it
was “not difficult at all” to translate and adapt the paper questionnaires to the national language; however one
participant reported the process was “very difficult”. Six participants found the process “somewhat difficult”.
The most frequently reported problem was the difficulty of preparing an accurate translation that precisely
conveyed the meaning of the source text and yet read naturally and fluently in the target language. This was
especially true for items that did not fit well with the country’s national context – e.g. concerning principals’
authority to hire and fire teachers – for which NPMs had to take special care not to introduce misunderstandings.
Of documenting national adaptations, 92% reported it was “not difficult at all”. However, several commented
that the process was very time-consuming.

In rating the usefulness of the translation verification process, 92% reported it was “very useful”, with no
participants reporting it was “not useful at all”. Furthermore, all respondents were able to make full use of the
feedback from translation verifiers when preparing the final version of their instruments. Countries that reported
some difficulties with translation mentioned that they resolved these problems with advice from the OECD
Secretariat, before translation verification began.

International Quality Control Monitor review of translation

The TVR for each country was sent to the relevant IQCM, who had the task of comparing the TVR against the
final version of the data collection instruments. The IQCM marked the verifier comment on the TVR with “yes”
if the verifier suggestion was implemented, and “no” if it was not. The IEA Secretariat retained this annotated
TVR, together with a copy of the final version of instruments used in schools, for future reference in the event
of unusual item characteristics showing in the data. According to these data, across all 31 sets of instruments,
there were only two unchanged or unexplained Code 1 errors in the final version of the instruments.

Layout verification: paper and on line data collection

The ISC performed layout verification both for paper and for ODC instruments. After translation verification had
been finalised, NPMs submitted their questionnaires and cover letters (if ODC was used) together with the latest
version of the NAF to the ISC.

The ISC performed a careful check of the national versions of the instruments against the international English
version and the NAF. The aim of layout verification was to ensure that the national versions of the TALIS
instruments looked as much as possible like the international source version.

The paper instruments were verified and administered in a total of 31 languages. ISC staff checked each
questionnaire for font size, font changes, and adjustment of cells, response options, blank pages, word emphasis,

© OECD 2010 TALIS 2008 Technical Report

Translation and Cultural Adaptation chapter 4
53

track changes and comments. All deviations from the source version were listed in a standardised document
and sent back to the NPMs for review. Another staff member at the ISC then verified the revised version of the
instruments. This procedure was repeated until the instruments looked as much as possible like the international
source. For the majority of languages two to four rounds were needed before the ISC approved the layout of
the instruments.

In a few rare cases NPMs detected some minor inconsistencies regarding spelling or punctuation after layout
approval and prior to printing. The NPMS changed these inconsistencies and sent the updated version to the
ISC for documentation. However these instruments were checked again to ensure that the overall layout had
not been affected.

The ODC versions of the instruments were checked for 16 participating countries in a total of 17 languages.
ODC instruments were checked against the national paper version after paper layout verification. This was to
ensure that the instruments within one country were the same regardless of whether they were administered
on paper or on line.

Visual checks were run using the same standards and procedures as for paper layout verification. For most of
the languages up to two rounds were needed to finally approve the ODC instruments. Additionally, the ISC
performed technical ODC load checks and load testing to ensure smooth operations. For more details on ODC
upload, activation and shut down as managed by the ISC, see Chapter 7.

Notes

1. Mexico joined the study late and adapted the Spanish instruments from Spain for use in the FT; for the MS, Mexico produced its
own translations. Austria produced its own German translations, and adapted the international materials for English-speaking schools;
however for Slovenian (a minority language in Austria) the instruments used in the MS were adapted from the translation produced
by the TALIS national team in Slovenia.

2. Additional questions for the questionnaires could be placed after all the international questions. However, guidelines stipulated
that they should be few in number in order to keep the time it would take respondents to complete the questionnaire to a minimum.

3. The exception was Austria, which prepared ODC cover letters in German only. This is because English and Slovenian are both
minority languages, administered in only a small number of Austrian schools.

4. Bulgaria joined the study after the FT was complete and the international instruments for the MS had been released. Therefore,
Bulgaria submitted the MS instruments for verification and used them in a single-country FT. The Bulgarian instruments were further
refined after the FT, and these revised instruments were submitted for verification again prior to the main data collection.

TALIS 2008 Technical Report © OECD 2010

54
chapter 4 Translation and Cultural Adaptation

References
IEA (2007), TALIS Manual for National Project Managers (MS-01-03), prepared by the IEA DPC, Hamburg.

IEA (2007), TALIS Translation Verification Guidelines, prepared by the IEA Secretariat, Amsterdam.

UNESCO Institute for Statistics (2006), Operational manual for ISCED-1997: International standard classification of education,
Re-edition.

© OECD 2010 TALIS 2008 Technical Report

Chapter 5

Sample Design

56 Abstract
56 Overview
56 International sampling plan
56 Target population and survey population:
International requirements and national
implementations
58 Sample size requirements
58 National sampling strategies
59 Sampling frames
59 Stratification
59 Sample selection
60 Sampling for the field trial
60 National sampling plans

TALIS 2008 Technical Report © OECD 2010

56
chapter 5 Sample Design

Abstract

This chapter offers an overview of the international sampling plan prepared for the participants
in TALIS, including the international sampling strategy and sample size. Appendix B presents
the characteristics of each national sampling plan. Strategies for estimating population
characteristics and their sampling error are covered in detail in Chapter 9. This chapter deals
primarily with the TALIS “core survey” of ISCED Level 2 teachers.

Overview
This chapter concerns only the TALIS “core survey”, that is, the survey of ISCED Level 2 teachers. Participating
countries were offered the option of linking their TALIS sample to that of PISA 2006 but none chose to pursue
this. Participating countries could also opt to survey ISCED Level 1 and Level 3 teachers. Only Iceland chose to
cover ISCED Level 1 teachers and none chose to survey ISCED Level 3.

A more detailed description of the survey design and its recommended implementation can be found in the
TALIS Sampling Manual (MS-02-03).

International sampling plan

The international sampling plan prepared for the TALIS core survey is a stratified two-stage probability sampling
design (Lohr, 1999). This means that teachers (second stage units or secondary sampling units – SSU) were
randomly selected from the list of in-scope teachers in each of the randomly selected schools (first stage units,
or primary sampling units – PSU).

The universes of interest comprised schools where ISCED Level 2 education is provided, along with the affiliated
principals and teachers. Following the Indicators of Education Systems (INES) data collection definitions, “the
formal definition of a classroom teacher is a person whose professional activity involves the planning, organising
and conducting of group activities whereby students’ knowledge, skills and attitudes develop as stipulated by
educational programmes. In short, it is one whose main activity is teaching.” (OECD, 2004).

Target population and survey population: International requirements

and national implementations
TALIS was designed to cover all ISCED Level 2 teachers in a participating country. TALIS identified policy
issues that encompass the classroom, the teacher, the school and school management. No subject matter was
excluded from the scope of TALIS. Thus, coverage of TALIS extends to all teachers of ISCED Level 2 and to the
principals of the schools where they teach.

An ISCED Level 2 teacher is one who, as part of his or her regular duties in school, provides instruction in
programmes at ISCED Level 2. Teachers who teach a mixture of programmes at different levels including ISCED
Level 2 programmes in the target school are included in the TALIS universe. There is no minimum cut-off for
how much ISCED Level 2 teaching these teachers need to be engaged in.

The international target population of TALIS restricts the survey to those teachers who teach regular classes in
ordinary schools and to the principals of those schools. Teachers teaching to adults and teachers working with
children with special needs are not part of the international target population and are deemed “out of scope”.

© OECD 2010 TALIS 2008 Technical Report

Sample Design chapter 5
57

When schools are comprised exclusively of these teachers, the school itself is said to be “out of scope”. Teacher
aides, pedagogical support staff (e.g. guidance counsellors, librarians) and health and social support staff
(e.g. doctors, nurses, psychiatrists, psychologists, occupational therapists and social workers) were not
considered as teachers and thus not part of the TALIS international target population.

For national reasons, some participating countries chose to restrict the coverage of their national implementation
of TALIS to parts of the country. For example, a province or state experiencing civil unrest or an area struck by a
natural disaster could be removed from the international target population to create a national target population.
Participating countries were invited to keep these exclusions to a minimum (see MS-02-03, paragraph 38).

Ideally, all the members of the target population ought to be eligible for sampling and data collection. This
is the option that TALIS chose and, as a consequence, the international survey population (those who can be
surveyed) is identical to the international target population (those who should be surveyed).

TALIS recognised that attempting to survey teachers in very small schools, those in schools with no more than
three teachers at ISCED Level 2, and those teaching in schools located in geographically remote areas could be
a costly, time-consuming and statistically inefficient exercise. Therefore, participating countries were allowed
to exclude those teachers for TALIS data collection, thus creating a national survey population different from
the national target population. The NPM was required to document the reasons for exclusion, the size, the
location, the clientele and so on for each excluded school. Moreover, as discussed later in this section, during
data collection in the selected schools, some teachers could be excused from data collection.

Ultimately, samples of schools and teachers were selected from the national survey population.

Figure 5.1 illustrates how these concepts relate to one another.

Figure 5.1
TALIS international target and survey populations

TALIS out of scope TALIS International target population = TALIS International survey population

National out of scope National target population

Adult education,
special needs Entire province, state, National exclusions National survey population
sub-population
Remote, small schools, etc. Not sampled In sample

Source: OECD, TALIS Database.

Table 5.1 describes how the survey population is defined with respect to the target population (details of
how these relate to the TALIS international target population are given in Appendix B); the information
was provided by the NPMs using the Sampling Forms (templates for each Sampling Form can be found in
Appendix C).

Within a selected in-scope school, the following teachers were to be excluded:

• teachers teaching only to special needs students (out of scope);

• teachers who also act as principals: no teacher data collected, but principal data collected (labelled as
NEXCL5 in Chapter 10);

TALIS 2008 Technical Report © OECD 2010

58
chapter 5 Sample Design

• substitute, emergency or occasional teachers (out of scope);

• teachers on long-term leave: out of scope;
• teachers teaching exclusively to adults (out of scope); and
• in Malta and Iceland, teachers who had taken part in the TALIS 2007 Field Trial: no teacher data collected
and labelled as NEXCL6 in Chapter 10.

Detailed guidelines concerning the application of those categories of exclusion were given to the NPMs in the
Sampling Manual or in separate correspondence between Statistics Canada, the ISC and the interested countries.
Moreover, the School Co-ordinator Manual provided operational advice to those who had to assign codes.

Teachers who taught at more than one school were not excluded. Rather, the number of schools in which they
taught was recorded (see weight adjustments in Chapter 10).

Sample size requirements

To allow for reliable estimation and modelling, while allowing for some amount of non-response, the minimum
sample size was set at 20 teachers within each participating school. A minimum sample of 200 schools was to
be drawn from the population of in-scope schools. Thus, the nominal international sample size was a minimum
of 4 000 teachers. Teachers teaching in the same school might tend to share opinions and behave in similar
ways, more so than teachers from different schools, cities or provinces in a given country. This tendency for
two teachers from the same school to be “more alike” than two teachers from different schools is called a
“clustering effect” and is often measured by the “intracluster correlation coefficient”. In essence, the stronger
the intracluster correlation, the fewer sampled teachers one needs from one school, as one responding teacher
becomes a good predictor of the other teachers of his school. In other words, in a sample of 20 teachers from
the same school, there are, in a sense, fewer than 20 original data points. This also is a manifestation of the
clustering effect or design effect, and the larger the cluster, the larger the loss of information. In preparing
TALIS, the working hypothesis, based on previous studies of student achievement, was to use an intracluster
correlation coefficient of 0.30, supposing that teachers are as homogeneous as their students. The loss in sample
size due to clustering, added to the losses due to non-response, reduces the nominal sample of 4 000 teachers
to an effective sample of approximately 400 as depicted in Table 5.2.

Thus, the nominal sample of 4 000 teachers obtained by the complex sampling design is equivalent to a simple
random sample of 433 teachers. The precision that is expected from the sample of 20 teachers in 200 schools is
equivalent to that of a simple random sample of 433 teachers selected from the (often unavailable) national list of
teachers. The expected margin of error for a simple random sample of this size is ± (1.96) × (1/ 433) = ± 9.4%.

Participating countries could choose to augment their national sample by selecting more schools, or by selecting
more teachers within each selected school, or by increasing both. Some countries were asked to increase the
within-school sample to counterbalance the effect of selecting too many schools with fewer than 20 teachers.

The sample size requirement was reduced for some participating countries because of the smaller number of
schools available for sampling (see Appendix B). In a few cases, because the average number of teachers in the
schools was less than expected in the international plan, the number of schools to be sampled was increased to
maintain a minimum total number of participating teachers.

National sampling strategies

Participating countries could suggest variations or adaptations of the international sampling plan to better suit their
national needs. The TALIS sampling team reviewed and approved all changes to the international sampling plan.

© OECD 2010 TALIS 2008 Technical Report

Sample Design chapter 5
59

Sampling frames
Participating countries provided Statistics Canada with a current and complete list of schools providing
education at ISCED Level 2. This list constituted the school sampling frame for TALIS and was expected to
correspond to the survey population as defined and described on the Sampling Forms.

The sampling frame had to contain certain key variables: a national school identifier, a measure of size (MOS),
preferably the number of ISCED Level 2 teachers, and values for those variables to be used for stratification;
whenever possible, the type of funding (private or public) and the type of education stream (academic or
vocational) were also to appear on the frame.

Additional sampling frames were required for the sampling of teachers, namely, the list of eligible ISCED
Level 2 teachers in each selected school.

Stratification
The international sampling plan did not anticipate any stratification of the schools nor of the teachers within the
selected schools. Participating countries that chose to implement some form of stratification to answer national
requirements were invited to discuss their strategy with the TALIS sampling team.

Stratification could be done explicitly (whereby a fixed portion of the total sample is allocated to the stratum) or
implicitly (whereby the stratification variable is used to sort the sampling frame prior to sample selection thus
giving on average a proportional representation of the implicit strata in the sample).

When explicit stratification was used, the participating country and the TALIS sampling team agreed to a sample
allocation scheme.

In most cases, stratification resulted in a combination of some or all of geography, source of financing, type of
educational programme and size of schools. Appendix B gives details for each participating country.

Sample selection
Samples of schools were selected by systematic random sampling with probability proportional to size (PPS)
within explicit strata, according to the national sampling plans. When implicit stratification was used, schools
in explicit strata were sorted by implicit strata and MOS prior to sampling. Sampling frames were always sorted
by MOS prior to sampling, whether stratification was applied or not. Sorting by MOS was done in a serpentine
manner, alternating increasing order and decreasing order so that adjacent schools would be of similar sizes even
across strata. This is useful when creating replication zones for estimation of sampling error (see Chapter 10).

Systematic random sampling with PPS can be described as follows. Let M be the total MOS in an explicit stratum,
let mi be the MOS for school i in the explicit stratum and Mi be the cumulative sum of the school sizes up to and
including school i, and let n be the number of schools to be sampled from that explicit stratum. Then, a sampling
step k is computed as the integer part of M ÷ n. A random starting point d is drawn at random from the interval
[1, …, k]. The sample is selected by taking steps of fixed length k along the (ordered) sampling frame. Where the
step lands points to the school to be added to the sample. The procedure is illustrated in Table 5.3.

Whenever possible, two replacement schools were assigned for each sampled school: the school just above and
the school just below the selected school on the sampling frame sorted by implicit strata (where needed) and
MOS. The replacement schools had to come from the same explicit stratum as the sampled school. This strategy
was expected to help maintain the sample size and minimise non-response biases by replacing originally
sampled non-responding schools with schools having similar characteristics. Schools selected for the original
sample could not also be selected as replacement schools.

TALIS 2008 Technical Report © OECD 2010

60
chapter 5 Sample Design

Although participating countries were given the option of selecting the school sample themselves, in the event
the TALIS sampling team performed this task.

At the end of school selection, participating countries were returned a copy of their school sampling frame
where the selected schools were identified (marked “S” for the original sample, marked “R1” and “R2” for the
replacement schools) and given a standardised TALIS school identification number.

Table 5.3 illustrates how systematic random sampling with PPS may be implemented using an ordinary
spreadsheet. In this illustration, explicit stratum “A” is comprised of 12 schools and a sample of n = 3 schools is
needed from this stratum; the sampling step k = [209 ÷ 3] = 69.7 and suppose that the random start is d = 49;
then the j th school selected is such that Mj-1 < d+ (j – 1) × k Mj , with M0 = 0 and j = 1, 2, 3. Here, for the first
selection, j = 1 and the pointer is 49 + (1 – 1) × 69.7 = 49; if j = 2, the pointer is at 49 + (2 – 1) × 69.7 = 118.7
(rounded down to 118), and finally the pointer is at 118.7 + 69.7 = 188.4 (rounded down to 188). Replacement
schools are selected automatically as the schools immediately before and after a selected school, if available;
note that school 12 has no second replacement.

IEA Data Processing and Research Center (IEA DPC) provided each participating country with Windows Within-
School Sampling Software (WinW3S) to help in the creation of sampling frames and selection of teachers,
ensuring compliance with the sample design as well as complete documentation.

As a series of questions within the teacher questionnaire were concerned with events within the classroom
setting, the final sampling step was the selection of a reference class or course for each selected teacher. This
happened as the teachers were filling in their questionnaires: they were asked to identify the first ISCED Level 2
class or course they typically taught after 11:00 a.m. on Tuesdays.

The sizes of the school and teacher samples for each participating country are listed in Appendix B.

Sampling for the field trial

Prior to the main data collection, each participating country conducted a field trial (FT) during March and April
2007. For that purpose, a sample of twenty schools (plus their one replacement1) was selected during sample
selection for the main survey (MS). The simultaneous selection of the school samples for the FT and the MS
allowed some control of the overlap between the two samples and helped in reducing response burden on
participating schools. When the number of schools in an explicit stratum was such that overlap of FT and MS
samples was unavoidable, teachers who took part in the FT could be excused from the MS (see Chapter 10 on
weighting).

National sampling plans

Table 5.4 gives an overview of the sampling plan for each participating country. More details are given in the
country reports found in Appendix B.

Note
1. Only one replacement school was selected for the Field Trial to minimise the overlap with the sample for the Main Survey.

© OECD 2010 TALIS 2008 Technical Report

Sample Design chapter 5
61

Table 5.1 (1/2)

Reasons for and magnitude of school exclusion, by country
Reasons for exclusion Schools % Teachers %
Australia Target Population 2 617 100.0 100.0
Non mainstream schools, non English language schools, distance,
110 4.2 N/A
adult education
Survey Population 2 507 95.8
Austria Target Population 1 540 100.0
N/A
Survey Population 1 540 100.0
Belgium (Fl.) Target Population 675 100.0 22 130 100.0
Survey Population 675 100.0 22 130 100.0
Brazil Target Population 57 479 100.0 843 951 100.0
Small schools (fewer than 4 ISCED Level 2 teachers) 4 636 10 124
8.1 1.4
Federal schools 34 1 683
Survey Population 52 809 91.9 832 144 98.6
Bulgaria Target Population 2 408 100.0 30 782 100.0
Small schools (fewer than 4 ISCED Level 2 teachers) 108 4.5 282 0.9
Survey Population 2 300 95.5 30 500 99.1
Denmark Target Population 2 176 100.0 60 905 100.0
Small schools (fewer than 6 ISCED Level 2 teachers ) 70 274
Public Youth Schools (Ungdomsskoler) 111 9.6 300 >0.9
No measurement of size available 29 N/A
Survey Population 1 966 90.4 60 331
Estonia Target Population 448 100.0 8 747 100.0
Remote schools 5 27
Small schools (fewer than 7 ISCED Level 2 teachers ) 12 7.4 65 5.7
Bilingual schools (15 Estonian / Russian – 1 Estonian / Finnish) 16 410
Survey Population 415 92.6 8 245 94.3
Hungary Target Population 2 897 100.0 46 594 100.0
Small schools (fewer than 4 ISCED Level 2 teachers) 45 1.6 103 0.2
Survey Population 2 852 98.4 46 491 99.8
Ireland Target Population 702 100.0
N/A
Survey Population 702 100.0
Iceland Target Population 152 100.0 2 537 100.0
Survey Population 152 100.0 2 537 100.0
Italy Target Population 7 894 100.0 191 346 100.0
Small schools (fewer than 4 ISCED Level 2 teachers) 262 604
Remote schools 31 233
4.9 1.0
Schools attached to art academies 71 919
Private schools outside national education system 21 244
Survey Population 7 509 95.1 189 346
Korea Target Population 2987 100.0 103 877 100.0
Survey Population 2987 100.0 103 877 100.0
Lithuania Target Population 1 296 100.0 47 382 100.0
Survey Population 1 296 100.0 47 382 100.0
Mexico Target Population 15 220 100.0 286 379 100.0
CONAFE 918 1 050
Small schools (fewer than 4 ISCED Level 2 teachers) 87 159
6.8 0.9
Field Trial 23 758
Others 8 506
Survey Population 14 184 93.2 283 906 99.1
Malta Target Population 64 100.0 3 013 100.0
Schools not following mainstream curriculum 3 88
6.3 2.9
Small schools (fewer than 4 ISCED Level 2 teachers) 1 1
Survey Population 60 93.7 2 924 97.1
Malaysia Target Population 2 361 100.0 100.0
Language, curriculum 106
Small schools (less than 100 ISCED Level 2 students) 109 9.2 N/A
Remote schools 2
Survey Population 2 144 90.8
Note: “N / A” appears when the country did not or could not provide the information; in such cases, the corresponding proportions could not be computed and are left blank.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

62
chapter 5 Sample Design

Table 5.1 (2/2)

Reasons for and magnitude of school exclusion, by country
Reasons for exclusion Schools % Teachers %
Netherlands Target Population 587 100.0 100.0
Vocational schools 42 7.2 N/A
Survey Population 545 92.8
Norway Target Population 1 212 100.0 21 898 100.0
Schools outside Norwegian school regulation 4 10.1 69 1.5
Schools abroad 14 48
Small schools (fewer than 10 students or fewer than 3 teachers) 104 211
Survey Population 1 090 89.9 21 570 98.5
Poland Target Population 6 218 100.0 139 290 100.0
Small schools (fewer than 4 ISCED Level 2 teachers) 908 14.6 1 816 (est.) 1.3
Survey Population 5 310 85.4 137 474 98.7
Portugal Target Population 1 307 100.0 41 807 100.0
Survey Population 1 307 100.0 41 807 100.0
Slovak Republic Target Population 1 655 100.0 28 182 100.0
Small schools (fewer than 4 ISCED Level 2 teachers) 21 2.1 40 0.6
Language other than Slovak or Hungarian 14 132
Survey Population 1 620 97.8 28 010 99.4
Slovenia Target Population 446 100.0 9 450 100.0
Survey Population 446 100.0 9 450 100.0
Spain Target Population 7 106 100.0 235 060 100.0
(excluding Rioja Survey Population 7 106 100.0 235 060 100.0
and Canarias)
Turkey Target Population 16 315 100.0 157 635 100.0
Small schools (fewer than 4 ISCED Level 2 teachers) 3 838 23.5 8 648 5.5
Survey Population 12 477 76.5 148 987 94.5
Note: “N / A” appears when the country did not or could not provide the information; in such cases, the corresponding proportions could not be computed and are left blank.
Source: OECD, TALIS Database.

Table 5.2
Derivation of the required sample size
Schools a 200
Teachers per school b 20
Total number of teachers c=a×b 4 000
School response rate d 75%
Teacher response within school e 75%
Overall response rate f=d×e 56%
Net number of responding teachers g=c×f 2 250
Intra-cluster correlation h 0.30
Design effect (deff) deff = 1 + {(e ×b) –1} × h 5.2
Effective sample = g / deff 433
Source: OECD, TALIS Database.

Table 5.3
Illustration of systematic random sampling with PPS
Measurement Cumulative Selections and
National school id Explicit stratum Implicit stratum of size Mi measurement of size Mi Sampling steps replacements
1 A 1 10 10
2 A 1 12 22
3 A 1 15 37 R1
4 A 1 17 54 49 S
5 A 2 20 74 R2
6 A 2 18 92
7 A 2 16 108 R1
8 A 2 16 124 118 S
9 A 3 15 139 R2
10 A 3 17 156
11 A 3 26 182 R1
12 A 3 27 M = 209 188 S
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Sample Design chapter 5
63

Table 5.4
Overview of the national sampling plans
Number of ISCED Number of ISCED Teacher sample
Explicit stratification Level 2 schools Level 2 teachers School sample size (expected size)
Australia Geography (8) 2 507 * 200 4 000
Austria School type (3) 1 540 * 279 5 580
Belgium (Fl.) Network (3) 675 22 130 260 5 200
Brazil School size (3) × School type (3) 52 809 832 144 400 7 161
Bulgaria School size (4) × School type (3) 2 300 30 500 203 4 133
Denmark School type (3) 1 966 60 331 200 4 000
Estonia Region (2) × School type (2) 415 8 245 200 3 316
Hungary School size (4) 2 852 46 491 200 3 618
Ireland School size (3) 702 * 200 4 000
Iceland None 152 2 537 152 2 537
Italy Geography (3) 7 509 189 346 300 6 000
Korea None 2 987 103 877 200 4 000
Lithuania School type (4) 1 296 47 382 220 4 400
Mexico School size (4) × School type (3) 14 184 283 906 200 4 164
Malta None 60 2 924 60 1 200
Malaysia School type (3) 2 144 * 219 4 380
Netherlands School type (4) 545 * 150 3 000
Norway School size (4) × Density (2) 1 090 21 570 200 4 875
Poland Density (3) × Funding (2) 5 310 137 474 200 4 000
Portugal Funding (2) × Region (5) 1 307 41 807 200 4 000
Slovak Republic School type (2) 1 620 28 010 200 4 000
Slovenia None 446 9 450 200 4 000
Spain Group of autonomous 7 106 235 060 200 4 000
(excluding Rioja communities (2)
and Canarias)
Turkey School size (4) 12 477 148 987 200 4 105
Note: “*” appears when the size of the ISCED Level 2 teacher population is unknown.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

64
chapter 5 Sample Design

References
Cochran, W.G. (1977), Sampling Techniques, 3rd edition, Wiley, New York.

Lohr, S. (1999), Sampling: Design and Analysis, Duxbury Press, New York.

OECD (2004), OECD Handbook for Internationally Comparative Education Statistics: Concepts, Standards, Definitions, and
Classifications, OECD, Paris.

Statistics Canada (2003), TALIS Sampling Manual (MS-02-03), Ottawa.

Statistics Canada (2003), Survey Methods and Practices, Catalogue Number 12-587-XPE, Ottawa.

© OECD 2010 TALIS 2008 Technical Report

Chapter 6

Survey Operations
Procedures
66 Abstract
66 Manuals and software
67 Contacting schools and within-school sampling procedures
67 Identification numbers, Teacher Listing Forms and
Teacher Tracking Forms
68 Assigning materials to teachers and school principals
69 Administering the questionnaires and national quality
control
69 Monitoring the on line questionnaires
69 Material receipt and preparing for data entry
69 Survey Activities Questionnaire

TALIS 2008 Technical Report © OECD 2010

66
chapter 6 Survey Operations Procedures

Abstract

This chapter focuses on the survey operation procedures implemented for TALIS, including
the materials and software that the International Study Centre (ISC) provided to all National
Project Managers (NPMs). Other tasks of the NPMs as well as the three phases of the survey
were explained in more detail in Chapter 1of this report. The International Association for the
Evaluation of Educational Achievement Data Processing Center (IEA DPC) had developed a
series software packages for previous IEA surveys, and adapted them to fit the needs of TALIS.
The chapter concludes with an explanation of the quality control checks at different levels
which ensure the high quality of the TALIS data.

Manuals and software

During all phases of the survey NPMs followed the standardised procedures prepared by the ISC and its
consortium partners. This section lists only the latest versions of the six manuals and three software packages
that were used for the main survey (MS). The ISC provided the following manuals, each in English and French.1
• The TALIS Manual for National Project Managers (MS-01-03) guided NPMs through all steps of the survey
from the production of the national instruments to the submission of data to the ISC. The manual also
included information on how to raise participation rates and how to manage confidentiality concerns.
• Statistics Canada prepared the TALIS Sampling Manual (MS-02-03), which defined the target population of
ISCED Level 2 teachers. The manual described how to establish a national sampling plan, how to prepare
the school sampling frame and how to select the school sample.
• The TALIS School Co-ordinator Manual (MS-03-02) addressed the school co-ordinator (SC) who played a
key role within the school. The manual described in detail the steps for listing and tracking teachers and for
organising the survey administration on site. NPMs were responsible for translating the manual into their
survey administration language(s) and for adding national information where necessary. Responsibility for
translations and adaptations rested solely with the NPMs.
• The TALIS Data Management Manual (MS-04-02) provided the national data manager with instructions on
how to use the software for collecting, capturing and verifying the data. The ISC held a three-day training
seminar prior to the field test, giving data managers additional skills in using the software.
• The IEA Secretariat prepared the TALIS Manual for International Quality Control Monitors (MS-05-01) and
delivered it directly to the International Quality Control Monitors (IQCM) contracted by the IEA. The manual
outlined the tasks to be undertaken by the IQCMs in order to check the quality of survey operation procedures
within participating countries. IQCMs visited NPMs and schools to document the outcomes of the visits.
• The TALIS Manual for National Quality Control Monitors (MS-06-01) guided NPMs in how to conduct a
national quality control programme. The procedures were closely related to those for the IQCMs. However,
NPMs were free to adapt the manual according to their needs.

Additionally, the ISC supplied NPMs with three software packages to assist with data collection:
• The Windows Within-School Sampling Software (WinW3S) aided national data managers in preparing the
survey listing forms, qualifying and randomly sampling teachers in selected schools, and producing tracking
forms for the sampled individuals. The software stored all tracking information in a single database so that the
information could later be used to verify the integrity of the sampling procedures, to verify the completeness
of the response data and eventually to compute sampling weights and participation rates.

© OECD 2010 TALIS 2008 Technical Report

Survey Operations Procedures chapter 6
67

• The Windows Data Entry Manager (WinDEM) enabled national centre staff to capture the data through
keyboard data entry and to perform a range of validity checks on the entered data. The WinDEM databases
included codebooks for each of the questionnaires, providing all the information necessary for producing
data files for each instrument in a standard international format (see Chapter 9).

• The IEA SurveySystem converted paper questionnaires’ text passages for on line administration (see Chapter 7)
and delivered these to respondents via the Internet. National centres performed the conversion. The on line
questionnaires were then sent to the ISC for technical checks and layout verification.

Contacting schools and within-school sampling procedures

Statistics Canada sent all NPMs a selected school sample based on the sampling frame the NPM had already
submitted.2 In order to achieve the highest possible participation rates at school level, two replacement schools
were sampled in addition to each originally sampled school. The TALIS Manual for National Project Managers
(MS-0-03) Appendix 8.5 gave detailed instructions about how to secure high participation rates within schools.
These suggestions were based on the experiences of NPMs during the field trial.

Once NPMs received the sample, national centres started contacting the designated schools to secure their
participation. If one of the sampled schools declined participation the national centre contacted its first
replacement school. If this school also refused participation NPMs approached a second replacement school.

Each school nominated an SC to be responsible for carrying out all TALIS-related tasks within the school.
Due to confidentiality concerns, it was preferable that the SC be a person other than the principal. Since SCs
played a key role within the survey in almost half the participating countries, the NPMs provided them with
formal training.

Identification numbers, Teacher Listing Forms and Teacher Tracking Forms

Teacher Listing Forms and Teacher Tracking Forms were needed to record information about ISCED Level 2
teachers. National centres produced these forms using WinW3S. The software created hierarchical identification
numbers that uniquely identified the sampled schools and teachers within a country. A unique four-digit school
identification number was assigned to each sampled school within each participating country. This number was
also the identification code for the school principal who answered the principal questionnaire.

According to the instructions in the TALIS School Co-ordinator Manual (MS-03-02), SCs listed teachers and
their name, year of birth, gender, main teaching domain and exclusion status. The main teaching domain
was divided into four groups: i) language and arts; ii) human sciences; iii) mathematics and science; and iv)
other. The classification of teachers into the appropriate groups was sometimes a demanding task, requiring
close co-operation between the SC and the NPM. Although TALIS surveyed ISCED Level 2 teachers, not
every teacher teaching at this level was within the scope. For example, teachers teaching only adults or
special needs students had to be excluded, as were teachers on long-term leave, and substitute, emergency
or occasional teachers.3

The national centre entered information from the Teacher Listing Forms into WinW3S and then drew the random
within-school teacher sample of 20 teachers per school. After within-school sampling was completed, WinW3S
created Teacher Tracking Forms that listed all sampled teachers. The national centre sent the Teacher Tracking
Forms to schools so that SCs knew to whom to distribute the instruments.

The Teacher Tracking Forms monitored the participation status of the sampled teachers and included teacher
names, teacher ID, year of birth, gender, questionnaire mode (on line or paper) and participation status.

TALIS 2008 Technical Report © OECD 2010

68
chapter 6 Survey Operations Procedures

The ISC did not receive any teacher names, only teacher IDs. (Since the names on the Teacher Tracking Forms
could be cut off the form, all names were kept confidential.) Copies of the Teacher Tracking Forms, without
names, were sent to the ISC together with the survey data. Appendix C contains blank Teacher Listing Forms
and Teacher Tracking Forms.

Assigning materials to teachers and school principals

Each school principal was asked to complete one principal questionnaire. The SC assigned a teacher
questionnaire to each teacher listed on the Teacher Tracking Forms. Chapter 5 gives detailed information about
the sampling algorithm and linkage of teacher lists.

The NPM sent the SC of each school a package containing all paper questionnaires and cover letters for on
line administration, the Teacher Tracking Forms and any other relevant materials prepared for briefing the SCs.
To address confidentiality concerns, several countries chose to provide teachers with pre-paid envelopes that
could be sent directly to the national centre, so that they did not have to return the completed questionnaire
to the SC.

Figure 6.1 outlines the different responsibilities of the NPM and the SC for correct assignment of questionnaires
to teachers.

Figure 6.1
Responsibilities of NPMs and the SC during survey administration

National Project Manager Activity School Co-ordinator Activity

1. Contacting participating schools

2. Preparing Teacher Listing Forms to be completed
by schools

3. Completing the Teacher Listing Form listing

all eligible ISCED Level 2 teachers within schools

4. Sampling 20 teachers per school using the information

on the Teacher Listing Form
5. Preparing Teacher Tracking Forms for administration
of the teacher questionnaires
6. Administering the questionnaires to principals
and teachers

7. After the questionnaires were administered, recording

the participation status on Teacher Tracking Forms

8. Documenting participation of teachers and principals

in WinW3S according to Teacher Tracking Forms and
IEA SurveySystem

Source: OECD.

© OECD 2010 TALIS 2008 Technical Report

Survey Operations Procedures chapter 6
69

Administering the questionnaires and national quality control

Each country had its own time frame for survey administration, from three days to four months. During this
period principals and teachers were free to fill in the questionnaires whenever they chose. It was a demanding
task for SCs to monitor the administration of the survey, especially in cases where the teachers could send the
completed questionnaires directly to the national centre.

Countries were requested to run a national quality control monitoring programme in order to guarantee high
survey standards. Outcomes of national quality control had to be reported in the Survey Activities Questionnaire
(SAQ) after survey administration and are discussed in Chapter 7.

Monitoring the on line questionnaires

The SCs recorded the return status of the paper questionnaires on the Teacher Tracking Forms. Naturally, the
tracking procedure for on line questionnaires was different. SCs indicated on the Teacher Tracking Forms
whether a teacher was assigned an on line questionnaire. National centres tracked the completion status of all
on line questionnaires using the IEA SurveySystem Monitor module. Through a secured Internet website only
available to the respective NPM, the real-time status of all respondents filling in the questionnaire could be
monitored. If a teacher or school principal who was expected to participate was not listed in the monitor, the
NPM asked the SCs to follow up.

After survey administration, national centre staff imported the participation information from the IEA
SurveySystem Monitor reports into WinW3S to record their participation status.

Material receipt and preparing for data entry

Immediately following the administration of TALIS, the major tasks for NPMs included retrieving and collating
the materials from schools and verifying their integrity. When they received survey materials from the schools,
NPMs were required to i) check that the complete and appropriate questionnaires were received for every
teacher listed on the Teacher Tracking Form; ii) verify that all identification numbers on all paper instruments
were accurate and legible; iii) check that the participation status recorded on the Teacher Tracking Form
matched the availability of questionnaires, the information on the paper questionnaires and the information
in the on line monitor; and iv) follow up with schools that did not return all the survey materials or for which
forms were missing, incomplete or otherwise inconsistent.

At the national centre, all necessary information about schools, principals and teachers as well as the return
status of the questionnaires was recorded in WinW3S. NPMs then organised the paper questionnaires and
corresponding forms for data entry (see Chapter 8).

Survey Activities Questionnaire

NPMs completed the SAQ to report their experiences during all steps of survey preparation and administration.
The ISC set up the questionnaire and administered it on line when data collection activities were completed.
All data went to the ISC.

The SAQ was built upon 9 content sections with 67 item blocks and a total of 113 items. The questions
pertained to problems or unusual occurrences, with respect to within-school sampling, establishing school
contact, preparing paper and (if applicable) on line materials, administering paper and (if applicable) on line
materials, manual data entry and submission, and the national quality control monitoring programme.

The ISC carefully reviewed responses to the SAQ and the outcomes are discussed in this report. This section
covers only within-school sampling, training of SCs and confidentiality issues. All TALIS countries, with the

TALIS 2008 Technical Report © OECD 2010

70
chapter 6 Survey Operations Procedures

exception of one, accepted the definition of “school”. In this single case the definition of “school” was clarified
with the sampling experts from Statistics Canada. All participating countries used WinW3S for within school-
sampling, including Iceland, Malta and Norway, which sampled all teachers within schools.

Nineteen out of 22 NPMs reported that the sampling selection process was not at all difficult; the remaining
3 NPMs reported it to be somewhat difficult due to the amount of paper work in large schools, communication
problems with the schools or the fact that sometimes additional explanation was needed. Seventeen out of
22 NPMs found the Teacher Listing Forms and Teacher Tracking Forms easy to work with. Fifteen countries also
used other means to list and track teachers, including Excel or Word sheets, email or personal communication
to follow up on the process.

Seven out of 22 NPMs held formal training sessions for the SCs prior to survey administration. Thirteen provided
information to the SCs through the School Co-ordinator Manual, written instructions or telephone calls.

Due to data protection rules 9 out of 24 participating countries were restricted in their use of teacher names
on the questionnaires. They replaced the names with numbers, aliases, codes or symbols. The data did not
indicate, nor did the International Quality Control Monitor report that these restrictions jeopardised the random
sampling process or the allocation of questionnaires or the quality of data.

Notes

1. Although they were written exclusively for TALIS, the manuals incorporate procedures, best practices and standards that were set
for previous IEA studies such as TIMSS, PIRLS and SITES and that were similar to those used in the OECD PISA study.

2. See Chapter 5 for more details on school sampling.

3. For more details, see Chapter 5.

© OECD 2010 TALIS 2008 Technical Report

Survey Operations Procedures chapter 6
71

References
IEA DPC (2007), TALIS Manual for National Project Managers (MS-01-03), Hamburg.

IEA DPC (2007), TALIS School Co-ordinator Manual (MS-03-02), Hamburg.

IEA DPC (2007), TALIS Data Management Manual (MS-04-02), Hamburg.

IEA (2007), TALIS Manual for International Quality Control Monitors (MS-05-01), Amsterdam.

IEA (2007), TALIS Manual for National Quality Control Monitors (MS-06-01), Amsterdam.

Statistics Canada (2007), TALIS Sampling Manual (MS-02-03), Ottawa.

TALIS 2008 Technical Report © OECD 2010

Chapter 7

On Line Data Collection

74 Abstract
74 Overview
74 Design and mixed-mode considerations
76 Technical implementation
78 Operations
79 Field trial
80 Main study participation, mode distribution and evaluation

TALIS 2008 Technical Report © OECD 2010

74
chapter 7 On Line Data Collection

Abstract

This chapter discusses the on line and electronically delivered questionnaires that have become
an increasingly useful option for international large-scale sample surveys and assessments.
TALIS offered on line data collection (ODC) with a mixed-mode design as an international
option: countries could use the option as a default means to collect data for all schools and
respondents, selected schools only or a particular population, i.e. school principals or teachers.
This chapter provides information on the design, operations, technical solutions, field trial,
main study and on line data collection processes.

Overview
The on line mode of questionnaire delivery can offer operational benefits, significantly reduce paper handling
and data entry costs for national centres, as well as yield a more accurate and timely available international
analysis database. The Second Information Technology in Education Study (SITES) 2006 (Carstens & Pelgrum,
2009), operated by the International Association for the Evaluation of Educational Achievement (IEA), was
one of the first international large-scale surveys of teachers that used on line questionnaires to collect data for
the majority of participating countries. More recently, several IEA studies (International Civic and Citizenship
Education Study, Trends in International Mathematics and Science Study and Progress in International Reading
Literacy Study) and several OECD studies (Programme for International Student Assessment and the Programme
for the International Assessment of Adult Competencies) are actively using or planning to use electronic delivery
of questionnaires and assessments.

In TALIS, the Board of Participating Countries (BPC) believed that principals and teachers would be receptive
to using a more convenient, interesting or simply “up-to-date” mode of survey administration. Most countries
participating in TALIS had already used on line data collection (ODC) in some way and for a quite a few of them,
the administration of electronic questionnaires to schools and teachers had become commonplace. Previously,
large-scale educational surveys at the international level had been based entirely on paper questionnaires. If the
alterative approach was to be successful within and across countries, it not only had to meet certain established
standards and best practices (see for example Couper, 2000; Dillman and Bowker, 2001; Reips, 2002) but also to
address the issue of reliably administering paper-based and on line questionnaires side by side where countries or
individual institutions could not guarantee a flawless overall on line delivery. The BPC consequently called for an
detailed evaluation of the appropriateness of on line-delivered questionnaires for each participating country, for
example with respect to acceptance within the target population of ISECD Level 2 teachers.

Design and mixed-mode considerations

On line data collection in TALIS was offered as an international option and conducted using a mixed-mode
design, meaning that the participating countries could adopt the option as a default means of data collection
for all schools and respondents, for selected schools only or for a particular population, i.e. school principals
or teachers. National centres had to ensure that individual respondents who refused to participate in the on
line mode or did not have access to the Internet were provided with a paper questionnaire, thereby ruling out
non-response as a result of a forced administration mode.

Data from different collection modes were merged to a single dataset within and across countries. Potential
sources of error originating from the use of the two parallel modes had to be controlled for and reduced as
much as possible to ensure uniform and comparable conditions across modes as well as countries. The design

© OECD 2010 TALIS 2008 Technical Report

On Line Data Collection chapter 7
75

established several general similarities to achieve this. The questionnaires in both modes were self-administered
and equally situated in the visual domain, in contrast to mixed-mode surveys that, say, simultaneously employ
self-administered questionnaires and telephone or face-to-face interviews. Moreover, respondents were
identified by the same sample design and procedures, contact with respondents and their validation was
established by equal means, and data from both modes were collected over the same period of time.

The electronic versions of the TALIS questionnaires could only be filled in via the Internet. No other options
were permissible, such as sending PDF documents via email or printing out the on line questionnaires and
mailing them to the national centre. As the on line data collection option for TALIS was designed specifically
with respect to educational surveys and complex operations, a precondition for a successful administration of
electronic questionnaires was that countries had to use centrally provided software.

To properly sequence preparation tasks and processes and to ensure comparability of data, the paper versions of
the two questionnaire types (i.e. principal and teacher) had first to be finalised in terms of their translation and
layout verification, even if the expectation was that all or nearly all of the data would be collected on line. From
these final paper versions, the questionnaires were converted for the on line mode followed by final structural,
optical and textual verification (see Chapter 3 for more details).

In addition to these considerations, the design had to address certain technical issues. Respondents needed only
an Internet connection and a standard Internet browser. No additional software or particular operating system
was required.

The navigational concept for the on line questionnaire had to be as similar as possible to that of the paper
questionnaires. Respondents could use “next” and “previous” buttons to navigate to an adjacent page, as if
they were flipping physical pages. In addition, the inclusion of a hypertext “table of contents” mirrored the
experience of opening a specific page or question of a paper questionnaire. While most respondents followed
the sequence of questions directly, these two features allowed respondents to skip or omit questions just as if
they were answering a self-administered paper questionnaire.

To further ensure the similarity of the two sets of instrumentation, responses to the on line questionnaires were
not made mandatory, evaluated or enforced in detail (e.g. using hard validations). Instead, some questions
used soft validation: respondents were asked to give several percentage numbers that were supposed to add up
to 100%. On these questions the sum was constantly updated according to the respondent’s entries and was
highlighted in red as long as it differed from 100%. Even if their response was still highlighted red, respondents
were able to proceed to the next question.

Certain differences in the representation of the two modes remained, however. To reduce response burden and
complexity, the on line survey automatically skipped questions not applicable to the respondent, in contrast
to the paper questionnaire in which respondents were instructed to proceed to the next applicable question.
Rather than presenting multiple questions per page, the on line questionnaire proceeded question by question.
Vertical scrolling was required for a few questions, particularly the longer questions with multiple yes/no or
Likert-type items. No horizontal scrolling was required. The visual or sensory impression of the length and
burden of a paper questionnaire can be estimated easily. The on line questionnaires attempted to offer this
through progress counters and a “table of contents” that listed each question and its response status. Multiple-
choice questions were implemented with standard HTML radio buttons. While it was possible for respondents
to change the answer to any other option, it was not possible for them to uncheck an answer completely as they
could in the paper questionnaires by crossing out an answer. The consortium considered adding extra “don’t
know” or “cancel” categories to these questions or utilising JavaScript to uncheck options, but decided against
this. The level of “cancelled” responses observed during the TALIS field trial was extremely low and did not
warrant the use of these options.

TALIS 2008 Technical Report © OECD 2010

76
chapter 7 On Line Data Collection

Overall, a near-identical representation between modes (Denscombe, 2006) was achieved, an accomplishment
that yielded identically structured and comparable data with the highest possible response rates, both at the
questionnaire and at the variable level. Great care was taken to present questions in ways that were easy to
read on screen and self-explanatory to complete. Both the terminology and the technical hurdles were carefully
considered and implemented in a way that reduced to a bare minimum the computer skills respondents needed
to access and answer the questions.

Technical implementation
After addressing procedural requirements and methodological necessities, the consortium created a plan for
implementation. No single “off-the-shelf” solution could be found that would satisfy all requirements, most
importantly in the areas of i) decentralised translation, adaptation and verification (see Chapter 4); ii) mixed-
mode data collection and subsequent data processing; and iii) minimal prerequisites on the side of respondents’
or schools’ computers. The consortium accordingly decided to re-use and extend the “IEA SurveySystem”
software which was initially developed in the context of the IEA SITES 2006 survey, although with numerous
enhancements and new features.

The SurveySystem software stored the hierarchical model of a survey’s instruments and managed all questionnaire-
related information, including text passages, translations, adaptations, validation rules, conditions and skip
logic, variable names, and other information needed for post-processing. The SurveySystem’s consolidation of
metadata in a single set of files that the TALIS national and international centres could easily send to one another
over the Internet allowed for a consistent way of managing the localised on line versions of the questionnaires.

To serve the different usage scenarios, three distinct components of the SurveySystem were developed (see
Figure 7.1).

Figure 7.1
Architectural overview of the SurveySystem

Designer application Web application

Monitor application
(Windows based) for (browser based) for
(browser based)
editiong structure, texts, taking surveys in multiple
for auditing participation
conditions ect. languages

Core provides all survey

information and manages
sessions, users,
authentication, etc.

Definition and
Database resources files Export
(SQL Server) stores structure, to WinDEM, SAS,
stores responses conditions, SPSS, RAW,
and log data texts etc. etc.

Source: OECD.

© OECD 2010 TALIS 2008 Technical Report

On Line Data Collection chapter 7
77

The Designer component was a Microsoft® Windows-based application used to create, delete, disable and
adapt survey components (e.g. questions and categories) and their properties. It allowed for translation of
all text passages in the existing national paper questionnaires and additional system texts, and it included a
complete local web server to verify and test-drive the survey as if under live conditions. The Designer also
supported the export of codebooks to the generic data-entry software used by IEA in TALIS, WinDEM, to allow
for isomorphic data entry of on line and paper questionnaires.

The Web component was a compiled ASP.NET application that served HTML questionnaires to the respondents
for completion using a standard Internet browser. Given the overall goal of securing the maximum possible
coverage, no respondents were to be excluded because of incompatible or outdated browsers or disabled
features in these. This was especially important, as requirements in terms of connection speed and available
software (browsers) were identified as crucial obstacles during initial discussions and the review of literature
(see, for example, Reips, 2002). Because computer literacy was likely to vary greatly among respondents, the
design sought a balance between minimally desirable capabilities and simplicity. In this sense, the approach
taken in TALIS (selected aspects are discussed later in this chapter) was similar to that of the “respondent-
friendly design” explicated by Dillman, Tortora and Bowker (1998). For a more recent and thorough discussion,
refer to Couper (2008).

In detail, the output was tested to assure near identical representation at minimal (i.e. 800 x 600 pixels) as well
as typical screen sizes (i.e. 1024 x 768, 1280 x 1024) in all supported browsers, which were required to support
HTML 4.0 (a standard since 1998), bi-directional Unicode text and cascading style sheets for basic formatting.
A few users had browsers that did not meet this requirement (e.g. embedded browsers on cell phones or other
mobile devices). They received a translated list of supported browsers and information on contacting their
national centre.

The Web component deliberately made use of plain HTML controls only and therefore did not require advanced
technologies such as cookies, JavaScript, Flash or pop-ups that might not be available or activated for all users.
With the exception of the welcome screen, no graphics were used. There was no automatic increase of font
sizes or question widths for larger screen resolutions.

Finally, the access-secured, web-based Monitor component allowed national centres to audit participation in
real time, including the respondent’s ID number, the first and last date of login, the total number of logins, the
current question, the response status for each individual question, and the questionnaire completion status. This
allowed the national centres to contact schools regarding incomplete or not returned questionnaires as they did
with paper questionnaires.

All systems were programmed at the IEA Data Processing and Research Center (DPC) on the basis of
Microsoft®’s .NET framework because of its proven robustness and excellent support for multilingual (Unicode)
and Internet applications. The IEA DPC used industry standards and patterns in developing the applications and
verified them through embedded unit tests and extensive internal and external testing. The live systems were
hosted on dedicated high-performance servers rented from a large and reliable solution provider in Germany.
Load and stress testing, simulating far more than the expected number of simultaneous questionnaire sessions,
were carried out prior to the production use.

Appropriate measures were taken to secure the data, and these were further strengthened by a professional
security audit conducted by an external expert. The IEA DPC developed backup and disaster recovery strategies
and constantly monitored the systems for permanent availability during the data-collection periods.

TALIS 2008 Technical Report © OECD 2010

78
chapter 7 On Line Data Collection

Operations
Once the paper questionnaires had successfully been translated and the layout verified (see Chapter 4), national
centres used the SurveySystem Designer software to convert the questionnaires into the on line mode. Along
with the software, national centres received the international English survey files prepared by the consortium.
The IEA DPC provided detailed documentation and training as part of a data management seminar in November
2006. The conversion to the on line mode was based on the concept of cultures, meaning a certain language
within a certain cultural context, for example “Norwegian (Nynorsk)” in “(Norway)”.1 Because the translation
was already verified and finalised for the paper questionnaires, this conversion involved copying and pasting
the text passages in both modes. Prior to this, any structural adaptations to the questionnaire (i.e. adapted or not
administered international variables, additional national variables) had to be reflected in the survey structure.
In addition to the questionnaire passages, certain translations were needed exclusively for on line purposes,
such as texts on the welcome screen, navigation buttons and error messages. Before submitting the files to the
IEA DPC, national centres were required to perform a visual side-by-side comparison of the paper and on line
versions using the integrated preview component.

After receiving the files containing all information needed to run the on line survey from the national centre,
experts at the IEA DPC performed a comprehensive structural and visual question-by-question check for
differences between the on line and paper versions as a quality control measure prior to activating a country’s
survey. Any detected deviations, such as mistakes in copying passages into the correct location or formatting
mistakes, were reported back to the national centres. The IEA DPC approved the on line questionnaires and
made them accessible only after any remaining issues had been resolved, thereby ensuring an isomorphic
representation of questions in both modes.

The national centres decided whether to assign the on line or the paper questionnaire to respondents based on
prior experience gained from participation in similar surveys and in the TALIS field trial. In most participating
countries, the default mode was set at the school level. All respondents at each school – the principal and the
sampled teachers – were assigned the same mode, either on line or paper. In Ireland and Portugal, respondents
received both a paper questionnaire and login instructions for the electronic version, allowing them to choose
their preferred mode.

To minimise non-responses resulting from mode assignment, NPMs were required to determine the mode that
a specific school or individual preferred and implement procedures to reliably track these requests. NPMs had
to ensure that every respondent initially assigned to the on line mode (by default or preference) had the option
of requesting and completing a paper questionnaire at any time.

To ensure confidentiality and security, each respondent received individualised login information consisting of
a numerical respondent ID and a corresponding password. National centres sent this information, along with a
confidentiality statement and information on how to access the on line questionnaire, to respondents in a letter.
As with the procedures for the paper questionnaires, the school co-ordinator (SC) distributed the information to
the designated individuals. No direct identifiers such as names were used or stored at any time. The anonymous
login procedure, together with corresponding provision during the assignment of operational IDs, guaranteed
that local data protection laws and provisions were met.

During the administration period, respondents could log in and out as many times as needed and later resume
answering the questionnaire at the point at which they had left until the end of the fielding time. Answers were
automatically saved whenever respondents moved to another question, and respondents could change any
answer at any time before completing the questionnaire. During administration, national centres provided
support and could, in turn, contact the IEA DPC if they were unable to solve the problem locally.

© OECD 2010 TALIS 2008 Technical Report

On Line Data Collection chapter 7
79

National centres were able to monitor the responses to the on line questionnaires in real-time and to dispatch
reminders to schools where respondents had not completed their surveys within the expected timeframe.
School co-ordinators were then asked to follow up with the individuals concerned.

In summary, the operational procedures needed to support both ODC and the conventional paper-and-pencil
track were designed to ensure that standard survey operations typically employed in large-scale surveys
could be used with few or no modifications. The main challenges were catering for isomorphic versions of
the instrumentation in both modes, reliably administering the resulting mixed-mode survey, and subsequently
integrating the two data sources. The overall conclusion is that TALIS was successful in achieving this.

Although countries using the on line mode in TALIS faced parallel workload and complexity before and
during data collection, they had the benefit of a reduction in workload afterwards. Because answers to on line
questionnaires were already in electronic format, and responses were stored on servers maintained by the IEA
DPC, there was also no need for separate hard copy data entry.

Field trial
As with all other complex operations, the ODC technology and methodology had to be field trialled prior to
implementation in the main survey. All NPMs were asked whether they intended to use ODC for the main data
collection. Those who intended to do so were obliged to field trial the ODC procedures.

The field trial took place in 24 countries in early 2007, mainly in a six- to eight-week period between the
second week of March and the first week of May. Of these, 16 implemented the ODC option and used a feature-
complete ODC software: Australia, Austria, Belgium (Fl.), Denmark, Estonia, Iceland, Ireland (in English and
Irish), Italy, Korea, Lithuania, Malaysia, Norway, Portugal, Slovak Republic, Slovenia and Turkey.

Approximately 3000 principal and teacher questionnaires were completed on line, or 53.1% of the school and
50.9% of the teacher data. Within the 16 countries opting for ODC, 73.1% of school questionnaires and 70.0%
of teacher questionnaires were administered on line. Using the teacher questionnaire to illustrate this further,
4 countries administered all of their questionnaires on line and another 8 countries achieved rates of more than
50% but less than 100%. Four countries administered fewer than half their questionnaires on line and for two
of those, the rate was below 10%. Nonetheless, these countries decided to continue using the option in the
main study although they were not likely to be significant reductions in terms of costs or logistics. The rates for
the principal questionnaire were similar.

The main goals of the TALIS field trial were to test and evaluate the instruments and their scaling characteristics.
The field trial was also designed to yield basic information on both the feasibility and the validity of the on
line data collection procedures. However, TALIS did not implement a strict experimental design that allocated
respondents randomly to either the paper or on line administration mode and therefore there could be no
formal tests of mode differences with respect to response rates, drop-out, response style, variable distributions,
reliability or invariance of scales and indicators.

Previous research and statistical analysis carried out on the basis of the IEA SITES 2006 survey, which targeted a
similar school and teacher audience, indicated no strong measurement effects or that these, if they existed, were
too small to be detected by the available samples (Bre ko & Carstens, 2007). For the TALIS field trial, the mode
seemed to have no reported or detectable effect on unit and variable non-response (see also Lozar et al., 2008).
Partial completion (drop-out) was less than 2% on average. While field trial results did not allow strong assumptions
about the main study, they were seen as acceptable estimators.

Overall, the procedures and tasks, supported by manuals, training and direct support, went as intended during
the field trial.

TALIS 2008 Technical Report © OECD 2010

80
chapter 7 On Line Data Collection

Main study participation, mode distribution and evaluation

The consortium recommended that countries assess their own level of confidence in regard to using the on
line mode, based on factors such as the within-country or within-school computer and Internet penetration, on
line response rates in previous surveys, and (most importantly) the outcomes of the mandatory field trial. After
taking these factors into account, all 16 participating countries that implemented ODC during the field trial
opted to do so in the main study, usually as the default mode for collecting data. Again, Ireland administered its
questionnaires in both English and Irish. Both Ireland and Portugal allowed their respondent to select the mode
by providing both paper and on line materials to all selected schools and teachers.

The surveys were usually active for a period of 7 to 12 weeks between October and November of 2007 for
the southern hemisphere countries (Australia, Korea, Malaysia) and between February and May of 2008 for the
remaining northern hemisphere countries.

Table 7.1 provides the unweighted counts and percentages of paper and on line principal questionnaires for all
participating countries. Four countries managed to collect all data entirely using on line questionnaires without
compromising coverage or increasing non-response rates. Another eight countries reached rates of more than
50% but less than 100%. Three more countries returned a significant proportion of on line questionnaires
although the rate was less than 50%. Questionnaires in Ireland were administered on line in only 4.2% of all
cases and this is reportedly due to the fact that respondents were given both a printed questionnaire and login
details and could choose between the two. On average, 50.2% of all completed principal questionnaires were
administered on line. Among the 16 countries that opted to use ODC, the majority of completed principal
questionnaires (75.3%) were administered on line.

Table 7.2 provides the unweighted counts and percentages of paper and on line teacher questionnaires for all
participating countries. Three countries managed to collect all data using on line questionnaires, 8 had rates
of more than 50% but less than 100%, 4 had a significant proportion of on line questionnaires for less than
half the respondents, and Ireland administered on line questionnaires to a small number of teachers (3.1%).
On average, 47.4% of all completed teacher questionnaires were administered on line, with 71.0% of teachers
within the 16 countries completing their questionnaires on line. The proportion of paper versus on line mode
was highly consistent across questionnaire types as well as with the proportions observed during the field trial.

In the main study, national centres and respondents agreed on the actual administration mode, meaning that
TALIS could not formally analyse or test for mode effects. The evaluation of ODC procedures after the main
survey was therefore largely based on observations and reports received directly from participating countries or
sent via the Survey Activities Questionnaire (SAQ). The following list contains selected aspects of the feedback
from the MS:
• NPMs described the conversion and verification procedures as easy to implement although repetitive. There
were no major observed or reported problems with respect to translation, conversion and representation
of complex scripts, such as Korean. The IEA DPC gave support for the more challenging modifications
and adaptations. NPMs reported that the preparation, on average, took about one person week, including
any additional time needed to communicate and resolve the differences identified during the side-by-side
comparison of instruments at the IEA DPC. All problems were resolved successfully before administration.
• The verification of ODC resource files against paper versions and the corresponding national adaptation
forms showed that all initial conversions were in good to very good condition and that typically two to
three (to a maximum five) rounds of feedback and corrections were needed to finalise the survey definitions
and resources. Minor issues included incorrect or missing line breaks, inconsistent or missing formatting
(underline, bold, italic), missing full stops after items, or incomplete translation of system texts such as those

© OECD 2010 TALIS 2008 Technical Report

On Line Data Collection chapter 7
81

for the “Next” or “Previous” buttons. Only a few substantial issues were flagged and corrected such as
inconsistent wording between the paper and on line version, accidentally disabled or re-ordered response
categories or missing variable names for national extensions.
• No major technical problems were reported during administration (e.g. unplanned down-time as a result of
hardware failures or hacking). As with the field trial, a few issues arose with respect to server access, firewalls,
virus software, URLs blocked by local networks, page caching, and the reliability of Internet connections in
general. A severe problem was only reported for two schools (affecting approx. 30 teachers) in one country
which experienced wireless network difficulties that could neither be reproduced nor fully explained by
either the IEA DPC or the NPM. Having the paper questionnaires as a fall-back strategy was essential.
• NPMs reported that respondents had several types of difficulties in accessing the questionnaire, including:
i) entering the access URL given to them in, for example, a search engine box rather than the address bar;
ii) confusing IDs and passwords during login; and iii) losing or misplacing the cover letters with login details.
A more detailed review of these difficulties is included below.
• Achieving high response rates was a key challenge for many NPMs and their teams. Reportedly, reluctance
to participate in TALIS was not related to the mode of data collection itself, but rather to “survey fatigue” and
overburdening of teachers and principals. In those few countries in which some respondents “resisted” the
on line mode, doubts about the confidentiality and anonymity of responses were the main reasons reported.
These respondents received paper questionnaires and NPMs reported that there were no further challenges
to completion. Given the importance of this matter and the numerous questions received, both the NPMs
and the consortium emphasised that the information was indeed and would remain stored securely and
confidentially at all times and that no direct identifiers or personally identifiable information was stored or
accessible to any third party.
• Follow-up procedures were in place in all countries and, typically, school co-ordinators or principals were
updated on the least once by phone or email, sometimes even on a continuous basis. National centres made
extensive use of the monitor application to facilitate follow up with non-respondents.
• NPMs reported that the actual administration including the work for organising, preparing invitations,
monitoring, and following up with non-respondents, on average, took about one person month.

Experiences and reports during the field trial and the main study stimulated more detailed explorations
and descriptive analyses using not only the actual response data but also the metadata collected during
administration, for example the date and time of the first and last login. The results of these analyses are
summarised in the following paragraphs. It should be noted that these analyses make use of unweighted data
and hence descriptions of the achieved samples in the final international database. They were not intended to
yield population estimates.

Table 7.3 presents the rate of on line administered questionnaires by respondent type (principals and teachers)
and gender for all countries participating in the on line data collection option. While the prevalence of on
line questionnaire use by country mirrors the data presented in earlier tables, it can also be seen that the
administration mode was largely consistent across principals’ and teachers’ genders. A slightly higher preference
for on line questionnaires apparently existed among male teachers, although the result was less consistent
among principals.

Given that teachers were able to self-select their preferred administration mode, it was interesting to explore the
preference for on line administered questionnaires by age group and to what extent preferences were consistent
within schools.

TALIS 2008 Technical Report © OECD 2010

82
chapter 7 On Line Data Collection

Table 7.4 presents the percentage of questionnaires administered on line by age group. In some cases, for
example in Austria, Estonia, Italy and Slovenia, the preference for the on line mode apparently decreased as the
teachers’ reported age group increased. For other countries, the values were either more homogeneous or not
apparently associated with teachers’ age in a uniform way.

Table 7.5 shows, by country, the percentage of schools that administered questionnaires on line to a certain
extent. For example, questionnaires in almost all schools in Denmark were administered exclusively on line
(97.8%) and only 1.5% (i.e. 2 out of 137 schools) used paper questionnaires for all of their teachers. In Portugal,
the majority of schools used on line questionnaires for less than half of their teachers and only 4.6% (i.e. 8 out
of 173 schools) used on line questionnaires for all teachers.

Of particular interest in the domain of self-administered electronic questionnaires is the question of access.
Paper questionnaires are immediately accessible whereas electronic questionnaires, unless their access is
completely uncontrolled, require some form of validation. In TALIS, validation took the form of a fairly typical
numeric ID and password combination. The login procedures and accompanying instructions were concise
and straightforward. However, it was expected that some respondents would face difficulties in managing this
hurdle. This, in turn, could have had an adverse affect on response rates or introduces bias to the survey results
if individuals or certain demographic groups had been unable to access the questionnaire reliably. Although the
fall-back-to-paper strategy was designed to address this issue, it was nonetheless of interest to identify access
problems and derive ways to minimise these for future surveys.

A labour-intensive manual analysis was conducted during the main study administration in May 2008 and
covered the first 40 000 out of approximately 55 000 login attempts recorded in the systems. Of these,
about 4 800 were recorded as initially unsuccessful, a number that appeared high. During the analysis, several
types of systematic errors were detected and classified into an error typology. The distribution of this failed login
typology was not uniform, however. The majority of all failed login attempts were due to four main reasons,
listed in Table 7.6; three other reasons existed in small proportions. The analysis continued by attempting to
identify corrections that respondents have used and their eventual success following the initial failure. Most
respondents were able to correct their initial error immediately by reviewing the login details or requesting help
from colleagues, the school co-ordinator or the national centre (help information was included as part of the
cover letter given to respondents). Table 6.6 shows that 94.4 to 100% of all initially failed login attempts were
corrected and eventually successful.

In summary, the login procedure was not a significant source of non-response or total survey error. For the residual
failed attempts, paper questionnaires were most likely issued by the NPMs. The consortium, the NPMs and the
OECD considered as acceptable the observed final rate of failed logins for which no immediate or delayed
correction was identified or identifiable, i.e. 119 out of 40 000. The true level of failed attempts was believed to
be even smaller given that it was impossible in the analysis to clarify whether a later login attempt for some error
types succeeded. The results indicate that even more simplified login procedures could be explored.

Little is known or documented about how respondents fill out paper-based, self-administered questionnaires
in educational surveys. For the on line mode, the complete set of login records described the number of logins
per respondent and the duration, defined as the number of days between the first and last login date less one, in
which each questionnaire was completed. The actual duration may have been affected by a number of factors.
The principal questionnaire asked for factual information that principals had to gather from school databases
and other sources. For both teachers and principals, questionnaire completion may have been interrupted by
other, school-related activities. NPMs reported that a sizable number of respondents had initially started to
complete the questionnaire, left it incomplete, and were later asked to finalise it as part of the national centres’
follow-up activities.

© OECD 2010 TALIS 2008 Technical Report

On Line Data Collection chapter 7
83

Table 7.7 presents some key measures of response duration and login behaviour. While the mean and maximum
number of days taken to complete the questionnaires were relatively high (most probably because respondents
failed to complete the questionnaire until reminded by the school co-ordinator) the majority of questionnaires
were completed within one day (e.g. 95% of teachers in Austria) and a higher percentage (e.g. 98.9% of
teachers in Austria) were completed within a week. On average respondents at all levels logged in slightly fewer
than two times. In line with expectations, principals consistently logged in at a higher than average rate and
took more time to complete their questionnaires, expressed in lower completion rates within one login, one day
and one week. Malaysia is a noteworthy exception to this picture. The Malaysian NPM provided one possible
explanation, namely that some schools experienced slow Internet connections or could not access the on line
questionnaire. When this happened, they normally tried again on a different day. This seems to be supported by
the high average number of logins for both principals and teachers.

Another concern with electronically delivered questionnaires is that they lack a usable overview of the time
required for completion. Item non-response in long questionnaires may increase as a function of time, exhaustion
or rejection. Such an effect, for example, was observed during the field trial in IEA SITES 2006 (see Bre ko &
Carstens, 2007) and questionnaires were significantly shortened for the main study. The TALIS questionnaires
were also reviewed after the field trial and reduced to such a length that, again on average, required a net
response time of 30 to 45 minutes. Drop-out, i.e. prematurely leaving the questionnaire before its end, was
therefore not expected to be a major concern in TALIS. Analyses investigated the magnitude and the increases
in trailing omitted variables in the TALIS main study questionnaires. The result suggested that there was no clear
direction of drop-out across countries and administration modes, although differences existed within individual
countries. The main conclusion was that the on line mode itself did not systematically increase the amount of
drop-out observed in the data.

Overall, on line data collection in TALIS was implemented successfully, due to the commitment and hard work
of NPMs and their teams. However, experiences showed that surveys of this type cannot yet go “e” completely
and should be rolled out with care. Mixed-mode systems are admissible but if countries or institutions cannot
guarantee flawless delivery of electronic questionnaires, a fall-back to paper-based administration is essential
to ensure that sampling and coverage principles are not compromised.

Note

1. Culture names follow the RFC 1766 standard in the format «<languagecode2>-<country/regioncode2>», where <languagecode2> is
a lowercase two-letter code derived from ISO 639-1 and <country/regioncode2> is an uppercase two-letter code derived from ISO 3166.

TALIS 2008 Technical Report © OECD 2010

84
chapter 7 On Line Data Collection

Table 7.1
Extent to which paper and on line administration modes were used for the Principal Questionnaire
during the main study (sorted in descending order of on line usage and country name)

Total On line Paper
Number of teachers Number of teachers % Number of teachers %
Iceland 102 102 100.0 0 0.0
Korea 153 153 100.0 0 0.0
Norway 153 153 100.0 0 0.0
Turkey 188 188 100.0 0 0.0
Denmark 117 116 99.1 1 0.9
Australia 148 145 98.0 3 2.0
Malaysia 217 210 96.8 7 3.2
Estonia 193 182 94.3 11 5.7
Belgium (Fl.) 180 168 93.3 12 6.7
Lithuania 204 190 93.1 14 6.9
Slovak Republic 181 113 62.4 68 37.6
Italy 287 149 51.9 138 48.1
Austria 242 110 45.5 132 54.5
Portugal 161 62 38.5 99 61.5
Slovenia 178 48 27.0 130 73.0
Ireland 120 5 4.2 115 95.8
Brazil 377 0 0.0 377 100.0
Bulgaria 197 0 0.0 197 100.0
Hungary 183 0 0.0 183 100.0
Malta 58 0 0.0 58 100.0
Mexico 189 0 0.0 189 100.0
Netherlands 33 0 0.0 33 100.0
Poland 172 0 0.0 172 100.0
Spain 183 0 0.0 183 100.0
Totals and averages >> 2 094 50.2 2 122 49.8
(49.7) (50.3)
Totals and averages 2 824 2 094 75.3 730 24.7
(ODC using countries only) (74.2) (25.8)

Notes: Percentage values in the totals and averages rows represent the averages of the percentages for each country.
Values in parentheses represent the percentages of all individual questionnaires.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

On Line Data Collection chapter 7
85

Table 7.2
Extent to which paper and on line administration modes were used for the Teacher Questionnaire
during the main study (sorted in descending order of on line usage and country name)

Total On line Paper
Number of teachers Number of teachers % Number of teachers %
Iceland 1 394 1 394 100.0 0 0.0
Korea 2 970 2 970 100.0 0 0.0
Turkey 3 224 3 224 100.0 0 0.0
Norway 2 458 2 438 99.2 20 0.8
Denmark 1 722 1 702 98.8 20 1.2
Malaysia 4 248 4 156 97.8 92 2.2
Australia 2 275 1 972 86.7 303 13.3
Estonia 3 154 2 698 85.5 456 14.5
Belgium (Fl.) 3 473 2 962 85.3 511 14.7
Lithuania 3 535 2 980 84.3 555 15.7
Slovak Republic 3 157 2 529 80.1 628 19.9
Portugal 3 046 940 30.9 2 106 69.1
Italy 5 263 1 538 29.2 3 725 70.8
Slovenia 3 069 893 29.1 2 176 70.9
Austria 4 265 1 136 26.6 3 129 73.4
Ireland 2 227 69 3.1 2 158 96.9
Brazil 5 834 0 0.0 5 834 100.0
Bulgaria 3 796 0 0.0 3 796 100.0
Hungary 2 934 0 0.0 2 934 100.0
Malta 1 142 0 0.0 1 142 100.0
Mexico 3 368 0 0.0 3 368 100.0
Netherlands 484 0 0.0 484 100.0
Poland 3 184 0 0.0 3 184 100.0
Spain 3 362 0 0.0 3 362 100.0
Totals and averages 73 584 33 601 47.4 39 983 52.6
(45.7) (54.3)
Totals and averages 49 480 33 601 71.0 15 879 29.0
(ODC using countries only) (67.9) (32.1)
Notes: Percentage values in the totals and averages rows represent the averages of the country percentages.
Values in parentheses represent the percentages of all individual questionnaires.
Source: OECD, TALIS Database.

Table 7.3
Percentage of questionnaires administered on line, by respondent type and gender
Principals Teachers

Female Male Female Male

Australia 98.1 97.9 86.8 86.5
Austria 43.3 46.3 25.9 28.2
Belgium (Fl.) 91.3 94.5 85.0 86.0
Denmark 100.0 98.7 98.9 98.7
Estonia 93.1 95.6 85.5 85.6
Iceland 100.0 100.0 100.0 100.0
Ireland 7.3 2.6 2.8 3.8
Italy 49.3 54.7 28.0 33.7
Korea 100.0 100.0 100.0 100.0
Lithuania 89.1 97.1 84.4 83.9
Malaysia 96.6 96.9 97.8 97.8
Norway 100.0 100.0 99.2 99.2
Portugal 37.1 38.9 28.5 36.4
Slovak Republic 64.8 58.3 80.3 79.2
Slovenia 22.9 32.9 27.6 35.0
Turkey 100.0 100.0 100.0 100.0

Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

86
chapter 7 On Line Data Collection

Table 7.4
Percentage of questionnaires administered on line, by teachers’ age group
<25 25-29 30-39 40-49 50-59 60+
% % % % % %
Australia 94.7 90.7 83.7 88.1 83.8 92.4
Austria 37.5 29.5 27.8 28.2 24.3 19.6
Belgium (Fl.) 89.5 89.7 86.1 84.5 80.3 88.2
Denmark 100.0 96.2 98.8 99.5 98.7 100.0
Estonia 97.5 93.9 90.7 89.0 81.4 69.3
Iceland 100.0 100.0 100.0 100.0 100.0 100.0
Ireland 5.2 4.0 3.6 1.2 2.9 5.3
Italy 40.0 38.8 36.4 30.2 27.2 17.1
Korea 100.0 100.0 100.0 100.0 100.0 100.0
Lithuania 84.8 86.2 89.4 87.7 79.1 74.1
Malaysia 100.0 96.5 97.6 98.8 98.2 100.0
Norway 100.0 99.0 99.4 98.8 99.3 99.3
Portugal 17.6 33.1 34.6 29.8 22.3 19.1
Slovak Republic 84.7 85.0 86.4 80.8 75.3 56.8
Slovenia 50.0 38.9 34.0 28.9 19.3 12.2
Turkey 100.0 100.0 100.0 100.0 100.0 100.0
Source: OECD, TALIS Database.

Table 7.5
Percentage of schools that administered questionnaires on line, by country
Half or more but
None on line Less than half on line less than all on line All on line
N % % % %
Australia 149 2.0 5.4 36.2 56.4
Austria 248 70.6 1.2 2.4 25.8
Belgium (Fl.) 197 4.6 5.1 26.9 63.5
Denmark 137 1.5 0.0 0.7 97.8
Estonia 195 3.1 6.2 31.3 59.5
Iceland 133 0.0 0.0 0.0 100.0
Ireland 142 69.0 31.0 0.0 0.0
Italy 298 36.2 39.6 13.1 11.1
Korea 171 0.0 0.0 0.0 100.0
Lithuania 206 4.9 10.7 15.0 69.4
Malaysia 217 1.4 0.9 0.9 96.8
Norway 156 0.0 0.0 0.6 99.4
Portugal 173 17.9 61.8 15.6 4.6
Slovak Republic 186 11.3 6.5 32.8 49.5
Slovenia 184 45.7 24.5 18.5 11.4
Turkey 193 0.0 0.0 0.0 100.0
Source: OECD, TALIS Database.

Table 7.6
Error typology: type, initial percentage of all analysed login attempts,
percentage of successful corrections, and percentage of residual failed login attempts
(sorted in descending order of magnitude of initial error rate)
Of these,
percentage of Residual
Percentage corrected (ultimately percentage of
of all login attempts successful) logins all login attempts
Respondent typed incorrect password (e.g. added or deleted one digit). 3.16 99.76 0.01
Respondent typed his or her name, an e-mail address, the project name, or the school
3.10 94.36 0.18
name instead of his or her personal ID.
Respondent used the school ID also printed on the label instead of his or her teacher ID. 3.07 99.18 0.03
Respondent typed incorrect ID (e.g. added or deleted one digit). 2.53 96.64 0.09
Respondent confused ID and password. 0.15 100.00 0.00
Respondent confused 0 (digit) and O (letter). 0.12 97.83 0.00
Respondent typed the word “password” or similar. 0.04 92.86 0.00
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

On Line Data Collection chapter 7
87

Table 7.7
Maximum, mean, and median duration for completing an on line questionnaire, as well as percentage
of completed questionnaires within one login, one day, and one week, by country
Maximum duration Mean duration Completed in Completed in Completed in
Mean logins (days) (days) one login % one day % one week %
P T P T P T P T P T P T
Australia 1.6 1.4 47 57 2.5 2.0 59.9 73.3 83.4 93.1 93.6 96.2
Austria 1.5 1.4 10 21 1.2 1.3 68.0 76.9 94.5 95.0 98.4 98.9
Belgium (Fl.) 1.9 1.4 39 43 2.7 1.7 55.1 74.6 83.0 91.2 91.5 96.2
Denmark 1.4 1.2 45 62 2.7 1.9 73.8 83.5 88.5 95.2 93.1 97.1
Estonia 1.9 1.7 43 35 2.9 2.3 58.6 64.0 82.3 83.4 92.5 93.2
Iceland 1.5 1.3 24 62 1.4 1.3 70.6 78.8 94.5 95.9 98.2 98.8
Ireland 1.7 1.5 1 27 1.0 1.5 63.6 70.1 100.0 94.8 100.0 97.4
Italy 2.2 1.9 63 64 4.7 4.1 48.4 56.8 76.5 76.4 85.6 86.6
Korea 2.0 1.8 28 33 2.4 2.0 44.7 56.4 85.5 89.2 92.5 94.5
Lithuania 2.1 1.8 30 37 2.7 2.2 51.0 57.3 80.7 82.2 91.7 93.3
Malaysia 3.8 3.0 50 53 7.8 6.8 22.2 27.1 50.5 46.6 65.3 68.0
Norway 1.5 1.5 32 52 1.8 1.8 73.6 77.3 91.4 94.3 96.6 96.5
Portugal 2.1 1.7 65 70 5.8 2.6 52.1 59.6 75.3 86.9 82.2 92.3
Slovak Republic 2.3 2.1 27 33 4.0 3.6 44.9 49.8 72.9 72.9 81.4 83.2
Slovenia 1.8 1.6 22 27 2.5 1.8 54.7 63.3 77.4 87.7 90.6 95.6
Turkey 3.4 2.0 52 48 8.5 2.9 30.1 50.2 54.1 80.4 67.3 90.6
Note: P = Principal, T = Teacher
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

88
chapter 7 Online Data Collection

References
Bre ko, B.N. and R. Carstens (2007), “On line data collection in SITES 2006: Paper versus web survey – Do they provide
comparable results?”, The Second IEA International Research Conference: Proceedings of the IRC-2006 (Vol. 2), International
Association for the Evaluation of Educational Achievement, Amsterdam, pp. 261-270.

Carstens, R. and W.J. Pelgrum (eds.) (2009), IEA SITES 2006 Technical Report, International Association for the Evaluation
of Educational Achievement, Amsterdam.

Couper, M.P. (2000), “Web Surveys: A Review of Issues and Approaches”, Public Opinion Quarterly, 64, pp. 464-494.

Couper, M.P. (2008), Designing Effective Web Surveys, Cambridge University Press, Cambridge.

Denscombe, M. (2006), “Web-based Questionnaires and the Mode Effect: An Evaluation Based on Completion Rates and
Data Contents of Near-identical Questionnaires Delivered in Different Modes”, Social Science Computer Review, 24 (2),
pp. 246-254.

Dillman, D.A. and D. Bowker (2001), “The Web Questionnaire Challenge to Survey Methodologists”, In U.D. Reips and
M. Bosnjak (eds.), Dimensions of Internet Science, Pabst Science Publishers, Lengerich, pp. 159-178.

Dillman, D.A., R.D. Tortora, and D. Bowker (1998), Principles for Constructing Web Surveys: An Initial Statement
(Technical Report No. 98-50), Washington State University Social and Economic Sciences Research Center, Pullman, WA.

Lozar, M.K., et al. (2008), “Web Surveys versus Other Survey Modes – A Meta-Analysis Comparing Response Rates”,
International Journal of Market Research, 50 (1), pp. 79-104.

Reips, U.D. (2002), Standards for Internet-based Experimenting. Experimental Psychology, 49 (4), 243-256.

© OECD 2010 TALIS 2008 Technical Report

Chapter 8

Quality Assurance

90 Abstract
90 Overview
90 Quality control in TALIS survey administration
91 International quality control monitoring programme
91 Survey anonymity
92 School co-ordinator interviews
92 Initial preparations
93 Survey administration activities
93 General observations
94 Survey Activities Questionnaire
94 Contacting schools
95 Preparing survey materials
96 Conversion to on line data collection
96 Survey administration
97 Data entry and verification
97 National quality control monitoring programme

TALIS 2008 Technical Report © OECD 2010

90
chapter 8 Quality Assurance

Abstract

This chapter explores the quality control programme prepared for administering TALIS. It
discusses the standardised procedures taken for survey preparation, administration and
data entry in order to ensure a high quality collection and processing of TALIS data. Quality
control in TALIS comprised three parts: organisation and oversight by the IEA Secretariat of an
international programme of school and national centre visits by International Quality Control
Monitors (IQCMs); a national quality control programme of school visits operated by National
Project Managers; and a follow-up on line Survey Activities Questionnaire seeking experiential
information from the National Project Managers.

Overview
For the TALIS main survey (MS) the IEA Secretariat prepared a standardised quality control programme of
school visits. The programme consisted of an international and a national component; its major aim was to
document the quality of the survey administration in each country and flag any issues that might influence the
comparability of the data. A secondary aim was to learn about experiences with TALIS directly from the people
administering it, so as to better understand how to improve procedures for subsequent cycles.

Quality control in TALIS survey administration

The materials and procedures developed for the TALIS survey administration were standardised across all
participating countries and languages to ensure, as far as possible, that participants in each country received
comparable survey materials under comparable survey conditions (see Chapter 6 for more details). The purpose
of the TALIS quality control programme was to document the extent to which the standard operating procedures
were followed in each country.

Quality control of data collection in TALIS comprised three parts:

• The IEA Secretariat organised and oversaw an international programme of school and national centre visits
by International Quality Control Monitors (IQCMs).
• National Project Managers (NPMs) operated a national quality control programme of school visits. The IEA
Secretariat supplied a manual template that could be adapted according to countries’ needs, and that was
used by 19 out of 21 countries that ran a national quality control programme.
• The International Study Centre (ISC) administered an on line Survey Activities Questionnaire (SAQ) to be
completed by NPMs after survey administration. NPMs were asked about their experiences with the TALIS
survey administration.1 Outcomes of the national quality control programme were reported in the final
section of the SAQ.
• The full quality control programme was administered only for the MS. Quality control for the field trial (FT)
at the international level consisted of the Field Trial Operations Checklist, which outlined major steps in
survey administration activities: sampling, preparing survey materials, data collection, data entry and data
submission. The checklist asked NPMs to fill in the date each task was completed, and to list any comments
or any problems they experienced. The ISC used completed checklists to identify weak points in survey
administration and to improve survey operation procedures for the MS.

© OECD 2010 TALIS 2008 Technical Report

Quality Assurance chapter 8
91

International quality control monitoring programme

For the international programme, the IEA Secretariat, in co-operation with national centres, identified and appointed
an IQCM in each country to visit 10% of the sampled TALIS schools and to interview the school co-ordinators
(SCs) about aspects of study administration. Since the international sample size for TALIS was 200 schools,
IQCMs visited 20 schools in every country except Brazil. The Brazilian IQCM visited 40 schools because Brazil
had sampled 400 schools for the MS. IQCMs were asked to select the schools following a standardised procedure.
Schools to be visited were randomly selected from a subset of schools that met specific criteria.2

In large countries like Australia and Brazil IQCMs were permitted to recruit and train assistants to conduct the
school visits in more distant regions of the country. Assistants were also necessary if the survey was conducted
within a short timeframe, as in the case of Mexico where all TALIS data were collected within one week. Overall,
five IQCMs engaged one assistant; in the case of one country three assistants were necessary. Altogether, the
IQCMs and those trained by them successfully visited and interviewed SCs at 500 schools. The results from
these school visits are discussed in later in the chapter.

Prior to the MS administration, IQCMs from twenty-four TALIS countries participated in a one-day training
seminar in Amsterdam, Netherlands. For southern hemisphere countries the training session took place in
September 2007; for northern hemisphere countries, it took place in January 2008.

During training IQCMs received the following materials:

• the TALIS Manual for International Quality Control Monitors (MS-05-02), outlining the IQCM’s roles and
responsibilities;
• the School Visit Record, a standardised, structured interview format concerning survey implementation;
• the Translation Verification Report from the translation verifier. The IQCMs commented on a comparison
between the report from the translation verifier and the final version of instruments used in the country
(see Chapter 4 for more details); and
• a CD-ROM containing all TALIS manuals and forms used during data collection.

Survey anonymity
A major concern among teachers sampled for participation in TALIS was whether the completed questionnaires
and survey results would be anonymous and confidential. While confidentiality was guaranteed in the written
introduction to the survey, many respondents remained unconvinced. SCs and NPMs in 50% of participating
countries reported teacher questions or concerns regarding the confidentiality of responses. Teacher names
were recorded on questionnaires and tracking forms for only 33% of these countries. The others relied on ID
numbers, codes or aliases to disguise teacher identities.

The importance of maintaining respondent confidentiality was impressed upon both SCs and QCMs. Many
SCs mentioned that the completed questionnaires were in sealed envelopes or did not have teacher names on
them. In 58% of countries, NPMs elected to identify teachers on tracking forms and questionnaire or cover letter
labels by some means other than name to follow legal requirements or to meet teachers’ concerns. Although
this procedure was more difficult and more prone to error, informal reports suggested that some respondents
felt more comfortable when their name was not on the documents.

When asked about provisions for the security of the Teacher Listing and Tracking Forms, SCs indicated that
security and confidentiality were taken very seriously at the school level. Fewer than 2% of Teacher Listing and
Teacher Tracking Forms were stored with questionable security, for example that they were left in a staffroom
pigeonhole. In 84% of cases SCs indicated that nobody other than themselves had access to the forms.3

TALIS 2008 Technical Report © OECD 2010

92
chapter 8 Quality Assurance

School co-ordinator interviews

As mentioned earlier, the School Visit Record is a structured SC interview covering four topics: the initial
preparations of the SC, survey administration activities, the SC’s general impressions and the IQCM’s
general impressions regarding this particular school. In most cases IQCMs were able to complete their tasks
successfully; however, IQCMs in three countries encountered at least one unhelpful SC who forgot or refused
the appointment.

In most countries SCs were members of the school staff: 29% were vice-principals, 22% were principals,
14% were classroom teachers and 11% were directors or heads of teaching and learning. However, some
countries like Brazil, Hungary, Mexico and Spain employed an external surveying organisation to distribute the
questionnaires, explain the survey and collect the questionnaires, so the role of the SC there was correspondingly
smaller. Approximately 86% of SCs were responsible for one school only. Generally these people held positions
within the school. The maximum number of schools for which one SC was responsible was 15 in Spain, where
SCs came from an external agency.

Initial preparations
Section A of the School Visit Record asked SCs about the training and other preparations they had completed
for their role in TALIS. Formal training sessions were offered in 42% of participating countries to the majority
of SCs. Approximately half of all SCs interviewed reported they had received training for their role, and of
those who did receive training, more than 95% reported it was adequate. Training typically consisted of formal
face-to-face sessions – others described “training” as a combination of email and phone contact and written
instructions. The greatest barriers to training were distance and time. Ninety-two percent of SCs described the
TALIS School Co-ordinator Manual (MS-03-02) as having worked well. Several SCs noted that the checklists
were very helpful, and some thought that the manual explained things so well and so clearly that an additional
training session was unnecessary.

Prior to the process of within-school sampling, SCs completed Teacher Listing Forms. Page 5 of the TALIS
School Co-ordinator Manual (MS-03-02) provided SCs with detailed information about whom to include on the
form. Among SCs interviewed, 8.6% experienced some difficulty in completing the Teacher Listing Form – the
major complaint being that it was a lengthy and time-consuming process to locate and list all the information,
especially for large schools. There was only one country where SCs returned Teacher Listing Forms with exactly
20 teachers listed, suggesting that they had not understood procedures properly.

Approximately 87% of SCs reported that they corresponded with their NPM and familiarised themselves with
the survey process prior to distributing materials; a proportion of those who did not have contact with the NPM
had some other external co-ordinator, such as an external surveying organisation, that they could rely on for
advice and support. After national centres sent the materials to schools, SCs had to check their completeness.
Fewer than 5% of SCs reported that materials were missing from their shipment – in most cases, these problems
were resolved swiftly with the assistance of the national centre. Eighty-seven percent of SCs verified adequate
supplies of questionnaires or cover letters prior to questionnaire administration. Only one of those who did not
verify adequate supplies had to arrange replacements. In 98% of the total sample, paper questionnaires were of
good quality and defective questionnaires were present in only 2%.4

School co-ordinators in countries participating in on line data collection (ODC) were asked two additional
questions about their effectiveness at explaining the ODC procedures, and whether teachers received the survey in
the format they were expecting. In ODC countries, 99.6% of SCs thought they were “very effective” or “somewhat
effective” at explaining the ODC procedures; 94% were confident that teachers received the questionnaire in the
format they were expecting. Table 8.1 gives an overview of the SC tasks prior to survey administration.

© OECD 2010 TALIS 2008 Technical Report

Quality Assurance chapter 8
93

Survey administration activities

Section B of the School Visit Record asked about survey administration activities. Table 8.2 presents SC responses
to these questions.

Eighty percent of SCs distributed the Principal Questionnaires as soon as the materials arrived at the school;
only 3.4% reported any problems identifying teachers or the principal. About 84% of SCs reported that they
explained the purpose of the survey, the estimated time to completion and the survey return procedures.

Seventy-six percent of SCs reported that there were no special instructions or motivational talk or incentives
offered to teachers to encourage them to participate. When there were motivational talks or encouragement,
these were usually held at a group session for survey distribution, and were usually conducted by the principal
or deputy. Some schools published information in their staff newsletter.

When distributing questionnaires to teachers, 54% of respondents claimed they located each teacher at the school
and handed the questionnaire to him or her personally. Nineteen percent held a group session for teachers at
which they handed out the information individually, and 13% placed the questionnaire in teachers’ pigeonhole
or other private mailbox. Fourteen percent enlisted the principal’s assistance in distributing questionnaires – either
relying on the principal to hold a group session, or leaving the questionnaires with the principal to distribute.

Table 8.3 shows the use of forms in survey administration. Ninety-seven percent of SCs interviewed responded
that they distributed the questionnaires or cover letters in accordance with the Teacher Tracking Form. Of those
who responded in the negative, in most cases the problem was rectified or flagged in co-operation with the NPM.

Among SCs, 87.4% understand the return procedures. Only 4% of those surveyed indicated that they had doubts
or problems and usually these were resolved in co-operation with the NPM. External surveyors experienced
some difficulties in retrieving completed questionnaires; some had to return to the school more than once to
collect them.

Completion of the Teacher Tracking and Reference Class Listing Forms went smoothly in more than 97% of
cases. Fewer than 3% of surveyed SCs prepared Teacher Tracking Forms that listed someone who was not a
teacher. Just under half of these cases involved pedagogical support staff, and a similar number were teachers
who were not yet fully qualified. The remainder referred to principals who were also teachers – in some cases
principals were listed on the Teacher Tracking Form but only completed the Principal Questionnaire.

The Reference Class Listing Form proved a little more difficult, and was not collected at approximately 10% of
all schools visited by IQCMs. No Reference Class Listing Forms were collected for two participating countries.
In eight countries, SCs communicated problems with the Reference Class Listing Form to NPMs. When asked
if SCs used a school timetable or other official document to complete the form, 97% who completed the
form responded “yes”. In 88% of cases, there were no problems completing the form. However, if there
were problems, the most common included: recently changed timetables; use of a 6-day or 10-day (or other)
timetable; difficulty understanding instructions and the meaning of “after 11 o’clock Tuesday”; and difficulties
interpreting subject codes.

General observations
Section C of the School Visit Record asked SCs to indicate whether any principals or teachers approached them
to discuss any aspect of the survey: 20% of SCs were approached about the purpose of the survey, 9% were
asked about the survey return procedures, 21% were asked for clarification of items, 3% were approached
about an error spotted and 13% heard other questions about the survey. Only 3% of SCs indicated they were
asked questions they could not answer.

TALIS 2008 Technical Report © OECD 2010

94
chapter 8 Quality Assurance

For the most part, teachers were described as co-operative, and 47% of SCs described teachers as “extremely
co-operative”. None were described as “hardly co-operative at all”. Approximately 17% of SCs who had
found teachers to be somewhat unco-operative reported that special efforts were made to encourage their co-
operation. These efforts consisted of placing the survey into a context, and detailing the purpose of the survey.
In many schools, principals also gave short motivational talks or otherwise encouraged teachers to participate.

More than 80% of SCs thought the distribution of surveys went “very well”. Fifty-eight percent described the
attitude of school staff towards the survey as positive and 40% as neutral. Fewer than 10% of SCs were faced
with teachers who refused to participate in the survey. Reasons for non-participation included: lack of time; no
motivation; advice from the teachers’ union to not co-operate; absence; too many surveys; technical problems
or lack of computer skills; and objections to participating in international surveys.

Seventy-five percent of SCs reported that NPMs were responsive to their questions and concerns, although 20%
had no need to contact the NPM at all. Ninety-one percent of SCs interviewed said they would be willing to
serve as SC in another international survey of this kind.

For 81% of northern hemisphere countries, SCs reported that the survey was conducted at an appropriate time
of year; however only 37% of the SCs of the four southern hemisphere countries reported the same. Although
for both hemispheres the survey was administered close to the examination period at the end of the school
year, some southern hemisphere SCs considered this period too busy for implementing surveys such as TALIS.

IQCM’s overall impressions about implementing of TALIS (Section D of the School Visit Record) were positive.
In 95% of visited schools they had no doubts about SCs taking their role seriously and being well prepared for
their job. No more than 3% of SCs were, in the opinion of the IQCM, unsatisfied with their tasks for reasons
that included “too much work” as well as “too little involvement in the project”.

Survey Activities Questionnaire

The SAQ covered all aspects of survey administration. The ISC delivered it on line to NPMs after all data had
arrived at the ISC. The questionnaire obtained information about activities and the extent to which procedures
and guidelines were followed. It also gave NPMs an opportunity to give feedback about all aspects of survey
administration, including procedures and manuals.

Contacting schools
First contact with sampled schools5 was typically made by the NPM or other member of the national team (71%).
In 29% of cases, the Ministry of Education made first contact, sometimes in conjunction with the national centre.
In cases where an external surveying agency administered the questionnaires (four countries), staff from the
agency contacted the schools. Although overall participation rates for TALIS were high, 71% of NPMs reported
difficulties in convincing schools to participate. Several national centres reported spending considerable time
following up with school contacts. In some cases schools did not respond definitively until it was too late to
replace them. Strategies to overcome school reluctance to participate included: multiple follow-up attempts and
co-operation with teacher unions or regional, state or national education authorities in requests to participate.
About half the participating countries successfully extended the internal survey deadline in order to improve the
overall response rate. This was because schools felt “surveyed out”, because it was a difficult time of year, because
of concerns about confidentiality provisions, or because principals did not wish to place an extra burden on
teachers. However, the international deadline for data submission was not affected by these internal extensions.

Half the NPMs reported difficulty in identifying or contacting SCs. In several cases this was because it was not
obvious who – other than the principal – should be the SC, or because the prospective SC was difficult to reach
by telephone or e-mail.

© OECD 2010 TALIS 2008 Technical Report

Quality Assurance chapter 8
95

Formal training sessions for SCs were held in 42% of countries. Those who did not prepare training sessions
ensured that SCs were equipped with adequate written instructions, and contact details in case of difficulties.
In most cases the written material consisted of the translated TALIS School Co-ordinator Manual (MS-03-01).
For 83% of NPMs the adaptation and translation of this manual was “not difficult at all”. The remaining 17%
found this process “somewhat difficult”.

Preparing survey materials

All countries were required to prepare paper instruments, even if they administered the survey on line only.
Instruments and cover letters underwent a rigorous process of translation verification and layout verification
before printing.6

NPMs reported that all versions of the questionnaires were translated or adapted from the international English
source. The only exception was Slovenian in Austria,7 where the Slovenian version from Slovenia was used
as template for the Austrian instruments. When selecting translators and reviewers NPMs were advised to
employ language specialists who were fluent in English, had the survey language as their mother tongue,
were experienced with the country’s cultural context (preferably living there) and were familiar with survey
instruments (see Chapter 4 for more details). These conditions were met almost without exception.

Sixty-seven percent of NPMs reported that it was “not difficult at all” to translate and adapt the questionnaires
to the national language(s). The most frequently reported problem was the difficulty in preparing an accurate
translation of items that did not have a good fit with the country’s national context – for these NPMs had
to take special care not to introduce misunderstandings. Of documenting national adaptations, 92% of
participants reported it was “not difficult at all”; however several commented that the process was very time-
consuming.

Table 8.4 shows that all NPMs were able to find a translator who was fluent in English and was experienced
in the country’s cultural context, whereas 95.83% had the survey language as mother tongue. One hundred
percent of the reviewers were experienced in the country’s educational context and were familiar with survey
instruments. These findings underline that the TALIS instruments were translated and reviewed by experienced
experts throughout all participating countries.

Instrument verification was conducted over four stages8 if ODC was used and over three stages if the survey
was administered via paper and pencil only (see Chapter 3 for more details). The IEA Secretariat co-ordinated
the second stage, translation verification. NPMs were asked in the SAQ about the kinds of changes they
made to instruments following each stage of verification. For translation verification, 87% responded that
they made changes to their instruments in accordance with verifier suggestions. Three countries made further
improvements that were not suggested by the verifier. In one country the NPM identified a translation error
in the paper instruments after they had been printed and sent to the schools. The error was corrected in the
on line version of instruments prior to the start of survey administration, and an annex was sent to the 6% of
principals who had already received the paper instruments. Data analysis has shown that this did not have
an impact upon data quality.

In rating the usefulness of the translation verification process organised by the IEA Secretariat, 92% reported
it was “very useful”, with no participants reporting it was “not useful at all”. Furthermore, all national centres
were able to make full use of the feedback from translation verifiers when preparing the final version of their
instruments. Countries that reported some difficulties with translation mentioned that they resolved these
problems with advice from the OECD Secretariat before translation verification began.

TALIS 2008 Technical Report © OECD 2010

96
chapter 8 Quality Assurance

National Adaptation Forms (NAF) approval, layout verification and ODC verification were performed at the
ISC. Regarding layout verification, 71% of NPMs reported that they made changes in accordance with verifier
suggestions. These suggestions mainly included blank spaces, inappropriate page breaks, changes in the font
sizes and so on. However, 21% of NPMs detected and corrected punctuation or typographical errors that had
not been identified at earlier stages of the verification process.

Ninety-two percent of national centres experienced no problems during printing. However, one country identified
missing pages, and another country had difficulty with the layout of one page in the Principal Questionnaire:
the text had become compressed and it was difficult for both respondents and data entry personnel to match
the check boxes with the corresponding questions.

Conversion to on line data collection

The 16 countries collecting data on line also required verification for ODC. In order to guarantee comparability
of paper and ODC instruments, the ODC check was performed after layout verification of the paper instruments.
Sixty-nine percent of NPMs reported making changes to instruments that were suggested by the verifier; 19%
made changes not identified by the verifier. As with paper layout verification, changes made after verification –
changes that had not been identified by the verifiers – were rather minor, consisting mainly of punctuation or
typographical errors.

Most of the 16 countries participating in ODC had few problems with the TALIS administration. Eighty-eight
percent of NPMs reported that converting the paper questionnaires into on line questionnaires using the IEA
SurveySystem Designer software was “not difficult at all”; the remainder found it “somewhat difficult”.

Survey administration
National centres in 42% of participating countries implemented a procedure for schools to confirm receipt of
the questionnaires and Teacher Tracking Forms. Thirty-eight percent of NPMs reported difficulty in obtaining
a high degree of participation from teachers and principals; 42% had difficulty ensuring questionnaires were
returned on time; and 50% found it difficult to ensure schools returned the completed Teacher Tracking Forms.

Among the 16 countries collecting data on line, 12.5% of NPMs reported that they did not supply fall-back
paper questionnaires, which they had been requested to do in case of technical problems, or in case respondents
refused to complete the survey on line. Thirty-one percent reported severe technical problems that prevented
respondents from completing the questionnaires – this included firewalls and Internet connection problems.
Login problems were reported by 37.5% of countries, ranging from mistyped URLs and difficulties identifying
ID or checksum information.

In ODC countries, 69% of NPMs reported difficulty persuading teachers and principals to participate; 27%
of these NPMs considered that the difficulty was related to the on line mode itself. More specifically, a chief
concern was the perceived lack of anonymity in the on line mode. In 56% of countries SCs reported teachers
and principals who were unwilling or unable to complete the questionnaires on line: within countries this
referred to 1 to 68 principals; 10 to 628 teachers; and 4 to 30 schools. In 89% of these countries fall-back paper
questionnaires were made available, and this strategy proved successful for increasing participation.

The SurveySystem monitor application was developed to enable national centres to track the completion status of
on line questionnaires. Ninety-four percent of NPMs reported that the application was useful, and 75% referred
to it several times per week or daily. NPMs reported that they were able to contact SCs with the response rate for
their school and encourage SCs to remind teachers about the survey in order to improve participation rates.

© OECD 2010 TALIS 2008 Technical Report

Quality Assurance chapter 8
97

Data entry and verification

Data entry was required only for paper questionnaires. Seventy-nine percent of NPMs reported that they
entered data manually. In 42% of these cases, staff from the national centre completed all the data entry;
21% of national centres used an external data company; and 21% used a combination of their own staff and
external staff. The remaining 16% hired and trained students or other staff to work as data punchers. Only two
national centres did not rely on the IEA DPC WinDEM application to enter all data from paper questionnaires:
one of these agreed that they were in error and resolved it with the ISC. The other case reflected a split between
internal staff that used WinDEM and external staff that did not use WinDEM. However, no countries reported
unacceptable levels of error during the double entry of data.

National quality control monitoring programme

Each NPM organised and directed a national quality control monitoring (NQCM) programme. The aim of this
programme was to deliver structured feedback to national centres about survey administration in their country.
NPMs were asked about their national quality control programmes in the final section of the SAQ, but were not
required to supply data collected by National Quality Control Monitors to the ISC.

The IEA Secretariat prepared manual and interview templates for the NQCM programme, which national
centres could use either in the original version or adapt for the situation in their country. One country reported
it did not conduct an NQCM programme, but maintained close contact with SCs. Hungary developed its own
NQCM programme rather than use the template supplied. In Brazil, the survey was administered by an external
agency, which in turn was closely supervised by the national centre, and – as agreed with the ISC – no separate
NQCM programme was run. Two countries did not respond to the section of the SAQ concerning the NQCM
programme.

For the 21 countries that did conduct an NQCM programme, 33% appointed one NQCM. The maximum of
26 NQCMs were appointed in one country. In most countries NQCMs visited an average of 20 schools.9 Of
the 19 countries that used the manual provided by the IEA Secretariat for their NQCM programmes, 16% made
minor adaptations (e.g. removed a question that was not relevant) to the template supplied.

As previously noted, the school visits formed the central part of the data collection quality control programme
and are a primary source of information about how well SCs understood the requirements of their role. Three
countries reported suggestions for changing the NQCM procedures. In some cases the programme of school
visits highlighted the need to provide more training to SCs.

NPMs were asked if they acted on the results of the NQCM reports. In most cases no action was taken,
either because there were no serious problems with the survey or because the NQCM reports reached the
national centre too late to make changes. However, in 14% of cases the NQCM programme revealed general
problems and this information was fed back to the national centre, which in turn contacted schools, supplied
additional information or corrected any misunderstandings. In two countries, for example, the NQCM reported
that completed questionnaires were being centrally stored without the additional protection of an envelope.
The national centres in these countries were able to contact schools and explain the rationale and procedures
for ensuring questionnaire security. In one other country, reports from the NQCM suggested a need to extend
the survey window, and this was achieved in agreement with the ISC.

TALIS 2008 Technical Report © OECD 2010

98
chapter 8 Quality Assurance

Notes

1. As already outlined in Chapter 6, outcomes of the SAQ are reported in different chapters of this report.

2. Schools had to be within a reachable driving distance to allow IQCMs to visit an average of two schools in one working day.
Schools selected for the IQCM programme were not to take part in the national quality control monitoring programme. For more
details about the school selection process, see Section 3.1 of the TALIS Manual for International Quality Control Monitors (MS-05-02).

3. Chapter 9 describes in detail provisions for preserving the anonymity of responses in the international database.
4. There was one country where questionnaires or missing pages had to be replaced for teachers at 7 of the 20 schools visited by the
IQCM.

5. Details on within-school sampling are reported in Chapter 6.

6. See Chapter 4 for more details on translation and layout verification.

7. There was one Slovenian school in Austria that participated in TALIS.

8. Verification steps were: NAF approval, translation verification, layout verification and ODC verification.

9. One country reported that the NQCM visited 6 schools; the maximum of schools visited by NQCMs was 87.

© OECD 2010 TALIS 2008 Technical Report

Quality Assurance chapter 8
99

Table 8.1
Preliminary activities of the School Co-ordinator
Not
Yes No applicable Total Missing
(%) (%) (%) (%) (%)
Did you receive training for your role as school co-ordinator? 50.2 49.6 99.8 0.2
Was the training you received adequate? 47.4 4.4 32.6 84.4 15.6
Were any of the materials to conduct the stud y missing from your shipment? 4.4 87.4 91.8 8.2
Did you experience any difficulties completing the Teacher Listing Form? 8.6 90.4 99.0 1.0
Prior to distributing materials, did you verify adequate supplies of the
87.0 4.0 91.0 9.0
questionnaires and cover letters?
Did you need to arrange replacement questionnaires or cover letters for any
3.4 88.4 91.8 8.2
reason?
Did you correspond with the National Project Manager, read the introductory
page of the survey and familiarise yourself with the survey and return 86.6 5.6 92.2 7.8
procedures prior to distributing materials?

Source: OECD, TALIS Database.

Table 8.2
Questionnaire distribution and returns
Yes No Somewhat Total Missing
(%) (%) (%) (%) (%)
Did you explain the following to each teacher?
Purpose of the survey 84.6 2.2 4.6 91.4 8.6
Estimated time to complete (approximately 45 minutes) 84.6 3.2 3.4 91.2 8.8
Survey return procedures 84.0 2.4 3.8 90.2 9.8
Were any defective questionnaires/cover letters detected and replaced either
2.2 89.8 92.0 8.0
before or after the questionnaires had been distributed?
Did you have any doubts or problems understanding the return procedures? 4.0 87.4 91.4 8.6
Did anyone other than you talk to or otherwise contact the teachers to
23.4 76.0 99.4 0.6
encourage them to participate?

Source: OECD, TALIS Database.

Table 8.3
Use of forms in survey administration
Yes No Total Missing
(%) (%) (%) (%)
Did you distribute the questionnaires/ODC cover letters to the correct teachers in accordance
89.2 2.6 91.8 8.2
with the Teacher Tracking Form?
Does anyone but you have access to the Teacher Tracking Form? 16.0 82.2 98.2 1.8
Does anyone other than you have access to the completed questionnaires and/or tracking
7.8 84.0 91.8 8.2
forms?
Is there anyone listed on the Teacher Tracking Form who is NOT a teacher? 2.6 88.8 91.4 8.6
Did you refer to an official school document (such as school timetable, database, etc.) to
73.0 2.6 75.6 24.4
complete the Reference Class Listing Form?
Did you have any problems completing the Reference Class Listing Form? 10.6 80.4 91.0 9.0

Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

100
chapter 8 Quality Assurance

Table 8.4
Use of specialist translators and reviewers
Yes No
(%) (%)
Were you able to identify a translator who:
Is fluent in English. 100.00 0.00
Has the survey language as mother tongue. 95.83 4.17
Is experienced in your country’s cultural context. 100.00 0.00
Is familiar with survey instruments. 91.67 8.33
Were you able to identify a reviewer who:
Is fluent in English. 91.67 8.33
Has the survey language as mother tongue. 95.83 4.17
Is experienced in your country’s educational context. 100.00 0.00
Is familiar with survey instruments. 100.00 0.00
Is familiar with the subject of the study. 91.67 8.33

Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Quality Assurance chapter 8
101

References
IEA DPC (2007), TALIS Manual for National Project Managers (MS-01-03), Hamburg.

IEA DPC (2007), TALIS School Co-ordinator Manual (MS-03-02), Hamburg.

IEA DPC (2007), TALIS Data Management Manual (MS-04-02), Hamburg.

IEA (2007), TALIS Manual for International Quality Control Monitors (MS-05-02), Amsterdam.

IEA (2007), TALIS Manual for National Quality Control Monitors (MS-06-01), Amsterdam.

IEA (2007), TALIS Main Survey: School Visit Record/Interview with School Co-ordinator, Amsterdam.

Statistics Canada (2007), TALIS Sampling Manual (MS-02-03), Ottawa.

TALIS 2008 Technical Report © OECD 2010

103

Chapter 9

Creating and Checking

the International Database
104 Abstract
104 Overview
104 Data entry and verification at national centres
106 Data checking, editing and quality control at the IEA Data
Processing and Researcher Center
107 Import, documentation and structure check
108 Identification variable and linkage cleaning
108 Resolving inconsistencies in questionnaire data
110 Handling missing data
111 Interim data products
111 Building the international database

TALIS 2008 Technical Report © OECD 2010

104
chapter 9 Creating and Checking the International Database

Abstract

This chapter offers an overview of the strategy used to create the TALIS 2008 international
database (IDB). It describes the data entry and verification tasks employed by the national
centres, the integration of data from the paper and online administration modes, the data-
editing and database creation procedures implemented by the International Study Centre, and
the steps that all involved centres took to confirm the integrity of the international database.

Overview
Creating the TALIS 2008 international database (IDB) and ensuring its integrity required close co-ordination
and co-operation among the International Study Centre (ISC), Statistics Canada, the OECD and the National
Project Managers (NPMs). This chapter describes the data entry and verification tasks that the national centres
undertook, the integration of data from the paper and online administration modes, the data-editing and
database creation procedures the ISC implemented, and the steps that all involved centres took to confirm the
integrity of the international database.

The primary goals were to ensure that any national adaptations to questionnaires were reflected appropriately
in the codebooks and corresponding documentation, that all national information eventually conformed to the
international data structure and coding scheme, and that errors such as logical inconsistencies or implausible
values as a result of the response or data capture process were minimised as much as possible. Quality control
measures were applied throughout the process.

Data entry and verification at national centres

Each national centre was responsible for transcribing into computer data files the information from the principal
and teacher questionnaires administered at the school level. The International Association for the Evaluation
of Educational Achievement’s Data Processing Center (IEA DPC) supplied national centres with the Windows
Data Entry Manager software (WinDEM) and supporting documentation in the TALIS Data Management Manual
(MS-04-02).

In addition, the IEA DPC held a three-day data management seminar in Hamburg, Germany, in November 2006
covering software usage, procedures for national adaptations, and rules and procedures for data entry
(see Chapter 1). The seminar was specifically targeted at the national team member(s) responsible for data
management and liaising with the IEA DPC.

National centres entered responses from the principal and teacher questionnaires into data files created from
internationally predefined codebooks, which contained information about the names, lengths, locations,
labels, valid ranges (for continuous measures or counts) or valid values (for nominal or ordinal questions) and
missing codes for each variable in each of the two questionnaire types. Before data entry commenced, data
managers were required to adapt the codebook structure to reflect any approved adaptations made to the
national questionnaire versions, for example a nationally added response category. These adapted codebooks
then served as templates for creating the corresponding data entry file(s).

In general, national centres were instructed to discard any questionnaires that were unused or returned
completely empty, but to enter any questionnaire that contained at least one valid response. To ensure
consistency across participating countries, the basic rule for data entry in WinDEM required national staff to

© OECD 2010 TALIS 2008 Technical Report

Creating and Checking the International Database chapter 9
105

enter data “as is” without any interpretation, correction, truncation, imputation or cleaning. The resolution of
any inconsistencies remaining after the stage of data entry was delayed until the data cleaning stage (see below).
The rules for data entry were:

• Responses to categorical questions were generally coded as “1” if the first option (checkbox) was used, “2”
if the second option was marked, and so on.
• Responses to “check-all-that-apply” questions were coded as either “1” (checked) or “2” (not checked).
• Responses to numerical or scale questions (e.g. school enrolment) were entered “as is”, that is, without any
correction or truncation, even if the value was outside of the originally expected range, for example if a
teacher reported that he or she spent 80 hours a week on teaching students in school.
• Likewise, responses to filter questions and filter-dependent questions were entered exactly as filled in by the
respondent, even if the information provided was logically inconsistent.
• If responses were not given at all, not given in the expected format, ambiguous, or in any other way conflicting
(e.g. two options in a multiple-choice question were selected), the corresponding variable was coded as
“omitted or invalid”.
• TALIS did not use a separate code to identify “not administered” questions, such as those that were misprinted.
In these highly infrequent cases, the “omitted or invalid” code was used.

When data was entered with WinDEM it was automatically validated. First, the entered respondent ID had to
be validated with a three-digit code, the checksum (generated by WinW3S). A mistype in either the ID or the
checksum resulted in an error message that prompted the puncher to check the entered values. Additionally,
the data verification module of WinDEM identified a range of problems such as inconsistencies in identification
codes and out-of-range or otherwise invalid codes. These potential problems had to be resolved or confirmed
in order to resume data entry.

To check the reliability of the data entry within the participating country, national centres were required to have
at least 100 completed principal and 5% or at least 100 teacher questionnaires entered twice by different staff
members as early as possible during the data capture period. This procedure allowed data managers and the IEA
DPC to identify possible systematic or incidental misunderstandings or mishandlings of data entry rules and to
initiate appropriate reactions, for example, the re‑training of staff within national centres. The acceptable level
of disagreement between the originally entered and double‑entered data was established at 1% or less; above
this level a complete re-entry of data would have been requested. The margin of error observed for all countries
participating in the main data collection was well below this threshold.

Before sending the data to the IEA DPC for further processing, national centres carried out mandatory verification
steps on all entered data and undertook corrections as necessary. The corresponding routines were included
in the WinDEM software and the data files were systematically checked for duplicate identification codes and
data outside the expected valid range or values defined as valid. Data managers reviewed the corresponding
reports, and resolved any inconsistencies and, where possible, corrected problems by looking up the original
survey questionnaires. Additionally, data managers verified that all returned and non-empty questionnaires
were in fact entered and that the availability of data corresponded to the participation indicator variables and
entries on the tracking forms.

While the IEA DPC strongly encouraged every country to meet all standards and rules by using the WinDEM
software for manual data entry, Spain and Ireland used different data entry systems, such as a system routinely
used by an external survey company. Australia and Norway used the online data collection system to enter
a small number of paper questionnaires that were returned late. These countries were nonetheless required
to conform to all specifications established in the international codebooks and to verify their data using the

TALIS 2008 Technical Report © OECD 2010

106
chapter 9 Creating and Checking the International Database

same consistency checks as defined within the WinDEM software. The IEA DPC checked and confirmed the
consistency and quality of the data captured by these alternative means and did not detect any systematic or
incidental issues.

In addition to the data files described above, national centres provided the ISC with detailed data documentation,
including hard copies or electronic scans of all original Teacher Tracking Forms and a report on data-capture
activities collected as part of the online survey activities questionnaire (SAQ). The DPC already had access to
electronic copies of the national versions of all questionnaires and the final national adaptation forms (NAFs) as
part of the layout verification process.

While the questionnaire data was being entered, the data manager at each national centre used the information from
the Teacher Tracking Forms (see Chapter 6) to verify the completeness of the materials. Participation information,
for example whether the concerned teacher had left the school permanently between the time of sampling and the
time of administration, was entered in the WinW3S within‑school sampling software (see Chapter 6).

Data checking, editing and quality control at the IEA Data Processing and
Researcher Center
Once the data were submitted to the ISC, a process referred to as “data cleaning” commenced. The objective of
the process was to ensure that the data adhered to international formats, that information from principals and
teachers could be linked across different survey files, and that the data accurately and consistently reflected the
information collected within each participating country. The IEA DPC went to great lengths to ensure that the
data received from participating countries were internationally comparable and of high quality. The foundation
for quality assurance was laid before the data first arrived at the IEA DPC through the provision of software
designed to standardise a range of operational and data‑related tasks, manuals and training.

For instance, the WinW3S software performed the within‑school sampling operations, strictly adhering to the
sampling rules defined by TALIS. The software also created all necessary listing and tracking forms and stored
school- and teacher-specific information, such as gender and participation status. Furthermore, the WinDEM
software enabled entry of all questionnaire data in a standard, internationally defined format. The software also
included a range of checks for data verification.

A complex study such as TALIS requires a correspondingly complex data-cleaning design. The IEA DPC
developed processing tools in Statistical Analysis Software (SAS) and, where necessary, in Statistical Package
for the Social Sciences (SPSS) for data cleaning. To ensure that programmes ran in the correct sequence, that
no special requirements were overlooked and that the cleaning process was implemented independently of the
persons in charge, the following steps were undertaken:

• Before being used with real data, all data‑cleaning programmes were thoroughly tested using simulated data
sets containing all the expected problems or inconsistencies.

• To document versions and updates, all incoming data and documents were registered in a specific material
receipt database. The date of arrival was recorded, along with any specific issues meriting attention.

• All national adaptations and all detected deviations from the international data structure were recorded in a
“National Adaptation Database” and verified against both the national instruments, the codebooks and the
contents of the data itself. The reports from this process are available for data analysts in the TALIS 2008 User
Guide (OECD, 2009).

• The cleaning was organised according to rules that were strictly and consistently applied to all national data
sets so that deviations in the cleaning sequence were impossible.

© OECD 2010 TALIS 2008 Technical Report

Creating and Checking the International Database chapter 9
107

• All systematic or manual corrections made to data files were implemented and recorded in specific cleaning
reports for consortium and NPM review and approval.
• Once the data cleaning was completed for a participating country, all cleaning checks were repeated from
the beginning to detect any problems that might have been inadvertently introduced during the cleaning
process itself.

Figure 9.1 provides a schematic overview of this iterative process conducted in co-operation with the national
centres. The sequential data-cleaning steps displayed in the exhibit are described in more detail in the following
sections.

Figure 9.1
Overview of iterative data processing at the IEA DPC

Data Database
Structure check Content cleaning
(online and paper)
codebooks Input Output Reports
ID Linkage statistics
Documentation cleaning cleaning documentation

National Centre
(Communication during the cleaning process)

Source: OECD.

Import, documentation and structure check

Data cleaning began with an analysis of the submitted data-file structures and a review of data documentation,
that is, the Teacher Tracking Forms. Most countries submitted all required documentation along with their data,
which greatly facilitated the data checking. The IEA DPC contacted those countries returning incomplete data
or documentation to obtain any missing material. As soon as all required material was received, further data
processing began.

Next, all available codebooks and data were imported from the source files and combined into SAS databases.
Again, each questionnaire type corresponded to one SAS database and one SAS codebook file. In this step, both
the data originating from paper questionnaires and the online questionnaires were combined and checked for
structural agreement (see Chapter 6 on online data collection). In all cases, the data from both administration
modes were structurally equivalent and made use of the same valid and missing codes. The early combination
of these data in the import stage ensured that data resulting from both administration modes were fed through
the same data-processing steps and checks as described in the remainder of this chapter.

The structure check implemented at the IEA DPC looked for differences between the international and the
national file structures. As described above, some countries made structural adaptations to the questionnaires;
the extent and nature of these changes differed greatly across participating countries. While some countries
administered the questionnaires without any changes, except for translations and necessary cultural adaptations,
others inserted questions or options within existing international variables or added entirely new national

TALIS 2008 Technical Report © OECD 2010

108
chapter 9 Creating and Checking the International Database

variables. Given the associated risk of deviating from the international data structure, NPMs wishing to make
such changes followed certain strict rules to allow unequivocal integration of nationally adapted variables for
international comparison.

In general, very few adaptations were made to the international questionnaires. Where necessary, the IEA
DPC modified the codebooks according to the international design or values to ensure that the resulting data
were internationally comparable. For instance, additional national options in multiple-choice questions were
recoded (mapped) in such a way that they adhered to the international code scheme. National variables were
created to hold the original values for later use in national reports

NPMs and data managers received detailed reports on structural deviations together with documentation on
how the DPC resolved the deviations. In a few cases, data were not available for certain variables because
the corresponding question was not administered nationally (see TALIS User Guide). There was no case in
which data had to be removed from the international database because the information was not internationally
comparable.

Identification variable and linkage cleaning

To uniquely identify, track and document each participant and each corresponding questionnaire in a survey,
each record in a data file needs to have a unique identification number. The existence of records with duplicate
identification (ID) numbers in a file implies an error of some kind. In TALIS, if two records shared the same ID
number, and contained exactly the same data, one of the records was deleted and the other remained in the
database. If the records contained different data (apart from the ID numbers), and it was impossible to identify
which record contained the “authentic” data, and if consultations with the NPM did not resolve the matter, both
records were removed from the database. The IEA DPC deleted data in only a very small number of cases. In
addition, only a small number of records were present in both the paper and the online data files.

In TALIS, data collected at the school level were recorded in the principal file. It was crucial that the records
from these files could be linked to the multiple teacher-level records for that school, that is 1:n. In both cases,
the linkage was implemented through a hierarchical ID numbering system and was cross-checked against the
tracking forms and corrected when necessary.

Further ID cleaning focused on consistent tracking of information between the data used for listing, sampling
and tracking in WinW3S and the actual responses in the questionnaire. When necessary, variables pertaining to
teachers’ gender, year of birth, exclusion status and participation status were verified and checked against the
original paper teacher tracking forms.

Where possible, the DPC sought close co-operation with the national centre in resolving ID or linkage
inconsistencies. For this purpose, NPMs and data managers received standardised reports comprising each
identified inconsistency. Once the ID, linkage, participation and exclusion information was finalised, data were
transferred to Statistics Canada and used to calculate participation rates, exclusion rates and, finally, sampling
weights.

Resolving inconsistencies in questionnaire data

After each data file was matched the international standard as specified in the international codebooks, a
series of standard cleaning rules was applied to the files. The process was conducted through the SAS programs
developed at the IEA DPC, identifying and in many cases automatically correcting inconsistencies in the data.
The DPC documented details about all cleaning checks, procedures and actions applied to the data, sent these
to the national centres and explained them during the fourth NPM meeting in October 2008.

© OECD 2010 TALIS 2008 Technical Report

Creating and Checking the International Database chapter 9
109

Filter questions, which appear in certain positions in the questionnaires, were used to direct the respondent
to a particular question or section of the questionnaire. Filter questions and their dependent questions were
treated automatically in most cases. If the filter question contained a value and the dependent questions were
validly skipped, dependent variables were coded as “logically not applicable”. If a response to a filter question
was equivalent to “no”, meaning that the dependent questions were not applicable, and yet the dependent
questions were answered in an unambiguous pattern, the dependent variables were set to “logically not
applicable” regardless of the value originally recorded in the dependent variable. Questions 4 and 5 in the
Teacher Questionnaire (TQ) and questions 8 and 9 in the Principal Questionnaire (PQ) constituted exceptions
to this general rule.

For weighting purposes and calculation of the teacher multiplicity factor (WGTADJ4), a special treatment for
TQ 4/5 was agreed upon with Statistics Canada. If TQ-4 was “yes” (1) and TQ-5 was omitted or zero (0), then
TQ-4 was recoded to “no” (2) and TQ-5 to “logically not applicable”. If TQ-4 was “no” (2) and TQ-5 was
zero (0) or one (1) then TQ-5 was recoded to “logically not applicable”. If TQ-4 was “no” (2) but TQ-5 was
two (2) or more then TQ-4 was recoded to “yes” (1).

Split variable checks were applied to “yes/no” lists and “check-all-that-apply” questions where the responses
were coded into several variables. For example, question 11 in the Teacher Questionnaire listed a number
of developments and asked teachers to mark whether they participated in them with “yes”. Occasionally,
teachers marked either some “yes” and “no” boxes or just the “yes” boxes but also left some of the “no” boxes
unchecked, resulting in “omitted” values in the data file. Because in these cases it could be assumed that the
unmarked boxes actually meant “no”, the corresponding variables were imputed.

The individual responses to percentage questions were summed and, if they fell outside of the 90 to 110 range
or if any individual values were larger than 100, they were set to “omitted”.

Variables with implausible numerical values were set to “omitted”. For example, question 38 in the Teacher
Questionnaire asked about the average number of students in the target class. Values that exceeded 100 were
set to “omitted”.

Finally, variables within and across data files were verified against one another to identify and resolve
inconsistent response patterns or multivariate outliers. For example, TQ 12 asked for the total number of days
spent on professional development, while TQ 13 asked how many of these days were compulsory. Clearly, the
number given for TQ 13 should not exceed the number given for TQ 12; values for TQ 13 were set to the value
recorded in TQ 12 in these cases.

The number of inconsistent or implausible responses in the data files varied from one country to another, but
no national data were completely free of inconsistent responses. Each problem was recorded in a database,
identified by a unique problem number along with a description of the problem and the automatic action
taken by the programme or the manual action taken by DPC staff. Issues that could not be corrected using
systematic rules were reported to the NPM so that original data-collection instruments and tracking forms
could be checked to trace the source of the inconsistency. Whenever possible, staff at the IEA DPC suggested
a solution and asked the NPMs either to accept it or to propose an alternative. Data files then were updated to
reflect the agreed-upon solutions. Both systematic corrections as well as those apparent on a case‑by‑case level
were applied directly in SAS program syntax and carried out automatically for each cleaning run.

Where the NPM could not solve problems by inspecting the instruments and forms or could not suggest a
satisfying solution or explanation, final cleaning rules were defined by the consortium. The following systematic
content edits were agreed upon by the IEA DPC and OECD and documented for use by the NPM.

TALIS 2008 Technical Report © OECD 2010

110
chapter 9 Creating and Checking the International Database

• PQ-25, PQ-33, TQ-11, TQ-20, TQ-33: For lists that were partially answered with “yes”, “no” and “omitted”,
all omitted responses were recoded to “no”.
• PQ-33/34, PQ-35/36, TQ-4/5, TQ-11 (part a vs. b), TQ-17 (part a vs. b): The dependent variables were set to
“logically not applicable” if the filter question was answered negatively (“no”).
• PQ-8/9: The dependent variables were set to “logically not applicable” if the filter question was answered
negatively (“public”).
• PQ-17, TQ-41: The entire set of variables was set to “omitted” if the sum of percentages fell outside of 90 to
110 or if any individual variable was larger than 100.
• PQ-11A: The variable was set to “omitted” if the answer was 0 in the questionnaire.
• TQ-12/13: Set affected value to “omitted” if the number of days was higher than a plausible maximum within
18 months (i.e. 1.5*365 or ~ 550).
• TQ-12/13: Adjusted number of compulsory days to number of days of professional development attended if
number of compulsory days was higher than number of days of professional development attended.
• TQ-38: The variable was set to “omitted” if enrolment was 0 or larger than 100.
• TQ-8A: The variable was set to “omitted” if the number of hours was greater than 50.
• TQ-8A: The variable was set to “omitted” if the number of hours was 0.
• TQ-8A-D: The variables were set to “omitted” if the sum of hours was greater than 80.
• ITBIRTHY (Listing information on respondents’ year of birth collected prior to questionnaire administration):
the variable was set to “omitted” if it was outside of a plausible range of 1935 to 1989.
• Gender [TQ-1 vs. ITSEX (Listing information on respondents’ gender collected prior to questionnaire
administration)]: a) believe questionnaire information and substitute listing information gender in case it
is missing or inconsistent b) impute missing questionnaire value from listing if questionnaire variable was
omitted.
• Age (TQ-2, ITBIRTHY): a) believe questionnaire information and delete listing information if inconsistent;
b) impute missing questionnaire value from listing form.

In some instances in which a clear and unambiguous decision was not possible, the data remained unchanged.

Handling missing data

During the TALIS data entry at the national centres using WinDEM, two types of entries were possible: valid
data values and missing data values. Data entry staff were able to assign either the valid values or a value for
“omitted/invalid.” Later at the IEA DPC, additional missing values were applied to the data for further analyses
and to differentiate response behaviour.

In the international database, two missing codes were used:

• Omitted / invalid (9): the respondent had the opportunity to respond to the question, but did not do so or
provided an invalid response. The value was also assigned in extremely rare cases where questions were
misprinted or otherwise not legible.
• Logically not applicable (6): the respondent answered a preceding filter question in a way that made the
following dependent questions not applicable to him or her. This value was assigned during data processing
only.

© OECD 2010 TALIS 2008 Technical Report

Creating and Checking the International Database chapter 9
111

Interim data products

Building the TALIS international database was an iterative process during which the IEA DPC provided the
OECD and NPMs with a new version of data files whenever a major step in data processing was completed. This
process guaranteed that NPMs had a chance to review their data and to run additional plausibility and statistical
checks to validate the data. Data products sent out by the IEA DPC to the OECD and each NPM included the
teacher and principal data file as well as data summaries. All interim data were made available to the OECD in
full whereas each participating country received its own data only.

The first version of cleaned and weighted data was sent to the OECD at the end of August 2008, two months
after all data have arrived at the IEA DPC. In this data, all known identification, linkage and content issues
were resolved. Estimation weights and variables facilitating variance estimation were also included. The
OECD used these to produce the first set of draft tables for the international report and presented them at the
fourth NPM meeting in Dublin, Ireland in October 2008. Prior to this meeting, all NPMs received a version
of their own cleaned and weighted data, giving them a chance to review their data and the tables produced
by the OECD.

During the fourth NPM meeting and for two weeks following it, NPMs were able to raise any issues concerning
their data that had thus far gone unnoticed. This resulted in a second, updated data version that concluded the
field work and that was sent to the OECD and NPMs in November 2008.

In February 2009 NPMs received an update of their data, reflecting minor issues that had been raised after the
November 2008 data release and final cleaning. The OECD and its partners used this version of the data to
produce the updated, final tables for the international report.

All interim data products were accompanied by detailed data processing and weighting documentation,
codebooks, and summary statistics. The latter contained weighted univariate statistics for all questionnaire
variables for each country. For categorical variables, which represent the majority of variables in TALIS, the
percentages of respondents choosing each of the response options were displayed. For numeric or count
variables, various descriptive measures were reported. These included the minimum, the maximum, the
mean, the standard deviation, the median, the mode, percentiles and quartiles. For both types of variables,
the percentages of missing information due to respondents omitting or not reaching a particular question were
reported. These summaries were used for a more in-depth review of the data at the international and national
levels in terms of plausibility, unexpected response patterns, suspicious profiles and so on.

Building the international database

For the draft and final IDB, data cleaning at the IEA DPC ensured that information coded in each variable was
in fact internationally comparable, that national adaptations were reflected appropriately in all concerned
variables and that all records could be successfully linked across the two levels.

The interim data products described above and the draft and final (public‑use) international databases had
two key differences. First of all, all interim products included one record for each sampled unit (school or
teacher) even if the corresponding questionnaire was not returned or returned empty. The draft and final IDB,
by contrast, included only records that satisfied the sampling standards. Data from those units that either did
not participate or did not pass adjudication (for example, because within-school participation was insufficient)
were removed. Secondly, in order to protect the confidentiality of respondents, disclosure avoidance measures
were applied at the international level i) consistently for all countries and ii) concerning only specific national
datasets. These measures were implemented for all data versions and exports of the IDB for use by all other
countries and public users.

TALIS 2008 Technical Report © OECD 2010

112
chapter 9 Creating and Checking the International Database

The measures applied to all international-level datasets included:

• The teacher (IDTEACH) and school identifiers (IDSCHOOL) were scrambled and did thus not match those
used during data collection; however, the structural link between the school and teacher level (the variable
IDSCHOOL in the teacher file and the first four digits of any IDTEACH) was maintained. For each country,
unique matching tables were created and made available to authorised individuals.
• Variables used purely for the stratification of the teacher sample, i.e. birth year (ITBIRTHY), gender (ITSEX)
and main teaching domain (ITDOMAIN) were removed. Only the gender (BTG01) and age group (BTG02)
variables as collected in the questionnaire were retained.
• Variables used purely for stratification of schools were removed (IDSTRATE and IDSTRATI) to avoid the
identification of geographical or organisational groups. It should be noted that the stratum information is
mostly of interest for national-level analysis and was of course made available to the concerned country.
Experience showed that researchers from other countries might also wish to conduct analysis by stratification,
in which case the stratification variables could be requested directly from the country.
• Information used in the calculation of final sample and replicate weights was removed (for the school level:
WGTFAC1 and WGTADJ1; for the teacher level: WGTFAC1, WGTADJ1, WGTFAC2, WGTADJ2, WGTADJ3,
and WGTADJ4) as these could allow the identification of stratification cells.
• Replication zone and unit variables (BRRSZONE, BRRSREP, BRRTZONE, and BRRTREP), which could cause
indirect identification of schools, were also dropped from public-use micro-data.

TALIS BPC members from Belgium (Fl.), Denmark, Ireland, Italy, Malta and the Netherlands requested several
confidentiality measures, and these were applied to the respective national datasets. In these cases, the original
variables were set to “not administered” in the IDB and in some cases derived variables were created that
include grouped (binned) values. All measures and their resulting derived variables are described in detail in
the TALIS User Guide.

Iceland decided to withdraw all data from the IDB. It is available directly from the country only (see Appendix B).

All data for the Netherlands were retained as part of the IDB following agreements with the BPC. However,
all weight variables for the Netherlands were set to 0 because the achieved participation rates were too low to
allow population estimates and comparisons.

Following the data release policy and confidentiality agreements between each NPM and OECD, a draft IDB
that included data from all participating countries except Iceland was made available. This occurred in early
March 2009, prior to the publication of the international report Creating Effective Teaching and Learning
Environments: First Results from TALIS in June 2009. This enabled countries to replicate the results presented in
the draft chapters of the international report (with the exception of estimates for Iceland) and this data version
was also used in an international database training session held by IEA DPC staff in Hamburg, Germany, in
April 2009.

The final, public-use international database was scheduled for release in late 2009 and was supplemented by
full documentation in the TALIS 2008 User Guide (OECD, 2009). It is a unique resource for policy-makers and
analysts and contains data from representative samples of schools and ISCED Level 2 teachers in 23 countries
across 4 continents.

© OECD 2010 TALIS 2008 Technical Report

Creating and Checking the International Database chapter 9
113

References
IEA DPC (2007), TALIS Data Management Manual (MS-04-02), Hamburg.

OECD (2008), TALIS 2008 Data Processing and Cleaning Documentation, OECD, Paris.

OECD (2009), TALIS 2008 User Guide for the International Database, OECD, Paris.

TALIS 2008 Technical Report © OECD 2010

115

Chapter 10
Estimation Weights,
Participation Rates
and Sampling Error
116 Abstract
116 Overview
116 Elements of the estimation weight (final weight)
117 School base weight (school design weight)
117 School non-response adjustment factor
118 Final school weight
118 Teacher base weight (teacher design weight)
119 Teacher non-response adjustment factor
119 Teacher adjustment factor for incidental exclusions
119 Teacher multiplicity adjustment factor
120 Final teacher weight
120 Participation rates
121 Participation rates for schools
121 Participation rate for teachers
122 Overall participation rates
122 Reporting participation rates
122 Meeting participation rates standard for TALIS
123 Sampling error with Balanced Repeated Replication
123 Creating replicates for Balanced Repeated Replication
125 Estimating the sampling error
125 Design effect and effective sample size

TALIS 2008 Technical Report © OECD 2010

116
chapter 10 Estimation Weights, Participation Rates and Sampling Error

Abstract

This chapter covers three important aspects of the quality of the TALIS outcomes: weighting of
the data to produce estimates, participation rates and an estimation of the sampling error. It
begins by detailing how each component of the final estimation weight is defined and how those
components are assembled into the final estimation weight, before describing participation
rates and how they were computed.

Overview
Although the international sampling plan was prepared as a self-weighting design (with each individual having
the same final estimation weight), field conditions rendered that plan impossible. In the end, each national
sampling plan is a stratified multi-stage probability sampling plan with unequal probabilities of selection.
Iceland and Malta are the two exceptions as a census of schools and teachers was organised in those two
countries.

The following section details how each component of the final estimation weight is defined and how
those components are assembled into the final estimation weight. Following this, there is a description of
the participation rates and how they were computed. Results for each participant are given in this section.
Because of the unequal weights and the structure of the samples, sampling error was estimated using the
design and weights. Any other method would have yielded severely biased estimates of the sampling error.
Estimating sampling errors correctly is often a daunting task, but simple and approximately unbiased methods
are available. TALIS opted for Balanced Repeated Replication (BRR) for its statistical properties (consistency,
asymptotic unbiasedness), its portability (one formula fits all types of parameter estimates) and because it is
comparatively easy to compute. Finally, the chapter explains how the replicates were created and how the BRR
estimates of the sampling error were computed. These estimates of the sampling error are a key element of the
statistical quality of survey outcomes.

A more detailed description of the survey design and its implementation can be found in Chapter 5 of this
report, the TALIS Sampling Manual (TALIS reference number MS-02), the National Project Manager Manual
(TALIS reference number MS-01) and the School Co-ordinator Manual (TALIS reference number MS-04).

Elements of the estimation weight (final weight)

The statistics produced for TALIS are derived from data obtained through samples of schools, school principals
and ISCED Level 2 teachers. For these statistics to be meaningful for a country, they need to reflect the whole
population from which they were drawn and not merely the sample used to collect them. The process of going
from the sample data to information about the parent population is called estimation. When the sample is
equiprobable, unstratified and unclustered, then simple sample averages may suffice as estimates of population
averages (e.g. the average number of ISCED Level 2 teachers per school). However, sample counts do not suffice
as estimates of population totals (e.g. the total number of ISCED Level 2 teachers in a country).

The estimation weight (or final weight) is a device that allows the production of country-level estimates from
the observed sample data. The estimation weight indicates how many population units are represented by a
sampled unit. The final weight is the combination of many factors reflecting the probabilities of selection at the
various stages of sampling and the response obtained at each stage. Other factors may also come into play as

© OECD 2010 TALIS 2008 Technical Report

Estimation Weights, Participation Rates and Sampling Error chapter 10
117

dictated by special conditions so as to maintain unbiased estimates (e.g. adjustment for teachers working in
more than one school).

Basically, final weights are the product of a design or base weight and of one or many adjustment factors; the
former is the inverse of the selection probability, the latter compensates for non-response and other random
occurrences that could – if not accounted for – induce biases in the estimates. These design weights and
adjustment factors are specific to each stage of the sample design and to each explicit stratum used by the
design. In cases where countries adapted the general sample design of TALIS to their own conditions, the
estimation weights have to conform to the national adaptations.

Following are the conventional notations that are used in this chapter: the letters h, i, and j are used as
subscripts, the lower case letters k, l, m, n, r, t refer to the sample, while the upper case letters H, M, N refer to
the population:

• in each participating country, there are H explicit strata; the index h =1,…, H points to the explicit stratum;
if no explicit strata were defined, then H = 1;
• in each explicit stratum, a sample of size nh schools was drawn from the Nh schools comprising stratum h;
the index i =1, …, nh points to the ith sampled school in stratum h;
• each school i =1, …, nh within the explicit stratum h has a measure of size (MOS) noted MOShi; the sum of
the individual measures of size is noted MOSh
• in each responding school, the number of eligible ISCED Level 2 teachers is noted Mhi
• in each responding school, a sample of mhi teachers was drawn; if the selected school is large enough, mhi =
20 by design; the index j =1, …, mhi points to the teachers; mhi may be different from 20 if local conditions
dictate that the sample size should be different (e.g. if the MOS is 18, all teachers are selected and mhi = 18).

School base weight (school design weight)

The first stage of sampling is the school sample; in most countries school sampling followed a systematic random
sampling scheme with probability proportional to size. Thus, a school base weight is needed to represent the
first stage of sampling. If a census sample of schools was implemented in a country or in an explicit stratum of
a country, then the school base weight is set to 1.

Using the notation given above, for each school i =1, …, nh and each explicit stratum h =1, …, H, the school
base weight is given by:

MOSh
WGTFAC 1hi =
nh × MOShi

In Iceland and Malta, since all schools were selected, there is only one stratum (h = 1) and WGTFAC1hi =1, for
all i = 1, ..., n.

School non-response adjustment factor

In spite of efforts to secure the participation of all selected schools, some may have been unable or unwilling
to participate. The schools represented by the non-participating schools must somehow be represented by
those that did. Assuming that non-response happens for reasons unrelated to the topic of the study (also
referred to as “missing completely at random”), a non-response adjustment factor is required, within each
explicit stratum.

TALIS 2008 Technical Report © OECD 2010

118
chapter 10 Estimation Weights, Participation Rates and Sampling Error

For each explicit stratum h =1,…, H , if rh schools participated in TALIS out of the nh selected schools, and if dh
schools are found closed or out-of-scope, then for each school i =1, …, nh the non-response adjustment factor
is given by:
 nh − dh
 r , for participating schools
 h
WGTADJ 1hi = 1, for closed or out-of-scope schools

 0, for non- participat ing schools

In Iceland and Malta, the school non-response adjustment factor WGTADJ1i = (N-d) /r, for all schools i = 1,
..., N, where N is the total number of schools listed as in-scope in the country, d is the number of schools later
found to be closed or out-of scope and r is the number of participating schools.

Final school weight

As described earlier, the school estimation weight is the product of the school base weight and the school non-
response adjustment factor; it should be used for the estimation of school-related parameters.

The final school weight (school estimation weight) for each participating school i =1, …, rh and each explicit
stratum h =1, …, H is given by:

SCHWGThi = WGTFAC 1hi ×WGTADJ 1h

MOSh n − dh
= × h
nh × MOShi rh
In Iceland and Malta, the final school weight is SCHWGThi = 1 × (N-d) / r = (N-d) / r.

Teacher base weight (teacher design weight)

In some countries, or in some smaller schools, school principals also have teaching duties. In an effort to
maintain the response burden at a tolerable level, those individuals were considered incidental exclusions
while remaining in the scope of the survey. In Iceland and Malta, given the number of teachers in the country,
those teachers who had participated in the TALIS Field Trial could be excused for the same reason. These groups
of teachers were given exclusion codes of 5 (noted NEXCL5) and 6 (noted NEXCL6) respectively at the time of
compiling the school list in WinW3S. Both groups need to be accounted for in the estimates.

In a school where these exclusions happened, the sample of teachers was drawn from a reduced list. Let
Mhi = Mhi – NEXCL5hi – NEXCL6hi be the reduced size of the list used for teacher sampling, where Mhi is the total
number of ISCED Level 2 teachers eligible for sampling, as defined earlier. Note that when the measure of size
used was the number of eligible ISCED Level 2 teachers, then Mhi should be very close to MOShi.

In each participating school, a systematic random sample with equal probability of ISCED Level 2 teachers was
selected. The nominal sample size within each school was set at mhi = 20, but the number of in-scope ISCED
Level 2 teachers at each selected school could require that the size of the teacher sample be modified. The teacher
base weight (or design weight) is used to bring the individual teachers’ information to the level of their school.

For each selected teacher j =1 ,…, mhi of school i =1,…, nh in explicit stratum h =1,…, H, the teacher base
weight is given by:
M hi−
WGTFAC 2hij =
mhi

© OECD 2010 TALIS 2008 Technical Report

Estimation Weights, Participation Rates and Sampling Error chapter 10
119

Teacher non-response adjustment factor

Unfortunately, not all selected teachers were able or willing to participate in TALIS; the teachers that choose not
to participate must be represented by the participating ones. Under the assumption of missing at random, that
is achieved by the teacher non-response adjustment factor.

In each participating school i =1,…, rh of each explicit stratum h =1,…, H, there are three kinds of sampled
teachers: those who responded (noted thi), those who did not respond but who are still at the selected school
(noted qhi), and those who left the school permanently after the sample had been selected. Then, for each
selected teacher j =1 ,…, mhi, the teacher non-response adjustment factor is given by:
 t hi + qhi
 t , for responding teachers
 hi
WGTADJ 2hij =  0, for non-responding teachers

 1, for those who left the school permanently

While the “teachers who left the school permanently” will not provide data to most of the estimates of interest,
they still carry a positive weight as they represent those other “teachers who have left school permanently” who
are not in the sample.

Teacher adjustment factor for incidental exclusions

Since some teachers were excluded from sampling while they were in-scope, they need to be represented by
the sample. An adjustment factor is required to account for those so-called incidental exclusions.

For each teacher j = 1, ..., mhi in participating school i =1, …, rh in explicit stratum h =1, …, H, the teacher
adjustment factor for incidental exclusions is given by:
M hi
WGTADJ 3hij =
M hi−
In this adjustment factor, the numerator is the full teacher list and the denominator is the reduced list from which
the sample was actually selected. Mhi = Mhi when there were no incidental exclusions and then WGTADJ3hij = 1
for all sampled teachers.

Teacher multiplicity adjustment factor

Some teachers work in more than one school at ISCED Level 2. Since the lists of teachers were drawn
independently, those teachers could have been listed more than once. Moreover, the samples of teachers being
independent between schools, selecting the same teacher more than once was possible (though in practice not
very likely). An adjustment is needed to account for the number of schools in which a given teacher works and
this information was collected through the teacher questionnaire. For most teachers, the adjustment factor is 1.
For the others, it is the inverse of the number of schools in which they teach.

For each responding teacher j =1, …, thj, in each participating school i =1, …, rh, in explicit stratum h =1 , …, H,
the teacher adjustment factor for multiplicity is given by:
 1
 nbr _ schools , for teachers teaching in more than 1 school
WGTADJ 4hij = hij
 1, for teachers teaching in 1 school


where nbr_schoolshij is the number of schools where teacher j teaches.

This factor is set to 1 for teachers who have left the school permanently.

TALIS 2008 Technical Report © OECD 2010

120
chapter 10 Estimation Weights, Participation Rates and Sampling Error

Final teacher weight

The final teacher weight (estimation weight) is the product of the teacher base weight, the three adjustment
factors associated with each participating teacher, and of the final school weight. All estimates pertaining to the
populations of teachers should use the final teacher weight.

For each participating teacher j =1,…, thi, in each participating school i =1, …, rh, in explicit stratum h =1, …,
H, the final teacher weight is given by:

TCHWGThij = SCHWGThi × WGTFAC 2hij × WGTADJ 2hij × WGTADJ 3hij × WGTADJ 4hij
 MOSh n − dh   Mhi−   Mhi   t hi + qhi   1 
=  × h × × − ×  ×  
 nh × MOShi rh   mhi  M
  hi
  t
  hi  
 nbr _ schoolshij 


and for each teacher who has left school permanently, the final weight is given by:

TCHWGThij = SCHWGThi × WGTFAC 2hij × WGTADJ 2hij × WGTADJ 3hij × WGTADJ 4hij
 MOSh n − dh   Mhi−  M 
=  × h  ×   × 1 ×  hi−  × 1
  M 
 nh × MOShi rh   mhi   hi 

It can be remarked that, in the simplest of cases, the sampling design prepared for TALIS yields equal weights for
all teachers. Assuming that the measure of size is the full list of eligible ISCED Level 2 teachers (MOShi = Mhi), that
the sample size of 200 schools is distributed among the explicit strata proportionally to the number of teachers
in each stratum (nh = 200 × MOSh / MOS• where MOS• is the total number of eligible ISCED Level 2 teachers
in the country), that samples of 20 teachers can be selected from every selected school, that the school listings
contain nobody but in-scope teachers, that no incidental exclusion occurred, that each selected school and
teacher participates, that each teacher teaches in only one school, then the final teacher weight is effectively the
same for all the teachers in the sample:

TCHWGThij = SCHWGThi × WGTFAC 2hi × WGTADJ 2hi × WGTADJ 3hij × WGTADJ 4hij
 MOSh   Mhi  MOSh
=   ×  
n × MOS  20  × (1) × (1) × (1) = n × 20
 h hi    h

MOS• MOSh MOS •

= =
200× MOSh 20 4000

for h = 1, …, H; i = 1, …, nh; j = 1, …, mhi .

Participation rates
The quality requirements for TALIS translate into participation rates (response rates) for schools and
for teachers. Reaching these levels of participation does not preclude that some bias may be present
in the results but should minimise the negative impact of non-response biases. As TALIS is one of the
first large-scale international surveys of active teachers, little is known of “reasonable” response rates for
this population. Hence, when compared to large-scale student-level international surveys on education
(e.g. Programme for International Student Assessment [PISA], Progress in International Reading Literacy
Study [PIRLS], Trends in International Mathematics and Science Study [TIMSS]), the requirements for TALIS
may appear somewhat low.

© OECD 2010 TALIS 2008 Technical Report

Estimation Weights, Participation Rates and Sampling Error chapter 10
121

Participation rates for schools

The minimum school participation rate was set at 75% after replacement. Although replacement schools could
be called upon as substitutes for non-responding schools, NPMs were encouraged to do all they could to obtain
the participation of the schools in the original sample. As the number of replacement schools increases, the
sample loses its probabilistic features and becomes increasingly “purposive”. This can undermine the reliability,
validity and interpretability of the country’s results.

Responding schools that yield at least 50% of responding teachers will be considered as “participating” schools;
schools that fail to meet that threshold will be considered as “non-participating” even though the number of
responding teachers may be enough to contribute to some of the analyses.

Countries that experience less than 75% school participation after replacement had to demonstrate convincingly
that their sample was not significantly biased.

The unweighted school participation rate is computed as :

H rh H

∑∑1
h =1 i =1
∑r
h =1
h
UNWSCPART = H nh− dh
= H

∑ ∑1 ∑ (n
h =1 i =1 h =1
h − dh )

where, rh, nh and dh are as defined earlier. This represents the crude proportion of schools that achieved at least
50% response from their sample of teachers.

The weighted school participation rate is computed as the proportion of the population of teachers accounted
for by participating schools. To better display how the weighted rates are computed, TWGTinSCL is defined as
the “teacher weight within his or her school”, adjusted within school:

TWGTinSCLhij = (wgtfac 2hij × wgtadj 2hij × wgtadj 3hij × wgtadj 4hij )

Then, the weighted school participation rate is defined as:

H rh thi

∑∑ ∑ WGTFAC 1
h =1 i =1 j =1
hi
× TWGTinSCLhij
WTDSCPART = H rh thi

∑∑ ∑ SCHWGT
h =1 i =1 j =1
hi
× TWGTinSCLhij

Note that the numerator is not adjusted for school non-response while the denominator is. In both the numerator
and denominator, the full estimated number of teachers in the school is used.

Both rates were computed once over the complete set of participating schools (after replacement) and once
over the subset of participating schools in the original selection (before replacement).

Participation rate for teachers

TALIS expected that at least 75% of selected teachers in participating schools (original sample or replacement
schools) would take part in the assessment.

Teacher participation was calculated over all participating schools, whether the schools were in the original
sample or used as replacements, and thus the participation rate for teachers is a requirement only at the national
level, not at the school level.

TALIS 2008 Technical Report © OECD 2010

122
chapter 10 Estimation Weights, Participation Rates and Sampling Error

The unweighted teacher participation rate is defined as :

H rh thi H rh

∑∑ ∑1 ∑∑ t
h =1 i =1 j =1 h =1 i =1
hi

UNWTPART = H rh mhi
= H rh

∑∑ ∑1 ∑∑ m
h =1 i =1 j =1 h =1 i =1
hi

This gives the crude ratio of the number of responding teachers in participating schools with respect to the
expected sample size from participating schools.

Again, to better show the structure of the participation rate, a “responding teacher weight” in a school is defined
as RESPinSCLhij:

RESPinSCLhij = wgtfac 2hij × wgtadj 3hij × wgtadj 4hij

Then the weighted teacher participation rate is given by:

H rh thi

∑∑ ∑ WGTFAC 1
h =1 i =1 j =1
hi
× RESPinSCLhij
WTDSCPART = H rh thi

∑∑ ∑ WGTFAC 1
h =1 i =1 j =1
hi
× TWGTinSCLhij

Note that the numerator is not adjusted for teacher non-response while the denominator is. In both the numerator
and denominator, schools are not adjusted for non-response.

Overall participation rates

The overall unweighted and weighted participation rates are the product of the respective school and teacher
participation rates.

Reporting participation rates

Both weighted and unweighted participation rates, with and without replacement schools were produced. As
well, the weighted and unweighted participation rates for teachers were computed.

The analytical results for each country were annotated (OECD, 2009), based on whether the response rate
requirements were adequately met.

Meeting participation rates standard for TALIS

Each country’s data received one of three response ratings: good, fair or poor. The “good” rating means that the
country’s data were included in international comparisons. The “fair” rating means that the country’s data were
a candidate for not being reported in international comparisons because the participation rate after replacement
was less than 75%. However, in most cases, evidence was provided by the countries concerned that non-
response bias was negligible. Finally, the “poor” rating means that the country’s data were not included in the
international comparisons. The TALIS Board of Participating Countries made the final decision on whether to
include the country’s data in international comparisons while taking into account various other factors.

The ratings depend on participation rates before and after replacements and on the apparent severity of the
non-response biases; these ratings are summarised in Table 10.1.

Table 10.2 gives the unweighted school participation rates, before and after replacement of non-participating
schools, the unweighted teacher participation rate and the unweighted overall participation rates by country.

© OECD 2010 TALIS 2008 Technical Report

Estimation Weights, Participation Rates and Sampling Error chapter 10
123

This table is a crude measure of the efficiency or effectiveness of the data collection activities. In all, nearly 74
000 teachers participated, that is, 78% of all teachers sampled. Table 10.3 shows weighted participation rates
and thus the estimated proportion of each national population of teachers who took part in TALIS. With this in
mind, “TALIS participation rates” might not convey much meaning.

Table 10.3 gives the weighted school participation rates before and after replacement of non-participating
schools, the teacher participation rate in participating schools and the overall participation rate for each
country.

Sampling error with Balanced Repeated Replication

Surveys with complex designs like TALIS require special attention when it comes to estimation, especially
estimation of the sampling error. Both the survey design and the unequal weights are needed to obtain
(approximately) unbiased estimates of sampling error. Failing to do so can lead to severe underestimation
of the sampling error. While exact formulae exist in theory for stratified PPS sample designs, the required
computations become practically impossible as soon as the number of primary units selected per stratum
exceeds two. In those cases, approximate solutions have been proposed over the years. An important class
of solutions is that of resampling or replication. Interpenetrating sub-samples (Mahalanobis), Balanced Half-
Samples or Balanced Repeated Replication (McCarthy, Fay), the Jackknife (Quenouille, Tukey, Durbin, Frankel),
and the Bootstrap (Efron) are the best known examples of replication methods (see, for example, Lohr [1999],
Rust and Rao [1996], or Wolter [2007] for a review of these methods).

The Balanced Repeated Replication (BRR) was adopted for the estimation of the sampling error of the estimates
produced for TALIS. This is similar to what was done for PISA, for example (OECD, 2008). BRR is a replication
method suited to sample designs where exactly two primary sampling units (PSUs) are selected in each stratum.
The principle of BRR is the following: each of the two PSUs can provide an unbiased estimate of the total
(or other parameter of interest) of its stratum; if the sampling design comprises H strata, there are then 2H
possible unbiased estimates of the parameter of interest by combining either PSU from each of the H strata.
The sampling error of the estimate of the parameter of interest can be directly computed by comparing each of
the 2H estimates with their mean, as one usually does in simple statistics. Even with moderate values of H, the
number of unbiased estimates may be quite large (e.g. 25=32, 210=1 024, 220=1 048 576,…). BRR provides a
way to extract from the complete set of 2H possible replicates a much smaller subset that will give the very same
measure of sampling error as the full set would.

Creating replicates for Balanced Repeated Replication

BRR was developed for sample designs using only two PSUs per stratum. Clearly, none of the countries
participating in TALIS implemented such a sample design. Fortunately, the implemented sample design can
be approximated by a superimposed “BRR-ready” sample plan. Listing the schools in the order in which
they appear on the sampling frame, the participating schools (of the original sample or the replacements)
are paired within explicit strata and each pair is dubbed “pseudo stratum” or “zone”. If the number of
participating schools in an explicit stratum is odd, then a triplet is formed with the last three schools. The
pairs (or triplets) are then numbered sequentially from 1 to G, spanning the whole sample. Within each
pseudo stratum or zone, each school is assigned a random pseudo PSU number 1 or 2 (or 3 for a triplet) as
depicted in Table 10.4.

As with the jackknife repeated replication, one of the two pseudo PSUs will be dropped and the remaining
pseudo PSU will see its weight doubled and be used to compute an estimate of the parameter of interest.
Rather than randomising which pseudo PSU will be dropped, a special matrix (of order 4t ) of +1’s and –1’s –

TALIS 2008 Technical Report © OECD 2010

124
chapter 10 Estimation Weights, Participation Rates and Sampling Error

the so-called Hadamard matrix – indicates which pseudo PSU is to be kept from each pseudo stratum in BRR,
associating the +1’s with the PSUs numbered 1 and the –1’s with the PSUs numbered 2. For example, the
Hadamard matrix of order 8 can be written as:

+1 +1 +1 −1 +1 −1 −1 −1
 
−1 +1 +1 +1 −1 +1 −1 −1
−1 −1 +1 +1 +1 −1 +1 −1
 
+1 −1 −1 +1 +1 +1 −1 −1
Hadamard8 = 
−1 +1 −1 −1 +1 +1 +1 −1
 
+1 −1 +1 −1 −1 +1 +1 −1
+1 +1 −1 −1 −1 −1 +1 −1
 
−1 −1 −1 −1 −1 −1 −1 −1

In this matrix, each column is a BRR replicate and each line is a pseudo stratum or zone; the matrix entry
indicates which pseudo PSU should be kept from each pseudo stratum to create the BRR replicate. For example,
the previous matrix translates into:

BRR 1 BRR 2 BRR 3 BRR 4 BRR 5 BRR 6 BRR 7 BRR 8

ZONE 1 PSU1 PSU1 PSU1 PSU2 PSU1 PSU2 PSU2 PSU2
ZONE 2 PSU2 PSU1 PSU1 PSU1 PSU2 PSU1 PSU2 PSU2
ZONE 3 PSU2 PSU2 PSU1 PSU1 PSU1 PSU2 PSU1 PSU2
ZONE 4 PSU1 PSU2 PSU2 PSU1 PSU1 PSU1 PSU2 PSU2
ZONE 5 PSU2 PSU1 PSU2 PSU2 PSU1 PSU1 PSU1 PSU2
ZONE 6 PSU1 PSU2 PSU1 PSU2 PSU2 PSU1 PSU1 PSU2
ZONE 7 PSU1 PSU1 PSU2 PSU2 PSU2 PSU2 PSU1 PSU2
ZONE 8 PSU2 PSU2 PSU2 PSU2 PSU2 PSU2 PSU2 PSU2

In the case of TALIS, as was the case in PISA, a variation of the BRR developed by Fay (1989) was implemented.
Rather than completely dropping a PSU and doubling the weight of the other one, the weight of the PSU
indicated by the Hadamard matrix is multiplied by 1.5 and the weight of the remaining PSU is multiplied by
0.5. This strategy removes the risk of deleting a domain completely.

In cases where there is an odd number of PSUs in an explicit stratum, the last three PSUs are treated as a zone
in the following manner: one of the PSUs is randomly designated as “+1” while the remaining two are both
designated as “-1”. For each replicate, as indicated by the Hadamard matrix, the weight of the selected unit
is multiplied by 1.7071 if it is the single unit and the weights of the remaining pair are multiplied by 0.6464.
If the matrix indicates that the pair should be selected, then the weights of the paired units are multiplied by
1.3536 and the weight of the single unit is multiplied by 0.2929. This ensures that the sum of the factors is 3.
This strategy was developed by Judkins (OECD, 2002).

Since the nominal sample size for TALIS was n = 200 schools, a maximum of G = 100 zones or pseudo strata
were created for each participating country and a series of G = 100 BRR replicate weights were computed and
stored as well.

© OECD 2010 TALIS 2008 Technical Report

Estimation Weights, Participation Rates and Sampling Error chapter 10
125

Estimating the sampling error

*
Let q be the population parameter of interest. Let q̂ be the full-sample estimate for q obtained by using the final
weight and let q̂g , g = 1, ..., 100, be the BRR replicate estimates of the same parameter of interest obtained by
using the BRR weights described above. Then, setting k = 0.5 and G = 100, Fay’s BRR estimate of the sampling
*
error of q̂ is given by:

∑(θˆ − θˆ ) = 0.04∑(θˆ − θˆ )
G 100
1 2 2
VˆFAY (θˆ * ) = g
*
g
*

G (1 − k ) 2 g =1 g =1

Design effect and effective sample size

Complex surveys like TALIS are known to be “less efficient” than simple random samples of the same size. The
usual explanation notes that respondents are selected in groups of individuals sharing many characteristics:
school environment, professional training, classroom equipment, textbooks and so on. The loss in efficiency
is often summarised in a statistic called “design effect” or deff (Kish, 1965). The design effect, for a statistic
and a sampling plan, is the ratio of the variance of the estimate under the sampling plan to the variance of the
same estimate under simple random sampling of the same size. In the case of TALIS, the true design effect is
approximated by:

( Vˆ
deff θˆ , BRR = BRR) (θˆ)
VˆSRS (θˆ)
Alternatively, the design effect can be regarded as the ratio of sample sizes; then, the term “effective sample
size” may be used to describe the sample size of the complex survey adjusted for the design effect:
nBRR
n effective =
deff

Tables 10.5 and 10.6 give the estimated design effect for selected key variables, the actual and effective sample
sizes, by participating country, and for TALIS as a whole.

TALIS 2008 Technical Report © OECD 2010

126
chapter 10 Estimation Weights, Participation Rates and Sampling Error

Table 10.1
Quality ratings and unweighted participation rates
Before replacement <75% 75%
After replacement <75% 75%
Non-response bias high low GOOD
FAIR
Quality rating POOR FAIR
Source: OECD, TALIS Database.

Table 10.2
Unweighted participation rates, by country
Responding teachers Teacher participation
Number of in participating School participation School participation in participating
participating schools schools before replacement after replacement schools Overall participation
Australia 149 2 275 45.0 74.5 78.6 58.6
Austria 248 4 265 78.7 89.5 84.8 75.9
Belgium (Fl.) 197 3 473 61.8 76.1 83.8 63.7
Brazil 380 5 834 90.6 96.2 90.6 87.1
Bulgaria 199 3 796 97.5 99.0 95.4 94.5
Denmark 137 1 722 47.0 68.5 79.4 54.4
Estonia 195 3 154 94.9 98.5 96.3 94.8
Hungary 183 2 934 89.4 96.8 91.7 88.8
Ireland 142 2 227 63.5 71.0 76.4 54.2
Iceland 133 1 394 92.4 92.4 79.7 73.6
Italy 298 5 263 87.0 99.3 92.9 92.2
Korea 171 2 970 66.5 85.5 92.5 79.1
Lithuania 206 3 535 96.6 99.5 96.1 95.6
Mexico 192 3 368 95.5 96.0 87.5 84.0
Malta 58 1 142 100.0 100.0 97.2 97.2
Malaysia 217 4 248 98.6 99.1 98.1 97.2
Netherlands 39 484 11.4 26.2 63.7 16.7
Norway 156 2 458 49.2 78.4 75.7 59.4
Poland 172 3 184 85.0 86.0 96.3 82.8
Portugal 173 3 046 81.3 87.4 86.6 75.7
Slovak Republic 186 3 157 86.8 94.4 93.1 87.9
Slovenia 184 3 069 88.5 92.0 88.6 81.5
Spain (excluding
193 3 362 93.0 97.0 88.7 86.1
Rioja and Canarias)
Turkey 193 3 224 93.5 96.5 90.9 87.7
TALIS total 4 401 73 584 79.3 88.2 88.4 78.0
Source: OECD, TALIS Database.

Table 10.3
Weighted participation rates, by country
School participation School participation Teacher participation Estimated size
before replacement after replacement in participating schools Overall participation of teacher population
Australia 47.5 75.1 79.0 59.3 92 691
Austria 79.5 89.7 85.3 76.6 42 372
Belgium (Fl.) 56.9 78.0 85.1 66.4 19 580
Brazil 94.4 98.0 91.8 89.9 569 553
Bulgaria 98.1 99.5 96.2 95.7 29 166
Denmark 47.6 68.8 79.6 54.8 25 735
Estonia 94.6 97.7 96.1 93.9 7 567
Hungary 88.7 96.6 92.0 88.9 47 492
Ireland 65.1 72.3 77.6 56.1 22 039
Iceland 92.6 92.6 79.8 73.9 1 916
Italy 84.6 99.2 93.0 92.2 177 539
Korea 66.5 85.5 92.8 79.4 78 052
Lithuania 95.8 99.3 95.9 95.3 28 961
Mexico 95.2 95.5 87.4 83.5 248 197
Malta 100.0 100.0 97.0 97.0 2 618
Malaysia 99.1 99.4 98.3 97.7 81 958
Netherlands 11.8 26.6 63.2 16.8 28 316
Norway 49.3 78.6 76.6 60.2 18 990
Poland 84.9 86.2 96.3 83.0 120 604
Portugal 82.8 88.7 86.4 76.6 48 381
Slovak Republic 90.1 95.9 93.7 89.9 25 738
Slovenia 88.4 92.0 88.8 81.7 7 244
Spain (excluding
93.2 97.0 88.8 86.2 200 101
Rioja and Canarias)
Turkey 92.3 95.8 91.0 87.2 148 304
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Estimation Weights, Participation Rates and Sampling Error chapter 10
127

Table 10.4
Example of BRR-ready sample design and random assignation of pseudo PSUs

Explicit stratum School ID Zone = pseudo stratum Pseudo PSU Other variables of interest…
1 1001 1 1 … …
1 1002 1 2
1 1003 2 1
1 1004 2 2
2 1005 3 2
2 1006 3 1
2 1007 4 1
2 1008 4 2
… …
H … G-1 2
H … G-1 1
H … G 1
H ... G 2
Source: OECD, TALIS Database.

Table 10.5
Estimated design effects and effective sample sizes for selected key variables, their average
and the original and effective sample sizes, by country
(Teacher Questionnaire variables)
Average Participating Effective
BTG10(4) BTG21A(4) BTG31A(3) BTG12 BTG8(A) design effect teachers sample size
Australia 2.06 2.01 1.06 1.40 2.20 1.74 2 275 1 307
Austria 1.74 2.19 1.23 1.35 1.31 1.56 4 265 2 734
Belgium (Fl.) 2.22 2.41 1.71 1.27 2.30 1.98 3 473 1 754
Brazil 3.51 4.08 4.19 5.33 3.50 4.12 5 834 1 416
Bulgaria 2.39 11.87 5.46 2.96 7.82 6.10 3 796 622
Denmark 1.65 1.82 1.25 1.10 2.07 1.58 1 722 1 090
Estonia 1.09 1.23 1.36 1.35 1.70 1.35 3 154 2 336
Hungary 4.38 10.13 2.33 2.38 2.23 4.29 2 934 684
Ireland 0.99 1.37 1.37 1.37 1.28 1.28 2 227 1 740
Iceland 1.01 1.18 1.09 0.93 1.01 1.04 1 394 1 340
Italy 1.65 3.16 1.27 2.28 2.51 2.17 5 263 2 425
Korea 1.62 1.09 0.98 1.37 2.23 1.46 2 970 2 034
Lithuania 1.43 1.87 1.26 1.51 2.06 1.63 3 535 2 169
Mexico 1.63 2.76 1.55 1.69 3.59 2.25 3 368 1 497
Malta 1.41 1.59 0.99 1.10 1.26 1.27 1 142 899
Malaysia 1.38 3.26 2.16 3.54 3.21 2.71 4 248 1 568
Netherlands 1.51 1.71 0.99 1.15 1.61 1.39 484 348
Norway 0.74 1.93 1.28 1.24 1.46 1.33 2 458 1 848
Poland 1.78 1.80 1.55 1.87 3.67 2.13 3 184 1 495
Portugal 1.93 1.70 1.26 1.41 1.56 1.57 3 046 1 940
Slovak Republic 1.76 2.72 3.09 2.65 2.74 2.59 3 157 1 219
Slovenia 1.64 1.93 1.90 1.75 1.27 1.70 3 069 1 805
Spain (excluding
1.72 1.94 1.47 1.93 2.61 1.93 3 362 1 742
Rioja and Canarias)
Turkey 4.69 2.46 4.72 3.95 7.26 4.62 3 224 698
TALIS total 73 584 36 711
Notes:
BTG10(4) How long have you been working as a teacher at this school? (4) 6-10 years.
BTG21A(4) From the following people, how often have you received appraisal and/or feedback about your work as a teacher in this school? (A) Principal? (4) once
per year.
BTG31A(3) How strongly do you agree or disagree with the following statements about yourself as a teacher in this school? (A) All in all, I am satisfied with my job
(3) Agree.
BTG12 In all, how many days of professional development did you attend during the last 18 months?
BTG8A In a typical school week, estimate the number of (60-minute) hours you spend on the following for this school (A) Teaching of students in school.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

128
chapter 10 Estimation Weights, Participation Rates and Sampling Error

Table 10.6
Estimated design effects and effective sample size for selected key variables, their average
and the original and effective sample sizes, by country
(Principal Questionnaire variables)

BCG05 BCG17(A) Average design effect Participating schools Effective sample size
Australia 1.59 1.95 1.74 149 84
Austria 1.24 1.42 1.56 248 187
Belgium (Fl.) 1.53 1.46 1.98 197 132
Brazil 1.38 1.15 4.12 380 301
Bulgaria 3.22 2.86 6.10 199 65
Denmark 1.37 3.20 1.58 137 60
Estonia 1.06 1.31 1.35 195 165
Hungary 1.93 1.40 4.29 183 110
Ireland 0.71 0.94 1.28 142 172
Iceland 1.12 1.33 1.04 133 109
Italy 1.79 2.20 2.17 298 150
Korea 2.17 1.96 1.46 171 83
Lithuania 1.00 1.46 1.63 206 168
Mexico 1.30 1.44 2.25 192 140
Malta 0.91 0.90 1.27 58 64
Malaysia 1.40 1.52 2.71 217 149
Netherlands 2.13 2.23 1.39 39 18
Norway 1.06 1.46 1.33 156 124
Poland 1.56 1.84 2.13 172 101
Portugal 1.07 1.21 1.57 173 152
Slovak Republic 1.61 1.65 2.59 186 114
Slovenia 0.86 1.42 1.70 184 162
Spain (excluding
2.05 1.30 1.93 193 115
Rioja and Canarias)
Turkey 1.55 5.31 4.62 193 56
TALIS total 4 401 2 979
Notes:
BCG05 How many years’ experience do you have working as principal? 3 = “3-5 years”, 4 = “6‑10 years”.
BCG17(A) As principal of this school, on average throughout the school year, what percentage of the time do you estimate that you spend on the following tasks in
this school? (A) Internal administrative tasks.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Estimation Weights, Participation Rates, and Sampling Error chapter 10
129

References
Cochran, W.G. (1977), Sampling Techniques, 3rd edition, Wiley, New York.

Fay, R.E. III (1989), “Theoretical Application of Weighting for Variance Calculation”, Proceedings of the Section on Survey
Research Methods of the ASA, pp. 212-217.

Kish, L. (1965), Survey Sampling, Wiley, New York.

Lohr, S. (1999), Sampling: Design and Analysis, Duxbury Press, New York.

OECD (2002), PISA 2000 Technical Report, OECD, Paris.

OECD (2008), PISA 2006 Technical Report, OECD, Paris.

OECD (2009), Creating Effective Teaching and Learning Environments: First Results from TALIS, OECD, Paris.

Rust, K. and J.N.K. Rao (1996), Variance Estimation for Complex Estimators in Sample Surveys, Statistics in Medical Research,
No. 5, pp. 381-397.

Statistics Canada (2003), Survey Methods and Practices, Catalogue Number 12-587-XPE, Ottawa.

Statistics Canada (2007), TALIS Sampling Manual (MS-02-03), Ottawa.

Wolter, K. (2007), Introduction to Variance Estimation, 2nd edition, Springer Verlag, New York.

TALIS 2008 Technical Report © OECD 2010

131

Chapter 11

Construction and Validation

of Scales and Indices
132 Abstract
132 Overview
132 Simple questionnaire indices, ratios and indicators
132 Student-teacher ratio
135 Validation of complex questionnaire scale indices
135 Indices derived through Principal Components Analysis
145 Description of complex scale indices and their parameters
145 School leadership indices

TALIS 2008 Technical Report © OECD 2010

132
chapter 11 Construction and Validation of Scales and Indices

Abstract

This chapter outlines the design and validation of scales and indices in TALIS. The TALIS
questionnaires were comprised of many items; while some were intended to be used in single
item analyses, others were intended to be combined to measure latent constructs. This chapter
explains how simple indices were constructed, describes the methodology used for construct
validation and scaling and finally details the construction, validation, and computation of each
scaled index and its characteristics.

Overview
The TALIS questionnaires included numerous items on school characteristics, school background, teacher
perceptions, and perceptions of school principals. Some of the items were designed to be used in analyses
as single items (for example, teachers’ amount of professional development). However, a large number of
questionnaire items were designed to be combined in some way so as to measure latent constructs that cannot
be observed directly. For these items, transformations or scaling procedures are needed to construct meaningful
indices.

As in previous surveys of this kind, two different types of indices can be distinguished:
• Simple indices (ratios, averages, and binary indicators): these indices were constructed through the
arithmetical transformation or recoding of one or more items;
• Complex scale indices: these indices were constructed through the scaling of items. Typically, scale scores
for these indices are estimates of latent traits derived through scaling of dichotomous or Likert-type items
using more complex methodology.

Some indices were already used in previous surveys and are constructed based on similar scaling methodology,
whereas others were based on the elaboration of a questionnaire framework (see Chapters 1 and 2).

This chapter outlines how simple indices were constructed, describes the methodology used for construct
validation and scaling and proceeds to detail the construction, validation, and computation of each scaled
index and its characteristics.

Simple questionnaire indices, ratios and indicators

This section describes the simple indices, including ratios, averages and binary indicators, that were constructed
through the arithmetical transformation or recoding of one or more items. It discusses these indices at both the
school level and individual teacher level.

Student-teacher ratio
This is a school-level ratio derived from school principals’ responses to questions about the number of staff
(headcounts) currently working in the school and the total number of students (headcounts) of all grades in the
school. The measure is not therefore restricted to those teaching or supporting ISCED level 2 education in the
school but covers education of all levels provided in the school. The ratio (STRATIO) is derived by dividing the
number of students in the school (BCG12) by the number of teachers in the school (those whose main activity
is the provision of instruction to students) (BCG11A).

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
133

Ratio of students to number of personnel for pedagogical support

This is a school-level ratio derived from school principals’ responses to questions about the number of staff
(headcounts) currently working in the school and the total number of students (headcounts) of all grades in the
school. The measure is not therefore restricted to those teaching or supporting ISCED level 2 education in the
school but covers education of all levels provided in the school. The ratio (SPRATIO) is derived by dividing the
number of students in the school (BCG12) by the number of personnel for pedagogical support in the school
(BCG11B). Pedagogical support personnel include all teacher aides or other non-professional personnel who
provide instruction or support teachers in providing instruction, professional curricular/instructional specialists
and educational media specialists.

Ratio of students to number of school administrative or management personnel

This is a school-level ratio derived from school principals responses to questions about the number of staff
(headcounts) currently working in the whole school and the total number of students (headcounts) of all
grades in the school. The measure is therefore not restricted to only those teaching or supporting ISCED level
2 education in the school but covers education of all levels provided in the school. The ratio (SARATIO) is
derived by dividing the number of students in the school (BCG12) by the number of school administrative
or management personnel in the school (BCG11C). School administrative or management personnel include
principals, assistant principals, other management staff, receptionists, secretaries and administration assistants
whose main activity is administration or management.

Ratio of teachers to number of personnel for pedagogical support

This is a school-level ratio derived from school principals’ responses to a question about the number of staff
(headcounts) currently working in the whole school and so is not restricted to only those teaching or supporting
ISCED level 2 education in the school. The ratio (TPRATIO) is derived by dividing the number of teachers
(those whose main activity is the provision of instruction to students) (BCG11A) by the number of personnel
for pedagogical support (BCG11B). Pedagogical support personnel include all teacher aides or other non-
professional personnel who provide instruction or support teachers in providing instruction, professional
curricular/instructional specialists and educational media specialists.

Ratio of teachers to number of school administrative or management personnel

This is a school-level ratio derived from school principals responses to a question about the number of staff
(headcounts) currently working in the whole school and so is not restricted to only those teaching or supporting
ISCED level 2 education in the school. The ratio (TARATIO) is derived by dividing the number of teachers
(those whose main activity is the provision of instruction to students) (BCG11A) by the number of school
administrative or management personnel (BCG11C). School administrative or management personnel include
principals, assistant principals, other management staff, receptionists, secretaries and administration assistants
whose main activity is administration or management.

Percentage of professional development that is compulsory

This is a percentage at the individual teacher level and was derived from teachers’ responses to the questions
“In all, how many days of professional development did you attend during the last 18 months?” (rounded to
whole days) and “Of these, how many were compulsory for you to attend as part of your job as a teacher?” In
the international database, for each teacher, the variable COMPULPD was calculated by dividing the number
of compulsory days (BTG13) by the total number of days (BTG12) and multiplying by 100.

TALIS 2008 Technical Report © OECD 2010

134
chapter 11 Construction and Validation of Scales and Indices

Average class size

In the section of the teacher questionnaire where teachers were asked about their classroom teaching practices,
they were asked to report on a “target class” that they taught. This “target class” was defined as the first ISCED
level 2 class that the teacher (typically) taught in the school after 11am on Tuesdays. This formulation was used
to introduce randomisation in the selection of the “target class”. As this approach is less rigorous than a truly
randomised selection of classes, some caution is needed in interpreting the results at the teacher level and in
the aggregation to the school level.

Among the characteristics of the “target class”, teachers were asked to report the number of students in this class
on average throughout the year (BTG38). In the international database the AVGCLSIZ is calculated at the school
level as the mean of the values reported by teachers in BTG38 for that school.

Language difference
In the section of the teacher questionnaire where teachers were asked about their classroom teaching practices,
they were asked to report on a “target class” that they taught. This “target class” was defined as the first ISCED
level 2 class that the teacher (typically) taught in the school after 11am on Tuesdays (see cautionary note above).

Among the characteristics of the “target class”, teachers were asked to report the percentage of students whose
first language is different from the language of instruction (BTG40A). In the international database LANGDIFF
is calculated at the school level as the mean of the response categories (rather than the percentages that these
response categories represent) reported by teachers in BTG40A for that school.

Education level of parents

Among the characteristics of the “target class”, teachers were asked to report the percentage of students who have
at least one parent/guardian who has completed at least upper secondary education (ISCED level 3 or higher)
(BTG40B). In the international database PEDUATT3 is calculated at the school level as the mean of the values
reported by teachers in BTG40B for that school.

Teachers were also asked to report the percentage of students who have at least one parent/guardian who has
completed higher education (ISCED level 5 or higher) (BTG40C). In the international database PEDUATT5 is
calculated at the school level as the mean of the response categories (rather than the percentages that these
response categories represent) reported by teachers in BTG40C for that school.

No evaluation of the school

This school-level derived variable (NVREVAL) indicates whether or not a school evaluation has been conducted
on the school in the five years prior to the survey (either a school self-evaluation or an external evaluation). The
variable is coded “1” if no such evaluation was conducted (BCG18A=1 and BCG18B=1) and “0” otherwise.

No appraisal or feedback received by the teacher

This teacher-level derived variable (NEVERAF) indicates whether or not a teacher has received an appraisal or
feedback about their work as a teacher in their current school (either from the school principal, other teachers or
members of the school management team or an external individual or body). The variable is coded “1” if no such
appraisal or feedback was received by the teacher (BTG21A=1 and BTG21B=1 and BTG21C=1) and “0” otherwise.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
135

Validation of complex questionnaire scale indices

This section explains the construct validation and scaling analyses used to develop the complex questionnaire
scale indices, including the Principal Component Analysis and the Confirmatory Factor Analysis. These
techniques are employed to search for characteristics and identify patterns within the data.

Indices derived through Principal Components Analysis

Indices for school autonomy and school resources were derived through Principal Components Analysis (PCA).
PCA is a variable-reduction procedure but differs from common factor analysis in that it considers the total
variability of the variables in the analysis and extracts the variance that is common among the factors identified
rather than considering the unique variance of the individual variables.

PCA begins by extracting the maximum variance and allocates that to the first factor and proceeds to extract
the maximum of the remaining variance for the second factor and so on until all of the variance in the data has
been accounted for. In PCA, the full variance is therefore brought into the factor matrix. The factor matrix in
PCA is the matrix that contains the factor loadings of all the variables on all of the factors extracted. The factor
loadings in PCA are simply the correlations between the factors and the variables and thus the diagonal of the
correlation matrix consists of unities.

PCA is a common technique for searching for patterns in data that consist of a high number of dimensions
and is recommended when the researcher’s primary concern is to determine the minimum number of factors
that will account for the maximum variance in the data in a given multivariate analysis. PCA was chosen
over Confirmatory Factor Analysis (CFA) to derive the indices for school autonomy and school resources in
TALIS because these scales are based on responses by the principal that are essentially factual reports. This
is in contrast to the CFA approach used to derive the scales in the remainder of this chapter, which sought to
model underlying traits of the attitudinal data that comprise these scales. Moreover, the indices for school
autonomy and school resources were more readily postulated from the questionnaire items and therefore
less in need of empirical derivation. For these same reasons, the scales on school autonomy and school
resources were not subject to the extent of cross-cultural scrutiny that the other scales were.

List-wise deletion was used to deal with missing teacher and principal questionnaire data and SPSS 17.0 was
used for computing component scores.

School autonomy indices

To describe the extent of school autonomy in decision making, indices were derived from question BCG31 of
the school principals’ questionnaire. For a list of 13 tasks, the question asked the school principal to indicate
who, among a range of stakeholders, had a considerable responsibility in the decision making for these
tasks. Considerable responsibility could be attributed to one or more of: the principal, teachers, the school
governing board, regional or local authority and national education authority (Table 11.1). For a particular
task, the extent of school-level autonomy was determined by whether a considerable responsibility lay at
the school level (i.e. with principal, the teachers or the school governing board) or with other authorities
(i.e. regional or local authority and national education authority) or shared between both groups. Thus, for
example, if BCG31A1=1 or BCG31A2=1 or BCG31A3=1, then this indicates a school-level responsibility.

Table 11.2 shows the rotated component matrix from the PCA of the 13 items of question 31 of the principal
questionnaire.

TALIS 2008 Technical Report © OECD 2010

136
chapter 11 Construction and Validation of Scales and Indices

The rotated component matrix derived from the PCA allows four components to be derived:
• School autonomy in hiring teachers, determining salaries (AUTHIRE).
• School autonomy in budgeting (AUTBUDGT).
• School autonomy in curriculum (AUTCURR).
• School autonomy in student policy and textbooks (AUTSTUDP).

Each of the components has an EIGENVALUE above 1 and together they explained 66% of the total variance of
12 items (respectively 32%, 15%, 10% and 9% for Indices 1, 2, 3, and 4 shown in Table 11.2).

The index AUTHIRE is derived by regrouping five items related to school autonomy in hiring teachers and
determining salaries: Selecting teachers for hire (BCG31A); Firing teachers (BCG31B); Establishing teachers’
starting salaries (BCG31C); Determining teachers’ salary increases (BCG31D); and Allocating funds for teachers’
professional development’ (BCG31M).

The index AUTSTUDP is derived by regrouping three items related to school autonomy in student policy
and textbooks: Establishing student disciplinary policies (BCG31G); Establishing student assessment policies
(BCG31H); and Approving students for admission to the school (BCG31I).

The index AUTCURR is from three items related to school autonomy in curriculum: Choosing which textbooks
are used (BCG31J); Determining course content (BCG31K); and Deciding which courses are offered (BCG31L)
and the index AUTBUDGT is derived from two items related to autonomy in budget: Formulating the school
budget (BCG31E); and Deciding on budget allocations within the school (BCG31F).

These four indices were obtained for the first principal component, with zero being the score of an average TALIS
country and 1 being the standard deviation across TALIS countries weighted equally with school weight, except
in the Netherlands, which was excluded from the weighting. Higher values on the scale of the four indices
(AUTHIRE, AUTBUDGT, AUTSTUDP, AUTCURR) indicate relatively higher levels of school responsibility in
this area.

Concerning the reliability of the indices, Cronbach’s Alpha is acceptable or good for all of the autonomy scales
(see Table 11.1211.4). However, given that the reliabilities for the index of school autonomy in budgeting
(AUTBUDGT) are below 0.50 for around half of the countries, the reliability of this index is more open to
question and care should therefore be taken in its interpretation.

School resources indices

To describe the relative level of resources available in schools, indices were derived from nine items in question
BCG29 of the principals’ questionnaire in which school principals were asked to indicate the extent (“not at
all”, “very little”, “to some extent” or “a lot”) to which the school’s capacity to provide instruction was hindered
by a shortage or lack of resources in a range of areas (Table 11.5). For the calculation of the indices, responses
“not at all” and “very little” were coded to the value of -1 and the responses “to some extent” and “a lot” were
recoded to the value of 1.

Table 11.6 shows the rotated component matrix from a Principal Components Analysis (PCA) of the nine items
of question 29 of the principal questionnaire.

The rotated component matrix derived from the PCA allows two components to be derived:
• Lack of personnel resources (LACKPERS).
• Lack of material resources (LACKMAT).

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
137

The index LACKPERS is derived by regrouping four items of question 29 related to a lack of personnel: Teachers
(BCG29A); Laboratory technicians (BCG29B); Instructional support personnel (BCG29C); Other support
personnel (BCG29D). The index of LACKMAT is derived by regrouping four variables related to a shortage
or inadequacy in materials: Instructional materials (BCG29E); Computers for instruction (BCG29F); Other
equipment (BCG29G); and Library materials (BCG29H). The item BCG29I “Other” was dropped due to lack of
reliability with the two indices (less than 20% of school principals ticked this option).

The LACKPERS and LACKMAT scores were obtained as component scores for the first principal component with
zero being the score of an average TALIS country and one the standard deviation across TALIS countries weighted
with school weight except the Netherlands, which was excluded from the weighting. Higher values on the indices
LACKPERS or LACKMAT indicate relatively high levels of a lack or inadequacy or resources in these areas.

Table 11.8 shows the scale reliabilities (Cronbach’s Alpha) in TALIS countries for the two resource indices. The
scale reliabilities for each scale are generally acceptable or good, with only a few countries having reliability
indices below 0.50.

Indices developed through Confirmatory factor analysis (CFA)

TALIS measures teacher beliefs, attitudes and practices and principals’ leadership styles with single items that
are combined (reduced) to form scales. The basic advantages of scales are their higher reliability and validity, as
well as the possibility to alleviate issues of multicollinearity in models. Confirmatory factor analysis (CFA) with
mean and covariance structure (MACS) models was used to confirm and, if necessary, re-specify the expected
dimensional structure of the scales. The analysis was carried out with the software Mplus, version 5.1 (Muthén
& Muthén, 1998-2007).

CFA treats the constructs of interest as latent response variables. Latent variables are variables that cannot be
directly observed but are rather inferred from other variables that are directly measured. In the CFA model the
responses to each item y are predicted from the latent factor h. In addition to the observed variables y and the
latent factor h the model contains a matrix of factor loadings L, a vector of intercepts t and a vector of residual
variances e.

y = ty + Lyh + e

Figure 11.1
Illustration of CFA parameters
Value for item y
Factor loading

Residual

Intercept
Latent variable

Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

138
chapter 11 Construction and Validation of Scales and Indices

The factor loadings are the regression slopes for predicting each item y from the latent factor. The intercept
is the predicted value for item y when the value for the latent trait h is zero. The residual variance is the
variance in item y that is not explained by the latent variable h. It is a combination of variance that is specific
to the indicator and random error variance. Figure 11.1 further illustrates the meaning of these parameters.
The association of item y and the latent factor h is described with a regression line. The factor loading l is
the regression slope, defined as the ratio of the “rise” divided by the “run” between two points on a line, or in
other words, the ratio of the altitude change to the horizontal distance between any two points on the line, for
predicting the item y from the latent factor h. The intercept t is the value for item y where the regression line
crosses the y-axis. Finally the deviation of each observed value from the regression line is the residual e and its
variance across all observations is the residual variance q.

Figure 11.2 illustrates the latent y notation for a CFA-model.1 Here the latent variable h is represented by an
oval, while the manifest items y1 to y4 are represented by boxes. The factor loadings describe the relationships
between the latent variable h and the items y1 to y4 e1 to e4 are the residuals and the four qs (qe1 to qe4) are the
residual variances. The triangle represents a mean structure, with the mean vector a1 and the intercepts t1 to t 4.

Figure 11.2
Latent y notation for a one-factor CFA model
qe1
1
y1: Educational difference e1

l1 qe2
1
l2 y2: Progress difficult for students e2
h Self-efficacy qe3
l3
1
y3: Successful with students e3
t1
t2 qe4
a1 l4
t3 1
y4: Know how to get through e4
t4

Source: OECD.

The model can also be written in expanded matrix form. Here S represents the covariance matrix of the items
y, Ly is the matrix of factor loadings l, L`y is the transposed matrix of factor loadings, Y is the symmetric matrix
of the factor covariances, and Q is the diagonal matrix of residual variances q.

S = LyYL‘y + Qe

The mean vector m of y equals a vector of intercepts t plus a matrix of factor loadings multiplied by the mean
vector a of h.
m = t + La

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
139

The goal in CFA models is to find a set of parameters that yields an estimated mean vector μ and variance-
covariance matrix S that best reproduces the input matrix. To minimise the difference between the input and the
predicted matrix, a fitting function is used. The most common fitting function is maximum likelihood estimation,
which generally requires continuous data and a multivariate normal distribution. However, e.g. Muthén & Kaplan
(1985) showed that the use of Likert data and skewed items does not significantly influence the probability of
incorrect conclusions in CFA. The estimation is an iterative procedure: first, an initial set of starting values for the
parameters is selected; next, the difference between input and estimated matrices is computed. The parameters
are then refined, with the difference between input and estimated matrices being computed again, and so on,
until a set of parameters that cannot be improved substantially is found (e.g. Brown, 2006; Muthén, 1998-2004).

For the estimation of parameters in models with missing data, the model based approach for categorical
and continuous data implemented in Mplus is used. Model based approaches treat the missing data and
estimate the parameters in one step (Lüdtke, Robitzsch, Trautwein & Köller, 2007). Mplus uses the Expectation
Maximisation (EM) algorithm (for a detailed description, see Dempster, Laird & Rubin, 1977). The procedure
assumes that the data is missing at random (MAR). MAR means that the probability of a missing observation
does not depend on the true score of a person with the variable of interest, but can be correlated with other
covariates (Schafer & Graham, 2002).

As described in Chapter 4, a two-stage stratified sampling design was used for TALIS, sampling schools within
countries and teachers within schools. Because of similar working conditions and a common socialisation,
teachers within a school are likely to have more similar responses than teachers in different schools. If this is
the case, the variance and standard errors would be underestimated with regular procedures (e.g. Hox, 2002;
Raudenbush & Bryk, 2002; Snijders & Bosker, 1994). To avoid this, the Mplus “type is complex” procedure was
used for CFA and MGCFA, which corrected for cluster effects (Muthén & Muthén, 1998-2007).

In CFA, the constructs are each measured by a number of different items and treated as latent response variables.
Because these variables are latent, their metrics (units of measurement) are not determined. It is common to
standardise the mean of the latent response to zero and the factor loading of one item to one to give the scale
a metric invariance (Bollen, 1989; Kline, 2005). This was also done in TALIS.

To determine whether the theoretically expected model fits the data, different fit indices were used: the Comparative
Fit Index (CFI), the Root Mean-Square Error of Approximation (RMSEA) and the Standardized Root Mean-Square
Residual (SRMR). These indices all evaluate the correspondence between the observed data with the data pattern
that would be expected based on the estimated model (for a more detailed description see e.g. Brown, 2006).
In accordance with scientific conventions (Hu & Bentler, 1999; Schermelleh-Engel & Moosbrugger, 2002),
CFI > 0.90, RMSEA < .08 and SRMR < .08 were seen as indicative of an acceptable model fit.

It is generally desirable to give participating countries an equal impact and contribution on the estimation
of model parameters. In the presence of large datasets, smaller calibration samples are drawn from the
entire sample to compute international parameters that are equally influenced by all participating countries
(e.g. OECD, 2009). For the calibration of the TALIS teacher-scales, a sample of 1 000 cases was randomly
selected in each country. Records were selected proportional to the final estimation weight (see Chapter 9),
i.e. giving those cases with relatively larger weights a higher probability of being selected. Likewise a calibration
sample of 150 principals per country was drawn, except for those countries who did not reach the minimum
sampling standard of 150. Here the whole sample was used.

Analysis of cross-cultural invariance and validity

Cross-national data allows countries to identify other countries facing similar challenges and to learn from
other policy approaches. But it also entails special challenges. Flaws in translation, cross-cultural differences
in the handling of questionnaires, different meanings of certain aspects of a construct in different cultures and

TALIS 2008 Technical Report © OECD 2010

140
chapter 11 Construction and Validation of Scales and Indices

other factors may threaten the validity of cross-cultural comparisons. Countries can only be validly compared
if the scales used have an equivalent meaning across all countries (Van de Vijver and Leung, 1997). To assure
cross-cultural validity of the TALIS instruments, the translation process was closely monitored. Furthermore,
psychometric methods were used to examine cross-cultural equivalence of the measurement and of the
measured constructs. Tests of invariance were carried out with multiple group confirmatory factor analysis
(MGCFA). In MGCFA models factor loadings, intercepts, residual variances, means and standard deviations are
estimated for each country separately. They can be restricted to be equal or they can be allowed to vary across
groups. Three levels of invariance were examined: Configural, metric and scalar invariance (e.g. Cheung &
Rensvold, 2002; Davidov, 2008; Steenkamp & Baumgartner, 1998).

Configural invariance holds if the same number of factors is found in all the participating countries and the
same items are associated with each of the underlying factors (Bollen, 1989; Meredith, 1993). This means
that the same pattern of zero loadings and loadings different from zero can be found in all countries,
while the exact value of the loadings is allowed to vary. Thus configural invariance requires an adequate
model fit (CFI > 0.90, RMSEA < 0.08 and SRMR < 0.08) when models for all groups (countries) are estimated
simultaneously using the same factor structure, while the model parameters do not need to be equal across
countries. Only one factor loading and one intercept need to be restricted to be equal for model identification.2

Figure 11.3
Illustration of configural invariance of the construct Self-efficacy across two countries
qe1
1
y1: Educational difference e1

l1 qe2
1
Country 1
l2 y2: Progress difficult students e2
h Self-efficacy qe3
l3
1
y3: Successful with students e3
t1
t2 qe4
a1 l4
t3 1
y4: Know how to get through e4
t4

qe5
1 t5 1
y1: Educational difference e5
t6
t7 l5 qe6
t8
a2 1
y2: Progress difficult students e6
l6
qe7

l7 1
y3: Successful with students e7
h Self-efficacy
Country 2 qe8
l8
1
y4: Know how to get through e8

Source: OECD.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
141

Figure 11.3 illustrates configural invariance of the scale Self-efficacy. Here, in both countries 1 and 2, the same
four items belong to the scale Self-efficacy, while the factor loadings l, the intercepts t and the residual variances
q are allowed to vary.

Factors are metric-invariant, if the same dimensional structure is found across countries and the strength of
the associations between the items and the factor they constitute are also equal for all participating countries.
Only in the case of equal factor loadings, a change in the value of item y of one unit is associated with the
same change in the latent construct for all countries (Steenkamp & Baumgartner, 1998, Meredith, 1993).
Figure 11.4 shows a case of metric non-invariance. Here, item y has a larger factor loading in country 1
than in country 2. Comparing correlations or mean scores of these two countries, it would be impossible to
separate real attitudinal differences from those that are only due to differences in the relative importance of
single responses (Cheung & Rensvold, 1998).

Figure 11.4
Illustration of metric non-invariance for two countries
Value for item y

Country 1

Country 2

Latent variable

Source: OECD, TALIS Database.

When scales are scalar-invariant, equal intercepts are observed for all countries. Thus, equal values for each item
y are predicted for participants from different countries who have a value of zero on the underlying trait h. This
means that all items indicate the same cross-cultural differences in latent means (Cheung & Rensvold, 2002;
Davidov, 2008; Steenkamp & Baumgartner, 1998). Figure 11.5 illustrates scalar non-invariance. Here, the
same slopes are found for both countries, but country 1 has a higher intercept than country 2. With scalar
non-invariance it is impossible to decide whether mean score differences are due to differences in the latent
construct or to differences that concern single items only.

Equality of residual variances implies that the portion of item variance not attributable to variance in the latent
variable is also the same across countries. This means that the items have the same quality as measures of the
latent variable in all countries (Cheung & Rensvold, 2002). In Figure 11.6, non-invariance of residual variances
for countries 1 and 2 is depicted. The residual variance for country 2 is larger while for country 1, the observed
values are closer to the regression line. This level of invariance is a prerequisite for comparing manifest means
across countries (Davidov, 2008).

TALIS 2008 Technical Report © OECD 2010

142
chapter 11 Construction and Validation of Scales and Indices

Figure 11.5
Illustration of scalar non-invariance across two countries
Value for item y
Country 1

Country 2

Latent variable

Source: OECD, TALIS Database.

Figure 11.6
Illustration of non-invariance of residual variance across two countries
Value for item y Value for item y

Country 1 Country 2

Residual Residual

Latent variable Latent variable

Source: OECD, TALIS Database.

The different levels of invariance form a hierarchy. Metric invariance requires configural invariance, and scalar
invariance requires metric invariance (e.g. Meredith, 1993). Accordingly, models testing the three levels of
invariance are nested, and fit indices can be compared across models. To determine whether the model fit
significantly decreases, when loadings and intercepts are restricted, differences in CFI, RMSEA and SRMR were
compared for the nested models. The c2 difference test was not employed because of its strong sensitivity to
sample size (e.g. Cheung & Rensvold, 1998). A certain level of cross-cultural variation of the parameters is to
be expected. Up to now it remains unclear, which difference in model fit and between model parameters is
indicative of serious bias and to what extent variations are acceptable (Schulz, 2005). Based on a simulation
study, Chen (2007) recommends to view models as invariant if CFI changes < -0.010, RMSEA changes < 0.010
and SRMR changes < 0.005. However, Chen compared two groups only. Given that TALIS examined 23 groups,
Chen’s recommendation was only considered to be a rough orientation.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
143

Generally, the MGCFA models were specified similar to the simple CFA models, as described above. In addition
to fixing the mean to zero and the factor loading of one item to one in one of the countries, one factor loading and
one intercept were restricted to be equal across countries for identification purposes (e.g. Cheung & Rensvold,
1999). For the estimation of parameters, maximum likelihood and the EM-algorithm were used (see above).

Latent correlations between dimensions of a construct were also compared across countries. It should be noted,
though, that differences in the strength of the relationships might be due to real cross-national differences, and
do not necessarily indicate bias or inconsistency.

Scaling procedures
Different procedures can be used for the estimation of scale characteristics and composite scores. The simplest
method is to compute a sum score or mean score over all items that measure the same construct. Other
approaches encompass factor scores that are computed based on the classical test theory or the structural
equation modelling (SEM) framework or person parameters that are based on the item response theory (IRT)
framework as used in, for example, the scaling of context questionnaire data in PISA using the Partial Credit
Model (OECD, 2009). Methods are typically highly correlated but are not completely congruent and all three
methods entail advantages as well as disadvantages.

To assure high quality standards for TALIS, the Board or Participating Countries (BPC) requested a description
of the different methods and the definition of criteria by which to decide the best scoring procedure for TALIS.
Based on analysis carried out using main study data in late 2008, it was found that composite scores computed
with different methods were in fact highly correlated and that there were no large differences between mean
scores, IRT-scores (using weighted likelihood estimation, WLE) and SEM factor scores. Differences between the
factor scores assuming different levels of cross-cultural invariance were highly correlated, while correlations of
factor scores and IRT-scores were found to be slightly lower. This suggested that differences between methods
were larger than those due to cross-cultural non-invariance of the scales. The scoring methods did influence
the comparisons of country means to some extent. Differences were generally small, but larger for those
scales, which show a poorer model fit and were less cross-culturally invariant. Given this, it seems more
important to consider psychometric and strategic arguments regarding the selection of a method. The TALIS BPC
consequently preferred the Structural Equation Modelling (SEM) approach in light of: i) the sounder scientific
basis and flexibility, given the invariance results for the TALIS scales; ii) the fact that SEM and the employed
modelling software are better equipped to deal with missing values, and iii) the fact that the analysis and scoring
could be carried out within one (i.e. SEM) instead of two frameworks (SEM and IRT).

For most of the TALIS scales, factor scores were computed as representations of the latent constructs with the
program Mplus version 5.1. The use of factor scores minimises measurement error in the items contributing to
each of the scales, thus increasing the reliability (and validity) of the computed scale scores, even though – as
opposed to latent variables in SEM – factor scores are not completely free of measurement error (Hansen, Rosén
& Gustafsson, 2006). Another advantage of factor scores is that – compared to simple sum scores – they contain
more information and they are a more realistic approximation of a person’s value on the construct of interest,
because they account for differences in the relative strength of the relationships between the latent construct
and the items (e.g. Cheung & Rensvold, 1998).

Factor scores are based on the general structural equation modelling framework. As described above, the
items y are predicted from the latent factor h, which is multiplied with the factor loadings l. The vector of item
intercepts t and the vector of residual variances e are both added to the product.

y = ty + Lyh + e

TALIS 2008 Technical Report © OECD 2010

144
chapter 11 Construction and Validation of Scales and Indices

To estimate factor scores, Mplus uses the Bayes method (Muthén, 1998-2004). If all y variables are continuous,
this results in the usual factor score estimates based on the regression method with correlated factors (Muthén,
1977). For continuous items, the factor score for individual i is computed from the mean vector of y variables
m, the factor score coefficient matrix C, the vector of observations vi, the vector of intercepts t, and the matrix
of factor loadings L multiplied by the mean vector m:

i = my + C (vi – tv – Ly my )

The score coefficient matrix in turn is based on the item covariance matrix S, the matrix of factor loadings L
and the matrix of residual variances and covariances Q:

C = Sy L’y (LySyL’y + Qy )-1

These formulas imply that higher factor loadings of an item are associated with a stronger influence of this item
on the factor score estimate. Likewise, the larger the residual variance of an item, the smaller its influence on
the factor score estimate. The mean vector and the variance of the latent variable also affect the estimated scores
for different countries in multiple group models.

A score was computed for respondents who answered at least one of the items that belong to the respective
scale, using the EM algorithm as described above to deal with missing data. The specification of the MGCFA
model depended on the level of invariance established in previous analysis. If metric invariance had been
established, a model with equal factor loadings across countries, but different intercepts and residual
variances, was used. For each scale, one of the intercepts was restricted to be equal across countries for
identification purposes. The choice of the item was based on previous analysis examining the proportion
of invariance each individual item adds to the total invariance. This was done by freeing the intercept of
each item successively and comparing the model fits for these models to that of a model assuming metric
invariance. If scalar invariance had been established, a model with equal factor loadings and intercepts
across countries, but different residual variances, was used. Finally, for scales whose uniqueness invariance
had been established, a model restricting factor loadings, intercepts and residual variances to be equal across
countries was used.

For a given factor analysis, there is an infinite number of sets of factor scores that are consistent with the
factor loadings. This phenomenon is called “factor score indeterminacy” (see e.g. Grice, 2001). The degree
of indeterminacy varies in different CFA models depending on several factors like the general model fit and
the number of items included in the model. As an indicator of the quality of factor scores, the degree of
indeterminacy can be estimated. For TALIS, validity coefficients are requested in Mplus that inform about the
correlation between the factor score estimates and their respective factors (Muthén & Muthén, 1998-2007). In
the following, factor score determinacies for complete data will be reported for each of the scales. According
to Gorsuch (1983), validity coefficients of > .80 indicate an acceptably small magnitude of indeterminacy.

Once individual factor scores were estimated, each complex questionnaire scale index was transformed to an
international metric with an average of zero and a standard deviation of one. The transformation to the scores
in the international metric was achieved by subtracting the international mean of factor scores, computed from
the pooled data with equally weighted country sub-samples (i.e. giving each country a weight of one)3, from
the original factor scores and dividing the remainder by the corresponding international standard deviation.
The Netherlands were excluded from all transformations. The means and standard deviations used for the
transformation into the international metric are shown in Table 11.9.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
145

Description of complex scale indices and their parameters

This section details the construction and computation of a variety of complex scale indices and their
characteristics. The indices described take into account school leadership, teacher-student relationships,
classroom environments, teachers’ self-efficacy and teaching practices and beliefs.

School leadership indices

Five indices describing the school leadership and management styles of school principals in TALIS were
derived from questions BCG15 and BCG16, which asked school principals about the frequency with which
they engaged in a range of school management activities and behaviours and how strongly they agreed with
statements about their role in the school. The five indices are: Framing and communicating the school’s
goals and the curricular development (FCSGCD) consisting of 6 items; Promoting instructional improvement
and professional development (PROIIPD) consisting of 4 items; Supervision of the instruction in the school
(SUPINSTR) consisting of 4 items; Accountability role of the principal (ACCROLE) consisting of 4 items; and
Bureaucratic rule-following (BURRULEF) consisting of 5 items (Table 11.10). The items BCG15N, BCG16B,
BCG16C, BCG16G, BCG16L, and BCG16N (Table 11.11) were excluded due to poor inter-item correlation
and poor fit in the different scales. However, these items could be analysed as single items in further analysis.

For the items from BCG15, response options were “never”, “seldom”, “quite often” and “very often”; while for
the items in BCG16, the response options were “strongly disagree”, “disagree”, “agree”, and “strongly agree”.

Reliabilities (Cronbach’s Alpha) are acceptable or good for all of the leadership scales (see Table 11.12). Only a
few countries have reliability indices (for some of the scales) below 0.50 (Bulgaria, Denmark, Estonia, Iceland,
Italy, the Netherlands, and Poland). As noted earlier, the Netherlands was excluded from further analysis
because it did not meet international sampling standards.

As can be seen from Table 11.13-11.17, model fit was acceptable at the national and international level for
each of the scales.

Tables 11.18-11.22 show the results of the tests of cross-cultural invariance of the school leadership scales
using multiple group confirmatory factor analysis. The results show that model fit only deteriorates slightly
when factor loadings are restricted to be equal. The difference between the models examining configural
and metric invariance is small: with DCFI ranging from -0.04 (BURRULEF) to -0.01 (FCSGCD) and DRMSEA
ranging from -0.02 (FCSGCD) to 0.00 (SUPINSTR). These changes in model fit are close to the criteria
established by Chen (see earlier section in this chapter).

However, continuing the analysis to test scalar invariance by restricting intercepts to be equal, leads to a
noticeable drop in model fit (Tables 11.18-11.22). This suggests that school principals from different countries
differ with regards to the relative tendency to endorse each of the single items given the same level of the
underlying trait. Thus, while the results confirm the validity of cross-cultural comparisons of correlations
of leadership styles with other constructs across countries, mean score comparisons should be interpreted
carefully as the mean scores may have a slightly different meaning for each country.

Following the results of the analysis given in Tables 11.18-11.23, for the computation of factor scores, a model
assuming metric invariance was used. Thus, the same items with the same factor loadings were used for all
participating countries in a multiple group CFA model, but all item intercepts (except for BCG15A for FCSGCD,
BCG15G for PROIIPD, BCG15C for SUPINSTR, BCG16A for ACCROLE and BCG16H for BURRULEF, which
were restricted to be equal for identification purposes) and all unique variance variances were allowed to
vary across countries. The parameters used are detailed in Table 11.23. As can be seen from Table 11.38, the
factor score determinacy indicator is acceptable (close to 1 in the majority of countries) for all scales across
all participating countries. This indicates good measurement of the latent construct by the observed variables.

TALIS 2008 Technical Report © OECD 2010

146
chapter 11 Construction and Validation of Scales and Indices

School leadership composite indices

The five different dimensions of school management outlined in the previous sections were further summarised
into two indices of leadership styles: Instructional Leadership and Administrative Leadership.

The Instructional Leadership index was defined as the combination of three of the school management indices:
i) Framing and communicating the school’s goals and curricular development; ii) Promoting instructional
improvements and professional development; and iii) Supervision of the instruction in the school. Together,
these indices relate to tasks seeking to enhance and improve the learning process in the schools.

The Administrative Leadership index was defined as the combination of the two remaining school management
indices: i) Accountability role of the principal and ii) Bureaucratic rule-following. Together these indices relate
to administrative tasks, enforcing rules and procedures, and accountability role of the school principal.

The composite scores for each style of leadership were built by taking a simple average of the component
management indices:
N

∑X i
Yj = i =1
N
where Yj is the composite leadership style j, j is the index for each leadership style (instructional and
administrative), N is the number of school management scales for each of the composite scores, Xi is the school
management scale i, and i is the index for each of the school management scales.

Once the simple average of the three school management scales for Instructional Leadership and the two
school management scales for Administrative Leadership were calculated, the scores were transformed to an
international metric of mean zero and standard deviation one.

Finally, it was necessary to impute the value for the Administrative Leadership index for Lithuania, as this
country did not have complete information to estimate the index for the Accountability role of the school
principal. In this case, the Administrative Leadership composite index was derived only from the Bureaucratic
rule-following scale.

School climate indices

Three indices describe the climate of the schools that teachers and principals work in. Two of these indices
are based on principals’ reports: School climate: student delinquency and School climate: teachers’ working
morale. School climate: student delinquency (SCDELINQ) consists of 6 items and School climate: teachers’
working morale (SCTMORAL) of 3 items, that are described in detail in Table 11.39. Response categories were
not at all, very little, to some extent and a lot. Item BCG30A, BCG30B, BCG30C, BCG30D, BCG30E were
excluded due to their low item-total correlation and a poor fit of a model including all items (see Table 11).
These items may be used as single items in further analysis. Additionally teachers were asked to describe
Teacher-student relations (TSRELAT). Four items were used to compute this index (see Table 11.41). Response
categories were strongly disagree, disagree, agree and strongly agree. The response categories disagree and
strongly disagree were collapsed for this scale, because generally only few teachers used the response category
strongly disagree and empty cells for some countries hindered further analysis.

Cronbach’s Alpha is acceptable or good for all of the scales asking about school climate (see Table 11.42).
Only in Estonia and Slovak Republic are the three items measuring School climate: teachers’ working morale
not substantively intercorrelated with reliabilities below 0.400. Latent correlations between student and

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
147

School climate: teachers’ working morale (see Table 11.43) are positive and significant in most countries.
But the strength of the association between the two scales considerably varies across countries. Table 11.44
shows that a CFA model for the two scales in the Principal questionnaire has an acceptable fit in all but four
countries.

Analysis of cross-cultural invariance of the scales shows that model fit hardly deteriorates when factor loadings
are restricted to be equal. The difference between the models examining configural and metric invariance is
small: with DCFI=0.025 and DRMSEA = 0.004 the drop in model fit is close to the criteria established by Chen
(see earlier in the chapter). Additionally, restricting intercepts to be equal leads to a noticeable drop in model fit
(see Table 11.44). This means that the strength of the relations between the scale and each of the items is equal for
all countries, while subjects from different countries differ with regards to the relative tendency to endorse each
of the single items given the same level of the underlying trait. The results confirm the validity of cross-cultural
comparisons of correlations of school climate with other constructs across countries. Mean score comparisons
should be interpreted carefully as the mean scores may have a slightly different meaning for each country.

For the computation of factor scores a model assuming metric invariance was used, because only configural
and metric invariance have been established for both scales. Thus, the same items with the same factor loadings
were used for all participating countries in a multiple group CFA model, but all item intercepts except for
one (BCG30G for the scale School climate: student delinquency and BCG30M for the scale School climate:
teachers’ working morale) and all unique variances were allowed to vary across countries. The parameters
used are detailed in Tables 11.45 to 11.50. The factor score determinacy is acceptable for both scales across all
participating countries, as can be seen in Table 11.51.

Table 11.53 shows that analysing cross-cultural invariance of the scale “teacher student relations” model
fit hardly changes when factor loadings are restricted to be equal across countries compared to a baseline
model with free parameters. The drop in model fit is larger when intercepts are additionally restricted to be
equal. Consequently, relationships with other scales can be validly compared across countries, but mean score
comparisons should be interpreted carefully as the mean scores have a slightly different meaning for each
country.

Factor scores are computed based on the model assuming metric invariance, as this is the highest level of
invariance that has been established. Thus, all item intercepts except for one (BTG31G) and all unique variances
were allowed to vary across countries. The parameters used are detailed in Tables 11.54 to 11.56. The factor
score determinacy shown in Table 11.57 is acceptable for all participating countries.

Classroom disciplinary climate index

To describe the classroom level environment, TALIS measures Classroom disciplinary climate (CCLIMATE).
Table 11.58 shows the wording of the items belonging to this scale. Response categories were “strongly agree”,
“agree”, “disagree” and “strongly disagree”. Items BTG43A, BTG43C and BTG43D are phrased negatively and
were inverted for scaling, so that high scores indicate a positive Classroom disciplinary climate and low scores
a negative climate.

Table 11.59 shows that the index for Classroom disciplinary climate has a remarkable degree of internal
consistency across participating countries. The fit of CFA models is also highly satisfactory at both the national
and the international levels (see Table 11.60).

A comparison of multiple group models restricting different parameters across countries generally supports
the cross-cultural invariance of this scale (see Table 11.61). Adding restrictions on factor loadings only leads
to a small decrease in model fit. The model examining scalar invariance fits worse than the other two models.

TALIS 2008 Technical Report © OECD 2010

148
chapter 11 Construction and Validation of Scales and Indices

This indicates that mean score comparisons should be interpreted carefully as the mean scores may have a
slightly different meaning for each country. Correlations with other constructs can be validly compared across
countries.

For the computation of factor scores a model assuming metric invariance was used. All item intercepts except
for one (BTG43C) and all unique variance variances were allowed to vary across countries. The parameters used
are detailed in Tables 11.62 to 11.64. Table 11.65 shows good a factor score determinacy for all participating
countries.

Self-efficacy index
Self-efficacy (SELFEF) was measured with four items. These are described in detail in Table 11.66. Response
categories were “strongly agree”, “agree”, “disagree” and “strongly disagree”. Only few teachers used the
response category “strongly disagree” and empty cells were found for some countries. Therefore the response
categories “disagree” and “strongly disagree” were collapsed.

The scale Self-efficacy shows acceptable reliabilities, both for the international sample and for the country
sub-samples (see Table 11.67). Table 11.68 shows the results of confirmatory factor analysis. The model fit is
satisfactory for the pooled international sample and for all country sub-samples.

Analysis of cross-cultural invariance of the scale Self-efficacy shows that the scale is valid for international
comparisons of relationships with other constructs. The difference in model fit between the models testing
configural and metric invariance is small. However, mean score comparisons should be conducted with great
care as the drop in model fit adding constraints on the intercepts indicates a slightly different meaning of mean
scores across countries. The results of MGCFA are detailed in Table 11.69.

Metric invariance is the highest level of invariance that was established for Self-efficacy. Therefore the corresponding
model was used for the computation of factor scores. All item intercepts except for one (BTG31D) and all unique
variances were allowed to vary across countries. The parameters used are detailed in Table 11.70 to Table 11.72.
The degree of factor score determinacy is acceptable across countries (see Table 11.73).

Beliefs about instruction indices

To describe teachers’ and principals’ beliefs about instruction three indices were formed: For teachers the
index Direct transmission beliefs about instruction (TBTRAD) and for teachers and principals the index
Constructivist beliefs about instruction (TBCONS and PBCONS). The item wording of the four items measuring
direct transmission beliefs and the four items measuring constructivist beliefs is detailed in Table 11.74. Items
BTG29C and BTG29E were not included in the scales (See Table 11.75). These items did not show a clear
loading pattern, had low item-total-correlations and model fit could be improved by excluding them. They
may still be used as single items in further analysis. Response categories were “strongly agree”, “agree”,
“disagree” and “strongly disagree”. The categories “strongly disagree” and “disagree” were collapsed,
because “strongly disagree” was only utilised by a few teachers and principals resulting in empty cells for
some of the countries.

Reliabilities for the two scales measuring teachers’ beliefs tended to be rather poor (see Table 11.76).
Furthermore, CFA results (see Table 11.77) show only a moderate fit of the two-dimensional model across sub-
samples. In some of the countries the fit of this model is unsatisfactory. Latent correlations between the two scales
vary strongly between countries. For Asian, Latin American, Southern European and some of the Eastern European
countries direct transmission and constructivist beliefs are moderately positively associated. Non-significant or
negative correlations can be found for Northern and Central European countries as well as for Australia.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
149

A comparison between the unrestricted multiple-group model and the model with constrained factor loadings
shows a high degree of invariance for these parameters and provides support for the cross-country validity of this
model. When additional constraints are imposed on the intercepts, a large drop in model fit can be observed. This
signifies that mean score comparisons for these scales cannot unequivocally be interpreted, while there are no
objections to comparing relationships with other constructs across countries (see Table 11.78 and Table 11.79).

For the computation of factor scores, a model assuming metric invariance was used. Thus, the same items with
the same factor loadings were used for all participating countries in a multiple group CFA model. All item
intercepts except for two (BTG29A for the scale Direct transmission beliefs about instruction and BTG29L
for the scale Constructivist beliefs about instruction) and all unique variances were allowed to vary across
countries. The parameters used are detailed in Tables 11.80 to 11.82. Table 11.83 shows a rather poor factor
score determinacy for most of the participating countries. This is due to the comparatively poor fit and the cross-
cultural variance of model parameters for these two scales.

For the principals’ scale measuring Constructivist beliefs about instruction reliabilities were also comparatively
poor, but confirmatory factor analysis shows an excellent fit in all countries, except for Bulgaria and Turkey (see
Tables 11.84 and 11.85).

Comparing the multiple group models, the fit indices with constrained factor loadings are only slightly different
from those with unconstrained loadings for principals’ constructivist beliefs (see Table 11.86). The drop in
CFI is above Chen’s criterion with DCFI = 0.04, but this is still a relatively small drop and the drop in RMSEA
is with DRMSEA = 0.01 consistent with the rule of thumb. A substantial decrease in model fit is observed
when intercepts are additionally restricted. Therefore it must be concluded that principals’ constructivist beliefs
can be used for comparisons of relationships with other constructs. Mean scores of both scales cannot be
unequivocally compared across countries. The poor fit of the models testing scalar invariance indicates that
mean scores may have a different meaning across countries.

For the computation of factor scores, a model assuming metric invariance was used. All item intercepts except
for BCG32L and all unique variance variances were allowed to vary across countries. The parameters used are
detailed in Tables 11.87 to 11.89. Factor score determinancies for the scale Principals’ constructivist beliefs
about instruction indicate an acceptable correlation between the factor score estimates and their respective
factors for most countries. However they are rather poor for 8 of the 23 countries, especially for Lithuania (see
Table 11.90).

Classroom teaching practices indices

Thirteen items measuring teachers’ instructional practices were administered to the teachers. Three scales
were formed: Classroom teaching practice: structuring (TPSTRUC) consisting of five items; Classroom teaching
practice: student-oriented (TPSTUD) consisting of four items; and Classroom teaching practice: enhanced
activities (TPACTIV) also consisting of four items (Table 11.91). Items BTG42A, BTG42G, BTG42K, BTG42L
and BTG42P were not included in the three scales (see Table 11.92). These items either did not show a clear
loading pattern or had low item-total-correlations and model fit could be improved by excluding them. They
can be used as single items for further analysis. The items were answered on six point ordinal scales. Response
categories were “never or hardly ever”, “in about one-quarter of lessons”, “in about one-half of lessons”, “in
about three-quarters of lessons” and “in almost every lesson”.

Reliabilities for the three scales measuring classroom teaching practices are mostly satisfactory (see Table11.93).
Only for the scale Classroom teaching practice: enhanced activities the reliabilities tended to be low for some
countries. Table 11.95 further shows an acceptable model fit for most participating countries. In Table11.93,

TALIS 2008 Technical Report © OECD 2010

150
chapter 11 Construction and Validation of Scales and Indices

reliabilities are presented for the whole sample, but some of the items belonging to the scale Classroom teaching
practice: enhanced activities may only be meaningful for teachers of certain subjects. For example it is not very
likely that a physical education teacher often holds a debate, while it is for a social science teacher. Therefore,
subject specific reliabilities are also detailed in Table 11.94. Furthermore Table 11.96 shows that the model fit
is satisfactory for the subjects reading, writing and literature, mathematics, science, social studies and modern
foreign languages. But it is rather poor for the other subjects.

Correlations between Classroom teaching practice: student-oriented and Classroom teaching practice: enhanced
activities are generally high. Correlations of Classroom teaching practice: structuring and Classroom teaching
practice: student-oriented and Classroom teaching practice: structuring and Classroom teaching practice:
enhanced activities are non-significant to moderate and markedly vary by country. A similar pattern of correlations
can be found for all subject groups.

Tables 11.97, 11.98 and 11.99 show that the fit for multiple group models with unrestricted factor loadings is
only marginally superior to the models with constrained factor loadings. This supports the assumption of metric
invariance across TALIS countries. However, adding restrictions of intercepts leads to a noticeable drop in
model fit for all three scales, indicating that differences in item means are not unequivocally due to differences
in the underlying construct.

Thus, for the three scales measuring classroom teaching practices, only configural and metric invariance were
established. Therefore a model with similar factor loadings, but different item intercepts (except for the items
BTG42M for the scale Classroom teaching practice: structuring, BTG42N for the scale Classroom teaching
practice: student-oriented BTG42Q for the scale Classroom teaching practice: enhanced activities, which were
restricted to be equal for identification purposes) and unique variances was used for the computation of factor
scores. The parameters used are detailed in Tables 11.100 to 11.108. Table 11.109 shows that the factor score
determinacy is acceptably small across all participating countries.

Co-operation among staff indices

Co-operation among staff was measured with 12 items. Two scales were formed: Exchange and co-ordination
for teaching (TCEXCHAN) and Professional collaboration (TCCOLLAB). Both scales consist of five items. Table
11.110 shows a list of items and constructs. Items BTG30A and BTG30B were not included in the scales (see
Table 11.111), because they did not show a clear loading pattern, but rather formed a third factor in many
countries. All items were answered on six point ordinal scales. Response categories were “never”, “less than
once per year”, “once per year”, “3-4 times per year”, “monthly” and “weekly”.

Reliabilities and model fit were satisfactory for the two scales measuring co-operation among staff (see Table 11.112
and Table 11.113). Only for three of the countries, model fit or reliabilities were noticeably below the common
boundaries. Correlations of the two scales measuring co-operation among staff are generally high, showing that
both dimensions are closely related. Still model fit indices and loading patterns suggest two factors. The strength of
the correlation varies between countries. In Denmark model fit is relatively poor and the latent correlation equals
0.900, showing that the two dimensions are not clearly distinguishable for the Danish sample.

Tables 11.114 and 11.115 show that the two scales measuring co-operation among staff are metric-invariant.
Change in model fit is relatively small when additional constraints on factor loadings are added. The drop in
model fit is large when the intercepts are also constricted to be equal across countries. For a given level of the
latent trait, the probability to endorse each of the single items varies across countries and consequently mean
scores for these scales may have a slightly different meaning from one country to the next. This should be
considered analysing descriptive data.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
151

For the computation of factor scores, a model assuming metric invariance was used. All item intercepts except
for three (BTG30C for the scale Exchange and co-ordination for teaching, BTG30J for the scale Professional
collaboration) and all unique variances were allowed to vary across countries. The parameters used are detailed
in Table 11.116 to Table 11.121. The factor score determinacy is rather small for both scales and across countries
(see Table 11.122).

Notes

1. The latent y notation treats the items as endogenous variables that are caused by other variables. However, for CFA with Mplus it
does not make a difference whether the latent x or the latent y notation is used.

2. Model identification concerns the relative ratio of freely estimated parameters (unknowns) to the number of known parameters.
Only when the former exceeds the latter, there is enough information available to obtain a unique set of parameter estimates for each
parameter in the model whose value is unknown.

3. Given the i) presence of missing data, ii) varying levels of missing data in each country, and iii) the fact that a single set of scaled
equal weights were computed, the effective weight with which each country contributed to the transformation may have been slightly
smaller than one.

TALIS 2008 Technical Report © OECD 2010

152
chapter 11 Construction and Validation of Scales and Indices

Table 11.1
Item wording for school autonomy indices
BCG31 Regarding this school, who has a considerable responsibility for the following tasks?
BCG31A Selecting teachers for hire.
BCG31B Firing teachers.
BCG31C Establishing teachers’ starting salaries.
BCG31D Determining teachers’ salary increases.
BCG31E Formulating the school budget.
BCG31F Deciding on budget allocations within the school.
BCG31G Establishing student disciplinary policies.
BCG31H Establishing student assessment policies.
BCG31I Approving students for admission to the school.
BCG31J Choosing which textbooks are used.
BCG31K Determining course content.
BCG31L Deciding which courses are offered.
BCG31M Allocating funds for teachers’ professional development.
Source: OECD, TALIS Database.

Table 11.2
Selection of indices – rotated component matrix
Component
1 2 3 4
BCG31A Selecting teachers for hire. 0.642 0.587 -0.199 0.077
BCG31B Firing teachers. 0.689 0.533 -0.228 0.063
BCG31C Establishing teachers’ starting salaries. 0.866 -0.040 0.213 0.086
BCG31D Determining teachers’ salary increases. 0.855 -0.074 0.170 0.135
BCG31E Formulating the school budget. 0.153 -0.039 0.104 0.828
BCG31F Deciding on budget allocations within the school. 0.073 0.277 0.018 0.768
BCG31G Establishing student disciplinary policies. 0.012 0.689 0.130 0.051
BCG31H Establishing student assessment policies. -0.053 0.622 0.330 0.106
BCG31I Approving students for admission to the school. 0.185 0.534 0.150 0.219
BCG31J Choosing which textbooks are used. 0.077 0.633 0.361 0.026
BCG31K Determining course content. 0.135 0.210 0.805 0.034
BCG31L Deciding which courses are offered. 0.086 0.281 0.742 0.137
BCG31M Allocating funds for teachers’ professional development. 0.453 0.293 0.205 0.351

Source: OECD, TALIS Database.

Table 11.3
Factor loadings used for computation of factor scores for the indices of school autonomy
AUTHIRE AUTBUDGT AUTSTUDP AUTCURR
BCG31A 0.813 BCG31E 0.834 BCG31G 0.793 BCG31J 0.720
BCG31B 0.834 BCG31F 0.834 BCG31H 0.790 BCG31K 0.819
BCG31C 0.785 BCG31I 0.663 BCG31L 0.817
BCG31D 0.772
BCG31M 0.635

Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
153

Table 11.4
Reliabilities for indices of school autonomy
AUTHIRE AUTSTUDP AUTCURR AUTBUDGT
Australia 0.848 0.587 0.239 0.135
Austria 0.699 0.293 0.619 0.244
Belgium (Fl.) 0.379 0.361 0.517 0.466
Brazil 0.944 0.631 0.658 0.696
Bulgaria 0.473 0.381 0.480 0.585
Denmark 0.647 0.560 0.630 0.264
Estonia 0.315 0.529 0.192 0.530
Hungary 0.653 0.651 0.310 0.672
Iceland 0.494 0.309 0.327 0.122
Ireland 0.566 0.691 0.185 0.509
Italy 0.673 0.378 0.377 0.557
Korea 0.768 0.483 0.688 0.379
Lithuania 0.473 0.519 0.682 0.420
Malaysia 0.822 0.659 0.626 0.655
Malta 0.813 0.636 0.854 0.465
Mexico 0.969 0.539 0.854 0.831
Norway 0.567 0.318 0.509 No data
Poland 0.544 0.691 0.678 0.313
Portugal 0.728 0.346 0.411 0.524
Slovak Republic 0.595 0.379 0.648 0.376
Slovenia 0.494 0.668 0.700 0.211
Spain 0.749 0.386 0.591 0.410
Turkey 0.962 0.678 0.684 0.818
Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.5
Item wording for school resource indices
BCG29 Is this school’s capacity to provide instruction hindered by any of the following?
BCG29A Lack of qualified teachers.
BCG29B Lack of laboratory technicians.
BCG29C Lack of instructional support personnel.
BCG29D Lack of other support personnel.
BCG29E Shortage or inadequacy of instructional materials.
BCG29F Shortage or inadequacy of computers for instruction.
BCG29G Shortage or inadequacy of other equipment.
BCG29H Shortage or inadequacy of library materials.
BCG29I Other (Please specify).
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

154
chapter 11 Construction and Validation of Scales and Indices

Table 11.6
Selection of indices-rotated component matrix
Component
1 2
BCG29A Lack of qualified teachers. 0.167 0.562
BCG29B Lack of laboratory technicians. 0.284 0.629
BCG29C Lack of instructional support personnel. 0.149 0.862
BCG29D Lack of other support personnel. 0.177 0.823
BCG29E Shortage or inadequacy of instructional materials. 0.723 0.231
BCG29F Shortage or inadequacy of computers for instruction. 0.802 0.248
BCG29G Shortage or inadequacy of other equipment. 0.858 0.204
BCG29H Shortage or inadequacy of library materials. 0.796 0.161

Source: OECD, TALIS Database.

Table 11.7
Factor loadings used for computation of factor scores for the indices of school resources
LACKPERS LACKMAT
BCG29A 0.594 BCG29E 0.762
BCG29B 0.706 BCG29F 0.845
BCG29C 0.859 BCG29G 0.885
BCG29D 0.830 BCG29H 0.807

Source: OECD, TALIS Database.

Table 11.8
Reliabilities for indices of school resources
LACKPERS LACKMAT
Australia 0.898 0.925
Austria 0.376 0.631
Belgium (Fl.) 0.857 0.945
Brazil 0.789 0.868
Bulgaria 0.716 0.909
Denmark 0.864 0.942
Estonia 0.832 0.906
Hungary 0.610 0.814
Iceland 0.975 0.977
Ireland 0.494 0.679
Italy 0.537 0.775
Korea 0.961 0.943
Lithuania 0.671 0.823
Malaysia 0.934 0.897
Malta 0.808 0.499
Mexico 0.830 0.885
Norway 0.800 0.856
Poland 0.832 0.776
Portugal 0.721 0.794
Slovak Republic 0.686 0.751
Slovenia 0.760 0.779
Spain 0.790 0.879
Turkey 0.881 0.907
Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
155

Table 11.9
International means and standard deviations of school- and teacher-level factor
score estimates using equally weighted pooled data
Mean Standard deviation
School-level indices
PBCONS -0.113 0.269
SCDELINQ 0.000 0.626
SCTMORAL -0.137 0.693
FCSGCD -0.247 0.388
PROIIPD 0.284 0.379
SUPINSTR 0.097 0.483
ACCROLE 0.069 0.310
BURRULEF 0.332 0.351
ADMINL -0.004 0.897
INSTRL 0.000 0.744

Teacher-level indices
CCLIMATE -0.029 0.658
TSRELAT -0.071 0.297
SELFEF -0.107 0.335
TPSTRUC -0.069 0.615
TPSTUD 0.221 0.770
TPACTIV 0.087 0.608
TBTRAD -0.072 0.203
TBCONS 0.049 0.235
TCEXCHAN -0.327 0.516
TCCOLLAB -0.308 0.789
Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.10
Item wording of school leadership indices and dimensions
Please indicate the frequency of these activities and behaviours in this school during the current year (BCG15);
How strongly do you agree or disagree with these statements as applied to this school,
your job and the teachers at this school (BCG16)
Framing and communicating I make sure that the professional development activities of teachers are in accordance with the teaching goals
BCG15A
the school’s goals and of the school.
curricular development scale BCG15B I ensure that teachers work according to the school’s educational goals.
BCG15D I use student performance results to develop the school’s educational goals.
BCG15J I take exam results into account in decisions regarding curriculum development.
BCG15K I ensure that there is clarity concerning the responsibility for co-ordinating the curriculum.
BCG16M In this school, we work on goals and/or a school development plan.
Promoting instructional BCG15G When a teacher has problems in his/her classroom, I take the initiative to discuss matters.
improvements and BCG15H I inform teachers about possibilities for updating their knowledge and skills.
professional development
BCG15L When a teacher brings up a classroom problem, we solve the problem together.
BCG15M I pay attention to disruptive behaviour in classrooms.
Supervision of the instruction BCG15C I observe instruction in classrooms.
in the school scale BCG15E I give teachers suggestions as to how they can improve their teaching.
BCG15F I monitor students’ work.
BCG15I I check to see whether classroom activities are in keeping with our educational goals.
Accountability role of An important part of my job is to ensure that ministry-approved instructional approaches are explained to new
BCG16A
the principal teachers, and that more experienced teachers are using these approaches.
BCG16D A main part of my job is to ensure that the teaching skills of the staff are always improving.
BCG16E An important part of my job is to ensure that teachers are held accountable for the attainment of the school’s goals.
BCG16F An important part of my job is to present new ideas to the parents in a convincing way.
Bureaucratic BCG16H It is important for the school that I see to it that everyone sticks to the rules.
Rule-Following scale BCG16I It is important for the school that I check for mistakes and errors in administrative procedures and reports.
BCG16J An important part of my job is to resolve problems with the timetable and/or lesson planning.
BCG16K An important part of my job is to create an orderly atmosphere in the school.
BCG16O I stimulate a task-oriented atmosphere in this school.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

156
chapter 11 Construction and Validation of Scales and Indices

Table 11.11
Single items measuring school leadership and management behaviours
Please indicate the frequency of these activities and behaviours in this school during the current year (BCG15).
How strongly do you agree or disagree with these statements as applied to this school,
your job and the teachers at this school (BCG16)
Single items BCG15N I take over lessons from teachers who are unexpectedly absent.
BCG16B Using test scores of students to evaluate a teacher’s performance devalues the teacher’s professional judgment.
BCG16C Giving teachers too much freedom to choose their own instructional techniques can lead to poor teaching.
BCG16G I influence decisions about this school taken at a higher administrative level.
BCG16L I have no way of knowing whether teachers are performing well or badly in their teaching duties.
BCG16N I define goals to be accomplished by the staff of this school.

Source: OECD, TALIS Database.

Table 11.12
Reliabilities for indices of school leadership

FCSGCD PROIIPD SUPINSTR ACCROLE BURRULEF

Australia 0.777 0.700 0.682 0.700 0.694
Austria 0.681 0.596 0.533 0.558 0.616
Belgium (Fl.) 0.739 0.677 0.607 0.632 0.569
Brazil 0.823 0.664 0.734 0.668 0.685
Bulgaria 0.649 0.455 0.628 0.563 0.669
Denmark 0.631 0.652 0.402 0.517 0.394
Estonia 0.692 0.668 0.612 0.644 0.440
Hungary 0.647 0.586 0.670 0.634 0.693
Iceland 0.748 0.634 0.464 0.671 0.654
Ireland 0.739 0.649 0.638 0.627 0.635
Italy 0.653 0.597 0.739 0.605 0.477
Korea 0.688 0.681 0.671 0.627 0.782
Lithuania 0.763 0.622 0.597 m 0.743
Malaysia 0.817 0.780 0.761 0.642 0.726
Malta 0.715 0.631 0.687 0.621 0.674
Mexico 0.762 0.639 0.676 0.678 0.715
Norway 0.744 0.662 0.520 0.575 0.500
Poland 0.730 0.679 0.537 0.470 0.446
Portugal 0.699 0.607 0.728 0.683 0.683
Slovak Republic 0.714 0.737 0.589 0.539 0.590
Slovenia 0.681 0.607 0.673 0.529 0.558
Spain 0.720 0.619 0.737 0.687 0.729
Turkey 0.832 0.797 0.749 0.712 0.789
International sample 0.799 0.724 0.745 0.650 0.694

Netherlands 0.488 0.747 0.546 0.447 0.590

Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
m = missing
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
157

Table 11.13
Model fit for the scale Framing and communicating the school’s goals and curricular development
CFI TLI RMSEA SRMR
Australia 0.946 0.911 0.086 0.055
Austria 1.000 1.023 0.000 0.035
Belgium (Fl.) 0.981 0.959 0.057 0.035
Brazil 0.976 0.954 0.070 0.043
Bulgaria 0.878 0.797 0.107 0.056
Denmark 0.905 0.821 0.099 0.064
Estonia 0.956 0.927 0.064 0.046
Hungary 0.834 0.689 0.127 0.073
Iceland 0.986 0.973 0.047 0.041
Ireland1 0.929 0.867 0.094 0.050
Italy1 0.926 0.877 0.067 0.047
Korea1 1.000 1.012 0.000 0.036
Lithuania 1.000 1.014 0.000 0.028
Malaysia 1.000 1.032 0.000 0.022
Malta 0.944 0.880 0.103 0.079
Mexico 0.943 0.905 0.079 0.050
Norway 0.947 0.900 0.086 0.044
Poland 0.976 0.955 0.071 0.038
Portugal 0.995 0.990 0.027 0.040
Slovak Republic 1.000 1.037 0.000 0.029
Slovenia1 1.000 0.999 0.004 0.038
Spain 0.942 0.876 0.084 0.047
Turkey1 0.981 0.959 0.073 0.030
International Sample 0.999 0.997 0.013 0.008

Netherlands 0.853 0.755 0.073 0.093

1. The model estimation terminated normally but the standard errors are not trustworthy.
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.14
Model fit for the scale Promoting instructional improvements and professional development
CFI TLI RMSEA SRMR
Australia 0.991 0.973 0.052 0.027
Austria 0.955 0.864 0.109 0.034
Belgium (Fl.) 0.916 0.747 0.151 0.045
Brazil1 1.000 1.077 0.000 0.007
Bulgaria 0.943 0.828 0.078 0.032
Denmark 0.978 0.933 0.052 0.030
Estonia 1.000 1.082 0.000 0.006
Hungary 0.911 0.734 0.153 0.051
Iceland1 0.956 0.868 0.102 0.038
Ireland 1.000 1.005 0.000 0.027
Italy 0.967 0.900 0.077 0.035
Korea 1.000 1.009 0.000 0.019
Lithuania 0.948 0.845 0.113 0.034
Malaysia 0.929 0.788 0.186 0.039
Malta1 1.000 1.075 0.000 0.034
Mexico 1.000 1.067 0.000 0.012
Norway 1.000 1.039 0.000 0.016
Poland 0.888 0.664 0.151 0.061
Portugal1 0.951 0.852 0.090 0.036
Slovak Republic 0.989 0.966 0.068 0.025
Slovenia 0.916 0.748 0.132 0.040
Spain 0.969 0.907 0.069 0.032
Turkey 0.952 0.855 0.138 0.045
International Sample 0.995 0.986 0.032 0.010

Netherlands 1.000 1.109 0.000 0.006

TALIS 2008 Technical Report © OECD 2010

158
chapter 11 Construction and Validation of Scales and Indices

Table 11.15
Model fit for the scale Supervision of instruction in the school
CFI TLI RMSEA SRMR
Australia 0.914 0.742 0.152 0.037
Austria 1.000 1.144 0.000 0.015
Belgium (Fl.) 0.997 0.980 0.039 0.011
Brazil 1.000 1.023 0.000 0.015
Bulgaria 1.000 1.073 0.000 0.011
Denmark1 1.000 1.095 0.000 0.025
Estonia 1.000 1.083 0.000 0.019
Hungary 0.930 0.581 0.233 0.036
Iceland 1.000 1.194 0.000 0.024
Ireland 1.000 1.108 0.000 0.015
Italy 0.982 0.947 0.087 0.022
Korea 1.000 1.017 0.000 0.017
Lithuania 0.980 0.939 0.064 0.028
Malaysia 0.976 0.929 0.099 0.026
Malta 0.902 0.707 0.162 0.057
Mexico 1.000 1.029 0.000 0.016
Norway 0.953 0.858 0.099 0.040
Poland 0.998 0.994 0.013 0.030
Portugal 1.000 1.025 0.000 0.016
Slovak Republic 1.000 1.025 0.000 0.021
Slovenia 1.000 1.080 0.000 0.012
Spain 0.988 0.963 0.064 0.019
Turkey 1.000 1.008 0.000 0.015
International Sample 1.000 1.001 0.000 0.004

Netherlands 1.000 2.754 0.000 0.032

Table 11.16
Model fit for the scale Accountability role of the principal
CFI TLI RMSEA SRMR
Australia 1.000 1.020 0.000 0.017
Austria 0.802 0.407 0.175 0.044
Belgium (Fl.) 0.977 0.931 0.065 0.030
Brazil 0.865 0.596 0.154 0.040
Bulgaria 1.000 1.030 0.000 0.012
Denmark 1.000 1.197 0.000 0.009
Estonia 0.981 0.943 0.059 0.028
Hungary1 0.999 0.998 0.013 0.023
Iceland 1.000 1.096 0.000 0.016
Ireland 0.872 0.617 0.162 0.039
Italy1 0.989 0.966 0.051 0.033
Korea 0.995 0.986 0.036 0.022
Lithuania m m m m
Malaysia 1.000 1.097 0.000 0.009
Malta 0.867 0.602 0.151 0.050
Mexico 0.979 0.938 0.079 0.026
Norway 1.000 1.083 0.000 0.029
Poland 0.990 0.971 0.042 0.021
Portugal 0.940 0.819 0.123 0.044
Slovak Republic1 0.972 0.917 0.066 0.040
Slovenia 1.000 1.023 0.000 0.026
Spain 1.000 0.999 0.012 0.020
Turkey1 1.000 1.045 0.000 0.005
International Sample 0.998 0.993 0.020 0.008

Netherlands 0.591 -0.227 0.232 0.090

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
159

Table 11.17
Model fit for the scale Bureaucratic rule-following
CFI TLI RMSEA SRMR
Australia 0.910 0.821 0.104 0.047
Austria 0.938 0.876 0.086 0.047
Belgium (Fl.) 0.882 0.764 0.102 0.105
Brazil 0.990 0.979 0.030 0.036
Bulgaria1 0.983 0.957 0.062 0.034
Denmark1 0.712 0.423 0.155 0.069
Estonia 0.931 0.861 0.062 0.042
Hungary1 0.993 0.983 0.033 0.026
Iceland1 1.000 1.167 0.000 0.018
Ireland1 1.000 1.025 0.000 0.035
Italy1 0.646 0.292 0.124 0.055
Korea 0.972 0.930 0.097 0.034
Lithuania1 1.000 1.032 0.000 0.017
Malaysia 0.969 0.938 0.059 0.040
Malta 1.000 1.158 0.000 0.037
Mexico1 0.982 0.963 0.044 0.035
Norway 0.964 0.929 0.050 0.037
Poland 0.834 0.667 0.097 0.050
Portugal 0.970 0.941 0.070 0.036
Slovak Republic 0.896 0.792 0.089 0.046
Slovenia 1.000 1.003 0.000 0.032
Spain 0.940 0.849 0.103 0.041
Turkey 1.000 1.005 0.000 0.021
International Sample 1.000 1.001 0.000 0.004

Netherlands 0.938 0.876 0.068 0.070

Table 11.18
Multiple group CFA examining different levels of cross-cultural invariance
of the scale Framing and communicating the school’s goals and curricular development
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural Invariance 0.824 0.710 0.143 0.067 - -
Metric Invariance 0.810 0.794 0.121 0.145 -0.014 -0.022
Scalar Invariance 0.458 0.562 0.186 0.281 -0.352 0.065
Uniqueness 0.000 0.264 0.241 1.166 -0.458 0.055
Note: Model estimates based on international calibration sample (1 000 teachers per country).
Source: OECD, TALIS Database.

Table 11.19
Multiple group CFA examining different levels of cross-cultural invariance
of the scale Promoting instructional improvements and professional development
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural Invariance 1 1 1 1 - -
Metric Invariance 0.907 0.885 0.097 0.139 - -
Scalar Invariance 0.509 0.619 0.177 0.213 -0.398 0.080
Uniqueness 0.000 0.036 0.282 1.088 -0.509 0.105
1. The model estimation terminated normally but the standard errors are not trustworthy and the model fit index are not used for comparison purposes.
Note: Model estimates based on international calibration sample (1 000 teachers per country).
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

160
chapter 11 Construction and Validation of Scales and Indices

Table 11.20
Multiple group CFA examining different levels of cross-cultural invariance
of the scale Supervision of instruction in the school
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA
Configural Invariance 0.974 0.922 0.080 0.030 - -
Metric Invariance 0.937 0.922 0.080 0.099 -0.037 0.000
Scalar Invariance 0.243 0.413 0.220 0.276 -0.694 0.140
Uniqueness 0.000 -0.241 0.321 1.106 -0.243 0.101
Note: Model estimates based on international calibration sample (1 000 teachers per country).
Source: OECD, TALIS Database.

Table 11.21
Multiple group CFA examining different levels of cross-cultural invariance
of the scale Accountability role of the principal
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural Invariance * * * * - -
Metric Invariance 0.892 0.866 0.098 0.145 - -
Scalar Invariance 0.235 0.406 0.207 0.242 -0.657 0.109
Uniqueness 0.000 -0.312 0.307 1.117 -0.235 0.100
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
“*” appears when model estimation terminated normally but the standard errors are not trustworthy and the model fit index are not used for comparison purposes.
Source: OECD, TALIS Database.

Table 11.22
Multiple group CFA examining different levels of cross-cultural invariance
of principals’ Bureaucratic rule-following
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA
Configural Invariance 0.879 0.759 0.120 0.051 - -
Metric Invariance 0.838 0.817 0.105 0.133 -0.041 -0.015
Scalar Invariance 0.258 0.413 0.188 0.338 -0.580 0.083
Uniqueness 0.000 -0.304 0.280 1.225 -0.258 0.092
Note: Model estimates based on international calibration sample (1 000 teachers per country).
Source: OECD, TALIS Database.

Table 11.23
Factor loadings used for the computation of factor scores for the indicator
Framing and communicating the school’s goals and curricular development

BCG15A BCG15B BCG15D BCG15J BCG15K BCG16M
International sample 1.000 0.908 0.840 0.973 0.951 0.631
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
161

Table 11.24
Item intercepts used for the computation of factor scores for the indicator
Framing and communicating the school’s goals and curricular development, by country
BCG15A BCG15B BCG15D BCG15J BCG15K BCG16M
Australia 3.537 3.423 2.997 3.537 3.493
Austria 3.783 3.248 2.612 3.449 3.876
Belgium (Fl.) 3.697 2.986 3.167 3.260 3.543
Brazil 3.657 3.609 3.375 3.524 3.544
Bulgaria 3.771 3.256 2.841 3.441 3.567
Denmark 3.589 2.908 2.832 3.619 3.609
Estonia 3.584 3.366 2.950 3.468 3.507
Hungary 3.728 3.332 2.864 3.079 3.314
Iceland 3.470 2.848 2.880 3.308 3.458
Ireland 3.627 2.964 3.065 3.456 3.574
Italy 3.942 3.490 3.527 3.725 3.723
Korea 3.608 3.670 3.339 3.029 3.584 3.594
Lithuania 3.736 3.473 3.117 3.598 3.599
Malaysia 3.872 3.459 3.827 3.746 3.923
Malta 3.526 3.029 3.139 3.119 3.659
Mexico 3.696 3.465 3.328 3.428 3.613
Norway 3.600 3.294 3.042 3.436 3.757
Poland 3.743 3.460 3.097 3.113 3.274
Portugal 3.675 3.557 3.498 3.425 3.343
Slovak Republic 3.615 3.067 2.817 3.401 3.584
Slovenia 3.796 2.982 2.943 3.053 3.301
Spain 3.732 3.469 3.160 3.589 3.503
Turkey 3.740 3.465 3.161 3.553 3.635

Netherlands 3.514 3.065 3.109 3.445 3.382

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.25
Unique variances used for the computation of factor scores for the indicator
Framing and communicating the school’s goals and curricular development, by country

BCG15A BCG15B BCG15D BCG15J BCG15K BCG16M
Australia 0.131 0.144 0.320 0.424 0.178 0.319
Austria 0.354 0.259 0.286 0.294 0.388 0.269
Belgium (Fl.) 0.147 0.129 0.311 0.499 0.262 0.186
Brazil 0.137 0.125 0.200 0.286 0.253 0.235
Bulgaria 0.130 0.120 0.245 0.592 0.259 0.182
Denmark 0.169 0.225 0.432 0.543 0.297 0.254
Estonia 0.126 0.157 0.262 0.431 0.323 0.300
Hungary 0.200 0.172 0.289 0.643 0.364 0.225
Iceland 0.216 0.212 0.326 0.236 0.351 0.258
Ireland 0.241 0.191 0.385 0.419 0.212 0.191
Italy 0.260 0.168 0.404 0.384 0.245 0.248
Korea 0.095 0.117 0.231 0.388 0.223 0.250
Lithuania 0.171 0.155 0.154 0.347 0.270 0.202
Malaysia 0.214 0.158 0.224 0.175 0.133 0.213
Malta 0.257 0.195 0.223 0.518 0.231 0.246
Mexico 0.244 0.161 0.248 0.256 0.153 0.365
Norway 0.216 0.149 0.234 0.399 0.224 0.288
Poland 0.107 0.076 0.188 0.550 0.377 0.239
Portugal 0.176 0.119 0.238 0.332 0.232 0.245
Slovak Republic 0.264 0.178 0.351 0.321 0.246 0.208
Slovenia 0.188 0.124 0.271 0.248 0.223 0.210
Spain 0.446 0.207 0.383 0.417 0.273 0.290
Turkey 0.117 0.119 0.290 0.523 0.149 0.251

Netherlands 0.206 0.140 0.412 0.292 0.390 0.403

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

162
chapter 11 Construction and Validation of Scales and Indices

Table 11.26
Factor loadings used for the computation of factor scores for the indicator
Promoting instructional improvements and professional development

BCG15G BCG15H BCG15L BCG15M
International sample 1.000 0.730 1.073 0.888
Source: OECD, TALIS Database.

Table 11.27
Item intercepts used for the computation of factor scores for the indicator
Promoting instructional improvements and professional development, by country
BCG15G BCG15H BCG15L BCG15M
Australia 3.210 2.989 3.265
Austria 2.919 3.073 3.130
Belgium (Fl.) 3.070 3.171 3.166
Brazil 3.142 3.033 3.198
Bulgaria 3.308 3.091 3.342
Denmark 2.799 2.816 2.919
Estonia 3.375 3.396 3.197
Hungary 3.059 2.510 3.205
Iceland 3.247 3.435 2.896
Ireland 3.186 3.327 3.505
Italy 3.223 3.061 3.034
Korea 2.860 2.973 3.223 3.007
Lithuania 3.215 3.127 3.050
Malaysia 3.193 3.239 3.458
Malta 2.808 3.008 3.200
Mexico 3.300 3.359 3.444
Norway 2.820 3.083 3.119
Poland 3.069 3.128 3.080
Portugal 3.070 3.244 3.398
Slovak Republic 3.397 3.457 3.316
Slovenia 3.116 3.061 2.901
Spain 2.920 3.203 3.483
Turkey 2.862 2.877 3.136

Netherlands 2.908 2.865 2.482

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
163

Table 11.28
Unique variances used for the computation of factor scores for the indicator
Promoting instructional improvements and professional development, by country
BCG15G BCG15H BCG15L BCG15M
Australia 0.262 0.282 0.235 0.294
Austria 0.248 0.368 0.242 0.214
Belgium (Fl.) 0.248 0.314 0.157 0.178
Brazil 0.211 0.204 0.058 0.100
Bulgaria 0.291 0.254 0.165 0.247
Denmark 0.228 0.205 0.156 0.217
Estonia 0.240 0.282 0.192 0.366
Hungary 0.239 0.353 0.244 0.199
Iceland 0.230 0.317 0.114 0.368
Ireland 0.238 0.261 0.130 0.144
Italy 0.285 0.233 0.123 0.198
Korea 0.210 0.174 0.154 0.122
Lithuania 0.188 0.271 0.137 0.453
Malaysia 0.215 0.216 0.161 0.232
Malta 0.176 0.365 0.105 0.157
Mexico 0.414 0.254 0.166 0.187
Norway 0.181 0.217 0.164 0.216
Poland 0.261 0.263 0.187 0.305
Portugal 0.284 0.368 0.141 0.115
Slovak Republic 0.376 0.207 0.201 0.223
Slovenia 0.222 0.238 0.169 0.242
Spain 0.481 0.483 0.219 0.182
Turkey 0.158 0.240 0.152 0.207

Netherlands 0.256 0.116 0.104 0.291

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.29
Factor loadings used for the computation of factor scores for the indicator
Supervision of instruction in the school

BCG15C BCG15E BCG15F BCG15I
International sample 1.000 0.965 0.960 1.015
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

164
chapter 11 Construction and Validation of Scales and Indices

Table 11.30
Item intercepts used for the computation of factor scores for the indicator
Supervision of instruction in the school, by country
BCG15C BCG15E BCG15F BCG15I
Australia 2.704 2.382 2.741
Austria 2.760 3.236 2.949
Belgium (Fl.) 2.721 2.681 2.705
Brazil 2.775 2.327 2.714
Bulgaria 2.439 2.716 2.930
Denmark 2.814 3.493 3.169
Estonia 2.844 2.957 2.752
Hungary 2.829 3.065 2.818
Iceland 2.743 2.926 2.609
Ireland 2.930 3.071 3.169
Italy 2.501 1.906 2.671
Korea 2.486 2.835 3.030 2.628
Lithuania 3.010 2.520 2.997
Malaysia 2.905 2.523 3.019
Malta 2.795 2.572 2.986
Mexico 2.714 2.520 2.779
Norway 2.833 2.387 2.914
Poland 2.515 2.667 2.527
Portugal 3.405 3.231 3.353
Slovak Republic 2.209 2.425 2.422
Slovenia 2.371 2.598 2.295
Spain 2.836 2.466 2.846
Turkey 2.699 2.989 2.883

Netherlands 2.664 1.414 2.599

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.31
Unique variances used for the computation of factor scores for the indicator
Supervision of instruction in the school, by country
BCG15C BCG15E BCG15F BCG15I
Australia 0.188 0.292 0.369 0.315
Austria 0.310 0.248 0.280 0.336
Belgium (Fl.) 0.178 0.234 0.394 0.227
Brazil 0.339 0.270 0.297 0.204
Bulgaria 0.178 0.375 0.136 0.248
Denmark 0.239 0.335 0.218 0.300
Estonia 0.206 0.289 0.317 0.329
Hungary 0.211 0.318 0.305 0.257
Iceland 0.208 0.333 0.214 0.212
Ireland 0.283 0.354 0.465 0.342
Italy 0.402 0.316 0.230 0.277
Korea 0.218 0.175 0.157 0.188
Lithuania 0.251 0.164 0.273 0.227
Malaysia 0.241 0.205 0.240 0.267
Malta 0.299 0.258 0.251 0.191
Mexico 0.232 0.242 0.340 0.147
Norway 0.216 0.248 0.208 0.268
Poland 0.231 0.138 0.299 0.193
Portugal 0.313 0.256 0.493 0.318
Slovak Republic 0.384 0.342 0.342 0.202
Slovenia 0.233 0.187 0.183 0.199
Spain 0.371 0.291 0.415 0.288
Turkey 0.345 0.227 0.323 0.089

Netherlands 0.341 0.208 0.377 0.239

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
165

Table 11.32
Factor loadings used for the computation of factor scores for the indicator
Accountability role of the principal

BCG16A BCG16D BCG16E BCG16F
International sample 1.000 1.182 1.301 1.173
Source: OECD, TALIS Database.

Table 11.33
Item intercepts used for the computation of factor scores for the indicator
Accountability role of the principal, by country
BCG16A BCG16D BCG16E BCG16F
Australia 3.642 3.324 3.371
Austria 2.875 2.743 3.127
Belgium (Fl.) 3.396 3.345 3.059
Brazil 3.245 2.880 3.015
Bulgaria 2.854 2.811 2.583
Denmark 3.622 3.670 3.523
3.021
Estonia 3.466 2.923 2.988
Hungary 3.562 3.280 3.428
Iceland 3.118 3.425 3.372
Ireland 3.206 2.962 3.199
Italy 3.249 3.434 3.099
Korea 3.435 3.483 2.995
Lithuania m m m m
Malaysia 3.079 3.259 2.741
Malta 3.574 3.251 3.384
Mexico 3.230 3.404 3.159
Norway 3.260 2.928 2.934
Poland 3.056 3.131 3.150
3.021
Portugal 2.985 3.297 3.029
Slovak Republic 2.557 3.303 3.086
Slovenia 2.782 3.258 3.123
Spain 3.295 3.667 3.497
Turkey 3.079 3.341 3.181

Netherlands - - - -
Notes: The Netherlands was excluded due to stability problems with the scale.
Lithuania did not administer item BCG16A and was therefore excluded from this scale.
m = missing
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

166
chapter 11 Construction and Validation of Scales and Indices

Table 11.34
Unique variances used for the computation of factor scores for the indicator
Accountability role of the principal, by country
BCG16A BCG16D BCG16E BCG16F
Australia 0.423 0.168 0.151 0.153
Austria 0.510 0.410 0.400 0.335
Belgium (Fl.) 0.228 0.158 0.144 0.304
Brazil 0.307 0.167 0.232 0.217
Bulgaria 0.491 0.199 0.106 0.236
Denmark 0.539 0.184 0.189 0.356
Estonia 0.441 0.311 0.396 0.353
Hungary 0.320 0.148 0.158 0.206
Iceland 0.434 0.368 0.085 0.149
Ireland 0.316 0.201 0.210 0.211
Italy 0.308 0.160 0.090 0.376
Korea 0.148 0.175 0.141 0.198
Lithuania m m m m
Malaysia 0.294 0.182 0.134 0.246
Malta 0.324 0.187 0.205 0.254
Mexico 0.455 0.221 0.118 0.160
Norway 0.352 0.171 0.381 0.303
Poland 0.503 0.338 0.142 0.243
Portugal 0.291 0.171 0.167 0.224
Slovak Republic 0.276 0.377 0.130 0.201
Slovenia 0.314 0.317 0.187 0.223
Spain 0.389 0.290 0.168 0.216
Turkey 0.242 0.122 0.166 0.211

Table 11.35
Factor loadings used for the computation of factor scores for the indicator
Bureaucratic rule-following

BCG16H BCG16I BCG16J BCG16K BCG16O
International sample 1.000 1.208 1.063 0.823 0.539
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
167

Table 11.36
Item intercepts used for the computation of factor scores for the indicator
Bureaucratic rule-following, by country
BCG16H BCG16I BCG16J BCG16K BCG16O
Australia 2.850 2.426 3.421 3.021
Austria 2.510 2.485 3.189 3.014
Belgium (Fl.) 2.771 2.326 3.011 2.916
Brazil 2.990 2.721 3.162 2.916
Bulgaria 2.628 2.729 3.226 3.048
Denmark 2.900 2.647 3.937 3.436
Estonia 2.688 1.596 2.989 3.237
Hungary 2.634 2.648 3.202 3.277
Iceland 2.659 3.020 3.176 3.026
Ireland 2.758 3.102 3.452 2.745
Italy 2.829 2.311 3.371 3.141
Korea 3.025 2.940 2.594 3.158 3.082
Lithuania 2.761 2.295 3.188 3.300
Malaysia 2.701 2.749 3.201 2.930
Malta 2.707 2.800 3.524 2.984
Mexico 2.868 2.905 3.230 3.374
Norway 2.852 2.537 3.221 3.340
Poland 2.544 2.840 3.268 2.927
Portugal 2.640 2.635 3.362 3.146
Slovak Republic 2.570 2.431 3.460 3.264
Slovenia 2.675 2.416 3.335 3.032
Spain 3.106 2.748 3.166 3.213
Turkey 2.668 2.423 3.233 2.902

Netherlands 2.908 2.290 2.950 2.902

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.37
Unique variances used for the computation of factor scores for the indicator
Bureaucratic rule-following, by country
BCG16H BCG16I BCG16J BCG16K BCG16O
Australia 0.254 0.272 0.383 0.316 0.309
Austria 0.269 0.311 0.708 0.341 0.349
Belgium (Fl.) 0.183 0.187 0.482 0.316 0.187
Brazil 0.218 0.102 0.365 0.284 0.259
Bulgaria 0.165 0.121 0.248 0.118 0.229
Denmark 0.295 0.314 0.525 0.141 0.248
Estonia 0.278 0.328 0.442 0.341 0.254
Hungary 0.175 0.130 0.351 0.183 0.226
Iceland 0.170 0.326 0.370 0.328 0.216
Ireland 0.240 0.263 0.187 0.135 0.430
Italy 0.199 0.240 0.391 0.137 0.238
Korea 0.155 0.096 0.330 0.202 0.259
Lithuania 0.135 0.173 0.333 0.168 0.207
Malaysia 0.181 0.129 0.239 0.128 0.238
Malta 0.209 0.233 0.472 0.165 0.368
Mexico 0.199 0.239 0.245 0.235 0.195
Norway 0.326 0.197 0.572 0.207 0.305
Poland 0.155 0.493 0.275 0.361 0.212
Portugal 0.203 0.204 0.354 0.179 0.212
Slovak Republic 0.234 0.260 0.402 0.170 0.255
Slovenia 0.162 0.299 0.412 0.191 0.162
Spain 0.283 0.136 0.478 0.282 0.288
Turkey 0.183 0.086 0.341 0.122 0.222

Netherlands 0.151 0.123 0.538 0.289 0.077

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

168
chapter 11 Construction and Validation of Scales and Indices

Table 11.38
Factor score determinacy for the scales related to school principal’s leadership, by country
FCSGCD PROIIPD SUPINSTR ACCROLE BURRULEF
Australia 0.896 0.855 0.853 0.865 0.835
Austria 0.830 0.798 0.724 0.783 0.830
Belgium (Fl.) 0.866 0.817 0.794 0.808 0.795
Brazil 0.914 0.864 0.866 0.832 0.890
Bulgaria 0.845 0.798 0.827 0.856 0.870
Denmark 0.828 0.805 0.611 0.765 0.680
Estonia 0.876 0.826 0.758 0.784 0.698
Hungary 0.785 0.816 0.809 0.809 0.864
Iceland 0.875 0.831 0.727 0.858 0.836
Ireland 0.862 0.822 0.805 0.817 0.833
Italy 0.833 0.804 0.832 0.850 0.720
Korea 0.874 0.849 0.829 0.833 0.925
Lithuania 0.870 0.854 0.822 m 0.883
Malaysia 0.908 0.887 0.861 0.780 0.856
Malta 0.863 0.846 0.816 0.789 0.845
Mexico 0.888 0.823 0.843 0.872 0.853
Norway 0.867 0.851 0.740 0.777 0.804
Poland 0.905 0.858 0.785 0.754 0.769
Portugal 0.871 0.798 0.857 0.835 0.865
Slovak Republic 0.844 0.879 0.825 0.783 0.785
Slovenia 0.811 0.794 0.820 0.747 0.781
Spain 0.876 0.803 0.874 0.832 0.852
Turkey 0.897 0.871 0.878 0.870 0.892

Netherlands 0.655 0.907 0.688 - 0.830

Notes: The Netherlands was excluded due to stability problems with the scale.
Lithuania did not administer item BCG16A and was therefore excluded from this scale.
m = missing
Source: OECD, TALIS Database.

Table 11.39
Item wording of school climate items and dimensions – Principal Questionnaire
In this school, to what extent is the learning of students hindered by the following behaviours?
School climate: ctudent BCG30F Vandalism.
delinquency BCG30G Theft.
BCG30H Intimidation or verbal abuse of other students (or other forms of bullying).
BCG30I Physical injury to other students.
BCG30J Intimidation or verbal abuse of teachers or staff.
BCG30K Use/possession of drugs and/or alcohol.
School climate: teachers’ BCG30L Arriving late at school.
working morale BCG30M Absenteeism.
BCG30N Lack of pedagogical preparation.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
169

Table 11.40
Single items measuring aspects of school climate
In this school, to what extent is the learning of students hindered by the following behaviours?
Single items BCG30A Arriving late at school.
BCG30B Absenteeism (i.e. unjustified absences).
BCG30C Classroom disturbance.
BCG30D Cheating.
BCG30E Profanity/Swearing.
Source: OECD, TALIS Database.

Table 11.41
Item wording of school climate items and dimensions – Teacher Questionnaire
How strongly do you agree or disagree with the following statements …
… about what happens in this school?
Teacher-student relations BTG31G In this school, teachers and students usually get on well with each other.
BTG31H Most teachers in this school believe that students’ well-being is important.
BTG31I Most teachers in this school are interested in what students have to say.
BTG31J If a student from this school needs extra assistance, the school provides it.
Source: OECD, TALIS Database.

Table 11.42
Reliabilities for school climate indices, by country

SCDELINQ (PQ) SCTMORAL (PQ) TSRELAT (TQ)
Australia 0.846 0.669 0.815
Austria 0.877 0.752 0.779
Belgium (Fl.) 0.839 0.671 0.780
Brazil 0.887 0.733 0.741
Bulgaria 0.824 0.626 0.802
Denmark 0.901 0.860 0.813
Estonia 0.779 0.367 0.723
Hungary 0.862 0.751 0.819
Iceland 0.899 0.564 0.756
Ireland 0.864 0.613 0.802
Italy 0.882 0.778 0.741
Korea 0.947 0.909 0.723
Lithuania 0.939 0.837 0.782
Malaysia 0.927 0.863 0.772
Malta 0.883 0.741 0.732
Mexico 0.955 0.949 0.784
Norway 0.819 0.600 0.702
Poland 0.898 0.616 0.775
Portugal 0.910 0.718 0.729
Slovak Republic 0.818 0.260 0.717
Slovenia 0.873 0.769 0.711
Spain 0.952 0.854 0.748
Turkey 0.942 0.966 0.811
International sample 0.925 0.854 0.763

Netherlands 0.761 0.688 0.800

Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

170
chapter 11 Construction and Validation of Scales and Indices

Table 11.43
Model fit and latent correlations for factors influencing school climate, by country
Latent correlations
Model fit between

CFI TLI RMSEA SRMR SCDELINQ/ SCTMORAL

Australia 0.951 0.932 0.074 0.052 0.549
Austria 0.991 0.988 0.033 0.039 0.572
Belgium (Fl.) 0.884 0.840 0.120 0.060 0.781
Brazil 0.966 0.952 0.072 0.043 0.672
Bulgaria 0.828 0.762 0.129 0.079 0.547
Denmark 0.971 0.960 0.090 0.035 0.885
Estonia 0.985 0.980 0.030 0.055 0.194
Hungary 0.793 0.713 0.191 0.100 0.545
Iceland 0.933 0.907 0.103 0.077 0.351
Ireland 0.930 0.902 0.085 0.044 0.432
Italy 0.953 0.935 0.093 0.044 0.769
Korea 0.929 0.902 0.147 0.041 0.923
Lithuania 0.975 0.965 0.071 0.033 0.839
Malaysia 0.919 0.888 0.141 0.053 0.730
Malta 0.950 0.931 0.088 0.058 0.798
Mexico 0.959 0.943 0.100 0.031 0.718
Norway 0.868 0.818 0.125 0.062 0.525
Poland 0.922 0.891 0.106 0.056 0.651
Portugal 0.913 0.879 0.124 0.048 0.565
Slovak Republic 0.887 0.844 0.098 0.064 0.276
Slovenia 0.956 0.940 0.080 0.044 0.688
Spain 0.931 0.904 0.144 0.042 0.762
Turkey 0.854 0.798 0.196 0.057 0.894
International sample 0.950 0.931 0.093 0.038 0.725

Netherlands Model fit could not be computed due to the small sample size
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
p < 0.05
The Netherlands was excluded because it did not meet international sampling standards. p < 0.01
Source: OECD, TALIS Database.

Table 11.44
Multiple group CFA examining different levels of cross-cultural invariance
of factors influencing school climate
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.924 0.894 0.116 0.055 - -
Metric invariance 0.899 0.889 0.120 0.106 -0.025 0.004
Scalar invariance 0.747 0.768 0.172 0.184 -0.152 0.052
Uniqueness invariance 0.691 0.766 0.173 0.199 -0.056 0.001
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.45
Factor loadings used for the computation of factor scores for the indicator
School climate: student delinquency
BCG30F BCG30G BCG30H BCG30I BCG30J BCG30K
International sample 1.078 1.000 1.018 1.105 1.125 0.907
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
171

Table 11.46
Item intercepts used for the computation of factor scores for the indicator
School climate: student delinquency, by country
BCG30F BCG30G BCG30H BCG30I BCG30J BCG30K
Australia 1.834 2.258 1.702 1.845 1.645
Austria 2.139 2.275 1.682 1.565 1.404
Belgium (Fl.) 1.899 2.310 1.543 1.883 1.808
Brazil 2.152 2.255 1.890 1.971 1.675
Bulgaria 2.555 2.540 2.000 1.849 1.501
Denmark 1.905 2.283 1.837 1.860 1.532
Estonia 2.079 2.675 1.663 2.369 1.887
Hungary 2.404 2.279 2.048 1.577 1.255
Iceland 2.034 2.205 1.964 2.015 1.573
Ireland 1.901 2.413 1.663 2.028 1.859
Italy 2.122 2.441 2.095 1.874 1.479
Korea 1.912 1.830 2.125 1.903 1.744 1.413
Lithuania 1.896 2.371 1.854 1.818 1.728
Malaysia 1.987 1.809 1.552 1.329 1.421
Malta 2.086 2.661 1.747 1.837 1.398
Mexico 1.833 1.812 1.806 1.575 1.831
Norway 2.111 2.302 1.784 1.959 1.395
Poland 2.332 2.289 2.217 1.595 1.488
Portugal 1.763 2.078 1.936 1.670 1.472
Slovak Republic 2.100 2.088 1.685 1.506 1.401
Slovenia 2.177 2.420 1.659 1.743 1.377
Spain 1.963 2.209 1.849 1.888 1.740
Turkey 2.050 1.897 1.951 1.588 1.657

Netherlands 1.747 2.007 1.224 1.521 1.706

Notes: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.47
Unique variances used for the computation of factor scores for the indicator
School climate: student delinquency, by country
BCG30F BCG30G BCG30H BCG30I BCG30J BCG30K
Australia 0.163 0.179 0.197 0.122 0.213 0.183
Austria 0.276 0.215 0.315 0.123 0.240 0.232
Belgium (Fl.) 0.125 0.116 0.235 0.192 0.187 0.205
Brazil 0.295 0.228 0.248 0.159 0.187 0.297
Bulgaria 0.382 0.244 0.257 0.163 0.103 0.155
Denmark 0.205 0.198 0.192 0.135 0.156 0.328
Estonia 0.306 0.204 0.156 0.163 0.207 0.299
Hungary 0.411 0.311 0.238 0.234 0.300 0.415
Iceland 0.213 0.172 0.199 0.138 0.141 0.168
Ireland 0.133 0.125 0.208 0.181 0.266 0.347
Italy 0.255 0.312 0.190 0.198 0.159 0.223
Korea 0.304 0.224 0.210 0.102 0.186 0.380
Lithuania 0.343 0.214 0.349 0.202 0.211 0.364
Malaysia 0.231 0.146 0.107 0.131 0.247 0.380
Malta 0.340 0.206 0.201 0.150 0.168 0.220
Mexico 0.414 0.290 0.231 0.138 0.385 0.484
Norway 0.243 0.182 0.229 0.125 0.174 0.132
Poland 0.267 0.201 0.174 0.159 0.289 0.237
Portugal 0.179 0.227 0.197 0.178 0.238 0.304
Slovak Republic 0.363 0.232 0.246 0.141 0.175 0.244
Slovenia 0.228 0.217 0.236 0.245 0.232 0.196
Spain 0.222 0.210 0.166 0.170 0.212 0.313
Turkey 0.246 0.094 0.340 0.366 0.237 0.487

Netherlands 0.095 0.208 0.217 0.192 0.145 0.099

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

172
chapter 11 Construction and Validation of Scales and Indices

Table 11.48
Factor loadings used for the computation of factor scores for the indicator
School climate: teachers’ working morale
BCG30L BCG30M BCG30N
International sample 1.000 1.058 0.958
Source: OECD, TALIS Database.

Table 11.49
Item intercepts used for the computation of factor scores for the indicator
School climate: teachers’ working morale, by country
BCG30L BCG30M BCG30N
Australia 1.723 2.234
Austria 1.860 2.020
Belgium (Fl.) 2.331 2.389
Brazil 2.069 2.092
Bulgaria 2.265 2.260
Denmark 2.555 2.519
Estonia 2.238 2.366
Hungary 1.575 1.861
Iceland 1.627 2.079
Ireland 1.686 1.943
Italy 1.806 2.556
Korea 2.044 2.075 2.530
Lithuania 2.318 2.725
Malaysia 1.999 2.345
Malta 1.782 1.851
Mexico 2.052 2.163
Norway 1.590 1.767
Poland 1.155 0.864
Portugal 1.930 1.925
Slovak Republic 1.901 2.144
Slovenia 1.435 1.729
Spain 2.043 2.249
Turkey 2.030 2.311

Netherlands 1.594 1.710

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
173

Table 11.50
Unique variances used for the computation of factor scores for the indicator
School climate: teachers’ working morale, by country
BCG30L BCG30M BCG30N
Australia 0.150 0.267 0.366
Austria 0.219 0.348 0.272
Belgium (Fl.) 0.164 0.174 0.202
Brazil 0.240 0.293 0.529
Bulgaria 0.172 0.058 0.233
Denmark 0.161 0.244 0.122
Estonia 0.255 0.170 0.387
Hungary 0.218 0.404 0.295
Iceland 0.206 0.357 0.275
Ireland 0.183 0.349 0.344
Italy 0.185 0.397 0.481
Korea 0.118 0.171 0.283
Lithuania 0.170 0.283 0.597
Malaysia 0.162 0.257 0.230
Malta 0.176 0.303 0.438
Mexico 0.169 0.082 0.253
Norway 0.267 0.398 0.186
Poland 0.276 0.313 0.192
Portugal 0.165 0.106 0.335
Slovak Republic 0.195 0.317 0.294
Slovenia 0.233 0.316 0.253
Spain 0.238 0.321 0.433
Turkey 0.109 0.080 0.365

Netherlands 0.173 0.175 0.140

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.51
Factor score determinacy for the indicators School climate: student delinquency
and School climate: teachers’ working morale, by country
SCDELINQ SCTMORAL
Australia 0.932 0.875
Austria 0.938 0.860
Belgium (Fl.) 0.943 0.873
Brazil 0.956 0.906
Bulgaria 0.921 0.851
Denmark 0.969 0.953
Estonia 0.887 0.681
Hungary 0.940 0.871
Iceland 0.951 0.773
Ireland 0.933 0.832
Italy 0.960 0.924
Korea 0.981 0.971
Lithuania 0.971 0.950
Malaysia 0.970 0.936
Malta 0.953 0.906
Mexico 0.977 0.977
Norway 0.926 0.817
Poland 0.943 0.821
Portugal 0.959 0.900
Slovak Republic 0.905 0.649
Slovenia 0.944 0.889
Spain 0.977 0.940
Turkey 0.982 0.985

Netherlands 0.890 0.885

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

174
chapter 11 Construction and Validation of Scales and Indices

Table 11.52
Model fit of Teacher-student relations (TQ), by country
CFI TLI RMSEA SRMR
Australia 0.991 0.974 0.064 0.019
Austria 0.990 0.969 0.072 0.015
Belgium (Fl.) 0.999 0.997 0.024 0.010
Brazil 0.999 0.997 0.018 0.009
Bulgaria 1.000 1.001 0.000 0.009
Denmark 0.988 0.964 0.095 0.018
Estonia 1.000 1.005 0.000 0.006
Hungary 0.995 0.984 0.044 0.012
Iceland 0.998 0.995 0.029 0.010
Ireland 0.998 0.994 0.031 0.009
Italy 0.998 0.995 0.022 0.009
Korea 0.999 0.997 0.017 0.011
Lithuania 0.996 0.989 0.036 0.011
Malaysia 1.000 1.003 0.000 0.007
Malta 0.990 0.969 0.079 0.018
Mexico 0.996 0.989 0.041 0.011
Norway 0.982 0.946 0.079 0.019
Poland 0.970 0.909 0.097 0.025
Portugal 0.995 0.986 0.040 0.015
Slovak Republic 0.984 0.951 0.063 0.019
Slovenia 1.000 1.008 0.000 0.005
Spain 0.978 0.935 0.089 0.021
Turkey 0.980 0.939 0.107 0.023
International sample 0.998 0.995 0.034 0.012

Netherlands 0.955 0.866 0.144 0.033

Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.53
Multiple group CFA examining different levels of cross-cultural invariance
of Teacher-student relations
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.992 0.977 0.054 0.015 - -
Metric invariance 0.985 0.982 0.049 0.060 -0.007 -0.005
Scalar invariance 0.840 0.876 0.126 0.113 -0.145 0.077
Uniqueness invariance 0.774 0.883 0.123 0.228 -0.066 -0.003
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.54
Factor loadings used for the computation of factor scores for the indicator
Teacher-student relations
BTG31G BTG31H BTG31I BTG31J
International sample 1.000 1.413 1.395 1.098
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
175

Table 11.55
Item intercepts used for the computation of factor scores for the indicator
Teacher-student relations, by country
BTG31G BTG31H BTG31I BTG31J
Australia 2.395 2.188 2.237
Austria 2.158 1.981 2.111
Belgium (Fl.) 2.305 2.156 2.452
Brazil 2.370 2.154 1.964
Bulgaria 2.329 2.279 2.457
Denmark 2.261 2.038 1.901
Estonia 2.298 2.139 2.384
Hungary 2.266 2.227 2.301
Iceland 2.449 2.244 2.111
Ireland 2.287 2.073 2.194
Italy 2.380 2.294 2.228
Korea 2.183 2.229 2.276 2.141
Lithuania 2.448 2.393 2.420
Malaysia 2.606 2.385 2.437
Malta 2.407 2.181 2.241
Mexico 2.463 2.184 2.141
Norway 2.174 1.922 1.721
Poland 2.265 2.271 2.421
Portugal 2.429 2.223 2.256
Slovak Republic 2.421 2.260 2.448
Slovenia 2.377 2.207 2.545
Spain 2.327 2.151 2.208
Turkey 2.242 2.161 2.199

Netherlands 2.203 2.064 2.231

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.56
Unique variances used for the computation of factor scores for the indicator
Teacher-student relations, by country
BTG31G BTG31H BTG31I BTG31J
Australia 0.139 0.103 0.095 0.219
Austria 0.200 0.134 0.140 0.287
Belgium (Fl.) 0.118 0.074 0.084 0.198
Brazil 0.144 0.107 0.124 0.341
Bulgaria 0.129 0.081 0.100 0.156
Denmark 0.116 0.073 0.112 0.279
Estonia 0.144 0.083 0.112 0.188
Hungary 0.103 0.072 0.088 0.166
Iceland 0.149 0.098 0.112 0.322
Ireland 0.139 0.070 0.119 0.239
Italy 0.108 0.096 0.081 0.236
Korea 0.122 0.126 0.079 0.154
Lithuania 0.119 0.092 0.086 0.144
Malaysia 0.189 0.113 0.122 0.167
Malta 0.155 0.123 0.134 0.287
Mexico 0.210 0.154 0.157 0.296
Norway 0.197 0.063 0.180 0.440
Poland 0.129 0.083 0.075 0.203
Portugal 0.128 0.114 0.109 0.224
Slovak Republic 0.139 0.087 0.116 0.188
Slovenia 0.110 0.093 0.102 0.195
Spain 0.123 0.086 0.130 0.222
Turkey 0.199 0.093 0.111 0.206

Netherlands 0.126 0.067 0.090 0.163

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

176
chapter 11 Construction and Validation of Scales and Indices

Table 11.57
Factor score determinacy for the indicators Teacher-student relations, by country
TSRELAT
Australia 0.919
Austria 0.906
Belgium (Fl.) 0.911
Brazil 0.901
Bulgaria 0.911
Denmark 0.925
Estonia 0.876
Hungary 0.922
Iceland 0.904
Ireland 0.927
Italy 0.893
Korea 0.873
Lithuania 0.899
Malaysia 0.883
Malta 0.884
Mexico 0.901
Norway 0.894
Poland 0.896
Portugal 0.881
Slovak Republic 0.869
Slovenia 0.860
Spain 0.895
Turkey 0.934

Netherlands 0.909
Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.58
Item wording of Classroom disciplinary climate items and dimensions
How strongly do you agree or disagree with the following statements about this <target class>?
Classroom disciplinary climate BTG43A When the lesson begins, I have to wait quite a long time for students to quieten down.
BTG43B Students in this class take care to create a pleasant learning atmosphere.
BTG43C I lose quite a lot of time because of students interrupting the lesson.
BTG43D There is much noise in this classroom.

Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
177

Table 11.59
Reliabilities for Classroom disciplinary climate, by country

CCLIMATE
Australia 0.827
Austria 0.854
Belgium (Fl.) 0.867
Brazil 0.837
Bulgaria 0.815
Denmark 0.852
Estonia 0.879
Hungary 0.857
Iceland 0.803
Ireland 0.855
Italy 0.850
Korea 0.760
Lithuania 0.809
Malaysia 0.841
Malta 0.830
Mexico 0.696
Norway 0.835
Poland 0.840
Portugal 0.871
Slovak Republic 0.858
Slovenia 0.875
Spain 0.877
Turkey 0.839
International sample 0.837

Netherlands 0.779
Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.60
Model fit Classroom disciplinary climate, by country
CFI TLI RMSEA SRMR
Australia 0.999 0.996 0.028 0.010
Austria 1.000 1.003 0.000 0.004
Belgium (Fl.) 0.999 0.996 0.031 0.006
Brazil 1.000 1.007 0.000 0.003
Bulgaria 1.000 0.999 0.012 0.007
Denmark 0.994 0.982 0.056 0.012
Estonia 1.000 0.999 0.016 0.005
Hungary 0.998 0.994 0.035 0.008
Iceland 1.000 1.006 0.000 0.003
Ireland 0.999 0.996 0.026 0.010
Italy 1.000 1.000 0.000 0.006
Korea 1.000 1.005 0.000 0.005
Lithuania 0.989 0.968 0.063 0.016
Malaysia 0.998 0.995 0.025 0.009
Malta 1.000 1.001 0.000 0.006
Mexico 1.000 1.000 0.000 0.006
Norway 1.000 1.000 0.000 0.006
Poland 0.996 0.989 0.041 0.011
Portugal 0.995 0.984 0.055 0.012
Slovak Republic 1.000 1.005 0.000 0.002
Slovenia 1.000 1.003 0.000 0.003
Spain 0.999 0.996 0.027 0.008
Turkey 0.995 0.985 0.042 0.015
International sample 1.000 1.000 0.005 0.002

Netherlands 1.000 1.011 0.000 0.009

TALIS 2008 Technical Report © OECD 2010

178
chapter 11 Construction and Validation of Scales and Indices

Table 11.61
Multiple group CFA examining different levels of cross-cultural invariance
of Classroom disciplinary climate
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.999 0.996 0.025 0.008 - -
Metric invariance 0.981 0.976 0.064 0.087 -0.018 0.039
Scalar invariance 0.883 0.909 0.125 0.124 -0.098 0.061
Uniqueness invariance 0.831 0.912 0.123 0.162 -0.052 -0.002
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.62
Factor loadings used for the computation of factor scores for the indicator
Classroom disciplinary climate
BTG43A BTG43B BTG43C BTG43D
International sample 0.890 0.606 1.000 0.919
Source: OECD, TALIS Database.

Table 11.63
Item intercepts used for the computation of factor scores for the indicator
Classroom disciplinary climate, by country
BTG43A BTG43B BTG43C BTG43D
Australia 2.971 2.764 2.695
Austria 3.015 2.838 3.058
Belgium (Fl.) 2.967 2.683 3.012
Brazil 2.774 2.842 2.802
Bulgaria 3.088 2.887 3.040
Denmark 2.993 2.968 2.983
Estonia 2.864 2.443 2.931
Hungary 2.859 2.653 2.939
Iceland 2.262 2.858 2.948
Ireland 3.062 2.728 2.965
Italy 3.030 2.938 3.213
Korea 2.794 2.939 2.890 2.954
Lithuania 3.024 2.858 2.941
Malaysia 2.987 2.931 3.060
Malta 2.967 2.708 3.163
Mexico 3.058 2.959 2.935
Norway 2.680 2.839 3.145
Poland 3.125 2.803 3.048
Portugal 2.905 2.968 3.159
Slovak Republic 3.031 2.883 2.969
Slovenia 2.911 2.727 2.953
Spain 2.940 2.845 2.964
Turkey 3.057 2.761 2.938

Netherlands 2.506 3.108 3.072

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
179

Table 11.64
Unique variances used for the computation of factor scores for the indicator
Classroom disciplinary climate, by country
BTG43A BTG43B BTG43C BTG43D
Australia 0.232 0.309 0.160 0.377
Austria 0.264 0.468 0.188 0.254
Belgium (Fl.) 0.203 0.348 0.138 0.123
Brazil 0.290 0.285 0.125 0.162
Bulgaria 0.256 0.321 0.185 0.205
Denmark 0.227 0.318 0.212 0.203
Estonia 0.213 0.360 0.090 0.154
Hungary 0.328 0.309 0.149 0.182
Iceland 0.448 0.350 0.179 0.289
Ireland 0.234 0.394 0.166 0.223
Italy 0.202 0.293 0.160 0.188
Korea 0.281 0.321 0.126 0.150
Lithuania 0.208 0.254 0.146 0.258
Malaysia 0.254 0.266 0.174 0.163
Malta 0.228 0.355 0.195 0.215
Mexico 0.370 0.334 0.187 0.305
Norway 0.522 0.502 0.214 0.270
Poland 0.220 0.242 0.149 0.150
Portugal 0.279 0.257 0.162 0.163
Slovak Republic 0.208 0.246 0.113 0.187
Slovenia 0.172 0.236 0.083 0.129
Spain 0.285 0.333 0.151 0.219
Turkey 0.275 0.394 0.150 0.161

Netherlands 0.348 0.246 0.165 0.242

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.65
Factor score determinacy for the indicators Classroom disciplinary climate, by country
CCLIMATE
Australia 0.930
Austria 0.937
Belgium (Fl.) 0.952
Brazil 0.935
Bulgaria 0.924
Denmark 0.934
Estonia 0.960
Hungary 0.936
Iceland 0.924
Ireland 0.942
Italy 0.932
Korea 0.921
Lithuania 0.919
Malaysia 0.929
Malta 0.932
Mexico 0.873
Norway 0.940
Poland 0.932
Portugal 0.939
Slovak Republic 0.939
Slovenia 0.959
Spain 0.948
Turkey 0.938

Netherlands 0.901
Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

180
chapter 11 Construction and Validation of Scales and Indices

Table 11.66
Item wording of the scale Self-efficacy
How strongly do you agree or disagree with the following statements about yourself as a teacher in this school?
BTG31B I feel that I am making a significant educational difference in the lives of my students.
BTG31C If I try really hard, I can make progress with even the most difficult and unmotivated students.
BTG31D I am successful with the students in my class.
BTG31E I usually know how to get through to students.

Source: OECD, TALIS Database.

Table 11.67
Reliabilities for Self-efficacy, by country

SELFEF
Australia 0.817
Austria 0.752
Belgium (Fl.) 0.787
Brazil 0.771
Bulgaria 0.702
Denmark 0.805
Estonia 0.648
Hungary 0.670
Iceland 0.762
Ireland 0.815
Italy 0.771
Korea 0.744
Lithuania 0.710
Malaysia 0.783
Malta 0.745
Mexico 0.746
Norway 0.717
Poland 0.706
Portugal 0.674
Slovak Republic 0.713
Slovenia 0.613
Spain 0.738
Turkey 0.771
International sample 0.763

Netherlands 0.715
Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
181

Table 11.68
Model fit for Self-efficacy, by country
CFI TLI RMSEA SRMR
Australia 0.949 0.847 0.154 0.038
Austria 0.993 0.978 0.050 0.015
Belgium (Fl.) 0.996 0.989 0.036 0.011
Brazil 0.947 0.842 0.140 0.035
Bulgaria 1.000 1.008 0.000 0.005
Denmark 0.996 0.987 0.043 0.012
Estonia 0.981 0.942 0.060 0.019
Hungary 0.958 0.874 0.097 0.028
Iceland 0.944 0.832 0.135 0.039
Ireland 0.987 0.962 0.075 0.018
Italy 0.970 0.910 0.095 0.031
Korea 0.911 0.734 0.164 0.039
Lithuania 0.975 0.926 0.074 0.024
Malaysia 0.959 0.876 0.119 0.028
Malta 0.976 0.927 0.109 0.032
Mexico 0.992 0.975 0.055 0.015
Norway 0.965 0.896 0.105 0.028
Poland 0.995 0.986 0.031 0.013
Portugal 0.998 0.994 0.024 0.011
Slovak Republic 0.995 0.984 0.029 0.015
Slovenia 0.962 0.887 0.078 0.024
Spain 0.998 0.994 0.024 0.011
Turkey 0.995 0.985 0.046 0.013
International sample 0.978 0.934 0.082 0.022

Netherlands 0.959 0.876 0.093 0.034

Table 11.69
Multiple group CFA examining different levels of cross-cultural invariance of Self-efficacy
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.977 0.931 0.086 0.024 - -
Metric invariance 0.966 0.958 0.067 0.061 -0.011 -0.019
Scalar invariance 0.773 0.824 0.136 0.141 -0.193 0.069
Uniqueness invariance 0.712 0.851 0.126 0.208 -0.061 -0.01
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.70
Factor loadings used for the computation of factor scores for the indicator Self-efficacy
BTG31B BTG31C BTG31D BTG31E
International sample 0.882 0.897 1.000 0.933
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

182
chapter 11 Construction and Validation of Scales and Indices

Table 11.71
Item intercepts used for the computation of factor scores for the indicator Self-efficacy, by country
BTG31B BTG31C BTG31D BTG31E
Australia 2.228 2.097 2.218
Austria 2.165 2.056 2.313
Belgium (Fl.) 2.332 2.042 2.181
Brazil 2.244 2.238 2.291
Bulgaria 2.291 2.008 2.256
Denmark 2.311 1.935 2.291
Estonia 2.199 2.092 2.308
Hungary 2.151 1.999 2.493
Iceland 2.001 2.138 2.219
Ireland 2.245 2.146 2.170
Italy 2.173 2.112 2.216
Korea 2.279 2.313 2.237 2.309
Lithuania 2.164 2.036 2.205
Malaysia 2.418 2.491 2.338
Malta 2.270 2.183 2.296
Mexico 2.406 2.449 2.365
Norway 2.597 2.246 2.377
Poland 2.276 2.197 2.158
Portugal 2.200 1.920 2.385
Slovak Republic 2.279 2.163 2.247
Slovenia 2.033 2.225 2.402
Spain 2.238 2.007 2.287
Turkey 2.226 2.212 2.319

Netherlands 2.340 2.052 2.267

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.72
Unique variances used for the computation of factor scores for the indicator Self-efficacy, by country
BTG31B BTG31C BTG31D BTG31E
Australia 0.166 0.213 0.062 0.075
Austria 0.184 0.322 0.089 0.132
Belgium (Fl.) 0.142 0.222 0.062 0.095
Brazil 0.142 0.291 0.082 0.105
Bulgaria 0.156 0.336 0.068 0.090
Denmark 0.142 0.289 0.074 0.076
Estonia 0.230 0.291 0.077 0.094
Hungary 0.152 0.223 0.099 0.193
Iceland 0.230 0.229 0.088 0.100
Ireland 0.150 0.242 0.055 0.084
Italy 0.128 0.243 0.084 0.088
Korea 0.152 0.207 0.107 0.078
Lithuania 0.114 0.257 0.075 0.118
Malaysia 0.148 0.207 0.085 0.128
Malta 0.161 0.272 0.086 0.106
Mexico 0.143 0.204 0.101 0.201
Norway 0.141 0.280 0.146 0.161
Poland 0.123 0.238 0.072 0.093
Portugal 0.180 0.324 0.072 0.126
Slovak Republic 0.115 0.197 0.098 0.109
Slovenia 0.194 0.196 0.064 0.162
Spain 0.158 0.299 0.079 0.114
Turkey 0.178 0.301 0.113 0.118

Netherlands 0.154 0.295 0.051 0.072

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
183

Table 11.73
Factor score determinacy for the indicators Self-efficacy, by country
SELFEF
Australia 0.928
Austria 0.886
Belgium (Fl.) 0.907
Brazil 0.906
Bulgaria 0.910
Denmark 0.924
Estonia 0.819
Hungary 0.835
Iceland 0.895
Ireland 0.928
Italy 0.897
Korea 0.881
Lithuania 0.876
Malaysia 0.896
Malta 0.888
Mexico 0.880
Norway 0.861
Poland 0.861
Portugal 0.869
Slovak Republic 0.833
Slovenia 0.830
Spain 0.879
Turkey 0.898

Netherlands 0.886
Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.74
Item wording of teachers’ and principals’ Beliefs items and dimensions
We would like to ask about your personal beliefs on teaching and learning.
Please indicate how much you disagree or agree with each of the following statements.
Direct transmission beliefs BTG29A Effective/good teachers demonstrate the correct way to solve a problem.
about instruction Instruction should be built around problems with clear, correct answers, and around ideas that most students can grasp
BTG29G
quickly.
How much students learn depends on how much background knowledge they have – that is why teaching facts is so
BTG29H
necessary.
BTG29K A quiet classroom is generally needed for effective learning.
Constructivist beliefs BTG29D/
My role as a teacher is to facilitate students’ own inquiry.
about instruction BCG32D
BTG29F/
Students learn best by finding solutions to problems on their own
BCG32F
BTG29I/ Students should be allowed to think of solutions to practical problems themselves before the teacher shows them how they
BCG32I are solved.
BTG29L/
Thinking and reasoning processes are more important than specific curriculum content.
BCG32L
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

184
chapter 11 Construction and Validation of Scales and Indices

Table 11.75
Single items measuring teachers’ and principals’ Beliefs about instruction
We would like to ask about your personal beliefs on teaching and learning.
Please indicate how much you disagree or agree with each of the following statements.
Single items measuring an BTG29B/ When referring to a “poor performance”, I mean a performance that lies below the previous achievement level of
internal frame of reference BCG32B the student.
BTG29J/ When referring to a “good performance”, I mean a performance that lies above the previous achievement level of the
BCG32J student.
Single items BTG29C/
It is better when the teacher – not the student – decides what activities are to be done.
BCG32C
BTG29E/ Teachers know a lot more than students; they shouldn’t let students develop answers that may be incorrect when they
BCG32E can just explain the answers directly.
Single items measuring direct BCG32A Effective/good teachers demonstrate the correct way to solve a problem.
transmission beliefs about Instruction should be built around problems with clear, correct answers, and around ideas that most students can
instruction BCG32G
grasp quickly.
How much students learn depends on how much background knowledge they have – that is why teaching facts is so
BCG32H
necessary.
BCG32K A quiet classroom is generally needed for effective learning.
Source: OECD, TALIS Database.

Table 11.76
Reliabilities for indices of teachers’ Beliefs about instruction, by country
TBTRAD TBCONS
Australia 0.493 0.599
Austria 0.477 0.648
Belgium (Fl.) 0.422 0.629
Brazil 0.526 0.625
Bulgaria 0.533 0.586
Denmark 0.415 0.517
Estonia 0.499 0.580
Hungary 0.481 0.648
Iceland 0.535 0.660
Ireland 0.525 0.611
Italy 0.459 0.557
Korea 0.440 0.667
Lithuania 0.589 0.544
Malaysia 0.488 0.618
Malta 0.440 0.583
Mexico 0.496 0.534
Norway 0.504 0.441
Poland 0.451 0.625
Portugal 0.497 0.603
Slovak Republic 0.515 0.525
Slovenia 0.418 0.501
Spain 0.432 0.571
Turkey 0.515 0.715
International sample 0.467 0.607

Netherlands 0.488 0.579

Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
185

Table 11.77
Model fit and latent correlations for teachers’ Direct transmission beliefs about instruction
and Constructivist beliefs about instruction, by country

Model fit Latent correlations

CFI TLI RMSEA SRMR TBTRAD/TBCONS

Australia 0.897 0.848 0.057 0.047 0.008
Austria 0.977 0.966 0.027 0.029 -0.129
Belgium (Fl.) 0.877 0.819 0.063 0.049 -0.083
Brazil 0.982 0.973 0.026 0.025 0.515
Bulgaria 0.921 0.883 0.053 0.042 0.487
Denmark 0.818 0.732 0.068 0.055 0.073
Estonia 0.898 0.850 0.051 0.038 0.074
Hungary 0.886 0.833 0.067 0.054 0.083
Iceland 0.919 0.881 0.059 0.051 -0.166
Ireland 0.945 0.920 0.040 0.037 0.060
Italy 0.878 0.820 0.057 0.052 0.219
Korea 0.876 0.817 0.072 0.051 0.517
Lithuania 0.947 0.922 0.042 0.035 0.295
Malaysia 0.952 0.929 0.051 0.035 0.990
Malta 0.852 0.782 0.061 0.050 0.231
Mexico 0.870 0.809 0.065 0.047 0.578
Norway 0.979 0.969 0.021 0.024 0.142
Poland 0.870 0.809 0.065 0.059 0.109
Portugal 0.977 0.966 0.026 0.029 0.233
Slovak Republic 0.832 0.752 0.071 0.050 0.259
Slovenia 0.847 0.775 0.059 0.048 0.209
Spain 0.904 0.858 0.049 0.041 0.204
Turkey 0.962 0.944 0.046 0.033 0.643
International sample 0.940 0.912 0.040 0.031 0.262

Netherlands 0.880 0.823 0.065 0.052 -0.397

Notes: Model estimates based on international calibration sample (1 000 teachers per country).
p < 0.05
The Netherlands was excluded because it did not meet international sampling standards. p < 0.01
Source: OECD, TALIS Database.

Table 11.78
Multiple group CFA examining different levels of cross-cultural invariance of teachers’
Direct transmission beliefs about instruction
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.978 0.935 0.047 0.018 - -
Metric invariance 0.939 0.925 0.050 0.041 -0.039 0.003
Scalar invariance1 0.000 -0.179 0.199 0.205 -0.939 0.149
Uniqueness invariance 0.000 -0.005 0.183 0.222 0.000 -0.016
1. Latent correlation in Malaysia > 1.0 for this model. Therefore the latent variable covariance matrix for Malaysia is not positive definite.
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

186
chapter 11 Construction and Validation of Scales and Indices

Table 11.79
Multiple group CFA examining different levels of cross-cultural invariance of teachers’
Constructivist beliefs about instruction
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.995 0.985 0.028 0.014 - -
Metric invariance 0.976 0.971 0.039 0.041 -0.019 0.011
Scalar invariance1 0.431 0.559 0.152 0.147 -0.545 0.113
Uniqueness invariance 0.283 0.622 0.141 0.200 -0.148 -0.011
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands were excluded because they did not meet international sampling standards.
1. Latent correlation in Malaysia > 1.0 for this model. Therefore the latent variable covariance matrix for Malaysia is not positive definite.
Source: OECD, TALIS Database.

Table 11.80
Factor loadings used for the computation of factor scores for the indicators
Direct transmission beliefs about instruction and Constructivist beliefs about instruction
Direct transmission beliefs Constructivist beliefs

BTG29A BTG29G BTG29H BTG29K BTG29D BTG29F BTG29I BTG29L

International sample 1.000 1.985 1.791 1.249 1.001 1.462 1.427 1.000
Source: OECD, TALIS Database.

Table 11.81
Item intercepts used for the computation of factor scores for the indicators Direct transmission beliefs
about instruction and Constructivist beliefs about instruction, by country
Direct transmission beliefs Constructivist beliefs

BTG29A BTG29G BTG29H BTG29K BTG29D BTG29F BTG29I BTG29L

Australia 1.582 1.469 1.325 2.163 2.029 2.196
Austria 1.755 1.690 2.367 2.317 2.187 2.069
Belgium (Fl.) 1.852 1.646 2.028 2.287 2.376 2.346
Brazil 2.636 2.583 2.565 2.274 2.358 2.379
Bulgaria 2.152 1.912 2.195 1.943 1.847 1.980
Denmark 1.613 1.702 2.193 2.106 2.296 2.320
Estonia 1.984 2.065 2.314 2.083 1.732 2.071
Hungary 1.398 1.753 1.656 2.210 2.058 2.117
Iceland 1.409 1.373 1.513 2.506 2.111 2.117
Ireland 1.959 1.695 1.625 2.159 2.240 2.406
Italy 2.301 2.629 2.369 1.979 1.527 1.636
Korea 2.174 2.141 2.271 2.054 2.099 2.394 2.297 2.061
Lithuania 2.504 2.331 2.174 2.233 1.577 1.855
Malaysia 1.703 1.563 1.404 2.189 1.964 2.348
Malta 1.839 1.568 1.594 2.009 1.900 2.150
Mexico 2.269 2.269 1.443 2.375 2.438 2.590
Norway 1.623 1.986 2.071 2.575 2.075 2.626
Poland 1.778 1.852 1.906 2.073 2.037 2.083
Portugal 2.117 2.015 2.474 2.262 1.939 2.342
Slovak Republic 2.373 1.898 1.866 2.241 1.831 2.218
Slovenia 1.893 1.631 1.900 1.886 2.349 2.170
Spain 1.656 1.818 1.730 2.172 2.037 2.106
Turkey 2.083 1.946 2.103 1.961 2.154 2.180

Netherlands 1.637 1.777 1.945 2.543 2.630 2.624

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
187

Table 11.82
Unique variances used for the computation of factor scores for the indicators
Direct transmission beliefs about instruction and Constructivist beliefs about instruction, by country
Direct transmission beliefs Constructivist beliefs

BTG29A BTG29G BTG29H BTG29K BTG29D BTG29F BTG29I BTG29L

Australia 0.349 0.272 0.230 0.243 0.249 0.276 0.173 0.308
Austria 0.455 0.327 0.273 0.328 0.277 0.222 0.239 0.301
Belgium (Fl.) 0.250 0.295 0.232 0.339 0.192 0.200 0.151 0.250
Brazil 0.388 0.326 0.230 0.388 0.265 0.231 0.191 0.387;
Bulgaria 0.296 0.202 0.230 0.405 0.258 0.317 0.149 0.327
Denmark 0.290 0.205 0.271 0.381 0.295 0.245 0.155 0.308
Estonia 0.409 0.202 0.218 0.349 0.220 0.361 0.194 0.293
Hungary 0.335 0.261 0.238 0.325 0.241 0.218 0.143 0.266
Iceland 0.384 0.208 0.192 0.313 0.200 0.211 0.169 0.270
Ireland 0.312 0.242 0.256 0.370 0.290 0.293 0.143 0.321
Italy 0.289 0.261 0.235 0.377 0.199 0.288 0.211 0.267
Korea 0.254 0.326 0.234 0.306 0.177 0.193 0.126 0.228
Lithuania 0.352 0.203 0.251 0.385 0.257 0.318 0.188 0.291
Malaysia 0.224 0.145 0.253 0.490 0.201 0.329 0.145 0.240
Malta 0.282 0.278 0.318 0.406 0.265 0.335 0.173 0.243
Mexico 0.451 0.377 0.298 0.317 0.452 0.346 0.226 0.417
Norway 0.446 0.242 0.312 0.427 0.342 0.327 0.264 0.353
Poland 0.282 0.221 0.189 0.355 0.190 0.215 0.163 0.243
Portugal 0.300 0.206 0.234 0.351 0.208 0.337 0.159 0.279
Slovak Republic 0.219 0.246 0.239 0.353 0.209 0.325 0.172 0.241
Slovenia 0.270 0.310 0.250 0.374 0.302 0.218 0.160 0.222
Spain 0.318 0.303 0.281 0.465 0.259 0.288 0.230 0.320
Turkey 0.408 0.265 0.258 0.425 0.307 0.167 0.132 0.249

Netherlands 0.260 0.175 0.256 0.357 0.184 0.205 0.139 0.270

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

188
chapter 11 Construction and Validation of Scales and Indices

Table 11.83
Factor score determinacy for the indicators Direct transmission beliefs about instruction
and Constructivist beliefs about instruction, by country
TBTRAD TBCONS
Australia 0.747 0.794
Austria 0.723 0.814
Belgium (Fl.) 0.679 0.821
Brazil 0.780 0.815
Bulgaria 0.798 0.818
Denmark 0.684 0.765
Estonia 0.762 0.765
Hungary 0.738 0.834
Iceland 0.789 0.834
Ireland 0.762 0.811
Italy 0.711 0.777
Korea 0.709 0.852
Lithuania 0.799 0.741
Malaysia 0.854 0.860
Malta 0.700 0.787
Mexico 0.760 0.802
Norway 0.742 0.669
Poland 0.749 0.805
Portugal 0.769 0.800
Slovak Republic 0.722 0.738
Slovenia 0.681 0.775
Spain 0.711 0.770
Turkey 0.806 0.885

Netherlands 0.741 0.806

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.84
Reliabilities for indices of principals’ Constructivist beliefs about instruction, by country
PBCONS
Australia 0.653
Austria 0.674
Belgium (Fl.) 0.649
Brazil 0.637
Bulgaria 0.251
Denmark 0.655
Estonia 0.419
Hungary 0.563
Iceland 0.517
Ireland 0.650
Italy 0.617
Korea 0.714
Lithuania 0.456
Malaysia 0.668
Malta 0.696
Mexico 0.659
Norway 0.489
Poland 0.508
Portugal 0.640
Slovak Republic 0.609
Slovenia 0.470
Spain 0.608
Turkey 0.463
International sample 0.621

Netherlands 0.698
Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
189

Table 11.85
Model fit for principals’ Constructivist beliefs about instruction, by country
CFI TLI RMSEA SRMR
Australia 0.876 0.629 0.161 0.040
Austria 1.000 1.067 0.000 0.014
Belgium (Fl.) 1.000 1.092 0.000 0.006
Brazil 0.976 0.928 0.082 0.030
Bulgaria1 1.000 1.115 0.000 0.029
Denmark 1.000 1.119 0.000 0.002
Estonia 0.531 -0.407 0.215 0.040
Hungary 1.000 1.110 0.000 0.009
Iceland 0.993 0.980 0.029 0.045
Ireland 1.000 1.063 0.000 0.021
Italy 0.978 0.935 0.074 0.025
Korea 1.000 1.053 0.000 0.014
Lithuania 1.000 1.205 0.000 0.016
Malaysia 0.978 0.933 0.074 0.031
Malta 1.000 1.097 0.000 0.025
Mexico 1.000 1.061 0.000 0.008
Norway 1.000 1.182 0.000 0.011
Poland 1.000 1.182 0.000 0.016
Portugal 1.000 1.100 0.000 0.002
Slovak Republic 0.985 0.956 0.054 0.030
Slovenia 0.981 0.943 0.047 0.026
Spain 1.000 1.197 0.000 0.010
Turkey 0.144 -1.569 0.321 0.077
International sample 0.999 0.996 0.015 0.007

Netherlands Model fit could not be computed due to the small sample size
1. Unique variance of item BCG32L was fixed at zero, to avoid model non-convergence.
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.86
Multiple group CFA examining different levels of cross-cultural invariance
of principals’ Constructivist beliefs about instruction
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.990 0.969 0.042 0.028 - -
Metric invariance 0.958 0.949 0.054 0.078 -0.032 0.012
Scalar invariance 0.317 0.470 0.174 0.192 -0.641 0.120
Uniqueness invariance 0.106 0.529 0.164 0.285 -0.211 -0.010
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.87
Factor loadings used for the computation of factor scores for the indicator principals’
Constructivist beliefs about instruction
BCG32D BCG32F BCG32I BCG32L
International sample 0..948 1.332 1.226 1.000
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

190
chapter 11 Construction and Validation of Scales and Indices

Table 11.88
Item intercepts used for the computation of factor scores for the indicator principals’
Constructivist beliefs about instruction, by country
BCG32D BCG32F BCG32I BCG32L
Australia 2.200 2.098 2.254
Austria 2.529 2.529 2.366
Belgium (Fl.) 2.680 2.589 2.544
Brazil 2.677 2.821 2.770
Bulgaria 2.138 2.161 2.269
Denmark 2.504 2.768 2.701
Estonia 2.462 2.080 2.240
Hungary 2.451 2.451 2.485
Iceland 2.765 2.404 2.321
Ireland 2.285 2.358 2.432
Italy 2.265 1.803 1.919
Korea 2.513 2.721 2.606 2.297
Lithuania 2.656 1.959 2.535
Malaysia 2.550 2.367 2.719
Malta 2.366 2.430 2.705
Mexico 2.652 2.837 2.859
Norway 2.904 2.500 3.039
Poland 2.466 2.474 2.398
Portugal 2.325 2.031 2.527
Slovak Republic 2.530 2.217 2.596
Slovenia 2.197 2.669 2.577
Spain 2.399 2.211 2.289
Turkey 2.457 2.667 2.610

Netherlands 2.974 2.901 2.923

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.89
Unique variances used for the computation of factor scores for the indicator principals’
Constructivist beliefs about instruction, by country
BCG32D BCG32F BCG32I BCG32L
Australia 0.270 0.279 0.194 0.230
Austria 0.180 0.202 0.220 0.241
Belgium (Fl.) 0.210 0.192 0.166 0.232
Brazil 0.220 0.138 0.178 0.329
Bulgaria 0.348 0.356 0.269 0.199
Denmark 0.245 0.177 0.163 0.337
Estonia 0.190 0.367 0.224 0.265
Hungary 0.274 0.193 0.148 0.287
Iceland 0.164 0.195 0.249 0.277
Ireland 0.240 0.341 0.189 0.254
Italy 0.215 0.258 0.219 0.235
Korea 0.234 0.130 0.146 0.208
Lithuania 0.224 0.278 0.254 0.357
Malaysia 0.234 0.360 0.183 0.205
Malta 0.224 0.302 0.120 0.327
Mexico 0.356 0.258 0.168 0.452
Norway 0.352 0.272 0.201 0.318
Poland 0.271 0.325 0.219 0.272
Portugal 0.208 0.332 0.175 0.253
Slovak Republic 0.291 0.323 0.111 0.267
Slovenia 0.462 0.183 0.201 0.241
Spain 0.323 0.254 0.233 0.259
Turkey 0.339 0.123 0.221 0.265

Netherlands 0.205 0.153 0.098 0.251

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
191

Table 11.90
Factor score determinacy for the indicators principals’
Constructivist beliefs about instruction, by country
PBCONS
Australia 0.810
Austria 0.829
Belgium (Fl.) 0.829
Brazil 0.861
Bulgaria 0.717
Denmark 0.835
Estonia 0.720
Hungary 0.822
Iceland 0.759
Ireland 0.818
Italy 0.832
Korea 0.868
Lithuania 0.674
Malaysia 0.815
Malta 0.855
Mexico 0.828
Norway 0.734
Poland 0.741
Portugal 0.797
Slovak Republic 0.843
Slovenia 0.749
Spain 0.796
Turkey 0.802

Netherlands 0.879
Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.91
Item wording of classroom teaching practices items and dimensions
How often do each of the following activities happen in this <target class> throughout the school year?
Classroom teaching practice: BTG42B I explicitly state learning goals.
structuring BTG30C I review with the students the homework they have prepared.
BTG42H I ask my students to remember every step in a procedure.
BTG42I At the beginning of the lesson I present a short summary of the previous lesson.
BTG42M Students evaluate and reflect upon their own work.
Classroom teaching practice: BTG42D Students work in small groups to come up with a joint solution to a problem or task.
student-Oriented BTG42E I give different work to the students that have difficulties learning and/or to those who can advance faster.
BTG42F I ask my students to suggest or to help plan classroom activities or topics.
BTG42N Students work in groups based upon their abilities.
Classroom teaching practice: BTG42J Students work on projects that require at least one week to complete.
enhanced activities BTG42O Students make a product that will be used by someone else.
BTG42Q I ask my students to write an essay in which they are expected to explain their thinking or reasoning at some length.
BTG42S Students hold a debate and argue for a particular point of view which may not be their own.
Source: OECD, TALIS Database.

Table 11.92
Single items measuring classroom teaching practices items and dimensions
How often do each of the following activities happen in this <target class> throughout the school year?
Single items BTG42A I present new topics to the class (lecture-style presentation).
BTG42G I ask my students to remember every step in a procedure.
BTG42K I work with individual students.
BTG42L Students evaluate and reflect upon their own work.
BTG42P I administer a test or quiz to assess student learning.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

192
chapter 11 Construction and Validation of Scales and Indices

Table 11.93
Reliabilities for indices of classroom teaching practices, by country
TPSTRUC TPSTUD TPACTIV
Australia 0.699 0.691 0.598
Austria 0.642 0.701 0.602
Belgium (Fl.) 0.616 0.640 0.515
Brazil 0.763 0.731 0.708
Bulgaria 0.698 0.759 0.734
Denmark 0.622 0.601 0.606
Estonia 0.695 0.678 0.595
Hungary 0.665 0.666 0.610
Iceland 0.711 0.512 0.501
Ireland 0.669 0.638 0.533
Italy 0.712 0.611 0.575
Korea 0.771 0.814 0.825
Lithuania 0.747 0.724 0.704
Malaysia 0.841 0.786 0.764
Malta 0.609 0.701 0.563
Mexico 0.711 0.643 0.663
Norway 0.660 0.483 0.575
Poland 0.719 0.684 0.642
Portugal 0.683 0.690 0.640
Slovak Republic 0.750 0.723 0.575
Slovenia 0.716 0.685 0.621
Spain 0.670 0.655 0.633
Turkey 0.742 0.777 0.794
International sample 0.733 0.702 0.723

Netherlands 0.590 0.634 0.477

Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.94
Reliabilities for indices of classroom teaching practices, by subject
TPSTRUC TPSTUD TPACTIV
Reading, writing and literature 0.745 0.702 0.730
Mathematics 0.699 0.698 0.731
Science 0.694 0.711 0.757
Social studies 0.709 0.690 0.721
Modern foreign languages 0.681 0.679 0.706
Technology 0.726 0.696 0.673
Arts 0.725 0.658 0.608
Physical education 0.746 0.683 0.718
Religion 0.778 0.771 0.747
Practical and vocational skills 0.756 0.731 0.668
Other 0.767 0.764 0.715
International Sample 0.590 0.634 0.729
Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
193

Table 11.95
Model fit and latent correlations for classroom teaching practices, by country

Model fit Latent correlations

CFI TLI RMSEA SRMR TPSTRUC/ TPSTUD TPSTRUC/ TPACTIV TPSTUD/ TPACTIV
Australia 0.862 0.827 0.068 0.052 0.374 0.412 0.666
Austria 0.782 0.725 0.083 0.069 0.229 0.108 0.735
Belgium (Fl.) 0.817 0.770 0.068 0.060 0.199 0.141 0.809
Brazil 0.911 0.887 0.066 0.051 0.465 0.499 0.852
Bulgaria 0.886 0.857 0.069 0.056 0.444 0.369 0.721
Denmark 0.849 0.810 0.066 0.051 0.608 0.535 0.746
Estonia 0.830 0.787 0.078 0.062 0.371 0.358 0.727
Hungary 0.849 0.811 0.062 0.056 0.190 0.366 0.745
Iceland 0.835 0.792 0.060 0.052 0.549 0.520 0.775
Ireland 0.757 0.695 0.083 0.065 0.129 -0.114 0.562
Italy 0.829 0.785 0.072 0.059 0.378 0.382 0.759
Korea 0.903 0.878 0.081 0.077 0.470 0.380 0.853
Lithuania 0.858 0.821 0.082 0.066 0.459 0.297 0.779
Malaysia 0.943 0.928 0.061 0.042 0.680 0.420 0.862
Malta 0.829 0.785 0.070 0.051 0.303 0.208 0.757
Mexico 0.924 0.904 0.050 0.038 0.580 0.528 0.829
Norway 0.827 0.783 0.059 0.053 0.646 0.481 0.693
Poland 0.831 0.788 0.074 0.060 0.557 0.521 0.700
Portugal 0.830 0.786 0.076 0.072 0.022 0.078 0.749
Slovak Republic 0.881 0.850 0.071 0.054 0.480 0.381 0.773
Slovenia 0.869 0.835 0.065 0.051 0.376 0.267 0.663
Spain 0.857 0.821 0.064 0.053 0.425 0.321 0.803
Turkey 0.906 0.882 0.070 0.058 0.642 0.500 0.857
International sample 0.899 0.873 0.060 0.043 0.436 0.363 0.786

Netherlands 0.801 0.749 0.070 0.066 0.380 0.296 0.778

Table 11.96
Model fit and latent correlations for classroom teaching practices, by subject

Model fit Latent correlations

CFI TLI RMSEA SRMR TPSTRUC/ TPSTUD TPSTRUC/ TPACTIV TPSTUD/ TPACTIV
Reading, writing and literature 0.921 0.901 0.057 0.042 0.480 0.349 0.828
Mathematics 0.912 0.889 0.054 0.046 0.373 0.220 0.788
Science 0.913 0.891 0.056 0.046 0.506 0.349 0.849
Social studies 0.901 0.876 0.064 0.044 0.502 0.376 0.846
Modern foreign languages 0.920 0.900 0.054 0.041 0.511 0.308 0.792
Technology 0.881 0.851 0.067 0.046 0.496 0.514 0.796
Arts 0.888 0.859 0.063 0.046 0.596 0.512 0.867
Physical education 0.850 0.811 0.081 0.065 0.704 0.772 0.659
Religion 0.879 0.848 0.073 0.051 0.606 0.537 0.861
Practical and vocational skills 0.883 0.852 0.067 0.050 0.421 0.341 0.847
Other 0.891 0.863 0.072 0.054 0.539 0.441 0.899
International Sample 0.899 0.873 0.060 0.043 0.436 0.363 0.786
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
p < 0.05
The Netherlands was excluded because it did not meet international sampling standards. p < 0.01
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

194
chapter 11 Construction and Validation of Scales and Indices

Table 11.97
Multiple group CFA examining different levels of cross-cultural invariance
of Classroom teaching practice: structuring
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.925 0.853 0.101 0.039 - -
Metric invariance 0.895 0.882 0.090 0.070 -0.030 -0.011
Scalar invariance 0.485 0.593 0.167 0.171 -0.410 0.077
Uniqueness invariance 0.338 0.616 0.163 0.275 -0.147 -0.004
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.98
Multiple group CFA examining different levels of cross-cultural invariance
of Classroom teaching practice: student-oriented
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.986 0.959 0.056 0.018 - -
Metric invariance 0.966 0.958 0.056 0.045 -0.020 0.000
Scalar invariance 0.607 0.695 0.153 0.142 -0.359 0.097
Uniqueness invariance 0.382 0.675 0.158 0.159 -0.225 0.005
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.99
Multiple group CFA examining different levels of cross-cultural invariance
of Classroom teaching practice: enhanced activities
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.859 0.557 0.161 0.049 - -
Metric invariance 0.868 0.837 0.100 0.064 0.009 -0.061
Scalar invariance 0.460 0.581 0.160 0.137 -0.408 0.060
Uniqueness invariance 0.183 0.570 0.162 0.225 -0.277 0.002
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.100
Factor loadings used for the computation of factor scores for the indicator
Classroom teaching practice: structuring
BTG42B BTG42C BTG42H BTG42I BTG42M
International sample 0.899 1.245 1.233 1.219 1.000
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
195

Table 11.101
Item intercepts used for the computation of factor scores for the indicator
Classroom teaching practice: structuring, by country
BTG42B BTG42C BTG42H BTG42I BTG42M
Australia 3.533 2.895 3.481 2.977
Austria 3.412 3.197 3.604 2.936
Belgium (Fl.) 3.118 2.600 3.545 2.186
Brazil 3.507 3.685 3.680 3.748
Bulgaria 4.303 3.252 3.943 2.912
Denmark 2.797 3.157 3.156 2.478
Estonia 4.140 3.575 3.392 2.739
Hungary 4.657 4.074 3.747 3.091
Iceland 3.173 2.900 2.901 3.279
Ireland 3.507 3.755 3.426 3.399
Italy 3.349 3.508 3.600 3.048
Korea 4.455 3.566 4.450 3.709 4.218
Lithuania 4.365 3.858 3.969 3.044
Malaysia 3.940 3.954 3.583 4.123
Malta 3.252 3.031 3.532 3.050
Mexico 3.935 3.914 3.541 3.674
Norway 3.359 3.110 3.568 3.112
Poland 4.019 3.123 3.217 2.426
Portugal 3.575 3.467 3.779 2.685
Slovak Republic 3.798 3.377 3.966 2.972
Slovenia 3.939 3.394 3.666 2.660
Spain 3.263 4.058 3.497 3.080
Turkey 3.612 2.843 3.739 2.896

Netherlands 3.209 3.729 3.565 3.498

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.102
Unique variances used for the computation of factor scores for the indicator
Classroom teaching practice: structuring, by country
BTG42B BTG42C BTG42H BTG42I BTG42M
Australia 1.228 1.141 1.106 0.974 0.661
Austria 1.415 2.083 1.442 1.247 0.921
Belgium (Fl.) 1.390 1.423 1.188 1.195 0.681
Brazil 1.322 0.952 1.037 1.036 0.620
Bulgaria 0.493 1.407 0.536 1.154 0.447
Denmark 1.178 1.189 1.518 0.814 0.964
Estonia 1.144 1.428 1.384 1.024 0.756
Hungary 0.770 1.539 1.194 1.047 0.836
Iceland 1.113 1.213 1.068 1.188 1.073
Ireland 1.331 0.978 1.191 0.899 0.536
Italy 1.522 1.092 1.051 1.217 0.398
Korea 1.104 0.922 0.959 0.955 0.874
Lithuania 0.630 1.004 0.822 1.082 0.621
Malaysia 0.843 0.478 0.787 0.793 0.683
Malta 1.558 1.292 1.245 1.063 0.562
Mexico 0.989 0.962 1.276 1.143 0.719
Norway 1.140 0.863 1.313 1.007 0.895
Poland 1.328 1.487 1.756 1.114 0.819
Portugal 1.473 1.501 1.093 1.295 0.569
Slovak Republic 1.429 1.442 0.905 1.040 0.715
Slovenia 1.003 1.700 0.990 1.087 0.542
Spain 1.325 1.218 1.737 1.315 0.808
Turkey 1.262 1.274 1.175 1.261 0.732

Netherlands 1.501 1.842 1.451 1.714 1.060

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

196
chapter 11 Construction and Validation of Scales and Indices

Table 11.103
Factor loadings used for the computation of factor scores for the indicator
Classroom teaching practice: student-oriented
BTG42D BTG42E BTG42F BTG42N
International sample 0.949 0.990 0.880 1.000
Source: OECD, TALIS Database.

Table 11.104
Item intercepts used for the computation of factor scores for the indicator
Classroom teaching practice: student-oriented, by country
BTG42D BTG42E BTG42F BTG42N
Australia 2.587 2.696 1.946
Austria 2.520 2.396 1.736
Belgium (Fl.) 2.641 2.423 1.929
Brazil 2.253 1.729 1.429
Bulgaria 2.122 2.390 2.055
Denmark 2.817 2.401 1.638
Estonia 2.335 2.649 1.719
Hungary 2.157 2.385 1.542
Iceland 2.773 3.041 2.211
Ireland 2.296 2.538 1.923
Italy 2.170 2.802 2.963
Korea 2.257 2.096 2.270 2.078
Lithuania 2.287 2.913 1.423
Malaysia 2.272 1.742 1.679
Malta 2.415 2.037 1.816
Mexico 2.763 2.106 1.334
Norway 3.010 3.474 2.054
Poland 2.064 2.497 1.484
Portugal 2.600 2.455 1.838
Slovak Republic 2.167 2.440 1.818
Slovenia 2.103 2.315 1.878
Spain 2.391 2.787 1.522
Turkey 2.201 2.450 2.585

Netherlands 2.981 2.375 2.063

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
197

Table 11.105
Unique variances used for the computation of factor scores for the indicator
Classroom teaching practice: student-oriented, by country
BTG42D BTG42E BTG42F BTG42N
Australia 0.771 1.268 0.568 0.996
Austria 0.774 1.177 0.610 0.876
Belgium (Fl.) 0.720 0.898 0.440 0.629
Brazil 0.833 1.332 1.028 1.131
Bulgaria 0.812 0.847 1.199 0.848
Denmark 0.912 1.301 0.527 1.016
Estonia 0.712 1.214 0.498 0.855
Hungary 0.486 0.848 0.484 0.896
Iceland 1.248 1.753 0.729 0.995
Ireland 0.673 1.438 0.533 0.881
Italy 0.686 1.287 1.695 0.926
Korea 0.492 0.547 0.648 0.733
Lithuania 0.643 1.174 0.743 0.831
Malaysia 0.517 0.634 0.648 0.658
Malta 0.804 0.793 0.500 0.859
Mexico 0.905 1.393 1.081 1.486
Norway 0.992 1.697 0.605 0.945
Poland 0.722 1.132 1.054 1.060
Portugal 0.952 1.157 0.720 1.044
Slovak Republic 0.590 0.986 0.829 0.905
Slovenia 0.589 0.863 0.788 0.865
Spain 1.120 1.671 0.440 1.244
Turkey 0.812 1.132 1.187 0.986

Netherlands 1.317 0.962 0.508 0.741

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.106
Factor loadings used for the computation of factor scores for the indicator
Classroom teaching practice: enhanced activities
BTG42J BTG42O BTG42Q BTG42S
International sample 1.147 1.134 1.000 1.192
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

198
chapter 11 Construction and Validation of Scales and Indices

Table 11.107
Item intercepts used for the computation of factor scores for the indicator
Classroom teaching practice: enhanced activities, by country
BTG42J BTG42O BTG42Q BTG42S
Australia 2.629 1.698 1.729
Austria 2.221 1.726 2.204
Belgium (Fl.) 2.289 1.786 2.062
Brazil 1.202 0.975 1.808
Bulgaria 1.935 1.808 2.476
Denmark 1.605 1.256 1.677
Estonia 1.708 1.807 2.105
Hungary 1.972 1.607 2.162
Iceland 2.344 2.286 1.529
Ireland 2.165 1.560 1.761
Italy 2.236 1.828 2.720
Korea 1.801 1.735 1.704 1.754
Lithuania 1.714 1.453 2.117
Malaysia 1.945 1.439 1.444
Malta 2.083 1.336 2.013
Mexico 1.649 1.007 1.492
Norway 1.696 1.233 1.784
Poland 1.642 1.632 2.557
Portugal 2.087 1.739 1.857
Slovak Republic 2.115 1.739 2.837
Slovenia 2.106 1.742 2.339
Spain 1.753 1.120 1.618
Turkey 2.227 2.055 1.859

Netherlands 2.041 1.421 1.728

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.108
Unique variances used for the computation of factor scores for the indicator
Classroom teaching practice: enhanced activities, by country
BTG42J BTG42O BTG42Q BTG42S
Australia 1.241 0.975 0.535 0.504
Austria 0.894 0.456 0.379 0.851
Belgium (Fl.) 1.201 0.376 0.280 0.506
Brazil 1.062 1.096 1.103 1.189
Bulgaria 0.587 0.579 0.568 0.847
Denmark 0.665 0.574 0.674 0.990
Estonia 0.542 0.554 0.463 0.746
Hungary 0.517 0.275 0.277 0.612
Iceland 1.409 1.193 0.380 0.234
Ireland 1.178 0.380 0.527 0.434
Italy 1.283 0.935 0.719 1.245
Korea 0.720 0.480 0.509 0.363
Lithuania 0.457 0.625 0.544 0.848
Malaysia 0.872 0.653 1.009 0.490
Malta 1.053 0.277 0.730 0.910
Mexico 1.324 1.197 1.313 1.197
Norway 0.865 0.513 0.666 0.759
Poland 0.585 0.731 0.695 1.255
Portugal 1.006 0.662 0.486 0.587
Slovak Republic 0.566 0.464 0.501 1.242
Slovenia 0.484 0.265 0.211 0.747
Spain 1.013 0.387 0.712 1.014
Turkey 1.067 0.779 0.810 0.934

Netherlands 1.232 0.206 0.349 0.457

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
199

Table 11.109
Factor score determinacy for the indicators of Classroom teaching practice, by country
TPSTRUC TPSTUD TPACTIV
Australia 0.838 0.866 0.840
Austria 0.819 0.859 0.830
Belgium (Fl.) 0.808 0.863 0.840
Brazil 0.894 0.898 0.893
Bulgaria 0.861 0.890 0.875
Denmark 0.815 0.838 0.842
Estonia 0.849 0.858 0.834
Hungary 0.827 0.852 0.835
Iceland 0.851 0.789 0.829
Ireland 0.829 0.832 0.794
Italy 0.866 0.850 0.827
Korea 0.888 0.926 0.931
Lithuania 0.874 0.895 0.885
Malaysia 0.928 0.928 0.913
Malta 0.805 0.873 0.832
Mexico 0.859 0.862 0.866
Norway 0.835 0.792 0.800
Poland 0.862 0.857 0.840
Portugal 0.850 0.858 0.841
Slovak Republic 0.888 0.882 0.840
Slovenia 0.862 0.853 0.827
Spain 0.838 0.858 0.850
Turkey 0.889 0.915 0.913

Netherlands 0.790 0.835 0.805

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.110
Item wording of Co-operation among staff
How often do you do the following in this school?
Exchange and co-ordination BTG30C Discuss and decide on the selection of instructional media (e.g. textbooks, exercise books).
for teaching BTG30D Exchange teaching materials with colleagues.
BTG30E Attend team conferences for the age group I teach.
BTG30F Ensure common standards in evaluations for assessing student progress.
BTG30G Engage in discussion of the learning developments of specific students.
Professional collaboration BTG30H Teach jointly as a team in the same class.
BTG30I Take part in professional learning activities (e.g. team supervision).
BTG30J Observe other teachers’ classes and provide feedback.
BTG30K Engage in joint activities across different classes and age groups (e.g. projects).
BTG30L Discuss and coordinate homework practice across subjects.
Source: OECD, TALIS Database.

Table 11.111
Single items measuring Co-operation among staff
How often do you do the following in this school?
Exchange and co-ordination BTG30A Attend staff meetings to discuss the vision and mission of the school.
for teaching BTG30B Develop a school curriculum or part of it.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

200
chapter 11 Construction and Validation of Scales and Indices

Table 11.112
Reliabilities for indices of Co-operation among staff, by country
TCEXCHAN TCCOLLAB
Australia 0.650 0.740
Austria 0.689 0.658
Belgium (Fl.) 0.646 0.503
Brazil 0.772 0.739
Bulgaria 0.488 0.603
Denmark 0.732 0.660
Estonia 0.628 0.631
Hungary 0.645 0.677
Iceland 0.685 0.710
Ireland 0.670 0.555
Italy 0.612 0.592
Korea 0.716 0.808
Lithuania 0.664 0.686
Malaysia 0.761 0.760
Malta 0.655 0.633
Mexico 0.663 0.682
Norway 0.611 0.631
Poland 0.704 0.714
Portugal 0.633 0.572
Slovak Republic 0.723 0.711
Slovenia 0.680 0.652
Spain 0.595 0.600
Turkey 0.754 0.722
International sample 0.700 0.689

Netherlands 0.421 0.593

Notes: Reliabilities (Cronbach’s Alpha) computed with weighted national samples.
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.113
Model fit and latent correlations for indices of Co-operation among staff, by country
Model fit Latent correlations
CFI TLI RMSEA SRMR TCEXCHAN/TCCOLLAB
Australia 0.914 0.886 0.066 0.048 0.670
Austria 0.929 0.905 0.061 0.038 0.768
Belgium (Fl.) 0.913 0.885 0.049 0.038 0.668
Brazil 0.926 0.902 0.072 0.050 0.716
Bulgaria 0.863 0.818 0.060 0.045 0.752
Denmark 0.866 0.823 0.086 0.053 0.902
Estonia 0.934 0.913 0.049 0.036 0.779
Hungary 0.953 0.937 0.039 0.032 0.753
Iceland 0.926 0.902 0.059 0.043 0.678
Ireland 0.897 0.863 0.062 0.042 0.828
Italy 0.922 0.896 0.053 0.038 0.820
Korea 0.916 0.889 0.082 0.055 0.690
Lithuania 0.923 0.899 0.061 0.043 0.826
Malaysia 0.944 0.926 0.056 0.036 0.709
Malta 0.941 0.922 0.044 0.039 0.699
Mexico 0.923 0.898 0.057 0.039 0.762
Norway 0.892 0.858 0.061 0.043 0.788
Poland 0.916 0.888 0.062 0.043 0.762
Portugal 0.902 0.870 0.057 0.042 0.666
Slovak Republic 0.938 0.918 0.055 0.035 0.767
Slovenia 0.928 0.905 0.055 0.039 0.601
Spain 0.917 0.891 0.046 0.037 0.569
Turkey 0.891 0.856 0.090 0.055 0.856
International sample 0.949 0.932 0.042 0.028 0.764

Netherlands 0.897 0.863 0.050 0.044 0.809

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
201

Table 11.114
Multiple group CFA examining different levels of cross-cultural invariance
of Exchange and co-ordination for teaching
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.953 0.905 0.072 0.031 - -
Metric invariance 0.904 0.891 0.077 0.083 -0.049 0.005
Scalar invariance 0.044 0.244 0.204 0.326 -0.860 0.127
Uniqueness invariance 0.000 0.199 0.210 0.419 -0.044 0.006
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.115
Multiple group CFA examining different levels of cross-cultural invariance
of Professional collaboration
Model fit Difference

CFI TLI RMSEA SRMR Δ CFI Δ RMSEA

Configural invariance 0.952 0.904 0.074 0.031 - -
Metric invariance 0.908 0.896 0.077 0.064 -0.044 0.003
Scalar invariance 0.152 0.330 0.196 0.268 -0.756 0.119
Uniqueness invariance 0.000 0.368 0.190 0.263 -0.152 -0.006
Notes: Model estimates based on international calibration sample (1 000 teachers per country).
The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.116
Factor loadings used for the computation of factor scores for the indicator
Exchange and co-ordination for teaching
BTG30C BTG30D BTG30E BTG30F BTG30G
International sample 1.000 1.649 1.576 1.636 1.631
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

202
chapter 11 Construction and Validation of Scales and Indices

Table 11.117
Item intercepts used for the computation of factor scores for the indicator
Exchange and co-ordination for teaching, by country
BTG30C BTG30D BTG30E BTG30F BTG30G
Australia 5.153 3.487 4.290 4.924
Austria 5.337 4.121 3.439 5.275
Belgium (Fl.) 5.269 4.298 3.960 4.587
Brazil 4.516 4.887 4.927 5.139
Bulgaria 4.895 3.731 5.735 5.221
Denmark 3.315 3.650 2.124 3.124
Estonia 4.810 4.875 4.689 5.942
Hungary 4.739 4.721 4.624 4.625
Iceland 4.023 4.787 3.548 4.599
Ireland 5.015 3.645 3.965 4.787
Italy 4.019 3.987 3.770 4.747
Korea 3.481 4.997 4.638 4.707 4.010
Lithuania 4.610 3.900 4.644 4.703
Malaysia 4.172 3.613 4.095 3.899
Malta 5.319 3.352 4.639 5.220
Mexico 4.290 4.663 5.198 4.579
Norway 5.919 6.733 5.561 5.959
Poland 5.044 4.901 4.687 5.880
Portugal 5.161 2.333 4.409 4.493
Slovak Republic 4.862 4.194 3.906 4.428
Slovenia 5.277 5.102 4.599 4.633
Spain 4.977 4.641 4.282 4.606
Turkey 4.988 3.684 4.600 5.004

Netherlands 4.972 4.081 4.522 5.339

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.118
Unique variances used for the computation of factor scores for the indicator
Exchange and co-ordination for teaching, by country

BTG30C BTG30D BTG30E BTG30F BTG30G

Australia 1.126 0.781 2.357 0.805 0.792
Austria 0.406 0.956 1.731 1.327 1.078
Belgium (Fl.) 0.715 1.087 1.226 1.138 0.644
Brazil 0.899 1.774 1.322 0.790 0.653
Bulgaria 0.485 1.468 1.075 0.639 1.304
Denmark 0.898 0.702 0.683 1.021 0.861
Estonia 0.603 1.210 1.012 2.042 0.828
Hungary 0.351 1.504 1.307 1.308 1.083
Iceland 1.435 1.752 2.103 1.706 1.308
Ireland 0.547 1.136 1.460 1.024 1.495
Italy 0.856 1.265 1.373 0.959 0.345
Korea 1.499 1.067 2.006 0.437 1.294
Lithuania 0.741 0.868 1.108 1.696 0.794
Malaysia 1.046 0.713 1.398 0.389 0.608
Malta 1.485 1.949 1.729 1.355 1.704
Mexico 1.941 1.465 0.775 1.313 1.092
Norway 0.863 1.043 0.833 0.910 0.861
Poland 0.399 1.014 0.942 0.922 0.822
Portugal 0.752 0.957 1.114 0.729 1.071
Slovak Republic 0.707 0.908 0.843 0.629 0.857
Slovenia 0.414 1.544 0.763 0.893 0.565
Spain 1.221 1.297 1.916 0.962 1.106
Turkey 1.275 1.886 1.012 0.816 1.093

Netherlands 0.736 1.319 2.745 1.497 0.589

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
203

Table 11.119
Factor loadings used for the computation of factor scores for the indicator
Professional collaboration
BTG30H BTG30I BTG30J BTG30K BTG30L
International sample 1.449 1.360 1.000 1.249 1.510
Source: OECD, TALIS Database.

Table 11.120
Item intercepts used for the computation of factor scores for the indicator
Professional collaboration, by country
BTG30H BTG30I BTG30J BTG30K BTG30L
Australia 3.073 3.275 2.711 2.668
Austria 4.369 2.601 4.025 3.298
Belgium (Fl.) 3.106 3.286 4.476 3.530
Brazil 3.972 3.409 4.126 3.901
Bulgaria 1.787 3.394 2.884 3.959
Denmark 5.129 3.297 3.911 3.603
Estonia 3.219 3.755 3.015 3.457
Hungary 1.433 2.306 2.910 3.123
Iceland 3.548 2.971 3.554 3.608
Ireland 3.727 5.115 3.773 4.064
Italy 4.509 3.587 4.288 4.072
Korea 2.528 1.662 2.389 1.243 1.221
Lithuania 3.112 2.430 2.551 2.270
Malaysia 3.080 3.011 3.156 3.783
Malta 3.365 3.553 3.510 3.857
Mexico 4.534 3.843 3.775 3.269
Norway 4.410 2.577 3.421 4.687
Poland 1.904 2.227 2.462 1.792
Portugal 4.230 3.243 4.228 3.881
Slovak Republic 2.587 1.724 2.509 3.169
Slovenia 3.766 2.796 3.873 3.429
Spain 3.645 3.508 3.721 4.104
Turkey 2.706 3.144 3.357 4.365

Netherlands 2.411 3.131 3.433 2.088

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

TALIS 2008 Technical Report © OECD 2010

204
chapter 11 Construction and Validation of Scales and Indices

Table 11.121
Unique variances used for the computation of factor scores for the indicator
Professional collaboration, by country
BTG30H BTG30I BTG30J BTG30K BTG30L
Australia 2.373 1.348 1.547 1.430 1.847
Austria 3.267 0.865 1.458 0.826 1.379
Belgium (Fl.) 1.644 1.060 0.709 1.011 1.564
Brazil 2.144 1.654 1.280 1.042 2.369
Bulgaria 1.945 1.073 1.369 1.381 3.166
Denmark 1.155 1.948 2.091 1.081 1.675
Estonia 2.633 1.170 1.245 1.045 2.301
Hungary 1.318 1.469 1.566 1.127 1.986
Iceland 3.604 1.660 1.567 1.430 1.920
Ireland 2.631 0.858 0.511 1.517 1.462
Italy 4.708 0.962 1.506 2.040 1.940
Korea 2.056 0.737 0.553 0.672 0.834
Lithuania 2.617 0.789 0.926 0.869 1.842
Malaysia 1.876 1.002 1.365 1.131 1.709
Malta 1.747 1.418 0.673 1.199 1.814
Mexico 3.634 1.748 1.343 1.290 1.696
Norway 3.128 1.533 2.071 1.197 1.537
Poland 2.843 1.081 1.257 0.873 2.085
Portugal 3.259 1.248 1.151 1.495 2.511
Slovak Republic 1.974 1.040 1.378 1.017 1.463
Slovenia 2.639 1.177 0.914 1.064 1.876
Spain 2.716 1.469 0.723 1.519 3.009
Turkey 1.650 0.888 1.081 1.208 1.752

Netherlands 2.808 2.531 1.473 0.821 2.039

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

Table 11.122
Factor score determinacy for the indicators of co-operation among staff, by country
TCEXCHAN TCCOLLAB
Australia 0.870 0.874
Austria 0.869 0.858
Belgium (Fl.) 0.843 0.782
Brazil 0.908 0.889
Bulgaria 0.770 0.843
Denmark 0.894 0.882
Estonia 0.846 0.850
Hungary 0.834 0.850
Iceland 0.860 0.874
Ireland 0.854 0.814
Italy 0.833 0.836
Korea 0.887 0.923
Lithuania 0.867 0.876
Malaysia 0.893 0.894
Malta 0.837 0.840
Mexico 0.861 0.868
Norway 0.834 0.842
Poland 0.870 0.876
Portugal 0.840 0.814
Slovak Republic 0.892 0.881
Slovenia 0.844 0.836
Spain 0.805 0.805
Turkey 0.911 0.907

Netherlands 0.776 0.815

Note: The Netherlands was excluded because it did not meet international sampling standards.
Source: OECD, TALIS Database.

© OECD 2010 TALIS 2008 Technical Report

Construction and Validation of Scales and Indices chapter 11
205

References
Bollen, K.A. (1989), Structural Equations with Latent Variables, John Wiley & Sons, New York.

Brown, T.A. (2006), Confirmatory Factor Analysis for Applied Research, Guilford Press, New York.

Chen, F.F. (2007), “Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance”, Structural Equation Modeling,
Vol. 14, No. 3, pp. 464-504, Psychology Press.

Cheung, G.W. and R.B. Rensvold (1998), “Cross-Cultural Comparisons Using Non-invariant Measurement Items”, Applied
Behavioral Science Review, Vol. 6, No. 1, NTL Institute, pp. 93-110.

Cheung, G.W. and R.B. Rensvold (1999), “Testing Factorial Invariance Across Groups: A Reconceptualization and Proposed
New Method”, Journal of Management, Vol. 25, No. 1, pp. 1-27, Southern Management Association.

Cheung, G.W. and R.B. Rensvold (2002), “Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance”, Structural
Equation Modeling, Vol. 9, No. 2, pp. 233-255, Psychology Press.

Davidov, E. (2008), “A Cross-Country and Cross-Time Comparison of the Human Values Measurements with the Second Round
of the European Social Survey”, Survey Research Methods, Vol. 2, No. 1, pp. 33-46, European Survey Research Association.

Dempster, A.P., N.M. Laird and D.B. Rubin (1977), “Maximum Likelihood from Incomplete Data via the EM Algorithm”,
Journal of the Royal Statistical Society, Series B (Methodological), Vol. 39, No. 1, pp. 1-38.

Gorsuch, R.L. (1983), Factor Analysis, Lawrence Erlbaum Associates, Hillsdale, NJ.

Grice, J.W. (2001), “Computing and evaluating factor scores”, Psychological Methods, No. 6 (4), pp. 430-450.

Hansen, Y.K., M. Rosén, and J.-E. Gustafsson, (2006), “Measures of Self-reported Reading Resources, Attitudes and Activities
Based on Latent Variable Modelling”, International Journal of Research & Method in Education, Vol. 29, No. 2, Routledge,
pp. 221-237.

Hox, J.J. (2002), Multilevel Analysis: Techniques and Applications, Lawrence Erlbaum Associates, Mahwah, NJ.

Hu, L. and P.M. Bentler (1999), “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus
New Alternatives”, Structural Equation Modeling, Vol. 6, No. 1, pp. 1-55, Psychology Press.

Kline, R.B. (2005), Principles and Practice of Structural Equation Modeling (2nd edition), Guilford Press, New York.

Lüdtke, O., A. Robitzsch, U. Trautwein and O. Köller (2007), “Umgang mit fehlenden Werten in der psychologischen
Forschung: Probleme und Lösungen” (Handling of Missing Data in Psychological Research), Psychologische Rundschau,
Vol. 58, pp. 103-117, Deutsche Gesellschaft für Psychologie.

Meredith, W. (1993), “Measurement Invariance, Factor Analysis and Factorial Invariance”, Psychometrika, Vol. 58, No. 4, pp.
525-543, Psychometric Society.

Muthén, B. (1977), SomeResults on Using Summed Raw Scores and Factor Scores from Dichotomous Items in the Estimation
of Structural Equation Model, Unpublished Technical Report, University of Uppsala, Sweden, http://www.gseis.ucla.edu/
faculty/muthen/articles/Muthen_Unpublished_01.pdf, accessed 15 December 2008.

Muthén, B.O. and D. Kaplan (1985), “A Comparison of some Methodologies for the Factor Analysis of Non-Normal Likert
Variables”, British Journal of Mathematical and Statistical Psychology, Vol. 38, pp. 171-189, The British Psychological Society.

Muthén, B.O. (1998-2004), Mplus Technical Appendices, Muthén & Muthén, Los Angeles, CA.

Muthén, L.K. and B.O. Muthén (1998-2007), Mplus User’s Guide (Fifth edition). Muthén & Muthén, Los Angeles, CA.

OECD (2009), PISA 2006 Technical Report, OECD, Paris.

TALIS 2008 Technical Report © OECD 2010

206
chapter 11 Construction and Validation of Scales and Indices

Raudenbush, S.W. and A.S. Bryk (2002), Hierarchical Linear Models: Applications and Data Analysis Methods (2nd edition),
SAGE Publications, London.

Schafer. J.L. and J.W. Graham (2002), Missing Data: Our View of the State of the Art, Psychological Methods, Vol. 7, No. 2,
pp. 147-177, American Psychological Association.

Schermelleh-Engel, K. and H. Moosbrugger (2002), “Beurteilung der Modellgüte von Strukturgleichungsmodellen” (Evaluating
the Fit of Structural Equation Model), Arbeiten aus dem Institut für Psychologie, Heft 4/2002, J.W. Goethe-Universität, Frankfurt
am Main.

Schulz, W. (2005), “Testing Parameter Invariance for Questionnaire Indices using Confirmatory Factor Analysis and Item
Response Theory”, paper presented at the 2006 Annual Meeting of the American Educational Research Association (AERA),
San Francisco.

Snijders, T.A.B. and R .J. Bosker (1994), “Modeled Variance in Two-Level Models”, Sociological Methods & Research, Vol. 22,
No. 3, pp. 342-363, SAGE Publications.

Steenkamp, J.-B.E.M. and H. Baumgartner (1998), “Assessing Measurement Invariance in Cross-National Consumer Research”,
Journal of Consumer Research, Vol. 25, No. 1, The University of Chicago Press, pp. 78-90.

Van de Vijver, F.J.R and K. Leung (1997), Methods and Data Analysis for Cross-Cultural Research, SAGE Publications, London.

© OECD 2010 TALIS 2008 Technical Report

207

Annex A
TALIS Consortium, Experts
and Consultants

TALIS 2008 Technical Report © OECD 2010

208
Annex A TALIS Consortium, Experts and Consultants

IEA Data Processing and Research Center (Hamburg, Germany)

Dirk Hastedt (International Project Co-Director)
Steffen Knoll (International Project Co-Director)
Friederike Westphal (International Project Co-ordinator, field operations)
Ralph Carstens (International Project Manager, data and analysis)
Alena Becker (International Deputy Project Manager, data and analysis)

At the ISC Mr. Dirk Hastedt and Dr. Steffen Knoll acted as co-directors of TALIS. Mr. Hastedt was responsible for the overall
budget, contractual agreements and consulting, while Dr. Knoll took care of the international schedules, co-ordinating
the consortium and the internal budget. As study co-ordinator Ms. Friederike Westphal maintained close contact with the
National Project Managers (NPMs) and co-ordinated the survey administration at the international level. Mr. Ralph Carstens
acted as director of data management and was involved in the analysis of the TALIS data. As deputy data manager Ms. Alena
Becker was responsible for the everyday business of data processing in co-operation with the national data managers.

Simone Uecker (layout verification, national adaptations)

Daniel Radtke (layout verification, national adaptations)
Sebastian Meyer (layout verification, national adaptations)
Laura Romero de Rosenbusch (layout verification, national adaptations)

Dirk Oehler (data processing)

Tim Daniel (data processing)
Michael Jung (data processing)
Keith Hanmer (data processing)
Limiao Duan (data processing)

Plamen Mirazchiyski (data analysis and quality control)

Leslie Rutkowski (data analysis and quality control)

Stephan Petzchen (software development)

Harpreet Singh Choudry (software development)
Siddharth Somasundaram (software development)
Martin Olszewski (software development)
Bastian Deppe (software development)
Christian Harries (software development)
Alexander Konn (software development)
Lukas Bujara (software development)
Hauke Heyen (software development)

Bettina Wietzorek (meeting organisation)

IEA Secretariat (Amsterdam, Netherlands)

Barbara Malak-Minkiewicz (translation verification and international quality control)

Suzanne Morony (translation verification and international quality control)

At the IEA Secretariat Dr. Barbara Malak-Minkiewicz and Dr. Suzanne Morony co-ordinated the translation verification
and the implementation of international quality control procedures and instruments. cApStAn Linguistic Quality Control,
an independent linguistic quality control agency located in Brussels, Belgium performed the translation verification for 24
participants in a total of 31 languages. The IEA Secretariat appointed, contracted and trained independent quality control
monitors to watch over survey implementation in each participating country.

© OECD 2010 TALIS 2008 Technical Report

TALIS Consortium, Experts and Consultants Annex A
209

Statistics Canada (Ottawa, Canada)

Jean Dumais (sampling referee)

Sylvie LaRoche (sampling and weighting)

The sampling referee, Mr. Jean Dumais, and his sampling team manager, Ms. Sylvie LaRoche, both of Statistics Canada,
conducted the sampling, weighting and adjudication. Based on the sampling frame information provided by the countries,
the sampling team drew school samples of all participating countries prior to the field trial and performed the weighting and
sample adjudication for the main survey prior to data analysis.

TALIS Expert Groups

Instrument Development Expert Group
David Baker (Pennsylvania State University, United States)
Michael Davidson (OECD Secretariat)
Aletta Grisay (Consultant, Paris, France)
Ben Jensen (OECD Secretariat)
Eckhard Klieme (German Institute for International Educational Research (DIPF), Frankfurt, Germany)
Jaap Scheerens (University of Twente, the Netherlands)

Analysis Group
David Baker (Pennsylvania State University, United States)
Juan León (Pennsylvania State University, United States)
Eckhard Klieme (German Institute for International Educational Research (DIPF), Frankfurt, Germany)
Svenja Vieluf (German Institute for International Educational Research (DIPF), Frankfurt, Germany)
David Kaplan (University of Wisconsin – Madison, United States)
Fons van de Vijver (University of Tilburg)

TALIS 2008 Technical Report © OECD 2010

211

Annex B
Characteristics of National Samples

TALIS 2008 Technical Report © OECD 2010

212
Annex B Characteristics of National Samples

Australia

In Australia, the education system is the responsibility of each individual state or territory. ISCED Level 2 education corresponds
to junior secondary schooling and covers Grades 7 to 10 in Australian Capital Territory, New South Wales, Tasmania and
Victoria, while it covers Grades 8 to 10 in Northern Territory, Queensland, South Australia and Western Australia.

Sample design ISCED Level 2

Schools Students
International target population 2 617 969 509
Out of scope - -
National target population 2 617 969 509
Exclusions
Non mainstream schools (correctional, hospital, environment, distance education, language support centres,
110 3 852
mature age, non-English curriculum and special schools)
Survey population 2 507 965 657
Coverage rate 95.8% 99.6%

Sample allocation: Proportional to the number of schools, then adjusted to have a minimum of four schools
selected in the smaller strata.
Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Method of sample selection: Probability proportional to size.
Measure of size (MOS): Count of ISCED Level 2 students in school.
Explicit stratification: Explicit stratification was organised by state and territory, for a total of eight explicit strata.
Implicit stratification: Implicit stratification was organised by sector (Catholic, Government or Independent),
geography (nine categories from metropolitan to remote areas) and Quintiles of the Education
and Occupation index from SEIFA (Socio-Economic Index for Areas) based on the postal code
of the school (five categories).

Allocation of school sample in Australia

Population counts Field trial Main survey
Expected number Expected number
State or territory Schools MOS Schools of sampled teachers* Schools of sampled teachers*
Australian Capital Territory 36 19 475 1 20 4 80
New South Wales 802 346 593 4 80 63 1 260
Northern Territory 44 7 552 1 20 4 80
Queensland 456 164 472 4 80 36 720
South Australia 223 58 138 2 40 18 360
Tasmania 93 26 969 1 20 7 140
Victoria 542 257 638 4 80 43 860
Western Australia 311 84 820 3 60 25 500
Total 2 507 965 657 20 400 200 4 000
* Calculated using an estimate of 20 selected teachers per school.

Data sources
The Australian Council for Educational Research (ACER) developed the school sampling frame by coordinating information
from multiple sources including the Australian Bureau of Statistics (ABS) and commonwealth, state and territory education
department databases, from the school year 2005.

© OECD 2010 TALIS 2008 Technical Report

Characteristics of National Samples Annex B
213

Austria

In Austria, ISCED Level 2 education covers Grades 5 to 8. There are two major tracks (AHS – Allgemeinbildende höhere
Schulen /Academic secondary school and HS – Hauptschulen/General secondary school) and another, smaller category of
privately organised schools. For sampling purposes, tracks within schools are considered as separate sampling units.

Sample design ISCED Level 2

School tracks Students

International target population 1 540 385 402
Out of scope - -
National target population 1 540 385 402
Exclusions - -
Survey population 1 540 385 402
Coverage rate 100% 100%

Sample allocation: Disproportional allocation to allow some data comparisons between the two larger strata.
Sample sizes: 20 school tracks selected for the field trial and 279 school tracks selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 students in school track.
Explicit stratification: Explicit stratification was organised by school type (the two school tracks AHS and HS and an
additional category “other”), for a total of three explicit strata.
Implicit stratification: Implicit stratification was organised by district categories, for a total of 96 implicit strata.

Allocation of school sample in Austria

Population counts Field trial Main survey
MOS Expected number Expected number
District Schools (ISCED Level 2 students) Schools of sampled teachers* Schools of sampled teachers*
AHS 269 116 691 5 100 109 2 180
HS 1 181 264 221 13 260 160 3 200
OTHER 90 4 490 2 40 10 200
Total 1 540 385 402 20 400 279 5 580
* Calculated using an estimate of 20 selected teachers per school.

Data sources
Statistics Austria created the school sampling frame with information from the official school database.

TALIS 2008 Technical Report © OECD 2010

214
Annex B Characteristics of National Samples

Belgium (Flanders)

In Belgium (Flanders), ISCED Level 2 education corresponds to the first stage of secondary education. It consists of the first
and the second year of secondary education. There are 789 schools offering ISCED Level 2 education.

Sample design ISCED Level 2

Schools Teachers
International target population 789 26 127
Out of scope
Schools exclusively devoted to special needs students 114 3 997
National target population 675 22 130
Exclusions - -
Survey population 675 22 130
Coverage rate 100% 100%

Sample allocation: Disproportional allocation to allow comparison between educational networks (3).
Sample sizes: 20 schools selected for the field trial and 260 schools selected for the main survey.
Method of sample selection: Probability proportional to size.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: Explicit stratification was organised by educational networks. There are three educational networks
in Flanders: community-run education subsidised publicly run education and subsidised privately
run education, for a total of three explicit strata.
Implicit stratification: No implicit variable used.

Allocation of school sample in Belgium (Flanders)

Population counts Field trial Main survey
Expected number Expected number
Educational networks Schools MOS Schools of sampled teachers* Schools of sampled teachers*
Subsidized privately run education 467 15 764 14 280 132 2 640
Community-run education 149 4 398 4 80 83 1 660
Subsidized publicly run education 59 1 968 2 40 45 900
Total 675 22 130 20 400 260 5 200
* Calculated using an estimate of 20 selected teachers per school.

Data sources
The school sampling frame was a combination of three Access data files from the Flemish Ministry of Education, including
schools, number of teachers (double counts) and number of teachers (without double counts) respectively, from school year
2005-06 (June 2006).

© OECD 2010 TALIS 2008 Technical Report

Characteristics of National Samples Annex B
215

Brazil

Target Population
In Brazil, elementary education includes both ISCED Level 1 and ISCED Level 2 programmes. The ISCED Level 2 regular
programme lasts either eight or nine years. It covers Grades 5 to 8 in elementary education, offering eight years of schooling,
and Grades 6 to 9 in elementary education, offering nine years of schooling.

Sample design ISCED Level 2

Schools Teachers
International target population 57 704 847 423
Out of scope
Schools exclusively devoted to special needs students 225 3 472
National target population 57 479 843 951
Exclusions
Schools with fewer than four ISCED Level 2 teachers 4 636 10 124
Federal schools 34 1 683
Survey population 52 809 832 144
Coverage rate 91.9% 98.6%

Sample allocation: Proportional allocation to the number of schools per type but not to the number of schools by
school size groups.
Sample sizes: 20 schools selected for the field trial and 400 schools selected for the main survey.
Method of sample selection: Probability proportional to size with equal probability sampling in small school strata.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: Explicit stratification was organised by school type (private, municipal or state) and school size
groups (4-19, 20- 29 or 30+), for a total of nine explicit strata.
Implicit stratification: Implicit stratification was organised by region (North, Northeast, Middle West, South, Southeast),
location (urban, rural) and State Federal District (27 districts).

Allocation of school sample in Brazil

Population counts Field trial Main survey
Expected number Expected number
Explicit strata (school size and type) Schools MOS Schools of sampled teachers* Schools of sampled teachers*
1.4-19 teachers, Private 8 923 95 795 1 11 52 558
1.4-19 teachers, Municipal 17 279 157 913 1 9 54 494
1.4-19 teachers, State 12 163 144 749 1 12 34 402
2.20-29 teachers, Private 890 20 604 2 49 46 1 053
2.20-29 teachers, Municipal 2 632 62 559 3 72 47 1 123
2.20-29 teachers, State 5 121 122 549 3 68 51 1 211
3.30+ teachers, Private 309 11 780 3 75 32 640
3.30+ teachers, Municipal 1 817 74 441 3 75 34 680
3.30+ teachers, State 3 675 141 754 3 75 50 1 000
Total 52 809 832 144 20 446 400 7 161
* Calculated using the average number of teachers in each strata and an estimate of 20 selected teachers for the “30+ teachers” strata.

Data sources
INEP/MEC (INEP – Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixiera, MEC – Ministry of Education)
developed the school sampling frame. The data were taken from the 2006 School Census on Basic Education, carried out by
INEP/MEC.

TALIS 2008 Technical Report © OECD 2010

216
Annex B Characteristics of National Samples

Bulgaria

Target Population
In Bulgaria, ISCED Level 2 education corresponds to the second stage of basic education and covers Grades 5 to 8. There are
2 520 schools offering ISCED Level 2 education.

Sample design ISCED Level 2

Schools Teachers
International target population 2 520 31 765
Out of scope
Schools exclusively devoted to special needs students 112 983
National target population 2 408 30 782
Exclusions - -
Small schools with fewer than 4 ISCED Level 2 teachers 108 282
Survey population 2 300 30 500
Coverage rate 95.5% 99%

Sample allocation: Allocation proportional to the number of schools per type but not to the number of schools by
school size groups.
Sample sizes: 20 schools selected for the field trial and 203 schools selected for the main survey.
Method of sample selection: Probability proportional to size with equal probability sampling in small school stratum.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: Explicit stratification was organised by school type (general, vocational or profiled) and school
size groups (4-9, 10-19, 20-29 or 30+), for a total of 12 explicit strata.
Implicit stratification: Implicit stratification was organised by location (capital, large city or other location).

Allocation of school sample in Bulgaria

Population counts Field trial Main survey
Expected number Expected number
Explicit strata (school size and type) Schools MOS Schools of sampled teachers* Schools of sampled teachers*
1. 4-9 teachers, General 1 137 6 822 1 6 16 98
2. 4-9 teachers, Vocational 58 348 1 8 4 33
3. 4-9 teachers, Profiled 21 126 1 6 4 26
4. 10-19 teachers, General 439 6 363 1 14 32 464
5. 10-19 teachers, Vocational 83 1 078 1 13 4 52
6. 10-19 teachers, Profiled 55 839 2 31 4 61
7. 20-29 teachers, General 273 6 502 5 119 56 1 334
8. 20-29 teachers, Vocational 15 358 1 24 5 119
9. 20-29 teachers, Profiled 10 246 0 0 10 246
10. 30+ teachers, General 188 6 964 6 150 56 1 400
11. 30+ teachers, Vocational 13 540 1 25 4 100
12. 30+ teachers, Profiled 8 314 0 0 8 200
Total 2 300 30 500 20 396 203 4 133
* Calculated using the average number of teachers as the number of teachers selected for the first 9 strata and a total of 25 teachers selected for “30+ teachers” strata.

Data sources
The National Center for Informatics Coverage of Education developed the school sampling frame. Data from the school year
2006-07 were used.

© OECD 2010 TALIS 2008 Technical Report

Characteristics of National Samples Annex B
217

Denmark

Target Population
In Denmark, education is compulsory for nine years: first to ninth form. Approximately 40% of pupils in the ninth form
continue in a voluntary one-year tenth form. ISCED Level 2 education covers the seventh to tenth forms.

Sample design ISCED Level 2

Schools Teachers 1
International target population 2 509 62 105
Out of scope
Schools exclusively devoted to special needs students 333 1 200 2
National target population 2 176 60 905
Exclusions
Schools with five or fewer teachers (either ISCED Levels 1 or 2) 70 274
No information available 29 N/A
Public Youth schools (Ungdomsskoler): Very special schools offering only a small proportion of activities at
ISCED Level 2 (activities like cooking) 111 300

Survey population 1 966 60 331

Coverage rate 90,3% <99,1%
1. Includes all teachers from Grade 1 to Grade 10. ISCED Level 2 is from Grade 7 to Grade 10.
2. Number of teachers is underestimated because it is known for only 91 out of 333 schools for students with special needs.

Sample allocation: Proportional to the number of schools per stratum.

Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 students in school.
Explicit stratification: Explicit stratification was organised by school type (Continuation, Public and Private Schools), for a
total of three explicit strata.
Implicit stratification: Implicit stratification was organised by geographical categories: city around Greater Copenhagen,
city outside Greater Copenhagen and country municipal, for a total of three implicit strata.

Allocation of school sample in Denmark

Population counts Field trial Main survey
Expected number Expected number
School type Schools MOS 1 Schools of sampled teachers* Schools of sampled teachers*
Continuation Schools (C) 205 20 966 2 40 21 420
Public schools (P) 1 346 174 297 14 280 137 2 740
Private Schools (R) 415 31 621 4 80 42 840
Total 1 966 226 884 20 400 200 4 000
* Calculated using an estimate of 20 selected teachers per school.
1. Includes all teachers from Grades 1 to 10. ISCED Level 2 is from Grade 7 to Grade 10.

Data sources
The information used to create of the school sampling frame came from UNI-C, a government institution under the Danish
Ministry of Education, which collects data from all schools at the beginning of each school year. The data provided refer to
the school year 2005-06.

TALIS 2008 Technical Report © OECD 2010

218
Annex B Characteristics of National Samples

Estonia

In Estonia, ISCED Level 2 education covers Grades 7 to 9.

Sample design ISCED Level 2

Schools Teachers
International target population 508 9 624
Out of scope
Schools devoted to students with special needs 44 719
Institutions providing adult education (adult gymnasiums) 16 158
National target population 448 8 747
Exclusions
Very small schools with fewer than 7 ISCED Level 2 teachers* 12 65
Remote schools 5 27
Bilingual schools (Estonian/Russian (15) Finnish/Estonian (1)) 16 410
Survey population 415 8 245
Coverage rate 92.6% 94.3%
* At the time of sample preparation, these schools were expected to be closed at the time of data collection.

Sample allocation: Proportional to the number of schools.

Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: Explicit stratification was organised by region (city and municipality) and by school type (schools
providing lower secondary education (põhikool) and schools providing lower and upper secondary
education (Gûmnaasium)), for a total of four explicit strata.
Implicit stratification: Implicit stratification was organised by socio-economic status (poor and rich), school size (fewer
than 120 pupils and 120 pupils or more) and language of instruction (Estonian and Russian).

Allocation of school sample in Estonia

Population counts Field trial Main survey
Expected number Expected number
Region by school type Schools MOS Schools of sampled teachers* Schools of sampled teachers*
Municipality – Põhikool 176 1 869 8 88 85 935
Municipality – Gûmnaasium 73 1 654 3 69 35 805
City – Põhikool 24 432 2 36 12 216
City –Gûmnaasium 142 4 290 7 140 68 1 360
Total 415 8 245 20 333 200 3 316
* Calculated using the average number of teachers as the number of teachers selected for the first three strata and an average of 20 teachers per selected school for
the last stratum.

Data sources
The Estonian Educational Information System (EEJS / EHIS) collected information for the school sampling frame for the school
year 2005-06.

© OECD 2010 TALIS 2008 Technical Report

Characteristics of National Samples Annex B
219

Hungary

In Hungary, ISCED Level 2 education corresponds to Grades 5 to 8. It is offered in both primary and secondary schools.
Primary schools offer education from Grade 1 to Grade 8. Some secondary schools cover Grade 5 to Grade 12 while others
cover only Grade 7 to Grade 12. Some schools also offer both primary and secondary schooling.

Sample design ISCED Level 2

Schools Teachers*
International target population 3 078 49 045
Out of scope
Schools devoted to students with special needs 174 2 436
Schools devoted to adult education 7 15
National target population 2 897 46 594
Exclusions
Very small schools with fewer than four ISCED Level 2 teachers 45 103
Survey population 2 852 46 491
Coverage rate 98.4% 99.8%
* Estimated from the number of ISCED Level 1 and ISCED Level 2 classes and the number of teaching staff.

Sample allocation: Disproportional.

Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: Explicit stratification was organised by school size groups (4-9, 10-19, 20-29, 30 or more).
Implicit stratification: Implicit stratification was organised by type of school (primary, secondary and mixed), type of
settlement (capital, town and village) and school ownership (public/state, church and private).

Allocation of school sample in Hungary

Population counts Field trial 2 Main survey
Expected number Expected number
Schools MOS 1 Schools of sampled teachers* Schools of sampled teachers 3
4 to 9 teachers 831 6 136 16 118
10 to 19 teachers 1 173 16 288 70 969
20 400
20 to 29 teachers 591 13 941 70 1650
30 or more teachers 257 10 126 44 880
Total 2 852 46 491 20 400 200 3618
* Calculated using an estimate of 20 selected teachers per school.
1. Estimated from the number of ISCED Level 1 and ISCED Level 2 classes and the number of teaching staff.
2. No explicit stratification done for the Field Test.
3. Calculated using the average number of teachers as the number of teachers selected for the first three strata and an average of 20 teachers per selected school for
the last stratum.

Data sources
The school sampling frame comes from the Annual Statistical Database of the Ministry of Education, for school year 2007-08
(for the field trial, the frame used was based on data from the school year 2004-05).

220
Annex B Characteristics of National Samples

Iceland

The Icelandic education system covers Grades 1 to 10. ISCED Level 1 corresponds to Grades 1 to 7 (typically starting at 6
years of age) while ISCED Level 2 covers Grades 8 to 10 (typically starting at 13 years of age).
Iceland chose to survey all ISCED Level 2 teachers. As a national option, all ISCED Level 1 teachers in Iceland were also
surveyed.

Sample design ISCED Level 2

School tracks Teachers

International target population 146 2 315
Out of scope: Special needs 2 12
National target population 144 2 303
Exclusions - -
Survey population 144 2 303
Coverage rate 100% 100%

Sample allocation: Not applicable.

Sample sizes: The 10 largest schools were selected for the field trial and all schools were selected for the main
survey.
Measure of size (MOS): Not applicable.
Explicit stratification: Not applicable.
Implicit stratification: Not applicable.

Allocation of school sample in Iceland

Population counts Field trial Main survey
MOS Expected number Expected number
District Schools (ISCED Level 2 teachers) Schools of sampled teachers* Schools of sampled teachers*
All 144 2 303 10 200 144 2 303
Total 144 2 303 10 200 144 2 303
* All teachers are selected in the sample.

Data sources
The Icelandic statistical bureau provided the information for the creation of the school frame. A centralized database with
information on both teachers and students was used to create the school frame.

Characteristics of National Samples Annex B
221

Ireland

In Ireland, ISCED Level 2 education covers the first, second and third years of post-primary education. It corresponds to the
seventh, eighth and ninth years of education.

Sample design ISCED Level 2

School tracks Students 1

International target population 702 170 387
Out of scope - -
National target population 702 170 387
Exclusions - -
Survey population 702 170 387
Coverage rate 100% 100%
1. Country was able to provide only counts of ISCED Level 2 students.

Sample allocation: Proportional to the number of schools by school size group.

Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 students in school.
Explicit stratification: Explicit stratification variable was organised by school size groups (small, medium and large)
according to the number of ISCED Level 2 students in the school.
Implicit stratification: Implicit stratification variable was organised by school type (secondary, vocational and community/
comprehensive), student gender (boys, girls and mixed) and school socio-economic status (schools
classified as socio-economically disadvantaged and other schools that do not meet this criteria).

Allocation of school sample in Ireland

Population counts Field trial Main survey
Expected number Expected number
School size Schools MOS 1 Schools of sampled teachers* Schools of sampled teachers*
Small – Up to 120 students 127 9 776 4 80 36 720
Medium - 121 to 240 students 246 44 954 7 140 70 1 400
Large – 240 students or more 329 115 657 9 180 94 1 880
Total 702 170 387 20 400 200 4 000
* Calculated using an estimate of 20 teachers selected per school.
1. Counts of ISCED Level 2 students.

Data sources
The school sampling frame comes from the Irish Department of Education and Science Database of Post-Primary Schools, for
the school year 2005-06.

222
Annex B Characteristics of National Samples

Italy

In Italy, ISCED Level 2 education corresponds to lower secondary schooling and lasts three years. There are state and non-
state ISCED Level 2 schools, consisting of public schools managed by Central Government, public schools managed by Local
Government (in two regions only) and private schools.

Sample design ISCED Level 2

Schools Teachers
International target population 7 941 191 725
Out of scope
Schools exclusively devoted to special needs students 43 256
Laboratory schools with special ordinance (different from other regular schools) 4 123
National target population 7 894 191 346
Exclusions
Schools with fewer than 4 ISCED Level 2 teachers 262 604
Private schools outside the national education system 21 244
Schools attached to art academies 71 919
Remote schools 31 233
Survey population 7 509 189 346
Coverage rate 95.1% 99%

Sample allocation: Fixed allocation of 100 schools per geographical region (North Italy, Central Italy and combined
South and Insular Italy) to allow comparison between the three geographical areas.
Sample sizes: 20 schools selected for the field trial and 300 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: The explicit stratification variable was organised by geographical region as described above.
Implicit stratification: The implicit stratification variable was organised by school outcomes measured as the ratio
of students who passed with a sufficient evaluation over the total of students who passed (five
categories).

Allocation of School Sample in Italy

Population counts Field trial Main survey
Expected number Expected number
Geography Schools MOS Schools of sampled teachers* Schools of sampled teachers*
North Italy 3 200 75 293 8 160 100 2 000
South and Insular Italy 3 036 82 067 4 80 100 2 000
Central Italy 1 273 31 986 8 160 100 2 000
Total 7 509 189 346 20 400 300 6 000

* Calculated using an estimate of 20 selected teachers per school.

Data sources
The school sampling frame was created by combining data from different sources: administrative data owned by the Ministry
of Education (for public schools managed by the central government), data from the census survey managed by the Ministry
of Education’s Statistics Office (for public schools managed by the local government and other administrative data for private
schools.

Characteristics of National Samples Annex B
223

Korea

In Korea, ISCED Level 2 education is offered in middle schools (also called junior high school) and covers Grade 7 to Grade 9.

Sample design ISCED Level 2

Schools Teachers
International target population 2 987 103 877
Out of scope - -
National target population 2 987 103 877
Exclusions - -
Survey population 2 987 103 877
Coverage rate 100% 100%

Sample allocation: Not applicable.

Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: None.
Implicit stratification: Implicit stratification variable was organised by region (16) and by type of funding (private, public
and national).

Allocation of School Sample in Korea

Population counts Field trial Main survey
Expected number Expected number
Geography Schools MOS Schools of sampled teachers* Schools of sampled teachers*
All 2 987 103 877 20 400 200 4 000
Total 2 987 103 877 20 400 200 4 000
* Calculated using an estimate of 20 selected teachers per school.

Data sources
The Center for Education Statistics and Information (Korean Educational Development Institution) provided the school frame
as designated by Ministry of Education and Human Resource Development.

224
Annex B Characteristics of National Samples

Lithuania

In Lithuania, ISCED Level 2 education corresponds to Grades 5 to 10. It is offered in basic schools (from Grade 1 to Grade 10,
covering ISCED Levels 1 and 2), secondary schools (from Grade 1 to Grade 12, covering ISCED Levels 1 to 3), Gymnasium
(from Grade 9 to Grade 12, covering ISCED Levels 2 and 3) and vocational schools (also covering ISCED Levels 2 and 3).

Sample design ISCED Level 2

Schools Teachers*
International target population 1 361 49 154
Out of scope
Schools exclusively devoted to special needs students 65 1 772
National target population 1 296 47 382
Survey population 1 296 47 382
Coverage rate 95.2% 96.4%
*Includes the total number of teachers in the school.

Sample allocation: Proportional to the number of schools in each stratum.

Sample sizes: 20 schools selected for the field trial and 220 schools selected for the main survey.
Measure of size (MOS): Total number of teachers in the school.
Explicit stratification: Explicit stratification was organised by school type in four categories (basic, secondary, gymnasium
and vocational).
Implicit stratification: Implicit stratification was organised by urbanisation (city, town or village) and type of funding (public
and private).

Allocation of school sample in Lithuania

Population counts Field trial Main survey
Expected number Expected number
School type Schools MOS Schools of sampled teachers* Schools of sampled teachers*
Basic 630 12 998 9 180 107 2 140
Secondary 128 7 220 2 40 22 440
Gymnasium 465 22 564 7 140 79 1 580
Vocational 73 4 600 2 40 12 240
Total 1 296 47 382 20 400 220 4 400
* Calculated using an estimate of 20 selected teachers per school.

Data sources
The information used to create the school sampling frame came from the database of schools from the Ministry of Education
and Science of Lithuania from the school year 2005-06.

Characteristics of National Samples Annex B
225

Malaysia

In Malaysia, ISCED Level 2 education covers Forms 1, 2 and 3, which are equivalent to Grades 7, 8 and 9 respectively. It is
offered in government schools (MOE), MARA schools and religious schools (SMAR/SMAN).

Sample design ISCED Level 2

Schools Students 1
International target population 2 363 1 408 601 2
Out of scope
Schools exclusively devoted to special needs students 2 175
National target population 2 361 1 408 426 2
Exclusions
Schools with different curriculum (expatriate schools, international schools and Chinese private schools) 106 N/A
Very small schools (fewer than 100 enrolled ISCED Level 2 students) 109 6,554
Remote schools 2 591
Survey population 2 144 1 401 281
Coverage rate 90.8% <99.5%*
1. Country was able to provide counts of ISCED Level 2 students only.
2. Does not account for the unavailable figures.

Sample allocation: Proportional to the number of schools, with the exception of the smaller stratum in which all schools
were selected.
Sample sizes: 20 schools selected for the field trial and 219 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 students in school.
Explicit stratification: Explicit stratification variable was organised by school type (government schools [MOE], MARA
schools and religious schools [SMAR/SMAN]), for a total of 3 explicit strata.
Implicit stratification: Implicit stratification variables was organised by State (14) and location (urban and rural).

Allocation of school sample in Malaysia

Population counts Field trial Main survey
Expected number Expected number
Type of school Schools MOS 1 Schools of sampled teachers* Schools of sampled teachers*
Government schools (MOE) 1 945 1 336 257 17 340 183 3 660
MARA schools 20 8 608 1 20 19 380
Religious schools (SMAR/SMAN) 179 56 416 2 40 17 340
Total 2 144 1 401 281 20 400 219 4 380
* Calculated using an estimate of 20 selected teachers per school.
1. Counts of ISCED Level 2 students.

Data sources
The information used to create the school sampling frame came from the Ministry of Education (MOE) for government
schools, MARA Junior Science Colleges for MARA schools and from the EPDR Database for the Religious Schools, for the
school year 2006 (January).

226
Annex B Characteristics of National Samples

Malta

Target Population
In Malta, ISCED Level 2 lasts five years, from Form I to Form V. Children are typically 11 years of age when they start ISCED
Level 2 education. Sixty-four schools offer ISCED Level 2 education.

Sample design ISCED Level 2

Schools Teachers
International target population 64 3 013
Out of scope - -
National target population 64 3 013
Exclusions
Schools not following the mainstream curriculum 3 88
Very small schools 2 >1
International Baccalaureate 1 >1
Survey population 58 <2 924
Coverage rate 93,7% <97%

Sample allocation: Not applicable.

Sample sizes: Four largest schools selected for the field trial and all schools selected for the main survey.
Measure of size (MOS): Not applicable.
Explicit stratification: Not applicable.
Implicit stratification: Not applicable.

Allocation of school sample in Malta

Population counts Field trial Main survey
Expected number Expected number
Malta Schools MOS Schools of sampled teachers* Schools of sampled teachers*
All 58 2 924 4 80 58 1 160
Total 58 2 924 4 80 58 1 160
* Calculated using an average of 20 teachers per school.

Data sources
The Ministry of Education, Youth and Employment - Education Division provided information from the school year 2007-08
for the creation of the school sampling frame.

Characteristics of National Samples Annex B
227

Mexico

In Mexico, ISCED Level 2 is compulsory and corresponds to lower secondary education. It comprises three grades for the
Secondary Certificate (Year 7 to Year 9) or four grades for the Job Training Certificate (Year 7 to Year 10). ISCED Level 2
education is offered in regular (General, particular and Técnica) schools and in Telesecundaria schools.
Following discussions held among OECD, Mexico and Statistics Canada, it was decided that teachers in Telesecundaria
schools did not meet the TALIS definition of an ISCED Level 2 teacher. These schools and their teachers were classified as out
of the scope for TALIS, but were part of a national option for Mexico.

Sample design ISCED Level 2

Schools Teachers
International target population 32 079 351 454
Out of scope
Telesecundaria schools 1 16 529 60 493
Secondary schools (Secundaria) for workers 330 4 582
National target population 15 220 286 379
Exclusions
Very small schools (fewer than 4 teachers) 87 159
Schools selected for the Field Test 23 758
CONAFE 918 1 050
Others 8 506
Survey population 14 184 283 906
Coverage rate 93.2% 99.1%
1. Distance learning schools.

Sample allocation: Proportional to the number of schools, then adjusted to have a minimum of four schools selected in
the smaller strata.
Sample sizes: 24 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of the number of teachers in the school.
Explicit stratification: Explicit stratification was organised by school size group based on the number of teachers in the
school (4 to 9, 10 to 19, 20 to 29 and 30 or more) and by school type (General, Particular and
Técnica), for a total of 12 explicit strata.
Implicit stratification: Implicit stratification was organised by state, for a total of 32 implicit strata.

Allocation of school sample in Mexico

Population counts Field trial 1 Main survey
Schools MOS Schools Expected sample size Schools Expected sample size*
1. 4-9 teachers – General 816 5 741 n/a n/a 5 35
2. 4-9 teachers – Particular 813 6 530 n/a n/a 4 32
3. 4-9 teachers – Técnica 479 3 618 n/a n/a 4 30
4. 10-19 teachers – General 2 044 29 271 n/a n/a 24 344
5. 10-19 teachers – Particular 2 422 31 747 n/a n/a 11 144
6. 10-19 teachers – Técnica 1 490 21 298 n/a n/a 15 214
7. 20-29 teachers – General 1 839 44 970 n/a n/a 39 954
8. 20-29 teachers – Particular 339 7 824 n/a n/a 13 300
9. 20-29 teachers – Técnica 1 248 30 325 n/a n/a 20 486
10. 30+ teachers – General 1 877 72 749 n/a n/a 37 925
11. 30+ teachers – Particular 75 2 917 n/a n/a 9 225
12. 30+ teachers – Técnica 742 26 916 n/a n/a 19 475
Total 14 184 283 906 24 480 200 4 164
* Calculated using the average number of teachers per school in the first 9 strata and an estimate of 25 selected teachers in “30+ teachers” strata.
n/a = not applicable
1. A convenience sample was selected by Mexico for the Field trial.

Data sources
The government’s annual Census of Schools was the source of information for the school frame.

228
Annex B Characteristics of National Samples

Norway

In Norway, ISCED Level 2 corresponds to lower secondary education and covers Grades 8 to 10.

Sample design ISCED Level 2

Schools Teachers
International target population 1 271 22 337
Out of scope
Schools exclusively devoted to special needs students 59 439
National target population 1 212 21 898
Exclusions
Small schools (fewer than ten students or fewer than three ISCED Level 2 teachers 104 211
Schools outside Norwegian school regulation 4 69
Schools abroad 14 48
Survey population 1 090 21 570
Coverage rate 89.9% 98.5%

Sample allocation: Proportional to the number of school per type but not to the number of schools by school size group.
Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey. All teachers in
the selected schools will be surveyed to compensate for the large proportion of very small schools.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: Explicit stratification was organised by school size groups crossed by urban/rural categories, for a
total of eight explicit strata (see below).
Implicit stratification: Implicit stratification was organised by 20 county categories.

Allocation of school sample in Norway

Population counts Field trial Main survey
Expected number Expected number
School size by urban/rural Schools MOS Schools of sampled teachers* Schools of sampled teachers*
3-9 teachers – Urban 61 364 1 20 9 54
3-9 teachers – Rural 236 1 372 4 80 38 221
10-19 teachers – Urban 82 1 202 1 20 12 176
10-19 teachers – Rural 220 3 099 3 60 31 437
20-29 teachers – Urban 83 2 064 1 20 15 733
20-29 teachers – Rural 151 3 628 3 60 27 648
30-62 teachers – Urban 133 5 170 4 80 37 1 438
30-62 teachers – Rural 124 4 671 3 60 31 1 168
Total 1 090 21 570 20 400 200 4 875
* Calculated using the average number of teachers per strata.

Data sources
The information used to create of the school sampling frame came from the Primary and Lower Secondary School Information
System (GSI / Grunnskolens informasjons system) for the school year 2006-07.

Characteristics of National Samples Annex B
229

Poland

Target Population
In Poland, ISCED Level 2 education is referred to as gimnazjum (junior high and lower secondary). It covers three grades (first,
second and third) which generally correspond to the seventh, eighth, and ninth years in school.

Sample design ISCED Level 2

Schools Teachers 1
International target population 7 156 146 730
Out of scope
Schools exclusively devoted to special needs students 816 7 025
Adult education schools 122 415
National target population 6 218 139 290
Exclusions
Very small schools (fewer than three ISCED Level 2 teachers) 908 1 816 2
Survey population 5 310 137 474
Coverage rate 85.4% 98.7% 1
1. Poland was able to provide aggregated counts of teachers but did not have that information at the school level, which explains why the measure of size is based
on student counts.
2. Estimated due to some missing information about the number of ISCED Level 2 teachers in excluded schools. An average of two ISCED Level 2 teachers per school
was used to estimate the number of excluded ISCED Level 2 teachers.

Sample allocation: Proportional to the number of schools per strata, then adjusted to have a minimum of four schools
selected in the smaller strata.
Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 students in school.
Explicit stratification: Explicit stratification was organised by location type (rural, urban, rural-urban) and by type of
funding (public and private), for a total of six explicit strata.
Implicit stratification: Implicit stratification was organised by region (16 categories).

Allocation of school sample in Poland

Population counts Field trial Main survey
Expected number Expected number
Type of school Schools MOS(1) Schools of sampled teachers* Schools of sampled teachers*
Rural – Public 2 302 415 775 9 180 84 1 680
Rural – Private 28 1 802 0 0 4 80
Urban – Public 1 479 572 387 5 100 55 1 100
Urban – Private 294 18 062 1 20 11 220
Urban – Rural – Public 1 158 331 359 4 80 42 840
Urban – Rural – Private 49 3 519 1 20 4 80
Total 5 310 1 342 904 20 400 200 4 000
* Calculated using an estimate of 20 selected teachers per school.
1. Count of ISCED Level 2 students in school.

Data sources
The information used to create the school sampling frame came from the the Ministry of National Education’s System of
Information on Education (System Informacji O wiatowej) for the school year 2006-07.

230
Annex B Characteristics of National Samples

Portugal

In Portugal compulsory education includes ISCED Levels 1 and 2. ISCED Level 1 has two cycles: the first cycle lasts four years
and the second cycle, two years. ISCED Level 2 covers three years of schooling.

Sample design ISCED Level 2

Schools Teachers
International target population 1 310 41 807*
Out of scope
Schools exclusively devoted to special needs students 3 N/A
National target population 1 307 41 807
Exclusions - -
Survey population 1 307 41 807
Coverage rate 100% 100%
* Does not account for the unavailable figures.

Sample allocation: Proportional to the number of schools, then adjusted to select all schools in two very small strata.
Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit Stratification: Explicit stratification was organised by type of funding (private and public) and by region (Alentejo,
Algarve, Centro, Lisboa, Norte), for a total of 10 explicit strata.
Implicit Stratification: Implicit stratification was organised by school size (fewer than 31 teachers, 31 to 44 teachers, 45 or
more teachers).

Allocation of school sample in Portugal

Population counts Field trial Main survey
Expected number Expected number
Explicit strata (school type and region) Schools MOS Schools of sampled teachers* Schools of sampled teachers*
1 Publico – Alentejo 118 3 105 3 60 18 360
2 Publico – Algarve 56 1 641 2 40 9 180
3 Publico – Centro 289 9 140 3 60 41 820
4 Publico – Lisboa 242 9 320 3 60 35 700
5 Publico – Norte 391 14 455 3 60 55 1100
6 Privado – Alentejo 9 167 0 0 9 180
7 Privado – Algarve 3 43 0 0 3 60
8 Privado – Centro 54 1 361 2 40 8 160
9 Privado – Lisboa 74 1 187 2 40 11 220
10 Privado – Norte 71 1 388 2 40 11 220
Total 1 307 41 807 20 400 200 4 000
* Calculated using an estimate of 20 selected teachers per school.

Data sources
The information used to create the school sampling frame is taken from the Annual School Census, for the school year 2006-07.

Characteristics of National Samples Annex B
231

Slovak Republic

In the Slovak Republic, ISCED Level 2 education is offered in elementary and grammar schools. It covers Grades 5 to 9 in
elementary schools and Grades 1 to 4 in grammar schools.

Sample design ISCED Level 2

Schools Teachers
International target population 1 863 30 724
Out of scope
Schools exclusively devoted to special needs students 208 2 542
National target population 1 655 28 182
Exclusions
Language other than Slovak or Hungarian 14 132
Very small schools (fewer than 100 enrolled ISCED Level 2 students) 21 40
Survey population 1 620 28 010
Coverage rate 97.9% 99.4%

Sample allocation: Disproportional allocation to allow some comparison between the elementary and grammar schools.
Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: Explicit stratification was organised by school type (elementary and grammar).
Implicit stratification: Implicit stratification was organised by language of instruction (Slovak and Hungarian), region (8
categories) and school size (small, medium and large).

Allocation of school sample in the Slovak Republic

Population counts Field trial Main survey
Expected number Expected number
School type Schools MOS Schools of sampled teachers* Schools of sampled teachers*
Elementary Schools 1 455 21 940 18 360 130 2 600
Grammar Schools 165 6 070 2 40 70 1 400
Total 1 620 28 010 20 400 200 4 000

* Calculated using an average of 20 teachers per school.

Data sources
The information used to create the school sampling frame was drawn from the UIPS schools databases for the school year
2005-06.

232
Annex B Characteristics of National Samples

Slovenia

Target Population
In Slovenia, ISCED Level 2 corresponds to Grades 7 to 9.

Sample design ISCED Level 2

Schools Teachers
International target population 478 9 722
Out of scope
Schools exclusively devoted to special needs students 32 272
National target population 446 9 450
Exclusions - -
Survey population 446 9 450
Coverage rate 100% 100%

Allocation: Not applicable.

Allocation of school sample in Slovenia

Population counts Field trial Main survey
Expected number Expected number
Slovenia Schools MOS Schools of sampled teachers* Schools of sampled teachers*
All 446 9 450 20 400 200 4 000
Total 446 9 450 20 400 200 4 000

* Calculated using an estimate of 20 selected teachers per school.

Data sources
The Slovene Ministry of Education and Sports provided the information used to createthe school sampling frame. It corresponds
to the school year 2005-06.

Characteristics of National Samples Annex B
233

Spain (Excluding La Rioja, Canarias)

In Spain, the education system is under the responsibility of 18 autonomous communities. The national target population in
Spain covers 16 communities: the Rioja and Canary Islands communities did not take part in the TALIS Survey.
ISCED Level 2 education consists of four grades, from First Compulsory Secondary to Fourth Compulsory Secondary. There
are 7 106 schools offering ISCED Level 2 education in the covered autonomous communities.

Sample design ISCED Level 2 1

Schools Teachers 2
International target population 7 106 235 060
Out of scope - -
National target population 7 106 235 060
Exclusions - -
Survey population 7 106 235 060
Coverage rate 100% 100%
1. All counts exclude La Rioja and Canarias communities, which are not covered by the national target population.
2. The reported number of teachers covers both ISCED Level 2 and 3 teachers. NPM reports that 80% of those teachers are ISCED Level 2.

Allocation: Proportional to the number of schools in each stratum for the main survey.
Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Levels 2 and 3 teachers in school. Approximately 80% of these teachers teach at
ISCED Level 2.
Explicit stratification: Since the field trial was done in only two autonomous communities, the explicit stratification was
organised in two groups: field trial communities and all other communities.
Implicit stratification: Implicit stratification was organised by autonomous communities (16), school type (public and
private) and school size groups (less than 18, 18 to 41, and 42 or more ISCED Level 2 and 3 teachers.

Allocation of school sample in Spain

Population counts Field trial Main survey
Expected number Expected number
District Schools MOS 1 Schools of sampled teachers* Schools of sampled teachers*
Field Test communities 846 27 687 20 400 24 480
All other communities 6 260 207 373 0 0 176 3 520
Total 7 106 235 060 20 400 200 4 000
* Calculated using an estimate of 20 selected teachers per school.
1. Counts of ISCED Level 2 and 3 teachers.

Data sources
The school sampling frame was created using data provided by the 16 autonomous communities.

234
Annex B Characteristics of National Samples

The Netherlands

In the Netherlands, ISCED Level 2 consists of lower secondary education (the first three years of pre-university education (six
years total), the first three years of senior secondary education (five years total) and all four years of pre-vocational secondary
education.

Sample design ISCED Level 2

Schools Students 1
International target population 697 722 122
Out of scope
Schools exclusively devoted to special needs students 110 27 212
National target population 587 694 910
Exclusions
Schools for vocational training almost exclusively at ISCED Level 3 (limited education at ISCED Level 2
42 23 900 1
representing 5% of total enrolment in these schools)
Survey population 545 671 010
Coverage rate 92.8% 96.6%*
1. Country was able to provide counts of ISCED Level 2 students only.
2. Estimated.

Sample allocation: Proportional to the number of schools in each stratum.

Sample sizes: Ten schools selected for the field trial and 150 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 students in school.
Explicit stratification: Explicit stratification was organised by education stream categories (mainly vocational with
examination at ISCED Level 2 only, academic with examination at ISCED Level 3 only, academic
with examination at ISCED Levels 2 and 3, academic and vocational with examination at ISCED
Levels 2 and 3), for a total of four explicit strata.
Implicit stratification: None.

Allocation of school sample in the Netherlands

Population counts Field trial Main survey
Expected number Expected number
Education stream Schools MOS 1 Schools of sampled teachers* Schools of sampled teachers*
Mainly vocational / examination only
75 45 621 2 40 21 420
at ISCED Level 2
Academic / examination only at ISCED Level 3 79 38 599 2 40 22 440
Academic / examination at ISCED Levels 2 and 3 107 90 454 2 40 30 600
Academic as well as vocational / examination at
284 496 336 4 80 77 1 540
ISCED Levels 2 and 3
Total 545 671 010 10 200 150 3 000
* Calculated using an estimate of 20 selected teachers per school.
1. Counts of ISCED Level 2 students.

Data sources
The Ministry of Education provided the information used for the school sampling frame from the school year 2005-06.

Characteristics of National Samples Annex B
235

Turkey

In Turkey, ISCED Level 2 education covers Grades 6 to 8. Education at that level is compulsory. Schools can offers different
combinations of ISCED levels. For example, some schools offer primary and lower secondary education only. Others offer
ISCED Levels 1 to 3 or ISCED Levels 2 and 3 only (lower and upper secondary).

Sample design ISCED Level 2

Schools Teachers
International target population 16 626 161 552
Out of scope
Schools exclusively devoted to special needs students 311 3 917
National target population 16 315 157 635
Exclusions
Very small schools (fewer than 4 ISCED Level 2 teachers) 3 838 8 648
Survey population 12 477 148 987
Coverage rate 76 5% 94.5%

Sample allocation: Disproportional.

Sample sizes: 20 schools selected for the field trial and 200 schools selected for the main survey.
Measure of size (MOS): Count of ISCED Level 2 teachers in school.
Explicit stratification: Explicit stratification was organised by school size group defined according to the number of ISCED
Level 2 teachers in the school (4 to 9, 10 to 19, 20 to 29 and 30 or more).
Implicit stratification: Implicit stratification was organised by region (12 regions based on the socio-economic development
index) and type of funding (private and public).

Allocation of school sample in Turkey

Population counts Field trial 1 Main survey
Expected number Expected number
Explicit stratification (school size) Schools MOS Schools of sampled teachers* Schools of sampled teachers*
4 to 9 teachers 6 899 41 373 5 30 20 120
10 to 19 teachers 3 850 51 678 10 134 35 470
20 to 29 teachers 937 22 060 1 24 75 1 766
30 or more teachers 791 33 876 4 100 70 1 750
Total 12 477 148 987 20 288 200 4 105
* Calculated using an average number of teachers in the three smaller strata and an estimate of 25 selected teachers in the last category.
1. The field trial sample was selected using region as the explicit strata. The strategy for the main survey sample was changed to take into account the large proportion
of small schools in Turkey.

Data sources
The source of information used to create the school sampling frame was the ILSIS DATABASE from the school year 2006-07.

237

Annex C
Sampling Forms

238
Annex C Sampling Forms

Sampling Form 1 Participation

TALIS 2008 participant:

National project manager:

1. Please specify the usual start date of the school year, the expected date of surveying for the Main Survey.
Start of school year: Survey Administration period: End of school year:

2. Describe the grade structure through ISCED Level 2

3. Has your country chosen to survey teachers of 15-year-old students in schools selected for PISA 2006?
(Please enter ‘X’ in the appropriate box)
Yes No

4. Does your country/institution have some experience with other, similar international surveys
(e.g. PISA, TIMSS, SITES, PIRLS)? (Please enter “X” in the appropriate box)
Yes No

5. Indicate the language(s) in which the survey will be administered.

6. Would you be interested in using online data collection? (Please enter “X” in the appropriate box)
Yes No

7. Do you plan to outsource all or parts of survey operations to a third party?

(Please enter “X” in the appropriate box)
Yes No

If yes, which parts :

Sampling Forms Annex C
239

Sampling Form 2 National Target Population

TALIS 2008 participant:

National project manager:

1. Total number of schools and ISCED Level 2

Number Number of teachers
teachers in the target population:
of schools at ISCED Level 2
[a]

School-level exclusions
2. Describe the reasons for school exclusion from the national target population (if applicable).

Number Number of teachers

Reason for exclusion of schools at ISCED Level 2

TOTAL

Number Number of teachers

of schools at ISCED Level 2
3. Total number of schools and ISCED Level 2 teachers [b]
excluded from the national target population:

4. Total number of schools and ISCED Level 2 teachers in [c]

the national target population:
(box [a] – box [b])
5. Percentage of coverage of the national target population [d]
in terms of number of schools of ISCED Level 2 teachers:
(box [c] ÷ box [a])
6. Describe your data sources (provide copies of relevant tables).

240
Annex C Sampling Forms

Sampling Form 3 Stratification

TALIS 2008 participant:

National project manager:

Explicit stratification of schools

1. List and describe the variables used for explicit stratification:
Explicit stratification variables
Name Description Number of levels
1

2

3

4

5

2. Total number of explicit strata:

Implicit stratification of schools

3. List and describe the variables used for implicit stratification:
(Please list variables in the order you want them to be used)
Implicit stratification variables
Name Description Number of levels
1

2

3

4

5

4. Total number of implicit strata:

Sampling Forms Annex C
241

Sampling Form 4 Sampling Frame Description

TALIS 2008 participant:

National project manager:

1. Specify the school measure of size (MOS) to be used. (Please enter “X” in the appropriate box)

Number of ISCED Level 2 teachers in school

Number of ISCED Level 2 students in school

Total number of teachers in school

Total number of students in school

Other (please describe)

2. Specify the school year for which employment data will be used for the school MOS.

3. Describe the source of information used in creating the school sampling frame.

4. Define the units used in the sampling frame (i.e. whole schools, shifts, tracks, programmes, etc.).

5. If your country also surveys teachers of 15-year-olds in schools selected for PISA 2006, please define the units used in
the PISA 2006 sampling frame (i.e. whole schools, shifts, tracks, programmes, etc.).

242
Annex C Sampling Forms

Sampling Form 5 Excluded schools

TALIS 2008 participant:

National project manager:

School ID Reason for exclusion School MOS

page of (Use additional sheets if necessary)

Sampling Forms Annex C
243

Sampling Form 6 Population counts by Strata

TALIS 2008 participant: Target population

National project manager: ISCED: _______________

Population counts
Explicit strata Implicit strata Schools MOS
1 2 3 4

page of (Use additional sheets if necessary)

244
Annex C Sampling Forms

Sampling Form 7 Sample allocation

TALIS 2008 participant: Target population

National project manager: ISCED: ______________

Sample Allocation
Main Survey Field Trial
Explicit Strata MCS Schools MOS Schools MOS
1 2 3 4 5 6

Country Total:

page of (Use additional sheets if necessary)

245

Annex D
TALIS Principal and
Teacher Questionnaire

246
Annex D TALIS Principal and Teacher Questionnaire

[Placeholder
for identification label]
(105 x 35 mm)

OECD Teaching and Learning International Survey (TALIS)

Principal Questionnaire
Main study version (MS-11-01)
[International English, UK Spelling]
[National Project Information]

International project consortium:

International Association for the Evaluation of Educational Achievement (IEA), The Netherlands
IEA Data Processing and Research Center (IEA DPC), Germany
Statistics Canada, Canada

About TALIS
The first Teaching and Learning International Survey (TALIS) is an international survey that offers the opportunity for teachers
and principals to provide input into education analysis and policy development. TALIS is being conducted by the Organisation
for Economic Co-operation and Development (OECD) and [Name of country], along with some 23 other countries, is taking
part in the survey.
Cross-country analysis of this data will allow countries to identify other countries facing similar challenges and to learn
from other policy approaches. School principals and teachers will provide information about issues such as the professional
development they have received; their teaching beliefs and practices; the review of teachers’ work and the feedback and
recognition they receive about their work; and various other school leadership, management and workplace issues.
Being an international survey, it is possible that some questions do not fit very well within your national context. In these
cases, please answer as best as you can.

Confidentiality
All information that is collected in this study will be treated confidentially. While results will be made available by country and
by type of school within a country, you are guaranteed that neither you, this school nor any of its personnel will be identified
in any report of the results of the study. [Participation in this survey is voluntary and any individual may withdraw at any time.]

About the questionnaire

• This questionnaire asks for information about school education and policy matters.
• The person who completes this questionnaire should be the principal of this school. If you do not have the information to
answer particular questions, please consult other persons in this school.
• This questionnaire should take approximately 45 minutes to complete.
• When questions refer to “this school” we mean by “school”: national school definition.
• Guidelines for answering the questions are typed in italics. Most questions can be answered by marking the one most
appropriate answer.
• When you have completed this questionnaire, please [National Return Procedures and Date].
• When in doubt about any aspect of the questionnaire, or if you would like more information about it or the study, you can
reach us by phone at the following numbers: [National Center Contact Information]

Thank you very much for your co-operation!

TALIS Principal and Teacher Questionnaire Annex D
247

Background information
These questions are about you, your education and your position as school principal. In responding to the questions, please
mark the appropriate box.

1 What is your gender?

Female Male

1 2

2 How old are you?

Under 40 40-49 50-59 60+

1 2 3 4

3 Do you have principal responsibilities for more than one school?

Yes No

1 2

4 What is the highest level of formal education you have completed?

Please mark one choice.

1
<Below ISCED Level 5>

2
<ISCED Level 5B>

3
<ISCED Level 5A Bachelor degree>

4
<ISCED Level 5A Masters degree>

5
<ISCED Level 6>

5 How many years experience do you have working as a principal?

This is my first year 1-2 years 3-5 years 6-10 years 11-15 years 16-20 years More than 20 years

1 2 3 4 5 6 7

6 How many years experience do you have working as a principal at this school?
This is my first year 1-2 years 3-5 years 6-10 years 11-15 years 16-20 years More than 20 years

1 2 3 4 5 6 7

7 How many years did you spend as a subject/class teacher before you became a principal?
None Less than 3 years 3-5 years 6-10 years 11-15 years 16-20 years More than 20 years

1 2 3 4 5 6 7

248
Annex D TALIS Principal and Teacher Questionnaire

School background information

8 Is this school a public or private school?

Please mark one choice.

A public school à Please go to question 10.

1
(This is a school managed directly or indirectly by a public education authority, government agency, or
governing board appointed by government or elected by public franchise.)

A private school à Please go to question 9.

2
(This is a school managed directly or indirectly by a non-government organisation; e.g. a church, trade union,
business or other private institution.)

9 Thinking about the funding of this school in a typical year, which of the following applies?
Please only answer this question if you marked “private school” in question 8 before.
Please mark one choice in each row.

Yes No
a) 50% or more of the school’s funding comes from the <government>
(Includes departments, local, regional, state and national) 1 2

b) Teaching personnel are funded by the <government>

(Includes departments, local, regional, state and national) 1 2

10 Which of the following best describes the community in which this school is located?
Please mark one choice.
A <village, hamlet or rural area> (fewer than 3 000 people)
1

A <small town> (3 000 to about 15 000 people)

A <town> (15 000 to about 100 000 people)

A <city> (100 000 to about 1 000 000 people)

A large <city> with over 1 000 000 people

11 For each type of position listed below, indicate the number of staff currently working in this school.
Please indicate the number of persons (in head counts) who work at this school.
Please write a number in each row. Write 0 (zero) if there are none.
a) Teachers, irrespective of the grades/ages they teach
(Those whose main activity at this school is the provision of instruction to students)
b) Personnel for pedagogical support, irrespective of the grades/ages they support
(Including all teacher aides or other non-professional personnel who provide instruction or support
teachers in providing instruction, professional curricular/instructional specialists and educational
media specialists)
c) School administrative or management personnel
(Including principals, assistant principals, other management staff, receptionists, secretaries,
administration assistants whose main activity is administration or management)

TALIS Principal and Teacher Questionnaire Annex D
249

12 What is the current school enrolment (number of students of all grades in this school)?
Please write a number.

Number of students

13 Please estimate the broad percentage of students at <ISCED 2> level in this school who have the following
characteristics.
It is acceptable to base your replies on rough estimates.
Please mark one choice in each row.
10% or 20% or 40% or
more but more but more but
Less than less than less than less than 60% or
10% 20% 40% 60% more
a) Students whose <first language> is different from
the language(s) of instruction or a dialect of this/ 1 2 3 4 5
these.
b) Students who have at least one parent/guardian
who has completed <ISCED 3> or higher. 1 2 3 4 5

c) Students who have at least one parent/guardian

who has completed <ISCED 5> or higher. 1 2 3 4 5

14 How much consideration is given to the following factors when students are considered for admission to this school?
Please mark one choice in each row.
Not High
considered Considered priority Prerequisite
a) Residence in a particular area.
1 2 3 4

b) Students’ academic record (including placement

tests). 1 2 3 4

c) Recommendation of feeder schools.

1 2 3 4

d) Parents’ endorsement of the instructional or

religious philosophy of the school. 1 2 3 4

e) Students’ need or desire for a special programme.

1 2 3 4

f) Attendance of other family members at the school

(past or present). 1 2 3 4

250
Annex D TALIS Principal and Teacher Questionnaire

School management

15 Below you can find statements about your management of this school. Please indicate the frequency of these
activities and behaviours in this school during the current school year.

Please mark one choice in each row.

Quite Very
Never Seldom often often
a) I make sure that the professional development activities of teachers
are in accordance with the teaching goals of the school. 1 2 3 4

b) I ensure that teachers work according to the school’s educational

goals. 1 2 3 4

c) I observe instruction in classrooms.

1 2 3 4

d) I use student performance results to develop the school’s

educational goals. 1 2 3 4

e) I give teachers suggestions as to how they can improve their

teaching. 1 2 3 4

f)
I monitor students’ work. 1 2 3 4

g) When a teacher has problems in his/her classroom, I take the

initiative to discuss matters. 1 2 3 4

h) I inform teachers about possibilities for updating their knowledge

and skills. 1 2 3 4

i) I check to see whether classroom activities are in keeping with our

educational goals. 1 2 3 4

j) I take exam results into account in decisions regarding curriculum

development. 1 2 3 4

k) I ensure that there is clarity concerning the responsibility for

co-ordinating the curriculum. 1 2 3 4

l) When a teacher brings up a classroom problem, we solve the

problem together. 1 2 3 4

m)
I pay attention to disruptive behaviour in classrooms. 1 2 3 4

n) I take over lessons from teachers who are unexpectedly absent.

1 2 3 4

16 How strongly do you agree or disagree with these statements as applied to this school, your job, and the teachers
at this school?
[1/2]

Please mark one choice in each row.

Strongly Strongly
disagree Disagree Agree agree
a) An important part of my job is to ensure ministry-approved
instructional approaches are explained to new teachers, and that 1 2 3 4
more experienced teachers are using these approaches.
b) Using test scores of students to evaluate a teacher’s performance
devalues the teacher’s professional judgment. 1 2 3 4

c) Giving teachers too much freedom to choose their own

instructional techniques can lead to poor teaching. 1 2 3 4

d) A main part of my job is to ensure that the teaching skills of the

staff are always improving. 1 2 3 4

TALIS Principal and Teacher Questionnaire Annex D
251

16 How strongly do you agree or disagree with these statements as applied to this school, your job, and the teachers
at this school?
[2/2]

Please mark one choice in each row.

Strongly Strongly
disagree Disagree Agree agree
e) An important part of my job is to ensure that teachers are held
accountable for the attainment of the school’s goals. 1 2 3 4

f) An important part of my job is to present new ideas to the parents

in a convincing way. 1 2 3 4

g) I influence decisions about this school taken at a higher

administrative level. 1 2 3 4

h) It is important for the school that I see to it that everyone sticks to

the rules. 1 2 3 4

i) It is important for the school that I check for mistakes and errors in
administrative procedures and reports. 1 2 3 4

j) An important part of my job is to resolve problems with the

timetable and/or lesson planning. 1 2 3 4

k) An important part of my job is to create an orderly atmosphere in

the school. 1 2 3 4

l) I have no way of knowing whether teachers are performing well or

badly in their teaching duties. 1 2 3 4

m) In this school, we work on goals and/or a school development

plan. 1 2 3 4

n) I define goals to be accomplished by the staff of this school.

1 2 3 4
o) I stimulate a task-oriented atmosphere in this school.
1 2 3 4

17 As principal of this school, on average throughout the school year, what percentage of time do you estimate that
you spend on the following tasks in this school?
Rough estimates are sufficient.
Please write a number in each row. Write 0 (zero) if none.
Please ensure that responses add up to 100%.
a) % Internal administrative tasks (including human resource/personnel issues, regulations, reports,
school budget, timetable).
b) % Curriculum and teaching-related tasks (including teaching, lesson preparation, classroom
observations, mentoring teachers).
c) % Responding to requests from district, state, or national education officials.
d) % Representing the school at meetings or in the community and networking.
e) % Other.
100 % Total

18 How often during the last 5 years did this school produce a school self-evaluation document and/or was the school
evaluated by an external agency or body (e.g. external inspector)?
This refers to an evaluation of the whole school rather than of individual subjects or departments.
Please mark one choice in each row.
More than
Once per once per
Never Once 2-4 times year year
a) A school self-evaluation report was produced.
1 2 3 4 5
b) An external evaluation was conducted.
1 2 3 4 5

If you replied “Never” to both parts a) and b) above à Please go to question 23.

252
Annex D TALIS Principal and Teacher Questionnaire

19 In your opinion, how important were the following aspects considered to be in these school evaluations?
Please consider both school self-evaluation and external evaluation. We realise these evaluations may have attached
different importance to various aspects, but please consider both types of evaluations in your response to each row.
Please mark one choice in each row.
I do not Considered
know if Not Considered with Considered
it was considered with low moderate with high
considered at all importance importance importance
a) Student test scores.
1 2 3 4 5
b) Retention and pass rates of students.
1 2 3 4 5
c) Other student learning outcomes.
1 2 3 4 5
d) Student feedback on the teaching they receive.
1 2 3 4 5
e) Feedback from parents.
1 2 3 4 5
f) How well teachers work with you, the
principal, and their colleagues. 1 2 3 4 5

g) Direct appraisal of classroom teaching.

1 2 3 4 5
h) Innovative teaching practices.
1 2 3 4 5
i) Relations between teachers and students.
1 2 3 4 5
j) Professional development undertaken by
teachers. 1 2 3 4 5

k) Teachers’ classroom management.

1 2 3 4 5
l) Teachers’ knowledge and understanding of
their main subject field(s). 1 2 3 4 5

m) Teachers’ knowledge and understanding of

instructional practices (knowledge mediation) 1 2 3 4 5
in their main subject field(s).
n) Teaching of students with special learning
needs. 1 2 3 4 5

o) Student discipline and behaviour.

1 2 3 4 5
p) Teaching in a multicultural setting.
1 2 3 4 5
q) Extra-curricular activities with students (e.g. school
plays and performances, sporting activities). 1 2 3 4 5

20 To what extent did these school evaluations have an influence upon the following?
Please mark one choice in each row.
No Low High
influence level of Moderate level of
at all influence influence influence
a) The school budget.
1 2 3 4

b) The performance feedback to this school.

1 2 3 4

c) The performance appraisal of the school management.

1 2 3 4

d) The performance appraisals of individual teachers.

1 2 3 4

e) The assistance provided to teachers to improve their

teaching skills. 1 2 3 4

f) The remuneration and bonuses received by teachers.

1 2 3 4

TALIS Principal and Teacher Questionnaire Annex D
253

21 Are these school evaluations published?

Yes No

1 2

22 Are these school evaluations used by <government> in the publication of tables that compare the performance of
individual schools?
Yes No

1 2

Teacher appraisal
We would like to ask you about the appraisal (defined below) of teachers in this school.
In this survey, appraisal is defined as when a teacher’s work is reviewed by the principal, an external inspector or by his
or her colleagues. This appraisal can be conducted in a range of ways from a more formal, objective approach (e.g. as part
of a formal performance management system, involving set procedures and criteria) to the more informal, more subjective
approach (e.g. through informal discussions with the teacher).

23 How often is the work of teachers in this school appraised by either you, other colleagues in the school, or an
external individual or body (e.g. inspector)?
Less than Once Twice
once every every Once or more
Never 2 years 2 years per year per year
a) You (the principal) 1 2 3 4 5
b) Other teachers or members of the school
management team 1 2 3 4 5

c) External individual or body (e.g. external inspector)

1 2 3 4 5
If you answered “Never” to all of the above (a, b, and c) à Please go to question 29.

24
In your opinion, how important were the following aspects considered to be in these appraisals?
[1/2]

Please mark one choice in each row.

I do not Considered
know if Not Considered with Considered
it was considered with low moderate with high
considered at all importance importance importance
a) Student test scores.
1 2 3 4 5
b) Retention and pass rates of students.
1 2 3 4 5
c) Other student learning outcomes.
1 2 3 4 5
d) Student feedback on the teaching they receive.
1 2 3 4 5
e) Feedback from parents.
1 2 3 4 5
f) How well the teacher works with you, the
principal, and their colleagues. 1 2 3 4 5

g) Direct appraisal of classroom teaching.

1 2 3 4 5
h) Innovative teaching practices.
1 2 3 4 5
i) Relations between the teacher and students.
1 2 3 4 5

254
Annex D TALIS Principal and Teacher Questionnaire

24
In your opinion, how important were the following aspects considered to be in these appraisals?
[2/2]

Please mark one choice in each row.

I do not Considered
know if Not Considered with Considered
it was considered with low moderate with high
considered at all importance importance importance
j) Professional development undertaken by the teacher.
1 2 3 4 5
k) Teacher’s classroom management.
1 2 3 4 5
l) Teacher’s knowledge and understanding of their
main subject field(s). 1 2 3 4 5

m) Teacher’s knowledge and understanding of

instructional practices (knowledge mediation) 1 2 3 4 5
in their main subject field(s).
n) Teaching of students with special learning needs.
1 2 3 4 5
o) Student discipline and behaviour in the teacher’s
classes. 1 2 3 4 5

p) Teaching in a multicultural setting.

1 2 3 4 5
q) Extra-curricular activities with students (e.g. school
plays and performances, sporting activities). 1 2 3 4 5

25 When teachers’ work is appraised in this school, can these appraisals directly lead to any of the following for the teacher?
Please mark one choice in each row.
Can result from Can not result
an appraisal of from an appraisal
teachers’ work of teachers’ work
a) A change in salary.
1 2
b) A financial bonus or another kind of monetary reward.
1 2
c) A change in the likelihood of career advancement.
1 2
d) Opportunities for professional development activities.
1 2
e) Changes in teachers’ work responsibilities that make their job more attractive.
1 2
f) A development or training plan to improve their teaching.
1 2

26 We would like to ask your opinion on the objectives of the appraisal of teachers’ work at this school. Can you
please rate the importance of each of the following objectives in the appraisal of teachers’ work?
Please mark one choice in each row.
No Low Moderate High
importance importance importance importance
a) To determine the career advancement of individual teachers.
1 2 3 4
b) To inform an administrative level above the school (school
board, municipality, school district, school inspectorate). 1 2 3 4

c) To evaluate the performance of the whole school.

1 2 3 4
d) To evaluate the teaching in a particular subject.
1 2 3 4
e) To address a crisis or problem in the school.
1 2 3 4
f) To identify the professional development needs of teachers.
1 2 3 4
g) To take decisions about remuneration and bonuses of teachers.
1 2 3 4
h) To take decisions about school improvement.
1 2 3 4

TALIS Principal and Teacher Questionnaire Annex D
255

27 How often are appraisals of teachers’ work conducted that include a written report that is kept as a record? Please
also indicate who provides this report.
Please mark one choice in each row.
Less than
once Once Twice
every every Once or more
Never 2 years 2 years per year per year
a) You (the principal)
1 2 3 4 5
b) Other teachers or members of the school
management team 1 2 3 4 5

c) External individual or body (e.g. external inspector)

1 2 3 4 5

28 Please indicate the frequency with which each of the following occurs if an appraisal of teachers’ work identifies
weaknesses or you consider a teacher to be underperforming in their teaching duties.
Please mark one choice in each row.
Most of
Never Sometimes the time Always
a) I ensure that the outcome is reported to the teacher.
1 2 3 4
b) I ensure measures to remedy the weaknesses in teaching are
discussed with the teacher. 1 2 3 4

c) I, or others in the school, establish a development or training

plan for the teacher to address the weaknesses in their teaching. 1 2 3 4

d) I, or others in the school, impose material sanctions on the

teacher (e.g. reduced annual increases in pay). 1 2 3 4

e) I, or others in the school, report the underperformance to

another body to take action (e.g. governing board, local 1 2 3 4
authority, school inspector).
f) I ensure the teacher has more frequent appraisals of their work.
1 2 3 4
g) Other (please specify below).
1 2 3 4

School resources

29 Is this school’s capacity to provide instruction hindered by any of the following?

Please mark one choice in each row.
To some
Not at all Very little extent A lot
a) A lack of qualified teachers.
1 2 3 4
b) A lack of laboratory technicians.
1 2 3 4
c) A lack of instructional support personnel.
1 2 3 4
d) A lack of other support personnel.
1 2 3 4
e) Shortage or inadequacy of instructional materials (e.g.
textbooks). 1 2 3 4

f) Shortage or inadequacy of computers for instruction.

1 2 3 4
g) Shortage or inadequacy of other equipment.
1 2 3 4
h) Shortage or inadequacy of library materials.
1 2 3 4
i) Other (please specify below).
1 2 3 4

256
Annex D TALIS Principal and Teacher Questionnaire

30 In this school, to what extent is the learning of students hindered by the following behaviours?
Please mark one choice in each row.
To some
By students in this school: Not at all Very little extent A lot
a) Arriving late at school.
1 2 3 4
b) Absenteeism (i.e. unjustified absences).
1 2 3 4
c) Classroom disturbance.
1 2 3 4
d) Cheating.
1 2 3 4
e) Profanity/Swearing.
1 2 3 4
f) Vandalism.
1 2 3 4
g) Theft.
1 2 3 4
h) Intimidation or verbal abuse of other students (or other forms
of bullying). 1 2 3 4

i) Physical injury to other students.

1 2 3 4
j) Intimidation or verbal abuse of teachers or staff.
1 2 3 4
k) Use/possession of drugs and/or alcohol.
1 2 3 4
To some
By teachers in this school: Not at all Very little extent A lot
l) Arriving late at school.
1 2 3 4
m) Absenteeism.
1 2 3 4
n) Lack of pedagogical preparation.
1 2 3 4

31 Regarding this school, who has a considerable responsibility for the following tasks?
A “considerable responsibility” is one where an active role is played in decision making.
Please mark as many choices as appropriate in each row.
Regional
School or local National
governing education education
Principal Teachers board authority authority
a) Selecting teachers for hire.
1 1 1 1 1
b) Firing teachers.
1 1 1 1 1
c) Establishing teachers’ starting salaries.
1 1 1 1 1
d) Determining teachers’ salary increases.
1 1 1 1 1
e) Formulating the school budget.
1 1 1 1 1
f) Deciding on budget allocations within the school.
1 1 1 1 1
g) Establishing student disciplinary policies.
1 1 1 1 1
h) Establishing student assessment policies.
1 1 1 1 1
i) Approving students for admission to the school.
1 1 1 1 1
j) Choosing which textbooks are used.
1 1 1 1 1
k) Determining course content.
1 1 1 1 1
l) Deciding which courses are offered.
1 1 1 1 1
m) Allocating funds for teachers’ professional
development. 1 1 1 1 1

TALIS Principal and Teacher Questionnaire Annex D
257

32 How strongly do you agree or disagree with each of the following statements about teaching and learning in general?
Please mark one choice in each row.
Strongly Strongly
disagree Disagree Agree agree
a) Effective/good teachers demonstrate the correct way to solve a
problem. 1 2 3 4

b) When referring to a “poor performance”, I mean a performance

that lies below the previous achievement level of the student. 1 2 3 4

c) It is better when the teacher – not the student – decides what

activities are to be done. 1 2 3 4

d) The role of teachers is to facilitate students’ own inquiry.

1 2 3 4
e) Teachers know a lot more than students; they shouldn’t let students
develop answers that may be incorrect when they can just explain 1 2 3 4
the answers directly.
f) Students learn best by finding solutions to problems on their own.
1 2 3 4
g) Instruction should be built around problems with clear, correct
answers, and around ideas that most students can grasp quickly. 1 2 3 4

h) How much students learn depends on how much background

knowledge they have – that is why teaching facts is so necessary. 1 2 3 4

i) Students should be allowed to think of solutions to practical problems

themselves before the teacher shows them how they are solved. 1 2 3 4

j) When referring to a “good performance”, I mean a performance

that lies above the previous achievement level of the student. 1 2 3 4

k) A quiet classroom is generally needed for effective learning.

1 2 3 4
l) Thinking and reasoning processes are more important than specific
curriculum content. 1 2 3 4

33 When a teacher begins teaching at this school, does he/she undertake a formal <induction> process?
Please mark one choice.
Yes, for all teachers who are new to this school.
1
Yes, but only for teachers for whom this is their first teaching job.
2
No, there is no <induction> process for teachers who are new to this school
3 à Go to question 35.

34 If “Yes” in the previous question, who organises the <induction> process?

Please mark one choice.
The school alone.
1
The school together with agencies or institutions outside of the school.
2
Outside agencies or institutions alone.
3

35 When a teacher begins teaching at this school, is there a programme or policy by which he/she works with an
experienced teacher or teachers who act as their mentor?
Please mark one choice.
Yes, for all teachers who are new to this school.
1
Yes, but only for teachers for whom this is their first teaching job.
2
No, there is no mentoring programme or policy in this school à Go to question 37.
3

258
Annex D TALIS Principal and Teacher Questionnaire

36 If “Yes” in the previous question, is the mentor teacher’s main subject area(s) usually the same as that of the new
teacher?
Yes No

1 2

37 How would you rate the importance of mentoring new teachers in helping them to improve their instructional
effectiveness?
Please mark one choice.
Not important Of low Of moderate Of high
at all importance importance importance

1 2 3 4

This is the end of the questionnaire.

Thank you very much for your co-operation!

Please [National Return Procedures and Date]

TALIS Principal and Teacher Questionnaire Annex D
259

[Placeholder
for identification label]
(105 x 35 mm)

OECD Teaching and Learning International Survey (TALIS)

Teacher Questionnaire
Main study version (MS-12-01)
[International English, UK Spelling]
[National Project Information]

International Project Consortium:

International Association for the Evaluation of Educational Achievement (IEA), The Netherlands
IEA Data Processing and Research Center (IEA DPC), Germany
Statistics Canada, Canada

Confidentiality
All information that is collected in this study will be treated confidentially. While results will be made available by country
and by type of school within a country, you are guaranteed that neither you, this school nor any of its personnel will be
identified in any report of the results of the study. [Participation in this survey is voluntary and any individual may withdraw
at any time.]

About the Questionnaire

• This questionnaire asks for information about school education and policy matters.
• This questionnaire should take approximately 45 minutes to complete.
• When questions refer to “this school” we mean by “school”: national school definition.
• Guidelines for answering the questions are typed in italics. Most questions can be answered by marking the one most
appropriate answer.
• When you have completed this questionnaire, please [National Return Procedures and Date].
• When in doubt about any aspect of the questionnaire, or if you would like more information about it or the study, you can
reach us by phone at the following numbers: [National Center Contact Information]

Thank you very much for your co-operation!

260
Annex D TALIS Principal and Teacher Questionnaire

Background information

These questions are about you, your education and the time you have spent in teaching. In responding to the questions, please
mark the appropriate box.

1 What is your gender?

Female Male

1 2

2 How old are you?

Under 25 25-29 30–39 40-49 50-59 60+

1 2 3 4 5 6

3 What is your employment status as a teacher?

Part-time employment is where the contracted hours of work represent less than 90 per cent of the normal or
statutory number of hours of work for a full-time employee over a complete school year. Please consider your
employment status for all of your teaching jobs combined.
Full-time
1

Part-time (50-90% of full-time hours)

Part-time (less than 50% of full-time hours)

4 Do you work as a teacher of <ISCED level 2> at another school as well as this school?
Yes
1

No à Please go to question 6.
2

5 If “Yes” in the previous question, please indicate in how many other schools you work as a <ISCED level 2> teacher.
Please write in a number.
Schools

6 What is your employment status as a teacher at this school?

Please do not consider the probationary period of a contract as a separate contract.
Permanent employment (an on-going contract with no fixed end-point before the age of retirement)
1

Fixed term contract for a period of more than 1 school-year

Fixed-term contract for a period of 1 school-year or less

TALIS Principal and Teacher Questionnaire Annex D
261

7 What is the highest level of formal education that you have completed?
Please mark one choice.
<Below ISCED Level 5>
1
<ISCED Level 5B>
2
<ISCED Level 5A Bachelor degree>
3
<ISCED Level 5A Masters degree>
4
<ISCED Level 6>
5

8 In a typical school week, estimate the number of (60-minute) hours you spend on the following for this school.
This question concerns your work for this school only. Please do not include the work you do for other schools.
Please write a number in each row and round to the nearest hour in your responses.
Write 0 (zero) if none.
a) Teaching of students in school (either whole class, in groups or individually)

b) Planning or preparation of lessons either in school or out of school (including marking of student work)
Administrative duties either in school or out of school (including school administrative duties, paperwork
c)
and other clerical duties you undertake in your job as a teacher)
d) Other (please specify):

9 How long have you been working as a teacher?

Where possible exclude extended periods of absence (e.g. career breaks).
This is my More than
first year 1-2 years 3-5 years 6-10 years 11-15 years 16-20 years 20 years

1 2 3 4 5 6 7

10 How long have you been working as a teacher at this school?

Where possible exclude extended periods of absence (e.g. career breaks).
This is my More than
first year 1-2 years 3-5 years 6-10 years 11-15 years 16-20 years 20 years

1 2 3 4 5 6 7

262
Annex D TALIS Principal and Teacher Questionnaire

Professional development

In this survey, professional development is defined as activities that develop an individual’s skills, knowledge, expertise and
other characteristics as a teacher.
Please only consider professional development you have taken after your initial teacher training/education.

11 During the last 18 months, did you participate in any of the following kinds of professional development activities,
and what was the impact of these activities on your development as a teacher?
For each question below, please mark one choice in part (A). If you answer “Yes” in part (A) then please mark one
choice in part (B) to indicate how much impact it had upon your development as a teacher.
(A) (B)
Participation Impact
No A small A moderate A large
Yes No impact impact impact impact
a) Courses/workshops (e.g. on subject matter or
methods and/or other education-related topics). 1 2 1 2 3 4

b) Education conferences or seminars (where

teachers and/or researchers present their
research results and discuss educational 1 2 1 2 3 4
problems).
c) Qualification programme (e.g. a degree
programme). 1 2 1 2 3 4

d) Observation visits to other schools.

1 2 1 2 3 4
e) Participation in a network of teachers formed
specifically for the professional development 1 2 1 2 3 4
of teachers.
f) Individual or collaborative research on a topic
of interest to you professionally. 1 2 1 2 3 4

g) Mentoring and/or peer observation and

coaching, as part of a formal school 1 2 1 2 3 4
arrangement.

12 In all, how many days of professional development did you attend during the last 18 months?
Please round to whole days. Write 0 (zero) if none.
Days

If you answered “0” (zero) à Please go to question 17.

13 Of these, how many days were compulsory for you to attend as part of your job as a teacher?
Please round to whole days. Write 0 (zero) if none.
Days

14 For the professional development in which you participated in the last 18 months, how much did you personally
have to pay for?
Please mark one choice.
None Some All

1 2 3

TALIS Principal and Teacher Questionnaire Annex D
263

15 For the professional development in which you participated in the last 18 months, did you receive scheduled time for
undertaking the professional development that took place during regular work hours?
Please mark one choice.

1
Yes

2
No

3
Did not take place during regular work hours

16 For the professional development in which you participated in the last 18 months, did you receive a salary
supplement for undertaking the professional development activities that took place outside regular work hours?
Please mark one choice.

1
Yes

2
No

3
Did not take place outside of regular work hours

17 Thinking about less formal professional development, during the last 18 months, did you participate in any of the
following activities, and what was the impact of these activities on your development as a teacher?
For each question below, please mark one choice in part (A). If you answer “Yes” in part (A) then please mark one
choice in part (B) to indicate how much impact it had upon your development as a teacher.
(A) (B)
Participation Impact
No A small A moderate A large
Yes No impact impact impact impact
a) Reading professional literature (e.g. journals,
evidence-based papers, thesis papers). 1 2 1 2 3 4

b) Engaging in informal dialogue with your

colleagues on how to improve your teaching. 1 2 1 2 3 4

18 Thinking of your own professional development needs, please indicate the extent to which you have such needs in
each of the areas listed.
Please mark one choice in each row.
Moderate
No need Low level level of High level
at all of need need of need
a) Content and performance standards in my main subject field(s).
1 2 3 4
b) Student assessment practices.
1 2 3 4
c) Classroom management.
1 2 3 4
d) Knowledge and understanding of my main subject field(s).
1 2 3 4
e) Knowledge and understanding of instructional practices
(knowledge mediation) in my main subject field(s). 1 2 3 4
f) ICT skills for teaching.
1 2 3 4
g) Teaching students with special learning needs.
1 2 3 4
h) Student discipline and behaviour problems.
1 2 3 4
i) School management and administration.
1 2 3 4
j) Teaching in a multicultural setting.
1 2 3 4
k) Student counselling.
1 2 3 4

264
Annex D TALIS Principal and Teacher Questionnaire

19 In the last 18 months, did you want to participate in more professional development than you did?

1
Yes

2
No à Please go to question 21.

20 If “Yes” in the previous question, which of the following reasons best explain what prevented you from
participating in more professional development than you did?
Please mark as many choices as appropriate.

1
I did not have the pre-requisites (e.g. qualifications, experience, seniority).

1
Professional development was too expensive/I could not afford it.

1
There was a lack of employer support.

1
Professional development conflicted with my work schedule.

1
I didn’t have time because of family responsibilities.

1
There was no suitable professional development offered.

1
Other (please specify):

Teacher appraisal and feedback

We would like to ask you about the appraisal (defined below) of your work as a teacher and the feedback (defined below)
you receive about your work in this school.
In this survey, Appraisal is defined as when a teacher’s work is reviewed by the principal, an external inspector or by his
or her colleagues. This appraisal can be conducted in a range of ways from a more formal, objective approach (e.g. as part
of a formal performance management system, involving set procedures and criteria) to the more informal, more subjective
approach (e.g. through informal discussions with the teacher).
In this survey, Feedback is defined as the reporting of the results of a review of your work (however formal or informal
that review has been) back to the teacher, often with the purpose of noting good performance or identifying areas for
development. Again, the feedback may be provided formally (e.g. through a written report) or informally (e.g. through
discussions with the teacher).

21 From the following people, how often have you received appraisal and/or feedback about your work as a teacher
in this school?
Please mark one choice in each row.
Less than 3 or More
once Once more than
every every Once Twice times once per
Never two years two years per year per year per year Monthly month
a) Principal
1 2 3 4 5 6 7 8

b) Other teachers or members of

the school management team 1 2 3 4 5 6 7 8

c) External individual or body

(e.g. external inspector) 1 2 3 4 5 6 7 8

If you answered “Never” for all of the above (a, b, and c) à Please go to question 28.

TALIS Principal and Teacher Questionnaire Annex D
265

22 In your opinion, how important were the following aspects considered to be when you received this appraisal
and/or feedback?
Please mark one choice in each row.
I do not Considered
know if Not Considered with Considered
it was considered with low moderate with high
considered at all importance importance importance
a) Student test scores.
1 2 3 4 5
b) Retention and pass rates of students.
1 2 3 4 5
c) Other student learning outcomes.
1 2 3 4 5
d) Student feedback on my teaching.
1 2 3 4 5
e) Feedback from parents.
1 2 3 4 5
f) How well I work with the principal and my
colleagues. 1 2 3 4 5

g) Direct appraisal of my classroom teaching.

1 2 3 4 5
h) Innovative teaching practices.
1 2 3 4 5
i) Relations with students.
1 2 3 4 5
j) Professional development I have undertaken.
1 2 3 4 5
k) Classroom management.
1 2 3 4 5
l) Knowledge and understanding of my main
subject field(s). 1 2 3 4 5

m) Knowledge and understanding of

instructional practices (knowledge mediation) 1 2 3 4 5
in my main subject field(s).
n) Teaching students with special learning needs.
1 2 3 4 5
o) Student discipline and behaviour.
1 2 3 4 5
p) Teaching in a multicultural setting.
1 2 3 4 5
q) Extra-curricular activities with students (e.g.
school plays and performances, sporting 1 2 3 4 5
activities).
r) Other (please specify below).
1 2 3 4 5

23 Concerning the appraisal and/or feedback you have received at this school, to what extent have they directly led
to any of the following?
Please mark one choice in each row.
No A small A moderate A large
change change change change
a) A change in salary.
1 2 3 4
b) A financial bonus or another kind of monetary reward.
1 2 3 4
c) Opportunities for professional development activities.
1 2 3 4
d) A change in the likelihood of career advancement.
1 2 3 4
e) Public recognition from the principal and/or your colleagues.
1 2 3 4
f) Changes in your work responsibilities that make the job
more attractive. 1 2 3 4

g) Role in school development initiatives (e.g. curriculum

development group, development of school objectives). 1 2 3 4

266
Annex D TALIS Principal and Teacher Questionnaire

24 Concerning the appraisal and/or feedback you have received at this school, to what extent have they directly led to
or involved changes in any of the following?
Please mark one choice in each row.
No A small A moderate A large
change change change change
a) Your classroom management practices.
1 2 3 4

b) Your knowledge and understanding of your main subject field(s).

1 2 3 4

c) Your knowledge and understanding of instructional practices

(knowledge mediation) in you main subject field(s). 1 2 3 4

d) A development or training plan to improve your teaching.

1 2 3 4

e) Your teaching of students with special learning needs.

1 2 3 4

f) Your handling of student discipline and behaviour problems.

1 2 3 4

g) Your teaching of students in a multicultural setting.

1 2 3 4

h) The emphasis you place upon improving student test scores in

your teaching. 1 2 3 4

25 How would you describe the appraisal and/or feedback you received?
Please mark one choice in each row.
Yes No
a) The appraisal and/or feedback contained a judgment about the quality of my work.
1 2

b) The appraisal and/or feedback contained suggestions for improving certain aspects of my work.
1 2

26 Regarding the appraisal and/or feedback you received at this school, to what extent do you agree or disagree with
the following statements?
Please mark one choice in each row.
Strongly Strongly
disagree Disagree Agree agree
a) I think the appraisal of my work and/or feedback received was a
fair assessment of my work as a teacher in this school. 1 2 3 4

b) I think the appraisal of my work and/or feedback received was

helpful in the development of my work as a teacher in this school. 1 2 3 4

27 Concerning the appraisal and/or feedback you have received at this school, to what extent have they directly led to
any of the following?
Please mark one choice in each row.

A large A small A small A large

decrease decrease No change increase increase
a) Changes in your job satisfaction.
1 2 3 4 5

b) Changes in your job security.

1 2 3 4 5

TALIS Principal and Teacher Questionnaire Annex D
267

28 We would like to ask you about appraisal and/or feedback to teachers in this school more generally. To what extent
do you agree or disagree with the following statements?
Please mark one choice in each row.
Strongly Strongly
disagree Disagree Agree agree
a) In my opinion, in this school the principal takes steps to alter the
monetary rewards of a persistently underperforming teacher. 1 2 3 4

b) In my opinion, in this school the sustained poor performance of

a teacher would be tolerated by the rest of the staff. 1 2 3 4

c) In this school, teachers will be dismissed because of sustained

poor performance. 1 2 3 4

d) In my opinion, in this school the principal uses effective methods

to determine whether teachers are performing well or badly. 1 2 3 4

e) In my opinion, in this school a development or training plan is

established for teachers to improve their work as a teacher. 1 2 3 4

f) In my opinion, the most effective teachers in this school receive

the greatest monetary or non-monetary rewards. 1 2 3 4

g) If I improve the quality of my teaching at this school, I will

receive increased monetary or non-monetary rewards. 1 2 3 4

h) If I am more innovative in my teaching at this school, I will

receive increased monetary or non-monetary rewards. 1 2 3 4

i) In my opinion, in this school the review of teachers’ work is

largely done to fulfil administrative requirements. 1 2 3 4

j) In my opinion, in this school the review of teachers’ work has

little impact upon the way teachers teach in the classroom. 1 2 3 4

Teaching practices, beliefs and attitudes

29 We would like to ask about your personal beliefs on teaching and learning. Please indicate how much you disagree
or agree with each of the following statements.
[1/2]

Please mark one choice in each row.

Strongly Strongly
disagree Disagree Agree agree
a) Effective/good teachers demonstrate the correct way to solve
a problem. 1 2 3 4

b) When referring to a “poor performance”, I mean a

performance that lies below the previous achievement level of 1 2 3 4
the student.
c) It is better when the teacher – not the student – decides what
activities are to be done. 1 2 3 4

d) My role as a teacher is to facilitate students’ own inquiry.

1 2 3 4
e) Teachers know a lot more than students; they shouldn’t let
students develop answers that may be incorrect when they can 1 2 3 4
just explain the answers directly.
f) Students learn best by finding solutions to problems on their
own. 1 2 3 4

g) Instruction should be built around problems with clear, correct

answers, and around ideas that most students can grasp 1 2 3 4
quickly.

268
Annex D TALIS Principal and Teacher Questionnaire

29 We would like to ask about your personal beliefs on teaching and learning. Please indicate how much you disagree
or agree with each of the following statements.
[2/2]

Please mark one choice in each row.

Strongly Strongly
disagree Disagree Agree agree
h) How much students learn depends on how much background
knowledge they have – that is why teaching facts is so 1 2 3 4
necessary.
i) Students should be allowed to think of solutions to practical
problems themselves before the teacher shows them how they 1 2 3 4
are solved.
j) When referring to a “good performance”, I mean a
performance that lies above the previous achievement level of 1 2 3 4
the student.
k) A quiet classroom is generally needed for effective learning.
1 2 3 4
l) Thinking and reasoning processes are more important than
specific curriculum content. 1 2 3 4

30 How often do you do the following in this school?

Please mark one choice in each row.
Less than
once Once 3-4 times
Never per year per year per year Monthly Weekly
a) Attend staff meetings to discuss the vision
and mission of the school. 1 2 3 4 5 6

b) Develop a school curriculum or part of it.

1 2 3 4 5 6

c) Discuss and decide on the selection of

instructional media (e.g. textbooks, exercise 1 2 3 4 5 6
books).
d) Exchange teaching materials with
colleagues. 1 2 3 4 5 6

e) Attend team conferences for the age group

I teach. 1 2 3 4 5 6

f) Ensure common standards in evaluations

for assessing student progress. 1 2 3 4 5 6

g) Engage in discussion about the learning

development of specific students. 1 2 3 4 5 6

h) Teach jointly as a team in the same class.

1 2 3 4 5 6

i) Take part in professional learning activities

(e.g. team supervision). 1 2 3 4 5 6

j) Observe other teachers’ classes and provide

feedback. 1 2 3 4 5 6

k) Engage in joint activities across different

classes and age groups (e.g. projects). 1 2 3 4 5 6

l) Discuss and co-ordinate homework

practice across subjects. 1 2 3 4 5 6

TALIS Principal and Teacher Questionnaire Annex D
269

31 How strongly do you agree or disagree with the following statements …

Please mark one choice in each row.
Strongly Strongly
… about yourself as a teacher in this school? disagree Disagree Agree agree
a) All in all, I am satisfied with my job.
1 2 3 4

b) I feel that I am making a significant educational difference in

the lives of my students. 1 2 3 4

c) If I try really hard, I can make progress with even the most
difficult and unmotivated students. 1 2 3 4

d) I am successful with the students in my class.

1 2 3 4

e) I usually know how to get through to students.

1 2 3 4

f) Teachers in this local community are well respected.

1 2 3 4

Strongly Strongly
… about what happens in this school? Disagree Disagree Agree Agree
g) In this school, teachers and students usually get on well with
each other. 1 2 3 4

h) Most teachers in this school believe that students’ well-being is

important. 1 2 3 4

i) Most teachers in this school are interested in what students

have to say. 1 2 3 4

j) If a student from this school needs extra assistance, the school

provides it. 1 2 3 4

32 Below you can find statements about the management of your school. Please indicate your perceptions of the
frequency with which these activities took place during the current school year.
Please mark one choice in each row.
Never Seldom Quite often Very often
a) In meetings, the principal discusses educational goals with
teachers. 1 2 3 4

b) The principal ensures that teachers work according to the

school’s educational goals. 1 2 3 4

c) The principal or someone else in the management team

observes teaching in classes. 1 2 3 4

d) The principal gives teachers suggestions as to how they can

improve their teaching. 1 2 3 4

e) When a teacher has problems in his/her classroom, the principal

takes the initiative to discuss the matter. 1 2 3 4

f) The principal ensures that teachers are informed about

possibilities for updating their knowledge and skills. 1 2 3 4

g) The principal compliments teachers for special effort or

accomplishments. 1 2 3 4

h) In this school, the principal and teachers work on a school

development plan. 1 2 3 4

i) The principal defines goals to be accomplished by the staff of

this school. 1 2 3 4

j) The principal ensures that a task-oriented atmosphere is fostered

in this school. 1 2 3 4

k) In this school, the principal and teachers act to ensure that

education quality issues are a collective responsibility. 1 2 3 4

270
Annex D TALIS Principal and Teacher Questionnaire

33 We would like to ask you about the main <ISCED Level 2> subjects that you teach in this school in this school year.
Please indicate the <ISCED Level 2> subjects that you teach in this school (indicate only those that individually
account for at least 20% of your teaching time in this school). The exact name of your subjects may not appear in the
list below each category. If it does not, please mark the category you think best fits the subject.
Yes No
a) Reading, writing and literature
Includes reading and writing (and literature) in the mother tongue, reading and writing (and
literature) in the language of instruction, reading and writing in the tongue of the country 1 2
(region) as a second language (for non-natives), language studies, public speaking, literature.
b) Mathematics
Includes mathematics, mathematics with statistics, geometry, algebra etc. 1 2

c) Science
Includes science, physics, physical science, chemistry, biology, human biology, environmental 1 2
science, agriculture/horticulture/forestry.
d) Social studies
Includes social studies, community studies, contemporary studies, economics, environmental
studies, geography, history, humanities, legal studies, studies of the own country, social 1 2
sciences, ethical thinking, philosophy.
e) Modern foreign languages
Includes languages different from the language of instruction. 1 2

f) Technology
Includes orientation in technology, including information technology, computer studies,
construction/surveying, electronics, graphics and design, keyboard skills, word processing, 1 2
workshop technology / design technology.
g) Arts
Includes arts, music, visual arts, practical art, drama, performance music, photography, 1 2
drawing, creative handicraft, creative needlework.
h) Physical education
Includes physical education, gymnastics, dance, health. 1 2

i) Religion and/or ethics

Includes religion, history of religions, religion culture, ethics. 1 2

j) Practical and vocational skills

Includes vocational skills (preparation for a specific occupation), technics, domestic science,
accountancy, business studies, career education, clothing and textiles, driving, home 1 2
economics, polytechnic courses, secretarial studies, tourism and hospitality, handicraft.
k) Other (please specify below)
1 2

TALIS Principal and Teacher Questionnaire Annex D
271

Your teaching in a particular <class> at this school

The following questions ask you about a particular<ISCED Level 2> <class> that you teach in one of the main subjects you
identified in question 33.
The <class> that we would like you to respond about is the first <ISCED Level 2> <class> that you (typically) teach in this
school in one of these subjects after 11am on Tuesdays. Please note that the <class> can occur on a day following Tuesday if
you do not teach the <class> on Tuesday.
In the questions below, this <class> will be referred to as the <target class>.

34 Into which subject category in question 33 does this <target class> fall?
Please mark one choice.

1
a) Reading, writing and literature

2
b) Mathematics

3
c) Science

4
d) Social studies

5
e) Modern foreign languages

6
f) Technology

7
g) Arts

8
h) Physical education

9
i) Religion

10
j) Practical and vocational skills

11
k) Other

35 What is the actual name of the subject you teach in this <target class>?
Please write the name of the subject as it is used within this school.

36 Was the teaching of this subject part of your academic training?

Yes No

1 2

37 What is the year/grade level of this <target class>?

Please mark one choice.

1
<ISCED level 2 grade 1>

2
<ISCED level 2 grade 2>

3
<ISCED level 2 grade 3>

4
<ISCED level 2 grade 4>

5
<ISCED level 2 grade 5>

272
Annex D TALIS Principal and Teacher Questionnaire

38 On average throughout the year how many students are in this <target class>?
Please write a number.
Number of students

39 How would you describe the ability of students in this <target class>?
Please mark one choice in each row.
Slightly Slightly
Much lower lower than higher than Much higher
than average average Average average than average
ability ability ability ability ability
a) Compared to other students in the same
grade/year level in this school? 1 2 3 4 5
b) Compared to other students in the same
grade/year level more generally? 1 2 3 4 5

40 For this <target class>, please estimate the broad percentage of students who have the following characteristics.
It is acceptable to base your replies on rough estimates.
Please mark one choice in each row.
10% or 20% or 40% or
more but more but more but
Less than less than less than less than 60% or
10% 20% 40% 60% more
a) Students whose <first language> is different from the
language(s) of instruction or a dialect of this/these 1 2 3 4 5
b) Students who have at least one parent/guardian who
has completed <ISCED 3> or higher 1 2 3 4 5
c) Students who have at least one parent/guardian who
has completed <ISCED 5> or higher 1 2 3 4 5

41 For this <target class>, what percentage of <class> time is typically spent on each of the following activities?
Write a percentage for each activity. Write 0 (zero) if none.
Please ensure that responses add up to 100%.
a) % Administrative tasks (e.g. recording attendance, handing out school information/forms)
b) % Keeping order in the classroom (maintaining discipline)
c) % Actual teaching and learning
100 % Total

TALIS Principal and Teacher Questionnaire Annex D
273

42 How often do each of the following activities happen in this <target class> throughout the school year?
Please note that not all questions in this section are fully adapted to all sorts of teachers. Therefore, please just
answer as best you can.
Never or In about In about In about In almost
hardly one-quarter one-half three-quarters every
ever of <lessons> of <lessons> of <lessons> <lesson>
a) I present new topics to the class (lecture-
style presentation). 1 2 3 4 5

b) I explicitly state learning goals.

1 2 3 4 5

c) I review with the students the homework

they have prepared. 1 2 3 4 5

d) Students work in small groups to come up

with a joint solution to a problem or task. 1 2 3 4 5

e) I give different work to the students that

have difficulties learning and/or to those 1 2 3 4 5
who can advance faster.
f) I ask my students to suggest or to help plan
classroom activities or topics. 1 2 3 4 5

g) I ask my students to remember every step in

a procedure. 1 2 3 4 5

h) At the beginning of the lesson I present a

short summary of the previous lesson. 1 2 3 4 5

i) I check my students’ exercise books.

1 2 3 4 5

j) Students work on projects that require at

least one week to complete. 1 2 3 4 5

k) I work with individual students.

1 2 3 4 5

l) Students evaluate and reflect upon their

own work. 1 2 3 4 5

m) I check, by asking questions, whether or not

the subject matter has been understood. 1 2 3 4 5

n) Students work in groups based upon their

abilities. 1 2 3 4 5

o) Students make a product that will be used

by someone else. 1 2 3 4 5

p) I administer a test or quiz to assess student

learning. 1 2 3 4 5

q) I ask my students to write an essay in which

they are expected to explain their thinking 1 2 3 4 5
or reasoning at some length.
r) Students work individually with the
textbook or worksheets to practice newly 1 2 3 4 5
taught subject matter.
s) Students hold a debate and argue for a
particular point of view which may not be 1 2 3 4 5
their own.

274
Annex D TALIS Principal and Teacher Questionnaire

43 How strongly do you agree or disagree with the following statements about this <target class>?
Please mark one choice in each row.
Strongly Strongly
disagree Disagree Agree agree
a) When the lesson begins, I have to wait quite a long time for
students to <quieten down>. 1 2 3 4

b) Students in this class take care to create a pleasant learning

atmosphere. 1 2 3 4

c) I lose quite a lot of time because of students interrupting the

lesson. 1 2 3 4

d) There is much noise in this classroom.

1 2 3 4

This is the end of the questionnaire.

Thank you very much for your co-operation!

Please [National Return Procedures and Date]

OECD PUBLICATIONS, 2, rue André-Pascal, 75775 PARIS CEDEX 16
PRINTED IN FRANCE
(87 2010 01 1P) ISBN 978-92-64-07985-4 – No. 57309 2010
TALIS 2008 Technical Report
The OECD’s new Teaching and Learning International Survey (TALIS) has been designed to provide
data and analyses on the conditions needed for effective teaching and learning in schools. As the TALIS 2008
Technical Report
first international survey with this focus, it seeks to fill important information gaps that have been
identified at the national and international levels of education systems.

TALIS 2008 Technical Report

ISBN 978-92-64-07985-4
87 2010 01 1P
www.oecd.org/publishing

Educating Teachers For Diversity-OECD PRESS
100% (1)
Educating Teachers For Diversity-OECD PRESS
318 pages
Incidental Vocabulary Acquisition
No ratings yet
Incidental Vocabulary Acquisition
4 pages
Meaning and Significance of The Humanities
96% (27)
Meaning and Significance of The Humanities
4 pages
Intellectual Disability Brochure PDF
50% (2)
Intellectual Disability Brochure PDF
2 pages
MORPHOLOGY Notes
No ratings yet
MORPHOLOGY Notes
5 pages
TALIS 2018 Technical Report
No ratings yet
TALIS 2018 Technical Report
496 pages
Talis 2018 Talis Starting Strong 2018 User Guide
No ratings yet
Talis 2018 Talis Starting Strong 2018 User Guide
699 pages
Creating Effective Teaching and Learning Environments First Results From TALIS 2009 Ed. Edition Oecd Instant Download
No ratings yet
Creating Effective Teaching and Learning Environments First Results From TALIS 2009 Ed. Edition Oecd Instant Download
38 pages
OECD Education Complete
No ratings yet
OECD Education Complete
451 pages
Ocde Informe2006
No ratings yet
Ocde Informe2006
525 pages
Join Talis2024
No ratings yet
Join Talis2024
2 pages
1d0bc92a en
No ratings yet
1d0bc92a en
220 pages
OECD Trends Shaping Education 2008
No ratings yet
OECD Trends Shaping Education 2008
90 pages
3b9551db en
No ratings yet
3b9551db en
185 pages
Education at A Glance 2020: Oecd Indicators
No ratings yet
Education at A Glance 2020: Oecd Indicators
476 pages
Teaching in Focus #29 Supporting and Guiding Novice Teachers - Evidence From TALIS 2018
No ratings yet
Teaching in Focus #29 Supporting and Guiding Novice Teachers - Evidence From TALIS 2018
6 pages
Education at A Glance 2016 OECD Indicators
No ratings yet
Education at A Glance 2016 OECD Indicators
510 pages
Education at A Glance
No ratings yet
Education at A Glance
474 pages
Belgium Coronavirus Education Country Note
No ratings yet
Belgium Coronavirus Education Country Note
9 pages
Pisa 2009 Results
100% (1)
Pisa 2009 Results
21 pages
Talis2018 CN Fin
No ratings yet
Talis2018 CN Fin
6 pages
Innovating To Learn, Learn To Innovate
No ratings yet
Innovating To Learn, Learn To Innovate
260 pages
E13bef63 en
No ratings yet
E13bef63 en
472 pages
Marketing Teacher Quality Critical Discourse Analysis of OECD Documents On Effective Teaching and TALIS
No ratings yet
Marketing Teacher Quality Critical Discourse Analysis of OECD Documents On Effective Teaching and TALIS
17 pages
Learning and Teaching in Portugal: An Analysis of TALIS 2013
No ratings yet
Learning and Teaching in Portugal: An Analysis of TALIS 2013
7 pages
9604131e PDF
No ratings yet
9604131e PDF
154 pages
Cross-Border Tertiary Education
No ratings yet
Cross-Border Tertiary Education
204 pages
Pedagogical Knowledge PDF
No ratings yet
Pedagogical Knowledge PDF
278 pages
Final Publication Version of The 7 Lessons Mainstreaming Cross Cutting Issues
No ratings yet
Final Publication Version of The 7 Lessons Mainstreaming Cross Cutting Issues
40 pages
OECD (2021) - Teaching in Focus #38, What Can Schools and Teachers Do To Boost Students Academically
No ratings yet
OECD (2021) - Teaching in Focus #38, What Can Schools and Teachers Do To Boost Students Academically
6 pages
The OECD Diffusion Mechanisms and Its Link With Teacher Policy Worldwide
No ratings yet
The OECD Diffusion Mechanisms and Its Link With Teacher Policy Worldwide
24 pages
Comparable Corpora and Computer-assisted Translation
From Everand
Comparable Corpora and Computer-assisted Translation
Estelle Maryline Delpech
No ratings yet
Global Education Digest 2011 en
No ratings yet
Global Education Digest 2011 en
312 pages
Finland: Education Policy Outlook
No ratings yet
Finland: Education Policy Outlook
36 pages
Education at Glance
No ratings yet
Education at Glance
58 pages
Andreas Schleicher
No ratings yet
Andreas Schleicher
3 pages
Isced 2011 Operational Manual PDF
No ratings yet
Isced 2011 Operational Manual PDF
118 pages
Learningwiththeworld
No ratings yet
Learningwiththeworld
15 pages
Trends Shaping Education 2008 Edition Oecd Download
100% (1)
Trends Shaping Education 2008 Edition Oecd Download
36 pages
274155-Education Policy Outlook Spain - en
No ratings yet
274155-Education Policy Outlook Spain - en
26 pages
Education at A Glance OECD Indicators 2006 Edition Oecd PDF Download
100% (1)
Education at A Glance OECD Indicators 2006 Edition Oecd PDF Download
72 pages
PISA 2018 Assessment and Analytical Framework
No ratings yet
PISA 2018 Assessment and Analytical Framework
308 pages
Organization For Economic Co-Operation and Development - Globalisation, Poverty and Inequality (Development Centre Seminars)
No ratings yet
Organization For Economic Co-Operation and Development - Globalisation, Poverty and Inequality (Development Centre Seminars)
136 pages
Economics: Productivity and Technology Shocks
From Everand
Economics: Productivity and Technology Shocks
Guthlac N. Kirk Anyalezu
No ratings yet
Country Note Key Fi Ndings From PISA 2015 For The United States
No ratings yet
Country Note Key Fi Ndings From PISA 2015 For The United States
76 pages
Monitoring, Evaluation and Research
From Everand
Monitoring, Evaluation and Research
Ityai Muvandi
No ratings yet
OECD Report 69096873-En
No ratings yet
OECD Report 69096873-En
476 pages
2020 Book AudaciousEducation
No ratings yet
2020 Book AudaciousEducation
256 pages
Policy Implications of School Management
No ratings yet
Policy Implications of School Management
6 pages
Ce2b1a62 en PDF
No ratings yet
Ce2b1a62 en PDF
118 pages
Australia Edfreeucation - OECD
No ratings yet
Australia Edfreeucation - OECD
24 pages
OCDE - Studying Pedagogical Knowledge Across Education Systems
No ratings yet
OCDE - Studying Pedagogical Knowledge Across Education Systems
165 pages
Pisa - 2014
No ratings yet
Pisa - 2014
254 pages
Education Trends in Perspective Analysis of The World Education Indicators 2005 Edition Oecd Download
No ratings yet
Education Trends in Perspective Analysis of The World Education Indicators 2005 Edition Oecd Download
52 pages
Romania Vs Sweden
No ratings yet
Romania Vs Sweden
2 pages
Understanding Teachers' Pedagogical Knowledge, 2017
No ratings yet
Understanding Teachers' Pedagogical Knowledge, 2017
151 pages
Information Systems Architecture for University Hospitals
From Everand
Information Systems Architecture for University Hospitals
Silvano Herculano da Luz Júnior
No ratings yet
Education Policy Outlook Country Profile Belgium
No ratings yet
Education Policy Outlook Country Profile Belgium
32 pages
373142eng PDF
No ratings yet
373142eng PDF
180 pages
UIS 2024 Report Summary
No ratings yet
UIS 2024 Report Summary
27 pages
The Comprehensive Professional
No ratings yet
The Comprehensive Professional
20 pages
OECD-South Eastern Europe Albania, Bosnia-Herzegovina, Bulgaria, Croatia, Kosovo
No ratings yet
OECD-South Eastern Europe Albania, Bosnia-Herzegovina, Bulgaria, Croatia, Kosovo
381 pages
Towards best practice in the Archetype Development Process
From Everand
Towards best practice in the Archetype Development Process
Alberto Moreno Conde
No ratings yet
Vocab-Đã G P
No ratings yet
Vocab-Đã G P
2 pages
International Journal of Educational Research: Gavin T.L. Brown, Harish Chaudhry, Ratna Dhamija
No ratings yet
International Journal of Educational Research: Gavin T.L. Brown, Harish Chaudhry, Ratna Dhamija
15 pages
Present Tense
No ratings yet
Present Tense
5 pages
Bo de Kiem Tra 4 Ky Nang Tieng Anh 7 Tap 1
No ratings yet
Bo de Kiem Tra 4 Ky Nang Tieng Anh 7 Tap 1
119 pages
Beliefs, Practices, and Reflection: Exploring A Science Teacher's Classroom Assessment Through The Assessment Triangle Model
No ratings yet
Beliefs, Practices, and Reflection: Exploring A Science Teacher's Classroom Assessment Through The Assessment Triangle Model
19 pages
Wang 2020
No ratings yet
Wang 2020
11 pages
Assessing Assessment Literacy and Practices Among Lecturers: Seyed Ali Rezvani Kalajahi, Ain Nadzimah Abdullah
100% (1)
Assessing Assessment Literacy and Practices Among Lecturers: Seyed Ali Rezvani Kalajahi, Ain Nadzimah Abdullah
17 pages
Subjunctive Mood PDF
No ratings yet
Subjunctive Mood PDF
9 pages
Conditional Sentences 3
No ratings yet
Conditional Sentences 3
9 pages
1-The Portrait of A Lady Notes
No ratings yet
1-The Portrait of A Lady Notes
6 pages
Frequently Asked Questions MSPM
No ratings yet
Frequently Asked Questions MSPM
6 pages
Science Education in The Philippines
No ratings yet
Science Education in The Philippines
15 pages
SNR Stage1 Draft2 Study Religion 16
No ratings yet
SNR Stage1 Draft2 Study Religion 16
58 pages
Math Action Plan
No ratings yet
Math Action Plan
4 pages
Impact of Dr. Apj Abdul Kalam On Youth
No ratings yet
Impact of Dr. Apj Abdul Kalam On Youth
1 page
8 The Impact of Globalization On Communication
No ratings yet
8 The Impact of Globalization On Communication
1 page
Level 3 - Lesson 5 - Likes & Dislikes 2
No ratings yet
Level 3 - Lesson 5 - Likes & Dislikes 2
8 pages
IAN-JOEL-RESEARCH-Chat-GPT Final
No ratings yet
IAN-JOEL-RESEARCH-Chat-GPT Final
41 pages
SIT Indicative Grade Profile
No ratings yet
SIT Indicative Grade Profile
5 pages
Registration For The 50th National and 11th International PAFTE Convention
No ratings yet
Registration For The 50th National and 11th International PAFTE Convention
5 pages
List of Documents
No ratings yet
List of Documents
8 pages
Teaching Vocabulary: Presentation For Methods in Tesl
100% (2)
Teaching Vocabulary: Presentation For Methods in Tesl
19 pages
Supervision of Extra-Class Activities
90% (10)
Supervision of Extra-Class Activities
4 pages
EDU 480 - Assignment Guidelines (Project)
No ratings yet
EDU 480 - Assignment Guidelines (Project)
5 pages
No Child Left Behind and Students With Disabilities
No ratings yet
No Child Left Behind and Students With Disabilities
10 pages
TESOL - Ass 2
No ratings yet
TESOL - Ass 2
2 pages
Tos Midterm 2021
No ratings yet
Tos Midterm 2021
1 page
Chapter 7 Succession Management - C07 - BelcourtSHRP7e - PPT C
No ratings yet
Chapter 7 Succession Management - C07 - BelcourtSHRP7e - PPT C
90 pages
Homework Finder
100% (1)
Homework Finder
5 pages
Drug Education and Mass Media
No ratings yet
Drug Education and Mass Media
4 pages
Educ 312 Prelim Module
100% (1)
Educ 312 Prelim Module
32 pages
Accomplishment Report - Ivy 2
No ratings yet
Accomplishment Report - Ivy 2
1 page
Public Speaking Competition Concept Paper
No ratings yet
Public Speaking Competition Concept Paper
5 pages
19-Criteria For Weekly Evaluation of The Student Teachers Performance
No ratings yet
19-Criteria For Weekly Evaluation of The Student Teachers Performance
2 pages
Staff Brunswick High School
No ratings yet
Staff Brunswick High School
1 page
Francis Nichole A. Borja
No ratings yet
Francis Nichole A. Borja
11 pages

Talis 2008

Uploaded by

Talis 2008

Uploaded by

TALIS 2008 Technical Report

TALIS 2008 Technical Report

Teaching And Learning International Survey

Corrigenda to OECD publications may be found on line at: www.oecd.org/publishing/corrigenda.

TALIS 2008 Technical Report © OECD 2010

List of abbreviations and acronyms.........................................................................................................................15

Chapter 1 Introduction and Summary..................................................................................................................17

Chapter 2 Overview of TALIS 2008 and Framework Development.................................................23

Chapter 3 Development of Teacher and Principal Questionnaires..........................................29

Chapter 4 Translation and Cultural Adaptation......................................................................................45

TALIS 2008 Technical Report © OECD 2010

International translation verification. ........................................................................................................................................51

Chapter 5 Sample Design.........................................................................................................................................................55

Chapter 6 Survey Operations Procedures...........................................................................................................65

Chapter 7 On line Data Collection.............................................................................................................................73

Chapter 8 Quality Assurance.............................................................................................................................................89

© OECD 2010 TALIS 2008 Technical Report

Survey Activities Questionnaire....................................................................................................................................................94

Chapter 9 Creating and Checking the International Database...............................................103

Chapter 10 Estimation Weights, Participation Rates and Sampling Error. ...................115

Chapter 11 Construction and validation of scales and indices............................................131

TALIS 2008 Technical Report © OECD 2010

Annex A TALIS consortium, experts and consultants..........................................................................207

Annex B Characteristics of National Samples. ...........................................................................................211

Annex C Sampling Forms......................................................................................................................................................237

Annex D TALIS Principal and Teacher Questionnaire.............................................................................245

© OECD 2010 TALIS 2008 Technical Report

Figure 3.1 Survey themes for TALIS.............................................................................................................................................. 32

Figure 4.1 Countries and languages participating in TALIS. .............................................................................................. 48

Figure 5.1 TALIS international target and survey populations.......................................................................................... 57

Figure 6.1 Responsibilities of NPMs and the SC during survey administration........................................................ 68

Figure 7.1 Architectural overview of the SurveySystem...................................................................................................... 76

Figure 9.1 Overview of iterative data processing at the IEA DPC.................................................................................107

Figure 11.1 Illustration of CFA parameters. ...............................................................................................................................137

TALIS 2008 Technical Report © OECD 2010

Table 8.1 Preliminary activities of the School Co-ordinator............................................................................................ 99

Table 10.1  Quality ratings and unweighted participation rates......................................................................................126

Table 11.1 Item wording for school autonomy indices......................................................................................................152

© OECD 2010 TALIS 2008 Technical Report

TALIS 2008 Technical Report © OECD 2010

© OECD 2010 TALIS 2008 Technical Report

TALIS 2008 Technical Report © OECD 2010

© OECD 2010 TALIS 2008 Technical Report

TALIS 2008 Technical Report © OECD 2010

TALIS 2008 Technical Report © OECD 2010

Management of the study

© OECD 2010 TALIS 2008 Technical Report

Standardised procedures to ensure high-quality data

National centres and National Project Managers

Main tasks of the National Project Managers

TALIS 2008 Technical Report © OECD 2010

Phases of the TALIS study

© OECD 2010 TALIS 2008 Technical Report

TALIS 2008 Technical Report © OECD 2010

© OECD 2010 TALIS 2008 Technical Report

Overview of TALIS 2008

TALIS 2008 Technical Report © OECD 2010

The rationale for and aims of TALIS

© OECD 2010 TALIS 2008 Technical Report

The TALIS survey programme design

TALIS linked to PISA

The conceptual and analytic framework of the programme

TALIS 2008 Technical Report © OECD 2010

Policy Issue 1 Attracting teachers to the profession

Choosing the policy focus for TALIS

© OECD 2010 TALIS 2008 Technical Report

Ranking Indicator number Indicator

TALIS 2008 Technical Report © OECD 2010

OECD (2008), Education at a Glance 2008 – OECD Indicators, OECD, Paris.

© OECD 2010 TALIS 2008 Technical Report

TALIS 2008 Technical Report © OECD 2010

© OECD 2010 TALIS 2008 Technical Report

Timeline and phases of questionnaire development

In summary, the main steps in the development of the questionnaires were:

Frameworks guiding questionnaire development

TALIS 2008 Technical Report © OECD 2010

Table 8.1 Preliminary activities of the School Co-ordinator............................................................................................ 99

Table 10.1 Quality ratings and unweighted participation rates......................................................................................126

Table 11.1 Item wording for school autonomy indices......................................................................................................152