January 2022

Measurement Strategy
for SDG Global Indicator
4.4.2 using International
Largescale Assessments
MEASUREMENT STRATEGY FOR SDG GLOBAL
INDICATOR 4.4.2 USING INTERNATIONAL
LARGESCALE ASSESSMENTS1

1This paper has been prepared by Andrés Sandoval-Hernández, Eliana Osorio-Saez, Nurullah Eryilmaz (University
of Bath) as part of the UNESCO Institute for Statistics (UIS) methodological agenda.

1
Abstract
This document aims to describe and implement a measurement strategy for SDG Global
Indicator 4.4.2 using International Large-Scale Assessments (ILSAs) in Education. To do that,
this document is divided into three main sections. In the first one, we identify a global content
framework for the indicator based on existing mapping exercises. The second section
evaluates the extent to which the concepts contained in the content framework can be
measured with the instruments and procedures of existing ILSAs. In the third section, the
document presents a proposal to define proficiency levels for the indicator together with a
set of tables showing the average percentage of students who reach the SDG indicator 4.4.2
in each of the countries for which there is available data. Finally, we conclude with an
overview of the four sections described above and their limitations.

2
Contents
Introduction ........................................................................................................................ 5

Global indicator .............................................................................................................. 5

Thematic indicators ........................................................................................................ 5

A. Global content framework for SDG thematic indicator 4.4.2 .................................... 6

A.1 Concepts ................................................................................................................... 6

A.2 Operationalization ................................................................................................... 6

B. Mapping existing tools from ILSAs into SDG thematic indicator 4.4.2 ................... 12

B.1 PISA - OECD - The ICT Familiarity Questionnaire ................................................ 12

B.2 ICILS ......................................................................................................................... 12

B.3 PIAAC ....................................................................................................................... 13

C. SDG thematic indicator 4.4.2 ...................................................................................... 15

C.1 Availability ............................................................................................................... 18

C.2 Calculation method................................................................................................ 18

C.3 Definition of cut-off points (standards) ............................................................... 19

Conclusions ...................................................................................................................... 23

Bibliography...................................................................................................................... 25

Appendix 1. ....................................................................................................................... 26

Appendix 2. ....................................................................................................................... 31

3
List of tables
Table 1. Global Content Framework for SDG indicators 4.4.2 8
Table 2. First cycle of PIAAC and countries participating in each rou 13
Table 3. Selected test items to measure indicator 4.4.2. 16
Table 4. Selected test items to measure indicator 4.4.2. 16
Table 5. Selected test items to measure indicator 4.4.2. 16
Table 6. Mapping of PIAAC 2012-2017 scales into the indicator categories 17
Table 7. Description of the PSTRE proficiency levels. 20
Table 8. Data disaggregation. 31
Table 9. Proportion of youth/adults who have achieved SDG 4.4.2. 33
Table 10. Proportion of youth/adults who have achieved SDG 4.4.2. by sex 34
Table 11. Proportion of youth/adults who have achieved SDG 4.4.2. by SES 35
Table 12. Proportion of youth/adults who have achieved SDG 4.4.2. by age 36
Table 13. Proportion of youth/adults who have achieved SDG 4.4.2. by education 37

List of figures
Figure 1. Core dimensions of PIAAC's PSTRE tasks. 15
Figure 2. Proportion of youth/adults who have achieved SDG 4.4.2. 22
Figure 3. Screenshot 1 of sample item 1. 26
Figure 4. Screenshot 2 of sample item 1. 27
Figure 5. Screenshot 3 of sample item 1. 28
Figure 6. Screenshot 1 of sample item 2. 30

4
Introduction
The Sustainable Development Goals (SDGs) are the blueprint to achieve a better and
more sustainable future for all. They address the most important global challenges
we face, including those related to poverty, inequality, climate, environmental
degradation, prosperity, and peace and justice. The agenda contains 17 goals
including a global education goal (SDG4). SDG4 establishes that by 2030 we have to
“Ensure inclusive and equitable quality education and promote lifelong learning
opportunities for all”. SDG4 contains 10 specific targets. One of these targets, 4.4
(skills for work), sets the goal to “substantially increase the number of youth and
adults who have relevant skills, including technical and vocational skills, for
employment, decent jobs and entrepreneurship” (UNESCO, 2020, p. 247) by 2030.
Target 4.4 has one global and five thematic indicators.
Global indicator
4.4 By 2030, substantially increase the number of youth and adults who have relevant
skills, including technical and vocational skills, for employment, decent jobs and
entrepreneurship
Thematic indicators
4.4.1 Proportion of youth/adults with information and communications
technology (ICT) skills, by type of skill.
4.4.2 Percentage of youth/adults who have achieved at least a minimum level
of proficiency in digital literacy skills
4.4.3 Youth/adult educational attainment rates by age group, economic
activity status, levels of education and programme orientation
In this document, we focus on the second thematic indicator (4.4.2) which refers to
learning outcomes that are achieved as a result of the development of digital literacy
skills in the contexts described in the global indicator. The main objective of this
document is to describe and implement a measurement strategy for this thematic
indicator using data from International Large-Scale Assessments (ILSAs) in education.
Apart from this introduction, this document is divided into four sections. The first
section is dedicated to identifying a global content framework for indicator 4.4.2
based on a mapping exercise of existing International Large-Scale Assessments. In
the second section, we evaluate the extent to which the different concepts included
in the global content framework can be measured with the instruments and
procedures of existing International Large-Scale Assessments (ILSAs). The third
section describes the methods we used to produce scores and thresholds to

5
measure and monitor the progress towards SDG 4.4.2. Finally, the last section is
dedicated to the discussion and conclusions.

A. Global content framework for SDG thematic indicator 4.4.2

A.1 Concepts
As mentioned above, indicator 4.4.2 refers to the proportion of youth/adults who
have achieved at least a minimum level of proficiency in digital literacy skills (DLS).
In this section, we establish an operational definition of DLS that will constitute the
base of a global content framework for the construction of specific indicators.
There is a clear consensus on the importance of digital literacy. This is evidenced by
the many national and global initiatives to promote the development of citizens'
digital literacy. However, there is no definite agreement on the definition of this
concept. While some frameworks understand it as a new form of literacy, some
others consider it to be a combination of pre-existent forms of literacy (e.g.
traditional literacy and computer literacy) (Ala-Mutka, 2011).
Reaching consensus on a definition of digital literacy is not an easy task. Digital
literacy has a distinct history within UNESCO and beyond; and it is considered as an
umbrella concept that encompasses a broad range of knowledge, skills, attitudes,
values, identities and behaviours. To establish a unique operational definition, we
build on the Digital Literacy Global Framework (Law et al., 2018) and the
Recommendations on Assessment Tools for Monitoring Digital Literacy (Laanpere,
2019). We used these two reports to establish both an operational definition of DLS
and a global content framework for indicator 4.4.2. In reviewing a wide array of
related frameworks, Law and colleagues (2018, p. 6) found that the following notions
were recurrent: access, manage, understand, integrate, communicate, evaluate and
create. Hence, they propose (and we adopt) the following definition for digital
literacy:
Digital literacy is the ability to access, manage, understand, integrate, communicate,
evaluate and create information safely and appropriately through digital technologies for
employment, decent jobs and entrepreneurship. It includes competences that are
variously referred to as computer literacy, ICT literacy, information literacy and media
literacy.
A.2 Operationalization
As mentioned before, we have adopted the Digital Literacy Global Framework
developed by Law and colleagues (2018) for this project. This global content
framework (see Table 1) is based on the extensive work already conducted by
UNESCO to define and operationalise Digital Literacy (DL); and adopts the definitions

6
and operationalizations proposed in the European Commission’s Digital Competence
Framework for Citizens (Carretero et al., 2017).

7
Table 1. Global Content Framework for SDG indicators 4.4.2
Competence areas and
Description
competences

0. Devices and software To identify and use hardware tools and technologies. To identify data, information and digital content
operations** needed to operate software tools and technologies.

0.1 Physical operations of To identify and use the functions and features of the hardware tools and technologies.
digital devices**

0.2 Software operations in To know and understand the data, information and/or digital content that are needed to operate software
digital devices** tools and technologies.

1. Information and data To articulate information needs, to locate and retrieve digital data, information and content. To judge the
literacy relevance of the source and its content. To store, manage and organise digital data, information and
content.

1.1 Browsing, searching and To articulate information needs, to search for data, information and content in digital environments, to
filtering data, information access them and to navigate between them. To create and update personal search strategies.
and digital content

1.2 Evaluating data, To analyse, compare and critically evaluate the credibility and reliability of sources of data, information and
information and digital digital content. To analyse, interpret and critically evaluate the data, information and digital content.
content

1.3 Managing data, To organise, store and retrieve data, information and content in digital environments. To organise and
information and digital process them in a structured environment.
content

2. Communication and To interact, communicate and collaborate through digital technologies while being aware of cultural and
collaboration generational diversity. To participate in society through public and private digital services and participatory
citizenship. To manage one’s digital identity and reputation.

2.1 Interacting through To interact through a variety of digital technologies and to understand appropriate digital communication

8
 Competence areas and
Description
competences

digital technologies means for a given context.

2.2 Sharing through digital To share data, information and digital content with others through appropriate digital technologies. To act as
technologies an intermediary, to know about referencing and attribution practices.

2.3 Engaging in citizenship To participate in society through the use of public and private digital services. To seek opportunities for self-
through digital technologies empowerment and for participatory citizenship through appropriate digital technologies.

2.4 Collaborating through To use digital tools and technologies for collaborative processes and for co-construction and co-creation of
digital technologies resources and knowledge.

2.5 Netiquette To be aware of behavioural norms and know-how while using digital technologies and interacting in digital
environments. To adapt communication strategies to the specific audience and to be aware of cultural and
generational diversity in digital environments.

2.6 Managing digital identity To create and manage one or multiple digital identities, to be able to protect one's own reputation, to deal
with the data that one produces through several digital tools, environments and services.

3. Digital content creation To create and edit digital content. To improve and integrate information and content into an existing body of
knowledge while understanding how copyright and licenses are to be applied. To know how to give
understandable instructions for a computer system.

3.1 Developing digital To create and edit digital content in different formats, to express oneself through digital means.
content

3.2 Integrating and re- To modify, refine, improve and integrate information and content into an existing body of knowledge to
elaborating digital content create new, original and relevant content and knowledge.

3.3 Copyright and licences To understand how copyright and licences apply to data, information and digital content.

3.4 Programming To plan and develop a sequence of understandable instructions for a computing system to solve a given
problem or perform a specific task.

9
 Competence areas and
Description
competences

4. Safety To protect devices, content, personal data and privacy in digital environments. To protect physical and
psychological health, and to be aware of digital technologies for social well-being and social inclusion. To be
aware of the environmental impact of digital technologies and their use.

4.1 Protecting devices To protect devices and digital content, and to understand risks and threats in digital environments. To know
about safety and security measures and to have due regard to reliability and privacy.

4.2 Protecting personal data To protect personal data and privacy in digital environments. To understand how to use and share
and privacy personally identifiable information while being able to protect oneself and others from damages. To
understand that digital services use a “Privacy policy” to inform how personal data is used.

4.3 Protecting health and To be able to avoid health risks and threats to physical and psychological well-being while using digital
well-being technologies. To be able to protect oneself and others from possible dangers in digital environments (e.g.
cyberbullying). To be aware of digital technologies for social well-being and social inclusion.

4.4 Protecting the To be aware of the environmental impact of digital technologies and their use.
environment

5. Problem-solving To identify needs and problems and to resolve conceptual problems and problem situations in digital
environments. To use digital tools to innovate processes and products. To keep up to date with the digital
evolution.

5.1 Solving technical To identify technical problems when operating devices and using digital environments, and to solve them
problems (from trouble-shooting to solving more complex problems).

5.2 Identifying needs and To assess needs and to identify, evaluate, select and use digital tools and possible technological responses to
technological responses solve them. To adjust and customise digital environments to personal needs (e.g. accessibility).

5.3 Creatively using digital To use digital tools and technologies to create knowledge and to innovate processes and products. To
technologies engage individually and collectively in cognitive processing to understand and resolve conceptual problems
and problem situations in digital environments.

10
 Competence areas and
Description
competences

5.4 Identifying digital To understand where one’s own digital competence needs to be improved or updated. To be able to support
competence gaps others with their digital competence development. To seek opportunities for self-development and to keep
up-to-date with the digital evolution.

5.5 Computational To process a computable problem into sequential and logical steps as a solution for human and computer
thinking** systems.

6. Career-related To operate specialised digital technologies and to understand, analyse and evaluate specialised data,
competences** information and digital content for a particular field.

6.1 Operating specialised To identify and use specialised digital tools and technologies for a particular field.
digital technologies for a
particular field**

6.2 Interpreting and To understand, analyse and evaluate specialised data, information and digital content for a particular field
manipulating data, within a digital environment.
information and digital
content for a particular
field**

Source: Law, et al. (2018)

Note: In the competence areas and competences column, text in bold indicates competence areas and the plain text indicates
competences.
** Added competence areas and competences which are not in the DigComp 2.0 framework.

11
B. Mapping existing tools from ILSAs into SDG thematic indicator
4.4.2
Once the operational definition of DLS was established and operationalised into a
Digital Literacy Global Framework, the next step was to map existing tools from
different ILSAs into the concepts included in the framework. The objective was to
evaluate if and to what extent existing instruments and data could be used to
measure and monitor SDG 4.4.2.
This mapping exercise used the following strategy. To carry out the mapping of this
indicator we used the following analytic strategy:
First, informed by the operationalizations identified above, we consulted the latest
version of the frameworks and the instruments applied in several ILSAs. These were
the Programme for International Student Assessment (PISA) (OECD, 2003), the IEA
International Computer and Information Literacy Study (ICILS) (Fraillon et al., 2019)
and the OECD Programme for the International Assessment of Adult Competencies
(PIAAC) (OECD, 2012). We assessed these sources of information with the following
criteria in mind: the assessment framework should (at least partially) refer to the
concepts relevant to SDG 4.4.2, the instruments should provide sufficient
information on many of the aspects/concepts involved, and they should potentially
allow long-term monitoring.
Below is an overview of the International Large-Scale Assessments (ILSAs) considered
for this exercise and a summary of our evaluation.
B.1 PISA - OECD - The ICT Familiarity Questionnaire
PISA 2003 introduced the ICT familiarity computer-based student questionnaire as
an option and gave it to all participating 15-year-olds students if a country/economy
chose to use it. Among the questions included were those regarding students' digital
and electronic device usage, as well as their confidence and attitudes towards ICT.
PISA offers limited coverage for the indicator 4.4.2. Furthermore, when contrasting
the Global Content Framework for SDG indicators 4.4.2 and the items in the ICT
familiarity questionnaire, only three out of the six competence areas are compatible.
Due to the above fundamental reasons, PISA was disregarded as the tool for
measuring the 4.4.2 indicator.
B.2 ICILS
In response to the growing use of ICT in modern society, the IEA developed the
International Computer and Information Literacy Study (ICILS). There have been two
cycles, the first one in 2013 focused on the use of computers as information

12
searching, management, and communication tools, which are vital to participating in
the digital age. While the second one in 2018 looked into students' computational
thinking, along with their computer and information literacy (Fraillon et al., 2019).
Although the ILCIS measured a broad range of competences, including some
concepts directly related to indicator 4.4.2, it only assessed eighth graders.
Consequently, this resource was dismissed.
B.3 PIAAC
The OECD's International Assessments of Adult Competencies (PIAAC) were
administered between 2012-2017 in 39 countries/economies. This first cycle was
divided into three rounds of data collection. (See Table 2).
Table 2. First cycle of PIAAC and countries participating in each rou
Round 1 (2011-2012) Australia, Austria, Belgium (Flanders),
Canada, Czech Republic, Denmark,
Estonia, Finland, France, Germany,
Ireland, Italy, Japan, Korea, Netherlands,
Norway, Poland, Russian Federation,
Slovak Republic, Spain, Sweden, United
Kingdom (England and Northern
PIAAC 1st Cycle Ireland), United States.

Round 2 (2014-15) Chile, Greece, Indonesia, Israel,

Lithuania, New Zealand, Singapore,
Slovenia, Turkey.

Round 3 (2017) Ecuador, Hungary, Kazakhstan, Mexico,

Peru, United States.

PIAAC included both a background questionnaire and three cognitive assessments,

focusing on literacy, numeracy, and Problem-solving in technology-rich
environments (PSTRE).
The background questionnaire collects data about information and communication
technologies (ICTs), skills at work and at home; specifically how often individuals use
computers, their attitudes towards using technology, and their self-assessment of
how well they use technology.
PIAAC introduced PSTRE as a new domain, and it was the first time that it was
evaluated on a large scale and as a single component.

13
PSTRE is defined as using digital technology, communication tools and networks to
gather and evaluate information, communicate with others and complete tasks. The
first cycle of PIAAC emphasised “the abilities to solve problems for personal, work
and civic purposes by setting up appropriate goals and plans, and accessing and
making use of information through computers and computer networks” (OECD,
2012).
PSTRE encompasses the intersection of skills sometimes referred to as "computer
literacy" (i.e., using computer applications and tools) and cognitive skills needed to
resolve problems. Knowledge of basic ICT input devices (e.g., keyboards, mice, and
screens), file management tools, applications (e.g., word processing, email) and
graphic interfaces are essential for completing the assessment tasks. Rather, its
objective is to assess the capacity of adults to access, process, evaluate and analyze
information effectively using ICT tools and applications.
Our mapping exercise identified the OECD’s Programme for the International
Assessment of Adult Competencies (PIAAC) as the most valuable source of
information for SGD indicator 4.4.2. This study was chosen due to its conceptual
framework (OECD, 2012), which showed the highest coverage of the topics relevant
to this indicator. Additional reasons for the selection of PIAAC were that its target
population covers the two groups mentioned in the indicator (youth and adults); as
well as its potential to inform long-term monitoring.

14
C. SDG thematic indicator 4.4.2
The items used to operationalise SDG Indicator 4.4.2 were the ones corresponding
to the PIAAC’s dimension of Problem-Solving in Technology-Rich Environments. This
dimension refers to the ability to use technology to solve problems and accomplish
complex tasks. It is not a direct measure of computer literacy, as it also measures the
capacity to operate within a digital environment to solve the types of problems that
adults face in their everyday life as users of digital technologies (see OECD, 2012 for
more details).
In PIAAC, the Problem-Solving in Technology-Rich Environments (PSTRE) dimension
is measured with 14 tasks based on problem-solving scenarios, which are conceived
along three dimensions (see Figure 1Error! Reference source not found.).
Figure 1. Core dimensions of PIAAC's problem-solving in technology-rich environments
tasks.

Source: OECD (2012, p. 48).

PSTRE items were developed for testing participants' ability to handle tasks that may
involve multiple cognitive processes (see Table 3), multiple technologies (see Table
4), or different contexts (see Table 5). Participants may, for example, need to move
between email and spreadsheet environments when creating a table that represents
the information for a specific purpose either personal or work-related. The PSTRE
tasks are all scenario-based and vary in their difficulty.

15
Table 3. Selected test items to measure indicator 4.4.2 - Distribution of tasks as a function
of the cognitive processes measured by PIAAC.

Cognitive processes Number of items 1

Goal setting and monitoring progress 4

Planning 7

Acquiring and evaluating information 8

Making use of information 6

1
The number of items do not sum to 14 because some tasks are coded to more than one cognitive
process.
Source: Literacy, Numeracy and Problem-Solving in Technology-Rich Environments: Framework for the
OECD Survey of Adult Skills. Paris: OECD, 2012. Page 52.

Table 4. Selected test items to measure indicator 4.4.2 - Distribution of tasks as a

function of technology environments.
Technology Environments Number of items 1

Web 7
Spreadsheet 4
E-mail 9

1
The number of items do not sum to 14 because some tasks are coded to more than one technology
environment.
Source: Literacy, Numeracy and Problem-Solving in Technology-Rich Environments: Framework for the
OECD Survey of Adult Skills. Paris: OECD, 2012. Page 52.

Table 5. Selected test items to measure indicator 4.4.2 - Distribution of tasks by context.
Context Number of items

Personal 8
Work / Occupation 4
Civic 2

Source: Literacy, Numeracy and Problem-Solving in Technology-Rich Environments: Framework for the
OECD Survey of Adult Skills. Paris: OECD, 2012. Page 52.

16
In summary, PIAAC-PSTRE instruments contain items/tasks which overlap with all the
areas of competences described in the framework for SDG indicator 4.4.2 (see Table
6, OECD, 2012 and Law et al., 2018).

Table 6. Mapping of PIAAC 2012-2017 scales into the indicator categories

Areas of Competences Framework for SDG indicator 4.4.2

PIAAC 2012- Devices &

Digital Career-
2018 Construct software Inf. & data Comms. & content Problem-
Safety related
operation literacy collab. solving
creation comp.
s

Setting
goals &
x x x x x
monitorin
g progress
Cognitive
process Planning x x x x

Acquiring
x x x x x
& eval inf.

Using inf. x x

Web x x

Technology Spread
x x
environment sheet

E-mail x x

Personal x x
Context Work x x

Civic x x

By intrinsic Single
x x
complexity step
(num of Multiple
steps) x x
steps

Single
By intrinsic constraint x x
complexity
(num of Multiple
constraints) constraint x x
s

Examples of the specific items/tasks used for the PSTRE dimension of PIAAc can be
found in Annex 1.

17
C.1 Availability
PIAAC data has been collected in 40 countries/economies over three cycles between
2011 and 2017. However, PSTRE data was available only for 31 countries: Austria,
Belgium, Canada, Czech Republic, Denmark, Estonia, Finland, Germany, Ireland,
Japan, Republic of Korea, Netherlands, Norway, Poland, Russian Federation, Slovak
Republic, Sweden, United Kingdom, Chile, Greece, Israel, Lithuania, New Zealand,
Singapore, Slovenia, Turkey, Ecuador, Hungary, Kazakhstan, Mexico, Peru.
C.2 Calculation method
Since the test design for PIAAC is based on a variant of matrix sampling (using
different sets of items, multistage adaptive testing, and different assessment modes)
where each respondent was administered a subset of items from the total item pool.
The responses to the subset of test items are scaled using item response theory (IRT)
methodology and combined with other background information (provided by the
respondent) and model parameters to produce a set of 10 plausible values (PVs).
These PVs can be used to produce group-level estimations of proficiency values
(OECD, 2013).
According to the PIAAC Technical Report (OECD, 2013), the following steps can be
followed to calculate an estimate Τ of the proficiency values Θ using PVs and to
calculate an estimate of the variance of Τ:
1. Using the first vector of plausible values for each respondent, evaluate Τ as if
the plausible values were the true values of Θ. Denote the result Τ1 .
2. In the same manner as in step 1 above, evaluate the sampling variance of Τ,
or Var(Τ1 ), with respect to respondents’ first vectors of plausible values.
Denote the result Var1.
3. Carry out steps 1 and 2 for the second through all 10 vectors of plausible
values, thus obtaining Τ𝜐𝜐 and Var𝜐𝜐 for 𝜐𝜐=2,. . ., 10.
4. The best estimate of T obtainable from the plausible values is the average of
the 10 values obtained from the different sets of plausible values:

∑𝜐𝜐 Τ𝜐𝜐
Τ. =
10 (1)

18
 5. An estimate of the variance of Τ is the sum of two components: an estimate
of Var(Τ𝜐𝜐 ) obtained as in step 4 and the variance among the Τ𝜐𝜐 s:

∑𝑛𝑛 𝑉𝑉𝑉𝑉𝑉𝑉𝜐𝜐 1 ∑𝜐𝜐(Τ𝜐𝜐 − Τ. )2

𝑉𝑉𝑉𝑉𝑉𝑉Τ. = + �1 + � (2)
10 10 10 − 1

The first component in 𝑉𝑉𝑉𝑉𝑉𝑉Τ. reflects uncertainty due to sampling from the
population; the second component reflects uncertainty because the
respondents' proficiencies Θ are only indirectly observed.
Then, using the cut-off points established for the scale (see below), the proportion of
students respondents reaching the corresponding standard is estimated within each
country or region as a simple proportion (P).

𝑋𝑋
𝑃𝑃 =
𝑛𝑛 (3)

Where 𝑋𝑋 is the number of respondents that reach the standard in each country and
𝑛𝑛 is the total number of respondents in the same country.
C.3 Definition of cut-off points (standards)
The performance of the participants in PIAAC-PSTRE is used to produce a proficiency
scale (i.e., score) that ranges from 0 to 500. This scale is then divided into four
proficiency levels (i.e., below 1, 1, 2 and 3) based on the knowledge and skills required
to complete the tasks within those levels. Respondents at a particular level not only
demonstrate knowledge and skills associated with that level but also the
proficiencies required at lower levels. So, for example, respondents scoring at Level
2 are also proficient at Level 1.
To create the proficiency levels, an expert group in problem-solving in technology-
rich environments met with psychometricians and test developers and reviewed
data, looked at the tasks along the 500-point scale, and determined the requisite
skills and knowledge to complete those tasks progressively increased along the scale.
These proficiency levels of PSTRE are defined as shown in Table 7.

19
Table 7. Description of the PSTRE proficiency levels.
Below Level 1 (0 to240 score points)
Tasks are based on well-defined problems involving the use of only one function within a generic
interface to meet one explicit criterion without any categorical, inferential reasoning or
transforming of information. Few steps are required and no subgoal has to be generated.

Level 1 (241 to 290 score points)

At this level, tasks typically require the use of widely available and familiar technology applications,
such as email software or a Web browser. There is little or no navigation required to access the
information or commands required to solve the problem. The problem may be solved regardless
of one’s awareness and use of specific tools and functions (e.g., a sort function). The task involves
few steps and a minimal number of operators. At a cognitive level, the person can readily infer the
goal from the task statement; problem resolution requires one to apply explicit criteria; there are
few monitoring demands (e.g., the person does not have to check whether he or she has used the
adequate procedure or made progress toward the solution). Identifying contents and operators can
be done through simple match; only simple forms of reasoning, for example, assigning items to
categories are required. There is no need to contrast or integrate information.

Level 2 (291 to 340 score points)

At this level, tasks typically require the use of both generic and more specific technology
applications. For instance, the person may have to make use of a novel online form. Some
navigation across pages and applications is required to solve the problem. The use of tools (e.g., a
sort function) can facilitate the resolution of the problem. The task may involve multiple steps and
operators. In terms of cognitive processing, the problem goal may have to be defined by the person,
though the criteria to be met are explicit. There are higher monitoring demands. Some unexpected
outcomes or impasses may appear. The task may require evaluating the relevance of a set of items
to discard distractors. Some integration and inferential reasoning may be needed.

Level 3 (341 to 500 score points)

At this level, tasks typically require the use of both generic and more specific technology
applications. Some navigation across pages and applications is required to solve the problem. The
use of tools (e.g., a sort function) is required to make progress toward the solution. The task may
involve multiple steps and operators. In terms of cognitive processing, the problem goal may have
to be defined by the person, and the criteria to be met may or may not be explicit. There are typically
high monitoring demands. Unexpected outcomes and impasses are likely to occur. The task may
require evaluating the relevance and the reliability of information in order to discard distractors.
Integration and inferential reasoning may be needed to a large extent.

Source: PIAAC Technical Report (OECD, 2013)

By comparing the definition of SDG Indicator 4.4.2 and the description of the
problem-solving in technology-rich environments proficiency levels, we identified
Level 2 as the threshold or cut-off point to estimate the proportion of respondents
reaching the indicator within each country. At Level 2, tasks typically require the use
of both generic and more specific technology applications.

20
At the threshold, respondents typically require the use of both generic and specific
technology applications. Adults at this level are typically able to use software they
have never seen before to solve problems, even when unexpected
impasses/outcomes occur. For example, they are likely able to:
• Figure out how to send an email message to a number of contacts using an
unfamiliar bulk email function;
• Use a sorting tool to make it easier to locate sales numbers for a specific
product in a company spreadsheet;
• Conduct a web search to find out how to solve a problem with other software,
such as how to view a column that won’t display properly in a spreadsheet;
and
• Find an email message or file that has been “lost” somewhere on a computer
hard drive.
The proportion of youth/adults who reach the SDG 4.4.2 in each of the countries for
which data is available is shown in Figure 2. Tables with the proportions of
youth/adults reaching SDG 4.4.2 in each country disaggregated by sex,
socioeconomic status, age and educational level can be found in Annex 2.

21
Figure 2. Proportion of youth/adults who have achieved at least a minimum level of
proficiency in digital literacy skills.

NEW ZEALAND 0.44

SWEDEN 0.44
NETHERLANDS 0.42
FINLAND 0.42
NORWAY 0.41
DENMARK 0.39
SINGAPORE 0.37
CANADA 0.37
GERMANY 0.36
UNITED KINGDOM 0.35
JAPAN 0.35
BELGIUM 0.35
CZECH REPUBLIC 0.33
AUSTRIA 0.32
KOREA, REP. OF 0.30
HUNGARY 0.28
ESTONIA 0.28
ISRAEL 0.27
SLOVAK REPUBLIC 0.26
RUSSIAN FEDERATION 0.26
SLOVENIA 0.25
IRELAND 0.25
POLAND 0.19
LITHUANIA 0.18
KAZAKHSTAN 0.16
CHILE 0.15
GREECE 0.14
MEXICO 0.10
TURKEY 0.08
PERU 0.07
ECUADOR 0.05
0.00 0.10 0.20 0.30 0.40 0.50
Proportion who reach SDG 4.4.2

22
Conclusions
Our analysis suggests that the items, scales and potential indicators (scores) based
on the PSTRE dimension of PIAAC can be used to measure and monitor progress
towards SDG 4.4.2. PIAAC-PSTRE is certainly well suited for providing (at least a proxy)
measurement of SDG 4.4.2. It provides high coverage for the digital literacy
competences included in our Global Content Framework and incorporates the
relevant concepts naturally in its frameworks. It also systematically collects the same
data across countries and would allow long-term monitoring (provided that there are
further cycles of PIAAC). Finally, it has high-quality data quality assurance
mechanisms in place (ensuring data accuracy, validity and comparability).
Nevertheless, some aspects must be kept in mind when interpreting the scores and
proportions presented here.
In very simple terms, cut-off scores refer to a point in a scale used to classify
individuals according to the level of the attribute being measured between those
above and below a threshold. As such, this threshold should represent a meaningful
interpretation of the level of the attribute under study, in this case, “digital literacy
skills”. In other words, individuals scoring above the threshold should be able to
demonstrate “a minimum level of proficiency in digital literacy skills”. We have
decided to follow the methodology proposed by the OECD to determine the
thresholds for SDG Indicator 4.4.2. That is, we have selected proficiency Level 2 of
the scale “problem-solving in technology-rich environments” as the threshold or cut-
off point. Additionally, we have described what this threshold means according to
the PIAAC framework (e.g., the types of tasks that can be completed by adults who
reach the threshold). The selection and interpretation of this particular threshold are,
however, open to discussion among the relevant stakeholders (see OECD, 2013 for
details on the methodology and description of the proficiency levels).
PIAAC data are uniquely suited to contribute to measuring SDG Indicator 4.4.2
because its methods ensure that comparable information is collected across all
participating countries. This is a significant advantage compared to the alternative of
compiling and harmonizing national datasets or developing a purpose-built study.
However, it is important to keep in mind that PIAAC was not designed to measure
SDG Indicator 4.4.2. For this reason, the information used here has limitations
related to at least two areas: availability (e.g. the country coverage), and relevance
(e.g. the scales produced here can only be considered as proxy measures of the
concepts established in SDG Indicator 4.4.2).
Finally, it is important to consider that the “problem-solving in technology-rich
environments” assessment was not done if the respondent had insufficient
computer skills, or if the respondent opted to do a paper-and-pencil-based
23
assessment, or if the respondent did not do the computer assessment for literacy-
related reasons. As a result, there are missing values that are not addressed through
imputation or weighting—as their characteristics are different from those that did
complete the assessment. The estimates reported here assume that the individuals
that, for any of the three reasons described above, did not complete the assessment
did not reach the target established by SDG Indicator 4.4.2. We believe that this is a
reasonable assumption since those individuals who have insufficient computer or
literacy skills to answer the test are extremely unlikely to reach proficiency level 2 if
they had taken the test. However, there is some degree of uncertainty because they
did not take the “problem-solving in technology-rich environments” assessment.

24
Bibliography
Ala-Mutka, K. (2011). Mapping Digital Competence: Towards a Conceptual
Understanding (No. JRC67075-2011). European Commission.
Carretero, S., Vuorikari, R., & Punie, Y. (2017). DigComp 2.1: The Digital Competence
Framework for Citizens with eight proficiency levels and examples of use (EUR
28558 EN). European Comission.
http://svwo.be/sites/default/files/DigComp%202.1.pdf
Fraillon, J., Ainley, J., Schulz, W., Duckworth, D., & Friedman, T. (2019). IEA
International Computer and Information Literacy Study 2018 Assessment
Framework. Springer Open. 10.1007/978-3-030-19389-8
Laanpere, M. (2019). Recommendations on assessment tools for monitoring digital
literacy within unesco’s digital literacy global framework. UNESCO Institute for
Statistics. http://uis.unesco.org/sites/default/files/documents/ip56-
recommendations-assessment-tools-digital-literacy-2019-en.pdf
Law, N., Woo, D., de la Torre, J., & Wong, G. (2018). A global framework of reference on
digital literacy skills for indicator 4.4. 2 (UIS/2018/ICT/IP/51). UNESCO Institute
for Statistics. http://uis.unesco.org/sites/default/files/documents/ip51-global-
framework-reference-digital-literacy-skills-2018-en.pdf
OECD. (2003). PISA 2003 Assessment Framework: Mathematics, Reading, Science and
Problem Solving and Knowledge Skills. OECD Publishing.
http://dx.doi.org/10.1787/9789264281820-en
OECD. (2012). Literacy, Numeracy and Problem Solving in Technology-Rich
Environments. Framework for the OECD Survey of Adult Skills. OECD Publishing.
https://doi.org/10.1787/9789264128859-en
OECD. (2013). Technical Report of the Survey of Adult Skills (PIAAC). OECD Publishing.

25
Appendix 1.
The examples of the PSTRE items presented below are taken from the PIAAC
Framework (OECD, 2012, pp. 53–55). Please note that these items are administered
in electronic format, and these examples correspond to the print layout.
Sample item 1
In this item, respondents must access and evaluate information in the context of a
simulated job search. The instructions, located on the left side of the screen, require
respondents to identify and then bookmark one or more sites that do not require
users to register or pay a fee.

Figure 3. Screenshot 1 of sample item 1.

26
Figure 4. Screenshot 2 of sample item 1.

As can be seen, this item requires that respondents work within a simulated web
environment that includes tools and functionality similar to those found in real-life
applications. users are able to:
• Click on links on both the results page and associated web pages;
• Navigate using the back and forward arrows or the home icon; and
• Bookmark web pages and view or change those bookmarks.

27
Figure 5. Screenshot 3 of sample item 1.

In order to perform this task correctly, respondents may have to search through
several pages on a website. One of the features of PIAAC is that the process and
paths by which a respondent responds to the tasks are captured. For example, one
of the websites, presented below, does not meet the criteria of not requiring
registration or the payment of a fee, but the relevant information is not on the
opening page. If a respondent bookmarks this site without clicking on the “Learn
more” link to view the relevant information (see the website on the following page),
this response may be interpreted in a different way than if the relevant page had
been viewed. The breadth of information, combined with frameworks that specify
behaviours of interest, allow PIAAC to learn more about what adults know and can
do relative to the construct of problem-solving.
The relevant information is located on the form that indicates that users must sign
up (register) and pay a fee.

28
Sample item 2
In this item, respondents select a set of files to download onto a portable music
player. The files must meet specified criteria in terms of genre (jazz and rock) and not
exceed the capacity of the device (maximum of 20 MB).
The software includes an automatic summing functionality (“total Size Selected”) that
facilitates the task by updating the total file size as files are selected or de-selected.
Respondents must monitor progress as they select files, checking against the
specified criteria to know when they have satisfied the constraints presented in the
problem.
It is also possible to sort the spreadsheet by file size and/or genre, a strategy that can
improve task efficiency. The connection between the use of resources in a
technology-rich environment and resulting efficiencies for solving problems is
emphasised in the framework and therefore included across items in the
assessment.

29
Figure 6. Screenshot 1 of sample item 2.

30
Appendix 2.
Table 8. Data disaggregation.
DEFINITION METRICS ITEM AND DESCRIPTION CATEGORIES INSTRUMENT

Person resolved gender from Female, Male, Not stated or inferred Background
Sex Nominal
Background questionnaire (derived) (missing) questionnaire (link)

- No formal qualification or below ISCED 1

- ISCED 1

- ISCED 2
Which of the qualifications on this card - ISCED 3C shorter than 2 years
is the highest you have obtained?
- ISCED 3C 2 years or more
*Response categories were collapsed
into ‘Tertiary education’ (ISCED 5A, 5B - ISCED 3A-B
and 6); and ‘Non-tertiary education’
Educational - ISCED 3 (without distinction A-B-C, 2y+) Background
Ordinal (the rest).
level - ISCED 4C questionnaire (link)
* In order to account for the fact that
many of the youngest participants in - ISCED 4A-B
PIAAC are still in education, the
- ISCED 4 (without distinction A-B-C)
analysis here is restricted to adults
aged 25-65 years. - ISCED 5B
- ISCED 5A, bachelor degree

- ISCED 5A, master degree

- ISCED 6

- No formal qualification or below ISCED 1

Highest of mother or father's level of
Socioeconomic education (derived) - ISCED 1
status (SES) Background
Ordinal *Response categories were collapsed - ISCED 2
(parental questionnaire (link)
into ‘High SES’ or at least one parent
education) - ISCED 3C shorter than 2 years
with tertiary education (ISCED 5A, 5B
and 6); ‘Low SES’ or none of the - ISCED 3C 2 years or more

31
DEFINITION METRICS ITEM AND DESCRIPTION CATEGORIES INSTRUMENT
parents with tertiary education (the - ISCED 3A-B
rest).
- ISCED 3 (without distinction A-B-C, 2y+)
- ISCED 4C

- ISCED 4A-B
- ISCED 4 (without distinction A-B-C)

- ISCED 5B
- ISCED 5A, bachelor degree
- ISCED 5A, master degree
- ISCED 6

- 24 or less
- 25-34
Person resolved age from Background - 35-44
Questionnaire (derived)
- 45-54 Background
Age Ordinal *Response categories were collapsed
- 55 plus questionnaire (link)
into ‘Older adults’ (55 plus) and
‘Younger adults’ (the rest). - <16
- >65

- Not stated or inferred

32
Table 9. Proportion of youth/adults who have achieved at least a minimum level of
proficiency in digital literacy skills
Country Prcentage
Austria .32 (.01)
Belgium .35 (.01)
Canada .37 (.01)
Czech Republic .33 (.01)
Denmark .39 (.01)
Estonia .28 (.01)
Finland .42 (.01)
Germany .36(.01)
Ireland .25 (.01)
Japan .35 (.01)
Korea, Rep. of .30 (.01)
Netherlands .42 (.01)
Norway .41 (.01)
Poland .19 (.01)
Russian Federation .26 (.02)
Slovak Republic .26 (.01)
Sweden .44 (.01)
United Kingdom .35 (.01)
Chile .15 (.02)
Greece .14 (.01)
Israel .27 (.01)
Lithuania .18 (.01)
New Zealand .44 (.01)
Singapore .37 (.01)
Slovenia .25 (.01)
Turkey .08 (.01)
Ecuador .05 (.00)
Hungary .28 (.01)
Kazakhstan .16 (.01)
Mexico .10 (.01)
Peru .07 (.00)

33
Table 10. Proportion of youth/adults who have achieved at least a minimum level of
proficiency in digital literacy skills by sex
Male Female
Country Below Above Below Above
Austria .63 (.01) .37 (.01) .72 (.01) .28 (.01)
Belgium .63 (.01) .37 (.01) .68 (.01) .32 (.01)
Canada .63 (.01) .37 (.01) .64 (.01) .36 (.01)
Czech Republic .64 (.02) .36 (.02) .69 (.01) .31 (.01)
Denmark .60 (.01) .40 (.01) .63 (.01) .37 (.01)
Estonia .72 (.01) .28 (.01) .73 (.01) .27 (.01)
Finland .57 (.01) .43 (.01) .60 (.01) .40 (.01)
Germany .60 (.01) .40 (.01) .68 (.01) .32 (.01)
Ireland .73 (.01) .27 (.01) .76 (.01) .24 (.01)
Japan .60 (.01) .40 (.01) .71 (.01) .29 (.01)
Korea, Rep. of .67 (.01) .33 (.01) .72 (.01) .28 (.01)
Netherlands .55 (.01) .45 (.01) .62 (.01) .38 (.01)
Norway .56 (.01) .44 (.01) .62 (.01) .38 (.01)
Poland .79 (.01) .21 (.01) .82 (.01) .18 (.01)
Russian Federation .74 (.02) .26 (.02) .74 (.03) .26 (.03)
Slovak Republic .74 (.01) .26 (.01) .75 (.01) .25 (.01)
Sweden .54 (.01) .46 (.01) .58 (.01) .42 (.01)
United Kingdom .61 (.01) .39 (.01) .69 (.01) .31 (.01)
Chile .83 (.02) .17 (.02) .88 (.03) .12 (.02)
Greece .85 (.01) .15 (.01) .87 (.01) .13 (.01)
Israel .72 (.01) .28 (.01) .75 (.01) .25 (.01)
Lithuania .83 (.02) .17 (.01) .82 (.01) .18 (.01)
New Zealand .56 (.01) .44 (.01) .55 (.01) .45 (.01)
Singapore .60 (.01) .40 (.01) .66 (.01) .34 (.01)
Slovenia .74 (.01) .26 (.01) .75 (.01) .25 (.01)
Turkey .91 (.02) .09 (.01) .94 (.01) .06 (.01)
Ecuador .94 (.02) .06 (.01) .96 (.01) .04 (.01)
Hungary .70 (.01) .30 (.01) .73 (.01) .27 (.01)
Kazakhstan .84 (.02) .16 (.01) .84 (.02) .16 (.01)
Mexico .87 (.01) .13 (.01) .92 (.01) .08 (.01)
Peru .93 (.02) .07 (.01) .94 (.02) .06 (.01)

34
Table 11. Proportion of youth/adults who have achieved at least a minimum level of
proficiency in digital literacy skills by SES (low SES = none of the parents with tertiary
education, high SES = at least one parent with tertiary education)
Low High
Country Below Above Below Above
Austria .71 (.01) .29 (.01) .48 (.02) .52 (.02)
Belgium .71 (.01) .29 (.01) .39 (.01) .61 (.02)
Canada .71 (.01) .29 (.01) .49 (.01) .51 (.01)
Czech Republic .71 (.01) .29 (.01) .40 (.02) .60 (.03)
Denmark .69 (.01) .31 (.01) .44 (.01) .56 (.01)
Estonia .81 (.01) .19 (.01) .54 (.01) .46 (.01)
Finland .65 (.01) .35 (.01) .32 (.01) .68 (.02)
Germany .70 (.01) .30 (.01) .47 (.01) .53 (.01)
Ireland .80 (.01) .20 (.01) .52 (.01) .48 (.02)
Japan .73 (.01) .27 (.01) .48 (.01) .52 (.01)
Korea, Rep. of .75 (.01) .25 (.01) .46 (.01) .54 (.02)
Netherlands .64 (.01) .36 (.01) .36 (.01) .64 (.01)
Norway .67 (.01) .33 (.01) .40 (.01) .60 (.01)
Poland .85 (.01) .15 (.01) .55 (.02) .45 (.02)
Russian Federation .79 (.02) .21 (.02) .64 (.04) .36 (.03)
Slovak Republic .78 (.01) .22 (.01) .49 (.01) .51 (.03)
Sweden .65 (.01) .35 (.01) .37 (.01) .63 (.01)
United Kingdom .68 (.01) .32 (.01) .42 (.01) .58 (.02)
Chile .90 (.02) .10 (.01) .65 (.03) .35 (.03)
Greece .88 (.01) .12 (.01) .68 (.03) .32 (.03)
Israel .82 (.02) .18 (.01) .56 (.01) .44 (.01)
Lithuania .93 (.01) .07 (.01) .66 (.01) .34 (.02)
New Zealand .62 (.01) .38 (.01) .40 (.01) .60 (.01)
Singapore .69 (.01) .31 (.01) .34 (.01) .66 (.02)
Slovenia .81 (.02) .19 (.01) .46 (.02) .54 (.02)
Turkey .93 (.01) .07 (.01) .72 (.04) .28 (.04)
Ecuador .96 (.01) .04 (.00) .81 (.04) .19 (.03)
Hungary .79 (.01) .21 (.01) .45 (.01) .55 (.02)
Kazakhstan .88 (.02) .12 (.01) .73 (.02) .27 (.02)
Mexico .92 (.01) .08 (.01) .67 (.03) .33 (.03)
Peru .96 (.02) .04 (.00) .79 (.02) .21 (.02)

35
Table 12. Proportion of youth/adults who have achieved at least a minimum level of
proficiency in digital literacy skills by age (younger adults < 55, older adults ≥ 55)
younger older
Country Below Above Below Above
Austria .62 (.01) .38 (.01) .92 (.02) .08 (.01)
Belgium .59 (.01) .41 (.01) .88 (.02) .12 (.02)
Canada .58 (.01) .42 (.01) .83 (.01) .17 (.01)
Czech Republic .61 (.01) .39 (.01) .88 (.02) .12 (.02)
Denmark .54 (.01) .46 (.01) .87 (.01) .13 (.01)
Estonia .67 (.01) .33 (.01) .95 (.01) .05 (.01)
Finland .48 (.01) .52 (.01) .91 (.01) .09 (.01)
Germany .58 (.01) .42 (.01) .87 (.02) .13 (.02)
Ireland .71 (.01) .29 (.01) .95 (.01) .05 (.01)
Japan .57 (.01) .43 (.01) .90 (.02) .10 (.01)
Korea, Rep. of .64 (.01) .36 (.01) .96 (.01) .04 (.01)
Netherlands .52 (.01) .48 (.01) .83 (.02) .17 (.01)
Norway .53 (.01) .47 (.01) .86 (.02) .14 (.01)
Poland .76 (.01) .24 (.01) .97 (.05) .03 (.03)
Russian Federation .70 (.02) .30 (.03) .91 (.02) .09 (.02)
Slovak Republic .70 (.01) .30 (.01) .91 (.01) .09 (.01)
Sweden .49 (.01) .51 (.01) .83 (.02) .17 (.01)
United Kingdom .61 (.01) .39 (.01) .83 (.02) .17 (.02)
Chile .83 (.03) .17 (.02) .98 (.02) .02 (.01)
Greece .84 (.01) .16 (.01) .95 (.01) .05 (.01)
Israel .70 (.01) .30 (.01) .91 (.02) .09 (.01)
Lithuania .79 (.01) .21 (.01) .96 (.01) .04 (.01)
New Zealand .51 (.01) .49 (.01) .76 (.02) .24 (.02)
Singapore .56 (.01) .44 (.01) .94 (.01) .06 (.01)
Slovenia .69 (.01) .31 (.01) .95 (.01) .05 (.02)
Turkey .91 (.01) .09 (.01) .98 (.02) .02 (.01)
Ecuador .94 (.02) .06 (.01) .99 (.01) .01 (.01)
Hungary .66 (.01) .34 (.01) .91 (.02) .09 (.01)
Kazakhstan .82 (.01) .18 (.01) .92 (.02) .08 (.01)
Mexico .89 (.01) .11 (.01) .98 (.01) .02 (.01)
Peru .93 (.01) .07 (.01) .99 (.01) .01 (.00)

36
Table 13. Proportion of youth/adults who have achieved at least a minimum level of
proficiency in digital literacy skills by educational level
non-tertiary tertiary
Country Below Above Below Above
Austria .75 (.01) .25 (.01) .50 (.01) .50 (.01)
Belgium .82 (.01) .18 (.01) .46 (.01) .54 (.04)
Canada .79 (.01) .21 (.01) .54 (.01) .46 (.01)
Czech Republic .78 (.01) .22 (.01) .42 (.02) .58 (.03)
Denmark .76 (.01) .24 (.01) .45 (.01) .55 (.01)
Estonia .86 (.01) .14 (.01) .65 (.01) .35 (.01)
Finland .77 (.01) .23 (.01) .44 (.01) .56 (.01)
Germany .77 (.01) .23 (.01) .47 (.01) .53 (.02)
Ireland .89 (.01) .11 (.01) .56 (.01) .44 (.02)
Japan .80 (.01) .20 (.01) .51 (.01) .49 (.01)
Korea, Rep. of .89 (.01) .11 (.01) .56 (.01) .44 (.02)
Netherlands .74 (.01) .26 (.01) .37 (.01) .63 (.02)
Norway .75 (.01) .25 (.01) .41 (.01) .59 (.02)
Poland .93 (.01) .07 (.01) .63 (.02) .37 (.02)
Russian Federation .84 (.03) .16 (.03) .73 (.03) .27 (.02)
Slovak Republic .84 (.01) .16 (.01) .52 (.02) .48 (.02)
Sweden .70 (.01) .30 (.01) .39 (.01) .61 (.01)
United Kingdom .78 (.01) .22 (.01) .48 (.02) .52 (.02)
Chile .96 (.02) .04 (.01) .70 (.03) .30 (.03)
Greece .93 (.01) .07 (.01) .72 (.02) .28 (.02)
Israel .86 (.01) .14 (.01) .63 (.01) .37 (.02)
Lithuania .93 (.01) .07 (.01) .64 (.02) .36 (.02)
New Zealand .69 (.02) .31 (.01) .45 (.01) .55 (.01)
Singapore .91 (.01) .09 (.01) .46 (.01) .54 (.03)
Slovenia .87 (.01) .13 (.01) .52 (.01) .48 (.02)
Turkey .96 (.01) .04 (.00) .74 (.02) .26 (.03)
Ecuador .99 (.01) .01 (.00) .88 (.02) .12 (.02)
Hungary .85 (.01) .15 (.01) .48 (.01) .52 (.02)
Kazakhstan .90 (.02) .10 (.01) .76 (.02) .24 (.02)
Mexico .96 (.02) .04 (.01) .74 (.03) .26 (.02)
Peru .98 (.01) .02 (.00) .86 (.02) .14 (.01)

37