Topic 4-

Assessment of EIL
Contents
 Key terms
• Measurement • What are the usual
• Test standards of
• Assessment language
competence?
• evaluation
 Assessment vs Test

• Assessment: collecting and interpreting

information about learners/test takers’
attributes
• Test: an instrument of
assessment/collecting scores
 Test Development Procedure
Lead-in question: How do you (think people)
develop a test?
Test Development Procedure
Watch a video about test development

Do Activity 1, handout
 Stage 1: Test design
(Bachman & Palmer, 1996)

• identifying assessment purposes

• identifying target language abilities (communicative
language competence)
• identifying and describing the distinctive characteristics of
the real-life language use tasks (authenticity)
• describing the prospective test takers’ characteristics
• making a list of available and required resources
• planning the allocation and management of resources
• working out a plan for evaluating test usefulness.
 Stage 2: Test construction
(Bachman & Palmer, 1996)

• Writing test specification (structure of the assessment,

target candidates, types and topics of texts, task types,
language skills and elements, test techniques, time
allotment, criteria assessment, scoring procedure, etc.)
• Writing task specifications
• Selecting/ Writing materials
• Writing test items
 Stage 3: Test try-out
(Bachman & Palmer, 1996)

• Trialing the test on a sample of target learners

• Analyzing the piloted test results (use of CTT or IRT)
• Calibrating items and scales (if any)
• Collecting qualitative and quantitative feedback (testing
environment, administrative procedure, instructions, test-
takers’ perceptions of the test tasks, time allotment, etc.)
• Revising and refining the test
 Stage 4: Test implementation
(Bachman & Palmer, 1996)

• Preparing materials for test operation

• Training testing staff, interviewers, proctors, scorers, etc.
• Marking the test
• Reporting scores
• Validating the test
 Principles of test usefulness
(Bachman & Palmer, 1996)
Reliability: consistency of measurement across parallel forms of a test,
different administrations of the same test, and/or different
raters;
Construct the extent to which we can interpret a given test score as an
validity indicator of the ability(ies), or construct(s), we want to
measure

Authenticity the extent to which the characteristics of a test task

correspond to those of a target language use task

Interactiveness “the extent and type of involvement of the test taker’s

individual characteristics in accomplishing a test task

Impact/ the influences of a test on individuals (e.g., test takers and

washback teachers), institutional practices, educational systems (i.e.,
washback), and society at large
Practicality the extent to which the available resources exceed the
resources required for developing and implementing a test
Approaches in the
assessment of EIL/
ELF/ WE
 Reading and researching
• Group 1: make a mind map of the features, limitations and
strengths of the weak approach in EIL assessment. Is this
approach visible in VSTEP?
• Group 2: make a mind map of the features, limitations and
strengths of the strong approach in EIL assessment. Is
this approach visible in VSTEP?
• Group 3: Please find more evidence in Reading 1 and 2
that international tests have demonstrated some attempt
toward EIL assessment. Are the attempts weak or strong?
• Group 4: EIL assessment principles and reasons. Are
they feasible in Vietnam?
 The ‘weak’ approach
1. The target language norms of EIL tests: a
standard native-speaker variety or one based on
such variety.
2. Texts used in such tests are screened for potential
(topic) bias against users of other English varieties
3. Gloss [add a note to explain difficult word] or avoid
altogether any lexical items or structures which
are likely to be unfamiliar to non-native users.

4.
 The ‘weak’ approach
4. Use interlocutors (either examiners or other
candidates) who are expert NNS/ELF users and
therefore have experience in ELF contexts
5. Train raters to be lenient (only those errors
which result in miscommunication are penalized)
6. Involve ELF users in standard setting exercises
(deciding the passing levels or the excellent levels in
tests).

1.
The ‘weak’ approach: the case of IELTS & TOEFL
Attempts of change found in IELTS
•the use of reading and listening texts that reflect social and
regional (restricted to the Inner Circle) language variations
•the incorporation of material writers from the UK, Australia,
and New Zealand
•the inclusion of proficient nonnative speakers as examiners
for the oral and written tests
Attempts of change found in TOEFL
•the inclusion of accented non-native speakers in the
listening section
 Limitations of the ‘weak’ approach
• The continued dominance of native-speaker norms in
test design and construction
• The irrelevance of the tests to multilingual test takers’
communicative needs: inappropriate test use
• Even native speakers do not have the highest bands in
the same test for native and non-native English speakers
• The possible reinforcement of a deficit view of nonnative
English users’ competence by positive discrimination
(e.g pragmatic competence to communicate L2-L2)
 The ‘strong’ approach

• Sample test items directly from domains (task types,

topics, genres) relevant to EIL/WE/ELF communication.
• Focus on performance tasks that are evaluated in terms
of functional effectiveness or task fulfillment
• Give priority to strategic competence over formal
accuracy
• Use pair or group tasks on various communicative roles
• include peer assessment
 Benefits of the ‘strong’ approach

• A more equal picture of Outer- and Expanding- Circle

varieties of English
• A greater meaningfulness and relevance of the chosen
language norms to test takers from communities outside
the Inner Circle
• More valid representations of target language use
domains
• Positive washback on the teaching of endonormative
standards (the way the language is used by local
speakers, not from the country where the language
comes from)
 Limitations of the ‘strong’ approach

• A description of ELF competence is unavailable to define

construct of an English test. (Davies, 2009).
• ELF competence must still be localized and not globalized
– difficult to do
• The question of linguistic imperialism: ELF competence
may still oppress other languages/other local EIL.
 IMPLICATIONS
AND SUGGESTED
PRINCIPLES
Principles for assessment of EIL
1. Targets: Determine the English norms for a test
according to its intended use
Eg: TOEIC for business to North American partners; TOEFL and
IELTS for academic pathways.
2. Targets: Choose the most popular standard variety
of English if more than one variety is adequate
3. Input for test construction: Provide candidates with
exposure to multiple native and non-native varieties of
English
4. Targets: Broaden the construct of EIL tests to
incorporate intercultural strategic competence
5. Make allowances for individual aspirations to Inner-
circle norms
Practice
 Discussion 1 (Principle 1)
• Investigate high-stake tests in Vietnam,
their purposes, prevailing linguistic norms,
the alignment between these norms and
the purposes
Tests Purposes Norms Alignment
(variety/genres/
communication
strategies, topics,
tasks)
VSTEP
Highschool exit
Tests Purposes Norms Alignment

VSTEP Academic American Yes and no

studies English and
Working Bristish English
qualification Topics:
Teacher academic
license English (EAP)

High school Certifying k12 British English yes

exit graduates Topics: familiar
Admitting to students in
students to the curriculum
universities/coll 2018
eges
Discussion 2: How EIL oriented
are some international tests?
• Group 1: Investigate a PTE test for
evidence of EIL assessment
• Group 2: Investigate a VSTEP test for
evidence of EIL assessment
• Group 3: Investigate the CPE test for
evidence of EIL assessment
• Contents for evaluation:
– Material evaluation principles
– Assessment principles
Cooke (2020)
Watch a video on PTE
 Discussion 3: Practical
applications
• Look at a specific high school exit test in
2022 and make practical suggestions on
how to change it to reflect the EIL
assessment principles and the realities of
English used in the country
 Discussion 4: What are the
challenges to realize the teaching
and assessment of EIL in
Vietnam? What can be solutions?
 Learners’ challenges
• Learners’ challenges if EIL perspectives are
undertaken in Vietnam
– Grammar-based exams in Vietnam in contrast to the
communicative focus of ELT in Vietnam and the
focused targets in EIL
– Learners do not have exposure to different varieties
and may have negative attitudes to non-native
varieties
– Learners may have difficulty distinguishing spoken n
written language
– Learners may react negatively to new targets such as
intelligibility and pragmatic competence and new
teaching methods of eil such as L1 use
• Teachers/Testers’ challenges if having to
apply eil
– Limited training for teachers available for the
large number of teachers in Vietnam
– Negative attitude and misunderstanding from
students and parents about teachers’
proficiency and about focus of teaching
– Skills and time in material adaptation
• Parent’s challenges if eil becomes a real
– Struggle between the common and the new
perspectives
– No clear and qualified instruction to help
children to learn according to eil at home
• Policy makers’ challenges to apply Eil
– Uneven students’ english competence across
the countries: easier to apply eil in certain
areas
– Can’t make a standard and overall eil methods
and materials for all vietnamese students
– Need to organize teachers training: we need a
high quality teacher workforce to teach EIL
– Need to redesign test according to eil: who is
going to design the test?
 Elder and Harding (2008)
• Tests need to have validity, fairness and
accountability while the in teaching EIL,
norms are fluid => unfair to test takers who
would not know what kinds of language
use were acceptable, what sources to
draw upon in preparing themselves to
pass a test, or what standards were being
applied (contruct validity)
 Elder and Harding (2008)
• Including non-native accents may lead to
unfairness to those not familiar with that
accent
• The test takers want to be recognized to
be capable in a variety/SE of wider
acceptability than in a local variety
(eg GSLPA in Hongkong, or SEAMEO test
for teachers of English in Indonesia)
 Attempts in define EIL
proficiency in test (Brown, 2019)