Principles of Language

Assessment

Daniela Bettoni Méndez

Valeria Carboni J iménez
How do you know if a test is effective? Does it
accurately measure what you want it to
measure?

There are five cardinal criteria for testing a test, which are,
practicality, reliability, validity, authenticity, and washback.
 1. Practicality

An effective test can be considered practical.

• Not excessively expensive.
• Stays within appropriate time constraints.
• Easy to administer.
• Has a scoring/evaluation procedure that is specific
and time-efficient.
 1. Practicality
Practical Test: it does not consume time or money than necessary to
accomplish the objective, it does not take a student five hours to complete the
test, it is not a test that only takes minutes to complete and hours to evaluate, it I
not a test that is only scored by computer; otherwise, the test is impractical.

Example: If a test has 20 listening items based on an audio tape, and 80 item on
grammar, vocabulary, reading comprehension, and multiple choice format, and a
scoring grid that accompanies the test, is considered practical.
 2. Reliability

• A reliable test is consistent and dependable.

• If you give the same test to the same student, the results are going to be
similar.

• There happens to be some fluctuations,

1. Student related reliability: A student’s score can be deviated from the

true score by illness, a bad day, anxiety, and other physical and psychological
factors.
 2. Reliability
2. Rater Reliability: Human error, bias, and subjectivity may enter into the scoring
process.

• Inter-rater reliability: When two or more scorers yield inconsistent scores of the
same test, because of lack of attention, inexperience, inattention, and preconceived
biases. When two scorers are not applying the same standards, like the example before.

• Intra-rater reliability: Teachers with unclear scoring criteria, fatigue, bias toward
particular good and bad students, or carelessness. The first revision of tests with the last
ones is different because the teachers get more tired, and the result can be an
inconsistent evaluation across all tests. The careful specification of analytical scoring can
increase rater reliability.
 2. Reliability
3. Test Administration Reliability: The conditions in which the test is
administrated. Listening comprehension with a lot of noise from the streets,
and students sitting next to windows cannot hear well. Photocopying
variations, too dark, too blurry, the amount of light in different parts of the
room, variations in temperature, and the conditions of chairs and desks.

4. Test Reliability: There could be measurement errors. If a test is too long,

it could cause fatigue from the test-takers, and they could respond
incorrectly. Timed tests may discriminate with students who do not perform
well in a test with a time limit. Even test answers that are ambiguous may
cause trouble.
 3. Validity
• Most complex criterion and most important principle
• “The extent to which inferences made from assessment results are appropriate,
meaningful, and useful in terms of the purpose of the assessment”

Example:
• A valid test of reading ability actually measures reading ability, not a 20/20 vision
test
• To measure writing ability, one might ask students to write as many words as they
can in 15 minutes, and then simply count the words for the final score. This is easy
to apply but it is not considered a valid test of writing ability without some
consideration of comprehension.
 3. Validity
How is the validity of a test established?

There is no final measure of validity, but there are certain aspects that one can take in
consideration:
• Examine the extent to which a test calls for performance that matches that of the
course or unit of study being tested
• Being concerned of how well a test determines whether or not students have
reached an established set of goals or level of competence
• Statistical correlation with other related but independent measures
• Focus on the consequences of a test or even the test-takers perception of validity
 Types of evidence to establish the validity
of a test
1. Content-Related Evidence
• If a test samples the subject matter about which conclusions are to be drawn, and if it
requires the test taker to perform the behavior that is being measured, it can claim content-
related evidence of validity
• You can identify it if you can clearly define the achievement that you are measuring
Example:
• If you are trying to assess a person’s ability to speak a second language in a conversational
meaning, asking the learner to answer paper-and-pencil multiple choice questions regarding
grammatical judgements dos not achieve content validity. A test that requires the learner to
actually speak within some sort of authentic context does.
 Types of evidence to establish the validity
of a test
1. Content-Related Evidence
• Cases of highly specialized and sophisticated testing instruments: questionable
content validity.

• The standard language proficiency tests lack content validity since they do not
require the full spectrum of communicative performance on the part of the
learner

• What such proficiency tests lack In content validity, they may gain in other
forms of evidence
 Types of evidence to establish the validity
of a test
1. Content-Related Evidence
Understand the difference between direct and indirect testing
• Direct testing: actually performing the target task.
• Indirect testing: performing a task that is related in some way.

Example: to test a learner oral production of syllable stress

• Test task: to have learners mark stressed syllables in a list of written words (indirect testing)
• Test task: require students to actually produce target words orally (direct testing)

The most feasible rule of thumb for achieving content validity is to test performance directly.
 Types of evidence to establish the validity
of a test
2. Criterion-Related Evidence
• The extent to which the criterion of the test has actually been reached
• Specified classroom objectives are measured, and implied predetermined levels of
performance are expected to be reached
• In the case of teacher-made classroom assessments, criterion-related evidence is
demonstrated through a comparison of results of an assessment with results of some
other measure of the same criterion

Example: In a course units where students are asked to be able to produce orally voiced and
voiceless stops in all phonetic environments, the results of one teacher’s test unit might be
compared with an independent assessment of the same phonemic proficiency.
 Types of evidence to establish the validity
of a test
2. Criterion-Related Evidence
• It falls into one of two categories: concurrent and predictive validity.
• Concurrent validity: if a tests results are supported by other concurrent performance
beyond the assessment itself

Example: the validity of a high score on the final exam of a foreign language course will be
sustained by actual proficiency in the language

• Predictive validity: it becomes important in the case of placement tests, admissions

assessment batteries, language aptitude tests. It assess a test taker’s likelihood of
future success
 Types of evidence to establish the validity
of a test

3. Construct-Related Evidence
• A construct is any theory, hypothesis or model that attempts to explain
observed phenomena in our universe of perceptions

• Constructs may or may not be empirically measured

• Every issue in language learning and teaching involves theoretical

constructs

• In the field of assessment, construct validity asks: Does this test actually
tap into the theoretical constructs as it has been defined?
 Types of evidence to establish the validity
of a test

3. Construct-Related Evidence

• Construct validity is a major issue in validating large-scale standardized tests of

proficiency.

• Such tests adhere to the principle of practicality and because they must sample a
limited number of domains of language, they may not be able to contain all the
content of a particular skill

• The TOEFL has until recently not attempted to sample oral production, yet, oral
production is an important part of the test.
 Types of evidence to establish the validity
of a test

4. Consequential Validity

• It encompasses all the consequences of a test, including such considerations

as its accuracy in measuring intended criteria, its impact on the preparation of
test-takers, its effect on the learner and the social consequences of a test
interpretations

• One aspect: the effect of test preparation courses and manuals on

performance. It reflects on socioeconomic conditions such as opportunities for
coaching.

• Another consequence of a test falls into the category of washback

 Types of evidence to establish the validity
of a test

5. Face Validity
• The extent in which students view the assessment as fair, relevant and useful
for improving learning

• The degree in which a test looks right and appears to measure the knowledge
or abilities it claims to measure

• Sometimes students do not know what is being tested.

• They may feel that a test is not testing what it is supposed to test.

• Face validity means that the students perceive the test to be valid.
 Types of evidence to establish the validity
of a test
5. Face Validity
Face validity will be high if students encounter:

• A well constructed format with familiar tasks

• A test that is doable within the time limit

• Items and instructions that are clear

• Tasks related to their course work

• Difficulty level that is reasonable

 Types of evidence to establish the validity
of a test
5. Face Validity
• Face-validity is not something that can be empirically tested by a teacher or
even by a testing expert

• Purely a factor of the “eye of the beholder”

• Some assessment experts view face validity as a superficial factor that is

dependent on the eyes of the receiver

• The other side of this issue reminds us the psychological state of the learner
 4. Authenticity
• The degree of correspondence of the characteristics of a given language test
task to the features of a target language task.

• Claim for authenticity: task is likely to be enacted in the real world.

• Many test items fail to simulate real-world tasks

• Authenticity has increased noticeably

• Reading passages are selected from real-world sources

• Listening comprehension sections feature natural language

 4. Authenticity
In a test, authenticity may be presented in the following ways:

• The language is as natural as possible

• Items are contextualized

• Topics are meaningful

• Some thematic organization to items is provided

• Tasks represent real world tasks

 5. Washback
• The effect of testing on teaching and learning
• It refers to the effects the tests have on instruction in terms of how students
prepare for the test
• A form of washback that occurs in classroom assessment is the information that
“washes back” to students in the form of useful diagnoses of strengths and
weaknesses
• Effects of an assessment on teaching and learning prior to the assessment itself
• Informal performance assessment is more likely to have built-in washback effects
because the teacher is usually providing interactive feedback
• Formal tests can also have positive washback, but they provide no washback if
students only receive a grade.
 5. Washback
• Challenge to teachers: to create classroom tests that serve as learning devices
through which washback is achieved

• Students incorrect responses: windows of insight into further work.

• The correct responses need to be praised

• It enhances a number of basic principles of language acquisition: intrinsic

motivation, autonomy, self-confidence, language ego, interlanguage and
strategic investment

• The teacher must comment generously and specifically on task performance

 5. Washback
• Letter grades and numerical scores: no information of intrinsic interest. They
foster a competitive, not cooperative, learning

Tips for teachers:

• When you return a written test, consider giving more than a number as your
feedback.

• Give praise for strengths, as well as constructive criticism for weaknesses.

• Give strategic hints on how a student might improve

• Give the student a motivating experience in which he will gain a sense pf

accomplishment and challenge
 5. Washback
• Students have ready access to you to discuss the feedback and evaluation you
have given

• An interactive, cooperative, collaborative classroom can promote an

environment of dialogue between students and teachers regarding evaluation
aspects

• Students need to have a chance to feed back on your feedback to seek

clarification of any issue
 Applying principles to the evaluation of
classroom tests

Are the test procedures practical?

• Determined by students and teachers’ time constraints, costs,

administrative details. Practicality Checklist*

• Teachers should avoid the temptation of only adding multiple

choice items that might not be even well designed.

• Teachers must give feedback to students, comments and

suggestions on their tests.
 Applying principles to the evaluation of classroom
tests

Is the test reliable?

• It applies to both the test and the teacher. Students have to receive the
same quality of input, whether written or auditory.

• Clean photocopies of the test, the sound amplification is clearly audible,

video input is visible, there is good temperature, lightning, optimal
classroom conditions, and equality for all students.

• Intra-rater reliability for open-ended responses has to follow the

next guidelines: use consistent sets of criteria for a correct response, give
uniform attention to students during the test, read through tests at least
twice to check for your consistency.
 Applying principles to the evaluation of classroom tests

Does the procedure demonstrate content validity?

• The students have to perform tasks that were included in the previous
classroom lessons and that represent the objectives of the unit seen.
• Are classroom objectives identified? A valid objective would be this one
Valid
Students will produce yes/no questions with final rising intonation
Invalid
Students should be able to demonstrate some reading comprehension.
Practice vocabulary in context because it is ambiguous and no standards are
implied.

• Test specifications means that a test should have a structure by dividing it

into a number of sections that correspond to the objectives assessed.
 Applying principles to the evaluation of classroom tests

• Is the procedure face valid and ‘’biased for best’’?

• The test must have clear instructions, appropriate timing, there are
no surprises, and the test is logically organized.

• ‘’Biased for best’’ refers to giving students proper review of the

content seen, suggesting strategies that can be helpful for them,
helps students who have more difficulty and those who do not.

• It is important to give instructions before, during and after the test

to put all the ideas of the test clear. This makes the students
comfortable and it is valid.
 Applying principles to the evaluation of
classroom tests

Are the test tasks as authentic as possible?

• The language has to be natural, the items need to
approximate real world tasks, and the items need to
be contextualized.
• Contextualizing the items for example in multiple
choice, allows students to follow a story line.
 Applying principles to the evaluation of classroom tests

Does the test offer beneficial washback to the learner?

• Preparation time before the test.
• The learners can review the content, and after the test, the
students find their weaknesses and strengths.
• Self-assessment can help students discover their own
mistakes, and this is effective for writing performance.
• Peer discussion of the test results is an alternative, and even
with the teacher.
 Bibliography

Brown, D. (n.d.). Language assessment: Principles and classroom practices.

Retrieved from
http://images.pcmac.org/Uploads/JeffersonCountySchools/JeffersonC
ountySchools/Departments/DocumentsCategories/Documents/Langu
age%20Assessment%20-
%20Principles%20and%20Classroom%20Practices.pdf
Thank you!