A COMPILATION MATERIAL OF

ENGLISH LANGUAGE
TESTING
Compiled By:
BERTARIA SOHNATA HUTAURUK

Prodi PendidikanBahasaInggris
FAKULTAS KEGURUAN DAN ILMU PENDIDIKAN
UNIVERSITAS HKBP NOMMENSEN
PEMATANGSIANTAR
2015

English Language Testing 1

 INTRODUCTION

This book is a compilation material for English Language

Testing. General outlines of material as an introduction to
English Language Testing that has been compiled for the
students in S1 degree. This collection of material consists
of Testing, Assessment, Meassurement, Evaluation, Kinds
of Testing, Validity and Reliability of the tests, and
Interpreting the test Score. Hopefully, this compilation
will be useful for the students and yet not perfect, so any
critism is welcomed.

Compiled by:
Bertaria Sohnata Hutauruk

English Language Testing 2

 CONTENS

1. What is the difference between assessment and evaluation?…………. 1

2. Testing, Assessment, Measurement and Evaluation…………………... 4

3. Informal vs. Formal Assessments: Tests are not the only end-all-be-all

of how we assess.………………………………………………………….. 6

4. Norm-referenced test and Criterion-referenced test…………………... 11

5. Discrete Point Testing and Integrative Testing………………………… 19

6. Communicative Language Testing……………………………………… 22

7. Testing Communicative Competence…………………………………… 24

8. Testing Reading and Writing……………………………………………. 30

9. Performance-Based Assessment……………………………………….... 32

10. Validity and Reliability…………………………………………………... 42

11. Constructing Tests……………………………………………………….. 61

12. Types of Listening Testing………………………………………………. 75

13. Testing Grammar………………………………………………………… 97

14. Interpreting Test Score…………………………………………………... 103

English Language Testing 3

 1

What is the difference between

assessment and evaluation?
There is a lot of confusion over these two terms as well as other terms associated with
assessment, testing, and evaluation. The big difference can be summarized as this:
assessment is information gathered by the teacher and student to drive instruction, while
evaluation is when a teacher uses some instrument (such as the CMT or an end-of-unit
test) to rate a student so that this information can be used to compare or sort students.
Assessment is for the student and the teacher in the act of learning while evaluation is
usually for others.

“If mathematics teachers were to focus their efforts on classroom assessment that is
primarily formative in nature, students’ learning gains would be impressive. These
efforts would include gathering data through classroom questioning and discourse,
using a variety of assessment tasks, and attending primarily to what students know and
understand” (Wilson & Kenney, page 55).

Assessment is a lot more important because it is integral to instruction. Unfortunately, it

is being hampered by the demands of evaluation. The biggest demand for evaluation is
grading or report cards. There shouldn’t be a problem with that, except historically
evaluation (grades) were determined exclusively by computing a student’s numeric
average on paper and pencil assessments called quizzes or tests.

“Most experienced teachers will say that they know a great deal about their students in
terms of what the students know, how they perform in different situations, their attitudes
and beliefs, and their various levels of skill attainment. Unfortunately, when it comes to
grades, they often ignore this rich storehouse of information and rely on test scores and
rigid averages that tell only a small fraction of the story.

English Language Testing 4

The myth of grading by statistical number crunching is so firmly ingrained in schooling
at all levels that you may find it hard to abandon. But it is unfair to students, to parents,
and to you as the teacher to ignore all of the information you get almost daily from a
problem-based approach in favor of a handful of numbers based on tests that usually
focus on low-level skills” (Van de Walle and Lovin, page 35).

The reason this is a problem is that students learn what is valued and they strive to do
well on those things. If the end-of-unit tests are what are used to determine your grade,
guess what kids want to do well on, the end-of-unit test! You can do all the great
activities you want, but if the bottom line is the test, then that is what is going to be
valued most by everyone: teachers, students, and parents, alike.

What we need to get better at is valuing the day-to-day

activities we do and learn how to use them for both
assessment and evaluation.

This will not be an easy task.

It is very different from what we are used to doing. We are used to teaching and then
assessing. In reality, the line between teaching and assessment should be blurred
(NCTM, 2000). “Interestingly, in some languages, learning and teaching are the same
word”(Fosnot and Dolk, page 1). We need to assess on a daily basis to give us the
information to make choices about what to teach the next day. If we just teach the
whole unit and wait until the end-of-unit test to find out what the kids know, we may be
very unhappily surprised. On the other hand, if we are assessing on a daily basis
throughout the unit, we do not need to average all those assessments to come up with a
final evaluation. Instead, we could just use the most recent assessments to make that
evaluation. In this way, we do not penalize the student that did not know much at the
beginning of the unit and worked really hard to learn what you felt were the big ideas.
Instead we rate them on where they are when you finished the unit. This gives a more
accurate report or evaluation of where they are performing when the evaluation is made.

English Language Testing 5

English Language Testing 6
 2

Testing, Assessment, Measurement and

Evaluation
The definition for each are:

Test: a method to determine a student’s ability to complete certain tasks or demonstrate

mastery of a skill or knowledge of content. Some types would be multiple choice tests,
or a weekly spelling test. While it is commonly used interchangeably with assessment
or even evaluation, it cab be distinguished by the fact that a test is one form of an
assessment.

Assessment: The process of gathering information to monitor progress and make

educational decisions if necessary. As noted an assessment may include a test, but also
includes methods such as obeservations, interviews, behavior monitoring etc.

Evaluation: Procedured used to determine whether the subject (i.e. student) meets a
preset criteria such as qualifying for special education services. This uses assessment
(remember that an assessement may be a test) to make a determination of qualification
in accordance with a predetrmined criteria.

Meassurement, beyond its general definition, refers to the set of procedures and the
principles for how to use the procedures in educational evaluations would be raw
scores, percentiles ranks, derrived scores, standard scores etc.

English Language Testing 7

English Language Testing 8
 3

Informal vs. Formal Assessments: Tests

are not the only end-all-be-all of how
we assess.
Formal assessment

Formal assessment uses formal tests or structured continuous assessment to evaluate a

learner's level of language. It can be compared to informal assessment, which involves
observing the learners' performance as they learn and evaluating them from the data
gathered.

Example
At the end of the course, the learners have a final exam to see if they pass to the next
course or not. Alternatively, the results of a structured continuous assessment process
are used to make the same decision.

In the classroom Informal and formal assessments are both useful for making valid and
useful assessments of learners' knowledge and performance. Many teachers combine the
two, for example by evaluating one skill using informal assessment such as observing
group work, and another using formal tools, for example a discrete item grammar test.

Formative assessment

Formative assessment is the use of assessment to give the learner and the teacher
information about how well something has been learnt so that they can decide what to
do next. It normally occurs during a course. Formative assessment can be compared

English Language Testing 9

with summative assessment, which evaluates how well something has been learnt in
order to give a learner a grade.

Example
The learners have just finished a project on animals, which had as a language aim better
understanding of the use of the present simple to describe habits. The learners now
prepare gap-fill exercises for each other based on some of their texts. They analyse the
results and give each other feedback.

In the classroom ,One of the advantages of formative feedback is that peers can do it.
Learners can test each other on language they have been learning, with the additional
aim of revising the language themselves. It has been once said that ““Everybody is a
genius. But if you judge a fish by its ability to climb a tree, it will live its whole life
believing that it is stupid.” Our students must be assessed relative to what their skills
are. It could be done by doing formal assessments or informal assessments or
combination of both.

I realized that beyond giving formal assessments (i.e. Summative assessments:

Quizzes, long tests, periodical exams, etc.), our main role as teachers is determined by

English Language Testing 10

how we recognize our students’ progress/stagnation through informal assessments (i.e.
formative assessments: port folios, role play, record tracking, etc.) These methods allow
the teacher to easily maneuver where and how his/her instuction is going.

The result of a formal test (e.g. long test) alone would not necessarily dictate the entire
academic ability of our students. It does not mean that when a student fails a formal test
(e.g. periodical test), we could already conclude that he’s entire learning capabilities for
that subject failed as well.

Assessing students is not monopolized by just doing it formally (e.g. giving out tests,
quizzes, summative exams, etc.), but rather depends on the other informal assessments
(e.g. coaching sessions, reflective logs, fly-by-question and answers, etc.) that reinforce
formal ones.

There are many factors why a student could fail from a test (e.g. lack of sleep,
emotional and family distress, etc.), but there would only be few factors why he/she
would not be able to provide a reflective insight on the lesson. But how do we separate
formal assessments from informal ones?

The table and concept map I incorporated below could give some help (you could click
the picture or open it in a new tab to see it clearer =).

English Language Testing 11

When are informal assessments useful (versus formal assessments)?

The most applicable time to use informal assessments is when:

1. We want to gauge the students cognitive, affective and manipulative skills in the
simplest way possible. We ask students to recite or write down essays to easily
determine if they understood a specific lesson well or poorly, if they are enthusiastic or
bored with the lesson, if they are already familiar or completely unfamiliar with the
topic, etc.

2. We deem that the results of the formal examinations are not enough to give a
concluding mark for the students’ performance. If a specific student performs
excellently in class activities but suddenly failed a summative test, it could tell us that
there could be a deviation between our formal against our informal assessments, or
other factors might have been involved with such event (e.g. student factor: did not
review, physically/emotionally troubled, etc.)

How valuable are informal assessments? Can informal assessments be

good replacements for formal assessments?

Although informal assessments provide teachers with solid bases of how the students
are performing, it would not imply that it could already replace formal assessments.
They should work hand-in-hand and interdependently. One should complement the
other.For instance, if we opt to use role plays and recitals in assessing students’
communications skills informally, we should also align our formal exams with the
activities our students previously engaged on. In this way, we could ensure validity and
fairness of our assessments. Moreover, we could find that these methods relieve our
burdens with analyzing, comparing, and understanding our students “true” abilities.

We cannot just give (formal) tests or quizzes in the same manner as we cannot just
consume course-time with just giving out (informal) class activities. Arriving at a valid
and reliable grades for our students is a combination of maximing both formal and
informal assessments.

English Language Testing 12

Tosummarize:Informal assessment being a systematic observation = knowing what,
when and where we are going to assess + (How) Establishing criteria for assessing
students

English Language Testing 13

 Norm-referenced test and Criterion-
referenced test
A norm-referenced test (NRT) is a type of test, assessment, or evaluation which yields
an estimate of the position of the tested individual in a predefined population, with
respect to the trait being measured. The estimate is derived from the analysis of test
scores and possibly other relevant data from a sample drawn from the population.That
is, this type of test identifies whether the test taker performed better or worse than other
test takers, not whether the test taker knows either more or less material than is
necessary for a given purpose.The term normative assessment refers to the process of
comparing one test-taker to his or her peers.Norm-referenced assessment can be
contrasted with criterion-referenced assessment and ipsative assessment. In a criterion-
referenced assessment, the score shows whether or not test takers performed well or
poorly on a given task, not how that compares to other test takers; in an ipsative system,
test takers are compared to previous performance.Alternative to normative testing, tests
can be ipsative, in which individuals' assessment is compared to themselves through
time.

By contrast, a test is criterion-referenced when provision is made for translating the test
score into a statement about the behavior to be expected of a person with that score. The
same test can be used in both ways. Robert Glaser originally coined the terms norm-
referenced test and criterion-referenced test.

Standards-based education reform is based on the belief that public education should
establish what every student should know and be able to do.Students should be tested
against a fixed yardstick, rather than against each other or sorted into a mathematical
bell curve.

By assessing that every student must pass these new, higher standards, education
officials believe that all students will achieve a diploma that prepares them for success
in the 21st century.Most state achievement tests are criterion-referenced. In other words,

English Language Testing 14

a predetermined level of acceptable performance is developed and students pass or fail
in achieving or not achieving this level. Tests that set goals for students based on the
average student's performance are norm-referenced tests. Tests that set goals for
students based on a set standard (e.g., 80 words spelled correctly) are criterion-
referenced tests.

Many college entrance exams and nationally used school tests use norm-referenced
tests. The SAT, Graduate Record Examination (GRE), and Wechsler Intelligence Scale
for Children (WISC) compare individual student performance to the performance of a
normative sample. Test takers cannot "fail" a norm-referenced test, as each testtaker
receives a score that compares the individual to others that have taken the test, usually
given by a percentile. This is useful when there is a wide range of acceptable scores that
is different for each college.

By contrast, nearly two-thirds of US high school students will be required to pass a

criterion-referenced high school graduation examination. One high fixed score is set at a
level adequate for university admission whether the high school graduate is college
bound or not. Each state gives its own test and sets its own passing level, with states like
Massachusetts showing very high pass rates, while in Washington State, even average
students are failing, as well as 80 percent of some minority groups. This practice is
opposed by many in the education community such as Alfie Kohn as unfair to groups
and individuals who score lower than others.

Advantages and limitations

An obvious disadvantage of norm-referenced tests is that it cannot measure progress of

the population as a whole, only where individuals fall within the whole. Thus,
measuring against only a fixed goal can be used to measure the success of an
educational reform program that seeks to raise the achievement of all students against
new standards that seek to assess skills beyond choosing among multiple choices.
However, while this is attractive in theory, in practice, the bar has often been moved in
the face of excessive failure rates, and improvement sometimes occurs simply because
of familiarity with and teaching to the same test.

English Language Testing 15

With a norm-referenced test, grade level was traditionally set at the level set by the
middle 50 percent of scores.By contrast, the National Children's Reading Foundation
believes that it is essential to assure that virtually all of read at or above grade level by
third grade, a goal which cannot be achieved with a norm referenced definition of grade
level.

Advantages to this type of assessment include that students and teachers know what to
expect from the test and just how the test will be conducted and graded. Likewise, all
schools will conduct the exam in the same manner, reducing such inaccuracies as time
differences or environmental differences that may cause distractions to the students.
This also makes these assessments fairly accurate as far as results are concerned, a
major advantage for a test.

Critics of criterion-referenced tests point out that judges set bookmarks around items of
varying difficulty without considering whether the items actually are compliant with
grade level content standards or are developmentally appropriate Thus, the original
1997 sample problems published for the WASL 4th grade mathematics contained items
that were difficult for college educated adults, or easily solved with 10th grade level
methods such as similar triangles.The difficulty level of items themselves and the cut-
scores to determine passing levels are also changed from year to year. Pass rates also
vary greatly from the 4th to the 7th and 10th grade graduation tests in some states.

One of the limitations of No Child Left Behind is that each state can choose or construct
its own test, which cannot be compared to any other state. A Rand study of Kentucky
results found indications of artificial inflation of pass rates which were not reflected in
increasing scores in other tests such as the NAEP or SAT given to the same student
populations over the same time.Graduation test standards are typically set at a level
consistent for native born 4 year university applicants.Unusual side effect is that while
colleges often admit immigrants with very strong math skills who may be deficient in
English, there is no such leeway in high school graduation tests, which usually require
passing all sections, including language. Thus, it is not unusual for institutions like the
University of Washington to admit strong Asian American or Latino students who did

English Language Testing 16

not pass the writing portion of the state WASL test, but such students would not even
receive a diploma once the testing requirement is in place.

Although the tests such as the WASL are intended as a minimal bar for high school, 27
percent of 10th graders applying for Running Start in Washington State failed the math
portion of the WASL. These students applied to take college level courses in high
school, and achieve at a much higher level than average students. The same study
concluded the level of difficulty was comparable to, or greater than that of tests
intended to place students already admitted to the college.

A norm-referenced test has none of these problems because it does not seek to enforce
any expectation of what all students should know or be able to do other than what actual
students demonstrate. Present levels of performance and inequity are taken as fact, not
as defects to be removed by a redesigned system. Goals of student performance are not
raised every year until all are proficient. Scores are not required to show continuous
improvement through Total Quality Management systems. Disadvantages include
standards based assessments measure the level that students are currently by measuring
against where their peers are currently at instead of the level that both students should
be at.

A rank-based system produces only data that tell which average students perform at an
average level, which students do better, and which students do worse, contradicting
fundamental beliefs, whether optimistic or simply unfounded, that all will perform at
one uniformly high level in a standards based system if enough incentives and
punishments are put into place. This difference in beliefs underlies the most significant
differences between a traditional and a standards based education system.

Examples

1. IQ tests are norm-referenced tests, because their goal is to see which test taker is
more intelligent than the other test takers.

English Language Testing 17

 2. Theater auditions and job interviews are norm-referenced tests, because their
goal is to identify the best candidate compared to the other candidates, not to
determine how many of the candidates meet a fixed list of standards.

A criterion-referenced test is one that provides for translating test scores into a
statement about the behavior to be expected of a person with that score or their
relationship to a specified subject matter. Most tests and quizzes that are written by
school teachers can be considered criterion-referenced tests. The objective is simply to
see whether the student has learned the material. Criterion-referenced assessment can be
contrasted with norm-referenced assessment and ipsative assessment.

A common misunderstanding regarding the term is the meaning of criterion. Many, if

not most, criterion-referenced tests involve a cutscore, where the examinee passes if
their score exceeds the cutscore and fails if it does not (often called a mastery test). The
criterion is not the cutscore; the criterion is the domain of subject matter that the test is
designed to assess. For example, the criterion may be "Students should be able to
correctly add two single-digit numbers," and the cutscore may be that students should
correctly answer a minimum of 80% of the questions to pass.

The criterion-referenced interpretation of a test score identifies the relationship to the

subject matter. In the case of a mastery test, this does mean identifying whether the
examinee has "mastered" a specified level of the subject matter by comparing their
score to the cutscore. However, not all criterion-referenced tests have a cutscore, and the
score can simply refer to a person's standing on the subject domain.The ACT is an
example of this; there is no cutscore, it simply is an assessment of the student's
knowledge of high-school level subject matter.Because of this common
misunderstanding, criterion-referenced tests have also been called standards-based
assessments by some education agencies,as students are assessed with regards to
standards that define what they "should" know, as defined by the state.

English Language Testing 18

Comparison of criterion-referenced and norm-referenced tests

Both terms criterion-referenced and norm-referenced were originally coined by Robert

Glaser.Unlike a criterion-reference test, a norm-referenced test indicates whether the
test-taker did better or worse than other people who took the test. For example, if the
criterion is "Students should be able to correctly add two single-digit numbers," then
reasonable test questions might look like " " or " " A criterion-
referenced test would report the student's performance strictly according to whether the
individual student correctly answered these questions. A norm-referenced test would
report primarily whether this student correctly answered more questions compared to
other students in the group. Even when testing similar topics, a test which is designed to
accurately assess mastery may use different questions than one which is intended to
show relative ranking. This is because some questions are better at reflecting actual
achievement of students, and some test questions are better at differentiating between
the best students and the worst students. (Many questions will do both.) A criterion-
referenced test will use questions which were correctly answered by students who know
the specific material. A norm-referenced test will use questions which were correctly
answered by the "best" students and not correctly answered by the "worst" students (e.g.
Cambridge University's pre-entry 'S' paper). Some tests can provide useful information
about both actual achievement and relative ranking. The ACT provides both a ranking,
and indication of what level is considered necessary to likely success in college. Some
argue that the term "criterion-referenced test" is a misnomer, since it can refer to the
interpretation of the score as well as the test itself.In the previous example, the same
score on the ACT can be interpreted in a norm-referenced or criterion-referenced
manner.

Sample scoring for the history question: What caused World War II?

Criterion-
Student answers referenced Norm-referenced assessment
assessment

Student #1: This answer is This answer is worse than Student #2's

English Language Testing 19

WWII was caused by Hitler correct. answer, but better than Student #3's
and Germany invading answer.
Poland.

Student #2:
WWII was caused by multiple
factors, including the Great
Depression and the general
economic situation, the rise of
This answer is This answer is better than Student #1's
nationalism, fascism, and
correct. and Student #3's answers.
imperialist expansionism, and
unresolved resentments
related to WWI. The war in
Europe began with the
German invasion of Poland.

Student #3:
WWII was caused by the This answer is This answer is worse than Student #1's
assassination of Archduke wrong. and Student #2's answers.
Ferdinand.

Relationship to high-stakes testing

Many high-profile criterion-referenced tests are also high-stakes tests, where the results
of the test have important implications for the individual examinee. Examples of this
include high school graduation examinations and licensure testing where the test must
be passed to work in a profession, such as to become a physician or attorney. However,
being a high-stakes test is not specifically a feature of a criterion-referenced test. It is
instead a feature of how an educational or government agency chooses to use the results
of the test.

English Language Testing 20

Examples

1. Driving tests are criterion-referenced tests, because their goal is to see whether
the test taker is skilled enough to be granted a driver's license, not to see whether
one test taker is more skilled than another test taker.
2. Citizenship tests are usually criterion-referenced tests, because their goal is to
see whether the test taker is sufficiently familiar with the new country's history
and government, not to see whether one test taker is more knowledgeable than
another test taker.

English Language Testing 21

 5

Discrete Point Testing and Integrative

Testing
Electronic quiz tools usually involve a discrete point approach to testing as opposed to
an integrated or authentic approach, such as papers and projects. Discrete point tests are
made up of test questions each of which is meant to measure one content point. Discrete
point testing is associated with multiple choice and true/false formats, which have been
criticized for testing only recognition knowledge and facilitating guessing and cheating.
However, if they are used for an appropriate PURPOSE and if the test questions are
well constructed, discrete point tests can be used for effective teaching and learning.

Should language be tested by discrete points or by integrative testing?Traditionally, language test have
been constructed on the assumption that: language can be broken down intoits component and those
component parts are duly tested.What is discrete point?
Language is segmented into many small linguistic points and the four language skills of
listening, speaking,reading and writing. Test questions are designed to test these skills
and linguistic points. A discrete point testconsists of many questions on a large number
of linguistic points, but each question tests only one linguisticpoint.Examples of
Discrete point test are:1. Phoneme recognition.2. Yes/No, True/ False answers.3.
Spelling.4. Word completion.5. Grammar items.6. Multiple choice tests.Such tests have
a down side in that they take language out of context and usually bear no relationship to
theconcept or use of whole language.Discrete point test met with some criticism,
particularly in the view of more recent trends toward viewing theunits of language and
its communicative nature and purpose, and viewing language as the arithmetic sum ofall
its parts.That is why John Oller (1976) introduced“INTEGRATIVE TESTING”.

English Language Testing 22

According to him“language competence is a unified set of interacting abilities which
cannot be separated apart and testedadequately.“ Oller (1979:37) “Whereasdiscrete
items attempt to test knowledge of language one bit at a time, integrativetests attempt to
assess a learner's capacity to use many bits all at the same time, and possibly while
exercisingseveral presumed components of a grammatical system, and perhaps more
than one of the traditional skills oraspects of skills. Therefore, communicative
competence is so global and requires such“integration”for its“pragmatic”use in thereal
world that it cannot be captured in additive tests of grammar or reading or vocabulary
and other discretepoints of language.This emphasizes the simultaneous testing of the
testee's multiple linguistic competence from variousperspectives.Examples of
integrative test are:
1.Cloze tests
2. Dictation
3.Translation
4.Essays and other coherent writing tasks
5.Oral interviews and conversation
6.Reading, or other extended samples of real text

Oller (1979:38) has refined the integrative concept further by proposing what he calls
pragmatic test.A pragmatic test is“...any procedure or task that causes the learner to
process sequences of elements in alanguage that conform to the normal contextual
constraints of that language and which requires learner torelate sequences of linguistics
elements via pragmatic mappings to extra linguistic contexts.
” A step in a positive direction would be to concentrate on tests of communicative
competence.The recent direction of linguistic study has been toward viewing language
as an integrated and pragmatic skill,we cannot be certain that a test like a cloze test
meets the criterion of predicting or assessing a unified andintegrated underlying
linguistic competence we must be cautious in selecting and constructing test
oflanguage.There is nothing wrong to use the traditional tests of discrete points of
language especially in achievement andother classroom-oriented testing in which
certain discrete points are very important.

English Language Testing 23

English Language Testing 24
 6

Communicative Language Testing

The notion of communicative competence is broad and needs to be fully understood
before being considered as a basis for a research testing regime. As previously indicated
assessment can be viewed in terms of two distinct paradigms as follows: 1) The
Psychometric-Structuralist era: Testing is based on discrete linguistic points related to
four language skill areas, reading, writing, speaking and listening. Additionally there is
the Psycholinguistic-Sociolinguistic era: Integrative tests were conceived in response to
the language proficiency limitations associated with discrete point testing. According to
Oller (in Weir, 1988), Integrative testing could measure the ability to integrate disparate
language skills in ways that more closely resembled the actual process of language use.
The communicative paradigm is founded on the notion of competence. According to
Morrow (in Weir, 1988; pp8) communicative language testing should be concerned
with :1) what the learner knows about the form of the language and how to use it
appropriately in context (Competence). 2) the extent to which the learner is able to
demonstrate this knowledge in a meaningful situation (Performance) i.e what can he do
with the language. Performance testing should therefore be representative of a real-life
situation where an integration of communicative skills is required. The performance test
criteria should relate closely to the effective communication of ideas in that context.
Weir emphasises the importance of context and related tasks as an important dimension
in communicative (performance) language assessment (ibid, pp11). In conclusion a
variety of tests different tests are required for a range of different purposeds and the
associated instruments are no longer uniform in content or method.

In recognising the broad definitions of communication, Carroll (Testing Communicative

Performance, 1980) adopts a rationalist approach to test requirement definition. The

English Language Testing 25

basis of the methodology therefore is a detailed analysis including the identification of
events and activities (communication functions) that drive the communicative need.
Having identified the test requirements, they are divided between the principle
communicative domains of speaking, listening, writing and reading.

This approach is no doubt reminiscent of the requirements definition related to English

for Specific Purposes (ESP) i.e functional language appropriate for Tourist, Students,
Lawyers etc. However, this strategy (and associated methodology) would seem
inappropriate in the given research context for the following salient reasons:

1. No practical to undertake a meaningful needs analysis for all participants

2. The entirely process is far too complex and labour intensive
3. ESP is not aimed at marginalised communities or children

Sabria and Samer (other students) have pointed me in the direction of Cambridge
ToEFL exams (conformant with the Common European Framework of Reference for
Languages) as a potential basis for communicative testing. The tests are divided into the
4 principal language dimensions (Speaking, Listening, Writing and Reading) and
provide tests and marking criteria at all levels of competency including that for the
research context (Young Learners English – YLE starters).

English Language Testing 26

 7

Testing Communicative Competence

Testing language has traditionally taken the form of testing knowledge about language,
usually the testing of knowledge of vocabulary and grammar. However, there is much
more to being able to use language than knowledge about it. Dell Hymes proposed the
concept of communicative competence. He argued that a speaker can be able to produce
grammatical sentences that are completely inappropriate. In communicative
competence, he included not only the ability to form correct sentences but to use them at
appropriate times. Since Hymes proposed the idea in the early 1970s, it has been
expanded considerably, and various types of competencies have been proposed.
However, the basic idea of communicative competence remains the ability to use
language appropriately, both receptively and productively, in real situations.

The Communicative Approach to Testing

What Communicative Language Tests Measure

Communicative language tests are intended to be a measure of how the testees are able
to use language in real life situations. In testing productive skills, emphasis is placed on
appropriateness rather than on ability to form grammatically correct sentences. In
testing receptive skills, emphasis is placed on understanding the communicative intent
of the speaker or writer rather than on picking out specific details. And, in fact, the two
are often combined in communicative testing, so that the testee must both comprehend

English Language Testing 27

and respond in real time. In real life, the different skills are not often used entirely in
isolation. Students in a class may listen to a lecture, but they later need to use
information from the lecture in a paper. In taking part in a group discussion, they need
to use both listening and speaking skills. Even reading a book for pleasure may be
followed by recommending it to a friend and telling the friend why you liked it.

The "communicativeness" of a test might be seen as being on a continuum. Few tests

are completely communicative; many tests have some element of communicativeness.
For example, a test in which testees listen to an utterance on a tape and then choose
from among three choices the most appropriate response is more communicative than
one in which the testees answer a question about the meaning of the utterance.
However, it is less communicative than one in which the testees are face- to-face with
the interlocutor (rather than listening to a tape) and are required to produce an
appropriate response.

Tasks

Communicative tests are often very context-specific. A test for testees who are going to
British universities as students would be very different from one for testees who are
going to their company's branch office in the United States. If at all possible, a
communicative language test should be based on a description of the language that the
testees need to use. Though communicative testing is not limited to English for Specific
Purposes situations, the test should reflect the communicative situation in which the
testees are likely to find themselves. In cases where the testees do not have a specific
purpose, the language that they are tested on can be directed toward general social
situations where they might be in a position to use English.

This basic assumption influences the tasks chosen to test language in communicative
situations. A communicative test of listening, then, would test not whether the testee
could understand what the utterance, "Would you mind putting the groceries away
before you leave" means, but place it in a context and see if the testee can respond
appropriately to it.

English Language Testing 28

If students are going to be tested over communicative tasks in an achievement test
situation, it is necessary that they be prepared for that kind of test, that is, that the course
material cover the sorts of tasks they are being asked to perform. For example, you
cannot expect testees to correctly perform such functions as requests and apologies
appropriately and evaluate them on it if they have been studying from a structural
syllabus. Similarly, if they have not been studying writing business letters, you cannot
expect them to write a business letter for a test.

Tests intended to test communicative language are judged, then, on the extent to which
they simulate real life communicative situations rather than on how reliable the results
are. In fact, there is an almost inevitable loss of reliability as a result of the loss of
control in a communicative testing situation. If, for example, a test is intended to test the
ability to participate in a group discussion for students who are going to a British
university, it is impossible to control what the other participants in the discussion will
say, so not every testee will be observed in the same situation, which would be ideal for
test reliability. However, according to the basic assumptions of communicative
language testing, this is compensated for by the realism of the situation.

Evaluation

There is necessarily a subjective element to the evaluation of communicative tests. Real

life situations don't always have objectively right or wrong answers, and so band scales
need to be developed to evaluate the results. Each band has a description of the quality
(and sometimes quantity) of the receptive or productive performance of the testee.

Examples of Communicative Test Tasks

Speaking/Listening

Information gap. An information gap activity is one in which two or more testees work
together, though it is possible for a confederate of the examiner rather than a testee to

English Language Testing 29

take one of the parts. Each testee is given certain information but also lacks some
necessary information. The task requires the testees to ask for and give information. The
task should provide a context in which it is logical for the testees to be sharing
information.

The following is an example of an information gap activity.

Student A

You are planning to buy a tape recorder. You don't want to spend more than about 80
pounds, but you think that a tape recorder that costs less than 50 pounds is probably not
of good quality. You definitely want a tape recorder with auto reverse, and one with a
radio built in would be nice. You have investigated three models of tape recorder and
your friend has investigated three models. Get the information from him/her and share
your information. You should start the conversation and make the final decision, but
you must get his/her opinion, too.

(information about three kinds of tape recorders)

Student B

Your friend is planning to buy a tape recorder, and each of you investigated three types
of tape recorder. You think it is best to get a small, light tape recorder. Share your
information with your friend, and find out about the three tape recorders that your friend
investigated. Let him/her begin the conversation and make the final decision, but don't
hesitate to express your opinion.

(information about three kinds of tape recorders)

This kind of task would be evaluated using a system of band scales. The band scales
would emphasize the testee's ability to give and receive information, express and elicit
opinions, etc. If its intention were communicative, it would probably not emphasize
pronunciation, grammatical correctness, etc., except to the extent that these might
interfere with communication. The examiner should be an observer and not take part in

English Language Testing 30

the activity, since it is difficult to both take part in the activity and evaluate it. Also, the
activity should be tape recorded, if possible, so that it could be evaluated later and it
does not have to be evaluated in real time.

Role Play. In a role play, the testee is given a situation to play out with another person.
The testee is given in advance information about what his/her role is, what specific
functions he/she needs to carry out, etc. A role play task would be similar to the above
information gap activity, except that it would not involve an information gap. Usually
the examiner or a confederate takes one part of the role play.

The following is an example of a role play activity.

Student

You missed class yesterday. Go to the teacher's office and apologize for having missed
the class. Ask for the handout from the class. Find out what the homework was.

Examiner

You are a teacher. A student who missed your class yesterday comes to your office.
Accept her/his apology, but emphasize the importance of attending classes. You do not
have any extra handouts from the class, so suggest that she/he copy one from a friend.
Tell her/him what the homework was.

Again, if the intention of this test were to test communicative language, the testee would
be assessed on his/her ability to carry out the functions (apologizing, requesting, asking
for information, responding to a suggestion, etc.) required by the role.

English Language Testing 31

English Language Testing 32
 8

Testing Reading and Writing

Some tests combine reading and writing in communicative situations. Testees can be
given a task in which they are presented with instructions to write a letter, memo,
summary, etc., answering certain questions, based on information that they are
given.Letter writing. In many situations, testees might have to write business letters,
letters asking for information, etc.The following is an example of such a task.

Your boss has received a letter from a customer complaining about problems with a
coffee maker that he bought six months ago. Your boss has instructed you to check the
company policy on returns and repairs and reply to the letter. Read the letter from the
customer and the statement of the company policy about returns and repairs below and
write a formal business letter to the customer.

(the customer's complaint letter; the company policy)

The letter would be evaluated using a band scale, based on compliance with formal
letter writing layout, the content of the letter, inclusion of correct and relevant
information, etc.

Summarizing. Testees might be given a long passage--for example, 400 words--and be

asked to summarize the main points in less than 100 words. To make this task
communicative, the testees should be given realistic reasons for doing such a task. For
example, the longer text might be an article that their boss would like to have
summarized so that he/she can incorporate the main points into a talk.The summary
would be evaluated, based on the inclusion of the main points of the longer text.

Testing Listening and Writing/Note Taking

English Language Testing 33

Listening and writing may also be tested in combination. In this case, testees are given a
listening text and they are instructed to write down certain information from the text.
Again, although this is not interactive, it should somehow simulate a situation where
information would be written down from a spoken text.

English Language Testing 34

 9

Performance-Based Assessment
Performance-based assessment is an alternative form of assessment that moves away
from traditional paper and pencil tests. Performance-based assessment involves having
the students produce a project, whether it is oral, written or a group performance. The
students are engaged in creating a final project that exhibits their understanding of a
concept they have learned.

A unique quality of performance-based assessment is that is allows the students to be

assessed based on a process. The teacher is able to see first hand how the students
produce language in real-world situations. In addition, performance-based assessments
tend to have a higher content validity because a process is being measured. The focus
remains on the process, rather than the product in performance-based assessment.

There are two parts to performance-based assessments. The first part is a clearly
defined task for the students to complete. This is called the product descriptor. The
assessments are either product related, specific to certain content or specific to a given
task. The second part is a list of explicit criteria that are used to assess the students.
Generally this comes in the form of a rubric. The rubrics can either be analytical,
meaning it assesses the final product in parts, or holistic, meaning that is assesses the
final product as a whole.

Performance-based assessment tasks are generally not as formally structured. There

is room for creativity and student design in performance-based tasks. Generally, these
tasks measure the students when they are actually performing the given task. Due to the
nature of these tasks, performance-based assessment is highly interactive. Students are
interacting with each other in order to complete real-world examples of language tasks.

English Language Testing 35

Also, performance-based assessment tends to integrate many different skills. For
example, reading and writing can be involved in one task or speaking and listening can
be involved in the same task.

As previously mentioned, there are many types of performance-based assessments. Each

type of assessment brings with it different strengths and deficiencies relative to credible
and dependable information. Because it is virtually impossible for a single assessment
tool to adequately assess all aspects of student performance, the real challenge comes in
selecting or developing performance-based assessments that complement both each
other and more traditional assessments to equitably assess students in physical
education and human performance.

The goal for assessment is to accurately determine whether students have learned the
materials or information taught and reveal whether they have complete mastery of the
content with no misunderstandings. Just as researchers use multiple data sources to
determine the truthfulness of the results, teachers can use multiple types of assessment
to evaluate the level of student learning. Because assessments involve the gathering of
data or information, some type of product, performance, or recording sheet must be
generated. The following are some examples of various types of performance-based
assessments used in physical education.

Performance-based assessment is an opportunity to allow students to produce

language in real-world contexts while being assessed. This type of assessment is unique
because it is not a traditional test format. Some examples of performance-based
assessment tasks are as follows:

Types of Performance-Based Assessment:

1. Journals

English Language Testing 36

Students will write regularly in a journal about anything relevant to their life, school or
thoughts. Their writing will be in the target language. The teacher will collect the
journals periodically and provide feedback to the students. This can serve as a
communication log between the teacher and students.Journals can be used to record
student feelings, thoughts, perceptions, or reflections about actual events or results. The
entries in journals often report social or psychological perspectives, both positive and
negative, and may be used to document the personal meaning associated with one’s
participation (NASPE Standard 6). Journal entries would not be an appropriate
summative assessment by themselves, but might be included as an artifact in a portfolio.
Journal entries are excellent ways for teachers to “take the pulse” of a class and
determine whether students are valuing the content of the class. Teachers must be
careful not to assess affective domain journal entries for the actual content, because
doing so may cause students to write what teachers want to hear (or give credit for)
instead of true and genuine feelings. Teachers could hold students accountable for
completing journal entries. Some teachers use journals as a way to log participation
over time.

2.Letters
The students will create original language compositions through producing a letter.
They will be asked to write about something relevant to their own life using the target
language. The letter assignment will be accompanied by a rubric for assessment
purposes.

3. Oral Reports
The students will need to do research in groups about a given topic. After they have
completed their research, the students will prepare an oral presentation to present to the
class explaining their research. The main component of this project will be the oral
production of the target language.

4. Original Stories
The students will write an original fictional story. The students will be asked to
include several specified grammatical structures and vocabulary words. This assignment
will be assessed analytically, each component will have a point value.

English Language Testing 37

5. Oral Interview
An oral interview will take place between two students. One student will ask the
questions and listen to the responses of the other student. From the given responses,
more questions can be asked. Each student will be responsible for listening and
speaking.

6. skit
The students will work in groups in order to create a skit about a real-world situation.
They will use the target language. The vocabulary used should be specific to the
situation. The students will be assess holistically, based on the overall presentation of
the skit.

7.Poetry Recitations
After studying poetry, the students will select a poem in the target langugage of their
choice to recite to the class. The students will be assessed based on their pronunciation,
rhythm and speed. The students will also have an opportunity to share with the class
what they think the poem means.
8.Portfolios
Portfolios allow students to compile their work over a period of time. The students
will have a checklist and rubric along with the assignment description. The students will
assemble their best work, including their drafts so that the teacher can assess the
process.

9.PuppetShow
The students can work in groups or individually to create a short puppet show. The
puppet show can have several characters that are involved in a conversation of real-
world context. These would most likely be assessed holistically.

10. Art Work/ Designs/Drawings

This is a creative way to assess students. They can choose a short story or piece or
writing, read it and interpret it. Their interpretation can be represented through artistic

English Language Testing 38

expression. The students will present their art work to the class, explaining what they
did and why.

Using Observation in the Assessment Process

Human performance provides many opportunities for students to exhibit behaviors that
may be directly observed by others, a unique advantage of working in the psychomotor
domain. Wiggins (1998) uses physical activity when providing examples to illustrate
complex assessment concepts, as they are easier to visualize than would be the case
with a cognitive example. The nature of performing a motor skill makes assessment
through observational analysis a logical choice for many physical education teachers. In
fact, investigations of measurement practices of physical educators have consistently
shown a reliance on observation and related assessment methods (Hensley and East
1989; Matanin and Tannehill 1994; Mintah 2003).

Observation is a skill used with several performance-based assessments. It is often used

to provide students with feedback to improve performance. However, without some way
to record results, observation alone is not an assessment. Going back to the definition of
assessment provided earlier in the chapter, assessment is the gathering of information,
analyzing the data, and then using the information to make an evaluation. Therefore,
some type of written product must be produced if the task is considered an assessment.

Teachers and peers can assess others using observation. They might use a checklist or
some type of event recording scheme to tally the number of times a behavior occurred.
Keeping game play statistics is an example of recording data using event recording
techniques. Students can self-analyze their own performance and record their
performances using criteria provided on a checklist or a game play rubric. Table 14.1 is
an example of a recording form that could be used for peer assessment. When using
peer assessment, it is best to have the assessor do only the assessment. When the person
recording assessment results is also expected to take part in the assessment (e.g., tossing
the ball to the person being assessed), he or she cannot both toss and do an accurate
observation. In the case of large classes, teachers might even use groups of four, in

English Language Testing 39

which one person is being evaluated, a second person is feeding the ball, the third
person is doing the observation, and a fourth person is recording the results.

Individual or Group Projects

Projects have long been used in education to assess a student’s understanding of a

subject or a particular topic. Projects typically require students to apply their knowledge
and skills while completing the prescribed task, which often calls for creativity, critical
thinking, analysis, and synthesis. Examples of student projects used in physical
education and human performance include the following: demonstrating knowledge of
invasion game strategies by designing a new game; demonstrating knowledge of how to
become an active participant in the community by doing research on obesity and then
developing a brochure for people in the community that presents ideas for developing a
physically active lifestyle; demonstrating knowledge of fitness components and how to
stay fit by designing one’s own fitness program using personal fitness test results;
demonstrating knowledge of how to create a dance by video recording a dance that
members of the group choreographed; and doing research on childhood games and
teaching children from a local elementary school how to play them. Criteria for
evaluating the projects are developed and the results of the project are recorded.

Group projects involve a number of students working together on a complex problem

that requires planning, research, internal discussion, and presentation. Group projects
should include a component that each student completes individually to avoid having a
student receive credit for work that he or she did not do. Another way to avoid this issue
is to have members of the group award paychecks to the various members of the group
(e.g., split a $10,000 check) and provide justifications about the amount given to each
person. To encourage reflections on the contributions of others, students are not allowed
to give an equal amount to everyone. These “checks” are confidential and submitted
directly to the teacher in an envelope that others in the group are not allowed to see.

The following example of a project designed for middle school or high school students
involves a research component, analysis and synthesis of information, problem solving,
and effective communication.

English Language Testing 40

Portfolios

Portfolios are systematic, purposeful, and meaningful collections of an individual’s

work designed to document learning over time. Since a portfolio provides
documentation of student learning, the knowledge and skills that the teacher desires to
have students document guides the structure of the portfolio. The type of portfolio, its
format, and the general contents are usually prescribed by the teacher. Portfolio
collections may also include input provided by teachers, parents, peers, administrators,
or others.The guidelines used to format a portfolio will be based on the type of learning
that the portfolio is used to document. The following are two basic types of portfolios:

Working portfolio—A repository of portfolio documents that the student accumulates

over a certain period of time. Other types of process information may also be included,
such as drafts of student work or records of student achievement or progress over time.

Showcase or model portfolio—A portfolio consisting of work samples selected by the

student that document the student’s best work. The student has consciously evaluated
his or her work and selected only those products that best represent the type of learning
identified for this assessment. Each artifact selected is accompanied by a reflection, in
which the student explains the significance of the item and the type of learning it
represents.

It’s a good idea to limit the portfolio to a certain number of pieces of work to prevent
the portfolio from becoming a scrapbook that has little meaning to the student and to
avoid giving teachers a monumental evaluation task. This also requires students to
exercise some judgment about which artifacts best fulfill the requirements of the
portfolio task and document their level of achievement. The portfolio itself is usually a
file or folder that contains the student’s collected work. The contents could include
items such as a training log, student journal or diary, written reports, photographs or
sketches, letters, charts or graphs, maps, copies of certificates, computer disks or
computer-generated products, completed rating scales, fitness test results, game

English Language Testing 41

statistics, training plans, report of dietary analyses, and even video- or audio recordings.
Collectively, the artifacts selected will document student growth and learning over time
as well as current levels of achievement. The potential items that could become
portfolio artifacts are almost limitless. Kirk (1997) suggests the following list of
possible portfolio artifacts that may be useful for physical activity settings. A teacher
would never require that a portfolio contain all of these items. The list is offered as a
way to generate ideas for possible artifacts.

A rubric (scoring tool) should be used to evaluate portfolios in much the same manner
as any other product or performance. Providing a rubric to students in advance allows
them to self-assess their work and thus be more likely to produce a portfolio of high
quality. Portfolios, since they are designed to show growth and improvement in student
learning, are evaluated holistically. The reflections that describe the artifact and why the
artifact was selected for inclusion in the portfolio provide insights into levels of student
learning and achievement. Teachers should remember that format is less important than
content and that the rubric should be weighted to reflect this. Table 14.2 illustrates a
qualitative analytic rubric for judging a portfolio along three dimensions.

For additional information about portfolio assessments, Lund and Kirk (2010) have a
chapter on developing portfolio assessments. An article published as part of a JOPERD
feature presents a suggested scoring scale for a portfolio (Kirk 1997). Melograno’s
Assessment Series publication (2000) on portfolios also contains helpful information.

Performances

Student performances can be used as culminating assessments at the completion of an

instructional unit. Teachers might organize a gymnastics or track and field meet at the
conclusion of one of those units to allow students to demonstrate the skills and
knowledge that they gained during instruction. Game play during a tournament is also
considered a student performance. Rubrics for game play can be written so that students
are evaluated on all three learning domains (psychomotor, cognitive, and affective).

English Language Testing 42

Students might demonstrate their skills and learning in one of the following ways:

Performing an aerobics routine for a school assembly

Organizing and performing a jump rope show at the half-time of a basketball game

Performing in a folk dance festival at the county fair

Demonstrating wushu (a Chinese martial art) at the local shopping mall

Training for and participating in a local road race or cycling competition

Although performances do not produce a written product, there are several ways to
gather data to use for assessment purposes. A score sheet can be used to record student
performance using the criteria from a game play rubric. Game play statistics are another
example of a way to document performance. Performances can also be video recorded
to provide evidence of learning. In some cases teachers might want to shorten the time
used to gather evidence of learning from a performance. Event tasks are performances
that are completed in a single class period. Students might demonstrate their knowledge
of net or wall game strategies by playing a scripted game that is video recorded during a
single class. The ability to create movement sequences or a dance that uses different
levels, effort, or relationships could be demonstrated during a single class period with
an event task. Many adventure education activities that demonstrate affective domain
attributes can be assessed using event tasks.

Student Logs

Documenting student participation in physical activity (NASPE Standard 3) is often

difficult. Teachers can assess participation in an activity or skill practice trials
completed outside of class using logs. Practice trials during class that demonstrate
student effort can also be documented with logs. A log records behaviors over a period
of time (see figure 14.1). Often the information recorded shows changes in behavior,
trends in performance, results of participation, progress, or the regularity of physical

English Language Testing 43

activity. A student log is an excellent artifact for use in a portfolio. Because logs are
usually a self-recorded document, they are not used for summative assessments unless
as an artifact in a portfolio or for a project. If teachers wanted to increase the importance
placed on a log, a method of verification by an adult or someone in authority should be
added.

English Language Testing 44

 10
VALIDITY AND RELIABILITY

For the statistical consultant working with social science researchers the estimation of
reliability and validity is a task frequently encountered. Measurement issues differ in
the social sciences in that they are related to the quantification of abstract, intangible
and unobservable constructs. In many instances, then, the meaning of quantities is only
inferred.

Let us begin by a general description of the paradigm that we are dealing with. Most
concepts in the behavioral sciences have meaning within the context of the theory that
they are a part of. Each concept, thus, has an operational definition which is governed
by the overarching theory. If a concept is involved in the testing of hypothesis to
support the theory it has to be measured. So the first decision that the research is faced
with is “how shall the concept be measured?” That is the type of measure. At a very
broad level the type of measure can be observational, self-report, interview, etc. These
types ultimately take shape of a more specific form like observation of ongoing activity,
observing video-taped events, self-report measures like questionnaires that can be open-
ended or close-ended, Likert-type scales, interviews that are structured, semi-structured
or unstructured and open-ended or close-ended. Needless to say, each type of measure
has specific types of issues that need to be addressed to make the measurement
meaningful, accurate, and efficient.
Another important feature is the population for which the measure is intended. This
decision is not entirely dependent on the theoretical paradigm but more to the
immediate research question at hand.

English Language Testing 45

A third point that needs mentioning is the purpose of the scale or measure. What is it
that the researcher wants to do with the measure? Is it developed for a specific study or
is it developed with the anticipation of extensive use with similar populations?
Once some of these decisions are made and a measure is developed, which is a careful
and tedious process, the relevant questions to raise are “how do we know that we are
indeed measuring what we want to measure?” since the construct that we are measuring
is abstract, and “can we be sure that if we repeated the measurement we will get the
same result?”. The first question is related to validity and second to reliability. Validity
and reliability are two important characteristics of behavioral measure and are referred
to as psychometric properties.

It is important to bear in mind that validity and reliability are not an all or none issue but
a matter of degree.
Measurement Error
All measurements may contain some element of error; validity and reliability
concern the amount and type of error that typically occurs, and they also show how we
can estimate the amount of error in a measurement.
There are three chief sources of error:
1. in the thing being measured (my weight may fluctuate so it's difficult to get an
accurate picture of it);
2. the observer (on Mondays I may knock a pound off my weight if I binged on my
mother's cooking at the week-end. Obviously the binging doesn't reflect my true
weight!);
3. or in the recording device (our clinic weigh scale has been acting up; we really
should get it recalibrated).And there are two types of error:
Random errors are not attributable to a specific cause. If sufficiently large numbers of
observations are made, random errors average to zero, because some readings over-
estimate and some under-estimate.Systematic errors tend to fall in a particular direction
and are likely due to a specific cause. Because systematic errors fall in one direction
(e.g., I always exaggerate my athletic abilities) they bias a measurement.Random errors

English Language Testing 46

are considered part of the reliability of a measurement.Systematic errors are considered
part of the validity of a measurement.

Reliability and validity

The reliability of an assessment tool is the extent to which it measures learning
consistently.
The validity of an assessment tool is the extent by which it measures what it was
designed to measure.

Reliability
The reliability of an assessment tool is the extent to which it consistently and accurately
measures learning. When the results of an assessment are reliable, we can be confident
that repeated or equivalent assessments will provide consistent results. This puts us in a
better position to make generalised statements about a student’s level of achievement,
which is especially important when we are using the results of an assessment to make
decisions about teaching and learning, or when we are reporting back to students and
their parents or caregivers. No results, however, can be completely reliable. There is
always some random variation that may affect the assessment, so educators should
always be prepared to question results.

Factors which can affect reliability:

The length of the assessment – a longer assessment generally produces more reliable
results.
The suitability of the questions or tasks for the students being assessed.
The phrasing and terminology of the questions.
The consistency in test administration – for example, the length of time given for the
assessment, instructions given to students before the test.
The design of the marking schedule and moderation of marking procedures.
The readiness of students for the assessment – for example, a hot afternoon or straight
after physical activity might not be the best time for students to be assessed.

English Language Testing 47

How to be sure that a formal assessment tool is reliable

Check in the user manual for evidence of the reliability coefficient. These are measured
between zero and 1. A coefficient of 0.9 or more indicates a high degree of reliability.

Assessment tool manuals contain comprehensive administration guidelines. It is

essential to read the manual thoroughly before conducting the assessment.
Validity

Educational assessment should always have a clear purpose. Nothing will be gained
from assessment unless the assessment has some validity for the purpose. For that
reason, validity is the most important single attribute of a good test.
The validity of an assessment tool is the extent to which it measures what it was
designed to measure, without contamination from other characteristics. For example, a
test of reading comprehension should not require mathematical ability.

There are several different types of validity:

Face validity: do the assessment items appear to be appropriate?

Content validity: does the assessment content cover what you want to assess?
Criterion-related validity: how well does the test measure what you want it to?
Construct validity: are you measuring what you think you're measuring?

It is fairly obvious that a valid assessment should have a good coverage of the criteria
(concepts, skills and knowledge) relevant to the purpose of the examination. The
important notion here is the purpose. For example:

The PROBE test is a form of reading running record which measures reading
behaviours and includes some comprehension questions. It allows teachers to see the
reading strategies that students are using, and potential problems with decoding. The
test would not, however, provide in-depth information about a student’s comprehension
strategies across a range of texts.
English Language Testing 48
 STAR (Supplementary Test of Achievement in Reading) is not designed as a
comprehensive test of reading ability. It focuses on assessing students’ vocabulary
understanding, basic sentence comprehension and paragraph comprehension. It is most
appropriately used for students who don’t score well on more general testing (such as
PAT or e-asTTle) as it provides a more fine grained analysis of basic comprehension
strategies.
There is an important relationship between reliability and validity. An assessment that
has very low reliability will also have low validity; clearly a measurement with very
poor accuracy or consistency is unlikely to be fit for its purpose. But, by the same token,
the things required to achieve a very high degree of reliability can impact negatively on
validity. For example, consistency in assessment conditions leads to greater reliability
because it reduces 'noise' (variability) in the results. On the other hand, one of the things
that can improve validity is flexibility in assessment tasks and conditions. Such
flexibility allows assessment to be set appropriate to the learning context and to be
made relevant to particular groups of students. Insisting on highly consistent assessment
conditions to attain high reliability will result in little flexibility, and might therefore
limit validity.

Validity:
Very simply, validity is the extent to which a test measures what it is supposed
to measure. The question of validity is raised in the context of the three points made
above, the form of the test, the purpose of the test and the population for whom it is
intended. Therefore, we cannot ask the general question “Is this a valid test?”. The
question to ask is “how valid is this test for the decision that I need to make?” or “how
valid is the interpretation I propose for the test?” We can divide the types of validity
into logical and empirical.
VALIDITY refers to what conclusions we can draw from the results of a measurement.
Introductory-level definitions are "Does the test measure what we are intending to
measure?", or "How closely do the results of a measurement correspond to the true state
of the phenomenon being measured?"

English Language Testing 49

Nerd's Corner: These ideas of validity fit under a more general conception in terms of
"How can we interpret the test results?" or "What does this measurement actually
mean?" This approach is useful because sometimes information collected for one
purpose can also tell us about something quite different. So, the World Bank records the
gross national product of each country for economic monitoring, but this also gives us a
pretty good idea of how countries will rank in terms of child health.
Nerd's Corner: Putting these ideas together, we get a table showing how validity and
reliability may be assessed:

Thing being measured

Observer
Recording device
(e.g., screening test)
Random error
test re-test reliability
correlation between observers
calibration trial (variation with standard object)
Systematic
record diurnal (etc) variation (e.g. BP higher on Mondays)
agreement between observers (e.g. nurses or patients)
construct& criterion validity;
sensitivity& specificity
Validity of a screening test. This can be used to illustrate the way validity is assessed.
Here, it is commonly reported in terms of sensitivity and specificity.
Sensitivity refers to what fraction of all the actual cases of disease a test detects.
If the test is not very good, it may miss cases it should detect. Its sensitivity is low and it
generates "false negatives" (i.e., people score negatively on the test when they should
have scored positive). This can be extremely serious if early treatment would have
saved the person's life.
Mnemonics to help you: The word 'sensitivity' is intuitive: a sensitive test is one that
can identify the disease.
English Language Testing 50
SeNsitivity is inversely associated with the false Negative rate of a test (high sensitivity
= few false negatives).
Specificity refers to whether the test identifies only those with the disease, or
does it mistakenly classify some healthy people as being sick? Errors of this type are
called "false positives." This can lead to worry and expensive further investigations.

Types of Validity

1. Content Validity:
When we want to find out if the entire content of the behavior/construct/area is
represented in the test we compare the test task with the content of the behavior. This is
a logical method, not an empirical one. Example, if we want to test knowledge on
American Geography it is not fair to have most questions limited to the geography of
New England.

2. Face Validity:
Basically face validity refers to the degree to which a test appears to measure
what it purports to measure. Face Validity ascertains that the measure appears to be
assessing the intended construct under study. The stakeholders can easily assess face
validity. Although this is not a very “scientific” type of validity, it may be an essential
component in enlisting motivation of stakeholders. If the stakeholders do not believe the
measure is an accurate assessment of the ability, they may become disengaged with the
task. Example: If a measure of art appreciation is created all of the items should be
related to the different components and types of art. If the questions are regarding
historical time periods, with no reference to any artistic movement, stakeholders may
not be motivated to give their best effort or invest in this measure because they do not
believe it is a true assessment of art appreciation.

3. Criterion-Oriented or Predictive Validity:

English Language Testing 51

Criterion-Related Validity is used to predict future or current performance - it correlates
test results with another criterion of interest.
Example: If a physics program designed a measure to assess cumulative student
learning throughout the major. The new measure could be correlated with a
standardized measure of ability in this discipline, such as an ETS field test or the GRE
subject test. The higher the correlation between the established measure and new
measure, the more faith stakeholders can have in the new assessment tool.
When you are expecting a future performance based on the scores obtained
currently by the measure, correlate the scores obtained with the performance. The later
performance is called the criterion and the current score is the prediction. This is an
empirical check on the value of the test – a criterion-oriented or predictive validation.

4. Concurrent Validity:
Concurrent validity is the degree to which the scores on a test are related to the
scores on another, already established, test administered at the same time, or to some
other valid criterion available at the same time. Example, a new simple test is to be
used in place of an old cumbersome one, which is considered useful, measurements are
obtained on both at the same time. Logically, predictive and concurrent validation are
the same, the term concurrent validation is used to indicate that no time elapsed between
measures.

5. Construct Validity:
Construct Validity is used to ensure that the measure is actually measure what it is
intended to measure (i.e. the construct), and not other variables. Using a panel of
“experts” familiar with the construct is a way in which this type of validity can be
assessed. The experts can examine the items and decide what that specific item is
intended to measure. Students can be involved in this process to obtain their feedback.
Example: A women’s studies program may design a cumulative assessment of learning
throughout the major. The questions are written with complicated wording and
phrasing. This can cause the test inadvertently becoming a test of reading
comprehension, rather than a test of women’s studies. It is important that the measure is
actually assessing the intended construct, rather than an extraneous factor.
English Language Testing 52
 Construct validity is the degree to which a test measures an intended
hypothetical construct. Many times psychologists assess/measure abstract attributes or
constructs. The process of validating the interpretations about that construct as
indicated by the test score is construct validation. This can be done experimentally,
e.g., if we want to validate a measure of anxiety. We have a hypothesis that anxiety
increases when subjects are under the threat of an electric shock, then the threat of an
electric shock should increase anxiety scores (note: not all construct validation is this
dramatic!)
A correlation coefficient is a statistical summary of the relation between two
variables. It is the most common way of reporting the answer to such questions as the
following: Does this test predict performance on the job? Do these two tests measure
the same thing? Do the ranks of these people today agree with their ranks a year ago?
(rank correlation and product-moment correlation)
According to Cronbach, to the question “what is a good validity coefficient?”
the only sensible answer is “the best you can get”, and it is unusual for a validity
coefficient to rise above 0.60, though that is far from perfect prediction.
All in all we need to always keep in mind the contextual questions: what is the
test going to be used for? how expensive is it in terms of time, energy and money? what
implications are we intending to draw from test scores?
Formative Validity when applied to outcomes assessment it is used to assess how well a
measure is able to provide information to help improve the program under study.
Example: When designing a rubric for history one could assess student’s knowledge
across the discipline. If the measure can provide information that students are lacking
knowledge in a certain area, for instance the Civil Rights Movement, then that
assessment tool is providing meaningful information that can be used to improve the
course or program requirements.
Sampling Validity (similar to content validity) ensures that the measure covers
the broad range of areas within the concept under study. Not everything can be
covered, so items need to be sampled from all of the domains. This may need to be
completed using a panel of “experts” to ensure that the content area is adequately
sampled. Additionally, a panel can help limit “expert” bias (i.e. a test reflecting what an
individual personally feels are the most important or relevant areas).
English Language Testing 53
Example: When designing an assessment of learning in the theatre department, it would
not be sufficient to only cover issues related to acting. Other areas of theatre such as
lighting, sound, functions of stage managers should all be included. The assessment
should reflect the content area in its entirety.

What are some ways to improve validity?

1. Make sure your goals and objectives are clearly defined and operationalized.
Expectations of students should be written down.
2. Match your assessment measure to your goals and objectives. Additionally, have
the test reviewed by faculty at other schools to obtain feedback from an outside
party who is less invested in the instrument.
3. Get students involved; have the students look over the assessment for
troublesome wording, or other difficulties.
4. If possible, compare your measure with other measures, or data that may be
available.

Reliability:
Research requires dependable measurement. (Nunnally) Measurements are
reliable to the extent that they are repeatable and that any random influence which tends
to make measurements different from occasion to occasion or circumstance to
circumstance is a source of measurement error. (Gay) Reliability is the degree to which
a test consistently measures whatever it measures. Errors of measurement that affect
reliability are random errors and errors of measurement that affect validity are
systematic or constant errors.
Test-retest, equivalent forms and split-half reliability are all determined through
correlation.
RELIABILITY refers to consistency or dependability. Your patient Jim is
unpredictable; sometimes he comes to his appointment on time, sometimes he's late and
once or twice he was early.

One way to estimate reliability of a measurement is to record its stability: do you

get the same blood pressure reading if you repeat the measurement? This is sometimes
English Language Testing 54
called "test-retest stability" or "intra-rater reliability" and focuses on the observer and
the instrument as potential sources of error. (Note that we must assume that no actual
change in BP occurred between the measurements: there is no error in the thing being
measured).
You can also estimate reliability by comparing the agreement between different people
making a rating (e.g., if several nurses measure a patient's blood pressure, do they get
the same reading?). This can be called "inter-rater reliability" or "inter-rater agreement."

Nerd's Corner: This is a simplification. Sometimes it's difficult to figure out if an

error is random or systematic: the disagreement between the nurses could really be
random, or it could arise because one of them tends to under-record the BP. Further
testing would be needed to trace the origin of the inaccuracy.

Types of Reliability

1. Test-retest Reliability:
Test-retest reliability is the degree to which scores are consistent over time. It
indicates score variation that occurs from testing session to testing session as a result of
errors of measurement. Problems: Memory, Maturation, Learning.
Test-retest reliability is a measure of reliability obtained by administering the same
test twice over a period of time to a group of individuals. The scores from Time 1 and
Time 2 can then be correlated in order to evaluate the test for stability over time.
Example: A test designed to assess student learning in psychology could be given to a
group of students twice, with the second administration perhaps coming a week after the
first. The obtained correlation coefficient would indicate the stability of the scores.

2. Equivalent-Forms or Alternate-Forms Reliability

Parallel forms reliability is a measure of reliability obtained by administering
different versions of an assessment tool (both versions must contain items that probe the
same construct, skill, knowledge base, etc.) to the same group of individuals. The
scores from the two versions can then be correlated in order to evaluate the consistency
of results across alternate versions.
English Language Testing 55
Example: If you wanted to evaluate the reliability of a critical thinking assessment, you
might create a large set of items that all pertain to critical thinking and then randomly
split the questions up into two sets, which would represent the parallel forms.
Equivalent-Forms or Alternate-Forms Reliability:
Two tests that are identical in every way except for the actual items included.
Used when it is likely that test takers will recall responses made during the first session
and when alternate forms are available. Correlate the two scores. The obtained
coefficient is called the coefficient of stability or coefficient of equivalence. Problem:
Difficulty of constructing two forms that are essentially equivalent.
Both of the above require two administrations.

3. Inter-rater reliability
Inter-rater reliabilit is a measure of reliability used to assess the degree to which
different judges or raters agree in their assessment decisions. Inter-rater reliability is
useful because human observers will not necessarily interpret answers the same way;
raters may disagree as to how well certain responses or material demonstrate knowledge
of the construct or skill being assessed.

Example: Inter-rater reliability might be employed when different judges are evaluating
the degree to which art portfolios meet certain standards. Inter-rater reliability is
especially useful when judgments can be considered relatively subjective. Thus, the use
of this type of reliability would probably be more likely when evaluating artwork as
opposed to math problems.

4. Internal consistency reliability

Internal consistency reliability is a measure of reliability used to evaluate the
degree to which different test items that probe the same construct produce similar
results.
A. Average inter-item correlation is a subtype of internal consistency reliability.
It is obtained by taking all of the items on a test that probe the same construct
(e.g., reading comprehension), determining the correlation coefficient for each

English Language Testing 56

 pair of items, and finally taking the average of all of these correlation
coefficients. This final step yields the average inter-item correlation.
B. Split-half reliability is another subtype of internal consistency reliability. The
process of obtaining split-half reliability is begun by “splitting in half” all items
of a test that are intended to probe the same area of knowledge (e.g., World War
II) in order to form two “sets” of items. The entire test is administered to a
group of individuals, the total score for each “set” is computed, and finally the
split-half reliability is obtained by determining the correlation between the two
total “set” scores.

Split-Half Reliability:
Requires only one administration. Especially appropriate when the test is very
long. The most commonly used method to split the test into two is using the odd-even
strategy. Since longer tests tend to be more reliable, and since split-half reliability
represents the reliability of a test only half as long as the actual test, a correction
formula must be applied to the coefficient. Spearman-Brown prophecy formula.
Split-half reliability is a form of internal consistency reliability.
Internal Consistency Reliability:
Determining how all items on the test relate to all other items. Kudser-
Richardson-> is an estimate of reliability that is essentially equivalent to the average of
the split-half reliabilities computed for all possible halves.

Rationale Equivalence Reliability:

Rationale equivalence reliability is not established through correlation but rather
estimates internal consistency by determining how all items on a test relate to all other
items and to the total test.

Inter-rater reliability is a measure of reliability used to assess the degree to which

different judges or raters agree in their assessment decisions. Inter-rater reliability is
useful because human observers will not necessarily interpret answers the same way;
raters may disagree as to how well certain responses or material demonstrate knowledge
of the construct or skill being assessed.
English Language Testing 57
Example: Inter-rater reliability might be employed when different judges are evaluating
the degree to which art portfolios meet certain standards. Inter-rater reliability is
especially useful when judgments can be considered relatively subjective. Thus, the use
of this type of reliability would probably be more likely when evaluating artwork as
opposed to math problems.

Standard Error of Measurement:

Reliability can also be expressed in terms of the standard error of measurement.
It is an estimate of how often you can expect errors of a given size.

Principles of Language Testing

1. What are the principles of language testing?

2. How can we define them?
3. What factors can influence them?
4. How can we measure them?
5. How do they interrelate?

Three Important Characteristics of Tests:

1.Reliability: consistency and free from extraneous sources of error
2.Validity : how well a test measures what it is supposed to measure

Refers to measuring what we intend to measure. How well a test measures what it is
supposed to measure. For eexampleIf math and vocabulary truly represent intelligence
then a math and vocabulary test might be said to have high validity when used as a
measure of intelligence. Estimating the Validity of a Measure:

1. A good measure must not only be reliable, but also valid

2. A valid measure measures what it is intended to measure
3. Validity is not a property of a measure, but an indication of the extent to which
an assessment measures a particular construct in a particular context—thus a
measure may be valid for one purpose but not another

English Language Testing 58

 4. A measure cannot be valid unless it is reliable, but a reliable measure may not be
valid

Content Validity:

1. Does the test contain items from the desired “content domain”?
2. Based on assessment by experts in that content domain.
3. Is especially important when a test is designed to have low face validity.
4. Is generally simpler for “other tests” than for “psychological constructs ”

For Example - Easier for math experts to agree on an item for an algebra test than it is
for psych experts to agree whether or not an item should be placed in a EI or a
personality measure.

5. Content Validity is not “tested for”. Rather it is assured by experts in the

domain.

Basic Procedure for Assessing Content Validity:

1. Describe the content domain

2. Determine the areas of the content domain that are measured by each test item
3. Compare the structure of the test with the structure of the content domain

For Example:

In developing a nursing licensure exam, experts on the field of nursing would identify
the information and issues required to be an effective nurse and then choose (or rate)
items that represent those areas of information and skills.

A test is to measure foreign students’mastery of English sentence structure, an analysis

must first be made of the language itself and decisions made on which matters need to
be tested and in what proportions.

Face Validity
English Language Testing 59
 1. Face validity refers to the extent to which a measure ‘appears’ to measure what
it is supposed to measure
2. Not statistical—involves the judgment of the researcher (and the participants)
3. A measure has face validity—’if people think it does’
4. Just because a measure has face validity does not ensure that it is a valid
measure (and measures lacking face validity can be valid)

Relationship Between Reliability & Validity

• usefulness of a test. Though different, they work together. It would not be

beneficial to design a test with good reliability that did not measure what it was
intended to measure. The inverse, accurately measuring what you desire to
measure with a test that is so flawed that results are not reproducible, is
impossible. Reliability is a necessary requirement for validity. This means that
you have to have good reliability in order to have validity. Reliability actually
puts a cap or limit on validity, and if a test is not reliable, it can not be valid.
Establishing good reliability is only the first part of establishing validity.
Validity has to be established separately. Having good reliability does not mean
you have good validity, it just means you are measuring something consistently.
Now you must establish what it is that you are measuring consistently. The main
point here is reliability is necessary but not sufficient for validity. Tests that are
reliable are not necessarily valid or predictive. If the reliability of a
psychological measure increases, the validity of the measure is also expected to
increase.

FACTORS THAT INFLUENCE VALIDITY:

1. Inadequate sample
2. Items that do not function as intended
3. Improper arrangement/unclear directions
4. Too few items for interpretation
5. Improper test administration
6. Scoring that is subjective

English Language Testing 60

Reliability is influenced by:

1. the longer the test, the more reliable it is likely to be [though there is a point of
no extra return]
2. items which discriminate will add to reliability, therefore, if the items are too
easy / too difficult, reliability is likely to be lower
3. if there is a wide range of abilities amongst the test takers, test is likely to have
higher reliability
4. the more homogeneous the items are, the higher the reliability is likely to be

Practicality:

The ease with which the test:

1. items can be replicated in terms of resources needed e.g. time, materials, people
2. can be administered
3. can be graded
4. results can be interpreted

Factors which can influence reliability, validity and practicality:

From the TEST:

1. quality of items
2. number of items
3. difficulty level of items
4. level of item discrimination
5. type of test methods
6. number of test methods
7. time allowed
8. clarity of instructions

English Language Testing 61

 9. use of the test
10. selection of content
11. sampling of content
12. invalid constructs

From the TEST TAKERS:

1. familiarity with test method

2. attitude towards the test i.e. interest, motivation, emotional/mental state
3. degree of guessing employed
4. level of ability

From the Test Administration

1. consistency of administration procedure

2. degree of interaction between invigilators and test takers
3. time of day the test is administered
4. clarity of instructions
5. test environment – light / heat / noise / space / layout of room
6. quality of equipment used e.g. for listening tests

From the Scoring

1. accuracy of the key e.g. does it include all possible alternatives?

2. inter-rater reliability e.g. in writing, speaking
3. intra-rater reliability e.g. in writing, speaking
4. machine vs. human

How can we measure reliability?

Test-retest:same test administered to the same test takers following an interval of no

more than 2 weeks

English Language Testing 62

Inter-rater reliability: two or more independent estimates on a test e.g. written scripts
marked by two raters independently and results compared

3.Practicality

English Language Testing 63

 11

CONSTRUCTING TESTS

Writing items requires a decision about the nature of the item or question to which we
ask students to respond, that is, whether discreet or integrative, how we will score the
item; for example, objectively or subjectively, the skill we purport to test, and so on. We
also consider the characteristics of the test takers and the test taking strategies
respondents will need to use. What follows is a short description of these considerations
for constructing items.

Test Items

A test item is a specific task test takers are asked to perform.Test items can assess one
or more points or objectives, and the actual item itself may take on a different
constellation depending on the context. For example, an item may test one point
(understaning of a given vocabulary word) or several points (the ability to obtain facts
from a passage and then make inferences based on the facts). Likewise, a given
objective may be tested by a series of items. For example, there could be five items all
testing one grammatical point (e.g., tag questions). Items of a similar kind may also be
grouped together to form subtests within a given test.

Classifying Items

Discrete – A completely discrete-point item would test simply one point or objective
such as testing for the meaning of a word in isolation. For example:
English Language Testing 64
Choose the correct meaning of the word paralysis.

(A) inability to move

(B) state of unconscious

(C) state of shock

(D) being in pain

Integrative – An integrative item would test more than one point or objective at a time.
(e.g., comprehension of words, and ability to use them correctly in context). For
example:

Demonstrate your comprehension of the following words by using them together in a

written paragraph: “paralysis,” “accident,” and “skiing.”

Sometimes an integrative item is really more a procedure than an item, as in the case of
a free composition, which could test a number of objectives; for example, use of
appropriate vocabulary, use of sentence level discourse, organization, statement of
thesis and supporting evidence. For example:

Write a one-page essay describing three sports and the relative likelihood of being
injured while playing them competitively.

Objective – A multiple-choice item, for example, is objective in that there is only one
right answer.

Subjective – A free composition may be more subjective in nature if the scorer is not
looking for any one right answer, but rather for a series of factors (creativity, style,
cohesion and coherence, grammar, and mechanics).

The Skill Tested

English Language Testing 65

The language skills that we test include the more receptive skills on a continuum –
listening and reading, and the more productive skills – speaking and writing. There are,
of course, other language skills that cross-cut these four skills, such as vocabulary.
Assessing vocabulary will most likely vary to a certain extent across the four skills, with
assessment of vocabulary in listening and reading – perhaps covering a broader range
than assessment of vocabulary in speaking and writing. We can also assess nonverbal
skills, such as gesturing, and this can be both receptive (interpreting someone else’s
gestures) and productive (making one’s own gestures).

The Intellectual Operation Required

Items may require test takers to employ different levels of intellectual operation in order
to produce a response (Valette, 1969, after Bloom et al., 1956). The following levels of
intellectual operation have been identified:

knowledge (bringing to mind the appropriate material);

comprehension (understanding the basic meaning of the material);

application (applying the knowledge of the elements of language and comprehension to

how they interrelate in the production of a correct oral or written message);

analysis (breaking down a message into its constituent parts in order to make explicit
the relationships between ideas, including tasks like recognizing the connotative
meanings of words and correctly processing a dictation, and making inferences);

synthesis (arranging parts so as to produce a pattern not clearly there before, such as in
effectively organizing ideas in a written composition); and

evaluation (making quantitative and qualitative judgments about material).

it has been popularly held that these levels demand increasingly greater cognitive
control as one moves from knowledge to evaluation – that, for example, effective
operation at more advanced levels, such as synthesis and evaluation, would call for
more advanced control of the second language. Yet this has not necessarily been borne

English Language Testing 66

out by research (see Alderson &Lukmani, 1989). The truth is that what makes items
difficult, sometimes defies the intuitions of the test constructors.

The Tested Response Behavior

Items can also assess different types of response behavior. Respondents may be tested
for accuracy in pronunciation or grammar. Likewise, they could be assessed for fluency,
for example, without concern for grammatical correctness. Aside from accuracy and
fluency, respondents could also be assessed for speed – namely, how quickly they can
produce a response, to determine how effectively the respondent replies under time
pressure.In recent years, there has also been an increased concern for developing
measures of performance – that is, measures of the ability to perform real-world tasks,
with criteria for successful performance based on a needs analysis for the given task
(Brown, 1998; Norris, Brown, Hudson, & Yoshioka, 1998).

Performance tasks might include “comparing credit card offers and arguing for the best
choice” or “maximizing the benefits from a given dating service.” At the same time that
there is a call for tasks that are more reflective of the real world, there is a
commensurate concern for more authentic language assessment. At least one study,
however, notes that the differences between authentic and pedagogic written and spoken
texts may not be readily apparent, even to an audience specifically listening for
differences (Lewkowicz, 1997). In addition, test takers may not necessarily concern
themselves with task authenticity in a test situation. Test familiarity may be the
overriding factor affecting performance.

Characteristics of Respondents

Items can be designed to be appropriate for groups of test-takers with differing

characteristics. Bachman and Palmer (1996: 64-78) classify these characteristics into
four categories: the personal characteristics of the respondents – for example, their age,
gender, and native language; the knowledge of the topic that they bring to the language
testing situation; their affective schemata (that is, their prior likes and dislikes with
regard to assessment); and their language ability.

English Language Testing 67

Research into the impact of these characteristics continues. For example, with regard to
the age variable, researchers have suggested that educators revisit this issue and perhaps
conceive of new ways to consider the impact of the age variable in assessing language
ability (Marinova-Todd, Marshall, & Snow, 2000). With regard to performance on
language measures, it would appear that age interacts with other variables such as
attitudes, motivation, the length of exposure to the target language, as well as the nature
and quality of language instruction (see García Mayo &GarcíaLecumberri, 2003).

With regard to language ability, both Bachman and Palmer (1996) and Alderson (2000)
detail the many types of knowledge that respondents may need to draw on to perform
well on a given item or task:world knowledge and culturally-specific knowledge,
knowledge of how the specific grammar works, knowledge of different oral and written
text types, knowledge of the subject matter or topic, and knowledge of how to perform
well on the given task.

Item-Elicitation Format

the format for item elicitation has to be determined for any given item. An item can
have a spoken, written, or visual stimulus, as well as any combination of the three.
Thus, while an item or task may ostensibly assess one modality, it may also be testing
some other as well. So, for example, a subtest referred to as “listening” which has
respondents answer oral questions by means of written multiple-choice responses is
testing reading as well as listening. It would be possible to avoid introducing this
reading element by having the multiple-choice alternatives presented orally as well. But
then the tester would be introducing yet another factor, namely, short-term memory
ability, since the respondents would have to remember all the alternatives long enough
to make an informed choice.

Item-Response Format

The item-response format can be fixed, structured, or open-ended. Item responses with a
fixed format include true/false, multiple-choice, and matching items.Item responses,
which call for a structured format include ordering (where respondents are requested to

English Language Testing 68

arrange words to make a sentence, and several orders are possible), duplication – both
written (such as., dictation) and oral (for example, recitation, repetition, mimicry),
identification (explaining the part of speech of a form), and completion.Those item
responses calling for an open-ended format include composition – both written (for
example, creative fiction, expository essays) and oral (such as a speech) – as well as
other activities, such as free oral response in role-playing situations.

Grammatical competence

According to Canale and Swain (1980, p. 29), grammatical competence includes

phonology, morphology, syntax, knowledge of lexical items, and semantics, as well as
matters of mechanics (spelling, punctuation, capitalization, and handwriting). It would
seem that this definition is perhaps too broad for practical purposes. A truly perplexing
issue is determining what constitutes a grammatical error, as well as determining the
severity of this error. In other words, will the use of the error stigmatize the speaker?
Let us say that we are using a grammatical scale which deals with how acceptably
words, phrases, and sentences are formed and pronounced in the respondents'
utterances. Let us assume that the focus is on both of the following: clear cases of
errors in form, such as the use of the present perfect for an action completed in the past
(e.g., ”We have had a great time at your house last night."), and matters of style, such
as the use of a passive verb form in a context where a native would use the active form
(e.g., Question - “What happened to the CD I lent you, Jorge?” Reply - "The CD was
lost." vs. "I lost your CD.").

Major grammatical errors might be considered those that either interfere with
intelligibility or stigmatize the speaker. Minor errors would be those that do not get in
the way of the listener's comprehension nor would they annoy the listener to any
extent.Thus, getting the tense wrong in the above example, "We have had a great time at
your house last night" could be viewed as a minor error, whereas in another case,
producing "I don't have what to say" ("I really have no excuse" by translating directly
from the appropriate Hebrew language) could be considered a major error since it is not
only ungrammatical but also could stigmatize the speaker as rude and unconcerned,
rather than apologetic.

English Language Testing 69

Rational for Tests:

Measures of student performance (testing) may have as many as five purposes:

Student Placement,
Diagnosis of Difficulties,
Checking Student Progress,
Reports to Student and Superiors,
Evaluation of Instruction.

Unfortunately the most common perception is that tests are designed to statistically rank
all students according to a sampling of their knowledge of a subject and to report that
ranking to superiors or anyone else interested in using that information to adversely
influence the student's feeling of self-worth. It is even more unfortunate that the
perception matches reality in the majority of testing situations. Consequently tests are
highly stressful anxiety producing events for most persons.

All too often tests are constructed to determine how much a student knows rather than
determining what he/she must learn. Frequently tests are designed to "trap" the student
and in still other situations tests are designed to insure a "bell curve" distribution of
results. Most of the other numerous testing designs and strategies fail to help the student
in his learning process and in many cases are quite detrimental to that process.

In a Mastery Based system of instruction the two main reasons for testing are to
determine mastery and to diagnose difficulties. When tests are constructed for these
purposes, the other four purposes will also be satisfied. For example, consider a test
which requires the student to demonstrate mastery and at the same time rigorously
diagnoses learning difficulties. If no difficulties are indicated, it may be safely assumed
that the learner has mastered the concept. That information may then be used to record
student progress and to make reports to the student and superiors. Examining student
performance collectively for a group of students provides information about the quality
of instruction. Examining a single student's performance collectively for a group of

English Language Testing 70

learning objectives may be used to determine proper placement within that group of
learning objectives.

It is therefore important that the instructional developer construct each question so that a
correct response indicates mastery of the learning objective and any incorrect response
provides information about the nature of the student's lack of mastery. Furthermore,
each student should have ample opportunity to "inform" the instructor of any form of
lack of mastery. Unfortunately the mere presence of a test question influences the
student's response to the question. The developer should minimize that influence by
constructing questions which permit the student to make any error he would make in the
absence of such influence. For example, a multiple choice question should have all the
wrong answers the student might want to select and should also have as many correct
answers as the student might want to provide.

True/False Questions:

True/false questions should be written without ambiguity. That is, the statement of the
question should be clear and the decision whether the statement is true or false should
not depend on an obscure interpretation of the statement. A true/false question may
easily be used, and most commonly is used, to determine if the student recalls facts.
However, a true/false question may also be used to determine if the learner has mastered
the learning objective well enough to correctly analyze a statement.

It is important to be aware that only two choices are available to the student and
therefore the nature of the question gives the student a 50% chance of being correct. A
single True/False question therefore is helpful only if the student answers the question
incorrectly and the incorrect response indicates a specific misunderstanding of the
learning objective. A collection of true/false questions, about a single learning objective,
all answered correctly by a student is a much stronger indication of mastery. It is
therefore important that the instructional developer construct a "test bank" containing a
large number of true/false questions. It is also important to include numerous true/false
questions on any test which utilizes true/false questions. Ideally a true/false question
should be constructed so that an incorrect response indicates something about the

English Language Testing 71

student's misunderstanding of the learning objective. This may be a difficult task,
especially when constructing a true statement. The instructional developer should try to
accomplish the ideal, but should recognize that in some instances he/she will not reach
that goal.

Multiple Choice Questions:

Multiple choice questions should be written without ambiguity. That is, the statement of
the question stem should be clear and should leave no doubt about how to select
choices. Additionally the choices should be written without ambiguity and should
contain all information required to make a decision whether or not to choose it. The
decision whether to select or not select a choice should not depend on an obscure
interpretation of either the stem or the choice. A multiple choice question may easily be
used to determine if the student recalls facts. However, a multiple choice question may
also be used to determine if the student has mastered the learning objective well enough
to correctly analyze a statement.

The instructional developer should not construct multiple choice questions with a
uniform number of choices, a uniform number of valid choices, or any other
recognizable pattern for construction of choices. Instead the instructional developer
should include as many valid and invalid choices as is required to determine the
student's deficiencies with respect to the learning objective. Moreover, each choice
should appear to be a valid choice to some student.

Multiple choice questions should therefore contain any number of choices with one or
more valid choices. The student is of course required to select all valid choices and
failure to select any one of the valid choices will provide information about the student's
misunderstanding of the learning objective in the same way that selection of an invalid
choice reveals the nature of his/her misunderstanding. The nature of the choices
provided in a multiple choice question may be of two types: those which require merely
recall of facts and those which require additionally activity such as synthesis, analysis,
computation, comparison, or diagramming. The instructional developer who is seriously
concerned with the student's success will use both types extensively.

English Language Testing 72

Fill-in-the-Blank Questions:

The temptation, when constructing fillintheblank questions, is to construct traps for the
student. The instructional developer should avoid this problem. Ensure that there is only
one acceptable word for the student to provide and that the word (or words) is
significant. Avoid asking the student to supply "minor" words. Avoid fillintheblank
question with so many blanks that the student is unable to determine what is to be
completed.

Sometime/Always/Never Questions:

The collection of Sometime/Always/Never (referred to as SAN) statements are

statements which are: true sometimes, always true, and never true. The statements used
in these questions must be stated carefully and should contain enough information to
permit the student to decide whether the statement is true sometimes, always, or never.

SAN questions (especially the sometimes statements) are the most difficult to construct
but can be the most significant part of a test. SAN questions should be constructed to
force the student to engage in some critical thinking about the learning objective. When
used properly, SAN questions force the student to consider important details about the
learning objective. Careful use of this type of question and careful analysis of student's
response will provide detailed information about some of the student's deficiencies.

SAN questions are especially appropriate, and easy to construct, for learning objectives
addressing concepts which are "black" or "white" except in a few cases. The true
statements in a collection of true/false questions are of course always true statements
while the set of false statement may be further subdivided into those which are true
sometimes and those which are never true.

Test Construction

Closed-Answer or “Objective” Tests

Although by definition no test can be truly “objective” (existing as an object of fact,

independent of the mind), this handbook refers to tests made up of multiple choice,
English Language Testing 73
matching, fill-in, true/false, or fill-in-the-blank items as objective tests. Objective tests
have the advantages of allowing an instructor to assess a large and potentially
representative sample of course material and allow for reliable and efficient scoring.
The disadvantages of objective tests include a tendency to emphasize only “recognition”
skills, the ease with which correct answers can be guessed on many item types, and the
inability to measure students’ organization and synthesis of material (Adapted with
permission from Yonge, 1977).

Since the practical arguments for giving objective exams are compelling, we offer a few
suggestions for writing multiple-choice items. The first is to find and adapt existing test
items. Teachers’ manuals containing collections of items accompany many textbooks.
(AIs: Your course supervisor or former teachers of the same course may be willing to
share items with you.) However, the general rule is adapt rather than adopt. Existing
items will rarely fit your specific needs; you should tailor them to more adequately
reflect your objectives.

Second, design multiple choice items so that students who know the subject or material
adequately are more likely to choose the correct alternative and students with less
adequate knowledge are more likely to choose a wrong alternative. That sounds simple
enough, but you want to avoid writing items that lead students to choose the right
answer for the wrong reasons. For instance, avoid making the correct alternative the
longest or most qualified one, or the only one that is grammatically appropriate to the
stem. Even a careless shift in tense or verb-subject agreement can often suggest the
correct answer.

Finally, it is very easy to disregard the above advice and slip into writing items which
require only rote recall but are nonetheless difficult because they are taken from obscure
passages (footnotes, for instance). Some items requiring only recall might be
appropriate, but try to design most of the items to tap the students’ understanding of the
subject (Adapted with permission from Farris, 1985). One way to write multiple choice
questions that require more than recall is to develop questions that resemble miniature
“cases” or situations. Provide a small collection of data, such as a description of a
situation, a series of graphs, quotes, a paragraph, or any cluster of the kinds of raw

English Language Testing 74

information that might be appropriate material for the activities of your discipline. Then
develop a series of questions based on that material. These questions might require
students to apply learned concepts to the case, to combine data, to make a prediction on
the outcome of a process, to analyze a relationship between pieces of the information, or
to synthesize pieces of information into a new concept.

Here are a few additional guidelines to keep in mind when writing multiple-choice tests
(Adapted with permission from Yonge, 1977):

The item-stem (the lead-in to the choices) should clearly formulate a problem.

As much of the question as possible should be included in the stem.

Randomize occurrence of the correct response (e.g., you don’t always want “C” to be
the right answer).

Make sure there is only one clearly correct answer (unless you are instructing
students to select more than one).

Make the wording in the response choices consistent with the item stem.

Don’t load down the stem with irrelevant material.

Beware of using answers such as “none of these” or “all of the above.”

Use negatives sparingly in the question or stem; do not use double negatives.

Beware of using sets of opposite answers unless more than one pair is presented (e.g.,
go to work, not go to work).

Beware of providing irrelevant grammatical cues.

Grading of multiple choice exams can be done by hand or through the use of computer
scannable answer sheets available from your departmental office. Take completed
answer sheets to IUB Evaluation Services and Testing (BEST) located in Franklin Hall

English Language Testing 75

M014. If you have your test scored by BEST, they will provide statistics on difficulty
and reliability, which will help you to improve your tests.

If you choose the computer-grading route, you must be sure students have number 2
pencils to mark answers on their sheets. These are often available from your
department’s main office. At the time of the exam it is helpful to write on the
chalkboard all pertinent information required on the answer sheet (course name, course
number, section number, instructor’s name, etc.). Also, remind students to fill in their
university identification numbers carefully so that you can have a roster showing the ID
number and grade for each student. If you would like to consult with someone about
developing test items, call theCenter for Innovative Teaching and Learning at 855-9023.

If you would like to consult with someone about how to interpret your test results, call
BEST at 855-1595.

Essay Tests

Conventional wisdom accurately portrays short-answer and essay examinations as the

easiest to write and the most difficult to grade, particularly if they are graded well. You
should give students an exam question for each crucial concept that they must
understand.

If you want students to study in both depth and breadth, don't give them a choice among
topics. This allows them to choose not to answer questions about those things they
didn’t study. Instructors generally expect a great deal from students, but remember that
their mastery of a subject depends as much on prior preparation and experience as it
does on diligence and intelligence; even at the end of the semester some students will be
struggling to understand the material. Design your questions so that all students can
answer at their own levels.

The following are some suggestions that may enhance the quality of the essay tests that
you produce (Adapted with permission from Ronkowski, 1986):

1. Have in mind the processes that you want measured (e.g., analysis, synthesis).

English Language Testing 76

 2. Start questions with words such as “compare,” “contrast,” “explain why.” Don’t
use “what,” “when,” or “list.” (These latter types are better measured with
objective-type items). Writing Tutorial Services, Ballantine Hall 207, 855-6738,
has a handout for students which defines these terms and explains how to study
for and respond to essay questions.
3. Write items that define the parameters of expected answers as clearly as
possible.
4. Make sure that the essay question is specific enough to invite the level of detail
you expect in the answer. A question such as “Discuss the causes of the
American Civil War,” might get a wide range of answers, and therefore be
impossible to grade reliably. A more controlled question would be, “Explain
how the differing economic systems of the North and South contributed to the
conflicts that led to the Civil War.
5. Don’t have too many questions for the time available.

English Language Testing 77

 12
TYPES OF LISTENING TESTING

1. DISCRIMINATIVE LISTENING

Discriminative Listening is an awareness of changes in pitch and loudness of

sounds and it is determining if sounds are different or the same.These activities are
designed to enahnce this listening skill:
1) Same or different? - Call out two words andhave the children determine if they are
the same or different. For example, say bat/ bat, bat/bet.
2) Rhyming words- Practice rhyming discriminative listening skills by calling out a
few rhyming words, such as“hat, bat, rat, cat, and so on” Have the children take
turns calling out a word that rhymes with “at” as well as other rhyming words you
want to use.
3) What’s the problem? - After reading a storybook to children (one that’s very
familiar to them) have them tell you what the problem is. As you read the story
change things around so the story isdifferent somehow, to see if they catch the
changes and can tell you what theproblem is.
4) Musical moods- Play music, but change it up some by changing the pace, make it
fast, slow, loud, soft, high and low. Have the children tell you when a sound change
is made and what the change is.
5) Clap it out- After talking about syllables of words, clap out the syllables of some
words you call out, starting with a two syllable word, then three, and so on. Repeat a
word at least twice (or more if needed) so the concept is fully graspe.

Lastly, we have discriminative listening which has to do with the identification of

different variations in sounds and words in order to understand the different messages.
This is the most important listening and it spans all the other forms of listening. It
involves being sensitive to pitch, volume, emphasis and rate of speech in order to detect

English Language Testing 78

the messages that may be hidden. This form of listening usually requires one to be
efficient in two factors: have a good hearing ability and the knowledge of sound
structure (Kline, 2010).

Hearing ability
The ability to hear helps in sound differentiation and therefore is one can hear well, then
there is a high likelihood that they can get the message well (Lengel, 1998).

Knowledge of sound structure

The knowledge of sound structure enables an individual to differentiate different sounds

and be able to tell what is being said. For example the difference between “I would rank
it first” and “I drank it first” requires such kind of ability in order to get the message
clearly.In conclusion there are various forms of listening and these include listening for
the sake of making critical evaluations, building relationships, making discriminations
and obtaining information or gaining appreciation and each of the needs in listening
calls for a different form of listening. These forms of listening depend on basic factors
such as concentration, attention, memory, perception, experience, presentation style and
the determination of ethos, pathos and logos under the various forms of listening. The
lack of these may imply that there will be no communication that would be ongoing.

For example of discriminative listening

Exercise
Difference sounds is identified
1) “I would rank it first” and “I drank it first”
2) bat/ bat, bat/bet.
3) Safe/save
4) Made/mate
5) Age/h

English Language Testing 79

COMPREHENSION LISTENING

The next step beyond discriminating between different sound and sights is to make
sense of them. To comprehend the meaning requires first having a lexicon of words at
our fingertips and also all rules of grammar and syntax by which we can understand
what others are saying.
The same is true, of course, for the visual components of communication, and an
understanding of body language helps us understand what the other person is really
meaning.
In communication, some words are more important and some less so, and
comprehension often benefits from extraction of key facts and items from a long
spiel.Comprehension listening is also known as content listening, informative listening
and full listening.Listening Comprehension Sample Questions Transcript
Sample Item A
On the recording, you will hear:

(Narrator): Listen to a high school principal talking to the school's students.

(Man): I have a very special announcement to make. This year, not just
one,but three of our students will be receiving national awards for
their academic achievements. Krista Conner, Martin Chan, and
Shriya Patel have all been chosen for their hard work and
consistently high marks.It is very unusual for one school to have so
many students receive this award in a single year.

(Narrator): What is the subject of the announcement?

In your test book, you will read:

1. What is the subject of the announcement?
A. The school will be adding new classes.
B. Three new teachers will be working at the school.
C. Some students have received an award.

English Language Testing 80

 D. The school is getting its own newspaper.
Sample Item B
On the recording, you will hear:

(Narrator): Listen to a teacher making an announcement at the end of the day.

(Man): Remember that a team of painters is coming in tomorrow to paint the

walls. In this box on my desk are sheets of plastic that I want you to
slip over your desks. Make sure you cover your desks completely so
that no paint gets on them. Everything will be finished and the plastic
will be removed by the time we return on Monday.

(Narrator): What does the teacher want the students to do?

In your test book, you will read:

2. What does the teacher want the students to do?
A. Take everything out of their desks
B. Put the painting supplies in plastic bags
C. Bring paints with them to school on Monday
D. Put covers on their desks to keep the paint off
Sample Set A
On the recording, you will hear:

(Narrator): Listen to a conversation between two friends at school.

(Boy): Hi, Lisa.

(Girl): Hi, Jeff. Hey, have you been to the art room today?

(Boy): No, why?

(Girl): Well, Mr. Jennings hung up a notice about a big project that's going
ondowntown. You know how the city's been doing a lot of work to
fix up MainStreet—you know, to make it look nicer? Well, they're
going to create a mural.

English Language Testing 81

 (Boy): You mean, like, make a painting on the entire wall of a building?

(Girl): It's that big wall on the side of the public library. And students from
this school are going to do the whole thing ... create a design, and paint
it, and everything. I wish I could be a part of it, but I'm too busy.

(Boy): [excitedly] Cool! I'd love to help design a mural. Imagine everyone in
town walking past that wall and seeing my artwork, every day.

(Girl): I thought you'd be interested. They want the mural to be about nature,
so I guess all the design ideas students come up with should have a
nature theme.

(Boy): That makes sense—they've been planting so many trees and plants
along the streets and in the park.

(Girl): If you're interested you should talk with Mr. Jennings.

(Boy): [half listening, daydreaming] This could be so much fun. Maybe I'll try
to visit the zoo this weekend ... you know, to see the wild animals and
get some ideas, something to inspire me!

(Girl): [with humor] Well maybe you should go to the art room first to get
more information from Mr. Jennings.

(Boy): [slightly sheepishly] Oh yeah. Good idea. Thanks for letting me know,
Lisa! I'll go there right away.

(Narrator): Now answer the questions.

In your test book,you will read:

3. What are the speakers mainly discussing?
A. A new art project in the city
B. An assignment for their art class
C. An art display inside the public library

English Language Testing 82

 D. A painting that the girl saw downtown
4. Why is the boy excited?
A. A famous artist is going to visit his class.
B. His artwork might be seen by many people.
C. His class might visit an art museum.
D. He is getting a good grade in his art class.

5. Where does the boy say he may go this weekend?

A. To the zoo
B. To an art store
C. To Main Street
D. To the public library
6. Why does the girl suggest that the boy go to the art room?
A. So that he can hand in his homework
B. So that he can sign up for a class trip
C. So that he can see a new painting
D. So that he can talk to the teacher

Sample Set B
On the recording, you will hear:

Script Text:

(Narrator): Listen to a teacher talking in a biology class.

(Woman): We've talked before about how ants live and work together in huge
communities. Well, one particular kind of ant community also grows
its own food. So you could say these ants are like people like farmers.
And what do these ants grow? They grow fungi [FUN-guy]. Fungi are
kind of like plants—mushrooms are a kind of fungi. These ants have
gardens, you could say, in their underground nests. This is where the
fungi are grown.

English Language Testing 83

 Now, this particular kind of ant is called a leafcutter ant. Because of
their name, people often think that leafcutter ants eat leaves. If they cut
up leaves they must eat them, right? Well, they don't! They actually use
the leaves as a kind of fertilizer. Leafcutter ants go out of their nests
looking for leaves from plants or trees. They cut the leaves off and
carry them underground . . . and then feed the leaves to the fungi—the
fungi are able to absorb nutrients from the leaves. What the ants eat are
the fungi that they grow. In that way, they are like farmers!
The amazing thing about these ants is that the leaves they get are often
larger and heavier than the ants themselves. If a leaf is too large,
leafcutter ants will often cut it up into smaller pieces—but not all the
time. Some ants carry whole leaves back into the nest. In fact, some
experiments have been done to measure the heaviest leaf a leafcutter
ant can lift without cutting it. It turns out, it depends on the individual
ant. Some are stronger than others. The experiments showed that some
"super ants" can lift leaves about 100 times the weight of their body!

(Narrator): Now answer the questions.

In your test book, you will read:

7. What is the main topic of the talk?
A. A newly discovered type of ant
B. A type of ant with unusual skills
C. An increase in the population of one type of ant
D. A type of ant that could be dangerous to humans
8. According to the teacher, what is one activity that both leafcutter ants and
people do?
A. Clean their food
B. Grow their own food
C. Eat several times a day
D. Feed their young special food
English Language Testing 84
9. What does the teacher say many people think must be true about leafcutter
ants?
A. They eat leaves.
B. They live in plants.
C. They have sharp teeth.
D. They are especially large.

10. What did the experiments show about leafcutter ants?

A. How fast they grow
B. Which plants they eat
C. Where they look for leaves
D. How much weight they can carry

Answer Key for Listening Comprehension

1. C
2. D
3. A
4. B
5. A
6. D
7. B
8. B
9. A
10. D

English Language Testing 85

 1. CRITICAL LISTENING

Critical listening is listening in order to evaluate and judge, forming opinion about what
is being said. Judgment includes assessing strengths and weaknesses, agreement and
approval.
This form of listening requires significant real-time cognitive effort as the listener
analyzes what is being said, relating it to existing knowledge and rules, whilst
simultaneously listening to the ongoing words from the speaker.

2. BIASED LISTENING

Biased listening happens when the person hears only what they want to hear, typically
misinterpreting what the other person says based on the stereotypes and other biases that
they have. Such biased listening is often very evaluative in nature.

3. EVALUATIVE LISTENING

In evaluative listening, or critical listening, we make judgments about what the other
person is saying. We seek to assess the truth of what is being said. We also judge what
they say against our values, assessing them as good or bad, worthy or unworthy.
Evaluative listening is particularly pertinent when the other person is trying to persuade
us, perhaps to change our behavior and maybe even to change our beliefs. Within this,
we also discriminate between subtleties of language and comprehend the inner meaning
of what is said. Typically also we weigh up the pros and cons of an argument,
determining whether it makes sense logically as well as whether it is helpful to
us.Evaluative listening is also called critical, judgmental or interpretive listening.

4. APPRECIATIVE LISTENING

In appreciative listening, we seek certain information which will appreciate, for

example that which helps meet our needs and goals. We use appreciative listening when
English Language Testing 86
we are listening to good music, poetry or maybe even the stirring words of a great
leader. The student use ppreciative listening when they are listening this poetry and they
seek certain information which will appreciate
Adventure Quotient (AQ) Test 77 questions, 30 min
How adventurous are you? Thrill-seeking can come in different forms, whether it's
doing a swan dive bungee jump off the Auckland Harbour Bridge in New Zealand, or
trying that new exotic restaurant around the corner from work. The type of adventure
you enjoy (or avoid) depends a great deal on your personality. Are you more of a
planner or spontaneous? Courageous or careful? Do you have the energy level of a bee
or a sloth? Find out more about your adventure personality with this test!

Examine the following statements and choose the answer option that best applies to you.
There may be some questions describing situations that may not be relevant to you. In
such cases, select the answer you would most likely choose if you ever found yourself
in that type of situation. In order to receive the most accurate results, please answer as
truthfully as possible.After finishing the test, you will receive a Snapshot Report with an
introduction, a graph and a personalized interpretation for one of your test scores. You
will then have the option to purchase the full results.

Adventure Quotient (AQ) Test 50 questions, 30 min

1. I _____ repetitive tasks.
enjoy
don't mind
can't stand
2. I take pride in my appearance and upkeep.
Agree
Somewhat agree
Disagree
3. I have already been or would consider any of the following: skydiving, bungee
jumping, hang gliding, or free climbing.
Definitely
Maybe
English Language Testing 87
 No way
4. I see getting away from it all as a chance to:
Connect with people and places
Connect with myself

5. I would travel to a developing country and leave the airport/train station:

With pleasure.
Only with a friend.
Only with a hired guide.
6. I seek new experiences more...
To learn about new places, people, and things.
For the way they make me feel
7. I am more likely to ask myself:
"When is break time?"
"What's next?"
8. I am more likely to get my thrills from:
Doing something physically or emotionally gutsy
Watching someone else do something physically or emotionally gutsy
9. The lowest comfort I would consider for sleeping is:
Outside on the ground.
A tent.
An RV or camper.
A motel.
A bed and breakfast.
A furnished apartment or house.
A 3 or 4 star hotel.
10. Adrenaline is a chemical that:
I avoid
I enjoy from time to time
I seek
I am addicted to
11. Having a daily routine is:
English Language Testing 88
 Oppressive and stifling
Annoying and limiting
Sometimes a good thing, sometimes not
Helpful and comforting
Totally necessary
12. Not knowing what the future might hold is:
Terrifying
A little disconcerting, but that's just the way life is
Exhilarating
13. At a theme park, I'll try:
The highest, scariest ride
Something fast, but no upside-down stuff
The kiddie train or merry-go-round
The park bench
14. Having nice things and looking good is important to me
Extremely
Somewhat
Not very
15. A life without luxury is:
Not worth living
Difficult to imagine.
Perfectly acceptable
Expected.
16. Knowing what others think of me is:
Essential
Important
Helpful
Not important
17. When visiting new places, I am more interested in:
Soaking up the environment
Interacting with people
18. Others are more likely to wonder...
English Language Testing 89
 Where my energy goes.
Where my energy comes from
19. Life's experiences are most rich and interesting when I contemplate them...
With others
In my own mind
An old friend is in town. Where are you most likely to eat?

A. We'd eat at:

A fast food joint
An ethnic café
B. We'd eat at:
A themed restaurant or dinner theater
At the kitchen table in my house, warming up something in the microwave
C. We'd eat at a:
Chain restaurant
Upscale restaurant
You inherit $100,000 from a distant uncle. What are you more likely to do with it?
D. I'd take my wallet out and:
Go on an epic shopping spree
Donate some, or all, to charity
E. I'd take my wallet out and:
Go on a casino fling
Put it in the bank
F. I'd take my wallet out and:
Go on a dream vacation
Throw a gigantic party

It's time to learn something new. Which class would you be most interested in taking
up?
G. I would rather take:
Acting classes
Creative writing classes
English Language Testing 90
H. I would rather take:
Survival skills classes
Speed reading classes
I. I would rather take:
Kickboxing classes
Tai Chi classes
Which of the following would you rather visit or spend some time in?
J. I would rather go to:
An Inuit igloo
A Buddhist monastery

K. I would rather go to:

An African hut
A European hostel
L. I would rather go to:
A Japanese pagoda
A California spa
Pick your preferred pet
M. I'd rather have a:
Parrot
Hamster
N. I'd rather have a:
Goldfish
Snake
O. I'd rather have a:
Tarantula
Horse
Which is your preferred adrenaline rush?
P. There's nothing like the thrill of:
A looming deadline
A charging rhino
Q. There's nothing like the thrill of:
English Language Testing 91
 Running cross-country
Running with the bulls
R. There's nothing like the thrill of:
Swimming with dolphins
Swimming with sharks
Which is your preferred adrenaline rush?
S. There's nothing like the thrill of:
Finding something I really like on sale.
Finding an ancient Egyptian artifact in Valley of the Kings
T. There's nothing like the thrill of:
Cycling or hiking
Taking a scenic drive

U. There's nothing like the thrill of:

Getting a tattoo or piercing
Skydiving or hang gliding
Pick the adjective that best describes you.
V. I am more:
Bold
Timid
W. I am more:
Impulsive
Deliberate
X. I am more:
Of an improviser
Of a planner
Y. What's your most favorite way to get from point A to point B?
First class or Business class
The scenic railroad route
Automobile - the classic "road trip"
Budget airline - who needs legroom?
Tour bus - sit back and relax
English Language Testing 92
 An all-terrain vehicle. No road? No problem
My bike - and the wind in my hair
Z. What's your comfort zone when it comes to heights?
Top shelf of the bookcase
The 3-meter diving board
A bungee jump
A skydive
A spacewalk
AA. What is the one form of footwear you could never live without?
Skis
Cycling shoes
Stiletto heels
Cross-trainers
Walking shoes
Flip-flops
Hiking boots
Dress shoes
BB. Which voice mail message are you most likely to leave on a friend's phone?
"How about a movie and some take out?"
"Got an extra ticket to a show, let's go!"
"Party of the century! Pick you up at 9."
"Meet me at the airport with a suitcase and your passport."
CC. Which phrase do you agree with more?
"Better safe than sorry."
"Nothing ventured, nothing gained."
DD. How much of Mother Nature's wrath will you endure for adventure?
Monsoon, tornado, ice storm - bring it on!
Thundershowers, extreme hot and cold
Some wind, clouds, and drizzle
If it's not blue skies, forget it
EE. How often do you pick up new fashions?
Daily to weekly.
English Language Testing 93
 Monthly to yearly.
Every decade or so

5. SYMPATHETIC LISTENING

In sympathetic listening we care about the other person and show this concern in
the way we pay close attention and express our sorrow for their ills and happiness at
their joys.

EMPATHETIC LISTENING

When we listen empathetically, we go beyond sympathy to seek a truer understand how

others are feeling. This requires excellent discrimination and close attention to the
nuances of emotional signals. When we are being truly empathetic, we actually feel
what they are feeling.In order to get others to expose these deep parts of themselves to
us, we also need to demonstrate our empathy in our demeanor towards them, asking
sensitively and in a way that encourages self-disclosure.

6. THERAPEUTIC LISTENING

In therapeutic listening, the listener has a purpose of not only empathizing with the
speaker but also to use this deep connection in order to help the speaker understand,
change or develop in some way.This not only happens when you go to see a therapist
but also in many social situations, where friends and family seek to both diagnose
problems from listening and also to help the speaker cure themselves, perhaps by some
cathartic process. This also happens in work situations, where managers, HR people,
trainers and coaches seek to help employees learn and develop.

7. DIALOGIC LISTENING
English Language Testing 94
The word 'dialogue' stems from the Greek words 'dia', meaning 'through' and 'logos'
meaning 'words'. Thus dialogic listening mean learning through conversation and an
engaged interchange of ideas and information in which we actively seek to learn more
about the person and how they think.Dialogic listening is sometimes known as
'relational listening'.

The example of dialogic listening

A : I was working as a training director for a national homelessness foundation. I was
traveling around the country doing a lot of teaching and consulting. I was mostly
the only white male wherever I went. So I was doing big urban shelters and city
governments in Detroit and places like that. I was always coming up against race,
class, and gender issues between myself and the participants.
Q : Because they weren't white males?
A : Right, they were mostly females of color, and I could always deal with it, but it
was by the seat of my pants. So I came to PCP for consultation initially and then I
was accepted into their first workshop back in 1994. I found it to be such a
revolutionary approach to difference, one that I had never experienced before in
all my training in diversity and all that other stuff. I found out after my first class
that I had to do my training in Louisville, Kentucky for the homelessness network
there. The issue there was that the staff of the homeless shelters were mostly
women of color, and the volunteers were mostly affluent white women from the
suburbs and they differed in many ways and had different ideas about each other
as well. So I started doing this training. One of the goals of this group was that
they wanted the people to work more effectively together.
About half way through the first day, an African American woman stood up and
she was very angry. She said, "You don't know shit about my life, you're a white
man with privilege." I had some choices to make there. But because I had been to
this one PCP class, I decided that I was going to deal with this differently than I
would have dealt with this prior. I said, "You're absolutely right. I am white. I'm a
guy. I have certain level of power. I wear a tie. I live in suburbs. I drive a nice car.
And I imagine that your story has a lot to do with why you're here. I imagine that
English Language Testing 95
 a lot of other people's stories have a lot to do with why they're here. I'm
wondering if we can make a choice together as a group to hear your story, and
what it is that you want people to understand about you. Would you be willing to
hear the stories of others?" She said, "Yeah." So I had everyone go around and tell
the group how their personal story connected to why they were there. Everybody
went around the room. Women told these incredible stories.
I remember there was one white woman who told how she had been homeless for
the last two years. That she had been beaten by her husband, but because they
were wealthy and lived in the suburbs, he was basically able to buy off the police,
and she was basically in prison because of her wealth. Finally, when he started
beating the children, she took them. She was cut off completely from his wealth
and lived on the streets for two years. She had just gotten out of shelter. This
tremendous bonding happened among these women. We were all brought to tears
by it. That affected me deeply. I came home and a couple days later my kids were
fighting. I was always the type, and I still give into this temptation, of getting
involved in the middle and trying to referee, thinking I know what's going on. In
this instance, I tried taking what is called a not-knowing attitude. I suggested that
each kid take five minutes to explain what's going on. I was using the "what's-at-
the-heart-of-the-matter-for-you approach," but in a way that was easier for them to
understand because they were younger. So each kid had five minutes.
Once they spoke, I realized that I certainly didn't have a clue about what their
concerns were. I had a completely different idea about what they were concerned
about, and they had completely different ideas about each other. They were then
able to say, "Oh, so that's all you want," and then move along. Now it doesn't
always happen like that, but it made a really deep impression on me. The biggest
thing for me is being a father, it's the most important thing in my life, and the fact
that I can do it well is my biggest accomplishment. To think that I was doing it so
well, yet I was doing it so ineffectively that I could not know my own kids. I
could be with them ten hours a day but still not know them because I wasn't
listening to them deeply. It blew my mind. I just thought that this is the best thing
since sliced bread. So those two things really catapulted me into the whole PCP
mindset.
English Language Testing 96
Q : So you were really struck by the real power of letting the parties speak for
themselves, without being the convoy, without being the person who summarizes
and says, "This is what's going on."
A : Right. Exactly. Yeah, because I could have said, "This is what I hear." I try to
relate it to my own experience in some way, but basically I don't know. Being
asked, "Can we use your wisdom and tap the rest of the wisdom in the room and
make it work for us here?" Then leaving it in their hands afterward was big. It just
was not my style to do that before.

Listen carefully to the dialog between nick and jimmy,then complete the
conversation
Nick : I heard (1)..........as a computer pragrammer
Jimmy : Yes,and I had already(2)..............
Nick : Really?i’m happy(3)...
Jimmy : Thank you.
Nick : Your parents must be(4)........
Jimmy : They want me to run their business.they’re(5)......
Nick : That’s a pity!did you explain your reasons?
Jimmy : I did and I hope they’ll accept my decision.

Dialog II
Margaret : Look at you!you look so great now.what have you been doing?
Joe : Really?(1).................i’ve been in canada for two weeks.by the way,how
about your job?
Margaret : (2)............it’s in a big new hospital.My working conditions aremuch
better than the the last place.
Tony : Attention,please.today,we have a surprise.we’ve been offered a trip from
our boss
Joe : Really?(3)........................?
Tony : Bandung
Joe : (4)..................but where is it located?
Tony : Aren’t you pleased?
English Language Testing 97
Joe : Yes,of course.(5)........................but tell me where it is.
Margare : It’s in indonesia.
Joe : Oh,I see.that’s not so good
Tony : Don’t worry joe.my friend,lisa,who lives there,wrote to me about the
conditions in indonesia.indonesia is safe now,especially in that twon.there
is no riot.it’s just a rumour.

Key Answer
1) I think it’s usual
2) That’s great
3) Where to
4) Marvellous
5) I’m delighted to hear that

8. RELATIONSHIP LISTENING

Sometimes the most important factor in listening is in order to develop or sustain a

relationship. This is why lovers talk for hours and attend closely to what each other has
to say when the same words from someone else would seem to be rather
boring.Relationship listening is also important in areas such as negotiation and sales,
where it is helpful if the other person likes you and trusts you.

English Language Testing 98

 13
Testing Grammar

English is a very important language in the world. It plays a very big rule in
communication and education. Everything which is served by technology should be
related to English. By and by, English will be the global language in every part of the
world.Since English is an international language, people all over the world try to learn
as much as Possible about english. To develop our skill in english we always meet the
grammar. And we practice our english by the testing grammar, so that we know how far
we understand the english.

A. Definition of grammar

Grammar is the structural foundation of our ability to express ourselves. The more we
are aware of how it works, the more we can monitor the meaning and effectiveness of
the way we and others use language. It can help foster precision, detect ambiguity, and

English Language Testing 99

exploit the richness of expression available in English. And it can help everyone--not
only teachers of English, but teachers of anything, for all teaching are ultimately a
matter of getting to grips with meaning.

1. Descriptive grammar refers to the structure of a language as it is actually

used by speakers and writers.
2. Prescriptive grammar refers to the structure of a language as certain
people think it should be used.

Both kinds of grammar are concerned with rules--but in different ways. Specialists in
descriptive grammar (called linguists) study the rules or patterns that underlie our use of
words, phrases, clauses, and sentences. On the other hand, prescriptive grammarians
(such as most editors and teachers) lay out rules about what they believe to be the
“correct” or “incorrect” use of language.

B. Types of test
Before writing a test it is vital to think about what it is you want to test and what its
purpose is. We must make a distinction here between proficiency tests, achievement
tests, diagnostic tests and prognostic tests.

21. A proficiency test is one that measures a candidate's overall ability in a

language; it isn't related to a specific course.
22. An achievement test on the other hand tests the students' knowledge of
the material that has been taught on a course.
23. A diagnostic test highlights the strong and weak points that a learner may
have in a particular area.
24. A prognostic test attempts to predict how a student will perform on a
course.

There are of course many other types of tests. It is important to choose elicitation
techniques carefully when you prepare one of the aforementioned tests.There are many
elicitation techniques that can be used when writing a test. Below are some widely used
types with some guidance on their strengths and weaknesses. Using the right kind of

English Language Testing 100

question at the right time canbe enormously important in giving us a clear
understanding of our students' abilities, but we must also be aware of the limitations of
each of these task or question types so that we use each one appropriately

1. Multiple choice
Choose the correct word to complete the sentence.
Cook is ________________today for being one of Britain's most famous explorers.

a) Recommended b) reminded c) recognized d) remembered

In this question type there is a stem and various options to choose from. The advantages
of this question type are that it is easy to mark and minimizes guess work by having
multiple distracters. The disadvantage is that it
can be very time-consuming to create, effective multiple choice items are surprisingly
difficult to write. Also it takes time for the candidate to process the information which
leads to problems with the validity of the exam. If a low level candidate has to read
through lots of complicated information before they can answer the question, you may
find you are testing their reading skills more than their lexical knowledge.

Multiple choice can be used to test most things such as grammar, vocabulary, reading,
listening etc. but you must remember that it is still possible for students to just 'guess'
without knowing the correct answer.

2. Transformation
Complete the second sentence so that it has the same meaning as the first.
'Do you know what the time is, John?' asked Dave.
Dave asked John __________ (what) _______________ it was.
This time a candidate has to rewrite a sentence based on an instruction or a key word
given. This type of task is fairly easy to mark, but the problem is that it doesn't test
understanding. A candidate may simply be able to rewrite sentences to a formula. The
fact that a candidate has to paraphrase the whole meaning of the sentence in the
example above however minimizes this drawback.

English Language Testing 101

Transformations are particularly effective for testing grammar and understanding of
form. This wouldn't be an appropriate question type if you wanted to test skills such as
reading or listening.

3. Gap-filling
Complete the sentence.

Check the exchange ______________ to see how much your money is worth.

The candidate fills the gap to complete the sentence. A hint may sometimes be included
such as a root verb that needs to be changed, or the first letter of the word etc. This
usually tests grammar or vocabulary. Again this type of task is easy to mark and
relatively easy to write. The teacher must bear in mind though that in some cases there
may be many possible correct answers.

 Gap-fills can be used to test a variety of areas such as vocabulary, grammar and
are very effective at testing listening for specific words

4. True / False
Decide if the statement is true or false.

England won the world cup in 1966. T/F

Here the candidate must decide if a statement is true or false. Again this type is easy to
mark but guessing can result in many correct answers. The best way to counteract this
effect is to have a lot of items.

 This question type is mostly used to test listening and reading comprehension

5. Open questions
Answer the questions.

English Language Testing 102

Why did John steal the money?

Here the candidate must answer simple questions after a reading or listening or as part
of an oral interview. It can be used to test anything. If the answer is open-ended it will
be more difficult and time consuming to mark and there may also be a an element of
subjectivity involved in judging how 'complete' the answer is, but it may also be a more
accurate test.

 These question types are very useful for testing any of the four skills, but less
useful for testing grammar or vocabulary.

6.Error Correction
Find the mistakes in the sentence and correct them.

Ipswich Town was the more better team on the night.

Errors must be found and corrected in a sentence or passage. It could be an extra word,
mistakes with verb forms, words missed etc. One problem with this question type is that
some errors can be corrected in more than one way.

 Error correction is useful for testing grammar and vocabulary as well as readings
and listening.

7. Other Techniques
There are of course many other elicitation techniques such as translation, essays,
dictations, ordering words/phrases into a sequence and sentence construction
(He/go/school/yesterday).

It is important to ask yourself what exactly you are trying to test, which techniques suit
this purpose best and to bear in mind the drawbacks of each technique. Awareness of
this will help you to minimize the problems and produce a more effective test.

English Language Testing 103

 C.The Value of Studying Grammar

The study of grammar all by itself will not necessarily make you a better writer. But by
gaining a clearer understanding of how our language works, you should also gain
greater control over the way you shape words into sentences and sentences into
paragraphs. In short, studying grammar may help you become a more effective
writer.Descriptive grammarians generally advise us not to be overly concerned with
matters of correctness: language, they say, isn't good or bad; it simply is. As the history
of the glamorous word grammar demonstrates, the English language is a living system
of communication, a continually evolving affair. Within a generation or two, words and
phrases come into fashion and fall out again. Over centuries, word endings and entire
sentence structures can change or disappear.

Prescriptive grammarians prefer giving practical advice about using language:

straightforward rules to help us avoid making errors. The rules may be over-simplified
at times, but they are meant to keep us out of trouble--the kind of trouble that may
distract or even confuse our readers.

English Language Testing 104

 14
INTERPRETING TEST SCORE

Introduction

What does interpret mean? To interpret is to decide what the intended meaning of
something is (Cambridge Advanced Learner’s Dictionary). To interpret is to conceive
the significance of; construe (thefreedictionary.com). Thus, to interpret is to understand
the meaning and the significance of something.Interpreting test scores is to understand
the meaning and the significance of test scores, which can be used to plan next action -
to fix or to retain. There are many ways to do it, but the most common three are
frequency distribution, measures of central tendency, and measures of dispersion.
Frequency distribution here is talking about the distribution of scores and the frequency
of each category. On the other hand, measures of central tendency refer to measure of
“middle” value, and are measured using the mode, median, and mean. The last but not
least, is the measures of dispersion. It is related to the range or spread of scores. All
three can help teachers interpret the meaning behind test scores.

English Language Testing 105

II. Content

A. Frequency Distribution
Frequency distribution deals with the distribution of scores and the frequency of the
distribution. Each entry in the table contains the frequency or count of the occurrences
of scores within a particular name, and in this way, the table summarizes the
distribution of scores.
The example case here is: a teacher administers a test of 40 questions to 26 students.
Marks are awarded by counting the number of correct answers on the test scripts. These
are known as raw marks.
Here are the steps to create a table of frequency distribution:

1. Create Table 1 and put the raw mark of every student in it.
TABLE 1
Testee Mark
A 20
B 25
C 33
D 35
E 29
F 25
G 30
H 26
I 19
J 27
K 26
L 32
M 34
N 27
O 27
P 29
Q 25

English Language Testing 106

 R 23
S 30
T 26
U 22
V 23
W 33
X 26
Y 24
Z 26

2. Create Table 2. Sort the marks from the highest to the lowest score. This is called
descending sorting. It is easier and faster to use tool like Microsoft Excel to do the
sorting.

TABLE 2
Testee Mark
D 35
M 34
C 33
W 33
L 32
G 30
S 30
E 29
P 29
J 27
N 27
O 27
H 26
K 26
T 26

English Language Testing 107

 X 26
Z 26
B 25
F 25
Q 25
Y 24
R 23
V 23
U 22
A 20
I 19

Now, we determine the rank. We start form rank 1 up to rank 26, for there are 26
students.
The problem comes when there are two or more students with the same mark. Here we
highlight the same mark to make it easier to distinguish. Then, we write imaginary rank
on the right of Rank column from 1 to 26. The imaginary rank of the same mark is then
added and divided by how many people who get the same mark. For example, student C
and W have the same mark, 33. Their imaginary rank is 3 and 4. To get the actual rank,
we add 3 and 4 (3+4=7). The result, 7, is then divided by the number of people of the
same score, which is 2 here. The final result is 3.5. Thus, the final result is 3.5. Thus,
the final result is 3.5. Thus, the ranks of both of them are 3.5.

TABLE 2
Imaginary
Testee Mark Rank
rank
D 35 ? 1
M 34 ? 2
C 33 ? 3 (3+4) / 2 = 3.5
W 33 ? 4
L 32 ? 5

English Language Testing 108

 G 30 ? 6 (6+7) / 2 = 6.5
S 30 ? 7
E 29 ? 8 (8+9) / 2 = 8.5
P 29 ? 9
J 27 ? 10
(10+11+12) / 3 =
N 27 ? 11
11
O 27 ? 12
H 26 ? 13
K 26 ? 14 (13+14+15+16+17) / 5
T 26 ? 15 = 15 8.5
X 26 ? 16
Z 26 ? 17
B 25 ? 18
(18+19+20) / 3 =
F 25 ? 19 19
Q 25 ? 20
Y 24 ? 21
R 23 ? 22 (22+23) / 2 =
V 23 ? 23 22.5
U 22 ? 24
A 20 ? 25
I 19 ? 26

The result will be like this. Table 2 shows the students’ scores in order of merit and
their rank as well.
TABLE 2
Testee Mark Rank
D 35 1
M 34 2
C 33 3.5
W 33 3.5
L 32 5

English Language Testing 109

 G 30 6.5
S 30 6.5
E 29 8.5
P 29 8.5
J 27 11
N 27 11
O 27 11
H 26 15
K 26 15
T 26 15
X 26 15
Z 26 15
B 25 19
F 25 19
Q 25 19
Y 24 21
R 23 22.5
V 23 22.5
U 22 24
A 20 25
I 19 26

3. Create Table 3, which consists of Mark column, Tally column, and Frequency
column.
In Mark column, we can expand the range from 40 up to 15, for the highest score is
35 and the lowest score is 19. We usually do this to give more space to enhance
readability.
Tally is the stroke of how many students get a certain score. It is simply a method of
counting the frequency of scores.
Frequency column lists the number of students obtaining each score. It is easier to
count due to the tallies.

English Language Testing 110

 Table 3 is the table of frequency distribution.
TABLE 3
Mark Tally Frequency
40
39
38
37
36
35 / 1
34 / 1
33 // 2
32 / 1
31
30 // 2
29 // 2
28
27 /// 3
26 //// 5
25 /// 3
24 / 1
23 // 2
22 / 1
21
20 / 1
19 / 1
18
17
16
15
TOTAL 26

English Language Testing 111

B. Measures of Central Tendency
A measure of central tendency is a measure that tells us where the “middle” of a bunch
of data lies. The three most common measures of central tendency are the mode, the
median, and the mean.

B.1. Mode
Mode refers to the score which most candidates obtained. We can easily spot it from
Table 3. The most frequent score in Table 3 is 26, as five testees have scored this mark.
Thus, the mode is 26.

B.2. Median
Median refers to the score gained by the middle candidate after the data is put in order.
We use Table 2, which has been ordered in descending order, to find the median. In the
case of 26 students here, there can obviously be no middle student and thus the score
halfway between the lowest score in the top half and the highest score in the bottom half
is taken as the median. The median score in this case is 26.

English Language Testing 112

TABLE 2
Testee Mark Rank
D 35 1
M 34 2
C 33 3.5
W 33 3.5
L 32 5
G 30 6.5 TOP HALF
S 30 6.5
E 29 8.5
P 29 8.5
J 27 11
N 27 11
O 27 11 Lowest score of top half: 26
(26+26)/2=26
H 26 15 Highest score of bottom half: 26
K 26 15
T 26 15
X 26 15
Z 26 15
B 25 19
F 25 19
BOTTOM HALF
Q 25 19
Y 24 21
R 23 22.5
V 23 22.5
U 22 24
A 20 25
I 19 26

B.3. Mean

English Language Testing 113

Mean or average score is the sum of the scores divided by the total number of testees.
The mean is the most efficient measure of central tendency, but it is not always
appropriate.
Now, we are going to create Table 4, to count the mean. Note that symbol x is used to
denote the score, N the number of the testees, and m the mean. The symbol fdenotes the
frequency with which a score occurs. The symbol ∑ mean the sum of.
First, we gather the data from previous Table 3. We get the scores and their frequencies
from it. The score (x) is then multiplied by the frequency (f). The result is put in column
fx. After that, the total of fxis summed up as ∑fx.
TABLE 4
x . f Fx
35 x 1 35
34 x 1 34
33 x 2 66
32 x 1 32
30 x 1 60
29 x 2 58
27 x 3 81
26 x 5 130
25 x 3 75
24 x 1 24
23 x 2 46
22 x 1 22
20 x 1 20
19 x 1 19
TOTAL 702
∑fx

To get the mean, we use formula m = ∑fx / N.

English Language Testing 114

 ∑ 702
N 26
Thus, the mean is 27.

C. Measures of Dispersion
Measures of dispersion are important for describing the spread of the scores, or its
variation around a central value. There are various methods that can be used to measure
the dispersion of a dataset, but the most common ones are the range and the standard
deviation.

C.1. Range
A simple way of measuring the spread of marks is based on the difference between the
highest and the lowest scores. It is called the range. From previous Table 2, we can see
the highest score is 35 and the lowest score is 19. The range is 16.
Range =Xmax – Xmin
Range = 35 – 19 = 16

C.2. Standard Deviation

The standard deviation (s.d.) is another way of showing the spread of scores. It shows
how all the scores are spread out and gives a fuller description of test scores than the
range.
One simple method of calculating s.d. is shown below:

Σd
s. d.
N

N is the number of scores

d is the deviation of each score from the mean
From previous calculation, mean is 27. The steps to calculate s.d. are as followings:

English Language Testing 115

1. Step 1: Find out the amount by which each score deviates from the mean (d).

Mean deviation (d)

Score
(Score - 27)
35 8
34 7
33 6
33 6
32 5
30 3
30 3
29 2
29 2
27 0
27 0
27 0
26 -1
26 -1
26 -1
26 -1
26 -1
25 -2
25 -2
25 -2
24 -3
23 -4
23 -4
22 -5
20 -7
19 -8

2. Step 2: Square each result (d2)

English Language Testing 116

 Mean
Score deviation (d) d2
(Score - 27)
35 8 64
34 7 49
33 6 36
33 6 36
32 5 25
30 3 9
30 3 9
29 2 4
29 2 4
27 0 0
27 0 0
27 0 0
26 -1 1
26 -1 1
26 -1 1
26 -1 1
26 -1 1
25 -2 4
25 -2 4
25 -2 4
24 -3 9
23 -4 16
23 -4 16
22 -5 25
20 -7 49
19 -8 64

3. Step 3: Total all the results ( )

English Language Testing 117

4. Step 4: Divide the total by the number of testees( / )

/ = 432 / 26 = 16.62

5. Step 5: Find the square root of the result /

/ √ . = 4.077= 4.08

Thus, standard deviation (s.d.) is 4,08. That means that on average, the scores are about
4 points away from the average.

English Language Testing 118

 References:
Teaching Student-Centered Mathematics (K-3). John van de Walle and
LouAnnLovin, Pearson Publishing, 2006.

Classroom and Large- Scale Assessment. Wilson and Kenney. This article appeared
in A Research Companion to Principles and Standards for School Mathematics
(NCTM), 2003, (pages 53-67).

Principles and Standards for School Mathematics. National Council of Teachers of

Mathematics (NCTM), 2000.

Young Mathematicians at Work, Constructing Number Sense, Addition, and

Subtraction. By Catherine TwomeyFosnot and Maarten Dolk, Heinemann

Assessing Learners with Special Needs: 6TH ED. By Terry Overton

Weaver, B. Formal versus Informal Assessments.

http://www.scholastic.com/teachers/article/formal-versus-informal-assessments
Morrison, G. Informal Methods of Assessment.
http://www.education.com/reference/article/informal-methods-assessment/
Forlizzi, L. Informal assessment: The
Basics. http://aded.tiu11.org/disted/FamLitAdminSite/fn04assessinformal.pdf
Navarete, C., et al. Informal Assessment In Educational Evaluation.
Mind map retrieved february 20, 2013 from the URL:
http://www.mindmeister.com/122645400/formal-vs-informal-assessments

TESTING: BASIC CONCEPTS: BASIC TERMINOLOGY by

Anthony Bynom, Ph.D., December 2001

A Statistical Analysis of Different Instruments to Measure Short-term In an L2

ImmersionProgram1. by

Kyle Perkins Southern Illinois University at Carbondale U.S.A.

TESL Journal, Vol. II, No. 5, May 1996

http://iteslj.org/

English Language Testing 119

Berk, R., 1979. Generalizability of Behavioral Observations: A Clarification of
Interobserver Agreement and Interobserver Reliability. American Journal of Mental
Deficiency, Vol. 83, No. 5, p. 460-472.

Cronbach, L., 1990. Essentials of psychological testing. Harper & Row, New York.

Carmines, E., and Zeller, R., 1979. Reliability and Validity Assessment. Sage
Publications, Beverly Hills, California.

Gay, L., 1987. Eductional research: competencies for analysis and application. Merrill
Pub. Co., Columbus.

Guilford, J., 1954. Psychometric Methods. McGraw-Hill, New York.

Nunnally, J., 1978. Psychometric Theory. McGraw-Hill, New York.

Winer, B., Brown, D., and Michels, K., 1991. Statistical Principles in Experimental
Design, Third Edition. McGraw-Hill, New York.

American Educational Research Association, American Psychological Association, &

National Council on Measurement in Education. (1985). Standards for educational and

psychological testing. Washington, DC: Authors.

Cozby, P.C. (2001). Measurement Concepts. Methods in Behavioral Research (7th ed.).

California: Mayfield Publishing Company.

Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.). Educational

Measurement (2nd ed.). Washington, D. C.: American Council on Education.

Moskal, B.M., &Leydens, J.A. (2000). Scoring rubric development: Validity and

reliability. Practical Assessment, Research & Evaluation, 7(10). [Available online:

http://pareonline.net/getvn.asp?v=7&n=10].

The Center for the Enhancement of Teaching. How to improve test reliability and

validity: Implications for grading. [Available online:

http://oct.sfsu.edu/assessment/evaluating/htmls/improve_rel_val.html].

http://spiritize.blogspot.com/2007/10/active-listening.html

http://wiki.answers.com/Q/Examples_of_poetry#ixzz1xYYnZXmU

English Language Testing 120

Alderson, J. C 2002 Conceptions of validity and validation. Paper presented at a
conference in Bucharest, June 2002.

Angoff, 1988 Validity: An evolving concept. In H. Wainer& H. Braun [Eds.] Test

validity [pp. 19-32], Hillsdale, NJ: Erlbaum.

Bachman, L. F. 1990 Fundamental considerations in language testing. Oxford: O.U.P.

Cumming A. & Berwick R. [Eds.] Validation in Language Testing Multilingual

Matters 1996

Hatch, E. &Lazaraton, A. 1991 The Research Manual - Design & Statistics for Applied
Linguistics Newbury House

Henning, G. 1987 A guide to language testing: Development, evaluation and research

Cambridge, Mass: Newbury House

Hubley, A. M. &Zumbo, B. D. A dialectic on validity: where we have been and where

we are going. The Journal of General Psychology 1996. 123[3] 207-215

Messick, S. 1988 The once and future issues of validity: Assessing the meaning and
consequences of measurement. In H. Wainer& H. Braun [Eds.] Test validity [pp. 33-
45], Hillsdale, NJ: Erlbaum.

Messick, S. 1989 Validity. In R. L. Linn [Ed.] Educational measurement. [3rd ed., pp

13-103]. New York: Macmillan

English Language Testing 121